# Content
* Libraries & Contants
* Introduction to Problem
    * Loading Dataset
    * Visualizing Raw Dataset
    * Preprocessing
    * Visualizing Proprocessed Dataset
* Logistic Regression
    * Forward Propagation
    * Backward Propagation
    * Complete Propagation
    * Combining All Together
* Training
* Prediction
* Evaluation

# Libraries & Contants


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sb
import pandas as pd
from sklearn import datasets
from utils import plot_prediction
```

# Introduction to Problem
Given a set of inputs X as features of flowers, we want to assign them to one of two possible categories 0 or 1 that means what type of flower they are.
For solving this problem we use logistic regression because it models the probability that each input belongs to a particular category.

## Loading Dataset


```python
iris = pd.read_csv('./data/Iris.csv')
iris.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Id</th>
      <th>SepalLengthCm</th>
      <th>SepalWidthCm</th>
      <th>PetalLengthCm</th>
      <th>PetalWidthCm</th>
      <th>Species</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>5.1</td>
      <td>3.5</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>4.9</td>
      <td>3.0</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>2</th>
      <td>3</td>
      <td>4.7</td>
      <td>3.2</td>
      <td>1.3</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4</td>
      <td>4.6</td>
      <td>3.1</td>
      <td>1.5</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
    <tr>
      <th>4</th>
      <td>5</td>
      <td>5.0</td>
      <td>3.6</td>
      <td>1.4</td>
      <td>0.2</td>
      <td>Iris-setosa</td>
    </tr>
  </tbody>
</table>
</div>



## Visualizing Dataset


```python
sepalPlt = sb.FacetGrid(iris, hue="Species", size=6).map(plt.scatter, "SepalLengthCm", "SepalWidthCm")
plt.legend(loc='upper left');
```

# Preprocessing

In this step we will simplify the problem into a binary classification problem with just 2 features.


```python
X = iris.iloc[:, 1:3]
Y = (iris['Species'] != 'Iris-setosa') * 1
```


```python
plt.figure(figsize=(10, 6))
plt.scatter(X[Y == 0].iloc[:, 0], X[Y == 0].iloc[:, 1], color='b', label='Iris-setosa')
plt.scatter(X[Y == 1].iloc[:, 0], X[Y == 1].iloc[:, 1], color='r', label='Others')
plt.xlabel('SepalLengthCm')
plt.ylabel('SepalWidthCm')
plt.legend()
```

# Logistic Regression
  <a href="https://towardsdatascience.com/building-a-logistic-regression-in-python-step-by-step-becd4d56c9c8">Logistic Regression</a> is a Machine Learning classification algorithm that is used to predict the probability of a categorical dependent variable. In logistic regression, the dependent variable is a binary variable that contains data coded as 1 (yes, success, etc.) or 0 (no, failure, etc.). In other words, the logistic regression model predicts P(Y=1) as a function of X.


## Forward Propagation

First we will implement linear multiplication:

\begin{align}
Z = W^TX + b
\end{align}


```python
def linear_mult(X, w, b):
    return np.dot(w.T, X) + b
```

Now we implement a function could generate W and b for us:


```python
def initialize_with_zeros(dim):
    """
    This function creates a vector of zeros of shape (dim, 1) for w and initializes b to 0.
    
    Argument:
    dim -- size of the w vector we want (or number of parameters in this case)
    
    Returns:
    w -- initialized vector of shape (dim, 1)
    b -- initialized scalar (corresponds to the bias)
    """
    
    w = np.zeros((dim,1))
    b = 0

    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))
    
    return w, b
```

Next we will implement sigmoid function to map calculated value to a probablity:

\begin{align}
\sigma(Z) = \frac{1}{1 + e^{-Z}}
\end{align}


```python
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
```

Now we implement the cost function, cost function represent the difference between our predictions and actual labels(y is the actual label and a is our predicted label):

\begin{align}
J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})
\end{align}


```python
def cost_function(y, a):
    return -np.mean(y*np.log(a) + (1-y)*np.log(1-a))
```

Now we implement the whole forward propagation which will calculate cost and the predicted value for the each data point:


```python
def forward_propagate(w, b, X, Y):
    
    m = X.shape[1]
    
    Z = linear_mult(X, w, b)
    A = sigmoid(Z)                                 
    cost = cost_function(Y, A)  
    cost = np.squeeze(cost)

    assert(cost.shape == ())
    
    back_require = {
        'A': A
    }
    
    return back_require, cost
```

## Backward Propagation

Now we calculate W and b derivative as follow:

$$ \frac{\partial J}{\partial w} = \frac{1}{m}X(A-Y)^T\tag{7}$$
$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_{i=1}^m (a^{(i)}-y^{(i)})\tag{8}$$


```python
def backward_propagate(w, b, X, Y, back_require):
    
    m = X.shape[1]
    
    A = back_require['A']
    
    dw = (1/m) * np.dot(X,(A-Y).T)
    db = (1/m) * np.sum(A - Y)
    
    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    
    grads = {"dw": dw,
             "db": db}
    
    return grads
```

# Complete Propagation


```python
def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b
    
    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """
    
    # FORWARD PROPAGATION
    back_require, cost = forward_propagate(w, b, X, Y)
    
    # BACKWARD PROPAGATION
    grads = backward_propagate(w, b, X, Y, back_require)

    
    return grads, cost
```

# Combining All Together

Now we combine all our implemented functions together to create an optimizer which can find a linear function to devide the zero labeled data from one labeled data points by optimizng W and b as follow:
$$W=W−\alpha{dw}$$
$$b=b−\alpha{db}$$
$\alpha$ is the learning rate


```python
# GRADED FUNCTION: optimize

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm
    
    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps
    
    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    
    Tips:
    You basically need to write down two steps and iterate through them:
        1) Calculate the cost and the gradient for the current parameters. Use propagate().
        2) Update the parameters using gradient descent rule for w and b.
    """
    
    costs = []
    
    for i in range(num_iterations):
        
        
        grads, cost = propagate(w,b,X,Y)
        
        dw = grads["dw"]
        db = grads["db"]
        

        w -= learning_rate*dw
        b -= learning_rate*db
        
        # Record the costs
        if i % 100 == 0:
            costs.append(cost)
        
        # Print the cost every 100 training iterations
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))
    
    params = {"w": w,
              "b": b}
    
    grads = {"dw": dw,
             "db": db}
    
    return params, grads, costs
```

# Training


```python
X_t, Y_t = np.array(X.T), np.array(Y.T)
w, b = initialize_with_zeros(2)
params, grads, costs = optimize(w, b, X_t, Y_t, num_iterations= 5000, learning_rate = 0.009, print_cost = False)
```


```python
plt.plot(range(len(costs)),costs)
plt.xlabel('Iterations')
plt.ylabel('Cost(loss) value')
```

# Prediction


```python
def predict(w, b, X):
    '''
    Predict whether the label is 0 or 1 using learned logistic regression parameters (w, b)
    
    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    
    Returns:
    Y_prediction -- a numpy array (vector) containing all predictions (0/1) for the examples in X
    '''
    
    m = X.shape[1]
    Y_prediction = np.zeros((1,m))
    
    Z = linear_mult(X, w, b)
    A = sigmoid(Z)
    
    for i in range(m):
        
        Y_prediction[0][i] = 1 if A[0][i] > .5 else 0
    
    assert(Y_prediction.shape == (1, m))
    
    return Y_prediction
```

# Evaluation


```python
preds = predict(params['w'], params['b'], X_t)
print('Accuracy on training set: %{}'.format((preds[0] == Y).mean()*100))
```

    Accuracy on training set: %99.33333333333333



```python
plot_prediction(X, Y, predict, params)
```

f85f87b75e12ee244abf47bb6c0017004823b1fc
ipynb
Jupyter Notebook
Week 7 - Logistic Regression/Logistic Regression.ipynb
Computational-Intelligence-Fall18/Python-Hands-on
18018dd5258ec0563513076c8e46bba8113d65b4
Apache-2.0
2018-11-07T14:30:51.000Z
2022-02-06T16:35:20.000Z
Week 7 - Logistic Regression/Logistic Regression.ipynb
Computational-Intelligence-Fall18/Python-Numpy-Instructor-Codes
18018dd5258ec0563513076c8e46bba8113d65b4
Apache-2.0
2018-10-25T22:04:23.000Z
2020-10-04T11:27:44.000Z
Week 7 - Logistic Regression/Logistic Regression.ipynb
Computational-Intelligence-Fall18/Python-Numpy-Instructor-Codes
18018dd5258ec0563513076c8e46bba8113d65b4
Apache-2.0
2018-11-28T10:16:37.000Z
2022-02-24T13:09:17.000Z
Qwen/Qwen-72B
1. YES
2. YES