"""
**Note:**  These exercise series are from coursera machine learning course. Please keep Prof. Andrew Ng's each video and lecture note side by side for better understanding about this kernel. 
"""
"""
**1 Logistic Regression**

In this part of the exercise, we will build a logistic regression model to predict whether a student gets admitted into a university.
Suppose that you are the administrator of a university department and you want to determine each applicant’s chance of admission based on their
results on two exams. 

We have historical data from previous applicants that we can use as a training set for logistic regression. For each training
example, we have the applicant’s scores on two exams and the admissions decision.

Our task is to build a classification model that estimates an applicant’s
probability of admission based the scores from those two exams. 

"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

import os
print(os.listdir("../input"))

# Any results you write to the current directory are saved as output.
data = pd.read_csv('../input/ex2data1.txt')
data.head(5)
X = np.array(data.iloc[:, 0:2]) #for plotting data we need array instead of dataframe slice
y = np.array(data.iloc[:, 2]) #for plotting data we need array instead of series slice
#type(y)
"""
**1.1 Visualizing the data**

Before starting to implement any learning algorithm, it is always good to visualize the data if possible.

"""
def PlotData(X,y):
    """
    Plots the data points X and y.
    n_samples = 99 can varify from (X.shape, y.shape)
    """
    pos = np.argwhere(y == 1)#it will return np.ndarray instead of tuple if we use where.
    neg = np.argwhere(y == 0)#it will return np.ndarray instead of tuple if we use where.
  
    plt.plot(X[pos, 0], X[pos, 1], linestyle='', marker='+', color='k')
    plt.plot(X[neg, 0], X[neg, 1], linestyle='', marker='o', color='y')
plt.figure()
PlotData(X, y)
# Labels and Legend
plt.xlabel('Exam 1 score')
plt.ylabel('Exam 2 score')
plt.legend(['Admitted', 'Not admitted'], loc='upper right', numpoints=1)
plt.show()
"""
**1.2 Implementation**

**1.2.1 Warmup exercise: sigmoid function**

Before we start with the actual cost function, recall that the logistic regression
hypothesis is defined as: $$ h_\theta(x) = g(\theta^T) $$,

where function $g$ is the sigmoid function. The sigmoid function is defined as: $$g(z) = \frac{1}{1+e^{-z}}$$ 

**Note:** Your first step is to implement this function sigmoid so it can be called by the rest of your program. When you are finished, try testing a few values by calling sigmoid(x) in a new cell. For large positive values of x, the sigmoid should be close to 1, while for large negative values, the sigmoid should be close to 0. Evaluating sigmoid(0) should give you exactly 0.5. Your code should also work with vectors and matrices. For a matrix, your function should perform the sigmoid function on every element.
"""
#Initialze training parameters
m, n =  X.shape

# Add intercept term
X = np.hstack((np.ones((m, 1)), X))

# Initialize fitting parameters
theta = np.zeros(n + 1)  
#Compute sigmoid function.
def sigmoid(z):
    g = 1 / (1 + np.exp(-z))
    return g   

#evaluate sigmoid(0), it should give you 0.5
#sigmoid(0)
def h(theta,X):
    return sigmoid(np.dot(X,theta))
"""
**1.2.2 Cost function and gradient**

Let’s implement the cost function for the Logistic Regression.

the cost function in logistic regression is

$$ J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \left[ -y^{(i)} \log\left(h_\theta\left( x^{(i)} \right) \right) - \left( 1 - y^{(i)}\right) \log \left( 1 - h_\theta\left( x^{(i)} \right) \right) \right]$$ 

and the gradient of the cost is a vector of the same length as $\theta$ where the $j^{th}$ element (for $j = 0, 1, \cdots , n$) is defined as follows:
$$ \frac{\partial J(\theta)}{\partial \theta_j} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left( x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} $$

Note that while this gradient looks identical to the linear regression gradient, the formula is actually different because linear and logistic regression have different definitions of $h_\theta(x)$



"""
def costFunction(X,y,theta):
    J = (1/m) * np.sum((-y * np.log(h(theta,X))) - ((1-y)*np.log(1-h(theta,X))))
    
    #grad = (1/m) * ((h(theta,X)) - y)*X.T
    grad = (1/m) * (h(theta,X) - y).T.dot(X)
    
    return (J, grad)
cost, grad = costFunction(X,y,theta)
print('cost for intial theta value', cost)
print('gradient for initial theta value ', grad)
print(data.shape, X.shape, y.shape, theta.shape)
y = y.reshape(m,1)
theta = np.zeros((n+1,1))

print(data.shape, X.shape, y.shape, theta.shape)
import scipy.optimize as opt

theta, nfeval, rc = opt.fmin_tnc(func=costFunction, x0=theta, args=(X, y))

cost, grad = costFunction(X,y,theta)
print('cost for intial theta value', cost)
print('gradient for initial theta value ', grad)
"""
This kernel is not finished yet. Thanks for stopping by.
"""