"""
**Created by Berkay Alan**

**Principal Components Analysis(PCA)**

**13 October 2021**

**For more Tutorial:** https://github.com/berkayalan
"""
"""
## Content

"""
"""
**Principal Components Analysis(PCA)**

***

- Principal Components Analysis(PCA) (Theory - Manual Implementation of PCA - Model)
"""
"""
## Resources
"""
"""
- **The Elements of  Statistical Learning** - Trevor Hastie,  Robert Tibshirani, Jerome Friedman -  Data Mining, Inference, and Prediction (Springer Series in Statistics) 

- **An Introduction to Statistical Learning** - Trevor Hastie,  Robert Tibshirani, Daniela Witten, Gareth James

- [**Understanding Principal Component Analysis**](https://towardsdatascience.com/understanding-principal-component-analysis-ddaf350a363a)

- [**Principal Component Analysis for Dimensionality Reduction in Python**](https://machinelearningmastery.com/principal-components-analysis-for-dimensionality-reduction-in-python/)

- [**Principal Component Analysis explained visually**](https://setosa.io/ev/principal-component-analysis/)
"""
"""
## Importing Libraries
"""
from warnings import filterwarnings
filterwarnings("ignore")
pip install skompiler
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mping
import seaborn as sns
from scipy.cluster.hierarchy import dendrogram
from scipy.cluster import hierarchy
import statsmodels.api as sm
import statsmodels.formula.api as smf
from sklearn.preprocessing import StandardScaler,MinMaxScaler
from sklearn import preprocessing
from sklearn.decomposition import PCA
from sklearn.datasets import make_blobs,make_moons
from yellowbrick.cluster import KElbowVisualizer
import time
from matplotlib.colors import ListedColormap
from skompiler import skompile
from joblib import dump, load
"""
In order to see all rows and columns, we will increase max display numbers of dataframe.
"""
pd.set_option('display.max_rows', 1000)
pd.set_option('display.max_columns', 1000)
pd.set_option('display.width', 1000)
"""
## Principal Components Analysis(PCA) (Theory - Model)
"""
"""
### Theory
"""
"""
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.

 When faced with a large set of correlated variables, principal components allow us to summarize this set with a smaller number of representative variables that collectively explain most of the variability in the original set.
 
 Principal Components Analysis(PCA)refers to the process by which principal components are computed, and the subsequent use of these components in understanding the data. PCA is an unsupervised approach, since it involves only a set of features X1, X2, . . . , Xp, and no associated response Y .
 
 Suppose that we wish to visualize n observations with measurements on a set of p features, X1, X2, . . . , Xp, as part of an exploratory data analysis. We could do this by examining two-dimensional scatterplots of the data, each of which contains the n observations’ measurements on two of the features. If p is large, then it will certainly not be possible to look at all of them; moreover, most likely none of them will be informative since they each contain just a small fraction of the total information present in the data set. In particular, we would like to find a low-dimensional representation of the data that captures as much of the information as possible. For instance, if we can obtain a two-dimensional representation of the data that captures most of the information, then we can plot the observations in this low-dimensional space.
 
***

There are two important outcames of PCA:

- It reduces number of dimensions in data.

- It shows which features explain the most variance in the data.
"""
"""
 PCA seeks a small number of dimensions that are as interesting as possible, where the concept of interesting is measured by the amount that the observations vary along each dimension. Each of the dimensions found by PCA is a linear combination of the p features.
"""
"""
The first principal component of a set of features X1, X2, . . . , Xp is the normalized linear combination of the features

![Screen%20Shot%202021-10-12%20at%2015.08.12.png](attachment:Screen%20Shot%202021-10-12%20at%2015.08.12.png)
"""
"""
that has the largest variance.

 Since we are only interested in variance, we assume that each of the variables in X has been centered to have mean zero (that is, the col- umn means of X are zero). We then look for the linear combination of the sample feature values of the form

![Screen%20Shot%202021-10-12%20at%2021.38.40.png](attachment:Screen%20Shot%202021-10-12%20at%2021.38.40.png)

that has largest sample variance.

***

**Steps of PCA**

1. After Getting original data, calculate covariance matrix.

2. Calculate EigenVectors.

3. Sort EigenVectors by Eigenvalues.

4. Choose N largest EigenValues.

5. Project Original data onto EigenVectors.
"""
"""
![image.png](attachment:image.png)

<div class="alert alert-block alert-info">
<b>Reference 6:</b>  Image is taken from *Dataiku Knowledge Base* website.
</div>
"""
"""
### Manual Implementation of PCA
"""
"""
For a real world example, we will use *Breast Cancer Wisconsin* dataset. Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image. 

It can be downloaded from [here](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)).

We will understand the dataset first.
"""
df = pd.read_csv("../input/breast-cancer-wisconsin-data/data.csv")
df.head()
df.drop("Unnamed: 32",axis=1,inplace=True)
df.drop("diagnosis",axis=1,inplace=True)
df.shape
"""
First we need to standardize the data.
"""
scaler = StandardScaler()
scaled_X = scaler.fit_transform(df)
scaled_X
"""
As of step 1, we will calculate covariance matrix.
"""
covariance_matrix = np.cov(scaled_X, rowvar=False)
"""
Now we will get Eigen Vectors and Eigen Values.
"""
eigen_values, eigen_vectors = np.linalg.eig(covariance_matrix)
"""
Now we will sort EigenVectors by Eigenvalues.
"""
sorted_key = np.argsort(eigen_values)[::-1][:2]
"""
Let's get num_components of Eigen Values and Eigen Vectors.
"""
eigen_values, eigen_vectors = eigen_values[sorted_key], eigen_vectors[:, sorted_key]
"""
 Dot product of original data and eigen_vectors are the principal component values. This is the "projection" step of the original points on to the Principal Component.
"""
principal_components=np.dot(scaled_X,eigen_vectors)
principal_components
plt.figure(figsize=(8,6))
plt.scatter(principal_components[:,0],principal_components[:,1])
plt.xlabel('First principal component')
plt.ylabel('Second Principal Component')
plt.grid(False)
plt.show()
"""
### Model
"""
"""
For a real world example, we will use *Breast Cancer Wisconsin* dataset. Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image. 

It can be downloaded from [here](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)).

We will understand the dataset first.
"""
df = pd.read_csv("../input/breast-cancer-wisconsin-data/data.csv")
df.drop("Unnamed: 32",axis=1,inplace=True)
df.drop("diagnosis",axis=1,inplace=True)
df.shape
"""
First we need to standardize the data.
"""
scaler = StandardScaler()
scaled_X = scaler.fit_transform(df)
scaled_X
"""
Let's execute the algorithm.
"""
principal_components = PCA(n_components=2)
fitted_principal_components = principal_components.fit_transform(scaled_X)
component_df = pd.DataFrame(data=fitted_principal_components,columns=["First Component","Second Component"])
component_df.head(10)
"""
Let's look how much these components explains our data.
"""
principal_components.explained_variance_ratio_
principal_components = PCA(n_components=5)
fitted_principal_components = principal_components.fit_transform(scaled_X)
component_df = pd.DataFrame(data=fitted_principal_components,columns=["First Component","Second Component",
                                                                     "Third Component","Fourth Component","Fifth Component"
                                                                     ])
component_df.head(10)
"""
Let's look how much these components explains our data.
"""
principal_components.explained_variance_ratio_
np.sum(principal_components.explained_variance_ratio_)
"""
Let's try to find optimal component number.
"""
principal_components = PCA().fit(scaled_X)
plt.plot(np.cumsum(principal_components.explained_variance_ratio_))
plt.grid(False)
plt.show()
"""
If you want to learn more about Unsupervised Learning - Clustering - Principal Components Analysis(PCA), you can check my [Github Notebook.](https://github.com/berkayalan/Data-Science-Tutorials/tree/master/Unsupervised%20Learning%20-%20Clustering%20-%20Principal%20Components%20Analysis(PCA))
"""