"""
## **To put it simply**, 
## **Dimensionality** is the number of dimensions, features, or variables associated with a sample of data. Often, this can be thought of as a number of columns in a dataframe,where each sample is on a new row, and each column describes some attribute of the sample.

"""
"""
![table](https://media.geeksforgeeks.org/wp-content/uploads/20200710135837/Screenshot-9461.png)
"""
"""
## In this figure, we have eight samples of data, each with **four independent features or** **dimensions**. Depending upon the problem being solved, or the origin of this dataset, we may want to reduce the number of dimensions per sample without losing the provided information. This is where dimensionality reduction can be helpful
"""
"""
# **Dimensionality reduction** helps counteract 
# —the curse of dimensionality—in which algorithms cannot effectively and efficiently train on the data because of the sheer size of the feature space

## **Dimensionality reduction algorithms project high-dimensional data to a low dimensional space, retaining as much of the salient information as possible while removing redundant information**

### **Once the data is in the low-dimensional space, machine learning algorithms are able to identify interesting patterns more effectively and efficiently because a lot of the noise has been reduced**



There are various branches of dimensionality reduction. 

# **Linear projection**: 

## Linearly projecting data from a high dimensional space to a low-dimensional space

Techniques include:
* ## Principal component analysis
 * Incremental PCA
 * Sparse PCA 

* ## Singular value decomposition 
* ## Random projection

 * Gaussian random projection
 * sparse random projection

# **Manifold learning**:

## It is referred to as nonlinear dimensionality reduction. This involves techniques such as 

Techniques include:

* ## Kernel PCA 
* ## Isomap
* ## Multidimensional scaling (MDS)
* ## Locally linear embedding (LLE)
* ## t-distributed stochastic neighbor embedding (t-SNE)

# **Other Dimensionality Reduction Methods**:

## Do not rely on any sort of geometry or distance metric.

Techniques include:

* ## Dictionary learning
* ## Independent component analysis.

"""
#Import libraries
import numpy as np
import pandas as pd
import os,time
import pickle,gzip
#Data visualization
import matplotlib.pyplot as plt
import seaborn as sns
color=sns.color_palette()
import matplotlib as mpl
%matplotlib inline
#Data prep and Model Evaluation
from sklearn import preprocessing as pp
from scipy.stats import pearsonr
from numpy.testing import assert_array_almost_equal
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import log_loss
from sklearn.metrics import precision_recall_curve,average_precision_score
from sklearn.metrics import roc_curve,auc,roc_auc_score
from sklearn.metrics import confusion_matrix,classification_report
#Algorithms
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
import xgboost as xgb
import lightgbm as lgb
"""
## We will be using the MNIST dataset.

**Regrading the dataset:**

* Divided into three sets—a training set with 50,000 examples, a validation set with 10,000 examples, and a test set with 10,000 examples. We have labels for all the examples. 
* This dataset consists of 28x28 pixel images of handwritten digits. To make this simpler, we can flatten each array into a 28x28, or 784, dimensional vector. 
* Each component of the vector is a float between zero and one—representing the intensity of each pixel in the image. Zero stands for black; one stands for white. The labels are numbers between zero and nine, and indicate which digit the image represents.

"""
import pandas as pd

train_set, validation_set, test_set = pd.read_pickle(r'../input/d/towhidultonmoy/mnist-pickle-dataset/mnist.pkl')
X_train,y_train=train_set[0],train_set[1]
X_validation,y_validation=validation_set[0],validation_set[1]
X_test,y_test=test_set[0],test_set[1]
print("Shape of X_train: ",X_train.shape)
print("Shape of y_train: ",y_train.shape)
print("Shape of X_validation: ", X_validation.shape)
print("Shape of y_validation: ", y_validation.shape)
print("Shape of X_test: ", X_test.shape)
print("Shape of y_test: ", y_test.shape)
#Creat Pandas DataFrames from the datasets
train_index=range(0,len(X_train))
validation_index=range(len(X_train),len(X_train)+len(X_validation))
test_index=range(len(X_train)+len(X_validation),len(X_train)+len(X_validation)+len(X_test))
X_train=pd.DataFrame(data=X_train,index=train_index)
y_train=pd.Series(data=y_train,index=train_index)

X_validation=pd.DataFrame(data=X_validation,index=validation_index)
y_validation=pd.Series(data=y_validation,index=validation_index)

X_test=pd.DataFrame(data=X_test,index=test_index)
y_test=pd.Series(data=y_test,index=test_index)
X_train
#Exploring the data
X_train.describe()
y_train.head()
#Displaying the image
def view_digit(example):
    label = y_train.loc[example]
    image = X_train.loc[example,:].values.reshape([28,28])
    plt.title('Example: %d Label: %d' % (example, label))
    plt.imshow(image, cmap=plt.get_cmap('gray'))
    plt.show()

view_digit(2)
"""
## **PCA**
"""
"""
### In PCA, the algorithm finds a **low-dimensional representation of the data while retaining as much of the variation (information) as possible**. 

* PCA does this by addressing the correlation among features. 
* If the correlation is very high among a subset of the features, PCA will attempt to combine the highly correlated features and represent this data with a smaller number of linearly uncorrelated features. 
* The algorithm keeps performing this correlation eduction, finding the directions of maximum variance in the original high dimensional data and projecting them onto a smaller dimensional space.  **These newly derived components are known as principal components.**
"""
"""
### It is essential to **perform feature scaling before running PCA**. PCA is very sensitive to the relative ranges of the original features.

#### However, for our MNIST digits dataset, the features are already scaled to a range of zero to one, so we can skip this step.
"""
#Applying PCA
from sklearn.decomposition import PCA

n_components=784
whiten= False
random_state=2018

pca=PCA(n_components=n_components,whiten=whiten,random_state=random_state)
X_train_PCA=pca.fit_transform(X_train)
X_train_PCA=pd.DataFrame(data=X_train_PCA,index=train_index)
X_train_PCA
#Percenatage of Variance Captured by 784 principal components
print("Variance Explained by all 784 principle components: ",sum(pca.explained_variance_ratio_))
#Percenatage of Variance Captured by X principal components
importanceOfPrincipalComponents= pd.DataFrame(data=pca.explained_variance_ratio_)
importanceOfPrincipalComponents=importanceOfPrincipalComponents.T
importanceOfPrincipalComponents
print("Variance Captured by First 10 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:9].sum(axis=1).values)
print("Variance Captured by First 20 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:19].sum(axis=1).values)
print("Variance Captured by First 50 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:49].sum(axis=1).values)
print("Variance Captured by First 100 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:99].sum(axis=1).values)
print("Variance Captured by First 200 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:199].sum(axis=1).values)
print("Variance Captured by First 300 Principal Components: ",importanceOfPrincipalComponents.loc[:,0:299].sum(axis=1).values)
# View percentage captured by first X components
sns.set(rc={'figure.figsize':(10,10)})
sns.barplot(data=importanceOfPrincipalComponents.loc[:,0:10],color='k')
X_train_PCA
def scatterPlot(xDF,yDF,algoName):
    tempDF=pd.DataFrame(data=xDF.loc[:,0:1],index=xDF.index)
    tempDF=pd.concat((tempDF,yDF),axis=1,join="inner")
    tempDF.columns=["First Vector","Second Vector","Label"]
    sns.lmplot(x="First Vector",y="Second Vector",hue="Label",data=tempDF,fit_reg=False)
    ax=plt.gca()
    ax.set_title("Separation of Observations using " +algoName)
scatterPlot(X_train_PCA,y_train,"PCA")
# View two random dimensions
X_train_scatter = pd.DataFrame(data=X_train.loc[:,[350,406]], index=X_train.index)
X_train_scatter = pd.concat((X_train_scatter,y_train), axis=1, join="inner")
X_train_scatter.columns = ["First Vector", "Second Vector", "Label"]
sns.lmplot(x="First Vector", y="Second Vector", hue="Label", data=X_train_scatter, fit_reg=False)
ax = plt.gca()
ax.set_title("Separation of Observations Using Original Feature Set")
"""
# **Incremental PCA**
"""
"""
### For **datasets that are very large and cannot fit in memory** , we can perform **PCA incrementally in small batches** , where each batch is able to fit in memory. The batch size can be either set manually or determined automatically. This batchbased form of PCA is known as incremental PCA. The resulting principal components of PCA and incremental PCA are generally pretty similar
"""
from sklearn.decomposition import IncrementalPCA

n_components=784
batch_size=None

incrementalPCA=IncrementalPCA(n_components=n_components,batch_size=batch_size)

X_train_incrementalPCA=incrementalPCA.fit_transform(X_train)
X_train_incrementalPCA=pd.DataFrame(data=X_train_incrementalPCA,index=train_index)

X_validation_incrementalPCA=incrementalPCA.fit_transform(X_validation)
X_validation_incrementalPCA=pd.DataFrame(data=X_validation_incrementalPCA,index=validation_index)

scatterPlot(X_train_incrementalPCA,y_train,"Incremental PCA")
"""
# **Sparse PCA**
"""
"""
# First, clarify what is **sparsity**,

## A matrix in which most entries are **0** is called a **sparse matrix**. These matrices can be stored more efficiently and certain computations can be carried out more efficiently on them provided the matrix is sufficiently large and sparse. Neural networks can leverage the efficiency gained from sparsity by assuming most connection weights are equal to 0.

## **The goal is to reduce the mounds of matrix multiplication deep learning requires**, shortening the time to good results. 


### The normal PCA algorithm 
* searches for linear combinations in all the input variables, reducing the original feature space as densely as possible. 
* But for some machine learning problems, some degree of **sparsity** may be preferred. 
* A version of PCA that retains some degree of sparsity—controlled by a hyperparameter called alpha—is known as sparse PCA. 
* The sparse PCA algorithm searches for linear combinations in just some of the input variables, reducing the original feature space to some degree but not as compactly as normal PCA.
"""
from sklearn.decomposition import SparsePCA
n_components=100
alpha=0.0001
random_state=2018
n_jobs=-1

sparsePCA=SparsePCA(n_components=n_components,alpha=alpha,random_state=random_state,n_jobs=n_jobs)

sparsePCA.fit(X_train.loc[:10000,:])  #Because this algorithm trains a bit more slowly than normal PCA, we will train on just the first 10,000 examples in our training set
X_train_sparsePCA = sparsePCA.transform(X_train)
X_train_sparsePCA = pd.DataFrame(data=X_train_sparsePCA, index=train_index)
X_validation_sparsePCA = sparsePCA.transform(X_validation)
X_validation_sparsePCA = \
pd.DataFrame(data=X_validation_sparsePCA, index=validation_index)
scatterPlot(X_train_sparsePCA, y_train, "Sparse PCA")

"""
# **Kernel PCA**
"""
"""
### **Normal PCA, incremental PCA, and sparse PCA linearly project the original data onto a lower dimensional space**, but there is also a **nonlinear form of PCA** known as kernel PCA, which runs a similarity function over pairs of original data points in order to perform nonlinear dimensionality reduction.

### For the kernel PCA algorithm, we need to set the number of components we desire, the type of kernel, and the kernel coefficient, which is known as the **gamma**

* Kernel is used as a measure of similarity. In particular, the kernel function k(x,.) defines the distribution of similarities of points around a given point x. k(x,y) denotes the similarity of point x with another given point y.

* In machine learning "kernel" is usually used to refer to the kernel trick, a method of using a linear classifier to solve a non-linear problem "by mapping the original non-linear observations into a higher-dimensional space".

"""
from sklearn.decomposition import KernelPCA
n_components=100
kernel="rbf"
gamma=None
random_state=2018
n_jobs=1

kernelPCA=KernelPCA(n_components=n_components, kernel=kernel, \
gamma=gamma, n_jobs=n_jobs, random_state=random_state)
kernelPCA.fit(X_train.loc[:10000,:])  #Because this algorithm trains a bit more slowly than normal PCA, we will train on just the first 10,000 examples in our training set
X_train_sparsePCA = sparsePCA.transform(X_train)
X_train_kernelPCA = kernelPCA.transform(X_train)
X_train_kernelPCA = pd.DataFrame(data=X_train_kernelPCA,index=train_index)
X_validation_kernelPCA = kernelPCA.transform(X_validation)
X_validation_kernelPCA = \
pd.DataFrame(data=X_validation_kernelPCA, index=validation_index)
scatterPlot(X_train_kernelPCA, y_train, "Kernel PCA")
"""
# **Singular Value Decomposition**
"""
"""
### Another approach to learning the underlying structure of the data is to **reduce the rank of the original matrix** of features to a smaller rank such that the original matrix can be recreated using a linear combination of some of the vectors in the smaller rank matrix. This is known as singular value decomposition (SVD).

### **The maximum number of its linearly independent columns (or rows ) of a matrix is called the rank of a matrix. The rank of a matrix cannot exceed the number of its rows or columns**

"""
from sklearn.decomposition import TruncatedSVD
n_components=200
algorithm="randomized"
n_iter=5
random_state=2018

svd = TruncatedSVD(n_components=n_components, algorithm=algorithm, \
n_iter=n_iter, random_state=random_state)
X_train_svd = svd.fit_transform(X_train)
X_train_svd = pd.DataFrame(data=X_train_svd, index=train_index)
X_validation_svd = svd.transform(X_validation)
X_validation_svd = pd.DataFrame(data=X_validation_svd, index=validation_index)
scatterPlot(X_train_svd, y_train, "Singular Value Decomposition")
"""
# **Random projection**
"""
"""
### Another linear dimensionality reduction technique is random projection, which relies on the **Johnson–Lindenstrauss lemma**. According to the **Johnson–Lindenstrauss lemma**, points in a high-dimensional space can be embedded into a much lower-dimensional space so that distances between the points are nearly preserved. 
"""
"""
## **Gaussian Random Projection**
"""
"""
### For Gaussian random projection, we can either specify the number of components we would like to have in the reduced feature space, or we can set the hyperparameter **eps**. The **eps** controls the quality of the embedding according to the Johnson–Lindenstrauss lemma, where smaller values generate a higher number of dimensions. 
"""
#Gaussian Random Projection
from sklearn.random_projection import GaussianRandomProjection

n_components="auto"
eps=0.5
random_state=2018

GRP= GaussianRandomProjection(n_components=n_components, eps=eps, \
random_state=random_state)

X_train_GRP = GRP.fit_transform(X_train)
X_train_GRP = pd.DataFrame(data=X_train_GRP, index=train_index)
X_validation_GRP = GRP.transform(X_validation)
X_validation_GRP = pd.DataFrame(data=X_validation_GRP, index=validation_index)
scatterPlot(X_train_GRP, y_train, "Gaussian Random Projection")
"""
## **Sparse Random Projection**

"""
"""
### Just as there is a sparse version of PCA, there is a sparse version of random projection known as sparse random projection
"""
from sklearn.random_projection import SparseRandomProjection
n_components = 'auto'
density = 'auto'
eps = 0.5
dense_output = False
random_state = 2018
SRP = SparseRandomProjection(n_components=n_components, \
density=density, eps=eps, dense_output=dense_output, \
random_state=random_state)

X_train_SRP = SRP.fit_transform(X_train)
X_train_SRP = pd.DataFrame(data=X_train_SRP, index=train_index)
X_validation_SRP = SRP.transform(X_validation)
X_validation_SRP = pd.DataFrame(data=X_validation_SRP, index=validation_index)
scatterPlot(X_train_SRP, y_train, "Sparse Random Projection")
"""
# **Isomap**
"""
"""
### It learns a new, low-dimensional embedding of the original feature set by calculating the pairwise distances of all the points, where distance is curved or geodesic distance rather than Euclidean distance. 
"""
# Isomap
from sklearn.manifold import Isomap
n_neighbors = 5
n_components = 10
n_jobs = 4
isomap = Isomap(n_neighbors=n_neighbors, \
n_components=n_components, n_jobs=n_jobs)
isomap.fit(X_train.loc[0:5000,:])
X_train_isomap = isomap.transform(X_train)
X_train_isomap = pd.DataFrame(data=X_train_isomap, index=train_index)
X_validation_isomap = isomap.transform(X_validation)
X_validation_isomap = pd.DataFrame(data=X_validation_isomap, \
index=validation_index)
scatterPlot(X_train_isomap, y_train, "Isomap")
"""
# **Multidimensional Scaling**
"""
"""
### Multidimensional scaling (MDS) is a form of **nonlinear dimensionality reduction** that learns the similarity of points in the original dataset and, using this similarity learning, models this in a lower dimensional space
"""
# Multidimensional Scaling
from sklearn.manifold import MDS
n_components = 2
n_init = 12
max_iter = 1200
metric = True
n_jobs = 4
random_state = 2018
mds = MDS(n_components=n_components, n_init=n_init, max_iter=max_iter, \
metric=metric, n_jobs=n_jobs, random_state=random_state)
X_train_mds = mds.fit_transform(X_train.loc[0:1000,:])
X_train_mds = pd.DataFrame(data=X_train_mds, index=train_index[0:1001])
scatterPlot(X_train_mds, y_train, "Multidimensional Scaling")

"""
# **Locally Linear Embedding (LLE)**
"""
"""
### Another popular **nonlinear dimensionality reduction** method is called locally linear embedding (LLE). This method preserves distances within local neighborhoods as it projects the data from the original feature space to a reduced space. LLE discovers the nonlinear structure in the original, high-dimensional data by segmenting the data into smaller components and modeling each component as a linear embedding.

"""
# Locally Linear Embedding (LLE)
from sklearn.manifold import LocallyLinearEmbedding
n_neighbors = 10
n_components = 2
method = 'modified'
n_jobs = 4
random_state = 2018
lle = LocallyLinearEmbedding(n_neighbors=n_neighbors, \
n_components=n_components, method=method, \
random_state=random_state, n_jobs=n_jobs)
lle.fit(X_train.loc[0:5000,:])
X_train_lle = lle.transform(X_train)
X_train_lle = pd.DataFrame(data=X_train_lle, index=train_index)
X_validation_lle = lle.transform(X_validation)
X_validation_lle = pd.DataFrame(data=X_validation_lle, index=validation_index)
scatterPlot(X_train_lle, y_train, "Locally Linear Embedding")
"""
# **t-SNE**
"""
"""
### t-distributed stochastic neighbor embedding (t-SNE) is a **nonlinear dimensionality reduction** technique. It does this by constructing **two probability distributions**, one over pairs of points in the high-dimensional space and another over pairs of points in the low-dimensional space such that similar points have a high probability and dissimilar points have a lower probability. Specifically, t-SNE minimizes the Kullback–Leibler divergence between the two probability distributions.

The **Kullback-Leibler Divergence score** , or KL divergence score, quantifies how much one probability distribution differs from another probability distribution.
"""
"""
### In real-world applications of t-SNE, it is best to use another dimensionality reduction technique (such as PCA, as we do here) to reduce the number of dimensions before applying t-SNE. By applying another form of dimensionality reduction first, we reduce the noise in the features that are fed into t-SNE and speed up the computation of the algorithm
"""
"""
### t-SNE has a nonconvex cost function, which means that different initializations of the algorithm will generate different results. There is no stable solution.
"""
from sklearn.manifold import TSNE
n_components=2
learning_rate=300
perplexity=30
early_exaggeration=12
init="random"
random_state=2018

tSNE=TSNE(n_components=n_components,learning_rate=learning_rate,
         perplexity=perplexity,early_exaggeration=early_exaggeration,
         init=init,random_state=random_state)

X_train_tSNE=tSNE.fit_transform(X_train_PCA.loc[:5000,:9])
X_train_tSNE=pd.DataFrame(data=X_train_tSNE,index=train_index[:5001])

scatterPlot(X_train_tSNE,y_train,"t-SNE")
"""
# **Mini-batch dictionary learning**
"""
"""
### Dictionary learning learns the **sparse representation** of the original data. The resulting matrix is known as the dictionary, and the vectors in the dictionary are known as atoms. These atoms are simple, binary vectors, populated by zeros and ones. Each instance in the original data can be reconstructed as a weighted sum of these atoms.
"""
from sklearn.decomposition import MiniBatchDictionaryLearning

n_components=50
alpha=1
batch_size=200
n_iter=25
random_state=2018

miniBatchDictLearning=MiniBatchDictionaryLearning( \
n_components=n_components, alpha=alpha, \
batch_size=batch_size, n_iter=n_iter, \
random_state=random_state)

miniBatchDictLearning.fit(X_train.loc[:,:10000])
X_train_miniBatchDictLearning = miniBatchDictLearning.fit_transform(X_train)
X_train_miniBatchDictLearning = pd.DataFrame( \
data=X_train_miniBatchDictLearning, index=train_index)
X_validation_miniBatchDictLearning = \
miniBatchDictLearning.transform(X_validation)
X_validation_miniBatchDictLearning = \
pd.DataFrame(data=X_validation_miniBatchDictLearning, \
index=validation_index)
scatterPlot(X_train_miniBatchDictLearning, y_train, \
"Mini-batch Dictionary Learning")
"""
# **Independent Component Analysis**

"""
"""
### One common problem with unlabeled data is that there are many independent signals embedded together into the features we are given. Using independent component analysis (ICA), we can separate these blended signals into their individual components. After the separation is complete, we can reconstruct any of the original features by adding together some combination of the individual components we generate. ICA is commonly used in signal processing tasks (for example, to identify the individual voices in an audio clip of a busy coffeehouse)
"""
# Independent Component Analysis
from sklearn.decomposition import FastICA
n_components = 25
algorithm = 'parallel'
whiten = True
max_iter = 100
random_state = 2018
fastICA = FastICA(n_components=n_components, algorithm=algorithm, \
whiten=whiten, max_iter=max_iter, random_state=random_state)
X_train_fastICA = fastICA.fit_transform(X_train)
X_train_fastICA = pd.DataFrame(data=X_train_fastICA, index=train_index)
X_validation_fastICA = fastICA.transform(X_validation)
X_validation_fastICA = pd.DataFrame(data=X_validation_fastICA, \
index=validation_index)
scatterPlot(X_train_fastICA, y_train, "Independent Component Analysis")
"""
# References:
* Ankur A. Patel - Hands-On Unsupervised Learning Using Python_ How to Build Applied Machine Learning Solutions from Unlabeled Data-O'Reilly Media (2019)
* Rowel Atienza - Advanced Deep Learning with TensorFlow 2 and Keras_ Apply DL, GANs, VAEs, deep RL, unsupervised learning, object detection and segmentation, and more, 2nd Edition
* Johnston, Benjamin_Jones, Aaron_Kruger, Christopher - Applied Unsupervised Learning with Python-Packt Publishing (2019)
* https://machinelearningmastery.com/divergence-between-probability-distributions/
* https://www.analyticsvidhya.com/blog/2018/08/dimensionality-reduction-techniques-python/
* https://byjus.com/jee/rank-of-a-matrix-and-special-matrices/
* https://stats.stackexchange.com/questions/2499/what-is-a-kernel-in-plain-english
* https://stackoverflow.com/questions/41325673/what-does-sparse-mean-in-the-context-of-neural-nets


"""