# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Preprocessing

# We must transform data into a state suitable for machine learning.  Whatever transformation we need, the scikit-learn preprocessing module lets us accomplish it in just a couple lines of code.

# ## Standardization

# Standardization transforms data to have a mean and unit variance of zero, as expected by many machine learning algorithms.  Therefore, we subtract the mean from each datum and divide all data by the standard deviation.

from sklearn import preprocessing
import numpy as np

# Here's some example training data that we want to scale down.

X_train = np.array([[1.0, -1.0, 2.0],
                    [2.0, 0.0, 0.0],
                    [0.0, 1.0, -1.0]])

# First, we create a StandardScaler for our data.

scaler = preprocessing.StandardScaler().fit(X_train)

scaler.mean_

scaler.scale_

# Then, we create a new, scaled dataset!

X_scaled = scaler.transform(X_train)

X_scaled

# Use this class early in your pipeline to ensure all downstream code is using data of the same scale.

# ## Non-linear Transformation

# Some transforms are more than just subtraction and division.

# ### Mapping to a uniform distribution

# We can establish the quantile of each datum by mapping our data onto a uniform distribution.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

# To demonstrate, we load and split the iris dataset.

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# Next, we create a QuantileTransformer from the preprocessing module. 

quantile_transformer = preprocessing.QuantileTransformer(random_state=0, n_quantiles = len(X_train))

# Finally, we create uniformly mapped datasets with our transformer.

X_train_trans = quantile_transformer.fit_transform(X_train)
X_test_trans = quantile_transformer.transform(X_test)

# Checking the results with the percentile module of numpy reveals the results we would expect.

np.percentile(X_train[:, 0], [0, 25, 50, 75, 100]) 

# ### Mapping to a Gaussian Distribution

# Other algorithms require their data to fit a Gaussian distribution because it minimizes the variance and skewness of the data.  The PowerTransformer class offers the Yeo-Johnson and Box-Cox methods, the latter of which works for negative as well as positive numbers and is demonstrated below.

pt = preprocessing.PowerTransformer(method='box-cox', standardize=False)
X_lognormal = np.random.RandomState(616).lognormal(size=(3, 3))

X_lognormal

pt.fit_transform(X_lognormal)

# We can also use the QuantileTransformer class to map data onto normal distributions.

quantile_transformer = preprocessing.QuantileTransformer(
    output_distribution='normal', random_state=0, n_quantiles = len(X))

X_trans = quantile_transformer.fit_transform(X)

quantile_transformer.quantiles_

# ### Normalization

# Scaling data to have unit norm enables such quadratic methods as dot-product or a kernel to quantify the separation of data.  The preprocessing module has a method for just this purpose! 

X = [[ 1., -1.,  2.],
     [ 2.,  0.,  0.],
     [ 0.,  1., -1.]]

X_normalized = preprocessing.normalize(X, norm='l2')

X_normalized

StarcoderdataJupyterScriptsDedupFiltered