# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python (learn-env)
#     language: python
#     name: learn-env
# ---

# # Phase 4 Code Challenge Review: PCA and NLP

from src.student_caller import one_random_student
from src.student_list import quanggang 

# TOC:
#
#   - [PCA](#pca)
#   - [NLP](#nlp)
#

# <a id='pca'></a>

# # PCA

# When creating principle components, PCA aims to find a vector in the direction of our feature space that is fit to what?

# fit to a line that captures as much variance in the data as possible.

# PCA aims to find the directions of maximum variance in high-dimensional data and projects it onto a new subspace with equal or fewer dimensions than the original one.
#

# How is the 1st principle component related to the 2nd?

# The first principal component is the direction in space along which projections have the largest variance. The second principal component is the direction which maximizes variance among all directions orthogonal to the first.

# What are some reasons for using PCA?
#

# Reduce effects of multicollinearity.

# Reduce computation time.

# Helps to visualize high dimensional data. 

# Avoid curse of dimensionality 

# How can one determine how many principle components to use in a model?

# Look at cumulative variance explained and stop at a level of comfort, say 90%.
# Stop at a point of diminishing returns. Each addtional PCA does not add enough variance explanation.

# +
# Now let's implement PCA in code.
# -

import pandas as pd
from sklearn.datasets import  load_breast_cancer
data = load_breast_cancer()
X = pd.DataFrame(data['data'], columns = data['feature_names'])
X.head()

# +
from sklearn.decomposition import PCA

pca = PCA(n_components = 0.9, random_state = 23)
# -

pca.fit(X)

pca.n_components_

# <a id='nlp'></a>
#

# # NLP

# For NLP data, what is the entire data of records called?

# The Corpus

# What is an individual record called?

# Document

# What is a group of two words that appear next to one-another in a document?

# Bi-grams

# What is a high frequency, semantically low value word called? 

# Stop word

# List the preprocessing steps we can employ to create a cleaner feature set to our models.

# Split on spaces.
# Lower casing.
# Stop words removal.
# Stemming.
# Lemmatization.
# Tokenization.

# What sklearn tools do we have at our disposal to turn our raw text into numerical representations?

# CountVectorizer, TFIDVectorizer

# Explain the difference between the two main vectorizors we employ to transform the data into the document-term matrix.

# CountVectorizer returns integers (count within document), TFIDVectorizer returns floats (frequency in this doc, v. frequency in corpus)

# What form do the two main vectorizors expect our data to be fed to them?

# list of strings, each element a document

# Now let's write some code.

# +
# As a group, preprocess the data using an appropriate cross-validation strategy. 
# Test out different parameters in the vectorizer of choice
# Select a model and score it on the test set.
# If we have time.  Look at the frequency distributions of the words for a specific candidate.
# -

policies = pd.read_csv('data/2020_policies_feb_24.csv')
policies.head()

# +
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer()
# -

tfidf.fit_transform(policies['policy'])



StarcoderdataJupyterScriptsDedupFiltered