# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/danielburch/DS-Unit-2-Kaggle-Challenge/blob/master/module3-cross-validation/LS_DS_223_assignment.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="27EjTMlRTvAJ" colab_type="text"
# Lambda School Data Science
#
# *Unit 2, Sprint 2, Module 3*
#
# ---

# + [markdown] id="IwSNbBaWTvAL" colab_type="text"
# # Cross-Validation
#
#
# ## Assignment
# - [ ] [Review requirements for your portfolio project](https://lambdaschool.github.io/ds/unit2), then submit your dataset.
# - [ ] Continue to participate in our Kaggle challenge. 
# - [ ] Use scikit-learn for hyperparameter optimization with RandomizedSearchCV.
# - [ ] Submit your predictions to our Kaggle competition. (Go to our Kaggle InClass competition webpage. Use the blue **Submit Predictions** button to upload your CSV file. Or you can use the Kaggle API to submit your predictions.)
# - [ ] Commit your notebook to your fork of the GitHub repo.
#
#
# **You can't just copy** from the lesson notebook to this assignment.
#
# - Because the lesson was **regression**, but the assignment is **classification.**
# - Because the lesson used [TargetEncoder](https://contrib.scikit-learn.org/categorical-encoding/targetencoder.html), which doesn't work as-is for _multi-class_ classification.
#
# So you will have to adapt the example, which is good real-world practice.
#
# 1. Use a model for classification, such as [RandomForestClassifier](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)
# 2. Use hyperparameters that match the classifier, such as `randomforestclassifier__ ...`
# 3. Use a metric for classification, such as [`scoring='accuracy'`](https://scikit-learn.org/stable/modules/model_evaluation.html#common-cases-predefined-values)
# 4. If you’re doing a multi-class classification problem — such as whether a waterpump is functional, functional needs repair, or nonfunctional — then use a categorical encoding that works for multi-class classification, such as [OrdinalEncoder](https://contrib.scikit-learn.org/categorical-encoding/ordinal.html) (not [TargetEncoder](https://contrib.scikit-learn.org/categorical-encoding/targetencoder.html))
#
#
#
# ## Stretch Goals
#
# ### Reading
# - <NAME>, [Python Data Science Handbook, Chapter 5.3](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html), Hyperparameters and Model Validation
# - <NAME>, [Statistics for Hackers](https://speakerdeck.com/jakevdp/statistics-for-hackers?slide=107)
# - <NAME>, [A Programmer's Guide to Data Mining, Chapter 5](http://guidetodatamining.com/chapter5/), 10-fold cross validation
# - <NAME>, [A Basic Pipeline and Grid Search Setup](https://github.com/rasbt/python-machine-learning-book/blob/master/code/bonus/svm_iris_pipeline_and_gridsearch.ipynb)
# - <NAME>, [A Comparison of Grid Search and Randomized Search Using Scikit Learn](https://blog.usejournal.com/a-comparison-of-grid-search-and-randomized-search-using-scikit-learn-29823179bc85)
#
# ### Doing
# - Add your own stretch goals!
# - Try other [categorical encodings](https://contrib.scikit-learn.org/categorical-encoding/). See the previous assignment notebook for details.
# - In additon to `RandomizedSearchCV`, scikit-learn has [`GridSearchCV`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html). Another library called scikit-optimize has [`BayesSearchCV`](https://scikit-optimize.github.io/notebooks/sklearn-gridsearchcv-replacement.html). Experiment with these alternatives.
# - _[Introduction to Machine Learning with Python](http://shop.oreilly.com/product/0636920030515.do)_ discusses options for "Grid-Searching Which Model To Use" in Chapter 6:
#
# > You can even go further in combining GridSearchCV and Pipeline: it is also possible to search over the actual steps being performed in the pipeline (say whether to use StandardScaler or MinMaxScaler). This leads to an even bigger search space and should be considered carefully. Trying all possible solutions is usually not a viable machine learning strategy. However, here is an example comparing a RandomForestClassifier and an SVC ...
#
# The example is shown in [the accompanying notebook](https://github.com/amueller/introduction_to_ml_with_python/blob/master/06-algorithm-chains-and-pipelines.ipynb), code cells 35-37. Could you apply this concept to your own pipelines?
#

# + [markdown] id="fYBugPRpTvAL" colab_type="text"
# ### BONUS: Stacking!
#
# Here's some code you can use to "stack" multiple submissions, which is another form of ensembling:
#
# ```python
# import pandas as pd
#
# # Filenames of your submissions you want to ensemble
# files = ['submission-01.csv', 'submission-02.csv', 'submission-03.csv']
#
# target = 'status_group'
# submissions = (pd.read_csv(file)[[target]] for file in files)
# ensemble = pd.concat(submissions, axis='columns')
# majority_vote = ensemble.mode(axis='columns')[0]
#
# sample_submission = pd.read_csv('sample_submission.csv')
# submission = sample_submission.copy()
# submission[target] = majority_vote
# submission.to_csv('my-ultimate-ensemble-submission.csv', index=False)
# ```

# + id="lg2vkrriTvAN" colab_type="code" colab={}
# %%capture
import sys

# If you're on Colab:
if 'google.colab' in sys.modules:
    DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'
    # !pip install category_encoders==2.*

# If you're working locally:
else:
    DATA_PATH = '../data/'

# + id="eiLi5OskTvAS" colab_type="code" colab={}
import pandas as pd


# Merge train_features.csv & train_labels.csv
train = pd.merge(pd.read_csv(DATA_PATH+'waterpumps/train_features.csv'), 
                 pd.read_csv(DATA_PATH+'waterpumps/train_labels.csv'))

# Read test_features.csv & sample_submission.csv
test = pd.read_csv(DATA_PATH+'waterpumps/test_features.csv')
sample_submission = pd.read_csv(DATA_PATH+'waterpumps/sample_submission.csv')

# + id="UDAGP0v6kTIR" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 428} outputId="30004fa3-84a1-422c-d27a-8f37b0f55f8d"
test.head()

# + id="YMpJcpHcW_So" colab_type="code" colab={}
import numpy as np

def wrangle(X):
    
    # Make a copy 
    X = X.copy()

    #Using Nicholas' code from class and added a few more columns with zeros
    X['latitude'] = X['latitude'].replace(-2e-08, 0)
    cols_with_zeros = ['longitude', 'latitude', 'population',
                        'construction_year','district_code','gps_height']
    for col in cols_with_zeros:
        X[col] = X[col].replace(0, np.nan)
            
    # quantity & quantity_group are duplicates, so drop one
    X = X.drop(columns='quantity_group')
    
    # return the wrangled dataframe
    return X


train = wrangle(train)
test = wrangle(test)


# + id="AB-On0fr8FlB" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 411} outputId="7fd728a4-2aab-41ba-c5ee-bb41e729c597"

train.head()

# + id="0Q8kI-jRYAA7" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 119} outputId="774319f2-db59-41e2-e5c5-88a4498ccbc7"
# Set the target to the status group column
target ='status_group'

# Create a dataframe with the other features
df_train = train.drop(columns=[target,'id'])

print(train[target].head())

# + id="YSg4U5veYCey" colab_type="code" colab={}
#Remove High Cardinality Features


# Get a list of the numeric features
numeric_features = df_train.select_dtypes(include='number').columns.tolist()

# Get a series with the cardinality of the nonnumeric features
cardinality = df_train.select_dtypes(exclude='number').nunique()

# Get a list of all categorical features with cardinality <= 50
categorical_features = cardinality[cardinality <= 50].index.tolist()

# Combine the lists 
features = numeric_features + categorical_features

X_train = train[features]
y_train = train[target]
X_test = test[features]


# + id="GchQ7tCFYIcw" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 204} outputId="f922d973-d9d0-4e56-bd41-e11cc1a292b5"

from scipy.stats import randint, uniform
import category_encoders as ce
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import VarianceThreshold
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import RandomizedSearchCV

pipeline = make_pipeline(
    ce.OrdinalEncoder(),
    SimpleImputer(strategy = 'most_frequent'), 
    RandomForestClassifier(random_state=56)
)

param_distributions = {      
    'randomforestclassifier__n_estimators': randint(50, 500), 
    'randomforestclassifier__max_depth': [5, 10, 15, 20, None], 
    'randomforestclassifier__max_features': uniform(0, 1), 
}


search = RandomizedSearchCV(
    pipeline, 
    param_distributions=param_distributions, 
    n_iter=25, 
    cv=2, 
    scoring='accuracy', 
    verbose=10, 
    return_train_score=True, 
    n_jobs=-1
)

search.fit(X_train, y_train);

# + id="_IgbnyHD2fGy" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 71} outputId="d58763d4-6e92-42ed-986f-46c206c5a094"
print('Best hyperparameters', search.best_params_)
print('Cross-validation Accuracy', -search.best_score_)


# + id="n2M66GJh8Mqq" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 613} outputId="a688d588-d09b-41ee-90ef-c863d16281d2"
pd.DataFrame(search.cv_results_).sort_values(by='rank_test_score').T

# + id="s9H7tvSU9eRt" colab_type="code" colab={"base_uri": "https://localhost:8080/", "height": 17} outputId="5ad71ce1-5a94-4464-cc9b-1d8ec9546188"
#Save for Kaggle Submission
y_pred = search.predict(X_test)

DATA_PATH = 'https://raw.githubusercontent.com/LambdaSchool/DS-Unit-2-Kaggle-Challenge/master/data/'
sample_submission = pd.read_csv(DATA_PATH+'waterpumps/sample_submission.csv')
submission = sample_submission.copy()
submission['status_group'] = y_pred
submission.to_csv('daniel_burch_kaggle_randomizedsearch4.csv', index=False)

from google.colab import files
files.download('daniel_burch_kaggle_randomizedsearch4.csv')

StarcoderdataJupyterScriptsDedupFiltered