%%HTML
<style type="text/css">
div.h1 {
    background-color: #009C37; 
    color: white; padding: 8px; padding-right: 300px; font-size: 35px; max-width: 1500px; margin: auto; margin-top: 50px;}
div.h2 {
    background-color: #002277; 
    color: white; padding: 8px; padding-right: 300px; font-size: 25px; max-width: 1500px; margin: auto; margin-top: 50px;}
div.h3 {
    color: #002277;
    font-size: 16px; margin-top: 20px; margin-bottom:4px;}
div.h4 {
    font-size: 15px; margin-top: 20px; margin-bottom: 8px;
span.note {
    font-size: 5; color: #002277;
    font-style: italic;}
span.captiona {
    font-size: 5; color: dimgray; font-style: italic; margin-left: 130px; vertical-align: top;}
hr {
    display: block; color: #002277;
    height: 1px; border: 0;  border-top: 1px solid;}
hr.light {
    display: block; color: lightgray; height: 1px; border: 0; border-top: 1px solid;}
</style>
"""
<div class="h2">Diagnosis of COVID-19 and its clinical spectrum</div>
### AI and Data Science supporting clinical decisions (from 28th Mar to 1st Apr)

![](https://img.olhardigital.com.br/uploads/acervo_imagens/2020/03/r16x9/20200320062922_1200_675_-_coronavirus_brasil_arte.jpg)
[source](https://img.olhardigital.com.br/uploads/acervo_imagens/2020/03/r16x9/20200320062922_1200_675_-_coronavirus_brasil_arte.jpg)

<div class="h3">Background</div>
The World Health Organization (WHO) characterized the COVID-19, caused by the SARS-CoV-2, as a pandemic on March 11, while the exponential increase in the number of cases was risking to overwhelm health systems around the world with a demand for ICU beds far above the existing capacity, with regions of Italy being prominent examples.

Brazil recorded the first case of SARS-CoV-2 on February 26, and the virus transmission evolved from imported cases only, to local and finally community transmission very rapidly, with the federal government declaring nationwide community transmission on March 20.

**Until March 27**, the state of São Paulo had recorded 1,223 confirmed cases of COVID-19, with 68 related deaths, while the county of São Paulo, with a population of approximately 12 million people and where Hospital Israelita Albert Einstein is located, had 477 confirmed cases and 30 associated death, as of March 23. Both the state and the county of São Paulo decided to establish quarantine and social distancing measures, that will be enforced at least until early April, in an effort to slow the virus spread.

One of the motivations for this challenge is the fact that in the context of an overwhelmed health system with the possible limitation to perform tests for the detection of SARS-CoV-2, testing every case would be impractical and tests results could be delayed even if only a target subpopulation would be tested.

<div class="h3">Dataset</div>
This dataset contains anonymized data from patients seen at the [Hospital Israelita Albert Einstein](https://www.einstein.br/Pages/Home.aspx), at São Paulo, Brazil, and who had samples collected to perform the SARS-CoV-2 RT-PCR and additional laboratory tests during a visit to the hospital.

All data were anonymized following the best international practices and recommendations. All clinical data were standardized to have a mean of zero and a unit standard deviation.
"""
"""
<div class="h2">Task Details</div>
Predict confirmed **COVID-19** cases among suspected cases.
Based on the results of laboratory tests commonly collected for a suspected **COVID-19** case during a visit to the emergency room, would it be possible to predict the test result for **SARS-Cov-2** (positive/negative)?
"""
"""
<a id='top'></a>
<div class="h1">Contents</div>

1. [Glimpse of Data](#t1)

    1.1 [Imports](#t1_1)
    1.2 [Read in Data](#t1_2)

2. [Exploratory Data Analysis](#t2)
    2.1 [Examine the Distribution of the Target Column](#t2_1)
    2.2 [Examine Missing Values](#t2_2)
    2.3 [Handling missing values](#t2_3)
    2.4 [Column Types](#t2_4)
    2.5 [Correlations](#t2_5)
    
3. [Visualizations](#t3)

4. [Encoding Variables](#t4)
    4.1 [Label Encoding](#t4_1)

5. [Feature Selection](#t5)
    5.1 [Remove Collinear features](#t5_1)
    5.2 [Remove features with greater than a threshold percentage of missing values](#t5_2)
    5.3 [Feature Selection through Feature Importances](#t5_3)
    5.4 [LGB Shap Values](#t5_4)

6. [Modeling](#t6)
    6.1 [Confusion Matrix function](#t6_1)
    6.2 [Gradient Boosting Model function](#t6_2)
    6.3 [XGB](#t6_3)

7. [Ensembling With StackNet](#t7)
    7.1 [StackNet Modeling](#t7_1)
    7.1 [Evaluation StackNet ROC_AUC](#t7_2)

8. [General Findings](#t8)

9. [End Notebooks](#end)

"""
"""
# Acknowledgments
[XGB | LGB | CB](https://www.kaggle.com/vincentlugat/fraud-detection-xgb-lgb-cb/) by @vincentlugat
[LigthGBM Simple fe](https://www.kaggle.com/caesarlupum/ashrae-ligthgbm-simple-fe) by @caesarlupum
[Ensembling With Stacknet](https://www.kaggle.com/carlolepelaars/ensembling-with-stacknet/notebook) by @carlolepelaars
[Stacknet on gpu lgb xgb cb](https://www.kaggle.com/kirankunapuli/ieee-fraud-stacknet-on-gpu-lgb-xgb-cb/notebook) by @kirankunapuli
[eda and prediction](https://www.kaggle.com/gpreda/santander-eda-and-prediction) by @gpreda
[Introduction to Feature Selection](https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection#Introduction:-Feature-Selection) by @willkoehrsen
[EDA  lgb shap](https://www.kaggle.com/rspadim/eda-first-try-python-lgb-shap) by @rspadim

<div class="h3">Remember the upvote button is next to the fork button, and it's free too! ;) Don't hesitate to give your suggestions in the comment section, It motivates me to produce more quality content :)
</div>
"""
# Suppress warnings 
import warnings
warnings.filterwarnings("ignore", category=DeprecationWarning)
warnings.filterwarnings("ignore", category=UserWarning)
warnings.filterwarnings("ignore", category=FutureWarning)
from IPython.display import HTML
HTML('<iframe width="800" height="500" src="https://www.youtube.com/embed/zlzU0hytEks" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
"""
<div class="h3"> Heroes in Action</div>
Coronavirus: The routine of doctors at Albert Einstein Hospital in the fight against the advance of Covid-19
"""
"""
<a id='t1'></a>
# <div class="h1"> Glimpse of Data</div>
[Back to Contents](#top)

[Next](#t1_1)

"""
"""
<a id='t1_1'></a>
# <div class="h3"> Imports</div>
[Next](#t1_2)

We are using a typical data science stack: ``numpy``, ``pandas``, ``sklearn``, ``matplotlib``.
"""
# Data processing, metrics and modeling
from sklearn.preprocessing import LabelEncoder, StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from datetime import datetime
from sklearn.metrics import precision_score, recall_score, confusion_matrix, accuracy_score, roc_auc_score, f1_score, roc_curve, auc,precision_recall_curve
from sklearn import metrics
# Lgbm
import lightgbm as lgb
import catboost
from catboost import Pool
import xgboost as xgb

# Suppr warning
import warnings
warnings.filterwarnings("ignore")
"""
<div class="h3"> Import PyStackNet Package</div>
"""
# Import PyStackNet Package
# Source: https://www.kaggle.com/kirankunapuli/pystacknet
import os
import sys
package_dir = "/kaggle/input/pystacknet/h2oai-pystacknet-af571e0"
sys.path.append(package_dir)
import pystacknet
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# matplotlib and seaborn for plotting
import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns
import matplotlib.patches as patches

from plotly import tools, subplots
import plotly.offline as py
py.init_notebook_mode(connected=True)
import plotly.graph_objs as go
import plotly.express as px
py.init_notebook_mode(connected=True)
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)

pd.set_option('max_columns', 200)
pd.set_option('max_rows', 200)
print(os.listdir("../input"))
"""
<a id='t1_2'></a>
# <div class="h3"> Read in Data</div>
[Next](#t2)
"""
%%time
dataset = pd.read_excel('/kaggle/input/covid19/dataset.xlsx')
dataset.columns = [x.lower().strip().replace(' ','_') for x in dataset.columns]
print('Size of the data', dataset.shape)
"""
dataset columns values
"""
dataset.columns.values
"""
dataset head
"""
dataset.head()
%%time
dataset.describe()
"""
<a id='t2'></a>
# <div class="h1"> Exploratory Data Analysis</div>

[Back to Contents](#top)

[Next](#t2_1)


Exploratory Data Analysis (EDA) is an open-ended process where we calculate statistics and make figures to find trends, anomalies, patterns, or relationships within the data.
"""
"""
<a id='t2'></a>
# <div class="h3"> Examine the Distribution of the Target Column</div>
[Next](#t2_2)

"""
"""
Transform the target variable into numeric
"""
dataset['sars-cov-2_exam_result'] = dataset['sars-cov-2_exam_result'].replace(['negative','positive'], [0,1])
sns.countplot(dataset['sars-cov-2_exam_result'])

print("There are {}% target values with 1".format(100 * dataset['sars-cov-2_exam_result'].value_counts()[1]/dataset.shape[0]))
"""
<a id='t2_2'></a>
# <div class="h3"> Examine Missing Values</div>
[Next](#t2_3)

Next we can look at the number and percentage of missing values in each column
"""
def missing_data(data):
    total = data.isnull().sum()
    percent = (data.isnull().sum()/data.isnull().count()*100)
    tt = pd.concat([total, percent], axis=1, keys=['Total', 'Percent'])
    types = []
    for col in data.columns:
        dtype = str(data[col].dtype)
        types.append(dtype)
    tt['Types'] = types
    return(np.transpose(tt))
%%time
missing_data(dataset)
"""
<a id='t2_3'></a>
# <div class="h3">Handling missing values</div>
[Next](#t2_4)

Package called missingno (https://github.com/ResidentMario/missingno)
 !pip install quilt
"""
import missingno as msno
"""
**Nullity Matrix**.
The msno.matrix nullity matrix is a data-dense display which lets you quickly visually analyse data completion.
"""
msno.matrix(dataset.head(20000))
"""
**Heatmap**. The missingno correlation heatmap measures nullity correlation: how strongly the presence or absence of one variable affects the presence of another:
"""
%%time
a = msno.heatmap(dataset, sort='ascending')
a
"""
**Dendrogram**.The dendrogram allows you to more fully correlate variable completion, revealing trends deeper than the pairwise ones visible in the correlation heatmap:
"""
%%time
a2 = msno.dendrogram(dataset)
a2
"""
<a id='t2_4'></a>
# <div class="h3">Column Types</div>
[Next](#t2_5)

Let's look at the number of columns of each data type. int64 and float64 are numeric variables [(which can be either discrete or continuous)](https://stats.stackexchange.com/questions/206/what-is-the-difference-between-discrete-data-and-continuous-data). object columns contain strings and are [categorical features](https://stats.stackexchange.com/questions/206/what-is-the-difference-between-discrete-data-and-continuous-data). 
"""
"""
Number of each type of column
"""
# Number of each type of column
dataset.dtypes.value_counts()
"""
Number of unique classes in each object column
"""
# Number of unique classes in each object column
dataset.select_dtypes('object').apply(pd.Series.nunique, axis = 0)
"""
<a id='t2_5'></a>
# <div class="h3">Correlations</div>
[Next](#t3)


Now that we have dealt with the categorical variables and the outliers, let's continue with the EDA. One way to try and understand the data is by looking for correlations between the features and the target. We can calculate the Pearson correlation coefficient between every variable and the target using the .corr dataframe method.

The correlation coefficient is not the greatest method to represent "relevance" of a feature, but it does give us an idea of possible relationships within the data. Some [general interpretations of the absolute value of the correlation coefficent are](http://www.statstutor.ac.uk/resources/uploaded/pearsons.pdf):

- .00-.19 <b>“very weak”</b>
- .20-.39 <b>“weak”</b>
- .40-.59 <b>“moderate”</b>
- .60-.79 <b>“strong”</b>
- .80-1.0 <b>“very strong”</b>
"""
# Find correlations with the target and sort
correlations = dataset.corr()['sars-cov-2_exam_result'].sort_values()

# Display correlations
print('Most Positive Correlations:\n', correlations.tail(15))
print('\nMost Negative Correlations:\n', correlations.head(15))
"""
We calculate now the correlations between the features in dataset.
The following table shows the first 10 the least correlated features.
"""
%%time
features = dataset.columns.values[2:112]
corrs_ = dataset[features].corr().abs().unstack().sort_values(kind="quicksort").reset_index()
corrs_ = corrs_[corrs_['level_0'] != corrs_['level_1']]
corrs_.head(10)
"""
Let's see also the least correlated features.
"""
corrs_.head(10)
"""
Let's look to the top most correlated features, besides the same feature pairs.
"""
corrs_.tail(10)
"""
The correlation between the features is very small.
"""
"""
Correlations clustermap

"""
%%time
corrs = dataset.corr()
plt.figure(figsize = (20, 8))
# Heatmap of correlations
sns.heatmap(corrs, cmap = plt.cm.RdYlBu_r, vmin = -0.25, annot = False, vmax = 0.8)
plt.title('Clustermap');
"""
<a id='t3'></a>
# <div class="h1">Visualizations</div>

[Back to Contents](#top)

[Next](#t4)

"""
"""
Patient age quantile by sars cov2 exam result
"""
def plot_dist_col(column):
    pos__df = dataset[dataset['sars-cov-2_exam_result'] ==1]
    neg__df = dataset[dataset['sars-cov-2_exam_result'] ==0]

    '''plot dist curves for train and test weather data for the given column name'''
    fig, ax = plt.subplots(figsize=(10, 10))
    sns.distplot(pos__df[column].dropna(), color='green', ax=ax).set_title(column, fontsize=16)
    sns.distplot(neg__df[column].dropna(), color='purple', ax=ax).set_title(column, fontsize=16)
    plt.xlabel(column, fontsize=15)
    plt.legend(['Positive sars cov 2 exam result', 'Negative sars cov 2 exam result'])
    plt.show()
plot_dist_col('patient_age_quantile')
"""
% of SARS COV2 exam result (0=Negative, 1=Positive)
"""
%%time
fig = px.pie( values=dataset.groupby(['sars-cov-2_exam_result']).size().values,names=dataset.groupby(['sars-cov-2_exam_result']).size().index)
fig.update_layout(
    font=dict(
        size=15,
        color="#FEE000" ) )   
py.iplot(fig)

"""
<a id='t4'></a>
# <div class="h1">Encoding Variables</div>

[Back to Contents](#top)

[Next](#t5)

Before we go any further, we need to deal with pesky categorical variables. A machine learning model unfortunately cannot deal with categorical variables (except for some models such as [LightGBM](Before we go any further, we need to deal with pesky categorical variables. A machine learning model unfortunately cannot deal with categorical variables (except for some models such as LightGBM. Therefore, we have to find a way to encode (represent) these variables as numbers before handing them off to the model. There are two main ways to carry out this process). Therefore, we have to find a way to encode (represent) these variables as numbers before handing them off to the model. There are two main ways to carry out this process.
You can see this kaggle course: [Intermediate Machine Learning Home Page](https://www.kaggle.com/alexisbcook/categorical-variables)
"""
"""
<a id='t4_1'></a>
# <div class="h2">Label Encoding</div>

[Back to Contents](#top)

[Next](#t5)

Label encoding assigns each unique value to a different integer.
![](https://raw.githubusercontent.com/WillKoehrsen/Machine-Learning-Projects/master/label_encoding.png)

This approach assumes an ordering of the categories: "Never" (0) < "Rarely" (1) < "Most days" (2) < "Every day" (3).
This assumption makes sense in this example, because there is an indisputable ranking to the categories. Not all categorical variables have a clear ordering in the values, but we refer to those that do as ordinal variables. For tree-based models (like decision trees and random forests), you can expect label encoding to work well with ordinal variables.
"""
#fill in mean for floats
for c in dataset.columns:
    if dataset[c].dtype=='float16' or  dataset[c].dtype=='float32' or  dataset[c].dtype=='float64':
        dataset[c].fillna(dataset[c].mean())

#fill in -999 for categoricals
dataset = dataset.fillna(-999)
# Label Encoding
for f in dataset.columns:
    if dataset[f].dtype=='object': 
        lbl = LabelEncoder()
        lbl.fit(list(dataset[f].values))
        dataset[f] = lbl.transform(list(dataset[f].values))
        
print('Labelling done.')    
"""
<a id='t5'></a>
# <div class="h1">Feature Selection</div>

[Back to Contents](#top)

[Next](#t5_1)

Find the optimal feature subset using an evaluation measure. The choice of evaluation metric distinguish the three main strategies of feature selection algorithms: the wrapper strategy, the filter strategy, and the embedded strategy

Filter methods:
   - information gain
   - chi-square test
   - correlation coefficient
   - variance threshold

Wrapper methods:
   - recursive feature elimination
   - sequential feature selection algorithms

Embedded methods:
  - L1 (LASSO) regularization
   - decision tree

In our case, we remove some useless, redundant variables.
We will use three methods for feature selection: Remove collinear features, remove features with greater than a threshold percentage of missing values, keep only the most relevant features using feature importances from a model
"""
"""
<a id='t5_1'></a>
# <div class="h2">Remove Collinear Variables</div>

[Back to Contents](#top)

[Next](#t5_2)

Collinear variables are those which are highly correlated with one another. These can decrease the model's availablility to learn, decrease model interpretability, and decrease generalization performance on the test set. 

guidelines:[Drop Highly Correlated Features](https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/)

"""
# Threshold for removing correlated variables
threshold = 0.92

# Absolute value correlation matrix
corr_matrix = dataset.corr().abs()
corr_matrix.head()
# Upper triangle of correlations
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
upper.head()
# Select columns with correlations above threshold
to_drop = [column for column in upper.columns if any(upper[column] > threshold)]

print('There are %d columns to remove.' % (len(to_drop)))
dataset = dataset.drop(columns = to_drop)
print('Data shape: ', dataset.shape)
print('Size of the data', dataset.shape)
"""
Applying this on the entire dataset results in **64** collinear features removed.
"""
to_drop
"""
<a id='t5_2'></a>
# <div class="h2">Remove Missing Values</div>

[Back to Contents](#top)

[Next](#t5_3)

In this implementation, if any columns have greater than 85% missing values, they will be removed.

guidelines:[LightGBM Advanced Topics](https://github.com/Microsoft/LightGBM/blob/master/docs/Advanced-Topics.rst)

"""
## Dataset missing values (in percent)
dataset_missing = (dataset.isnull().sum() / len(dataset)).sort_values(ascending = False)
dataset_missing.head()
# Identify missing values above threshold
dataset_missing_ = dataset_missing.index[dataset_missing > 0.85]

all_missing = list(set(dataset_missing_))
print('There are %d columns with more than 85%% missing values' % len(all_missing))
dataset = dataset.drop(columns = all_missing)
print('Data shape: ', dataset.shape)
"""
<a id='t5_3'></a>
# <div class="h2">Feature Selection through Feature Importances</div>

[Back to Contents](#top)

[Next](#t5_4)

Tree-based models (and consequently ensembles of trees) can determine an "importance" for each feature by measuring the reduction in impurity for including the feature in the model.
Since the LightGBM model does not need missing values to be imputed, we can directly fit on the training data. We will use Early Stopping to determine the optimal number of iterations and run the model twice, averaging the feature importances to try and avoid overfitting to a certain set of features.

guidelines: [Feature Selection by @willkoehrsen](https://www.kaggle.com/willkoehrsen/introduction-to-feature-selection#Introduction:-Feature-Selection)
"""
cat_features = [i for i in dataset.columns if str(dataset[i].dtype) in ['object', 'category']]

if len(cat_features) > 0:
    dataset[cat_features] = dataset[cat_features].astype('category')


df_lgb = dataset.copy()
for i in cat_features:
    df_lgb[i] = dataset[i].cat.codes

df_lgb.columns = ["".join (c if c.isalnum() else "_" for c in str(x)) for x in df_lgb.columns]

dataset_labels = df_lgb['sars_cov_2_exam_result']
df_lgb_ = df_lgb.copy()
df_lgb = df_lgb.drop(['patient_id', 
                 'sars_cov_2_exam_result', 
                 'patient_addmited_to_regular_ward__1_yes__0_no_', 
                 'patient_addmited_to_semi_intensive_unit__1_yes__0_no_', 
                 'patient_addmited_to_intensive_care_unit__1_yes__0_no_'
                ], axis=1)
x = df_lgb.copy()
"""
#### Initialize an empty array to hold feature importances
"""
# Initialize an empty array to hold feature importances
feature_importances = np.zeros(df_lgb.shape[1])

# Create the model with several hyperparameters
model = lgb.LGBMClassifier(objective='binary', boosting_type = 'goss', n_estimators = 5000, class_weight = 'balanced')

# Fit the model twice to avoid overfitting
for i in range(2):
    # Split into training and validation set
    dataset_features, valid_features, dataset_features_y, valid_y = train_test_split(x, dataset_labels, test_size = 0.20, random_state = i)
    
    # Train using early stopping
    model.fit(dataset_features, dataset_features_y, early_stopping_rounds=100, eval_set = [(valid_features, valid_y)], 
              eval_metric = 'auc', verbose = 200)
    
    # Record the feature importances
    feature_importances += model.feature_importances_
"""
#### Average feature importances! 
"""
# Make sure to average feature importances! 
feature_importances = feature_importances / 2
feature_importances = pd.DataFrame({'feature': list(df_lgb.columns), 'importance': feature_importances}).sort_values('importance', ascending = False)
feature_importances.head()
"""
#### Find the features with zero importance
"""
# Find the features with zero importance
zero_features = list(feature_importances[feature_importances['importance'] == 0.0]['feature'])
print('There are %d features with 0.0 importance' % len(zero_features))
feature_importances.tail()
def plot_feature_importances(df, threshold = 0.9):
    """
    Plots 15 most important features and the cumulative importance of features.
    Prints the number of features needed to reach threshold cumulative importance.
    
    Parameters
    --------
    df : dataframe
        Dataframe of feature importances. Columns must be feature and importance
    threshold : float, default = 0.9
        Threshold for prining information about cumulative importances
        
    Return
    --------
    df : dataframe
        Dataframe ordered by feature importances with a normalized column (sums to 1)
        and a cumulative importance column
    
    """
    
    plt.rcParams['font.size'] = 18
    
    # Sort features according to importance
    df = df.sort_values('importance', ascending = False).reset_index()
    
    # Normalize the feature importances to add up to one
    df['importance_normalized'] = df['importance'] / df['importance'].sum()
    df['cumulative_importance'] = np.cumsum(df['importance_normalized'])

    # Make a horizontal bar chart of feature importances
    plt.figure(figsize = (10, 6))
    ax = plt.subplot()
    
    # Need to reverse the index to plot most important on top
    ax.barh(list(reversed(list(df.index[:15]))), 
            df['importance_normalized'].head(15), 
            align = 'center', edgecolor = 'k')
    
    # Set the yticks and labels
    ax.set_yticks(list(reversed(list(df.index[:15]))))
    ax.set_yticklabels(df['feature'].head(15))
    
    # Plot labeling
    plt.xlabel('Normalized Importance'); plt.title('Feature Importances')
    plt.show()
    
    # Cumulative importance plot
    plt.figure(figsize = (8, 6))
    plt.plot(list(range(len(df))), df['cumulative_importance'], 'r-')
    plt.xlabel('Number of Features'); plt.ylabel('Cumulative Importance'); 
    plt.title('Cumulative Feature Importance');
    plt.show();
    
    importance_index = np.min(np.where(df['cumulative_importance'] > threshold))
    print('%d features required for %0.2f of cumulative importance' % (importance_index + 1, threshold))
    
    return df
norm_feature_importances = plot_feature_importances(feature_importances)
"""
It also looks like many of the features we made have literally 0 importance. For the gradient boosting machine, features with 0 importance are not used at all to make any splits. Therefore, we can remove these features from the model with no effect on performance.

Let's remove the features that have zero importance.
"""
df_lgb = df_lgb.drop(columns = zero_features)
print('Dataset shape: ', df_lgb.shape)

"""
At this point, we can re-run the model to see if it identifies any more features with zero importance. In a way, we are implementing our own form of recursive feature elimination. Since we are repeating work, we should probably put the zero feature importance identification code in a function.
"""
def identify_zero_importance_features(train, train_labels, iterations = 2):
    """
    Identify zero importance features in a training dataset based on the 
    feature importances from a gradient boosting model. 
    
    Parameters
    --------
    train : dataframe
        Training features
        
    train_labels : np.array
        Labels for training data
        
    iterations : integer, default = 2
        Number of cross validation splits to use for determining feature importances
    """
    
    # Initialize an empty array to hold feature importances
    feature_importances = np.zeros(train.shape[1])

    # Create the model with several hyperparameters
    model = lgb.LGBMClassifier(objective='binary', boosting_type = 'goss', n_estimators = 10000, class_weight = 'balanced')
    
    # Fit the model multiple times to avoid overfitting
    for i in range(iterations):

        # Split into training and validation set
        train_features, valid_features, train_y, valid_y = train_test_split(train, train_labels, test_size = 0.25, random_state = i)

        # Train using early stopping
        model.fit(train_features, train_y, early_stopping_rounds=100, eval_set = [(valid_features, valid_y)], 
                  eval_metric = 'auc', verbose = 200)

        # Record the feature importances
        feature_importances += model.feature_importances_ / iterations
    
    feature_importances = pd.DataFrame({'feature': list(train.columns), 'importance': feature_importances}).sort_values('importance', ascending = False)
    
    # Find the features with zero importance
    zero_features = list(feature_importances[feature_importances['importance'] == 0.0]['feature'])
    print('\nThere are %d features with 0.0 importance' % len(zero_features))
    
    return zero_features, feature_importances
second_round_zero_features, feature_importances = identify_zero_importance_features(df_lgb, dataset_labels)
norm_feature_importances = plot_feature_importances(feature_importances, threshold = 0.95)
"""
#### Threshold for cumulative importance

"""
# # Threshold for cumulative importance
threshold = 0.95

# Extract the features to keep
features_to_keep = list(norm_feature_importances[norm_feature_importances['cumulative_importance'] < threshold]['feature'])
features_to_keep.append('patient_addmited_to_intensive_care_unit__1_yes__0_no_')
features_to_keep.append( 'patient_addmited_to_regular_ward__1_yes__0_no_')
features_to_keep.append( 'patient_addmited_to_semi_intensive_unit__1_yes__0_no_')
features_to_keep.append( 'patient_id')
features_to_keep.append( 'sars_cov_2_exam_result')
                        
# Create new datasets with smaller features
dataset_small = df_lgb_[features_to_keep]
    
"""
<a id='t5_4'></a>
# <div class="h2">Shap values LGB</div>

[Back to Contents](#top)

[Next](#t6)

guidelines:[LightGBM Shap](https://www.kaggle.com/rspadim/eda-first-try-python-lgb-shap/notebook), [LightGBM Predictions Explained with SHAP ](https://www.kaggle.com/hmendonca/lightgbm-predictions-explained-with-shap-0-796)

"""
import shap
%time shap_values = shap.TreeExplainer(model).shap_values(valid_features)
print(shap_values[0].shape)
%time shap.summary_plot(shap_values, valid_features)
"""
#### summarize the effects of top features
"""
# summarize the effects of top features
shap.summary_plot(shap_values[0], valid_features, max_display=30)    
print("Total columns to display:", len(valid_features.columns))
for i in valid_features.columns:
    shap.dependence_plot(i, shap_values[0], valid_features)
    plt.show()
"""
<a id='t6'></a>
# <div class="h1">Modeling</div>

[Back to Contents](#top)

[Next](#t6_1)

"""
train_df = dataset_small
features = list(train_df)
features.remove( 'patient_id')
features.remove( 'sars_cov_2_exam_result')
features.remove( 'patient_addmited_to_intensive_care_unit__1_yes__0_no_')
features.remove( 'patient_addmited_to_regular_ward__1_yes__0_no_')
features.remove( 'patient_addmited_to_semi_intensive_unit__1_yes__0_no_')
target = 'sars_cov_2_exam_result'
"""
<a id='t6_1'></a>
# <div class="h2">Confusion Matrix function</div>

[Back to Contents](#top)

[Next](#t6_2)

"""
# Confusion matrix 
def plot_confusion_matrix(cm, classes,
                          normalize = False,
                          title = 'Confusion matrix"',
                          cmap = plt.cm.Blues) :
    plt.imshow(cm, interpolation = 'nearest', cmap = cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation = 0)
    plt.yticks(tick_marks, classes)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])) :
        plt.text(j, i, cm[i, j],
                 horizontalalignment = 'center',
                 color = 'white' if cm[i, j] > thresh else 'black')

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
"""
<a id='t6_2'></a>
# <div class="h2">Gradient Boosting Model function</div>

[Back to Contents](#top)

[Next](#t6_3)

"""
import itertools
from scipy import interp
def gradient_boosting_model(params, folds, model='LGB',stack = False):    
    print(str(model)+' modeling...')
    start_time = timer(None)
    plt.rcParams["axes.grid"] = True
    nfold = 5
    skf = StratifiedKFold(n_splits=nfold, shuffle=True, random_state=2020)

    oof = np.zeros(len(train_df))
    mean_fpr = np.linspace(0,1,100)
    cms= []
    tprs = []
    aucs = []
    y_real = []
    y_proba = []
    recalls = []
    roc_aucs = []
    f1_scores = []
    accuracies = []
    precisions = []
    feature_importance_df = pd.DataFrame()

    i = 1
    for train_idx, valid_idx in skf.split(train_df, train_df['sars_cov_2_exam_result'].values):
        print("\nfold {}".format(i))
            
        if model == 'XGB':
            trn_data = xgb.DMatrix(train_df.iloc[train_idx][features], 
                                   label=train_df.iloc[train_idx][target].values)
            val_data = xgb.DMatrix(train_df.iloc[valid_idx][features], 
                                   label=train_df.iloc[valid_idx][target].values)

            watchlist = [(trn_data, 'train'), (val_data, 'valid')]

            clf = xgb.train(params, dtrain = trn_data, evals=watchlist, early_stopping_rounds=100, maximize=True, verbose_eval=100)
            oof[valid_idx] = clf.predict(val_data, ntree_limit=clf.best_ntree_limit)

            
        # Scores 
        roc_aucs.append(roc_auc_score(train_df.iloc[valid_idx][target].values, oof[valid_idx]))
        accuracies.append(accuracy_score(train_df.iloc[valid_idx][target].values, oof[valid_idx].round()))
        recalls.append(recall_score(train_df.iloc[valid_idx][target].values, oof[valid_idx].round()))
        precisions.append(precision_score(train_df.iloc[valid_idx][target].values, oof[valid_idx].round()))
        f1_scores.append(f1_score(train_df.iloc[valid_idx][target].values, oof[valid_idx].round()))

        # Roc curve by folds
        f = plt.figure(1)
        fpr, tpr, t = roc_curve(train_df.iloc[valid_idx][target].values, oof[valid_idx])
        tprs.append(interp(mean_fpr, fpr, tpr))
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)
        plt.plot(fpr, tpr, lw=2, alpha=0.3, label='ROC fold %d (AUC = %0.4f)' % (i,roc_auc))

        # Precion recall by folds
        g = plt.figure(2)
        precision, recall, _ = precision_recall_curve(train_df.iloc[valid_idx][target].values, oof[valid_idx])
        y_real.append(train_df.iloc[valid_idx][target].values)
        y_proba.append(oof[valid_idx])
        plt.plot(recall, precision, lw=2, alpha=0.3, label='P|R fold %d' % (i))  

        i= i+1
        
        # Confusion matrix by folds
        cms.append(confusion_matrix(train_df.iloc[valid_idx][target].values, oof[valid_idx].round()))
        

    # Metrics
    print(
            '\nCV roc score        : {0:.4f}, std: {1:.4f}.'.format(np.mean(roc_aucs), np.std(roc_aucs)),
            '\nCV accuracy score   : {0:.4f}, std: {1:.4f}.'.format(np.mean(accuracies), np.std(accuracies)),
            '\nCV recall score     : {0:.4f}, std: {1:.4f}.'.format(np.mean(recalls), np.std(recalls)),
            '\nCV precision score  : {0:.4f}, std: {1:.4f}.'.format(np.mean(precisions), np.std(precisions)),
            '\nCV f1 score         : {0:.4f}, std: {1:.4f}.'.format(np.mean(f1_scores), np.std(f1_scores))
    )
    
    # Roc plt
    f = plt.figure(1)
    plt.plot([0,1],[0,1],linestyle = '--',lw = 2,color = 'grey')
    mean_tpr = np.mean(tprs, axis=0)
    mean_auc = auc(mean_fpr, mean_tpr)
    plt.plot(mean_fpr, mean_tpr, color='blue',
             label=r'Mean ROC (AUC = %0.4f)' % (np.mean(roc_aucs)),lw=2, alpha=1)

    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title(str(model)+' ROC curve by folds')
    plt.legend(loc="lower right")
    
    # PR plt
    g = plt.figure(2)
    plt.plot([0,1],[1,0],linestyle = '--',lw = 2,color = 'grey')
    y_real = np.concatenate(y_real)
    y_proba = np.concatenate(y_proba)
    precision, recall, _ = precision_recall_curve(y_real, y_proba)
    plt.plot(recall, precision, color='blue',
             label=r'Mean P|R')
    plt.xlabel('Recall')
    plt.ylabel('Precision')
    plt.title(str(model)+' P|R curve by folds')
    plt.legend(loc="lower left")

    # Confusion maxtrix
    plt.rcParams["axes.grid"] = False
    cm = np.average(cms, axis=0)
    class_names = [0,1]
    plt.figure()
    plot_confusion_matrix(cm, 
                          classes=class_names, 
                          title= str(model).title()+' Confusion matrix [averaged/folds]')
    
    # Feat imp plt
    if model != 'XGB':
        cols = (feature_importance_df[["Feature", "importance"]]
            .groupby("Feature")
            .mean()
            .sort_values(by="importance", ascending=False)[:30].index)
        best_features = feature_importance_df.loc[feature_importance_df.Feature.isin(cols)]

        plt.figure(figsize=(10,10))
        sns.barplot(x="importance", y="Feature", data=best_features.sort_values(by="importance",ascending=False),
                edgecolor=('white'), linewidth=2, palette="rocket")
        plt.title(str(model)+' Features importance (averaged/folds)', fontsize=18)
        plt.tight_layout()
        
    # Timer end    
    timer(start_time)
    
#Timer
def timer(start_time=None):
    if not start_time:
        start_time = datetime.now()
        return start_time
    elif start_time:
        thour, temp_sec = divmod((datetime.now() - start_time).total_seconds(), 3600)
        tmin, tsec = divmod(temp_sec, 60)
        print('Time taken for Modeling: %i hours %i minutes and %s seconds.' % (thour, tmin, round(tsec, 2)))
"""
<a id='t6_3'></a>
# <div class="h2">XGBoost</div>

[Back to Contents](#top)

[Next](#t6_4)

"""
param_xgb = {
            'n_jobs' : -1, 'n_estimators' : 500, 'seed' : 1919,
            'random_state':2020, 'eval_metric':'auc' }
gradient_boosting_model(param_xgb, 5, 'XGB')
"""
<a id='t7'></a>
# <div class="h1">Ensembling With StackNet</div>

[Back to Contents](#top)

[Next](#t7_1)
![](https://github.com/kaz-Anova/StackNet/raw/master/images/StackNet_Logo.png?raw=true)
StackNet was created by Kaggle Grandmaster Marios Michailidis ([kazanova](https://www.kaggle.com/kazanova)) as part of his PhD. Thanks to [Kiran Kunapuli](https://www.kaggle.com/kirankunapuli) for uploading the package as a Kaggle dataset so it can conveniently be used with Kaggle kernels.
"""
# Machine Learning
from lightgbm import LGBMRegressor
from catboost import CatBoostRegressor
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
"""
#### Split Train and Validation
"""
# Split Train and Validation
X_train = train_df.drop('patient_id',axis=1)
X_train = X_train.drop(['sars_cov_2_exam_result','patient_addmited_to_intensive_care_unit__1_yes__0_no_','patient_addmited_to_regular_ward__1_yes__0_no_','patient_addmited_to_semi_intensive_unit__1_yes__0_no_'],axis=1)
target = train_df['sars_cov_2_exam_result']

X_train, X_val, y_train, y_val = train_test_split(X_train,
                                                  target,
                                                  test_size=0.30, 
                                                  random_state=2020, 
                                                  stratify=target)
"""
<a id='t7_2'></a>
# <div class="h2">StackNet Modeling</div>

[Back to Contents](#top)

[Next](#t7_3)
 
StackNet allows you to define all kinds of models. For example, Sklearn models, LightGBM, XGBoost, CatBoost and Keras models can all be used with StackNet.For the individual models, you are responsible for not overfitting.
"""
"""
Level 1 are the base models that take the training dataset as input
"""
# Level 1 are the base models that take the training dataset as input

l1_clf1 = LGBMRegressor(boosting_type='gbdt',
                        objective="binary",
                        metric="AUC",
                        boost_from_average="false",
                        learning_rate=0.0045,
                        num_leaves=491,
                        max_depth=20,
                        min_child_weight=0.035,
                        feature_fraction=0.38,
                        bagging_fraction=0.42,
                        min_data_in_leaf=100,
                        max_bin=255,
                        importance_type='split',
                        reg_alpha=0.4,
                        reg_lambda=0.65,
                        bagging_seed=2020,
                        random_state=2020,
                        verbosity=-1,
                        subsample=0.85,
                        colsample_bytree=0.8,
                        min_child_samples=79)

l1_clf3 = CatBoostRegressor(learning_rate=0.1,
                            bagging_temperature=0.1, 
                            l2_leaf_reg=30,
                            depth=12, 
                            max_bin=255,
                            iterations=100,
                            loss_function='Logloss',
                            objective='RMSE',
                            eval_metric="AUC",
                            bootstrap_type='Bayesian',
                            random_seed=2020,
                            early_stopping_rounds=10)


# Level 2 models will take predictions from level 1 models as input
# Remember to keep level 2 models smaller
# Basic models like Ridge Regression with large regularization or small random forests work well
l2_clf1 = RandomForestRegressor(n_estimators=250, 
                                max_depth=5, 
                                max_features='sqrt', 
                                random_state=2020)
"""
The model is compiled and fitted through the a familiar sklearn-like API. The StackNetClassifier will perform cross-validation (CV) and will output the CV scores for each model. To make sure we can output a probability of pacient result for sarscov2 exam result we specify "use_proba=True".
"""
"""
Specify model tree for StackNet
"""
# Specify model tree for StackNet
models = [[l1_clf1, l1_clf3], # Level 1
          [l2_clf1]] # Level 2
"""
Specify parameters for stacked model and begin training

"""
from pystacknet.pystacknet import StackNetClassifier

# Specify parameters for stacked model and begin training
model = StackNetClassifier(models, 
                           metric="auc", 
                           folds=5,
                           restacking=False,
                           use_retraining=True,
                           use_proba=True, # To use predict_proba after training
                           random_state=2020,
                           n_jobs=-1, 
                           verbose=1)

# Fit the entire model tree
model.fit(X_train, y_train)

# source: https://www.kaggle.com/carlolepelaars/ensembling-with-stacknet/notebook
def auc_score(y_true, y_pred):
    """
    Calculates the Area Under ROC Curve (AUC)
    """
    return roc_auc_score(y_true, y_pred)
def plot_curve(y_true_train, y_pred_train, y_true_val, y_pred_val, model_name):
    """
    Plots the ROC Curve given predictions and labels
    """
    fpr_train, tpr_train, _ = roc_curve(y_true_train, y_pred_train, pos_label=1)
    fpr_val, tpr_val, _ = roc_curve(y_true_val, y_pred_val, pos_label=1)
    plt.figure(figsize=(8, 8))
    plt.plot(fpr_train, tpr_train, color='black',
             lw=2, label=f"ROC train curve (AUC = {round(roc_auc_score(y_true_train, y_pred_train), 4)})")
    plt.plot(fpr_val, tpr_val, color='darkorange',
             lw=2, label=f"ROC validation curve (AUC = {round(roc_auc_score(y_true_val, y_pred_val), 4)})")
    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
    plt.xlim([-0.05, 1.05])
    plt.ylim([-0.05, 1.05])
    plt.xlabel('False Positive Rate', fontsize=16)
    plt.ylabel('True Positive Rate', fontsize=16)
    plt.xticks(fontsize=14)
    plt.yticks(fontsize=14)
    plt.title(f'ROC Plot for {model_name}', weight="bold", fontsize=20)
    plt.legend(loc="lower right", fontsize=16)
"""
### Get score on training set and validation set for our StackNetClassifier
"""
# Get score on training set and validation set for our StackNetClassifier
train_preds = model.predict_proba(X_train)[:, 1]
val_preds = model.predict_proba(X_val)[:, 1]
train_score = auc_score(y_train, train_preds)
val_score = auc_score(y_val, val_preds)
"""
<a id='t7_3'></a>
# <div class="h2">Evaluation StackNet ROC_AUC</div>

[Back to Contents](#top)

[Next](#t8)


The blue line signifies the baseline AUC which is 0.5. The final validation score is the area under the orange curve, which is mentioned in the plot
"""
print(f"StackNet AUC on training set: {round(train_score, 4)}")
print(f"StackNet AUC on validation set: {round(val_score, 4)}")
# Plot ROC curve
plot_curve(y_train, train_preds, y_val, val_preds, "StackNet Baseline")
"""
probabilities of sarscov2 exam result
"""
df_cov19pos = pd.DataFrame(val_preds, columns=['prob_sars-cov-2_exam_result'])
df_cov19pos['sars-cov-2_exam_result'] = y_val.values
df_cov19pos['patient_id'] = X_val.index
df_cov19pos.head()
"""
#### Classification report train
"""
from sklearn.metrics import classification_report
train_cm = confusion_matrix(y_train,train_preds.round())
print('Confusion matrix: \n',train_cm)
print('Classification report: \n',classification_report(y_train, train_preds.round()))
# visualize with seaborn library
sns.heatmap(train_cm,annot=True,fmt="d") 
plt.show()
"""
# Determining the Threshold for sars-cov2 exam positive result 

It is necessary to define a value to help the model make decisions for sars-cov2 exam positive result.
To find the ideal threshold that classifies Positive and Negative  sars-cov2 exam.  It's elaborated the heuristic,
, first, we found the proportion **p** of cases that were for sars-cov2 exam positive result in the entire dataset, **~9.99%**, the final result its the highest prob proportion correspondent. The central idea is that the model should output positive (sars-cov-2_exam_result Positive) cases in the same proportion as **p**.
"""
"""
###  9.8866% of the cases are Positive for SarsCov2 Exam
"""
dataset['sars-cov-2_exam_result'].value_counts() / dataset.shape[0]
#  0.098866% of the cases are Positive for SarsCov2 Exam
grouped = df_cov19pos.groupby('prob_sars-cov-2_exam_result').sum().reset_index()
grouped.sort_values('prob_sars-cov-2_exam_result', ascending=False)
"""
<a id='t8'></a>
# <div class="h1">General Findings</div>

[Back to Contents](#top)

[End Notebook](#end)

"""
"""
### Useful links: 
- [Classification: Precision and Recall](https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall)
- [Classification: True vs. False and Positive vs. Negative](https://developers.google.com/machine-learning/crash-course/classification/true-false-positive-negative)
- [What is LightGBM, How to implement it? How to fine tune the parameters?](https://towardsdatascience.com/boosting-techniques-in-python-predicting-hotel-cancellations-62b7a76ffa6c)
- [XGBoost and Imbalanced Classes](https://towardsdatascience.com/boosting-techniques-in-python-predicting-hotel-cancellations-62b7a76ffa6c)
 <hr>
"""
HTML('<iframe width="800" height="500" src="https://www.youtube.com/embed/HXmt0j1gtDU" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
"""
Live Coronavirus from 03/22 - Why is it important to stay home?
"""
HTML('<iframe width="800" height="500" src="https://www.youtube.com/embed/zF2pXXJIAGM" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
"""
Live Coronavirus from 03/20 - What Brazil needs to do in the coming days
"""
HTML('<iframe width="800" height="500" src="https://www.youtube.com/embed/7jHgS4yxS0A" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
"""
Live Coronavirus of 03/18 - How the pandemic can happen in Brazil
"""
HTML('<iframe width="800" height="500" src="https://www.youtube.com/embed/2DAIe1SlLMo" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')
"""
A situation in Brazil can be better #StayHome
"""
"""
<a class="anchor" id="end"></a>
<a id='end'></a>
# <div class="h1"> End Notebook</div>
[Back to Contents](#top)
"""