plt.scatter(train['avg_blocktime_6'], train['avg_blocktime_60'])

"""
Explanation: Columns to remove:
Possible leakage: avg_price_6, avg_price_60, avg_gasUsed_b_6, avg_gasUsed_t_6,  avg_gasUsed_b_60, avg_gasUsed_t_60, avg_price_60
Poor performance: gasLimit_t, gasUsed_t, newContract, avg_uncle_count_6, avg_txcnt_second_6, type_enc, avg_uncle_count_60
Candidate features:

avg_difficulty_60, r2=0.99
avg_tx_count_60, r2=0.5
avg_blocktime_6, 
avg_tx_count_6, r2=0.99
avg_difficulty_6, 
avg_blocktime_60
avg_txcnt_second_60, r2=0.12
mv, r2=0.75
End of explanation
"""


# select features
sub_cols = ['gasLimit_t', 
            'gasUsed_t', 
            'newContract', 
       'avg_blocktime_6', 
            'avg_tx_count_6',
       'avg_uncle_count_6', 
            'avg_difficulty_6', 
            'avg_txcnt_second_6',
       'avg_gasUsed_t_6', 
            'avg_price_6', 
            'avg_blocktime_60',
       'avg_gasUsed_b_60', 
            'avg_uncle_count_60',
       'avg_difficulty_60', 
            'avg_txcnt_second_60', 
            'avg_gasUsed_t_60',
        'avg_price_60', 
            'mv', 
            'type_enc']

sub_train = train[sub_cols]
X = sub_train.values
y = y_label
X_train, X_test, y_train, y_test = train_test_split(X, y)

matrix_rank(X), len(sub_cols)

X.shape, y.shape

"""
Explanation: Run decision tree
End of explanation
"""


for i, col in enumerate(sub_train.columns):
    print('VIF col {}: {}'.format(col,variance_inflation_factor(X,i)))

dt = tree.DecisionTreeRegressor()
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)

r2_score(y_test, y_pred)

mean_squared_error(y_test, y_pred)

print('Mean CV r2_score: {}'.format(np.mean(cross_val_score(dt, X_test, y_test, scoring='r2', cv=3))))

plt.scatter(y_test, y_pred)
#plt.xlim(0,1000)
#plt.ylim(0,1000)
plt.xlabel('y_test')
plt.ylabel('y_pred')

"""
Explanation: Check variance inflation factors
If the VIF is equal to 1 there is no multicollinearity among factors, but if the VIF is greater than 1, the predictors may be moderately correlated. The output above shows that the VIF for the Publication and Years factors are about 1.5, which indicates some correlation, but not enough to be overly concerned about. A VIF between 5 and 10 indicates high correlation that may be problematic. And if the VIF goes above 10, you can assume that the regression coefficients are poorly estimated due to multicollinearity.
End of explanation
"""


# Create a scaler object
#sc = StandardScaler()

# Fit the scaler to the feature data and transform
#X_std = sc.fit_transform(X)

"""
Explanation: Nested cross validation
Often we want to tune the parameters of a model (for example, C in a support vector machine). That is, we want to find the value of a parameter that minimizes our loss function. The best way to do this is cross validation:

Set the parameter you want to tune to some value.
Split your data into K 'folds' (sections).
Train your model using K-1 folds using the parameter value.
Test your model on the remaining fold.
Repeat steps 3 and 4 so that every fold is the test data once.
Repeat steps 1 to 5 for every possible value of the parameter.
Report the parameter that produced the best result.

However, as Cawley and Talbot point out in their 2010 paper, since we used the test set to both select the values of the parameter and evaluate the model, we risk optimistically biasing our model evaluations. For this reason, if a test set is used to select model parameters, then we need a different test set to get an unbiased evaluation of that selected model.
One way to overcome this problem is to have nested cross validations. First, an inner cross validation is used to tune the parameters and select the best model. Second, an outer cross validation is used to evaluate the model selected by the inner cross validation.
Standardize data
End of explanation
"""


# Create a list of 10 candidate values for the C parameter
#max_depth_candidates = dict(max_depth=np.arange(1, 7, 1))

# Create a gridsearch object with the decision tree regressor and the max_depth value candidates
#reg = GridSearchCV(estimator=tree.DecisionTreeRegressor(), param_grid=max_depth_candidates)

"""
Explanation: Create Inner Cross Validation (For Parameter Tuning)
This is our inner cross validation. We will use this to hunt for the best parameters for C, the penalty for misclassifying a data point. GridSearchCV will conduct steps 1-6 listed at the top of this tutorial.
End of explanation
"""


# Fit the cross validated grid search on the data 
#reg.fit(X_std, y)

# Show the best value for C
#reg.best_estimator_.max_depth

"""
Explanation: The code below isn't necessary for parameter tuning using nested cross validation, however to demonstrate that our inner cross validation grid search can find the best value for the parameter C, we will run it once here:
End of explanation
"""


#print('Mean CV r2_score: {}'.format(np.mean(cross_val_score(reg, X_std, y, scoring='r2', cv=3))))

"""
Explanation: Create Outer Cross Validation (For Model Evaluation)
With our inner cross validation constructed, we can use cross_val_score to evaluate the model with a second (outer) cross validation.
The code below splits the data into three folds, running the inner cross validation on two of the folds (merged together) and then evaluating the model on the third fold. This is repeated three times so that every fold is used for testing once.
End of explanation
"""


