df = unpickle_object("final_dataframe_for_analysis.pkl") #dataframe we got from webscraping and cleaning!
#see other notebooks for more info.

df.dtypes # there are all our features. Our target variable is Box_office

df.shape

"""
Explanation: The objective of this notebook, and more broadly, this project is to see whether we can discern a linear relationship between metrics found on Rotton Tomatoes and Box Office performance.
Box office performance is measured in millions as is budget.
Because we have used scaling, interpretation of the raw coefficients will be difficult. Luckily, sklearns standard scaler has an inverse_transform method, thus, if we had to, we could reverse transform the coefficients (sc_X_train for the holdout group and sc_x for the non-holdout group) to get some interpretation. The same logic follows for interpreting target variables should we use the model for prediction.
The year, country, language and month will all be made into dummy variables. I will do this with the built in pd.get_dummies() function! This will turn all columns into type object into dummies! I will also use the optional parameter drop_first to avoid the dummy variable trap!
I will use sklearn's standard scaler on all of my variables, except for the dummies! This is important since we will be using regularized regression i.e. Lasso, Ridge, Elastic Net
I will shuffle my dataframe before the train, test split. I will utilise the X_train, y_train, X_test and y_test variables in GridsearchCV. This is an example of employing cross-validation with a holdout set. This will help guard against overfitting.
To be truly honest, I do not have enough data to justify using a hold out set, however, I want to implement it as an academic exercise! It also give me more code to write!
I will then re-implement the models with using the hold out set to compare results!
Let's get to it!
CROSS-VALIDATION WITH HOLDOUT SECTION
End of explanation
"""


df['Month'] = df['Month'].astype(object)
df['Year'] = df['Year'].astype(object)
del df['Rank_in_genre']
df.reset_index(inplace=True)
del df['index']

percentage_missing(df)

df.hist(layout=(4,2), figsize=(50,50))

"""
Explanation: Upon further thought, it doesnt make sense to have rank_in_genre as a predictor variable for box office budget. When the movie is release, it is not ranked immeadiately. The ranks assigned often occur many years after the movie is released and so it not related to the amount of money accrued at the box office. We will drop this variable.
Right now, our index is the name of the movie! We dont need these to be indicies, it would be cleaner to have a numeric index
The month and year columns are currently in numerical form, however, for our analysis, we require that these be of type object!
End of explanation
"""


plot_corr_matrix(df)

X = unpickle_object("X_features_selection.pkl") #all features from the suffled dataframe. Numpy array
y = unpickle_object("y_variable_selection.pkl") #target variable from shuffled dataframe. Numpy array
final_df = unpickle_object("analysis_dataframe.pkl") #this is the shuffled dataframe!

"""
Explanation: From the above plots, we see that we have heavy skewness in all of our features and our target variable.
The features will be scaled using standard scaler.
When splitting the data into training and test. I will fit my scaler according to the training data!
There is no sign of multi-collinearity $(>= 0.9)$ - good to go!
End of explanation
"""


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state = 0) #train on 75% of data

sc_X_train = StandardScaler()
sc_y_train = StandardScaler()

sc_X_train.fit(X_train[:,:6])#only need to learn fit of first 6 - rest are dummies
sc_y_train.fit(y_train)

X_train[:,:6] = sc_X_train.transform(X_train[:,:6]) #only need to transform first 6 columns - rest are dummies
X_test[:,:6] = sc_X_train.transform(X_test[:,:6]) #same as above

y_train = sc_y_train.transform(y_train)
y_test = sc_y_train.transform(y_test)

"""
Explanation: Baseline Model and Cross-Validation with Holdout Sets
Creation of Holdout Set
End of explanation
"""


baseline_model(X_train, X_test, y_train, y_test)

"""
Explanation: Baseline Model
As we can see - the baseline model of regular linear regression is dreadful! Let's move on to more sophisticated methods!
End of explanation
"""


holdout_results = holdout_grid(["Ridge", "Lasso", "Elastic Net"], X_train, X_test, y_train, y_test)

pickle_object(holdout_results, "holdout_model_results")

"""
Explanation: Ridge, Lasso and Elastic Net regression - Holdouts
End of explanation
"""


sc_X = StandardScaler()
sc_y = StandardScaler()

sc_X.fit(X[:,:6])#only need to learn fit of first 6 - rest are dummies
sc_y.fit(y)

X[:,:6] = sc_X.transform(X[:,:6]) #only need to transform first 6 columns - rest are dummies
y = sc_y.transform(y)

no_holdout_results = regular_grid(["Ridge", "Lasso", "Elastic Net"], X, y)

pickle_object(no_holdout_results, "no_holdout_model_results")

"""
Explanation: Cross-Validation - No Holdout Sets
End of explanation
"""


extract_model_comparisons(holdout_results, no_holdout_results, "Ridge")

"""
Explanation: Analysis of Results!
Ridge Analysis
End of explanation
"""


extract_model_comparisons(holdout_results, no_holdout_results, "Lasso")

"""
Explanation: Lasso Analysis
End of explanation
"""


extract_model_comparisons(holdout_results, no_holdout_results, "Elastic Net")

"""
Explanation: Elastic Net Analysis
End of explanation
"""


