# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # **Movie Dataset**

# The movie dataset on which this Project is based is a database of 5000 movies catalogued by The Movie Database (TMDb). The information available about each movie is its budget, revenue, rating, actors and actresses, etc. In this case study, we will use this dataset to determine whether any information about a movie can predict the total revenue of a movie. We will also attempt to predict whether a movie's revenue will exceed its budget.
#
# In this Project, we will inspect, clean, and transform the data.
# We will use Linear and logistic regression and random forests to perform prediction and classification. We will use these methods to predict revenue, and we will use logistic regression to classify whether a movie was profitable.

# First, we will import several libraries. scikit-learn (sklearn) contains helpful statistical models, and we'll use the matplotlib.pyplot library for visualizations. Of course, we will use numpy and pandas for data manipulation throughout.

# +
import pandas as pd
import numpy as np

from sklearn.model_selection import cross_val_predict
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import r2_score
          positive_revenue_df = df[df["revenue"] > 0]

regression_outcome = positive_revenue_df[regression_target]
classification_outcome = positive_revenue_df[classification_target]
covariates = positive_revenue_df[all_covariates]

linear_regression = LinearRegression()
logistic_regression = LogisticRegression()
forest_regression = RandomForestRegressor(max_depth=4, random_state=0)
forest_classifier = RandomForestClassifier(max_depth=4, random_state=0)
linear_regression_scores = cross_val_score(linear_regression, covariates, regression_outcome, cv=10, scoring=correlation)
forest_regression_scores = cross_val_score(forest_regression, covariates, regression_outcome, cv=10, scoring=correlation)
logistic_regression_scores = cross_val_score(logistic_regression, covariates, classification_outcome, cv=10, scoring=accuracy)
forest_classification_scores = cross_val_score(forest_classifier, covariates, classification_outcome, cv=10, scoring=accuracy)

np.mean(forest_regression_scores)
import matplotlib.pyplot as plt

df = pd.read_csv("https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@movie_data.csv", index_col=0)
df.head()
# -

# We will define the regression and classification outcomes. Specifically, we will use the revenue column as the target for regression. For classification, we will construct an indicator of profitability for each movie.
#
#
# * Create a new column in df called profitable, defined as 1 if the movie revenue (revenue) is greater than the movie budget (budget), and 0 otherwise.
# * Next, define and store the outcomes we will use for regression and classification. Define regression_target as the string 'revenue'. Define classification_target as the string 'profitable'.
#

# +
df['profitable'] = df.revenue > df.budget
df['profitable'] = df['profitable'].astype(int)

regression_target = 'revenue'
classification_target = 'profitable'

df['profitable'].value_counts()
# -

# We will proceed by analyzing only the rows without any missing data. We will remove rows with any infinite or missing values.
# * Use df.replace() to replace any cells with type np.inf or -np.inf with np.nan.
# * Drop all rows with any np.nan values in that row using df.dropna().

# +
df = df.replace([np.inf, -np.inf], np.nan)
df = df.dropna(how="any")

df.shape
# -

# Many of the variables in our dataframe contain the names of genre, actors/actresses, and keywords. Let's add indicator columns for each genre.
#
# * Determine all the genres in the genre column. Make sure to use the strip() function on each genre to remove trailing characters.
# * Next, include each listed genre as a new column in the dataframe. Each element of these genre columns should be 1 if the movie belongs to that particular genre, and 0 otherwise. Keep in mind that a movie may belong to several genres at once.
# * Call df[genres].head() to view your results.
#

# +
list_genres = df.genres.apply(lambda x: x.split(","))
genres = []
for row in list_genres:
    row = [genre.strip() for genre in row]
    for genre in row:
        if genre not in genres:
            genres.append(genre)

for genre in genres:
    df[genre] = df['genres'].str.contains(genre).astype(int)

df[genres].head()
# -

# Some variables in the dataset are already numeric and perhaps useful for regression and classification. We will store the names of these variables for future use. We will also take a look at some of the continuous variables and outcomes by plotting each pair in a scatter plot. Finally, we will evaluate the skew of each variable.
#
# * Call plt.show() to observe the plot generated by the code given below. Which of the covariates and/or outcomes are correlated with each other?
# * Call skew() on the columns outcomes_and_continuous_covariates in df. 
#

# +
continuous_covariates = ['budget', 'popularity', 'runtime', 'vote_count', 'vote_average']
outcomes_and_continuous_covariates = continuous_covariates + [regression_target, classification_target]
plotting_variables = ['budget', 'popularity', regression_target]

axes = pd.plotting.scatter_matrix(df[plotting_variables], alpha=0.15, \
       color=(0,0,0), hist_kwds={"color":(0,0,0)}, facecolor=(1,0,0))
# show the plot.
plt.show()

print(df[outcomes_and_continuous_covariates].skew())
# -

# It appears that the variables budget, popularity, runtime, vote_count, and revenue are all right-skewed. We will transform these variables to eliminate this skewness. Specifically, we will use the np.log10() method. Because some of these variable values are exactly 0, we will add a small positive value to each to ensure it is defined; this is necessary because log(0) is negative infinity.
#
# For each above-mentioned variable in df, transform value x into np.log10(1+x)

# +
for covariate in ['budget', 'popularity', 'runtime', 'vote_count', 'revenue']:
    df[covariate] = df[covariate].apply(lambda x: np.log10(1+x))
    
print(df[outcomes_and_continuous_covariates].skew())
# -

# Now we're going to save our dataset to use this later.
# * Use to_csv() to save the df object as movies_clean.csv.

df.to_csv("movies_clean.csv")

# +
import pandas as pd
import numpy as np

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import accuracy_score
from sklearn.metrics import r2_score

import matplotlib.pyplot as plt

import warnings
warnings.filterwarnings("ignore")

# EDIT THIS CODE TO LOAD THE SAVED DF FROM THE LAST HOMEWORK
df = pd.read_csv('movies_clean.csv')
# -

# We will instantiate regression and classification models. Code is provided that prepares the covariates and outcomes we will use for data analysis.
#
# * Instantiate LinearRegression(), LogisticRegression(), RandomForestRegressor(), and RandomForestClassifier() objects, and assign them to linear_regression, logistic_regression, forest_regression, and forest_classifier, respectively.
# * For the random forests models, specify max_depth=4 and random_state=0
#

# +
      # Define all covariates and outcomes from `df`.
regression_target = 'revenue'
classification_target = 'profitable'
all_covariates = ['budget', 'popularity', 'runtime', 'vote_count', 'vote_average', 'Action', 'Adventure', 'Fantasy', 
                  'Science Fiction', 'Crime', 'Drama', 'Thriller', 'Animation', 'Family', 'Western', 'Comedy', 'Romance', 
                  'Horror', 'Mystery', 'War', 'History', 'Music', 'Documentary', 'TV Movie', 'Foreign']

regression_outcome = df[regression_target]
classification_outcome = df[classification_target]
covariates = df[all_covariates]

# Instantiate all regression models and classifiers.
linear_regression = LinearRegression()
logistic_regression = LogisticRegression()
forest_regression = RandomForestRegressor(max_depth=4, random_state=0)
forest_classifier = RandomForestClassifier(max_depth=4, random_state=0)


# -

# We will create two functions that compute a model's score. For regression models, we will use correlation as the score. For classification models, we will use accuracy as the score.
#
# * Define a function called correlation with arguments estimator, X, and y. The function should compute the correlation between the observed outcome y and the outcome predicted by the model.
# * To obtain predictions, the function should first use the fit method of estimator and then use the predict method from the fitted object.
# * The function should return the first argument from r2_score comparing predictions and y.
# * Define a function called accuracy with the same arguments and code, substituting accuracy_score for r2_score.
#

# +
def correlation(estimator, X, y):
    predictions = estimator.fit(X, y).predict(X)
    return r2_score(y, predictions)
    
def accuracy(estimator, X, y):
    predictions = estimator.fit(X, y).predict(X)
    return accuracy_score(y, predictions)


# -

# Compute the cross-validated performance for the linear and random forest classification models.
#
# * Call cross_val_score using logistic_regression and forest_classifier as models. Store the output as logistic_regression_scores and forest_classification_scores, respectively.
# * Set the parameters cv=10 to use 10-fold cross-validation and scoring=correlation to use the accuracy function defined.
# * Plotting code has been provided to compare the performance of the two models. Use plt.show() to plot the accuracy of predicted profitability for each cross-validation fold using the logistic and random forest classification models.
# * Consider which of the two models exhibits a better fit.
#

# +
# Determine the cross-validated correlation for linear and random forest models.
linear_regression_scores = cross_val_score(linear_regression, covariates, regression_outcome, cv=10, scoring=correlation)
forest_regression_scores = cross_val_score(forest_regression, covariates, regression_outcome, cv=10, scoring=correlation)

# Plot Results
plt.axes().set_aspect('equal', 'box')
plt.scatter(linear_regression_scores, forest_regression_scores)
plt.plot((0, 1), (0, 1), 'k-')

plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Linear Regression Score")
plt.ylabel("Forest Regression Score")
plt.show()
        
# -

# Cross-validated performance for the linear and random forest classification models.
#
# * Call cross_val_score using logistic_regression and forest_classifier as models. Store the output as logistic_regression_scores and forest_classification_scores, respectively.
# * Set the parameters cv=10 to use 10-fold cross-validation and scoring=correlation to use the accuracy function defined.
# * Plotting code has been provided to compare the performance of the two models. Use plt.show() to plot the accuracy of predicted profitability for each cross-validation fold using the logistic and random forest classification models.
# * Consider which of the two models exhibits a better fit.
#

# +
logistic_regression_scores = cross_val_score(logistic_regression, covariates, classification_outcome, cv=10, scoring=accuracy)
forest_classification_scores = cross_val_score(forest_classifier, covariates, classification_outcome, cv=10, scoring=accuracy)

plt.axes().set_aspect('equal', 'box')
plt.scatter(logistic_regression_scores, forest_classification_scores)
plt.plot((0, 1), (0, 1), 'k-')

plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Linear Classification Score")
plt.ylabel("Forest Classification Score")

plt.show()
# -

# We will rerun the regression analysis for this subsetted dataset.
#
# * Define positive_revenue_df as the subset of movies in df with revenue greater than zero.
# * Code is provided below that creates new instances of model objects. Replace all instances of df with positive_revenue_df, and run the given code.
#

# +
positive_revenue_df = df[df["revenue"] > 0]

# Replace the dataframe in the following code, and run.
regression_outcome = positive_revenue_df[regression_target]
classification_outcome = positive_revenue_df[classification_target]
covariates = positive_revenue_df[all_covariates]

# Reinstantiate all regression models and classifiers.
linear_regression = LinearRegression()
logistic_regression = LogisticRegression()
forest_regression = RandomForestRegressor(max_depth=4, random_state=0)
forest_classifier = RandomForestClassifier(max_depth=4, random_state=0)
linear_regression_scores = cross_val_score(linear_regression, covariates, regression_outcome, cv=10, scoring=correlation)
forest_regression_scores = cross_val_score(forest_regression, covariates, regression_outcome, cv=10, scoring=correlation)
logistic_regression_scores = cross_val_score(logistic_regression, covariates, classification_outcome, cv=10, scoring=accuracy)
forest_classification_scores = cross_val_score(forest_classifier, covariates, classification_outcome, cv=10, scoring=accuracy)

np.mean(forest_regression_scores)
# -

# Compute the cross-validated performance for the logistic regression and random forest classification models for positive revenue movies only.
#
# * Call cross_val_score using logistic_regression and forest_classifier as models. Store the output as logistic_regression_scores and forest_classification_scores, respectively.
# * Set the parameters cv=10 to use 10-fold cross-validation and scoring=correlation to use the accuracy function defined.
# * Plotting code has been provided to compare the performance of the two models. Use plt.show() to plot the correlation between actual and predicted revenue for each cross-validation fold using the logistic regression and random forest classification models. Consider which of the two models exhibits a better fit. Is this result different from what we observed when considering all movies?
# * Code is provided for you that prints the importance of each covariate in predicting revenue using the random forest classifier. Consider which variables are the most important.
#

# +
# Determine the cross-validated accuracy for logistic and random forest models.
logistic_regression_scores = cross_val_score(logistic_regression, covariates, classification_outcome, cv=10, scoring=accuracy)
forest_classification_scores = cross_val_score(forest_classifier, covariates, classification_outcome, cv=10, scoring=accuracy)

# Plot Results
plt.axes().set_aspect('equal', 'box')
plt.scatter(logistic_regression_scores, forest_classification_scores)
plt.plot((0, 1), (0, 1), 'k-')
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel("Linear Classification Score")
plt.ylabel("Forest Classification Score")

plt.show();

# Print the importance of each covariate in the random forest classification.
forest_classifier.fit(positive_revenue_df[all_covariates], positive_revenue_df[classification_target])
sorted(list(zip(all_covariates, forest_classifier.feature_importances_)), key=lambda tup: tup[1])
# -





StarcoderdataJupyterScriptsDedupFiltered