"""
# MOVIE RECOMMENDATION

## Author: Max Duong

### Date: April, 2021
"""
"""
# CREDITS

This work is inspired by multiple great sources done before:
- https://www.kaggle.com/rounakbanik/movie-recommender-systems
- https://www.kaggle.com/ibtesama/getting-started-with-a-movie-recommendation-system
- https://www.udacity.com/course/data-scientist-nanodegree--nd025
"""
"""
# INTRODUCTION

<a href="https://ibb.co/3RKmCvX"><img src="https://i.ibb.co/Vw6Yqxy/netflix.jpg" alt="netflix" border="0"></a>

The rapid growth of data collection has led to a new era of information. Data is being used to create more efficient systems and this is where Recommendation Systems come into play. Recommendation Systems are a type of information filtering systems as they improve the quality of search results and provides items that are more relevant to the search item or are realted to the search history of the user.
They are used to predict the rating or preference that a user would give to an item. Amazon uses it to suggest products to customers, Facebook uses it to recommend pages to like and people to follow. Moreover YouTube, Netflix, Spotify use it to decide which video to play next on autoplay.


### Let's explore the overview of the recommender systems
There are basically three types of recommender systems:

> *  **Demographic Filtering**- They offer generalized recommendations to every user, based on movie popularity and/or genre. The System recommends the same movies to users with similar demographic features. Since each user is different , this approach is considered to be too simple. The basic idea behind this system is that movies that are more popular and critically acclaimed will have a higher probability of being liked by the average audience.

> *  **Content Based Filtering**- They suggest similar items based on a particular item. This system uses item metadata, such as genre, director, description, actors, etc. for movies, to make these recommendations. The general idea behind these recommender systems is that if a person liked a particular item, he or she will also like an item that is similar to it.

> *  **Collaborative Filtering**- This system matches persons with similar interests and provides recommendations based on this matching. Collaborative filters do not require item metadata like its content-based counterparts.
"""
"""
# 1. LIBRARY
"""
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

from scipy import stats
from ast import literal_eval

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from sklearn.metrics.pairwise import linear_kernel, cosine_similarity

from nltk.stem.snowball import SnowballStemmer
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.corpus import wordnet

from surprise import Reader, Dataset, accuracy
from surprise import SVD, KNNBasic 
from surprise.model_selection import cross_validate
from surprise.model_selection import train_test_split

import warnings; warnings.simplefilter('ignore')
%matplotlib inline
"""
# 2. DATA EXTRACTION
"""
md = pd. read_csv('../input/the-movies-dataset/movies_metadata.csv')
links_small = pd.read_csv('../input/the-movies-dataset/links_small.csv')
credits = pd.read_csv('../input/the-movies-dataset/credits.csv')
keywords = pd.read_csv('../input/the-movies-dataset/keywords.csv')
ratings = pd.read_csv('../input/the-movies-dataset/ratings_small.csv')
"""
# 3. DATA EXPLORATION
"""
features = ['id', 'imdb_id', 'title', 
            'genres', 'overview', 'tagline', 'release_date', 
            'popularity', 'vote_average', 'vote_count']
md = md[features]
md[:1]
features = ['id', 'cast', 'crew']
credits = credits[features]
credits[:1]
keywords[:1]
features = ['tmdbId', 'movieId', 'imdbId']
links_small = links_small[features]
links_small.columns = ['id', 'movieId', 'imdbId']
links_small[:1]
ratings[:1]
"""
# 4. DATA MANIPULATION
"""
"""
#### Clean id
"""
def clean_id(x):
    try:
        x = int(x)
    except:
        x = np.NaN
    return x
md['id'] = md['id'].apply(clean_id)
md.dropna(subset=['id'], inplace=True)
df = pd.merge(md, credits, how='inner', on='id')
df = pd.merge(df, keywords, how='inner', on='id')
df[:1]
"""
### Content Based Rec - Plot Description Recs - Cleaning
"""
df['tagline'] = df['tagline'].fillna('')
df['overview'] = df['overview'].fillna('')
df['description'] = df['overview'] + df['tagline']
"""
### Content Based Rec - Metadata Recs - Cleaning
The next steps are the same as what we did with our plot description based recommender. One important difference is that we use the CountVectorizer() instead of TF-IDF. This is because we do not want to down-weight the presence of an actor/director if he or she has acted or directed in relatively more movies. It doesn't make much intuitive sense.
"""
"""
#### Clean genres
"""
df['genres'] = df['genres'].fillna('[]')
df['genres'] = df['genres'].apply(literal_eval)
df['genres'] = df['genres'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])
def genres_list(df_genres):
    genres = set()
        
    for genres_list in df_genres:
        try:
            genres.update(genres_list)
        except AttributeError:
            pass
    
    return genres
    
genres = genres_list(df['genres'])
print(genres)
def split_genres(val):
    try:
        if gene in val:
            return 1
        else:
            return 0
    except AttributeError:
        return 0

# Apply function for each genre
for gene in genres:        
    df[gene] = df['genres'].apply(split_genres)
"""
#### Clean year
"""
def clean_year(x):
    if x != np.nan:
        year = str(x).split('-')[0]
        return year
    else:
        return np.NaN
    
df.dropna(subset=['release_date'], inplace=True)
df['year'] = df['release_date'].apply(clean_year)
df = df.drop(['release_date'], axis=1)
df.dropna(subset=['year'], inplace=True)
df['year'] = df['year'].astype(int)
def add_movie_year_period(movie_year):
    if movie_year < 1900:
        return '1800'
    elif movie_year < 2000:
        return '1900'
    elif movie_year < 2010:
        return '2000'
    else :
        return '2010'
df['year_period'] = df['year'].apply(add_movie_year_period)
"""
#### Clean cast, crew, keywords
"""
df['cast'] = df['cast'].apply(literal_eval)
df['crew'] = df['crew'].apply(literal_eval)
df['keywords'] = df['keywords'].apply(literal_eval)
def get_director(x):
    for i in x:
        if i['job'] == 'Director':
            return i['name']
    return np.NaN
df['director'] = df['crew'].apply(get_director)
df['director'] = df['director'].astype('str').apply(lambda x: str.lower(x.replace(" ", "")))
df['director'] = df['director'].apply(lambda x: [x, x, x])

df['cast'] = df['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])
df['cast'] = df['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)
df['cast'] = df['cast'].apply(lambda x: [str.lower(i.replace(" ", "")) for i in x])
df['keywords'] = df['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])
dictionary = {}

def count_words(word_list):
    for word in word_list:
        if dictionary.get(word) == None:
            dictionary[word] = 1
        else:
            dictionary[word] += 1


df['keywords'].apply(count_words)


dictionary_copy = dictionary.copy()
for key, value in dictionary_copy.items():
    if value == 1:
        dictionary.pop(key)
def filter_keywords(word_list):
    words = []
    for word in word_list:
        if dictionary.get(word):
            words.append(word)
    return words
stemmer = SnowballStemmer('english')
stemmer.stem('Dogs')
df['keywords'] = df['keywords'].apply(filter_keywords)
df['keywords'] = df['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])
df['keywords'] = df['keywords'].apply(lambda x: [str.lower(i.replace(" ", "")) for i in x])
"""
# 5. RECOMMENDER SYSTEM TECHNIQUES
"""
"""
## 5.1 Demographic Recommender / Simple Recommender

   Before getting started with this  -
* we need a metric to score or rate movie 
* Calculate the score for every movie 
* Sort the scores and recommend the best rated movie to the users.

We can use the average ratings of the movie as the score but using this won't be fair enough since a movie with 8.9 average rating and only 3 votes cannot be considered better than the movie with 7.8 as as average rating but 40 votes.
So, I'll be using IMDB's weighted rating (wr) which is given as :-

![](https://image.ibb.co/jYWZp9/wr.png)
where,
* v is the number of votes for the movie;
* m is the minimum votes required to be listed in the chart;
* R is the average rating of the movie; And
* C is the mean vote across the whole report

We already have v(**vote_count**) and R (**vote_average**) and C can be calculated as 
"""
def weighted_rating(x, C, m):
    v = x['vote_count']
    R = x['vote_average']
    return (v/(v+m) * R) + (m/(m+v) * C)

def build_chart(genre, percentile=0.85):
    df_genre = df[df[genre] == 1]
    
    vote_counts = df_genre[df_genre['vote_count'].notnull()]['vote_count'].astype('int')
    vote_averages = df_genre[df_genre['vote_average'].notnull()]['vote_average'].astype('int')
    C = vote_averages.mean()
    m = vote_counts.quantile(percentile)
    
    features = ['title', 'year', 'vote_count', 'vote_average', 'popularity']
    qualified = df_genre[(df_genre['vote_count'] >= m) & (df_genre['vote_count'].notnull()) & (df_genre['vote_average'].notnull())][features]
    
    # qualified['wr'] = qualified.apply(lambda x: (x['vote_count']/(x['vote_count']+m) * x['vote_average']) + (m/(m+x['vote_count']) * C), axis=1)
    qualified['wr'] = qualified.apply(lambda x: weighted_rating(x, C, m), axis=1)
    qualified = qualified.sort_values('wr', ascending=False)[:50]
    
    return qualified
build_chart('Romance').head(5)
"""
## 5.2 Content Based: 
In this recommender system the content of the movie (overview, cast, crew, keyword, tagline etc) is used to find its similarity with other movies. Then the movies that are most likely to be similar are recommended.


### 5.2.1 Movie Description Based Recommender

![](https://image.ibb.co/f6mDXU/conten.png)

We will compute pairwise similarity scores for all movies based on their plot descriptions and recommend movies based on that similarity score.


**TF * IDF** is used to indicate the overall importance of each word to the documents. This will give you a matrix where each column represents a word in the overview vocabulary (all the words that appear in at least one document) and each row represents a movie, as before.This is done to reduce the importance of words that occur frequently in plot overviews and therefore, their significance in computing the final similarity score.

We will be using the cosine similarity to calculate a numeric quantity that denotes the similarity between two movies. We use the cosine similarity score since it is independent of magnitude and is relatively easy and fast to calculate. Mathematically, it is defined as follows:
<a href="https://imgbb.com/"><img src="https://i.ibb.co/nB1x1MG/4.png" alt="4" border="0"></a><br /><a target='_blank' href='https://imgbb.com/'>image upload</a><br />
"""
df = pd.merge(df, links_small, how='inner', on='id')

features = ['id', 'movieId','imdb_id', 'title', 'genres', 'overview', 'tagline', 'description','popularity',
       'vote_average', 'vote_count', 'year', 'year_period', 'cast', 'crew','director', 'keywords',
       'History', 'Horror', 'Foreign', 'Action', 'Documentary',
       'Science Fiction', 'Music', 'Romance', 'Animation', 'TV Movie',
       'Fantasy', 'War', 'Comedy', 'Family', 'Drama', 'Western', 'Crime',
       'Adventure', 'Mystery', 'Thriller']
df = df[features]
df[:1]
tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')
tfidf_matrix = tf.fit_transform(df['description'])

cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)

df = df.reset_index()
df[:1]
"""
We are now in a good position to define our recommendation function. These are the following steps we'll follow :-
* Get the index of the movie given its title.
* Get the list of cosine similarity scores for that particular movie with all movies. Convert it into a list of tuples where the first element is its position and the second is the similarity score.
* Sort the aforementioned list of tuples based on the similarity scores; that is, the second element.
* Get the top 10 elements of this list. Ignore the first element as it refers to self (the movie most similar to a particular movie is the movie itself).
* Return the titles corresponding to the indices of the top elements.
"""
indices_to_titles = df['title'] # titles
title_to_index = pd.Series(data=df.index, index=df['title']) # indices
def get_recommendations(title):
    idx = title_to_index[title]
    sim_scores = list(enumerate(cosine_sim[idx]))
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
    sim_scores = sim_scores[1:31]
    movie_indices = [i[0] for i in sim_scores]
    
    # return df.iloc[movie_indices]
    return indices_to_titles.iloc[movie_indices] 
get_recommendations('The Dark Knight').head(5)
"""
### 5.2.2 Metadata Based Recommender
- It goes without saying that the quality of our recommender would be increased with the usage of better metadata. We are going to build a recommender based on the following metadata: the actors, the director, related genres and the movie plot keywords, year.
- The next step would be to convert the names and keyword instances into lowercase and strip all the spaces between them. This is done so that our vectorizer doesn't count the Johnny of "Johnny Depp" and "Johnny Galecki" as the same.
- Creating our "metadata soup", which is a string that contains all the metadata that we want to feed to our vectorizer (namely actors, director and keywords).
"""
df['soup'] = df['cast'] + df['director'] + df['keywords'] + df['genres']
df['soup'] = df['soup'].apply(lambda x: ' '.join(x))
df['soup'] = df['soup'] + ' ' + df['year_period']
df['soup'][:1]
count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')
count_matrix = count.fit_transform(df['soup'])

cosine_sim = cosine_similarity(count_matrix, count_matrix)
get_recommendations('The Dark Knight').head(5)
"""
## 5.3 Simple Recommender and Metadata Based Recommender
"""
def improved_recommendations(title):
    idx = title_to_index[title]
    sim_scores = list(enumerate(cosine_sim[idx]))
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
    sim_scores = sim_scores[1:26]
    movie_indices = [i[0] for i in sim_scores]
    
    movies = df.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year']]
    vote_counts = movies[movies['vote_count'].notnull()]['vote_count'].astype('int')
    vote_averages = movies[movies['vote_average'].notnull()]['vote_average'].astype('int')
    C = vote_averages.mean()
    m = vote_counts.quantile(0.60)
    qualified = movies[(movies['vote_count'] >= m) & (movies['vote_count'].notnull()) & (movies['vote_average'].notnull())]
    qualified['vote_count'] = qualified['vote_count'].astype('int')
    qualified['vote_average'] = qualified['vote_average'].astype('int')

    # qualified['wr'] = qualified.apply(lambda x: (x['vote_count']/(x['vote_count']+m) * x['vote_average']) + (m/(m+x['vote_count']) * C), axis=1)
    qualified['wr'] = qualified.apply(lambda x: weighted_rating(x, C, m), axis=1)    
    qualified = qualified.sort_values('wr', ascending=False).head(10)
    return qualified
improved_recommendations('The Dark Knight').head(5)
reader = Reader()
data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)
"""
## 5.4 Collaborative Filtering

Our content based engine suffers from some severe limitations. It is only capable of suggesting movies which are close to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.

Also, the engine that we built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying our engine for recommendations based on a movie will receive the same recommendations for that movie, regardless of who she/he is.

Therefore, in this section, we will use a technique called Collaborative Filtering to make recommendations to Movie Watchers.
It is basically of two types:-

*  **User based filtering**-  These systems recommend products to a user that similar users have liked. For measuring the similarity between two users we can either use pearson correlation or cosine similarity.
This filtering technique can be illustrated with an example. In the following matrixes, each row represents a user, while the columns correspond to different movies except the last one which records the similarity between that user and the target user. Each cell represents the rating that the user gives to that movie. Assume user E is the target.
![](https://cdn-images-1.medium.com/max/1000/1*9NBFo4AUQABKfoUOpE3F8Q.png)

It suffers from several problems. One main issue is that users’ preference can change over time. It indicates that precomputing the matrix based on their neighboring users may lead to bad performance. To tackle this problem, we can apply item-based CF.

* **Item Based Collaborative Filtering** - Instead of measuring the similarity between users, the item-based CF recommends items based on their similarity with the items that the target user rated. Likewise, the similarity can be computed with Pearson Correlation or Cosine Similarity. The major difference is that, with item-based collaborative filtering, we fill in the blank vertically, as oppose to the horizontal manner that user-based CF does. The following table shows how to do so for the movie Me Before You.
![](https://cdn-images-1.medium.com/max/1000/1*LqFnWb-cm92HoMYBL840Ew.png)


However, several problems remain for this method. First, the main issue is ***scalability***. The computation grows with both the customer and the product. The worst case complexity is O(mn) with m users and n items. In addition, ***sparsity*** is another concern. Take a look at the above table again. Although there is only one user that rated both Matrix and Titanic rated, the similarity between them is 1. In extreme cases, we can have millions of users and the similarity between two fairly different movies could be very high simply because they have similar rank for the only user who ranked them both.
"""
"""
### **Single Value Decomposition**
One way to handle the scalability and sparsity issue created by CF is to leverage a **latent factor model** to capture the similarity between users and items. Essentially, we want to turn the recommendation problem into an optimization problem. We can view it as how good we are in predicting the rating for items given a user. One common metric is Root Mean Square Error (RMSE). **The lower the RMSE, the better the performance**.

Now talking about latent factor you might be wondering what is it ?It is a broad idea which describes a property or concept that a user or an item have. For instance, for music, latent factor can refer to the genre that the music belongs to. SVD decreases the dimension of the utility matrix by extracting its latent factors. Essentially, we map each user and each item into a latent space with dimension r. Therefore, it helps us better understand the relationship between users and items as they become directly comparable. The below figure illustrates this idea.

![](https://cdn-images-1.medium.com/max/800/1*GUw90kG2ltTd2k_iv3Vo0Q.png)
"""
#knn = KNNBasic()
svd = SVD()

# Run 5-fold cross-validation and then print results
cross_validate(svd, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)
# test set is made of 25% of the ratings.
trainset, testset = train_test_split(data, test_size=.25)
svd.fit(trainset)

predictions = svd.test(testset)

# Then compute RMSE
accuracy.rmse(predictions)
"""
For user_id of 514 rating movie_id of 505, we get an estimated prediction of 3.11 where the true rating is 3. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have predicted the movie.
"""
user_id = 514
movie_id = 505
real_rating = 3
svd.predict(user_id, movie_id, real_rating, verbose=True)
"""
## 5.5 Hybrid Recommender: Metadata Based and Collaborative Filtering

In this section, I will try to build a simple hybrid recommender that brings together techniques we have implemented in the content based and collaborative filter based engines. This is how it will work:
- Input: User ID and the Title of a Movie
- Output: Similar movies sorted on the basis of expected ratings by that particular user.
"""
indices_map = df.set_index('id')
indices_map.head(1)
def hybrid(userId, title):
    idx = title_to_index[title]
    
    sim_scores = list(enumerate(cosine_sim[int(idx)]))
    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)
    sim_scores = sim_scores[1:26]
    movie_indices = [i[0] for i in sim_scores]
    
    movies = df.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year', 'id']]
    movies['est'] = movies['id'].apply(lambda x: svd.predict(userId, indices_map.loc[x]['movieId']).est)
    movies = movies.sort_values('est', ascending=False)
    return movies.head(10)
hybrid(1, 'Avatar')