"""
### Trending YouTube Video Statistics
Daily statistics for trending YouTube videos

### Description
YouTube (the world-famous video sharing website) maintains a list of the top trending videos on the platform. According to Variety magazine, “To determine the year’s top-trending videos, YouTube uses a combination of factors including measuring users interactions (number of views, shares, comments and likes). Note that they’re not the most-viewed videos overall for the calendar year”. Top performers on the YouTube trending list are music videos (such as the famously virile “Gangam Style”), celebrity and/or reality TV performances, and the random dude-with-a-camera viral videos that YouTube is well-known for.

This dataset is a daily record of the top trending YouTube videos in USA for several months.
"""
"""
First we are going to import everything we need to do the analysis
"""
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

from datetime import datetime
import itertools
import json
import string
import re    #for regex
import nltk
from nltk.corpus import stopwords

from nltk import pos_tag
from nltk.stem.wordnet import WordNetLemmatizer 
from nltk.tokenize import word_tokenize
from nltk.tokenize import TweetTokenizer 
from wordcloud import WordCloud


plt.style.use('seaborn-whitegrid')
# Set Matplotlib defaults
plt.rc('figure', autolayout=True)
plt.rc('axes', labelweight='bold', labelsize='large',
       titleweight='bold', titlesize=14, titlepad=10)
"""
Let's have a look at our dataset! First I will take care of the time format columns; from them I will extract three additional columns that will be helpful for the exploratory analysis (day, hour and month of publication)
"""

data = pd.read_csv("../input/youtube-new/USvideos.csv",parse_dates=["publish_time"])
data.head(3)
"""
### Analyzing the time !
"""
# first convert trending date to datetime format
data["trending_date"] = data["trending_date"].apply(lambda x : datetime.strptime(x,"%y.%d.%m"))
# create three columns , using dt function to extract dayofweek, hour and month
data["publish_day_week"] = data["publish_time"].dt.dayofweek
data["publish_hour"] = data["publish_time"].dt.hour
data["publish_month"] = data["publish_time"].dt.month
"""
I will see the evolution of the number of views of the videos over time; for this I use the Rolling function and I group them by week to better appreciate the trend
"""
# groupby trending date by views
time = data.groupby("trending_date")["views"].sum()


# let,s plot using rolling (7 -> week)
plt.figure(figsize=(8,4))
sns.lineplot(data=time.rolling(7).mean(), linewidth=2.5,color="#EB7827",label="Weekly")
sns.lineplot(data=time, linewidth=2.5,color="grey",alpha=0.2,label="Dayly")
# using plt.grid and sns.despine you get a cleaner look in the graphics
plt.grid(None)
sns.despine()
plt.title("Trending videos: Total views per week ")
plt.ylabel("Total Views")
"""
From April 2018 the sights will go off like a rocket! BTS has surely released a new song :) 
"""
# Let's check how many videos there are per year in the dataset; using the same dataframe as before (Time) and reset_index, to facilitate their visualization.
year = time.reset_index()

#now with reset_index I have two columns, the number of views and trending_date; as before I use the dt.year function to extract the year and value_counts to count them.
year_count = year.trending_date.dt.year.value_counts()
year_count.plot(kind="bar",color=["#F6794B","#E3E3E3"])
plt.xticks(rotation=45)
plt.title("Total videos per year")
sns.despine()
plt.grid(None)
"""
Most videos belong to 2018!

Let's continue with time charts; let's see what day, month and time the videos are usually published!
"""
"""
First I group the views by day of the week; then I create a dictionary with the days of the week to convert the numbers into strings and make the display more enjoyable
"""
# creating a dict
numeros = [*range(7)]
day = ["Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday"]
dict_day = {}
for key,item in zip(day,numeros):
   dict_day[item] = key
# groupby by day of week
day_week = data.groupby("publish_day_week").size().reset_index()
# apply dict to publish day week column using map
day_week["publish_day_week"] = day_week["publish_day_week"].map(dict_day)

# let,s plot ¡¡
plt.figure(figsize=(8,4))
colors = ["#FAEFBE","#FAE0A7","#FAD19E","#FABC96","#EFA492","#18BADA","#6B6D8C"]
sns.barplot(x="publish_day_week",y=0,data=day_week,palette=colors,saturation=0.8)
plt.title("Total published videos per day of week")
plt.xlabel("Day of week")
plt.ylabel("Total")
plt.xticks(rotation=45)
sns.despine(left=True)
"""
We see that on weekends the Youtubers tend to rest and they tend to upload more videos on Thursday and Friday. Now I will do the same but with the months of the year
"""
numeros_año = [*range(1,13)]
month = [ 'Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov','Dec']

dict_month = {}
for key,item in zip(month,numeros_año):
   dict_month[item] = key

month_df = data.groupby("publish_month").size().reset_index()

month_df["publish_month"] = month_df["publish_month"].map(dict_month)

plt.figure(figsize=(8,4))

sns.barplot(x="publish_month",y=0,data=month_df,palette="Paired",saturation=0.8)
plt.xlabel("Month")
plt.ylabel("Total")
plt.title("Total videos per month")
plt.xticks(rotation=45)
sns.despine(left=True)
"""
Ahhhh the summer months people prefer to go to the beach or the mountains instead of uploading videos, and it makes a lot of sense, you have to enjoy the good weather!

And finally, let's see what time the Youtubers usually upload their videos!
"""
hour = data.groupby("publish_hour").size()

plt.figure(figsize=(8,4))
sns.barplot(x=hour.index.values,y=hour.values,palette="mako_r")
sns.despine()
plt.title("Number of videos uploaded per hour")
"""
There are many Youtubers who do not sleep at night! The truth is that the night is when it is quieter, there are fewer distractions and noise.
"""
"""
### Channels, Youtubers...
There is a column in our dataframe called category_id but they are numbers; luckily there is a Json file, where we can find the video categories. 
Let's create, as we did before with the dates, a dictionary of video categories!
"""
# First we load the json file

with open("../input/youtube-new/US_category_id.json") as f:
    data_json = json.load(f)["items"]
# # how is a json file? well, let,say they are dictionaries within dictionaries..
data_json[0]
# create a dict with titles (we go first to snippet and them title)
title_dict = {}
for cat in data_json:

    title_dict[int(cat["id"])] = cat["snippet"]["title"]
    
# now we use map to apply it to the dataframe so we can work better!
data["category_id"] = data["category_id"].map(title_dict)
# let's see what percentage of videos per category are in our dataframe.

# let's see what percentage of videos per category are in our dataframe. First, with value_counts and normalize we get the relative frequency
categorys = data["category_id"].value_counts(normalize=True).reset_index()
# rename columns names
categorys.rename(columns={"index":"Category","category_id":"Percentage"},inplace=True)
# get percentage
categorys["Percentage"] = round(categorys["Percentage"] *100,2)
# we use style background ¡
categorys.style.background_gradient(cmap='mako_r')
"""
Forty percent of the videos belong to the categories Music and Enterteiment ¡¡

In our dataframe there are columns that we can play with, for example likes, dislikes; we can create new columns like the percentage of likes by views, or the percentage of dislikes by views...
"""
# Well, neither one nor the other, I will do better the sum of both, that is, the percentage of like/dislikes by views....
total = data["likes"] + data["dislikes"]
"""
On second thought, I will also take into account the comments; I will create a column that will measure the percentage of user interactions based on the number of views
"""
# let's add up the comments.
total_def = data["comment_count"] + total
# we created a new column...percentage iterations per view ¡¡¡
data["percentage_iterations_per_view"] = round((total_def / data["views"]) * 100,2)
"""
I created a small function to group, average and sort the following columns ("views", "likes", "dislikes", "comment_count", "percentage_iterations_per_view")
"""
def data_views(groupby="channel_title",by="percentage_iterations_per_view",ascending=False):
    """
    return dataframe groupby channels, showing mean from columns:
    "views","likes","dislikes","comment_count","percentage_iterations_per_view"
    ,sort by and ascending included
    """
    return  (data
            .groupby(groupby)["views","likes","dislikes","comment_count","percentage_iterations_per_view"]
            .mean()
            .sort_values(by=by,ascending=ascending)
            )
# Let's take a look; these are the channels ordered by "percentage_iterations_per_view"
data_views().head(10)
"""
These are the channels with more interactions (likes, dislikes, comments); Daily caller , Desimpedidos or KickThePj for example are the ones that have more; it would be curious to see their theme ...

https://www.youtube.com/user/dailycaller
"""
# Which are the channels with more videos in the dataset ?
total_chanel = data.groupby("channel_title")["video_id"].count().sort_values(ascending=False).head(10).reset_index()
total_chanel.rename(columns={"channel_title":"Channel_title","video_id":"Total_videos"},inplace=True)
total_chanel.style.background_gradient(cmap='mako_r')
# is there any relation between the number of comments and the likes/dislikes ? well, let's chart it with a scatter plot and with size="comment_coun
plt.figsize=(10,6)
sns.scatterplot(x="likes",y="dislikes",size="comment_count", hue="comment_count",data=data,alpha=0.7)
plt.title("Relation between the number of comments and the likes/dislikes")
plt.grid(False)
plt.legend( bbox_to_anchor=(1.05, 1), loc='upper left',fontsize='xx-small')
"""
If we can see that the greater the increase of both likes and dislikes, the number of comments increases.

Let's check the average, in percentage, of iterations per category
"""
most_comment_category= data.groupby("category_id")["percentage_iterations_per_view"].mean()                                                                 .sort_values(ascending=False)
                                    
colors = ["#EAEBE9" for _ in range(len(most_comment_category))]
colors[0] = "#FD0B3B"
plt.figure(figsize=(10,6))
sns.barplot(x=most_comment_category.index.values,y=most_comment_category.values,palette=colors,saturation=0.8,errwidth=0.4)
plt.title("Average percentage iterations per category")
plt.xlabel("Categorys")
plt.ylabel("Percentage iterations")
plt.xticks(rotation=45)
sns.despine(left=True)
"""
We see that Music, Style or Comedy tend to have the greatest number of iteractions for videos; on the other hand, Politics and Sports do not tend to have as many, which is surprising since they are always controversial topics.
"""
"""
### WordCloud

Now finally we will make a WordCloud; it is a very visual and fast way to do text analysis without going into too much detail; let's see what result we get with the titles.
"""
#Setting the stopwords
eng_stopwords = set(stopwords.words("english"))

wordcloud = WordCloud(
                          background_color='white',
                          stopwords=eng_stopwords,
                          max_words=1000,
                          max_font_size=120, 
                          random_state=42
                         ).generate(str(data['title']))

print(wordcloud)
fig = plt.figure(1)
plt.imshow(wordcloud)
plt.title("WordCloud Titles")
plt.axis('off')
plt.show()
"""
Well this has been a quick analysis of the videos that are trends in Youtube, in a certain period, in Usa. I hope you liked it, draw your conclusions and if you have any comments or ideas for improvement, please do not hesitate to tell me, it will be of great help to improve!

a hug to everyone!
"""