"""
# Top Spotify Songs of 2018
"""
"""
<font size="4">_Audio features of top Spotify songs_</font>
"""
"""
### Task Desctiption
"""
"""
At the end of each year, Spotify compiles a playlist of the songs streamed most often over the course of that year. This year's playlist (Top Tracks of 2018) includes 100 songs. 

**The question is: What do these top songs have in common? Why do people like them?**

Original Data Source: The audio features for each song were extracted using the Spotify Web API and the spotipy Python library. Credit goes to Spotify for calculating the audio feature values.

**Exploring the Data: Some suggestions for what to do with the data:**
- Look for patterns in the audio features of the songs. Why do people stream these songs the most?
- Try to predict one audio feature based on the others
- See which features correlate the most
"""
"""
### Data Description
"""
"""
- `id`: Spotify URI of the song
- `name`: Name of the song
- `artists`: Artist(s) of the song
- `danceability`: Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
- `energy`: Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
- `key`: The key the track is in. Integers map to pitches using standard Pitch Class notation. E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on.
- `loudness`: The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.
- `mode`: Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.
- `speechiness`: Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
- `acousticness`: A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
- `instrumentalness`: Predicts whether a track contains no vocals. "Ooh" and "aah" sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly "vocal". The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
- `liveness`: Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
- `valence`: A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
- `tempo`: The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
- `duration_ms`: The duration of the track in milliseconds.
- `time_signature`: An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure).
"""
"""
### Importing The Tools
"""
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'svg'

import seaborn as sns
sns.set()
"""
### Loading The Data
"""
top_songs = pd.read_csv("../input/top2018.csv")
"""
### Exploratory Data Analysis
"""
"""
#### General info on the dataset
"""
top_songs.describe()
top_songs.info()
"""
Firsty, let's analyse the data column by column to get a general idea of what it represents.
"""
"""
#### `Name`
"""
"""
What can be explored about the name of the song?
- Length of the song name in characters
 - **Answer**: Mostly ranging from 1 to ~15 characters. 
- Length of the song name in words
 - **Answer**: Mostly songs with 1-2-3 words in their name.
- Presence of `"feat."` – we can calculate how many co-op songs make it to the top, for example. Because as we can see in the data, each song is corresponding to single artist. However, the song may be a collaboration, as denoted by the `"feat."` mark in the title
 - **Answer**: Most of the songs are not collabs.
"""
top_songs["name_len"] = top_songs["name"].apply(lambda x: len(x))
top_songs["name_len"].plot(kind="hist")
plt.title("Length of the song name in characters")
"""
Let's write a function that clears the song names from `<feat.>` and `<with>` words and extracts useful information – either the original song title, or its length in words, or the name of a collaborator.
"""
def get_additional_info_from_songname(songname, info_needed = "songname"):
    """
    Accepts:
        songname (str). String from dataframe.
        info_needed (str). Either "songname" (cleaned from <feat.> and <with>) or 
                           "coauthor" (in colab with whom this song was recorded) or
                           "wordcount" (how many words the title contains)
    Splits it.
    Finds <(feat.> and <(with> parts.
    Gets rid of them.
    Returns either the length of a song name without this additional info or the info itself.
    """
    song_components = songname.split()
    
    if "(feat." in song_components:
        if info_needed == "songname":
            return " ".join(song_components[ : song_components.index("(feat.")])
        if info_needed == "coauthor":
            coauth = song_components[song_components.index("(feat.")+1:]
            # this is to get rid of the last parenthesis
            coauth[-1] = coauth[-1][:-1]
            return " ".join(coauth)
        if info_needed == "wordcount":
            return len(song_components[ : song_components.index("(feat.")])
    elif "(with" in song_components:
        if info_needed == "songname":
            return " ".join(song_components[ : song_components.index("(with")])
        if info_needed == "coauthor":
            coauth = song_components[song_components.index("(with")+1:]
            # this is to get rid of the last parenthesis
            coauth[-1] = coauth[-1][:-1]
            return " ".join(coauth)
        if info_needed == "wordcount":
            return len(song_components[ : song_components.index("(with")])
    else:
        # if there was no feat. or with in the song title
        if info_needed == "songname":
            return songname
        elif info_needed == "coauthor":
            return "No colab"
        elif info_needed == "wordcount":
            return len(songname.split())
top_songs["clean_name"] = top_songs["name"].apply(
                            lambda x: get_additional_info_from_songname(x, 
                                                                        info_needed="songname"))

top_songs["name_wordcount"] = top_songs["name"].apply(
                                lambda x: get_additional_info_from_songname(x,
                                                                            info_needed="wordcount"))

top_songs["collaborator"] = top_songs["name"].apply(
                                lambda x: get_additional_info_from_songname(x,
                                                                            info_needed="coauthor"))
top_songs["name_wordcount"].value_counts().sort_values(ascending=False).plot(kind="bar")
plt.title("Number of words in song name")
plt.xticks(rotation=0)
"""
As we can see, most of the songs have from 1 to 3 words in their name. However, I am curious to see what are the notable outliers with 5, 6, and even 12 words in the name?
"""
top_songs[top_songs["name_wordcount"] > 4]["clean_name"]
"""
I've looked through the values in the `clean_names` column and figured out there is still some unclean data. Let's correct it.
"""
"""
We have to correct songs number 21, 46, 47, 84, 98.
"""
top_songs.at[21, "clean_name"] = "Te Bot?"

top_songs.at[46, "clean_name"] = "Finesse (Remix)"
top_songs.at[46, "collaborator"] = "Cardi B"

top_songs.at[47, "clean_name"] = "Back To You"

top_songs.at[84, "clean_name"] = "Perfect"
top_songs.at[84, "collaborator"] = "Beyonce"

top_songs.at[98, "clean_name"] = "Dusk Till Dawn"
"""
That is better.
"""
top_songs["is_collab"] = top_songs["collaborator"].apply(lambda x: 0 if x == "No colab" else 1)

top_songs["is_collab"].value_counts().plot(kind="bar")
plt.title("Is the song a collaboration or not?")
plt.xticks([0,1],["No", "Yes"], rotation=0)
"""
So it seems like most of the songs are not collabs! Interesting.
"""
"""
#### `artists`
"""
"""
What can we get from the artists column?
- Are there any artists that have several songs on the Top-100 of 2018? If yes, then how many songs do they have?
 - **Answer**: Yes, there are artists with more than 1 song on the top-list. There is 21 such artists. Most songs were provided by Post Malone and XXXTENTACION (6 each).
"""
top_songs["artists"].value_counts()[top_songs["artists"].value_counts() >= 2]
"""
It is interesting, because the count does not include the collabs. So what needs to be done is to count artists in the `collaborator` column and add the numbers to the respected artists in the above Series. I am pretty sure that Cardi B has several collabs, which technically means she is also responsible for the song; however, she has only 1 count in the overall counts.
"""
top_songs["collaborator"].value_counts().head()
total_number_of_works = top_songs["artists"].value_counts()

for artist, number in top_songs["collaborator"].value_counts().iteritems():
    try:
        total_number_of_works.loc[artist] += number
    except:
        pass
total_number_of_works[total_number_of_works >= 2].sort_values(ascending=False)
plt.figure(figsize=(10, 3))
total_number_of_works[total_number_of_works >= 2].sort_values(ascending=False).plot(kind="bar")
plt.title("Number of songs per artist")
plt.xticks(rotation=81)
"""
#### `danceability`
"""
"""
Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
"""
top_songs["danceability"].plot(kind='hist')
plt.title("Distribution of songs danceability")
"""
It can be seen that most of the songs are quite danceable (value being bigger than 0.5). 
"""
"""
Let's take a look at the danceability of the songs of artists with the highest amount of songs.
"""
top_artists = list(total_number_of_works[total_number_of_works >= 2].sort_values(ascending=False).index)
top_songs[top_songs["artists"].isin(top_artists)]["danceability"].plot(kind="hist")
plt.title("Danceability of most popular artists' songs")
"""
Well, danceability surely has to be high enough in order for a song to become popular!
"""
"""
#### `energy`
"""
"""
Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
"""
top_songs["energy"].plot(kind='hist', bins=20)
plt.title("Energy Distribution")
"""
Energy distribution doesn't really tell that much – well, it seems a song has to be danceable and quite energetic, in order to become popular. I am interested, however, is there a correlation between danceability and energy? Can a danceable song be not-energetic?
"""
fig = sns.jointplot(x="danceability", y="energy", data=top_songs)
# fig.fig.set_figwidth(10)
# fig.fig.set_figheight(5)
"""
It is interesting to see that high danceability doesn't necessarily mean high energy.
"""
"""
Let's add a hue on whether a certain point on the jointplot corresponds to a song that is made by most popular artists.
"""
top_songs["is_by_top_artist"] = top_songs["artists"].apply(lambda x: 1 if x in top_artists else 0)
fig = sns.scatterplot(x="danceability", y="energy", data=top_songs, hue="is_by_top_artist")
"""
Seems like most popular artists tend to care more for the danceability of the songs, not really bothering about energy.
"""
"""
#### `key`
"""
"""
The key the track is in. Integers map to pitches using standard Pitch Class notation. E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on.
"""
top_songs["key"].value_counts().sort_index().plot(kind="bar")
plt.title("How many songs of given key are there?")
_ = plt.xticks(range(len(top_songs["key"].value_counts().sort_index().index)),
           ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"],
           rotation=0)
"""
We can see that most songs are written in C#, G#, C, F, A, B keys. 

It might be interesting to see whether a key determines a song's danceability or energy or not.
"""
"""
In order to do so, we first need to convert the key variable from numeric to object (for seaborn to correctly identify it as a variable to hue on).
"""
top_songs["key_string"] = top_songs["key"].replace({0: "C", 1: "C#", 2: "D", 3: "D#", 
                                                    4: "E", 5: "F", 6: "F#", 7: "G",
                                                    8: "G#", 9: "A", 10: "A#", 11: "B"})
plt.figure(figsize=(8, 5))
sns.scatterplot(x="danceability", y="energy", data=top_songs, hue="key_string")
plt.title("Does the key has any influence on danceability / energy of the song?")
"""
There doesn't seem to be any connection.
"""
"""
#### `loudness`
"""
"""
The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.
"""
top_songs["loudness"].plot(kind="hist", bins=25)
plt.title("Loudness distribution")
"""
This doesn't tell us much. Maybe there is an interesting correlation, though.
"""
g = sns.jointplot(x="loudness", y="energy", data=top_songs)
g.fig.suptitle("Loudness vs. Energy")
"""
Here we go! This looks interesting. The louder the song, the more energy the song has.
"""
sns.lmplot(x="loudness", y="energy", data=top_songs)
"""
Songs in which key are louder?
"""
sns.boxplot(x="key_string", y="loudness", data=top_songs)
"""
Usually C#, A#, D, B and E songs are somewhat louder than the rest. Why whould that be?..
"""
"""
#### `mode`
"""
"""
Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0.
"""
top_songs["mode"].value_counts().plot(kind='bar', color=sns.color_palette("deep", 2))
_ = plt.xticks([0,1],["Major", "Minor"], rotation=0)
"""
I am quite surprised to see that it is somewhat equal between the two. In my mind I thought that maybe major songs will dominate. Guess I was wrong.
"""
"""
Any interesting correlation with either danceability or energy?
"""
sns.scatterplot(x="danceability", 
                y="energy", 
                data=top_songs, 
                hue="mode", 
                palette=sns.color_palette("deep", 2)[::-1])
"""
Minor songs tend to have more energy and a bit more danceability.
"""
sns.scatterplot(x="key_string", y="loudness", data=top_songs, 
                hue="mode", palette=sns.color_palette("deep", 2)[::-1])
top_songs.groupby(["key_string","mode"]).size().unstack().plot(kind="bar", 
                                                               stacked=True, 
                                                               color = sns.color_palette("deep", 2)[::-1])
_ = plt.xticks(rotation=0)
_ = plt.title("Major / Minor Mode by Key")
"""
So we can see that A, F and B often come in minor mode. 
"""
"""
#### `speechiness`
"""
"""
Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks.
"""
top_songs["speechiness"].plot(kind='hist', bins=20)
_ = plt.title("Speechiness Distribution")
"""
There is something strange about speechiness data; can it really be that most of the songs are instrumental?
"""
"""
I am pretty sure this feature is messed up a little bit. 
"""
"""
#### `acousticness`
"""
"""
A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic.
"""
top_songs["acousticness"].plot(kind="hist", bins=20)
_ = plt.title("Acousticness Distribution")
"""
Well, it doesn't seem that people like acoustic songs thet much, really. 
"""
"""
Let's see if all acoustic songs are written in a common key (who knows?)
"""
sns.scatterplot(x="acousticness", y="loudness", 
                data=top_songs, hue="key_string",
                palette=sns.color_palette("magma", 12))
_ = plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
"""
It doesn't seem so. But acoustic songs are definitely much more quiet than non-acoustic ones.
"""
sns.scatterplot(x="acousticness", y="energy", 
                data=top_songs)
_ = plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
_ = plt.title("Acousticness vs. Energy")
"""
Well, obivously, the more acoustic a song is, the less energy it has, and the more quiet it gets.
"""
"""
#### `liveness`
"""
"""
Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live.
"""
top_songs["liveness"].plot(kind="hist", bins=20)
_ = plt.title("Liveness distribution")
"""
There doesn't seem to be many live-versions of the songs. Which is explainable, I think people like the studio quality of sound the most. 
"""
"""
#### `valence`
"""
"""
A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
"""
top_songs["valence"].plot(kind="hist", bins=30)
_ = plt.title("Valence Distribution")
"""
This is interesting – looks like it is more possible to have a sad or angry song... Neutral at best.
"""
sns.scatterplot(x="valence", y="danceability", data=top_songs)
_ = plt.title("Valence vs. Danceability")
sns.scatterplot(x="valence", y="energy", data=top_songs)
_ = plt.title("Valence vs. Energy")
"""
Happier songs tend to get more energetic. 
"""
"""
#### `tempo`
"""
"""
The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration.
"""
top_songs["tempo"].plot(kind="hist", bins=20)
_ = plt.title("Tempo Distribution")
fig, ax = plt.subplots(1, 2, figsize=(10, 3))
sns.scatterplot(x="tempo", y="danceability", data=top_songs, ax=ax[0])
sns.scatterplot(x="tempo", y="energy", data=top_songs, ax=ax[1])
sns.boxplot(x="key_string", y="tempo", data=top_songs)
_ = plt.title("Keys vs. Tempo")
"""
Contrary to my belief, tempo doesn't have that much of interesting correlations with other variables.
"""
"""
#### `duration_ms`
"""
"""
The duration of the track in milliseconds.
"""
top_songs["duration_ms"].plot(kind="hist", bins=30)
_ = plt.title("Duration Distribution")
sns.boxplot(x="key_string", y="duration_ms", data=top_songs)
"""
TODO: Finish the analysis.
"""