# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Changing Music Preferences
# ### Do Spotify users listen to different music during the COVID-19 pandemic? 
# ### Does this relate to the amount of confirmed cases?
#
# To Do: 
#
# 1. Scraping Spotify Top 200 data
# 2. Get song metadata through Spotify API
# 3. Data Visualization & Modelling
# 4. Give reference values for happy/sad music
# 5. Create interactive dashboard in Shiny
#
# cool plot: https://towardsdatascience.com/discovering-your-music-taste-with-python-and-spotify-api-b51b0d2744d 
#

# ## 1.
# ## Scraping the Top 200 data for 2019 (baseline) and 2020 (pandemic).
#
# https://spotifycharts.com contains the official Spotify Charts. 
#
# Luckily, each day/week is available as a .csv file!  
# I will download each file and add them together in one dataframe.
#

import pandas as pd
import numpy as np
from tqdm import tqdm
import requests
from bs4 import BeautifulSoup
import io

# +
# Scraping all the links to the official Spotify Top 200 .csv's
url = 'https://spotifycharts.com/regional/nl/weekly/latest'

r = requests.get(url)

soup = BeautifulSoup(r.content, 'html.parser')
# -

dates = [item["data-value"] for item in soup.find_all(attrs={"data-value": True}) if item["data-value"].startswith('2')]
dates[:5]

urls = ["https://spotifycharts.com/regional/nl/weekly/" + date + "/download" for date in dates]
urls[:5]

# ## Attention: running the block below will result in downloading 200 .csv's!

# +
data = []

for date in tqdm(dates): 
    url = "https://spotifycharts.com/regional/nl/weekly/" + date + "/download"
    response = requests.get(url)
    file_object = io.StringIO(response.content.decode('utf-8'))
    df = pd.read_csv(file_object, header=1)
    df["Date"] = date
    data.append(df)

# merge all csv's
df = pd.concat(data)
df.reset_index(drop=True, inplace=True)
df.to_csv("..\\data\\raw\\top200_2017_2020.csv", index=False)
data = []
# -

df = pd.read_csv("..\\data\\raw\\top200_2017_2020.csv")
df

# +
print(df.isna().sum())
df.dropna(inplace=True) # removing 8 rows that contain a missing value

df[['Start Week', 'End Week']] = df['Date'].str.split('--', 1, expand=True) # add columns for start and end dates
df['ID'] = df['URL'].str.rsplit('https://open.spotify.com/track/', 1, expand=True)[1] # add ID column for merge in task 2

df.to_csv("..\\data\\processed\\top200_2017_2020.csv", index=False)
# -

df = pd.read_csv("..\\data\\processed\\top200_2017_2020.csv")
df.head()

# + [markdown] pycharm={"name": "#%% md\n"}
# ## 2.
# ## Retrieve and Combine music features through Spotify API
# Spotify generates features on all their music, which can be accessed through the API.  
# For each song (ID) in the Top 200 dataset, I will requrest these features and add it to the dataset.
#
# More on the features: https://developer.spotify.com/documentation/web-api/reference/tracks/get-audio-features/

# +
# Fill this in using your Spotify API Credentials from https://developer.spotify.com/dashboard/applications
import spotipy
from spotipy.oauth2 import SpotifyClientCredentials

cid ="XXX" 
secret = "XXX"

client_credentials_manager = SpotifyClientCredentials(client_id=cid, client_secret=secret)
sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager)
# -

# Where I save my Spotify API Credentials, outside of this public repository :)
import spotipy
# %run credentials.py

# +
df_ids = df["ID"].drop_duplicates()     # store all unique ID's in a seperate df

rows = []
batchsize = 100     # maximum batchsize is 100

for i in tqdm(range(0, len(df_ids), batchsize)):
    batch = df_ids[i:i+batchsize]
    feature_results = sp.audio_features(batch)
    for i, t in enumerate(feature_results):
        if t == None:
            continue
        else:
            rows.append(t)
# -

features = pd.DataFrame.from_dict(rows, orient="columns")
features.head()

# + pycharm={"name": "#%%\n"}
# drop redundant features
features.drop(['analysis_url','track_href','type','uri'], axis=1, inplace=True) 

# + pycharm={"name": "#%%\n"}
features.describe()

# + pycharm={"name": "#%%\n"}
# merge Top 200 data with song features

df_merge = pd.merge(df, features, left_on="ID", right_on="id", how="inner")

df_merge.to_csv("..\\data\\processed\\top200_2017_2020_w_features.csv", index=False)

df_merge.head()
# -

# ## 3.
# ## Data Visualization and Modelling
#
# To see if Dutch Spotify users actually listened to different music during the pandemic, I will focuss on 3 features.  
# - Danceability: Describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
#  
# - Energy: A measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.  
#
# And maybe the most important feature:
# - Valence:  A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).

# + pycharm={"name": "#%%\n"}
# Load the data and transform date-time columns
df = pd.read_csv("..\\data\\processed\\top200_2017_2020_w_features.csv")
df[['Start Week', 'End Week']] = df[['Start Week', 'End Week']].apply(pd.to_datetime, format="%Y-%m-%d")
#df.info()

# + pycharm={"name": "#%%\n"}
# calculate the weekly average on the three features
week_average = df[['Start Week', 'danceability', 'energy','valence']].groupby(pd.Grouper(key="Start Week", freq='W')).mean()

# add seperate features for the dates for easier computation
week_average["Y"] = week_average.index.year
week_average["M"] = week_average.index.month
week_average["YMD"] = week_average.index.date

week_average.head()

# +
import plotly.express as px

fig = px.line(week_average, x="YMD", y="energy", color='Y')
fig.show()
# -

fig = px.line(week_average, x="YMD", y="valence", color='Y')
fig.show()

# ### But is this right? It seems that the number of streams in each week follows the power law, which means the first positions have much more streams than the 50th or 100th. I will, therefore, weigh the track features (valence and others) on the number of streams.

fig = px.line(df[df["Start Week"] >= '2020'].sort_values(by="Position"), x="Position", y="Streams", color="Start Week",
title="Distribution of streams per week in 2020")
fig.show()

# ### 3.2. Calculating Weighted Weekly Averages

# +
# Calculating the weighted average for the most important features
weighted_week = df[['Streams', 'Start Week', 'danceability', 'energy','valence']]

for feature in ['danceability', 'energy','valence']:
    weighted_week[feature] = weighted_week[feature] * weighted_week['Streams']

weighted_week = weighted_week.groupby(["Start Week"]).sum()

for feature in ['danceability', 'energy','valence']:
    weighted_week[feature] = weighted_week[feature] / weighted_week['Streams']

# add seperate features for dates
weighted_week["Y"] = weighted_week.index.year
weighted_week["M"] = weighted_week.index.month
weighted_week["W"] = weighted_week.index.week
weighted_week["YMD"] = weighted_week.index.date

weighted_week.head()
# -

fig = px.line(weighted_week, x="YMD", y="valence", color='Y')
fig.show()

fig = px.line(weighted_week, x="W", y="valence", color='Y')
fig.show()

fig = px.box(weighted_week, x="Y", y="valence", color="Y")
fig.show()

# ### 2.3. Observations
#
# 1. There is an overall downward trend in valence over the 4 observed years.  
#
# 2. During the first COVID-19 wave (March 1 - June 1 2020), the valence is lower than in 2017 and 2018, but not very different from 2019.
# 3. The "Summer Peak", that starts at the end of April, is observed much later in 2020 than in other years. The 2020 peak starts exactly one week after the end of the lockdown in the Netherlands. The peak is the steepest observed.  
#
# 4. The "Holiday Peak" in the last weeks of 2020 is the same as in other years. The second/third COVID-19 wave and the second lockdown during that time did not change the valence.
#
# Concluding: Valence (as main factor in determining music preference) did not change drasticly during the COVID-19 pandemic in 2020. Only the start of the "Summer Peak" was delayed and started just after the end of the first lockdown.

# ## 4. Compare with confirmed COVID-19 cases  
# [CoronaWatchNL](https://github.com/J535D165/CoronaWatchNL) contains open data on COVID-19 provided by the local authorities.

# +
cases = pd.read_csv("https://raw.githubusercontent.com/J535D165/CoronaWatchNL/master/data-geo/data-national/RIVM_NL_national.csv", parse_dates=["Datum"])
cases["W"] = cases["Datum"].dt.week

cases.tail(3)
# -

fig = px.line(cases[cases["Type"]=="Totaal"], x="Datum", y="Aantal")
fig.show()

fig = px.line(weighted_week[(weighted_week["Y"]==2020) & (weighted_week["W"]>=cases["W"].min())], x="YMD", y="valence", color='Y')
fig.show()



StarcoderdataJupyterScriptsDedupFiltered