"""
# **Covid-19 EDA in the United States and New York** 
"""
"""
From Wuhan to cruise ships to at least 110 counties within three months, coronavirus (COVID-19) is rapidly sweeping the world, causing million infections and deathtolls. Kaggle has launched several [COVID-19 challenges](http://https://www.kaggle.com/c/covid19-global-forecasting-week-1) to call for better understanding of the disease. This EDA is to visualize the geospatially and demographical analysis of the pandemic across the States. The second part of this project focus on New York. I live in NYC and we have the highest fatalities outside China. Hope this project will help us to determine which factors impact the transmission behavior of COVID-19 in order to prevent the next pandemic.

In this kernel, we will walk through various uses of Matplotlib in this EDA for different types of data analysis. All the interaction effects and the visualization of this EDA is done by Plotly, it may take you awhile to load the kernel.

## CONTENTS

* [Sources of Data](#1)
* [Worldwide spread](#2)
* [How was the COVID-19 explosion in the USA?](#3)
* [New York City - demographical analysis of my hometown](#4)

"""
# This Python 3 environment comes with many helpful analytics libraries installed
# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory

import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# Any results you write to the current directory are saved as output.
"""
Importing the Essential Libraries
"""
# visualization
import matplotlib.pyplot as plt
import plotly.express as px
import plotly
import json

# racing chart
import matplotlib.ticker as ticker
import matplotlib.animation as animation
from IPython.display import HTML
from random import randint
import colorsys
import matplotlib.colors as mc
import re

# Disable the warnings
import warnings
warnings.filterwarnings("ignore")


%matplotlib inline
"""
<a id="1"></a>
## Sources of Data


The data is from Kaggle, New York Times, New York State of Health. Kaggle update the data on weekly basis. New York Times and New York State of Health have daily updates on the COVID-19. This EDA use the dataset as of 04/13/2020. You may find the data from the following links:

https://www.kaggle.com/c/covid19-global-forecasting-week-4New

https://github.com/nytimes/covid-19-data

https://covid19tracker.health.ny.gov/views/NYS-COVID19-Tracker/NYSDOHCOVID-19Tracker-Map?%3Aembed=yes&%3Atoolbar=no&%3Atabs=n
"""
# Import data
world_data = pd.read_csv('../input/covid19-week4/train.csv')
by_counties = pd.read_csv('../input/nytimes/us-counties.csv', index_col = 0)
by_states = pd.read_csv('../input/nytimes/us-states.csv')
"""
<a id = '2'></a>
## Worldwide Spread

The first COVID-19 wave started in China. The second COVID-19 hit Europe. The third COVID-19 now comes to the U.S. The fourth? Could be Africa due to the high population density.

Taiwan shows the successful example of combating the COVID-19. Taiwan also reports very low cases whilst it is one of the closest country of China. The reason? The Chinese Government stopped issuing visas for mainlanders to travel to Taiwan as a 'punishment' of the result of the Taiwan's presidential election.

The map does not show the dynamic of Hong Kong well because Hong Kong is a tiny city in China. It is surpsing that Hong Kong has very low confirmed cases comparing to mainland China. The reason? The Hong Kong anti-extradition protests. To pressure the protestors, the Chinese Government 'penalized' the Hong Kong's economy by banning all tourist groups to Hong Kong since the beginning of the protest. Many mainland Chinese avoid travelling to Hong Kong. 

Moreover, both Hong Kong and Taiwan has learned a painful and valuable lesson from SARS in 2003. Hong Kong has lost 299 during SARS. Therefore, both Hong Kongers and Taiwanese always keep some gloves and mask supplies at home for precaution. 

What else we find from the above worldwide map? Let's breakdown the Choropleth Graphs with the Coronavirus timeline from NBC news.

Jan. 23
COVID-19 have spread from China, with the epicenter of Wuhan.

Feb. 9
The death toll in mainland China reached 811, surpassing the number of fatalities from the SARS outbreak in 2003.

Feb. 4
Egypt became the first country in Africa with the confirmed case.

Feb. 14
France reported the first fatality outside Asia coronavirus death was recorded outside Asia. 

Feb. 19
Iran reported the first fatalities of two patients.

Feb. 24
Italy became the epicenter in Europe.

Feb. 26
Brazil is the first in Latin America to confirm its first coronavirus case.

March 16
Canada's number of confirmed cases hit 339 with one death. 

March 20
Death tolls in Spain surpassed 1,000, as the total number of confirmed cases in the country inched near 20,000.

March 26
U.S. coronavirus cases surpassed China. The U.S. reported at least 82,474, with more than 1,100 deaths.
China reported 81,961 cases and more than 3,000 deaths.

March 27
Italy’s deadliest day with 919 deaths with the accumulative of 9,134.

April 1
Spain’s deadliest day with 864 deaths.

April 11
Coronavirus deaths in the United States passed the 20,000 mark.


reference read:
https://www.nbcnews.com/health/health-news/coronavirus-timeline-tracking-critical-moments-covid-19-n1154341
"""
world_data.head(3)
# Rename the column 'Date'
world_data = world_data.rename(columns={'Date': 'Date_worldwide'})
# The graph will only show countries with confirmed cases
# Each animated graph will be grouped by the date and the country
df_countrydate = world_data[world_data['ConfirmedCases']>0]
df_countrydate = df_countrydate.groupby(['Date_worldwide','Country_Region']).sum().reset_index()
# Create the Choropleth

fig = px.choropleth(df_countrydate, 
                    locations = 'Country_Region', # Spatial coordinates
                    locationmode = 'country names',
                    color = 'ConfirmedCases', 
                    color_continuous_scale = 'tempo',
                    hover_name = 'Country_Region',
                    animation_frame = 'Date_worldwide',
                   )

fig.update_layout(
    title_text = 'Reported Confirmed Cases Worldwide between 1/22/2010 and 04/10/2020',
    title_x = 0.5,
    geo = dict(
        showframe = False,
        showcoastlines = False,
    ))
    
fig.show()
# Create the Choropleth
fig = px.choropleth(df_countrydate, 
                    locations = 'Country_Region', 
                    locationmode = 'country names', # Spatial coordinates
                    color = 'Fatalities', 
                    color_continuous_scale = 'reds',
                    hover_name = 'Country_Region', 
                    animation_frame = 'Date_worldwide',
                   )

fig.update_layout(
    title_text = 'Reported Fatalities Worldwide between 1/22/2010 and 04/10/2020',
    title_x = 0.5,
    geo = dict(
        showframe = False,
        showcoastlines = False,
    ))
    
fig.show()

"""
<a id = '3'></a>

## How was the COVID-19 explosion in the USA?

The bar chart race is inspired by YouHan Lee. I was thinking to do it on Tableau but found Lee has built this amazing kernel. I just added few minor personal touchups. 

Big shoutout:
<br>https://www.kaggle.com/youhanlee/comfirmed-bar-chart-race-depending-on-country

One of my favourite settings of Matplotlib is the style sheets. It is an easy trick to personalize your graphs without any significants change of codes. :)

plt.style.use()

Let's take a look how the COVID-19 spreading across the State. This pandemic is sweeping America with horrifiying speed. Why the disease sweep across America so quickly? One of the major reasons is because Americans do not wear masks; rather, many of them detest other people wear masks in public. Also, America is a very big country with different life-styles. There are still many people in the U.S. do not believe in the pandemic and still doing social events. It makes America into such severe situation and New York City is the epicenter out of the epicenter.

What else we find from the bar chart race? Here are the highlights from timeline from NBC news again.

Feb. 26 California announced the first case in the U.S.

Feb. 29 Washington State recorded the first coronavirus fatality in the U.S.

March 16 San Francisco is the first city in the U.S. to imposed strict prohibitions on residents leaving their homes.

March 21 New York State hit 10,000 confirmed cases.

March 26 U.S. coronavirus cases surpassed China. The U.S. reported at least 82,474, with more than 1,100 deaths. China reported 81,961 cases and more than 3,000 deaths.

April 9 New York State’s deadliest day with 799 deaths overnight with the total fatalities of 7,067.

April 11 The deaths in the U.S. passed the 20,000 mark, with over a half million confirmed cases nationwide. More than half of the deaths were concentrated in three states: New York, with 8,627; New Jersey with 2,183; and Michigan, with 1,392.

Reference read:
<br>https://www.nbcnews.com/health/health-news/coronavirus-timeline-tracking-critical-moments-covid-19-n1154341
"""
# change date format from string to datetime
by_states['date'] =  pd.to_datetime(by_states['date'], format='%Y-%m-%d')
# The racing bar chart is built by a series of charts grouped by 'state' along the date
by_states = by_states.groupby(['date', 'state']).sum().reset_index()
by_states['date'] = by_states['date'].astype('str') # The racing chart will go with string

# define a new dataframe for the racing chart
df = by_states[['date', 'state', 'cases', 'deaths']]
df.columns = ['date', 'state', 'value_confirmed', 'value_deaths']
fnames_list = df['date'].unique().tolist()

def random_color_generator(number_of_colors):
    random.seed(30)
    color = ["#"+''.join([random.choice('987654321ABCDEF') for j in range(6)])
                 for i in range(number_of_colors)]
    return color

state_list = df['state'].unique().tolist()

num_of_elements = 10
def transform_color(color, amount = 0.5):

    try:
        c = mc.cnames[color]
    except:
        c = color
        c = colorsys.rgb_to_hls(*mc.to_rgb(c))
    return colorsys.hls_to_rgb(c[0], 1 - amount * (1 - c[1]), c[2])

random_hex_colors = []
for i in range(len(state_list)):
    random_hex_colors.append('#' + '%06X' % randint(0, 0xFFFFFF))

rgb_colors = [transform_color(i, 1) for i in random_hex_colors]
rgb_colors_opacity = [rgb_colors[x] + (0.825,) for x in range(len(rgb_colors))]
rgb_colors_dark = [transform_color(i, 1.12) for i in random_hex_colors]

normal_colors = dict(zip(state_list, rgb_colors_opacity))
dark_colors = dict(zip(state_list, rgb_colors_dark))
# Let's check the different available style sheets on Matplotlib
print(plt.style.available)
plt.style.use('grayscale')
fig, ax = plt.subplots(figsize = (36, 20))

def draw_barchart(current_date):
    dff = df[df['date'].eq(current_date)].sort_values(by='value_confirmed', ascending=True).tail(num_of_elements)
    ax.clear()
    
    ax.barh(dff['state'], dff['value_confirmed'], color=[normal_colors[p] for p in dff['state']],
                edgecolor =([dark_colors[x] for x in dff['state']]), linewidth = '6')
    dx = dff['value_confirmed'].max() / 200


    for i, (value, name) in enumerate(zip(dff['value_confirmed'], dff['state'])):
        ax.text(value + dx, 
                i + (num_of_elements / 50), '    ' + name,
                size = 32,
                ha = 'left',
                va = 'center',
                fontdict = {'fontname': 'Trebuchet MS'})

        ax.text(value + dx,
                i - (num_of_elements / 50), 
                f'    {value:,.0f}', 
                size = 32, 
                ha = 'left', 
                va = 'center')   
    
    
    time_unit_displayed = re.sub(r'\^(.*)', r'', str(current_date))
    ax.text(1, 
            0.5, 
            time_unit_displayed,
            transform = ax.transAxes, 
            color = '#777777',
            size = 55,
            ha = 'right', 
            weight = 'bold', 
            fontdict = {'fontname': 'Trebuchet MS'})

    ax.text(-0.005, 
            1.05, 
            'cases', 
            transform = ax.transAxes, 
            size = 40, 
            color = '#666666')

    ax.text(0.07, 
            1.1, 
            'Confirmed Cases in the United States from 2020-01-21 to 2020-04-13', 
            transform = ax.transAxes,
            size = 50, 
            weight = 'bold', 
            color = 'royalblue',
            ha = 'left')

    ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
    ax.xaxis.set_ticks_position('top')
    ax.tick_params(axis = 'x', colors = '#666666', labelsize = 28)
    ax.set_yticks([])
    ax.set_axisbelow(True)
    ax.margins(0, 0.01)
    ax.grid(which = 'major', axis = 'x', linestyle = '-')
    

    plt.locator_params(axis = 'x', nbins = 4)
    plt.box(False)
    plt.subplots_adjust(left = 0.075, right = 0.75, top = 0.825, bottom = 0.05, wspace = 0.2, hspace = 0.2)
    plt.box(False)    

draw_barchart('2020-04-13')
fig, ax = plt.subplots(figsize = (36, 20))
animator = animation.FuncAnimation(fig, draw_barchart, frames=fnames_list)
racing_chart = HTML(animator.to_jshtml())

racing_chart
plt.style.use('dark_background')
fig, ax = plt.subplots(figsize = (36, 20))

def draw_barchart(current_date):
    dff = df[df['date'].eq(current_date)].sort_values(by='value_deaths', ascending=True).tail(num_of_elements)
    ax.clear()
    
    ax.barh(dff['state'], dff['value_deaths'], color=[normal_colors[p] for p in dff['state']],
                edgecolor =([dark_colors[x] for x in dff['state']]), linewidth = '6')
    dx = dff['value_deaths'].max() / 200


    for i, (value, name) in enumerate(zip(dff['value_deaths'], dff['state'])):
        ax.text(value + dx, 
                i + (num_of_elements / 50), '    ' + name,
                size = 32,
                ha = 'left',
                va = 'center',
                fontdict = {'fontname': 'Trebuchet MS'})

        ax.text(value + dx,
                i - (num_of_elements / 50), 
                f'    {value:,.0f}', 
                size = 32, 
                ha = 'left', 
                va = 'center')   
    
    
    time_unit_displayed = re.sub(r'\^(.*)', r'', str(current_date))
    ax.text(1, 
            0.5, 
            time_unit_displayed,
            transform = ax.transAxes, 
            color = '#777777',
            size = 55,
            ha = 'right', 
            weight = 'bold', 
            fontdict = {'fontname': 'Trebuchet MS'})

    ax.text(-0.005, 
            1.05, 
            'cases', 
            transform = ax.transAxes, 
            size = 40, 
            color = '#666666')

    ax.text(0.2, 
            1.1, 
            'Fatalities in US from 2020-01-21 to 2020-04-13', 
            transform = ax.transAxes,
            size = 50, 
            weight = 'bold',
            color = 'firebrick',
            ha = 'left')

    ax.xaxis.set_major_formatter(ticker.StrMethodFormatter('{x:,.0f}'))
    ax.xaxis.set_ticks_position('top')
    ax.tick_params(axis = 'x', colors = '#666666', labelsize = 28)
    ax.set_yticks([])
    ax.set_axisbelow(True)
    ax.margins(0, 0.01)
    ax.grid(which = 'major', axis = 'x', linestyle = '-')
    

    plt.locator_params(axis = 'x', nbins = 4)
    plt.box(False)
    plt.subplots_adjust(left = 0.075, right = 0.75, top = 0.825, bottom = 0.05, wspace = 0.2, hspace = 0.2)
    plt.box(False)    
draw_barchart('2020-04-13')
fig, ax = plt.subplots(figsize = (36, 20))
animator = animation.FuncAnimation(fig, draw_barchart, frames=fnames_list)
chart_race = HTML(animator.to_jshtml())
chart_race
"""
<a id = '4'></a>
## New York City - a demographical analysis of my hometown

I heard the siren of ambulance at least twice per hour since few weeks  ago. At the apex day, I counted 6 sirens within an hour. We are running out of medical supplies and the hospitals were flooded with patients. Our healthcare system was collapsing. New York City in the epicenter of the epicenter.

'What makes Asian community the low fatalities in New York City?'You may be asking. As an Asian, I have few assumptions. Asians have aknowledged the pandemic will hit to the States at its first outbreak in China. We started wearing masks way before CDC advised the public to do so. The next thought may not be pleasant: many Asian businesses close weeks before the non-Asian businesses did. Many New Yorkers stopped going to Asian stores during the pandemic. There are even reported violents against the Asian community across the U.S. Also, the majority of the second-generation Asian American work at tech or corporate jobs, which allow them to work fromhome. As many Asian do not commute to work due to job loss or remote work, the infections rate among Asian communties slow down. Unfortunately, it also means the low-paid 'essential workers' are potentially vulnerble COVID-19. I will make another analysis on wealth distribution and COVID-19 in New York City later.

I hope this data will demostrate the non-Asian community the importance of wearing masks after the pandemic. The COVID-19 is a wake-up call to us that no country can spare from the globalization. Pandemic treat everyone equally. We all are responsible to the more acurate and updated data..

Suggested reading:
<br>https://www.nytimes.com/aponline/2020/04/08/us/ap-us-virus-outbreak-race.html
<br>https://thehill.com/changing-america/respect/equality/490373-attacks-on-asian-americans-at-about-100-per-day-due-to
"""
def pie(ax, values, **kwargs):
    total = sum(values)
    def formatter(pct):
        return '{:0.0f}%'.format(pct*total/100)
        #return '${:0.0f}M\n({:0.1f}%)'.format(pct*total/100, pct)
    wedges, _, labels = ax.pie(values, autopct=formatter, **kwargs)
    return wedges

plt.style.use('classic')
fig, ax = plt.subplots(figsize=(10,10))
fig.suptitle('Fatalities by Race/ Ethnicity from Covid-19 in New York City', color='black', fontsize=22)

ax.axis('equal')

width = 0.35
kwargs = dict(colors=['#9999ff', '#ffff99', '#99ff99', '#ff9999', '#99ccff'], startangle=90)

outside = pie(ax, [29, 22, 32, 14, 3], radius=1, pctdistance=1-width/2, **kwargs)
inside = pie(ax, [34, 28, 27, 7, 4], radius=1-width,
             pctdistance=1 - (width/2) / (1-width), **kwargs)
plt.setp(inside + outside, width=width, edgecolor='white')

ax.legend(inside[:], ['Hispanic', 'Black', 'White', 'Asian','Other' ], prop={'size': 13}, frameon=False)

kwargs = dict(size=15, va='center', fontweight='bold')
ax.text(0, 0, '% of Fatalities', ha='center', color='red',
        bbox=dict(boxstyle='round', facecolor='silver', edgecolor='none'),
        **kwargs)
ax.annotate('% of NYC Population', (0, 0), color='blue', xytext=(np.radians(-45), 1.1) ,
            bbox=dict(boxstyle='round', facecolor='silver', edgecolor='none'),
            textcoords='polar', ha='left', **kwargs)

plt.show()
# the bar chart will use the time as index and the x-axis
by_counties.index = pd.to_datetime(by_counties.index)

# take of the unnecessary columns
by_counties = by_counties.drop(columns=['fips', 'state'])
by_counties = by_counties.loc[by_counties['county'] == 'New York City']
plt.rcParams['figure.figsize']=(20,10) # set the figure size # set the figure size
plt.style.use('fivethirtyeight') # using the fivethirtyeight matplotlib theme

# Divide the timeframe into three segments for different colors
first = by_counties[(by_counties.index >= '2020-03-01') & (by_counties.index < '2020-03-15')]
second = by_counties[(by_counties.index >= '2020-03-15') & (by_counties.index < '2020-03-31')]
third = by_counties[(by_counties.index >= '2020-04-01') & (by_counties.index < '2020-04-13')]

# Build our plot
fig, ax1 = plt.subplots()
ax2 = ax1.twinx()  # set up the 2nd axis
ax1.plot(by_counties.deaths) #plot the Revenue on axis #1

# the next few lines plot the fiscal year data as bar plots and changes the color for each.
ax2.bar(first.index, first.cases,width=0.5, alpha=0.5, color='green')
ax2.bar(second.index, second.cases,width=0.5, alpha=0.5, color='blue')
ax2.bar(third.index, third.cases,width=0.5, alpha=0.5, color='orange')

ax2.grid(b=False) # turn off grid #2

ax1.set_title('Confirmed Cases and Fatalites in NYC from 2020-03-01 to 2020-04-13)', color='black')
ax1.set_ylabel('Fatalies')
ax2.set_ylabel('Confirmed Cases')

# Set the x-axis labels to be more meaningful than just some random dates.
labels = ['2020-03-01','2020-03-15','2020-04-01'] 
ax1.set_xticks(labels)
ax1.axes.set_xticklabels(labels)