## Unemployment in different countries and age groups

**Team:** Aristochats

**Members:** Theresa Berz (txj188), Adrian Moise (bln333), Nam Anh Nguyen (xgw631), Karen Thule (lnc394)

In this project we want to investigate the unemployment rate in six different OECD countries over the time period 2007 to 2019. Using data from OECD statistics we first examine how the unemployment rate has developped over the timespan across countries and different ages groups. This in done in order to see if there is any patterns or significant differences across country or age.

**Data**

The data we use is the unemploymentrate in different age groups taken from the OECD database: 'Labour market statistics' and later in the porject we add data for GDP using GDP per capita in current US dollars, which is also taken from the OECD.

The variables are:
- **U_15_24** : Unemployment rate age 15-24
- **U_15_64** : Unemployment rate age 15-64
- **U_25_54** : Unemployment rate age 25-54
- **U_55_64** : Unemployment rate age 55-64
- **GDP_USD** : GDP per. capita in US dollars

## Load and clean data

```python
#Import packages
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
import matplotlib as mpl
%matplotlib inline
plt.style.use('seaborn-whitegrid')

from IPython.display import Markdown, display
from numpy import array
import sympy as sm
```

First we load the data, that can be found in the dataproject folder:

```python
#Load unemployment dataset from folder
filename = 'data_u.xlsx' # open the file and have a look at it
data=pd.read_excel(filename)
data
```

      Country
      Time
      U_15_24
      U_15_64
      U_25_54
      U_55_64

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      373
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3

      374
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2

      375
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7

      376
      Netherlands
      Q1-2020
      6.6
      3.2
      2.4
      2.6

      377
      Netherlands
      Q2-2020
      9.7
      3.9
      2.8
      2.5

378 rows × 6 columns

We then want to investgate if there is any missing values. This is done by using the `isnull()` function that seaches for any missing values in the data-set.

```python
# Search for any missing values
is_NaN = data.isnull()
row_has_NaN = is_NaN.any(axis=1)

#define a dataset with the missing values
rows_with_NaN = data[row_has_NaN]

# print the observations with missing values
rows_with_NaN

```

      Country
      Time
      U_15_24
      U_15_64
      U_25_54
      U_55_64

      160
      Germany
      Q1-2020
      NaN
      NaN
      NaN
      NaN

      161
      Germany
      Q2-2020
      NaN
      NaN
      NaN
      NaN

      214
      OECD - Average
      Q1-2020
      11.7
      5.5
      4.8
      NaN

      215
      OECD - Average
      Q2-2020
      17.7
      8.6
      7.5
      NaN

We want to remove the observations with missing values, this is done by using the `notna()`which does the oposite of `isnull()` i.e it finds all the observations with values which we then keep.

```python
#only keep observations with values (no data for 2020 unemployment in Germany and OECD-Average)
I = data['U_55_64'].notna() #use the U_55_64 since we have seen data is missing here
data = data[I] #overwrite data with data containing values

#rename time to quarter
rename_dict = {}
rename_dict['Time'] = 'Quarter'
data.rename(columns=rename_dict,inplace=True)
data

```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      373
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3

      374
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2

      375
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7

      376
      Netherlands
      Q1-2020
      6.6
      3.2
      2.4
      2.6

      377
      Netherlands
      Q2-2020
      9.7
      3.9
      2.8
      2.5

374 rows × 6 columns

# Descriptive statistic

We are now ready to look at the data. First we want to examine the data. In order to do so we use `groupby`  and `describe`

```python
var = ['U_15_24', 'U_15_64']
data.groupby('Country')[var ].describe()
```

      U_15_24
      U_15_64

      count
      mean
      std
      min
      25%
      50%
      75%
      max
      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Country

      Austria
      54.0
      9.729630
      1.054841
      7.6
      9.125
      9.60
      10.400
      11.8
      54.0
      5.161111
      0.588704
      3.8
      4.800
      5.10
      5.600
      6.3

      Belgium
      54.0
      19.653704
      3.346004
      12.2
      17.325
      20.25
      21.575
      26.2
      54.0
      7.427778
      1.082348
      5.1
      6.925
      7.65
      8.375
      8.7

      Denmark
      54.0
      12.600000
      2.684688
      7.2
      10.525
      12.50
      14.975
      17.0
      54.0
      6.246296
      1.437072
      3.4
      5.225
      6.30
      7.600
      8.3

      France
      54.0
      22.081481
      2.234386
      17.0
      20.325
      22.50
      23.850
      25.5
      54.0
      9.079630
      1.020984
      6.9
      8.625
      9.10
      9.975
      10.6

      Germany
      52.0
      8.359615
      1.920504
      5.4
      6.900
      7.85
      9.725
      12.4
      52.0
      5.584615
      1.736842
      3.2
      4.150
      5.35
      7.150
      9.2

      Netherlands
      54.0
      10.051852
      1.967351
      6.5
      8.625
      10.15
      11.375
      13.6
      54.0
      5.162963
      1.381180
      3.2
      3.925
      5.00
      6.275
      7.7

      OECD - Average
      52.0
      14.448077
      2.107269
      11.4
      12.400
      14.35
      16.525
      17.6
      52.0
      7.117308
      1.133919
      5.4
      5.975
      7.10
      8.200
      8.8

```python
var2 = ['U_25_54', 'U_55_64']
data.groupby('Country')[var2].describe()
```

      U_25_54
      U_55_64

      count
      mean
      std
      min
      25%
      50%
      75%
      max
      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Country

      Austria
      54.0
      4.588889
      0.575823
      3.4
      4.200
      4.60
      5.100
      5.5
      54.0
      3.598148
      0.832022
      1.9
      3.200
      3.60
      4.000
      5.2

      Belgium
      54.0
      6.538889
      0.952141
      4.4
      6.125
      6.65
      7.375
      8.0
      54.0
      3.598148
      0.832022
      1.9
      3.200
      3.60
      4.000
      5.2

      Denmark
      54.0
      5.272222
      1.285098
      2.5
      4.500
      5.45
      6.275
      7.0
      54.0
      4.420370
      1.098606
      2.3
      3.600
      4.15
      5.500
      6.3

      France
      54.0
      7.911111
      0.978768
      6.0
      7.400
      8.00
      8.700
      9.4
      54.0
      6.162963
      1.096529
      3.8
      5.700
      6.35
      7.075
      7.7

      Germany
      52.0
      5.138462
      1.581034
      3.0
      3.850
      4.85
      6.650
      8.5
      52.0
      5.763462
      2.280783
      2.6
      3.750
      5.70
      7.725
      11.1

      Netherlands
      54.0
      3.996296
      1.279145
      2.3
      2.800
      3.80
      5.025
      6.4
      54.0
      5.224074
      1.646028
      2.5
      4.200
      4.70
      6.675
      8.2

      OECD - Average
      52.0
      6.251923
      1.013341
      4.8
      5.100
      6.30
      7.200
      7.7
      52.0
      4.915385
      0.811531
      3.7
      4.075
      5.00
      5.700
      6.1

From the above descriptive statistics we see that generally the unemployment rate is highest for the youngest group age 15-24 in all countries as well as for the OECD average, where France has the higest mean of 22.08, and a max of the period of 25.5 pct, i.e a some point during the period a quarter of the persons in the age 15-24 were unemployed in France. The lowest means are found in the oldest age group 55-64, except for Germany, where the lowest mean is found in the age group 25-54.

We now want to plot the unemployment rate of each country in order to see, how they have developed over the time period. This is done via. a loop over the four age groups.

```python
#plot the first age group by looping over the four unemployment groups

def plot_unempl(column,name,title=''):
    fig = plt.figure()
    ax = plt.subplot(111)
    data.set_index('Quarter').groupby('Country')[column].plot(kind='line', legend=True, ax=ax)
    ax.set_ylabel(name)
    ax.set_title(title)
    box = ax.get_position()
    ax.set_title(f'Development in the {name} age group',fontweight="bold")
    ax.set_position([box.x0, box.y0 + box.height * 0.1,box.width, box.height * 0.9]) # shrink height by 10% at bottom
    ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.15),ncol=5); # Put a legend below current axis
columns = ['U_15_24','U_15_64','U_25_54','U_55_64']
names   = ['Unemployment 15-24','Unemployment 15-64','Unemployment 25-64','Unemployment 55-64']
for column,name in zip(columns,names):
    plot_unempl(column,name)

```

When looking at the figures, one sees a pattern of falling unemployment rates in all countries in the quarters leading up to the financial crisis in 2008, whereafter the general pattern is rising unemployment rates. In the three first age group (15-24, 15-64 and 25-54) France and Belgium has had the highest rates of unemployment throughout most of the investigated period, while Austria seems to have had a lower and a bit more steady unemployment rate.
When looking at Germany we see quite a different pattern compared to the rest of the countries and the OECD - Average. Germany has (except for small jumps around 2008) had a clear falling trend throughout the entire period, the strongest one for the oldest age group and the weakest for the youngest. This shows, that even though we are comparing six OECD countries, that have all been exposed to the financial crisis and what followed, we are able to see very different patterns. This leads us to asking the question, what might explain this difference. However, before doing so, we will have a look at how each country is compared to the OECD average.

## Deviation from OECD

First we create a list to make it possible to drop 'Country', and 'Quarter' for the loop that follows

```python
list__ = data.columns.drop(pd.Index(['Country','Quarter']))
list__

```

    Index(['U_15_24', 'U_15_64', 'U_25_54', 'U_55_64'], dtype='object')

We create a loop that takes the OECD - Average data for all unemployment groups and stores it as seperate columns.

```python
for name in list__:
    B = (data['Country']=='OECD - Average')
    New2 = data.loc[B,['Quarter',name]].rename(columns={name:f'{name}_OECD'})
    data = data.merge(New2,on='Quarter', how='left')

data
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      369
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3
      11.6
      5.6
      4.9
      3.8

      370
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2
      11.9
      5.6
      4.9
      3.7

      371
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7
      11.5
      5.4
      4.8
      3.7

      372
      Netherlands
      Q1-2020
      6.6
      3.2
      2.4
      2.6
      NaN
      NaN
      NaN
      NaN

      373
      Netherlands
      Q2-2020
      9.7
      3.9
      2.8
      2.5
      NaN
      NaN
      NaN
      NaN

374 rows × 10 columns

Now that we have the OECD - Average for all groups on seperate columns, we are able to create new variables. We loop over the unemployment groups and use the `lambda` function in order to store the results as columns.

```python
age_groups= ['15_24','15_64','25_54','55_64']
for age_group in age_groups:
        data[f'diff_oecd_{age_group}'] = data.apply(lambda x: x[f'U_{age_group}_OECD'] - x[f'U_{age_group}'], axis=1)
data
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2
      5.3
      2.0
      1.5
      0.6

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0
      5.0
      2.1
      2.1
      0.1

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0
      4.1
      2.0
      1.9
      0.6

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0
      5.1
      2.3
      2.1
      1.2

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9
      3.7
      2.4
      2.4
      1.1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      369
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3
      11.6
      5.6
      4.9
      3.8
      5.1
      2.3
      2.4
      0.5

      370
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2
      11.9
      5.6
      4.9
      3.7
      5.2
      2.3
      2.4
      0.5

      371
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7
      11.5
      5.4
      4.8
      3.7
      4.5
      2.0
      2.2
      1.0

      372
      Netherlands
      Q1-2020
      6.6
      3.2
      2.4
      2.6
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      373
      Netherlands
      Q2-2020
      9.7
      3.9
      2.8
      2.5
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

374 rows × 14 columns

We see that there are some missing values, therefor we remove these as done previous using `notna()`

```python
# Search for any missing values
is_NaN = data.isnull()
row_has_NaN = is_NaN.any(axis=1)

#define a dataset with the missing values
rows_with_NaN = data[row_has_NaN]

# print the observations with missing values
rows_with_NaN
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64

      52
      Denmark
      Q1-2020
      10.0
      5.1
      4.3
      3.7
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      53
      Denmark
      Q2-2020
      11.7
      5.5
      4.6
      4.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      106
      France
      Q1-2020
      19.1
      7.9
      6.8
      5.9
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      107
      France
      Q2-2020
      21.0
      7.2
      6.1
      4.7
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      264
      Austria
      Q1-2020
      9.6
      4.6
      3.8
      3.8
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      265
      Austria
      Q2-2020
      11.8
      5.8
      5.3
      3.7
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      318
      Belgium
      Q1-2020
      12.6
      5.1
      4.5
      3.8
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      319
      Belgium
      Q2-2020
      16.1
      5.1
      4.4
      3.7
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      372
      Netherlands
      Q1-2020
      6.6
      3.2
      2.4
      2.6
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      373
      Netherlands
      Q2-2020
      9.7
      3.9
      2.8
      2.5
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

```python
#only keep observations with values
I = data['U_55_64_OECD'].notna()
data = data[I]

#round to 1 decimal
data = data.round(decimals=1)

#rename time to quarter
rename_dict = {}
rename_dict['Time'] = 'Quarter'
data.rename(columns=rename_dict,inplace=True)
data
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2
      5.3
      2.0
      1.5
      0.6

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0
      5.0
      2.1
      2.1
      0.1

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0
      4.1
      2.0
      1.9
      0.6

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0
      5.1
      2.3
      2.1
      1.2

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9
      3.7
      2.4
      2.4
      1.1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      367
      Netherlands
      Q4-2018
      6.8
      3.6
      2.6
      4.2
      11.8
      5.6
      5.0
      3.9
      5.0
      2.0
      2.4
      -0.3

      368
      Netherlands
      Q1-2019
      6.7
      3.5
      2.6
      3.6
      11.8
      5.7
      5.0
      3.9
      5.1
      2.2
      2.4
      0.3

      369
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3
      11.6
      5.6
      4.9
      3.8
      5.1
      2.3
      2.4
      0.5

      370
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2
      11.9
      5.6
      4.9
      3.7
      5.2
      2.3
      2.4
      0.5

      371
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7
      11.5
      5.4
      4.8
      3.7
      4.5
      2.0
      2.2
      1.0

364 rows × 14 columns

We are now able to plot the results. y=0 is the OECD average. Thus, for all positive values of y>0, the country's respective unemployment rate is lower than the OECD average and vice versa.

```python
#create a loop that plots the diviation from OECD average
def plot_unempl_deviation(column,name,title=''):
    fig = plt.figure()
    ax = plt.subplot(111)
    data.set_index('Quarter').groupby('Country')[column].plot(kind='line', legend=True, ax=ax)
    ax.set_ylabel(name)
    ax.set_title(title)
    box = ax.get_position()
    ax.set_title(f'Deviation in unemployment from OECD average for {name}',fontweight="bold")
    ax.set_position([box.x0, box.y0 + box.height * 0.1,box.width, box.height * 0.9]) # shrink height by 10% at bottom
    ax.legend(loc='upper center', bbox_to_anchor=(0.5, -0.15),ncol=5); # Put a legend below current axis
columns = ['diff_oecd_15_24','diff_oecd_15_64','diff_oecd_25_54','diff_oecd_55_64']
names   = ['Unemployment, 15-24','Unemployment, 15-64','Unemployment, 25-64','Unemployment, 55-64']
for column,name in zip(columns,names):
    plot_unempl_deviation(column,name)

```

The plots display that the unemployment devation from OECD average varies a lot between the six chosen countries, where especially France stands out with negative diviation in all age groups except the age group 55-65. Notice that, except for the youngest age group, Germany has the highest negative deviation for all age groups, in the beginning of the time periode i.e. the highest unemployment rate compared to the OECD-Average. However in the following years they "catch up" and move towards a positive deviation i.e a lower unemployment rate than OECD average.

## Adding GDP

In the previous tables and figures we have seen how six different OECD countries have had very different unemployment rates - both in terms of levels and developments. This naturally leads to asking what might have caused this difference. As mentioned before, one factor that might explain some of the differences could be the financial crisis. We therefore try to add GDP to the data in order to see, if there might be any connection between the development in the two variables.

We therefore load another datafile containing data for GDP per capita in the six chosen countries.

```python
#load GDP dataset from folder
filename = 'data_gdp.xlsx' # open the file and have a look at it
dataGDP=pd.read_excel(filename)

#rename time to quarter
rename_dict = {}
rename_dict['Time'] = 'Quarter'
dataGDP.rename(columns=rename_dict,inplace=True)
dataGDP
```

      Country
      Quarter
      GDP_USD

      0
      Denmark
      Q1-2007
      38222.3

      1
      Denmark
      Q2-2007
      38342.6

      2
      Denmark
      Q3-2007
      39136.4

      3
      Denmark
      Q4-2007
      40178.8

      4
      Denmark
      Q1-2008
      40983.6

      ...
      ...
      ...
      ...

      373
      Netherlands
      Q2-2019
      59387.3

      374
      Netherlands
      Q3-2019
      59627.0

      375
      Netherlands
      Q4-2019
      59985.6

      376
      Netherlands
      Q1-2020
      59243.9

      377
      Netherlands
      Q2-2020
      53934.0

378 rows × 3 columns

We then merge it with the unemployment data

```python
#merge gdp data and unemployment data
data2 = pd.merge(data,dataGDP,on=['Country','Quarter'],how='outer')
data2
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64
      GDP_USD

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2
      5.3
      2.0
      1.5
      0.6
      38222.3

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0
      5.0
      2.1
      2.1
      0.1
      38342.6

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0
      4.1
      2.0
      1.9
      0.6
      39136.4

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0
      5.1
      2.3
      2.1
      1.2
      40178.8

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9
      3.7
      2.4
      2.4
      1.1
      40983.6

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      373
      Austria
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      50794.5

      374
      Belgium
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      53695.7

      375
      Belgium
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      47097.7

      376
      Netherlands
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      59243.9

      377
      Netherlands
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      53934.0

378 rows × 15 columns

As earlier we seach for any missing values and remove them

```python
# Search for any missing values
is_NaN = data2.isnull()
row_has_NaN = is_NaN.any(axis=1)
rows_with_NaN = data2[row_has_NaN]

rows_with_NaN
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64
      GDP_USD

      364
      Denmark
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      60786.6

      365
      Denmark
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      56527.9

      366
      France
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      46793.8

      367
      France
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      39965.9

      368
      Germany
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      55401.2

      369
      Germany
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      49784.8

      370
      OECD - Average
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      46186.6

      371
      OECD - Average
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      41132.4

      372
      Austria
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      57203.7

      373
      Austria
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      50794.5

      374
      Belgium
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      53695.7

      375
      Belgium
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      47097.7

      376
      Netherlands
      Q1-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      59243.9

      377
      Netherlands
      Q2-2020
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      53934.0

```python
#remove missing values
I = data2['U_15_64_OECD'].notna()
data2 = data2[I]
data2

```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64
      GDP_USD

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2
      5.3
      2.0
      1.5
      0.6
      38222.3

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0
      5.0
      2.1
      2.1
      0.1
      38342.6

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0
      4.1
      2.0
      1.9
      0.6
      39136.4

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0
      5.1
      2.3
      2.1
      1.2
      40178.8

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9
      3.7
      2.4
      2.4
      1.1
      40983.6

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      359
      Netherlands
      Q4-2018
      6.8
      3.6
      2.6
      4.2
      11.8
      5.6
      5.0
      3.9
      5.0
      2.0
      2.4
      -0.3
      58596.7

      360
      Netherlands
      Q1-2019
      6.7
      3.5
      2.6
      3.6
      11.8
      5.7
      5.0
      3.9
      5.1
      2.2
      2.4
      0.3
      58904.7

      361
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3
      11.6
      5.6
      4.9
      3.8
      5.1
      2.3
      2.4
      0.5
      59387.3

      362
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2
      11.9
      5.6
      4.9
      3.7
      5.2
      2.3
      2.4
      0.5
      59627.0

      363
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7
      11.5
      5.4
      4.8
      3.7
      4.5
      2.0
      2.2
      1.0
      59985.6

364 rows × 15 columns

Before looking at umeployment and GDP together, we want to have a look at the development in the GDP per capita in the six countires and the OECD-average over the chosen period. We therefor plot the GDP data.

```python
# plot GDP
fig = plt.figure()
ax = plt.subplot(111)
data2.set_index("Quarter").groupby("Country")["GDP_USD"].plot(kind="line", legend=True, ax=ax)
ax.yaxis.set_major_formatter(mpl.ticker.StrMethodFormatter('{x:,.0f}')) #allows for thousand number seperator
ax.set_ylabel("GDP per capita")
ax.set_title("Development in quarterly GDP", fontweight="bold")
box = ax.get_position()
ax.set_position([box.x0, box.y0 + box.height * 0.1,box.width, box.height * 1])

```

Previously we showed that all countries had very different patterns of unemployment development. We have also mentioned the financial crisis as a factor that led to a decrease in employment. Thus, one could think, that unemployment and the economic development of a country could be very related . However, the above figure shows that the causal relationship between employment and the economy, aproximated by GDP, might not be as significant as one might thought. We can see that despite significant difference in the change en unemploment rate all countries had a very similar development in GDP over the chosen time period.

## Correlation between unemployment and GDP

In order to further investigate if the unemployment and the economic development of a country could be related, we plot the log difference between unemployment rate according to the log difference between the GDP per capita, i.e. checking if the variation of unemployment rate is related to the variation of GDP per capita.

First we drop the OECD-Average since we only want to look at the six countries:

```python
pd.options.mode.chained_assignment = None

#Dropping the OCED - Average
data2.drop(data2[data2["Country"] == "OECD - Average"].index, inplace=True)
```

We then apply logs to every variable using a loop, and apply `lambda` to store the new calculations as columns:

```python
#Applying logs on different age groupes
new_groups= ["U_15_24","U_15_64","U_25_54","U_55_64","GDP_USD"]
for new_group in new_groups:
        data2[f"log_{new_group}"] = data2[new_group].apply(lambda x: np.log(x))
data2
```

      Country
      Quarter
      U_15_24
      U_15_64
      U_25_54
      U_55_64
      U_15_24_OECD
      U_15_64_OECD
      U_25_54_OECD
      U_55_64_OECD
      diff_oecd_15_24
      diff_oecd_15_64
      diff_oecd_25_54
      diff_oecd_55_64
      GDP_USD
      log_U_15_24
      log_U_15_64
      log_U_25_54
      log_U_55_64
      log_GDP_USD

      0
      Denmark
      Q1-2007
      7.2
      4.0
      3.6
      3.6
      12.5
      6.0
      5.1
      4.2
      5.3
      2.0
      1.5
      0.6
      38222.3
      1.974081
      1.386294
      1.280934
      1.280934
      10.551174

      1
      Denmark
      Q2-2007
      7.3
      3.8
      2.9
      3.9
      12.3
      5.9
      5.0
      4.0
      5.0
      2.1
      2.1
      0.1
      38342.6
      1.987874
      1.335001
      1.064711
      1.360977
      10.554317

      2
      Denmark
      Q3-2007
      8.3
      3.9
      3.1
      3.4
      12.4
      5.9
      5.0
      4.0
      4.1
      2.0
      1.9
      0.6
      39136.4
      2.116256
      1.360977
      1.131402
      1.223775
      10.574808

      3
      Denmark
      Q4-2007
      7.3
      3.6
      2.9
      2.8
      12.4
      5.9
      5.0
      4.0
      5.1
      2.3
      2.1
      1.2
      40178.8
      1.987874
      1.280934
      1.064711
      1.029619
      10.601095

      4
      Denmark
      Q1-2008
      8.7
      3.5
      2.6
      2.8
      12.4
      5.9
      5.0
      3.9
      3.7
      2.4
      2.4
      1.1
      40983.6
      2.163323
      1.252763
      0.955511
      1.029619
      10.620927

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      359
      Netherlands
      Q4-2018
      6.8
      3.6
      2.6
      4.2
      11.8
      5.6
      5.0
      3.9
      5.0
      2.0
      2.4
      -0.3
      58596.7
      1.916923
      1.280934
      0.955511
      1.435085
      10.978434

      360
      Netherlands
      Q1-2019
      6.7
      3.5
      2.6
      3.6
      11.8
      5.7
      5.0
      3.9
      5.1
      2.2
      2.4
      0.3
      58904.7
      1.902108
      1.252763
      0.955511
      1.280934
      10.983676

      361
      Netherlands
      Q2-2019
      6.5
      3.3
      2.5
      3.3
      11.6
      5.6
      4.9
      3.8
      5.1
      2.3
      2.4
      0.5
      59387.3
      1.871802
      1.193922
      0.916291
      1.193922
      10.991836

      362
      Netherlands
      Q3-2019
      6.7
      3.3
      2.5
      3.2
      11.9
      5.6
      4.9
      3.7
      5.2
      2.3
      2.4
      0.5
      59627.0
      1.902108
      1.193922
      0.916291
      1.163151
      10.995864

      363
      Netherlands
      Q4-2019
      7.0
      3.4
      2.6
      2.7
      11.5
      5.4
      4.8
      3.7
      4.5
      2.0
      2.2
      1.0
      59985.6
      1.945910
      1.223775
      0.955511
      0.993252
      11.001860

312 rows × 20 columns

We now want to plot the correlation between unemployment rate and GDP per capita. First we create a dictionary in order to have proper names for our variables

```python
#We create a dictionary to attach a proper name for our variables
var = {}
var["log_U_15_24"] = "log unemployment 15-24 age group"
var["log_U_15_64"] = "log unemployment 15-64 age group"
var["log_U_25_54"] = "log unemployment 25-54 age group"
var["log_U_55_64"] = "log unemployment 55-64 age group"
var["log_GDP_USD"] = "log GDP per capita"

#Defining the axis
x_col = "log_GDP_USD"
y_columns = ["log_U_15_24","log_U_15_64","log_U_25_54","log_U_55_64"]

for y_col in y_columns:

    figure = plt.figure()
    ax = plt.gca()
    ax.scatter(data2[x_col].diff(1), data2[y_col].diff(1),color = "#FFC0CB")
    ax.set_xlim([-0.04,0.03])
    ax.set_ylim([-0.4,0.3])
    ax.set_xlabel(f"{var[x_col]}")
    ax.set_ylabel(f"{var[y_col]}")
    ax.set_title(f"Correlation plot for {var[y_col]}", fontweight="bold")
    plt.show()
```

The correlation coefficient between the unemployment rate and the GDP per capita is then found as:

```python

def fancy(string):
    """ A function that allows us to write markdown text (and LaTex) inside the fancy function
    args:
    string : a string
    returns : a string i an fancy way
    """
    display(Markdown(string))

for y_col in y_columns:
    """A loop that calculates the correlation coefficient
    args:
    y_col: log of unemployment for different age groups
    x_col: log of GDP per capita

    returns : a float
    """
    corr = data2[y_col].corr(data2[x_col])
    fancy(f"$\hat\sigma_{{YX}}=${corr:.3} for {var[y_col]} and {var[x_col]}.")

```

$\hat\sigma_{YX}=$-0.484 for log unemployment 15-24 age group and log GDP per capita.

$\hat\sigma_{YX}=$-0.503 for log unemployment 15-64 age group and log GDP per capita.

$\hat\sigma_{YX}=$-0.472 for log unemployment 25-54 age group and log GDP per capita.

$\hat\sigma_{YX}=$-0.156 for log unemployment 55-64 age group and log GDP per capita.

From the above plots we see that the distribution of the observations is very large, and there does not seem to be any clear connection between the two variables. However the shown correlation coefficients show that there is some correlation between the two variables, and as expected the correlation is negative. When looking at the correlation coefficients exspecially the oldest age group stands out, only having a correlation of -0.156. When looking at the plots of the development in the unemployment rates this is not so supricing, since they already gave an indicator of the oldest age group reacting less to the 2008 financial crisis.

## Conclusion

In this project we wanted to investigate the unemployment rate in six different OECD countries. Using two datasets: one with the unemployment rate and one with the GDP, we have shown how the countries have developed over the period of 2007 Q1 to 2019 Q4.

Using descriptive statistics and plotting we have seen very different patterns in unemployment. However, when adding the information of the development in the GDP we have seen, that these differences cannot only be explained by differences in GDP, as the causal relationship does not seem to be as significant as one might have expected.


# Pycheat - Python Cheatsheet

When not specified: Python3
Link in markdown: \[blue_text](url_here)
[Help for markdown](https://commonmark.org/help/tutorial/index.html)
[Built-in magic Jupyter commands](https://ipython.readthedocs.io/en/stable/interactive/magics.html)

```latex
%%latex
\begin{equation}
H← 60 + \frac{30(B−R)}{Vmax−Vmin}  , if Vmax = G
\end{equation}
```

\begin{equation}
H← 60 + \frac{30(B−R)}{Vmax−Vmin}  , if Vmax = G
\end{equation}

```python
To run a bash command simple use the exclamation mark at the beginning of a line
!echo test
```

## Naming convention
[difference between module/class/package](https://softwareengineering.stackexchange.com/a/111882/195918)

[PEP 0008](https://www.python.org/dev/peps/pep-0008/#package-and-module-names) tells that:

- **modules (filenames)**: should have short, all-lowercase names, and they can contain underscores;
- **packages (directories)**: should have short, all-lowercase names, preferably without underscores;
- **classes**: should use the CapWords convention.

## Misc

```python
# Computing time for a line
%timeit [i for i in range(100000)]
```

    12.9 ms ± 1.49 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

```python
%%timeit # must be at the top of the cell => -r 1 to run only one time

# Computing time for a cell
for i in range(1000):
    i
```

    36.3 µs ± 0 ns per loop (mean ± std. dev. of 1 run, 10000 loops each)

## Lists

```python
xs = [1,2,3,4,5]
ys = [6,7,8,9,10]
```

```python
xs + ys # List concatenation
```

    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

## Dictionnaries

```python
from collections import Counter
# Counter
c = Counter(['eggs', 'ham', 'eggs'])
c.update(['eggs', 'chesse'])
print(c)
print(c.most_common(2))
```

    Counter({'eggs': 3, 'ham': 1, 'chesse': 1})
    [('eggs', 3), ('ham', 1)]

## String formatting

[pyformat.info](https://pyformat.info/)

```python
'{1} {0}'.format('one', 'two')
```

    'two one'

```python
'{{ {} {} }}'.format('one', 2)
```

    '{ one 2 }'

```python
multiline_string = "a first line"  \
                   "a second line"
```

```python
# To keep the zeroes at the end
print('{:.2f}'.format(round(2606.89579999999, 2)))
print('{:.2f}'.format(21))
```

    2606.90
    21.00

## Loops

```python
some_list = ["bananas", "apples", "mangos"]
```

```python
for index, value in enumerate(some_list):
    print(value + " is at index " + str(index))
```

    bananas is at index 0
    apples is at index 1
    mangos is at index 2

```python
some_dict = {'three': 3, 'one': 1, 'two': 2}
```

```python
for key, value in some_dict.items(): # Python 2.7 : iteritems()
    print(key + " is " + str(value))
```

    three is 3
    one is 1
    two is 2

```python
# Filter elements in comprehension list
[x for x in some_list if x != 'bananas']
```

    ['apples', 'mangos']

## Files

```python
import os
# Check file existance
os.path.exists('./file_or_link_or_dir_or_sym')
os.path.isdir('./folder/')
os.path.isfile('./file')
```

    False

```python
os.listdir('.')
#os.remove("dir_or_file_or_etc")
```

    ['python_cheatsheet.ipynb', '__main__.log', '.ipynb_checkpoints']

```python
import shutil
import os
def copytree(src, dst, symlinks=False, ignore=None):
    for item in os.listdir(src):
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if os.path.isdir(s):
            shutil.copytree(s, d, symlinks, ignore)
        else:
            shutil.copy2(s, d)

```

```python
# source: https://gist.github.com/seanh/93666
def format_filename(s):
    """Take a string and return a valid filename constructed from the string.
Uses a whitelist approach: any characters not present in valid_chars are
removed. Also spaces are replaced with underscores.

Note: this method may produce invalid filenames such as ``, `.` or `..`
When I use this method I prepend a date string like '2009_01_15_19_46_32_'
and append a file extension like '.txt', so I avoid the potential of using
an invalid filename.

"""
    valid_chars = "-_.() %s%s" % (string.ascii_letters, string.digits)
    filename = ''.join(c for c in s if c in valid_chars)
    filename = filename.replace(' ','_') # I don't like spaces in filenames.
    return filename
```

## Functions

```python
def a_function():
    print("myfunction")

if __name__ == '__main__': # If module (file) imported this part of the code will not be executed
    a_function()
```

    myfunction

```python
def multi_parameters_func(*params):
    print("params : {}".format(', '.join(params)))
multi_parameters_func("first_param", "second_param")
# other example: see benchmark function below ;)
```

    params : first_param, second_param

## Functionnal

[reduce_fold_left_python_haskell](https://eli.thegreenplace.net/2017/right-and-left-folds-primitive-recursion-patterns-in-python-and-haskell/)

```python
xs = [1,2,3,4,5]
ys = [6,7,8,9,10]
```

```python
 list(map(lambda a: a*10, xs))
```

    [10, 20, 30, 40, 50]

```python
# map with 2 arguments
list(map(lambda x,y: x+y, xs,ys))
```

    [7, 9, 11, 13, 15]

```python
list(filter(lambda x: x%2==0,xs))
```

    [2, 4]

```python
import functools
functools.reduce(lambda acc, x: acc+x, xs, 0) # ((((1+2)+3)+4)+5)  == Fold left in Haskell
```

    15

```python
# Partial functions (e.g. useful to use with map when needing to pass a param)
import functools
def multiply(x,y):
        return x * y

dbl = functools.partial(multiply,2) # create a new function that multiplies by 2
print(dbl(4))
```

    8

## Logs

```python
import logging

def create_logger(loglevel):
    numeric_level = getattr(logging, loglevel.upper(), logging.INFO)
    if not isinstance(numeric_level, int):
        raise ValueError('Invalid log level: %s' % loglevel)

    #import sys
    #import os
    #module_name = str(os.path.basename(sys.modules['__main__'].__file__)).split('.')[0]
    module_name = __name__

    logger = logging.getLogger(module_name)
    logger.setLevel(numeric_level)
    # create file handler which logs even debug messages
    fh = logging.FileHandler(module_name + '.log')
    fh.setLevel(logging.DEBUG)
    # create console handler with a higher log level
    ch = logging.StreamHandler()
    ch.setLevel(logging.INFO)
    # create formatter and add it to the handlers
    formatter = logging.Formatter('%(asctime)s\t%(name)s\t%(levelname)s\t\t%(message)s')
    fh.setFormatter(formatter)
    ch.setFormatter(formatter)
    # add the handlers to the logger
    logger.addHandler(fh)
    logger.addHandler(ch)
    logger.info("Logger created!")
    return logger

logger = create_logger("info")
logger.info("This is a log")
```

    2018-07-23 15:30:33,168	__main__	INFO		Logger created!
    2018-07-23 15:30:33,172	__main__	INFO		This is a log

## Various tools

```python
def benchmark(func, *params):
    import datetime
    import time
    start_time = time.time()
    return_value = func(*params) if params else func()
    total_time = datetime.timedelta(seconds=time.time() - start_time)
    print("Function " + func.__name__ + " - execution time : " + str(total_time))#.strftime('%H:%M:%S'))
    return return_value

def test():
    total = 0
    for i in range(0, 10000):
        total +=i
    return total

def sum(param1, param2):
    return param1 + param2

result = benchmark(sum, 1, 2)
print("Result : " + str(result))

result = benchmark(test)
print("Result : " + str(result))
```

    Function sum - execution time : 0:00:00.000002
    Result : 3
    Function test - execution time : 0:00:00.000820
    Result : 49995000

```python
import math
def entropy(string):
    "Calculates the Shannon entropy of a string"

    # get probability of chars in string
    prob = [float(string.count(c)) / len(string) for c in dict.fromkeys(list(string))]

    # calculate the entropy
    entropy = - sum([p * math.log(p) / math.log(2.0) for p in prob])

    return entropy

print(entropy("www.google.com"))
```

    2.8423709931771093

```python
def is_ipv4(ipv4_string):
    l = ipv4_string.split('.')
    if len(l) != 4:
        return False
    try:
        ip = list(map(int, l))
    except ValueError:
        return False
    if len(list(filter(lambda x: 0 <= x <= 255, ip))) == 4:
        return True
    return False

# True
print(is_ipv4("192.168.1.1"))
print(is_ipv4("0.0.0.0"))
print(is_ipv4("255.255.255.255"))

# False
print(is_ipv4("255.255.255"))
print(is_ipv4("255.255.255.255.3"))
print(is_ipv4("255.255.255.erzr"))

```

    True
    True
    True
    False
    False
    False

# Pandas dataframes

[Pandas tips and tricks](https://towardsdatascience.com/pandas-tips-and-tricks-33bcc8a40bb9)

```python
import pandas as pd

df = pd.DataFrame(data, columns=features_name)

import csv
df.to_csv(c.model_folder + filename, sep=',', encoding='utf-8', index=False, quoting=csv.QUOTE_NONNUMERIC)

#df = pd.read_csv(c.model_folder + "features.csv")

# Print a summary of 5 first rows
df.head(5)

# check the data frame info
df.info()

# Get unique values of a column
df['continent'].unique().tolist()

# To add a new column and set all rows to a specific value
df['Name'] = 'abc'

# Using DataFrame.drop
df.drop(df.columns[[1, 2]], axis=1, inplace=True)

# drop by Name
df1 = df1.drop(['B', 'C'], axis=1)

# Select the ones you want
df1 = df[['a','d']]

column_names = df.index
data = df.values

# Where condition
df.loc[df['label'] == 'NORMAL']

# Select column(s)
df[f_name]

# Count number of != values
df[f_name].value_counts()

# Put a column at the end
df_label = df.pop('label') # remove column 'label' and store it in df_label
df['label'] = df_label # add label as a 'new' column.

# To modify a specific cell
df.loc[df['key'] == 'mykey', "column_name"] = 1

# To iter over rows
for index, row in df.iterrows(): # row is a copy of the row from the dataframe
    print row['c1'], row['c2']

    # To modify the row: use the index to access the row in the dataframe
    df.loc[index, 'wgs1984_latitude'] = dict_temp['lat']

# To sum rows
df = df.sum(axis=1)

# To sum columns
normal_counts = df.["col_name"].sum()
```

## OS

```python
import subprocess
# Run command
subprocess.Popen(["bro", "-C", "-r", "../"+filename], cwd=working_dir).wait()
```

## Web requests (requests & BeautifulSoup)
- [requests doc](http://docs.python-requests.org/en/master/)
- [BeautifulSoup doc](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)

```python
import requests
from bs4 import BeautifulSoup
```

```python
url = "http://www.test.com"
cookies = {
    'Cookie':'some_value',
}
r = requests.get(url, cookies=cookies)

#print(r.text)

soup = BeautifulSoup(r.content, "html5lib")
```

```python
#Scan the URLs present on the page
for link in soup.find_all('a'):
    href = link.get('href')
    if str(href).startswith("http"): # To exclude refs that are links to paragraphs on the page (like #maincontent)
        print(href)
```

```python
def is_downloadable(url):
    """
    Does the url contain a downloadable resource
    """
    h = requests.head(url, allow_redirects=True, cookies=cookies)
    header = h.headers
    content_type = header.get('content-type')
    if 'text' in content_type.lower():
        return False
    if 'html' in content_type.lower():
        return False
    return True
```

```python
# Decode URL
from urllib.parse import unquote
url = unquote(url)
```