# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Week 1
#
# ## Overview
# As explained in the [Before week 1: How to take this class](https://nbviewer.org/github/suneman/socialdata2022/blob/main/lectures/How_To_Take_This_Class.ipynb) notebook, each week of this class is an Jupyter notebook like this one. In order to follow the class, you simply start reading from the top, following the instructions.
#
# **Hint**: And you can ask us for help at any point if you get stuck!

# ## Today
#
# This first lecture will go over a few different topics to get you started 
#
# * First, I'll explain a little bit about what we'll be doing this year (hint, you may want to watch _Minority Report_ if you want to prepare deeply for the class 😊).
# * Second, we'll start by loading some real-world data into your very own computers and getting started with some data analysis.

# ## Part 1: Predictive policing. A case to learn from
#
# For a number of years I've been a little bit obsessed with [predictive policing](https://www.sciencemag.org/news/2016/09/can-predictive-policing-prevent-crime-it-happens). I guess there are various reasons. For example:
#
# * I think it's an interesting application of data science. 
# * It connects to popular culture in a big way. Both through TV shows, such as [NUMB3RS](https://en.wikipedia.org/wiki/Numbers_(TV_series)) (it also features in Bones ... or any of the CSI), and also any number of movies, my favorite of which has to be [Minority report](https://www.imdb.com/title/tt0181689/).
# * Predictive policing is also big business. Companies like [PredPol](https://www.predpol.com), [Palantir](https://www.theverge.com/2018/2/27/17054740/palantir-predictive-policing-tool-new-orleans-nopd), and many other companies offer their services law enforcement by analyzing crime data. 
# * It hints at the dark sides of Data Science. In these algorithms, concepts like [bias, fairness, and accountability](https://www.smithsonianmag.com/innovation/artificial-intelligence-is-now-used-predict-crime-is-it-biased-180968337/) become incredibly important when the potential outcome of an algorithm is real people going to prison.  
# * And, finally there's lots of data available!! Chicago, NYC, and San Francisco all have crime data available freely online.
#
# Below is a little video to pique your interest. 

from IPython.display import YouTubeVideo
YouTubeVideo("YxvyeaL7NEM",width=800, height=450)

# All this is to say that in the coming weeks, we'll be working to understand crime in San Francisco. We'll be using the SF crime data as a basis for our work on data analysis and data visualization. 
#
# We will draw on data from the project [SF OpenData](https://data.sfgov.org), looking into SFPD incidents which have been recorded back since January 2003.
#
# *Reading*
#
# Read [this article](https://www.sciencemag.org/news/2016/09/can-predictive-policing-prevent-crime-it-happens) from science magazine to get a bit deeper sense of the topic.
#
#
# > *Exercise*
# > 
# > Answer the following questions in your own words
# > 
# > * According to the article, is predictive policing better than best practice techniques for law enforcement? The article is from 2016. Take a look around the web, does this still seem to be the case in 2020? (hint, when you evaluate the evidence consider the source)
# > * List and explain some of the possible issues with predictive policing according to the article. 

# + [markdown] pycharm={"name": "#%% md\n"}
# _Answer_
#
# * The article argues that using predictive policing reduces bias and risk of civil liberties violations in complicated relationships (black vs white). In that manner it is easier. However, <NAME> mentions that the evidence of predictive policing work is scarce as there are few data points. Other articles argue that predictive policing would increase the bias that the current police force has (https://www.brennancenter.org/our-work/research-reports/predictive-policing-explained), it also highlights many projects that have failed and follow up with lawsuits due to their unfairness. Articles such as https://www.ojp.gov/pdffiles1/nij/230414.pdf highlights the positive use of policing where they have allocated resources according to the model's findings.
# * As mentioned above: unfairness and bias.
# -

# # Part 2: Load some crime-data into `pandas`
#
# Go and check out the [Python Bootcamp](https://nbviewer.org/github/suneman/socialdata2022/blob/main/lectures/PythonBootcamp.ipynb) lecture if you don't know what "loading data into Pandas" means. If you're used to using Pandas, then it's finally time to get your hands on some data!!
#
# > *Exercise 1*
# >
# > * Go to https://datasf.org/opendata/
# > * Click on "Public Safety"
# > * Download all police incidence reports, historical 2003 to may 2018. You can get everything as a big CSV file if you press the *Export* button (it's a snappy little ~500MB file).
# > * Load the data into `pandas` using thie tips and tricks described [here](https://www.shanelynn.ie/python-pandas-read_csv-load-data-from-csv-files/).
# > * Use pandas to generate the following simple statistics
# >   - Report the total number of crimes in the dataset.
# >   - List the various categories of crime. How many are there? 
# >   - List the number of crimes in each category.
#

# + pycharm={"name": "#%%\n"}
import pandas as pd

# + pycharm={"name": "#%%\n"}
df = pd.read_csv('Police_Department_Incident_Reports__Historical_2003_to_May_2018.csv')

# + pycharm={"name": "#%%\n"}
df.columns

# + pycharm={"name": "#%%\n"}
print(f"Total number of crimes: {len(df)}")

# + pycharm={"name": "#%%\n"}
print(f'Various categories of crimes: {df.Category.unique()} \nTotal: {len(df.Category.unique())}\n\nNumber of crimes in each category:')
df.groupby('Category').count()['IncidntNum']
# -

# In order to do awesome *predictive policing* later on in the class, we're going to dissect the SF crime-data quite thoroughly to figure out what has been going on over the last  years on the San Francisco crime scene.
#
# > *Exercise 2*: The types of crimes. The first field we'll dig into is the column "Category".
# > * We have already counted the number of crimes in each category. What is the most commonly occurring category of crime? What is the least frequently occurring?
# > * Create a bar-plot over crime occurrences (**N.B.** for a plot to be informative we have to remember to add any relevant info, e.g., title, labels, etc.). Mine looks like this
#
# <div>
# <img src="https://raw.githubusercontent.com/suneman/socialdata2022/main/files/CrimeOccurrencesByCategory.png" width="700"/>
# </div>
#

# + pycharm={"name": "#%%\n"}
import seaborn as sns
import matplotlib.pyplot as plt
sns.set(rc = {'figure.figsize':(15,8)})

# + pycharm={"name": "#%%\n"}
group = df.groupby('Category').count()['IncidntNum']

# + pycharm={"name": "#%%\n"}
print(f'Most frequent category of crime: {group.idxmax()}')

# + pycharm={"name": "#%%\n"}
print(f'Least frequent category of crime: {group.idxmin()}')

# + pycharm={"name": "#%%\n"}
sorted_ = group.sort_values(ascending=False)

# + pycharm={"name": "#%%\n"}
sns.barplot(x=sorted_.index, y=sorted_)
plt.xticks(rotation=85)
plt.title('Number of incidents for each category')
plt.ylabel('Frequency')
plt.tight_layout()
# -

# The plot above already gives us a bit of information but does not tell us anything about how crimes changed over the years. Let's have a look at it!
#
# > *Exercise 3*: Temporal patterns.
# > * To start off easily, let's count the number of crimes per year:
# >     * What is the year with most crimes?
# >     * What is the year with less crimes? (**hint** if your result is 2018, have a further look at the data you have. Is it complete? If not, how would you pre-process the data to answer these questions?).
# >     * What's the average number of crimes per year?
# > * Now, let's change our bar-plot above by replacing the count with the *average number of crimes per year by category* and by adding its *standard deviation*.
# > * Finally, Police chief Suneman is interested in the temporal development of only a subset of categories, the so-called focus crimes. Those categories are listed below (for convenient copy-paste action). Now create bar-charts displaying the year-by-year development of each of these categories across the years 2003-2017.
# >

# + pycharm={"name": "#%%\n"}
df.Date = pd.to_datetime(df.Date)

# + pycharm={"name": "#%%\n"}
crimes_pr_year = df.groupby(df.Date.dt.year).count()['IncidntNum']

# + pycharm={"name": "#%%\n"}
print(f"Year with most crimes: {crimes_pr_year.idxmax()} (count: {crimes_pr_year.max()})")
print(f"Year with least crimes: {crimes_pr_year.idxmin()} (count: {crimes_pr_year.min()})")

# + [markdown] pycharm={"name": "#%% md\n"}
# As we have 2018 with the least number of crimes, we investigate further. By reading the **documentation** of the dataset at https://data.sfgov.org/Public-Safety/Police-Department-Incident-Reports-Historical-2003/tmnf-yvry we see that the data is **incomplete** as it stops in 2018 May, thus we do not have a full year of data points (the data was moved to a new dataset due to unreliability issues). Therefore, to get the correct result, we should exclude 2018.

# + pycharm={"name": "#%%\n"}
print(f"Year with least crimes: {crimes_pr_year.drop(2018).idxmin()} (count: {crimes_pr_year.drop(2018).min()})")

# + pycharm={"name": "#%%\n"}
print(f"Average number of crimes per year: {crimes_pr_year.mean()}")
print(f"Average number of crimes per year (excluding 2018): {crimes_pr_year.drop(2018).mean()}")

# + pycharm={"name": "#%%\n"}
crimes_pr_year_pr_cat = df.groupby([df.Date.dt.year, df.Category], as_index=False).count()

# + pycharm={"name": "#%%\n"}
plot_order = crimes_pr_year_pr_cat.groupby('Category')['IncidntNum'].mean().sort_values(ascending=False).index.values

# + pycharm={"name": "#%%\n"}
sns.color_palette()[1]

# + pycharm={"name": "#%%\n"}
sns.color_palette()

# + pycharm={"name": "#%%\n"}
import matplotlib

g = sns.barplot(x=crimes_pr_year_pr_cat.Category, y=crimes_pr_year_pr_cat.IncidntNum, ci='sd', order=plot_order)
plt.xticks(rotation=85)
plt.title('Average number of incidents for each category pr year')
plt.ylabel('Mean')

plt.legend(handles=[matplotlib.patches.Patch(color=sns.color_palette()[0], label='Mean'), matplotlib.patches.Patch(color=(0.2, 0.2, 0.2), label='Error')])

plt.tight_layout()
# -

focuscrimes = {'WEAPON LAWS', 'PROSTITUTION', 'DRIVING UNDER THE INFLUENCE', 'ROBBERY', 'BURGLARY', 'ASSAULT',
               'DRUNKENNESS', 'DRUG/NARCOTIC', 'TRESPASS', 'LARCENY/THEFT', 'VANDALISM', 'VEHICLE THEFT',
               'STOLEN PROPERTY', 'DISORDERLY CONDUCT'}

# + pycharm={"name": "#%%\n"}
df_focus = df[df.Category.isin(focuscrimes) & (2003 <= df.Date.dt.year) & (df.Date.dt.year <= 2017)]

# + pycharm={"name": "#%%\n"}
df_plot = df_focus.groupby([df_focus.Category, df_focus.Date.dt.year]).count().IncidntNum.reset_index()

# + pycharm={"name": "#%%\n"}
g = sns.FacetGrid(df_plot, col='Category', height=4, aspect=1.5, col_wrap=2, sharey=False)
g.map(sns.barplot, "Date", "IncidntNum")
g.set_ylabels('Frequency')
g.set_titles('{col_name}')
g.set_xticklabels(rotation=85)
g.fig.subplots_adjust(top=.95)
g.fig.suptitle('Development of focus crimes from 2003 to 2017')
# -

# > * My plot looks like this for the 14 focus crimes:
#
#   ![Barplots](https://raw.githubusercontent.com/suneman/socialdata2022/main/files/CrimesYearCategory.png)
# >
# > (Note that titles are OVER the plots and the axes on the bottom are common for all plots.)
# > * Comment on at least three interesting trends in your plot.
# >
# > Also, here's a fun fact: The drop in car thefts is due to new technology called 'engine immobilizer systems' - get the full story [here](https://www.nytimes.com/2014/08/12/upshot/heres-why-stealing-cars-went-out-of-fashion.html).
#
# You've made it! You have successfully completed Lecture 1. If you didn't get enough, try to **personalize** your plots. We have already added titles, labels, etc., however, the look of default pandas plots is quite boring (that blue... and font... 😂).

StarcoderdataJupyterScriptsDedupFiltered