import pandas as pd
pd.options.display.max_rows = 20

%matplotlib inline

"""
Explanation: Introducing Pandas
From the docs:

A Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.

We also use matplotlib:

A Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.

Requirements:
(venv) $ pip install pandas matplotlib
We're going to see a sliver of the functionality provided by these packages.
End of explanation
"""


df = pd.DataFrame([
    {'integer': 1, 'float': 1.0, 'string': 'one'},
    {'integer': 2, 'float': 2.0, 'string': 'two'},
    {'integer': 2, 'float': 2.0, 'string': 'two'},
    {'integer': 3, 'float': 3.0, 'string': 'three'},
])

# Print some details about the DataFrame
df.info()
df

"""
Explanation: Introducing DataFrame
From the docs:

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table. It is generally the most commonly used pandas object.

There are many ways to get a DataFrame, but we'll start with a list of dictionaries.
End of explanation
"""


df_index = df.set_index('string')
df_index

"""
Explanation: The Jupyter Notebook automatically renders DataFrame as HTML!
Note the first column; this is an Index, and is an essential component of DataFrame. Here, it was auto-generated, but we can also set it:
End of explanation
"""


# Slice by label
df_index.loc['two']

# Slice by position
df_index.iloc[-2:]

"""
Explanation: The Index plays a key role in slicing the DataFrame:
End of explanation
"""


floats = df_index['float']
floats

"""
Explanation: We can also get individual columns:
End of explanation
"""


floats['two']

"""
Explanation: This is a Series, which is essentially a one-dimensional DataFrame, with a defined data type. Put another way, a DataFrame is a collection of Series.
Note that the Series retained the Index of our DataFrame, so we can use similar slicing:
End of explanation
"""


df_index['float'] * df_index['integer']

df_index * df_index

number_format = 'Number {}'.format
df_index['integer'].apply(number_format)

df_index.applymap(number_format)

"""
Explanation: Series and DataFrame support element-wise operations:
End of explanation
"""


gig_values = Gig.objects.past().published().values('date', 'venue__name', 'venue__city')
gig_values[:5]

"""
Explanation: Using DataFrame with Django
Django gives us a handy way to build a list of dictionaries:
End of explanation
"""


gigs = pd.DataFrame(gig_values.iterator())
gigs.info()
gigs

"""
Explanation: DataFrame doesn't know what to do with a QuerySet; it wants something that looks more like a list.
We could use list(gig_values), but gig_values.iterator() is more efficient.
End of explanation
"""


gig_values = Gig.objects.past().published().values_list('date', 'venue__name', 'venue__city')
gig_values[:5]

gigs = pd.DataFrame(gig_values.iterator(), columns=['date', 'venue', 'city'])

gigs['date'] = pd.to_datetime(gigs['date'])
gigs = gigs.set_index('date').sort_index()

gigs.info()
gigs.head()

"""
Explanation: This is a good place to start, and we've already got the answer to "How many gigs have we played"?
However, there are a few ways we can make this easier to work with:

Shorter column names
Predictable column order
Indexed and sorted by date

For more control, we'll use a list of tuples to initialize the DataFrame.
End of explanation
"""


gigs.loc['2016']

"""
Explanation: The previous cell demonstrates a good practice: make all of your modifications to a variable in one cell. This will help prevent surprises when you execute cells out of order as you play with code. If you need to make modifications later in the notebook, assign the result to a new variable.
Answering questions
What gigs  did we play last year?
Using the date as the Index allows for fast slicing:
End of explanation
"""


# resample('A') creates year-end groups like '2005-12-31'
# to_period() turns that into '2005'
years = gigs.resample('A').size().to_period()
years

years.plot.bar()

"""
Explanation: How many gigs have we played each year?
The date Index also allows for fast aggregration:
End of explanation
"""


gig_dates = gigs.reset_index()['date']
gig_dates

"""
Explanation: What are our most active months?
Get the dates as a Series:
End of explanation
"""


# Series.dt gives us access to date-related methods
gig_months = gig_dates.dt.strftime('%m %b')
gig_months

"""
Explanation: Convert those to sortable month names:
End of explanation
"""


months = gig_months.value_counts()
months

# matplotlib has lots of options for customization
ax = months.sort_index().plot.bar(table=True, figsize=(10,5))
ax.get_xaxis().set_visible(False)

"""
Explanation: Count the unique values:
End of explanation
"""


gig_cities = gigs['city']
gig_cities

cities = gig_cities.value_counts()
cities

cities.describe()

# Adding the ; suppresses `<matplotlib.axes._subplots.AxesSubplot ...>`
cities[:10].sort_values().plot.barh();

"""
Explanation: What cities have we played?
End of explanation
"""


# Series.str gives us access to string methods
in_pgh = gig_cities.str.contains('Pittsburgh')
in_pgh

gigs[in_pgh]

"""
Explanation: When did we play in Pittsburgh?
We can use a Series of boolean values to slice our DataFrame:
End of explanation
"""


'Boston, MA'.split(',')[1].strip()

"""
Explanation: What states have we played?
The Gig model doesn't have a state field, so we need to parse it out. In vanilla Python, we'd do:
End of explanation
"""


states = gig_cities.str.split(',').str.get(1).str.strip().value_counts()
states

states.describe()

# Don't rotate the x-axis labels
states[:5].plot.bar(rot=0);

"""
Explanation: With pandas, we can do the same thing for every Series element:
End of explanation
"""


venues = gigs.groupby(['venue', 'city']).size()
venues

"""
Explanation: What venues have we played?
DataFrame has powerful grouping and aggregration functionality:
End of explanation
"""


venues.describe()

top_venues = venues.nlargest(10)
top_venues

top_venues.sort_values().plot.barh();

"""
Explanation: This Series has a MultiIndex. Very useful, but beyond the scope of this presentation...
End of explanation
"""


