# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# ## Exploratory Analysis using Jupyter Notebook
# For further reading, we recommend: 
# - [the pandas documentation](http://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html#getting)  for information about using DataFrames
# - [this blog post](https://towardsdatascience.com/introduction-to-data-visualization-in-python-89a54c97fbed) for a jumpstart into visualizations
# - [the matplotlib documentation](https://matplotlib.org/users/pyplot_tutorial.html) for more info about visualizations

import pandas as pd
import numpy as np

# #### Loading data from our GPC bucket

import s3fs
s3 = s3fs.S3FileSystem(anon=True)
base = 'twde-datalab/raw'
for full_file in s3.ls(base):
    file = full_file[len(base)+1::]
    print(full_file)
    if (len(file)):
        s3.get(full_file, '../data/%s'%file)

# +
import glob
import os

csvs = glob.glob('../data/*.csv')
raw = {}

for file in csvs:
    size = int(os.path.getsize(file)/1024/1024)
    name = file.split('/')[-1].split('.')[0]
    print(name, size, 'MB')
    if size<512:
        print('loading %s'%file)
        raw[name] = pd.read_csv(file)
# -

for key,value in raw.items():
    print('\033[1m\033[91m' + key, str(value.shape) + '\033[0m')
    print(value.head())
    print()

# #### With just this glimpse, you can start to fill out your list of assumptions, hypotheses, and questions. Some of mine are:
# - Question: What is the span of dates we are provided?
# - Question: How many distinct store_nbr values are there?
# - Question: How many distinct item_nbr values are there?
# - Hypothesis: unit_sales are always positive
# - Hypothesis: onpromotion is always either True or False
# - Hypothesis: city and state are always going to be Quito and Pichincha
# - Hypothesis: cluster is always 5
# - Question: What does cluster mean and is it important to know?
# - Question: How many records does the data contain?
# - Question: What other data files are available?

# ### Here's some examples of how to address those first questions

# Access an entire dataframe column like you would
# the value in a python dictionary:
# (The returned object has similar pandas built-in 
# functions, like 'head' and 'max')
data = raw['store47_sample2016-2017']
date_min = data['date'].min()
date_max = data['date'].max()

# Dataframe columns also have a 'unique' method,
# which can answer several of our questions from above
store_list = data['store_nbr'].unique()
store_target = 47

print(data['item_nbr'].unique())
print("There are too many item numbers to display, so let's just count them for now:")
print("\n{} different item_nbr values in our data"
          .format(len(data['item_nbr'].unique())))

# #### It might be helpful to know the 'shape' of our data. We could count by hand (for now) the columns, but how many rows do we have altogether?

print(data.shape)
print("There are {} rows and {} columns in our data".format(data.shape[0], data.shape[1]))

# #### Moving along to answer our intial questions... Let's have a look at unit_sales. Keep in mind that unit sales is the variable we want to predict with our science.
#
# Each row in our data is essentially telling us a `unit_sales` number for a given `item_nbr` at a given `store_nbr` on a given `date`. That is, "how many of an item was sold at a store on a day".

print("unit sales smaller than zero: {}".format(np.sum(data['unit_sales'] < 0)))

on_promo = np.sum(data['onpromotion'] == True)
off_promo = np.sum(data['onpromotion'] == False)
print("unit sales on promotion: {}".format(on_promo))
print("unit sales off promotion: {}".format(off_promo))
print("unit neither on/off promotion: {}".format(data.shape[0] - on_promo - off_promo))

print('unique states: {}'.format(data['state'].unique()))



StarcoderdataJupyterScriptsDedupFiltered