"""
# **Title: EDA for sales dataset for an Online store**
"""
"""
> Welcome to this Exploratory Data Analysis (EDA) project, in which we will address a dataset that contains sales of an online store in the United States of America over the period from the beginning of January 2019 until the beginning of the same month of the following year 2020 across 8 US states [ CA ,NY ,TX ,MA ,GA ,WA ,OR ,ME ].

*  During this trip, we will explore the information inside the file to determine the problems in it and certainly address them, to be 100% valid and clean for the beginning of collecting the necessary information, and then put it in a comprehensive conclusion for all the information that must be mentioned. I hope that the presentation will be organized and comfortable, and I will be very welcome for any opinions or modifications. Thank you very much.
"""
"""
# **Stage 0: Exploring the given data**
"""
"""
* **Import the required modules.**
"""
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
"""
* **Import the dataset and start exploring.**
"""
df = pd.read_csv('../input/sales-data-set/all_data.csv')
df
"""
* **Number of rows and columns in this dataset.**
"""
df.shape
"""
* **Information about every column's data type.**
"""
df.info()
"""
* **Columns names.**
"""
df.columns
"""
* **Number of NaN values for each column.**
"""
df.isnull().sum()
df.loc[df.duplicated()]
"""
* **List of unique values in `Product` column.**
"""
df.Product.unique()
df.Product.value_counts()
"""
* **Generate descriptive statistics.**
"""
df.describe()
"""
* **Number of unique values in each column.**
"""
df.nunique()
"""
# **Stage 1: Assessing the data.**
"""
"""
> **Content issues detected:**
"""
"""
* **NaN Values** :
  * There are 545 NaN values. These values sgould be removed to ensure that the dataset is completely clean.
* **Data Entry errors** :
  * in **`Product`** column, There are 355 values with the name 'Product' . This should be a *data entry error* from combining table in csv file manually.
  * Another issue in the same column; there are some items duplicated in the same order for no reason. Maybe it's another software crash or data entry error, whatever.
* **Duplicated values** :
  * These duplicated values are Combination of NaN values and data entry errors.
"""
"""
> **Data type issues detected:**
"""
"""
* **`Quantity Ordered`** column :
  * should be changed to integer to count the total number of each product.
* **`Price Each`** column :
  * should be changed to float to count the total sales for each product.
* **`Order Date`** column:
  * should be changed to datetime to do some calucaltions about the preffered time of day or month or year, time classification calculations at all.
* New columns can be created:
  * **`ZIP Code`** column:
   * Contains a five-digit number extracted from `Purchase Address` column, represents the mail box for every area, and can be helpful when sending the product using postal address.
  * **`State`** Column:
   * to help in doing stats about sales in every state.
  * **`Day of week`** Column: 
   * to help in doing stats about the preffered day of week for order decision.
  * **`Hour of day`** Column:
   * to give information about the most frequent time of day to make an order.
"""
"""
> **I think it's all clear now about issues and how to fix it. One stage left before starting doing data science job; Cleaning the data.**
"""
"""
# **Stage 2: Cleaning the data.**
"""
"""
* **Taking a copy of the dataset to perform the cleaning process without changing the real data**
"""
df_clean = df.copy()
df
"""
> ## **Fixing content issues:**
"""
"""
> **Step 1: Removing data-entry-error values:**
"""
df_clean.loc[df_clean.duplicated()].sort_values('Order Date',ascending = False)
"""
* **Displaying the Duplicated values together:**
"""
"""
*Roughly speaking, these values ​​are duplicated because the data entry specialist has stitched the sales tables together as new values ​​with new indexes, ignoring the row and column order.*
"""
df_clean.loc[df_clean['Product']=='Product']
"""
* **Drop the previous values out of the dataset**
"""
df_clean.drop(df_clean.index[df_clean['Product'] == 'Product'], inplace = True)
"""
* **Check if it's done or not.**
"""
df_clean.loc[df_clean['Product']=='Product']
"""
*It seems like it's done, so we can jump into the next step!*
"""
"""
> **Step 2: Removing NaN values:**
"""
"""
* **Displaying NaN values together**
"""
df_clean.loc[df_clean['Order ID'].isnull()]
"""
* **Deleting NaN values**
"""
df_clean.dropna(inplace=True)
df_clean.loc[df_clean['Order ID'].isnull()]
"""
>  **Cool ; to the last step, before we start our EDA!!!**
"""
"""
> **Step 3: Removing duplicated values.**
"""
"""
* **Search for duplicated values:**
"""
df_clean.groupby(['Order ID','Product'])['Order ID'].count().nlargest()
"""
As an example, let's search with random `Order ID` value:
"""
df_clean.loc[df_clean['Order ID']=='142071']
"""
That's another big issue that makes the the data somekind missleading, so it's a must to drop these duplicated values.
"""
df_clean.drop_duplicates(subset=['Order ID','Product'],keep='first',inplace=True)
df_clean.groupby(['Order ID','Product'])['Order ID'].count()
df_clean.loc[df_clean['Order ID']=='142071']
"""
> **Now, finally we can say: WE HAVE CLEAN DATA 100%, but one another issue still not fixed: changing data types in some columns,and creating new columns to make your EDA more easier.** 
"""
"""
> ## **fixing data type issues:**
"""
df_clean.info()
"""
> **Step 1: changing some columns to the appropriate data type.**
"""
df_clean['Order Date']=pd.to_datetime(df_clean['Order Date'],errors='coerce')
df_clean['Price Each']= pd.to_numeric(df_clean['Price Each'])
df_clean['Quantity Ordered']=df_clean['Quantity Ordered'].astype(int)
"""
> **Step 2: Creating new columns.**
"""
"""
* **Creating `ZIP Code` column.**
"""
df_clean['ZIP Code']=df_clean['Purchase Address'].str[-6:]
"""
* **Editing `Purchase Address` column.**
"""
df_clean['Purchase Address']=df_clean['Purchase Address'].str[:-6]
"""
* **Creating `State` column.**
"""
df_clean['State']=df_clean['Purchase Address'].str[-2:]
df_clean['State']=df_clean['State'].astype('category')
"""
* **Creating `Order Year` column.**
"""
df_clean['Order Year']=df_clean['Order Date'].dt.year
df_clean['Order Year']=df_clean['Order Year'].astype('category')
"""
* **Creating `Order Month` column.**
"""
df_clean['Order Month']=df_clean['Order Date'].dt.month
df_clean['Order Month']=df_clean['Order Month'].astype('category')
"""
* **Creating `Order Day of Week` column.**
"""
df_clean['Order Day of Week']=df_clean['Order Date'].dt.dayofweek
"""
* **Creating `Order Hour of Day` column.**
"""
df_clean['Order Hour of Day']=df_clean['Order Date'].dt.hour
df_clean['Order Hour of Day']=df_clean['Order Hour of Day'].astype(int)
"""
> **Display the dataset in its clean state.**
"""
df_clean
"""
> **The dataset ordered by `Order Date`.**
"""
df_clean = df_clean.sort_values('Order Date')
df_clean.reset_index(drop=True,inplace=True)
df_clean
"""
> For more accurate analysis, we need to drop the values with date: `2020-01-01` to have the data for a complete year.
"""
df_clean.loc[df_clean['Order Year']==2020].count()[0]
df_clean.drop(df_clean.index[df_clean['Order Year'] ==2020], inplace = True)
df_clean
"""
> **Saving clean data**
"""
df_clean.to_csv('online_store_clean.csv')
df_clean.info()
df_clean.duplicated().sum()
df_clean.isnull().sum()
df_clean.describe()
"""
# **Stage 3: Exploratory Data Analysis (EDA) and Data Visualization**
"""
"""
## **Setting some parameters for a better data visualization**
"""
plt.style.use(['seaborn'])
SMALL_SIZE = 14
MEDIUM_SIZE = 16
BIGGER_SIZE = 18
plt.rc('font', size=SMALL_SIZE)          # controls default text sizes
plt.rc('axes', titlesize=BIGGER_SIZE)     # fontsize of the axes title
plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('legend', fontsize=SMALL_SIZE)    # legend fontsize
plt.rc('figure', titlesize=BIGGER_SIZE,figsize=(15,10))  # fontsize of the figure title and figure size

"""
## **Research Question 0: How many orders are made in 2019?**
"""
df_clean['Order ID'].nunique()
"""
## **Research Question 1: How much was the total sales value during 2019?**
"""
df_clean['Price Each'].sum().astype(int)
"""
## **Research Question 2: How much was the total sales for each month?**
"""
df_clean.groupby(['Order Month'])['Price Each'].sum().astype(int)
df_clean.groupby(['Order Month'])['Price Each'].sum().astype(int).plot(kind='line',color = 'black',linestyle = '--',figsize = (15,7.5))
df_clean.groupby(['Order Month'])['Price Each'].sum().astype(int).plot(kind='bar')
plt.xticks(np.arange(0,12,1),labels=['January','Febraury','March','April','May','June','July','August','September','October','November','December'],fontstyle = 'oblique')
plt.xlabel('Months of the year')
plt.ylabel('Number of Orders')
plt.title('The Total Sales For Each Month In 2019')
plt.show()
df_clean.groupby(['Order Month'])['Price Each'].sum().astype(int).plot(kind='pie',shadow=True,autopct='%.2f%%',figsize=(18,20))
plt.title('The Total Sales For Each Month In 2019')
plt.legend(title='Months', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.show()
"""
## **Research Question 3: How many pieces did the store sell during 2019?**
"""
df_clean['Quantity Ordered'].sum()
"""
## **Research Question 4: What is the most requested item from the store?**
"""
df_clean.groupby(['Product'])['Product'].count().nlargest(20)
df_clean.groupby('Product')['Product'].count().nlargest(20).plot(kind='bar',figsize=(15,7.5))
plt.title('The most requested item in 2019')
plt.ylabel('Number of Orders')
plt.show()
df_clean.groupby('Product')['Product'].count().nlargest(20).plot(kind='pie',figsize=(18,20),shadow=True,autopct='%.2f%%')
plt.title('The most requested item in 2019')
plt.legend(title='Products', bbox_to_anchor=(1.15, 1), loc='upper left')
plt.show()
"""
## **Research Question 5: How much did each item represent in total sales? And which piece was the highest share of sales?**
"""
df_clean.groupby('Product')['Price Each'].sum().nlargest(20)
df_clean.groupby('Product')['Price Each'].sum().nlargest(20).plot(kind='bar',figsize = (15,7.5))
plt.title('Total Sales For Each Product')
plt.ylabel('Sales per Million')
plt.show()
df_clean.groupby('Product')['Price Each'].sum().nlargest(20).plot(kind='pie',figsize = (18,20),shadow=True,autopct='%.2f%%')
plt.title('Total Sales For Each Product')
plt.ylabel('Sales per Million')
plt.legend(title='Products', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.show()
"""
## **Research Question 6: How much was each state's share of the purchases?**
"""
df_clean.groupby(['State'])['Price Each'].sum().astype(int).nlargest(10)
plt.subplot(2, 2, 1)
df_clean.groupby(['State'])['Price Each'].sum().astype(int).nlargest(10).plot(kind='bar',figsize=(15,15))
plt.subplot(2, 2, 2)
df_clean.groupby(['State'])['Price Each'].sum().astype(int).plot(kind='pie',figsize=(15,15),shadow=True,autopct='%.2f%%',explode = [0.1,0,0,0,0,0,0,0])
plt.ylabel('Percentage For Each State')
plt.title('Which state represents the highest precentage of purchases?',loc='center')
plt.legend(title='States', bbox_to_anchor=(1.05, 1), loc='upper left')
plt.show()
"""
## **Research Question 7: What are the details of the biggest order?**
"""
df_clean['Order ID'].value_counts()
df_clean.loc[df_clean['Order ID']=='160873']
"""
## **Research Question 8: Where is the address where the biggest number of purchases have been delivered?**
"""
df_clean.groupby(['Purchase Address'])['Purchase Address'].count().nlargest()
df_clean.loc[df_clean['Purchase Address']=='193 Forest St, San Francisco, CA'].sort_values('Purchase Address')
"""
## **Research Question 9: Which mailbox was the largest number of purchases sent to?**
"""
df_clean.groupby(['ZIP Code'])['ZIP Code'].count().nlargest(10)
df_clean.groupby(['ZIP Code'])['ZIP Code'].count().plot(kind='pie',shadow=False,autopct='%.2f%%',explode = [0,0,0.15,0,0,0,0.15,0.15,0,0],figsize=(15,15))
plt.title('Which Mailbox receives the most orders?')
plt.legend(title='ZIP Codes', bbox_to_anchor=(1.1, 1), loc='upper left')
plt.show()
"""
## **Research Question 10: What is the most popular day of the week on which products are purchased from the store?**
"""
df_clean['Order Day of Week'].value_counts().nlargest(7)
df_clean['Order Day of Week'].value_counts(normalize=True).nlargest(7)*100
df_clean['Order Day of Week'].value_counts(sort=False).plot(kind='line')
plt.xticks(np.arange(0,7,1),labels=['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'],fontstyle = 'oblique')
plt.title('The popular day of week to make an order')
plt.xlabel('Day of Week')
plt.ylabel('Number of orders')
plt.show()
"""
## **Research Question 11: What is the most popular time of the day on which products are purchased from the store?**
"""
df_clean['Order Hour of Day'].value_counts(sort=False).plot(kind='bar')
df_clean['Order Hour of Day'].value_counts(sort=False).plot(kind='line',color = 'black',linestyle = '--')
plt.xlabel('Hour')
plt.ylabel('Number of Orders')
plt.title("The most popular hour to make an order")
plt.show()
"""
## **Research Question 12: Which smart phone is the most popular to be ordered?**
"""
df_clean.loc[df_clean.Product=='iPhone'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
df_clean.loc[df_clean.Product=='Google Phone'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
df_clean.loc[df_clean.Product=='Vareebadd Phone'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
plt.xticks(np.arange(0,12,1),labels=['January','Febraury','March','April','May','June','July','August','September','October','November','December'],fontstyle = 'oblique')
plt.legend(title = 'Smart Phones',labels=['iPhone','Google Phone','Vareebadd Phone'])
plt.ylabel('Number of Orders')
plt.show()

"""
## **Research Question 13: Which headphone is the most popular to be ordered?**
"""
df_clean.loc[df_clean.Product=='Wired Headphones'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
df_clean.loc[df_clean.Product=='Bose SoundSport Headphones'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
df_clean.loc[df_clean.Product=='Apple Airpods Headphones'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
plt.xticks(np.arange(0,12,1),labels=['January','Febraury','March','April','May','June','July','August','September','October','November','December'],fontstyle = 'oblique')
plt.legend(title = 'Headphones',labels=['Wired Headphones','Bose SoundSport Headphones','Apple Airpods Headphones'])
plt.ylabel('Number of Orders')
plt.show()
"""
## **Research Question 14: Which laptop is the most popular to be ordered?**
"""
df_clean.loc[df_clean.Product =='ThinkPad Laptop'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
df_clean.loc[df_clean.Product=='Macbook Pro Laptop'].groupby(['Order Month'])['Order ID'].count().plot(kind='line')
plt.xticks(np.arange(0,12,1),labels=['January','Febraury','March','April','May','June','July','August','September','October','November','December'],fontstyle = 'oblique')
plt.legend(title = 'Laptops',labels=['ThinkPad Laptop','Macbook Pro Laptop'])
plt.ylabel('Number of Orders')
plt.show()

"""
# **Stage 4: Results and Conclusion**
"""
"""
In conclusion, and after this long process, I came out with these observations:
> * During the year 2019 : the store received **178,406** orders, during which **208,689** items were purchased, with a total sales estimated at approximately **$34 million**.
> * The most sales for a single month were in **December**, at nearly **$4.5 million (about 13.4% of the total sales)**.
> * The most requested item from the store was for **"USB-C Charging Cable"**, with about **21,900** pieces, and it was followed by a small difference, **"Lightning Charging Cable"**, with about **21,600** pieces.
 > * The highest share of sales profits was the **Macbook Pro Laptop** with about **$8 million (23.43% of the total sales)**, and the **iPhone** came in second place with about **$4.8 million (13.97% of the total sales)**.
 > * **California** topped the states in terms of sales, with sales accounting for **39.76% of total sales**, amounting to approximately **$13.6 million**.
 > * Products that were mailed to **California alone** via PO Boxes: *"90001"* and *"94016"* **(74,198 products)**, make up **37.43% of the products**.
 > * The most day of the week in which customers purchase products is **Tuesday (27,128 requests, or 14.61% of the total requests)**, but the percentages are very close among the rest of the week, as the lowest day in orders was on **Friday (26,211 requests, or 14.12% of the total requests)**.
 > * Throughout the day, the number of requests took a certain pattern, as it **decreased from midnight until four o’clock in the morning**, from which it gradually increased until it **gradually stabilized with the noon hours** and **decreased again until three o’clock in the afternoon**, from which **it rises again until it reaches its highest level at seven o’clock at night** and then **gradually decreases Until midnight**.
"""