# import pandas library
import pandas as pd

"""
Explanation: <div class="alert alert-block alert-info" style="margin-top: 20px">
 <a href="http://cocl.us/DA0101EN_NotbookLink_Top"><img src = "https://ibm.box.com/shared/static/fvp89yz8uzmr5q6bs6wnguxbf8x91z35.png" width = 750, align = "center"></a>
  <h1 align=center><font size = 5> Link</font></h1> 

 <a href="https://www.bigdatauniversity.com"><img src = "https://ibm.box.com/shared/static/ugcqz6ohbvff804xp84y4kqnvvk3bq1g.png" width = 300, align = "center"></a>

<h1 align=center><font size = 5>Data Analysis with Python</font></h1>

# Module 1 Introduction
### Welcome!

In this section you will learn how to approach data acquisition in various ways, and obtain necessary insights from a dataset. By the end of this lab, you will successfully load the data into Jupyter Notebook, and gain some fundamental insights via Pandas Library.




## Table of contents

<div class="alert alert-block alert-info" style="margin-top: 20px">
<li><a href="#ref1">Data Acquisition</a>
<li><a href="#ref2">Basic Insight of Dataset</a></li>
<p></p>
Estimated Time Needed: <strong>10 min</strong>
</div>
<hr>

<a id="ref1"></a>
# Data Acquisition
There are various formats for a dataset, .csv, .json, .xlsx  etc. The dataset can be stored in different places, on your local, machine or sometimes online.
In this section, you will learn how to load a dataset into our Jupyter Notebook.
In our case, the Automobile Dataset is an online source, and it is in CSV (comma separated value) format. Let's use this dataset as an example to practice data reading.
Data source: https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data
data type: csv
The Pandas Library is a useful tool that enables us to read various datasets into a data frame. Our Jupyter notebook platforms have a built-in **Pandas Library** so that all we need to do is import Pandas without installing.
End of explanation
"""


# import pandas library
import pandas as pd
# read the online file by the URL provides above, and assign it to variable "df"
path="https://archive.ics.uci.edu/ml/machine-learning-databases/autos/imports-85.data"

df = pd.read_csv(path,header=None)
print("Done")

"""
Explanation: Read Data
We use "pandas.read_csv()"  function to read the csv file. In the bracket, we put the file path along with a quotation mark, so that pandas will read the file into a data frame from that address. The file path can be either a URL or your local file address.
Because the data does not include headers, we can add an argument " header = None"  inside the  "read_csv()" method, so that pandas will not automatically set the first row as a header.
You can also assign the dataset to any variable you create.
End of explanation
"""


# show the first 5 rows using dataframe.head() method
df.head(5)

"""
Explanation: After reading the dataset, we can use the dataframe.head(n) method to check the top 'n' rows of the dataframe, where n is an integer. Contrary to dataframe.head(n), dataframe.tail(n) will show you the bottom n rows of the dataframe.
End of explanation
"""


df.tail(10)

"""
Explanation: <div class="alert alert-danger alertdanger" style="margin-top: 20px">
<h1> Question #1: </h1>
<b>Check the bottom 10 rows of data frame "df":</b>
</div>
End of explanation
"""


# create headers list
headers = ["symboling","normalized-losses","make","fuel-type","aspiration", "num-of-doors","body-style",
         "drive-wheels","engine-location","wheel-base", "length","width","height","curb-weight","engine-type",
         "num-of-cylinders", "engine-size","fuel-system","bore","stroke","compression-ratio","horsepower",
         "peak-rpm","city-mpg","highway-mpg","price"]
headers

"""
Explanation: <div class="alert alert-danger alertdanger" style="margin-top: 20px">
<h1> Question #1 Answer: </h1>
<b>Run the code below! Did you get the right code?</b>
</div>

<div align="right">
<a href="#q2" class="btn btn-default" data-toggle="collapse">Click here for the solution</a>

</div>
<div id="q2" class="collapse">
```
df.tail(10)
```
</div>

Add Headers
Take a look at our dataset; pandas automatically set the header by an integer from 0. 
<div>
To better describe our data we can introduce a header. This information is available at:  https://archive.ics.uci.edu/ml/datasets/Automobile</div>
<p></p>
<div>Thus, we have to add headers manually.</div>
<div>Firstly, we create a list "headers" that includes all column names in order.</div>
<div>Then, we use **`dataframe.columns = headers`** to replace the headers by the list we created.</div>
End of explanation
"""


df.columns = headers
df.head(10)

"""
Explanation: We replace headers and recheck our data frame:
End of explanation
"""


df.dropna(subset=["price"], axis=0)

"""
Explanation: We can drop missing values along the column "price" as follows:
End of explanation
"""


list(df.columns.values)

"""
Explanation: Now, we have successfully read the raw dataset and add the correct headers into the data frame.
<div class="alert alert-danger alertdanger" style="margin-top: 20px">
<h1> Question #2: </h1>
<b>Find the name of the columns of the dataframe:</b>
</div>
End of explanation
"""


# check the data type of data frame "df" by .dtypes
df.dtypes

"""
Explanation: <div align="right">
<a href="#q3" class="btn btn-default" data-toggle="collapse">Click here for the solution</a>

</div>
<div id="q3" class="collapse">
```
df.columns
```
</div>

Save Dataset
Correspondingly, Pandas enables us to save the dataset to csv  by using the dataframe.to_csv() method. You can add the file path and name along with quotation marks in the brackets.
For example, if you would save the dataframe "df" as "automobile.csv" to your local machine, you may use the syntax below:
~~~~
df.to_csv("automobile.csv")
~~~~
We can also read and save other file formats, and we can use similar functions to pd.read_csv() and df.to_csv() for other data formats. The functions are listed in the following table:
Read/Save Other Data Formats
| Data Formate  | Read           | Save             |
| ------------- |:--------------:| ----------------:|
| csv           | pd.read_csv()  |df.to_csv()     |
| json          | pd.read_json() |df.to_json()    |
| excel         | pd.read_excel()|df.to_excel()   |
| hdf           | pd.read_hdf()  |df.to_hdf()     |
| sql           | pd.read_sql()  |df.to_sql()     |
| ...           |   ...          |       ...        |
<a id="ref2"></a>
Basic Insight of Dataset
After reading data into Pandas dataframe, it is time for us to explore the dataset.
There are several ways to obtain essential insights of the data to help us better understand our dataset.
Data Types
Data has a variety of types.
The main types stored in Pandas dataframes are object, float, int, bool, and datetime64. In order to better learn about each attribute, it is always good for us to know the data type of each column. In Pandas:
~~~~
dataframe.dtypes
~~~~
Returns a Series with the data type of each column.
End of explanation
"""


df.describe()

"""
Explanation: As a result, as shown above, it is clear to see that the data type of "symboling" and "curb-weight" are int64, "normalized-losses" is object, and "wheel-base" is float64, etc.
These data types can be changed, and we will learn how to accomplish this in a later module. 
Describe
If we would like to get a statistical summary of each column, such as  count, column mean value, column standard deviation, etc, then we use the describe method:
~~~~
dataframe.describe()
~~~~
This method will provide various summary statistics, excluding NaN (Not a Number) values:
End of explanation
"""


# describe all the columns in "df" 
df.describe(include = "all")

"""
Explanation: This shows the statistical summary of all numeric-typed (int, float) columns.
For example, the attribute "symboling" has 205 counts, the mean value of this column is 0.83, the standard deviation is 1.25, the minimum value is -2, the 25th percentile is 0, 50th percentile is 1, 75th percentile is 2, and the maximum value is 3.
However, what if we would also like to check all the columns including those that are of type object?
You can add an argument include = "all" inside the bracket. Let's try it again:
End of explanation
"""


df[['length', 'compression-ratio']].describe()

"""
Explanation: Now it provides the statistical summary of all the columns, including object-typed attributes.
We can now see how many unique values there are, (top row), and the frequency of the top value in the third row.
Some values in the table above show as "NaN", this is because those numbers are not available regarding a particular column type.
<div class="alert alert-danger alertdanger" style="margin-top: 20px">
<h1> Question #3: </h1>

<p>You can select the columns of a data frame by indicating the name of  each column. For example, you can select the three columns as follows:</p>
<p></p>
<p>dataframe[[' column 1 ',column 2', 'column 3'] ]</p>
<p></p>
<p>Where "column" is the name of the column, you can apply the method  ".describe()" to get the statistics of those columns as follows:</p>
<p></p>
<p>dataframe[[' column 1 ',column 2', 'column 3'] ].describe() </p>
<p></p>

Apply the  method to ".describe()" to the columns 'length' and 'compression-ratio'.
</div>
End of explanation
"""


# look at the info of "df"
df.info

"""
Explanation: <div align="right">
<a href="#q4" class="btn btn-default" data-toggle="collapse">Click here for the solution</a>

</div>
<div id="q4" class="collapse">
```
 df[['length','compression-ratio']].describe()
```
</div>

Info
Another method you can use to check your dataset is:
~~~~
dataframe.info
~~~~
It provides a concise summary of your DataFrame.
End of explanation
"""


df.info()

"""
Explanation: Here we are able to see the information of our dataframe, with the top 30 rows and the bottom 30 rows.
It also shows us the whole data frame has 205 rows and 26 columns in total.
you can also use the method info as follows
End of explanation
"""


