"""
# Project Overview

While crowdfunding Kickstarter projects seem lucrative, it does not mean that it is without risk. When a project fails, the risk extends to both project owners and backers. Project owners who have invested a large amount of money into building the product will suffer from a huge loss if the goal is not reached. Backers who have contributed into the projects may end up not receiving the products that they expected to get.


This project aims to identify the different factors that affect the success of a Kickstarter project. The analysis will be based on data of Kickstarter projects in the year 2018.
"""
"""
## Imports
"""
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
# Getting glimpse of the data

df = pd.read_csv("../input/kickstarter-projects/ks-projects-201801.csv")
df.head()
# I noticed that there is an inconsistency in the format of the column name 'usd pledged'
# This will cause a problem later as Pandas cannot access columns that contain spaces
# Here, I'm replacing columns with ' ' space with '_' underscore

df.columns = [col.replace(" ", "_") for col in df.columns]
# Dropping the column pledged, usd pledged
# These columns are equal to usd_pledged_real & usd_pledged_goal fields
# The only difference is the conversion rate used; _real columns uses conversion provided by Fixer.io.

df = df.drop(labels=["pledged", "usd_pledged", "goal"], axis=1)
df.head()
"""
## Business Understanding

These are the specific questions that we aim to answer in the analysis: 

"Are there differences in project popularity or amount of goal across different categories?"

"Is the goal realistic?"

"What is the average amount pledged by each backer? Are people willing to contribute more of their money into certain projects?"

"Are there differences in project popularity or amount of goal across different durations?"
"""
"""
## Data Understanding & Analysis
"""
"""
In this section, we will deep dive into the data to familiarise ourselves further, identify data quality issues that can potentially affect our analysis and discover insights in the data that can answer our questions.
"""
# Computing summary statistics pertaining to the quantitative variables of the data

df.describe()
"""
**Interpretation**

It is interesting to see that there are a lot of variance between the median (50% percentile) and the maximum values of the quantitative variables...
"""
# Checking the number of rows and columns in the datasest

num_rows = df.shape[0]
num_cols = df.shape[1]

f"There are {num_rows} rows and {num_cols} columns in the dataset"
# Proportion of NaN values in each of the column, sorted from in descending order

(df.isnull().mean()).sort_values(ascending=False)
"""
**Interpretation**

The only column that has NaN values is 'name', but the proportion is very small (0.0011%). It doesn't seem that there is an alarming number of NaN values in any of the columns, which is a good sign.
"""
# A bar chart showing the count of projects by Main Category

count_by_main_cat = (df["main_category"].value_counts()).sort_values(ascending=False)
count_by_main_cat = count_by_main_cat.to_frame().reset_index()

count_by_main_cat.rename(
    columns={"index": "main_category", "main_category": "project_count"}, inplace=True
)

count_by_main_cat.style.bar(subset="project_count", align="mid", color=["#5fba7d"])
"""
**Interpretation**

The top 3 projects that are hosted on Kickstarter are:
1. 'Film&Video'
2. 'Music'
3. 'Publishing'

The bottom 3 projects that are hosted on Kickstarter are:
1. 'Dance',
2. 'Journalism'
3. 'Crafts'
"""
# Finding the number of backers per main_category

main_cat_backers_count = (
    df.groupby(df["main_category"]).sum()[["backers"]].reset_index()
)
main_cat_backers_count = main_cat_backers_count.sort_values(
    by="backers", ascending=False
)
main_cat_backers_count.style.bar(subset=["backers"], align="mid", color=["#5fba7d"])
"""
**Interpretation:**

From the chart above, we can see that these are the most popular 3 main category by the total number of backers:
1. 'Games'
2. 'Design'
3. 'Technology'.


On the bottom rows, these are the least backed projects:
1. 'Dance'
2. 'Journalism'
3. 'Crafts'

On the chart above, I'm actually totalling up the number of backers for each category. The caveat of doing this is that it is possible that there are fewer projects in that category to begin with and therefore, there would not be many backers.

.

.

.

_Let's find the median number of backers per each main category instead._
"""
"""
## Note

I'm using median instead of mean here as the distribution of the quantitative variables are very skewed, as we can see in the table at the beginning of this notebook. Median is a measure of central tendency that is more robust to outliers.
"""
# Median number of backers per category

main_cat_backers_median = (
    df.groupby(df["main_category"]).median()[["backers"]].reset_index()
)
main_cat_backers_median = main_cat_backers_median.sort_values(
    by=["backers"], ascending=False
)

main_cat_backers_median.style.bar(subset=["backers"], align="mid", color=["#5fba7d"])
"""
**Intepretation**

The top 3 main categories that has the highest median number of backers are:
1. Comics
2. Design
3. Games

The top 3 main categories that has the lowest median number of backers are:
1. Journalism
2. Crafts
3. Fashion

It is interesting to see that Comics has the highest median number of backer in a single project. We can also see that Design is second after Comics.

Journalism still has the lowest number of backer. The number of backer a project has doesn't always determine the success of a project. A project can have only 1 backer (sounds extreme, but why not), but he/she can contribute $1,000,000 dollar to a project and the goal can still be easily met. I know this example sounds unrealistic, but I hope you get my point.

.

.

.

.

Next, we will be looking into the median amount of money that each backer put to fund a project. 
"""
# Finding the MEDIAN of the total amount pledged by main_category

median_pledged_by_main_cat = (
    df.groupby(df["main_category"])
    .median()[["usd_pledged_real"]]
    .sort_values(by=["usd_pledged_real"], ascending=False)
    .reset_index()
)

median_pledged_by_main_cat.style.bar(
    subset=["usd_pledged_real"], align="mid", color=["#5fba7d"]
)
"""
**Interpretation**

Top 3 main categories with the highest median amount pledged:

1. Design
2. Dance
3. Theater

Bottom 3 main categories with the lowest median amount pledged:
1. Journalism
2. Crafts
3. Photography

Wow, Design didn't only have a lot of backers but it also does have the highest median amount pledged!
.

.

.

.

Let's also take a look at the median amount of goal set for each category. It doesn't matter if a project has a relatively high total pledged amount or number of backers but it still far away from the goal. On the other hand, a project can only have a few backers and little amount of money pledged, but if the goal set is relatively low, then there is still chance for it to be successful.
"""
# Finding the MEDIAN of the amount of project goal set, by main_category

median_goal_by_main_cat = (
    df.groupby(df["main_category"])
    .median()[["usd_goal_real"]]
    .sort_values(by=["usd_goal_real"], ascending=False)
    .reset_index()
)

median_goal_by_main_cat.style.bar(
    subset=["usd_goal_real"], align="mid", color=["#5fba7d"]
)
"""
**Interpretation**

Top 3 main categories by amount of project goal:
1. Technology
2. Design
3. Food

Bottom 3 main categories by amount of project goal:
1. Crafts
2. Art
3. Theater

Seems that 'Technology' requires the highest amount of money to fund the project! Wait..didn't technology have a relatively low number of backers and median amount pledged though? This means that it is less likely for 'technology' projects to reach the goal.

'Crafts' projects fortunately do not require lots of fund in the first place, even though it doesn't have high number of backers & amount pledged.

.

.

.

Let's find out if this is the case.
"""
# Finding the difference between the goal set and the total amount pledged

goal_vs_pledged = median_goal_by_main_cat.merge(
    median_pledged_by_main_cat, how="outer", on="main_category"
)

goal_vs_pledged["pledged_minus_goal"] = (
    goal_vs_pledged["usd_pledged_real"] - goal_vs_pledged["usd_goal_real"]
)


goal_vs_pledged["pledged_over_goal_rate"] = (
    goal_vs_pledged["usd_pledged_real"] / goal_vs_pledged["usd_goal_real"]
)

goal_vs_pledged.style.bar(
    subset=[
        "usd_goal_real",
        "usd_pledged_real",
        "pledged_minus_goal",
        "pledged_over_goal_rate",
    ],
    align="mid",
    color=["#d65f5f", "#5fba7d"],
)
"""
**Interpretation**

In the chart above, I plotted the median of USD Goal, median of USD Pledged side by side. Then I subtract USD Pledged from USD Goal to see which main category that has the highest amount of losses.

On the right hand side, I created another column that divides USD Pledged / USD Goal to see which project main category that has the highest success rate.

.

.

.


The highest success rate (pledged_over_goal_rate) is seen across the main categories of:
1. Dance
2. Theater
3. Comics

The lowest success rate (pledged_over_goal_rate) is seen across the main categories of:
1. Journalism
2. Technology
3. Crafts

Interestingly, even though 'Technology' seems to be the main_category that typically sets the highest goal (median of 20k USD), the median amount of total money pledged is the lowest.

Crowdfunding through sites like Kickstarter might seem lucrative at first glance, however if creators do not create a comprehensive budgeting plan, they might overlook 'unforeseen' costs that will make it even more difficult for the project to be fully funded

I'm wondering if the goal if duration also plays a part in a project success? We'll analyse the results in the coming sections.
"""
# Calculating 'duration_in_days' and add into the dataframe

df["duration_in_days"] = (
    pd.to_datetime(df["deadline"]) - pd.to_datetime(df["launched"])
).dt.days
df.head(3)
# Returning the unique duration_in_days to check for its values

df.duration_in_days.unique()
"""
In the array above, you can see that the the number of duration in days ranges from 58 to 16,738. I will those values that are above 14,000, which are outliers. This step is necessary as I will create a histogram to visualize the quantitative variables (backers, goals, amount pledged) into the binned duration.

If we include these outliers, then it would be difficult for us to interpret the histogram meaningfully.
"""
# Removing outliers in duration_in_days

df = df.loc[df["duration_in_days"] < 14000]
# Creating bins for duration_in_days

duration_in_days_bins = np.linspace(
    df.duration_in_days.min(), df.duration_in_days.max(), 10
)
df["duration_in_days_bins"] = pd.cut(df.duration_in_days, duration_in_days_bins)

numerical_col_list = ["usd_goal_real", "backers", "usd_pledged_real"]

# Calculating the median of goal, backers & pledged amount for each duration bin
median_by_duration_bins = (
    df.groupby(df["duration_in_days_bins"])[numerical_col_list].median().reset_index()
)
median_by_duration_bins
def plot_hist(
    x,
    bins,
    weights,
    xlabel=None,
    ylabel=None,
    title=None,
    ax=None,
    color="#5fba7d",
    **kwargs
):
    """Plotting a histogram.

    INPUT:
    x - input values of the histogram
    bins - bins of the histogram that the x should be put into
    weights - array of weights with a same shape as x
    xlabel - label of the histogram x axis (optional)
    ylabel - label of the histogram y axis (optional)
    ylabel - title of the histogram y axis (optional)
    ax - subplot axis to plot the histogram (optional)
    color - color of the histogram. default is green.

    OUTPUT
    hist - the histogram plot
    """
    ax = ax or plt.gca()
    # Do some cool data transformations...
    hist = ax.hist(x=x, bins=bins, weights=weights, color=color, **kwargs)
    ax.set_xlabel(xlabel)
    ax.set_ylabel(ylabel)
    ax.set_title("Median Goal by Duration")
    
    return hist
fig, ax = plt.subplots(2, 2, figsize=(10, 10))
fig.delaxes(ax[1, 1])

# 'Median Goal by Duration'
plot_hist(
    duration_in_days_bins[:-1],
    duration_in_days_bins,
    median_by_duration_bins['usd_goal_real'],
    xlabel='Duration in days bins',
    ylabel='Median goal amount',
    title='Median Goal by Duration',
    ax=ax[0,0]
);

# 'Median Backers Count by Duration'
plot_hist(
    duration_in_days_bins[:-1],
    duration_in_days_bins,
    median_by_duration_bins['backers'],
    xlabel='Duration in days bins',
    ylabel='Median backers amount',
    title='Median Backers by Duration',
    ax=ax[0, 1]
);

# 'Median Pledged Amount by Duration'
plot_hist(
    duration_in_days_bins[:-1],
    duration_in_days_bins,
    median_by_duration_bins['usd_pledged_real'],
    xlabel='Duration in days bins',
    ylabel='Median pledged amount',
    title='Median Pledged Amount by Duration',
    ax=ax[1, 0]
);

"""
**Interpretation:**


_Goal_

Initially, I thought that projects with high amount of goal set will have longer duration for the project to allow more time for the projects to attract backers.
Based on the 'Goal by duration' figure, the highest median goal amount interestingly belongs to projects with duration that ranges from 30-60 days.

_Backers_

The longer the duration of a project, does not necessarily mean that the project will end up attracting more backers, as can be seen in the 'Backers count by duration' histogram. The duration range that attracted the highest number of backers is 30-40 days.

_Pledged_

Again, the highest median pledged amount seen in Kickstarter projects in 2018 belong to the 30-40 duration day range.
"""
"""
## Conclusion
"""
"""
While crowdfunding Kickstarter projects seem lucrative, it does not mean that there is no risk. When a project is failing, the risk extends to both project owners and backers. Project owners who have invested a large amount of money into building the product will suffer from a huge loss if the goal is not met. Backers who have contributed the projects may end up not receiving the products that they expected to get.

In the analysis above, we did a comparison of several success metrics of a project (e.g. number of backers, amount pledged) across different categories & project duration. The aim is to help backers & project owners to be more discerning of the different factors that affect a project success.

Here is the summary of our findings:

_Riskiest Category_

- The riskiest category is 'Technology'. Not only does it require a lot of funds in the first place, but it also has the lowest success rate across all main categories.
- 'Journalism' & 'Crafts' also have low success rates even though the amount of funds required to build the project is relatively low. When these projects failed, the amount of loss tends to be a lot lesser than the loss of Technology projects.
- From the initial analysis, 'Design' seems to be pretty popular among backers and it has the highest median amount pledged. However, the goal that is set for Design projects tends to be high, which makes it more challenging for it to be successful.

_Safest Category_

- The safest category to go for is 'Dance', followed by 'Theater' and 'Comics'.
- These projects do not require a lot of fixed costs to start and have the highest success rates.
- If product owners can reduce the cost associated with Design projects significantly, then it can be a good category to launch your product.

_Ideal Duration_

- Duration does not seem to affect the success of a project much. Setting a longer duration does not necessarily increase the amount pledged and the number of backers.
- Based on the analysis, the ideal duration is around 30–60 days, where it can attract the highest number of backers & generated a good amount of money pledged.
"""