# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="view-in-github" colab_type="text"
# <a href="https://colab.research.google.com/github/nazianafis/StackOverflow-Data-Analysis/blob/main/StackOverflow_Survey_Analysis.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

# + [markdown] id="FvFRfXTSefZ4"
# # **Analyzing Stack Overflow Survey Data 2020**
# (Check out my related Medium blog [here.](https://nazianafis.medium.com/best-country-to-work-in-2020-15a790b00904))
#
#

# + [markdown] id="45afJcWwVF0m"
# In this project I will be analyzing the **Stack Overflow Annual Developer Survey 2020** data. The main survey results are in *survey_results_public.csv*, you can download it directly from Stack Overflow [here](https://insights.stackoverflow.com/survey).
#
# For this analysis I will be focusing on the following three questions:
#
# ### **Questions:**
#
# 1. Which country provides the highest job satisfaction for developers?
#
# 2. Is there a difference in job satisfaction between developers who had an IT related undergraduate major and those who didn’t?
#
# 3. Do developers from non IT backgrounds think differently about the importance of having a formal education, such as a university degree in Computer Science, for their career?
#
#
# ---
#
#

# + [markdown] id="wew78nFeYJAg"
# ### **Step 1 : Data Acquisition & Preparation**

# + colab={"base_uri": "https://localhost:8080/"} id="_JZjHgWnYPp1" outputId="11bc45dd-b73b-4fd1-9813-f01ba9fc48f0"
# !git clone https://github.com/nazianafis/StackOverflow-Data-Analysis.git

# + id="AL81jGVmVTlk" colab={"base_uri": "https://localhost:8080/"} outputId="99b3afb8-44b7-479c-cc39-de609bfe10b9"
# Import libraries
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

from matplotlib.colors import ListedColormap

import warnings
warnings.filterwarnings('ignore')

# Load the dataset
data = pd.read_csv('/content/StackOverflow-Data-Analysis/survey_results_public.csv')
data.head

# + [markdown] id="tEjFR6QZuIAb"
# The dataset contains 64461 rows and 61 columns.

# + id="a-iT_-JKVYlx" colab={"base_uri": "https://localhost:8080/"} outputId="6f50c8c9-08a9-47f0-9012-95f35b1c9340"
# Filter data so that it only contains respondents who identify as 'developer'
developer_data = data[data['MainBranch'] == 'I am a developer by profession']

# Fill the NA values in 'JobSat' column with the mode
developer_data['JobSat'] = developer_data['JobSat'].fillna(developer_data['JobSat'].mode()[0])

# Fill the NA values in 'NEWEdImpt' column with the mode
developer_data['NEWEdImpt'] = developer_data['NEWEdImpt'].fillna(developer_data['NEWEdImpt'].mode()[0])


# Fill the NA values in 'ConvertedComp' column with the mean of each country
salary_per_country = developer_data.groupby('Country')['ConvertedComp'].mean()
developer_data['ConvertedComp'] = developer_data.fillna(lambda x: salary_per_country[x['Country']] if pd.isnull(x['ConvertedComp']) else x['ConvertedComp'], axis=1)
developer_data = developer_data.dropna(subset=['ConvertedComp'])

# Set 'JobSat' and 'NEWEdImpt' columns into categorical datatype so that they will be sorted with the right order (ordinal data)
developer_data['JobSat'] = pd.Categorical(developer_data['JobSat'], categories=['Very dissatisfied','Slightly dissatisfied','Neither satisfied nor dissatisfied','Slightly satisfied','Very satisfied'], ordered=True)

developer_data['NEWEdImpt'] = pd.Categorical(developer_data['NEWEdImpt'], categories=['Not at all important/not necessary','Somewhat important','Fairly important','Very important','Critically important'], ordered=True)

developer_data.head

# + [markdown] id="FCEI5Twg9joF"
# The subset only contains 47193 rows.

# + [markdown] id="znpYHV1hVd0F"
# So far, I have done the following tasks:
#
# * I **filtered** the data so that it contains only those respondents who identify as 'developer'.
# * I set the *JobSat* and *NEWEdImpt* columns to categorical datatype.
# * Because *JobSat* and *NEWEdImpt* columns contain categorical data, I filled their empty cells with the respective **mode** values.
# * Because salaries depend on the country the developer lives in, I filled all NA values in *ConvertedComp* with the respective **mean** salary of developer's country.
# * There were still some empty values in *ConvertedComp* column since the mean *ConvertedComp* value for some countries was equal to NaN (i.e. all of the respondents from those countries hadn’t answered the salary question). So I **dropped** those records.
#
# ---
#
#
#

# + [markdown] id="BZgVV3ozAL2F"
# ### **Step 2 : Data Exploration**

# + [markdown] id="0e3amEy5AN5U"
# **1. Countries with the Highest Job Satisfaction Rates for Developers**

# + id="0N9ORtqFVjFK"
# Filter the data so that it only contains the survey result from top 20 countries based on the number of total respondents
top_20_countries = list(developer_data['Country'].value_counts().head(20).index)
filtered_developer_data = developer_data[developer_data['Country'].isin(top_20_countries)]

# Aggregate the data so we can visualize it later
aggregated_data = filtered_developer_data.groupby(['Country','JobSat'], as_index=False)['Respondent'].count()

# Add column percentage
total_respondents_per_country = filtered_developer_data['Country'].value_counts()

aggregated_data['Respondent Percentage'] = aggregated_data.apply(lambda x: x['Respondent'] / total_respondents_per_country[x['Country']], axis=1).round(3)

# Pivot the data
aggregated_data = aggregated_data[['Country', 'JobSat', 'Respondent Percentage']]
pivoted_data = pd.pivot_table(aggregated_data, values='Respondent Percentage', index='Country', columns='JobSat').sort_values('Very satisfied', ascending=False)

# + [markdown] id="4Vb_lZmbAXj9"
# Because the survey contains data of respondents from more than 180 countries, I narrowed it down to the top 20 based on each country’s total number of respondents. This was desirable because now each country in the top 20 had at least 400 respondents (which was a good enough number for my analysis).

# + id="1GlDTfoIVm91" colab={"base_uri": "https://localhost:8080/", "height": 465} outputId="f490130f-8123-4905-9181-4f153fad91af"
# Visualize the data
pivoted_data.plot(kind='bar',stacked=True,colormap=ListedColormap(sns.color_palette("ch:2.5,-.2,dark=.3")),figsize=(13,5))
plt.legend(loc="upper left", bbox_to_anchor=(1,1.02))
plt.ylabel('Satisfaction Percentages')
plt.title('\nJOB SATISFACTION PERCENTAGE BY COUNTRY\n')
plt.show()


# + [markdown] id="5IsIkK29AeO7"
# Analysis revealed that Sweden topped the charts, with almost 50% of the developers feeling very satisfied. United States stood second and Netherlands came in third.

# + [markdown] id="3RLSt0_wAi3_"
# **2. Undergraduate Major and Job Satisfaction**
#

# + id="ENw2eD5dVqXo" colab={"base_uri": "https://localhost:8080/"} outputId="2781fe1b-4155-45e8-bddd-286fd78baa49"
# Create a new column to flag each respondent if their major is related or not
def check_major(value):
    '''
    input: the value of each row in the selected column
    output: the mapped value based on the criteria in this function
    '''

    if value == 'Computer science, computer engineering, or software engineering' or value == 'Information systems, information technology, or system administration' or value == 'Web development or web design':
        return 'IT Related'
    return 'Non IT Related'

developer_data['Major'] = developer_data['UndergradMajor'].apply(check_major)

developer_data['Major'].value_counts(normalize=True)

# + [markdown] id="Wog0F8JPAqPs"
# I **split** the *UndergradMajor* column into two — ‘IT Related’ and ‘non IT Related’. Majority of the developers on Stack Overflow (about 65%) had an IT related undergraduate major.

# + id="NFHSaVsUVqih"
# Aggregate the data so that we know how many respondents are there in each type of major ('IT related' and 'non IT related')
aggregated_data = developer_data.groupby(['Major','JobSat'], as_index=False)['Respondent'].count()

# Add column percentage
total_respondents_per_major = developer_data['Major'].value_counts()

aggregated_data['Respondent Percentage'] = aggregated_data.apply(lambda x: x['Respondent'] / total_respondents_per_major[x['Major']], axis=1).round(4)

# Pivot the data
aggregated_data = aggregated_data[['Major', 'JobSat', 'Respondent Percentage']]
pivoted_data = pd.pivot_table(aggregated_data, values='Respondent Percentage', index='Major', columns='JobSat')

# + [markdown] id="MkJM6nBsAwAI"
# Then I calculated the percentages of each value in *JobSat* to measure job satisfaction for each type of major.

# + id="SinxIue2Vyt_" colab={"base_uri": "https://localhost:8080/", "height": 380} outputId="96649394-8851-4a3b-d0ba-0d128b5b41d3"
# Visualize the data
pivoted_data.plot(kind='barh',stacked=True,colormap=ListedColormap(sns.color_palette("ch:2.5,-.2,dark=.3")),figsize=(13,5))
plt.legend(loc="upper left", bbox_to_anchor=(1,1.02))
plt.xlabel('Satisfaction Percentages')
plt.ylabel('Undergraduate Major')
plt.title('\nJOB SATISFACTION PERCENTAGE BY UNDERGRADUATE MAJOR\n')
plt.show()

# + id="X3h7infoV2i6" colab={"base_uri": "https://localhost:8080/", "height": 143} outputId="276a06a3-5772-4674-dc24-b3a8e2e05aa8"
pivoted_data

# + [markdown] id="sd2pqlB_A5FP"
# I found that developers whose undergraduate major is ‘non IT related’ have better job satisfaction as compared those who have an ‘IT related’ background, with 53% and 35% respondents feeling very satisfied respectively.

# + [markdown] id="3m3ESk2zA9VU"
# **3. Importance of Formal Education**

# + id="jofJSbfZV6IN"
# Aggregate the data to know the number of respondents in each type of major ('IT related' / 'Non IT related')
aggregated_data = developer_data.groupby(['Major','NEWEdImpt'], as_index=False)['Respondent'].count()

# Add column percentage
total_respondents_per_major = developer_data['Major'].value_counts()

aggregated_data['Respondent Percentage'] = aggregated_data.apply(lambda x: x['Respondent'] / total_respondents_per_major[x['Major']], axis=1).round(4)

# Pivot the data
aggregated_data = aggregated_data[['Major', 'NEWEdImpt', 'Respondent Percentage']]
pivoted_data = pd.pivot_table(aggregated_data, values='Respondent Percentage', index='Major', columns='NEWEdImpt')

# + id="3AxbYY0-WE-P" colab={"base_uri": "https://localhost:8080/", "height": 380} outputId="53db1cde-fa81-4ba8-cf38-2f4293fc4c44"
# Visualize the data
pivoted_data.plot(kind='barh',stacked=True,colormap=ListedColormap(sns.color_palette("ch:2.5,-.2,dark=.3")),figsize=(13,5))
plt.legend(loc="upper left", bbox_to_anchor=(1,1.02))
plt.xlabel('Answer to "Is formal education important?"')
plt.ylabel('Undergraduate Major')
plt.title('\nIMPORTANCE OF FORMAL EDUCATION BY UNDERGRADUATE MAJOR\n')
plt.show()


# + [markdown] id="N2SZZNShWLCE"
# Analysis revealed that developers with IT related backgrounds agreed more with the importance of having a formal education, such as a university degree in Computer Science, for their career. The opposite was true for non IT background developers.

# + [markdown] id="DM496xOjWNZd"
#
#
# ---
#
#
# ### **Conclusion**
#
# Here is a quick summary of my analysis of the Stack Overflow Annual Developer Survey 2020 data:
#
# 1.     I measured job satisfaction levels for each country and found that Sweden topped the charts. United States stood second and Netherlands came in third.
# 2. I looked at undergraduate majors and found that developers from non IT backgrounds had better job satisfaction.
# 3. I found that developers from non IT fields believed that having a formal education was not as important for their career. Their belief was backed up by the fact that despite not having a background in IT, these respondents were not only working as developers but also had higher job satisfaction.
#
#
# ---

StarcoderdataJupyterScriptsDedupFiltered