"""
# Stack Overflow Developer Survey, 2020
Heads up!
* Survey tends to have some biasness as all programmers don't use stackover flow, so dataset may represent like minded people
* Survey dataset in general tends to have a lot of **human error** which must be looked for.
> a. Respondents may not feel encouraged to provide accurate, honest answers    
b. Respondents may not be fully aware of their reasons for any given answer because of lack of memory on the subject, or even boredom.  
c. Surveys with closed-ended questions may have a lower validity rate than other question types.  
d. Data errors due to question non-responses may exist. which leads to `NaN` Values  
e. The number of respondents who choose to respond to a survey question may be different from those who chose not to respond, thus creating bias.  

## `Errors in Survey dataset`
1. **Survey error** refers to `mistakes made in the construction and implementation of the survey instrument, as well as the interpretation of results.` Survey error is associated with `completeness, interpretability of the data, and the generalizability of the results.` *This type of error affects the validity of the measurement and what is called* ***measurement error***. Survey error is avoided, to some extent, by carefully designing the study, creating a comprehensive set of well-crafted survey items, and properly interpreting and presenting the results. 
2. **Purpose Creep Error** At times researchers (or clients) will add items to a survey that don’t directly align with the needs and purposes of the study.
3. **Sampling Error** This occurs when mistakes are made selecting a sample. Sampling error typically occurs when a specific subgroup within the population is under- or overrepresented in the sample. When this happens, the results obtained from the survey are not generalizable. This is because the characteristics of respondents in the sample do not match proportionally with the characteristics of those in the population. This can easily happen with small samples, especially when there are a number of small subgroups within the population. 
4. **Item Nonresponse Error** This happens when a participant fails to answer all the questions on a survey. It can happen by accident or intentionally. Nonresponse is unfortunate when you are administering a questionnaire but does not necessarily invalidate the results; the results depend on how many respondents failed to answer a specific question. 

## `Common Forms of Response Bias`
1. **Response bias** is a general term that describes the ways a respondent may be influenced when self-reporting their answers on a survey. Bias affects the accuracy and reliability of the results. There are several ways in which an individual’s response may be inaccurate. The inaccuracy, or bias, may be deliberate or subconscious on the part of the respondent; it is the result of a respondent being unduly influenced to respond in a certain way, or a respondent’s unwillingness to answer the questions honestly. Bias can be caused when items are unclear or poorly constructed but can also occur as a result of the response scale selected for participants to use when they provide their answers. Never underestimate the propensity of individuals to misunderstand what is being asked or otherwise provide information that is inaccurate to some degree.
2. **Random Response Bias** Random response bias can occur when a respondent honestly does not know the answer to the question but answers anyway.
3. **Demand Characteristic Bias** A demand characteristic is used to describe specific cues in research that may inadvertently influence a participant’s response.
4. **Central Tendency Bias** This bias refers to the tendency of some individuals to avoid responding in extreme ways.
5. **Mood Bias and Emotional Mind Sets** One’s mood or mindset will affect the way responses are provided. 
6. **Item and Option Order Effect Bias** Order bias can be the result of both item order and response option order. The order in which survey items are presented can affect a respondent’s answers due to a priming effect. People tend to contextualize their responses. Because of this, survey questions that come just before a particular query may provide information that respondents will use as context in formulating their subsequent answers. If a different primer was presented, the responses may be significantly different.
`Two common response biases associated with response option order` are **primacy and recency bias**.**Primacy bias** is the tendency for respondents to pick one of the first options presented to them. This can happen when a respondent quickly reads through the survey and picks one of the first response options they agree with.**Recency bias** is the tendency to pick an answer option presented at the end of a list. For example, in a long list of options, the choices respondents read last are more memorable to the respondent as they select an answer.  
7. **Prestige Bias** This bias is related to social desirability bias as it is based on an individual’s personal desire to be seen in a positive light. This bias is based on personal feelings, not a general instinct for conformity. 
8. **Acquiescence or Agreement Bias** This bias is like conformity bias. However, unlike conformity bias, in this case the respondent will, in general and inadvertently, agree with statements. With this bias, participants tend to select a positive response option or disproportionately indicate a positive connotation. `This bias will skew results towards the positive.`
"""
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import seaborn as sns
import matplotlib.pyplot as plt
pd.set_option('display.max_columns',None)
import warnings
warnings.filterwarnings('ignore')
schema= pd.read_csv('../input/stack-overflow-developer-survey-2020/survey_results_schema.csv', index_col='Column') ## setting column name as index so that we can easily access question
df=pd.read_csv('../input/stack-overflow-developer-survey-2020/survey_results_public.csv')
print('Questions asked in survey','\n')
display(schema.head(60))
print()
df
categoricat_feature= df.select_dtypes(include='object').columns
categoricat_feature
non_obj_feature= df.select_dtypes(exclude='object').columns
non_obj_feature
plt.figure(figsize=(7,7))
sns.heatmap(df.isnull());
"""
There are a lot of missing data
"""
(df.isnull().mean()*100).sort_values(ascending=False).head(45)
df.info()
df.describe()
"""
There are few outliers/strange data feeded
1. Age min is `1 year` and max is `279 years`
2. Working hour/ week min is `1hr` which to me seem false, and max is `475hr/week` which is again **not possible as 24*7=168**
3. Why does **CompTotal** have std infinite?
"""
### Extracting questions from schema
print(schema.loc['CompTotal'])
"""
This is printing half question so to solve this issue I will convert this df into series  
As this df only have 1 column and `Column` is set as index this won't be bothering   
Lets check it
"""
schema_df = schema.QuestionText
schema_df.loc['CompTotal']
"""
This question explicitly ask people to input `whole no` it seems people didn't follow through
"""
"""
## Data Preparation & Cleaning

While the survey responses contain a wealth of information, we'll limit our analysis to the following areas:

- Demographics of the survey respondents and the global programming community
- Distribution of programming skills, experience, and preferences
- Employment-related information, preferences, and opinions

Let's select a subset of columns with the relevant data for our analysis.
"""
selected_columns = [
    # Demographics
    'Country',
    'Age',
    'Gender',
    'EdLevel',
    'UndergradMajor',
    # Programming experience
    'Hobbyist',
    'Age1stCode',
    'YearsCode',
    'YearsCodePro',
    'LanguageWorkedWith',
    'LanguageDesireNextYear',
    'NEWLearn',
    'NEWStuck',
    # Employment
    'Employment',
    'DevType',
    'WorkWeekHrs',
    'JobSat',
    'JobFactors',
    'NEWOvertime',
    'NEWEdImpt'
]
len(selected_columns)
# Let's extract a copy of the data from these columns into a new data frame `survey_df`.
## We can continue to modify further without affecting the original data frame.
sdf = df[selected_columns].copy()
display(sdf.sample(10))

## Same for schema

schema = schema_df[selected_columns]
display(schema.head(5))
sdf.info()
"""
**Age1stCode**, **YearsCode**,**YearsCodePro** seems to have float datatype but are stored as object  
Lets explore it's reason...
"""
sdf['Age1stCode'].unique()
sdf['YearsCode'].unique()
sdf['YearsCodePro'].unique()
"""
It seems these columns store few object data type like :  "Less than 1 year",'More than 50 years' , 'Younger than 5 years' , 'Older than 85' etc  
we can solve this problem in 2 way
1. Use error management like, `errors='coerce'` 
> errors : {‘ignore’, ‘raise’, ‘coerce’}, default ‘raise’  
     If **‘raise’**, then invalid parsing will raise an exception  
     If **‘coerce’**, then invalid parsing will be set as NaN  
     If **‘ignore**’, then invalid parsing will return the input  

2. Use some numerical values to idicate they belong to outliers
"""
sdf[sdf['YearsCodePro']== 'More than 50 years']
sdf[sdf['Age1stCode']=='Younger than 5 years']
"""
**Observation:** 
* YearsCode & YearsCodePro are mostly having same values and this is the difference between  `age - Age1stCode`
* 'Age1stCode'='Younger than 5 years' seems data entered for fun and seems false entry
* `Way ahead:` Use error management like, errors='coerce'
"""
sdf['Age1stCode'] = pd.to_numeric(sdf.Age1stCode, errors='coerce')
sdf['YearsCode'] = pd.to_numeric(sdf.YearsCode, errors='coerce')
sdf['YearsCodePro'] = pd.to_numeric(sdf.YearsCodePro, errors='coerce')
print(sdf.Age.mean(), sdf.Age.min(), sdf.Age.max())
sdf.Age.hist(bins=30);
"""
There seems to be a problem with the age column, *as the minimum value is 1* and *the maximum is 279*.   
This is a common issue with surveys: responses may contain invalid values due to accidental or intentional errors while responding. A simple fix would be to ignore the rows where the age is higher than 100 years or lower than 10 years as invalid survey responses.  
We can do this using the `.drop` method, [as explained here](https://www.geeksforgeeks.org/drop-rows-from-the-dataframe-based-on-certain-condition-applied-on-a-column/).
"""
sdf.drop(sdf[sdf.Age < 10].index, inplace=True)
sdf.drop(sdf[sdf.Age > 100].index, inplace=True)
# The same holds for `WorkWeekHrs`. Let's ignore entries where the value for the column is higher than 140 hours. (~20 hours per day).
sdf.drop(sdf[sdf.WorkWeekHrs > 140].index, inplace=True)
"""
##### Lets explore Gender
"""
sdf['Gender'].value_counts()
## We will replace gender containing ';' with nan
# Below code will do that
sdf.where(~(sdf.Gender.str.contains(';', na=False)), np.nan, inplace=True)
sdf.sample(10)
"""
There are still many columns with NaN value  
 we can remove rows with more than 10 rows having NaN value
"""
"""
# Exploratory Analysis and Visualization

Before we ask questions about the survey responses, it would help to understand the respondents' demographics, i.e., country, age, gender, education level, employment level, etc. It's essential to explore these variables to understand how representative the survey is of the worldwide programming community. A survey of this scale generally tends to have some [selection bias](https://en.wikipedia.org/wiki/Selection_bias).

Let's begin by importing `matplotlib.pyplot` and `seaborn`.
"""
import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

sns.set_style('whitegrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'
###############~~~~~~~~~~~~ Creating Custom Color Palette ~~~~~~~~~~~~###############
#24477f : Mck deep blue
#0892d0 : Mck electric blue
#66b3ff : light blue

mck_color2 = sns.color_palette(['#0892d0', '#24477f'])
mck_color3 = sns.color_palette(['#0892d0', '#24477f', 'black'])
mck_color = sns.blend_palette(['#0892d0', '#24477f','black'],9)
mck_color12 = sns.blend_palette(['#0892d0', '#24477f','black'],12)
mck_color25 = sns.blend_palette(['#41aad9','#0892d0', '#24477f','black'],25)
sns.color_palette('Blues')
sns.color_palette(mck_color3)
"""
## `Country`

Let's look at the number of countries from which there are responses in the survey and plot the ten countries with the highest number of responses.
"""
top_countries = sdf.Country.value_counts().head(15)

plt.figure(figsize=(12,6))
plt.xticks(rotation=75)
plt.title(schema.Country)
sns.barplot(x=top_countries.index, y=top_countries, palette='Blues_r');
"""
It appears that a disproportionately high number of respondents are from the US and India, probably because the survey is in English, and these countries have the highest English-speaking populations. We can already see that the survey may not be representative of the global programming community - especially from non-English speaking countries. Programmers from non-English speaking countries are almost certainly underrepresented.  
***Conclusion***  
As **40% surveyees are from `USA`,`India`, `UK`**. This indicates this survey `bias towards english speaking nation`

**Way ahead for survey**:
Survey should be conducted in multiple language so to encorage participation from non-english speaking countries
"""
"""
## `Age`

The distribution of respondents' age is another crucial factor to look at. We can use a histogram to visualize it. 
"""
plt.figure(figsize=(12, 6))
plt.title(schema.Age)
plt.xlabel('Age')
plt.ylabel('Probability Density')
sns.distplot(sdf.Age , color='blue', rug=True, rug_kws = { 'height': 0.05, 'color': 'xkcd:navy'}, kde_kws={'lw':2, 'color': 'xkcd:navy', 'bw':0.08} )
plt.show();
print('Mean Age is: ', sdf.Age.mean())
# 'lw' Thickness of kde plot 
# 'bw' Bandwidth controls the smoothness of kdplot
# By default rug is false
"""
It appears that a large percentage of respondents are `20-45 years old`. It's somewhat representative of the programming community in general. Many young people have taken up computer science as their field of study or profession in the last 20 years.

`Note:` The x-axis is the value of the variable just like in a histogram, but what exactly does the y-axis represent?  
**ANS->** The y-axis in a density plot is the probability density function for the kernel density estimation. However, we need to be careful to specify this is a probability density and not a probability. The difference is the probability density is the probability per unit on the x-axis. To convert to an actual probability, we need to find the area under the curve for a specific interval on the x-axis. Somewhat confusingly, because this is a probability density and not a probability, the y-axis can take values greater than one. The only requirement of the density plot is that the total area under the curve integrates to one. I generally tend to think of the y-axis on a density plot as a value only for relative comparisons between different categories.
for [reference](https://towardsdatascience.com/histograms-and-density-plots-in-python-f6bda88f5ac0) 


## `Gender`

Let's look at the distribution of responses for the Gender. It's a well-known fact that women and non-binary genders are underrepresented in the programming community, so we might expect to see a skewed distribution here.
"""
schema.Gender
gender_counts = sdf.Gender.value_counts()
gender_counts
# Pie chart
###### Use when trying to COMPARE A CATEGORY or GROUP OF CATEGORIES to the TOTAL  #########
labels = gender_counts.index 
sizes = gender_counts
#colors
colors = ['#24477f','#0892d0','#66b3ff']
#explsion
explode = (0.05,0.05,0.075)
plt.figure(figsize=(12,6))
plt.pie(sizes, colors = colors, labels=labels, autopct='%1.1f%%', startangle=180, pctdistance=0.85, explode = explode)
#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)
# Equal aspect ratio ensures that pie is drawn as a circle  
plt.title(schema.Gender)
plt.tight_layout()
plt.show()
"""
Only about 8% of survey respondents who have answered the question identify as women or non-binary. This number is lower than the overall percentage of women & non-binary genders in the programming community - which is estimated to be around 12%. 

**Intresting**: This EDA notebook express `Gender Divide in Data Science Roles` [Gender Divide in Data Science](https://medium.com/datadriveninvestor/exploratory-data-analysis-eda-understanding-the-gender-divide-in-data-science-roles-9faa5da44f5b)

## `Education Level`

Formal education in computer science is often considered an essential requirement for becoming a programmer. However, there are many free resources & tutorials available online to learn programming. Let's compare the education levels of respondents to gain some insight into this. We'll use a horizontal bar plot here.
"""
z=sdf.EdLevel
sns.countplot(y= z, palette=mck_color)
plt.xticks(rotation=75);
plt.title(schema['EdLevel'])
plt.ylabel(None);
"""
It appears that well **over half of the respondents hold a bachelor's or master's degree**, so `most programmers seem to have some college education.` However, it's not clear from this graph alone if they hold a degree in computer science.
"""
schema.UndergradMajor
undergrad_pct = sdf.UndergradMajor.value_counts() * 100 / sdf.UndergradMajor.count()

sns.barplot(x=undergrad_pct, y=undergrad_pct.index, palette= mck_color12)

plt.title(schema.UndergradMajor)
plt.ylabel(None);
plt.xlabel('Percentage');
"""
It turns out that 40% of programmers holding a college degree have a field of study other than computer science - which is very encouraging. It seems to suggest that while a college education is helpful in general, you do not need to pursue a major in computer science to become a successful programmer.

**Way ahead**: Analyze the `NEWEdImpt` column for respondents who hold some college degree vs. those who don't. Do you notice any difference in opinion?
"""
"""
## `NEWEdImpt`
"""
schema.NEWEdImpt
sdf.NEWEdImpt.value_counts()
a=sdf.EdLevel
b=sdf.NEWEdImpt
temp= pd.concat([a,b],axis=1)
temp.dropna(inplace=True)
temp[temp['NEWEdImpt']=='Fairly important'].value_counts()
temp[temp['NEWEdImpt']=='Not at all important/not necessary'].value_counts()
temp[temp['EdLevel']=='Bachelor’s degree (B.A., B.S., B.Eng., etc.)'].value_counts()
temp[temp['EdLevel']=='Bachelor’s degree (B.A., B.S., B.Eng., etc.)'].value_counts()
"""
#### Analyse with respect to **UndergradMajor**
"""
"""
## `Employment`

Freelancing or contract work is a common choice among programmers, so it would be interesting to compare the breakdown between full-time, part-time, and freelance work. Let's visualize the data from the `Employment` column.
"""
schema.Employment
(sdf.Employment.value_counts(normalize=True, ascending=True)*100).plot(kind='barh', color= mck_color)
plt.title(schema.Employment)
plt.xlabel('Percentage');
"""
It appears that close to 10% of respondents are employed part time or as freelancers.

**Exercise**: Add a new column `EmploymentType` containing the values `Enthusiast` (student or not employed but looking for work), `Professional` (employed full-time, part-time or freelancing), and `Other` (not employed or retired). For each of the graphs that follow, show a comparison between `Enthusiast` and `Professional`.

"""
schema.DevType
sdf.DevType.value_counts()
## Creating a function for Feature Engineering

def split_multicolumn(col_series):
    result_df = col_series.to_frame()
    options = []
    # Iterate over the column
    for idx, value  in col_series[col_series.notnull()].iteritems():
        # Break each value into list of options
        for option in value.split(';'):
            # Add the option as a column to result
            if not option in result_df.columns:
                options.append(option)
                result_df[option] = False
            # Mark the value in the option column as True
            result_df.at[idx, option] = True
    return result_df[options]
dev_type_df = split_multicolumn(sdf.DevType)
dev = dev_type_df.copy()
dev_type_df
"""
The `dev_type_df` has one column for each option that can be selected as a response. If a respondent has chosen an option, the corresponding column's value is `True`. Otherwise, it is `False`.

We can now use the column-wise totals to identify the most common roles.
"""
dev_type_totals = dev_type_df.sum().sort_values(ascending=False)
dev_type_totals
"""
As one might expect, the most common roles include "Developer" in the name. 

**Way ahead**: 

* Figure out what percentage of respondents work in roles related to data science? 
* Which positions have the highest percentage of women?
"""
dev_type_prc = dev_type_df.mean().sort_values(ascending=False)*100
display(dev_type_prc )

### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[10] = '#ff781f'
custom[11] = '#ff6600'

### visualization the data 

plt.figure(figsize=(12, 12))
sns.barplot(x=dev_type_prc, y=dev_type_prc.index, palette= custom)
plt.title("Roles held by the respondents");
plt.xlabel('Percentage');
## Which positions have the highest percentage of women?

dev['Gender']= sdf['Gender']
dev.dropna(inplace=True)
gen_dev= dev.groupby('Gender').sum().T
a= gen_dev['Man']
b= gen_dev['Non-binary, genderqueer, or gender non-conforming']
gen_dev['Man']= gen_dev['Man']/(gen_dev['Man'] + gen_dev['Non-binary, genderqueer, or gender non-conforming']+ gen_dev['Woman'])*100
gen_dev['Non-binary, genderqueer, or gender non-conforming'] = b/(a+b+gen_dev['Woman'])*100
gen_dev['Woman'] = gen_dev['Woman']/ (a+b+ gen_dev['Woman'])*100
gen_dev.sort_values(by= 'Woman', ascending= False)
"""
#### Which positions have the highest percentage of women?
* **Marketing or sales professional** with 12.14%
* Followed by **Data scientist or machine learning specialist** with 9.05
"""
"""
## `Job Satisfaction`
"""
temp= sdf['JobSat'].copy()
display(temp.value_counts(dropna=True))
a= sdf['JobSat']
b = sdf['Gender']
temp= pd.concat([a, b], axis=1) 
temp.dropna(inplace=True)
male= temp.value_counts().head()
female= temp.value_counts().iloc[5:10]
third = temp.value_counts().iloc[10:16]

gender= [male, female, third]
# Pie chart
###### Use when trying to COMPARE A CATEGORY or GROUP OF CATEGORIES to the TOTAL  #########
j=1
for i in gender:
    plt.figure(figsize=(18,25))
    plt.subplot(1,3,j)
    labels = i.index 
    sizes = i
    #colors
    colors = ['#24477f','#0892d0','#66b3ff', '#6fd0fc','#80d7ff']
    #explsion
    explode = (0.05,0.05,0.05,0.05,0.05)
    plt.pie(sizes, colors = colors, labels=labels, autopct='%1.1f%%', startangle=180, pctdistance=0.85, explode =explode)
    #draw circle
    centre_circle = plt.Circle((0,0),0.70,fc='white')
    fig = plt.gcf()
    fig.gca().add_artist(centre_circle)
    # Equal aspect ratio ensures that pie is drawn as a circle  
    j=j+1
    plt.show()
"""
## `Work Week Hours`
"""
schema.WorkWeekHrs
plt.figure(figsize=(12, 6))
plt.title(schema.WorkWeekHrs)
plt.xlabel('WorkWeekHrs')
plt.ylabel('Probability Density')
sns.distplot(sdf.WorkWeekHrs , color='blue', rug=True, rug_kws = { 'height': 0.02, 'color': 'xkcd:navy'}, kde_kws={'lw':2, 'color': 'xkcd:navy', 'bw':0.1})
plt.show();
print('Mean Work hour/Week is: ', int(sdf.WorkWeekHrs.mean()),'Hour per Week')
"""
## `Hobbyist` relation with `JobSat`
"""
temp.info()
a= sdf.Hobbyist
a.replace(('Yes', 'No'), (True, False), inplace=True)
a= a.astype(bool)
b = sdf.JobSat
#temp = pd.DataFrame([a,b])
temp= pd.concat([a, b], axis=1) 
temp.dropna(inplace=True)
i=temp.groupby('JobSat').sum().sort_values(by= 'Hobbyist', ascending= False)
display(i)
print()
## Ploting a pie plot

labels = i.index 
sizes = i.Hobbyist
#colors
colors = ['#24477f','#0892d0','#66b3ff', '#6fd0fc','#80d7ff']
#explsion
explode = (0.05,0.05,0.05,0.05,0.05)
plt.pie(sizes, colors = colors, labels=labels, autopct='%1.1f%%', startangle=180, pctdistance=0.85, explode =explode)
#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)
# Equal aspect ratio ensures that pie is drawn as a circle  
plt.title('Relation of Job Satisfaction with relation to Hobbyist')
plt.show()
sdf
"""
## Asking and Answering Questions

We've already gained several insights about the respondents and the programming community by exploring individual columns of the dataset. Let's ask some specific questions and try to answer them using data frame operations and visualizations.
"""
"""
#### Q: What are the most popular programming languages in 2020? 

To answer, this we can use the `LanguageWorkedWith` column. Similar to `DevType`, respondents were allowed to choose multiple options here.
"""
sdf.LanguageWorkedWith
languages_worked_df = split_multicolumn(sdf.LanguageWorkedWith)
languages_worked_df
"""
It appears that a total of 25 languages were included among the options. Let's aggregate these to identify the percentage of respondents who selected each language.
"""
languages_worked_percentages = languages_worked_df.mean().sort_values(ascending=False) * 100
languages_worked_percentages
### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[3] = 'red'
custom[2] = 'green'

### visualization the data 

plt.figure(figsize=(12, 12))
sns.barplot(x=languages_worked_percentages, y=languages_worked_percentages.index, palette= custom)
plt.title("Languages used in the past year");
plt.xlabel('count');
"""
Perhaps unsurprisingly, Javascript & HTML/CSS comes out at the top as web development is one of today's most sought skills. It also happens to be one of the easiest to get started. SQL is necessary for working with relational databases, so it's no surprise that most programmers work with SQL regularly. Python seems to be the popular choice for other forms of development, beating out Java, which was the industry standard for server & application development for over two decades.

**Way ahead**:

* What are the most common languages used by students? How does the list compare with the most common languages used by professional developers?
* What are the most common languages among respondents who do not describe themselves as "Developer, front-end"?
* What are the most common languages among respondents who work in fields related to data science?
* What are the most common languages used by developers older than 40 years of age? 
* What are the most common languages used by developers in India?
"""
"""
#### Q: Which languages are the most people interested to learn over the next year?

For this, we can use the `LanguageDesireNextYear` column, with similar processing as the previous one.
"""
languages_interested_df = split_multicolumn(sdf.LanguageDesireNextYear)
languages_interested_percentages = languages_interested_df.mean().sort_values(ascending=False) * 100
languages_interested_percentages
### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[0] = 'red'
custom[3] = 'green'

### Visualization
plt.figure(figsize=(12, 12))
sns.barplot(x=languages_interested_percentages, y=languages_interested_percentages.index, palette = custom)
plt.title("Languages people are intersted in learning over the next year");
plt.xlabel('count');
"""
Once again, it's not surprising that Python is the language most people are interested in learning - since it is an easy-to-learn general-purpose programming language well suited for a variety of domains: application development, numerical computing, data analysis, machine learning, big data, cloud automation, web scraping, scripting, etc. We're using Python for this very analysis, so we're in good company!

**Way ahead**: Repeat the exercises from the previous question, replacing "most common languages" with "languages people are interested in learning/using."
"""
"""
#### Q:  Which are the most loved languages, i.e., a high percentage of people who have used the language want to continue learning & using it over the next year?

While this question may seem tricky at first, it's straightforward to solve using Pandas array operations. Here's what we can do:

- Create a new data frame `languages_loved_df` that contains a `True` value for a language only if the corresponding values in `languages_worked_df` and `languages_interested_df` are both `True`
- Take the column-wise sum of `languages_loved_df` and divide it by the column-wise sum of `languages_worked_df` to get the percentage of respondents who "love" the language
- Sort the results in decreasing order and plot a horizontal bar graph
"""
languages_loved_df = languages_worked_df & languages_interested_df
languages_loved_percentages = (languages_loved_df.sum() * 100/ languages_worked_df.sum()).sort_values(ascending=False)
### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[2] = 'red'
custom[10] = 'green'
### Visualization

plt.figure(figsize=(12, 12))
sns.barplot(x=languages_loved_percentages, y=languages_loved_percentages.index, palette= custom)
plt.title("Most loved languages");
plt.xlabel('count');
"""
[Rust](https://www.rust-lang.org) has been StackOverflow's most-loved language for [four years in a row](https://stackoverflow.blog/2020/01/20/what-is-rust-and-why-is-it-so-popular/). The second most-loved language is TypeScript, a popular alternative to JavaScript for web development.

Python features at number 3, despite already being one of the most widely-used languages in the world. Python has a solid foundation, is easy to learn & use, has a large ecosystem of domain-specific libraries, and a massive worldwide community.

**Way ahead:** What are the most dreaded languages, i.e., languages which people have used in the past year but do not want to learn/use over the next year. Hint: `~languages_interested_df`.
"""
"""
#### Q: In which countries do developers work the highest number of hours per week? Consider countries with more than 250 responses only.

To answer this question, we'll need to use the `groupby` data frame method to aggregate the rows for each country. We'll also need to filter the results to only include the countries with more than 250 respondents.
"""
countries_df = sdf.groupby('Country')[['WorkWeekHrs']].mean().sort_values('WorkWeekHrs', ascending=False)
high_response_countries_df = countries_df.loc[sdf.Country.value_counts() > 250].head(15)
high_response_countries_df
"""
The Asian countries like Iran, China, and Israel have the highest working hours, followed by the United States. However, there isn't too much variation overall, and the average working hours seem to be around 40 hours per week.

**Way ahead:**

* How do the average work hours compare across continents? You may find this list of [countries in each continent](https://hub.jovian.ml/wp-content/uploads/2020/09/countries.csv) useful.
* Which role has the highest average number of hours worked per week? Which one has the lowest?
* How do the hours worked compare between freelancers and developers working full-time?
"""
"""
#### Q: How important is it to start young to build a career in programming?

Let's create a scatter plot of `Age` vs. `YearsCodePro` (i.e., years of coding experience) to answer this question.
"""
schema.YearsCodePro
sns.scatterplot(x='Age', y='YearsCodePro', hue='Hobbyist', data=sdf, palette=mck_color2)
plt.xlabel("Age")
plt.ylabel("Years of professional coding experience");
"""
You can see points all over the graph, which indicates that you can **start programming professionally at any age**. Many people who have been coding for several decades professionally also seem to enjoy it as a hobby.

We can also view the distribution of the `Age1stCode` column to see when the respondents tried programming for the first time.
"""
plt.title(schema.Age1stCode)
sns.histplot(x=sdf.Age1stCode, bins=30, kde=True);
"""
As you might expect, most people seem to have had some exposure to programming before the age of 40. However, but there are people of all ages and walks of life learning to code.

"""
"""
#### Q: What are the most common languages used by developers older than 40 years of age?
"""
a=sdf[sdf["Age"]>40]
b = split_multicolumn(a.LanguageWorkedWith)

languages_worked_percentages = b.mean().sort_values(ascending=False) * 100

### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[4] = 'red'
custom[1] = 'green'

### visualization the data 

plt.figure(figsize=(12, 12))
sns.barplot(x=languages_worked_percentages, y=languages_worked_percentages.index, palette= custom)
plt.title("Languages used by users above 40 Years");
plt.xlabel('Percentage');
"""
#### Q: Check for popularly used language by users with 20+ years of experience
"""
a = sdf[sdf['YearsCodePro']>20]
b= split_multicolumn(a.LanguageWorkedWith)

languages_worked_percentages = b.mean().sort_values(ascending=False) * 100

### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[5] = 'red'
custom[0] = 'green'

### visualization the data 
plt.figure(figsize=(12, 12))
sns.barplot(x=languages_worked_percentages, y=languages_worked_percentages.index, palette= custom)
plt.title("Languages used by programmers with 20+ years of experience");
plt.xlabel('Percentage');
### For gender 
i=a['Gender']
i.dropna(inplace=True)
## Ploting a pie plot

labels = i.unique()
sizes = i.value_counts()

#colors
colors = ['#24477f','#0892d0','#66b3ff', '#6fd0fc','#80d7ff']
#explsion
explode = (0.05,0.05,0.05)
plt.pie(sizes, colors = colors, labels=labels, autopct='%1.1f%%', startangle=180, pctdistance=0.85, explode =explode)
#draw circle
centre_circle = plt.Circle((0,0),0.70,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)
# Equal aspect ratio ensures that pie is drawn as a circle  
plt.title('Gender-wise work force distribution of people with 20+ yrs of experience')
plt.show()
"""
#### Q: What are the most common languages used by developers in India?
"""
a=sdf[sdf['Country']=='India']
b = split_multicolumn(a.LanguageWorkedWith)

languages_worked_percentages = b.mean().sort_values(ascending=False) * 100

### Creating custom color for this visualization
custom = sns.color_palette(mck_color25)
custom[4] = 'red'
custom[2] = 'green'

### visualization the data 

plt.figure(figsize=(12, 12))
sns.barplot(x=languages_worked_percentages, y=languages_worked_percentages.index, palette= custom)
plt.title("Most common languages used by developers in India");
plt.xlabel('Percentage');
"""
# Q: Language usage with respect to devtype
"""
"""
# Inferences and Conclusions

We've drawn many inferences from the survey. Here's a summary of a few of them:

- Based on the survey respondents' demographics, we can infer that the survey is somewhat representative of the overall programming community. However, it has fewer responses from programmers in non-English-speaking countries and women & non-binary genders.

- The programming community is not as diverse as it can be. Although things are improving, we should make more efforts to support & encourage underrepresented communities, whether in terms of age, country, race, gender, or otherwise.


- Although most programmers hold a college degree, a reasonably large percentage did not have computer science as their college major. Hence, a computer science degree isn't compulsory for learning to code or building a career in programming.

- A significant percentage of programmers either work part-time or as freelancers, which can be a great way to break into the field, especially when you're just getting started.

- Javascript & HTML/CSS are the most used programming languages in 2020, closely followed by SQL & Python.

- Python is the language most people are interested in learning - since it is an easy-to-learn general-purpose programming language well suited for various domains.

- Rust and TypeScript are the most "loved" languages in 2020, both of which have small but fast-growing communities. Python is a close third, despite already being a widely used language.

- Programmers worldwide seem to be working for around 40 hours a week on average, with slight variations by country.

- You can learn and start programming professionally at any age. You're likely to have a long and fulfilling career if you also enjoy programming as a hobby.

- **SQL and Python** are 2nd and 5th most popular language in India which is used for Data Science. Where as **R is 17th** most popular language in India.
### Way ahead
**Will compare these finding with kaggle survey finding, as it more targeted towards data science community**
"""
"""
## If u like this Notebook Please upvote this notebook !!!
**And comment any suggestion in comment, `Thank You!!!`**
"""