# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # A2: Bias in data
#
# For this assignment, I'll compare [Wikipedia](https://www.wikipedia.org) articles about political figures, authored from various countries. By comparing the count of articles from a country to its population, and also the "quality" of articles about politicians, I hope to see and show that the level of coverage varies significantly and could produce bias within the content of the articles. The quality of a given article will be evaluated using the [ORES](https://www.mediawiki.org/wiki/ORES) service.
#
# First, I import the libraries that my Python 3 code will be using:

import requests   # For the ORES API call 
import numpy      # For replacement of unread values with NaNs
import pandas     # For dataframing, merging, numeric conversions, and reading CSVs

# Next, load the population data from the CSV file (WPDS_2018_data.csv) obtained from [here:](https://www.dropbox.com/s/5u7sy1xt7g0oi2c/WPDS_2018_data.csv?dl=0)

# +
# Load population data
data_population = pandas.read_csv('WPDS_2018_data.csv')

# Rename the columns, because we'll want 'country' to merge, later, and because I like short'population' better
data_population = pandas.DataFrame({'country':data_population['Geography'],
                          'population':data_population['Population mid-2018 (millions)']})
# Take a look at its data
data_population.head()
#data_population
# -

# To acquire the data regarding Wikipedia pages, I download the archive from [here](https://figshare.com/articles/Untitled_Item/5513449) and expand to be able to open \country\data\page_data.csv. For this assignment, this file is stored in the same working directory as this Python notebook. 

data_page = pandas.read_csv('page_data.csv')
# Take a look at its data
data_page.head(4)
#data_page

# To obtain a quality score from ORES, I adapted code from [here,](https://github.com/Ironholds/data-512-a2/blob/master/hcds-a2-bias_demo.ipynb) (credit to GitHub users [jtmorgan](https://github.com/jtmorgan) and [ironholds](https://github.com/Ironholds)). The following function makes an ORES API call, given a list of revision ides to search for and a call header identifying me as caller.

# +
headers = {'User-Agent' : 'https://github.com/pking70', 'From' : '<EMAIL>'}

def get_ores_data(revision_ids, headers):
    # Define the endpoint
    endpoint = 'https://ores.wikimedia.org/v3/scores/{project}/?models={model}&revids={revids}'
    
    # Specify the parameters - concatenating all the revision IDs together separated by | marks. 
    params = {'project' : 'enwiki',
              'model'   : 'wp10',
              'revids'  : '|'.join(str(x) for x in revision_ids)
              }
    
    # Make the call for a response in JSON format
    api_call = requests.get(endpoint.format(**params))
    response = api_call.json()
    #print(json.dumps(response, indent=4, sort_keys=True))
    return response


# -

# To loop through all the page data, I segment the ids into groups of 100. Then I call ORES and append the response to a new dataframe named predictions. For rev_ids for which there is not a valid response, I replace with a numpy NaN.
#
# This code can take a while to execute, depending on the count of pages to query (for this run, I have over 47,000). I uncomment the 'print(i)' statement when I want to see progress and not wonder if it is indefinitely looping, but for now it is commented out.

# +
# So if we grab some example revision IDs and turn them into a list and then call get_ores_data...
rev_ids = list(data_page['rev_id'])   # Extract the rev_ids from the page data
start = 0                             # Start at item 0
step = 100                            # How many ids to query ORES for at once. ORES does not work with large counts, but 100 works
predictions = pandas.DataFrame()      # A dataframe for the prediction results

for i in range(start, len(rev_ids), step): # Loop through all the rev_ids
    
    # print(i)                             # Uncomment this if you want to watch progress 
    rev_ids_set = rev_ids[i:i+step]        # Use this number of ids for the ORES call 
    response = get_ores_data(rev_ids_set, headers)   # Call ORES
    
    for revision in response['enwiki']['scores']:   # Loop through the JSON ORES call response
        try:
            prediction = response['enwiki']['scores'][revision]['wp10']['score']['prediction']   # Store predictions
        except:
            prediction = numpy.nan                  # When there is not a valid response, store a NaN
        
        # In a new dataframe, store revisions and predictions
        predictions = predictions.append({'revision':revision, 'prediction':prediction}, ignore_index=True)
# -

# To review the structure of the predictions dataframe:

predictions.head()

# The predictions data is merged with the page data, joined on their respective revision id fields. The revision field must be converted from string to numeric for this to work.

predictions['revision'] = pandas.to_numeric(predictions['revision'], errors='coerce')
data_page_prediction = data_page.merge(predictions, left_on='rev_id', right_on='revision') 
# Take a look at its data
data_page_prediction.head()

# The prediction+page data is merged with the population data that I loaded into data_population earlier, joined on their respective country fields. I was having trouble with this merge, so I trim any extra spaces from both country fields to possibly improve matching.

data_pagepredpop = data_page_prediction.merge(data_population, left_on='country'.strip(), right_on='country'.strip())
# Take a look at its data
data_pagepredpop.head()

# For the final dataframe, extract the fields we want with the titles requested by the assignment, <a href="https://wiki.communitydata.cc/Human_Centered_Data_Science_(Fall_2018)/Assignments#Combining_the_datasets">here.</a>

data_final = pandas.DataFrame({'country':data_pagepredpop['country'],
                               'population':data_pagepredpop['population'],
                               'article_name':data_pagepredpop['page'],
                               'revision_id':data_pagepredpop['rev_id'],
                               'article_quality':data_pagepredpop['prediction']})
# Take a look at its data
data_final

# To analyze this data, I want to examine which countries have the most (and least) amount of articles on Wikipedia regarding their political figures. I also want to examine the proportion of highly and lowly rated articles for each country.
#
# Note that the quality of an article has been returned by ORES, according to the scale defined <a href="https://wiki.communitydata.cc/Human_Centered_Data_Science_(Fall_2018)/Assignments#Getting_article_quality_predictions">here.</a> 
#
# In short, the prediction column of final_data now contains a value on this spectrum:
#
# 1. FA - Featured article
# 2. GA - Good article
# 3. B - B-class article
# 4. C - C-class article
# 5. Start - Start-class article
# 6. Stub - Stub-class article
#
# To prepare for analysis, I must calculate the count of articles per country, and the per capita ratio of articles per country:

data_article_count = data_final.groupby(['country']).size().reset_index(name='count')
# Take a look at its data
data_article_count.head()

# The article count data is merged with the population data, joined on their respective country fields.

data_popcount = data_population.merge(data_article_count, left_on='country'.strip(), right_on='country'.strip())
# Take a look at its data
data_popcount.head()

# I want the per capita proportion of articles to population. I have to convert population and count to numeric, and also multiply population by one million to scale it according to its defined format (remember, it was 'Population mid-2018 (millions)').

data_popcount['count'] = pandas.to_numeric(data_popcount['count'], errors='coerce')     # Numeric conversion
data_popcount['population'] = pandas.to_numeric(data_popcount['population'], errors='coerce') # Numeric conversion
data_popcount['per capita'] = 100*(data_popcount['count']/(data_popcount['population']*1000000)) # Ratio calculation
# Take a look at its data
data_popcount.head()

# To see the ten highest-ranked countries in terms of number of politician articles as a proportion of country population:
#

data_popcount.sort_values(by='per capita', ascending=False).head(10)

# To see the ten lowest-ranked countries in terms of number of politician articles as a proportion of country population:

data_popcount.sort_values(by='per capita', ascending=True).head(10)

# To see the ten highest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country:
#

data_quality = data_final[(data_final['article_quality']=='GA')|(data_final['article_quality']=='FA' )]
data_quality = data_quality.groupby(['country']).size().reset_index(name='count_quality')
# Take a look at its data
data_quality.head()

# Merge:

data_countqual = data_popcount.merge(data_quality, left_on='country'.strip(), right_on='country'.strip())
# Take a look at its data
data_countqual.head()

# I want the proportion of highly rated articles to total articles. 

data_countqual['proportion'] = data_countqual['count_quality']/data_countqual['count']
# Take a look at its data
data_countqual.head()

# To see the ten highest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country:

data_countqual.sort_values(by='proportion', ascending=False).head(10)

# To see the ten lowest-ranked countries in terms of number of GA and FA-quality articles as a proportion of all articles about politicians from that country:

data_countqual.sort_values(by='proportion', ascending=True).head(10)

# However, the above table ranks only countries for which there are high quality articles to count. All the countries that had zero (0) GA or FA articles have been omitted, as their proportion is 0. To rank countries that have no high quality articles, would be impossible. They are all tied at 0, which would make them all equally the "lowest." To see which countries completely lack high quality articles for comparison, first I create a dataframe that contains the counts of all level of quality articles by country:

data_allquality = data_final.groupby(['country']).size().reset_index(name='count_allquality')
# Take a look at its data
data_allquality

# There are 180 such countries.
#
# I can find the indexes within all countries that have quality articles with this merge:

data_indexes = pandas.merge(data_allquality.reset_index(), data_quality)
data_indexes

# Then, by dropping these indexes (the indexes of countries that have high quality articles) I produce a new dataset of countries that lack any high quality articles, which I call data_lowquality:

data_lowquality = data_allquality.drop(data_indexes['index'])
data_lowquality

# In essence, this is the list of countries that have zero (0) highly rated articles. It is not exactly meaningful to rank them; They are above, in alphabetic order: the 37 countries with the lowest possible proportion of highly rated articles.
#
# Finally, I save my data to a CSV for sharing and reproducability:

data_final.to_csv('data_final.csv')

# For further reflection upon the meaning of the data processing and analysis within this notebook, please see the Readme file.

StarcoderdataJupyterScriptsDedupFiltered