# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     name: python3
# ---

# + [markdown] id="_yruzaaA4Q-j"
# # Data Analysis of Chess Grandmasters (GM)
#
# Recently, the Indian-American Abhimanyu Mishra became the [youngest Grandmaster](https://www.chess.com/news/view/abhimanyu-mishra-youngest-grandmaster-in-chess-history) in chess history, qualifying for the title at the age of 12 years 4 months and 25 days, whereas the Venezuelan [Salvador Diaz Carias](https://www.chess.com/news/view/venezuela-chess-player-fm-title-88-salvador-diaz-carias) got the FIDE Master (FM) Title at the age of 88. Motivated by these news, which I came to know by the Brazilian YouTube channel [Xadrez Brasil](https://www.youtube.com/c/XadrezBrasil), I decided to do a data analysis and visualization about the GMs based on Wikipedia "[List of chess grandmasters](https://en.wikipedia.org/wiki/List_of_chess_grandmasters)".

# + [markdown] id="3--xoA6eSSfl"
# ## General Research Questions about GMs
#
# We aim at approaching the following questions. These questions guide our exploration, but does not limit it.
#
# 1. What is the distribution of GM title since 1950 (when it started)?
#
# 2. What is the relationship, if any, between age and receiving GM title?
#
# 3. What is the distribution of GM title among the countries and sex?

# + [markdown] id="V7a2NZYXYtxO"
# ## Python Libraries

# + id="j0WL4wMIXabU"
# Importing the necessary Python libraries to our tasks:

import numpy as np # data analysis
import pandas as pd # data analysis and manipulation
import matplotlib.pyplot as plt # data visualization
# %matplotlib inline
import seaborn as sns # data visualization

from IPython.core.display import display, HTML
# This package allows embedding here the visualization made by Flourish website: https://app.flourish.studio/.

# + [markdown] id="ltpzMCeaY360"
# ## Data Extraction

# + id="EpmBJXgvXaem"
# Our data source: the Wikipedia "List of chess grandmasters".

url = "https://en.wikipedia.org/wiki/List_of_chess_grandmasters"

# + [markdown] id="uTjPKi1HZIn7"
# ## Data Cleaning and Preparation

# + id="yFQAfqdwYQJw" colab={"base_uri": "https://localhost:8080/"} outputId="72908561-8b46-4df2-e187-6d6f55627ff3"
# Reading and selecting the table we are interested in:

html = pd.read_html(url, match = "Birthplace")
html

# Notice the output is a list.

# + colab={"base_uri": "https://localhost:8080/", "height": 572} id="SQKxuPE0YgWZ" outputId="5788da38-3ffe-47a3-a234-29f4f6bfc179"
# Taking the table as a DataFrame from the list:

html[0]

# + colab={"base_uri": "https://localhost:8080/", "height": 419} id="s7pEiMRcAnOh" outputId="decd9a68-817d-47e7-ba6c-a2ccee1a30dd"
# Reducing the table to just the columns we are interested in:

table1 = html[0][["Name", "Born", "TitleYear", "Federation", "Sex"]]
table1

# + id="UCr6NulSAqpX" colab={"base_uri": "https://localhost:8080/", "height": 419} outputId="eceed3c1-c33d-47b3-aa3e-de7bb9e0dcfb"
# Dropping the first nonsense line (line zero):

table2 = table1.drop(0)
table2

# + colab={"base_uri": "https://localhost:8080/"} id="rvvOR261Kna5" outputId="03de80ca-68e3-4e2f-8c97-0ec8d413ca4b"
# Changing the value of the column 'Born' so that we have only the year:

table2['Born'] = table2['Born'].apply(lambda x: x[:4])
table2['Born']

# This introduces a small imprecision in our analysis since we are not going to consider the exact birth date.
# However, as we do not have the exact date of the GM title acquisition, this is the best we can do.

# + colab={"base_uri": "https://localhost:8080/"} id="ci_pGFT-QAcF" outputId="98a1da0e-3a99-4f66-95c3-8bef74797f24"
# Changing the data type of the value of the column 'Born' to integer instead of string:

table2['Born'] = table2['Born'].apply(lambda x: float(x))
table2['Born']

# + colab={"base_uri": "https://localhost:8080/", "height": 419} id="ZP8ediW6LgIw" outputId="94bc1dd0-3d89-4f42-b8e5-08c6ffbfe544"
# Adding a column with TitleAge.
# The age of the GMs when they got the title is the title year minus his/her birth date.

table2['TitleAge'] = table2['TitleYear'] - table2['Born']
table2

# + [markdown] id="P4MfE6a8Yn8L"
# ## Data Analysis and Visualization

# + [markdown] id="RHQIX1qT8-XI"
# ### The age of the GM title receivers

# + colab={"base_uri": "https://localhost:8080/"} id="Pbg4pegtQ4zQ" outputId="9e646c2a-1c81-451c-f51b-8c9f179ccdcf"
# Descriptive statistics summary about the age of GMs when they got the title:

table2['TitleAge'].describe()

# + colab={"base_uri": "https://localhost:8080/", "height": 350} id="3s6y3utGRJsX" outputId="c40a79e4-1c0a-43a1-c1c6-8dec42250fd2"
# Scatter plot showing the distribution of title age of the GM and the year of the title:

sns.scatterplot(table2['TitleYear'], table2['TitleAge'])

# + [markdown] id="ftS5LNil1Q4Q"
# An interactive scatter plot is displayed at the end of this document.

# + colab={"base_uri": "https://localhost:8080/", "height": 350} id="mN3XG03gTfga" outputId="73382bbf-7bf7-472e-c6d4-fa6092d3522e"
# Box plot about the title age of the GMs when they received the title.

sns.boxplot(table2['TitleAge'])

# + [markdown] id="5nw4oY1wakaG"
# By the analysis and visualization above, we notice that:
#
# * The oldest GM title receiver is about 88 years old. But this is **not** entirely correct, because among the 10 oldest chess players (all being at least 77 years old) only [<NAME>](https://en.wikipedia.org/wiki/Jacques_Mieses) (1865-1954) was an active player. He received his title at the age of 85 in 1950 (inauguration of the GM title), but it is said his chess strength was not that great anymore. The other 9 players received honorary titles.
#
# * The mean of GM title receivers age is 27 years old. Indeed, 75% of them is at most 31 years old. The standard deviation is about 10 years old
#
# * Both the scatter plot and, especially, the box plot highlight that whoever receives the GM title at the age of 45 years or above is already an outlier among the GMs. We see this is even more true once we know most elderly GM title receivers got an honorary title, not a regular one.
#
# * The scatter plot also shows a huge growth of GM titles, in particular since 1990.

# + colab={"base_uri": "https://localhost:8080/", "height": 350} id="7EdIooJHUcL2" outputId="29b8b0de-66d5-423c-c9ec-14b145d618cd"
# Distribution of age of GM title receivers.

sns.distplot(table2['TitleAge'])

# + colab={"base_uri": "https://localhost:8080/", "height": 350} id="Aj5LanCKUn-5" outputId="49dd994e-4342-4b68-d60d-70fb829affaf"
# Distribution of GM titles along the years.

sns.distplot(table2['TitleYear'])

# + [markdown] id="giF1ZfKKz3xR"
# The two graphs above show that:
#
# * Again, the age of GM title receivers is concentrated between the 20s years old.
#
# * Since 1990 we have observed a lot more GM titles. Most likely this is due to the fact that chess became much more popular and accessible, particularly after the spreading of home computers and the internet. The more people interested in chess, the more GMs.

# + colab={"base_uri": "https://localhost:8080/", "height": 142} id="5FPXnRvCUcDP" outputId="e80ce2ce-3e3b-4caf-c75a-3dd7621bf1ae"
# Correlations

table2.corr()

# We see the year of birth and the title year are highly correlated with each other

# + colab={"base_uri": "https://localhost:8080/", "height": 359} id="JCskp1F9cN-x" outputId="a0ce9bd2-ddcc-4c74-f23d-37f01bd87c2d"
# The first 10 oldest GM title receivers.

table2.sort_values('TitleAge')[-10:]

# We notice very old GMs often have received honorary title for their career.

# By the way, there are only 19 GMs whose age is at least 70 years old.

# + [markdown] id="lA2suDFR9uul"
# ### Distributions of GMs per Sex and Country

# + colab={"base_uri": "https://localhost:8080/"} id="YhlakvnNgC3w" outputId="d22b9e1c-526c-4154-a792-a0c450fe6a85"
# GMs per country:

countries = dict()
for i in table2['Federation']:
  #print(i)
  countries[i] = countries.get(i, 0) + 1
countries

# + id="KGjakzS90Rq1" colab={"base_uri": "https://localhost:8080/"} outputId="6c693951-9ef2-48dd-fcf6-fc4bedc3ed02"
# GM per sex

sex = dict()
for i in table2['Sex']:
  #print(i)
  sex[i] = sex.get(i, 0) + 1
sex

# Female GMs represent only 1.95% of the total of GMs.

# + colab={"base_uri": "https://localhost:8080/", "height": 1000} id="tpt_-CGl2j2M" outputId="83faa87d-4fb1-4352-8de6-743e3f2a8c53"
# Distributions of GMs per sex and country displayed using Flourish.
# Interactive visualization!

display(HTML('https://public.flourish.studio/visualisation/6633672/'))

# + colab={"base_uri": "https://localhost:8080/", "height": 1000} id="rIrmK53q-x0v" outputId="e7df5d8e-cfc0-4c13-d5af-4cf9fe6ad2d4"
# Scatter plot (with regression line) showing the distribution of title age of the GM and the year of the title:
# Interactive visualization!

display(HTML('https://public.flourish.studio/visualisation/6633947/'))

# + id="bI5gMsM42jWm"
# For converting the dataset seen in the variable table2 (DataFrame) into a csv file:

table2.to_csv('chess.csv')

StarcoderdataJupyterScriptsDedupFiltered