import os

import pandas as pd
import seaborn as _

from bob_emploi.data_analysis.lib import read_data
from bob_emploi.data_analysis.lib import plot_helpers

data_folder = os.getenv('DATA_FOLDER')

"""
Explanation: ROME skills
Author: Stephan &#115;&#116;&#101;&#112;&#104;&#97;&#110;&#64;&#98;&#97;&#121;&#101;&#115;&#105;&#109;&#112;&#97;&#99;&#116;&#46;&#111;&#114;&#103;
Date: Jun 14, 2016
We want to allow our users to build a profile of skills they have, in order to suggest suitable jobs to them. A PRD for this feature can be found at http://go/pe:profile-prd. 
We want to use the skills taxonomy from the ROME dataset because it allows us to extract a mapping from skills to jobs. The main questions we want to answer in this notebook are:

How many different skills do we have in total?
How many skills are common for a job?
Any other properties of the skills dataset?
Are some skills more unique to a job than others?
TODO: Are there jobs that have the exact same set of skills?
End of explanation
"""


fiche_dicts = read_data.load_fiches_from_xml(os.path.join(data_folder, 'rome/ficheMetierXml'))
rome = [read_data.fiche_extractor(f) for f in fiche_dicts]
skills_data = [dict(skill, code_rome=job['code_rome'], job_group_name=job['intitule'])
               for job in rome
               for skill in job['skills']]
skills_raw = pd.DataFrame(skills_data)
skills_raw.columns = ['skill_id', 'job_group_id', 'type', 'job_group_name', 'name', 'position', 'priorisation']

"""
Explanation: Before using the XML version of the data, I had a look at the CSV data. I saw that i could use unix_coherence_item_v330_utf8 to establish a mapping between job_groups and skills, however I saw that the XML data contains this ordering of skills. I thought it could be quite useful, if that actually represents a ordering from more to less specific. That is why I used the XML version of the data in this case.
End of explanation
"""


skills_raw.head()

skills_raw.describe().transpose()

"""
Explanation: Overview of dataset structure
A peek at the raw data.
End of explanation
"""


# Mapping extracted from data/rome/csv/unix_referentiel_competence_v330_utf8.csv
COMPETENCE_TYPES = {
    '1': 'theoretical_skill',
    '2': 'action_skill'
}

skills = skills_raw.drop(['priorisation'], axis=1)
skills['type'] = skills.type.map(COMPETENCE_TYPES)
skills['position'] = skills.position.astype(int)

dedup_skills = skills.drop_duplicates(subset=['skill_id'])
dedup_skills['type'].value_counts(normalize=True)

"""
Explanation: So we got 1741 different skills for the 531 different job groups in ROME. They are of two different types where one of them makes up about 70% of all skill occurrences (3964 / 5520). The most common skill is Utilisation d'outils bureautiques.... The position column suggests that they are ordered and the priorisation column does not contain any information.
Let's clean up the data a little and look at the distribution of skill type on the deduplicated list of skills. With that I mean just investigating the list of 1741 unique skills, instead of skills being associated with jobs as in the table above.
End of explanation
"""


skills.groupby('skill_id')['type'].nunique().value_counts()

"""
Explanation: The distribution of theoretical to action skills is present also in the deduplicated list of skills, not only in the skills associated to jobs.
Just as a sanity check, make sure that each skill is only associated with one type.
End of explanation
"""


skills.duplicated(subset=['job_group_id', 'type', 'position']).sum()

skills.duplicated(subset=['job_group_id', 'position']).sum()

"""
Explanation: Good!
Ordering of skills
Does that ordering actually play a role?
Let's first check whether the position is unique per job_group or per job_group and type. From looking at the XML it looks like there are individual orderings per type, but I want to make sure.
End of explanation
"""


skills = skills.sort_values('position')
first = skills.groupby(['job_group_name', 'type']).first().name
last = skills.groupby(['job_group_name', 'type']).last().name
pd.concat([first, last], axis=1, keys=['first', 'last'])

"""
Explanation: Skills are ordered separately for the two types of skills.
are the skills higher up in the ordering maybe more specific to a certain job_group?
End of explanation
"""


skills.groupby('job_group_id').skill_id.nunique().hist();

"""
Explanation: Feedback I got from Pascal: I'm sorry but even if I read French quite fluently I cannot tell whether they are ordered, so I believe they are not.
"Éducation de jeunes enfants" (young kinds education), has its first theoritical skill being "Civil Law" and its last "Technics of listening and realtion to people": I believe the last one is more correlated to the job group.
"Élevage ovin ou caprin" (Breeding of sheep or goats) has its first action skill being "Setup of birth cages", and the last one "welding". I believe the first one is more correlated to the job group.
TODO: 
However I feel that we can compute some interesting measure of it:
1. assign a specificity score to each skill by computing some reverse function of its frequency among job groups (1 / # of job groups with this skill)
2. normalize positions from 0 to 1 for each job group
3. plot 1 & 2 and see if there's any kind of correlation
Supposedly we should see very specific skills being closer to 0.
To make it even closer to something useful, you can also weigh each job group with the number of persons in this job group, for instance using a proxy with the FHS.
Skills per job
How many different skills are usually associated to a job?
End of explanation
"""


by_job_by_type = skills.groupby(['job_group_id', 'type'])
unique_job_counts = by_job_by_type.skill_id.nunique().reset_index()
unique_job_counts.hist(by='type', normed=True, sharex=True, sharey=True);

"""
Explanation: That's a neat distribution with a clear peak around 10. Florian asked in the PRD 'how many skills do we plan to collect per user'. With users having had several jobs in their past, I would expect a user to pick between 15-30 skills, as a very rough estimate. This of course also depends on the typical skill overlap between jobs and a probobably higher skill overlap between jobs in the same field.
Also, as we saw in the summary statistics above, all 531 job groups in ROME have at least one skill associated with it.
Is there a difference between the number of theoretical and action skills?
End of explanation
"""


skill_frequency = skills.name.value_counts() / skills.job_group_id.nunique()
skill_frequency.index = pd.Series(range(len(skill_frequency))) / len(skill_frequency)
skill_frequency.plot(figsize=(8, 4));

"""
Explanation: The theoretical skills resemble the above distribution, but many jobs seem to have only one or two action skills.
Let's see how specific skills are to a job group.
Because each skill only appears once per job_group, I can simply divide the skill frequency by number of job_groups.
End of explanation
"""


counts = skills.groupby(['name', 'type']).skill_id.count().to_frame()
counts.sort_values('skill_id', ascending=False).head(10)

"""
Explanation: There seem to be a very few skills that are associated with lots of different jobs. Most of the skills seem to be very specific to a job. 
That means that we might have trouble suggesting totally new jobs for the skills a user imported from their old job, because there might be hardly any overlap between skills of his old job and a potential new job. Except of the few very common skills that probably don't provide much signal.
What are those very common skills?
End of explanation
"""


skills['category'] = skills.job_group_id.str.slice(0, 1)
skills.groupby('name').category.nunique().hist(bins=14);

skills.groupby(['name', 'type']).category.nunique().hist(bins=14);

skills.groupby(['name', 'type']).category.nunique().reset_index().hist(by='type', normed=True);

"""
Explanation: Looks like very general soft skills.
Skills spread over categories?
Paul asked: check if a given skill is present across many different job categories (look at the first letter of ROME for example) -> possible there's a bias towards only reusing the exact same skill if the jobs belong in the same job category, which would defeat some of the purpose.
End of explanation
"""


