summaryDf = pd.DataFrame([extractSummaryLine(l) for l in open('../../data/learnedModel/domain/summary.txt').readlines()],
                        columns=['domain', 'strict', 'clf', 'feature', 'post', 'precision', 'recall', 'f1'])

summaryDf = summaryDf[summaryDf['clf'] != 'KNeighborsClassifier'].sort_values('f1', ascending=False)
print len(summaryDf)
summaryDf[:5]

"""
Explanation: Experience
Based on annoted ground truth, we tried to learn a model to classify domains specific words.
We use as input a combinaison of 4 datasets:
* animal
* vehicle
* plant
* other - a random sample from the whole vocabulary
To do so we will explore the carthesian product of:
* domains: a combinaison of N previously presented domains
* strict: try to compose missing concept 
* randomForest / knn: knn allow us to check if there is anything consistent to learn, randomForest is a basic model as a first approach to learn the function
* feature: one of the feature presented in the guided tour
* postFeature: any extra processing to apply to the feature extraction (like normalise)
We use a 10 K-Fold cross validation.
Once you downloaded the files, you can use this script reproduce the experience at home:
python experiment/trainAll_domainClf.py &gt; ../data/learnedModel/domain/log.txt
Results
Here is the summary of the results we gathered,
You can find details reports in logs.
End of explanation
"""


summaryDf['f1'] = summaryDf['f1'].astype(float)
summaryDf[['feature', 'post', 'f1']].groupby(['feature', 'post']).describe().unstack(level=-1)

"""
Explanation: Considering the nature of the date (really close points for semanticly close concept - ie puppy, dog), Knn is not relevant.
As you can see we, there is a lot trained model (198), therefore,<br>
we need to find a method to select the best combinaison - ie: robust to the number and variety of domains
To do so, we'll select the best average model depending of the dataset combinaison
End of explanation
"""


summaryDf[summaryDf['domain'] == 'animal-plant-vehicle-other'][:1]

summaryDf[(summaryDf['feature'] == 'angular') & (summaryDf['post'] == 'noPost') & (summaryDf['strict'] == '')]

"""
Explanation: We observe several things here:
* The f1-score decrease as we add variety of domains (from ~95% for 2 to ~80% for 4)
* In average, the results are satisfying for the basic model.
* The feature selected angular, polar, carthesian have a litte impact on the average score.
* Adding possibility to compose concept (strict) improve very slightly the score
If we had to select one model, we could choose angular feature with no post processing. which is the best in the edge case (4 domains)
End of explanation
"""


!python ../../toolbox/script/detailConceptClfError.py ../../data/voc/npy/wikiEn-skipgram.npy ../../data/learnedModel/domain/animal-plant-vehicle__RandomForestClassifier_angular_noPost.dill ../../data/domain/luu_animal.txt animal ../../data/domain/luu_plant.txt plant ../../data/domain/luu_vehicle.txt vehicle

"""
Explanation: Study errors
Here is the detail of classification error for combined animal, plant and vehicle:
End of explanation
"""


!python ../../toolbox/script/detailConceptClfError.py ../../data/voc/npy/wikiEn-skipgram.npy ../../data/learnedModel/domain/animal-plant-vehicle-other__RandomForestClassifier_angular_noPost.dill ../../data/domain/luu_animal.txt animal ../../data/domain/luu_plant.txt plant ../../data/domain/luu_vehicle.txt vehicle ../../data/domain/all_1400.txt other

"""
Explanation: Be aware there is no cross validation here, so we are overfitting
Yet, we see collisions seems to be due to several meaning for one concept:<br>
rocket for example, is both a plant and a vehicle, which make it an unsolvable case for this model.
Let's compare with the same domains but adding other:
End of explanation
"""


