# convert text files to input for the classifiers
# usage: %run embem/machinelearningdata/txt2for_prediction.py <directory with input text files> <directory for output files>

%run embem/machinelearningdata/txt2for_prediction.py /home/jvdzwaan/data/embem/txt/ceneton/ /home/jvdzwaan/data/embem/txt/ceneton-for_prediction/

"""
Explanation: Pipeline to create data to classify
This pipeline can be used to create input for the classifiers from folia xml or text files.
Please adjust the arguments of scripts (usually refering to the location of input and/or output data) to your local location of the data.
End of explanation
"""


# convert additional folia files to input for prediction
# usage: !./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py <directory with input folia files> <directory for output files>

!./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py /home/jvdzwaan/data/embem_corpus_big/ /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction

# convert annotation folia files to input for prediction
# usage: !./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py <directory with input folia files> <directory for output files>

!./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py /home/jvdzwaan/data/folia-annotatie/ /home/jvdzwaan/data/embem/txt/annotation-for_prediction

"""
Explanation: Convert Folia Files
Because the folia files in corpus big (additional folia files) are not as nicely structured into acts as the texts for annotation, a custom script must be used.
End of explanation
"""


# normalize text data
# usage: %run embem/spellingnormalization/normalize_dataset.py <directory with input text files> <directory for output files>

# corpus big (nederlab texts)
#%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction-normalized

# ceneton texts
%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/ceneton-for_prediction /home/jvdzwaan/data/embem/txt/ceneton-for_prediction-normalized

# edbo texts
#%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/edbo-for_prediction /home/jvdzwaan/data/embem/txt/edbo-for_prediction-normalized

"""
Explanation: Create normalized data for prediction
Once text files for prediction have been created, these should be normalized 
(the best performing classifiers are trained on normalized data).
End of explanation
"""


