%matplotlib inline
import pystyl

"""
Explanation: Paradosis
We start by importing the pystyl library. The first command is to make sure that our matplotlib figures are rendered in the notebook.
End of explanation
"""


from pystyl.corpus import Corpus
corpus = Corpus(language='other')

"""
Explanation: Next, we create an empty corpus object. We have to specify that we are using language for which there is no full support yet by setting language to "other":
End of explanation
"""


print(corpus)

"""
Explanation: The corpus is empty right now:
End of explanation
"""


corpus.add_directory(directory='data')
print(corpus)

"""
Explanation: We can add our single text under the data directory using a single command:
End of explanation
"""


corpus.preprocess(alpha_only=True, lowercase=True)

"""
Explanation: I added the dummy author label (because we don't know the true author). We can now take some common preprocessing steps. We lowercose the text and only keep alphabetic characters:
End of explanation
"""


corpus.tokenize()

"""
Explanation: We tokenize the corpus (the default is to split along whitespace):
End of explanation
"""


corpus.segment(segment_size=2000, step_size=500)

"""
Explanation: This step is more crucial: we segment our corpus (a single text in this case) into consecutive windows or slices of length 2000 (cf. segment_size, expressed in words). The step_size controls to which extent these samples are allowed to overlap:
End of explanation
"""


len(corpus)

"""
Explanation: After this segmentation, our corpus contains 103 consecutive, equal-sized samples:
End of explanation
"""


mfi = corpus.vectorize(mfi=30,
                       ngram_type='word',
                       ngram_size=1,
                       vector_space='tf_std',
                       min_df=0.50)

"""
Explanation: We are now ready to vectorize our corpus, meaning that we will represent as a two-dimensional matrix, with a row for each sample, and a column for each stylistic feature (in this case word frequencies). With the following settings, we extract information word unigram features (ngram_size, ngram_type) for the 30 most frequent items (MFI) in the samples. We scale the original relative frequencies in our matrix, using their standard deviation. This sort of scaling is common in stylometry (cf. Burrows's Delta). The parametrization min_df=0.50 indicates that a feature should be present in at least 50% of the samples analyzed.
End of explanation
"""


print(' - '.join(mfi))

"""
Explanation: If this language is heavily inflected, it might also make sense to extract character n-grams (for character trigrams, use ngram_type='char' and ngram_size=1). As you can see, the function returns the MFI's selected, so that we can inspect the actual features used:
End of explanation
"""


from pystyl.analysis import pca
pca_coor, pca_loadings = pca(corpus, nb_dimensions=2)

"""
Explanation: We can now turn to some exploratory analysis. PCA is a common dimension reduction method in stylometry. It is especially useful because it allows us to plot word loadings. We first import the PCA functionality and apply it to our corpus, asking to retain only the top-2 components:
End of explanation
"""


from pystyl.visualization import scatterplot
scatterplot(corpus, coor=pca_coor,
            nb_clusters=3, loadings=pca_loadings,
            save=True, outputfile="/Users/mike/Desktop/pca.pdf")

"""
Explanation: We can now visualize the samples in the two PCs:
End of explanation
"""


from pystyl.analysis import distance_matrix
dm = distance_matrix(corpus, 'minmax')

"""
Explanation: (I can't make sense of the loadings, because I do not understand this language, but you might.) By changing the nb_clusters, we can control how many clusters will being detected and colored (which is just a reading aid). With the parameters in the previous code block, you will also have saved a (more readable version) of this plot to disk. Here we notice the clear separation of samples 12-18 (in purple) from the bulk of the text -- the other samples tend to fall apart in early (blue) and late samples. Likewise, we can also run a traditional cluster analyses on this data. We first import the functions we need and create a pairwise distance matrix, called dm, using the minmax distance measure.
End of explanation
"""


from pystyl.analysis import hierarchical_clustering
cluster_tree = hierarchical_clustering(dm, linkage='ward')

"""
Explanation: On this distance matrix, we can run a standard clustering algorithm with Ward's linkage method:
End of explanation
"""


from pystyl.visualization import scipy_dendrogram
scipy_dendrogram(corpus=corpus, tree=cluster_tree,
                 outputfile='~/Desktop/scipy_dendrogram.pdf',
                 fontsize=5, save=True, return_svg=False)

"""
Explanation: We can now visualize and export the resulting dendrogram, as we did with the PCA scatterplot:
End of explanation
"""


from pystyl.visualization import clustermap
clustermap(corpus, distance_matrix=dm, fontsize=4, color_leafs=True,\
           outputfile='/Users/mike/Desktop/cm.pdf',
          show=True, save=True, return_svg=False)

"""
Explanation: Here, a threefold clustering seems to emerge, distinghuising between three main groups:
- green samples (begin and and of the text?)
- the curious red part with roughly samples 10-120 (cf. PCA results)
- the remainder of the text in blue, with a divergent clade for samples 65-72 (in purple)
A slightly fancier visualization method is to run seaborn's clustermap on our distance matrix, which runs largely parallel:
End of explanation
"""


from pystyl.analysis import vnc_clustering
vnc_tree = vnc_clustering(dm, linkage='ward')
scipy_dendrogram(corpus=corpus, tree=vnc_tree,
                 outputfile='~/Desktop/vnc.pdf',\
                 fontsize=5, save=True, return_svg=False)

"""
Explanation: The previous block gives some errors, but yields the correct image in the end, which seems to show a similar structure. Finally, we can now apply the VNC algorithm, which will cluster the data while taking into account (and leave unchanged) the original order of the samples in the text. The setup is again similar:
End of explanation
"""


