# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# *best viewed in [nbviewer](https://nbviewer.jupyter.org/github/CambridgeSemiticsLab/BH_time_collocations/blob/master/results/notebooks/time_distribution_and_pos.ipynb)*

# # Time Distribution and "Parts of Speech"
# ## Beginning Analysis of Hebrew Time Adverbials
# ### <NAME>
#
# <a href="../../docs/sponsors.md"><img height=200px width=200px align='left' src="../../docs/images/CambridgeU_BW.png"></a>

# ! echo "last updated:"; date

# ## The Phenomena of "Time Adverbials"
#
# Time adverbials are crucial and ubiquitous constructions in the clause structures of worldwide languages ([Klein 1994; Haspelmath 1997](../../docs/bibliography.txt)). These arguments, which can consist of any unit from single words up to dependent clauses, coordinate with the main verb, alongside other arguments, to locate and modify events/predications. Word and phrasal time adverbials derive from two main sources: words prototypically used as "adverbs," and noun phrases. Adverbs are occasionally distinguished by morphological features. In Biblical Hebrew, for instance, the suffix *-am* distinguishes words like יומם as adverbial. But in Hebrew explicit morphological markers are otherwise rare. Adverbs have to be distinguished syntactically and behaviorally instead. Noun-based time adverbials are the genetic descendants of locative phrases (Haspelmath 1997). They are most frequently prepositional, and they utilize other elements common to noun phrases: definite articles, demonstratives, quantifiers, construct states, etc.
#
# In this notebook, we (me and you) will set out the primary surface forms of attested time adverbials in the Hebrew Bible, meaning we will compare and quantify all identical and similar phrases. We will visualize and model the distribution of time adverbials across the whole Hebrew Bible. And finally we will investigate the validity of so-called "parts of speech." By what criteria does one define a "part of speech"? How do these categories affect the interpretation of time adverbials in Biblical Hebrew? And is it possible to derive a data-driven approach to such an analysis?

# <hr>

# # Python
#
# Now we import the modules and data needed for the analysis.

# +
# standard packages
from pathlib import Path
import collections
import networkx as nx
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams['font.serif'] = ['SBL Biblit']
import seaborn as sns
#sns.set(font_scale=1.5, style='whitegrid')
#from sklearn.decomposition import PCA
from adjustText import adjust_text
from bidi.algorithm import get_display # bi-directional text support for plotting

# import custom modules (kept under /tools)
# # !NB! add to Python path to avoid Module Not Found errors
from stats.significance import contingency_table, apply_fishers
from stats.pca import apply_pca
from paths import figs, main_table
from tf_tools.load import load_tf
from cx_analysis.load import cxs
from cx_analysis.search import SearchCX

# set up Text-Fabric methods for navigating corpus data
TF, API, A = load_tf(silent='deep')
A.displaySetup(condenseType='phrase')
F, E, T, L = A.api.F, A.api.E, A.api.T, A.api.L

# import Construction objects for more corpus data + syntax structures
se = SearchCX(A)
phrase2cxs = cxs['phrase2cxs']
class2cx = cxs['class2cx']
time_cxs = list(phrase2cxs.values())

# set up path configs for figure images
name = 'time_distribution_and_pos'
figures = figs.joinpath(name)
if not figures.exists():
    figures.mkdir(parents=True)
def fg_path(fg_name):
    return figures.joinpath(fg_name)


# -

# # Dataset
#
# The dataset used by this analysis is originally based on the ETCBC's [BHSA syntactic corpus](https://github.com/ETCBC/bhsa). BHSA encodes phrases along with syntactic features for the entire Hebrew Bible. One of these features is that of `function`, referring to a phrase's role within a clause. A phrase with `function=Time` (time phrases) is the object of this analysis: phrases that modify time within the argument structure of a clause.
#
# The BHSA syntax data provides a useful starting point. One of the first challenges in using this data for research, however, is in evaluating the content and quality of the syntactic data. Tagging syntax, as with many things in linguistics, is far from an objective task. Another challenge is found in the BHSA datamodel itself. BHSA, in general, does not contain very much semantic data (e.g. word classes). It also emphasizes clause and discourse elements at the expense of the phrase model. This makes the task of navigating and classifying time phrases more difficult. In order to evaluate the quality and content of the BHSA time phrase data, we need to be able to navigate the phrase structure.
#
# To augment these shortcomings, this project adds some semantic intelligence and parsing to BHSA time phrases. This is achieved using a custom class called [`Construction`](../../tools/cx_analysis/cx.py). The class, using the [grammar](../../data/cxs) written for it, parse the BHSA time phrases to provide more detailed semantic distinctions (especially for prepositional words) and a more precise parsing of phrase structure. The phrase structure is stored as a directed graph (using NetworkX). That structure is then used to tag features of interest in the time phrases.
#
# Finally, a [script](../../data/cxs/dataset.py) uses the new data to build a dataset. We import that dataset below as a Pandas `DataFrame` and explore its contents.

times_full = pd.read_csv(main_table, sep='\t')
times_full.set_index(['node'], inplace=True)

# The full dimensions of the dataset are shown:

times_full.shape

# We show the top 5 rows of the `DataFrame` below.

times_full.head()

# **The dataset currently contains some complex phrases which consist of multiple time units coordinated together. For the time being we set those aside to focus on time adverbials which have no more than one profiled time unit.**
#
# This choice is necessary since complex time phrases have not yet been fully parsed. It is my hope to add them later.

times = times_full[~times_full.classi.str.contains('component')]

times.shape

# The difference in size is compared below:

print(f'Size difference in dataset: {times_full.shape[0]-times.shape[0]}')

# ## Distribution
#
# Before describing the time adverbials in depth, let's look at their distribution throughout the Hebrew Bible. We can visualize distribution as across a single dimension, a sequence of clauses.
#
# For a set of clauses within a single book, `1–N`, create clusters of clauses, where `cluster = a 50 clause stretch`. If a book ends without an even 50 clauses, keep the uneven cluster as either its own cluster (if `N-clauses > 30`) or add it to last cluster in the book. 

# +
# divide texts evenly into slices of 50 clauses

clause_segments = []

for book in F.otype.s('book'):
    
    clauses = list(L.d(book,'clause'))
    cluster = []
    
    for i, clause in enumerate(clauses):

        i += 1
        
        # skip non-Hebrew clauses
        lang = F.language.v(L.d(clause,'word')[0])
        if lang != 'Hebrew':
            continue

        cluster.append(clause)

        # create cluster of 50
        if (i and i % 50 == 0):
            clause_segments.append(cluster)
            cluster = []

        # deal with final uneven clusters
        elif i == len(clauses):
            if len(cluster) < 30:
                clause_segments[-1].extend(cluster) # add to last cluster
            else:
                clause_segments.append(cluster) # keep as cluster
# -

# Let's see how many segments have been made.

len(clause_segments)

# NB that several segments are slightly larger or smaller than 50. Here are the sizes of the under/over-sized clusters.

unevens = [cl for cl in clause_segments if len(cl) != 50]
print('lengths of uneven-sized clusters:')
for cl in unevens:
    print(len(cl), end='; ')

# We will now iterate through the clusters and tally the number of time adverbials contained within each one. We track along the way the starting points for each new book in the corpus. Those are recorded so they can be plotted. The plot is presented further below as a strip-plot. 

# +
# map book names for visualizing

# map grouped book names
book_map = {'1_Kings': 'Kings', '2_Kings':'Kings', '1_Samuel':'Samuel',
            '2_Samuel':'Samuel', '1_Chronicles':'Chronicles', '2_Chronicles':'Chronicles',}
# book of 12
for book in ('Hosea', 'Joel', 'Amos', 'Obadiah',
             'Jonah', 'Micah', 'Nahum', 'Habakkuk',
             'Zephaniah', 'Haggai', 'Zechariah',
             'Malachi'): 
    book_map[book] = 'Twelve'

# Megilloth
for book in ('Ruth', 'Lamentations', 'Ecclesiastes', 
             'Esther', 'Song_of_songs'): 
    book_map[book] = 'Megilloth'

# Dan-Neh
for book in ('Ezra', 'Nehemiah', 'Daniel'): 
    book_map[book] = 'Daniel-Neh'

# +
# build strip plot data

strip_data = []
covered_nodes = set()
bookboundaries = {}

# time adverbial slots for testing whether
# a clause contains a TA or not
ta_slots = set(
    s for cx in time_cxs
        for sp in cx
            for s in sp.slots
)

# iterate through constructions and gather book data
this_book = None

for i, seg in enumerate(clause_segments):
    
    for cl in seg:
        book, chapter, verse = T.sectionFromNode(cl)
        this_book = book_map.get(book, book)
        
        
        if set(L.d(cl,'word')) & ta_slots:
            strip_data.append(i+1)
            
        # add book boundaries for plotting
        if this_book not in bookboundaries: 
            bookboundaries[this_book] = i+1
# -

strip_title = 'Distribution of time adverbials by segments of ~50 clauses across Hebrew Bible'
plt.figure(figsize=(20, 6))
sns.stripplot(x=strip_data, edgecolor='black', linewidth=0.6, color='lightblue', jitter=0.3)
plt.xticks(ticks=list(bookboundaries.values()), labels=list(bookboundaries.keys()), rotation='vertical', size=16)
plt.ylabel('random "jitter" effect for visibility', size=11)
plt.xlabel('sequence of ~50 clause segments across books (x=1 is first 50 clauses of Genesis)', size=11)
plt.grid(axis='x')
plt.savefig(fg_path('distribution_by_50clauses.png'), dpi=300, bbox_inches='tight')
plt.title(strip_title, size=16)
print(strip_title) # keep title out of savefig
print('x-axis: Nth clause cluster')
print('y-axis: random jitter effect for visibility')
plt.show()

# We can see a slightly sparser population of time adverbials in some of the poetic books, especially Job-Proverbs.

# ## Top Surface Forms

# +
token_counts = pd.DataFrame(times.token.value_counts())

token_counts.head(25)
# -

# ## Noun / Adverb Distinction?
#
# In the surface forms we can recognize numerous features which are common to phrases in Hebrew: prepositions, plural morphemes, definite articles, demonstratives. We also see modifications of number with cardinal number quantifiers or qualitative quantifiers like כל. **In the quest to specify the semantics and build semantic classes, we need to know how these various constructions relate to one another.** What does each element contribute to the whole? What is their relationship to one another? How does their use contribute to the precise meaning of a phrase? 
#
# A cursory overview of the time adverbial surface forms above shows that not all time-words combine evenly with these features. Some regularly do, such as יום. But other words like עתה, אז, עולם seem to appear on their own more often, occasionally with a preposition. These are words that are often classified as "particles" or quintessential "adverbs".
#
# ### Variability Demonstrated: עולם
#
# A closer inspection of certain so-called particles reveals that these larger tendencies are not always absolute, but variable. Take עולם, for example. Below we count and reveal all of its surface forms.

olam = times.query('time == "עולם"')
olam_tokens = pd.DataFrame(olam.token.value_counts())
display(olam_tokens)

# Let's count up the relevant features in the phrases.

olam_features = pd.pivot_table(olam, index='time', values=['bare', 'definite', 'time_pl', 'genitive'], aggfunc=np.sum)
olam_features['total'] = olam_features.sum(1)
olam_features

# Inspection of the various forms occuring with עולם as the head shows variable preference for nominalizing constructions. The vast majority (166/175) appear without modifications ("bare") like definite articles and pluralization. But 8 cases do deploy nominalizing constructions such as definites and plurals.

# **How do we account for both the tendencies and the variability of words like these?** And how does that inform our analysis of the semantics?
#
# I propose, in line with the approach of Construction Grammar, to avoid assuming a univeral word class (e.g. "noun" versus "adverb"). Rather, parts-of-speech tendencies are manifested through collocational patterns. Some words may be highly associated with a given pattern, and the associations may even be semantically motivated. For instance, the word אישׁ will co-occur requently with patterns associated with (animate) objects. The high co-occurrence frequency is motivated by the meaning of the word itself. Framed in this way, we can talk more about a continuum of uses: a word may be more or less associated with contexts that indicate, for example, an object (traditionally a "noun"). 
#
# Croft also treats parts of speech as a continuum of semantic features rather than discrete categories (Croft, *Radical Construction Grammar*, 2001). He provides the following illustration for constructions in English which characterize that continuum (2001: 99):

# <img height=30% width=30% src="../../docs/images/figures/Croft_2001_POS_map.jpg">

# Croft notes two axes along with language encodes certain functions. The y-axis denotes an object-to-action continuum, whereas the x-axis denotes a reference-to-predication continuum. Words that are prototypical nouns will co-occur often with constructions indicating referentiality (REF) and object attributes (OBJECT). This is the upper left-hand corner. Croft lists indicators of "number" as such a marker. On the opposite corner are words which are heavily construed as actions with predication, i.e. prototypical verbs. These co-occur with morpheme markers like tense or modality. 
#
# Can we model this continuum somehow? Doing so would enable us to look for subtle collocational tendencies above the level of the phrase. For example, in a previous study, I found prelimary evidence that the *yiqṭol* verb tends to collocate with time adverbials which do not nominalize with features such as definite articles (see results [here](https://github.com/CambridgeSemiticsLab/BH_time_collocations/blob/master/archive/2019-10-31/analysis/exploratory/construction_clusters.ipynb)). Or, to put it simply, it seems that *yiq̣tol* prefers times with more particle-like than nominal behavior. If we were to have more precise data about time-word behavior, we could more accurately measure these kinds of clause-level tendencies.

# ## Test Collocational Tendencies within Time Adverbial Components

# The remaining time adverbials are now further filtered based on the exclusions noted above. They will be stored based on the lexeme string of their heads.

# +
head_cols = pd.pivot_table(
    times, 
    index=['time'], 
    values=['time_pl', 'quantified', 'definite', 
            'demonstrative', 'ordinal', 'time_sffx',
            'bare', 'genitive'],
    aggfunc=np.sum
)

head_cols = head_cols.loc[head_cols.sum(1).sort_values(ascending=False).index] # sort on sum
# -

head_cols

# We now have a co-occurrence matrix that contains all of our counts. Let's have a look at the top values.

head_cols.head(10)

# ## Dataset Stats

# ### Raw sums across features

head_cols.sum()

# ## Pruning
# Some heads will occur only once or so. We will remove any cases with a sample size `< 5`.

head_cols_pruned = head_cols[head_cols.sum(1) > 4]

# Here is the comparison of size after we've applied our pruning.

head_cols.shape # before

head_cols_pruned.shape # after

# ### Normalization: ratio
#
# There are a number of ways to normalize the counts so that more and less common words are evenly compared. We can use a contingency-based method, which looks at each count in relation to the size of the dataset. This would allow us to isolate relationships which are statistically significant to a particular word. We can also use a non-contingency method such as a simple ratio (percentage). In this case, the ratio makes a bit more sense because we want to focus more closely on each word's own distributional tendencies. None of the data can be "statistically insignificant". The data is normalized below. Each decimal value is out of 1.

# +
head_cols_ratio = head_cols_pruned.divide(head_cols_pruned.sum(1), axis=0)

head_cols_ratio.head(10)
# -

head_cols.index.shape

# ## Nominalizing Tendencies
#
# Here we will plot the nominalizing tendencies of words to get an idea about parts of speech.

pca_values, loadings = apply_pca(head_cols_ratio, 0, 1)

loadings[:3]

# +
# -- plot words --
x, y = (pca_values.iloc[:,0], pca_values.iloc[:,1])
plt.figure(figsize=(12, 10))
plt.scatter(x, y, color='')
#plt.grid(b=None)
plt.xticks(size=10)
plt.yticks(size=10)
plt.xlabel('PC1', size=16)
plt.ylabel('PC2', size=16)
plt.axhline(color='black', linestyle='-', linewidth=0.5)
plt.axvline(color='black', linestyle='-', linewidth=0.5)

# -- annotate words --
texts = []
text_strings = head_cols_ratio.index
text_strings = [get_display(s).replace('\u05C1','') for s in text_strings]
for i, txt in enumerate(text_strings):
    tx, ty = x[i], y[i]
    texts.append(
        plt.text(
            tx, ty, txt, 
            size=11,
            weight='heavy',
            family='serif',
        )
    )
    
# -- annotate features --
# configure offsets / feature-specific font sizes:
offsets = {
    'definite': (-0.04, 0.005, 11),
    'genitive': (-0.08, 0, 10),
    'quantified': (-0.1, -0.001, 10),
    'time_sffx': (0, -0.013, 9),
    'time_pl': (-0.03, -0.02, 11),
    'bare': (0, 0.005, 11),
}
skip_feat = ['ordinal', 'demonstrative']
for feature in loadings:
    if feature in skip_feat:
        continue
    x_off, y_off, size = offsets.get(feature, (0,0,12)) # config offsets / size
    fx, fy = loadings[feature][:2]
    plt.arrow(0, 0, fx, fy, color='blue', linewidth=1, head_width=0)
    plt.text(fx+x_off, fy+y_off, feature, color='blue', size=size) 

adjust_text(texts) # clean up overlapping annotations

plt.savefig(fg_path('time_head_clusters.png'), dpi=300, bbox_inches='tight') # save plot
plt.title('Time Head Clusters based on Collocating Constructions', size=18) 
plt.show()
# -

# Cf. Fisher's hyper-geometric

# ## Analysis of PCA Results
#
# Two primary groups with a potential third sub-group. The blue arrows show us the features which are influencing the locations on the graph. The two strongest influences are prepositions and zero-marking. These separate words along the x-axis. From the perspective of the x-axis alone, we see mainly two groups, based on colloca 
#
# Note that the presence of constructions indicating definiteness (ה), quantifiers, demonstratives, and plurality pull the largest cluster to the first and second quadrants (`y>0`).  

pca_values[:10]

# ## Prepositional Tendencies
#
# Here we plot the prepositional tendencies of various time words.

# +
prep_cols = times.pivot_table(index='time', columns='preposition', aggfunc='size').fillna(0)
prep_cols = prep_cols.loc[prep_cols.sum(1).sort_values(ascending=False).index]

prep_cols

# +
prep_col_ratio = prep_cols.div(prep_cols.sum(1), axis=0)

prep_col_ratio
# -

prep_col_ratio_pruned = prep_col_ratio[prep_cols.sum(1) > 5] # plot only values with > 10 observations

# +
fig, ax = plt.subplots(figsize=(10,8))
x, y = [prep_col_ratio_pruned.iloc[:,i] for i in (0,1)]
ax.scatter(x, y, color='')
ax.set_xlabel('ø prep')
ax.set_ylabel('prep')

# annotate
texts = []
text_strings = prep_col_ratio_pruned.index
text_strings = [get_display(s).replace('\u05C1','') for s in text_strings]
for i, txt in enumerate(text_strings):
    tx, ty = x[i], y[i]
    texts.append(
        ax.text(
            tx, ty, txt, 
            size=11,
            weight='heavy',
            family='serif',
        )
    )
    
adjust_text(texts) # clean up overlapping annotations

plt.savefig(fg_path('time_prep_ratios.png'), dpi=300, bbox_inches='tight')

plt.show()
# -

# **TODO: switch to a scatter plot [like this](https://analyse-it.com/docs/user-guide/distribution/continuous/dot-plot)**

StarcoderdataJupyterScriptsDedupFiltered