Men Are from Mars, Women Are from Venus: Evaluation and Modelling of Verbal Associations
Ekaterina Vylomova Andrei Shcherbakov Yuriy Philippovich Galina Cherkasova
cs.CL















Ekaterina Vylomova et al. 


Ekaterina Vylomova, Andrei Shcherbakov, Yuriy Philippovich, Galina Cherkasova














The University of Melbourne, Melbourne, Australia,

evylomova@gmail.com & ultrasparc@yandex.ru,
Moscow Polytech, Moscow, Russia,

Y_philippovich@mail.ru
Institute of the Science of Language, Moscow, Russia

gacherk@mail.ru


Men Are from Mars, Women Are from Venus: Evaluation and Modelling of Verbal Associations
    Ekaterina Vylomova1 Andrei Shcherbakov1 Yuriy Philippovich2 Galina Cherkasova3
    December 30, 2023
========================================================================================




We present a quantitative analysis of human word association pairs and study the types of relations presented in the associations. We put our main focus on the correlation between response types and respondent characteristics such as occupation and gender by contrasting  syntagmatic and paradigmatic associations. Finally, we propose a personalised distributed word association model and show the importance of incorporating demographic factors into the models commonly used in natural language processing.  






§ INTRODUCTION


Most of contemporary approaches in natural language processing (NLP) mainly rely on well-annotated and clean textual corpora. For instance, language as well as translation models are typically trained over Europarl<cit.> or the Wall Street Journal corpora. As Eisenstein<cit.> noted, most of such corpora present language used by a very specific social group. For example, Hovy<cit.> showed that the models trained over the Wall Street Journal perform better for old language users. And it becomes extremely troublesome to adapt the models trained over these corpora to new domains such as Twitter. In most cases researchers either normalize the data (for instance, by using string and distributional similarity as in Han<cit.>) or apply various techniques of domain adaptation and knowledge transfer.  
 
Recently several studies in sociolinguistics demonstrated how the NLP models could be improved by considering social factors (see Volkova<cit.>, Stoop<cit.>). This inspired us to exploit associative experiments approach to demonstrate how specialization and gender might affect associations. In this paper we first propose the dataset for associative pairs of Russian native speakers[The dataset is available at <http://github.com/ivri/RusAssoc>] and then show how association types vary across gender and occupation. We also present a simple PPMI-based model of associations and demonstrate the difference in the model's predictions depending on the social characteristics.

The paper is structured as follows. We first discuss previously organized associative experiments, then we introduce the dataset for Russian speakers associations. In Section 4 we analyse how the associations depend on demographic factors and, finally, we present a personalised associative vector model. 




§ RELATED WORK


Introduced by Sir Francis Halton in 1870s, associative experiments became a common approach to study human cognition. Nowadays various researchers organized the experiments on many languages. Most of the experiments present English (American and British) native speakers (see Deese<cit.>, Cramer<cit.>, Kiss<cit.>, Nelson<cit.>). De Groot<cit.> and De Deyne<cit.> conducted them for Dutch, and Shaps<cit.> for Swedish. There are also some for Eastern languages, such as Japanese (see Okamoto & Ishizaki<cit.> and Joyce<cit.>), Korean (Jung <cit.>); and Hebrew (Rubinstein<cit.>) for Semitic group. Lots of research had been done on Slavic languages as well. Novak<cit.> organized the experiment for Czech, Ufimtseva<cit.> presented Slavic Associative Thesaurus comprising of Russian, Belarusian, Bulgarian, and Ukrainian. Finally, Russian thesauri were developed by Leontiev<cit.> and Karaulov<cit.>. The latter one has been conducted in three stages during 1986-1997 and is one of the largest experiments. In addition to associations the dataset also contains demographic information such as age, gender, specialization, and location. 

Most of the previous research had been focused on the study of reactions: their distribution and cross-lingual commonalities. Some of the researchers (e.g. Steyvers & Tenenbaum<cit.>) also studied the structure of human associative networks. They represented a network as a directed graph in which stimuli and reactions correspond to nodes whereas associations are edges connecting them. They showed that the node's degree (the number of different reactions given for a stimulus) follows a power law distribution[i.e. associative networks are scale-free.]. In other words, there are several “hub” nodes with many connections and many “weak” nodes with small degree.

But very little had been done in terms of quantitative evaluation of demographic factors in associations. Current research fills up this gap. We investigate the reaction types distribution in regards to gender and speciality. 




§ DATASET



The experiment conducted by Karaulov's group, although being one of the most lasting ones, very quickly becomes outdated. Moreover, it has only been focused on the regions of Central Russia. To address these issues as well as to analyse the change of the associations over time, we additionally organized the associative experiments in various Russian regions, including Siberia and the Urals. The age of participants ranged from 16 to 26[People in psycholinguistics typically assume that the core of the verbal associations becomes stable and does not significantly change after the age of 18.], most of them were either undergraduate or postgraduate university students of ≈50 specialities.
The experiments were organised as follows. A respondent received a questionnaire of 100 single-word stimuli. For each stimulus the respondent had to provide a reaction. There were no constraints on the reaction types, but the total time was limited to 10-15 minutes, i.e. the participants had 6-9 seconds for each stimulus. Most of the reactions appeared to be also single-word. Several association pairs are presented on Table 1.

In total, the dataset contains 4,997 questionnaires. The list of stimuli comprises of 1,213 various lemmas partially taken from Leontiev's list as well as most common reactions of previous Russian associative experiments from Karaulov<cit.>. The total number of different reactions received from the respondents is 50,359 (37,895 lemmas). Table <ref> shows the top-10 most frequently used reactions. Surprisingly, we see a large overlap between current study and the experiment conducted in 1986-1997. Besides that, there is also an overlap with the most frequently used Russian words from Sharoff's list<cit.>. We also did not observe a significant cross-gender difference in the top reactions.






§ EXPERIMENTS



 §.§ Association types analysis

Aiming to observe possible differences in distribution of association patterns among categories of respondents, we match associations against two major patterns as follows. 

First, we check whether a stimulus - response pair matches an ngram observed in text corpora. We measure a smoothed sum of matched ngram frequencies:


    S = ∑_a ∈ A{[ log f(a),    f > 0,;        0,     f = 0 ]}

where a is an association pair, A is a set of all association pairs, f(a) is a corpus frequency of an ngram produced of a association.

We consider bigrams for single-word responses (the vast majority of cases). If we have two-word response, we match it against trigrams. Responses containing more than two words are treated as non-matching any known ngrams. We tried to match each association both in forward (stimulus→response) and backward (response→stimulus) direction, and each side (stimulus and response) was supplied both as is and in a lemmatized form.[We used mystem<cit.> to extract lemmas.] By doing that, we actually match each association against eight candidate ngrams, and we pick the maximum frequency observed over those ngrams. We used National Corpus of Russian Language  [http://www.ruscorpora.ru/corpora-freq.html] as source for ngram frequencies.




Second, we extract associations that correspond to basic thesaurus relations such as synonyms, antonyms, hypernyms/hyponyms, meronyms/holonyms, and cause/effect. We use Russian WordNet[http://wordnet.ru] <cit.> as our initial data source, where we count the number of matching associations for every relation type. Table 1 presents a list of associations and corresponding extracted ngram frequencies and relations.

We measure the values listed above as percentages to the total number of responses. We have done it for the full dataset and also for slices selected by respondent's gender or specialization.











Figures <ref> and <ref> show that men are more biased towards using semantically inspired associations (paradigmatic) whereas women are more likely to produce ngrams (syntagmatic)[With p-value<0.001]. We observe a similar pattern (i.e. syntagmatic inversely related to paradigmatic) by looking at the specializations. Figure <ref> presents S values for the top-20 most popular specializations. For instance, “chemistry” presents the highest scores in semantic relations whereas lower than average for ngrams. On the other hand, in the case of “sales” it is completely opposite. Note that most of the technical specializations and natural sciences demonstrate high scores for paradigmatic association types. We supposed that this is due to correlation between gender and occupation and the fact that gender still plays a significant role in the process of choosing the future career. In order to test that hypothesis, we calculated ngram usage figures normalized over gender (Fig. <ref>). In this context, a normalized value is a half-sum of two corresponding average values, each one being computed over its respective respondent gender. The normalization didn't seem to smooth differences in ngram usage over various specializations. Therefore, one may conclude that the specialization standalone plays as a significant factor influencing word association patterns.   




 §.§ Personalization of vector models

Now we turn to our experiments with associative vector models. We propose gender and specialization specific associative models. In order to create the models, we first slice the data by the proposed attribute. For instance, for “gender” we take two subsets corresponding to “male” and “female”, respectively.[The dataset is quite balanced and we have roughly the same number of questionnaires for both male and female participants.] As we described earlier, their association frequency distributions present some differences which we would like our model to capture.
Unlike traditional language modelling task, here we only rely on stimulus-reaction pair frequencies. Therefore, we consider SVD-PPMI<cit.> approach to get the distributed vector representations. The method had been shown to perform on par with neural models<cit.>, such as word2vec<cit.>. It is also less expensive in terms of the time complexity and better fits our task setting.[We used the model implementation from <https://bitbucket.org/omerlevy/hyperwords>. We set the size of the context window to 1 (left and right words), embedding size to 100, context distribution smoothing of 0.75, token threshold value of 5, all the other parameters were left with their default values.]  We additionally train a baseline model on the full dataset to compare it to the personalised models. We would like to emphasize that usage of distributed vector models allows us to go beyond the scope of direct associations and generalize better.

Table <ref> presents several examples for the top 10 nearest neighbours of “male”, “female” and baseline models. In this case, we observe mainly semantic differences. Notice a substantial variation in the predictions of the models if we provide them with “I” stimulus. Table <ref> illustrates the models work for “sales” and “publishing” occupation types. In general, we find the ability to provide gender- and specialization-sensitive information useful for addressing the issues related to social language variation. 

Table <ref> additionally provides the difference in the model's predictions for two  locations in Chelyabinsk oblast: a small town of Asha and an industrial city of Magnitogorsk.

There is no consensus in the research community on how to evaluate the associative models. Most of the methods are based on direct comparison of statistical characteristics of the distributions of reactions each of the models generates. Moreover, to our knowledge, no theoretical framework or quality assessment or measures had been proposed for that so far. Therefore, we consider this part of research for our future studies.













§ CONCLUSION


We presented a new dataset for Russian verbal associations. We also showed that social factors such as gender and specialization provide a significant amount of information on the type of association.
Finally, we also proposed a gender-sensitive associative model and demonstrated the significance of incorporating of social factors into the traditional NLP models. 



§ ACKNOWLEDGMENTS

We would like to thank all reviewers for their valuable comments and suggestions for future research directions. The first author was supported by the Melbourne International Research Scholarship (MIRS). 






























