Annotator-Centric Active Learning for Subjective NLP Tasks
Michiel van der Meer Neele Falk Pradeep K. Murukannaiah Enrico Liscio

Galaxies' properties in the Fundamental Plane across time
    M. D'Onofrio
          1
          C. Chiosi1
          
    Received February 5, 2024; accepted ...
============================================================





To accurately capture the variability in human judgments for subjective NLP tasks, incorporating a wide range of perspectives in the annotation process is crucial.
Active Learning (AL) addresses the high costs of collecting human annotations by strategically annotating the most informative samples. We introduce Annotator-Centric Active Learning (ACAL), which incorporates an annotator selection strategy following data sampling. Our objective is two-fold: [label=(*)]
    
  * to efficiently approximate the full diversity of human judgments, and
    
  * to assess model performance using annotator-centric metrics, which emphasize minority perspectives over a majority.

We experiment with multiple annotator selection strategies across seven subjective NLP tasks, employing both traditional and novel, human-centered evaluation metrics. Our findings indicate that ACAL improves data efficiency and excels in annotator-centric performance evaluations. However, its success depends on the availability of a sufficiently large and diverse pool of annotators to sample from.









§ INTRODUCTION

A challenging aspect of natural language understanding (NLU) is the variability of human judgment and interpretation in subjective tasks (e.g., hate speech detection) <cit.>.
While humans can navigate subjectivity naturally, most machine learning methods are insensitive to individual differences <cit.> and underrepresented perspectives <cit.>.

Modern NLU approaches are commonly trained and tested on annotated datasets.
In a subjective task, each data sample is typically labeled by a set of annotators, and differences in annotation are reconciled through aggregation techniques (e.g., majority voting), resulting in a single “gold label” <cit.>.
This approach, though effective for training ML algorithms, neglects the labels of minorities, which becomes problematic, especially, in the case of sensitive subjective tasks.

Subjectivity has been addressed by modeling the full distribution of annotations for each data sample as opposed to aggregating them <cit.>.
However, resources for such approaches are scarce, as most datasets do not (yet) make fine-grained annotation details available <cit.>, and representing a full range of perspectives is contingent on obtaining annotations from a diverse crowd <cit.>.


One way of accounting for a limited annotation budget is to use Active Learning <cit.>. Given a pool of unannotated data samples, AL employs a sample selection strategy to select maximally informative samples for training, retrieving the corresponding annotations from a ground truth oracle (e.g., a single human expert).
However, in subjective tasks there is no such oracle, instead we rely on a set of available annotators. Given this practical constraint, we argue that informativeness for AL manifests in both samples and annotations, as the model should also be guided to reflect the distribution of annotations. Demanding all available annotators to annotate all selected samples would provide a truthful representation of the annotation distribution, but is often unfeasible, especially if the pool of annotators is large. Thus, deciding which annotator(s) should annotate the selected samples is as critical as selecting which samples to annotate.


We introduce Annotator-Centric Active Learning (ACAL) to account for annotation diversity in subjective tasks. In ACAL, the sample selection strategy of traditional AL is followed by an annotator selection strategy as Figure <ref> shows. For each data sample selected through the sample selection strategy, the annotator selection strategy selects an annotator from the available annotators.
We make the following contributions:
[label=(*)]
    
  * We create ACAL, extending the AL approach to optimize for diversity among annotators when learning soft labels in subjective tasks.
    
  * We introduce a suite of annotator-centric evaluation metrics to measure both representativeness and diversity.
    
  * We demonstrate our approach's effectiveness on three diverse datasets with subjective tasks—hate speech detection, moral value classification, and safety judgments.


Our experiments show that ACAL works to better approximate the distribution of human judgments with a lower annotation budget. However, this effectiveness requires a large pool of diverse annotators, as is the case for one of our datasets. In other cases, the differences between ACAL and traditional AL become smaller. Through our annotator-centric evaluation, we show that task agreement and the number of available annotations both influence the effectiveness of ACAL, hinting at a direct trade-off between learning from a majority versus being sensitive to minority annotations.




§ RELATED WORK

We review related works on annotator disagreement and active learning. Our work is novel in combining these fields to [label=(*)]
    
  * represent annotation distributions through soft labels,
    
  * incorporate annotator selection strategies in the active learning loop, and
    
  * evaluate with annotator-centric metrics next to traditional evaluation.




 §.§ Learning with annotator disagreement

Modeling annotator disagreement is garnering increasing attention <cit.>. For instance, some aggregation methods can lead to a fairer representation than simple majority <cit.>. Alternatively, the full annotation distribution can be modeled using soft labels <cit.>.
Other approaches leverage annotator-specific information, e.g., by including individual classification heads per annotator <cit.>, embedding annotator-specific behavior <cit.>, or encoding the annotator's socio-demographic information <cit.>.

Yet, representing annotator diversity remains challenging. Standard calibration metrics under human label variation may be unsuitable, especially when the variation is high <cit.>. Trade-offs ought to be made between collecting more samples or more annotations <cit.>. Further, solely measuring differences among sociodemographic traits is not sufficient to fully capture opinion diversity <cit.>.
To this end, we represent diversity based on which annotators have annotated, what they annotated, and how they have annotated. We experiment with different annotator selection strategies to reveal what aspects impact task performance and annotation budget.




 §.§ Active Learning


AL enables a supervised learning model to achieve high performance with a few training examples if chosen judiciously <cit.>. In a typical AL scenario, a vast collection of unlabeled data is available, and an oracle (e.g., a human expert) can be asked to annotate this unlabeled data. A sampling strategy is employed to iteratively (and smartly) select the next batch of unlabeled data for annotation by the oracle <cit.>.

AL has found widespread application in the field of NLP <cit.>. Two main strategies are employed, either by selecting the unlabeled samples on which the model prediction is most uncertain <cit.>, or by selecting samples that are most representative of the unlabeled dataset <cit.>.

The combination of AL and annotator diversity is a novel direction that has not garnered much attention yet. Existing work proposes to align model uncertainty with annotator uncertainty <cit.>, whereas others adapt annotator-specific classification heads in AL settings <cit.>, or select texts to annotate based on annotator preferences <cit.>.

Existing methods ignore a crucial part of learning with human variation: the diversity among annotators. We focus on which annotators should annotate, such that it best informs us about the underlying label diversity.




§ METHOD

First, we define the soft-label prediction task we use to train a supervised model.
Then, we introduce the traditional AL and the novel ACAL approaches.



 §.§ Soft-Label prediction

Consider a dataset composed of triples (x_i, a_j, y_ij), where x_i is a data sample (i.e., a piece of text) and y_ij∈ C is the class label assigned by annotator a_j. The multiple labels assigned to a sample x_i by the different annotators are usually combined into an aggregated label ŷ_i. For training with soft labels, the aggregation typically takes the form of maximum likelihood estimation <cit.>:

    ŷ_i(x) = ∑^N_i=1[x_i = x][y_ij=c]/∑^N_i=1[x_i = x]


In our experiments, We use a passive learning approach that uses all available {x_i,ŷ_i} to train a model f_θ with cross-entropy loss as a baseline.



 §.§ Active Learning

AL imposes a sampling technique for inputs x_i, such that the most informative sample(s) are picked for learning.
In a typical AL approach, a set of unlabelled data points U is available. At every iteration, a sample selection strategy 𝒮 selects samples x_i ∈ U to be annotated by an oracle 𝒪 that provides the ground truth label distribution ŷ_i. The selected samples and annotations are added to the labeled data D, with which the model f_θ is trained. Alg. <ref> provides an overview of the procedure. In our sample selection strategies, a batch of data of a given size B is queried at each iteration. In our experiments, we compare the following strategies:




    
  Random (𝒮_R) selects a B samples uniformly at random from U.
    
  Uncertainty (𝒮_U) predicts a distribution over class labels with f_θ(x_i) for each x_i ∈ U. Select the B samples with the highest prediction entropy (i.e., the samples on which the model is most uncertain).







 §.§ Annotator-Centric Active Learning


The ACAL approach builds on the AL approach. In contrast to AL, which retrieves an aggregated annotation ŷ_i, ACAL employs an annotator selection strategy 𝒯 to select one annotator and their annotation for each selected data point x_i. Alg. <ref> describes the ACAL approach.







We propose annotator selection strategies that include annotations from diverse annotators. The strategies vary in the type of information used to represent differences between annotators, and include what or how the annotators have annotated thus far.
We test the following strategies:

    
  Random (𝒯_R) selects one random annotator a_j.
    
  Label Minority (𝒯_L) considers only the labels that annotators have assigned. Given a new sample x_i, 𝒯_L selects the available annotator that has the largest bias toward the minority label compared to the other available annotators, i.e., who has annotated other samples with the minority label the most. The minority label is selected as the class with the smallest annotation count in the available dataset D_n thus far.
    
  Semantic Diversity (𝒯_S) considers only information on what each annotator has annotated so far (i.e., the samples that they have annotated). Given a new sample x_i selected through 𝒮, 𝒯_S selects the available annotator for whom x_i is semantically the most different from what the annotator has labeled so far. To measure this difference for an annotator a_j, we employ a sentence embedding model to measure the cosine distance between the embeddings of x_i and embeddings of all the samples annotated by a_j. We then take the average of all semantic similarities. The annotator with the lowest average similarity score is selected.
    
  Representation Diversity (𝒯_D) selects the annotator that has the lowest similarity with the other annotators available for that item. We create a simple representation for each annotator based on the items together with the respective label that they have annotated, followed by computing the pair-wise cosine similarity between all annotators.





§ EXPERIMENTAL SETUP


We describe the experimental setup for the comparisons between ACAL strategies. In all our experiments, we employ a TinyBERT model <cit.> to reduce the number of trainable parameters.
Appendix <ref> includes a detailed overview of the computational setup and hyperparameters. We will provide our codebase upon publication.



 §.§ Datasets



Table <ref> introduces the three datasets that we use, with variation in domain, annotation task (in italics), annotator count, and annotations per instance.

The DICES Corpus <cit.> is composed of 990 conversations with an LLM where 172 annotators provided judgments on whether a generated response can be deemed safe (3-way judgments: yes, no, unsure). We perform a multi-class classification with the scores.

The MFTC Corpus <cit.> is composed of 35K tweets that 23 annotators annotated with any of the 10 moral elements from the Moral Foundation Theory <cit.>. We select the elements of loyalty (lowest annotation count), care (medium count), and betrayal (highest count) and perform three binary classifications to predict the presence of the respective elements. As most tweets were labeled non-moral (i.e., with no moral element), we balanced the datasets by subsampling the non-moral class.

The MHS Corpus <cit.> consists of 50K social media comments on which 8K annotators judged three hate speech aspects—dehumanize (low inter-rater agreement), respect (medium agreement), and genocide (high agreement)—on a 5-point Likert scale. We perform a multi-class classification with the annotated Likert scores for each task.

The datasets and tasks differ in the entropy scores over annotations (Appendix <ref>). DICES and MHS generally have medium normalized entropy scores (most lie between 0.15 < H < 0.85), whereas the MFTC entropy scores are highly polarized.





 §.§ Training procedure


We test the annotator selection strategies proposed in Section <ref> by comparing all possible combinations of the two different sample selection strategies (𝒮_R and 𝒮_U) with the annotator selection strategies (𝒯_R, 𝒯_L, 𝒯_S, and 𝒯_D). At each round, we use 𝒮 to select B unique samples from the unlabeled data pool U. We empirically select B to be the smallest between 5% of the number of available annotations and the number of unique samples in the training set. For each selected sample x_i, we use 𝒯 to select one annotator and retrieve their annotation y_ij.

To populate the annotation history for the annotation selection strategies, we perform a single warmup round with B randomly selected annotations before starting the ACAL iterations <cit.>. We report our training progress results on a validation set with 3-fold cross-validation, showing the average to account for stability across random data splits (into 80% train, 10% validation, and 10% test) and initialization. Then, we select the model iteration that led to the best performance (according to JS) on the validation set and evaluate it using a separate test set.

We compare our work with traditional Oracle-based AL approaches (𝒮_R𝒪 and 𝒮_U𝒪), which use the data sampling strategies but obtain all possible annotations for each sample (following Alg. <ref>). Moreover, we employ a passive learning (PL) approach as an upper bound by training the model on the full dataset, thus observing all available samples and annotations. Our baselines follow the analogous cross-validation setup.





 §.§ Evaluation metrics

The ACAL strategies aim to guide the algorithm to model a representative distribution of the annotator's perspectives while reducing human annotation effort. To this end, we evaluate the model both with a traditional evaluation metric and a metric aimed at comparing predicted and annotated distributions:


  Macro F_1-score (F_1) For each sample in the test set, we select the label predicted by the model with the highest confidence, determine the golden label through a majority agreement aggregation, and compute the resulting macro F_1-score.

  Jensen-Shannon Divergence (JS) The JS measures the divergence between the distribution of label annotation and prediction <cit.>. We report the average JS for the samples in the test set to measure how well the algorithm can model the annotation distribution.

Next, since our proposed annotator selection strategies aim to promote diversity, we introduce novel annotator-centric evaluation metrics. First, we report the average among annotators. Second, in line with Rawls' principle of maximum fairness <cit.>, the result for the worst-off annotators:


  Per-annotator F_1 (F_1^a) We compute the F_1 for each annotator in the test set using their annotations as golden labels, and average it.

  Per-annotator JS (JS^a) We compute the JS for each annotator in the test set using their annotations as target distribution, and average it.

  Worst per-annotator F_1 (F_1^w) We compute the F_1 for each annotator in the test set using their annotations as golden labels, and report the average of the lowest 10% (to mitigate noise).

  Worst per-annotator JS (JS^w) We compute the JS for each annotator in the test set using their annotations as target distribution, and report the average of the lowest 10% (to mitigate noise).


These evaluation metrics allow us to measure the trade-offs between modeling the majority agreement, a representative distribution of annotations, and accounting for minority voices.
We report these metrics on the validation set (as progress over the AL iterations) and test set (by using the best-performing model on the validation set), as described in Section <ref>.





§ RESULTS





 §.§ Test sets results

See Figure <ref> for the performance of the DICES, MFTC, and MHS, respectively. For MFTC, we initially focus on care, since it is the task with neither the most nor least amount of data. For MHS, we start with dehumanize, since it saw the most medium-level disagreement. The rest of the results can be observed in Appendix <ref>.


Combining our results across datasets, we see that data characteristics influence whether ACAL can learn performant models efficiently. In particular, we see that for DICES and MHS, ACAL may learn models that perform well using less data (38% and 62% reduction at best, respectively). Conversely, for MFTC, there is little impact of using ACAL over PL (5.6% less data used). A similar pattern holds when comparing ACAL to AL, though AL seems to be a strong baseline for MHS, where random sample selection leads to more efficient data usage (60%). AL with uncertainty sampling is more efficient for MFTC (13%).

When we compare the performance metrics, we see that the distributions obtained through ACAL are consistently closer to the ground truth distribution in DICES, as measured by JS than PL and AL. However, this pattern is not visible for MFTC and MHS.  In terms of majority-voted F_1, ACAL again leads to better scores in both DICES and MHS.
Since DICES and MHS are datasets with moderate disagreement, we may benefit from using ACAL in such scenarios. Further, if the dataset contains a large number of annotators per sample, annotator selection strategies are shown to pick a more informative set of annotators to learn from.

We highlight some further dataset-specific findings that shed light on the differences between the annotator selection strategies in ACAL. First, in DICES, we see that for three out of four annotator sampling strategies (𝒯_R, 𝒯_D, 𝒯_L), the choice of data sampling strategy has no impact on the performance of the model due to the low number of samples to choose from. Furthermore, only 𝒯_D performs worse in terms of JS, overrepresenting outlier annotators. This hints that selecting annotators based on the average embedding of the annotated content strongly emphasizes diverging label behavior.
Second, MFTC was annotated by a limited fixed set of annotators for whom we can construct a rich annotation history. However, since there are few annotators per sample to pick from, ACAL cannot leverage this information effectively. Again, we see that strategies perform relatively similarly to one another, except for the F_1 scores. Third, in MHS we observe that all strategies using random sample selection require less data.
Since the task has low inter-rater agreement scores, uncertainty-based sampling wrongly attempts to sample annotations for correct high-entropy predictions, while this is an accurate distribution.

Our findings highlight that with many labels per sample, ACAL is more data-efficient than traditional AL and passive learning in terms of overall evaluation. However, when data characteristics differ and few annotations are available per sample, ACAL has less of an impact. Polarized agreement scores (either high agreement or no agreement) make the use of ACAL and AL cause little to no improvements over passive learning. This corroborates that (AC)AL leads to improvements in specific cases <cit.>. Furthermore, we found conflicting results depending on the metric used (JS and F_1). We closely examine the relationship between the evaluation metrics by turning to annotator-centric evaluation, observing how ACAL impacted predictions for individual annotators.




 §.§ Annotator-centric evaluation

We show the annotator-centric evaluation metrics in Tables <ref>, <ref>, and <ref> for DICES, MFTC (care) and MHS (dehumanize), respectively. We again describe per-dataset results.

Again, for DICES and MHS, we observe a positive effect of using ACAL over PL, both in terms of data efficiency and final annotator-centric behavior. For these datasets, ACAL leads to a better representation of annotators on average (JS^a,F_1^a), as well as a better representation of the 10% most different annotators (JS^w, F_1^w). Compared ACAL to AL, we mainly observe improvements in the DICES dataset, showing less data used and a better annotator-centric F_1 score. We observe a strong JS^w for the 𝒯_D strategy and worse JS_a, corroborating our earlier finding that emphasizing diverging label behavior trades off with the averaged evaluation scores. Interestingly, this is not apparent in the F_1 scores.
For MHS, all approaches using random data sampling (𝒮_R) require considerably less data than passive learning. Further, since the pool of annotators for MHS is large (7K+), there will always be some annotators in disagreement with the output of our models, leading to a zero score on F_1^w.













 §.§ Training plots

While the evaluation shows a pattern of efficient data use with ACAL under certain data conditions, it reveals little about how the metrics behave during training or how individual annotator strategies behave. To this end, we provide a complete overview of all metrics (as computed on the validation set) during training in App. <ref>. Here we describe the major patterns reoccurring across our experiments using examples and show six of particular interest (Figure <ref>). Since the strategies only differ in what annotations are included during training, we only show plots related to the annotator-centric metrics.

We can see that there is an influence of both the data sampling and annotator strategies on the performance of the models. Only on DICES is the choice of 𝒮 irrelevant, probably due to the low number of samples. Specifically 𝒯_D deteriorates slower for the worst-off annotators than the other strategies but does so without being able to uphold a competitive F_1^a score.
In MFTC, we see that when using 𝒮_U, performance on F_1^a dips at the start of training. Selecting annotators for samples with high predicted entropy initially leads to a decrease in average performance. The strategy seeks to first lower the entropy for the labels already encountered, though some of the variation in labels is irreconcilable. A similar reasoning holds for MHS, where the differences between strategies are even less impacted by the choice of 𝒯. These two plots further underline our main finding that for ACAL to be impactful in representing diverse annotation perspectives, we need to ensure a [label=(*)]
    
  * heterogeneous pool of annotators, and
    
  * a task that facilitates human label variation.









 §.§ Change in task

In Fig.  <ref>, we present a comparative analysis of two annotator-centric metrics across the three distinct tasks of MFTC and MHS,
 for which we have seen the least impact of ACAL over AL and PL.
We cannot conclude that the chosen ACAL approach (𝒮_R𝒯_S) offers a consistent improvement over sampling all annotations (𝒮_R𝒪), particularly given that the models using ACAL occasionally require more data to converge (Tables <ref> to <ref>).

Initially, we hypothesized that tasks with a high degree of subjectivity would benefit from ACAL strategies, especially on metrics focused on the most marginalized (worst-off) annotators. These strategies typically involve selecting an annotator whose patterns of annotation diverge from the majority, either in terms of their annotation behavior or in the semantic content of their past annotations.

However, as depicted in Figure  <ref>, when examining the task of dehumanize (high entropy), it becomes apparent that ACAL does not consistently outperform AL. ACAL demonstrates a lower F_1^a-score than AL for this task, and on the other hand, a higher F_1^a-score for a task that is less subjective, such as genocide. Similarly, when evaluating loyalty, which involves the moral dimension with the highest disagreement among annotators, the lower 10% of annotators are more accurately approximated with PL.
This suggests that the effectiveness of annotation strategies varies depending on the task's degree of subjectivity and available pool of annotators. The more heterogeneous the annotation behavior, indicative of a highly subjective task, the larger the pool of annotators required for each item selection. However, due to the limited annotations available per item in both datasets MFTC and MHS, even carefully selecting specific annotators may not adequately represent divergent annotation behavior in general, which challenges the generalization to unseen data.
Finally, we can observe that there is a trade-off between modeling the majority of annotators equally, as reflected in the F_1^a-score and prioritizing the minority viewpoint (JS^w). A better performance in one aspect does not necessarily guarantee superiority in the other.




§ CONCLUSION

We introduce Annotator-Centric Active Learning (ACAL), an active learning approach that incorporates annotator selection strategies aimed at capturing label variation among annotators. We experiment with tasks across three different datasets, each leading to different ACAL behaviors. One of these datasets, DICES, is the most realistic application of ACAL since the pool of possible annotators is the largest. Here, ACAL leads to more diverse label distributions using fewer annotations. However, we find that the effectiveness of the ACAL paradigm is contingent on data characteristics. These characteristics include the number of annotations per sample, the number of annotations per annotator, and the nature of disagreement in the task annotations. Our analysis shows that we can use these conditions to help explain the often disappointing results for AL in NLP applications.

Including annotator-centric evaluation reveals how methods with similar averaged performance deal with different levels of disagreement among annotators. We show that evaluation can be enhanced by focusing on individual annotators, as there is a large gap between conventional, averaged, and worst-off performance. Furthermore, many aspects of our ACAL approach can be experimented with, e.g. by swapping the order in which samples are selected (in our case first) and annotators (second), or investigating the impact of including annotator-specific demographic information, as it is inconsistently predictive of annotation behavior <cit.>.





§ LIMITATIONS

The main limitation of this work is that the experiments are based on simulated active learning which is known to bear potential issues <cit.>. In our study, a primary challenge arises with two of the datasets (MFTC, MHS), which, despite having a large pool of annotators, lack annotations from every annotator for each item. Consequently, in real-world scenarios, the annotator selection strategies for these datasets would benefit from access to a more extensive pool of annotators. This limitation likely contributes to the underperformance of ACAL on these datasets compared to DICES. We emphasize the need for more datasets that feature a greater number of annotations per item, as this would significantly enhance research efforts aimed at modeling human disagreement.

Since we evaluate four different annotator selection strategies and two sample selection strategies across three datasets and seven tasks, the amount of experiments is high. This did not allow for further investigation of the difference using different classification models, the extensive turning of hyperparameters, or even different training paradigms.
Lastly, a limitation of our annotator selection strategies is that they rely on a small annotation history. This is why we require a warmup phase for some of the strategies, for which we decided to take a random sample of annotations. Incorporating more informed warmup strategies or incorporating ACAL strategies that do not rely on annotator history may positively impact its performance and data efficiency.




§ ETHICAL CONSIDERATIONS

Our goal is to approximate a good representation of human judgments over subjective tasks. We want to highlight the fact that the performance of the models differs a lot depending on which metric is used. We tried to account for a less majority-focussed view when evaluating the models which is very important, especially for more human-centered applications, such as hate-speech detection. However, the evaluation metrics we use do not fully capture the diversity of human judgments. The selection of metrics should align with the specific goals and motivations of the application, and there is a pressing need to develop more metrics to accurately reflect human variability in these tasks.

Our experiments are conducted on English datasets due to the scarcity of unaggregated datasets in other languages. In principle, ACAL can be applied to other languages (given the availability of multilingual models to semantically embed textual items for some particular strategies used in this work). We encourage the community to enrich the dataset landscape by incorporating more perspective-oriented datasets in various languages, ACAL potentially offers a more efficient method for creating such datasets in real-world scenarios.


acl_natbib





§ DETAILED EXPERIMENTAL SETUP




 §.§ Cross validation details

We split the data on samples, meaning that all annotations for any given sample are completely contained in each separate split.



 §.§ Hyperparameters

We report the hyperparameters for training passive, AL, and ACAL in Tables <ref>, <ref>, and <ref>, respectively. For turning the learning rate for passive learning, on each dataset, we started with a learning rate of 1e-06 and increased it by a factor of 3 in steps until the model showed a tendency to overfit quickly (within a single epoch). All other parameters are kept on their default setting.












 §.§ Training details


Experiments were largely run between January and April 2024. Obtaining the ACAL results for a single run takes up to an hour on a Nvidia RTX4070. For large-scale computation, our experiments were run on a cluster with heterogeneous computing infrastructure, including RTX2080 Ti, A100, and Tesla T4 GPUs.

Obtaining the results of all experiments required a total of 231 training runs, combining:
[label=(*)]
    
  * two data sampling strategies,
    
  * four annotator sampling strategies, plus an additional Oracle-based AL approach,
    
  * a passive learning approach.

Each of the above were run for [label=(*)]
    
  * three folds, each with a different seed, and
    
  * the seven tasks across three datasets.

For training all our models, we employ the AdamW optimizer <cit.>. Our code is based on the Huggingface library <cit.>, unmodified values are taken from their defaults.



 §.§ ACAL Annotator Strategy details

Some of the strategies used for selecting annotators to provide a label to a sample

    
  𝒯_S uses a sentence embedding model to represent the content that an annotator has annotated. We use [<https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2>]. We select annotators that have not annotated yet (empty history) before picking from those with a history to prioritize filling the annotation history for each annotator.
    
  𝒯_L creates an average embedding for the content annotated by each annotator and selects the most different annotator. We use the same sentence embedding model as 𝒯_S. To avoid overfitting, we perform PCA and retain only the top 10 most informative principal components for representing each annotator.





 §.§ Disagreement rates


We report the average disagreement rates per dataset and task in Figure <ref>, for each of the dataset and task combinations.






§ DETAILED RESULTS OVERVIEW




 §.§ Test set evaluation other MFTC and MHS tasks

See Table <ref> for the trade-off between data efficiency and test-set performance for the two conventional metrics (JS and F_1). We include copy the earlier mentioned results for MFTC (care) and MHS (dehumanize) for convenience.






 §.§ Annotator-Centric evaluation for other MFTC and MHS tasks

We show the full annotator-centric metrics results for MFTC betrayal (Table <ref>), MFTC loyalty (Table <ref>), MHS genocide (Table <ref>), and MHS respect (Table <ref>).














 §.§ Training process


In our main paper, we report a condensed version of all metrics during the training phase of the active learning approaches. Below, we provide a complete overview of all approaches over all metrics. The results can be seen in Figures <ref> through <ref>.




























