--- abstract: 'Linguistic style is an integral component of language. Recent advances in the development of style representations have increasingly used training objectives from *authorship verification (AV)*: Do two texts have the same author? The assumption underlying the AV training task (same author approximates same writing style) enables self-supervised and, thus, extensive training. However, a good performance on the AV task does not ensure good "general-purpose" style representations. For example, as the same author might typically write about certain topics, representations trained on AV might also encode content information instead of style alone. We introduce a variation of the AV training task that controls for content using conversation or domain labels. We evaluate whether known style dimensions are represented and preferred over content information through an original variation to the recently proposed `STEL` framework. We find that representations trained by controlling for conversation are better than representations trained with domain or no content control at representing style independent from content.' author: - | Anna Wegmann, Marijn Schraagen and Dong Nguyen\ Department of Information and Computing Sciences\ Utrecht University\ Utrecht, the Netherlands\ `a.m.wegmann, m.p.schraagen, wreynolds@example.com`\ bibliography: - 'anthology.bib' - 'custom.bib' title: ' Same Author or Just Same Topic? Towards Content-Independent Style Representations ' --- =1 Introduction ============ Linguistic style (i.e., how something is said) is an integral part of natural language. Style is relevant for natural language understanding and generation @nguyen-etal-2021-learning [@ficler-goldberg-2017-controlling] as well as the stylometric analysis of texts @el2014authorship [@Goswami_Sarkar_Rustagi_2009]. Applications include author profiling @10.1145/1871985.1871993 and style preservation in machine translation systems @niu-etal-2017-study [@rabinovich-etal-2017-personalized]. While authors are theoretically able to talk about any topic and (un-)consciously choose to use many styles (e.g., designed to fit an audience @bell-audience-design), it is typically assumed that there are combinations of style features that are distinctive for an author (sometimes called an author's <span>idiolect</span>). Based on this assumption, the *authorship verification* task (AV) aims to predict whether two texts have been written by the same author @coulthard2004author [@neal_aa-survey; @utility_content_AA]. Recently, training objectives based on the AV task have been used to train style representations @AV_SimilarityLearning [@hay-etal-2020-representation; @zhu-jurgens-2021-idiosyncratic]. Training objectives on AV are especially promising because they do not require any additional labeling when author identifiers are available. Similar to the distributional hypothesis, the assumption underlying the AV training task (same author approximates same writing style) enables extensive self-supervised learning. -- ------- ------------------------ -- $A_1$ <span>(A\_end);</span> -- ------- ------------------------ -- --------------------------------- ------------------------------------------------------------------------------- ------------------------- ----------------------------------- -- ------------------------- $A_2$ <span>(SA\_start);</span> <span>(SA\_top);</span> it's clear that these are wildly different situations <span>(SA\_end);</span> $B$ <span>(DA\_start);</span> <span>(DA\_end);</span> --------------------------------- ------------------------------------------------------------------------------- ------------------------- ----------------------------------- -- ------------------------- \[overlay, remember picture\] ($(DA_start.west) - (-0.1,0.3)$) –node\[below=14pt\]<span> CC - Same Topic as $A_1$</span> ($(DA_end.east) - (0.4,1)$); ($(SA_start.west) - (-0.1,0.45)$) –node\[below=14pt\]<span> Same Author as $A_1$</span> ($(SA_end.east) - (0.2,1)$); Style and content are often correlated @gero-etal-2019-low [@bischoff2020importance]: For example, people might write more formally about their professional career but more informally about personal hobbies. As a result, style representations might encode spurious content correlations @poliak-etal-2018-hypothesis, especially when their AV training objective does <span>not control for content</span> @Halvani_AV_bad-topic-control [@sundararajan-woodard-2018-represents]. Current style representation learning methods either use no or only limited control for content @hay-etal-2020-representation or use domain labels to approximate topic @AV_SimLearning_Attention. work with 24 domain labels (here: product categories) for more than 100k `Amazon` reviews to improve generalizability. However, using a small set of labels might be too coarse-grained to fully represent and thus control for content. In this paper, we use "content" and "topic" to refer to different concepts. We assume same content (fulfilled if two utterances are paraphrases of each other) implies same topic (e.g., two utterances that discuss personal hobbies), while same topic does not necessarily imply same content. **Approach.** We introduce two independent variations to the AV task (see Figure \[fig:Task\]): adding a contrastive sentence (CAV setup) and addressing content correlation with a topic proxy (CC). We train several siamese BERT-based neural networks @reimers-gurevych-2019-sentence to compare style representations learned with the new variations to the AV task. We train on utterances from the platform `Reddit` but our approach could be applied to any other conversation dataset as well. While previous work mostly aimed for learning representations that represent an author's individual style @AV_SimilarityLearning [@hay-etal-2020-representation; @zhu-jurgens-2021-idiosyncratic], we aim for general-purpose style representations. As a result, we evaluate the generated representations on (a) whether known style dimensions (e.g. formal vs. informal) are present in the embedding space (Section \[sec:eval-STEL\]) and preferred over content information (Section \[sec:eval-t-a-STEL\]) and (b) whether sentences written by the same author are closer to each other even when they have different content (Section \[sec:eval-TT\]). **Contribution.** With this paper, we (a) contribute an extension of the AV task that aims to control for content (CC) with conversation labels, (b) introduce a novel variation of the AV setup by adding a contrastive utterance (CAV setup), (c) compare style representations trained with different levels of content control (CC) on two task setups (AV and CAV), (d) introduce a variation of the `STEL` framework @wegmann-nguyen-2021-capture to evaluate whether representations prefer content over style information and (e) demonstrate found stylistic features via agglomerative clustering. We find that representations trained on the conversation topic proxy are better than representations trained with domain or no content control at representing style independent from content. Additionally, combining the conversation topic proxy with the CAV setup leads to better results than combining it with the AV setup. We show that our representations are sensitive to stylistic features like punctuation and apostrophe types such as ' vs. using agglomerative clustering. We hope to further the development of content-controlled style representations. Our code and data are available on GitHub.[^1] Related Work ============ Recently, deep learning approaches have been used in authorship verification @shrestha-etal-2017-convolutional [@AV_CNN; @AV_SimLearning_Attention; @AI_SimilarityLearning_Siamese; @hay-etal-2020-representation; @Hu_AA-Style-Embedding_Triplet; @zhu-jurgens-2021-idiosyncratic]. Training on transformer architectures like BERT has been shown to be competitive with other neural as well as non-neural approaches in AV and style representation @zhu-jurgens-2021-idiosyncratic [@wegmann-nguyen-2021-capture]. AV methods have controlled for content by restricting the feature space to contain "content-independent" features like function words or character n-grams @neal_aa-survey [@masking-topic_AA; @sundararajan-woodard-2018-represents]. However, even these features have been shown to not necessarily be content-independent @litvinova_context-stylometric. Semantic sentence embeddings are typically trained using supervised or self-supervised learning @reimers-gurevych-2019-sentence. For supervised learning, models are often trained on manually labelled natural language inference datasets @conneau-etal-2017-supervised. For self-supervised learning, *contrastive* learning objectives @contrastive-learning have been increasingly used. Contrastive objectives push semantically distant sentence pairs apart and pull semantically close sentence pairs together. Different strategies for selecting sentence pairs have been used, e.g., same sentences as semantically close vs. randomly sampled as semantically distant sentences @giorgi-etal-2021-declutr [@gao-etal-2021-simcse]. also experiment with a *triplet loss*, which pushes an anchor closer to a semantically close sentence and pulls the same anchor apart from a semantically distant sentence. Semantic representations are typically first evaluated on the task that they have been trained on, e.g., binary tasks for binary contrastive objectives and triplet tasks (similar to Figure \[fig:Task\]) for triplet objectives @reimers-gurevych-2019-sentence. Semantic representations are often also evaluated on the STS benchmark @cer-etal-2017-semeval or semantic downstream tasks like semantic search, NLI @bowman-etal-2015-large [@williams-etal-2018-broad] or SentEval @conneau-kiela-2018-senteval. Typically, objective functions that are known from semantic embedding learning have been used @hay-etal-2020-representation [@zhu-jurgens-2021-idiosyncratic] with AV training tasks to learn style representations. address possible spurious correlations by sampling half of the different and same author utterances from the same and the other half from different domains (e.g., subreddits for `Reddit`). Style representations are often trained and evaluated on the AV task @AV_SimLearning_Attention [@zhu-jurgens-2021-idiosyncratic; @bischoff2020importance]. Style Representation Learning ============================= We describe the new Contrastive Authorship Verification setup (CAV) and our approach to content control (CC) in Section \[sec:task\]. Then we describe the generation of training tasks (Section \[sec:task-generation\]) and the hyperparameters for model training (Section \[sec:models\]). Training Task {#sec:task} ------------- The authorship verification (AV) task is the task of predicting whether two texts are written by the same or different authors. In the following, we introduce two independent variations to the AV task: Adding (1) contrastive information with the CAV setup and (2) content control via topic proxies. **CAV setup.** We introduce an adaption of the Authorship Verification task — the Contrastive Authorship Verification setup (CAV, Figure \[fig:Task\]): Given an anchor utterance $A_1$ and two other utterances $A_2$ and $B$, the task is to identify which of the two sentences were written by the same author as $A_1$. Using a contrastive AV setup adds learnable information to the task (namely the contrast between $A_2$ and $B$ w.r.t. $A_1$) and enables the use of learning objectives that require three input sentences and have been successful in semantic embedding learning @reimers-gurevych-2019-sentence. We experiment with both CAV and AV setups for style representation learning. In the future, it is also possible to adapt this setup to include several instead of just one contrastive "negative" different author utterance (similar to contrastive semantic learning, e.g., in ). One task with the CAV setup, which consists of three utterances ($A_1$, $A_2$, $B$), can be split up into two AV tasks: ($A_1$, $A_2$) and ($A_1$, $B$). We compare the CAV and AV setups during evaluation (Section \[sec:eval\]). **Content Control (CC).** Models optimized for AV have been known to make use of semantic information @sari-etal-2018-topic [@sundararajan-woodard-2018-represents; @Stamatatos_av-w-topic] and to perform badly in cross-topic settings @Halvani_AV_bad-topic-control [@bischoff2020importance]. Recent studies use AV tasks to train style representations and address possible correlations by controlling for domain @zhu-jurgens-2021-idiosyncratic [@AV_SimilarityLearning]. However, it is unclear to what extent these domain labels are better (or worse) than other ways of controlling for content. We compare three different levels of content control by approximating content with the help of a topic proxy. We sample the utterance pairs written by different authors ($B$ and $A_1$ for CAV, c.f. Figure \[fig:Task\]) (i) from the same *conversation*, (ii) from the same *domain* (e.g., subreddit for `Reddit` as in ) or (iii) *randomly* (as a baseline, similar to ). Our newly proposed use of the same conversation "topic proxy" is inspired by semantic sentence representation learning, where conversations have previously been used as a proxy for semantic information encoded in utterances @yang-etal-2018-learning [@liu-etal-2021-dialoguecse]. We test to what extent the three different topic proxies are contributing to content-independent style representations during evaluation (Section \[sec:eval-t-a-STEL\]). \[t\] -------------- ---------------- ------------- ------------- ------------- ------------- ----- -------- -------- -------- -------- **CC level** **Data Split** \# AV \# CAV \# \# ma co do co do train set $420{,}000$ $210{,}000$ $546{,}757$ $194{,}836$ $9$ $0.27$ $0.56$ $1.00$ $1.00$ dev set $90{,}000$ $45{,}000$ $116{,}451$ $41{,}848$ $8$ $0.26$ $0.55$ $1.00$ $1.00$ test set $90{,}000$ $45{,}000$ $116{,}621$ $41{,}902$ $8$ $0.27$ $0.55$ $1.00$ $1.00$ train set $420{,}000$ $210{,}000$ $544{,}587$ $240{,}065$ $9$ $0.01$ $1.00$ dev set $90{,}000$ $45{,}000$ $116{,}490$ $50{,}939$ $8$ $0.02$ $1.00$ test set $90{,}000$ $45{,}000$ $116{,}586$ $51{,}182$ $8$ $0.02$ $1.00$ train set $420{,}000$ $210{,}000$ $548{,}082$ $270{,}079$ $9$ $0.00$ $0.01$ dev set $90{,}000$ $45{,}000$ $117{,}149$ $57{,}352$ $8$ $0.00$ $0.01$ test set $90{,}000$ $45{,}000$ $117{,}434$ $57{,}726$ $8$ $0.00$ $0.02$ -------------- ---------------- ------------- ------------- ------------- ------------- ----- -------- -------- -------- -------- Task Generation {#sec:task-generation} --------------- We use a 2018 `Reddit` sample with utterances from 100 active subreddits[^2] extracted via `ConvoKit` @chang-etal-2020-convokit[^3]. Per subreddit, we sample 600 conversations with at least 10 posts (which we call utterances). All subreddits are directed at an English audience, which we infer from the subreddit descriptions. **Generation.** We removed all invalid utterances[^4]. Then, we split the set of authors into a non-overlapping 70% train, 15% development and 15% test author split. For each CC level (conversation, domain, no) and each author split, we generated a set of training tasks, i.e., nine sets in total (see Table \[table:split-statistics\]). First, we generated the tasks for the train split of the dataset with conversation content control. We sampled 210k distinct utterances $A_1$ from the train author split. We use a weighted sampling process to not overrepresent authors that wrote more utterances than others. The maximum time one author wrote $A_1$ is 9 (c.f. "ma" in Table \[table:split-statistics\]). Then, for each utterance $A_1$, we randomly sampled an utterance $B$ that was part of the same conversation as $A_1$ but written by a different author. Then, for all 210k ($A_1$, $B$)-pairs, an utterance $A_2$ was sampled randomly from all utterances written by the same author as $A_1$ and for which $A_1 \neq A_2$ holds. We equivalently sampled 45k tasks for the dev and test. For the domain and no CC level, we reuse $A_1$ and $A_2$, to keep as many correlating variables constant as possible. Thus, we only resampled 210k utterances $B$ written by a different author from $A_1$ by sampling from the same domain or randomly. We make sure that each combination of ($A_1$, $A_2$, $B$) occurs only once. Thus there are no repeating CAV tasks.[^5] However, it is possible that some utterances occur more than once across tasks. In total, we generate 210k train, 45k dev and 45k test tasks for each CC level (see Table \[table:split-statistics\]), corresponding to a total of 420k, 90k and 90k AV-pairs when splitting the CAV task into (A, SA) and (A, DA) pairs (c.f. Section \[sec:task\]). \[th\] ----------- ------------------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- **Setup** **CC level** AUC $\pm\sigma$ AUC $\pm\sigma$ AUC $\pm\sigma$ acc $\pm\sigma$ acc $\pm\sigma$ acc $\pm\sigma$ $.53$ $.57$ $.61$ $.53$ $.58$ $.63$ <span>**Conversation**</span> $\mathbf{.69}\pm.02$ ${.70}\pm.02$ $.71\pm.02$ $\mathbf{.68}\pm.02$ ${.69}\pm.02$ $.70\pm.02$ <span>**Domain**</span> ${.68}\pm.01$ $\mathbf{.71}\pm.01$ ${.73}\pm.02$ ${.67}\pm.01$ $\mathbf{.70}\pm.01$ ${.73}\pm.00$ <span>**No**</span> $.58\pm.01$ $.63\pm.02$ $\mathbf{.79}\pm.00$ $.59\pm.01$ $.66\pm.01$ $\mathbf{.78}\pm.00$ <span>**Conversation**</span> $\mathbf{.69}\pm.00$ ${.70}\pm.00$ ${.71}\pm.00$ $\mathbf{.68}\pm.00$ ${.69}\pm.00$ ${.70}\pm.00$ <span>**Domain**</span> ${.68}\pm.00$ ${.70}\pm.00$ ${.72}\pm.00$ $\mathbf{.68}\pm.00$ $\mathbf{.70}\pm.00$ ${.72}\pm.01$ <span>**No**</span> $.58\pm.00$ $.63\pm.03$ $.77\pm.00$ $.59\pm.00$ $.65\pm.00$ ${.77}\pm.00$ ----------- ------------------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- ---------------------- Training {#sec:models} -------- We use the `Sentence-Transformers`[^6] python library @reimers-gurevych-2019-sentence[^7] to fine-tune several siamese networks based on (1) 'bert-base-uncased', (2) 'bert-base-cased' @devlin-etal-2019-bert and (3) 'roberta-base' @liu2019roberta. We expect those to perform well based on previous work @zhu-jurgens-2021-idiosyncratic [@wegmann-nguyen-2021-capture]. We compare using (a) contrastive loss @contrastive-learning with the AV setup (Section \[sec:task\]) tasks and (b) triplet loss @reimers-gurevych-2019-sentence with the CAV setup (Figure \[fig:Task\]). The binary contrastive loss function uses a pair of sentences as input while the triplet loss expects three input sentences. For the loss functions, we experiment with three different values for the margin hyperparameter (i) 0.4, (ii) 0.5, (iii) 0.6. We train with a batch size of 8 over 4 epochs using 10% of the training data as warm-up steps. We use the Adam optimizer with the default learning rate (0.00002). We leave all other parameters as default. We use the BinaryClassificationEvaluator on the AV setup with contrastive loss and the TripletEvaluator on the CAV setup with triplet loss from `Sentence-Transformers` to select the best model out of the 4 epochs. The BinaryClassificationEvaluator calculates the accuracy of identifying similar and dissimilar sentences, while the TripletEvaluator checks if the distance between A and SA is smaller than the distance between A and DA. We use cosine distance as the distance function. Evaluation {#sec:eval} ========== We evaluate the learned style representations on the Authorship Verification task (i.e., the training task) in Section \[sec:eval-TT\]. Then, we evaluate whether models learn to represent known style dimensions via the performance on the `STEL` framework @wegmann-nguyen-2021-capture in Section \[sec:eval-STEL\]. Last, we evaluate representations on their content-independence with an original manipulation of `STEL` (Section \[sec:eval-t-a-STEL\]). Authorship Verification {#sec:eval-TT} ----------------------- We display the AV and CAV performance of trained models in Table \[fig:test-task-results\]. On the development sets, RoBERTa models consistently outperformed the cased and uncased BERT models. Also, different margin values only led to small performance differences (Appendix \[sec:app\_dev-results\]). Consequently, in Table \[fig:test-task-results\], we only display the performance of the six fine-tuned RoBERTa models on the test sets using the three different content controls (CC) and two different task setups (AV and CAV setups) with constant margin values of 0.5. AV performance is usually calculated with either (i) AUC or (ii) accuracy using a predetermined threshold @zhu-jurgens-2021-idiosyncratic [@KestemontEtAl:CLEF-2021]. We use cosine similarity to calculate the similarity between sentence representations. Thus, there is no clear constant default threshold to decide between same and different author utterances. A threshold could be fine-tuned on the development set, however for simplicity we use AUC to calculate AV performance instead.We use accuracy for the CAV task — here no threshold is necessary (cosine similarity is calculated between $A_1$, $A_2$ and $A_1$, $B$ and the highest similarity utterance is chosen). This makes the performance scores on the test sets less comparable across setups – however, comparability of the CAV and AV performance scores are limited in any case as the AV vs. CAV setups are fundamentally different. Performance scores can be compared across the same column, i.e., within the same AV and CAV setup. We aggregate performance with mean and standard deviation for three different random seeds per model parameter combination.[^8] Overall, the AV & CAV training task setup (rows in Table \[fig:test-task-results\]) lead to similar performance on the test sets. As a result, we do not distinguish between them in this section's discussion. Generally, the representations tested on the CC level they were trained on (diagonal) outperform other models that were not trained with the same CC level. For example, representations trained with the conversation CC level, perform better on the test set with the conversation CC than representations trained with the domain or no CC. **Tasks with the conversation label are hardest to solve.** For all models, the performance is lowest on the conversation test set and increases on the domain and further on the random test set. This is in line with our assumption that the conversation test set has semantically closer different author utterance ($A_1$, $B$)-pairs that make the AV task harder due to reduced spurious content cues (Section \[sec:task\]). **Representations trained with the conversation CC might encode less content information.** The average performance across the three CC levels is slightly higher for the models trained with domain than conversation CC level and lowest for no CC. Across the three test sets with the different CC levels, the standard deviation in performance is biggest for models trained without CC and smallest for models trained with the conversation CC. Representations trained with domain or no CC might latch on to more semantic features because they are more helpful on the no and domain CC test sets. Models learned with the conversation CC might in turn learn more content-agnostic representations. Overall, a representation that performs well on the AV task alone might do so by latching on to content (not style) information. As a result, a good AV performance alone might not be indicative of a good representation of style. We further evaluate the quality of style representations and their content-independence in Sections \[sec:eval-STEL\] and \[sec:eval-t-a-STEL\]. \[t\] -- -------------------- ------- ---------------- ---------------------- ---------------------- ------------------ ---------------------- ----------------- ----------------- ----------------- ---------------------- o o-c o o-c o o-c o o-c acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ acc$\pm\sigma$ **.80** .05 .83 .09 **.73** .01 **.94** **.13** **1.0** .00 <span>**c**</span> .71 ${.35}$ ${.83}\pm.02$ ${.64}\pm.00$ $.57\pm.02$ ${.13}\pm.04$ $.61\pm.02$ $.04\pm.01$ ${.91}\pm.10$ ${.00}\pm.01$ <span>**d**</span> $.73$ $.28$ ${.84}\pm.01$ ${.56}\pm.04$ ${.69}\pm.05$ $.05\pm.02$ $.61\pm.02$ $.03\pm.02$ ${.98}\pm.03$ $.00\pm.00$ <span>**n**</span> $.72$ $.22$ $\mathbf{.85}\pm.01$ ${.46}\pm.04$ $.57\pm.01$ $.03\pm.01$ $.62\pm.04$ $.05\pm.02$ ${.98}\pm.01$ $.00\pm.00$ <span>**c**</span> $.71$ $\mathbf{.42}$ ${.81}\pm.02$ $\mathbf{.69}\pm.02$ ${.59}\pm.01$ $\mathbf{.24}\pm.02$ ${.65}\pm.09$ ${.03}\pm.01$ ${.99}\pm.02$ $\mathbf{.04}\pm.02$ <span>**d**</span> $.71$ $.32$ ${.82}\pm.01$ ${.61}\pm.02$ $.57\pm.01$ $.12\pm.01$ $.64\pm.05$ $.03\pm.01$ ${.99}\pm.01$ ${.01}\pm.01$ <span>**n**</span> $.71$ $.24$ $\mathbf{.85}\pm.00$ $.50\pm.02$ ${.56}\pm.01$ ${.04}\pm.01$ ${.59}\pm.03$ ${.06}\pm.01$ ${.98}\pm.04$ ${.00}\pm.00$ -- -------------------- ------- ---------------- ---------------------- ---------------------- ------------------ ---------------------- ----------------- ----------------- ----------------- ---------------------- <span>p<span>45pt</span> p<span>60pt</span> p<span>1pt</span> p<span>60pt</span></span> & 1 & & 2\ Anchor (A) & r u a fan of them or something? & &\ & &\ Sentence (S) & <span>(A2\_top);</span> & & Oh yea and that young dr got a bad haircut\ \[overlay, remember picture\] ($(A2_end.east) - (1.6,0.2)$) – ($(A2_top.east) - (-2.35,-0.3)$) ; STEL Task {#sec:eval-STEL} --------- We calculate the performance of the representations on the `STEL` framework @wegmann-nguyen-2021-capture[^9]. Here, models are evaluated on whether they are able to measure differences in style across four known dimensions of style (formal vs. informal style, complex vs. simple style, contraction usage and number substitution usage). Models are tested on 1830 tasks of the same setup: Two "sentences" S1 and S2 have to be matched to the style of two given "anchor" sentences A1 and A2. The task is binary. Sentences can either be matched without reordering (A1-S1 & A2-S2) or with reordering (A1-S2 & A2-S1). For example, consider the sentences in Figure \[tab:t-a-STEL\] before alterations. The correct solution to the task is to reorder the sentences, i.e., to match A1 with S2 because they both exhibit a more informal style and A2 with S1 because they both exhibit a more formal style. The `STEL` sentence pairs (S1, S2) and (A1, A2) are always paraphrases of each other (in contrast to $A_1$ and $B$ for the AV task which are only chosen to be about the same approximate topic, c.f. \[sec:task\]). The anchor pairs and sentence pairs are randomly matched and are thus otherwise expected to have no connection in content or topic. Representations can thus not make use of learned content features to solve the task. We display the `STEL` results for the RoBERTa models in Table \[table:results-stle-models\]. **`STEL` performance is comparable across all fine-tuned models — for all different CC levels and AV & CAV setups.** Surprisingly, the overall `STEL` performance for the fine-tuned models is lower than that of the original RoBERTa base model @liu2019roberta. Thus, models may have 'unlearned' some style information. In the remainder of this subsection, we analyze possible reasons for this `STEL` performance drop. Performance stays approximately the same or improves for the formal/informal and the contraction dimensions, but drops for the complex/simple and the nb3r substitution dimensions. Based on manual inspection, we notice nb3r substitution to regularly appear in specific conversations and for specific topics. Future work could investigate whether the use of nb3r substitution is less consistent for one author than other stylistic dimensions. As the nb3r dimension of `STEL` only consists of 100 instances, future work could increase the number of instances. Further, we perform an error analysis to investigate the `STEL` performance drop in the complex/simple dimension. We manually look at consistently unlearned (i.e., wrongly predicted by the fine-tuned but correctly predicted by the original RoBERTa model) or learned (i.e., wrongly predicted by the RoBERTa model and correctly predicted by the fine-tuned model) `STEL` instances (see details in Appendix \[sec:app\_STEL-error-analysis\]). We find several problematic examples where the correct solution to the task is at least ambiguous. We display two such examples in Table \[table:err-analysis-unlearned\_qualitative\]. The share of examples with problematic ambiguities is higher for the unlearned (50/55) than for the newly learned `STEL` instances (29/41). Generally, the number of complex/simple `STEL` instances with ambiguities is surprisingly high for both the learned as well as the unlearned instances, consistent with the lower performance of the models in this category. Several of the found ambiguities should be relatively easy to correct in the future (e.g., spelling mistakes or punctuation differences). Content-Independence of Style Representations {#sec:eval-t-a-STEL} --------------------------------------------- We tested whether models are able to distinguish between different authors (in Section \[sec:eval-TT\]) and represent styles when the content remains the same (Section \[sec:eval-STEL\]). However, we have not tested whether models learn to represent style independent from content. Different approaches have been used to test whether style representations encode unwanted content information, including (a) comparing performance on the AV task across domain @AV_SimilarityLearning [@zhu-jurgens-2021-idiosyncratic], (b) assessing performance on function vs. content words @hay-etal-2020-representation [@zhu-jurgens-2021-idiosyncratic] and (c) predicting domain labels from utterances using their style representations @zhu-jurgens-2021-idiosyncratic. However, these evaluation methods have limitations: Domain labels usually come from a small set of coarse-grained labels and function words have been shown to not necessarily be content-independent @litvinova_context-stylometric. Additionally, next to content, AV might include other spurious features that help increase performance without representing style. \[th\] **Agg.** **GT** **Anchor 1 (A1)** **Anchor 2 (A2)** **Sentence 1 (S1)** **Sentence 2 (S2)** **Ambiguity** ---------- -------- ------------------------------------------------------------------------ ---------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------ ---------------------------------------------------------------------------------------------------------------- ------------------------------------------ **un** TDL Group announced in March 2006, in response to a request \[...\] \[...\] storm names Alberto Helene Beryl Isaac Chris \[...\] Palestinian voters in the Gaza Strip \[...\] were eligible to participate in the election. 1\. Palestinian voters in the Gaza Strip \[...\] were eligible to participate in the election. A1/A2 have different content **l** \[...\] 51 Phantom \[...\] received nominations in that same category. \[...\] 1 phantom \[...\] received nominations in the same category. \[...\] the Port Jackson District Commandant could exchange with all military land with buildings on the harbor. \[...\] the Port Jackson District Commandant could communicate with all military installations on the harbour. A2 spelling mistake, S1 sounds unnatural To test if models learn to prefer style over content, we introduce a variation to the `STEL` framework — the *`STEL`-Or-Content* task: From one original `STEL` instance (Section \[sec:eval-STEL\]), we take the sentence that has the same style as A2 and replace it with A2. In Figure \[tab:t-a-STEL\], this leads to S1 being replaced by A2. The new task is to decide whether A1 matches with the new S1 (originally A2) or with S2. The task is more difficult than the original `STEL` task as S2 is written in the same style as A1 but has different content and the new S1 is written in a different style but has the same content. The representations will have to decide between giving 'style or content' more weight. This setup is similar to the CAV task (Figure \[fig:Task\]). The main differences to the CAV task are (i) that we do not use same author as a proxy for same style but instead use the predefined style dimensions from the `STEL` framework and (ii) that we control for content with the help of paraphrases (instead of using ony a topic proxy). We display the `STEL`-Or-Content results in Table \[table:results-stle-models\]. The performance for the new task is low ($<0.5$ which corresponds to a random baseline). However, the task is also very difficult as lexical overlap is usually high between the anchor and the false choice (i.e., the sentence that was written in a different style but has the same content). Nevertheless, performance should only be considered in combination with other evaluation approaches (Sections \[sec:eval-TT\] and \[sec:eval-STEL\]) as on this task alone models might perform well because they punish same content information. **Models trained on the CAV task with the conversation CC level are the best at representing style independent from content.** The performance increases from an accuracy of $0.05$ for the original RoBERTa model to up to $0.42\pm.01$ for the representation trained with the CAV task and the conversation CC. This 'CAV conversation representation' did not just learn to punish same content cues because of its performance on the AV task and the `STEL` framework: (1) On the AV task, the representation performed comparably on all three test sets. If the model had learned to just punish same content cues, we would expect a clearer difference in performance as confounding same content information should be more prevalent for the random than the conversation test set. (2) The representation performed comparably to the other representations on the `STEL` framework, where style information is needed to solve the task but content information cannot be used. Style Representation Analysis ============================= We want to further understand what the style representations learned to be similar styles. We take the best-performing style representation (RoBERTa trained on the CAV task with the conversation CC and seed 106) and perform agglomerative clustering on a sample of 5.000 CAV tasks of the conversation test set resulting in 14,756 unique utterances. We use 7 clusters based on an analysis of Silhouette scores (Appendix \[sec:app\_silhouette\]). Out of all utterance pairs that have the same author, $46.2\%$ appear in the same cluster. This is different from random assignments among 7 clusters[^10] which corresponds to $20.1\%\pm .00$. As authors will have a certain variability to their style, a perfect clustering according to general linguistic style would not assign all same author pairs to the same cluster. **C \#** **Consistent** **Example** ---------- ---------------------- -------------------------------------------------------------------------- **3** no last punct. I am living in china, they are experiencing an enormous baby boom **4** punctuation / casing huh thats odd im in the 97% percentile on iq tests, the sat, and the act **5** ' vs I assume it's the blind lady? **7** linebreaks I admire what youre doing but \[...\] I know Im \[...\] : **Clusters for RoBERTa Trained on CAV with Conversation Content Control.** We display one example for 4 out of 7 clusters. We mention noticeable consistencies within the cluster (Consistent). <span data-label="tab:clusters"></span> In Table \[tab:clusters\], we display examples for 4 out of 7 clusters. We manually looked at a few hundred examples per cluster to find consistencies. We found clear consistencies within clusters in the punctuation (e.g., 97% of utterances have no last punctuation mark in Cluster 3 vs. an average of $37\%$ in the other clusters), casing (e.g., $67\%$ of utterances that use *i* instead of *I* appear in Cluster 4), contraction spelling (e.g., 22 out of 27 utterances that use *didnt* instead of *didn't* appear in Cluster 4), the type of apostrophe used (e.g., $90\%$ of utterances use <span>'</span> vs in Cluster 5 vs. an average of $0\%$ in the other clusters) and line breaks within an utterance (e.g., $72\%$ of utterances in Cluster 7 include line breaks vs. an average of $22\%$ in the other clusters). We mostly found letter-level consistencies — likely because they are easiest to spot manually. We expect representations to also capture more complex stylometric information because of their performance on the AV and `STEL` tasks (Section \[sec:eval\]). Future work could analyze whether and what other stylistic consistencies are represented by the models. For comparison we also cluster with the base RoBERTa model (see Appendix \[sec:app\_cluster\]). The only three interesting RoBERTa clusters (i.e., clusters 2,3,4 that contain more than three elements and not as many as $86.7\%$ of all utterances), seem to mostly differ in utterance length (average number of characters are 15 in Cluster 2 vs. in 1278 in Cluster 3) and in the presence of hyperlinks ($84\%$ of utterances contain 'https://' in Cluster 4 vs. an overall average of $2\%$). Average utterance lengths are not as clearly separated by the clusters of the trained style representations. Limitations and Future Work =========================== We propose several directions for future research: First, conversation labels are already inherently available in conversation corpora like `Reddit`. However, it remains a difficulty to transfer the conversation CC to other than conversation datasets. Moreover, even when using the conversation CC, content information might still be useful for AV: If one person writes "my husband" and another writes "my wife" within the same conversation, it is highly unlikely that those utterances have been generated by the same person. With the recent advances in semantic sentence embeddings, it might be interesting to train style representations on CAV tasks with a new content control level: Two utterances could be labelled as having the same content if their semantic embeddings are close to each other (e.g., when cosine similarity is above a certain threshold). Second, for the `STEL`-Or-Content task, the so-called "triplet problem" @wegmann-nguyen-2021-capture remains a potential problem. Consider the example in Figure \[tab:t-a-STEL\]. Here, the `STEL` framework only guarantees that A1 is more informal than A2 and S2 is more informal than S1. Thus, in some cases A2 can be stylistically closer to A1 than S2. However, we expect this case to be less prevalent: A2 would need to be already pretty close in style to A1, or both S2 and S1 would need to be substantially more informal or formal than A1. In the future, removing problematic instances could alleviate a possible maximum performance cap. Third, the representation models may learn to represent individual stylistic variation as we use utterances from the same individual author as positive signals (c.f. ). However, because the representation models learn with same author pairs that are generated from thousands of authors, it is likely that they also learn consistencies along groups of authors that use similar style features (e.g., demographic groups based on age or education level, or subreddit communities). Future work could explore how different CC levels and training tasks influence the type of styles that are learned. Conclusion ========== Recent advances in the development of style representations have increasingly used training objectives from authorship verification @hay-etal-2020-representation [@zhu-jurgens-2021-idiosyncratic]. However, representations that perform well on the Authorship Verification (AV) task might do so not because they represent style well but because they latch on to spurious content correlations. We train different style representations by controlling for content (CC) using conversation or domain membership as a proxy for topic. We also introduce the new Contrastive Authorship Verification setup (CAV) and compare it to the usual AV setup. We propose an original adaptation of the recent `STEL` framework @wegmann-nguyen-2021-capture to test whether learned representations favor style over content information. We find that representations that were trained on the CAV setup with conversation CC represent style in a way that is more independent from content than models using other CC levels or the AV setup. We demonstrate some of the learned stylistic differences via agglomerative clustering — e.g., the use of a right single quotation mark vs. an apostrophe in contractions. We hope to contribute to increased efforts towards learning general-purpose content-controlled style representations. Ethical Considerations {#ethical-considerations .unnumbered} ====================== We use utterances taken from 100 subcommunities (i.e., subreddits) of the popular online platform `Reddit` to train style representations with different training tasks and compare their performance. With our work, we aim to contribute to the development of general style representations that are disentangled from content. Style representations have the potential to increase classification performance for diverse demographics and social groups @hovy-2015-demographic. The user demographics on the selected 100 subreddits are likely skewed towards particular demographics. For example, locally based subreddits (e.g., canada, singapore) might be over-represented. Generally, the average `Reddit` user is typically more likely to be young and male.[^11] Thus, our representations might not be representative of (English) language use across different social groups. However, experiments on the set of 100 distinct subreddits should still demonstrate the possibilities of the used approaches and methods. We hope the ethical impact of reusing the already published `Reddit` dataset @Baumgartner_Zannettou_Keegan_Squire_Blackburn_2020 [@chang-etal-2020-convokit] to be small but acknowledge that reusing it will lead to increased visibility of data that is potentially privacy infringing. As we aggregate the styles of thousands of users to calculate style representations, we expect it to not be indicative of individual users. We confirm to have read and that we abide by the ACL Code of Ethics. Acknowledgements {#acknowledgements .unnumbered} ================ We thank the an anonymous ARR reviewers for their helpful comments. This research was supported by the "Digital Society - The Informed Citizen" research programme, which is (partly) financed by the Dutch Research Council (NWO), project 410.19.007. Dong Nguyen was supported by the research programme Veni with project number VI.Veni.192.130, which is (partly) financed by the Dutch Research Council (NWO). Results on the Development Set {#sec:app_dev-results} ============================== Hyperparameter Tuning {#sec:hyperparameter} --------------------- We evaluated contrastive (on the AV training setup), triple (on the CAV training setup) and online contrastive loss (on the AV training setup) using implementations from `Sentence-Transformers`. We experiment with the loss hyperparameter "margin" with values of 0.4, 0.5, 0.6 for the uncased BERT model @devlin-etal-2019-bert on the domain training data. Results are displayed in Figure \[fig:hyperparameter-tuning\]. Contrastive and triplet loss perform better than online contrastive loss. The margin value only has a small influence on the performance scores. Based on these results, we decided to run all further models only with the contrastive and triplet loss functions and a margin value of 0.5. ----------------------- ------------------- ------------------- ------------------- ------------------- ---------- ---------- acc AUC acc AUC acc AUC <span>c 0.4</span> <span>0.63</span> 0.63 **0.68** **0.68** **0.71** **0.71** <span>c 0.5</span> <span>0.63</span> 0.63 **0.68** **0.68** **0.71** **0.71** <span>c 0.6</span> <span>0.62</span> 0.63 **0.68** **0.68** **0.71** **0.71** <span>0.63</span> <span>0.62</span> <span>0.68</span> <span>0.67</span> 0.70 0.70 <span>t 0.5</span> **0.64** **0.64** **0.68** **0.68** 0.70 0.70 <span>t 0.6</span> <span>0.63</span> <span>0.63</span> <span>0.67</span> <span>0.67</span> 0.70 0.70 <span>0.58</span> 0.58 0.64 0.64 0.67 0.67 <span>c-on 0.5</span> <span>0.58</span> 0.58 0.64 0.64 0.67 0.67 <span>c-on 0.6</span> <span>0.58</span> 0.58 0.64 0.64 0.67 0.67 ----------------------- ------------------- ------------------- ------------------- ------------------- ---------- ---------- : **Hyperparameter-tuning results on the dev AV and CAV datasets with varying content control.** Results for BERT uncased trained on the contrastive authorship verification tasks (CAV). With different loss functions (contrastive - c, triple - t, contrastive online - c-on) and margin values (0.4, 0.5, 0.6). For each dev set (conversation, domain and no content control), we display the accuracy of the models for the CAV task and the AUC for the authorship verification task (AV). For each dev set and CAV/AV setup, the best performance is boldfaced. contrastive and triple loss behave comparable. The margin value only has a small influence. <span data-label="fig:hyperparameter-tuning"></span> Detailed Results on the Development Sets {#sec:app_details-dev-results} ---------------------------------------- We display the performance of further fine-tuned models on the dev sets in Table \[fig:task-results\]. RoBERTa @liu2019roberta generally performs better than the uncased and cased BERT model @devlin-etal-2019-bert. Performance for the triplet and contrastive loss functions are comparable. We only use RoBERTa models in the main paper and both contrastive and triplet loss as a result. \[t\] Details on `STEL` results {#sec:app_stel-details} ========================= We display the `STEL` results on further trained models in Table \[tab:stel\_add\]. Interestingly, cased BERT seems to be the better choice for the contraction `STEL` dimension. \[t\] ------------ ---------------------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- train data model `STEL` o-c `STEL` o-c `STEL` o-c `STEL` o-c `STEL` o-c <span>BERT uncased (bert)</span> 0.75 0.03 0.76 0.05 0.70 0.00 **0.93** 0.09 1.00 0.00 <span>BERT cased (BERT)</span> **0.78** 0.05 0.80 0.10 **0.71** 0.00 0.92 **0.11** 1.00 0.00 <span>bert c 0.5</span> 0.68 <span>0.21</span> 0.72 <span>0.40</span> 0.59 <span>0.07</span> 0.73 <span>0.06</span> 1.00 <span>0.01</span> <span>bert t 0.5</span> 0.68 <span>0.30</span> 0.71 <span>0.52</span> 0.61 <span>0.15</span> 0.72 0.05 0.99 <span>0.06</span> <span>BERT c 0.5</span> 0.73 0.32 0.83 0.62 0.60 **0.19** 0.67 0.06 1.00 0.00 <span>BERT t 0.5</span> 0.73 **0.37** 0.79 **0.66** 0.63 0.15 0.74 0.05 1.00 **0.15** <span>bert c 0.4</span> <span>0.70</span> <span>0.12</span> <span>0.76</span> <span>0.26</span> <span>0.61</span> <span>0.01</span> <span>0.72</span> 0.02 <span>1.00</span> <span>0.00</span> <span>bert c 0.5</span> 0.69 <span>0.13</span> 0.74 <span>0.27</span> 0.59 <span>0.01</span> 0.68 <span>0.05</span> 1.00 <span>0.00</span> <span>bert c 0.6</span> <span>0.70</span> <span>0.13</span> <span>0.76</span> <span>0.26</span> <span>0.61</span> <span>0.01</span> <span>0.72</span> 0.04 <span>1.00</span> <span>0.00</span> <span>bert c-on 0.4</span> 0.65 <span>0.02</span> 0.67 <span>0.03</span> 0.60 <span>0.00</span> 0.69 0.02 0.84 <span>0.00</span> <span>bert c-on 0.5</span> 0.65 <span>0.02</span> 0.67 <span>0.03</span> 0.60 <span>0.00</span> 0.69 0.02 0.84 <span>0.00</span> <span>bert c-on 0.6</span> 0.65 <span>0.02</span> 0.67 <span>0.03</span> 0.60 <span>0.00</span> 0.69 0.02 0.84 <span>0.00</span> <span>bert t 0.4</span> <span>0.71</span> <span>0.15</span> <span>0.78</span> <span>0.31</span> <span>0.59</span> <span>0.01</span> <span>0.78</span> 0.05 1.00 <span>0.00</span> <span>bert t 0.5</span> 0.68 <span>0.18</span> 0.74 <span>0.37</span> 0.58 <span>0.03</span> 0.72 <span>0.06</span> 1.00 <span>0.00</span> <span>bert t 0.6</span> 0.69 <span>0.22</span> 0.76 <span>0.44</span> 0.58 <span>0.04</span> 0.69 0.06 1.00 <span>0.00</span> <span>BERT c-0.5</span> 0.73 0.23 0.82 0.48 0.61 0.02 0.77 0.03 1.00 0.00 <span>BERT t-0.5</span> 0.71 <span>0.28</span> 0.81 <span>0.56</span> 0.57 <span>0.06</span> 0.80 0.04 1.00 0.00 <span>bert c 0.5</span> 0.69 0.09 0.77 0.20 0.58 0.01 0.68 0.02 0.98 0.00 <span>bert t 0.5</span> 0.70 0.13 0.75 0.26 0.61 0.03 0.79 0.06 1.00 0.00 <span>BERT c-0.5</span> 0.72 0.21 **0.84** 0.44 0.55 0.02 0.75 0.07 1.00 0.01 <span>BERT t-0.5</span> 0.73 <span>0.23</span> **0.84** <span>0.48</span> 0.59 <span>0.03</span> 0.68 0.05 1.00 0.00 ------------ ---------------------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- ------------------- Error Analysis RoBERTa `STEL` results {#sec:app_STEL-error-analysis} ------------------------------------- In Table \[fig:error-analysis\_quantitative\], we display the number of learned and unlearned `STEL` instances across different aggregates for the RoBERTa models. We combine all such unique `STEL` instances across the aggregates and annotate if they contain ambiguities. In Table \[tab:error-analysis\_categories\], we display the results. Overall, the learned `STEL` instances contain fewer ambiguities. However, they still show considerable amounts of ambiguities. -- -------------- ----- ----- ----- ----- f/i c/s f/i c/s conversation 21 34 62 22 domain 13 34 62 24 no 21 44 67 24 AV 8 9 61 11 CAV 6 14 55 14 all 1 4 48 8 -- -------------- ----- ----- ----- ----- : **Error Analysis `STEL` Results.** For the formal/informal (f/i) and complex/simple (c/s) `STEL` dimension, we display the number of instances that were unlearned and learned by all RoBERTa models in an aggregate. We use three different aggregates: (i) all models trained with a given CC level, (ii) all models trained with a certain task setup and (iii) all models. <span data-label="fig:error-analysis_quantitative"></span> <span>unlearned</span> <span>learned</span> ----------------------- ---------------------------- ----------------------------- no ambiguity $\frac{5}{55}\approx9\%$ $\frac{12}{41}\approx29\% $ typo simple $\frac{21}{55}\approx38\%$ $\frac{13}{41}\approx32\%$ typo complex $\frac{11}{55}\approx20\%$ $\frac{6}{41}\approx15\%$ error grammar simple $\frac{15}{55}\approx27\%$ $\frac{9}{41}\approx22\%$ error grammar complex $\frac{5}{55}\approx9\%$ $\frac{3}{41}\approx7\%$ changed content $\frac{5}{55}\approx9\%$ $\frac{3}{41}\approx7\%$ word as/more complex $\frac{16}{55}\approx29\%$ $\frac{11}{41}\approx27\%$ naturalness $\frac{7}{55}\approx13\%$ $\frac{3}{41}\approx7\%$ : **Categories Error Analysis `STEL` Results.** For the six fine-tuned RoBERTa models, we manually looked at the common learned as well as the unlearned simple/complex examples. We put the examples in the displayed ambiguity classes. <span data-label="tab:error-analysis_categories"></span> Details on cluster parameters {#sec:app_silhouette} ============================= We use agglomerative clustering for the RoBERTa model trained on the CAV setup with a margin of 0.5 and conversations as CC with seed 106 (R CAV CONV 106). We experiment with different numbers of clusters and display the results in Table \[tab:cluster\_hyperparameter-tuning\]. The highest Silhouette scores are reached for cluster sizes of 5, 6, 7. We select a cluster size of 7 for evaluation. n avg. silhouette -- ----------------- ----------------- <span>2</span> 0.23 3 0.21 4 0.23 **5** **0.27** **6** **0.27** 7 0.26 8 0.23 9 0.19 10 0.20 11 0.19 12 0.18 13 0.19 14 0.17 15 0.16 16 0.16 17 0.16 18 0.17 19 0.17 20 0.17 21 0.16 22 0.16 23 0.15 24 0.15 <span>25</span> 0.15 26 0.15 30 0.15 40 0.15 50 0.15 100 0.13 150 0.13 200 0.12 : **Silhouette values.** We experiment with different numbers of clusters for one fine-tuned RoBERTa model (R CAV CONV 106). It was on the CAV task with conversation CC. The highest Silhouette score is reached for cluster sizes of 5–7. <span data-label="tab:cluster_hyperparameter-tuning"></span> Details on the cluster analysis {#sec:app_cluster} =============================== We give more examples of the seven clusters in Table \[table:app\_cl-examples\]. Refer to our Github repository for the complete clustering. We did not find obvious consistencies for clusters 1, 2 and 6. That does, however, not mean that more nuanced stylistic consistencies are not present. We recommend using a higher number of clusters, possibly different clustering algorithms and testing out statistics for known style features to pinpoint more consistencies. Out of all utterance pairs that have the same author, $46.2\%$ appear in the same cluster for the style embedding model. This is different from a random distribution among 7 clusters[^12] which corresponds to $20.1\%\pm.00$. As authors will have a certain variability to their style as well (e.g., ), a perfect clustering according to writing style would not assign all same author pairs to the same cluster. For the RoBERTa base model the fraction of same author pairs in the same cluster is closer to the random distribution ($75.4\%$ vs. $76.1\%$ for the random distribution[^13]). The fraction of utterance pairs that appear in the same domain are close to the random distribution for both the style embedding model ($23.6\%$ vs. $20.1\%$) and the RoBERTa base model ($77.6\%$ vs. $76.0\%$). The percentage for the RoBERTa base models is a lot higher as the first cluster contains almost $90\%$ of all utterances. Random assignment of utterances across the 7 clusters, that keeps the clustering size would already lead to $76.0\%$ same author pairs appearing in the same cluster (almost all of them in the first). Results are similar for utterance pairs that appear in the same conversation. \[th!\] **C** **\#** **Consistency** **Example 1** **Example 2** **Example 3** ------- -------- ------------------------------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ -------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------- **1** 4065 citing previous comments, standard punctuation, URLs Yes. Proportionally, this kid's feet are absolutely enormous. &gt; Please delete your account. Says the no life who always shits on anything Kanye or anti-Drake I can promise you that capitalism is very much alive in Norway. \[This should help.\](YOUTUBE-LINK) **2** 4016 short sentences? Nice catch! Well done. cookies are in the back of this Grammar party. You can have two. You can mute them we've been told! Came here to post this only to find it's already the top voted comment. This is a good sub. **3** 2165 no last punctuation mark I am living in china, they are experiencing an enormous baby boom Seems like sarcasm. But could also be Poe \[...\] The earth probably has two or more degrees of symmetry, but less than infinite (like a sphere), but Im honestly not too concerned about the minutiae of it **4** 1794 punctuation / casing huh thats odd im in the 97% percentile on iq tests, the sat, and the act Its not a problem if you a got a full game. Whats the problem if a game didnt get expansions? Fair point, I didnt know that. Just at glance I kind of went woah that doesnt seem right **5** 1555 ' instead of apostrophe I assume it's the blind lady? Oh I wasn't really dismissing them. I'm saying Ford will try their own thing compared to Fiat It's 4am in Brussels and I am still hyped **6** 781 similar to 1? Well, as your neighbors, I'd say Fuck you.. But we're not like that, see? We want to be part of the alliance, not part of the 'fuck you, we cant be competitive with jobs or innovate any more, so we're going to run massive tariffs against all our friendly nations Hah, thus the one calf larger than the other issue. I have it too ;) \[So you are saying that current encryption falls apart as long as the quantum computer is large enough\](URL). (for reference, the current highest qubit is 50)' **7** 380 linebreaks I admire what youre doing but \[...\] I know Im in the minority. \[...\] 75% of the problems I run into are solved by \[...\] I work in live streaming. All the suggestions others have given are excellent. RS7 makes the most sense to me. But \[...\] Meanwhile, \[...\] \[t\] **C** **\#** **Consistency** **Example 1** **Example 2** **Example 3** ------- -------- ------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ **1** 12798 wide variety Just googled it, looks like a great device for the price! If I weren't so impatient I would have bought this online. Great battery life! This is exactly why i believe iphone 5 body was perfect example of good balance with design(timeless) and utility \[...\] The earth probably has two or more degrees of symmetry, but less than infinite (like a sphere), but Im honestly not too concerned about the minutiae of it **2** 1110 short utterances here we go!! And her good posture. Not in California. **3** 310 long utterances I've never had the pleasure of seeing Neil live but I got on a big kick a few years ago after buying one of his live albums (can't remember which one) where I listened to all his live albums and then wanted to see as many of his live performance I could find on YouTube. \[...\] &gt; but the movie has the superior ending I think. \[...\] \[...\] So .... heavily influenced by the social economics ... but still voluntary, got it. \[...\] Then how about this. \[...\] Everyone still keeps their child that way, you even promote child birth. No sterilization, no stigmatization of poor people, no poor people stuck with child with heavy needs requiring care that they can't pay for. **4** 232 URLs <https://youtu.be/GmULc5VANsw> \[This\](<https://np.reddit.com/r/MakeupAddiction/comments/25hkqi/how_to_tell_if_your_foundationprimer_is_silicone/>) might help! I thought there was 51 stars because of Puerto Rico <https://en.m.wikipedia.org/wiki/51st_state> Computing Infrastructure ======================== The training of 23 RoBERTa @liu2019roberta, 13 uncased BERT and 6 cased BERT models @devlin-etal-2019-bert took about 846 GPU hours with one RTX6000 card with 24 GB RAM on a Linux computing cluster. Further analysis and clustering of two RoBERTa models took about 24 GPU hours. We used a machine with 32 GB RAM and 8 intel i7 CPUs using Ubuntu 20.04 LTS without GPU access to generate the training data. We used `Sentence-Transformers` 2.1.0 @reimers-gurevych-2019-sentence and `numpy` 1.18.5 @harris2020array, `scipy` 1.5.2 @2020SciPy-NMeth and `scikit-learn` 0.24.2 @scikit-learn. We use previous work, including code and data, consistent with their specified or implied intended use @reimers-gurevych-2019-sentence [@chang-etal-2020-convokit; @wegmann-nguyen-2021-capture]. The `ConvoKit` open-source Python framework invites NLP researchers and 'anyone with questions about conversations' to use it @chang-etal-2020-convokit. The `SentenceTransformers` Python framework can be used to compute sentence / text embeddings.[^14] We comply with asking permission for part of the dataset for `STEL` and citing the specified works @wegmann-nguyen-2021-capture. state the intended use of developing improved style(-sensitive) measures. Intended Use ============ We hope our work will inform further research into style and its representations. We invite researchers to reuse any of our provided results, code and data for this purpose. [^1]: <https://github.com/nlpsoc/Style-Embeddings> [^2]: <https://zissou.infosci.cornell.edu/convokit/datasets/subreddit-corpus/subreddits_small_sample.txt> [^3]: MIT license [^4]: Utterance of only spaces, tabs, line breaks or\ of the form: `""`, `" [removed] "`, `"[ removed ]"`,\ `"[removed]"`, `"[ deleted ]"`, `"[deleted]"`, `" [deleted] "` [^5]: Due to the sampling process, there might be same author ($A_1$, $A_2$) pairs that occur twice. However, this remains unlikely due to the high number of authors and utterances. Overall, the share of repeating pairs remains lower than 1%. [^6]: <https://sbert.net/> [^7]: with Apache License 2.0 [^8]: We used seeds 103-105. A total of 5 out of 18 models did not learn. We re-trained those with different seeds. [^9]: <https://github.com/nlpsoc/STEL>, with data from and with permission from Yahoo for the "L6 - Yahoo! Answers Comprehensive Questions and Answers version 1.0 (multi part)": <https://webscope.sandbox.yahoo.com/catalog.php?datatype=l>. Data and code available with MIT License with exceptions for proprietary Yahoo data. [^10]: Calculated mean and standard deviation of
{'source': 'arXiv_out/2204'}