---
abstract: 'As natural language processing methods are increasingly deployed in real-world scenarios such as healthcare, legal systems, and social science, it becomes necessary to recognize the role they potentially play in shaping social biases and stereotypes. Previous work has revealed the presence of *social biases* in widely used word embeddings involving gender, race, religion, and other social constructs. While some methods were proposed to debias these word-level embeddings, there is a need to perform debiasing at the sentence-level given the recent shift towards new contextualized sentence representations such as ELMo and BERT. In this paper, we investigate the presence of social biases in sentence-level representations and propose a new method, <span style="font-variant:small-caps;">Sent-Debias</span>, to reduce these biases. We show that <span style="font-variant:small-caps;">Sent-Debias</span> is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks such as sentiment analysis, linguistic acceptability, and natural language understanding. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.'
author:
- |
    Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim,\
    <span>**Ruslan Salakhutdinov, Louis-Philippe Morency**</span>\
    Machine Learning Department and Language Technologies Institute\
    Carnegie Mellon University\
    <span>jacquelinehopkins@example.com</span>
bibliography:
- 'acl2020.bib'
title: Towards Debiasing Sentence Representations
---

=1

Introduction
============

Machine learning tools for learning from language are increasingly deployed in real-world scenarios such as healthcare @VELUPILLAI201811, legal systems @journals/nle/Dale19, and computational social science @ws-2016-nlp-social. Key to the success of these models are powerful *embedding* layers which learn continuous representations of input information such as words, sentences, and documents from large amounts of data @DBLP:journals/corr/abs-1810-04805 [@mikolov2013efficient]. Although word-level embeddings @pennington2014glove [@mikolov2013efficient] are highly informative features useful for a variety of tasks in Natural Language Processing (NLP), recent work has shown that word-level embeddings reflect and propagate *social biases* present in training corpora @lauscher-glavas-2019-consistently [@caliskan2017semantics; @DBLP:journals/corr/abs-1812-08769; @bolukbasi2016man]. Machine learning systems that incorporate these word embeddings can further amplify biases @sun-etal-2019-mitigating [@zhao2017men; @barocas2016big] and unfairly discriminate against users, particularly those from disadvantaged social groups. Fortunately, researchers working on fairness and ethics in NLP have devised methods towards *debiasing* these word representations for both binary @bolukbasi2016man and multiclass @DBLP:journals/corr/abs-1904-04047 bias attributes such as gender, race, and religion.

More recently, sentence-level representations such as ELMo @DBLP:journals/corr/abs-1802-05365, BERT @DBLP:journals/corr/abs-1810-04805, and GPT @radford2019language have become the preferred choice for text sequence encoding. When compared to word-level representations, these models have achieved better performance on multiple tasks in NLP @Wu2019BetoBB, multimodal learning @Zellers2018FromRT [@Sun2019VideoBERTAJ], and grounded language learning @Urbanek2019LearningTS. As their usage proliferates across various real-world applications @Huang2019ClinicalBERTMC [@Alsentzer2019PubliclyAC], it becomes necessary to recognize the role they play in shaping social biases and stereotypes.

Debiasing sentence representations is difficult for two reasons. Firstly, it is usually *unfeasible* to fully retrain many of the state-of-the-art sentence-based embedding models. In contrast with conventional word-level embeddings such as GloVe @pennington2014glove and word2vec @mikolov2013efficient which can be retrained on a single machine within a few hours, the best sentence encoders such as BERT @DBLP:journals/corr/abs-1810-04805, and GPT @radford2019language are trained on massive amounts of text data over hundreds of machines for several weeks. As a result, it is difficult to retrain a new sentence encoder whenever a new source of bias is uncovered from data. We therefore focus on *post-hoc* debiasing techniques which add a post-training debiasing step to these sentence representations before they are used in downstream tasks @bolukbasi2016man [@DBLP:journals/corr/abs-1904-04047]. Secondly, sentences display large variety in how they are composed from individual words. This variety is driven by many factors such as topics, individuals, settings, and even differences between spoken and written text. As a result, it is difficult to scale traditional word-level debiasing approaches (which involve bias-attribute words such as *man, woman*) @bolukbasi2016man to sentences.

**Related Work:** Although there has been some recent work in measuring the presence of bias in sentence representations @DBLP:journals/corr/abs-1903-10561 [@DBLP:journals/corr/abs-1904-08783], none of them have been able to successfully *remove* bias from pretrained sentence representations. In particular, @DBLP:journals/corr/abs-1904-03310, @park-etal-2018-reducing, and @DBLP:journals/corr/abs-1809-10610 are not able to perform post-hoc debiasing and require changing the data or underlying word embeddings and retraining which is costly. @DBLP:journals/corr/abs-1904-03035 only study word-level language models and also requires re-training. Finally, @kurita_measuring_2019 only measure bias on BERT by extending the word-level Word Embedding Association Test (WEAT) @caliskan2017semantics metric in a manner similar to @DBLP:journals/corr/abs-1903-10561.

In this paper, as a compelling step towards generalizing debiasing methods to sentence representations, we capture the various ways in which bias-attribute words can be used in natural sentences. This is performed by *contextualizing* bias-attribute words using a *diverse* set of sentence templates from various text corpora into bias-attribute sentences. We propose <span style="font-variant:small-caps;">Sent-Debias</span>, an extension of the <span style="font-variant:small-caps;">Hard-Debias</span> method @bolukbasi2016man, to debias sentences for both binary[^1] and multiclass bias attributes spanning gender and religion. Key to our approach is the *contextualization* step in which bias-attribute words are converted into bias-attribute sentences by using a *diverse* set of sentence templates from text corpora. Our experimental results demonstrate the importance of using a large number of diverse sentence templates when estimating bias subspaces of sentence representations. Our experiments are performed on two widely popular sentence encoders BERT @DBLP:journals/corr/abs-1810-04805 and ELMo @DBLP:journals/corr/abs-1802-05365, showing that our approach reduces the bias while preserving performance on downstream sequence tasks. We end with a discussion about possible shortcomings and present some directions for future work towards accurately characterizing and removing social biases from sentence representations for fairer NLP.

Debiasing Sentence Representations {#debias}
==================================

Our proposed method for debiasing sentence representations, <span style="font-variant:small-caps;">Sent-Debias</span>, consists of four steps: 1) *defining* the words which exhibit bias attributes, 2) *contextualizing* these words into bias attribute sentences and subsequently their sentence representations, 3) *estimating* the sentence representation bias subspace, and finally 4) *debiasing* general sentences by removing the projection onto this bias subspace. We summarize these steps in Algorithm 1 and describe the algorithmic details in the following subsections.

**1) Defining Bias Attributes:** The first step involves *identifying* the bias attributes and defining a set of *bias attribute words* that are indicative of these attributes. For example, when characterizing bias across the male and female genders, we use the word pairs (*man*, *woman*), (*boy*, *girl*) that are indicative of gender. When estimating the $3$-class religion subspace across the Jewish, Christian, and Muslim religions, we use the tuples (*Judaism*, *Christianity*, *Islam*), (*Synagogue*, *Church*, *Mosque*). Each tuple should consist of words that have an equivalent meaning except for the bias attribute. In general, for $d$-class bias attributes, the set of words forms a dataset $\mathcal{D} = \{ (w_1^{(i)}, ..., w_d^{(i)}) \}_{i=1}^m$ of $m$ entries where each entry $(w_1, ..., w_d)$ is a $d$-tuple of words that are each representative of a particular bias attribute (we drop the superscript $^{(i)}$ when it is clear from the context). Table \[words\] shows some bias attribute words that we use to estimate the bias subspaces for binary gender and multiclass religious attributes (full pairs and triplets in appendix).

Existing methods that investigate biases tend to operate at the word-level which simplifies the problem since the set of tokens is bounded by the vocabulary size @bolukbasi2016man. This simple approach has the advantage of identifying the presence of biases using predefined sets of word associations, and estimate the bias subspace using the predefined bias word pairs. On the other hand, the potential number of sentences are unbounded which makes it harder to precisely characterize the sentences in which bias is present or absent. Therefore, it is not trivial to directly convert these words to sentences to obtain a representation from pre-trained sentence encoders. In the subsection below, we describe our solution to this problem.

**2) Contextualizing Words into Sentences:** A core step in our <span style="font-variant:small-caps;">Sent-Debias</span> approach involves *contextualizing* the predefined sets of bias attribute words to sentences so that sentence encoders can be applied to obtain sentence representations. One option is to use a simple template-based design to simplify the contextual associations a sentence encoder makes with a given term, similar to how @DBLP:journals/corr/abs-1903-10561 proposed to measure (but not remove) bias in sentence representations. For example, each word can be slotted into templates such as "This is $<$word$>$.", "I am a $<$word$>$.". We take an alternative perspective and hypothesize that for a given bias attribute (e.g. gender), a single bias subspace exists across all possible sentence representations. For example, the bias subspace should be the same in the sentences "The boy is coding.", "The girl is coding.", "The boys at the playground.", and "The girls at the playground.". In order to estimate this bias subspace accurately, it becomes important to use sentence templates that are as *diverse* as possible to account for all occurrences of that word in surrounding contexts. In our experiments, we empirically demonstrate that estimating the bias subspace using a large and diverse set of templates from text corpora leads to improved bias reduction as compared to using simple templates.

To capture the variety in syntax across sentences, we use large text corpora to find naturally occurring sentences. These naturally occurring sentences therefore become our sentence "templates". To use these templates to generate new sentences, we replace words representing a single class with another. For example, a sentence containing a male term "*he*" is used to generate a new sentence but replacing it with the corresponding female term "*she*". This contextualization process is repeated for all word tuples in the bias attribute word dataset $\mathcal{D}$, eventually contextualizing the given set of bias attribute words into *bias attribute sentences*. Since there are multiple templates which a bias attribute word can map to, the contextualization process results in a *bias attribute sentence dataset* $\mathcal{S}$ which is substantially larger in size: $$\begin{aligned}
    \mathcal{S} &= \bigcup_{i=1}^m \textsc{Contextualize}(w_1^{(i)}, ..., w_d^{(i)})\\
    &= \{ (s_1^{(i)}, ..., s_d^{(i)}) \}_{i=1}^n, \ |\mathcal{S}| > |\mathcal{D}|\end{aligned}$$ where $\textsc{Contextualize}(w_1, ..., w_d)$ is a function which returns a set of sentences obtained by matching words with naturally-occurring sentence templates from text corpora.

Our text corpora originate from the following five sources: 1) **WikiText-2** @DBLP:journals/corr/MerityXBS16, a dataset of formally written Wikipedia articles (we only use the first 10% of WikiText-2 which we found to be sufficient to capture formally written text), 2) **Stanford Sentiment Treebank** @socher-etal-2013-recursive, a collection of 10000 polarized written movie reviews, 3) **Reddit** data collected from discussion forums related to politics, electronics, and relationships, 4) **MELD** [@poria-etal-2019-meld], a large-scale multimodal multi-party emotional dialog dataset collected from the TV-series Friends, and 5) **POM** @Park:2014:CAP:2663204.2663260, a dataset of spoken review videos collected across 1,000 individuals spanning multiple topics. These datasets have been the subject of recent research in language understanding [@DBLP:conf/iclr/MerityX0S17; @DBLP:journals/corr/abs-1907-11692; @DBLP:journals/corr/abs-1904-09408] and multimodal human language [@liang-etal-2018-multimodal; @liang-etal-2019-strong]. Table \[datasets\] summarizes these datasets. We also give some examples of the diverse templates that occur naturally across various individuals, settings, and in both written and spoken text.

**3) Estimating the Bias Subspace:** Now that we have contextualized all $m$ word $d$-tuples in $\mathcal{D}$ into $n$ sentence $d$-tuples $\mathcal{S}$, we pass these sentences through a pre-trained sentence encoder (e.g. BERT, ELMo) to obtain sentence representations. Suppose we have a pre-trained encoder $M_\theta$ with parameters $\theta$. Define $\mathcal{R}_j, j \in [d]$ as sets that collect all sentence representations of the $j$-th entry in the $d$-tuple, $\mathcal{R}_j = \{ M_\theta( s_j^{(i)} ) \}_{i=1}^n$. Each of these sets $\mathcal{R}_j$ defines a vector space in which a specific bias attribute is present across its contexts. For example, when dealing with binary gender bias, $\mathcal{R}_1$ (likewise $\mathcal{R}_2$) defines the space of sentence representations with a male (likewise female) context. The only difference between the representations in $\mathcal{R}_1$ versus $\mathcal{R}_2$ should be the specific bias attribute present. Define the mean of set $j$ as $\bm{\mu}_j = \frac{1}{|\mathcal{R}_j|} \sum_{\mathbf{w} \in \mathcal{R}_j} \mathbf{w}$. The bias subspace $\mathbf{V} = \{ \mathbf{v}_1, ..., \mathbf{v}_k \}$ is given by the first $k$ components of principal component analysis (PCA) [@Abdi:2010:PCA:3160436.3160440]: $$\mathbf{V} = \mathbf{PCA}_k \left( \bigcup_{j=1}^d \bigcup_{\mathbf{w} \in \mathcal{R}_j} \left( \mathbf{w} - \bm{\mu}_j \right) \right).$$ $k$ is a hyperparameter in our experiments which determines the dimension of the bias subspace. Intuitively, $\mathbf{V}$ represents the top-$k$ orthogonal directions which most represent the bias subspace.

**4) Debiasing:** Given the estimated bias subspace $\mathbf{V}$, we apply a partial version of the <span style="font-variant:small-caps;">Hard-Debias</span> algorithm @bolukbasi2016man to remove bias from new sentence representations. Taking the example of binary gender bias, the <span style="font-variant:small-caps;">Hard-Debias</span> algorithm consists of two steps:

**4a) Neutralize:** Bias components are removed from sentences that are not gendered and should not contain gender bias (e.g., *I am a doctor.*, *That nurse is taking care of the patient.*) by removing the projection onto the bias subspace. More formally, given a representation $\mathbf{h}$ of a sentence and the previously estimated gender subspace $\mathbf{V} = \{ \mathbf{v}_1, ..., \mathbf{v}_k \}$, the debiased representation $\mathbf{\hat{h}}$ is given by first obtaining $\mathbf{h}_{\mathbf{V}}$, the projection of $\mathbf{h}$ onto the bias subspace $\mathbf{V}$ before subtracting $\mathbf{h}_{\mathbf{V}}$ from $\mathbf{h}$. This results in a vector that is *orthogonal* to the bias subspace $\mathbf{V}$ and therefore contains no bias: $$\begin{aligned}
    \mathbf{h}_{\mathbf{V}} &= \sum_{j=1}^k \langle \mathbf{h}, \mathbf{v}_j \rangle \mathbf{v}_j,\\
    \mathbf{\hat{h}} &= \mathbf{h} - \mathbf{h}_{\mathbf{V}}.\end{aligned}$$

**4b) Equalize:** Gendered representations are centered and their bias components are equalized (e.g. *man* and *woman* should have bias components in opposite directions, but of the same magnitude). This ensures that any neutral words are equidistant to biased words with respect to the bias subspace. In our implementation, we skip this Equalize step because it is hard to identify all or even the majority of sentence pairs to be equalized due to the complexity of natural sentences. For example, we can never find all the sentences that *man* and *woman* appear in to equalize them appropriately. Note that even if the magnitudes of sentence representations are not normalized, the debiased representations are still pointing in directions orthogonal to the bias subspace. Therefore, skipping the equalize step still results in debiased sentence representations as measured by our definition of bias.

Experiments {#exp}
===========

We test the effectiveness of <span style="font-variant:small-caps;">Sent-Debias</span> at removing biases and retaining performance on downstream tasks. All experiments are conducted on English terms and downstream tasks. We acknowledge that biases can manifest differently across different languages, in particular gendered languages [@zhou-etal-2019-examining], and emphasize the need for future extensions in these directions. Experimental details are in the appendix and code is released at <https://github.com/pliang279/sent_debias>.

Evaluating Biases
-----------------

Biases are traditionally measured using the Word Embedding Association Test (WEAT) @caliskan2017semantics. WEAT measures bias in word embeddings by comparing two sets of target words to two sets of attribute words. For example, to measure social bias surrounding genders with respect to careers, one could use the target words *programmer, engineer, scientist*, and *nurse, teacher, librarian*, and the attribute words *man, male*, and *woman, female*. Unbiased word representations should display no difference between the two target words in terms of their relative similarity to the two sets of attribute words. The relative similarity as measured by WEAT is commonly known as the *effect size*. An effect size with absolute value closer to 0 represents lower bias.

To measure the bias present in sentence representations, we use the method as described in @DBLP:journals/corr/abs-1903-10561 which extended WEAT to the Sentence Encoder Association Test (SEAT). For a given set of words for a particular test, words are converted into sentences using a template-based method. The WEAT metric can then be applied for fixed-length, pre-trained sentence representations. To measure bias over multiple classes, we use the Mean Average Cosine similarity (MAC) metric which extends SEAT to a multiclass setting @DBLP:journals/corr/abs-1904-04047. For the binary gender setting, we use words from the Caliskan Tests @caliskan2017semantics which measure biases in common stereotypes surrounding gendered names with respect to careers, math, and science @articleaaaa. To evaluate biases in the multiclass religion setting, we modify the Caliskan Tests used in @DBLP:journals/corr/abs-1903-10561 with lexicons used by @DBLP:journals/corr/abs-1904-04047.

Debiasing Setup
---------------

We first describe the details of applying <span style="font-variant:small-caps;">Sent-Debias</span> on two widely-used sentence encoders: BERT[^2] @DBLP:journals/corr/abs-1810-04805 and ELMo @DBLP:journals/corr/abs-1802-05365. Note that the pre-trained BERT encoder must be fine-tuned on task-specific data. This implies that the final BERT encoder used during debiasing changes from task to task. To account for these differences, we report two sets of metrics: 1) **BERT**: simply debiasing the pre-trained BERT encoder, and 2) **BERT post task**: first fine-tuning BERT and post-processing (i.e. normalization) on a specific task before the final BERT representations are debiased. We apply <span style="font-variant:small-caps;">Sent-Debias</span> on BERT fine-tuned on two single sentence datasets, Stanford Sentiment Treebank (**SST-2**) sentiment classification @socher-etal-2013-recursive and Corpus of Linguistic Acceptability (**CoLA**) grammatical acceptability judgment @warstadt2018neural. It is also possible to apply BERT @DBLP:journals/corr/abs-1810-04805 on downstream tasks that involve *two* sentences. The output sentence pair representation can also be debiased (after fine-tuning and normalization). We test the effect of <span style="font-variant:small-caps;">Sent-Debias</span> on Question Natural Language Inference (**QNLI**) @wang-etal-2018-glue which converts the Stanford Question Answering Dataset (SQuAD) @rajpurkar-etal-2016-squad into a binary classification task. These results are reported as **BERT post SST-2**, **BERT post CoLA**, and **BERT post QNLI** respectively.

For ELMo, the encoder stays the same for downstream tasks (no fine-tuning on different tasks) so we just debias the ELMo sentence encoder. We report this result as **ELMo**.

Debiasing Results
-----------------

We present these debiasing results in Table \[debias\], and see that for both binary gender bias and multiclass religion bias, our proposed method reduces the amount of bias as measured by the given tests and metrics. The reduction in bias is most pronounced when debiasing the pre-trained BERT encoder. We also observe that simply fine-tuning the BERT encoder for specific tasks also reduces the biases present as measured by the Caliskan tests, to some extent. However, fine-tuning does not lead to consistent decreases in bias and cannot be used as a standalone debiasing method. Furthermore, fine-tuning does not give us control over which type of bias to control for and may even amplify bias if the task data is skewed towards particular biases. For example, while the bias effect size as measured by Caliskan test C7 decreases from $+0.542$ to $-0.033$ and $+0.288$ after fine-tuning on SST-2 and CoLA respectively, the effect size as measured by the multiclass Caliskan test increases from $+0.035$ to $+1.200$ and $+0.243$ after fine-tuning on SST-2 and CoLA respectively.

Comparison with Baselines
-------------------------

We compare to three baseline methods for debiasing: 1) **FastText** derives debiased sentence embeddings using an average of debiased FastText word embeddings [@bojanowski2016enriching] using word-level debiasing methods [@bolukbasi2016man], 2) **BERT word** obtains a debiased sentence representation from average debiased BERT word representations, again debiased using word-level debiasing methods [@bolukbasi2016man], and 3) **BERT simple** adapts @DBLP:journals/corr/abs-1903-10561 by using simple templates to debias BERT sentence representations. From Table \[simple\], <span style="font-variant:small-caps;">Sent-Debias</span> achieves a lower average absolute effect size and outperforms the baselines based on debiasing at the word-level and averaging across all words. This indicates that it is not sufficient to debias words only and that biases in a sentence could arise from their debiased word constituents. In comparison with BERT simple, we observe that using diverse sentence templates obtained from naturally occurring written and spoken text makes a difference on how well we can remove biases from sentence representations. This supports our hypothesis that using increasingly diverse templates estimates a bias subspace that generalizes to different words in their context.

Effect of Templates
-------------------

We further test the importance of sentence templates through two experiments.

**1) Same Domain, More Quantity:** Firstly, we ask: *how does the number of sentence templates impact debiasing performance*? To answer this, we begin with the largest domain WikiText-2 (13750 templates) and divide it into 5 partitions each of size 2750. We collect sentence templates using all possible combinations of the 5 partitions and apply these sentence templates in the contextualization step of <span style="font-variant:small-caps;">Sent-Debias</span>. We then estimate the corresponding bias subspace, debias, and measure the average absolute values of all 6 SEAT effect sizes. Since different combinations of the 5 partitions result in a set of sentence templates of different sizes ($20\%$, $40\%$, $60\%$, $80\%$, $100\%$), this allows us to see the relationship between size and debiasing performance. Combinations with the same percentage of data are grouped together and for each group we compute the mean and standard deviation of the average absolute effect sizes. We perform the above steps to debias BERT fine-tuned on SST-2 and QNLI and plot these results in Figure \[fig:sent\_size\]. Please refer to the appendix for experiments with BERT fine-tuned on CoLA, which show similar results.

For BERT fine-tuned on SST-2, we observe a *decreasing trend* in the effect size as increasing subsets of the data is used. For BERT fine-tuned on QNLI, there is a decreasing trend that quickly tapers off. However, using a larger number of templates *reduces the variance* in average absolute effect size and *improves the stability* of the <span style="font-variant:small-caps;">Sent-Debias</span> algorithm. These observations allow us to conclude the importance of using a large *number* of templates from naturally occurring text corpora.

**2) Same Quantity, More Domains:** *How does the number of domains that sentence templates are extracted from impact debiasing performance?* We fix the total number of sentence templates to be $1080$ and vary the number of domains these templates are drawn from. Given a target number $k$, we first choose $k$ domains from our Reddit, SST, POM, WikiText-2 datasets and randomly sample $1080/k$ templates from each of the $k$ selected domains. We construct $1080$ templates using all possible subsets of $k$ domains and apply them in the contextualization step of <span style="font-variant:small-caps;">Sent-Debias</span>. We estimate the corresponding bias subspace, debias and measure the average absolute SEAT effect sizes. To see the relationship between the number of domains $k$ and debiasing performance, we group combinations with the same number of domains ($k$) and for each group compute the mean and standard deviation of the average absolute effect sizes. This experiment is also performed for BERT fine-tuned on SST-2 and QNLI datasets. Results are plotted in Figure \[fig:sent\_domain\].

We draw similar observations: there is a *decreasing trend* in effect size as templates are drawn from more domains. For BERT fine-tuned on QNLI, using a larger number of domains *reduces the variance* in effect size and *improves stability* of the algorithm. Therefore, it is important to use a large *variety* of templates across different domains.

Visualization
-------------

As a qualitative analysis of the debiasing process, we visualize how the distances between sentence representations shift after the debiasing process is performed. We average the sentence representations of a concept (e.g. *man, woman, science, art*) across its contexts (sentence templates) and plot the t-SNE [@vanDerMaaten2008] embeddings of these points in 2D space. From Figure \[fig:tsne\], we observe that BERT average representations of *science* and *technology* start off closer to *man* while *literature* and *art* are closer to *woman*. After debiasing, non gender-specific concepts (e.g *science, art*) become more equidistant to both *man* and *woman* average concepts.

Performance on Downstream Tasks
-------------------------------

To ensure that debiasing does not hurt the performance on downstream tasks, we report the performance of our debiased BERT and ELMo on SST-2 and CoLA by training a linear classifier on top of debiased BERT sentence representations. From Table \[downstream\], we observe that downstream task performance show a small decrease ranging from $1 - 3\%$ after the debiasing process. However, the performance of ELMo on SST-2 increases slightly from $89.6$ to $90.0$. We hypothesize that these differences in performance are due to the fact that CoLA tests for linguistic acceptability so it is more concerned with low-level syntactic structure such as verb usage, grammar, and tenses. As a result, changes in sentence representations across bias directions may impact its performance more. For example, sentence representations after the gender debiasing steps may display a mismatch between gendered pronouns and the sentence context. For SST, it has been shown that sentiment analysis datasets have labels that correlate with gender information and therefore contain gender bias [@kiritchenko-mohammad-2018-examining]. As a result, we do expect possible decreases in accuracy after debiasing. Finally, we test the effect of <span style="font-variant:small-caps;">Sent-Debias</span> on QNLI by training a classifier on top of debiased BERT sentence pair representations. We observe little impact on task performance: our debiased BERT fine-tuned on QNLI achieves $90.6\%$ performance as compared to the $91.3\%$ we obtained without debiasing.

Discussion {#discuss}
==========

We outline several limitations of our proposed approach as well as directions for future work

Limitations
-----------

Firstly, we would like to emphasize that both the WEAT, SEAT, and MAC metrics are not perfect since they only have positive predictive ability: they can be used to detect the presence of biases but not their absence [@gonen-goldberg-2019-lipstick]. This calls for new metrics that evaluate biases and can scale to the various types of sentences appearing across different individuals, topics, and in both spoken and written text. We believe that our positive results regarding contextualizing words into sentences implies that future work can build on our algorithms and tailor them for new metrics.

Secondly, a particular bias should only be removed from words and sentences that are neutral to that attribute. For example, gender bias should not be removed from the word "grandmother" or the sentence "she gave birth to me". Previous work on debiasing word representations tackled this issue by listing all attribute specific words based on dictionary definitions and only debiasing the remaining words. However, given the complexity of natural sentences, it is extremely hard to identify the set of neutral sentences and its complement. Thus, in downstream tasks, we removed bias from all sentences which could possibly harm downstream task performance if the dataset contains a significant number of non-neutral sentences.

Thirdly, a fundamental challenge lies in the fact that these representations are trained without explicit bias control mechanisms on large amounts of naturally occurring text. Given that it becomes infeasible (in standard settings) to completely retrain these large sentence encoders for debiasing @zhao-etal-2018-learning [@Zhang:2018:MUB:3278721.3278779], future work should focus on developing better post-hoc debiasing techniques. In our experiments, we need to re-estimate the bias subspace and perform debiasing whenever the BERT encoder was fine-tuned. It remains to be seen whether there are debiasing methods which are invariant to fine-tuning, or can be efficiently re-estimated as the encoders are fine-tuned.

Ethical Considerations
----------------------

Firstly, we would like to remind all users that our models, even after going through debiasing, are not perfect. They have only been examined in the definition of bias as stated in the paper and by related work [@bolukbasi2016man; @DBLP:journals/corr/abs-1903-10561], and even then, does NOT remove all biases measured. In addition, there are other definitions of bias [@gonen-goldberg-2019-lipstick] that researchers must also consider when applying these models to real data.

Furthermore, any model modified to have less bias can still potentially reproduce and amplify biases that appear in training data used for further fine-tuning. Therefore, we emphasize that using these model is NOT in itself enough and is not a one-size-fits-all method that automatically removes bias from sentence encoders. While we hope that some of our ideas can be useful in probing sentence encoders for bias, we strongly believe there still needs to be a critical investigation of the various ways in which bias can be introduced in each specific application these general pretrained models are used for.

Conclusion
==========

This paper investigated the post-hoc removal of social biases from pretrained sentence representations. We proposed the <span style="font-variant:small-caps;">Sent-Debias</span> method that accurately captures the bias subspace of sentence representations by using a diverse set of templates from naturally occurring text corpora. Our experiments show that we can remove biases that occur in BERT and ELMo while preserving performance on downstream tasks. We also demonstrate the importance of using a large number of diverse sentence templates when estimating bias subspaces. Leveraging these developments will allow researchers to further characterize and remove social biases from sentence representations for fairer NLP.

Acknowledgements {#acknowledgements .unnumbered}
================

PPL and LPM were supported in part by the National Science Foundation (Awards \#1750439, \#1722822) and National Institutes of Health. RS was supported in part by US Army, ONR, Apple, and NSF IIS1763562. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, National Institutes of Health, DARPA, and AFRL, and no official endorsement should be inferred. We also acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments.

[^1]: Although we recognize that gender is non-binary and there are many important ethical principles in the design, ascription of categories/variables to study participants, and reporting of results in studying gender as a variable in NLP [@larson-2017-gender], for the purpose of this study, we follow existing research and focus on female and male gendered terms.

[^2]: We used uncased BERT-Base throughout all experiments.

{'source': 'arXiv_out/2007'}