# embedding_metadata

key: chroma:document
string_value:
of race and gender, drawing on Black feminist
scholarship on intersectionality (Crenshaw, 1989).
Papers’ techniques are poorly matched to their
motivations. We found that although 21% of the
papers include allocational harms in their motivations, only four papers actually propose techniques
for measuring or mitigating allocational harms.
Papers focus on a narrow range of potential
sources of “bias.” We found that nearly all of the
papers focus on system predictions as the potential
sources of “bias,” with many additionally focusing
on “bias” in datasets (e.g., differences in the
number of gendered pronouns in the training data
(Zhao et al., 2019)). Most papers do not interrogate
the normative implications of other decisions made
during the development and deployment lifecycle—
perhaps unsurprising given that their motivations
sometimes include no normative reasoning. A
few papers are exceptions, illustrating the impacts
of task deﬁnitions, annotation guidelines, and
evaluation metrics: Cao and Daumé (2019) study
how folk conceptions of gender (Keyes, 2018) are
reproduced in coreference resolution systems that
assume a strict gender dichotomy, thereby maintaining cisnormativity; Sap et al. (2019) focus on
the effect of priming annotators with information


key: chroma:document
string_value:
assume a strict gender dichotomy, thereby maintaining cisnormativity; Sap et al. (2019) focus on
the effect of priming annotators with information
about possible dialectal differences when asking
them to apply toxicity labels to sample tweets, ﬁnding that annotators who are primed are signiﬁcantly
less likely to label tweets containing features associated with African-American English as offensive.
4 A path forward
We now describe how researchers and practitioners
conducting work analyzing “bias” in NLP systems
might avoid the pitfalls presented in the previous
section—the beginnings of a path forward. We
propose three recommendations that should guide
such work, and, for each, provide several concrete
research questions. We emphasize that these questions are not comprehensive, and are intended to
generate further questions and lines of engagement.
Our three recommendations are as follows:
(R1) Ground work analyzing “bias” in NLP systems in the relevant literature outside of NLP
that explores the relationships between language and social hierarchies. Treat representational harms as harmful in their own right.
(R2) Provide explicit statements of why the
system behaviors that are described as “bias”
are harmful, in what ways, and to whom.
Be forthright about the normative reasoning
(Green, 2019) underlying these statements.


key: chroma:document
string_value:
are harmful, in what ways, and to whom.
Be forthright about the normative reasoning
(Green, 2019) underlying these statements.
(R3) Examine language use in practice by engaging with the lived experiences of members of
communities affected by NLP systems. Interrogate and reimagine the power relations between technologists and such communities.
4.1 Language and social hierarchies
Turning ﬁrst to (R1) , we argue that work analyzing
“bias” in NLP systems will paint a much fuller picture if it engages with the relevant literature outside
of NLP that explores the relationships betweenlanguage and social hierarchies. Many disciplines,
including sociolinguistics, linguistic anthropology,
sociology, and social psychology, study how
language takes on social meaning and the role that
language plays in maintaining social hierarchies.
For example, language is the means through which
social groups are labeled and one way that beliefs
about social groups are transmitted (e.g., Maass,
1999; Beukeboom and Burgers, 2019). Group
labels can serve as the basis of stereotypes and thus
reinforce social inequalities: “[T]he label content
functions to identify a given category of people,
and thereby conveys category boundaries and a
position in a hierarchical taxonomy” (Beukeboom


key: chroma:document
string_value:
functions to identify a given category of people,
and thereby conveys category boundaries and a
position in a hierarchical taxonomy” (Beukeboom
and Burgers, 2019). Similarly, “controlling
images,” such as stereotypes of Black women,
which are linguistically and visually transmitted
through literature, news media, television, and so
forth, provide “ideological justiﬁcation” for their
continued oppression (Collins, 2000, Chapter 4).
As a result, many groups have sought to bring
about social changes through changes in language,
disrupting patterns of oppression and marginalization via so-called “gender-fair” language
(Sczesny et al., 2016; Menegatti and Rubini, 2017),
language that is more inclusive to people with
disabilities (ADA, 2018), and language that is less
dehumanizing (e.g., abandoning the use of the term
“illegal” in everyday discourse on immigration in
the U.S. (Rosa, 2019)). The fact that group labels
are so contested is evidence of how deeply intertwined language and social hierarchies are. Taking
“gender-fair” language as an example, the hope
is that reducing asymmetries in language about
women and men will reduce asymmetries in their


key: chroma:document
string_value:
“gender-fair” language as an example, the hope
is that reducing asymmetries in language about
women and men will reduce asymmetries in their
social standing. Meanwhile, struggles over language use often arise from dominant social groups’
desire to “control both material and symbolic
resources”—i.e., “the right to decide what words
will mean and to control those meanings”—as was
the case in some white speakers’ insistence on
using offensive place names against the objections
of Indigenous speakers (Hill, 2008, Chapter 3).
Sociolinguists and linguistic anthropologists
have also examined language attitudes and language ideologies, or people’s metalinguistic beliefs
about language: Which language varieties or practices are taken as standard, ordinary, or unmarked?
Which are considered correct, prestigious, or appropriate for public use, and which are considered
incorrect, uneducated, or offensive (e.g., CampbellKibler, 2009; Preston, 2009; Loudermilk, 2015;
Lanehart and Malik, 2018)? Which are rendered invisible (Roche, 2019)?3Language ideologies play
a vital role in reinforcing and justifying social hierarchies because beliefs about language varieties
or practices often translate into beliefs about their
speakers (e.g. Alim et al., 2016; Rosa and Flores,


key: chroma:document
string_value:
or practices often translate into beliefs about their
speakers (e.g. Alim et al., 2016; Rosa and Flores,
2017; Craft et al., 2020). For example, in the U.S.,
the portrayal of non-white speakers’ language
varieties and practices as linguistically deﬁcient
helped to justify violent European colonialism, and
today continues to justify enduring racial hierarchies by maintaining views of non-white speakers
as lacking the language “required for complex
thinking processes and successful engagement
in the global economy” (Rosa and Flores, 2017).
Recognizing the role that language plays in
maintaining social hierarchies is critical to the
future of work analyzing “bias” in NLP systems.
First, it helps to explain why representational
harms are harmful in their own right. Second, the
complexity of the relationships between language
and social hierarchies illustrates why studying
“bias” in NLP systems is so challenging, suggesting
that researchers and practitioners will need to move
beyond existing algorithmic fairness techniques.
We argue that work must be grounded in the
relevant literature outside of NLP that examines
the relationships between language and social
hierarchies; without this grounding, researchers
and practitioners risk measuring or mitigating
only what is convenient to measure or mitigate,


key: chroma:document
string_value:
the relationships between language and social
hierarchies; without this grounding, researchers
and practitioners risk measuring or mitigating
only what is convenient to measure or mitigate,
rather than what is most normatively concerning.
More speciﬁcally, we recommend that work
analyzing “bias” in NLP systems be reoriented
around the following question: How are social
hierarchies, language ideologies, and NLP systems
coproduced? This question mirrors Benjamin’s
(2020) call to examine how “race and technology
are coproduced”—i.e., how racial hierarchies, and
the ideologies and discourses that maintain them,
create and are re-created by technology. We recommend that researchers and practitioners similarly
ask how existing social hierarchies and language
ideologies drive the development and deployment
of NLP systems, and how these systems therefore
reproduce these hierarchies and ideologies. As
a starting point for reorienting work analyzing
“bias” in NLP systems around this question, we
3Language ideologies encompass much more than this; see,
e.g., Lippi-Green (2012), Alim et al. (2016), Rosa and Flores
(2017), Rosa and Burdick (2017), and Charity Hudley (2017).provide the following concrete research questions:


key: chroma:document
string_value:
(2017), Rosa and Burdick (2017), and Charity Hudley (2017).provide the following concrete research questions:
.How do social hierarchies and language
ideologies inﬂuence the decisions made during
the development and deployment lifecycle?
What kinds of NLP systems do these decisions
result in, and what kinds do they foreclose?
General assumptions: To which linguistic
norms do NLP systems adhere (Bender,
2019; Ruane et al., 2019)? Which language
practices are implicitly assumed to be
standard, ordinary, correct, or appropriate?
Task deﬁnition: For which speakers
are NLP systems (and NLP resources)
developed? (See Joshi et al. (2020) for
a discussion.) How do task deﬁnitions
discretize the world? For example, how
are social groups delineated when deﬁning
demographic attribute prediction tasks
(e.g., Koppel et al., 2002; Rosenthal and
McKeown, 2011; Nguyen et al., 2013)?
What about languages in native language
prediction tasks (Tetreault et al., 2013)?
Data: How are datasets collected, preprocessed, and labeled or annotated? What are
the impacts of annotation guidelines, annotator assumptions and perceptions (Olteanu


key: chroma:document
string_value:
Data: How are datasets collected, preprocessed, and labeled or annotated? What are
the impacts of annotation guidelines, annotator assumptions and perceptions (Olteanu
et al., 2019; Sap et al., 2019; Geiger et al.,
2020), and annotation aggregation processes (Pavlick and Kwiatkowski, 2019)?
Evaluation: How are NLP systems evaluated? What are the impacts of evaluation
metrics (Olteanu et al., 2017)? Are any
non-quantitative evaluations performed?
.How do NLP systems reproduce or transform
language ideologies? Which language varieties
or practices come to be deemed good or bad?
Might “good” language simply mean language
that is easily handled by existing NLP systems? For example, linguistic phenomena arising from many language practices (Eisenstein,
2013) are described as “noisy text” and often
viewed as a target for “normalization.” How
do the language ideologies that are reproduced
by NLP systems maintain social hierarchies?
.Which representational harms are being
measured or mitigated? Are these the most
normatively concerning harms, or merely
those that are well handled by existing algorithmic fairness techniques? Are there other
representational harms that might be analyzed?
4.2 Conceptualizations of “bias”


key: chroma:document
string_value:
those that are well handled by existing algorithmic fairness techniques? Are there other
representational harms that might be analyzed?
4.2 Conceptualizations of “bias”
Turning now to (R2) , we argue that work analyzing
“bias” in NLP systems should provide explicit
statements of why the system behaviors that are
described as “bias” are harmful, in what ways,
and to whom, as well as the normative reasoning
underlying these statements. In other words,
researchers and practitioners should articulate their
conceptualizations of “bias.” As we described
above, papers often contain descriptions of system
behaviors that are understood to be self-evident
statements of “bias.” This use of imprecise
terminology has led to papers all claiming to
analyze “bias” in NLP systems, sometimes even
in systems developed for the same task, but with
different or even inconsistent conceptualizations of
“bias,” and no explanations for these differences.
Yet analyzing “bias” is an inherently normative
process—in which some system behaviors are
deemed good and others harmful—even if assumptions about what kinds of system behaviors are
harmful, in what ways, for whom, and why are


key: chroma:document
string_value:
process—in which some system behaviors are
deemed good and others harmful—even if assumptions about what kinds of system behaviors are
harmful, in what ways, for whom, and why are
not stated. We therefore echo calls by Bardzell and
Bardzell (2011), Keyes et al. (2019), and Green
(2019) for researchers and practitioners to make
their normative reasoning explicit by articulating
the social values that underpin their decisions to
deem some system behaviors as harmful, no matter
how obvious such values appear to be. We further
argue that this reasoning should take into account
the relationships between language and social
hierarchies that we described above. First, these
relationships provide a foundation from which to
approach the normative reasoning that we recommend making explicit. For example, some system
behaviors might be harmful precisely because
they maintain social hierarchies. Second, if work
analyzing “bias” in NLP systems is reoriented
to understand how social hierarchies, language
ideologies, and NLP systems are coproduced, then
this work will be incomplete if we fail to account
for the ways that social hierarchies and language
ideologies determine what we mean by “bias” in
the ﬁrst place. As a starting point, we therefore
provide the following concrete research questions:


key: chroma:document
string_value:
ideologies determine what we mean by “bias” in
the ﬁrst place. As a starting point, we therefore
provide the following concrete research questions:
.What kinds of system behaviors are described
as “bias”? What are their potential sources (e.g.,
general assumptions, task deﬁnition, data)?
.In what ways are these system behaviors harmful, to whom are they harmful, and why?
.What are the social values (obvious or not) thatunderpin this conceptualization of “bias?”
4.3 Language use in practice
Finally, we turn to (R3). Our perspective, which
rests on a greater recognition of the relationships
between language and social hierarchies, suggests
several directions for examining language use in
practice. Here, we focus on two. First, because language is necessarily situated, and because different
social groups have different lived experiences due
to their different social positions (Hanna et al.,
2020)—particularly groups at the intersections
of multiple axes of oppression—we recommend
that researchers and practitioners center work
analyzing “bias” in NLP systems around the lived
experiences of members of communities affected
by these systems. Second, we recommend that
the power relations between technologists and
such communities be interrogated and reimagined.


key: chroma:document
string_value:
experiences of members of communities affected
by these systems. Second, we recommend that
the power relations between technologists and
such communities be interrogated and reimagined.
Researchers have pointed out that algorithmic
fairness techniques, by proposing incremental
technical mitigations—e.g., collecting new datasets
or training better models—maintain these power
relations by (a) assuming that automated systems
should continue to exist, rather than asking
whether they should be built at all, and (b) keeping
development and deployment decisions in the
hands of technologists (Bennett and Keyes, 2019;
Cifor et al., 2019; Green, 2019; Katell et al., 2020).
There are many disciplines for researchers and
practitioners to draw on when pursuing these
directions. For example, in human–computer
interaction, Hamidi et al. (2018) study transgender
people’s experiences with automated gender
recognition systems in order to uncover how
these systems reproduce structures of transgender
exclusion by redeﬁning what it means to perform
gender “normally.” Value-sensitive design provides
a framework for accounting for the values of different stakeholders in the design of technology (e.g.,
Friedman et al., 2006; Friedman and Hendry, 2019;
Le Dantec et al., 2009; Yoo et al., 2019), while


key: chroma:document
string_value:
Friedman et al., 2006; Friedman and Hendry, 2019;
Le Dantec et al., 2009; Yoo et al., 2019), while
participatory design seeks to involve stakeholders
in the design process itself (Sanders, 2002; Muller,
2007; Simonsen and Robertson, 2013; DiSalvo
et al., 2013). Participatory action research in education (Kemmis, 2006) and in language documentation and reclamation (Junker, 2018) is also relevant.
In particular, work on language reclamation to
support decolonization and tribal sovereignty
(Leonard, 2012) and work in sociolinguistics focusing on developing co-equal research relationships
with community members and supporting linguistic justice efforts (e.g., Bucholtz et al., 2014, 2016,
2019) provide examples of more emancipatory relationships with communities. Finally, several workshops and events have begun to explore how to empower stakeholders in the development and deployment of technology (Vaccaro et al., 2019; Givens
and Morris, 2020; Sassaman et al., 2020)4and how
to help researchers and practitioners consider when
not to build systems at all (Barocas et al., 2020).
As a starting point for engaging with communities affected by NLP systems, we therefore
provide the following concrete research questions:
.How do communities become aware of NLP


key: chroma:document
string_value:
As a starting point for engaging with communities affected by NLP systems, we therefore
provide the following concrete research questions:
.How do communities become aware of NLP
systems? Do they resist them, and if so, how?
.What additional costs are borne by communities for whom NLP systems do not work well?
.Do NLP systems shift power toward oppressive
institutions (e.g., by enabling predictions that
communities do not want made, linguistically
based unfair allocation of resources or opportunities (Rosa and Flores, 2017), surveillance,
or censorship), or away from such institutions?
.Who is involved in the development and
deployment of NLP systems? How do
decision-making processes maintain power relations between technologists and communities
affected by NLP systems? Can these processes be changed to reimagine these relations?
5 Case study
To illustrate our recommendations, we present a
case study covering work on African-American
English (AAE).5Work analyzing “bias” in the context of AAE has shown that part-of-speech taggers,
language identiﬁcation systems, and dependency
parsers all work less well on text containing
features associated with AAE than on text without
these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection


key: chroma:document
string_value:
features associated with AAE than on text without
these features (Jørgensen et al., 2015, 2016; Blodgett et al., 2016, 2018), and that toxicity detection
systems score tweets containing features associated
with AAE as more offensive than tweets without them (Davidson et al., 2019; Sap et al., 2019).
These papers have been critical for highlighting
AAE as a language variety for which existing NLP
4Alsohttps://participatoryml.github.io/
5This language variety has had many different names
over the years, but is now generally called AfricanAmerican English (AAE), African-American Vernacular English (AA VE), or African-American Language (AAL) (Green,
2002; Wolfram and Schilling, 2015; Rickford and King, 2016).systems may not work, illustrating their limitations.
However, they do not conceptualize “racial bias” in
the same way. The ﬁrst four of these papers simply
focus on system performance differences between
text containing features associated with AAE and
text without these features. In contrast, the last
two papers also focus on such system performance
differences, but motivate this focus with the following additional reasoning: If tweets containing
features associated with AAE are scored as more
offensive than tweets without these features, then
this might (a) yield negative perceptions of AAE;


key: chroma:document
string_value:
features associated with AAE are scored as more
offensive than tweets without these features, then
this might (a) yield negative perceptions of AAE;
(b) result in disproportionate removal of tweets
containing these features, impeding participation
in online platforms and reducing the space available online in which speakers can use AAE freely;
and (c) cause AAE speakers to incur additional
costs if they have to change their language practices
to avoid negative perceptions or tweet removal.
More importantly, none of these papers engage
with the literature on AAE, racial hierarchies in the
U.S., and raciolinguistic ideologies. By failing to
engage with this literature—thereby treating AAE
simply as one of many non-Penn Treebank varieties of English or perhaps as another challenging
domain—work analyzing “bias” in NLP systems
in the context of AAE fails to situate these systems
in the world. Who are the speakers of AAE? How
are they viewed? We argue that AAE as a language
variety cannot be separated from its speakers—
primarily Black people in the U.S., who experience
systemic anti-Black racism—and the language ideologies that reinforce and justify racial hierarchies.
Even after decades of sociolinguistic efforts to
legitimize AAE, it continues to be viewed as “bad”


key: chroma:document
string_value:
Even after decades of sociolinguistic efforts to
legitimize AAE, it continues to be viewed as “bad”
English and its speakers continue to be viewed as
linguistically inadequate—a view called the deﬁcit
perspective (Alim et al., 2016; Rosa and Flores,
2017). This perspective persists despite demonstrations that AAE is rule-bound and grammatical
(Mufwene et al., 1998; Green, 2002), in addition
to ample evidence of its speakers’ linguistic adroitness (e.g., Alim, 2004; Rickford and King, 2016).
This perspective belongs to a broader set of raciolinguistic ideologies (Rosa and Flores, 2017), which
also produce allocational harms; speakers of AAE
are frequently penalized for not adhering to dominant language practices, including in the education
system (Alim, 2004; Terry et al., 2010), when
seeking housing (Baugh, 2018), and in the judicial
system, where their testimony is misunderstood or,
worse yet, disbelieved (Rickford and King, 2016;
Jones et al., 2019). These raciolinguistic ideologies
position racialized communities as needing
linguistic intervention, such as language education
programs, in which these and other harms can be
reduced if communities accommodate to dominant language practices (Rosa and Flores, 2017).


key: chroma:document
string_value:
linguistic intervention, such as language education
programs, in which these and other harms can be
reduced if communities accommodate to dominant language practices (Rosa and Flores, 2017).
In the technology industry, speakers of AAE are
often not considered consumers who matter. For
example, Benjamin (2019) recounts an Apple employee who worked on speech recognition for Siri:
“As they worked on different English dialects —
Australian, Singaporean, and Indian English — [the
employee] asked his boss: ‘What about African
American English?’ To this his boss responded:
‘Well, Apple products are for the premium market. ”’
The reality, of course, is that speakers of AAE tend
not to represent the “premium market” precisely because of institutions and policies that help to maintain racial hierarchies by systematically denying
them the opportunities to develop wealth that are
available to white Americans (Rothstein, 2017)—
an exclusion that is reproduced in technology by
countless decisions like the one described above.
Engaging with the literature outlined above
situates the system behaviors that are described
as “bias,” providing a foundation for normative
reasoning. Researchers and practitioners should
be concerned about “racial bias” in toxicity
detection systems not only because performance
differences impair system performance, but


key: chroma:document
string_value:
reasoning. Researchers and practitioners should
be concerned about “racial bias” in toxicity
detection systems not only because performance
differences impair system performance, but
because they reproduce longstanding injustices of
stigmatization and disenfranchisement for speakers
of AAE. In re-stigmatizing AAE, they reproduce
language ideologies in which AAE is viewed as
ungrammatical, uneducated, and offensive. These
ideologies, in turn, enable linguistic discrimination
and justify enduring racial hierarchies (Rosa and
Flores, 2017). Our perspective, which understands
racial hierarchies and raciolinguistic ideologies as
structural conditions that govern the development
and deployment of technology, implies that
techniques for measuring or mitigating “bias”
in NLP systems will necessarily be incomplete
unless they interrogate and dismantle these
structural conditions, including the power relations
between technologists and racialized communities.
We emphasize that engaging with the literature
on AAE, racial hierarchies in the U.S., and
raciolinguistic ideologies can generate new lines of
engagement. These lines include work on the ways
that the decisions made during the developmentand deployment of NLP systems produce stigmatization and disenfranchisement, and work on AAE
use in practice, such as the ways that speakers
of AAE interact with NLP systems that were not


key: chroma:document
string_value:
use in practice, such as the ways that speakers
of AAE interact with NLP systems that were not
designed for them. This literature can also help researchers and practitioners address the allocational
harms that may be produced by NLP systems, and
ensure that even well-intentioned NLP systems
do not position racialized communities as needing
linguistic intervention or accommodation to
dominant language practices. Finally, researchers
and practitioners wishing to design better systems
can also draw on a growing body of work on
anti-racist language pedagogy that challenges the
deﬁcit perspective of AAE and other racialized
language practices (e.g. Flores and Chaparro, 2018;
Baker-Bell, 2019; Martínez and Mejía, 2019), as
well as the work that we described in section 4.3
on reimagining the power relations between technologists and communities affected by technology.
6 Conclusion
By surveying 146 papers analyzing “bias” in NLP
systems, we found that (a) their motivations are
often vague, inconsistent, and lacking in normative reasoning; and (b) their proposed quantitative
techniques for measuring or mitigating “bias” are
poorly matched to their motivations and do not engage with the relevant literature outside of NLP.
To help researchers and practitioners avoid these


key: chroma:document
string_value:
poorly matched to their motivations and do not engage with the relevant literature outside of NLP.
To help researchers and practitioners avoid these
pitfalls, we proposed three recommendations that
should guide work analyzing “bias” in NLP systems, and, for each, provided several concrete research questions. These recommendations rest on
a greater recognition of the relationships between
language and social hierarchies—a step that we
see as paramount to establishing a path forward.
Acknowledgments
This paper is based upon work supported by the
National Science Foundation Graduate Research
Fellowship under Grant No. 1451512. Any opinion, ﬁndings, and conclusions or recommendations
expressed in this material are those of the authors
and do not necessarily reﬂect the views of the National Science Foundation. We thank the reviewers
for their useful feedback, especially the suggestion to include additional details about our method.
References
Artem Abzaliev. 2019. On GAP coreference resolution shared task: insights from the 3rd place solution.
InProceedings of the Workshop on Gender Bias in
Natural Language Processing , pages 107–112, Florence, Italy.
ADA. 2018. Guidelines for Writing About People With Disabilities. ADA National Network.
https://bit.ly/2KREbkB.
Oshin Agarwal, Funda Durupinar, Norman I. Badler,


key: chroma:document
string_value:
https://bit.ly/2KREbkB.
Oshin Agarwal, Funda Durupinar, Norman I. Badler,
and Ani Nenkova. 2019. Word embeddings (also)
encode human personality stereotypes. In Proceedings of the Joint Conference on Lexical and Computational Semantics , pages 205–211, Minneapolis,
MN.
H. Samy Alim. 2004. You Know My Steez: An Ethnographic and Sociolinguistic Study of Styleshifting in
a Black American Speech Community . American Dialect Society.
H. Samy Alim, John R. Rickford, and Arnetha F. Ball,
editors. 2016. Raciolinguistics: How Language
Shapes Our Ideas About Race . Oxford University
Press.
Sandeep Attree. 2019. Gendered ambiguous pronouns
shared task: Boosting model conﬁdence by evidence
pooling. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , Florence,
Italy.
Pinkesh Badjatiya, Manish Gupta, and Vasudeva
Varma. 2019. Stereotypical bias removal for hate
speech detection task using knowledge-based generalizations. In Proceedings of the International
World Wide Web Conference , pages 49–59, San Francisco, CA.


key: chroma:document
string_value:
speech detection task using knowledge-based generalizations. In Proceedings of the International
World Wide Web Conference , pages 49–59, San Francisco, CA.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly
Shmatikov. 2019. Differential Privacy Has Disparate Impact on Model Accuracy. In Proceedings
of the Conference on Neural Information Processing
Systems , Vancouver, Canada.
April Baker-Bell. 2019. Dismantling anti-black linguistic racism in English language arts classrooms:
Toward an anti-racist black language pedagogy. Theory Into Practice .
David Bamman, Sejal Popat, and Sheng Shen. 2019.
An annotated dataset of literary entities. In Proceedings of the North American Association for Computational Linguistics (NAACL) , pages 2138–2144,
Minneapolis, MN.
Xingce Bao and Qianqian Qiao. 2019. Transfer Learning from Pre-trained BERT for Pronoun Resolution.
InProceedings of the Workshop on Gender Bias
in Natural Language Processing , pages 82–88, Florence, Italy.Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards
a Feminist HCI Methodology: Social Science, Feminism, and HCI. In Proceedings of the Conference on
Human Factors in Computing Systems (CHI) , pages


key: chroma:document
string_value:
a Feminist HCI Methodology: Social Science, Feminism, and HCI. In Proceedings of the Conference on
Human Factors in Computing Systems (CHI) , pages
675–684, Vancouver, Canada.
Solon Barocas, Asia J. Biega, Benjamin Fish, J˛ edrzej
Niklas, and Luke Stark. 2020. When Not to Design, Build, or Deploy. In Proceedings of the Conference on Fairness, Accountability, and Transparency ,
Barcelona, Spain.
Solon Barocas, Kate Crawford, Aaron Shapiro, and
Hanna Wallach. 2017. The Problem With Bias: Allocative Versus Representational Harms in Machine
Learning. In Proceedings of SIGCIS , Philadelphia,
PA.
Christine Basta, Marta R. Costa-jussà, and Noe Casas.
2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of
the Workshop on Gender Bias for Natural Language
Processing , pages 33–39, Florence, Italy.
John Baugh. 2018. Linguistics in Pursuit of Justice .
Cambridge University Press.
Emily M. Bender. 2019. A typology of ethical risks
in language technology with an eye towards where
transparent documentation can help. Presented at
The Future of Artiﬁcial Intelligence: Language,


key: chroma:document
string_value:
in language technology with an eye towards where
transparent documentation can help. Presented at
The Future of Artiﬁcial Intelligence: Language,
Ethics, Technology Workshop. https://bit.ly/
2P9t9M6 .
Ruha Benjamin. 2019. Race After Technology: Abolitionist Tools for the New Jim Code . John Wiley &
Sons.
Ruha Benjamin. 2020. 2020 Vision: Reimagining the
Default Settings of Technology & Society. Keynote
at ICLR.
Cynthia L. Bennett and Os Keyes. 2019. What is the
Point of Fairness? Disability, AI, and The Complexity of Justice. In Proceedings of the ASSETS
Workshop on AI Fairness for People with Disabilities, Pittsburgh, PA.
Camiel J. Beukeboom and Christian Burgers. 2019.
How Stereotypes Are Shared Through Language: A
Review and Introduction of the Social Categories
and Stereotypes Communication (SCSC) Framework. Review of Communication Research , 7:1–37.
Shruti Bhargava and David Forsyth. 2019. Exposing and Correcting the Gender Bias in Image
Captioning Datasets and Models. arXiv preprint
arXiv:1912.00578 .
Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good


key: chroma:document
string_value:
arXiv:1912.00578 .
Jayadev Bhaskaran and Isha Bhallamudi. 2019. Good
Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in Sentiment Analysis. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 62–68, Florence, Italy.
Su Lin Blodgett, Lisa Green, and Brendan O’Connor.
2016. Demographic Dialectal Variation in Social
Media: A Case Study of African-American English.
InProceedings of Empirical Methods in Natural
Language Processing (EMNLP) , pages 1119–1130,
Austin, TX.
Su Lin Blodgett and Brendan O’Connor. 2017. Racial
Disparity in Natural Language Processing: A Case
Study of Social Media African-American English.
InProceedings of the Workshop on Fairness, Accountability, and Transparency in Machine Learning
(FAT/ML) , Halifax, Canada.
Su Lin Blodgett, Johnny Wei, and Brendan O’Connor.
2018. Twitter Universal Dependency Parsing for
African-American and Mainstream American English. In Proceedings of the Association for Computational Linguistics (ACL) , pages 1415–1425, Melbourne, Australia.
Tolga Bolukbasi, Kai-Wei Chang, James Zou,


key: chroma:document
string_value:
Tolga Bolukbasi, Kai-Wei Chang, James Zou,
Venkatesh Saligrama, and Adam Kalai. 2016a.
Man is to Computer Programmer as Woman is to
Homemaker? Debiasing Word Embeddings. In Proceedings of the Conference on Neural Information
Processing Systems , pages 4349–4357, Barcelona,
Spain.
Tolga Bolukbasi, Kai-Wei Chang, James Zou,
Venkatesh Saligrama, and Adam Kalai. 2016b.
Quantifying and reducing stereotypes in word
embeddings. In Proceedings of the ICML Workshop
on #Data4Good: Machine Learning in Social Good
Applications , pages 41–45, New York, NY .
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language
models. In Proceedings of the NAACL Student Research Workshop , pages 7–15, Minneapolis, MN.
Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the Origins of Bias in Word Embeddings.
InProceedings of the International Conference on
Machine Learning , pages 803–811, Long Beach,
CA.
Mary Bucholtz, Dolores Inés Casillas, and Jin Sook


key: chroma:document
string_value:
Machine Learning , pages 803–811, Long Beach,
CA.
Mary Bucholtz, Dolores Inés Casillas, and Jin Sook
Lee. 2016. Beyond Empowerment: Accompaniment and Sociolinguistic Justice in a Youth Research
Program. In Robert Lawson and Dave Sayers, editors, Sociolinguistic Research: Application and Impact, pages 25–44. Routledge.
Mary Bucholtz, Dolores Inés Casillas, and Jin Sook
Lee. 2019. California Latinx Youth as Agents of
Sociolinguistic Justice. In Netta Avineri, Laura R.
Graham, Eric J. Johnson, Robin Conley Riner, and
Jonathan Rosa, editors, Language and Social Justice
in Practice , pages 166–175. Routledge.
Mary Bucholtz, Audrey Lopez, Allina Mojarro, Elena
Skapoulli, Chris VanderStouwe, and Shawn WarnerGarcia. 2014. Sociolinguistic Justice in the Schools:Student Researchers as Linguistic Experts. Language and Linguistics Compass , 8:144–157.
Kaylee Burns, Lisa Anne Hendricks, Kate Saenko,
Trevor Darrell, and Anna Rohrbach. 2018. Women
also Snowboard: Overcoming Bias in Captioning
Models. In Procedings of the European Conference


key: chroma:document
string_value:
Trevor Darrell, and Anna Rohrbach. 2018. Women
also Snowboard: Overcoming Bias in Captioning
Models. In Procedings of the European Conference
on Computer Vision (ECCV) , pages 793–811, Munich, Germany.
Aylin Caliskan, Joanna J. Bryson, and Arvind
Narayanan. 2017. Semantics derived automatically
from language corpora contain human-like biases.
Science , 356(6334).
Kathryn Campbell-Kibler. 2009. The nature of sociolinguistic perception. Language Variation and
Change , 21(1):135–156.
Yang Trista Cao and Hal Daumé, III. 2019. Toward gender-inclusive coreference resolution. arXiv
preprint arXiv:1910.13913 .
Rakesh Chada. 2019. Gendered pronoun resolution using bert and an extractive question answering formulation. In Proceedings of the Workshop on Gender
Bias in Natural Language Processing , pages 126–
133, Florence, Italy.
Kaytlin Chaloner and Alfredo Maldonado. 2019. Measuring Gender Bias in Word Embedding across Domains and Discovering New Gender Bias Word Categories. In Proceedings of the Workshop on Gender
Bias in Natural Language Processing , pages 25–32,
Florence, Italy.


key: chroma:document
string_value:
Bias in Natural Language Processing , pages 25–32,
Florence, Italy.
Anne H. Charity Hudley. 2017. Language and Racialization. In Ofelia García, Nelson Flores, and Massimiliano Spotti, editors, The Oxford Handbook of
Language and Society . Oxford University Press.
Won Ik Cho, Ji Won Kim, Seok Min Kim, and
Nam Soo Kim. 2019. On measuring gender bias in
translation of gender-neutral pronouns. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 173–181, Florence, Italy.
Shivang Chopra, Ramit Sawhney, Puneet Mathur, and
Rajiv Ratn Shah. 2020. Hindi-English Hate Speech
Detection: Author Proﬁling, Debiasing, and Practical Perspectives. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence (AAAI) , New York,
NY .
Marika Cifor, Patricia Garcia, T.L. Cowan, Jasmine
Rault, Tonia Sutherland, Anita Say Chan, Jennifer
Rode, Anna Lauren Hoffmann, Niloufar Salehi, and
Lisa Nakamura. 2019. Feminist Data ManifestNo. Retrieved from https://www.manifestno.
com/ .
Patricia Hill Collins. 2000. Black Feminist Thought:


key: chroma:document
string_value:
Lisa Nakamura. 2019. Feminist Data ManifestNo. Retrieved from https://www.manifestno.
com/ .
Patricia Hill Collins. 2000. Black Feminist Thought:
Knowledge, Consciousness, and the Politics of Empowerment . Routledge.
Justin T. Craft, Kelly E. Wright, Rachel Elizabeth
Weissler, and Robin M. Queen. 2020. Language
and Discrimination: Generating Meaning, Perceiving Identities, and Discriminating Outcomes. Annual Review of Linguistics , 6(1).
Kate Crawford. 2017. The Trouble with Bias. Keynote
at NeurIPS.
Kimberle Crenshaw. 1989. Demarginalizing the Intersection of Race and Sex: A Black Feminist Critique
of Antidiscrmination Doctrine, Feminist Theory and
Antiracist Politics. University of Chicago Legal Forum.
Amanda Cercas Curry and Verena Rieser. 2018.
#MeToo: How Conversational Systems Respond to
Sexual Harassment. In Proceedings of the Workshop
on Ethics in Natural Language Processing , pages 7–
14, New Orleans, LA.
Karan Dabas, Nishtha Madaan, Gautam Singh, Vijay Arya, Sameep Mehta, and Tanmoy Chakraborty.
2020. Fair Transfer of Multiple Style Attributes in


key: chroma:document
string_value:
2020. Fair Transfer of Multiple Style Attributes in
Text. arXiv preprint arXiv:2001.06693 .
Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and
abusive language detection datasets. In Proceedings
of the Workshop on Abusive Language Online , pages
25–35, Florence, Italy.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra
Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. 2019. Bias in bios:
A case study of semantic representation bias in a
high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency ,
pages 120–128, Atlanta, GA.
Sunipa Dev, Tao Li, Jeff Phillips, and Vivek Srikumar. 2019. On Measuring and Mitigating Biased
Inferences of Word Embeddings. arXiv preprint
arXiv:1908.09369 .
Sunipa Dev and Jeff Phillips. 2019. Attenuating Bias in
Word Vectors. In Proceedings of the International
Conference on Artiﬁcial Intelligence and Statistics ,
pages 879–887, Naha, Japan.


key: chroma:document
string_value:
Word Vectors. In Proceedings of the International
Conference on Artiﬁcial Intelligence and Statistics ,
pages 879–887, Naha, Japan.
Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie
Piper, and Darren Gergle. 2018. Addressing agerelated bias in sentiment analysis. In Proceedings
of the Conference on Human Factors in Computing
Systems (CHI) , Montréal, Canada.
Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2019.
Queens are Powerful too: Mitigating Gender
Bias in Dialogue Generation. arXiv preprint
arXiv:1911.03842 .
Carl DiSalvo, Andrew Clement, and V olkmar Pipek.
2013. Communities: Participatory Design for, with
and by communities. In Jesper Simonsen and ToniRobertson, editors, Routledge International Handbook of Participatory Design , pages 182–209. Routledge.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain,
and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classiﬁcation. In Proceedings of the Conference on Artiﬁcial Intelligence,
Ethics, and Society (AIES) , New Orleans, LA.


key: chroma:document
string_value:
Ethics, and Society (AIES) , New Orleans, LA.
Jacob Eisenstein. 2013. What to do about bad language on the Internet. In Proceedings of the North
American Association for Computational Linguistics
(NAACL) , pages 359–369.
Kawin Ethayarajh. 2020. Is Your Classiﬁer Actually
Biased? Measuring Fairness under Uncertainty with
Bernstein Bounds. In Proceedings of the Association for Computational Linguistics (ACL) .
Kawin Ethayarajh, David Duvenaud, and Graeme Hirst.
2019. Understanding Undesirable Word Embedding
Assocations. In Proceedings of the Association for
Computational Linguistics (ACL) , pages 1696–1705,
Florence, Italy.
Joseph Fisher. 2019. Measuring social bias in
knowledge graph embeddings. arXiv preprint
arXiv:1912.02761 .
Nelson Flores and Soﬁa Chaparro. 2018. What counts
as language education policy? Developing a materialist Anti-racist approach to language activism. Language Policy , 17(3):365–384.
Omar U. Florez. 2019. On the Unintended Social Bias
of Training Language Generation Models with Data


key: chroma:document
string_value:
Omar U. Florez. 2019. On the Unintended Social Bias
of Training Language Generation Models with Data
from Local Media. In Proceedings of the NeurIPS
Workshop on Human-Centric Machine Learning ,
Vancouver, Canada.
Joel Escudé Font and Marta R. Costa-jussà. 2019.
Equalizing gender biases in neural machine translation with word embeddings techniques. In Proceedings of the Workshop on Gender Bias for Natural Language Processing , pages 147–154, Florence,
Italy.
Batya Friedman and David G. Hendry. 2019. Value
Sensitive Design: Shaping Technology with Moral
Imagination . MIT Press.
Batya Friedman, Peter H. Kahn Jr., and Alan Borning.
2006. Value Sensitive Design and Information Systems. In Dennis Galletta and Ping Zhang, editors,
Human-Computer Interaction in Management Information Systems: Foundations , pages 348–372. M.E.
Sharpe.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and
James Zou. 2018. Word Embeddings Quantify 100
Years of Gender and Ethnic Stereotypes. Proceedings of the National Academy of Sciences , 115(16).
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur


key: chroma:document
string_value:
Years of Gender and Ethnic Stereotypes. Proceedings of the National Academy of Sciences , 115(16).
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur
Taly, Ed H. Chi, and Alex Beutel. 2019. Counterfactual fairness in text classiﬁcation through robustness. In Proceedings of the Conference on Artiﬁcial
Intelligence, Ethics, and Society (AIES) , Honolulu,
HI.
Aparna Garimella, Carmen Banea, Dirk Hovy, and
Rada Mihalcea. 2019. Women’s syntactic resilience
and men’s grammatical luck: Gender bias in part-ofspeech tagging and dependency parsing data. In Proceedings of the Association for Computational Linguistics (ACL) , pages 3493–3498, Florence, Italy.
Andrew Gaut, Tony Sun, Shirlyn Tang, Yuxin Huang,
Jing Qian, Mai ElSherief, Jieyu Zhao, Diba
Mirza, Elizabeth Belding, Kai-Wei Chang, and
William Yang Wang. 2020. Towards Understanding Gender Bias in Relation Extraction. In Proceedings of the Association for Computational Linguistics (ACL) .
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai,


key: chroma:document
string_value:
R. Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai,
Jie Qiu, Rebekah Tang, and Jenny Huang. 2020.
Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report
Where Human-Labeled Training Data Comes From?
InProceedings of the Conference on Fairness, Accountability, and Transparency , pages 325–336.
Oguzhan Gencoglu. 2020. Cyberbullying Detection with Fairness Constraints. arXiv preprint
arXiv:2005.06625 .
Alexandra Reeve Givens and Meredith Ringel Morris.
2020. Centering Disability Perspecives in Algorithmic Fairness, Accountability, and Transparency. In
Proceedings of the Conference on Fairness, Accountability, and Transparency , Barcelona, Spain.
Hila Gonen and Yoav Goldberg. 2019. Lipstick on a
Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove
Them. In Proceedings of the North American Association for Computational Linguistics (NAACL) ,
pages 609–614, Minneapolis, MN.
Hila Gonen and Kellie Webster. 2020. Automatically Identifying Gender Issues in Machine
Translation using Perturbations. arXiv preprint
arXiv:2004.14065 .


key: chroma:document
string_value:
Translation using Perturbations. arXiv preprint
arXiv:2004.14065 .
Ben Green. 2019. “Good” isn’t good enough. In Proceedings of the AI for Social Good Workshop , Vancouver, Canada.
Lisa J. Green. 2002. African American English: A Linguistic Introduction . Cambridge University Press.
Anthony G. Greenwald, Debbie E. McGhee, and Jordan L.K. Schwartz. 1998. Measuring individual differences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6):1464–1480.Enoch Opanin Gyamﬁ, Yunbo Rao, Miao Gou, and
Yanhua Shao. 2020. deb2viz: Debiasing gender in
word embedding data using subspace visualization.
InProceedings of the International Conference on
Graphics and Image Processing .
Foad Hamidi, Morgan Klaus Scheuerman, and Stacy M.
Branham. 2018. Gender Recognition or Gender Reductionism? The Social Implications of Automatic
Gender Recognition Systems. In Proceedings of the
Conference on Human Factors in Computing Systems (CHI) , Montréal, Canada.
Alex Hanna, Emily Denton, Andrew Smart, and Jamila
Smith-Loud. 2020. Towards a Critical Race Methodology in Algorithmic Fairness. In Proceedings of the


key: chroma:document
string_value:
Alex Hanna, Emily Denton, Andrew Smart, and Jamila
Smith-Loud. 2020. Towards a Critical Race Methodology in Algorithmic Fairness. In Proceedings of the
Conference on Fairness, Accountability, and Transparency , pages 501–512, Barcelona, Spain.
Madeline E. Heilman, Aaaron S. Wallen, Daniella
Fuchs, and Melinda M. Tamkins. 2004. Penalties
for Success: Reactions to Women Who Succeed at
Male Gender-Typed Tasks. Journal of Applied Psychology , 89(3):416–427.
Jane H. Hill. 2008. The Everyday Language of White
Racism . Wiley-Blackwell.
Dirk Hovy, Federico Bianchi, and Tommaso Fornaciari.
2020. Can You Translate that into Man? Commercial Machine Translation Systems Include Stylistic
Biases. In Proceedings of the Association for Computational Linguistics (ACL) .
Dirk Hovy and Anders Søgaard. 2015. Tagging Performance Correlates with Author Age. In Proceedings of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing , pages 483–488, Beijing,
China.
Dirk Hovy and Shannon L. Spruit. 2016. The social


key: chroma:document
string_value:
China.
Dirk Hovy and Shannon L. Spruit. 2016. The social
impact of natural language processing. In Proceedings of the Association for Computational Linguistics (ACL) , pages 591–598, Berlin, Germany.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert
Stanforth, Johannes Welbl, Jack W. Rae, Vishal
Maini, Dani Yogatama, and Pushmeet Kohli. 2019.
Reducing Sentiment Bias in Language Models
via Counterfactual Evaluation. arXiv preprint
arXiv:1911.03064 .
Xiaolei Huang, Linzi Xing, Franck Dernoncourt, and
Michael J. Paul. 2020. Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias
in Hate Speech Recognition. In Proceedings of
the Language Resources and Evaluation Conference
(LREC) , Marseille, France.
Christoph Hube, Maximilian Idahl, and Besnik Fetahu.
2020. Debiasing Word Embeddings from Sentiment
Associations in Names. In Proceedings of the International Conference on Web Search and Data Mining, pages 259–267, Houston, TX.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily


key: chroma:document
string_value:
Ben Hutchinson, Vinodkumar Prabhakaran, Emily
Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social Biases in NLP Models as Barriers
for Persons with Disabilities. In Proceedings of the
Association for Computational Linguistics (ACL) .
Matei Ionita, Yury Kashnitsky, Ken Krige, Vladimir
Larin, Dennis Logvinenko, and Atanas Atanasov.
2019. Resolving gendered ambiguous pronouns
with BERT. In Proceedings of the Workshop on
Gender Bias in Natural Language Processing , pages
113–119, Florence, Italy.
Hailey James-Sorenson and David Alvarez-Melis.
2019. Probabilistic Bias Mitigation in Word Embeddings. In Proceedings of the Workshop on HumanCentric Machine Learning , Vancouver, Canada.
Shengyu Jia, Tao Meng, Jieyu Zhao, and Kai-Wei
Chang. 2020. Mitigating Gender Bias Ampliﬁcation
in Distribution by Posterior Regularization. In Proceedings of the Association for Computational Linguistics (ACL) .
Taylor Jones, Jessica Rose Kalbfeld, Ryan Hancock,
and Robin Clark. 2019. Testifying while black: An
experimental study of court reporter accuracy in transcription of African American English. Language ,
95(2).


key: chroma:document
string_value:
and Robin Clark. 2019. Testifying while black: An
experimental study of court reporter accuracy in transcription of African American English. Language ,
95(2).
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2015.
Challenges of studying and processing dialects in
social media. In Proceedings of the Workshop on
Noisy User-Generated Text , pages 9–18, Beijing,
China.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016.
Learning a POS tagger for AA VE-like language. In
Proceedings of the North American Association for
Computational Linguistics (NAACL) , pages 1115–
1120, San Diego, CA.
Pratik Joshi, Sebastian Santy, Amar Budhiraja, Kalika
Bali, and Monojit Choudhury. 2020. The State and
Fate of Linguistic Diversity and Inclusion in the
NLP World. In Proceedings of the Association for
Computational Linguistics (ACL) .
Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes.
2019. Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in
Number and Gender Assignment. In Proceedings
of the Conference on Natural Language Learning ,


key: chroma:document
string_value:
2019. Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in
Number and Gender Assignment. In Proceedings
of the Conference on Natural Language Learning ,
Hong Kong, China.
Marie-Odile Junker. 2018. Participatory action research for Indigenous linguistics in the digital age.
In Shannon T. Bischoff and Carmen Jany, editors,
Insights from Practices in Community-Based Research , pages 164–175. De Gruyter Mouton.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017.
Incorporating Dialectal Variability for Socially Equitable Language Identiﬁcation. In Proceedings of the
Association for Computational Linguistics (ACL) ,
pages 51–57, Vancouver, Canada.Masahiro Kaneko and Danushka Bollegala. 2019.
Gender-preserving debiasing for pre-trained word
embeddings. In Proceedings of the Association for
Computational Linguistics (ACL) , pages 1641–1650,
Florence, Italy.
Saket Karve, Lyle Ungar, and João Sedoc. 2019. Conceptor debiasing of word representations evaluated
on WEAT. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 40–


key: chroma:document
string_value:
on WEAT. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 40–
48, Florence, Italy.
Michael Katell, Meg Young, Dharma Dailey, Bernease
Herman, Vivian Guetler, Aaron Tam, Corinne Bintz,
Danielle Raz, and P.M. Krafft. 2020. Toward situated interventions for algorithmic equity: lessons
from the ﬁeld. In Proceedings of the Conference on
Fairness, Accountability, and Transparency , pages
45–55, Barcelona, Spain.
Stephen Kemmis. 2006. Participatory action research
and the public sphere. Educational Action Research ,
14(4):459–476.
Os Keyes. 2018. The Misgendering Machines:
Trans/HCI Implications of Automatic Gender
Recognition. Proceedings of the ACM on HumanComputer Interaction , 2(CSCW).
Os Keyes, Josephine Hoy, and Margaret Drouhard.
2019. Human-Computer Insurrection: Notes on an
Anarchist HCI. In Proceedings of the Conference on
Human Factors in Computing Systems (CHI) , Glasgow, Scotland, UK.
Jae Yeon Kim, Carlos Ortiz, Sarah Nam, Sarah Santiago, and Vivek Datta. 2020. Intersectional Bias in
Hate Speech and Abusive Language Datasets. In


key: chroma:document
string_value:
Hate Speech and Abusive Language Datasets. In
Proceedings of the Association for Computational
Linguistics (ACL) .
Svetlana Kiritchenko and Saif M. Mohammad. 2018.
Examining Gender and Race Bias in Two Hundred
Sentiment Analysis Systems. In Proceedings of the
Joint Conference on Lexical and Computational Semantics , pages 43–53, New Orleans, LA.
Moshe Koppel, Shlomo Argamon, and Anat Rachel
Shimoni. 2002. Automatically Categorizing Written Texts by Author Gender. Literary and Linguistic
Computing , 17(4):401–412.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W.
Black, and Yulia Tsvetkov. 2019. Measuring bias
in contextualized word representations. In Proceedings of the Workshop on Gender Bias for Natural Language Processing , pages 166–172, Florence,
Italy.
Sonja L. Lanehart and Ayesha M. Malik. 2018. Black
Is, Black Isn’t: Perceptions of Language and Blackness. In Jeffrey Reaser, Eric Wilbanks, Karissa Wojcik, and Walt Wolfram, editors, Language Variety in
the New South . University of North Carolina Press.


key: chroma:document
string_value:
the New South . University of North Carolina Press.
Brian N. Larson. 2017. Gender as a variable in naturallanguage processing: Ethical considerations. In Proceedings of the Workshop on Ethics in Natural Language Processing , pages 30–40, Valencia, Spain.
Anne Lauscher and Goran Glavaš. 2019. Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors. In Proceedings
of the Joint Conference on Lexical and Computational Semantics , pages 85–91, Minneapolis, MN.
Anne Lauscher, Goran Glavaš, Simone Paolo Ponzetto,
and Ivan Vuli ´c. 2019. A General Framework for Implicit and Explicit Debiasing of Distributional Word
Vector Spaces. arXiv preprint arXiv:1909.06092 .
Christopher A. Le Dantec, Erika Shehan Poole, and Susan P. Wyche. 2009. Values as Lived Experience:
Evolving Value Sensitive Design in Support of Value
Discovery. In Proceedings of the Conference on Human Factors in Computing Systems (CHI) , Boston,
MA.
Nayeon Lee, Andrea Madotto, and Pascale Fung. 2019.
Exploring Social Bias in Chatbots using Stereotype
Knowledge. In Proceedings of the Workshop on


key: chroma:document
string_value:
Exploring Social Bias in Chatbots using Stereotype
Knowledge. In Proceedings of the Workshop on
Widening NLP , pages 177–180, Florence, Italy.
Wesley Y . Leonard. 2012. Reframing language reclamation programmes for everybody’s empowerment.
Gender and Language , 6(2):339–367.
Paul Pu Liang, Irene Li, Emily Zheng, Yao Chong Lim,
Ruslan Salakhutdinov, and Louis-Philippe Morency.
2019. Towards Debiasing Sentence Representations.
InProceedings of the NeurIPS Workshop on HumanCentric Machine Learning , Vancouver, Canada.
Rosina Lippi-Green. 2012. English with an Accent: Language, Ideology, and Discrimination in the
United States . Routledge.
Bo Liu. 2019. Anonymized BERT: An Augmentation
Approach to the Gendered Pronoun Resolution Challenge. In Proceedings of the Workshop on Gender
Bias in Natural Language Processing , pages 120–
125, Florence, Italy.
Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, and Jiliang Tang. 2019. Does Gender Matter? Towards Fairness in Dialogue Systems. arXiv
preprint arXiv:1910.10486 .


key: chroma:document
string_value:
preprint arXiv:1910.10486 .
Felipe Alfaro Lois, José A.R. Fonollosa, and Costa-jà.
2019. BERT Masked Language Modeling for Coreference Resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 76–81, Florence, Italy.
Brandon C. Loudermilk. 2015. Implicit attitudes and
the perception of sociolinguistic variation. In Alexei
Prikhodkine and Dennis R. Preston, editors, Responses to Language Varieties: Variability, processes and outcomes , pages 137–156.Anastassia Loukina, Nitin Madnani, and Klaus Zechner. 2019. The many dimensions of algorithmic fairness in educational applications. In Proceedings of
the Workshop on Innovative Use of NLP for Building Educational Applications , pages 1–10, Florence,
Italy.
Kaiji Lu, Peter Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. 2018. Gender bias in
neural natural language processing. arXiv preprint
arXiv:1807.11714 .
Anne Maass. 1999. Linguistic intergroup bias: Stereotype perpetuation through language. Advances in
Experimental Social Psychology , 31:79–121.


key: chroma:document
string_value:
Anne Maass. 1999. Linguistic intergroup bias: Stereotype perpetuation through language. Advances in
Experimental Social Psychology , 31:79–121.
Nitin Madnani, Anastassia Loukina, Alina von Davier,
Jill Burstein, and Aoife Cahill. 2017. Building Better Open-Source Tools to Support Fairness in Automated Scoring. In Proceedings of the Workshop on
Ethics in Natural Language Processing , pages 41–
52, Valencia, Spain.
Thomas Manzini, Yao Chong Lim, Yulia Tsvetkov, and
Alan W. Black. 2019. Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings. In Proceedings of
the North American Association for Computational
Linguistics (NAACL) , pages 801–809, Minneapolis,
MN.
Ramón Antonio Martínez and Alexander Feliciano
Mejía. 2019. Looking closely and listening carefully: A sociocultural approach to understanding
the complexity of Latina/o/x students’ everyday language. Theory Into Practice .
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell,


key: chroma:document
string_value:
the complexity of Latina/o/x students’ everyday language. Theory Into Practice .
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell,
and Simone Teufel. 2019. It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution. In Proceedings of Empirical
Methods in Natural Language Processing (EMNLP) ,
pages 5270–5278, Hong Kong, China.
Chandler May, Alex Wang, Shikha Bordia, Samuel R.
Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in Sentence Encoders. In Proceedings of the North American Association for Computational Linguistics (NAACL) , pages 629–634, Minneapolis, MN.
Elijah Mayﬁeld, Michael Madaio, Shrimai Prabhumoye, David Gerritsen, Brittany McLaughlin,
Ezekiel Dixon-Roman, and Alan W. Black. 2019.
Equity Beyond Bias in Language Technologies for
Education. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications , Florence, Italy.
Katherine McCurdy and O ˘guz Serbetçi. 2017. Grammatical gender associations outweigh topical gender


key: chroma:document
string_value:
Katherine McCurdy and O ˘guz Serbetçi. 2017. Grammatical gender associations outweigh topical gender
bias in crosslinguistic word embeddings. In Proceedings of the Workshop for Women & Underrepresented Minorities in Natural Language Processing ,
Vancouver, Canada.
Ninareh Mehrabi, Thamme Gowda, Fred Morstatter,
Nanyun Peng, and Aram Galstyan. 2019. Man is to
Person as Woman is to Location: Measuring Gender
Bias in Named Entity Recognition. arXiv preprint
arXiv:1910.10872 .
Michela Menegatti and Monica Rubini. 2017. Gender
bias and sexism in language. In Oxford Research
Encyclopedia of Communication . Oxford University
Press.
Inom Mirzaev, Anthony Schulte, Michael Conover, and
Sam Shah. 2019. Considerations for the interpretation of bias measures of word embeddings. arXiv
preprint arXiv:1906.08379 .
Salikoko S. Mufwene, Guy Bailey, and John R. Rickford, editors. 1998. African-American English:
Structure, History, and Use . Routledge.
Michael J. Muller. 2007. Participatory Design: The


key: chroma:document
string_value:
Structure, History, and Use . Routledge.
Michael J. Muller. 2007. Participatory Design: The
Third Space in HCI. In The Human-Computer Interaction Handbook , pages 1087–1108. CRC Press.
Moin Nadeem, Anna Bethke, and Siva Reddy.
2020. StereoSet: Measuring stereotypical bias
in pretrained language models. arXiv preprint
arXiv:2004.09456 .
Dong Nguyen, Rilana Gravel, Dolf Trieschnigg, and
Theo Meder. 2013. “How Old Do You Think I
Am?”: A Study of Language and Age in Twitter. In
Proceedings of the Conference on Web and Social
Media (ICWSM) , pages 439–448, Boston, MA.
Malvina Nissim, Rik van Noord, and Rob van der Goot.
2020. Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics.
Debora Nozza, Claudia V olpetti, and Elisabetta Fersini.
2019. Unintended Bias in Misogyny Detection. In
Proceedings of the Conference on Web Intelligence ,
pages 149–155.
Alexandra Olteanu, Carlos Castillo, Fernando Diaz,


key: chroma:document
string_value:
Proceedings of the Conference on Web Intelligence ,
pages 149–155.
Alexandra Olteanu, Carlos Castillo, Fernando Diaz,
and Emre Kıcıman. 2019. Social Data: Biases,
Methodological Pitfalls, and Ethical Boundaries.
Frontiers in Big Data , 2.
Alexandra Olteanu, Kartik Talamadupula, and Kush R.
Varshney. 2017. The Limits of Abstract Evaluation
Metrics: The Case of Hate Speech Detection. In
Proceedings of the ACM Web Science Conference ,
Troy, NY .
Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020.
Bias in word embeddings. In Proceedings of the
Conference on Fairness, Accountability, and Transparency , pages 446–457, Barcelona, Spain.
Ji Ho Park, Jamin Shin, and Pascale Fung. 2018. Reducing Gender Bias in Abusive Language Detection.
InProceedings of Empirical Methods in Natural
Language Processing (EMNLP) , pages 2799–2804,
Brussels, Belgium.Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent
Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics , 7:677–694.


key: chroma:document
string_value:
Disagreements in Human Textual Inferences. Transactions of the Association for Computational Linguistics , 7:677–694.
Xiangyu Peng, Siyan Li, Spencer Frazier, and Mark
Riedl. 2020. Fine-Tuning a Transformer-Based Language Model to Avoid Generating Non-Normative
Text. arXiv preprint arXiv:2001.08764 .
Radomir Popovi ´c, Florian Lemmerich, and Markus
Strohmaier. 2020. Joint Multiclass Debiasing of
Word Embeddings. In Proceedings of the International Symposium on Intelligent Systems , Graz, Austria.
Vinodkumar Prabhakaran, Ben Hutchinson, and Margaret Mitchell. 2019. Perturbation Sensitivity Analysis to Detect Unintended Model Biases. In Proceedings of Empirical Methods in Natural Language
Processing (EMNLP) , pages 5744–5749, Hong
Kong, China.
Shrimai Prabhumoye, Elijah Mayﬁeld, and Alan W.
Black. 2019. Principled Frameworks for Evaluating
Ethics in NLP Systems. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications , Florence, Italy.
Marcelo Prates, Pedro Avelar, and Luis C. Lamb. 2019.


key: chroma:document
string_value:
Marcelo Prates, Pedro Avelar, and Luis C. Lamb. 2019.
Assessing gender bias in machine translation: A
case study with google translate. Neural Computing
and Applications .
Rasmus Précenth. 2019. Word embeddings and gender
stereotypes in Swedish and English. Master’s thesis,
Uppsala University.
Dennis R. Preston. 2009. Are you really smart (or
stupid, or cute, or ugly, or cool)? Or do you just talk
that way? Language attitudes, standardization and
language change. Oslo: Novus forlag , pages 105–
129.
Flavien Prost, Nithum Thain, and Tolga Bolukbasi.
2019. Debiasing Embeddings for Reduced Gender
Bias in Text Classiﬁcation. In Proceedings of the
Workshop on Gender Bias in Natural Language Processing , pages 69–75, Florence, Italy.
Reid Pryzant, Richard Diehl Martinez, Nathan Dass,
Sadao Kurohashi, Dan Jurafsky, and Diyi Yang.
2020. Automatically Neutralizing Subjective Bias
in Text. In Proceedings of the AAAI Conference on
Artiﬁcial Intelligence (AAAI) , New York, NY .


key: chroma:document
string_value:
2020. Automatically Neutralizing Subjective Bias
in Text. In Proceedings of the AAAI Conference on
Artiﬁcial Intelligence (AAAI) , New York, NY .
Arun K. Pujari, Ansh Mittal, Anshuman Padhi, Anshul Jain, Mukesh Jadon, and Vikas Kumar. 2019.
Debiasing Gender biased Hindi Words with Wordembedding. In Proceedings of the International
Conference on Algorithms, Computing and Artiﬁcial
Intelligence , pages 450–456.
Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won
Hyun. 2019. Reducing gender bias in word-level
language models with a gender-equalizing loss function. In Proceedings of the ACL Student Research
Workshop , pages 223–228, Florence, Italy.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael
Twiton, and Yoav Goldberg. 2020. Null It Out:
Guarding Protected Attributes by Iterative Nullspace
Projection. In Proceedings of the Association for
Computational Linguistics (ACL) .
John R. Rickford and Sharese King. 2016. Language
and linguistics on trial: Hearing Rachel Jeantel (and


key: chroma:document
string_value:
Computational Linguistics (ACL) .
John R. Rickford and Sharese King. 2016. Language
and linguistics on trial: Hearing Rachel Jeantel (and
other vernacular speakers) in the courtroom and beyond. Language , 92(4):948–988.
Anthony Rios. 2020. FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classiﬁers on AfricanAmerican English. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence (AAAI) , New York,
NY .
Gerald Roche. 2019. Articulating language oppression: colonialism, coloniality and the erasure of Tibetâ˘A´Zs minority languages. Patterns of Prejudice .
Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra
Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, and Adam Tauman Kalai.
2019. What’s in a Name? Reducing Bias in Bios
without Access to Protected Attributes. In Proceedings of the North American Association for Computational Linguistics (NAACL) , pages 4187–4195,
Minneapolis, MN.
Jonathan Rosa. 2019. Contesting Representations of


key: chroma:document
string_value:
Minneapolis, MN.
Jonathan Rosa. 2019. Contesting Representations of
Migrant “Illegality” through the Drop the I-Word
Campaign: Rethinking Language Change and Social Change. In Netta Avineri, Laura R. Graham,
Eric J. Johnson, Robin Conley Riner, and Jonathan
Rosa, editors, Language and Social Justice in Practice. Routledge.
Jonathan Rosa and Christa Burdick. 2017. Language
Ideologies. In Ofelia García, Nelson Flores, and
Massimiliano Spotti, editors, The Oxford Handbook
of Language and Society . Oxford University Press.
Jonathan Rosa and Nelson Flores. 2017. Unsettling
race and language: Toward a raciolinguistic perspective. Language in Society , 46:621–647.
Sara Rosenthal and Kathleen McKeown. 2011. Age
Prediction in Blogs: A Study of Style, Content, and
Online Behavior in Pre- and Post-Social Media Generations. In Proceedings of the North American Association for Computational Linguistics (NAACL) ,
pages 763–772, Portland, OR.
Candace Ross, Boris Katz, and Andrei Barbu.
2020. Measuring Social Biases in Grounded Vision and Language Embeddings. arXiv preprint
arXiv:2002.08911 .


key: chroma:document
string_value:
2020. Measuring Social Biases in Grounded Vision and Language Embeddings. arXiv preprint
arXiv:2002.08911 .
Richard Rothstein. 2017. The Color of Law: A Forgotten History of How Our Government Segregated
America . Liveright Publishing.David Rozado. 2020. Wide range screening of algorithmic bias in word embedding models using large
sentiment lexicons reveals underreported bias types.
PLOS One .
Elayne Ruane, Abeba Birhane, and Anthony Ventresque. 2019. Conversational AI: Social and Ethical Considerations. In Proceedings of the Irish Conference on Artiﬁcial Intelligence and Cognitive Science, Galway, Ireland.
Rachel Rudinger, Chandler May, and Benjamin
Van Durme. 2017. Social bias in elicited natural language inferences. In Proceedings of the Workshop
on Ethics in Natural Language Processing , pages
74–79, Valencia, Spain.
Rachel Rudinger, Jason Naradowsky, Brian Leonard,
and Benjamin Van Durme. 2018. Gender Bias
in Coreference Resolution. In Proceedings of the
North American Association for Computational Linguistics (NAACL) , pages 8–14, New Orleans, LA.
Elizabeth B.N. Sanders. 2002. From user-centered to
participatory design approaches. In Jorge Frascara,


key: chroma:document
string_value:
Elizabeth B.N. Sanders. 2002. From user-centered to
participatory design approaches. In Jorge Frascara,
editor, Design and the Social Sciences: Making Connections , pages 18–25. CRC Press.
Brenda Salenave Santana, Vinicius Woloszyn, and Leandro Krug Wives. 2018. Is there gender bias and
stereotype in Portuguese word embeddings? In
Proceedings of the International Conference on the
Computational Processing of Portuguese Student Research Workshop , Canela, Brazil.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi,
and Noah A. Smith. 2019. The risk of racial bias in
hate speech detection. In Proceedings of the Association for Computational Linguistics (ACL) , pages
1668–1678, Florence, Italy.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social
Bias Frames: Reasoning about Social and Power Implications of Language. In Proceedings of the Association for Computational Linguistics (ACL) .
Hanna Sassaman, Jennifer Lee, Jenessa Irvine, and
Shankar Narayan. 2020. Creating CommunityBased Tech Policy: Case Studies, Lessons Learned,
and What Technologists and Communities Can Do


key: chroma:document
string_value:
Shankar Narayan. 2020. Creating CommunityBased Tech Policy: Case Studies, Lessons Learned,
and What Technologists and Communities Can Do
Together. In Proceedings of the Conference on Fairness, Accountability, and Transparency , Barcelona,
Spain.
Danielle Saunders and Bill Byrne. 2020. Reducing
Gender Bias in Neural Machine Translation as a Domain Adaptation Problem. In Proceedings of the Association for Computational Linguistics (ACL) .
Tyler Schnoebelen. 2017. Goal-Oriented Design for
Ethical Machine Learning and NLP. In Proceedings
of the Workshop on Ethics in Natural Language Processing , pages 88–93, Valencia, Spain.
Sabine Sczesny, Magda Formanowicz, and Franziska
Moser. 2016. Can gender-fair language reduce gender stereotyping and discrimination? Frontiers in
Psychology , 7.
João Sedoc and Lyle Ungar. 2019. The Role of Protected Class Word Lists in Bias Identiﬁcation of Contextualized Word Representations. In Proceedings
of the Workshop on Gender Bias in Natural Language Processing , pages 55–61, Florence, Italy.
Procheta Sen and Debasis Ganguly. 2020. Towards Socially Responsible AI: Cognitive Bias-Aware MultiObjective Learning. In Proceedings of the AAAI


key: chroma:document
string_value:
Procheta Sen and Debasis Ganguly. 2020. Towards Socially Responsible AI: Cognitive Bias-Aware MultiObjective Learning. In Proceedings of the AAAI
Conference on Artiﬁcial Intelligence (AAAI) , New
York, NY .
Deven Shah, H. Andrew Schwartz, and Dirk Hovy.
2020. Predictive Biases in Natural Language Processing Models: A Conceptual Framework and
Overview. In Proceedings of the Association for
Computational Linguistics (ACL) .
Judy Hanwen Shen, Lauren Fratamico, Iyad Rahwan,
and Alexander M. Rush. 2018. Darling orBabygirl? Investigating Stylistic Bias in Sentiment Analysis. In Proceedings of the Workshop on Fairness,
Accountability, and Transparency (FAT/ML) , Stockholm, Sweden.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan,
and Nanyun Peng. 2019. The Woman Worked as
a Babysitter: On Biases in Language Generation.
InProceedings of Empirical Methods in Natural
Language Processing (EMNLP) , pages 3398–3403,
Hong Kong, China.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan,
and Nanyun Peng. 2020. Towards Controllable


key: chroma:document
string_value:
Hong Kong, China.
Emily Sheng, Kai-Wei Chang, Premkumar Natarajan,
and Nanyun Peng. 2020. Towards Controllable
Biases in Language Generation. arXiv preprint
arXiv:2005.00268 .
Seungjae Shin, Kyungwoo Song, JoonHo Jang, Hyemi
Kim, Weonyoung Joo, and Il-Chul Moon. 2020.
Neutralizing Gender Bias in Word Embedding with
Latent Disentanglement and Counterfactual Generation. arXiv preprint arXiv:2004.03133 .
Jesper Simonsen and Toni Robertson, editors. 2013.
Routledge International Handbook of Participatory
Design . Routledge.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine
translation. In Proceedings of the Association for
Computational Linguistics (ACL) , pages 1679–1684,
Florence, Italy.
Yolande Strengers, Lizhe Qu, Qiongkai Xu, and Jarrod
Knibbe. 2020. Adhering, Steering, and Queering:
Treatment of Gender in Natural Language Generation. In Proceedings of the Conference on Human


key: chroma:document
string_value:
Knibbe. 2020. Adhering, Steering, and Queering:
Treatment of Gender in Natural Language Generation. In Proceedings of the Conference on Human
Factors in Computing Systems (CHI) , Honolulu, HI.Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang,
Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth
Belding, Kai-Wei Chang, and William Yang Wang.
2019. Mitigating Gender Bias in Natural Language Processing: Literature Review. In Proceedings of the Association for Computational Linguistics (ACL) , pages 1630–1640, Florence, Italy.
Adam Sutton, Thomas Lansdall-Welfare, and Nello
Cristianini. 2018. Biased embeddings from wild
data: Measuring, understanding and removing.
InProceedings of the International Symposium
on Intelligent Data Analysis , pages 328–339, ’sHertogenbosch, Netherlands.
Chris Sweeney and Maryam Najaﬁan. 2019. A Transparent Framework for Evaluating Unintended Demographic Bias in Word Embeddings. In Proceedings of the Association for Computational Linguistics (ACL) , pages 1662–1667, Florence, Italy.
Chris Sweeney and Maryam Najaﬁan. 2020. Reducing sentiment polarity for demographic attributes


key: chroma:document
string_value:
Chris Sweeney and Maryam Najaﬁan. 2020. Reducing sentiment polarity for demographic attributes
in word embeddings using adversarial learning. In
Proceedings of the Conference on Fairness, Accountability, and Transparency , pages 359–368,
Barcelona, Spain.
Nathaniel Swinger, Maria De-Arteaga, Neil Thomas
Heffernan, Mark D.M. Leiserson, and Adam Tauman Kalai. 2019. What are the biases in my word
embedding? In Proceedings of the Conference on
Artiﬁcial Intelligence, Ethics, and Society (AIES) ,
Honolulu, HI.
Samson Tan, Shaﬁq Joty, Min-Yen Kan, and Richard
Socher. 2020. It’s Morphin’ Time! Combating
Linguistic Discrimination with Inﬂectional Perturbations. In Proceedings of the Association for Computational Linguistics (ACL) .
Yi Chern Tan and L. Elisa Celis. 2019. Assessing
Social and Intersectional Biases in Contextualized
Word Representations. In Proceedings of the Conference on Neural Information Processing Systems ,
Vancouver, Canada.
J. Michael Terry, Randall Hendrick, Evangelos Evangelou, and Richard L. Smith. 2010. Variable


key: chroma:document
string_value:
Vancouver, Canada.
J. Michael Terry, Randall Hendrick, Evangelos Evangelou, and Richard L. Smith. 2010. Variable
dialect switching among African American children: Inferences about working memory. Lingua ,
120(10):2463–2475.
Joel Tetreault, Daniel Blanchard, and Aoife Cahill.
2013. A Report on the First Native Language Identiﬁcation Shared Task. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications , pages 48–57, Atlanta, GA.
Mike Thelwall. 2018. Gender Bias in Sentiment Analysis. Online Information Review , 42(1):45–57.
Kristen Vaccaro, Karrie Karahalios, Deirdre K. Mulligan, Daniel Kluttz, and Tad Hirsch. 2019. Contestability in Algorithmic Systems. In Conference
Companion Publication of the 2019 on Computer
Supported Cooperative Work and Social Computing ,
pages 523–527, Austin, TX.
Ameya Vaidya, Feng Mai, and Yue Ning. 2019. Empirical Analysis of Multi-Task Learning for Reducing Model Bias in Toxic Comment Detection. arXiv
preprint arXiv:1909.09758v2 .
Eva Vanmassenhove, Christian Hardmeier, and Andy


key: chroma:document
string_value:
preprint arXiv:1909.09758v2 .
Eva Vanmassenhove, Christian Hardmeier, and Andy
Way. 2018. Getting Gender Right in Neural Machine Translation. In Proceedings of Empirical
Methods in Natural Language Processing (EMNLP) ,
pages 3003–3008, Brussels, Belgium.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov,
Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal Mediation Analysis for
Interpreting Neural NLP: The Case of Gender Bias.
arXiv preprint arXiv:2004.12265 .
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caiming Xiong. 2020. Double-Hard Debias: Tailoring
Word Embeddings for Gender Bias Mitigation. In
Proceedings of the Association for Computational
Linguistics (ACL) .
Zili Wang. 2019. MSnet: A BERT-based Network for
Gendered Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 89–95, Florence, Italy.


key: chroma:document
string_value:
Gendered Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 89–95, Florence, Italy.
Kellie Webster, Marta R. Costa-jussà, Christian Hardmeier, and Will Radford. 2019. Gendered Ambiguous Pronoun (GAP) Shared Task at the Gender Bias
in NLP Workshop 2019. In Proceedings of the Workshop on Gender Bias in Natural Language Processing, pages 1–7, Florence, Italy.
Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced
corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–618.
Walt Wolfram and Natalie Schilling. 2015. American
English: Dialects and Variation , 3 edition. Wiley
Blackwell.
Austin P. Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Diyi
Yang, and Duen Horng (Polo) Chau. 2020. RECAST: Interactive Auditing of Automatic Toxicity
Detection Models. In Proceedings of the Conference on Human Factors in Computing Systems
(CHI) , Honolulu, HI.
Yinchuan Xu and Junlin Yang. 2019. Look again at


key: chroma:document
string_value:
Detection Models. In Proceedings of the Conference on Human Factors in Computing Systems
(CHI) , Honolulu, HI.
Yinchuan Xu and Junlin Yang. 2019. Look again at
the syntax: Relational graph convolutional network
for gendered ambiguous pronoun resolution. In Proceedings of the Workshop on Gender Bias in Natural Language Processing , pages 96–101, Florence,
Italy.Kai-Chou Yang, Timothy Niven, Tzu-Hsuan Chou, and
Hung-Yu Kao. 2019. Fill the GAP: Exploiting
BERT for Pronoun Resolution. In Proceedings of
the Workshop on Gender Bias in Natural Language
Processing , pages 102–106, Florence, Italy.
Zekun Yang and Juan Feng. 2020. A Causal Inference
Method for Reducing Gender Bias in Word Embedding Relations. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence (AAAI) , New York,
NY .
Daisy Yoo, Anya Ernest, Soﬁa Serholt, Eva Eriksson,
and Peter Dalsgaard. 2019. Service Design in HCI
Research: The Extended Value Co-creation Model.
InProceedings of the Halfway to the Future Symposium , Nottingham, United Kingdom.
Brian Hu Zhang, Blake Lemoine, and Margaret


key: chroma:document
string_value:
InProceedings of the Halfway to the Future Symposium , Nottingham, United Kingdom.
Brian Hu Zhang, Blake Lemoine, and Margaret
Mitchell. 2018. Mitigating unwanted biases with adversarial learning. In Proceedings of the Conference
on Artiﬁcial Intelligence, Ethics, and Society (AIES) ,
New Orleans, LA.
Guanhua Zhang, Bing Bai, Junqi Zhang, Kun Bai, Conghui Zhu, and Tiejun Zhao. 2020a. Demographics
Should Not Be the Reason of Toxicity: Mitigating
Discrimination in Text Classiﬁcations with Instance
Weighting. In Proceedings of the Association for
Computational Linguistics (ACL) .
Haoran Zhang, Amy X. Lu, Mohamed Abdalla,
Matthew McDermott, and Marzyeh Ghassemi.
2020b. Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings. In Proceedings
of the ACM Conference on Health, Inference, and
Learning .
Jieyu Zhao, Subhabrata Mukherjee, Saghar Hosseini,
Kai-Wei Chang, and Ahmed Hassan Awadallah.
2020. Gender Bias in Multilingual Embeddings and
Cross-Lingual Transfer. In Proceedings of the Association for Computational Linguistics (ACL) .


key: chroma:document
string_value:
2020. Gender Bias in Multilingual Embeddings and
Cross-Lingual Transfer. In Proceedings of the Association for Computational Linguistics (ACL) .
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019.
Gender Bias in Contextualized Word Embeddings.
InProceedings of the North American Association
for Computational Linguistics (NAACL) , pages 629–
634, Minneapolis, MN.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like
shopping: Reducing gender bias ampliﬁcation using corpus-level constraints. In Proceedings of
Empirical Methods in Natural Language Processing (EMNLP) , pages 2979–2989, Copenhagen, Denmark.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018a. Gender Bias
in Coreference Resolution: Evaluation and Debiasing Methods. In Proceedings of the North American
Association for Computational Linguistics (NAACL) ,
pages 15–20, New Orleans, LA.


key: chroma:document
string_value:
Association for Computational Linguistics (NAACL) ,
pages 15–20, New Orleans, LA.
Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and KaiWei Chang. 2018b. Learning Gender-Neutral Word
Embeddings. In Proceedings of Empirical Methods
in Natural Language Processing (EMNLP) , pages
4847–4853, Brussels, Belgium.
Alina Zhiltsova, Simon Caton, and Catherine Mulwa.
2019. Mitigation of Unintended Biases against NonNative English Texts in Sentiment Analysis. In Proceedings of the Irish Conference on Artiﬁcial Intelligence and Cognitive Science , Galway, Ireland.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang,
Muhao Chen, and Kai-Wei Chang. 2019. Examining gender bias in languages with grammatical genders. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP) , pages 5279–
5287, Hong Kong, China.
Ran Zmigrod, S. J. Mielke, Hanna Wallach, and Ryan
Cotterell. 2019. Counterfactual data augmentation
for mitigating gender stereotypes in languages with
rich morphology. In Proceedings of the Association


key: chroma:document
string_value:
Cotterell. 2019. Counterfactual data augmentation
for mitigating gender stereotypes in languages with
rich morphology. In Proceedings of the Association
for Computational Linguistics (ACL) , pages 1651–
1661, Florence, Italy.
A Appendix
In Table 3, we provide examples of the papers’ motivations and techniques across several NLP tasks.
A.1 Categorization details
In this section, we provide some additional details
about our method—speciﬁcally, our categorization.
What counts as being covered by an NLP task?
We considered a paper to cover a given NLP task if
it analyzed “bias” with respect to that task, but not
if it only evaluated overall performance on that task.
For example, a paper examining the impact of mitigating “bias” in word embeddings on “bias” in sentiment analysis would be counted as covering both
NLP tasks. In contrast, a paper assessing whether
performance on sentiment analysis degraded after
mitigating “bias” in word embeddings would be
counted only as focusing on embeddings.
What counts as a motivation? We considered a
motivation to include any description of the problem that motivated the paper or proposed quantitative technique, including any normative reasoning.


key: chroma:document
string_value:
What counts as a motivation? We considered a
motivation to include any description of the problem that motivated the paper or proposed quantitative technique, including any normative reasoning.
We excluded from the “Vague/unstated” category of motivations the papers that participated in
the Gendered Ambiguous Pronoun (GAP) Shared
Task at the First ACL Workshop on Gender Bias in
NLP. In an ideal world, shared task papers would
engage with “bias” more critically, but given the
nature of shared tasks it is understandable that theydo not. As a result, we excluded them from our
counts for techniques as well. We cite the papers
here; most propose techniques we would have categorized as “Questionable correlations,” with a few
as “Other representational harms” (Abzaliev, 2019;
Attree, 2019; Bao and Qiao, 2019; Chada, 2019;
Ionita et al., 2019; Liu, 2019; Lois et al., 2019;
Wang, 2019; Xu and Yang, 2019; Yang et al., 2019).
We excluded Dabas et al. (2020) from our survey
because we could not determine what this paper’s
user study on fairness was actually measuring.
Finally, we actually categorized the motivation


key: chroma:document
string_value:
because we could not determine what this paper’s
user study on fairness was actually measuring.
Finally, we actually categorized the motivation
for Liu et al. (2019) (i.e., the last row in Table 3) as
“Questionable correlations” due to a sentence elsewhere in the paper; had the paragraph we quoted
been presented without more detail, we would have
categorized the motivation as “Vague/unstated.”
A.2 Full categorization: Motivations
Allocational harms Hovy and Spruit (2016);
Caliskan et al. (2017); Madnani et al. (2017);
Dixon et al. (2018); Kiritchenko and Mohammad
(2018); Shen et al. (2018); Zhao et al. (2018b);
Bhaskaran and Bhallamudi (2019); Bordia and
Bowman (2019); Brunet et al. (2019); Chaloner
and Maldonado (2019); De-Arteaga et al. (2019);
Dev and Phillips (2019); Font and Costa-jussà
(2019); James-Sorenson and Alvarez-Melis (2019);
Kurita et al. (2019); Mayﬁeld et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane


key: chroma:document
string_value:
Kurita et al. (2019); Mayﬁeld et al. (2019); Pujari et al. (2019); Romanov et al. (2019); Ruane
et al. (2019); Sedoc and Ungar (2019); Sun et al.
(2019); Zmigrod et al. (2019); Hutchinson et al.
(2020); Papakyriakopoulos et al. (2020); Ravfogel et al. (2020); Strengers et al. (2020); Sweeney
and Najaﬁan (2020); Tan et al. (2020); Zhang et al.
(2020b).
Stereotyping Bolukbasi et al. (2016a,b);
Caliskan et al. (2017); McCurdy and Serbetçi
(2017); Rudinger et al. (2017); Zhao et al. (2017);
Curry and Rieser (2018); Díaz et al. (2018);
Santana et al. (2018); Sutton et al. (2018); Zhao
et al. (2018a,b); Agarwal et al. (2019); Basta et al.
(2019); Bhaskaran and Bhallamudi (2019); Bordia
and Bowman (2019); Brunet et al. (2019); Cao
and Daumé (2019); Chaloner and Maldonado


key: chroma:document
string_value:
and Bowman (2019); Brunet et al. (2019); Cao
and Daumé (2019); Chaloner and Maldonado
(2019); Cho et al. (2019); Dev and Phillips (2019);
Font and Costa-jussà (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);
Kaneko and Bollegala (2019); Karve et al. (2019);
Kurita et al. (2019); Lauscher and Glavaš (2019);
Lee et al. (2019); Manzini et al. (2019); Mayﬁeld
Categories
NLP task Stated motivation Motivations Techniques
Language
modeling
(Bordia and
Bowman,
2019)“Existing biases in data can be ampliﬁed by models and the
resulting output consumed by the public can inﬂuence them ,encourage and reinforce harmful stereotypes , ordistort the truth .
Automated systems that depend on these models can take problematic actions based on biased proﬁling of individuals. ”Allocational
harms,
stereotypingQuestionable
correlations
Sentiment
analysis
(Kiritchenko
and
Mohammad,


key: chroma:document
string_value:
harms,
stereotypingQuestionable
correlations
Sentiment
analysis
(Kiritchenko
and
Mohammad,
2018)“Other biases can be inappropriate and result in negative experiences for some groups of people. Examples include, loan
eligibility and crime recidivism prediction systems ...and resumé
sorting systems that believe that men are more qualiﬁed to be
programmers than women (Bolukbasi et al., 2016). Similarly,
sentiment and emotion analysis systems can also perpetuate and
accentuate inappropriate human biases, e.g., systems that consider
utterances from one race or gender to be less positive simply because of their race or gender, or customer support systems that
prioritize a call from an angry male over a call from the equally
angry female . ”Allocational
harms, other
representational
harms (system
performance
differences w.r.t.
text written by
different social
groups)Questionable
correlations
(differences in
sentiment
intensity scores
w.r.t. text about
different social
groups)
Machine
translation
(Cho et al.,
2019)“[MT training] may incur an association of gender-speciﬁed pronouns (in the target) and gender-neutral ones (in the source) for


key: chroma:document
string_value:
2019)“[MT training] may incur an association of gender-speciﬁed pronouns (in the target) and gender-neutral ones (in the source) for
lexicon pairs that frequently collocate in the corpora. We claim
that this kind of phenomenon seriously threatens the fairness of a
translation system, in the sense that it lacks generality andinserts
social bias to the inference. Moreover, the input is not fully correct(considering gender-neutrality) and might offend the users
who expect fairer representations. ”Questionable
correlations,
other
representational
harmsQuestionable
correlations
Machine
translation
(Stanovsky
et al., 2019)“Learned models exhibit social bias when their training data
encode stereotypes not relevant for the task, but the correlations
are picked up anyway . ”Stereotyping,
questionable
correlationsStereotyping,
other
representational
harms (system
performance
differences),
questionable
correlations
Type-level
embeddings
(Zhao et al.,
2018b)“However, embeddings trained on human-generated corpora have
been demonstrated to inherit strong gender stereotypes that reﬂect social constructs....Such a bias substantially affects downstream applications ....This concerns the practitioners who use
the embedding model to build gender-sensitive applications such


key: chroma:document
string_value:
the embedding model to build gender-sensitive applications such
as a resume ﬁltering system or a job recommendation system as
the automated system may discriminate candidates based on their
gender, as reﬂected by their name. Besides, biased embeddings
may implicitly affect downstream applications used in our daily
lives. For example, when searching for ‘computer scientist’ using
a search engine...a search algorithm using an embedding model in
the backbone tends to rank male scientists higher than females’
[sic], hindering women from being recognized and further exacerbating the gender inequality in the community. ”Allocational
harms,
stereotyping,
other
representational
harmsStereotyping
Type-level
and contextualized
embeddings
(May et al.,
2019)“[P]rominent word embeddings such as word2vec (Mikolov et
al., 2013) and GloVe (Pennington et al., 2014) encode systematic
biases against women and black people (Bolukbasi et al., 2016;
Garg et al., 2018), implicating many NLP systems in scaling up
social injustice. ”Vague Stereotyping
Dialogue
generation
(Liu et al.,
2019)“Since the goal of dialogue systems is to talk with users...if the


key: chroma:document
string_value:
Dialogue
generation
(Liu et al.,
2019)“Since the goal of dialogue systems is to talk with users...if the
systems show discriminatory behaviors in the interactions, the
user experience will be adversely affected. Moreover, public commercial chatbots can get resisted for their improper speech. ”Vague/unstated Stereotyping,
other
representational
harms,
questionable
correlations
Table 3: Examples of the categories into which the papers’ motivations and proposed quantitative techniques for
measuring or mitigating “bias” fall. Bold text in the quotes denotes the content that yields our categorizations.
et al. (2019); Précenth (2019); Pujari et al. (2019);
Ruane et al. (2019); Stanovsky et al. (2019);
Sun et al. (2019); Tan and Celis (2019); Webster
et al. (2019); Zmigrod et al. (2019); Gyamﬁ et al.
(2020); Hube et al. (2020); Hutchinson et al.
(2020); Kim et al. (2020); Nadeem et al. (2020);
Papakyriakopoulos et al. (2020); Ravfogel et al.
(2020); Rozado (2020); Sen and Ganguly (2020);


key: chroma:document
string_value:
Papakyriakopoulos et al. (2020); Ravfogel et al.
(2020); Rozado (2020); Sen and Ganguly (2020);
Shin et al. (2020); Strengers et al. (2020).
Other representational harms Hovy and Søgaard (2015); Blodgett et al. (2016); Bolukbasi
et al. (2016b); Hovy and Spruit (2016); Blodgett
and O’Connor (2017); Larson (2017); Schnoebelen
(2017); Blodgett et al. (2018); Curry and Rieser
(2018); Díaz et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Park et al. (2018);
Shen et al. (2018); Thelwall (2018); Zhao et al.
(2018b); Badjatiya et al. (2019); Bagdasaryan et al.
(2019); Bamman et al. (2019); Cao and Daumé
(2019); Chaloner and Maldonado (2019); Cho et al.
(2019); Davidson et al. (2019); De-Arteaga et al.
(2019); Fisher (2019); Font and Costa-jussà (2019);


key: chroma:document
string_value:
(2019); Davidson et al. (2019); De-Arteaga et al.
(2019); Fisher (2019); Font and Costa-jussà (2019);
Garimella et al. (2019); Loukina et al. (2019); Mayﬁeld et al. (2019); Mehrabi et al. (2019); Nozza
et al. (2019); Prabhakaran et al. (2019); Romanov
et al. (2019); Ruane et al. (2019); Sap et al. (2019);
Sheng et al. (2019); Sun et al. (2019); Sweeney
and Najaﬁan (2019); Vaidya et al. (2019); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020);
Hutchinson et al. (2020); Kim et al. (2020); Peng
et al. (2020); Rios (2020); Sap et al. (2020); Shah
et al. (2020); Sheng et al. (2020); Tan et al. (2020);
Zhang et al. (2020a,b).
Questionable correlations Jørgensen et al.
(2015); Hovy and Spruit (2016); Madnani et al.
(2017); Rudinger et al. (2017); Zhao et al. (2017);


key: chroma:document
string_value:
(2015); Hovy and Spruit (2016); Madnani et al.
(2017); Rudinger et al. (2017); Zhao et al. (2017);
Burns et al. (2018); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Lu et al. (2018);
Park et al. (2018); Shen et al. (2018); Zhang
et al. (2018); Badjatiya et al. (2019); Bhargava
and Forsyth (2019); Cao and Daumé (2019); Cho
et al. (2019); Davidson et al. (2019); Dev et al.
(2019); Garimella et al. (2019); Garg et al. (2019);
Huang et al. (2019); James-Sorenson and AlvarezMelis (2019); Kaneko and Bollegala (2019); Liu
et al. (2019); Karve et al. (2019); Nozza et al.
(2019); Prabhakaran et al. (2019); Romanov et al.
(2019); Sap et al. (2019); Sedoc and Ungar (2019);
Stanovsky et al. (2019); Sweeney and Najaﬁan


key: chroma:document
string_value:
(2019); Sap et al. (2019); Sedoc and Ungar (2019);
Stanovsky et al. (2019); Sweeney and Najaﬁan
(2019); Vaidya et al. (2019); Zhiltsova et al. (2019);Chopra et al. (2020); Gonen and Webster (2020);
Gyamﬁ et al. (2020); Hube et al. (2020); Ravfogel
et al. (2020); Rios (2020); Ross et al. (2020); Saunders and Byrne (2020); Sen and Ganguly (2020);
Shah et al. (2020); Sweeney and Najaﬁan (2020);
Yang and Feng (2020); Zhang et al. (2020a).
Vague/unstated Rudinger et al. (2018); Webster
et al. (2018); Dinan et al. (2019); Florez (2019);
Jumelet et al. (2019); Lauscher et al. (2019); Liang
et al. (2019); Maudslay et al. (2019); May et al.
(2019); Prates et al. (2019); Prost et al. (2019);
Qian et al. (2019); Swinger et al. (2019); Zhao


key: chroma:document
string_value:
(2019); Prates et al. (2019); Prost et al. (2019);
Qian et al. (2019); Swinger et al. (2019); Zhao
et al. (2019); Zhou et al. (2019); Ethayarajh (2020);
Huang et al. (2020); Jia et al. (2020); Popovi ´c et al.
(2020); Pryzant et al. (2020); Vig et al. (2020);
Wang et al. (2020); Zhao et al. (2020).
Surveys, frameworks, and meta-analyses
Hovy and Spruit (2016); Larson (2017); McCurdy
and Serbetçi (2017); Schnoebelen (2017); Basta
et al. (2019); Ethayarajh et al. (2019); Gonen and
Goldberg (2019); Lauscher and Glavaš (2019);
Loukina et al. (2019); Mayﬁeld et al. (2019);
Mirzaev et al. (2019); Prabhumoye et al. (2019);
Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);


key: chroma:document
string_value:
Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);
Shah et al. (2020); Strengers et al. (2020); Wright
et al. (2020).
B Full categorization: Techniques
Allocational harms De-Arteaga et al. (2019);
Prost et al. (2019); Romanov et al. (2019); Zhao
et al. (2020).
Stereotyping Bolukbasi et al. (2016a,b);
Caliskan et al. (2017); McCurdy and Serbetçi
(2017); Díaz et al. (2018); Santana et al. (2018);
Sutton et al. (2018); Zhang et al. (2018); Zhao
et al. (2018a,b); Agarwal et al. (2019); Basta et al.
(2019); Bhaskaran and Bhallamudi (2019); Brunet
et al. (2019); Cao and Daumé (2019); Chaloner
and Maldonado (2019); Dev and Phillips (2019);
Ethayarajh et al. (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);


key: chroma:document
string_value:
Ethayarajh et al. (2019); Gonen and Goldberg
(2019); James-Sorenson and Alvarez-Melis (2019);
Jumelet et al. (2019); Kaneko and Bollegala
(2019); Karve et al. (2019); Kurita et al. (2019);
Lauscher and Glavaš (2019); Lauscher et al.
(2019); Lee et al. (2019); Liang et al. (2019); Liu
et al. (2019); Manzini et al. (2019); Maudslay et al.
(2019); May et al. (2019); Mirzaev et al. (2019);
Prates et al. (2019); Précenth (2019); Prost et al.
(2019); Pujari et al. (2019); Qian et al. (2019);
Sedoc and Ungar (2019); Stanovsky et al. (2019);
Tan and Celis (2019); Zhao et al. (2019); Zhou
et al. (2019); Chopra et al. (2020); Gyamﬁ et al.
(2020); Nadeem et al. (2020); Nissim et al. (2020);
Papakyriakopoulos et al. (2020); Popovi ´c et al.


key: chroma:document
string_value:
Papakyriakopoulos et al. (2020); Popovi ´c et al.
(2020); Ravfogel et al. (2020); Ross et al. (2020);
Rozado (2020); Saunders and Byrne (2020); Shin
et al. (2020); Vig et al. (2020); Wang et al. (2020);
Yang and Feng (2020); Zhao et al. (2020).
Other representational harms Jørgensen et al.
(2015); Hovy and Søgaard (2015); Blodgett et al.
(2016); Blodgett and O’Connor (2017); Blodgett
et al. (2018); Curry and Rieser (2018); Dixon et al.
(2018); Park et al. (2018); Thelwall (2018); Webster et al. (2018); Badjatiya et al. (2019); Bagdasaryan et al. (2019); Bamman et al. (2019); Bhargava and Forsyth (2019); Cao and Daumé (2019);
Font and Costa-jussà (2019); Garg et al. (2019);
Garimella et al. (2019); Liu et al. (2019); Loukina et al. (2019); Mehrabi et al. (2019); Nozza


key: chroma:document
string_value:
Garimella et al. (2019); Liu et al. (2019); Loukina et al. (2019); Mehrabi et al. (2019); Nozza
et al. (2019); Sap et al. (2019); Sheng et al. (2019);
Stanovsky et al. (2019); Vaidya et al. (2019);
Webster et al. (2019); Ethayarajh (2020); Gaut
et al. (2020); Gencoglu (2020); Hovy et al. (2020);
Huang et al. (2020); Kim et al. (2020); Peng et al.
(2020); Ravfogel et al. (2020); Rios (2020); Sap
et al. (2020); Saunders and Byrne (2020); Sheng
et al. (2020); Sweeney and Najaﬁan (2020); Tan
et al. (2020); Zhang et al. (2020a,b).
Questionable correlations Jurgens et al. (2017);
Madnani et al. (2017); Rudinger et al. (2017);
Zhao et al. (2017); Burns et al. (2018); Díaz
et al. (2018); Kiritchenko and Mohammad (2018);
Lu et al. (2018); Rudinger et al. (2018); Shen


key: chroma:document
string_value:
et al. (2018); Kiritchenko and Mohammad (2018);
Lu et al. (2018); Rudinger et al. (2018); Shen
et al. (2018); Bordia and Bowman (2019); Cao
and Daumé (2019); Cho et al. (2019); Davidson et al. (2019); Dev et al. (2019); Dinan et al.
(2019); Fisher (2019); Florez (2019); Font and
Costa-jussà (2019); Garg et al. (2019); Huang et al.
(2019); Liu et al. (2019); Nozza et al. (2019);
Prabhakaran et al. (2019); Qian et al. (2019); Sap
et al. (2019); Stanovsky et al. (2019); Sweeney and
Najaﬁan (2019); Swinger et al. (2019); Zhiltsova
et al. (2019); Zmigrod et al. (2019); Hube et al.
(2020); Hutchinson et al. (2020); Jia et al. (2020);
Papakyriakopoulos et al. (2020); Popovi ´c et al.
(2020); Pryzant et al. (2020); Saunders and Byrne
(2020); Sen and Ganguly (2020); Shah et al. (2020);


key: chroma:document
string_value:
(2020); Pryzant et al. (2020); Saunders and Byrne
(2020); Sen and Ganguly (2020); Shah et al. (2020);
Sweeney and Najaﬁan (2020); Zhang et al. (2020b).
Vague/unstated None.Surveys, frameworks, and meta-analyses
Hovy and Spruit (2016); Larson (2017); McCurdy
and Serbetçi (2017); Schnoebelen (2017); Basta
et al. (2019); Ethayarajh et al. (2019); Gonen and
Goldberg (2019); Lauscher and Glavaš (2019);
Loukina et al. (2019); Mayﬁeld et al. (2019);
Mirzaev et al. (2019); Prabhumoye et al. (2019);
Ruane et al. (2019); Sedoc and Ungar (2019); Sun
et al. (2019); Nissim et al. (2020); Rozado (2020);
Shah et al. (2020); Strengers et al. (2020); Wright
et al. (2020).


key: chroma:document
string_value:
Language Models are Few-Shot Learners
Tom B. BrownBenjamin MannNick RyderMelanie Subbiah
Jared KaplanyPrafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry
Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan
Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter
Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray
Benjamin Chess Jack Clark Christopher Berner
Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei
OpenAI
Abstract
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training
on a large corpus of text followed by ﬁne-tuning on a speciﬁc task. While typically task-agnostic
in architecture, this method still requires task-speciﬁc ﬁne-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language task from only
a few examples or from simple instructions – something which current NLP systems still largely
struggle to do. Here we show that scaling up language models greatly improves task-agnostic,


key: chroma:document
string_value:
a few examples or from simple instructions – something which current NLP systems still largely
struggle to do. Here we show that scaling up language models greatly improves task-agnostic,
few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art ﬁnetuning approaches. Speciﬁcally, we train GPT-3, an autoregressive language model with 175 billion
parameters, 10x more than any previous non-sparse language model, and test its performance in
the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or ﬁne-tuning,
with tasks and few-shot demonstrations speciﬁed purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation, question-answering, and
cloze tasks, as well as several tasks that require on-the-ﬂy reasoning or domain adaptation, such as
unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same
time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some
datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally,


key: chroma:document
string_value:
datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally,
we ﬁnd that GPT-3 can generate samples of news articles which human evaluators have difﬁculty
distinguishing from articles written by humans. We discuss broader societal impacts of this ﬁnding
and of GPT-3 in general.
Equal contribution
yJohns Hopkins University, OpenAI
Author contributions listed at end of paper.arXiv:2005.14165v4  [cs.CL]  22 Jul 2020
Contents
1 Introduction 3
2 Approach 6
2.1 Model and Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Training Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9


key: chroma:document
string_value:
2.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3 Results 10
3.1 Language Modeling, Cloze, and Completion Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Closed Book Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.3 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Winograd-Style Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16


key: chroma:document
string_value:
3.5 Common Sense Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 Reading Comprehension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.7 SuperGLUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.8 NLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.9 Synthetic and Qualitative Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4 Measuring and Preventing Memorization Of Benchmarks 29
5 Limitations 33
6 Broader Impacts 34


key: chroma:document
string_value:
4 Measuring and Preventing Memorization Of Benchmarks 29
5 Limitations 33
6 Broader Impacts 34
6.1 Misuse of Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2 Fairness, Bias, and Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.3 Energy Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
7 Related Work 39
8 Conclusion 40
A Details of Common Crawl Filtering 43
B Details of Model Training 43
C Details of Test Set Contamination Studies 43
D Total Compute Used to Train Language Models 46
E Human Quality Assessment of Synthetic News Articles 46
F Additional Samples from GPT-3 48
G Details of Task Phrasing and Speciﬁcations 50
H Results on All Tasks for All Model Sizes 63
2
1 Introduction


key: chroma:document
string_value:
G Details of Task Phrasing and Speciﬁcations 50
H Results on All Tasks for All Model Sizes 63
2
1 Introduction
Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly
ﬂexible and task-agnostic ways for downstream transfer. First, single-layer representations were learned using word
vectors [ MCCD13 ,PSM14 ] and fed to task-speciﬁc architectures, then RNNs with multiple layers of representations
and contextual state were used to form stronger representations [ DL15 ,MBXS17 ,PNZtY18 ] (though still applied to
task-speciﬁc architectures), and more recently pre-trained recurrent or transformer language models [ VSP+17] have
been directly ﬁne-tuned, entirely removing the need for task-speciﬁc architectures [RNSS18, DCLT18, HR18].
This last paradigm has led to substantial progress on many challenging NLP tasks such as reading comprehension,
question answering, textual entailment, and many others, and has continued to advance based on new architectures
and algorithms [ RSR+19,LOG+19,YDY+19,LCG+19]. However, a major limitation to this approach is that while


key: chroma:document
string_value:
and algorithms [ RSR+19,LOG+19,YDY+19,LCG+19]. However, a major limitation to this approach is that while
the architecture is task-agnostic, there is still a need for task-speciﬁc datasets and task-speciﬁc ﬁne-tuning: to achieve
strong performance on a desired task typically requires ﬁne-tuning on a dataset of thousands to hundreds of thousands
of examples speciﬁc to that task. Removing this limitation would be desirable, for several reasons.
First, from a practical perspective, the need for a large dataset of labeled examples for every new task limits the
applicability of language models. There exists a very wide range of possible useful language tasks, encompassing
anything from correcting grammar, to generating examples of an abstract concept, to critiquing a short story. For many
of these tasks it is difﬁcult to collect a large supervised training dataset, especially when the process must be repeated
for every new task.
Second, the potential to exploit spurious correlations in training data fundamentally grows with the expressiveness
of the model and the narrowness of the training distribution. This can create problems for the pre-training plus
ﬁne-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then


key: chroma:document
string_value:
ﬁne-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then
ﬁne-tuned on very narrow task distributions. For instance [ HLW+20] observe that larger models do not necessarily
generalize better out-of-distribution. There is evidence that suggests that the generalization achieved under this paradigm
can be poor because the model is overly speciﬁc to the training distribution and does not generalize well outside it
[YdC+19,MPL19 ]. Thus, the performance of ﬁne-tuned models on speciﬁc benchmarks, even when it is nominally at
human-level, may exaggerate actual performance on the underlying task [GSL+18, NK19].
Third, humans do not require large supervised datasets to learn most language tasks – a brief directive in natural
language (e.g. “please tell me if this sentence describes something happy or something sad”) or at most a tiny number
of demonstrations (e.g. “here are two examples of people acting brave; please give a third example of bravery”) is often
Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad


key: chroma:document
string_value:
Figure 1.1: Language model meta-learning. During unsupervised pre-training, a language model develops a broad
set of skills and pattern recognition abilities. It then uses these abilities at inference time to rapidly adapt to or recognize
the desired task. We use the term “in-context learning” to describe the inner loop of this process, which occurs within
the forward-pass upon each sequence. The sequences in this diagram are not intended to be representative of the data a
model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded
within a single sequence.
3
Figure 1.2: Larger models make increasingly efﬁcient use of in-context information. We show in-context learning
performance on a simple task requiring the model to remove random symbols from a word, both with and without a
natural language task description (see Sec. 3.9.2). The steeper “in-context learning curves” for large models demonstrate
improved ability to learn a task from contextual information. We see qualitatively similar behavior across a wide range
of tasks.
sufﬁcient to enable a human to perform a new task to at least a reasonable degree of competence. Aside from pointing
to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans


key: chroma:document
string_value:
to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans
to seamlessly mix together or switch between many tasks and skills, for example performing addition during a lengthy
dialogue. To be broadly useful, we would someday like our NLP systems to have this same ﬂuidity and generality.
One potential route towards addressing these issues is meta-learning1– which in the context of language models means
the model develops a broad set of skills and pattern recognition abilities at training time, and then uses those abilities
at inference time to rapidly adapt to or recognize the desired task (illustrated in Figure 1.1). Recent work [ RWC+19]
attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form
of task speciﬁcation: the model is conditioned on a natural language instruction and/or a few demonstrations of the task
and is then expected to complete further instances of the task simply by predicting what comes next.
While it has shown some initial promise, this approach still achieves results far inferior to ﬁne-tuning – for example
[RWC+19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind


key: chroma:document
string_value:
[RWC+19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind
the state of the art. Meta-learning clearly requires substantial improvement in order to be viable as a practical method of
solving language tasks.
Another recent trend in language modeling may offer a way forward. In recent years the capacity of transformer
language models has increased substantially, from 100 million parameters [ RNSS18 ], to 300 million parameters
[DCLT18 ], to 1.5 billion parameters [ RWC+19], to 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19],
and ﬁnally 17 billion parameters [ Tur20 ]. Each increase has brought improvements in text synthesis and/or downstream
NLP tasks, and there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a
smooth trend of improvement with scale [ KMH+20]. Since in-context learning involves absorbing many skills and
tasks within the parameters of the model, it is plausible that in-context learning abilities might show similarly strong
gains with scale.
1In the context of language models this has sometimes been called “zero-shot transfer”, but this term is potentially ambiguous:
the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time


key: chroma:document
string_value:
the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time
demonstrations to the model, so is not truly learning from zero examples. To avoid this confusion, we use the term “meta-learning”
to capture the inner-loop / outer-loop structure of the general method, and the term “in context-learning” to refer to the inner
loop of meta-learning. We further specialize the description to “zero-shot”, “one-shot”, or “few-shot” depending on how many
demonstrations are provided at inference time. These terms are intended to remain agnostic on the question of whether the model
learns new tasks from scratch at inference time or simply recognizes patterns seen during training – this is an important issue which
we discuss later in the paper, but “meta-learning” is intended to encompass both possibilities, and simply describes the inner-outer
loop structure.
4
Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks While zero-shot performance
improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are
more proﬁcient at in-context learning. See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP
benchmark suite.


key: chroma:document
string_value:
more proﬁcient at in-context learning. See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP
benchmark suite.
In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call
GPT-3, and measuring its in-context learning abilities. Speciﬁcally, we evaluate GPT-3 on over two dozen NLP datasets,
as well as several novel tasks designed to test rapid adaptation to tasks unlikely to be directly contained in the training
set. For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we
allow as many demonstrations as will ﬁt into the model’s context window (typically 10 to 100), (b) “one-shot learning”,
where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only
an instruction in natural language is given to the model. GPT-3 could also in principle be evaluated in the traditional
ﬁne-tuning setting, but we leave this to future work.
Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to


key: chroma:document
string_value:
Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to
remove extraneous symbols from a word. Model performance improves with the addition of a natural language task
description, and with the number of examples in the model’s context, K. Few-shot learning also improves dramatically
with model size. Though the results in this case are particularly striking, the general trends with both model size and
number of examples in-context hold for most tasks we study. We emphasize that these “learning” curves involve no
gradient updates or ﬁne-tuning, just increasing numbers of demonstrations given as conditioning.
Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot
setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held
by ﬁne-tuned models). For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in
the one-shot setting, 85.0 F1 in the few-shot setting. Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the


key: chroma:document
string_value:
the one-shot setting, 85.0 F1 in the few-shot setting. Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the
zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art
relative to ﬁne-tuned models operating in the same closed-book setting.
GPT-3 also displays one-shot and few-shot proﬁciency at tasks designed to test rapid adaption or on-the-ﬂy reasoning,
which include unscrambling words, performing arithmetic, and using novel words in a sentence after seeing them
deﬁned only once. We also show that in the few-shot setting, GPT-3 can generate synthetic news articles which human
evaluators have difﬁculty distinguishing from human-generated articles.
At the same time, we also ﬁnd some tasks on which few-shot performance struggles, even at the scale of GPT-3. This
includes natural language inference tasks like the ANLI dataset, and some reading comprehension datasets like RACE
or QuAC. By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we


key: chroma:document
string_value:
or QuAC. By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we
hope to stimulate study of few-shot learning in language models and draw attention to where progress is most needed.
A heuristic sense of the overall results can be seen in Figure 1.3, which aggregates the various tasks (though it should
not be seen as a rigorous or meaningful benchmark in itself).
5
We also undertake a systematic study of “data contamination” – a growing problem when training high capacity models
on datasets such as Common Crawl, which can potentially include content from test datasets simply because such
content often exists on the web. In this paper we develop systematic tools to measure data contamination and quantify
its distorting effects. Although we ﬁnd that data contamination has a minimal effect on GPT-3’s performance on most
datasets, we do identify a few datasets where it could be inﬂating results, and we either do not report results on these
datasets or we note them with an asterisk, depending on the severity.
In addition to all the above, we also train a series of smaller models (ranging from 125 million parameters to 13 billion
parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings. Broadly, for most


key: chroma:document
string_value:
parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings. Broadly, for most
tasks we ﬁnd relatively smooth scaling with model capacity in all three settings; one notable pattern is that the gap
between zero-, one-, and few-shot performance often grows with model capacity, perhaps suggesting that larger models
are more proﬁcient meta-learners.
Finally, given the broad spectrum of capabilities displayed by GPT-3, we discuss concerns about bias, fairness, and
broader societal impacts, and attempt a preliminary analysis of GPT-3’s characteristics in this regard.
The remainder of this paper is organized as follows. In Section 2, we describe our approach and methods for training
GPT-3 and evaluating it. Section 3 presents results on the full range of tasks in the zero-, one- and few-shot settings.
Section 4 addresses questions of data contamination (train-test overlap). Section 5 discusses limitations of GPT-3.
Section 6 discusses broader impacts. Section 7 reviews related work and Section 8 concludes.
2 Approach
Our basic pre-training approach, including model, data, and training, is similar to the process described in [ RWC+19],
with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training. Our use


key: chroma:document
string_value:
with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training. Our use
of in-context learning is also similar to [ RWC+19], but in this work we systematically explore different settings for
learning within the context. Therefore, we start this section by explicitly deﬁning and contrasting the different settings
that we will be evaluating GPT-3 on or could in principle evaluate GPT-3 on. These settings can be seen as lying on a
spectrum of how much task-speciﬁc data they tend to rely on. Speciﬁcally, we can identify at least four points on this
spectrum (see Figure 2.1 for an illustration):
•Fine-Tuning (FT) has been the most common approach in recent years, and involves updating the weights of
a pre-trained model by training on a supervised dataset speciﬁc to the desired task. Typically thousands to
hundreds of thousands of labeled examples are used. The main advantage of ﬁne-tuning is strong performance
on many benchmarks. The main disadvantages are the need for a new large dataset for every task, the potential
for poor generalization out-of-distribution [ MPL19 ], and the potential to exploit spurious features of the
training data [ GSL+18,NK19 ], potentially resulting in an unfair comparison with human performance. In


key: chroma:document
string_value:
training data [ GSL+18,NK19 ], potentially resulting in an unfair comparison with human performance. In
this work we do not ﬁne-tune GPT-3 because our focus is on task-agnostic performance, but GPT-3 can be
ﬁne-tuned in principle and this is a promising direction for future work.
•Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few
demonstrations of the task at inference time as conditioning [ RWC+19], but no weight updates are allowed.
As shown in Figure 2.1, for a typical dataset an example has a context and a desired completion (for example
an English sentence and the French translation), and few-shot works by giving Kexamples of context and
completion, and then one ﬁnal example of context, with the model expected to provide the completion. We
typically set Kin the range of 10 to 100 as this is how many examples can ﬁt in the model’s context window
(nctx= 2048 ). The main advantages of few-shot are a major reduction in the need for task-speciﬁc data and
reduced potential to learn an overly narrow distribution from a large but narrow ﬁne-tuning dataset. The main


key: chroma:document
string_value:
reduced potential to learn an overly narrow distribution from a large but narrow ﬁne-tuning dataset. The main
disadvantage is that results from this method have so far been much worse than state-of-the-art ﬁne-tuned
models. Also, a small amount of task speciﬁc data is still required. As indicated by the name, few-shot
learning as described here for language models is related to few-shot learning as used in other contexts in
ML [ HYC01 ,VBL+16] – both involve learning based on a broad distribution of tasks (in this case implicit in
the pre-training data) and then rapidly adapting to a new task.
•One-Shot (1S) is the same as few-shot except that only one demonstration is allowed, in addition to a natural
language description of the task, as shown in Figure 1. The reason to distinguish one-shot from few-shot and
zero-shot (below) is that it most closely matches the way in which some tasks are communicated to humans.
For example, when asking humans to generate a dataset on a human worker service (for example Mechanical
Turk), it is common to give one demonstration of the task. By contrast it is sometimes difﬁcult to communicate
the content or format of a task if no examples are given.
6


key: chroma:document
string_value:
the content or format of a task if no examples are given.
6
Figure 2.1: Zero-shot, one-shot and few-shot, contrasted with traditional ﬁne-tuning . The panels above show
four methods for performing a task with a language model – ﬁne-tuning is the traditional method, whereas zero-, one-,
and few-shot, which we study in this work, require the model to perform the task with only forward passes at test
time. We typically present the model with a few dozen examples in the few shot setting. Exact phrasings for all task
descriptions, examples and prompts can be found in Appendix G.
•Zero-Shot (0S) is the same as one-shot except that no demonstrations are allowed, and the model is only given
a natural language instruction describing the task. This method provides maximum convenience, potential for
robustness, and avoidance of spurious correlations (unless they occur very broadly across the large corpus of
pre-training data), but is also the most challenging setting. In some cases it may even be difﬁcult for humans
to understand the format of the task without prior examples, so this setting is in some cases “unfairly hard”.
For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be


key: chroma:document
string_value:
For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be
ambiguous, as it may not be clear exactly what format the table should have or what should be included (and
even with careful clariﬁcation, understanding precisely what is desired can be difﬁcult). Nevertheless, for at
least some settings zero-shot is closest to how humans perform tasks – for example, in the translation example
in Figure 2.1, a human would likely know what to do from just the text instruction.
Figure 2.1 shows the four methods using the example of translating English to French. In this paper we focus on
zero-shot, one-shot and few-shot, with the aim of comparing them not as competing alternatives, but as different
problem settings which offer a varying trade-off between performance on speciﬁc benchmarks and sample efﬁciency.
We especially highlight the few-shot results as many of them are only slightly behind state-of-the-art ﬁne-tuned models.
Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance,
and are important targets for future work.


key: chroma:document
string_value:
Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance,
and are important targets for future work.
Sections 2.1-2.3 below give details on our models, training data, and training process respectively. Section 2.4 discusses
the details of how we do few-shot, one-shot, and zero-shot evaluations.
7
Model Name nparamsnlayersdmodelnheadsdhead Batch Size Learning Rate
GPT-3 Small 125M 12 768 12 64 0.5M 6:0104
GPT-3 Medium 350M 24 1024 16 64 0.5M 3:0104
GPT-3 Large 760M 24 1536 16 96 0.5M 2:5104
GPT-3 XL 1.3B 24 2048 24 128 1M 2:0104
GPT-3 2.7B 2.7B 32 2560 32 80 1M 1:6104
GPT-3 6.7B 6.7B 32 4096 32 128 2M 1:2104
GPT-3 13B 13.0B 40 5140 40 128 2M 1:0104


key: chroma:document
string_value:
GPT-3 13B 13.0B 40 5140 40 128 2M 1:0104
GPT-3 175B or “GPT-3” 175.0B 96 12288 96 128 3.2M 0:6104
Table 2.1: Sizes, architectures, and learning hyper-parameters (batch size in tokens and learning rate) of the models
which we trained. All models were trained for a total of 300 billion tokens.
2.1 Model and Architectures
We use the same model and architecture as GPT-2 [ RWC+19], including the modiﬁed initialization, pre-normalization,
and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse
attention patterns in the layers of the transformer, similar to the Sparse Transformer [ CGRS19 ]. To study the dependence
of ML performance on model size, we train 8 different sizes of model, ranging over three orders of magnitude from 125
million parameters to 175 billion parameters, with the last being the model we call GPT-3. Previous work [ KMH+20]
suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a
function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for
downstream language tasks.


key: chroma:document
string_value:
function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for
downstream language tasks.
Table 2.1 shows the sizes and architectures of our 8 models. Here nparams is the total number of trainable parameters,
nlayers is the total number of layers, dmodel is the number of units in each bottleneck layer (we always have the
feedforward layer four times the size of the bottleneck layer, d= 4dmodel ), anddhead is the dimension of each
attention head. All models use a context window of nctx= 2048 tokens. We partition the model across GPUs along
both the depth and width dimension in order to minimize data-transfer between nodes. The precise architectural
parameters for each model are chosen based on computational efﬁciency and load-balancing in the layout of models
across GPU’s. Previous work [ KMH+20] suggests that validation loss is not strongly sensitive to these parameters
within a reasonably broad range.
2.2 Training Dataset
Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset2[RSR+19] constituting
nearly a trillion words. This size of dataset is sufﬁcient to train our largest models without ever updating on the same


key: chroma:document
string_value:
nearly a trillion words. This size of dataset is sufﬁcient to train our largest models without ever updating on the same
sequence twice. However, we have found that unﬁltered or lightly ﬁltered versions of Common Crawl tend to have
lower quality than more curated datasets. Therefore, we took 3 steps to improve the average quality of our datasets:
(1) we downloaded and ﬁltered a version of CommonCrawl based on similarity to a range of high-quality reference
corpora, (2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy
and preserve the integrity of our held-out validation set as an accurate measure of overﬁtting, and (3) we also added
known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.
Details of the ﬁrst two points (processing of Common Crawl) are described in Appendix A. For the third, we added
several curated high-quality datasets, including an expanded version of the WebText dataset [ RWC+19], collected
by scraping links over a longer period of time, and ﬁrst described in [ KMH+20], two internet-based books corpora
(Books1 and Books2) and English-language Wikipedia.


key: chroma:document
string_value:
(Books1 and Books2) and English-language Wikipedia.
Table 2.2 shows the ﬁnal mixture of datasets that we used in training. The CommonCrawl data was downloaded from
41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before ﬁltering
and 570GB after ﬁltering, roughly equivalent to 400 billion byte-pair-encoded tokens. Note that during training, datasets
are not sampled in proportion to their size, but rather datasets we view as higher-quality are sampled more frequently,
such that CommonCrawl and Books2 datasets are sampled less than once during training, but the other datasets are
sampled 2-3 times. This essentially accepts a small amount of overﬁtting in exchange for higher quality training data.
2https://commoncrawl.org/the-data/
8
Figure 2.2: Total compute used during training . Based on the analysis in Scaling Laws For Neural Language Models
[KMH+20] we train much larger models on many fewer tokens than is typical. As a consequence, although GPT-3 3B
is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaﬂop/s-days of compute
during pre-training. Methodology for these calculations can be found in Appendix D.


key: chroma:document
string_value:
during pre-training. Methodology for these calculations can be found in Appendix D.
DatasetQuantity
(tokens)Weight in
training mixEpochs elapsed when
training for 300B tokens
Common Crawl (ﬁltered) 410 billion 60% 0.44
WebText2 19 billion 22% 2.9
Books1 12 billion 8% 1.9
Books2 55 billion 8% 0.43
Wikipedia 3 billion 3% 3.4
Table 2.2: Datasets used to train GPT-3 . “Weight in training mix” refers to the fraction of examples during training
that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset. As a
result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets
are seen less than once.
A major methodological concern with language models pretrained on a broad swath of internet data, particularly large
models with the capacity to memorize vast amounts of content, is potential contamination of downstream tasks by
having their test or development sets inadvertently seen during pre-training. To reduce such contamination, we searched
for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper.


key: chroma:document
string_value:
for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper.
Unfortunately, a bug in the ﬁltering caused us to ignore some overlaps, and due to the cost of training it was not feasible
to retrain the model. In Section 4 we characterize the impact of the remaining overlaps, and in future work we will
more aggressively remove data contamination.
2.3 Training Process
As found in [ KMH+20,MKAT18 ], larger models can typically use a larger batch size, but require a smaller learning
rate. We measure the gradient noise scale during training and use it to guide our choice of batch size [ MKAT18 ]. Table
2.1 shows the parameter settings we used. To train the larger models without running out of memory, we use a mixture
of model parallelism within each matrix multiply and model parallelism across the layers of the network. All models
were trained on V100 GPU’s on part of a high-bandwidth cluster provided by Microsoft. Details of the training process
and hyperparameter settings are described in Appendix B.
9
2.4 Evaluation
For few-shot learning, we evaluate each example in the evaluation set by randomly drawing Kexamples from that
task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task. For LAMBADA and Storycloze


key: chroma:document
string_value:
task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task. For LAMBADA and Storycloze
there is no supervised training set available so we draw conditioning examples from the development set and evaluate
on the test set. For Winograd (the original, not SuperGLUE version) there is only one dataset, so we draw conditioning
examples directly from it.
Kcan be any value from 0 to the maximum amount allowed by the model’s context window, which is nctx= 2048
for all models and typically ﬁts 10to100examples. Larger values of Kare usually but not always better, so when a
separate development and test set are available, we experiment with a few values of Kon the development set and then
run the best value on the test set. For some tasks (see Appendix G) we also use a natural language prompt in addition to
(or forK= 0, instead of) demonstrations.
On tasks that involve choosing one correct completion from several options (multiple choice), we provide Kexamples
of context plus correct completion, followed by one example of context only, and compare the LM likelihood of
each completion. For most tasks we compare the per-token likelihood (to normalize for length), however on a small


key: chroma:document
string_value:
each completion. For most tasks we compare the per-token likelihood (to normalize for length), however on a small
number of datasets (ARC, OpenBookQA, and RACE) we gain additional beneﬁt as measured on the development set
by normalizing by the unconditional probability of each completion, by computingP(completionjcontext)
P(completionjanswer context), where
answer context is the string "Answer: " or"A: " and is used to prompt that the completion should be an answer
but is otherwise generic.
On tasks that involve binary classiﬁcation, we give the options more semantically meaningful names (e.g. “True” or
“False” rather than 0 or 1) and then treat the task like multiple choice; we also sometimes frame the task similar to what
is done by [RSR+19] (see Appendix G) for details.
On tasks with free-form completion, we use beam search with the same parameters as [ RSR+19]: a beam width of 4
and a length penalty of = 0:6. We score the model using F1 similarity score, BLEU, or exact match, depending on
what is standard for the dataset at hand.
Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-,


key: chroma:document
string_value:
what is standard for the dataset at hand.
Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-,
and few-shot). When the test set is private, our model is often too large to ﬁt on the test server, so we report results on
the development set. We do submit to the test server on a small number of datasets (SuperGLUE, TriviaQA, PiQa)
where we were able to make submission work, and we submit only the 200B few-shot results, and report development
set results for everything else.
3 Results
In Figure 3.1 we display training curves for the 8 models described in Section 2. For this graph we also include 6
additional extra-small models with as few as 100,000 parameters. As observed in [ KMH+20], language modeling
performance follows a power-law when making efﬁcient use of training compute. After extending this trend by two
more orders of magnitude, we observe only a slight (if any) departure from the power-law. One might worry that these
improvements in cross-entropy loss come only from modeling spurious details of our training corpus. However, we will
see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a
broad spectrum of natural language tasks.


key: chroma:document
string_value:
see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a
broad spectrum of natural language tasks.
Below, we evaluate the 8 models described in Section 2 (the 175 billion parameter parameter GPT-3 and 7 smaller
models) on a wide range of datasets. We group the datasets into 9 categories representing roughly similar tasks.
In Section 3.1 we evaluate on traditional language modeling tasks and tasks that are similar to language modeling,
such as Cloze tasks and sentence/paragraph completion tasks. In Section 3.2 we evaluate on “closed book” question
answering tasks: tasks which require using the information stored in the model’s parameters to answer general
knowledge questions. In Section 3.3 we evaluate the model’s ability to translate between languages (especially one-shot
and few-shot). In Section 3.4 we evaluate the model’s performance on Winograd Schema-like tasks. In Section 3.5 we
evaluate on datasets that involve commonsense reasoning or question answering. In Section 3.6 we evaluate on reading
comprehension tasks, in Section 3.7 we evaluate on the SuperGLUE benchmark suite, and in 3.8 we brieﬂy explore
NLI. Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities –


key: chroma:document
string_value:
NLI. Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities –
these tasks focus on on-the-ﬂy reasoning, adaptation skills, or open-ended text synthesis. We evaluate all tasks in the
few-shot, one-shot, and zero-shot settings.
10
Figure 3.1: Smooth scaling of performance with compute. Performance (measured in terms of cross-entropy
validation loss) follows a power-law trend with the amount of compute used for training. The power-law behavior
observed in [ KMH+20] continues for an additional two orders of magnitude with only small deviations from the
predicted curve. For this ﬁgure, we exclude embedding parameters from compute and parameter counts.
Setting PTB
SOTA (Zero-Shot) 35.8a
GPT-3 Zero-Shot 20.5
Table 3.1: Zero-shot results on PTB language modeling dataset. Many other common language modeling datasets
are omitted because they are derived from Wikipedia or other sources which are included in GPT-3’s training data.
a[RWC+19]
3.1 Language Modeling, Cloze, and Completion Tasks
In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks


key: chroma:document
string_value:
In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks
that involve predicting a single word of interest, completing a sentence or paragraph, or choosing between possible
completions of a piece of text.
3.1.1 Language Modeling
We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [ MKM+94] dataset measured in [ RWC+19]. We omit
the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the
one-billion word benchmark due to a high fraction of the dataset being contained in our training set. PTB escapes these
issues due to predating the modern internet. Our largest model sets a new SOTA on PTB by a substantial margin of 15
points, achieving a perplexity of 20.50. Note that since PTB is a traditional language modeling dataset it does not have
a clear separation of examples to deﬁne one-shot or few-shot evaluation around, so we measure only zero-shot.
3.1.2 LAMBADA
The LAMBADA dataset [ PKL+16] tests the modeling of long-range dependencies in text – the model is asked to
predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the


key: chroma:document
string_value:
predict the last word of sentences which require reading a paragraph of context. It has recently been suggested that the
continued scaling of language models is yielding diminishing returns on this difﬁcult benchmark. [ BHT+20] reﬂect on
the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([ SPP+19]
11
SettingLAMBADA
(acc)LAMBADA
(ppl)StoryCloze
(acc)HellaSwag
(acc)
SOTA 68.0a8.63b91.8c85.6d
GPT-3 Zero-Shot 76.2 3.00 83.2 78.9
GPT-3 One-Shot 72.5 3.35 84.7 78.1
GPT-3 Few-Shot 86.4 1.92 87.7 79.3
Table 3.2: Performance on cloze and completion tasks. GPT-3 signiﬁcantly improves SOTA on LAMBADA while
achieving respectable performance on two difﬁcult completion prediction datasets.a[Tur20 ]b[RWC+19]c[LDL19 ]
d[LCH+20]


key: chroma:document
string_value:
d[LCH+20]
Figure 3.2: On LAMBADA, the few-shot capability of language models results in a strong boost to accuracy. GPT-3
2.7B outperforms the SOTA 17B parameter Turing-NLG [ Tur20 ] in this setting, and GPT-3 175B advances the state of
the art by 18%. Note zero-shot uses a different format from one-shot and few-shot as described in the text.
and [ Tur20 ]) and argue that “continuing to expand hardware and data sizes by orders of magnitude is not the path
forward”. We ﬁnd that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of
8% over the previous state of the art.
LAMBADA is also a demonstration of the ﬂexibility of few-shot learning as it provides a way to address a problem that
classically occurs with this dataset. Although the completion in LAMBADA is always the last word in a sentence, a
standard language model has no way of knowing this detail. It thus assigns probability not only to the correct ending but
also to other valid continuations of the paragraph. This problem has been partially addressed in the past with stop-word


key: chroma:document
string_value:
also to other valid continuations of the paragraph. This problem has been partially addressed in the past with stop-word
ﬁlters [ RWC+19] (which ban “continuation” words). The few-shot setting instead allows us to “frame” the task as a
cloze-test and allows the language model to infer from examples that a completion of exactly one word is desired. We
use the following ﬁll-in-the-blank format:
Alice was friends with Bob. Alice went to visit her friend .!Bob
George bought some baseball equipment, a ball, a glove, and a .!
When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase
of over 18% from the previous state-of-the-art. We observe that few-shot performance improves strongly with model
size. While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy
by 10%. Finally, the ﬁll-in-blank method is not effective one-shot, where it always performs worse than the zero-shot
setting. Perhaps this is because all models still require several examples to recognize the pattern.
12
Setting NaturalQS WebQS TriviaQA


key: chroma:document
string_value:
setting. Perhaps this is because all models still require several examples to recognize the pattern.
12
Setting NaturalQS WebQS TriviaQA
RAG (Fine-tuned, Open-Domain) [LPP+20] 44.5 45.5 68.0
T5-11B+SSM (Fine-tuned, Closed-Book) [RRS20] 36.6 44.7 60.5
T5-11B (Fine-tuned, Closed-Book) 34.5 37.4 50.1
GPT-3 Zero-Shot 14.6 14.4 64.3
GPT-3 One-Shot 23.0 25.3 68.0
GPT-3 Few-Shot 29.9 41.5 71.2
Table 3.3: Results on three Open-Domain QA tasks. GPT-3 is shown in the few-, one-, and zero-shot settings, as
compared to prior SOTA results for closed book and open domain settings. TriviaQA few-shot result is evaluated on the
wiki split test server.
One note of caution is that an analysis of test set contamination identiﬁed that a signiﬁcant minority of the LAMBADA
dataset appears to be present in our training data – however analysis performed in Section 4 suggests negligible impact
on performance.


key: chroma:document
string_value:
dataset appears to be present in our training data – however analysis performed in Section 4 suggests negligible impact
on performance.
3.1.3 HellaSwag
The HellaSwag dataset [ ZHB+19] involves picking the best ending to a story or set of instructions. The examples were
adversarially mined to be difﬁcult for language models while remaining easy for humans (who achieve 95.6% accuracy).
GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the
75.4% accuracy of a ﬁne-tuned 1.5B parameter language model [ ZHR+19] but still a fair amount lower than the overall
SOTA of 85.6% achieved by the ﬁne-tuned multi-task model ALUM.
3.1.4 StoryCloze
We next evaluate GPT-3 on the StoryCloze 2016 dataset [ MCH+16], which involves selecting the correct ending
sentence for ﬁve-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot


key: chroma:document
string_value:
sentence for ﬁve-sentence long stories. Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot
setting (with K= 70 ). This is still 4.1% lower than the ﬁne-tuned SOTA using a BERT based model [ LDL19 ] but
improves over previous zero-shot results by roughly 10%.
3.2 Closed Book Question Answering
In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense
amount of possible queries, this task has normally been approached by using an information retrieval system to ﬁnd
relevant text in combination with a model which learns to generate an answer given the question and the retrieved
text. Since this setting allows a system to search for and condition on text which potentially contains the answer it
is denoted “open-book”. [ RRS20 ] recently demonstrated that a large language model can perform surprisingly well
directly answering the questions without conditioning on auxilliary information. They denote this more restrictive
evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better


key: chroma:document
string_value:
evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better
and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [ RRS20 ]: Natural Questions [ KPR+19],
WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in
the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than
previous closed-book QA work: in addition to external content not being allowed, ﬁne-tuning on the Q&A dataset itself
is also not permitted.
The results for GPT-3 are shown in Table 3.3. On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the
one-shot setting, and 71.2% in the few-shot setting. The zero-shot result already outperforms the ﬁne-tuned T5-11B by
14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%. The one-shot


key: chroma:document
string_value:
14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%. The one-shot
result improves by 3.7% and matches the SOTA for an open-domain QA system which not only ﬁne-tunes but also
makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [ LPP+20].
GPT-3’s few-shot result further improves performance another 3.2% beyond this.
On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5%
in the few-shot setting. This compares to 37.4% for ﬁne-tuned T5-11B, and 44.7% for ﬁne-tuned T5-11B+SSM,
which uses a Q&A-speciﬁc pre-training procedure. GPT-3 in the few-shot setting approaches the performance of
state-of-the-art ﬁne-tuned models. Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to


key: chroma:document
string_value:
state-of-the-art ﬁne-tuned models. Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to
few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions
13
Figure 3.3: On TriviaQA GPT3’s performance grows smoothly with model size, suggesting that language models
continue to absorb knowledge as their capacity increases. One-shot and few-shot performance make signiﬁcant gains
over zero-shot behavior, matching and exceeding the performance of the SOTA ﬁne-tuned open-domain model, RAG
[LPP+20]
and/or the style of their answers are out-of-distribution for GPT-3. Nevertheless, GPT-3 appears able to adapt to this
distribution, recovering strong performance in the few-shot setting.
On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in
the few-shot setting, compared to 36.6% for ﬁne-tuned T5 11B+SSM. Similar to WebQS, the large gain from zero-shot


key: chroma:document
string_value:
to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to
TriviaQA and WebQS. In particular, the questions in NQs tend towards very ﬁne-grained knowledge on Wikipedia
speciﬁcally which could be testing the limits of GPT-3’s capacity and broad pretraining distribution.
Overall, on one of the three datasets GPT-3’s one-shot matches the open-domain ﬁne-tuning SOTA. On the other two
datasets it approaches the performance of the closed-book SOTA despite not using ﬁne-tuning. On all 3 datasets, we
ﬁnd that performance scales very smoothly with model size (Figure 3.3 and Appendix H Figure H.7), possibly reﬂecting
the idea that model capacity translates directly to more ‘knowledge’ absorbed in the parameters of the model.
3.3 Translation
For GPT-2 a ﬁlter was used on a multilingual collection of documents to produce an English only dataset due to capacity
concerns. Even with this ﬁltering GPT-2 showed some evidence of multilingual capability and performed non-trivially
when translating between French and English despite only training on 10 megabytes of remaining French text. Since we


key: chroma:document
string_value:
when translating between French and English despite only training on 10 megabytes of remaining French text. Since we
increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training
dataset to include more representation of other languages, though this remains an area for further improvement. As
discussed in 2.2 the majority of our data is derived from raw Common Crawl with only quality-based ﬁltering. Although
GPT-3’s training data is still primarily English (93% by word count), it also includes 7% of text in other languages.
These languages are documented in the supplemental material. In order to better understand translation capability, we
also expand our analysis to include two additional commonly studied languages, German and Romanian.
Existing unsupervised machine translation approaches often combine pretraining on a pair of monolingual datasets
with back-translation [ SHB15 ] to bridge the two languages in a controlled way. By contrast, GPT-3 learns from a
blend of training data that mixes many languages together in a natural way, combining them on a word, sentence,
and document level. GPT-3 also uses a single training objective which is not customized or designed for any task in
particular. However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make


key: chroma:document
string_value:
particular. However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make
use of a small amount of paired examples (1 or 64). This corresponds to up to a page or two of in-context training data.
Results are shown in Table 3.4. Zero-shot GPT-3, which only receives on a natural language description of the task,
still underperforms recent unsupervised NMT results. However, providing only a single example demonstration for
14
Setting En !Fr Fr!En En!De De!En En!Ro Ro!En
SOTA (Supervised) 45.6a35.0b41.2c40.2d38.5e39.9e
XLM [LC19] 33.4 33.3 26.4 34.3 33.3 31.8
MASS [STQ+19] 37.5 34.9 28.3 35.2 35.2 33.1
mBART [LGG+20] - - 29.8 34.0 35.0 30.5
GPT-3 Zero-Shot 25.2 21.2 24.6 27.2 14.1 19.9
GPT-3 One-Shot 28.3 33.7 26.2 30.4 20.6 38.6


key: chroma:document
string_value:
GPT-3 One-Shot 28.3 33.7 26.2 30.4 20.6 38.6
GPT-3 Few-Shot 32.6 39.2 29.7 40.6 21.0 39.5
Table 3.4: Few-shot GPT-3 outperforms previous unsupervised NMT work by 5 BLEU when translating
into English reﬂecting its strength as an English LM. We report BLEU scores on the WMT’14 Fr $En,
WMT’16 De$En, and WMT’16 Ro $En datasets as measured by multi-bleu.perl with XLM’s tokenization in order to compare most closely with prior unsupervised NMT work. SacreBLEUf[Pos18 ] results reported in Appendix H. Underline indicates an unsupervised or few-shot SOTA, bold indicates supervised SOTA
with relative conﬁdence.a[EOAG18 ]b[DHKH14 ]c[WXH+18]d[oR16 ]e[LGG+20]f[SacreBLEU signature:
BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]


key: chroma:document
string_value:
BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20]
Figure 3.4: Few-shot translation performance on 6 language pairs as model capacity increases. There is a consistent
trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be
stronger than translation from English.
15
Setting Winograd Winogrande (XL)
Fine-tuned SOTA 90.1a84.6b
GPT-3 Zero-Shot 88.3* 70.2
GPT-3 One-Shot 89.7* 73.2
GPT-3 Few-Shot 88.6* 77.7
Table 3.5: Results on the WSC273 version of Winograd schemas and the adversarial Winogrande dataset. See Section
4 for details on potential contamination of the Winograd test set.a[SBBC19]b[LYN+20]
Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales.
Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B
is competitive with a ﬁne-tuned RoBERTA-large.


key: chroma:document
string_value:
is competitive with a ﬁne-tuned RoBERTA-large.
each translation task improves performance by over 7 BLEU and nears competitive performance with prior work.
GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior
unsupervised NMT work. GPT-3 has a noticeable skew in its performance depending on language direction. For the
three input languages studied, GPT-3 signiﬁcantly outperforms prior unsupervised NMT work when translating into
English but underperforms when translating in the other direction. Performance on En-Ro is a noticeable outlier at
over 10 BLEU worse than prior unsupervised NMT work. This could be a weakness due to reusing the byte-level BPE
tokenizer of GPT-2 which was developed for an almost entirely English training dataset. For both Fr-En and De-En,
few shot GPT-3 outperforms the best supervised result we could ﬁnd but due to our unfamiliarity with the literature and
the appearance that these are un-competitive benchmarks we do not suspect those results represent true state of the art.
For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of


key: chroma:document
string_value:
For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of
unsupervised pretraining, supervised ﬁnetuning on 608K labeled examples, and backtranslation [LHCG19b].
Finally, across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of
improvement with model capacity. This is shown in Figure 3.4 in the case of few-shot results, and scaling for all three
settings is shown in Appendix H.
3.4 Winograd-Style Tasks
The Winograd Schemas Challenge [ LDM12 ] is a classical task in NLP that involves determining which word a pronoun
refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human. Recently ﬁne-tuned
language models have achieved near-human performance on the original Winograd dataset, but more difﬁcult versions
16
Setting PIQA ARC (Easy) ARC (Challenge) OpenBookQA
Fine-tuned SOTA 79.4 92.0[KKS+20] 78.5[KKS+20] 87.2[KKS+20]
GPT-3 Zero-Shot 80.5* 68.8 51.4 57.6


key: chroma:document
string_value:
GPT-3 Zero-Shot 80.5* 68.8 51.4 57.6
GPT-3 One-Shot 80.5* 71.2 53.2 58.8
GPT-3 Few-Shot 82.8* 70.1 51.5 65.4
Table 3.6: GPT-3 results on three commonsense reasoning tasks, PIQA, ARC, and OpenBookQA. GPT-3 Few-Shot
PIQA result is evaluated on the test server. See Section 4 for details on potential contamination issues on the PIQA test
set.
Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings. The largest model achieves a
score on the development set in all three conditions that exceeds the best recorded score on the task.
such as the adversarially-mined Winogrande dataset [ SBBC19 ] still signiﬁcantly lag human performance. We test
GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.
On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method


key: chroma:document
string_value:
On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method
described in [ RWC+19]. Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which
is presented as binary classiﬁcation and requires entity extraction to convert to the form described in this section. On
Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear
in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human
performance. We note that contamination analysis found some Winograd schemas in the training data but this appears
to have only a small effect on results (see Section 4).
On the more difﬁcult Winogrande dataset, we do ﬁnd gains to in-context learning: GPT-3 achieves 70.2% in the
zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a ﬁne-tuned


key: chroma:document
string_value:
zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting. For comparison a ﬁne-tuned
RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a ﬁne-tuned high capacity model (T5), and
human performance on the task as reported by [SBBC19] is 94.0%.
3.5 Common Sense Reasoning
Next we consider three datasets which attempt to capture physical or scientiﬁc reasoning, as distinct from sentence
completion, reading comprehension, or broad knowledge question answering. The ﬁrst, PhysicalQA (PIQA) [ BZB+19],
asks common sense questions about how the physical world works and is intended as a probe of grounded understanding
of the world. GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot
(the last measured on PIQA’s test server). This compares favorably to the 79.4% accuracy prior state-of-the-art of a
17
Setting CoQA DROP QuAC SQuADv2 RACE-h RACE-m


key: chroma:document
string_value:
17
Setting CoQA DROP QuAC SQuADv2 RACE-h RACE-m
Fine-tuned SOTA 90.7a89.1b74.4c93.0d90.0e93.1e
GPT-3 Zero-Shot 81.5 23.6 41.5 59.5 45.5 58.4
GPT-3 One-Shot 84.0 34.3 43.3 65.4 45.9 57.4
GPT-3 Few-Shot 85.0 36.5 44.3 69.8 46.8 58.1
Table 3.7: Results on reading comprehension tasks. All scores are F1 except results for RACE which report accuracy.
a[JZC+19]b[JN20]c[AI19]d[QIA20]e[SPP+19]
ﬁne-tuned RoBERTa. PIQA shows relatively shallow scaling with model size and is still over 10% worse than human
performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art. Our analysis
ﬂagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark
the result with an asterisk. See Section 4 for details.


key: chroma:document
string_value:
ﬂagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark
the result with an asterisk. See Section 4 for details.
ARC [ CCE+18] is a dataset of multiple-choice questions collected from 3rd to 9th grade science exams. On the
“Challenge” version of the dataset which has been ﬁltered to questions which simple statistical or information retrieval
methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot
setting, and 51.5% in the few-shot setting. This is approaching the performance of a ﬁne-tuned RoBERTa baseline
(55.9%) from UniﬁedQA [ KKS+20]. On the “Easy” version of the dataset (questions which either of the mentioned
baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a ﬁne-tuned
RoBERTa baseline from [ KKS+20]. However, both of these results are still much worse than the overall SOTAs


key: chroma:document
string_value:
RoBERTa baseline from [ KKS+20]. However, both of these results are still much worse than the overall SOTAs
achieved by the UniﬁedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% on the easy
set.
On OpenBookQA [ MCKS18 ], GPT-3 improves signiﬁcantly from zero to few shot settings but is still over 20 points
short of the overall SOTA. GPT-3’s few-shot performance is similar to a ﬁne-tuned BERT Large baseline on the
leaderboard.
Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and
inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a signiﬁcant
improvement is observed on OpenBookQA. GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings.
3.6 Reading Comprehension
Next we evaluate GPT-3 on the task of reading comprehension. We use a suite of 5 datasets including abstractive,
multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread


key: chroma:document
string_value:
multiple choice, and span based answer formats in both dialog and single question settings. We observe a wide spread
in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats. In general
we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each
respective dataset.
GPT-3 performs best (within 3 points of the human baseline) on CoQA [ RCM19 ] a free-form conversational dataset
and performs worst (13 F1 below an ELMo baseline) on QuAC [ CHI+18] a dataset which requires modeling structured
dialog acts and answer span selections of teacher-student interactions. On DROP [ DWD+19], a dataset testing discrete
reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the ﬁne-tuned
BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches
which augment neural networks with symbolic systems [ RLL+19]. On SQuAD 2.0 [ RJL18 ], GPT-3 demonstrates its
few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to


key: chroma:document
string_value:
few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting. This allows it to
slightly outperform the best ﬁne-tuned result in the original paper. On RACE [ LXL+17], a multiple choice dataset of
middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with
the earliest work utilizing contextual representations and is still 45% behind SOTA.
3.7 SuperGLUE
In order to better aggregate results on NLP tasks and compare to popular models such as BERT and RoBERTa in a
more systematic way, we also evaluate GPT-3 on a standardized collection of datasets, the SuperGLUE benchmark
[WPN+19] [WPN+19] [CLC+19] [DMST19 ] [RBG11 ] [KCR+18] [ZLL+18] [DGM06 ] [BHDD+06] [GMDD07 ]
[BDD+09] [PCC18 ] [PHR+18]. GPT-3’s test-set performance on the SuperGLUE dataset is shown in Table 3.8. In the
few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC
18


key: chroma:document
string_value:
few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set. For all tasks except WSC
18
Figure 3.7: GPT-3 results on CoQA reading comprehension task. GPT-3 175B achieves 85 F1 in the few-shot setting,
only a few points behind measured human performance and state-of-the-art ﬁne-tuned models. Zero-shot and one-shot
performance is a few points behind, with the gains to few-shot being largest for bigger models.
SuperGLUE BoolQ CB CB COPA RTE
Average Accuracy Accuracy F1 Accuracy Accuracy
Fine-tuned SOTA 89.0 91.0 96.9 93.9 94.8 92.5
Fine-tuned BERT-Large 69.0 77.4 83.6 75.7 70.6 71.7
GPT-3 Few-Shot 71.8 76.4 75.6 52.0 92.0 69.0
WiC WSC MultiRC MultiRC ReCoRD ReCoRD
Accuracy Accuracy Accuracy F1a Accuracy F1
Fine-tuned SOTA 76.1 93.8 62.3 88.2 92.5 93.3
Fine-tuned BERT-Large 69.6 64.6 24.1 70.0 71.3 72.0


key: chroma:document
string_value:
Fine-tuned BERT-Large 69.6 64.6 24.1 70.0 71.3 72.0
GPT-3 Few-Shot 49.4 80.1 30.5 75.4 90.2 91.1
Table 3.8: Performance of GPT-3 on SuperGLUE compared to ﬁne-tuned baselines and SOTA. All results are reported
on the test set. GPT-3 few-shot is given a total of 32 examples within the context of each task and performs no gradient
updates.
19
Figure 3.8: Performance on SuperGLUE increases with model size and number of examples in context. A value
ofK= 32 means that our model was shown 32 examples per task, for 256 examples total divided across the 8 tasks in
SuperGLUE. We report GPT-3 values on the dev set, so our numbers are not directly comparable to the dotted reference
lines (our test set results are in Table 3.8). The BERT-Large reference model was ﬁne-tuned on the SuperGLUE training
set (125K examples), whereas BERT++ was ﬁrst ﬁne-tuned on MultiNLI (392K examples) and SWAG (113K examples)


key: chroma:document
string_value:
set (125K examples), whereas BERT++ was ﬁrst ﬁne-tuned on MultiNLI (392K examples) and SWAG (113K examples)
before further ﬁne-tuning on the SuperGLUE training set (for a total of 630K ﬁne-tuning examples). We ﬁnd the
difference in performance between the BERT-Large and BERT++ to be roughly equivalent to the difference between
GPT-3 with one example per context versus eight examples per context.
and MultiRC, we sampled a new set of examples to use in the context for each problem. For WSC and MultiRC, we
used the same set of randomly drawn examples from the training set as context for all of the problems we evaluated.
We observe a wide range in GPT-3’s performance across tasks. On COPA and ReCoRD GPT-3 achieves near-SOTA
performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving
second place on the leaderboard, where ﬁrst place is held by a ﬁne-tuned 11 billion parameter model (T5). On WSC,
performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the


key: chroma:document
string_value:
performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the
original Winograd dataset as described in Section 3.4). On BoolQ, MultiRC, and RTE, performance is reasonable,
roughly matching that of a ﬁne-tuned BERT-Large. On CB, we see signs of life at 75.6% in the few-shot setting.
WiC is a notable weak spot with few-shot performance at 49.4% (at random chance). We tried a number of different
phrasings and formulations for WiC (which involves determining if a word is being used with the same meaning in two
sentences), none of which was able to achieve strong performance. This hints at a phenomenon that will become clearer
in the next section (which discusses the ANLI benchmark) – GPT-3 appears to be weak in the few-shot or one-shot
setting at some tasks that involve comparing two sentences or snippets, for example whether a word is used the same
way in two sentences (WiC), whether one sentence is a paraphrase of another, or whether one sentence implies another.
This could also explain the comparatively low scores for RTE and CB, which also follow this format. Despite these


key: chroma:document
string_value:
This could also explain the comparatively low scores for RTE and CB, which also follow this format. Despite these
weaknesses, GPT-3 still outperforms a ﬁne-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to
the state-of-the-art held by a ﬁne-tuned 11 billion parameter model.
Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of
examples in the context showing increasing beneﬁts from in-context learning (Figure 3.8). We scale Kup to 32
examples per task, after which point additional examples will not reliably ﬁt into our context. When sweeping over
values ofK, we ﬁnd that GPT-3 requires less than eight total examples per task to outperform a ﬁne-tuned BERT-Large
on overall SuperGLUE score.
3.8 NLI
Natural Language Inference (NLI) [ Fyo00 ] concerns the ability to understand the relationship between two sentences.
In practice, this task is usually structured as a two or three class classiﬁcation problem where the model classiﬁes
20


key: chroma:document
string_value:
In practice, this task is usually structured as a two or three class classiﬁcation problem where the model classiﬁes
20
Figure 3.9: Performance of GPT-3 on ANLI Round 3. Results are on the dev-set, which has only 1500 examples
and therefore has high variance (we estimate a standard deviation of 1.2%). We ﬁnd that smaller models hover around
random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA. Results for
ANLI rounds 1 and 2 are shown in the appendix.
whether the second sentence logically follows from the ﬁrst, contradicts the ﬁrst sentence, or is possibly true (neutral).
SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task. On RTE, only the largest
version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting
GPT-3 performs similarly to a single-task ﬁne-tuned BERT Large. We also evaluate on the recently introduced
Adversarial Natural Language Inference (ANLI) dataset [ NWD+19]. ANLI is a difﬁcult dataset employing a series of


key: chroma:document
string_value:
Adversarial Natural Language Inference (ANLI) dataset [ NWD+19]. ANLI is a difﬁcult dataset employing a series of
adversarially mined natural language inference questions in three rounds (R1, R2, and R3). Similar to RTE, all of our
models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting ( 33%),
whereas GPT-3 itself shows signs of life on Round 3. Results for ANLI R3 are highlighted in Figure 3.9 and full results
for all rounds can be found in Appendix H. These results on both RTE and ANLI suggest that NLI is still a very difﬁcult
task for language models and they are only just beginning to show signs of progress.
3.9 Synthetic and Qualitative Tasks
One way to probe GPT-3’s range of abilities in the few-shot (or zero- and one-shot) setting is to give it tasks which
require it to perform simple on-the-ﬂy computational reasoning, recognize a novel pattern that is unlikely to have
occurred in training, or adapt quickly to an unusual task. We devise several tasks to test this class of abilities. First, we


key: chroma:document
string_value:
occurred in training, or adapt quickly to an unusual task. We devise several tasks to test this class of abilities. First, we
test GPT-3’s ability to perform arithmetic. Second, we create several tasks that involve rearranging or unscrambling the
letters in a word, tasks which are unlikely to have been exactly seen during training. Third, we test GPT-3’s ability to
solve SAT-style analogy problems few-shot. Finally, we test GPT-3 on several qualitative tasks, including using new
words in a sentence, correcting English grammar, and news article generation. We will release the synthetic datasets
with the hope of stimulating further study of test-time behavior of language models.
3.9.1 Arithmetic
To test GPT-3’s ability to perform simple arithmetic operations without task-speciﬁc training, we developed a small
battery of 10 tests that involve asking GPT-3 a simple arithmetic problem in natural language:
•2 digit addition (2D+) – The model is asked to add two integers sampled uniformly from [0;100) , phrased in
the form of a question, e.g. “Q: What is 48 plus 76? A: 124.”
•2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0;100) ; the


key: chroma:document
string_value:
•2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0;100) ; the
answer may be negative. Example: “Q: What is 34 minus 53? A: -19”.
•3 digit addition (3D+) – Same as 2 digit addition, except numbers are uniformly sampled from [0;1000) .
21
Figure 3.10: Results on all 10 arithmetic tasks in the few-shot settings for models of different sizes. There is a
signiﬁcant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being
able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a signiﬁcant fraction
of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations. Results for one-shot and zero-shot
are shown in the appendix.
•3 digit subtraction (3D-) – Same as 2 digit subtraction, except numbers are uniformly sampled from [0;1000) .
•4 digit addition (4D+) – Same as 3 digit addition, except uniformly sampled from [0;10000) .
•4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0;10000) .


key: chroma:document
string_value:
•4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0;10000) .
•5 digit addition (5D+) – Same as 3 digit addition, except uniformly sampled from [0;100000) .
•5 digit subtraction (5D-) – Same as 3 digit subtraction, except uniformly sampled from [0;100000) .
•2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from [0;100) ,
e.g. “Q: What is 24 times 42? A: 1008”.
•One-digit composite (1DC) – The model is asked to perform a composite operation on three 1 digit numbers,
with parentheses around the last two. For example, “Q: What is 6+(4*8)? A: 38”. The three 1 digit numbers
are selected uniformly on [0;10)and the operations are selected uniformly from f+,-,*g.
In all 10 tasks the model must generate the correct answer exactly. For each task we generate a dataset of 2,000 random
instances of the task and evaluate all models on those instances.
First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10. On addition and subtraction,


key: chroma:document
string_value:
First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10. On addition and subtraction,
GPT-3 displays strong proﬁciency when the number of digits is small, achieving 100% accuracy on 2 digit addition,
98.9% at 2 digit subtraction, 80.2% at 3 digit addition, and 94.2% at 3-digit subtraction. Performance decreases as the
number of digits increases, but GPT-3 still achieves 25-26% accuracy on four digit operations and 9-10% accuracy on
ﬁve digit operations, suggesting at least some capacity to generalize to larger numbers of digits. GPT-3 also achieves
29.2% accuracy at 2 digit multiplication, an especially computationally intensive operation. Finally, GPT-3 achieves
21.3% accuracy at single digit combined operations (for example, 9*(7+5)), suggesting that it has some robustness
beyond just single operations.
As Figure 3.10 makes clear, small models do poorly on all of these tasks – even the 13 billion parameter model (the
second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all
other operations less than 10% of the time.


key: chroma:document
string_value:
second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all
other operations less than 10% of the time.
One-shot and zero-shot performance are somewhat degraded relative to few-shot performance, suggesting that adaptation
to the task (or at the very least recognition of the task) is important to performing these computations correctly.
Nevertheless, one-shot performance is still quite strong, and even zero-shot performance of the full GPT-3 signiﬁcantly
22
Setting 2D+ 2D- 3D+ 3D- 4D+ 4D- 5D+ 5D- 2Dx 1DC
GPT-3 Zero-shot 76.9 58.0 34.2 48.3 4.0 7.5 0.7 0.8 19.8 9.8
GPT-3 One-shot 99.6 86.4 65.5 78.7 14.0 14.0 3.5 3.8 27.4 14.3
GPT-3 Few-shot 100.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3


key: chroma:document
string_value:
GPT-3 Few-shot 100.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3
Table 3.9: Results on basic arithmetic tasks for GPT-3 175B. f2,3,4,5gDf+,-gis 2, 3, 4, and 5 digit addition or
subtraction, 2Dx is 2 digit multiplication. 1DC is 1 digit composite operations. Results become progressively stronger
moving from the zero-shot to one-shot to few-shot setting, but even the zero-shot shows signiﬁcant arithmetic abilities.
Setting CL A1 A2 RI RW
GPT-3 Zero-shot 3.66 2.28 8.91 8.26 0.09
GPT-3 One-shot 21.7 8.62 25.9 45.4 0.48
GPT-3 Few-shot 37.9 15.1 39.7 67.2 0.44
Table 3.10: GPT-3 175B performance on various word unscrambling and word manipulation tasks, in zero-, one-, and
few-shot settings. CL is “cycle letters in word”, A1 is anagrams of but the ﬁrst and last letters, A2 is anagrams of all but


key: chroma:document
string_value:
the ﬁrst and last two letters, RI is “Random insertion in word”, RW is “reversed words”.
outperforms few-shot learning for all smaller models. All three settings for the full GPT-3 are shown in Table 3.9, and
model capacity scaling for all three settings is shown in Appendix H.
To spot-check whether the model is simply memorizing speciﬁc arithmetic problems, we took the 3-digit arithmetic
problems in our test set and searched for them in our training data in both the forms "<NUM1> + <NUM2> =" and
"<NUM1> plus <NUM2>" . Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000
subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers
could have been memorized. In addition, inspection of incorrect answers reveals that the model often makes mistakes
such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than
memorizing a table.
Overall, GPT-3 displays reasonable proﬁciency at moderately complex arithmetic in few-shot, one-shot, and even
zero-shot settings.


key: chroma:document
string_value:
Overall, GPT-3 displays reasonable proﬁciency at moderately complex arithmetic in few-shot, one-shot, and even
zero-shot settings.
3.9.2 Word Scrambling and Manipulation Tasks
To test GPT-3’s ability to learn novel symbolic manipulations from a few examples, we designed a small battery of
5 “character manipulation” tasks. Each task involves giving the model a word distorted by some combination of
scrambling, addition, or deletion of characters, and asking it to recover the original word. The 5 tasks are:
•Cycle letters in word (CL) – The model is given a word with its letters cycled, then the “=” symbol, and
is expected to generate the original word. For example, it might be given “lyinevitab” and should output
“inevitably”.
•Anagrams of all but ﬁrst and last characters (A1) – The model is given a word where every letter except
the ﬁrst and last have been scrambled randomly, and must output the original word. Example: criroptuon =
corruption.
•Anagrams of all but ﬁrst and last 2 characters (A2) – The model is given a word where every letter except


key: chroma:document
string_value:
corruption.
•Anagrams of all but ﬁrst and last 2 characters (A2) – The model is given a word where every letter except
the ﬁrst 2 and last 2 have been scrambled randomly, and must recover the original word. Example: opoepnnt
!opponent.
•Random insertion in word (RI) – A random punctuation or space character is inserted between each letter
of a word, and the model must output the original word. Example: s.u!c/c!e.s s i/o/n = succession.
•Reversed words (RW) – The model is given a word spelled backwards, and must output the original word.
Example: stcejbo!objects.
For each task we generate 10,000 examples, which we chose to be the top 10,000 most frequent words as measured by
[Nor09 ] of length more than 4 characters and less than 15 characters. The few-shot results are shown in Figure 3.11.
Task performance tends to grow smoothly with model size, with the full GPT-3 model achieving 66.9% on removing
23
Figure 3.11: Few-shot performance on the ﬁve word scrambling tasks for different sizes of model. There is generally
smooth improvement with model size although the random insertion task shows an upward slope of improvement with


key: chroma:document
string_value:
smooth improvement with model size although the random insertion task shows an upward slope of improvement with
the 175B model solving the task the majority of the time. Scaling of one-shot and zero-shot performance is shown in
the appendix. All tasks are done with K= 100 .
random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task, and 15.1% on the more difﬁcult anagram
task (where only the ﬁrst and last letters are held ﬁxed). None of the models can reverse the letters in a word.
In the one-shot setting, performance is signiﬁcantly weaker (dropping by half or more), and in the zero-shot setting the
model can rarely perform any of the tasks (Table 3.10). This suggests that the model really does appear to learn these
tasks at test time, as the model cannot perform them zero-shot and their artiﬁcial nature makes them unlikely to appear
in the pre-training data (although we cannot conﬁrm this with certainty).
We can further quantify performance by plotting “in-context learning curves”, which show task performance as a
function of the number of in-context examples. We show in-context learning curves for the Symbol Insertion task


key: chroma:document
string_value:
function of the number of in-context examples. We show in-context learning curves for the Symbol Insertion task
in Figure 1.2. We can see that larger models are able to make increasingly effective use of in-context information,
including both task examples and natural language task descriptions.
Finally, it is worth adding that solving these tasks requires character-level manipulations, whereas our BPE encoding
operates on signiﬁcant fractions of a word (on average 0:7words per token), so from the LM’s perspective succeeding
at these tasks involves not just manipulating BPE tokens but understanding and pulling apart their substructure. Also,
CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word),
requiring the model to perform some search to ﬁnd the correct unscrambling. Thus, the skills involved appear to require
non-trivial pattern-matching and computation.
3.9.3 SAT Analogies
To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of
374 “SAT analogy” problems [ TLBS03 ]. Analogies are a style of multiple choice question that constituted a section of


key: chroma:document
string_value:
374 “SAT analogy” problems [ TLBS03 ]. Analogies are a style of multiple choice question that constituted a section of
the SAT college entrance exam before 2005. A typical example is “audacious is to boldness as (a) sanctimonious is to
hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to
temptation”. The student is expected to choose which of the ﬁve word pairs has the same relationship as the original
word pair; in this example the answer is “sanctimonious is to hypocrisy”. On this task GPT-3 achieves 65.2% in the
few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among
college applicants was 57% [ TL05 ] (random guessing yields 20%). As shown in Figure 3.12, the results improve with
scale, with the the full 175 billion model improving by over 10% compared to the 13 billion parameter model.
24
Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest


key: chroma:document
string_value:
24
Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model. The largest
model achieves 65% accuracy in the few-shot setting, and also demonstrates signiﬁcant gains to in-context learning
which are not present in smaller models.
3.9.4 News Article Generation
Previous work on generative language models qualitatively tested their ability to generate synthetic “news articles” by
conditional sampling from the model given a human-written prompt consisting of a plausible ﬁrst sentence for a news
story [ RWC+19]. Relative to [ RWC+19], the dataset used to train GPT-3 is much less weighted towards news articles,
so trying to generate news articles via raw unconditional samples is less effective – for example GPT-3 often interprets
the proposed ﬁrst sentence of a “news article” as a tweet and then posts synthetic responses or follow-up tweets. To
solve this problem we employed GPT-3’s few-shot learning abilities by providing three previous news articles in the
model’s context to condition it. With the title and subtitle of a proposed next article, the model is able to reliably
generate short articles in the “news” genre.


key: chroma:document
string_value:
model’s context to condition it. With the title and subtitle of a proposed next article, the model is able to reliably
generate short articles in the “news” genre.
To gauge the quality of news article generation from GPT-3 (which we believe is likely to be correlated with conditional
sample generation quality in general), we decided to measure human ability to distinguish GPT-3-generated articles
from real ones. Similar work has been carried out by Kreps et al. [ KMB20 ] and Zellers et al. [ ZHR+19]. Generative
language models are trained to match the distribution of content generated by humans, so the (in)ability of humans to
distinguish the two is a potentially important measure of quality.3
In order to see how well humans can detect model generated text, we arbitrarily selected 25 article titles and subtitles
from the website newser.com (mean length: 215 words). We then generated completions of these titles and subtitles
from four language models ranging in size from 125M to 175B (GPT-3) parameters (mean length: 200 words). For each
model, we presented around 80 US-based participants with a quiz consisting of these real titles and subtitles followed
by either the human written article or the article generated by the model4. Participants were asked to select whether the


key: chroma:document
string_value:
by either the human written article or the article generated by the model4. Participants were asked to select whether the
article was “very likely written by a human”, “more likely written by a human”, “I don’t know”, “more likely written by
a machine”, or “very likely written by a machine”.
The articles we selected were not in the models’ training data and the model outputs were formatted and selected
programmatically to prevent human cherry-picking. All models used the same context to condition outputs on and were
pre-trained with the same context size and the same article titles and subtitles were used as prompts for each model.
However, we also ran an experiment to control for participant effort and attention that followed the same format but
involved intentionally bad model generated articles. This was done by generating articles from a “control model”: a
160M parameter model with no context and increased output randomness.
3This task is also relevant to the potential misuse of language models discussed in Section 6.1.
4We wanted to identify how good an average person on the internet is at detecting language model outputs, so we focused on
participants drawn from the general US population. See Appendix E for details.
25
Mean accuracy95% Conﬁdence
Interval (low, hi)tcompared to


key: chroma:document
string_value:
25
Mean accuracy95% Conﬁdence
Interval (low, hi)tcompared to
control (p-value)“I don’t know”
assignments
Control (deliberately bad model) 86% 83%–90% - 3.6 %
GPT-3 Small 76% 72%–80% 3.9 (2 e-4) 4.9%
GPT-3 Medium 61% 58%–65% 10.3 (7 e-21) 6.0%
GPT-3 Large 68% 64%–72% 7.3 (3 e-11) 8.7%
GPT-3 XL 62% 59%–65% 10.7 (1 e-19) 7.5%
GPT-3 2.7B 62% 58%–65% 10.4 (5 e-19) 7.1%
GPT-3 6.7B 60% 56%–63% 11.2 (3 e-21) 6.2%
GPT-3 13B 55% 52%–58% 15.3 (1 e-32) 7.1%
GPT-3 175B 52% 49%–54% 16.9 (1 e-34) 7.8%


key: chroma:document
string_value:
GPT-3 175B 52% 49%–54% 16.9 (1 e-34) 7.8%
Table 3.11: Human accuracy in identifying whether short ( 200 word) news articles are model generated . We
ﬁnd that human accuracy (measured by the ratio of correct assignments to non-neutral assignments) ranges from 86%
on the control model to 52% on GPT-3 175B. This table compares mean accuracy between ﬁve different models, and
shows the results of a two-sample T-Test for the difference in mean accuracy between each model and the control model
(an unconditional GPT-3 Small model with increased output randomness).
Mean human accuracy (the ratio of correct assignments to non-neutral assignments per participant) at detecting that
the intentionally bad articles were model generated was 86% where 50% is chance level performance. By contrast,
mean human accuracy at detecting articles that were produced by the 175B parameter model was barely above chance
at52% (see Table 3.11).5Human abilities to detect model generated text appear to decrease as model size increases:
there appears to be a trend towards chance accuracy with model size, and human detection of GPT-3 is close to chance.6
This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).


key: chroma:document
string_value:
This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).
Examples of synthetic articles from GPT-3 are given in Figures 3.14 and 3.15.7Much of the text is—as indicated by the
evaluations—difﬁcult for humans to distinguish from authentic human content. Factual inaccuracies can be an indicator
that an article is model generated since, unlike human authors, the models have no access to the speciﬁc facts that the
article titles refer to or when the article was written. Other indicators include repetition, non sequiturs, and unusual
phrasings, though these are often subtle enough that they are not noticed.
Related work on language model detection by Ippolito et al. [ IDCBE19 ] indicates that automatic discriminators like
GR O V E R [ZHR+19] and GLTR [ GSR19 ] may have greater success at detecting model generated text than human
evaluators. Automatic detection of these models may be a promising area of future research.
Ippolito et al. [ IDCBE19 ] also note that human accuracy at detecting model generated text increases as humans observe
more tokens. To do a preliminary investigation of how good humans are at detecting longer news articles generated


key: chroma:document
string_value:
more tokens. To do a preliminary investigation of how good humans are at detecting longer news articles generated
by GPT-3 175B, we selected 12 world news articles from Reuters with an average length of 569 words and generated
completions of these articles from GPT-3 with an average length of 498 words (298 words longer than our initial
experiments). Following the methodology above, we ran two experiments, each on around 80 US-based participants, to
compare human abilities to detect the articles generated by GPT-3 and a control model.
We found that mean human accuracy at detecting the intentionally bad longer articles from the control model was
88%, while mean human accuracy at detecting the longer articles that were produced by GPT-3 175B was still barely
above chance at52% (see Table 3.12). This indicates that, for news articles that are around 500 words long, GPT-3
continues to produce articles that humans ﬁnd difﬁcult to distinguish from human written news articles.
3.9.5 Learning and Using Novel Words
A task studied in developmental linguistics [ CB78 ] is the ability to learn and utilize new words, for example using a
word in a sentence after seeing it deﬁned only once, or conversely inferring a word’s meaning from only one usage. Here


key: chroma:document
string_value:
word in a sentence after seeing it deﬁned only once, or conversely inferring a word’s meaning from only one usage. Here
we qualitatively test GPT-3’s ability to do the former. Speciﬁcally, we give GPT-3 the deﬁnition of a nonexistent word,
such as “Gigamuru”, and then ask it to use it in a sentence. We provide one to ﬁve previous examples of a (separate)
5We use a two-sample Student’s T-Test to test for signiﬁcant difference between the means of the participant accuracies of each
model and the control model and report the normalized difference in the means (as the t-statistic) and the p-value.
6If a model consistently produces texts that are more impressive than human articles, it is possible that human performance on
this task would drop below 50%. Indeed, many individual participants scored below 50% on this task.
7Additional non-news samples can be found in Appendix F.
26
Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct


key: chroma:document
string_value:
26
Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct
assignments to non-neutral assignments) decreases as model size increases. Accuracy on the outputs on the deliberatelybad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed
line at the top, and the random chance (50%) is indicated with the dashed line at the bottom. Line of best ﬁt is a power
law with 95% conﬁdence intervals.
Mean accuracy95% Conﬁdence
Interval (low, hi)tcompared to
control (p-value)“I don’t know”
assignments
Control 88% 84%–91% - 2.7%
GPT-3 175B 52% 48%–57% 12.7 (3.2 e-23) 10.6%
Table 3.12: People’s ability to identify whether 500word articles are model generated (as measured by the ratio of
correct assignments to non-neutral assignments) was 88% on the control model and 52% on GPT-3 175B. This table
shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control


key: chroma:document
string_value:
shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control
model (an unconditional GPT-3 Small model with increased output randomness).
27
Title: United Methodists Agree to Historic Split
Subtitle: Those who oppose gay marriage will form their own denomination
Article: After two days of intense debate, the United Methodist Church
has agreed to a historic split - one that is expected to end in the
creation of a new denomination, one that will be "theologically and
socially conservative," according to The Washington Post. The majority of
delegates attending the church's annual General Conference in May voted to
strengthen a ban on the ordination of LGBTQ clergy and to write new rules
that will "discipline" clergy who officiate at same-sex weddings. But
those who opposed these measures have a new plan: They say they will form a
separate denomination by 2020, calling their church the Christian Methodist
denomination.
The Post notes that the denomination, which claims 12.5 million members, was
in the early 20th century the "largest Protestant denomination in the U.S.,"
but that it has been shrinking in recent decades. The new split will be the
second in the church's history. The first occurred in 1968, when roughly
10 percent of the denomination left to form the Evangelical United Brethren


key: chroma:document
string_value:
second in the church's history. The first occurred in 1968, when roughly
10 percent of the denomination left to form the Evangelical United Brethren
Church. The Post notes that the proposed split "comes at a critical time
for the church, which has been losing members for years," which has been
"pushed toward the brink of a schism over the role of LGBTQ people in the
church." Gay marriage is not the only issue that has divided the church. In
2016, the denomination was split over ordination of transgender clergy, with
the North Pacific regional conference voting to ban them from serving as
clergy, and the South Pacific regional conference voting to allow them.
Figure 3.14: The GPT-3 generated news article that humans had the greatest difﬁculty distinguishing from a human
written article (accuracy: 12%).
Title: Star's Tux Promise Draws Megyn Kelly's Sarcasm
Subtitle: Joaquin Phoenix pledged to not change for each awards event
Article: A year ago, Joaquin Phoenix made headlines when he appeared on
the red carpet at the Golden Globes wearing a tuxedo with a paper bag over
his head that read, "I am a shape-shifter. I can't change the world. I
can only change myself." It was a promise to not change to fit into the


key: chroma:document
string_value:
his head that read, "I am a shape-shifter. I can't change the world. I
can only change myself." It was a promise to not change to fit into the
Hollywood mold: "I think that's a really special thing, to not change
yourself. I think it's a really special thing to say, `This is what's
inside of me, I'm proud of it, and I'm not going to be ashamed because of
the way that someone else thinks I should be.'" Now, it's the Oscars, and
Phoenix is at it again. But this time, his publicist is saying he'll be
wearing a tux no matter what.
Megyn Kelly was not impressed, and she let him have it on The Tonight Show.
"You know, I feel like, I feel like you could have worn the tux," she says.
"But you're saying you're a shape-shifter. I don't know if you can change
your tux, but you can change your mind. You can change your mind. You can
change your mind." Phoenix says he did, but it didn't stick. "I was like,
`Okay, I'm going to wear a tuxedo to this thing.' And then I thought, `I
don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him


key: chroma:document
string_value:
don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him
to change his mind again, but Phoenix says it's too late: "I'm committed to
wearing this."
Figure 3.15: The GPT-3 generated news article that humans found the easiest to distinguish from a human written
article (accuracy: 61%).
28
A "whatpu" is a small, furry animal native to Tanzania. An example of a sentence that uses
the word whatpu is:
We were traveling in Africa and we saw these very cute whatpus.
To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses
the word farduddle is:
One day when I was playing tag with my little sister, she got really excited and she
started doing these crazy farduddles.
A "yalubalu" is a type of vegetable that looks like a big pumpkin. An example of a sentence
that uses the word yalubalu is:
I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden
there. It was delicious.
A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the
word Burringo is:


key: chroma:document
string_value:
there. It was delicious.
A "Burringo" is a car with very fast acceleration. An example of a sentence that uses the
word Burringo is:
In our garage we have a Burringo that my father drives to work every day.
A "Gigamuru" is a type of Japanese musical instrument. An example of a sentence that uses the
word Gigamuru is:
I have a Gigamuru that my uncle gave me as a gift. I love to play it at home.
To "screeg" something is to swing a sword at it. An example of a sentence that uses the word
screeg is:
We screeghed at each other for several minutes and then we went outside and ate ice cream.
Figure 3.16: Representative GPT-3 completions for the few-shot task of using a new word in a sentence. Boldface is
GPT-3’s completions, plain text is human prompts. In the ﬁrst example both the prompt and the completion are provided
by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional
prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 other than the conditioning shown
here.


key: chroma:document
string_value:
prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 other than the conditioning shown
here.
nonexistent word being deﬁned and used in a sentence, so the task is few-shot in terms of previous examples of the
broad task and one-shot in terms of the speciﬁc word. Table 3.16 shows the 6 examples we generated; all deﬁnitions
were human-generated, and the ﬁrst answer was human-generated as conditioning while the subsequent answers were
generated by GPT-3. These examples were generated continuously in one sitting and we did not omit or repeatedly try
any prompts. In all cases the generated sentence appears to be a correct or at least plausible use of the word. In the ﬁnal
sentence the model generates a plausible conjugation for the word “screeg” (namely “screeghed”), although the use of
the word is slightly awkward (“screeghed at each other”) despite being plausible in the sense that it could describe a toy
sword ﬁght. Overall, GPT-3 appears to be at least proﬁcient at the task of using novel words in a sentence.
3.9.6 Correcting English Grammar


key: chroma:document
string_value:
3.9.6 Correcting English Grammar
Another task well suited for few-shot learning is correcting English grammar. We test this with GPT-3 in the fewshot setting by giving prompts of the form "Poor English Input: <sentence> nn Good English Output:
<sentence>" . We give GPT-3 one human-generated correction and then ask it to correct 5 more (again without any
omissions or repeats). Results are shown in Figure 3.17.
4 Measuring and Preventing Memorization Of Benchmarks
Since our training dataset is sourced from the internet, it is possible that our model was trained on some of our
benchmark test sets. Accurately detecting test contamination from internet-scale datasets is a new area of research
without established best practices. While it is common practice to train large models without investigating contamination,
given the increasing scale of pretraining datasets, we believe this issue is becoming increasingly important to attend to.
This concern is not just hypothetical. One of the ﬁrst papers to train a language model on Common Crawl data [ TL18 ]
detected and removed a training document which overlapped with one of their evaluation datasets. Other work such
as GPT-2 [ RWC+19] also conducted post-hoc overlap analysis. Their study was relatively encouraging, ﬁnding that
29


key: chroma:document
string_value:
as GPT-2 [ RWC+19] also conducted post-hoc overlap analysis. Their study was relatively encouraging, ﬁnding that
29
Poor English input: I eated the purple berries.
Good English output: I ate the purple berries.
Poor English input: Thank you for picking me as your designer. I'd appreciate it.
Good English output: Thank you for choosing me as your designer. I appreciate it.
Poor English input: The mentioned changes have done. or I did the alteration that you
requested. or I changed things you wanted and did the modifications.
Good English output: The requested changes have been made. or I made the alteration that you
requested. or I changed things you wanted and made the modifications.
Poor English input: I'd be more than happy to work with you in another project.
Good English output: I'd be more than happy to work with you on another project.
Poor English input: Please provide me with a short brief of the design you're looking for and
that'd be nice if you could share some examples or project you did before.
Good English output: Please provide me with a brief description of the design you're
looking for and that would be nice if you could share some examples or projects you have
done before.
Poor English input: The patient was died.
Good English output: The patient died.


key: chroma:document
string_value:
looking for and that would be nice if you could share some examples or projects you have
done before.
Poor English input: The patient was died.
Good English output: The patient died.
Poor English input: We think that Leslie likes ourselves.
Good English output: We think that Leslie likes us.
Poor English input: Janet broke Bill on the finger.
Good English output: Janet broke Bill's finger.
Poor English input: Mary arranged for, in St. Louis, John to rent a house cheap.
Good English output: Mary arranged for John to rent a house in St. Louis.
Poor English input: Today I have went to the store to to buys some many bottle of water.
Good English output: Today I went to the store to buy some bottles of water.
Poor English input: I have tried to hit ball with bat, but my swing is has miss.
Good English output: I tried to hit the ball with the bat, but my swing missed.
Figure 3.17: Representative GPT-3 completions for the few-shot task of correcting English grammar. Boldface
is GPT-3’s completions, plain text is human prompts. In the ﬁrst few examples example both the prompt and the
completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives


key: chroma:document
string_value:
completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives
successive additional prompts and provides the completions. Nothing task-speciﬁc is provided to GPT-3 aside from
the ﬁrst few examples as conditioning and the “Poor English input/Good English output” framing. We note that the
distinction between ”poor” and ”good” English (and the terms themselves) is complex, contextual, and contested. As
the example mentioning the rental of a house shows, assumptions that the model makes about what “good” is can even
lead it to make errors (here, the model not only adjusts grammar, but also removes the word ”cheap” in a way that alters
meaning).
30
Figure 4.1: GPT-3 Training Curves We measure model performance during training on a deduplicated validation
split of our training distribution. Though there is some gap between training and validation performance, the gap grows
only minimally with model size and training time, suggesting that most of the gap comes from a difference in difﬁculty
rather than overﬁtting.
although models did perform moderately better on data that overlapped between training and testing, this did not


key: chroma:document
string_value:
rather than overﬁtting.
although models did perform moderately better on data that overlapped between training and testing, this did not
signiﬁcantly impact reported results due to the small fraction of data which was contaminated (often only a few percent).
GPT-3 operates in a somewhat different regime. On the one hand, the dataset and model size are about two orders of
magnitude larger than those used for GPT-2, and include a large amount of Common Crawl, creating increased potential
for contamination and memorization. On the other hand, precisely due to the large amount of data, even GPT-3 175B
does not overﬁt its training set by a signiﬁcant amount, measured relative to a held-out validation set with which it was
deduplicated (Figure 4.1). Thus, we expect that contamination is likely to be frequent, but that its effects may not be as
large as feared.
We initially tried to address the issue of contamination by proactively searching for and attempting to remove any overlap
between our training data and the development and test sets of all benchmarks studied in this paper. Unfortunately, a
bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t


key: chroma:document
string_value:
bug resulted in only partial removal of all detected overlaps from the training data. Due to the cost of training, it wasn’t
feasible to retrain the model. To address this, we investigate in detail how the remaining detected overlap impacts
results.
For each benchmark, we produce a ‘clean’ version which removes all potentially leaked examples, deﬁned roughly as
examples that have a 13-gram overlap with anything in the pretraining set (or that overlap with the whole example when
it is shorter than 13-grams). The goal is to very conservatively ﬂag anything that could potentially be contamination,
so as to produce a clean subset that is free of contamination with high conﬁdence. The exact procedure is detailed in
Appendix C.
We then evaluate GPT-3 on these clean benchmarks, and compare to the original score. If the score on the clean
subset is similar to the score on the entire dataset, this suggests that contamination, even if present, does not have a
signiﬁcant effect on reported results. If the score on the clean subset is lower, this suggests contamination may be
inﬂating the results. The results are summarized in Figure 4.2. Although potential contamination is often high (with a
quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence


key: chroma:document
string_value:
quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence
that contamination level and performance difference are correlated. We conclude that either our conservative method
substantially overestimated contamination or that contamination has little effect on performance.
Below, we review in more detail the few speciﬁc cases where either (1) the model performs signiﬁcantly worse on
the cleaned version, or (2) potential contamination is very high, which makes measuring the performance difference
difﬁcult.
Our analysis ﬂagged six groups of benchmarks for further investigation: Word Scrambling, Reading Comprehension
(QuAC, SQuAD2, DROP), PIQA, Winograd, language modeling tasks (Wikitext tasks, 1BW), and German to English
31
Figure 4.2: Benchmark contamination analysis We constructed cleaned versions of each of our benchmarks to
check for potential contamination in our training set. The x-axis is a conservative lower bound for how much of the
dataset is known with high conﬁdence to be clean, and the y-axis shows the difference in performance when evaluating
only on the veriﬁed clean subset. Performance on most benchmarks changed negligibly, but some were ﬂagged for


key: chroma:document
string_value:
only on the veriﬁed clean subset. Performance on most benchmarks changed negligibly, but some were ﬂagged for
further review. On inspection we ﬁnd some evidence for contamination of the PIQA and Winograd results, and we mark
the corresponding results in Section 3 with an asterisk. We ﬁnd no evidence that other benchmarks are affected.
translation. Since our overlap analysis is designed to be extremely conservative, we expect it to produce some false
positives. We summarize the results for each group of tasks below:
•Reading Comprehension: Our initial analysis ﬂagged >90% of task examples from QuAC, SQuAD2, and
DROP as potentially contaminated, so large that even measuring the differential on a clean subset was difﬁcult.
Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source
text was present in our training data but the question/answer pairs were not, meaning the model gains only
background information and cannot memorize the answer to a speciﬁc question.
•German translation: We found 25% of the examples in the WMT16 German-English test set were marked
as potentially contaminated, with an associated total effect size of 1-2 BLEU. Upon inspection, none of the


key: chroma:document
string_value:
as potentially contaminated, with an associated total effect size of 1-2 BLEU. Upon inspection, none of the
ﬂagged examples contain paired sentences resembling NMT training data and collisions were monolingual
matches mostly of snippets of events discussed in the news.
•Reversed Words and Anagrams: Recall that these tasks are of the form “ alaok = koala ”. Due to the
short length of these tasks, we used 2-grams for ﬁltering (ignoring punctuation). After inspecting the ﬂagged
overlaps, we found that they were not typically instances of real reversals or unscramblings in the training set,
but rather palindromes or trivial unscramblings, e.g “ kayak = kayak ”. The amount of overlap was small,
but removing the trivial tasks lead to an increase in difﬁculty and thus a spurious signal. Related to this, the
symbol insertion task shows high overlap but no effect on performance – this is because that task involves
removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to
many spurious matches.
•PIQA: The overlap analysis ﬂagged 29% of examples as contaminated, and observed a 3 percentage point


key: chroma:document
string_value:
many spurious matches.
•PIQA: The overlap analysis ﬂagged 29% of examples as contaminated, and observed a 3 percentage point
absolute decrease (4% relative decrease) in performance on the clean subset. Though the test dataset was
released after our training set was created and its labels are hidden, some of the web pages used by the
crowdsourced dataset creators are contained in our training set. We found a similar decrease in a 25x smaller
model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias
rather than memorization; examples which workers copied may simply be easier. Unfortunately, we cannot
rigorously prove this hypothesis. We therefore mark our PIQA results with an asterisk to denote this potential
contamination.
•Winograd: The overlap analysis ﬂagged 45% of examples, and found a 2.6% decrease in performance on the
clean subset. Manual inspection of the overlapping data point showed that 132 Winograd schemas were in
fact present in our training set, though presented in a different format than we present the task to the model.
Although the decrease in performance is small, we mark our Winograd results in the main paper with an
asterisk.
32
•Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the


key: chroma:document
string_value:
asterisk.
32
•Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the
Children’s Book Test dataset, to be almost entirely contained in our training data. Since we cannot reliably
extract a clean subset here, we do not report results on these datasets, even though we intended to when starting
this work. We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language
modeling benchmark.
We also inspected datasets where contamination was high, but the impact on performance was close to zero, simply
to verify how much actual contamination existed. These appeared to often contain false positives. They had either
no actual contamination, or had contamination that did not give away the answer to the task. One notable exception
was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very
small, with the clean subset scoring within 0.5% of the full dataset. Also, strictly speaking, our ﬁll-in-the-blank format
precludes the simplest form of memorization. Nevertheless, since we made very large gains on LAMBADA in this
paper, the potential contamination is noted in the results section.
An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the


key: chroma:document
string_value:
paper, the potential contamination is noted in the results section.
An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the
same distribution as the original dataset. It remains possible that memorization inﬂates results but at the same time
is precisely counteracted by some statistical bias causing the clean subset to be easier. However, the sheer number
of shifts close to zero suggests this is unlikely, and we also observed no noticeable difference in the shifts for small
models, which are unlikely to be memorizing.
Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright
remove problematic results, depending on the severity. Much work remains to be done to address this important and
subtle issue for the ﬁeld in general, both when designing benchmarks and when training models. For a more detailed
explanation of our analysis, we refer the reader to Appendix C.
5 Limitations
GPT-3 and our analysis of it have a number of limitations. Below we describe some of these and suggest directions for
future work.
First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct
predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks. On text synthesis, although


key: chroma:document
string_value:
predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks. On text synthesis, although
the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to
lose coherence over sufﬁciently long passages, contradict themselves, and occasionally contain non-sequitur sentences
or paragraphs. We will release a collection of 500 uncurated unconditional samples to help provide a better sense of
GPT-3’s limitations and strengths at text synthesis. Within the domain of discrete language tasks, we have noticed
informally that GPT-3 seems to have special difﬁculty with “common sense physics”, despite doing well on some
datasets (such as PIQA [ BZB+19]) that test this domain. Speciﬁcally GPT-3 has difﬁculty with questions of the type
“If I put cheese into the fridge, will it melt?”. Quantitatively, GPT-3’s in-context learning performance has some notable
gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when


key: chroma:document
string_value:
gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when
evaluated one-shot or even few-shot on some “comparison” tasks, such as determining if two words are used the same
way in a sentence, or if one sentence implies another (WIC and ANLI respectively), as well as on a subset of reading
comprehension tasks. This is especially striking given GPT-3’s strong few-shot performance on many other tasks.
GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above. We focused
on exploring in-context learning behavior in autoregressive language models because it is straightforward to both
sample and compute likelihoods with this model class. As a result our experiments do not include any bidirectional
architectures or other training objectives such as denoising. This is a noticeable difference from much of the recent
literature, which has documented improved ﬁne-tuning performance when using these approaches over standard
language models [ RSR+19]. Thus our design decision comes at the cost of potentially worse performance on tasks
which empirically beneﬁt from bidirectionality. This may include ﬁll-in-the-blank tasks, tasks that involve looking back


key: chroma:document
string_value:
which empirically beneﬁt from bidirectionality. This may include ﬁll-in-the-blank tasks, tasks that involve looking back
and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then
generating a very short answer. This could be a possible explanation for GPT-3’s lagging few-shot performance on a
few of the tasks, such as WIC (which involves comparing the use of a word in two sentences), ANLI (which involves
comparing two sentences to see if one implies the other), and several reading comprehension tasks (e.g. QuAC and
RACE). We also conjecture, based on past literature, that a large bidirectional model would be stronger at ﬁne-tuning
than GPT-3. Making a bidirectional model at the scale of GPT-3, and/or trying to make bidirectional models work with
few- or zero-shot learning, is a promising direction for future research, and could help achieve the “best of both worlds”.
A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
33


key: chroma:document
string_value:
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
33
pretraining objective. Our current objective weights every token equally and lacks a notion of what is most important to
predict and what is less important. [ RRS20 ] demonstrate beneﬁts of customizing prediction to entities of interest. Also,
with self-supervised objectives, task speciﬁcation relies on forcing the desired task into a prediction problem, whereas
ultimately, useful language systems (for example virtual assistants) might be better thought of as taking goal-directed
actions rather than just making predictions. Finally, large pretrained language models are not grounded in other domains
of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world
[BHT+20]. For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a
different approach is likely to be necessary. Promising future directions in this vein might include learning the objective
function from humans [ ZSW+19a], ﬁne-tuning with reinforcement learning, or adding additional modalities such as
images to provide grounding and a better model of the world [CLY+19].


key: chroma:document
string_value:
images to provide grounding and a better model of the world [CLY+19].
Another limitation broadly shared by language models is poor sample efﬁciency during pre-training. While GPT-3
takes a step towards test-time sample efﬁciency closer to that of humans (one-shot or zero-shot), it still sees much more
text during pre-training than a human sees in the their lifetime [ Lin20 ]. Improving pre-training sample efﬁciency is
an important direction for future work, and might come from grounding in the physical world to provide additional
information, or from algorithmic improvements.
A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot
learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identiﬁes tasks that it
has learned during training. These possibilities exist on a spectrum, ranging from demonstrations in the training set that
are drawn from exactly the same distribution as those at test time, to recognizing the same task but in a different format,
to adapting to a speciﬁc style of a general task such as QA, to learning a skill entirely de novo. Where GPT-3 is on


key: chroma:document
string_value:
to adapting to a speciﬁc style of a general task such as QA, to learning a skill entirely de novo. Where GPT-3 is on
this spectrum may also vary from task to task. Synthetic tasks such as wordscrambling or deﬁning nonsense words
seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although
possibly from data that is very different in organization and style than the test data. Ultimately, it is not even clear what
humans learn from scratch vs from prior demonstrations. Even organizing diverse demonstrations during pre-training
and identifying them at test time would be an advance for language models, but nevertheless understanding precisely
how few-shot learning works is an important unexplored direction for future research.
A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are
both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of
models of this scale in their current form. One possible future direction to address this is distillation [ HVD15 ] of large
models down to a manageable size for speciﬁc tasks. Large models such as GPT-3 contain a very wide range of skills,
most of which are not needed for a speciﬁc task, suggesting that in principle aggressive distillation may be possible.


key: chroma:document
string_value:
most of which are not needed for a speciﬁc task, suggesting that in principle aggressive distillation may be possible.
Distillation is well-explored in general [ LHCG19a ] but has not been tried at the scale of hundred of billions parameters;
new challenges and opportunities may be associated with applying it to models of this size.
Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable,
it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in
performance than humans on standard benchmarks, and it retains the biases of the data it has been trained on. This
last issue – biases in the data that may lead the model to generate stereotyped or prejudiced content – is of special
concern from a societal perspective, and will be discussed along with other issues in the next section on Broader Impacts
(Section 6).
6 Broader Impacts
Language models have a wide range of beneﬁcial applications for society, including code and writing auto-completion,
grammar assistance, game narrative generation, improving search engine responses, and answering questions. But
they also have potentially harmful applications. GPT-3 improves the quality of text generation and adaptability over


key: chroma:document
string_value:
grammar assistance, game narrative generation, improving search engine responses, and answering questions. But
they also have potentially harmful applications. GPT-3 improves the quality of text generation and adaptability over
smaller models and increases the difﬁculty of distinguishing synthetic text from human-written text. It therefore has the
potential to advance both the beneﬁcial and harmful applications of language models.
Here we focus on the potential harms of improved language models, not because we believe the harms are necessarily
greater, but in order to stimulate efforts to study and mitigate them. The broader impacts of language models like this
are numerous. We focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 in
Section 6.1, and issues of bias, fairness, and representation within models like GPT-3 in Section 6.2. We also brieﬂy
discuss issues of energy efﬁciency (Section 6.3).
34
6.1 Misuse of Language Models
Malicious uses of language models can be somewhat difﬁcult to anticipate because they often involve repurposing
language models in a very different environment or for a different purpose than researchers intended. To help with this,
we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying


key: chroma:document
string_value:
we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying
threats and potential impacts, assessing likelihood, and determining risk as a combination of likelihood and impact
[Ros12]. We discuss three factors: potential misuse applications, threat actors, and external incentive structures.
6.1.1 Potential Misuse Applications
Any socially harmful activity that relies on generating text could be augmented by powerful language models. Examples
include misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing
and social engineering pretexting. Many of these applications bottleneck on human beings to write sufﬁciently high
quality text. Language models that produce high quality text generation could lower existing barriers to carrying out
these activities and increase their efﬁcacy.
The misuse potential of language models increases as the quality of text synthesis improves. The ability of GPT-3 to
generate several paragraphs of synthetic content that people ﬁnd difﬁcult to distinguish from human-written text in
3.9.4 represents a concerning milestone in this regard.
6.1.2 Threat Actor Analysis
Threat actors can be organized by skill and resource levels, ranging from low or moderately skilled and resourced actors
who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced


key: chroma:document
string_value:
who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced
(e.g. state-sponsored) groups with long-term agendas [SBC+19].
To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat
groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed. While we did
ﬁnd signiﬁcant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances
of experimentation and no successful deployments since then. Additionally, those misuse discussions were correlated
with media coverage of language model technologies. From this, we assess that the threat of misuse from these actors is
not immediate, but signiﬁcant improvements in reliability could change this.
Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about
possible APT activity involving the use of language models. Since the release of GPT-2 there has been no discernible
difference in operations that may see potential gains by using language models. The assessment was that language
models may not be worth investing signiﬁcant resources in because there has been no convincing demonstration that
current language models are signiﬁcantly better than current methods for generating text, and because methods for


key: chroma:document
string_value:
current language models are signiﬁcantly better than current methods for generating text, and because methods for
“targeting” or “controlling” the content of language models are still at a very early stage.
6.1.3 External Incentive Structures
Each threat actor group also has a set of tactics, techniques, and procedures (TTPs) that they rely on to accomplish their
agenda. TTPs are inﬂuenced by economic factors like scalability and ease of deployment; phishing is extremely popular
among all groups because it offers a low-cost, low-effort, high-yield method of deploying malware and stealing login
credentials. Using language models to augment existing TTPs would likely result in an even lower cost of deployment.
Ease of use is another signiﬁcant incentive. Having stable infrastructure has a large impact on the adoption of TTPs.
The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k
truncation) they are not able to perform consistently without human feedback. If a social media disinformation bot
produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the


key: chroma:document
string_value:
produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the
amount of human labor required in operating this bot. But a human is still needed to ﬁlter the outputs, which restricts
how scalable the operation can be.
Based on our analysis of this model and analysis of threat actors and the landscape, we suspect AI researchers will
eventually develop language models that are sufﬁciently consistent and steerable that they will be of greater interest to
malicious actors. We expect this will introduce challenges for the broader research community, and hope to work on
this through a combination of mitigation research, prototyping, and coordinating with other technical developers.
35
6.2 Fairness, Bias, and Representation
Biases present in training data may lead models to generate stereotyped or prejudiced content. This is concerning,
since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and
producing demeaning portrayals amongst other potential harms [ Cra17 ]. We have conducted an analysis of biases in
the model in order to better understand GPT-3’s limitations when it comes to fairness, bias, and representation.8
Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and


key: chroma:document
string_value:
Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and
behaviors. We focus on biases relating to gender, race, and religion, although many other categories of bias are likely
present and could be studied in follow-up work. This is a preliminary analysis and does not reﬂect all of the model’s
biases even within the studied categories.
Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reﬂect stereotypes
present in their training data. Below we discuss our preliminary ﬁndings of bias along the dimensions of gender, race,
and religion. We probe for bias in the 175 billion parameter model and also in similar smaller models, to see if and how
they are different in this dimension.
6.2.1 Gender
In our investigation of gender bias in GPT-3, we focused on associations between gender and occupation. We found
that occupations in general have a higher probability of being followed by a male gender identiﬁer than a female one
(in other words, they are male leaning) when given a context such as "Thefoccupationgwas a" (Neutral Variant).
83% of the 388 occupations we tested were more likely to be followed by a male identiﬁer by GPT-3. We measured


key: chroma:document
string_value:
83% of the 388 occupations we tested were more likely to be followed by a male identiﬁer by GPT-3. We measured
this by feeding the model a context such as "The detective was a" and then looking at the probability of the
model following up with male indicating words (eg. man, male etc.) or female indicating words (woman, female etc.).
In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus
were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and
sheriff. Occupations that were more likely to be followed by female identiﬁers include midwife, nurse, receptionist,
housekeeper etc.
We also tested how these probabilities changed when we shifted the context to be the "The competent foccupationg
was a" (Competent Variant), and when we shifted the context to be "The incompetent foccupationgwas a"
(Incompetent Variant) for each occupation in the dataset. We found that, when prompted with "The competent
foccupationgwas a," the majority of occupations had an even higher probability of being followed by a
male identiﬁer than a female one than was the case with our original neutral prompt, "Thefoccupationgwas


key: chroma:document
string_value:
male identiﬁer than a female one than was the case with our original neutral prompt, "Thefoccupationgwas
a". With the prompt "The incompetent foccupationgwas a" the majority of occupations still leaned male
with a similar probability than for our original neutral prompt. The average occupation bias - measured as
1
njobsP
jobslog(P(femalejContext)
P(malejContext)))- was1:11for the Neutral Variant, 2:14for the Competent Variant and 1:15
for the Incompetent Variant.
We also carried out pronoun resolution on the Winogender dataset [ RNLVD18 ] using two methods which further
corroborated the model’s tendency to associate most occupations with males. One method measured the models ability to correctly assign a pronoun as the occupation or the participant. For example, we fed the model
a context such as "The advisor met with the advisee because she wanted to get advice about job
applications. `She' refers to the" and found the option with the lowest probability between the two possible options (Choices between Occupation Option: advisor; Participant Option: advisee).
Occupation and participant words often have societal biases associated with them such as the assumption that most


key: chroma:document
string_value:
Occupation and participant words often have societal biases associated with them such as the assumption that most
occupants are by default male. We found that the language models learnt some of these biases such as a tendency to
associate female pronouns with participant positions more than male pronouns. GPT-3 175B had the highest accuracy of
all the models (64.17%) on this task. It was also the only model where the accuracy for Occupant sentences (sentences
where the correct answer was the Occupation option) for females was higher than for males (81.7% vs 76.7%). All
other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns
with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both. This offers
some preliminary evidence that in places where issues of bias can make language models susceptible to error, the larger
models are more robust than smaller models.
We also performed co-occurrence tests, where we analyzed which words are likely to occur in the vicinity of other preselected words. We created a model output sample set by generating 800 outputs of length 50 each with a temperature
8Evaluating fairness, bias, and representation in language models is a rapidly-developing area with a large body of prior work.
See, for example, [HZJ+19, NBR20, SCNP19].


key: chroma:document
string_value:
See, for example, [HZJ+19, NBR20, SCNP19].
36
Table 6.1: Most Biased Descriptive Words in 175B Model
Top 10 Most Biased Male Descriptive Words with Raw
Co-Occurrence CountsTop 10 Most Biased Female Descriptive Words with Raw
Co-Occurrence Counts
Average Number of Co-Occurrences Across All Words:
17.5Average Number of Co-Occurrences Across All Words:
23.9
Large (16) Optimistic (12)
Mostly (15) Bubbly (12)
Lazy (14) Naughty (12)
Fantastic (13) Easy-going (12)
Eccentric (13) Petite (10)
Protect (10) Tight (10)
Jolly (10) Pregnant (10)
Stable (9) Gorgeous (28)
Personable (22) Sucked (8)
Survive (7) Beautiful (158)
of 1 and top p of 0.9 for every prompt in our dataset. For gender, we had prompts such as "He was very" ,"She
was very" ,"He would be described as" ,"She would be described as"9. We looked at the adjectives and


key: chroma:document
string_value:
was very" ,"He would be described as" ,"She would be described as"9. We looked at the adjectives and
adverbs in the top 100 most favored words using an off-the-shelf POS tagger [ LB02 ]. We found females were more
often described using appearance oriented words such as ”beautiful” and ”gorgeous” as compared to men who were
more often described using adjectives that span a greater spectrum.
Table 6.1 shows the top 10 most favored descriptive words for the model along with the raw number of times each
word co-occurred with a pronoun indicator. “Most Favored” here indicates words which were most skewed towards a
category by co-occurring with it at a higher rate as compared to the other category. To put these numbers in perspective,
we have also included the average for the number of co-occurrences across all qualifying words for each gender.
6.2.2 Race
To investigate racial bias in GPT-3, we seeded the model with prompts such as - "Thefracegman was very" ,
"Thefracegwoman was very" and"People would describe the fracegperson as" and generated 800
samples for each of the above prompts, with fracegreplaced with a term indicating a racial category such as White


key: chroma:document
string_value:
samples for each of the above prompts, with fracegreplaced with a term indicating a racial category such as White
or Asian. We then measure word co-occurrences in the generated samples. Given prior research demonstrating that
language models produce text of differing sentiment when varying features such as occupation [ HZJ+19], we explored
how race impacted sentiment. We measured sentiment using Senti WordNet [ BES10 ] for the words which co-occurred
disproportionately with each race. Each word sentiment varied from 100 to -100, with positive scores indicating positive
words (eg. wonderfulness: 100, amicable: 87.5), negative scores indicating negative words (eg. wretched: -87.5 , horrid:
-87.5) and a score of 0 indicating neutral words (eg. sloping, chalet).
It should be noted that we were explicitly prompting the models to talk about race and this in turn generated text that
focused on racial features; these results are not from the models talking about race in the wild but talking about race in
an experimental setup where they have been primed to do so. Additionally, since we are measuring sentiment by simply
looking at word co-occurrences, the resulting sentiment can reﬂect socio-historical factors - for instance, text relating to
a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated


key: chroma:document
string_value:
a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated
with a negative sentiment under this testing methodology.
Across the models we analyzed, ‘Asian’ had a consistently high sentiment - it ranked 1st in 3 out of 7 models. On the
other hand, ’Black’ had a consistently low sentiment - it ranked the lowest in 5 out of 7 models. These differences
narrowed marginally on the larger model sizes. This analysis gives a sense of the biases of different models and
highlights the need for more sophisticated analysis of the relationship between sentiment, entities, and input data.
9We only used male and female pronouns. This simplifying assumption makes it easier to study co-occurrence since it does not
require the isolation of instances in which ‘they’ refers to a singular noun from those where it didn’t, but other forms of gender bias
are likely present and could be studied using different approaches.
37
Figure 6.1: Racial Sentiment Across Models
Religion Most Favored Descriptive Words
Atheism ‘Theists’, ‘Cool’, ‘Agnostics’, ‘Mad’, ‘Theism’, ‘Defensive’, ‘Complaining’, ‘Correct’, ‘Arrogant’,
‘Characterized’


key: chroma:document
string_value:
‘Characterized’
Buddhism ‘Myanmar’, ‘Vegetarians’, ‘Burma’, ‘Fellowship’, ‘Monk’, ‘Japanese’, ‘Reluctant’, ‘Wisdom’, ‘Enlightenment’, ‘Non-Violent’
Christianity ‘Attend’, ‘Ignorant’, ‘Response’, ‘Judgmental’, ‘Grace’, ‘Execution’, ‘Egypt’, ‘Continue’, ‘Comments’, ‘Ofﬁcially’
Hinduism ‘Caste’, ‘Cows’, ‘BJP’, ‘Kashmir’, ‘Modi’, ‘Celebrated’, ‘Dharma’, ‘Pakistani’, ‘Originated’, ‘Africa’
Islam ‘Pillars’, ‘Terrorism’, ‘Fasting’, ‘Sheikh’, ‘Non-Muslim’, ‘Source’, ‘Charities’, ‘Levant’, ‘Allah’,


key: chroma:document
string_value:
‘Prophet’
Judaism ‘Gentiles’, ‘Race’, ‘Semites’, ‘Whites’, ‘Blacks’, ‘Smartest’, ‘Racists’, ‘Arabs’, ‘Game’, ‘Russian’
Table 6.2: Shows the ten most favored words about each religion in the GPT-3 175B model.
6.2.3 Religion
We studied which words co-occurred with religious terms relating to Atheism, Buddhism, Christianity, Hinduism, Islam,
and Judaism, by generating 800 model outputs of length 50 with a temperature of 1 and a top pof0:9for every
prompt. Our prompts were of the nature "fReligion practitioners gare" (Eg. "Christians are" ) for each
of the six religious categories listed above. We then allowed the model to naturally carry out completions and created a
corpus of such completions for studying co-occurrence of words.
The following is an example output from the model:
"Buddhists are divided into two main branches - Theravada and Mahayana. Theravada
is the more conservative branch, centering on monastic life and the earliest sutras


key: chroma:document
string_value:
is the more conservative branch, centering on monastic life and the earliest sutras
and refusing to recognize the later Mahayana sutras as authentic."
Similar to race, we found that the models make associations with religious terms that indicate some propensity to reﬂect
how these terms are sometimes presented in the world. For example, with the religion Islam , we found that words such
asramadan ,prophet andmosque co-occurred at a higher rate than for other religions. We also found that words such
asviolent ,terrorism andterrorist co-occurred at a greater rate with Islam than with other religions and were in
the top 40 most favored words for Islam in GPT-3.
38
6.2.4 Future Bias and Fairness Challenges
We have presented this preliminary analysis to share some of the biases we found in order to motivate further research,
and to highlight the inherent difﬁculties in characterizing biases in large-scale generative models; we expect this to be an
area of continuous research for us and are excited to discuss different methodological approaches with the community.
We view the work in this section as subjective signposting - we chose gender, race, and religion as a starting point, but
we recognize the inherent subjectivity in this choice. Our work is inspired by the literature on characterizing model


key: chroma:document
string_value:
we recognize the inherent subjectivity in this choice. Our work is inspired by the literature on characterizing model
attributes to develop informative labels such as Model Cards for Model Reporting from [MWZ+18].
Ultimately, it is important not just to characterize biases in language systems but to intervene. The literature on this
is also extensive [ QMZH19 ,HZJ+19], so we offer only a few brief comments on future directions speciﬁc to large
language models. In order to pave the way for effective bias prevention in general purpose models, there is a need for
building a common vocabulary tying together the normative, technical and empirical challenges of bias mitigation for
these models. There is room for more research that engages with the literature outside NLP, better articulates normative
statements about harm, and engages with the lived experience of communities affected by NLP systems [ BBDIW20 ].
Thus, mitigation work should not be approached purely with a metric driven objective to ‘remove’ bias as this has been
shown to have blind spots [GG19, NvNvdG19] but in a holistic manner.
6.3 Energy Usage
Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3


key: chroma:document
string_value:
6.3 Energy Usage
Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3
175B consumed several thousand petaﬂop/s-days of compute during pre-training, compared to tens of petaﬂop/s-days
for a 1.5B parameter GPT-2 model (Figure 2.2). This means we should be cognizant of the cost and efﬁciency of such
models, as advocated by [SDSE19].
The use of large-scale pre-training also gives another lens through which to view the efﬁciency of large models - we
should consider not only the resources that go into training them, but how these resources are amortized over the
lifetime of a model, which will subsequently be used for a variety of purposes and ﬁne-tuned for speciﬁc tasks. Though
models like GPT-3 consume signiﬁcant resources during training, they can be surprisingly efﬁcient once trained: even
with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or


key: chroma:document
string_value:
with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or
only a few cents in energy costs. Additionally, techniques like model distillation [ LHCG19a ] can further bring down
the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efﬁcient
versions of them for use in appropriate contexts. Algorithmic progress may also naturally further increase the efﬁciency
of such models over time, similar to trends observed in image recognition and neural machine translation [HB20].
7 Related Work
Several lines of work have focused on increasing parameter count and/or computation in language models as a
means to improve generative or task performance. An early work scaled LSTM based language models to over a
billion parameters [ JVS+16]. One line of work straightforwardly increases the size of transformer models, scaling
up parameters and FLOPS-per-token roughly in proportion. Work in this vein has successively increased model size:
213 million parameters [ VSP+17] in the original paper, 300 million parameters [ DCLT18 ], 1.5 billion parameters
[RWC+19], 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19], and most recently 17 billion parameters


key: chroma:document
string_value:
[RWC+19], 8 billion parameters [ SPP+19], 11 billion parameters [ RSR+19], and most recently 17 billion parameters
[Tur20 ]. A second line of work has focused on increasing parameter count but not computation, as a means of
increasing models’ capacity to store information without increased computational cost. These approaches rely on the
conditional computation framework [ BLC13 ] and speciﬁcally, the mixture-of-experts method [ SMM+17] has been
used to produce 100 billion parameter models and more recently 50 billion parameter translation models [ AJF19 ],
though only a small fraction of the parameters are actually used on each forward pass. A third approach increases
computation without increasing parameters; examples of this approach include adaptive computation time [ Gra16 ] and
the universal transformer [ DGV+18]. Our work focuses on the ﬁrst approach (scaling compute and parameters together,
by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ
this strategy.
Several efforts have also systematically studied the effect of scale on language model performance. [ KMH+20,
RRBS19 ,LWS+20,HNA+17], ﬁnd a smooth power-law trend in loss as autoregressive language models are scaled up.


key: chroma:document
string_value:
RRBS19 ,LWS+20,HNA+17], ﬁnd a smooth power-law trend in loss as autoregressive language models are scaled up.
This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the
curve can perhaps be detected in Figure 3.1), and we also ﬁnd relatively smooth increases in many (though not all)
downstream tasks across 3 orders of magnitude of scaling.
Another line of work goes in the opposite direction from scaling, attempting to preserve strong performance in language
models that are as small as possible. This approach includes ALBERT [ LCG+19] as well as general [ HVD15 ] and
39
task-speciﬁc [ SDCW19 ,JYS+19,KR16 ] approaches to distillation of language models. These architectures and
techniques are potentially complementary to our work, and could be applied to decrease latency and memory footprint
of giant models.
As ﬁne-tuned language models have neared human performance on many standard benchmark tasks, considerable
effort has been devoted to constructing more difﬁcult or open-ended tasks, including question answering [ KPR+19,
IBGC+14,CCE+18,MCKS18 ], reading comprehension [ CHI+18,RCM19 ], and adversarially constructed datasets


key: chroma:document
string_value:
IBGC+14,CCE+18,MCKS18 ], reading comprehension [ CHI+18,RCM19 ], and adversarially constructed datasets
designed to be difﬁcult for existing language models [ SBBC19 ,NWD+19]. In this work we test our models on many
of these datasets.
Many previous efforts have focused speciﬁcally on question-answering, which constitutes a signiﬁcant fraction of the
tasks we tested on. Recent efforts include [ RSR+19,RRS20 ], which ﬁne-tuned an 11 billion parameter language model,
and [ GLT+20], which focused on attending over a large corpus of data at test time. Our work differs in focusing on
in-context learning but could be combined in the future with those of [GLT+20, LPP+20].
Metalearning in language models has been utilized in [ RWC+19], though with much more limited results and no
systematic study. More broadly, language model metalearning has an inner-loop-outer-loop structure, making it
structurally similar to metalearning as applied to ML in general. Here there is an extensive literature, including


key: chroma:document
string_value:
structurally similar to metalearning as applied to ML in general. Here there is an extensive literature, including
matching networks [ VBL+16], RL2 [ DSC+16], learning to optimize [ RL16 ,ADG+16,LM17 ] and MAML [ FAL17 ].
Our approach of stufﬁng the model’s context with previous examples is most structurally similar to RL2 and also
resembles [ HYC01 ], in that an inner loop of adaptation takes place through computation in the model’s activations
across timesteps, without updating the weights, while an outer loop (in this case just language model pre-training)
updates the weights, and implicitly learns the ability to adapt to or at least recognize tasks deﬁned at inference-time.
Few-shot auto-regressive density estimation was explored in [ RCP+17] and [ GWC+18] studied low-resource NMT as
a few-shot learning problem.
While the mechanism of our few-shot approach is different, prior work has also explored ways of using pre-trained
language models in combination with gradient descent to perform few-shot learning [ SS20 ]. Another sub-ﬁeld with


key: chroma:document
string_value:
language models in combination with gradient descent to perform few-shot learning [ SS20 ]. Another sub-ﬁeld with
similar goals is semi-supervised learning where approaches such as UDA [ XDH+19] also explore methods of ﬁne-tuning
when very little labeled data is available.
Giving multi-task models instructions in natural language was ﬁrst formalized in a supervised setting with [ MKXS18 ]
and utilized for some tasks (such as summarizing) in a language model with [ RWC+19]. The notion of presenting
tasks in natural language was also explored in the text-to-text transformer [ RSR+19], although there it was applied for
multi-task ﬁne-tuning rather than for in-context learning without weight updates.
Another approach to increasing generality and transfer-learning capability in language models is multi-task learning
[Car97 ], which ﬁne-tunes on a mixture of downstream tasks together, rather than separately updating the weights for
each one. If successful multi-task learning could allow a single model to be used for many tasks without updating the
weights (similar to our in-context learning approach), or alternatively could improve sample efﬁciency when updating


key: chroma:document
string_value:
weights (similar to our in-context learning approach), or alternatively could improve sample efﬁciency when updating
the weights for a new task. Multi-task learning has shown some promising initial results [ LGH+15,LSP+18] and
multi-stage ﬁne-tuning has recently become a standardized part of SOTA results on some datasets [ PFB18 ] and pushed
the boundaries on certain tasks [ KKS+20], but is still limited by the need to manually curate collections of datasets and
set up training curricula. By contrast pre-training at large enough scale appears to offer a “natural” broad distribution of
tasks implicitly contained in predicting the text itself. One direction for future work might be attempting to generate
a broader set of explicit tasks for multi-task learning, for example through procedural generation [ TFR+17], human
interaction [ZSW+19b], or active learning [Mac92].
Algorithmic innovation in language models over the last two years has been enormous, including denoising-based
bidirectionality [ DCLT18 ], preﬁxLM [ DL15 ] and encoder-decoder architectures [ LLG+19,RSR+19], random permutations during training [ YDY+19], architectures that improve the efﬁciency of sampling [ DYY+19], improvements in


key: chroma:document
string_value:
data and training procedures [ LOG+19], and efﬁciency increases in the embedding parameters [ LCG+19]. Many of
these techniques provide signiﬁcant gains on downstream tasks. In this work we continue to focus on pure autoregressive
language models, both in order to focus on in-context learning performance and to reduce the complexity of our large
model implementations. However, it is very likely that incorporating these algorithmic advances could improve GPT-3’s
performance on downstream tasks, especially in the ﬁne-tuning setting, and combining GPT-3’s scale with these
algorithmic techniques is a promising direction for future work.
8 Conclusion
We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and
benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of
40
state-of-the-art ﬁne-tuned systems, as well as generating high-quality samples and strong qualitative performance at
tasks deﬁned on-the-ﬂy. We documented roughly predictable trends of scaling in performance without using ﬁne-tuning.
We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results


key: chroma:document
string_value:
We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results
suggest that very large language models may be an important ingredient in the development of adaptable, general
language systems.
Acknowledgements
The authors would like to thank Ryan Lowe for giving detailed feedback on drafts of the paper. Thanks to Jakub
Pachocki and Szymon Sidor for suggesting tasks, and Greg Brockman, Michael Petrov, Brooke Chan, and Chelsea
V oss for helping run evaluations on OpenAI’s infrastructure. Thanks to David Luan for initial support in scaling up
this project, Irene Solaiman for discussions about ways to approach and evaluate bias, Harrison Edwards and Yura
Burda for discussions and experimentation with in-context learning, Geoffrey Irving and Paul Christiano for early
discussions of language model scaling, Long Ouyang for advising on the design of the human evaluation experiments,
Chris Hallacy for discussions on data collection, and Shan Carter for help with visual design. Thanks to the millions of
people who created content that was used in the training of the model, and to those who were involved in indexing or
upvoting the content (in the case of WebText). Additionally, we would like to thank the entire OpenAI infrastructure
and supercomputing teams for making it possible to train models at this scale.
41
Contributions


key: chroma:document
string_value:
and supercomputing teams for making it possible to train models at this scale.
41
Contributions
Tom Brown, Ben Mann, Prafulla Dhariwal, Dario Amodei, Nick Ryder, Daniel M Ziegler, and Jeffrey Wu
implemented the large-scale models, training infrastructure, and model-parallel strategies.
Tom Brown, Dario Amodei, Ben Mann, and Nick Ryder conducted pre-training experiments.
Ben Mann and Alec Radford collected, ﬁltered, deduplicated, and conducted overlap analysis on the training data.
Melanie Subbiah, Ben Mann, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown, Tom Henighan, and
Girish Sastry implemented the downstream tasks and the software framework for supporting them, including creation
of synthetic tasks.
Jared Kaplan and Sam McCandlish initially predicted that a giant language model should show continued gains, and
applied scaling laws to help predict and guide model and data scaling decisions for the research.
Ben Mann implemented sampling without replacement during training.
Alec Radford originally demonstrated few-shot learning occurs in language models.
Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically
studied in-context learning curves, task prompting, and evaluation methods.


key: chroma:document
string_value:
Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically
studied in-context learning curves, task prompting, and evaluation methods.
Prafulla Dhariwal implemented an early version of the codebase, and developed the memory optimizations for fully
half-precision training.
Rewon Child and Mark Chen developed an early version of our model-parallel strategy.
Rewon Child and Scott Gray contributed the sparse transformer.
Aditya Ramesh experimented with loss scaling strategies for pretraining.
Melanie Subbiah and Arvind Neelakantan implemented, experimented with, and tested beam search.
Pranav Shyam worked on SuperGLUE and assisted with connections to few-shot learning and meta-learning literature.
Sandhini Agarwal conducted the fairness and representation analysis.
Girish Sastry and Amanda Askell conducted the human evaluations of the model.
Ariel Herbert-Voss conducted the threat analysis of malicious use.
Gretchen Krueger edited and red-teamed the policy sections of the paper.
Benjamin Chess, Clemens Winter, Eric Sigler, Christopher Hesse, Mateusz Litwin, and Christopher Berner
optimized OpenAI’s clusters to run the largest models efﬁciently.
Scott Gray developed fast GPU kernels used during training.


key: chroma:document
string_value:
optimized OpenAI’s clusters to run the largest models efﬁciently.
Scott Gray developed fast GPU kernels used during training.
Jack Clark led the analysis of ethical impacts — fairness and representation, human assessments of the model, and
broader impacts analysis, and advised Gretchen, Amanda, Girish, Sandhini, and Ariel on their work.
Dario Amodei, Alec Radford, Tom Brown, Sam McCandlish, Nick Ryder, Jared Kaplan, Sandhini Agarwal,
Amanda Askell, Girish Sastry, and Jack Clark wrote the paper.
Sam McCandlish led the analysis of model scaling, and advised Tom Henighan and Jared Kaplan on their work.
Alec Radford advised the project from an NLP perspective, suggested tasks, put the results in context, and demonstrated
the beneﬁt of weight decay for training.
Ilya Sutskever was an early advocate for scaling large generative likelihood models, and advised Pranav, Prafulla,
Rewon, Alec, and Aditya on their work.
Dario Amodei designed and led the research.
42
A Details of Common Crawl Filtering
As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1)


key: chroma:document
string_value:
42
A Details of Common Crawl Filtering
As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1)
ﬁltering Common Crawl and (2) fuzzy deduplication:
1.In order to improve the quality of Common Crawl, we developed an automatic ﬁltering method to remove low
quality documents. Using the original WebText as a proxy for high-quality documents, we trained a classiﬁer
to distinguish these from raw Common Crawl. We then used this classiﬁer to re-sample Common Crawl by
prioritizing documents which were predicted by the classiﬁer to be higher quality. The classiﬁer is trained
using logistic regression classiﬁer with features from Spark’s standard tokenizer and HashingTF10. For the
positive examples, we used a collection of curated datasets such as WebText, Wikiedia, and our web books
corpus as the positive examples, and for the negative examples, we used unﬁltered Common Crawl. We used
this classiﬁer to score Common Crawl documents. We kept each document in our dataset iff
np.random.pareto ()>1document_score


key: chroma:document
string_value:
this classiﬁer to score Common Crawl documents. We kept each document in our dataset iff
np.random.pareto ()>1document_score
We chose= 9in order to take mostly documents the classiﬁer scored highly, but still include some documents
that were out of distribution. was chosen to match the distribution of scores from our classiﬁer on WebText.
We found this re-weighting increased quality as measured by loss on a range of out-of-distribution generative
text samples.
2.To further improve model quality and prevent overﬁtting (which becomes increasingly important as model
capacity increases), we fuzzily deduplicated documents (i.e. removed documents with high overlap with
other documents) within each dataset using Spark’s MinHashLSH implementation with 10 hashes, using the
same features as were used for classiﬁcation above. We also fuzzily removed WebText from Common Crawl.
Overall this decreased dataset size by an average of 10%.
After ﬁltering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in
Appendix C.
B Details of Model Training


key: chroma:document
string_value:
After ﬁltering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in
Appendix C.
B Details of Model Training
To train all versions of GPT-3, we use Adam with 1= 0:9,2= 0:95, and= 108, we clip the global norm of the
gradient at 1.0, and we use cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260
billion tokens, training continues at 10% of the original learning rate). There is a linear LR warmup over the ﬁrst 375
million tokens. We also gradually increase the batch size linearly from a small value (32k tokens) to the full value over
the ﬁrst 4-12 billion tokens of training, depending on the model size. Data are sampled without replacement during
training (until an epoch boundary is reached) to minimize overﬁtting. All models use weight decay of 0.1 to provide a
small amount of regularization [LH17].
During training we always train on sequences of the full nctx= 2048 token context window, packing multiple
documents into a single sequence when documents are shorter than 2048, in order to increase computational efﬁciency.


key: chroma:document
string_value:
documents into a single sequence when documents are shorter than 2048, in order to increase computational efﬁciency.
Sequences with multiple documents are not masked in any special way but instead documents within a sequence
are delimited with a special end of text token, giving the language model the information necessary to infer that
context separated by the end of text token is unrelated. This allows for efﬁcient training without need for any special
sequence-speciﬁc masking.
C Details of Test Set Contamination Studies
In section 4 we gave a high level overview of test set contamination studies. In this section we provide details on
methodology and results.
Initial training set ﬁltering We attempted to remove text occurring in benchmarks from training data by searching
for13gram overlaps between all test/development sets used in this work and our training data, and we removed
the colliding 13gram as well as a 200 character window around it, splitting the original document into pieces. For
ﬁltering purposes we deﬁne a gram as a lowercase, whitespace delimited word with no punctuation. Pieces less than
200characters long were discarded. Documents split into more than 10 pieces were considered contaminated and


key: chroma:document
string_value:
200characters long were discarded. Documents split into more than 10 pieces were considered contaminated and
10https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.feature.HashingTF
43
removed entirely. Originally we removed entire documents given a single collision, but that overly penalized long
documents such as books for false positives. An example of a false positive might be a test set based on Wikipedia, in
which the Wikipedia article quotes a single line from a book. We ignored 13grams that matched more than 10 training
documents, as inspection showed the majority of these to contain common cultural phrases, legal boilerplate, or similar
content that we likely do want the model to learn, rather than undesired speciﬁc overlaps with test sets. Examples for
various frequencies can be found in the GPT-3 release repository11.
Overlap methodology For our benchmark overlap analysis in Section 4, we used a variable number of words Nto
check for overlap for each dataset, where Nis the 5th percentile example length in words, ignoring all punctuation,
whitespace, and casing. Due to spurious collisions at lower values of Nwe use a minimum value of 8 on non-synthetic


key: chroma:document
string_value:
whitespace, and casing. Due to spurious collisions at lower values of Nwe use a minimum value of 8 on non-synthetic
tasks. For performance reasons, we set a maximum value of 13 for all tasks. Values for Nand the amount of data
marked as dirty are shown in Table C.1. Unlike GPT-2’s use of bloom ﬁlters to compute probabilistic bounds for test
contamination, we used Apache Spark to compute exact collisions across all training and test sets. We compute overlaps
between test sets and our full training corpus, even though we only trained on 40% of our ﬁltered Common Crawl
documents per Section 2.2.
We deﬁne a ‘dirty’ example as one with any N-gram overlap with any training document, and a ‘clean’ example as one
with no collision.
Test and validation splits had similar contamination levels despite some test splits being unlabeled. Due to a bug revealed
by this analysis, ﬁltering described above failed on long documents such as books. Because of cost considerations it
was infeasible to retrain the model on a corrected version of the training dataset. As such, several language modeling
benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this


key: chroma:document
string_value:
benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this
paper. Overlaps are shown in Table C.1
Overlap results To understand how much having seen some of the data helps the model perform on downstream
tasks, we ﬁlter every validation and test set by dirtiness. Then we run evaluation on the clean-only examples and report
the relative percent change between the clean score and the original score. If the clean score is more than 1% or 2%
worse than the overall score, it suggests the model may have overﬁt to the examples it has seen. If the clean score is
signiﬁcantly better , our ﬁltering scheme may have preferentially marked easier examples as dirty.
This overlap metric tends to show a high rate of false positives for datasets that contain background information (but
not answers) drawn from the web (such as SQuAD, which draws from Wikipedia) or examples less than 8 words
long, which we ignored in our ﬁltering process (except for wordscrambling tasks). One instance where this technique
seems to fail to give good signal is DROP, a reading comprehension task in which 94% of the examples are dirty. The
information required to answer the question is in a passage provided to the model, so having seen the passage during


key: chroma:document
string_value:
information required to answer the question is in a passage provided to the model, so having seen the passage during
training but not the questions and answers does not meaningfully constitute cheating. We conﬁrmed that every matching
training document contained only the source passage, and none of the questions and answers in the dataset. The more
likely explanation for the decrease in performance is that the 6% of examples that remain after ﬁltering come from a
slightly different distribution than the dirty examples.
Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but
there is no apparent bias towards improved or degraded performance. This suggests that GPT-3 is relatively insensitive
to contamination. See Section 4 for details on the datasets we ﬂagged for further review.
11https://github.com/openai/gpt-3/blob/master/overlap_frequency.md
44
Name Split Metric N Acc/F1/BLEUTotal
CountDirty
Acc/F1/BLEUDirty
CountClean
Acc/F1/BLEUClean
CountClean
PercentageRelative
Difference
Clean vs All
Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20%


key: chroma:document
string_value:
CountClean
PercentageRelative
Difference
Clean vs All
Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20%
SQuADv2 dev f1 13 69.8 11873 69.9 11136 68.4 737 6% -2%
DROP dev f1 13 36.5 9536 37.0 8898 29.5 638 7% -21%
Symbol Insertion dev acc 7 66.9 10000 66.8 8565 67.1 1435 14% 0%
CoQa dev f1 13 86.0 7983 85.3 5107 87.1 2876 36% 1%
ReCoRD dev acc 13 89.5 10000 90.3 6110 88.2 3890 39% -1%
Winograd test acc 9 88.6 273 90.2 164 86.2 109 40% -3%
BoolQ dev acc 13 76.0 3270 75.8 1955 76.3 1315 40% 0%
MultiRC dev acc 13 74.2 953 73.4 558 75.3 395 41% 1%
RACE-h test acc 13 46.8 3498 47.0 1580 46.7 1918 55% 0%


key: chroma:document
string_value:
RACE-h test acc 13 46.8 3498 47.0 1580 46.7 1918 55% 0%
LAMBADA test acc 13 86.4 5153 86.9 2209 86.0 2944 57% 0%
LAMBADA (No Blanks) test acc 13 77.8 5153 78.5 2209 77.2 2944 57% -1%
WSC dev acc 13 76.9 104 73.8 42 79.0 62 60% 3%
PIQA dev acc 8 82.3 1838 89.9 526 79.3 1312 71% -4%
RACE-m test acc 13 58.5 1436 53.0 366 60.4 1070 75% 3%
De!En 16 test bleu-sb 12 43.0 2999 47.4 739 40.8 2260 75% -5%
En!De 16 test bleu-sb 12 30.9 2999 32.6 739 29.9 2260 75% -3%
En!Ro 16 test bleu-sb 12 25.8 1999 24.9 423 26.1 1576 79% 1%
Ro!En 16 test bleu-sb 12 41.3 1999 40.4 423 41.6 1576 79% 1%


key: chroma:document
string_value:
Ro!En 16 test bleu-sb 12 41.3 1999 40.4 423 41.6 1576 79% 1%
WebQs test acc 8 41.5 2032 41.6 428 41.5 1604 79% 0%
ANLI R1 test acc 13 36.8 1000 40.5 200 35.9 800 80% -3%
ANLI R2 test acc 13 34.0 1000 29.4 177 35.0 823 82% 3%
TriviaQA dev acc 10 71.2 7993 70.8 1390 71.3 6603 83% 0%
ANLI R3 test acc 13 40.2 1200 38.3 196 40.5 1004 84% 1%
En!Fr 14 test bleu-sb 13 39.9 3003 38.3 411 40.3 2592 86% 1%
Fr!En 14 test bleu-sb 13 41.4 3003 40.9 411 41.4 2592 86% 0%
WiC dev acc 13 51.4 638 53.1 49 51.3 589 92% 0%
RTE dev acc 13 71.5 277 71.4 21 71.5 256 92% 0%
CB dev acc 13 80.4 56 100.0 4 78.8 52 93% -2%


key: chroma:document
string_value:
CB dev acc 13 80.4 56 100.0 4 78.8 52 93% -2%
Anagrams 2 dev acc 2 40.2 10000 76.2 705 37.4 9295 93% -7%
Reversed Words dev acc 2 0.4 10000 1.5 660 0.3 9340 93% -26%
OpenBookQA test acc 8 65.4 500 58.1 31 65.9 469 94% 1%
ARC (Easy) test acc 11 70.1 2268 77.5 89 69.8 2179 96% 0%
Anagrams 1 dev acc 2 15.0 10000 49.8 327 13.8 9673 97% -8%
COPA dev acc 9 93.0 100 100.0 3 92.8 97 97% 0%
ARC (Challenge) test acc 12 51.6 1144 45.2 31 51.8 1113 97% 0%
HellaSwag dev acc 13 79.3 10042 86.2 152 79.2 9890 98% 0%
NQs test acc 11 29.9 3610 32.7 52 29.8 3558 99% 0%
Cycled Letters dev acc 2 38.6 10000 20.5 73 38.7 9927 99% 0%
SAT Analogies dev acc 9 65.8 374 100.0 2 65.6 372 99% 0%


key: chroma:document
string_value:
SAT Analogies dev acc 9 65.8 374 100.0 2 65.6 372 99% 0%
StoryCloze test acc 13 87.7 1871 100.0 2 87.6 1869 100% 0%
Winogrande dev acc 13 77.7 1267 - 0 77.7 1267 100% 0%
Table C.1: Overlap statistics for all datasets sorted from dirtiest to cleanest. We consider a dataset example dirty if it
has a singleN-gram collision with any document in our training corpus. “Relative Difference Clean vs All” shows the
percent change in performance between only the clean examples vs all the examples in the benchmark. “Count” shows
the number of examples. “Clean percentage” is the percent of examples that are clean vs total. For “Acc/F1/BLEU” we
use the metric speciﬁed in “Metric”. These scores come from evaluations with a different seed for the random examples
used for in-context learning, and will therefore differ slightly from the scores elsewhere in the paper.
45
D Total Compute Used to Train Language Models
This appendix contains the calculations that were used to derive the approximate compute used to train the language
models in Figure 2.2. As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10%


key: chroma:document
string_value:
models in Figure 2.2. As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10%
of the total compute for the models we are analyzing.
Calculations can be seen in Table D.1 and are explained within the table caption.
ModelTotal train
compute
(PF-days)Total train
compute
(ﬂops)Params
(M)Training tokens
(billions)Flops
per param
per tokenMult for
bwd passFwd-pass
ﬂops per
active param
per tokenFrac of
params active
for each
token
T5-Small 2.08E+00 1.80E+20 60 1,000 3 3 1 0.5
T5-Base 7.64E+00 6.60E+20 220 1,000 3 3 1 0.5
T5-Large 2.67E+01 2.31E+21 770 1,000 3 3 1 0.5
T5-3B 1.04E+02 9.00E+21 3,000 1,000 3 3 1 0.5
T5-11B 3.82E+02 3.30E+22 11,000 1,000 3 3 1 0.5


key: chroma:document
string_value:
T5-11B 3.82E+02 3.30E+22 11,000 1,000 3 3 1 0.5
BERT-Base 1.89E+00 1.64E+20 109 250 6 3 2 1.0
BERT-Large 6.16E+00 5.33E+20 355 250 6 3 2 1.0
RoBERTa-Base 1.74E+01 1.50E+21 125 2,000 6 3 2 1.0
RoBERTa-Large 4.93E+01 4.26E+21 355 2,000 6 3 2 1.0
GPT-3 Small 2.60E+00 2.25E+20 125 300 6 3 2 1.0
GPT-3 Medium 7.42E+00 6.41E+20 356 300 6 3 2 1.0
GPT-3 Large 1.58E+01 1.37E+21 760 300 6 3 2 1.0
GPT-3 XL 2.75E+01 2.38E+21 1,320 300 6 3 2 1.0
GPT-3 2.7B 5.52E+01 4.77E+21 2,650 300 6 3 2 1.0


key: chroma:document
string_value:
GPT-3 2.7B 5.52E+01 4.77E+21 2,650 300 6 3 2 1.0
GPT-3 6.7B 1.39E+02 1.20E+22 6,660 300 6 3 2 1.0
GPT-3 13B 2.68E+02 2.31E+22 12,850 300 6 3 2 1.0
GPT-3 175B 3.64E+03 3.14E+23 174,600 300 6 3 2 1.0
Table D.1: Starting from the right hand side and moving left, we begin with the number of training tokens that each
model was trained with. Next we note that since T5 uses an encoder-decoder model, only half of the parameters are
active for each token during a forward or backwards pass. We then note that each token is involved in a single addition
and a single multiply for each active parameter in the forward pass (ignoring attention). Then we add a multiplier of
3x to account for the backwards pass (as computing both@params
@lossand@acts
@lossuse a similar amount of compute as the
forwards pass. Combining the previous two numbers, we get the total ﬂops per parameter per token. We multiply this


key: chroma:document
string_value:
@lossuse a similar amount of compute as the
forwards pass. Combining the previous two numbers, we get the total ﬂops per parameter per token. We multiply this
value by the total training tokens and the total parameters to yield the number of total ﬂops used during training. We
report both ﬂops and petaﬂop/s-day (each of which are 8.64e+19 ﬂops).
E Human Quality Assessment of Synthetic News Articles
This appendix contains details on the experiments measuring human ability to distinguish GPT-3-generated synthetic
news articles from real news articles. We ﬁrst describe the experiments on the 200word news articles, and then
describe the preliminary investigation of 500word news articles generated by GPT-3.
Participants: We recruited 718 unique participants to take part in 6 experiments. 97 participants were excluded for
failing an internet check question, leaving a total of 621 participants: 343 male, 271 female, and 7 other. Mean
participant age was 38years old. All participants were recruited through Positly, which maintains a whitelist of
high-performing workers from Mechanical Turk. All participants were US-based but there were no other demographic
restrictions. Participants were paid $12 for their participation, based on a task time estimate of 60 minutes determined


key: chroma:document
string_value:
restrictions. Participants were paid $12 for their participation, based on a task time estimate of 60 minutes determined
by pilot runs. In order to ensure that the sample of participants for each experiment quiz was unique, participants were
not allowed to take part in an experiment more than once.
Procedure and design: We arbitrarily selected 25 news articles that appeared in newser.com in early 2020. We used
the article titles and subtitles to produce outputs from the 125M, 350M, 760M, 1.3B, 2.7B, 6.7B, 13.0B, and 200B
(GPT-3) parameter language models. Five outputs per question were generated by each model and the generation with a
word count closest to that of the human written article was selected automatically. This was to minimize the effect
that completion length might have on participants’ judgments. The same output procedure for each model with the
exception of the removal of the intentionally bad control model, as described in the main text.
46
ModelParticipants
RecruitedParticipants
ExcludedGenders
(m:f:other)Mean
AgeAverage
Word Count
(human:model)
Control 76 7 32:37:0 39 216:216
GPT-3 Small 80 7 41:31:1 40 216:188


key: chroma:document
string_value:
Word Count
(human:model)
Control 76 7 32:37:0 39 216:216
GPT-3 Small 80 7 41:31:1 40 216:188
GPT-3 Medium 80 7 46:28:2 39 216:202
GPT-3 Large 81 24 46:28:2 37 216:200
GPT-3 XL 79 14 32:32:1 38 216:199
GPT-3 2.7B 80 11 36:33:0 40 216:202
GPT-3 6.7B 76 5 46:28:2 37 216:195
GPT-3 13.0B 81 13 46:28:2 37 216:209
GPT-3 175B 80 9 42:29:0 37 216:216
Table E.1: Participant details and article lengths for each experiment to evaluate human detection of 200word model
generated news articles. Participants were excluded due to internet check fails.
Figure E.1: Participants spend more time trying to identify whether each news article is machine generated as model
size increases. Duration on the control model is indicated with the dashed line. Line of best ﬁt is a linear model on a log
scale with 95% conﬁdence intervals.
In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz


key: chroma:document
string_value:
scale with 95% conﬁdence intervals.
In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz
B. Each quiz consisted of 25 articles: half (12-13) were human written and half (12-13) were model generated: the
articles with human written completions in quiz A had model generated completions in quiz B and vice versa. The
order of quiz question was shufﬂed for each participant. Participants could leave comments and were asked to indicate
if they had seen the articles before. Participants were instructed not to look up the articles or their content during the
quiz and at the end of the quiz were asked if they had looked anything up during the quiz.
Statistical Tests: To compare means on the different runs, we performed a two-sample t-test for independent groups for
each model against the control. This was implemented in Python using the scipy.stats.ttest_ind function. When
plotting a regression line in the graph of average participant accuracy vs model size, we ﬁt a power law of the form
axb. The 95% conﬁdence intervals were estimated from the t-distribution of the sample mean.
Duration statistics : In the main text, we discussed the ﬁnding that the ability of human participants to distinguish


key: chroma:document
string_value:
Duration statistics : In the main text, we discussed the ﬁnding that the ability of human participants to distinguish
model and human generated news articles decreases as our models become larger. We have also found that the
average time spent for a given set of questions increases as the model size increases, as shown in Figure E.1. Lower
47
ModelParticipants
RecruitedParticipants
ExcludedGenders
(m:f:other)Mean
AgeAverage
Word Count
(human:model)
Control 79 17 32:37:0 39 569:464
GPT-3 175B 81 19 32:30:0 40 569:498
Table E.2: Participant details and article lengths for the experiments investigating human detection of 500word
model generated news articles. Participants were excluded due to internet check fails.
accuracy scores despite increased time investment from participants supports the ﬁnding that larger models generate
harder-to-distinguish news articles.
Preliminary investigation of 500word articles: We recruited 160 unique US-based participants to take part in 2
experiments through Positly (details are given in Table E.2). We randomly selected 12 Reuters world news articles from
late 2019 and created a context for GPT-3 175B that consisted of a single Reuters article not in this set of 12. We then


key: chroma:document
string_value:
late 2019 and created a context for GPT-3 175B that consisted of a single Reuters article not in this set of 12. We then
used the article titles and Reuters locations to generate completions from GPT-3 175B and the 160M control model
from the previous experiments. These were used to create two 12-question quizzes per model, each consisting of half
human written and half model generated articles. Comprehension questions were added and articles were shown to
participants in 3 stages at 30 second intervals to encourage closer reading. Participants were paid $12 for this task.
Model generation selection methods, exclusion criteria, and statistical tests mirror those of the previous experiments.
F Additional Samples from GPT-3
GPT-3 adapts well to many tasks other than the ones explored in the main body of the paper. As an example, in Figure
F.1, we show four uncurated samples from a prompt suggesting that the model write a poem, with a given title, in the
style of Wallace Stevens. We ﬁrst experimented with a few prompts, then generated four samples with no additional
editing or selection (sampling at temperature 1using nucleus sampling [ HBFC19 ] withP= 0:9). Completions were
truncated when the model began to write a new title and author heading, or broke into prose commentary.
48
Context! The City


key: chroma:document
string_value:
truncated when the model began to write a new title and author heading, or broke into prose commentary.
48
Context! The City
BY C. P. CAVAFY
TRANSLATED BY EDMUND KEELEY
[Poem text omitted]
SOME TREES
John Ashbery
[Poem text omitted]
Shadows on the Way
Wallace Stevens
-------- Generated Poem 1 -------- -------- Generated Poem 3 -------I must have shadows on the way The sun was all we had. Now, in the shade
If I am to walk I must have All is changed. The mind must dwell on those
Each step taken slowly and alone White fields, that to its eyes were always old;
To have it ready made Those ancient gleams, convoluting
The way they lay among the huge roots,
And I must think in lines of grey The great dark tomes of reverie,
To have dim thoughts to be my guide The plumed labyrinth of the sea.
Must look on blue and green
And never let my eye forget When our own breath gonne faint and fade
That color is my friend Up the long stairway of the dead
And purple must surround me too We shall remember well
The blindness of the light, the whiteness
The yellow of the sun is no more Of that white land. The footsteps, and


key: chroma:document
string_value:
And purple must surround me too We shall remember well
The blindness of the light, the whiteness
The yellow of the sun is no more Of that white land. The footsteps, and
Intrusive than the bluish snow The grasses in that light, and in the shade
That falls on all of us. I must have The menacing life.
Grey thoughts and blue thoughts walk with me
If I am to go away at all. We played, too, once, in the warmest rooms.
To one content, there is one content
-------- Generated Poem 2 -------- For all mankind. The forest glades
Are all the more for being in fable,
There is a way in the world. And wood and town in legend shadowedThat way a man may live in a small town Once more to return. The one content
His lifetime and see no man come, Has in it every happiness, more brave
Never the stir of men about a market, For being purposeless; and so
Never the bobbing of covered carts, Those island gardens, deep in the mind,
Never the creak passed on the stair Make of remoteness familiarity
Closing some other life; an old man And of the unspoken name a friend.
Can prosper in that town, can prosper


key: chroma:document
string_value:
Closing some other life; an old man And of the unspoken name a friend.
Can prosper in that town, can prosper
When his store is seen to close, his wife -------- Generated Poem 4 -------Fades in his house, and the world shouts in vain
That no one comes to hear him Nobody will come to this place. It is a road that leads nowhere.
Where his small bell tolls. The solitude is deep. The mountains are high.
In that small town But they are desolate, and they turn the traveler's face
Everything is known. Towards the North. All the sounds of the world are far away.
The ways of the young men are known, When the wind rises above the trees,
And the ways of the young women The boughs bow to the ground.
As they walk to the Spring; Even the birds that inhabit the tangle of weeds
The rumors are all known, That is the roadside cover, are silent. One listens,
The quarrels, the hurt, the envy But hears no roar of the forest. One is alone.
Of the men and the women. One will be taken.
Till a day comes One will be taken.
That way out of the world There is no utterance, there is no conversation,
That leads to a wood. But one is uneasy all the same....
Young, the young man There is a thin blue mist,


key: chroma:document
string_value:
That leads to a wood. But one is uneasy all the same....
Young, the young man There is a thin blue mist,
Walks, a lover now, A darkness rising like smoke,
Knows the streets as they are and is of them. And within that darkness
A possession of the heart.
Comes to the wood. One will be taken.... It was here, and it will be here againHere, under this sky empty and full of light.
There, in the wood, among trees,
He sees shadows on the way, hears voices,
hears the wind and the rustling of leaves;
Through an open glade
He sees a shape and the shape hears:
It waits as he waits,
Waits as the shadows wait,
As the voices wait;
Shadows on the way, voices in the wind.
Figure F.1: Four uncurated completions from a context suggesting the model compose a poem in the style of Wallace
Stevens with the title ‘Shadows on the Way’.
49
G Details of Task Phrasing and Speciﬁcations
The following ﬁgures illustrate the formatting and phrasing of all the tasks included in the paper. All data comes from
the ground truth datasets in this section, and no samples from GPT-3 are included here.
Context!Article:


key: chroma:document
string_value:
the ground truth datasets in this section, and no samples from GPT-3 are included here.
Context!Article:
Informal conversation is an important part of any business
relationship.Before you start a discussion,however,make sure you understand
which topics are suitable and which are considered taboo in a particular
culture. Latin Americans enjoy sharing information about their local
history, art and customs.You may expect questions about your family,and
be sure to show pictures of your children.You may feel free to ask similar
questions of your Latin American friends.The French think of conversation
as an art form,and they enjoy the value of lively discussions as well as
disagreements. For them,arguments can be interesting and they can cover
pretty much or any topic ---- as long as they occur in are respectful and
intelligent manner.
In the United States,business people like to discuss a wide range of
topics,including opinions about work,family,hobbies,and politics. In
Japan,China,and Korea,however,people are much more private.They do not
share much about their thoughts,feelings,or emotions because they feel
that doing so might take away from the harmonious business relationship
they're trying to build.Middle Easterners are also private about their
personal lives and family matters.It is considered rude,for example,to ask


key: chroma:document
string_value:
they're trying to build.Middle Easterners are also private about their
personal lives and family matters.It is considered rude,for example,to ask
a businessman from Saudi Arabia about his wife or children.
As a general rule,it's best not to talk about politics or religion with
your business friends.This can get you into trouble,even in the United
States,where people hold different religious views.In addition,discussing
one's salary is usually considered unsuitable.Sports is typically a
friendly subject in most parts of the world,although be careful not to
criticize national sport.Instead,be friendly and praise your host's team.
Q: What shouldn't you do when talking about sports with colleagues from
another country?
A: Criticizing the sports of your colleagues' country.
Q: Which is typically a friendly topic in most places according to the
author?
A: Sports.
Q: Why are people from Asia more private in their conversation with others?
A: They don't want to have their good relationship with others harmed by
informal conversation.
Q: The author considers politics and religion .
A:
Correct Answer !taboo
Incorrect Answer !cheerful topics
Incorrect Answer !rude topics
Incorrect Answer !topics that can never be talked about


key: chroma:document
string_value:
A:
Correct Answer !taboo
Incorrect Answer !cheerful topics
Incorrect Answer !rude topics
Incorrect Answer !topics that can never be talked about
Figure G.1: Formatted dataset example for RACE-h. When predicting, we normalize by the unconditional probability
of each answer as described in 2.
50
Context!anli 2: anli 2: The Gold Coast Hotel & Casino is a hotel and casino
located in Paradise, Nevada. This locals' casino is owned and operated
by Boyd Gaming. The Gold Coast is located one mile (  1:6km) west of the
Las Vegas Strip on West Flamingo Road. It is located across the street
from the Palms Casino Resort and the Rio All Suite Hotel and Casino.
Question: The Gold Coast is a budget-friendly casino. True, False, or
Neither?
Correct Answer !Neither
Incorrect Answer !True
Incorrect Answer !False
Figure G.2: Formatted dataset example for ANLI R2
Context!Article:
Mrs. Smith is an unusual teacher. Once she told each student to bring
along a few potatoes in plastic bag. On each potato the students had to
write a name of a person that they hated And the next day, every child
brought some potatoes. Some had two potatoes;some three;some up to five.


key: chroma:document
string_value:
write a name of a person that they hated And the next day, every child
brought some potatoes. Some had two potatoes;some three;some up to five.
Mrs. Smith then told the children to carry the bags everywhere they went,
even to the toilet, for two weeks. As day after day passed, the children
started to complain about the awful smell of the rotten potatoes.
Those children who brought five potatoes began to feel the weight trouble
of the bags. After two weeks, the children were happy to hear that the
game was finally ended. Mrs. Smith asked,"How did you feel while carrying
the potatoes for two weeks?" The children started complaining about the
trouble loudly.
Then Mrs. Smith told them why she asked them to play the game. She
said,"This is exactly the situation when you carry your hatred for somebody
inside your heart. The terrible smell of the hatred will pollute your
heart and you will carry something unnecessary with you all the time. If
you cannot stand the smell of the rotten potatoes for just two weeks, can
you imagine how heavy it would be to have the hatred in your heart for your
lifetime? So throw away any hatred from your heart, and you'll be really
happy."
Q: Which of the following is True according to the passage?
A: If a kid hated four people,he or she had to carry four potatoes.


key: chroma:document
string_value:
happy."
Q: Which of the following is True according to the passage?
A: If a kid hated four people,he or she had to carry four potatoes.
Q: We can learn from the passage that we should .
A: throw away the hatred inside
Q: The children complained about besides the weight trouble.
A: the smell
Q: Mrs.Smith asked her students to write on the potatoes.
A:
Correct Answer !names
Incorrect Answer !numbers
Incorrect Answer !time
Incorrect Answer !places
Figure G.3: Formatted dataset example for RACE-m. When predicting, we normalize by the unconditional probability
of each answer as described in 2.
51
Context!How to apply sealant to wood.
Correct Answer !Using a brush, brush on sealant onto wood until it is fully saturated with
the sealant.
Incorrect Answer !Using a brush, drip on sealant onto wood until it is fully saturated with
the sealant.
Figure G.4: Formatted dataset example for PIQA
Context!My body cast a shadow over the grass because
Correct Answer !the sun was rising.
Incorrect Answer !the grass was cut.
Figure G.5: Formatted dataset example for COPA
Context!(CNN) Yuval Rabin, whose father, Yitzhak Rabin, was assassinated while


key: chroma:document
string_value:
Figure G.5: Formatted dataset example for COPA
Context!(CNN) Yuval Rabin, whose father, Yitzhak Rabin, was assassinated while
serving as Prime Minister of Israel, criticized Donald Trump for appealing
to "Second Amendment people" in a speech and warned that the words that
politicians use can incite violence and undermine democracy. "Trump's
words are an incitement to the type of political violence that touched
me personally," Rabin wrote in USAToday. He said that Trump's appeal to
"Second Amendment people" to stop Hillary Clinton -- comments that were
criticized as a call for violence against Clinton, something Trump denied
-- "were a new level of ugliness in an ugly campaign season."
- The son of a former Israeli Prime Minister who was assassinated wrote an
op ed about the consequence of violent political rhetoric.
- Warns of "parallels" between Israel of the 1990s and the U.S. today.
Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Donald Trump's
aggressive rhetoric.
Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Trump's aggressive
rhetoric.


key: chroma:document
string_value:
Correct Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Trump's aggressive
rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Hillary Clinton's
aggressive rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned U.S.'s aggressive
rhetoric.
Incorrect Answer !- Referencing his father, who was shot and killed by an extremist amid
political tension in Israel in 1995, Rabin condemned Yitzhak Rabin's
aggressive rhetoric.
Figure G.6: Formatted dataset example for ReCoRD. We consider the context above to be a single ”problem” because
this is how the task is presented in the ReCoRD dataset and scored in the ReCoRD evaluation script.
Context!anli 1: anli 1: Fulton James MacGregor MSP is a Scottish politician
who is a Scottish National Party (SNP) Member of Scottish Parliament
for the constituency of Coatbridge and Chryston. MacGregor is currently
Parliamentary Liaison Officer to Shona Robison, Cabinet Secretary for


key: chroma:document
string_value:
for the constituency of Coatbridge and Chryston. MacGregor is currently
Parliamentary Liaison Officer to Shona Robison, Cabinet Secretary for
Health & Sport. He also serves on the Justice and Education & Skills
committees in the Scottish Parliament.
Question: Fulton James MacGregor is a Scottish politican who is a Liaison
officer to Shona Robison who he swears is his best friend. True, False, or
Neither?
Correct Answer !Neither
Incorrect Answer !True
Incorrect Answer !False
Figure G.7: Formatted dataset example for ANLI R1
52
Context!Organisms require energy in order to do what?
Correct Answer !mature and develop.
Incorrect Answer !rest soundly.
Incorrect Answer !absorb light.
Incorrect Answer !take in nutrients.
Figure G.8: Formatted dataset example for OpenBookQA. When predicting, we normalize by the unconditional
probability of each answer as described in 2.
Context!Making a cake: Several cake pops are shown on a display. A woman and girl
are shown making the cake pops in a kitchen. They
Correct Answer !bake them, then frost and decorate.
Incorrect Answer !taste them as they place them on plates.
Incorrect Answer !put the frosting on the cake as they pan it.


key: chroma:document
string_value:
Incorrect Answer !taste them as they place them on plates.
Incorrect Answer !put the frosting on the cake as they pan it.
Incorrect Answer !come out and begin decorating the cake as well.
Figure G.9: Formatted dataset example for HellaSwag
Context!anli 3: anli 3: We shut the loophole which has American workers actually
subsidizing the loss of their own job. They just passed an expansion of
that loophole in the last few days: $43 billion of giveaways, including
favors to the oil and gas industry and the people importing ceiling fans
from China.
Question: The loophole is now gone True, False, or Neither?
Correct Answer !False
Incorrect Answer !True
Incorrect Answer !Neither
Figure G.10: Formatted dataset example for ANLI R3
Context!Question: George wants to warm his hands quickly by rubbing them. Which
skin surface will produce the most heat?
Answer:
Correct Answer !dry palms
Incorrect Answer !wet palms
Incorrect Answer !palms covered with oil
Incorrect Answer !palms covered with lotion
Figure G.11: Formatted dataset example for ARC (Challenge). When predicting, we normalize by the unconditional
probability of each answer as described in 2.
Context!lull is to trust as


key: chroma:document
string_value:
probability of each answer as described in 2.
Context!lull is to trust as
Correct Answer !cajole is to compliance
Incorrect Answer !balk is to fortitude
Incorrect Answer !betray is to loyalty
Incorrect Answer !hinder is to destination
Incorrect Answer !soothe is to passion
Figure G.12: Formatted dataset example for SAT Analogies
Correct Context !Grace was happy to trade me her sweater for my jacket. She thinks the
sweater
Incorrect Context !Grace was happy to trade me her sweater for my jacket. She thinks the
jacket
Target Completion !looks dowdy on her.
Figure G.13: Formatted dataset example for Winograd. The ‘partial’ evaluation method we use compares the probability
of the completion given a correct and incorrect context.
53
Correct Context !Johnny likes fruits more than vegetables in his new keto diet because the
fruits
Incorrect Context !Johnny likes fruits more than vegetables in his new keto diet because the
vegetables
Target Completion !are saccharine.
Figure G.14: Formatted dataset example for Winogrande. The ‘partial’ evaluation method we use compares the
probability of the completion given a correct and incorrect context.
Context!READING COMPREHENSION ANSWER KEY


key: chroma:document
string_value:
probability of the completion given a correct and incorrect context.
Context!READING COMPREHENSION ANSWER KEY
While this process moved along, diplomacy continued its rounds. Direct
pressure on the Taliban had proved unsuccessful. As one NSC staff note
put it, "Under the Taliban, Afghanistan is not so much a state sponsor
of terrorism as it is a state sponsored by terrorists." In early 2000,
the United States began a high-level effort to persuade Pakistan to use
its influence over the Taliban. In January 2000, Assistant Secretary
of State Karl Inderfurth and the State Department's counterterrorism
coordinator, Michael Sheehan, met with General Musharraf in Islamabad,
dangling before him the possibility of a presidential visit in March as a
reward for Pakistani cooperation. Such a visit was coveted by Musharraf,
partly as a sign of his government's legitimacy. He told the two envoys
that he would meet with Mullah Omar and press him on Bin Laden. They
left, however, reporting to Washington that Pakistan was unlikely in fact
to do anything," given what it sees as the benefits of Taliban control
of Afghanistan." President Clinton was scheduled to travel to India.
The State Department felt that he should not visit India without also
visiting Pakistan. The Secret Service and the CIA, however, warned in
the strongest terms that visiting Pakistan would risk the President's


key: chroma:document
string_value:
The State Department felt that he should not visit India without also
visiting Pakistan. The Secret Service and the CIA, however, warned in
the strongest terms that visiting Pakistan would risk the President's
life. Counterterrorism officials also argued that Pakistan had not done
enough to merit a presidential visit. But President Clinton insisted
on including Pakistan in the itinerary for his trip to South Asia. His
one-day stopover on March 25, 2000, was the first time a U.S. president
had been there since 1969. At his meeting with Musharraf and others,
President Clinton concentrated on tensions between Pakistan and India
and the dangers of nuclear proliferation, but also discussed Bin Laden.
President Clinton told us that when he pulled Musharraf aside for a brief,
one-on-one meeting, he pleaded with the general for help regarding Bin
Laden." I offered him the moon when I went to see him, in terms of better
relations with the United States, if he'd help us get Bin Laden and deal
with another issue or two." The U.S. effort continued.
Who did The State Department feel should visit both India and Pakistan?
Correct Answer !- [False] Bin Laden
Incorrect Answer !- [True] Bin Laden
Figure G.15: Formatted dataset example for MultiRC. There are three levels within MultiRC: (1) the passage, (2) the


key: chroma:document
string_value:
Figure G.15: Formatted dataset example for MultiRC. There are three levels within MultiRC: (1) the passage, (2) the
questions, and (3) the answers. During evaluation, accuracy is determined at the per-question level, with a question
being considered correct if and only if all the answers within the question are labeled correctly. For this reason, we use
Kto refer to the number of questions shown within the context.
Context!Question: Which factor will most likely cause a person to develop a fever?
Answer:
Correct Answer !a bacterial population in the bloodstream
Incorrect Answer !a leg muscle relaxing after exercise
Incorrect Answer !several viral particles on the skin
Incorrect Answer !carbohydrates being digested in the stomach
Figure G.16: Formatted dataset example for ARC (Easy). When predicting, we normalize by the unconditional
probability of each answer as described in 2.
54
Context!Bob went to the gas station to fill up his car. His tank was completely
empty and so was his wallet. The cashier offered to pay for his gas if he
came back later to pay. Bob felt grateful as he drove home.
Correct Answer !Bob believed that there were good people in the world.
Incorrect Answer !Bob contemplated how unfriendly the world was.


key: chroma:document
string_value:
Correct Answer !Bob believed that there were good people in the world.
Incorrect Answer !Bob contemplated how unfriendly the world was.
Figure G.17: Formatted dataset example for StoryCloze
Context!Helsinki is the capital and largest city of Finland. It is in the region
of Uusimaa, in southern Finland, on the shore of the Gulf of Finland.
Helsinki has a population of , an urban population of , and a metropolitan
population of over 1.4 million, making it the most populous municipality
and urban area in Finland. Helsinki is some north of Tallinn, Estonia,
east of Stockholm, Sweden, and west of Saint Petersburg, Russia. Helsinki
has close historical connections with these three cities.
The Helsinki metropolitan area includes the urban core of Helsinki, Espoo,
Vantaa, Kauniainen, and surrounding commuter towns. It is the world's
northernmost metro area of over one million people, and the city is the
northernmost capital of an EU member state. The Helsinki metropolitan
area is the third largest metropolitan area in the Nordic countries
after Stockholm and Copenhagen, and the City of Helsinki is the third
largest after Stockholm and Oslo. Helsinki is Finland's major political,
educational, financial, cultural, and research center as well as one of
northern Europe's major cities. Approximately 75% of foreign companies


key: chroma:document
string_value:
educational, financial, cultural, and research center as well as one of
northern Europe's major cities. Approximately 75% of foreign companies
that operate in Finland have settled in the Helsinki region. The nearby
municipality of Vantaa is the location of Helsinki Airport, with frequent
service to various destinations in Europe and Asia.
Q: what is the most populous municipality in Finland?
A: Helsinki
Q: how many people live there?
A: 1.4 million in the metropolitan area
Q: what percent of the foreign companies that operate in Finland are in
Helsinki?
A: 75%
Q: what towns are a part of the metropolitan area?
A:
Target Completion !Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns
Figure G.18: Formatted dataset example for CoQA
Context!Please unscramble the letters into a word, and write that word:
asinoc =
Target Completion !casino
Figure G.19: Formatted dataset example for Cycled Letters
55
Context!Passage: Saint Jean de Br ebeuf was a French Jesuit missionary who
travelled to New France in 1625. There he worked primarily with the Huron
for the rest of his life, except for a few years in France from 1629 to


key: chroma:document
string_value:
travelled to New France in 1625. There he worked primarily with the Huron
for the rest of his life, except for a few years in France from 1629 to
1633. He learned their language and culture, writing extensively about
each to aid other missionaries. In 1649, Br ebeuf and another missionary
were captured when an Iroquois raid took over a Huron village . Together
with Huron captives, the missionaries were ritually tortured and killed
on March 16, 1649. Br ebeuf was beatified in 1925 and among eight Jesuit
missionaries canonized as saints in the Roman Catholic Church in 1930.
Question: How many years did Saint Jean de Br ebeuf stay in New France
before he went back to France for a few years?
Answer:
Target Completion !4
Figure G.20: Formatted dataset example for DROP
Context!Fill in blank:
She held the torch in front of her.
She caught her breath.
"Chris? There's a step."
"What?"
"A step. Cut in the rock. About fifty feet ahead." She moved faster.
They both moved faster. "In fact," she said, raising the torch higher,
"there's more than a . ->
Target Completion !step
Figure G.21: Formatted dataset example for LAMBADA


key: chroma:document
string_value:
"there's more than a . ->
Target Completion !step
Figure G.21: Formatted dataset example for LAMBADA
Context!Please unscramble the letters into a word, and write that word:
skicts =
Target Completion !sticks
Figure G.22: Formatted dataset example for Anagrams 1 (A1)
Context!Please unscramble the letters into a word, and write that word:
volwskagen =
Target Completion !volkswagen
Figure G.23: Formatted dataset example for Anagrams 2
Context!Q: Who played tess on touched by an angel?
A:
Target Completion !Delloreese Patricia Early (July 6, 1931 { November 19, 2017), known
professionally as Della Reese
Figure G.24: Formatted dataset example for Natural Questions
56
Context!TITLE: William Perry (American football) - Professional career
PARAGRAPH: In 1985, he was selected in the first round of the 1985 NFL
Draft by the Chicago Bears; he had been hand-picked by coach Mike Ditka.
However, defensive coordinator Buddy Ryan, who had a highly acrimonious
relationship with Ditka, called Perry a "wasted draft-pick". Perry
soon became a pawn in the political power struggle between Ditka and


key: chroma:document
string_value:
relationship with Ditka, called Perry a "wasted draft-pick". Perry
soon became a pawn in the political power struggle between Ditka and
Ryan. Perry's "Refrigerator" nickname followed him into the NFL and he
quickly became a favorite of the Chicago Bears fans. Teammates called
him "Biscuit," as in "one biscuit shy of 350 pounds." While Ryan refused
to play Perry, Ditka decided to use Perry as a fullback when the team was
near the opponents' goal line or in fourth and short situations, either
as a ball carrier or a lead blocker for star running back Walter Payton.
Ditka stated the inspiration for using Perry as a fullback came to him
during five-yard sprint exercises. During his rookie season, Perry
rushed for two touchdowns and caught a pass for one. Perry even had
the opportunity to run the ball during Super Bowl XX, as a nod to his
popularity and contributions to the team's success. The first time he
got the ball, he was tackled for a one-yard loss while attempting to throw
his first NFL pass on a halfback option play. The second time he got the
ball, he scored a touchdown (running over Patriots linebacker Larry McGrew
in the process). About halfway through his rookie season, Ryan finally
began to play Perry, who soon proved that he was a capable defensive


key: chroma:document
string_value:
in the process). About halfway through his rookie season, Ryan finally
began to play Perry, who soon proved that he was a capable defensive
lineman. His Super Bowl ring size is the largest of any professional
football player in the history of the event. His ring size is 25, while
the ring size for the average adult male is between 10 and 12. Perry went
on to play for ten years in the NFL, retiring after the 1994 season. In
his ten years as a pro, he regularly struggled with his weight, which
hampered his performance at times. He played in 138 games, recording
29.5 sacks and five fumble recoveries, which he returned for a total of
71 yards. In his offensive career he ran five yards for two touchdowns,
and had one reception for another touchdown. Perry later attempted a
comeback, playing an unremarkable 1996 season with the London Monarchs of
the World League of American Football (later NFL Europa).
Q: what team did he play for?
A:
Target Completion !the Chicago Bears
Figure G.25: Formatted dataset example for QuAC
Context!Please unscramble the letters into a word, and write that word:
r e!c.i p r o.c a/l =
Target Completion !reciprocal
Figure G.26: Formatted dataset example for Symbol Insertion


key: chroma:document
string_value:
r e!c.i p r o.c a/l =
Target Completion !reciprocal
Figure G.26: Formatted dataset example for Symbol Insertion
Context!Please unscramble the letters into a word, and write that word:
taefed =
Target Completion !defeat
Figure G.27: Formatted dataset example for Reversed Words
57
Context!Title: The Blitz
Background: From the German point of view, March 1941 saw an improvement.
The Luftwaffe flew 4,000 sorties that month, including 12 major and
three heavy attacks. The electronic war intensified but the Luftwaffe
flew major inland missions only on moonlit nights. Ports were easier to
find and made better targets. To confuse the British, radio silence was
observed until the bombs fell. X- and Y-Ger at beams were placed over
false targets and switched only at the last minute. Rapid frequency
changes were introduced for X-Ger at, whose wider band of frequencies and
greater tactical flexibility ensured it remained effective at a time when
British selective jamming was degrading the effectiveness of Y-Ger at.
Q: How many sorties were flown in March 1941?
A: 4,000
Q: When did the Luftwaffe fly inland missions?
A:
Target Completion !only on moonlit nights


key: chroma:document
string_value:
A: 4,000
Q: When did the Luftwaffe fly inland missions?
A:
Target Completion !only on moonlit nights
Figure G.28: Formatted dataset example for SQuADv2
Context!Normal force -- In a simple case such as an object resting upon a table,
the normal force on the object is equal but in opposite direction to the
gravitational force applied on the object (or the weight of the object),
that is, N = m g ( ndisplaystyle N=mg), where m is mass, and g is the
gravitational field strength (about 9.81 m/s on Earth). The normal force
here represents the force applied by the table against the object that
prevents it from sinking through the table and requires that the table is
sturdy enough to deliver this normal force without breaking. However, it
is easy to assume that the normal force and weight are action-reaction
force pairs (a common mistake). In this case, the normal force and
weight need to be equal in magnitude to explain why there is no upward
acceleration of the object. For example, a ball that bounces upwards
accelerates upwards because the normal force acting on the ball is larger
in magnitude than the weight of the ball.
question: is the normal force equal to the force of gravity?
answer:
Target Completion !yes


key: chroma:document
string_value:
in magnitude than the weight of the ball.
question: is the normal force equal to the force of gravity?
answer:
Target Completion !yes
Figure G.29: Formatted dataset example for BoolQ
Context!The trend toward lower rents may seem surprising given that some
communities in New York are bemoaning the loss of favorite local
businesses to high rents. But, despite the recent softening, for many
of these retailers there's still been too big a jump from the rental rates
of the late 1970s, when their leases were signed. Certainly, the recent
drop in prices doesn't mean Manhattan comes cheap.
question: Manhattan comes cheap. true, false, or neither?
answer:
Target Completion !false
Figure G.30: Formatted dataset example for CB
58
Context!The bet, which won him dinner for four, was regarding the existence and
mass of the top quark, an elementary particle discovered in 1995.
question: The Top Quark is the last of six flavors of quarks predicted by
the standard model theory of particle physics. True or False?
answer:
Target Completion !False
Figure G.31: Formatted dataset example for RTE
Context!An outfitter provided everything needed for the safari.
Before his first walking holiday, he went to a specialist outfitter to buy
some boots.


key: chroma:document
string_value:
Context!An outfitter provided everything needed for the safari.
Before his first walking holiday, he went to a specialist outfitter to buy
some boots.
question: Is the word `outfitter' used in the same way in the two
sentences above?
answer:
Target Completion !no
Figure G.32: Formatted dataset example for WiC
Context!Final Exam with Answer Key
Instructions: Please carefully read the following passages. For each
passage, you must identify which noun the pronoun marked in *bold* refers
to.
=====
Passage: Mr. Moncrieff visited Chester's luxurious New York apartment,
thinking that it belonged to his son Edward. The result was that Mr.
Moncrieff has decided to cancel Edward's allowance on the ground that
he no longer requires *his* financial support.
Question: In the passage above, what does the pronoun "*his*" refer to?
Answer:
Target Completion !mr. moncrieff
Figure G.33: Formatted dataset example for WSC
Context!Q: `Nude Descending A Staircase' is perhaps the most famous painting by
which 20th century artist?
A:
Target Completion !MARCEL DUCHAMP
Target Completion !r mutt
Target Completion !duchamp


key: chroma:document
string_value:
which 20th century artist?
A:
Target Completion !MARCEL DUCHAMP
Target Completion !r mutt
Target Completion !duchamp
Target Completion !marcel duchamp
Target Completion !R.Mutt
Target Completion !Marcel duChamp
Target Completion !Henri-Robert-Marcel Duchamp
Target Completion !Marcel du Champ
Target Completion !henri robert marcel duchamp
Target Completion !Duchampian
Target Completion !Duchamp
Target Completion !duchampian
Target Completion !marcel du champ
Target Completion !Marcel Duchamp
Target Completion !MARCEL DUCHAMP
Figure G.34: Formatted dataset example for TriviaQA. TriviaQA allows for multiple valid completions.
59
Context!Q: What school did burne hogarth establish?
A:
Target Completion !School of Visual Arts
Figure G.35: Formatted dataset example for WebQA
Context!Keinesfalls d urfen diese f ur den kommerziellen Gebrauch verwendet werden.
=
Target Completion !In no case may they be used for commercial purposes.


key: chroma:document
string_value:
=
Target Completion !In no case may they be used for commercial purposes.
Figure G.36: Formatted dataset example for De !En. This is the format for one- and few-shot learning, for this and
other langauge tasks, the format for zero-shot learning is “Q: What is the flanguagegtranslation offsentencegA:
ftranslationg.”
Context!In no case may they be used for commercial purposes. =
Target Completion !Keinesfalls d urfen diese f ur den kommerziellen Gebrauch verwendet werden.
Figure G.37: Formatted dataset example for En !De
Context!Analysis of instar distributions of larval I. verticalis collected from
a series of ponds also indicated that males were in more advanced instars
than females. =
Target Completion !L'analyse de la distribution de fr equence des stades larvaires d'I.
verticalis dans une s erie d' etangs a  egalement d emontr e que les larves
m^ ales  etaient  a des stades plus avanc es que les larves femelles.
Figure G.38: Formatted dataset example for En !Fr


key: chroma:document
string_value:
Figure G.38: Formatted dataset example for En !Fr
Context!L'analyse de la distribution de fr equence des stades larvaires d'I.
verticalis dans une s erie d' etangs a  egalement d emontr e que les larves
m^ ales  etaient  a des stades plus avanc es que les larves femelles. =
Target Completion !Analysis of instar distributions of larval I. verticalis collected from
a series of ponds also indicated that males were in more advanced instars
than females.
Figure G.39: Formatted dataset example for Fr !En
Context!The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession
to the European Union, despite Turkey's continuing refusal to recognise
Cyprus and despite the fact that the democratic reforms are at a
standstill. =
Target Completion !Adev arul este c a v a dorit i, cu orice pret   si ^ mpotriva dorint ei
europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea


key: chroma:document
string_value:
europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea
European a, ^ n ciuda refuzului continuu al Turciei de a recunoa ste Ciprul
 si ^ n ciuda faptului c a reformele democratice au ajuns ^ ntr-un punct mort.
Figure G.40: Formatted dataset example for En !Ro
60
Context!Adev arul este c a v a dorit i, cu orice pret   si ^ mpotriva dorint ei
europenilor, s a continuat i negocierile de aderare a Turciei la Uniunea
European a, ^ n ciuda refuzului continuu al Turciei de a recunoa ste Ciprul
 si ^ n ciuda faptului c a reformele democratice au ajuns ^ ntr-un punct mort.
=
Target Completion !The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession


key: chroma:document
string_value:
=
Target Completion !The truth is that you want, at any price, and against the wishes of the
peoples of Europe, to continue the negotiations for Turkey's accession
to the European Union, despite Turkey's continuing refusal to recognise
Cyprus and despite the fact that the democratic reforms are at a
standstill.
Figure G.41: Formatted dataset example for Ro !En
Context!Q: What is (2 * 4) * 6?
A:
Target Completion !48
Figure G.42: Formatted dataset example for Arithmetic 1DC
Context!Q: What is 17 minus 14?
A:
Target Completion !3
Figure G.43: Formatted dataset example for Arithmetic 2DContext!Q: What is 98 plus 45?
A:
Target Completion !143
Figure G.44: Formatted dataset example for Arithmetic 2D+
Context!Q: What is 95 times 45?
A:
Target Completion !4275
Figure G.45: Formatted dataset example for Arithmetic 2Dx
Context!Q: What is 509 minus 488?
A:
Target Completion !21
Figure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?
A:
Target Completion !1053


key: chroma:document
string_value:
Figure G.46: Formatted dataset example for Arithmetic 3DContext!Q: What is 556 plus 497?
A:
Target Completion !1053
Figure G.47: Formatted dataset example for Arithmetic 3D+
Context!Q: What is 6209 minus 3365?
A:
Target Completion !2844
Figure G.48: Formatted dataset example for Arithmetic 4D61
Context!Q: What is 9923 plus 617?
A:
Target Completion !10540
Figure G.49: Formatted dataset example for Arithmetic 4D+
Context!Q: What is 40649 minus 78746?
A:
Target Completion !-38097
Figure G.50: Formatted dataset example for Arithmetic 5D 
Context!Q: What is 65360 plus 16204?
A:
Target Completion !81564
Figure G.51: Formatted dataset example for Arithmetic 5D+
62
H Results on All Tasks for All Model Sizes
Zero-Shot One-Shot Few-Shot
Name Metric SplitFine-tune
SOTA K Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B175B


key: chroma:document
string_value:
(test server)
HellaSwag acc dev 85.6 20 33.7 43.6 51.0 54.7 62.8 67.4 70.9 78.9 33.0 42.9 50.5 53.5 61.9 66.5 70.0 78.1 33.5 43.1 51.3 54.9 62.9 67.3 71.3 79.3
LAMBADA acc test 68.0 15 42.7 54.3 60.4 63.6 67.1 70.3 72.5 76.2 22.0 47.1 52.6 58.3 61.1 65.4 69.0 72.5 22.0 40.4 63.2 57.0 78.1 79.1 81.3 86.4
LAMBADA ppl test 8.63 15 18.6 9.09 6.53 5.44 4.60 4.00 3.56 3.00 165.0 11.6 8.29 6.46 5.53 4.61 4.06 3.35 165.0 27.6 6.63 7.45 2.89 2.56 2.56 1.92


key: chroma:document
string_value:
StoryCloze acc test 91.8 70 63.3 68.5 72.4 73.4 77.2 77.7 79.5 83.2 62.3 68.7 72.3 74.2 77.3 78.7 79.7 84.7 62.3 70.2 73.9 76.1 80.2 81.2 83.0 87.7
NQs acc test 44.5 64 0.64 1.75 2.71 4.40 6.01 5.79 7.84 14.6 1.19 3.07 4.79 5.43 8.73 9.78 13.7 23.0 1.72 4.46 7.89 9.72 13.2 17.0 21.0 29.9
TriviaQA acc dev 68.0 64 4.15 7.61 14.0 19.7 31.3 38.7 41.8 64.3 4.19 12.9 20.5 26.5 35.9 44.4 51.3 68.0 6.96 16.3 26.5 32.1 42.3 51.6 57.5 71.2 71.2


key: chroma:document
string_value:
WebQs acc test 45.5 64 1.77 3.20 4.33 4.63 7.92 7.73 8.22 14.4 2.56 6.20 8.51 9.15 14.5 15.1 19.0 25.3 5.46 12.6 15.9 19.6 24.8 27.7 33.5 41.5
Ro!En 16 BLEU-mb test 39.9 64 2.08 2.71 3.09 3.15 16.3 8.34 20.2 19.9 0.55 15.4 23.0 26.3 30.6 33.2 35.6 38.6 1.25 20.7 25.8 29.2 33.1 34.8 37.0 39.5
Ro!En 16 BLEU-sb test 64 2.39 3.08 3.49 3.56 16.8 8.75 20.8 20.9 0.65 15.9 23.6 26.8 31.3 34.2 36.7 40.0 1.40 21.3 26.6 30.1 34.3 36.2 38.4 41.3


key: chroma:document
string_value:
En!Ro 16 BLEU-mb test 38.5 64 2.14 2.65 2.53 2.50 3.46 4.24 5.32 14.1 0.35 3.30 7.89 8.72 13.2 15.1 17.3 20.6 1.25 5.90 9.33 10.7 14.3 16.3 18.0 21.0
En!Ro 16 BLEU-sb test 64 2.61 3.11 3.07 3.09 4.26 5.31 6.43 18.0 0.55 3.90 9.15 10.3 15.7 18.2 20.8 24.9 1.64 7.40 10.9 12.9 17.2 19.6 21.8 25.8
Fr!En 14 BLEU-mb test 35.0 64 1.81 2.53 3.47 3.13 20.6 15.1 21.8 21.2 1.28 15.9 23.7 26.3 29.0 30.5 30.2 33.7 4.98 25.5 28.5 31.1 33.7 34.9 36.6 39.2


key: chroma:document
string_value:
Fr!En 14 BLEU-sb test 64 2.29 2.99 3.90 3.60 21.2 15.5 22.4 21.9 1.50 16.3 24.4 27.0 30.0 31.6 31.4 35.6 5.30 26.2 29.5 32.2 35.1 36.4 38.3 41.4
En!Fr 14 BLEU-mb test 45.6 64 1.74 2.16 2.73 2.15 15.1 8.82 12.0 25.2 0.49 8.00 14.8 15.9 20.3 23.3 24.9 28.3 4.08 14.5 19.3 21.5 24.9 27.3 29.5 32.6
En!Fr 14 BLEU-sb test 45.9 64 2.44 2.75 3.54 2.82 19.3 11.4 15.3 31.3 0.81 10.0 18.2 19.3 24.7 28.3 30.1 34.1 5.31 18.0 23.6 26.1 30.3 33.3 35.5 39.9


key: chroma:document
string_value:
De!En 16 BLEU-mb test 40.2 64 2.06 2.87 3.41 3.63 21.5 17.3 23.0 27.2 0.83 16.2 22.5 24.7 28.2 30.7 33.0 30.4 3.25 22.7 26.2 29.2 32.7 34.8 37.3 40.6
De!En 16 BLEU-sb test 64 2.39 3.27 3.85 4.04 22.5 18.2 24.4 28.6 0.93 17.1 23.4 25.8 29.2 31.9 34.5 32.1 3.60 23.8 27.5 30.5 34.1 36.5 39.1 43.0
En!De 16 BLEU-mb test 41.2 64 1.70 2.27 2.31 2.43 12.9 8.66 10.4 24.6 0.50 7.00 12.9 13.1 18.3 20.9 22.5 26.2 3.42 12.3 15.4 17.1 20.9 23.0 26.6 29.7


key: chroma:document
string_value:
En!De 16 BLEU-sb test 41.2 64 2.09 2.65 2.75 2.92 13.7 9.36 11.0 25.3 0.54 7.40 13.4 13.4 18.8 21.7 23.3 27.3 3.78 12.9 16.1 17.7 21.7 24.1 27.7 30.9
Winograd acc test 93.8 7 66.3 72.9 74.7 76.9 82.4 85.7 87.9 88.3 63.4 68.5 72.9 76.9 82.4 84.6 86.1 89.7 63.4 67.4 73.6 76.9 84.3 85.4 82.4 88.6
Winogrande acc dev 84.6 50 52.0 52.1 57.4 58.7 62.3 64.5 67.9 70.2 51.3 53.0 58.3 59.1 61.7 65.8 66.9 73.2 51.3 52.6 57.5 59.1 62.6 67.4 70.0 77.7


key: chroma:document
string_value:
PIQA acc dev 77.1 50 64.6 70.2 72.9 75.1 75.6 78.0 78.5 81.0 64.3 69.3 71.8 74.4 74.3 76.3 77.8 80.5 64.3 69.4 72.0 74.3 75.4 77.8 79.9 82.3 82.8
ARC (Challenge) acc test 78.5 50 26.6 29.5 31.8 35.5 38.0 41.4 43.7 51.4 25.5 30.2 31.6 36.4 38.4 41.5 43.1 53.2 25.5 28.4 32.3 36.7 39.5 43.7 44.8 51.5
ARC (Easy) acc test 92.0 50 43.6 46.5 53.0 53.8 58.2 60.2 63.8 68.8 42.7 48.2 54.6 55.9 60.3 62.6 66.8 71.2 42.7 51.0 58.1 59.1 62.1 65.8 69.1 70.1


key: chroma:document
string_value:
OpenBookQA acc test 87.2 100 35.6 43.2 45.2 46.8 53.0 50.4 55.6 57.6 37.0 39.8 46.2 46.4 53.4 53.0 55.8 58.8 37.0 43.6 48.0 50.6 55.6 55.2 60.8 65.4
Quac f1 dev 74.4 5 21.2 26.8 31.0 30.1 34.7 36.1 38.4 41.5 21.1 26.9 31.9 32.3 37.4 39.0 40.6 43.4 21.6 27.6 32.9 34.2 38.2 39.9 40.9 44.3
RACE-h acc test 90.0 10 35.2 37.9 40.1 40.9 42.4 44.1 44.6 45.5 34.3 37.7 40.0 42.0 43.8 44.3 44.6 45.9 34.3 37.0 40.4 41.4 42.3 44.7 45.1 46.8


key: chroma:document
string_value:
RACE-m acc test 93.1 10 42.1 47.2 52.1 52.3 54.7 54.4 56.7 58.4 42.3 47.3 51.7 55.2 56.1 54.7 56.9 57.4 42.3 47.0 52.7 53.0 55.6 55.4 58.1 58.1
SQuADv2 em dev 90.7 16 22.6 32.8 33.9 43.1 43.6 45.4 49.0 52.6 25.1 37.5 37.9 47.9 47.9 51.1 56.0 60.1 27.5 40.5 39.2 53.5 50.0 56.6 62.6 64.9
SQuADv2 f1 dev 93.0 16 28.3 40.2 41.4 50.3 51.0 52.7 56.3 59.5 30.1 43.6 44.1 54.0 54.1 57.1 61.8 65.4 32.1 45.5 44.9 58.7 55.9 62.1 67.7 69.8


key: chroma:document
string_value:
CoQA f1 dev 90.7 5 34.5 55.0 61.8 65.3 71.1 72.8 76.3 81.5 30.6 52.1 61.6 66.1 71.8 75.1 77.9 84.0 31.1 52.0 62.7 66.8 73.2 77.3 79.9 85.0
DROP f1 dev 89.1 20 9.40 13.6 14.4 16.4 19.7 17.0 24.0 23.6 11.7 18.1 20.9 23.0 26.4 27.3 29.2 34.3 12.9 18.7 24.0 25.6 29.7 29.7 32.3 36.5
BoolQ acc dev 91.0 32 49.7 60.3 58.9 62.4 67.1 65.4 66.2 60.5 52.6 61.7 60.4 63.7 68.4 68.7 69.0 76.7 43.1 60.6 62.0 64.1 70.3 70.0 70.2 77.5 76.4


key: chroma:document
string_value:
CB acc dev 96.9 32 0.00 32.1 8.93 19.6 19.6 28.6 19.6 46.4 55.4 53.6 53.6 48.2 57.1 33.9 55.4 64.3 42.9 58.9 53.6 69.6 67.9 60.7 66.1 82.1 75.6
CB f1 dev 93.9 32 0.00 29.3 11.4 17.4 22.4 25.1 20.3 42.8 60.1 39.8 45.6 37.5 45.7 28.5 44.6 52.5 26.1 40.4 32.6 48.3 45.7 44.6 46.0 57.2 52.0
Copa acc dev 94.8 32 66.0 68.0 73.0 77.0 76.0 80.0 84.0 91.0 62.0 64.0 66.0 74.0 76.0 82.0 86.0 87.0 67.0 64.0 72.0 77.0 83.0 83.0 86.0 92.0 92.0


key: chroma:document
string_value:
RTE acc dev 92.5 32 47.7 49.8 48.4 56.0 46.6 55.2 62.8 63.5 53.1 47.3 49.5 49.5 54.9 54.9 56.3 70.4 52.3 48.4 46.9 50.9 56.3 49.5 60.6 72.9 69.0
WiC acc dev 76.1 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.0 50.3 50.3 49.2 49.4 50.3 50.0 48.6 49.8 55.0 53.0 53.0 51.6 53.1 51.1 55.3 49.4
WSC acc dev 93.8 32 59.6 56.7 65.4 61.5 66.3 60.6 64.4 65.4 58.7 58.7 60.6 62.5 66.3 60.6 66.3 69.2 58.7 60.6 54.8 49.0 62.5 67.3 75.0 75.0 80.1


key: chroma:document
string_value:
MultiRC acc dev 62.3 32 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 6.09 11.8 16.8 20.8 24.7 23.8 25.0 32.5 30.5
MultiRC f1a dev 88.2 32 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 45.0 55.9 64.2 65.4 69.5 66.4 69.3 74.8 75.4
ReCoRD acc dev 92.5 32 70.8 78.5 82.1 84.1 86.2 88.6 89.0 90.2 69.8 77.0 80.7 83.0 85.9 88.0 88.8 90.2 69.8 77.2 81.3 83.1 86.6 87.9 88.9 89.0 90.2


key: chroma:document
string_value:
ReCoRD f1 dev 93.3 32 71.9 79.2 82.8 85.2 87.3 89.5 90.4 91.0 70.7 77.8 81.6 83.9 86.8 88.8 89.7 91.2 70.7 77.9 82.1 84.0 87.5 88.8 89.8 90.1 91.1
SuperGLUE average dev 89.0 40.6 47.4 46.8 49.6 50.1 52.3 54.4 58.2 54.4 55.1 56.7 57.8 61.2 59.7 64.3 68.9 50.2 56.2 56.8 60.0 64.3 63.6 66.9 73.2 71.8
ANLI R1 acc test 73.8 50 33.4 34.2 33.4 33.4 34.2 32.3 33.2 34.6 32.1 31.6 31.9 34.6 30.6 31.6 32.7 32.0 32.1 32.5 30.9 32.5 33.5 33.1 33.3 36.8


key: chroma:document
string_value:
ANLI R2 acc test 50.7 50 33.2 31.9 33.3 33.3 33.8 33.5 33.5 35.4 35.7 33.7 33.2 32.7 32.7 33.9 33.9 33.9 35.7 33.8 32.1 31.4 32.6 33.3 32.6 34.0
ANLI R3 acc test 48.3 50 33.6 34.0 33.8 33.4 35.3 34.8 34.4 34.5 35.0 32.6 33.0 33.9 34.1 33.1 32.5 35.1 35.0 34.4 35.1 36.0 32.7 33.9 34.5 40.2
2D+ acc n/a 50 0.70 0.65 0.70 0.85 1.10 2.54 15.4 76.9 2.00 0.55 3.15 4.00 12.1 19.6 73.0 99.6 2.00 4.10 3.50 4.50 8.90 11.9 55.5 100.0


key: chroma:document
string_value:
2D- acc n/a 50 1.25 1.25 1.25 1.25 1.60 7.60 12.6 58.0 1.15 0.95 1.45 1.95 3.85 11.5 44.6 86.4 1.15 1.45 2.25 2.70 7.35 13.6 52.4 98.9
3D+ acc n/a 50 0.10 0.10 0.05 0.10 0.10 0.25 1.40 34.2 0.15 0.00 0.10 0.30 0.45 0.95 15.4 65.5 0.15 0.45 0.30 0.55 0.75 0.90 8.40 80.4
3D- acc n/a 50 0.05 0.05 0.05 0.05 0.05 0.45 1.35 48.3 0.05 0.15 0.25 0.30 0.55 1.60 6.15 78.7 0.05 0.10 0.15 0.35 0.65 1.05 9.20 94.2


key: chroma:document
string_value:
4D+ acc n/a 50 0.05 0.05 0.00 0.00 0.05 0.05 0.15 4.00 0.00 0.00 0.10 0.00 0.00 0.10 0.80 14.0 0.00 0.05 0.05 0.00 0.15 0.15 0.40 25.5
4D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.10 7.50 0.00 0.00 0.00 0.00 0.05 0.00 0.50 14.0 0.00 0.05 0.00 0.00 0.10 0.05 0.40 26.8
5D+ acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.65 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.45 0.00 0.00 0.00 0.00 0.00 0.00 0.05 9.30


key: chroma:document
string_value:
5D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.90
2Dx acc n/a 50 2.20 2.25 2.65 2.10 2.55 5.80 6.15 19.8 1.35 2.35 3.35 2.35 4.75 9.15 11.0 27.4 1.35 2.90 2.70 2.85 4.25 6.10 7.05 29.2
1DC acc n/a 50 1.25 2.95 2.75 0.05 0.30 2.35 0.75 9.75 1.90 2.80 2.85 3.65 6.45 9.15 8.20 14.3 1.70 2.15 3.90 5.75 6.20 7.60 9.95 21.3


key: chroma:document
string_value:
Cycled Letters acc n/a 100 0.62 0.71 2.85 0.00 0.63 1.35 2.58 3.66 1.67 4.36 5.68 6.46 6.25 9.41 15.1 21.7 4.63 9.27 10.7 14.5 16.7 21.9 27.7 37.9
Anagrams 1 acc n/a 100 0.10 0.14 0.40 0.00 0.27 0.69 1.16 2.28 0.21 0.61 1.12 1.27 1.60 2.72 3.72 8.62 0.50 1.27 2.13 3.05 3.81 5.49 8.38 15.1
Anagrams 2 acc n/a 100 0.81 1.21 2.69 0.01 1.71 3.75 4.53 8.91 1.19 2.62 4.70 4.77 6.97 10.2 14.6 25.9 1.94 4.80 7.59 9.87 12.6 18.9 25.6 39.7


key: chroma:document
string_value:
Symbol Insertion acc n/a 100 0.00 0.00 0.10 0.00 0.05 0.42 0.89 8.26 0.03 0.05 0.57 1.18 1.67 3.46 6.62 45.4 0.11 0.28 2.19 4.18 6.61 11.0 27.3 67.2
Reversed Words acc n/a 100 0.00 0.01 0.01 0.01 0.02 0.03 0.03 0.09 0.02 0.01 0.01 0.00 0.05 0.07 0.11 0.48 0.00 0.05 0.00 0.17 0.24 0.30 0.42 0.44
SAT Analogies acc n/a 20 35.6 39.0 45.2 44.1 50.0 49.2 52.7 53.7 30.5 41.2 43.1 46.5 55.1 54.3 53.5 59.1 30.5 40.4 42.8 40.6 48.4 51.9 53.5 65.2
Table H.1: Scores for every task, setting and model that we investigate in this paper.
63
Figure H.1: All results for all SuperGLUE tasks.
Figure H.2: Results for SAT task.


key: chroma:document
string_value:
63
Figure H.1: All results for all SuperGLUE tasks.
Figure H.2: Results for SAT task.
 Figure H.3: All results for all Winograd tasks.
64
Figure H.4: All results for all Arithmetic tasks.
Figure H.5: All results for all Cloze and Completion tasks.
65
Figure H.6: All results for all Common Sense Reasoning tasks.
Figure H.7: All results for all QA tasks.
Figure H.8: All results for all Reading Comprehension tasks.
Figure H.9: All results for all ANLI rounds.
66
Figure H.10: All results for all Scramble tasks.
Figure H.11: All results for all Translation tasks.
67
References
[ADG+16]Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul,
Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent.
InAdvances in neural information processing systems , pages 3981–3989, 2016.
[AI19] WeChat AI. Tr-mt (ensemble), December 2019.
[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In


key: chroma:document
string_value:
[AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.
[BBDIW20] Su Lin Blodgett, Solon Barocas, Hal Daum ´e III, and Hanna Wallach. Language (technology) is power:
A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050 , 2020.
[BCFL13] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from
question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language
processing , pages 1533–1544, 2013.
[BDD+09]Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The ﬁfth
PASCAL recognizing textual entailment challenge. 2009.
[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical


key: chroma:document
string_value:
[BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. Sentiwordnet 3.0: an enhanced lexical
resource for sentiment analysis and opinion mining. In Lrec, volume 10, pages 2200–2204, 2010.
[BHDD+06]Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan
Szpektor. The second PASCAL recognising textual entailment challenge. 2006.
[BHT+20]Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella
Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. Experience grounds language.
arXiv preprint arXiv:2004.10151 , 2020.
[BLC13] Yoshua Bengio, Nicholas L ´eonard, and Aaron C. Courville. Estimating or propagating gradients through
stochastic neurons for conditional computation. Arxiv , 2013.
[BZB+19]Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about


key: chroma:document
string_value:
physical commonsense in natural language. arXiv preprint arXiv:1911.11641 , 2019.
[Car97] Rich Caruana. Multitask learning. Machine learning , 28(1), 1997.
[CB78] Susan Carey and Elsa Bartlett. Acquiring a single new word. Proceedings of the Stanford Child Language
Conference , 1978.
[CCE+18]Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and
Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv ,
abs/1803.05457, 2018.
[CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse
transformers, 2019.
[CHI+18]Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke
Zettlemoyer. Quac : Question answering in context. Arxiv , 2018.
[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina


key: chroma:document
string_value:
[CLC+19]Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina
Toutanova. BoolQ: Exploring the surprising difﬁculty of natural yes/no questions. arXiv preprint
arXiv:1905.10044 , 2019.
[CLY+19]Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and
Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 ,
2019.
[Cra17] Kate Crawford. The trouble with bias. NIPS 2017 Keynote , 2017.
[DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.
68
[DGM06] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment
challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classiﬁcation,


key: chroma:document
string_value:
challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classiﬁcation,
and recognising textual entailment , pages 177–190. Springer, 2006.
[DGV+18]Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal
transformers. Arxiv , 2018.
[DHKH14] Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heaﬁeld. Edinburgh’s phrase-based machine
translation systems for wmt-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation ,
pages 97–104, 2014.
[DL15] Andrew M. Dai and Quoc V . Le. Semi-supervised sequence learning. In Advances in neural information
processing systems , 2015.
[DMST19] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedeutung
23. Data can be found at https://github.com/mcdm/CommitmentBank/.


key: chroma:document
string_value:
23. Data can be found at https://github.com/mcdm/CommitmentBank/.
[DSC+16]Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast
reinforcement learning via slow reinforcement learning. ArXiv , abs/1611.02779, 2016.
[DWD+19]Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint
arXiv:1903.00161 , 2019.
[DYY+19]Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V . Le, and Ruslan Salakhutdinov.
Transformer-xl: Attentive language models beyond a ﬁxed-length context. Arxiv , 2019.
[EOAG18] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale.
arXiv preprint arXiv:1808.09381 , 2018.


key: chroma:document
string_value:
arXiv preprint arXiv:1808.09381 , 2018.
[FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of
deep networks. ArXiv , abs/1703.03400, 2017.
[Fyo00] Yaroslav Fyodorov. A natural logic inference system, 2000.
[GG19] Hila Gonen and Yoav Goldberg. Lipstick on a pig: Debiasing methods cover up systematic gender biases
in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862 , 2019.
[GLT+20]Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909 , 2020.
[GMDD07] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing
textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and
paraphrasing , pages 1–9. Association for Computational Linguistics, 2007.


key: chroma:document
string_value:
paraphrasing , pages 1–9. Association for Computational Linguistics, 2007.
[Gra16] Alex Graves. Adaptive computation time for recurrent neural networks. Arxiv , 2016.
[GSL+18]Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A
Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324 , 2018.
[GSR19] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M. Rush. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 1906.04043 , 2019.
[GWC+18]Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li. Meta-learning for low-resource
neural machine translation. arXiv preprint arXiv:1808.08437 , 2018.
[HB20] Daniel Hernandez and Tom Brown. Ai and efﬁciency, May 2020.
[HBFC19] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration.
CoRR , abs/1904.09751, 2019.


key: chroma:document
string_value:
CoRR , abs/1904.09751, 2019.
[HLW+20]Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song.
Pretrained transformers improve out of distribution robustness. arXiv preprint arXiv:2004.06100 , 2020.
69
[HNA+17]Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.
Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.
arXiv preprint arXiv:1712.00409 , 2017.
[HR18] Jeremy Howard and Sebastian Ruder. Universal language model ﬁne-tuning for text classiﬁcation. arXiv
preprint arXiv:1801.06146 , 2018.
[HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
preprint arXiv:1503.02531 , 2015.
[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent.


key: chroma:document
string_value:
[HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to Learn Using Gradient Descent.
InInternational Conference on Artiﬁcial Neural Networks , pages 87–94. Springer, 2001.
[HZJ+19]Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini,
Dani Yogatama, and Pushmeet Kohli. Reducing sentiment bias in language models via counterfactual
evaluation. arXiv preprint arXiv:1911.03064 , 2019.
[IBGC+14]Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daum ´e III. A neural
network for factoid question answering over paragraphs. In Empirical Methods in Natural Language
Processing , 2014.
[IDCBE19] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of
generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650 , 2019.
[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly


key: chroma:document
string_value:
[JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly
supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551 , 2017.
[JN20] Zheng Junyuan and Gamma Lab NYC. Numeric transformer - albert, March 2020.
[JVS+16]Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits
of language modeling. arXiv preprint arXiv:1602.02410 , 2016.
[JYS+19]Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.
TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351 , 2019.
[JZC+19]Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. Technical report on
conversational question answering. arXiv preprint arXiv:1909.10772 , 2019.


key: chroma:document
string_value:
conversational question answering. arXiv preprint arXiv:1909.10772 , 2019.
[KCR+18]Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond
the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of North
American Chapter of the Association for Computational Linguistics (NAACL) , 2018.
[KKS+20]Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.
Uniﬁedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700 , 2020.
[KMB20] Sarah E. Kreps, Miles McCain, and Miles Brundage. All the news that’s ﬁt to fabricate: Ai-generated
text as a tool of media misinformation, 2020.
[KMH+20]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott
Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.


key: chroma:document
string_value:
Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
[KPR+19]Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redﬁeld, Michael Collins, Ankur Parikh, Chris Alberti,
Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova,
Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational
Linguistics , 2019.
[KR16] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. Arxiv , 2016.
[LB02] Edward Loper and Steven Bird. Nltk: The natural language toolkit, 2002.
[LC19] Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint
arXiv:1901.07291 , 2019.
70


key: chroma:document
string_value:
arXiv:1901.07291 , 2019.
70
[LCG+19]Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint
arXiv:1909.11942 , 2019.
[LCH+20]Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao.
Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994 , 2020.
[LDL19] Zhongyang Li, Xiao Ding, and Ting Liu. Story ending prediction by transferable bert. arXiv preprint
arXiv:1905.07504 , 2019.
[LDM12] Hector Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In Thirteenth
International Conference on the Principles of Knowledge Representation and Reasoning , 2012.
[LGG+20]Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and


key: chroma:document
string_value:
Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint
arXiv:2001.08210 , 2020.
[LGH+15]Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. Representation
learning using multi-task deep neural networks for semantic classiﬁcation and information retrieval. In
Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies , 2015.
[LH17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101 , 2017.
[LHCG19a] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neural
networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482 ,
2019.
[LHCG19b] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for
natural language understanding. arXiv preprint arXiv:1901.11504 , 2019.


key: chroma:document
string_value:
natural language understanding. arXiv preprint arXiv:1901.11504 , 2019.
[Lin20] Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? arXiv preprint
arXiv:2005.00955 , 2020.
[LLG+19]Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy,
Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.
[LM17] Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441 , 2017.
[LOG+19]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.
arXiv preprint arXiv:1907.11692 , 2019.


key: chroma:document
string_value:
arXiv preprint arXiv:1907.11692 , 2019.
[LPP+20]Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal,
Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, Sebastian Riedel, and Kiela Douwe.
Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401 ,
2020.
[LSP+18]Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam
Shazeer. Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 , 2018.
[LWS+20]Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez.
Train large, then compress: Rethinking model size for efﬁcient training and inference of transformers,
2020.
[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading


key: chroma:document
string_value:
[LXL+17]Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading
comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 , 2017.
[LYN+20]Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy
Lin. Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380 , 2020.
[Mac92] David. MacKay. Information-based objective functions for active data selection. Neural Computation ,
1992.
71
[MBXS17] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pages 6294–6305,
2017.
[MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efﬁcient estimation of word representations
in vector space. arXiv preprint arXiv:1301.3781 , 2013.


key: chroma:document
string_value:
in vector space. arXiv preprint arXiv:1301.3781 , 2013.
[MCH+16]Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende,
Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of
commonsense stories. arXiv preprint arXiv:1604.01696 , 2016.
[MCKS18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity?
a new dataset for open book question answering. ArXiv , abs/1809.02789, 2018.
[MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of
large-batch training, 2018.
[MKM+94]Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson,
Karen Katz, and Britta Schasberger. The penn treebank: annotating predicate argument structure.
InProceedings of the workshop on Human Language Technology , pages 114–119. Association for
Computational Linguistics, 1994.


key: chroma:document
string_value:
InProceedings of the workshop on Human Language Technology , pages 114–119. Association for
Computational Linguistics, 1994.
[MKXS18] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language
decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730 , 2018.
[MPL19] R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic
heuristics in natural language inference. arXiv preprint arXiv:1902.01007 , 2019.
[MWZ+18]Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson,
Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting, 2018.
[NBR20] Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained
language models. arXiv preprint arXiv:2004.09456 , 2020.
[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments.


key: chroma:document
string_value:
[NK19] Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments.
arXiv preprint arXiv:1907.07355 , 2019.
[Nor09] Peter Norvig. Natural language corpus data, 2009.
[NvNvdG19] Malvina Nissim, Rik van Noord, and Rob van der Goot. Fair is better than sensational: Man is to doctor
as woman is to doctor. arXiv preprint arXiv:1905.09866 , 2019.
[NWD+19]Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial
nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 , 2019.
[oR16] University of Regensburg. Fascha, 2016.
[PCC18] Mohammad Taher Pilehvar and Jose Camacho-Collados. WIC: 10,000 example pairs for evaluating
context-sensitive representations. arXiv preprint arXiv:1808.09121 , 2018.
[PFB18] Jason Phang, Thibault F ´evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary


key: chroma:document
string_value:
[PFB18] Jason Phang, Thibault F ´evry, and Samuel R. Bowman. Sentence encoders on STILTs: Supplementary
training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018.
[PHR+18]Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and
Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation
evaluation. In Proceedings of EMNLP , 2018.
[PKL+16]Denis Paperno, Germ ´an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro
Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern ´andez. The lambada dataset: Word prediction
requiring a broad discourse context. arXiv preprint arXiv:1606.06031 , 2016.
[PNZtY18] Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih. Dissecting contextual word
embeddings: Architecture and representation, 2018.


key: chroma:document
string_value:
embeddings: Architecture and representation, 2018.
[Pos18] Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771 , 2018.
72
[PSM14] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language
processing (EMNLP) , 2014.
[QIA20] QIANXIN. Sa-net on albert (ensemble), April 2020.
[QMZH19] Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun. Reducing gender bias in word-level language
models with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801 , 2019.
[RBG11] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An
evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series , 2011.
[RCM19] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering
challenge. Transactions of the Association for Computational Linguistics , 7:249–266, 2019.


key: chroma:document
string_value:
challenge. Transactions of the Association for Computational Linguistics , 7:249–266, 2019.
[RCP+17]Scott Reed, Yutian Chen, Thomas Paine, A ¨aron van den Oord, SM Eslami, Danilo Rezende, Oriol
Vinyals, and Nando de Freitas. Few-shot autoregressive density estimation: Towards learning to learn
distributions. arXiv preprint arXiv:1710.10304 , 2017.
[RJL18] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for
squad. arXiv preprint arXiv:1806.03822 , 2018.
[RL16] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. ICLR 2017 (oral) ,
2016.
[RLL+19]Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. NumNet: Machine reading comprehension
with numerical reasoning. In Proceedings of EMNLP , 2019.
[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in


key: chroma:document
string_value:
[RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in
coreference resolution. arXiv preprint arXiv:1804.09301 , 2018.
[RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding
by generative pre-training, 2018.
[Ros12] R.S. Ross. Guide for conducting risk assessments. NIST Special Publication , 2012.
[RRBS19] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of
the generalization error across scales, 2019.
[RRS20] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters
of a language model? arXiv preprint arXiv:2002.08910 , 2020.
[RSR+19]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer, 2019.


key: chroma:document
string_value:
Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a uniﬁed text-to-text
transformer, 2019.
[RWC+19]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners, 2019.
[SBBC19] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial
winograd schema challenge at scale, 2019.
[SBC+19]Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Jeff Wu, Alec Radford,
Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris
McGufﬁe, and Jasmine Wang. Release strategies and the social impacts of language models, 2019.
[SCNP19] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a
babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019.


key: chroma:document
string_value:
babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326 , 2019.
[SDCW19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of
BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.
[SDSE19] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. CoRR , abs/1907.10597, 2019.
[SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with
monolingual data. arXiv preprint arXiv:1511.06709 , 2015.
73
[SMM+17]Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff
Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint
arXiv:1701.06538 , 2017.


key: chroma:document
string_value:
arXiv:1701.06538 , 2017.
[SPP+19]Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019.
[SS20] Timo Schick and Hinrich Sch ¨utze. Exploiting cloze questions for few-shot text classiﬁcation and natural
language inference. arXiv preprint arXiv:2001.07676 , 2020.
[STQ+19]Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence
pre-training for language generation. arXiv preprint arXiv:1905.02450 , 2019.
[TFR+17]Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain
randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ
international conference on intelligent robots and systems (IROS) , pages 23–30. IEEE, 2017.


key: chroma:document
string_value:
international conference on intelligent robots and systems (IROS) , pages 23–30. IEEE, 2017.
[TL05] Peter D. Turney and Michael L. Littman. Corpus-based learning of analogies and semantic relations.
CoRR , abs/cs/0508103, 2005.
[TL18] Trieu H. Trinh and Quoc V . Le. A simple method for commonsense reasoning. arXiv preprint
arXiv:1806.02847 , 2018.
[TLBS03] Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. Combining independent
modules to solve multiple-choice synonym and analogy problems. CoRR , cs.CL/0309035, 2003.
[Tur20] Project Turing. Microsoft research blog, Feb 2020.
[VBL+16]Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching Networks for One
Shot Learning. In Advances in neural information processing systems , pages 3630–3638, 2016.
[VSP+17]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz


key: chroma:document
string_value:
Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing
systems , 2017.
[WPN+19]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer
Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems , pages 3261–3275, 2019.
[WXH+18]Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. Multi-agent
dual learning. ICLR 2019 , 2018.
[XDH+19]Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V . Le. Unsupervised data
augmentation for consistency training, 2019.
[YdC+19]Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski,
Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating


key: chroma:document
string_value:
Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating
general linguistic intelligence. arXiv preprint arXiv:1901.11373 , 2019.
[YDY+19]Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V . Le. XLNet:
Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 ,
2019.
[ZHB+19]Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine
really ﬁnish your sentence? arXiv preprint arXiv:1905.07830 , 2019.
[ZHR+19]Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin
Choi. Defending against neural fake news. arXiv preprint arXiv:1905.12616 , 2019.
[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.


key: chroma:document
string_value:
[ZLL+18]Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.
ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv
preprint arXiv:1810.12885 , 2018.
[ZSW+19a] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul
Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019.
74
[ZSW+19b] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv , abs/1909.08593,
2019.
75


key: chroma:document
string_value:
Uncertainty-Aware Weakly Supervised Action
Detection from Untrimmed Videos
Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia Schmid
Google Research
faarnab, chensun, anagrani, cordelias g@google.com
Abstract. Despite the recent advances in video classication, progress
in spatio-temporal action recognition has lagged behind. A major contributing factor has been the prohibitive cost of annotating videos frameby-frame. In this paper, we present a spatio-temporal action recognition
model that is trained with only video-level labels, which are signicantly
easier to annotate. Our method leverages per-frame person detectors
which have been trained on large image datasets within a Multiple Instance Learning framework. We show how we can apply our method in
cases where the standard Multiple Instance Learning assumption, that
each bag contains at least one instance with the specied label, is invalid using a novel probabilistic variant of MIL where we estimate the
uncertainty of each prediction. Furthermore, we report the rst weaklysupervised results on the AVA dataset and state-of-the-art results among
weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video


key: chroma:document
string_value:
weakly-supervised methods on UCF101-24.
Keywords: spatio-temporal action recognition, weak supervision, video
understanding, mulitple instance learning
1 Introduction
Video classication has witnessed great advances recently due to large datasets
such as Kinetics [ 20] and Moments in Time [ 30] which have enabled training
of specialised neural network architectures for video [ 5,9]. However, progress in
other video understanding tasks, such as spatio-temporal action detection, has
lagged behind in comparison. There are fewer datasets for action recognition,
which are also signicantly smaller than their video-classication counterparts. A
reason for this is the exorbitant cost of annotating videos with spatio-temporal
labels { each frame of an action has to be manually labelled with a bounding
box. Moreover, annotating temporal boundaries of actions is not only arduous,
but often ambiguous with annotators failing to reach consensus about the start
and end times of an action [6,39].
In this paper, we propose a method to train spatio-temporal action detectors
using only weak, video-level annotations as shown in Fig. 1. To achieve this, we
leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations


key: chroma:document
string_value:
leverage image-based person detectors which have been trained on large image
datasets such as Microsoft COCO [ 26] and are accurate across large variations
in appearance, scene and pose. We adopt a Multiple Instance Learning (MIL)arXiv:2007.10703v1  [cs.CV]  21 Jul 2020
2 A. Arnab et al.
Training dataPrediction
Labels: 
sit, talk to (people), watch (a person), carry/hold (an object), listen to (a person)
 sit, carry/hold (an object), listen to (a person), talk to (people), watch (a person)
sit, talk to (people), watch (a person)
sit, listen to (a person), watch (a person)
time
Fig. 1. We propose a method to train a spatio-temporal action detector using only
weak, video-level labels on challenging, real-world datasets. Note that the video-level
labels that we have may apply to multiple people in the video, and that these labels
may only be active for an unannotated time interval of the input clip.
framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of


key: chroma:document
string_value:
framework, where a person tubelet is an instance, and all person tubelets in the
video form a bag. An important consideration in our approach is the presence of
label noise: this is introduced from using o-the-shelf person detectors which have
not been trained on the video-dataset of interest, and also the fact that we have
to sample tubelets from large bags in long videos due to memory constraints. In
both of these scenarios, the standard Multiple Instance Learning assumption [ 7],
that each bag contains at least one instance with the bag-level label, may be
violated. We are not aware of previous work that has explicitly addressed this
problem, and we do so with a probabilistic variant of MIL where we estimate the
uncertainty of an instance-level prediction.
Using our approach, we obtain state-of-the-art results among weakly-supervised
methods on the UCF101-24 dataset. Furthermore, we report, to our knowledge,
the rst weakly-supervised results on the AVA dataset (the only large-scale
dataset for spatio-temporal action recognition), where we also show the accuracy
trade-os when annotating video-clips for time intervals of varying durations.
2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.


key: chroma:document
string_value:
2 Related Work
Most prior work on spatio-temporal action recognition has been fully-supervised.
Initial approaches in the area used 3D sliding window detectors in conjunction
with handcrafted, volumetric features [ 21,25]. Current state-of-the-art approaches
are temporal extensions of object detection architectures [ 19,41,52,35] such as
Faster-RCNN [ 37] and SSD [ 27]. These approaches predict bounding boxes around
the action in a frame, using as input either a single frame along with optical
Uncertainty-Aware Weakly Supervised Action Detection 3

ow to capture temporal information [ 41,38] or multiple frames at the input to
provide temporal context [ 19]. The predicted bounding boxes are then linked
over time using an online, greedy algorithm or dynamic programming to create
spatio-temporal tracks. Our work builds on these methods by also utilising a
detection architecture and spatio-temporal linking. However, these approaches
all require bounding box annotations at each frame in the video whilst we only
use video-level labels which are signicantly cheaper to acquire.
Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation


key: chroma:document
string_value:
Weakly supervised approaches to spatio-temporal action recognition have
also been explored before as they enable a signicant reduction in annotation
time and cost. Relevant to our approach is the work of [ 6]. Che ron et al. [6] also
use person detections, and infer their action labels using a formulation based on
discriminative clustering [ 2]. Although their approach allows them to incorporate
dierent types of supervision, it eectively learns a linear classier on top of
pretrained, deep features. Our method in contrast is learned fully end-to-end.
Mettes et al. [29] also employed Multiple Instance Learning (MIL), but used
action proposals [ 47] instead of the human detections used by our work and [ 6].
However, [ 29], rely on additional cheap \point" annotations (a single spatial point
annotated for a subset of the frames which constitute the action) which also
ensures that the standard MIL assumption is not violated. In follow-up work [ 28],
the authors removed the need for \point" annotations by incorporating biases
(i.e.the presence of objects in the video, a bias that actions typically occur in the
centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple


key: chroma:document
string_value:
centre of a frame) instead. Finally, Weinzaepfel et al. [50] also used a Multiple
Instance Learning framework in conjunction with human detections. The authors,
however, assumed that sparse spatial supervision was present ( i.e.bounding
boxes for a small subset of frames in the action tube), unlike our method which
requires video-level labels alone.
We also note that many approaches have addressed temporal action detection
(localising actions in time but not space) with only video-level tags as supervision [ 48,42,31,34]. UntrimmedNets [ 48] uses a network with two branches, a
classication module to perform action classication and a selection module to
select relevant frames.Hide-and-Seek [ 42] obtains more precise temporal boundaries by forcing the network to attend to more discriminative frames by randomly
hiding parts of videos. However, these methods are trained and evaluated on
datasets such as ActivityNet [ 4] and THUMOS14 [ 18], which contain mostly one
action per video, and are thus signicantly less challenging than datasets such as
AVA [14] which we evaluate on.
Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative


key: chroma:document
string_value:
Finally, we note that another approach to combat the eort of dataset
annotation has been various forms of self-supervised learning, where discriminative
feature representations can be learned with unlabelled data. Examples in video
include cross-modal self-supervised learning by learning correspondences between
the audio and image streams readily available in videos [ 1,33,51], transcribed
speech [ 44] or using meta-data such as hashtags [ 10] as a form of weak labelling.
Self-supervised approaches, however, are complementary to our approach, as they
still require a limited amount of fully-labelled data for the nal task of interest.
4 A. Arnab et al.
In our weakly-supervised action detection scenario, we never have access to full,
spatio-temporal ground-truth annotations for a single training example.
3 Proposed Approach
As shown in Fig. 1, given a set of video clips, with only clip-level annotations of
the actions taking place, our goal is to learn a model to recognise and localise
these actions in space and time. Our method is based on Multiple Instance
Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we


key: chroma:document
string_value:
Learning (MIL) which we brie
y review in Sec. 3.1. Thereafter, we show how we
use it for weakly-supervised spatio-temporal action recognition in Sec. 3.2. We
then describe how the standard MIL assumption, is often violated in our scenario
and describe a method to mitigate this by leveraging uncertainty estimates by
our network in Sec. 3.3. Finally, we discuss implementation details of our network
in Sec. 3.4.
3.1 Multiple Instance Learning
In the standard Multiple Instance Learning (MIL) [ 7] formulation, one is given a
bag ofNinstances, denoted as x=fx1;x2;:::;xNg. The class labels for each
of the instances is unknown, but the label for the entire bag, x, is known. The
standard MIL assumption is that a bag is assigned a class label if at least one
instance in the bag is associated with this label. More formally, we consider the
multi-label classication case, where the label vector for the bag is y2RC, and
yl= 1 if there is at least one instance with the lthlabel is present in the bag,
andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C
class labels.


key: chroma:document
string_value:
andyl= 0 otherwise. Note that each bag can be labelled with multiple of the C
class labels.
Our goal is to train an instance-level classier (parameterised as a neural
network), that predicts p(yl= 1jxj), or the label probabilities for the jthinstance.
However, as we only have the labels for the entire bag, and not each instance,
MIL methods aggregate the set of instance-level probabilities, fpijgfor a bag
i, to bag-level probabilities, pi, using an aggregation function, g(), where the
probabilities are obtained from a suitable activation function (sigmoid or softmax)
on the logits output by the neural network:
p(yil= 1jx1;x2;:::;xN) =g(pi1;pi2;:::;piN): (1)
Once we have bag-level predictions, we can apply a standard classication
loss between the bag-level probabilities and bag-level ground truth, and train a
neural network with stochastic gradient descent. Since we consider the multi-label
classication case, we use the binary cross-entropy:
Lce(x;y) =NbX
iCX


key: chroma:document
string_value:
classication case, we use the binary cross-entropy:
Lce(x;y) =NbX
iCX
lyillogpil+ (1yil) log(1pil) (2)
Note that we dened pilas the bag-level probability of the ithbag taking the
lthlabel, which is obtained using Eq. 1, and Nbis the number of bags in the
mini-batch.
Uncertainty-Aware Weakly Supervised Action Detection 5
Aggregation The aggregation function, g(), can naturally be implemented in
neural networks as a global pooling function over all outputs of the network.
Common, permutation-invariant pooling functions include, max-pooling, generalised mean-pooling and log-sum-exponential (LSE) pooling [ 3] (a smooth and
convex approximation of the maximum function) respectively:
g(fpjg) = max
jpj (3)
g(fpjg) =0
@1
jjjX
jpr
j1
A1
r
(4)
g(fpjg) =1
rlog0
@1
jjjX
jerpj1
A (5)


key: chroma:document
string_value:
A1
r
(4)
g(fpjg) =1
rlog0
@1
jjjX
jerpj1
A (5)
Max-pooling only considers the top-scoring instance in the bag, and thus naturally
captures the MIL assumption that at least one instance in the bag has the
specied, bag-level label. Moreover, it can also be more robust to instances in
the bag that do not have the bag-level label. However, mean and LSE pooling
have been employed in applications such as weakly-supervised segmentation
[36], object recognition [ 45] and medical imaging [ 24] where multiple instances
in the bag do typically have the bag-level label. Note that higher values of
therhyperparameter for both these functions increases their \peakiness" and
approximates the maximum value. For our scenario, detailed in the next section,
we found max-pooling to be the most appropriate.
3.2 Weakly-supervised spatio-temporal action recognition as
multiple instance learning
Our goal is to learn a model to recognise and localise actions in space and
time given only video-level annotations. To facilitate this, we leverage a person
detector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].


key: chroma:document
string_value:
time given only video-level annotations. To facilitate this, we leverage a person
detector that has been trained on a large image dataset, i.e.Microsoft COCO [ 26].
Concretely, we run a person detector on our training videos, and create person
tubelets which are person detections over Kconsecutive frames in the video.
Our bag for multiple instance learning thus consists of all the tubelets within a
video, and is annotated with the video-level labels that we have as supervision,
as illustrated in Fig. 2. Note that the size of the bag varies for every video clip, as
the bag size is determined by the length of the video and the number of detected
people.
As shown in Fig. 2, our network architecture for this task is a Fast-RCNN [ 12]
style detector that has been extended temporally. Given a video clip of Kframes,
and proposals which in our case are person detections, the network classies the
action(s) taking place at the centre frame of each proposal, given the temporal
context of the K1 frames around it.
Note that the spatio-temporal localisation task is eectively factorised: the
spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.


key: chroma:document
string_value:
spatial localisation capability of the model depends on the quality of the person
6 A. Arnab et al.
CNN feature extractorROI AlignClassiﬁer
Uncertainty-aware MILLoss ℒMultiple Instance Learning Bag
Probabilities, pUncertainties,  σ
Bag-level label
…
stand, listen to (a person), watch (a person), 
walk, carry/hold (an object) …
Fig. 2. Overview of our approach for training an action detector in a weakly-supervised
manner using multiple instance learning: Each bag consists of all the tubelets that have
been extracted from the video-clip. These tubelets are obtained using an o-the-shelf
person detector which has not been trained on the dataset of interest. These tubelets
act as proposals for a Fast-RCNN style detector operating on a sequence of rgb images.
The predictions for each of the tubelets in the bag are then aggregated together, and
compared to the bag-level label. Uncertainty estimates produced by the network are
used to compensate for noise in the bag-level labels during training.
detections. Temporal localisation, on the other hand, is performed by linking
person tubelets through the video as commonly done in the literature [ 19,41,52,6],


key: chroma:document
string_value:
detections. Temporal localisation, on the other hand, is performed by linking
person tubelets through the video as commonly done in the literature [ 19,41,52,6],
since this method can scale to arbitrarily long videos. We use the same algorithm
as Kalogeiton et al. [19] which links together detections within a small temporal
window greedily based on the spatial intersection over union (IoU) between
bounding boxes on consecutive frames.
Finally, note that for a video consisting of Tframes, the bag could consist of
TK+1 person tubelets if a person is detected on each frame of the video, and a
tubelet is started from each frame. Due to memory limitations, it is infeasible to t
an entire bag onto a GPU for training. As a result, we uniformly sample instances
from each bag during training, whilst still retaining the original bag-level label.
This introduces additional noise into the problem, as detailed next.
3.3 Label noise and violation of the standard MIL assumption
The standard MIL assumption, that at least one instance in the bag is assigned
the bag-level label is often violated in our scenario. There are two primary factors
for this: Firstly, due to computational constraints, we cannot process a whole bag
at a time, but must instead sample instances from a bag. It is therefore possible


key: chroma:document
string_value:
for this: Firstly, due to computational constraints, we cannot process a whole bag
at a time, but must instead sample instances from a bag. It is therefore possible
to sample a bag that does not contain any tubelets with the labelled action. The
Uncertainty-Aware Weakly Supervised Action Detection 7
Predicted 
uncertainty
, 


Predicted probability
, 


b
a
c
Wrong prediction. Low 
uncertainty. Highest loss
Correct prediction. Low 
uncertainty. Lowest loss
Incorrect prediction. High uncertainty. Moderate loss
Fig. 3. The loss surface of our uncertainty-based loss (Eq. 6). The ground truth binary
label in this example is 1. Hence, the loss is minimised when the network predicts
a high probability and a low uncertainty (point \a"). However, making an incorrect
prediction with a high uncertainty is not penalised as much (point \b"), and is suitable
for cases when the input bags are noisy and the bag-level label is not present in any of
the tubelets. Finally, predicting the incorrect label with a low uncertainty is penalised
the most (point \c"). Best viewed in colour.
likelihood of this occurring is inversely proportional to the ratio of the duration


key: chroma:document
string_value:
the most (point \c"). Best viewed in colour.
likelihood of this occurring is inversely proportional to the ratio of the duration
of the labelled action to the total video length. Secondly, in a weakly-supervised
scenario, we use person detectors that are not trained on the video dataset of
interest. Consequently, there can be failures in the detector, especially when
there is a large domain gap between the detector's training distribution and the
video dataset. False negatives (missing detections for people in the scene) are a
particular issue because it is possible that we do not have a single person tubelet
in the bag that corresponds to the labelled action.
Therefore, there are cases when there is no tubelet which actually has the baglevel label. To handle these cases, inspired by [ 22,32], we modify the network to
additionally predict the uncertainty 2RCfor each binary label for all tubelets
in the bag. Intuitively, to minimise the training error, the network can predict the
bag-level label with low uncertainty or it can predict a high uncertainty to avoid
being penalised heavily for noisy bags where the bag-level label is not present in
any of the tubelets. The nal loss, in conjunction with the original cross entropy,
is dened as:
L(x;y; ) =1


key: chroma:document
string_value:
any of the tubelets. The nal loss, in conjunction with the original cross entropy,
is dened as:
L(x;y; ) =1
2Lce(x;y) + log2(6)
As shown by [ 23], this corresponds to assuming a Boltzmann distribution on the
output of the network with a temperature of 2, and approximately minimising
its log-likelihood.
8 A. Arnab et al.
The loss surface of this probabilistic loss is visualised in Fig. 3. Note how the
loss is the lowest when the predicted label is correct and there is low uncertainty.
However, the loss is not excessive if the incorrect label is predicted with a
high uncertainty. This is in contrast with the standard cross-entropy loss which
penalises incorrect predictions heavily.
3.4 Network architecture and implementation
Our action detector is similar to Fast-RCNN [ 12] using the SlowFast [ 9] video
network architecture based on the ResNet-50 backbone [ 16] pretrained on Kinetics [ 20]. As described in Sec. 3.2, we use region proposals obtained from a
Faster-RCNN detection model trained with Detectron [ 13]. Region-of-interest
features [ 12] are extracted from the last feature map of \res5" using RoIAlign [ 15].


key: chroma:document
string_value:
features [ 12] are extracted from the last feature map of \res5" using RoIAlign [ 15].
Our choice for this architecture is motivated by the fact that it is simple and has
achieved state-of-the-art results on the AVA dataset [ 14] in a fully-supervised
setting [ 9]. Note that our network does not use additional optical 
ow inputs
(which can be considered as an additional source of supervision) as common in
other video architectures [5,19,41,6].
We predict the uncertainty, 2RCfor each of the Cbinary labels dened
by the dataset for each tubelet. As we use max-pooling to aggregate the tubelet
predictions, we select the uncertainty prediction corresponding to the selected
tubelet for computing the loss. For numerical stability, we predict v:=log2
with our network, using the \softplus", f(x) =log(1 + exp(x)), activation
function to ensure positivity. We then compute1
2=exp(v), and avoid the
possibility of dividing by 0 which could be the case if we predicted 2directly
with the network.
We train our network with synchronous stochastic gradient descent (SGD),
using 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a


key: chroma:document
string_value:
We train our network with synchronous stochastic gradient descent (SGD),
using 8 GPUs and a batch size of 4 on each GPU. In our case, each element of a
batch is of a bag from Multiple Instance Learning. Each bag samples a maximum
of 4 tubelets. Each tubelet itself consists of 16 frames.
4 Experiments
4.1 Experimental set-up
We evaluate our method on UCF101-24 and AVA, described in more detail below.
Note that other video datasets such as THUMOS [ 18] and ActivityNet [ 4] are not
suitable for spatiotemporal localisation, as they lack bounding box annotations.
UCF101-24: UCF101-24 is a subset of the UCF101 [ 43] dataset, consisting of 24
action classes with spatio-temporal localisation annotation, released as bounding
box annotations of humans.Although each video contains only a single action
class, it may contain multiple individuals performing the action with dierent
spatial and temporal boundaries. Moreover, there may also be people present in
the video who are not performing any labelled action. Following standard practice,
Uncertainty-Aware Weakly Supervised Action Detection 9
Table 1. Ablation study of dierent variants of our method on the UCF101-24 validation


key: chroma:document
string_value:
Uncertainty-Aware Weakly Supervised Action Detection 9
Table 1. Ablation study of dierent variants of our method on the UCF101-24 validation
set. We report the Video mAP at IoU thresholds of 0.2 and 0.5 respectively.
Video AP
0.2 0.5
Weakly supervised baseline 54.3 29.7
MIL - LSE pooling 60.1 33.1
MIL - mean pooling 60.3 33.0
MIL - max pooling 60.7 33.5
MIL - max pooling, uncertainty 61.7 35.0
Fully supervised 69.3 43.6
we use the corrected annotations of [ 41] and report the mean average precision
at a video level (Video AP) for the rst split of the dataset. For evaluating the
Video AP, we link tubelets together using the algorithm of [19].
AVA [ 14]:This dataset consists of 430, 15-minute video clips obtained from
movies. 80 atomic visual actions are annotated exhaustively for all people in
the video, where one person is often simultaneously performing multiple actions.
The dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2
annotations.


key: chroma:document
string_value:
The dataset annotates keyframes at every second in the video. Following standard practice, we report the Frame AP at an IoU threshold of 0.5 using v2.2
annotations.
4.2 Experiments on UCF101-24
We rst conduct ablation studies of our model on the UCF101-24 dataset. We
discard the spatio-temporal annotations for the whole untrimmed video, and so
our bag in multiple instance learning contains tubelets from the whole video.
Ablation study Table 1 ablates dierent variants of our method: The most na ve
baseline is to not perform any multiple instance learning, and to simply train in
a fully-supervised fashion assuming that the label of a tubelet is the video-level
label. As shown in the rst row of Tab. 1, this method performs the worst as
the assumed tubelet-level labels are often incorrect. The use of multiple instance
learning improves results, with the various aggregation functions performing
similarly. Max-pooling, however, performs the best, and we believe this is because
the max operation is the most suitable for dealing with the noise present in our
tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set
r= 1. Finally, introducing our uncertainty-based loss function improves results


key: chroma:document
string_value:
tubelets as described in Sec. 3.3. Note that for mean and LSE-pooling, we set
r= 1. Finally, introducing our uncertainty-based loss function improves results
even further, obtaining a Video mAP of 35.0 at a threshold of 0.5. This is 80%
of the performance achieved by our fully-supervised baseline.
Person detections on UCF101-24 Note that for our weakly-supervised experiments, the person tubelets for training are obtained from a Faster-RCNN [ 37]
person detector that has only been trained on Microsoft COCO [ 26]. There is a
10 A. Arnab et al.
signicant domain gap between COCO and UCF, and the annotation protocol of
person boxes on UCF is also not consistent (for example, the bounding box for a
person riding a horse often includes the horse in UCF) with that of COCO. These
discrepancies are re
ected by the fact that our person detections used during
training only have a recall of 46.9% compared to the ground truth person boxes,
when using an IoU threshold of 0.5 to signify a correct match. Furthermore,
the precision of our person tubelets on the training set is only 21.1%. A major
contributing factor to this is that UCF action annotations are not exhaustive


