This is an early draft. Please read/cite the published version instead:
Caliskan, Aylin, Joanna J. Bryson, and Arvind Narayanan. “Semantics derived automatically from language corpora contain human-like biases.” Science
356.6334 (2017): 183-186. http://opus.bath.ac.uk/55288/

arXiv:1608.07187v4 [cs.AI] 25 May 2017

Semantics derived automatically from language
corpora necessarily contain human biases
Aylin Caliskan1 , Joanna J. Bryson1,2 , and Arvind Narayanan1
1 Princeton

University
of Bath
∗ Address correspondence to aylinc@princeton.edu, bryson@conjugateprior.org, arvindn@cs.princeton.edu.
2 University

ABSTRACT
Artificial intelligence and machine learning are in a period of astounding growth. However, there are concerns that these
technologies may be used, either with or without intention, to perpetuate the prejudice and unfairness that unfortunately
characterizes many human institutions. Here we show for the first time that human-like semantic biases result from the
application of standard machine learning to ordinary language—the same sort of language humans are exposed to every
day. We replicate a spectrum of standard human biases as exposed by the Implicit Association Test and other well-known
psychological studies. We replicate these using a widely used, purely statistical machine-learning model—namely, the GloVe
word embedding—trained on a corpus of text from the Web. Our results indicate that language itself contains recoverable and
accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards
race or gender, or even simply veridical, reflecting the status quo for the distribution of gender with respect to careers or first
names. These regularities are captured by machine learning along with the rest of semantics. In addition to our empirical
findings concerning language, we also contribute new methods for evaluating bias in text, the Word Embedding Association
Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results have implications not only for AI and
machine learning, but also for the fields of psychology, sociology, and human ethics, since they raise the possibility that mere
exposure to everyday language can account for the biases we replicate here.

Introduction
Those astonished by the human-like capacities visible in the recent advances in artificial intelligence (AI) may be comforted
to know the source of this progress. Machine learning, exploiting the universality of computation (Turing, 1950), is able to
capture the knowledge and computation discovered and transmitted by humans and human culture. However, while leading to
spectacular advances, this strategy undermines the assumption of machine neutrality. The default assumption for many was that
computation, deriving from mathematics, would be pure and neutral, providing for AI a fairness beyond what is present in
human society. Instead, concerns about machine prejudice are now coming to the fore—concerns that our historic biases and
prejudices are being reified in machines. Documented cases of automated prejudice range from online advertising (Sweeney,
2013) to criminal sentencing (Angwin et al., 2016).
Most experts and commentators recommend that AI should always be applied transparently, and certainly without prejudice.
Both the code of the algorithm and the process for applying it must be open to the public. Transparency should allow courts,
companies, citizen watchdogs, and others to understand, monitor, and suggest improvements to algorithms (Oswald and Grace,
2016). Another recommendation has been diversity among AI developers, to address insensitive or under-informed training
of machine learning algorithms (Sweeney, 2013; Noble, 2013; Barr, 2015; Crawford, 2016). A third has been collaboration
between engineers and domain experts who are knowledgeable about historical inequalities (Sweeney, 2013).
Here we show that while all of these strategies might be helpful and even necessary, they could not be sufficient. We
document machine prejudice that derives so fundamentally from human culture that it is not possible to eliminate it through

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
strategies such as the above. We demonstrate here for the first time what some have long suspected (Quine, 1960)—that
semantics, the meaning of words, necessarily reflects regularities latent in our culture, some of which we now know to be
prejudiced. We demonstrate this by showing that standard, widely used Natural Language Processing tools share the same
biases humans demonstrate in psychological studies. These tools have their language model built through neutral automated
parsing of large corpora derived from the ordinary Web; that is, they are exposed to language much like any human would be.
Bias should be the expected result whenever even an unbiased algorithm is used to derive regularities from any data; bias is the
regularities discovered.
Human learning is also a form of computation. Therefore our finding that data derived from human culture will deliver
biases and prejudice have implications for the human sciences as well. They present a new “null hypothesis” for explaining the
transmission of prejudice between humans. Our findings also have implications for addressing prejudice, whether in humans or
machines. The fact that it is rooted in language makes prejudice difficult to address, but by no means impossible. We argue
that prejudice must be addressed as a component of any intelligent system learning from our culture. It cannot be entirely
eliminated from the system, but rather must be compensated for.
In this article, we begin by explaining meaning and the methods by which we determine human understanding, and interpret
it in machines. Then we present our results. We replicate previously-documented biases and prejudices in attitudes towards
ordinary objects, animals, and humans. We show that prejudices that reduce the number of interview invitations sent to people
because of the racial association of their name, and that associate women with arts rather than science or mathematics, can
be retrieved from standard language tools used in ordinary AI products. We also show that veridical information about the
proportions of women in particular job categories, or what proportion of men versus women have a particular name, can be
recovered using the same methods. We then present a detailed account of our methods, and further discussion of the implications
of our work.

Meaning and Bias in Humans and Machines
In AI and machine learning, bias refers to prior information, a necessary prerequisite for intelligence (Bishop, 2006). Yet bias
can be problematic where prior information is derived from precedents known to be harmful. For the purpose of this paper,
we will call harmful biases ‘prejudice’. We show here that prejudice is a special case of bias identifiable only by its negative
consequences, and therefore impossible to eliminate purely algorithmically. Rather, prejudice requires deliberate action based
on knowledge of a society and its outstanding ethical challenges.
If we are to demonstrate that AI incorporates the same bias as humans, we first have to be able to document human bias.
We will use several methods to do this below, but the one we use most is the Implicit Association Test (IAT). First introduced
by Greenwald et al. (1998), the IAT demonstrates enormous differences in response times when subjects are asked to pair two
concepts they find similar, in contrast to two concepts they find different. The IAT follows a reaction time paradigm, which
means subjects are encouraged to work as quickly as possible, and their response times are the quantified measure. For example,
subjects are much quicker if they are told to label insects as unpleasant and flowers as pleasant than if they are asked to label
these objects in reverse. The fact that a pairing is faster is taken to indicate that the task is more easy, and therefore that the two
subjects are linked in their mind. The IAT is ordinarily used to pair categories such as ‘male’ and ‘female’ with attributes such
as ‘violent’ or ‘peaceful’. The IAT has been used to describe and account for a wide range of implicit prejudices and other
phenomena, including stereotype threat (Kiefer and Sekaquaptewa, 2007; Stanley et al., 2011).
Our method for demonstrating both bias and prejudice in text is a variant of the implicit association test applied to a
widely-used semantic representation of words in AI, termed word embeddings. These are derived by representing the textual
context in which a word is found as a vector in a high-dimensional space. Roughly, for each word, its relationship to all other
words around it is summed across its many occurrences in a text. We can then measure the distances (more precisely, cosine
similarity scores) between these vectors. The thesis behind word embeddings is that words that are closer together in the vector
−−−−−−−→ is closer to −
−→ than to −
−−−→ it
space are semantically closer in some sense. Thus, for example, if we find that −
programmer
man
woman,
suggests (but is far from conclusive of) a gender stereotype. We assume here that this measure is analogous to reaction time in
the IAT, since the shorter time implies a semantic ‘nearness’ (McDonald and Lowe, 1998; Moss et al., 1995).
As with the IAT, we do not just compare two words. Many if not most words have multiple meanings, which makes
pairwise measurements “noisy”. To control for this, we use small baskets of terms to represent a concept. In the present
paper we have never invented our own basket of words, but rather have in every case used the same words as were used in the
psychological study we are replicating. We should note that distances / similarities of word embeddings notoriously lack any
intuitive interpretation. But this poses no problem for us: our results and their import do not depend on attaching meaning to
these distances. Our primary claim is that the associations revealed by relative nearness scores between categories match human
biases and stereotypes strongly (i.e., low p-values and high effect sizes) and across many categories. Thus, the associations in
the word vectors could not have arisen by chance, but instead reflect extant biases in human culture.
2/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
There are however several key differences between our method and the IAT. Most of these we will discuss in future versions
of this paper’s appendices, but one in particular is critical to our presentation of results. While the IAT applies to individual
human subjects, the embeddings of interest to us are derived from the aggregate writings of humans on the web. These corpora
are generated in an uncontrolled fashion and are not representative of any one population (though our results indicate they
may be disproportionately American; see below). The IAT has sometimes been used to draw conclusions about populations by
averaging individual results over samples, and these have led to important insights on racial bias and gender stereotypes, among
others. Our tests of word embeddings are loosely analogous to population-level IATs.
Nevertheless, this difference precludes a direct numerical comparison between human biases measured by the IAT and
algorithmic biases measured by our methods. In particular, an IAT allows rejecting the null hypothesis (of non-association
between two categories) via a p-value and quantification of the strength of association via an effect size. These are obtained
by administering the test to a statistically-significant sample of subjects (and multiple times to each subject). With word
embeddings, there is no notion of test subjects. Roughly, it is as if we are able to measure the mean of the association strength
over all the “subjects” who collectively created the corpora. But we have no way to observe variation between subjects or
between trials. We do report p-values and effect sizes resulting from the use of multiple words in each category, but the meaning
of these numbers is entirely different from those reported in IATs.

Results
Using the techniques described in the Methods section, we have found every linguistic bias documented in psychology that we
have looked for. Below are a sample that we think are persuasive. We have not cherry picked these for effect size—these are
uniformly high. Rather, we chose these to illustrate our assertion that we can account for a variety of implicit human biases
purely from language regularities, and that these are in fact part and parcel with the meaning of language. We demonstrate this
by showing that the same measures that replicate implicit bias also replicate prejudicial hiring practices, and further return
veridical information about employment and naming practices in contemporary America.
We ensure impartiality in our approach by using the benchmarks and keywords established in well-known and heavily cited
works of the human sciences, psychology and sociology. We use a state-of-the-art and widely used word embedding, namely
GloVe, made available by Pennington et al. (2014). We used one of GloVe’s standard semantic models trained on standard
corpora of ordinary language use found on the World Wide Web. We have also found similar results for other standard tools
and corpora, which we will also discuss in future versions of this paper’s appendices.
Following the lead of the IAT, for each result we report the two sets of target concepts about which we are attempting to
learn and the two sets of attribute words we are comparing each to. We then apply our method WEAT (defined in the Methods
section), and report the probability (p-value) that our observed similarity scores could have arisen with no semantic association
between the target concepts and the attribute. We report an effect size based on the number of standard deviations that separate
the two sets of target words in terms of their association with the attribute words; precise details of this measure are described
in the Methods section.
Baseline: Replication of Associations That Are Universally Accepted
The first results presented in the initial publication on the IAT (Greenwald et al., 1998) concerned biases that were found to be
universal in humans and about which there is no social concern. This allows the introduction and clarification of the method
and its validation on relatively morally neutral topics. We begin by replicating these inoffensive results for the same purposes.
Flowers and Insects
Original Finding: Greenwald et al. (1998, p. 1469) report that the IAT is able to demonstrate via reaction times that flowers

are significantly more pleasant than insects, and insects more unpleasant than flowers. Based on the reaction latencies of 32
participants, the IAT results in an effect size1 of 1.35, which is considered a large effect, and a p-value of 10−8 for statistical
significance.
We replicate this finding by looking at semantic similarity in GloVe for the same stimuli by using our WEAT
method. Flowers are more likely than insects to be closer to pleasant than to unpleasant. By applying our method, we observe
the expected association with an effect size of 1.50 and with p-value < 10−7 for statistical significance.2
Notice that GloVe “knows” this property of flowers and insects with no direct experience of the world, and no representation
of semantics other than the implicit metrics of words’ co-occurrence statistics that it is trained on.
Our Finding:

1 Effect size is Cohen’s d, which is the log-transformed mean of latencies in milliseconds divided by the standard deviation. Conventional small, medium,
and large values of d are 0.2, 0.5, and 0.8, respectively.
2 The maximum effect size can be is 2.0.

3/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
Stimuli:

We use the same stimuli as Greenwald et al. (1998, p. 1479).

• Flowers: aster, clover, hyacinth, marigold, poppy, azalea, crocus, iris, orchid, rose, bluebell, daffodil, lilac, pansy, tulip,
buttercup, daisy, lily, peony, violet, carnation, gladiola, magnolia, petunia, zinnia.
• Insects: ant, caterpillar, flea, locust, spider, bedbug, centipede, fly, maggot, tarantula, bee, cockroach, gnat, mosquito,
termite, beetle, cricket, hornet, moth, wasp, blackfly, dragonfly, horsefly, roach, weevil.
• Pleasant: caress, freedom, health, love, peace, cheer, friend, heaven, loyal, pleasure, diamond, gentle, honest, lucky,
rainbow, diploma, gift, honor, miracle, sunrise, family, happy, laughter, paradise, vacation.
• Unpleasant: abuse, crash, filth, murder, sickness, accident, death, grief, poison, stink, assault, disaster, hatred, pollute,
tragedy, divorce, jail, poverty, ugly, cancer, kill, rotten, vomit, agony, prison.
Musical Instruments and Weapons
Original Finding: Similarly, Greenwald et al. (1998, p. 1469) find that musical instruments are significantly more pleasant

than weapons. Based on the reaction latencies of 32 participants, the IAT results in an effect size of 1.66 and a p-value of 10−10 .
We replicate this finding by looking at semantic nearness in GloVe for the same stimuli. Musical instruments
are more likely than weapons to be closer to pleasant than to unpleasant. By applying our method, we observe the expected
association with an effect size of 1.53 and with p-value < 10−7 .
Our Finding:

Stimuli:

We use the same stimuli found in Greenwald et al. (1998, p. 1479).

• Musical instruments: bagpipe, cello, guitar, lute, trombone, banjo, clarinet, harmonica, mandolin, trumpet, bassoon,
drum, harp, oboe, tuba, bell, fiddle, harpsichord, piano, viola, bongo, flute, horn, saxophone, violin.
• Weapons: arrow, club, gun, missile, spear, axe, dagger, harpoon, pistol, sword, blade, dynamite, hatchet, rifle, tank,
bomb, firearm, knife, shotgun, teargas, cannon, grenade, mace, slingshot, whip.
• Pleasant and Unpleasant: as per previous experiment with insects and flowers.
Racial Biases
We now use the same technique to demonstrate that machine learning absorbs prejudice as easily as other biases. Here we
replicate not only the original IAT results on racial prejudice, but also the more recent and striking finding that names alone
have enormous impact on the probability of job candidates being called for an interview.
Replicating Implicit Associations for Valence
Original Finding: Greenwald et al. (1998, p. 1475) find extreme impacts of race as indicated simply by name. A bundle of

names associated with being European American was found to be significantly easier to associate with pleasant than unpleasant
terms, compared to a bundle of African American names. With 26 subjects, Greenwald et al. show that the European American
names are more likely to be implicitly associated as pleasant with an effect size of 1.17 and a p-value of 10−6 .
Our Finding: We were again able to replicate attitude towards two races by looking at semantic nearness in GloVe. We were
forced to slightly alter the stimuli because some of the original African American names did not occur in the corpus with
sufficient frequency. These are shown in italics below. We therefore also deleted the same number of European American
names, chosen at random, to balance the number of elements in the sets of two concepts. In our results, European American
names are more likely than African American names to be closer to pleasant than to unpleasant, with an effect size of 1.41 and
p-value < 10−8 .
Stimuli: We use a subset (see above) of the same stimuli found in Greenwald et al. (1998, p. 1479). Names that are marked
with italics are excluded from our replication.

• European American names: Adam, Chip, Harry, Josh, Roger, Alan, Frank, Ian, Justin, Ryan, Andrew, Fred, Jack,
Matthew, Stephen, Brad, Greg, Jed, Paul, Todd, Brandon, Hank, Jonathan, Peter, Wilbur, Amanda, Courtney, Heather,
Melanie, Sara, Amber, Crystal, Katie, Meredith, Shannon, Betsy, Donna, Kristin, Nancy, Stephanie, Bobbie-Sue, Ellen,
Lauren, Peggy, Sue-Ellen, Colleen, Emily, Megan, Rachel, Wendy (deleted names in italics).
• African American names: Alonzo, Jamel, Lerone, Percell, Theo, Alphonse, Jerome, Leroy, Rasaan, Torrance, Darnell,
Lamar, Lionel, Rashaun, Tvree, Deion, Lamont, Malik, Terrence, Tyrone, Everol, Lavon, Marcellus, Terryl, Wardell,
Aiesha, Lashelle, Nichelle, Shereen, Temeka, Ebony, Latisha, Shaniqua, Tameisha, Teretha, Jasmine, Latonya, Shanise,
Tanisha, Tia, Lakisha, Latoya, Sharise, Tashika, Yolanda, Lashandra, Malika, Shavonn, Tawanda, Yvette (deleted names
in italics).
4/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
• Pleasant: caress, freedom, health, love, peace, cheer, friend, heaven, loyal, pleasure, diamond, gentle, honest, lucky,
rainbow, diploma, gift, honor, miracle, sunrise, family, happy, laughter, paradise, vacation.
• Unpleasant: abuse, crash, filth, murder, sickness, accident, death, grief, poison, stink, assault, disaster, hatred, pollute,
tragedy, bomb, divorce, jail, poverty, ugly, cancer, evil, kill, rotten, vomit.
Replicating the Bertrand and Mullainathan (2004) Résumé Study
Original Finding: Bertrand and Mullainathan (2004) sent nearly 5,000 identical résumés to 1,300 job advertisements with

only one change made to the résumés: the names of the candidates. They found that European American candidates were 50%
more likely to be offered an opportunity to be interviewed.
Our Finding: Perhaps unsurprisingly, we again found a significant result for the names used by Bertrand and Mullainathan.
As before, we had to delete some low-frequency names. We also assumed semantic nearness to pleasantness as the correlate for
an invitation to interview. We did this with two different sets of ‘pleasant/unpleasant’ stimuli: those from the original IAT paper,
and also a revised shorter set used more recently, found in Nosek et al. (2002a). For both sets of attributes, European American
names are more likely than African American names to be invited for interviews (closer to pleasant than to unpleasant). Using
the Greenwald et al. (1998) attributes, the effect size is 1.50 and p-value < 10−4 ; and using the updated Nosek et al. (2002a)
attributes, the effect size is 1.28 and p-value < 10−3 .

For the names we use the stimuli found in Bertrand and Mullainathan (2004, p. 1012). The first set of pleasant and
unpleasant words are as per above, the second are from Nosek et al. (2002a, p. 114).

Stimuli:

• European American names: Brad, Brendan, Geoffrey, Greg, Brett, Jay, Matthew, Neil, Todd, Allison, Anne, Carrie,
Emily, Jill, Laurie, Kristen, Meredith, Sarah (deleted names in italics).
• African American names: Darnell, Hakim, Jermaine, Kareem, Jamal, Leroy, Rasheed, Tremayne, Tyrone, Aisha, Ebony,
Keisha, Kenya, Latonya, Lakisha, Latoya, Tamika, Tanisha (deleted names in italics).
• First set of Pleasant and Unpleasant: as per previous experiment with African American and European American
names.
• updated Pleasantness: joy, love, peace, wonderful, pleasure, friend, laughter, happy.
• updated Unpleasantness: agony, terrible, horrible, nasty, evil, war, awful, failure.
Gender Biases
We now turn to gender-related biases and stereotypes. We begin by returning to prejudice as demonstrated by the IAT, but then
we will turn to matching the biases we data mine against veridical information taken from published U.S. government statistics.
Replicating Implicit Associations for Career and Family

Whether or not it is appropriate for women to have careers has been a matter of significant cultural dispute. Historically, the
consensus was that they should not; today, most but by no means all Americans consider it as appropriate for a woman to have
a career as a man. Similarly, there have been historical biases against men who choose to take domestic roles. The IAT study
we compare to here was conducted online, and thus has a vastly larger subject pool. However, since there is more difficulty
ensuring that online subjects will complete their task with attention, it also has far fewer keywords examined. We are able to
replicate the results even with these reduced keyword sets.
With 38,797 interpretable subjects (those who fully completed the test), female names were found to be
more associated with family than career words with an effect size of 0.72 and p-value < 10−2 , Nosek et al. (2002a, p. 105).
Original Finding:

Our Finding: We found the same result that females are more associated with family and males with career, with an effect
size of 1.81 and p-value < 10−3 .
Stimuli:

We use the same stimuli found in Nosek et al. (2002a, p. 114).

• Male names: John, Paul, Mike, Kevin, Steve, Greg, Jeff, Bill.
• Female names: Amy, Joan, Lisa, Sarah, Diana, Kate, Ann, Donna.
• Career words : executive, management, professional, corporation, salary, office, business, career.
• Family words : home, parents, children, family, cousins, marriage, wedding, relatives.
5/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
Replicating Implicit Associations for Arts and Mathematics

In a similar result, both Nosek et al. and we find that female terms are more associated with arts than mathematics, compared to
male terms.
Original Finding: 28,108 subjects completed the online IAT and female terms were more associated with arts than mathematics
with an effect size of 0.82 and p-value < 10−2 , Nosek et al. (2002a, p. 105).
Our Finding:
Stimuli:

We found the expected association with an effect size of 1.06 and a p-value of 10−2 .

We use the stimuli found in Nosek et al. (2002a, p. 114).

• Math words : math, algebra, geometry, calculus, equations, computation, numbers, addition.
• Arts Words : poetry, art, dance, literature, novel, symphony, drama, sculpture.
• Male attributes: male, man, boy, brother, he, him, his, son.
• Female attributes: female, woman, girl, sister, she, her, hers, daughter.
Replicating Implicit Associations for Arts and Sciences

In another laboratory study, Nosek et al. (2002b) found that female terms are less associated with the sciences, and male terms
less associated with the arts.
83 subjects took the IAT with a combination of math/science and art/language attributes, and the expected
associations were observed with an effect size of 1.47 and a p-value of 10−24 , Nosek et al. (2002b, p. 51).
Original Finding:

By examining only arts and sciences attributes, we found that female terms were associated more with arts and
male terms with science with an effect size of 1.24 and a p-value of 10−2 .

Our Finding:
Stimuli:

We use the stimuli found in Nosek et al. (2002b, p. 59).

• Science words : science, technology, physics, chemistry, Einstein, NASA, experiment, astronomy.
• Arts words : poetry, art, Shakespeare, dance, literature, novel, symphony, drama.
• Male attributes: brother, father, uncle, grandfather, son, he, his, him.
• Female attributes: sister, mother, aunt, grandmother, daughter, she, hers, her.
Comparison to Real-World Data: Occupational Statistics

It has been suggested that implicit gender-occupation biases are linked to gender gaps in occupational participation (Nosek
et al., 2009); however the relationship between these is complex and may be mutually reinforcing. Here we examine the
correlation between the gender association of occupation words and labor-force participation data.
The x-axis of Figure 1 is derived from the 2015 U.S. Bureau of Labor Statistics3 , which provides information
about occupational categories and the percentage of women that have certain occupations under these categories. We generated
single word occupation names (as explained in the Methods section) based on the available data and calculated the percentage
of women for the set of single word occupation names.
Original Data:

By applying WEFAT, we are able to use word embeddings to predict the percentage of women in the 50 most
relevant occupations with a Pearson’s correlation coefficient of ρ = 0.90 with p-value < 10−18 .
Our Finding:

We use the gender stimuli found in Nosek et al. (2002a, p. 114) along with the occupation attributes we derived from
labor statistics.
Stimuli:

• Careers : technician, accountant, supervisor, engineer, worker, educator, clerk, counselor, inspector, mechanic, manager,
therapist, administrator, salesperson, receptionist, librarian, advisor, pharmacist, janitor, psychologist, physician, carpenter,
nurse, investigator, bartender, specialist, electrician, officer, pathologist, teacher, lawyer, planner, practitioner, plumber,
instructor, surgeon, veterinarian, paramedic, examiner, chemist, machinist, appraiser, nutritionist, architect, hairdresser,
baker, programmer, paralegal, hygienist, scientist.
• Female attributes: female, woman, girl, sister, she, her, hers, daughter.
• Male attributes: male, man, boy, brother, he, him, his, son.
3 http://www.bls.gov/cps/cpsaat11.htm

6/14

Strength of association of
occupation word vector with female gender

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
2

1

0

−1

−2

0
20
40
80
100
60
Percentage of workers in occupation who are women

Figure 1. Occupation-gender association
Pearson’s correlation coefficient ρ = 0.90 with p-value < 10−18 .

Comparison to Real-World Data: Androgynous Names

Similarly, we looked at the veridical association of gender to androgynous names, that is, names sometimes used by either
gender. In this case, the most recent information we were able to find was the 1990 census name and gender statistics. Perhaps
because of the age of our name data, our correlation was weaker than for the 2015 occupation statistics, but still strikingly
significant.

Strength of association of
name vector with female gender

2

1

0

−1

−2

0

20
40
80
100
60
Percentage of people with name who are women

Figure 2. People with androgynous names
Pearson’s correlation coefficient ρ = 0.84 with p-value < 10−13 .

The x-axis of Figure 2 is derived from the 1990 U.S. census data4 that provides first name and gender
information in population.
Original Data:

Our Finding: The y-axis reflects our calculation of the bias for how male or female each of the names is. By applying WEFAT,
we are able to predict the percentage of people with a name who were women with Pearson’s correlation coefficient of ρ = 0.84
4 http://www.census.gov/main/www/cen1990.html

7/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.

Figure 3. A 2D projection (first
two principal components) of the
300-dimensional vector space of
the GloVe word embedding
(Pennington et al., 2014). The lines
illustrate algebraic relationships
between related words: pairs of
words that differ only by gender
map to pairs of vectors whose
vector difference is roughly
constant. Similar algebraic
relationships have been shown for
other semantic relationships, such
as countries and their capital cities,
companies and their CEOs, or
simply different forms of the same
word.
with p-value < 10−13 .
Stimuli: We use the gender stimuli found in Nosek et al. (2002a, p. 114) along with the most popular androgynous names
from 1990’s public census data as targets.

• Names : Kelly, Tracy, Jamie, Jackie, Jesse, Courtney, Lynn, Taylor, Leslie, Shannon, Stacey, Jessie, Shawn, Stacy, Casey,
Bobby, Terry, Lee, Ashley, Eddie, Chris, Jody, Pat, Carey, Willie, Morgan, Robbie, Joan, Alexis, Kris, Frankie, Bobbie,
Dale, Robin, Billie, Adrian, Kim, Jaime, Jean, Francis, Marion, Dana, Rene, Johnnie, Jordan, Carmen, Ollie, Dominique,
Jimmie, Shelby.
• Female and Male attributes: as per previous experiment on occupations.

Methods
Data and training
A word embedding is a representation of words as points in a vector space. Loosely, embeddings satisfy the property that vectors
that are close to each other represent semantically “similar” words. Word embeddings derive their power from the discovery
that vector spaces with around 300 dimensions suffice to capture most aspects of similarity, enabling a computationally tractable
representation of all or most words in large corpora of text (Bengio et al., 2003; Lowe, 1997). Starting in 2013, the word2vec
family of word embedding techniques has gained popularity due to a new set of computational techniques for generating word
embeddings from large training corpora of text, with superior speed and predictive performance in various natural-language
processing tasks (Mikolov et al., 2013; Mikolov and Dean, 2013).
Most famously, word embeddings excel at solving “word analogy” tasks because the algebraic relationships between
vectors capture syntactic and semantic relationships between words (Figure 3). In addition, word embeddings have found use
in natural-language processing tasks such as web search and document classification. They have also found use in cognitive
science for understanding human memory and recall (Zaromb et al., 2006; McDonald and Lowe, 1998).
For all results in this paper we use the state-of-the-art GloVe word embedding method, in which, at a high level, the similarity
between a pair of vectors is related to the probability that the words co-occur close to each other in text (Pennington et al., 2014).
Word embedding algorithms such as GloVe substantially amplify the signal found in simple co-occurrence probabilities using
dimensionality reduction. In pilot-work experiments along the lines of those presented here (on free associations rather than
implicit associations) raw co-occurrence probabilities were shown to lead to substantially weaker results (Macfarlane, 2013).
Rather than train the embedding ourselves, we use pre-trained GloVe embeddings distributed by its authors. We aim to
replicate the effects that may be found in real applications to the extent possible, and using pre-trained embeddings minimizes
8/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
the choices available to us and simplifies reproducing our results. We pick the largest of the four corpora for which the
GloVe authors provide trained embeddings, which is a “Common Crawl” corpus obtained from a large-scale crawl of the web,
containing 840 billion tokens (roughly, words). Tokens in this corpus are case-sensitive and there are 2.2 million different ones,
each corresponding to a 300-dimensional vector. The large size of this corpus and the resulting model is important to us, since
it enables us to find word vectors for even relatively uncommon names. An important limitation is that there are no vectors for
multi-word phrases.
We can expect similar results to the ones presented here if we used other corpora and/or embedding algorithms. For example,
we repeated all the WEAT and WEFAT experiments presented above using a different pre-trained embedding: word2vec on
a Google News corpus (Mikolov and Dean, 2013). In all experiments, we observed statistically significant effects and high
effect sizes. Further, we found that the gender association strength of occupation words is highly correlated between the GloVe
embedding and the word2vec embedding (Pearson ρ = 0.88; Spearman ρ = 0.86). In concurrent work, Bolukbasi et al. (2016)
compared the same two embeddings, using a different measure of the gender bias of occupation words, also finding a high
correlation (Spearman ρ = 0.81).
Word Embedding Association Test (WEAT)
To demonstrate and quantify bias, we use the permutation test. Borrowing terminology from the IAT literature, consider two sets
of target words (e.g., programmer, engineer, scientist, ... and nurse, teacher, librarian, ...) and two sets of attribute words
(e.g., man, male, ... and woman, female ...). The null hypothesis is that there is no difference between the two sets of target
words in terms of their relative similarity to the two sets of attribute words. The permutation test measures the (un)likelihood of
the null hypothesis by computing the probability that a random permutation of the attribute words would produce the observed
(or greater) difference in sample means.
In formal terms, let X and Y be two sets of target words of equal size, and A, B the two sets of attribute words. Let cos(~a,~b)
denote the cosine of the angle between the vectors ~a and ~b.
• The test statistic is
s(X,Y, A, B) =

∑ s(x, A, B) − ∑ s(y, A, B)

x∈X

y∈Y

where
s(w, A, B) = meana∈A cos(~w,~a) − meanb∈B cos(~w,~b)
In other words, s(w, A, B) measures the association of the word w with the attribute, and s(X,Y, A, B) measures the
differential association of the two sets of target words with the attribute.
• Let {(Xi ,Yi )}i denote all the partitions of X ∪Y into two sets of equal size. The one-sided p-value of the permutation test
is
Pri [s(Xi ,Yi , A, B) > s(X,Y, A, B)]
• The effect size is

meanx∈X s(x, A, B) − meany∈Y s(y, A, B)
std-devw∈X∪Y s(w, A, B)

It is a normalized measure of how separated the two distributions (of associations between the target and attribute) are.
We re-iterate that these p-values and effect sizes don’t have the same interpretation as the IAT. The “subjects” in our
experiments are words, not people. While the IAT can measure the differential association between a single pair of target
concepts and an attribute, the WEAT can only measure the differential association between two sets of target concepts and an
attribute.
Word Embedding Factual Association Test (WEFAT)
To understand and demonstrate the necessity of human bias in word embeddings, we also wish to examine how word embeddings
capture empirical information about the world, which is also embedded in language. Consider a set of target concepts, such as
occupations, and a real-valued, factual property of the world associated with each concept, such as the percentage of workers in
the occupation who are women. We’d like to test if the vectors corresponding to the concepts embed knowledge of the property,
that is, if there is an algorithm that can extract or predict the property given the vector. In principle we could use any algorithm,
but in this work we test the association of the target concept with some set of attribute words, analogous to WEAT above.
Formally, consider a single set of target words W and two sets of attribute words A, B. There is a property pw associated
with each word w ∈ W .
9/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
• The statistic associated with each word vector is a normalized association score of the word with the attribute:

s(w, A, B) =

meana∈A cos(~w,~a) − meanb∈B cos(~w,~b)
std-devx∈A∪B cos(~w,~x)

• The null hypothesis is that there is no association between s(w, A, B) and pw . We test the null hypothesis using a linear
regression analysis to predict the latter from the former.
Now we discuss in more detail how we apply WEFAT in two cases. The first is to test if occupation word vectors embed
knowledge of the gender composition of the occupation in the real world. We use data released by the Bureau of Labor Statistics
in which occupations are categorized hierarchically, and for each occupation the number of workers and percentage of women
are given (some data is missing). The chief difficulty is that many occupation names are multi-word terms whereas word vectors
represent single words. Our strategy is to convert a multi-word term into a single word that represents a superset of the category
(e.g., chemical engineer → engineer), and filter out occupations where this is not possible.
Our second application of WEFAT is to test if androgynous names embed knowledge of how often the name is given to
boys versus girls. We picked the most popular names in each 10% window of gender frequency based on 1990 U.S. Census
data. Here again there is a difficulty: some names are also regular English words (e.g., Will). State-of-the-art word embeddings
are not yet sophisticated enough to handle words with multiple senses or meanings; all usages are lumped into a single vector.
To handle this, we algorithmically determine how “name-like” each vector is (by computing the distance of each vector to the
centroid of all the name vectors), and eliminate the 20% of vectors that are least name-like.
We plan to make the code used to generate our results publicly available.5

Discussion
We have shown that machine learning can acquire prejudicial biases from training data that reflect historical injustice. This is
not entirely a new finding. A recent line of work on fairness in machine learning tries to minimize or avoid such biases (Dwork
et al., 2012; Feldman et al., 2015; Zemel et al., 2013; Barocas and Selbst, 2014). However, unlike this literature, our setting is
not a particular, explicit decision-making task (known as “classification” in machine learning), but rather the often unconscious
consequences of all of language. We show for the first time that if AI is to exploit via our language the vast knowledge that
culture has compiled, it will inevitably inherit human-like prejudices. In other words, if AI learns enough about the properties
of language to be able to understand and produce it, it also acquires cultural associations that can be offensive, objectionable, or
harmful. These are much broader concerns than intentional discrimination, and possibly harder to address. This distinction
informs much of the rest of this section.
Implications for understanding human prejudice
The simplicity and strength of our results suggests a new null hypothesis for explaining origins of prejudicial behavior in
humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing
an explicit or institutional explanation for why individuals make decisions that disadvantage one group with regards to another,
one must show that the unjust decision was not a simple outcome of unthinking reproduction of statistical regularities absorbed
with language. Similarly, before positing complex models for how prejudicial attitudes perpetuate from one generation to
the next or from one group to another, we must check whether simply learning language is sufficient to explain the observed
transmission of prejudice. These new null hypotheses are important not because we necessarily expect them to be true in
most cases, but because Occam’s razor now requires that we eliminate them, or at least quantify findings about prejudice in
comparison to what is explainable from language transmission alone.
Our work lends credence to the highly parsimonious theory that all that is needed to create prejudicial discrimination is not
malice towards others, but preference for one’s ingroup (Greenwald and Pettigrew, 2014). This theory is also supported by
recent results showing that in times of conflict, rather than an increase in ingroup altruism, we see a decrease in baseline altruism
towards the outgroup (Silva and Mace, 2015). Our results also both explain and support empirical results from education
indicating that reducing prejudice requires directed interventions to facilitate “decategorizing and recategorizing outgroups”
(Dessel, 2010, p. 411). Simple contact with members of other groups is not enough. There needs to be specific bridging
experiences to facilitating the construction of new identities or to develop skills to work with people across group boundaries.
It has been known for some time that even newborn infants attend foremost to speakers sharing their mother’s dialect
(Kinzler et al., 2007); it has been conjectured that such ingroup signaling may even account for the origins of music and
language (Fitch, 2004). What we have shown here is that language identifies not only one’s own group, but also which group
5 We

thank Will Lowe for assistance with the methods WEAT and WEFAT that greatly improved the methodology.

10/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
is currently culturally dominant, or dominates particular regions of a culture. This may account for why in the IAT, Koreans
and Japanese people living in their own countries each associate the other as ‘less pleasant,’ but African Americans show
European-American-oriented biases, though not as strongly as European Americans (Greenwald et al., 1998). The dominance
of European-American orientation may change as the American demography changes; indeed it would be interesting to examine
corpora consisting of newspapers or other public language in towns or cities with different demographic makeup, particularly
where racial diversity is also represented consistently in public offices and media.
Of course, neither our work nor any other theory explaining the origins of prejudice justify prejudiced behavior. Humans
are (or can be) good at using explicit knowledge to better cooperate, including choosing to behave fairly. Lee (2016) has shown
very recently that the level of implicit bias displayed by subjects in the IAT does not predict cooperative performance. In other
words, the learned biases that affect rate of comprehension of test stimuli or construction of artificial pairings does not affect
deliberate choices about how to treat others, at least not in a laboratory setting. However, we have demonstrated here that
a known case of prejudicial decision making (about inviting job candidates, cf. Bertrand and Mullainathan, 2004) can be
replicated by biases latent in language. We therefore recommend continuing the program of research examining behavior that
does and does not correlate to human subject performance in the IAT. We recommend using our text processing tools to check
pilot predictions for likely IAT performance on comparisons where none is currently known.
Consequences of bias in humans and machines
We have shown that AI can and does inherit substantially the same biases that humans exhibit. However, the consequences of
bias are different in humans and machine-learning systems. Bias in AI is important because AI is increasingly given agency in
our society, for tasks ranging from predictive text in search to determining criminal sentences assigned by courts. Yet machines
are artifacts, owned and controlled by humans operators. That means that learning can be shut off completely once a product is
put into production or operation, and this is frequently done to create more efficient and uniform experiences. Such an approach
opens a potential downside: we may enshrine an imperfect procedure in a context where it will not be routinely reexamined by
other humans. Such artifacts could persist and perpetuate biases in society for a long time — digital analogs of Robert Moses’s
racially motivated overpasses (Winner, 1980).
One advantage of AI, at least where the algorithms and outcomes are open to inspection, is that it can at least make such
errors explicit and therefore potentially subject to monitoring and correction. After all, the same dependencies on history we
have uncovered here may very well also pollute individual expectations, public policy, and even law. Natural intelligence and
learning, just as in artifacts, may pick upon correlations without considering sufficiently carefully whether there is any causal
relationship, or whether the correlation is caused by some other unobserved factor, possibly a correctable injustice.
Effects of bias in NLP applications
To better understand the potential impact of bias in word embeddings, let us consider applications where they have found
use. Sentiment analysis classifies text as being positive, negative, or neutral. Two of its uses are in marketing to quantify
customer satisfaction (say, from a set of product reviews) and in finance to predict market trends (say, from tweets about
companies). Consider a straw-man sentiment analysis technique based on word embeddings: calculate the valence of each
word based on its association with designated positive and negative words, then sum up the sentiment scores. Now consider
applying this technique to movie reviews. Our results show that European-American names have more positive valence than
African-American names in a state-of-the-art word embedding. That means a sentence containing a European-American name
will have a higher sentiment score than a sentence with that name replaced by an African-American name. In other words, the
tool will display a racial bias in its output based on actor and character names.
We picked this example because the argument follows directly from our experiments on names. But our results suggest that
other imprints of human racial prejudice, not confined to names, will also be picked up by machine-learning models. Besides,
bias is known to creep in indirectly, by proxy (Barocas and Selbst, 2014). Thus, it would be simplistic to conclude that we can
fix the problem by withholding names from the inputs to NLP applications.
Next, consider statistical machine translation (SMT). Unsurprisingly, today’s SMT systems reflect existing gender stereotypes. Translations to English from many gender-neutral languages such as Finnish, Estonian, Hungarian, Persian, and Turkish
lead to gender-stereotyped sentences. For example, Google Translate converts these Turkish sentences with genderless pronouns:
“O bir doktor. O bir hemşire.” to these English sentences: “He is a doctor. She is a nurse.” A test of the 50 occupation words
used in the results presented in Figure 1 shows that the pronoun is translated to “he” in the majority of cases and “she” in about
a quarter of cases; tellingly, we found that the gender association of the word vectors almost perfectly predicts which pronoun
will appear in the translation.
Challenges in addressing bias
Redresses such as transparent development of AI technology and improving diversity and ethical training of developers, while
useful, do little to address the kind of prejudicial bias we expose here. Unfortunately, our work points to several additional
11/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
reasons why addressing bias in machine learning will be harder than one might expect. First, our results suggest that word
embeddings don’t merely pick up specific, enumerable biases such as gender stereotypes (Bolukbasi et al., 2016), but rather the
entire spectrum of human biases reflected in language. In fact, we show that bias is meaning. Bias is identical to meaning, and
it is impossible to employ language meaningfully without incorporating human bias. This is why we term unacceptable bias
prejudice in this paper. The biases we reveal aren’t about a particular application of machine learning, but rather about the basic
representation of knowledge — used possibly in human cognition, and certainly in an expanding variety of AI applications.
Second, the idea of correcting even prejudiced biases is also problematic. That is because societal understanding of prejudice
is constantly evolving, along with our understanding of humanity and human rights, and also varies between cultures. It is
therefore hard or impossible to specify algorithmically what is prejudiced. To give one example, Monteith and Pettit (2011)
using the IAT show that people with mental illnesses are stigmatized compared to people with physical illnesses — a result we
have also replicated in word embeddings (but not reported above). Is this a prejudice? Who determines whether it should be
corrected?
Third and finally, we have shown that biases result from extant as well as historic inequalities in the world. There may
be many other contexts where these inequalities are important to know about. More generally, shared awareness of the real
world is important for communication (Zue, 1985; Barsalou, 2009). Consider the gender stereotypes in occupations. If we
were using machine learning to evaluate the suitability of job applicants, these stereotypes would be bad. Yet if the task was
to analyze historical job ads and infer if more men or women worked in those roles, the stereotypical associations would be
exactly the information we would wish to utilize. The gender associations we found in the word embeddings of names might
be exceedingly useful, yet those same associations might lead to prejudicial expectations concerning names and occupations.
Remedies must be tailored to applications. Within a given context, such as college admissions, we can decide whether (and to
what extent) considerations of fairness should override the usual focus on predictive accuracy (as is the case with affirmative
action), but it is not meaningful to do this devoid of context. Put simply, eliminating bias is eliminating information; eliminating
prejudice takes more thought.
Awareness is better than blindness
For these reasons, we view the approach of “debiasing” word embeddings (Bolukbasi et al., 2016) with skepticism. If we view
AI as perception followed by action, debiasing alters the AI’s perception (and model) of the world, rather than how it acts on
that perception. This gives the AI an incomplete understanding of the world. We see debiasing as “fairness through blindness”.6
It has its place, but also important limits: prejudice can creep back in through proxies (although we should note that Bolukbasi
et al. (2016) do consider “indirect bias” in their paper). Efforts to fight prejudice at the level of the initial representation will
necessarily hurt meaning and accuracy, and will themselves be hard to adapt as societal understanding of fairness evolves.
Instead, we take inspiration from the fact that humans can express behavior different from their implicit biases (Lee, 2016).
Human intelligence is typified by behavior integrating multiple forms of memory and evidence (Purcell and Kiani, 2016; Bear
and Rand, 2016). It includes the capacity to recall one-shot exposure to highly context-specific information in the form of rules
and instructions. We can learn that “prejudice is bad”, that “women used to be trapped in their homes and men in their careers,
but now gender doesn’t necessarily determine family role” and so forth. If AI is not built in a similar way, then it would be
possible for prejudice absorbed by machine learning to have a much greater negative impact than when prejudice is absorbed in
the same way by children. This is because children also receive other kinds of instruction and social examples as a part of the
ordinary, painstaking process of child rearing. Normally when we design AI architectures, we try to keep them as simple as
possible to facilitate our capacity to debug and maintain AI systems. However, where AI is partially constructed automatically
by machine learning of human culture, we may also need an analog of human explicit memory and deliberate actions, that can
be trained or programmed to avoid the expression of prejudice.
Of course, such an approach doesn’t lend itself to a straightforward algorithmic formulation. Instead it requires a long-term,
interdisciplinary research program that includes cognitive scientists and ethicists. One concrete suggestion for the present is to
choose corpora for training machine learning to have as little prejudice as possible — the tools we have presented here can be
used to identify these. Another is that given the vulnerability of relying on purely statistical information for understanding and
operating within a culture, it may be advisable to consider more complex AI architectures such as cognitive systems (Thórisson,
2007; Hanheide et al., 2015). Heterogeneous approaches to representing knowledge and intelligence may allow us to exploit
both the great strengths of machine learning and the instructability of symbolic systems.
Acknowledgements
We are grateful to the following people: Will Lowe for substantial assistance in the design of our significance tests, Tim
Macfarlane for his pilot research as a part of his undergraduate dissertation, Solon Barocas and Miles Brundage for excellent
comments on an early version of this paper.
6 Our

use of this term is inspired by Dwork et al. (2012), but we use it slightly differently, and our argument is different from theirs.

12/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.

References
Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). Machine bias: There’s software used across the country to predict
future criminals. and it’s biased against blacks. ProPublica, May, 23.
Barocas, S. and Selbst, A. D. (2014). Big data’s disparate impact. California Law Review, 104.
Barr, A. (2015). Google mistakenly tags black people as ‘gorillas,’ showing limits of algorithms. The New York Times.
Barsalou, L. W. (2009). Simulation, situated conceptualization, and prediction. Philosophical Transactions of the Royal Society
B: Biological Sciences, 364(1521):1281–1289.
Bear, A. and Rand, D. G. (2016). Intuition, deliberation, and the evolution of cooperation. Proceedings of the National Academy
of Sciences, 113(4):936–941.
Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. journal of machine
learning research, 3(Feb):1137–1155.
Bertrand, M. and Mullainathan, S. (2004). Are emily and greg more employable than lakisha and jamal? a field experiment on
labor market discrimination. The American Economic Review, 94(4):991–1013.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer, London.
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., and Kalai, A. (2016). Man is to computer programmer as woman is to
homemaker? debiasing word embeddings. arXiv preprint arXiv:1607.06520.
Crawford, K. (2016). Artificial intelligence’s white guy problem. The New York Times.
Dessel, A. (2010). Prejudice in schools: Promotion of an inclusive culture and climate. Education and Urban Society,
42(4):407–429.
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012). Fairness through awareness. In Proceedings of the 3rd
Innovations in Theoretical Computer Science Conference, pages 214–226. ACM.
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015). Certifying and removing
disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data
Mining, pages 259–268. ACM.
Fitch, W. T. (2004). Kin selection and ‘mother tongues’: A neglected component in language evolution. In Oller, D. K. and
Griebel, U., editors, Evolution of Communication Systems: A Comparative Approach, pages 275–296. MIT Press, Cambridge,
MA.
Greenwald, A. G., McGhee, D. E., and Schwartz, J. L. (1998). Measuring individual differences in implicit cognition: the
implicit association test. Journal of personality and social psychology, 74(6):1464.
Greenwald, A. G. and Pettigrew, T. F. (2014). With malice toward none and charity for some: Ingroup favoritism enables
discrimination. American Psychologist, 69(7):669.
Hanheide, M., Göbelbecker, M., Horn, G. S., Pronobis, A., Sjöö, K., Aydemir, A., Jensfelt, P., Gretton, C., Dearden, R., Janicek,
M., Zender, H., Kruijff, G.-J., Hawes, N., and Wyatt, J. L. (2015). Robot task planning and explanation in open and uncertain
worlds. Artificial Intelligence, pages –. in press.
Kiefer, A. K. and Sekaquaptewa, D. (2007). Implicit stereotypes and women’s math performance: How implicit gender-math
stereotypes influence women’s susceptibility to stereotype threat. Journal of Experimental Social Psychology, 43(5):825–832.
Kinzler, K. D., Dupoux, E., and Spelke, E. S. (2007). The native language of social cognition. Proceedings of the National
Academy of Sciences, 104(30):12577–12580.
Lee, D. J. (2016). Racial bias and the validity of the implicit association test. Technical Report 35, Helsinki, Finland.
Lowe, W. (1997). Meaning and the mental lexicon. In Proceedings of the 15th International Joint Conference on Artificial
Intelligence, pages 1092–1097, Nagoya. Morgan Kaufmann.
Macfarlane, T. (2013). Extracting semantics from the enron corpus.
McDonald, S. and Lowe, W. (1998). Modelling functional priming and the associative boost. In Proceedings of the Twentieth
Annual Conference of the Cognitive Science Society, pages 667–680. LEA.
Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv
preprint arXiv:1301.3781.
Mikolov, T. and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in
neural information processing systems.
Monteith, L. L. and Pettit, J. W. (2011). Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal
of Social and Clinical Psychology, 30(5):484.
Moss, H. E., Ostrin, R. K., Tyler, L. K., and Marslen-Wilson, W. D. (1995). Accessing different types of lexical semantic
information: Evidence from priming. Journal of Experimental Psychology: Learning, Memory and Cognition, 21:863–883.
Noble, S. U. (2013). Google search: Hyper-visibility as a means of rendering black women and girls invisible. InVisible
Culture, 19.
Nosek, B. A., Banaji, M., and Greenwald, A. G. (2002a). Harvesting implicit group attitudes and beliefs from a demonstration
13/14

Outdated draft. See published version at http://opus.bath.ac.uk/55288/.
web site. Group Dynamics: Theory, Research, and Practice, 6(1):101.
Nosek, B. A., Banaji, M. R., and Greenwald, A. G. (2002b). Math= male, me= female, therefore math6= me. Journal of
personality and social psychology, 83(1):44.
Nosek, B. A., Smyth, F. L., Sriram, N., Lindner, N. M., Devos, T., Ayala, A., Bar-Anan, Y., Bergh, R., Cai, H., Gonsalkorale,
K., et al. (2009). National differences in gender–science stereotypes predict national sex differences in science and math
achievement. Proceedings of the National Academy of Sciences, 106(26):10593–10597.
Oswald, M. and Grace, J. (2016). Norman stanley fletcher and the case of the proprietary algorithmic risk assessment. Policing
Insight.
Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In EMNLP, volume 14,
pages 1532–43.
Purcell, B. A. and Kiani, R. (2016). Hierarchical decision processes that operate over distinct timescales underlie choice and
changes in strategy. Proceedings of the National Academy of Sciences, 113(31):E4531–E4540.
Quine, W. V. O. (1960). Word and Object. MIT Press, Cambridge, MA.
Silva, A. S. and Mace, R. (2015). Inter-group conflict and cooperation: field experiments before, during and after sectarian riots
in northern ireland. Frontiers in Psychology, 6:1790.
Stanley, D. A., Sokol-Hessner, P., Banaji, M. R., and Phelps, E. A. (2011). Implicit race attitudes predict trustworthiness
judgments and economic trust decisions. Proceedings of the National Academy of Sciences, 108(19):7710–7715.
Sweeney, L. (2013). Discrimination in online ad delivery. Queue, 11(3):10:10–10:29.
Thórisson, K. R. (2007). Integrated A.I. systems. Minds and Machines, 17(1):11–25.
Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236):433–460.
Winner, L. (1980). Do artifacts have politics? Daedalus, pages 121–136.
Zaromb, F. M., Howard, M. W., Dolan, E. D., Sirotin, Y. B., Tully, M., Wingfield, A., and Kahana, M. J. (2006). Temporal
associations and prior-list intrusions in free recall. Journal of Experimental Psychology: Learning, Memory, and Cognition,
32(4):792.
Zemel, R. S., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013). Learning fair representations. ICML (3), 28:325–333.
Zue, V. W. (1985). The use of speech knowledge in automatic speech recognition. Proceedings of the IEEE, 73(11):1602–1615.

14/14

