Think you have Solved Question Answering?
Try ARC, the AI2 Reasoning Challenge
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,
Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord

arXiv:1803.05457v1 [cs.AI] 14 Mar 2018

Allen Institute for Artificial Intelligence, Seattle, WA, U.S.A.
{peterc,isaacc,orene,tushark,ashishs,carissas,oyvindt}@allenai.org
Abstract
We present a new question set, text corpus, and baselines
assembled to encourage AI research in advanced question
answering. Together, these constitute the AI2 Reasoning
Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD
or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains
only questions answered incorrectly by both a retrieval-based
algorithm and a word co-occurence algorithm. The dataset
contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set
of this kind (7,787 questions). We test several baselines on
the Challenge Set, including leading neural models from the
SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult
nature of this task. We are also releasing the ARC Corpus, a
corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can
your model perform better? We pose ARC as a challenge to
the community.

Introduction
Datasets are increasingly driving progress in AI, resulting
in impressive solutions to several question-answering (QA)
tasks (e.g., Rajpurkar et al., 2016; Joshi et al., 2017). However, many of these datasets focused on retrieval-style tasks,
where surface-level cues alone were usually sufficient to
identify an answer. This has not encouraged progress on
questions requiring reasoning, use of commonsense knowledge, or other advanced methods for deeper text comprehension. The challenge presented here, called ARC (AI2 Reasoning Challenge), aims to address this limitation by posing
questions that are hard to answer with simple baselines.
The ARC Dataset consists of a collection of 7787 natural science questions, namely questions authored for use
on standardized tests. Standardized tests have previously
been proposed as a Grand Challenge for AI (Brachman,
2005; Clark and Etzioni, 2016) as they involve a wide variety of linguistic and inferential phenomena, have varying
levels of difficulty, and are measurable, motivating, and ambitious. However, making this challenge a reality is difficult, as such questions are difficult obtain (most examination boards release only limited practice tests to the public).

For ARC we have addressed this through several months of
extensive search and investigation.
In addition, to encourage focus on advanced phenomena,
we have partitioned ARC into a Challenge Set (2590 questions), containing questions answered incorrectly by both
a retrieval-based algorithm and a word co-occurrence algorithm, and an Easy Set (5197 questions), containing the remainder. For example, two typical challenge questions are:
Which property of a mineral can be determined just by
looking at it? (A) luster [correct] (B) mass (C) weight
(D) hardness
A student riding a bicycle observes that it moves
faster on a smooth road than on a rough road. This
happens because the smooth road has (A) less gravity
(B) more gravity (C) less friction [correct] (D) more
friction
Both these questions are difficult to answer via simple retrieval or word correlation. For example, there are no Web
sentences of the form “luster can be determined by looking at something”; similarly, “mineral” is strongly correlated with “hardness” (an incorrect answer option). Rather,
they require more advanced QA methods. We provide more
example questions, and a catagorization of the knowledge
and reasoning types that the questions appeal to, in Tables 4
and 5.
To help the community to engage with this dataset, we are
also releasing a science text corpus and two baseline neural
models as part of the ARC challenge:
1. The ARC Corpus, containing 14M science-related sentences with knowledge relevant to ARC. A sampled analysis suggests the corpus mentions knowledge relevant to
95% of the Challenge questions. (Use of the Corpus for
the Challenge is optional).
2. Three neural baseline models, DecompAttn, BiDAF, and
DGEM, for QA. These are multiple-choice QA adaptations of three neural models: the decomposable attention
model (Parikh et al., 2016), a top performer on SNLI;
Bidirectional Attention Flow (Seo et al., 2017b), a top performer on SQuAD; and the decomposed graph entailment
model, a top performer on SciTail (Khot et al., 2018).
While these score well on the Easy Set, they are unable

to perform significantly better than a random baseline on
the Challenge Set, illustrating its challenging nature.
This challenge differs from the Kaggle-hosted 2016 Allen
AI Science Challenge (Schoenick et al., 2017) in three important ways1 . First, the creation of a Challenge partition is
to avoid scores being dominated by the performance of simple algorithms, and thus encourage research on methods that
the more difficult questions demand. Second, we provide a
science corpus along with the questions to help get started
(use of the corpus is optional, and systems are not restricted
to this corpus). Finally, the questions, corpus, and models
are all publically available.
The paper is organized as follows. We first discuss related work, and then describe how the ARC dataset was
collected and partitioned. We also provide an analysis of the
types of problems present in the Challenge Set. We then describe the supporting ARC Corpus, and illustrate how it contains knowledge relevant to Challenge questions. Finally, we
describe several baselines and their performance on ARC.
Most notably, although some baseline algorithms perform
well on the Easy Set (scoring up to 61%), none are able to
perform significantly above random on the Challenge Set.
We conclude by posing ARC as a challenge to the community. The ARC Dataset, Corpus, Models, and Leaderboard
can be accessed at http://data.allenai.org/arc.

et al., 2014). In general, a risk of using large synthetic QA
datasets is that neural methods are remarkably powerful at
“reverse-engineering” the process by which a dataset was
generated, or picking up on its idiosyncrasies to excel at
it, without necessarily advancing language understanding or
reasoning.
More recently, Welbl et al. (2017b) created the WikiHop
dataset, containing questions that appear to require more
than one Wikipedia document to answer (“multihop questions”). The dataset takes a step towards a more challenging task, but has several limitations: questions are binary
predicates (e.g., date of birth(“jeanne c. stein”,?X)); the intended inference is typically a simple two-step chain (commonly a geographical substitution of a city for a country);
and in many cases the correct answer can be guessed from
the passage, without requiring multi-hop inference (∼44%
of the answerable questions are single-hop, according to the
authors (Welbl et al., 2017b)).
Datasets based on human standardized tests have also
been used in AI for several years (e.g. Clark et al., 2016;
Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and
as part of the NTCIR QALab (NII, 2017), and for the 2016
Allen AI Science Challenge described in the Introduction
(Schoenick et al., 2017). However, there are two potentially significant challenges with these datasets. First, these
datasets are often small (e.g., hundreds of questions), due
to the scarcity of public, real-world test data. Second, because tests were designed for people rather than machines,
large portions of these tests can be easily solved by simple
AI methods (e.g. Davis, 2016; Clark et al., 2016). The result
of this is that scores become dominated by the performance
of simple algorithms (information retrieval, statistical correlations). This then biases research towards incrementally
improving those algorithms, rather than exploring the larger
AI challenges that the more difficult questions demand. Indeed, it is easy to mistake progress on these datasets as implying equal progress on easy and hard questions, while in
reality progress may be heavily concentrated on easy questions alone (Gururangan et al., 2018), leaving more difficult
challenges unaddressed. The ARC Dataset addresses both
of these limitations.

Related Work
There are numerous datasets available to drive progress
in question-answering. Earlier reading comprehension
datasets, e.g., MCTest (Richardson, 2013), SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), and
CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level
cues alone (i.e., answers were “explicitly stated”). TriviaQA
(Joshi et al., 2017) broadened this task by providing several articles with a question, and used questions authored
independently of the articles. Again, though, the questions
were largely factoid-style, e.g., “Who won the Nobel Peace
Prize in 2009?”. Although systems can now perform well
on these datasets, even matching human performance (Simonite, 2018), they can be easily fooled (Jia and Liang,
2017); the degree to which they truly understand language
or domain-specific concepts remains unclear.
To push towards more complex QA tasks, one approach
has been to generate synthetic datasets, the most notable example being the bAbI dataset (Weston et al., 2015). bAbi
was generated using a simple world simulator and language
generator, producing data for 20 different tasks. It has stimulated work on use of memory network neural architectures
(Weston et al., 2014), supporting a form of multistep reasoning where a neural memory propagates information from
one step to another (e.g. Henaff et al., 2016; Seo et al.,
2017a). However, its use of synthetic text and a synthetic
world limits the realism and difficulty of the task, with many
systems scoring a perfect 100% on most tasks (e.g. Weston

The ARC Dataset
The ARC dataset consists of 7787 science questions, all
non-diagram, multiple choice (typically 4-way multiple
choice). They are drawn from a variety of sources, and
sorted into a Challenge Set of 2590 “hard” questions (those
that both a retrieval and a co-occurrence method fail to answer correctly) and an Easy Set of 5197 questions. Table 1
summarizes the sizes of the train/dev/test partitions in the
ARC dataset. The question vocabulary uses 6329 distinct
words (stemmed).
Questions vary in their target student grade level (as assigned by the examiners who authored the questions), ranging from 3rd grade to 9th, i.e., students typically of age 8
through 13 years. Table 2 shows a break-down of the set
based on grade level with absolute counts(#) and percentage(%) of the Challenge and Easy set. In practice, there
is substantial overlap in difficulty among grade levels (also

1

ARC includes the publically releasable subset of the Kaggle
questions (about 60% of the Kaggle set, making up 43% of the
ARC set).

2

Train
Dev
Test
TOTAL

Challenge
1119
299
1172
2590

Easy
2251
570
2376
5197

The Pointwise Mutual Information (PMI) Solver. The
second filter we apply is the PMI solver, also from Clark
et al. (2016), again described here for completeness. This
uses the same corpus as the IR solver, and formalizes a way
of computing and applying associational knowledge. Given
a question q and an answer option ai , it uses PMI or pointwise mutual information (Church and Hanks, 1989) to measure the strength of the associations between parts of q and
parts of ai . Given a large corpus C, the PMI for two n-grams
x and y is defined as

Total
3370
869
3548
7787

Table 1: Number of questions in the ARC partitions.
Grade
3
4
5
6
7
8
9

Challenge
% (# qns)
3.6 (94 qns)
9 (233)
19.5 (506)
3.2 (84)
14.4 (372)
41.4 (1072)
8.8 (229)

%
3.4
11.4
21.2
3.4
10.7
41.2
8.7

Easy
(# qns)
(176 qns)
(591)
(1101)
(179)
(557)
(2139)
(454)

PMI(x, y) = log

p(x, y)
p(x)p(y)

Here p(x, y) is the joint probability that x and y occur together in C, within a certain window of text (we use a 10
word window). The term p(x)p(y), on the other hand, represents the probability with which x and y would occur together if they were statistically independent. The ratio of
p(x, y) to p(x)p(y) is thus the ratio of the observed cooccurrence to the expected co-occurrence. The larger this
ratio, the stronger the association between x and y.
The solver extracts unigrams, bigrams, trigrams, and skipbigrams from the question q and each answer option ai .
It outputs the answer with the largest average PMI, calculated over all pairs of question n-grams and answer option
n-grams.

Table 2: Grade-level distribution of ARC questions
seen in the similar distribution of grade levels), as each grade
level contains a mixture of easy and difficult questions.
min / average / max
Property:
Challenge
Easy
Question (# words)
2 / 22.3 / 128 3 / 19.4 / 118
Question (# sentences)
1 / 1.8 / 11
1 / 1.6 / 9
Answer option (# words)
1 / 4.9 / 39
1 / 3.7 / 26
# answer options
3 / 4.0 / 5
3 / 4.0 / 5

The Challenge Set
To illustrate the impact of using these algorithms as filters
when defining the Challenge Set, consider the following example:

Table 3: Properties of the ARC Dataset

Which property of air does a barometer measure? (A)
speed (B) pressure [correct] (C) humidity (D) temperature

Summary statistics of ARC are provided in Table 3,
showing questions and answers vary considerably in length.
Finally, Table 7 in the Appendix lists the variety of sources
the questions were drawn from.

The question was excluded from the Challenge Set because
it is correctly answered by (here) both the IR and PMI algorithms (note that it would have been excluded even if it was
answered correctly by just one of the solvers). The IR algorithm finds multiple sentences supporting the correct answer,
e.g.,

Identifying Challenge Questions
Operationally, we define a Challenge question as one that
is answered incorrectly by both of two baseline solvers, described below. Although this only approximates the informal goal of it being a “hard” question, this definition nevertheless serves as a practical and useful filter, as reflected by
the low scores of various baselines on the Challenge Set.

• Air pressure is measured with a barometer.
• Air pressure will be measured with a barometer.
• The aneroid barometer is an instrument that does not use
liquid in measuring the pressure of the air.
• A barometer measures the pressure of air molecules.

Information Retrieval (IR) Solver. The first filter we apply is the IR solver from Clark et al. (2016), briefly described
here for completeness. The IR solver uses the Waterloo corpus from (Clark et al., 2016), a Web-based corpus of 5×1010
tokens (280GB). The solver searches to see if the question
q along with an answer option is explicitly stated in the corpus, and returns the confidence that such a statement was
found. To do this, for each answer option ai , it sends q + ai
as a query to a search engine (we use Elasticsearch), and returns the search engine’s score for the top retrieved sentence
s where s also has at least one non-stopword overlap with q,
and at least one with ai ; this ensures s has some relevance
to both q and ai . This is repeated for all options ai to score
them all, and the option with the highest score selected.

and similarly the PMI algorithm finds that “barometer” and
“pressure” (and also “air” and “pressure”) co-occur unusually frequently (high PMI) in its corpus.
In contrast, consider the following question:
Which property of a mineral can be determined just by
looking at it? (A) luster [correct] (B) mass (C) weight
(D) hardness
This is incorrectly answered by both algorithms: There are
no corpus sentences similar to “a material’s luster can be
determined by looking at it”. Similarly, “mineral” co-occurs
unusually frequently with several incorrect answer options
3

Knowledge Type
Definition

Example
What is a worldwide increase in temperature called? (A) greenhouse effect (B) global
warming (C) ozone depletion (D) solar heating

Basic Facts &
Properties

Which element makes up most of the air we breathe? (A) carbon (B) nitrogen (C) oxygen
(D) argon

Structure

The crust, the mantle, and the core are structures of Earth. Which description is a feature
of Earth’s mantle? (A) contains fossil remains (B) consists of tectonic plates (C) is located
at the center of Earth (D) has properties of both liquids and solids

Processes & Causal

What is the first step of the process in the formation of sedimentary rocks? (A) erosion (B)
deposition (C) compaction (D) cementation

Teleology / Purpose

What is the main function of the circulatory system? (1) secrete enzymes (2) digest proteins
(3) produce hormones (4) transport materials

Algebraic

If a red flowered plant (RR) is crossed with a white flowered plant (rr), what color will the
offspring be? (A) 100% pink (B) 100% red (C) 50% white, 50% red (D) 100% white

Experiments

Scientists perform experiments to test hypotheses. How do scientists try to remain objective during experiments? (A) Scientists analyze all results. (B) Scientists use safety
precautions. (C) Scientists conduct experiments once. (D) Scientists change at least two
variables.

Spatial / Kinematic

In studying layers of rock sediment, a geologist found an area where older rock was layered
on top of younger rock. Which best explains how this occurred? (A) Earthquake activity
folded the rock layers...

Table 4: Types of knowledge suggested by ARC Challenge Set questions
(e.g., mass, hardness), confusing the PMI algorithm. This
question is thus part of the Challenge Set, as it appears to
require a more advanced answering method.

Question Types
ARC questions appeal to both different styles of knowledge, and different styles of reasoning. In Tables 4 and 5,
we enumerate the broad classes of each that we observe in
the ARC challenge, based on a sample of 100 questions.
The relative sizes of these categories are shown in Figures 1
and 2. These sizes are necessarily approximate, as it requires
a subjective judgement about the main challenge exhibited
by different questions. Nevertheless, it helps to provide a
rough atlas of the knowledge and reasoning space underlying ARC.

Figure 1: Relative sizes of different knowledge types suggested by the ARC Challenge Set.

The ARC Corpus
In addition to the ARC question set, we are also releasing
the ARC Corpus, a large corpus of science-related sentences
mined from the Web. It contains 14M sentences (1.4GB of
text), and mentions much of the knowledge required to answer the Challenge Questions. Although some of these mentions are indirect, and exploiting them is not trivial, it nevertheless provides a starting point for attacking the ARC Challenge. Note that use of the corpus is optional, and also that
systems are not restricted to this corpus.
The ARC Corpus was created by utilizing a major search
engine to run a large series of search queries relevant to

science. Queries were automatically constructed by instantiating ∼100 hand-written templates for 80 science topics
covered by US elementary and middle schools, the subject
areas of ARC. For example, for the topic “celestial phenomena”, two templates “[astronomical-term] astronomy”
and “[astronomical-term] astrophysics” were authored, and
a list of (here, 360) terms for astronomical-term collected
and used, resulting in 720 queries. The top several documents from each search were collected and de-duplicated,
and then the content of these documents was stripped down
4

Reasoning Type
Question logic

Example
Which item below is not made from a material grown in nature? (A) a cotton shirt (B) a
wooden chair (C) a plastic spoon (D) a grass basket

Linguistic
Matching

Which of the following best describes a mineral? (A) the main nutrient in all foods (B) a
type of grain found in cereals (C) a natural substance that makes up rocks (D) the decomposed plant matter found in soil

Multihop
Reasoning

Which property of a mineral can be determined just by looking at it? (A) luster (B) mass
(C) weight (D) hardness

Comparison

Compared to the Sun, a red star most likely has a greater (A) volume. (B) rate of rotation.
(C) surface temperature. (D) number of orbiting planets

Algebraic

If a heterozygous smooth pea plant (Ss) is crossed with a homozygous smooth pea plant
(SS), which are the possible genotypes the offspring could have? (A) only SS (B) only Ss
(C) Ss or SS (D) ss or SS

Hypothetical /
Counterfactual

If the Sun were larger, what would most likely also have to be true for Earth to sustain life?
(A) Earth would have to be further from the Sun. (B) Earth would have to be closer to the
Sun. (C) Earth would have to be smaller. (D) Earth would have to be larger.

Explanation /
Meta-reasoning

Why can steam be used to cook food? (A) Steam does work on objects. (B) Steam is a
form of water. (C) Steam can transfer heat to cooler objects. (D) Steam is able to move
through small spaces.

Spatial / Kinematic

Where will a sidewalk feel hottest on a warm, clear day? (A) Under a picnic table (B) In
direct sunlight (C) Under a puddle (D) In the shade

Analogy

Inside cells, special molecules carry messages from the membrane to the nucleus. Which
body system uses a similar process? (A) endocrine system (B) lymphatic system (C) excretory system (D) integumentary system

Table 5: Types of reasoning suggested by ARC Challenge Set questions
corpus was then augmented with the AristoMini corpus2 , an
earlier corpus containing dictionary definitions from Wiktionary, articles from Simple Wikipedia tagged as science,
and additional science sentences collected from the Web.
From a vocabulary analysis, 99.8% of the ARC question
vocabulary is mentioned in the ARC Corpus3 .
In our baseline experiments discussed shortly, we find that
if we change the corpus behind the IR solver from Waterloo
to the ARC Corpus, this changes its Challenge Set score
from near zero (by definition, Challenge questions are those
that IR with the Waterloo corpus gets wrong) to a score similar to random guessing4 . However, from an informal, sampled analysis, we find that this is more a limitation of the IR
methodology than of the coverage of the ARC Corpus. The
ARC Corpus, in fact, appears to mention knowledge relevant to approximately 95% of the ARC Challenge questions
(from an analysis of a random sample of questions), even if
simple retrieval methods are not able to exploit it to correctly

Figure 2: Relative sizes of different reasoning types suggested by the ARC Challenge Set.

2

Also available separately at http://allenai.org/data.html
Only 11 question words, mainly proper nouns, do not appear
in the corpus: Daphne, Sooj, LaKeisha, Quickgrow, Hypergrow,
CCGCAT, nonsnow, Quickgrow, Coaccretion, HZn, MgBr
4
In fact slightly lower than random guessing, likely due to
some corpus similarities where the same distractors in Waterloo
were also present in theARC Corpus.

to capture just the text in each document. The resulting text
was then chunked into sentences. This was repeated for all
templates. Note that some templates were parameterized by
more than one parameter. From an informal analysis of a
random sample of 805 documents that were collected, approximately 75% were judged as “science relevant”. The

3

5

that (C) is possible:

answer the questions. For example, consider:
Scientists launch a rocket into space for a mission.
Once the rocket escapes the gravitational pull of Earth,
how will the mass and weight of the rocket be affected?
(A) The mass and weight will change.
(B) The mass and weight will stay the same.
(C) The mass will stay the same, but the weight will
change. [correct]
(D) The mass will change, but the weight will stay the
same.
While this particular scenario is of course not mentioned
explicitly in the ARC Corpus, there are several statements
about the relation between mass, weight, and gravity, for example:
• The main difference is that if you were to leave the Earth
and go to the Moon, your weight would change but your
mass would remain constant.
• Astronauts in orbit experience weightlessness just like objects in the falling aircraft.
• Weight is the force that something feels due to gravity:
so the brick would have a much larger weight near the
earth’s surface than it does in deep space.
Such sentences provide evidence that weight but not mass
will change when in space. Similarly, consider the question:
Which factor will prompt an animal’s fight-or-flight response? (A) population size (B) competition for food
[correct] (C) seasonal temperatures (D) protection of
the environment
While there are few sentences in the ARC Corpus that directly relate fight-or-flight with food competition, there are
several that indicate competition is a kind of threat, e.g.,:
• Insurgency, cross-border firing, competition with livestock for grazing ground, ...all threaten the Markhor.
• The main threats to the species include loss of habitat
through deforestation, hunting for meat and competition
with domestic livestock.
• Habitat fragmentation threatens this species, as does possible competition with the introduced non-native Ringnecked Pheasant.
and similarly that threats can trigger the fight-or-flight response:
• Whenever a reptile faces a threat they go into fight or
flight mode...
• The sympathetics command your fight or flight response,
and when they fire, your heart rate and your breathing
speed up, ...preparing your body to face a threat...
Finally, consider the question:
Which of the following distinguishes the organisms in
the kingdom Fungi from other eukaryotic organisms?
(A) Fungi are unicellular. (B) Fungi reproduce sexually. (C) Fungi obtain nutrients by absorption. [correct] (D) Fungi make food through photosynthesis
Again, there is distributed evidence for the answer, including
that (A) is incorrect:
• There are unicellular fungi and multicellular fungi.

• Fungi are heterotrophs that acquire their nutrients by absorption.
and that (C) is distinct for other eukaryotic organisms:
• Animals are multicellular eukaryotes; they are chemosynthetic heterotrophs that ingest their food.
Although these descriptions of reasoning paths are informal, and clearly many others are possible, they illustrate that
the ARC Corpus mentions knowledge relevant to a question, even if no single sentence alone provides the answer.
Of course, this does not address the challenge of correctly
identifying and reasoning with this knowledge, nor the challenge of injecting unstated commonsense knowledge that
may also be required. Rather, our claim is that the Corpus
contains substantial linguistic signal relevant to most of the
ARC questions, and is a useful starting point for corpusbased attacks on the Challenge.

Baseline Performance
Baseline Systems
We ran several baseline QA systems on the Challenge and
Easy Sets, including two neural models, DecompAttn and
BiDAF (details below), that have near state-of-the-art performance on the well-known SNLI and SQuAD datasets respectively. We scored systems using the following scoring
rubric: For each question, a system receives 1 point if it
chooses the correct answer and 1/k if it reports a k-way tie
(i.e., chooses multiple answers) that includes the correct answer. For a question set, the overall score of the system is
the sum of the points it receives for all questions, divided by
the number of questions and reported as a percentage. We
report performance of the following systems:
1. IR (dataset definition). IR method, described earlier.
2. PMI (dataset definition). PMI method, described earlier.
3. Guess-all (“random”). A naı̈ve baseline that selects all
answer options as equally valid, thereby scoring 1/k for
each question with k answer choices. A system that
chooses a single answer at random will also converge to
this score after enough trials.
4. IR (ARC Corpus). The IR algorithm, rerun with the
ARC Corpus. Note that changing the original corpus is
expected to result in a different score, unless the two corpora are highly correlated. A corpus containing random
strings, for instance, will have very low correlation with
the original corpus and will result in a random-guessing
score of around 25%.
5. TableILP (Khashabi et al., 2016), which performs matching and reasoning using a semi-structured knowledge base
of science knowledge, expressed in tables.
6. TupleInference (Khot et al., 2017), which performs semistructured matching of the question with retrieved sentences, where the structure consists of Open IE tuples.
7. DecompAttn, DGEM, and DGEM-OpenIE (Neural Entailment Models). We adapted two neural entailment
6

Test Scores
Challenge
Easy
Set
Set
†
IR (dataset defn)
(1.02)
(74.48)†
PMI (dataset defn)
(2.03)†
(77.82)†
IR (using ARC Corpus)
20.26
62.55
TupleInference
23.83
60.81
DecompAttn‡
24.34
58.27
Guess-all (“random”)
25.02
25.02
DGEM-OpenIE‡
26.41
57.45
BiDAF‡
26.54
50.11
TableILP
26.97
36.15
DGEM
27.11
58.97
†
These solvers were used to define the dataset, affecting scores.
‡
Code available at https://github.com/allenai/arc-solvers

models, DecompAttn (Parikh et al., 2016) and DGEM
(Khot et al., 2018), to the task of answering multiplechoice questions. The models were trained on an extended version of the SciTail dataset (Khot et al., 2018).
To adapt these to multiple-choice QA, we first convert
the question q plus an answer option ai into a hypothesis sentence (or paragraph) hi , use this as a search query
to retrieve text sentences tij from a corpus, then compute
the entailment scores between hi and each tij . This is
repeated for all answer options, and the option with the
overall highest entailment score selected. Further details
are given in the Appendix.
The DGEM model uses a structured representation of the
hypothesis hi , extracted with a proprietary parser plus
Open IE. To create a releasable version of DGEM, we also
evaluate (and release) a variant DGEM-OpenIE, a version
of DGEM that only uses Open IE to create the structured
representation of hi , thus avoiding proprietary tools.

Solver

Table 6: Performance of the different baseline systems. Scores are reported as percentages on the test
sets. For up-to-date results, see the ARC leaderboard at
http://data.allenai.org/arc.

8. BiDAF (Reading Comprehension Model).
We also
adapted BiDAF (Seo et al., 2017b), a top-performer on
the SQuAD dataset (Rajpurkar et al., 2016), to our task.
As BiDAF is a direct answer system, we adapted it to
multiple-choice QA following the approach used in several previous projects (Khashabi et al., 2018b; Welbl et al.,
2017a; Kembhavi et al., 2017) as follows: First, given a
question, we create a single paragraph by concatenating a
set of retrieved sentences. In this case, we use the same
sentences retrieved by the entailment models for all answer options (above). We then use BiDAF to select an
answer span from this paragraph, given the question. Finally, we pick the multiple-choice option that maximally
overlaps that answer span (here defined as the option with
the highest percentage of lemmatized, non-stopword tokens covered by the BiDAF answer span). BiDAF was
trained on SQuAD then further tuned to science questions
using continued training.

ture, attempting matching, attempting chaining, etc. However, the retrieval bias of the underlying IR methods is towards sentences that are all very similar to the question, and
away from sentences that individually only partially match
the question, but together fully explain the correct answer
(e.g., through chaining). This suggests the need for a more
advanced retrieval strategy for questions that require combining multiple facts, as well as new methods for combining
that information.

Conclusion
Datasets have become highly influential in driving the direction of research. Recent datasets for QA have led to impressive advances, but have focused on factoid questions where
surface-level cues alone are sufficient to find an answer, discouraging progress on questions requiring reasoning or other
advanced methods. To help the field move towards more difficult tasks, we have presented the AI2 Reasoning Challenge
(ARC), consisting of a new question set, text corpus, and
baselines, and whose Challenge partition is hard for retrieval
and co-occurence methods. We find that none of the baseline systems tested can significantly outperform a random
baseline on the Challenge set, including two neural models
with high performances on SNLI and SQuAD. Progress on
ARC would thus be an impressive achievement, given its
design, and be significant step forward for the community.
To access ARC, view the leaderboard, and submit new entries, visit the ARC Website at http://data.allenai.org/arc.

Results
Table 6 summarizes the scores obtained by various baseline algorithms on the test partitions of the Challenge and
Easy sets.
The IR and PMI (dataset definition) solvers, by design,
score near zero on the Challenge set. The slightly abovezero score is due to the solver occasionally picking multiple
(tied) answers, resulting in a partial credit for a few questions. We include these questions in the Challenge set.
The most striking observation is that none of the algorithms score significantly higher than the random baseline
on the Challenge set, where the 95% confidence interval is
±2.5%. In contrast, their performance on the Easy set is
generally between 55% and 65%. This highlights the different nature and difficulty of the Challenge set.
The poor performance of the non-IR solvers is partly explained by their correlation with the IR solver: the first step
for nearly all of them (except TableILP, which uses nonsentential knowledge but has low knowledge coverage) is
to use a simple IR method to obtain relevant sentences, and
then process them in different ways such as extracting struc-

Availability The ARC Dataset,
Corpus,
three
baseline neural models (DecompAttn, BiDAF, and
DGEM), and the leaderboard, are all available
from http://data.allenai.org/arc.

References
R. Brachman. Selected grand challenges in cognitive science. Technical Report Tech Report 05-1218, MITRE,
7

NII. Evaluation of information access technologies. In Proc.
13th NTCIR Conf, 2017.
A. P. Parikh, O. Täckström, D. Das, and J. Uszkoreit. A
decomposable attention model for natural language inference. In EMNLP, 2016.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad:
100,000+ questions for machine comprehension of text.
In Proc. EMNLP’16, 2016.
M. Richardson. Mctest: A challenge dataset for the opendomain machine comprehension of text. In EMNLP’13,
2013.
C. Schoenick, P. Clark, O. Tafjord, P. Turney, and O. Etzioni. Moving beyond the turing test with the allen ai
science challenge. Communications of the ACM, 60(9):
60–64, 2017.
M. Seo, S. Min, A. Farhadi, and H. Hajishirzi. Queryreduction networks for question answering. In ICLR,
2017a.
M. J. Seo, H. Hajishirzi, A. Farhadi, and O. Etzioni. Diagram understanding in geometry questions. In AAAI, pp.
2831–2838, 2014.
M. J. Seo, A. Kembhavi, A. Farhadi, and H. Hajishirzi. Bidirectional attention flow for machine comprehension. In
Proc. ICLR’17, 2017b.
T. Simonite. Ai beat humans at reading! maybe not. In
WIRED Magazine, 2018.
E. Strickland. Can an ai get into the university of tokyo?
IEEE Spectrum, August 2013.
A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni,
P. Bachman, and K. Suleman. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830,
2016.
J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing multiple
choice science questions. In Workshop on Noisy Usergenerated Text, 2017a.
J. Welbl, P. Stenetorp, and S. Riedel. Constructing datasets
for multi-hop reading comprehension across documents.
arXiv preprint arXiv:1710.06481, 2017b.
J. Weston, A. Bordes, S. Chopra, A. M. Rush, B. van
Merriënboer, A. Joulin, and T. Mikolov. Towards AIcomplete question answering: A set of prerequisite toy
tasks. arXiv preprint arXiv:1502.05698, 2015.
J. Weston, S. Chopra, and A. Bordes. Memory networks.
arXiv preprint arXiv:1410.3916, 2014.

2005.
K. W. Church and P. Hanks. Word association norms, mutual
information and lexicography. In 27th ACL, pp. 76–83,
1989.
P. Clark and O. Etzioni. My Computer is an Honor Student
but how Intelligent is it? Standardized Tests as a Measure
of AI. AI Magazine, 2016. (To appear).
P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D.
Turney, and D. Khashabi. Combining retrieval, statistics,
and inference to answer elementary science questions. In
AAAI, pp. 2580–2586, 2016.
E. Davis. How to write science questions that are easy for
people and hard for computers. AI magazine, 37(1):13–
22, 2016.
A. Fujita, A. Kameda, A. Kawazoe, and Y. Miyao. Overview
of todai robot project and evaluation framework of its
nlp-based problem solving. In Proc. LREC’14, vol. 36,
page 36, 2014.
S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz,
S. Bowman, and N. A. Smith. Annotation artifacts in natural language inference data. In Proc. of NAACL, 2018.
M. Henaff, J. Weston, A. Szlam, A. Bordes, and Y. LeCun.
Tracking the world state with recurrent entity networks.
CoRR, abs/1612.03969, 2016.
K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt,
W. Kay, M. Suleyman, and P. Blunsom. Teaching machines to read and comprehend. In Advances in Neural
Information Processing Systems, pp. 1693–1701, 2015.
R. Jia and P. Liang. Adversarial examples for evaluating
reading comprehension systems. In Proc. EMNLP’17,
2017.
M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset
for reading comprehension. In Proc. ACL’17, July 2017.
A. Kembhavi, M. Seo, D. Schwenk, J. Choi, A. Farhadi, and
H. Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, 2017.
D. Khashabi, T. Khot, A. Sabharwal, P. Clark, O. Etzioni,
and D. Roth. Question answering via integer programming over semi-structured knowledge. In IJCAI, 2016.
D. Khashabi, T. Khot, A. Sabharwal, and D. Roth. Question
answering as global reasoning over semantic abstractions.
In AAAI’18, 2018a.
D. Khashabi, T. K. A. Sabharwal, and D. Roth. Question
answering as global reasoning over semantic abstractions.
In AAAI, 2018b.

Appendix A: Question Sources
Table 7 provides a brief description of the question sources
used to assemble ARC.

T. Khot, A. Sabharwal, and P. Clark. Answering complex
questions using open information extraction. In ACL,
2017.

Appendix B: Question Answering via
Entailment

T. Khot, A. Sabharwal, and P. Clark. SciTail: A textual
entailment dataset from science question answering. In
AAAI, 2018.

Given a large corpus of sentences (such as the ARC Corpus), one natural way to answer multiple-choice questions
8

Source

Description

# Questions
Challenge Easy
ACTAAP
Arkansas Comprehensive Testing, Assessment, and Accountability Program
71
129
AIMS
Arizona’s Instrument to Measure Standards
15
32
Alaska Dept Ed
Alaska Department of Education & Early Development
38
62
AMP
Alaska Measures of Progress
0
1
CA Stnds Test
California Standards Test
19
59
FCAT
Florida Comprehensive Assessment Test
3
3
LA Ed Assm. Prog Louisiana Department of Education
36
61
MD School Assm.
Maryland School Assessment
66
89
MCAS
MCAS-Massachusetts Comprehensive Assessment System
221
411
MEA
Maine Educational Assessment
36
56
MEAP
Michigan Educational Assessment Program
14
19
Mercury
Our name for an anonymous content partner of AI2 who has generously provided 1762 3366
some of their science exam question data for non-commercial research purposes.
MSA
Maryland School Assessment
23
34
NAEP
National Assessment of Educational Progress
12
37
NC READY
Ready North Carolina
45
67
NY Regents
New York State Educational Department Regents exams
103
422
Ohio Achieve. Tests Ohio Achievement Tests
14
44
TAKS
Texas Assessment of Knowledge and Skills
12
29
TIMSS
TIMSS 1995, 2003, 2007, & 2011 Assessments. Copyright (c) 2013 International
60
151
Association for the Evaluation of Educational Achievement (IEA)
VA Stds of Learning Virginia Standards of Learning
37
120
WASL
Washington Assessment of Student Learning
3
5

Table 7: The various question sources for ARC.
is by identifying the answer choice entailed by a sentence in
this corpus. We can, therefore, use an entailment model to
answer multiple-choice questions by (1) converting a question and each answer choice into an assertion (2) computing the entailment scores for this assertion (hypothesis) from
sentences in our corpus (premise) (3) return the answer with
the highest support based on the entailment scores. We describe each step in further detail.

support from each sentence in the corpus. Since sentences
supporting the assertion are likely to have high lexical overlap, we use information retrieval as described earlier to find
relevant sentences. We query an ElasticSearch index on this
corpus with the question and answer choice as a query. We
also require the matched sentence must contain the answer
choice to avoid sentences that can not be used to answer
questions. We filter long sentences (> 300 characters), sentences with negation and noisy sentences from the retrieved
sentences. Similar to SemanticILP (Khashabi et al., 2018a),
we pick the top 8 sentences (based on their ElasticSearch
score, scoreir (pq,a , q + a)) across the answer choices for
each question. These sentences, Pq form the set of candidate premises used to compute the entailment score.
For an entailment model g, we can compute the score
for each question, answer choice and premise as pq,a ∈
Pq , scoree (q, a, pq,a ) = entailsg (pq,a , hq,a ). We present
the results with two neural entailment models, Decomposable Attention (Parikh et al., 2016) and Decomposed Graph
Entailment Model (DGEM) (Khot et al., 2018). Specifically
for DGEM, we also extract graph structure from the hypothesis using Open IE and a chunker. Since the chunker uses
proprietary software, we also evaluate and release DGEMOpenIE that only uses the graph structure from Open IE and
ignores sentences with no Open IE extraction. These models
were trained on the Science Entailment dataset (Khot et al.,
2018), a dataset created in a similar manner where crowdworkers annotated the premises and hypotheses as entailing
or not. Our released code 5 can also be easily extended to

Question to Assertion
We convert a question, q and answer choice, a to an assertion, hq,a by first converting the question into a Fill-In-TheBlank (FITB) statement, fq (e.g. Plants produce ). In certain cases, the questions were already in FITB form and in
other cases, we replace the wh-word with a blank. We also
drop phrases common in the multiple-choice question setting, e.g. “of the following”, “of these”. Given a FITB statement, we just replace the blank with the answer choice to
generate the assertion hq,a = replace(fq , , a). However,
this simple approach can lead to non-grammatical sentences.
E.g. “How many times does Earth rotate on its axis in one
day?” will be converted to “One many times does Earth rotate on its axis in one day.”. A more sophisticated rule-based
approach could be developed to handle each question type;
but we leave this to future work.

Entailment Computation
Next, we use the entailment model to identify the support
for the generated hypothesis, hq,a . For a large-scale text corpus, it would not be computationally feasible to compute the

5

9

https://github.com/allenai/arc-solvers

use any other neural entailment model implemented in AllenNLP.

Answer Scoring
Given the scores for each premise scoree (q, a, pq,a ), we
use the maximum supporting sentence score as the answer
choice scorec (q, a) = max scoree (q, a, pq,a ).

10

