Marmara Turkish Coreference Corpus
and Coreference Resolution Baseline
Peter Schüller, Kübra Cingilli, Ferit Tunçer,
Barış Gün Sürmeli, Ayşegül Pekel, Ayşe Hande Karatay,

arXiv:1706.01863v1 [cs.CL] 6 Jun 2017

Marmara University,
Faculty of Engineering
Istanbul, Turkey
peter.schuller@marmara.edu.tr kubracingilli@gmail.com
ferit@cryptolab.net barisgunsurmeli@gmail.com
aysegulpekel@gmail.com handekaratay7@gmail.com
Hacer Ezgi Karakaş
Sabanci University
Faculty of Engineering and Natural Science
Istanbul, Turkey
hezgikarakas@sabanciuniv.edu

Version 1, June 7, 2017
Technical Report: manuscript submitted to
Natural Language Engineering by Cambridge University Press
http://journals.cambridge.org/nle
Abstract
We describe the Marmara Turkish Coreference Corpus, which is an annotation of the whole
METU-Sabanci Turkish Treebank with mentions and coreference chains. Collecting nine or more
independent annotations for each document allowed for fully automatic adjudication. We provide
a baseline system for Turkish mention detection and coreference resolution and evaluate it on the
corpus.

1

Introduction

Coreference Resolution is the task of identifying groups of phrases in a text that refer to the same
discourse entity. Such referring phrases are called mentions, a set of mentions that all refer to the
same discourse entity is called a chain. Annotated corpora are important resources for developing and
evaluating automatic coreference resolution methods.
Turkish is an agglutinative language and Turkish coreference resolution poses several challenges different from many other languages, in particular the absence of grammatical gender, the possibility of
null pronouns in subject and object position, possessive pronouns that can be expressed as suffixes, and
ambiguities among possessive and number morphemes, e.g., ‘çocukları’ can be analyzed as ‘their children’
or as ‘his/her children’, depending on context (Oflazer and Bozşahin, 1994).
No coreference resolution corpus exists for Turkish so far. We here describe the result of an effort
to create such a corpus based on the METU-Sabanci Turkish Treebank (Atalay et al., 2003; Oflazer

1

et al., 2003; Say et al., 2004), which is, to the best of our knowledge, the only publicly available Turkish
Treebank.
Our contributions are as follows.
• We describe two stages of annotation: in Phase I, annotators created mentions and chains, which
did not yield sufficient inter-annotator agreement. In Phase II, mentions were given to annotators
who created only chains. We collected on average more than ten independent annotations per
document for each document in the METU-Sabanci Turkish Treebank.
• We describe annotator profiles and adjudication, which was done semi-automatically in Phase I
and fully automatically in Phase II. We describe the principles of our automatic adjudication tool
which uses a voting-like approach. Such an automatic approach is possible because we collected
enough (on average 10) annotations per document.
• We describe the XML format used to address documents, sentences, and tokens in the METUSabanci Turkish Treebank. We provide a public version of the corpus as XML, including tools
to convert the corpus to CoNLL format. (For licensing reasons we cannot re-publish the Turkish
Treebank data.)
• We describe and provide a baseline method for mention detection and coreference resolution, compatible with the format of the corpus. We evaluate this baseline method on the corpus with
leave-one-out cross-validation.
Section 2 gives preliminaries of Coreference Resolution and the Turkish language and describes related
work. Section 3 explains the annotation and adjudication process and discusses properties of the corpus,
annotator profiles, and supporting tools. Section 4 describes the baseline system and its evaluation on
the corpus. Section 5 concludes and gives an outlook on future work.
We provide the Marmara Turkish Coreference Corpus, tools, and the baseline system, at
https://bitbucket.org/knowlp/marmara-turkish-coreference-corpus .

2

Preliminaries and Related Work

Next we give background and related work on coreference resolution, the Turkish language, and specific
challenges of coreference resolution in Turkish.

2.1

Coreference Resolution

Coreference resolution is the task of marking noun phrases that refer to the same discourse entity as
coreferent. Mention detection, which identifies such noun phrases, is usually included in that task. Coreference resolution is not limited to resolving pronouns: in computer linguistics it was first introduced as
a benchmark for deep semantic understanding of text in the message understanding conference series
(Grishman and Sundheim, 1995) and later continued in the Automatic Content Extraction (ACE) program (Doddington et al., 2004) which required English coreference resolution as foundation for all tasks
of years 2000–2004. The SemEval competition series followed ACE and featured the first multilingual
coreference resolution challenge in 2010 (Recasens et al., 2010). Connected to that is the freely available,
large, and multilingual OntoNotes (Hovy et al., 2006) corpus, which was used in the multilingual coreference task in SemEval 2012 (Pradhan et al., 2012) and contains coreference annotations for English,
Arabic, and Mandarin (Pradhan et al., 2007).
Coreference resolution has been surveyed by Ng (2010). Approaches are manifold and include
machine-learning-based approaches, rule-based systems, and combinations of both. In most machinelearning approaches, equivalence relations of chains (sometimes called clusters) are assembled from predictions of the relatedness of pairs of mentions (mention-mention links). The earliest machine-learning
approach is due to Soon et al. (2001), methods for building chains from link predictions include local
greedy heuristics as done by Bengtson and Roth (2008) or Stoyanov and Eisner (2012), global optimization formulations such as relaxation labeling (Sapena et al., 2012) or ranking with ILP or Markov
Logic (Culotta et al., 2007; Denis and Baldridge, 2009), and representations of trees of mention-mention
links (Chang et al., 2013; Fernandes et al., 2012). The first rule-based algorithm for anaphora resolution
2

was done by Hobbs (1978). More recent rule-based systems merge chains based on several sets of rules in
a multi-stage filtering approach (Lee et al., 2013), moreover there are hybrid systems combining rules and
machine learning such as the one by Chen and Ng (2012). Other approaches use curated or distributed
knowledge sources such as WordNet, Google distance, and Wikipedia (Poesio et al., 2004; Zheng et al.,
2013).
Note that anaphora resolution (Hirst, 1981; Mitkov, 2002) is a problem orthogonal to coreference
resolution (van Deemter and Kibble, 2000), because anaphora resolution focuses on referring expressions
that point to previous expressions in the text. Cataphora (pointing to later occurrences in the text)
are exluded. On the other hand, different from most works on coreference, anaphora resolution includes
bound pronouns that do not refer to concrete entities because they are quantified using, e.g., ‘some’ or
‘none’.

2.2

Turkish

Turkish is a member of the family of Altaic languages, it is an agglutinative language where suffixes
are attached to a root word. Derivational and inflectional suffixes are very productive (Oflazer, 1993;
Oflazer et al., 1994) and are subject to vowel harmony from the root word. Morphological analysis
is challenging due to ambiguities between different types of suffixes, for example ‘izin’ can mean ‘your
trace’ (iz+P2Sg+Nom), ‘trace’ (iz+Pnon+Gen), or ‘permission’ (izin+Pnon+Nom) (Hakkani-Tür et al.,
2002).
The METU-Sabanci Turkish Treebank (in the following just called Turkish Treebank) (Atalay et al.,
2003; Oflazer et al., 2003; Say et al., 2004) contains a subset of the METU Turkish Corpus (Say et al.,
2004) in tokenized form. Each token is analyzed morphologically and split into inflectional groups (IGs).
Sentences are annotated with dependency parse information, where dependencies point to specific IGs
within tokens. The Turkish Treebank splits tokens into IGs on derivational boundaries, for example,
‘evimdekiler’ (those in my house) is analyzed (Oflazer et al., 2003) as
ev+Noun+A3sg+P1sg+Loc^DB+Adj^DB+Noun+Zero+A3pl+Pnon+Nom
where ^DB indicates derivation boundaries and the token consists of three IGs ‘evimde’ (in my house),
‘ki’ (adjectivization), and ‘ler’ (nominalization+plural). A CoNLL format that provides a CoNLL token
corresponding to each IG of a token has been created for Turkish dependency parsing (Buchholz and
Marsi, 2006).
Named entities in the Turkish Treebank are not marked specially, but multi-word named entities are
represented as single tokens.
Turkish Coreference Resolution. The following properties of Turkish are in particular relevant for
coreference resolution. In the above example, ‘those in my house’ as well as ‘my house’ as well as ‘my’
could be coreferent with mentions in the document. However, neither ‘my house’ nor ‘my’ is available as a
separate unit of analysis: both are parts of the first IG (‘evimde’). Gender is not marked in Turkish with
the exception of honorifics ‘Bey’ and ‘Hanım’ which corresponds to English ‘Mr’ and ‘Mrs’. Moreover
several common first names apply to both genders. Hence gender-based syntactic compatibility checks
for mentions are only possible in some cases. Personal pronoun subjects in Turkish are usually realized
as suffixes of the verb, e.g., ‘gidiyoruz’ (we are going) and ‘gidiyorlar’ (they are going) but they can also
be realized explicitly as in ‘biz gidiyoruz’, depending on discourse conditions (Turan, 1996). Suffixes
of proper nouns in written Turkish are systematically separated from the proper nouns using a single
quote, e.g., ‘Türkiye’den’ (from Turkey) and ‘Türkiye’deki’ (the thing in Turkey). This systematic rule
simplifies finding equal proper noun mentions in coreference resolution for Turkish.
Most works about referring expressions in Turkish focus on anaphora resolution and not on full
coreference resolution. One exception is the work of Küçük and Yazıcı (2008) on political news texts
extracted from videos: they focus on Gazetteers for extracting mentions (without considering general
NPs or syntactic information), provide a rule-based heuristic based on recency for creating coreference
chains, and evaluate their approach on three documents (which are not part of the Turkish Treebank).
In the following we describe work on Turkish anaphora resolution, which is related to coreference
resolution.

3

METU-Sabanci Turkish Treebank (1960 text fragments)

Phase I

Treebank documents (33 documents)
21 documents
GATE annotation

12 documents
Mention annotation

Semi-automatic adjudication

Phase II

Mention candidates (for 33 documents)
Annotation of chains (339 annotations)
Automatic adjudication
Marmara Turkish Coreference Corpus

Figure 1: Annotation process visualisation
Turkish Anaphora Resolution. Erkan and Akman (1998) describe an implementation of pronoun
anaphora resolution in a framework for situation theory which is based on knowledge representation and
logical reasoning. Hobb’s naïve pronoun resolution algorithm (Hobbs, 1978) was realized for Turkish and
tested on 10 toy sentences (Tüfekçi and Kılıçaslan, 2007).
Centering theory (Grosz et al., 1995) is the foundation of several works on Turkish pronouns. Turan
(1996) performed a study about discourse conditions for referring vs. nonreferring expressions and null
vs. overt pronouns, and evaluated the theory on 2500 annotated tokens. Yüksel and Bozsahin (2002)
created a system for generating referring expressions that was tested on a machine translation task.
Furthermore, there is a theoretical model of anaphora resolution based on centering theory by Yıldırım
et al. (2004).
Küçük and Yöndem (2007) described a system for finding and resolving Turkish pronominal anaphora
and annotated 12266 anaphor candidate instances in the METU Turkish Corpus to evaluate their candidate extractor and decision tree learner for anaphora resolution. Kılıçaslan et al. (2009) performed a
comprehensive study on pronoun resolution and evaluated various machine learning methods for resolving
overt and null pronouns in a corpus of 20 children stories.

3

Marmara Turkish Coreference Corpus

We next describe the annotation and adjudication process including formal adjudication criteria, key
properties of the resulting corpus, and supporting tools.

3.1

Annotation Process

Figure 1 visualizes the process that led to the final corpus. Annotations were collected in two phases:
Phase I took place in October–December 2015 and Phase II during October–December 2016.
Annotations were collected from computer engineering students participating in a lecture on natural
language processing, after educating them in basic linguistic analysis and coreference resolution. To
achieve reasonable annotation quality, we aimed to keep annotation simple and therefore based it on
few principles and examples. We designed an annotation manual (Sürmeli et al., 2016) for marking
coreference according to the following principles:
• all concrete entities that are mentioned more than once shall be annotated,
• mentions shall be marked as the biggest possible span of tokens that describes the entity,
• lists shall not be annotated (elements of lists can be annotated), and
4

Genre

Doc
#
sum

News
Short Story
Novel
Essay
Research Monogr.
Article
Travel
Other
Overall

Ann
#
avg

IAA1
%
avg

IAA2
%
avg

Tok
#
avg

CM
#
avg

AM
#
avg

AC
#
avg

Ph1
#
sum

9
8
7
2
2
3
1
1

10.2
10.1
10.4
9.5
10.5
10.7
11.0
10.0

87
64
67
85
72
90
79
62

94
84
87
93
89
96
92
83

1324
1540
1798
2058
2020
1506
2142
2284

139
170
183
92
182
121
178
301

126
164
176
87
179
120
173
275

35
23
22
28
27
32
44
44

2
5
5
2
2
3
1
1

33

10.3

76

90

1634

160

152

29

21

Table 1: Key metrics about corpus and inter-annotator agreement per genre and overall. Doc is the
number of documents, Ann the number of annotations received, IAA1 and IAA2 are inter-annotator
agreements, Tok and CM indicate the number of tokens and given mentions, AM and AC show the
number of received mentions and chains, and Ph1 indicates how many documents were annotated in
both annotation phases. Columns Doc through CM and Ph1 are accumulated over documents, while
AM and AC are accumulated over annotations.
• predications shall not be annotated.
Note that, by marking mentions as the biggest spans, mentions and potentially available appositives are
annotated as single mentions, which is different from OntoNotes where appositives are a special type of
coreference annotation. We do not mark predications because they are a different type of coreference as
argued by van Deemter and Kibble (2000).
In Phase I, annotations were created by 19 annotators with the GATE (Cunningham et al., 2013;
Gaizauskas et al., 1996) coreference annotation tool. This yielded on average 6.5 annotations per document for 21 documents in the Treebank. Adjudication of these documents was done semi-automatically
(see Sections 3.2 and 3.4). However, due to low inter-annotator agreement about mention boundaries,
decisions often depended on the adjudicator. Therefore, in order to make the setting simpler, we decided to collect more annotations with given mentions. We used the list of mentions resulting from the
adjudicated documents of Phase I. Mentions for those 12 documents that were not annotated in Phase I
were manually created in a collaboration of two annotators for each document.
In Phase II, 46 annotators were given CoNLL files with token and coreference columns where each
mention was given in its own chain. Annotators created files with equalities between chain IDs and
uploaded these files to a web service where they were checked for syntactical correctness. The submission
file format is described in (Sürmeli et al., 2016). Phase II yielded 339 individual annotations of sufficient
inter-annotator agreement to perform fully automatic adjudication (see next section).
This method of collecting annotation as text files might seem archaic, however, in practice, annotators
were more comfortable with such a system than with the graphical user interface of GATE in Phase I.
We were not able to use the BRAT (Stenetorp et al., 2012) annotation tool, because of difficulties
representing sentence and word addresses in a way that they can be extracted from annotations.

3.2

Adjudication

Table 1 shows key properties of the annotated corpus including inter-annotator agreement. Statistics
are accumulated per genre as well as over the whole corpus. Coreference IAA scores IAA1 and IAA2 are
calculated as described in (Passonneau, 2004) and (Passonneau et al., 2006), respectively. IAA1 is an
adaption of Krippendorff α (Krippendorff, 1980) where a pair of chains gets a score of 1 for a perfect
match, 32 if one chain is a subset of the other one, 31 if they are intersecting, and 0 otherwise. (Krippendorff
α assigns 1 for a perfect match and 0 in all other cases.) IAA2 uses Jaccard distance instead of fixed
scores for the above cases, which provides more realistic results if chains have heterogeneous sizes. (This
is often the case in coreference corpora.) Over all documents, IAA1 is 76% and IAA2 is 90%. We observe

5

worse IAA for genres that are based on writing as an art form, i.e., for short stories, novels, and the
Other genre (a first-person narrative).
The amount of annotations per document, combined with the observed IAA, allows us to automatically adjudicate the corpus. This is different from other coreference annotations, in particular in
OntoNotes, where two annotators created annotations followed by adjudication done by a single human expert (Weischedel et al., 2012). Automatic adjudication is based on combinatorial optimization,
where we search for a solution of chains that has overall minimal divergence from all annotator inputs.
Divergence is measured in terms of mention-mention links given and omitted by annotators.
Formally, given a set M of mentions in a document, a chain is a subset of these mentions, and k
annotators produce annotations (sets of chains that are nonintersecting) A1 , . . . , Ak over M . A solution
G is also a set of chains over M and we search for G such that the following objective becomes minimal:
X
2 · a(m, m0 , Ai ) · na(m, m0 , G) + na(m, m0 , Ai ) · a(m, m0 , G)
(1)
m,m0 ∈ M
i ∈ {1,...,k}

where a(· · · ) is an indicator function for mentions in the same chain within an annotation A, formally

1 if there is a chain C ∈ A with {m, m0 } ⊆ C
0
a(m, m , A) =
0 otherwise
and na(· · · ) is an indicator for mention pairs that are not in the same chain in an annotation A:
na(m1 , m2 , A) = 1 − a(m1 , m2 , A).
Objective (1) incurs a cost of 2j for each mention pair that is not in the same chain in the solution G
contrary to the opinion of j annotators. Moreover, (1) incurs a cost of l for each mention-pair that is in
the same chain in G contrary to the opinion of l annotators. This makes optimal solutions ignore as little
as possible information from annotators. The coefficient 2 has the effect, that not annotating a mention
to be in some chain has less weight than annotating a mention as part of some chain. We introduced this
preference into the objective, because not annotating some mention can be due to an oversight, while
putting a mention into a chain is more likely done intentionally.
Note that, if annotators produce different sets of mentions, we can build M from the union of all
mentions produced by annotators with unused mentions becoming singleton partitions. We describe
practical issues of our adjudication tool in Section 3.4.

3.3

Corpus Properties and Annotator Profiles

Table 1 shows key properties of the annotated corpus, accumulated over genres and overall figures.
Average number of tokens and mentions per genre varies a lot. In particular, the Essay genre contains
texts discussing abstract concepts like ‘home’ and ‘science’ which are not annotated. The narrative in
genre Other contains many person names which are repeatedly mentioned.
Column Ph1 indicates how many of the documents were annotated in both phases of the annotation
process. For example the News genre contains 9 documents. Mentions of 2 News documents were
obtained from Phase I, the others from Phase II (see Figure 1). We observe better IAA values for
Phase II than for Phase I (not shown in table). This might be due to the high ratio of documents from
genre News (7 out of 12) in Phase II, because we observed that IAA for News is higher than in other
genres, independent from the annotation phase.
By comparing column CM (given mentions) and AM (annotated mentions) we see that annotators
rarely use all mentions in the annotated chains. To reduce the chance that these mentions were omitted
due to an oversight, the annotation submission system indicated which mentions were left unused.
Annotator Profiles. Anonymized learner profiles were collected from all students in Phase II (written
permission for using and publishing the data was also obtained). Learner profiles include age, gender,
native language, languages spoken at home, in primary and secondary school, foreign language knowledge,
and the number of years the annotator spent in Turkish-speaking communities.
Annotators are on average 23 years old university students at Marmara University in Istanbul. Annotations were done after basic introduction to linguistics and coreference resolution and after discussing
the annotation manual (Sürmeli et al., 2016).
6

Out of 46 annotators, 29 are male and 17 are female. One annotator indicated Azerbaijani as a
native language, all others indicated Turkish as one of their native languages. (Azerbaijani is close to
Turkish.) Two annotators indicated Kurdish as further native language, and one each Arabic, English,
and Macedonian. Primary and secondary school education was Turkish for 43 annotators, English for
two and Azerbaijani for one. Moreover 43 annotators lived at least 20 years in predominantly Turkishspeaking communities, the remaining annotators answered 4, 5, and 14 years, respectively, for this
question.
According to this data we consider our annotators to be capable of understanding and annotating
the texts in the corpus.

3.4

Tools

For creating this corpus, we built several tools.
Document Extractor. The METU-Sabanci Turkish Treebank contains 1960 text fragments, distributed over 33 documents. Most documents are split over several XML files, however there is also
one XML file containing two distinct documents. We provide a tool for extracting documents from the
Turkish Treebank and store each document in a single XML file. The Turkish Treebank is licensed
in a way that it cannot be redistributed with the Marmara Turkish Coreference Corpus, therefore the
tool generates document files from a directory containing the unpacked Turkish Treebank. Our tool not
only creates one XML file for each document, it also recodes all data to UTF-8 and fixes problematic
(non-encoded) attributes that are present in the original corpus.
Coreference XML Format. For representing coreference information we created an XML format
that contains pointers to sentence and word IDs into documents extracted from the Turkish Treebank.
A sample of such an XML file with two mentions and one chain is as follows.
<coref>
<mentions>
<mention fromWordIX="1" id="0" sentenceNo="00016112313.1" toWordIX="1">
Prof._Dr._Semih_Koray’ın
</mention>
<mention fromWordIX="1" id="2" sentenceNo="00016112313.2" toWordIX="1">
Koray
</mention>
</mentions>
<chains>
<chain>
<mention mentionId="0">Prof._Dr._Semih_Koray’ın</mention>
<mention mentionId="2">Koray</mention>
</chain>
</chains>
</coref>

In this example, ‘Prof._Dr._Semih_Koray’ın’ is a mention with ID 0 containing token 1 (called ‘word’ in
the Treebank) in sentence ‘00016112313.1’ of the document assembled from the Treebank. Moreover there
is a chain containing that mention and another mention in the first token of sentence ‘00016112313.2’.
Note that the actual text within mentions is only for readability purposes, the information about
mention content is fully represented in attributes.
CoNLL ⇔ XML Converters. As the CoNLL reference coreference scorer (Pradhan et al., 2014) is
based on CoNLL format, we provide tools for converting a document and a coreference XML file into
a CoNLL file (and vice versa). CoNLL format is also more readable than XML Format for humans.
We use XML to be consistent with the Turkish Treebank and because the Treebank license prevents
redistribution.

7

(Semi-)automatic coreference adjudication tool. Merging several distinct coreference annotations
into a single gold standard is a complex task, in particular if annotators do not agree on mentions. To
simplify this task we created a tool that merges multiple annotations into a single solution according
to objective (1) from Section 3.2. Manual intervention for editing mentions and chains is also possible,
details about a preliminary (Phase I) version of the tool is described by Schüller (2016). Note that, in
Phase II we performed only automatic adjudication and did not need manual intervention.
For the purpose of this project, it was sufficient to use our tool directly on CoNLL files without a
GUI. In the future, to make the tool accessible to a wider part of the community, we consider integrating
it into an existing software, for example BART (Broscheit et al., 2010).

4

Baseline

We have created a baseline for mention detection, based on the work of Sapena et al. (2012), and for
coreference resolution, inspired by Bengtson and Roth (2008). The baseline is based on Python and
scikit-learn (Pedregosa et al., 2011). We considered to integrate also the Named Entity Recognition
(NER) module of the ITU-pipeline (Eryiğit, 2014) because NER is not annotated in the Turkish Treebank, however we found that the output the web service changed significantly several times during the
development of the baseline. To facilitate reproducibility of results we decided to create a stand-alone
baseline that allows reproducing our results using only the METU-Sabanci Turkish Treebank (Say et al.,
2004) and scikit-learn.
Mention Detection. Our Mention Detection baseline marks all
(i) noun phrases,
(ii) pronouns, and
(iii) capitalized common nouns or proper names that occur two or more times in the document
as mentions, similar to the approach of Sapena et al. (2012) for English.
Coreference Resolution. Our Coreference Resolution baseline is inspired by the work of Bengtson
and Roth (2008) who describe a simple method with reasonable accuracy and without the need for
custom machine learning models and algorithms that are specific to coreference resolution.
As input the baseline uses a set of candidate mentions (either gold or predicted), furthermore lemma
and dependency parsing information for obtaining mention heads, as available in the Turkish Treebank.
The type of a mention is marked as pronoun if the lemma of the head is one of ben, sen, biz, siz, bu, şu,
o, bura, şura, ora, kendi, birbiri. To separate proper noun from noun phrase mention types, we realized
our own heuristic which (i) collects all uppercase tokens not at sentence-initial position, (ii) strips case
markers, and (iii) uses the resulting set of strings to mark all (including sentence-initial) tokens as proper
nouns. All remaining mentions obtain the type noun phrase.
Based on mention types and head information we create the following features for each mentionmention pair (m1 , m2 ):
(i) type of m1 and type of m2 (2 features),
(ii) both mentions are pronouns, proper nouns, or noun phrases (3 features),
(iii) heads of m1 and m2 match, and the same for head lemmas (2 features),
(iv) last part of name is equal in m1 and m2 , and
(v) m1 is an acronym of m2 ,
(vi) head of m1 is a substring of head of m2 , and the same for head lemmas (2 features),
(vii) m1 is an acronym of m2 .

8

RMD

PMD

MUC

B3

CEAFe

CEAFm

BLANC

News
Short Story
Novel
Essay
Research.M
Article
Travel
Other

82.6
89.6
84.0
85.7
76.5
71.4
71.4
89.6

14.7
22.0
19.9
7.1
13.4
11.6
10.5
23.9

81.1
77.4
79.0
76.1
76.5
78.1
66.1
72.1

74.9
55.5
56.0
70.4
56.0
70.7
57.6
47.7

71.8
57.2
51.2
66.8
51.8
68.7
55.2
46.6

73.2
56.6
55.4
67.1
55.6
70.6
55.3
45.6

72.3
60.7
61.7
63.4
62.8
68.6
52.6
50.9

Total

83.2

16.9

77.8

62.1

59.7

61.4

63.8

Genre

Table 2: Evaluation of baseline by genres: RMD and PMD show recall and precision for mention detection,
other columns show F1 scores of the respective metrics for coreference resolution on gold mentions using
linear SVC all-links method. Accumulations are averages weighted by document size (# tokens).
Features (v)–(vii) are asymmetric, that means exchanging m1 and m2 can change the feature value. For
these features we also add the respective reverse direction feature, as well as the disjunction of features of
both directions. Moreover we add all possible pairs of features (i)–(ii) and (iii)–(vii) to allow the machine
learning to give separate weight to features (iii)–(vii) per mention type.
We implemented two machine learning methods for predicting coreference based on classification
(SVC) and regression (SVR).
SVC is based on classification with a linear-kernel SVM (Corinna Cortes and Vapnik, 1995). Positive
examples are mentions and their closest predecessors within all chains, while negative examples are all
mention pairs that are not in the same chain with less than 100 mentions distance. For predicting
chains, we first generate candidate mention pairs for all mentions except for noun phrases with pronoun
predecessor, then we predict whether they are in the same chain using the SVM. Finally, each mention
starts in its own chain and we go through mentions from the beginning of the document to the end,
and merge mentions to (potentially several) previous chains for all predicted mention-mention links. We
prevent merges that lead to chains with overlapping mentions.
SVR is based on support vector regression with a linear-kernel SVM (Drucker et al., 1997) trained
on the same examples as SVC. For prediction we generate the same candidate mentions as in SVC. For
building chains we also start with one chain per mention, but this time we use the Best-Link (Bengtson
and Roth, 2008) strategy: we iterate over mentions in order of occurrence in the document, and merge
each mention with at most one predecessor chain if its highest-scored candidate link to a predecessor
mention is above 0.1 and if the resulting chain does not contain overlapping mentions.
In addition to the above, when predicting coreference on predicted mentions, we include incorrect
mentions predicted on the training documents to generate negative examples. We randomly sample
at most as many incorrect mentions as already contained in the gold annotation. When predicting
coreference on gold mentions, we train only on gold mentions. We balance example weight by class size
(we have significantly more negative examples), and we use L2 regularization for both SVC and SVR.

4.1

Evaluation

We evaluate our baseline using the CoNLL reference coreference scorer (Pradhan et al., 2014) and report
MUC, B3 , CEAFm , CEAFe , and BLANC scores. Intuitively, MUC (Vilain et al., 1995) computes
precision and recall of mention-mention links over all gold chains compared with all predicted chains.
B3 (Bagga and Baldwin, 1998) computes precision and recall over each individual mention which makes
the score also applicable to singleton mentions. CEAFm (Luo, 2005) creates an optimal matching between
predicted and gold mentions and evaluates the percentage of mentions correctly assigned to chains,
while CEAFe does the same from the perspective of chains. BLANC (Recasens and Hovy, 2010) gives
equal importance to mention-mention links and non-existing mention-mention links, without the need
to consider per-mention or per-chain score accumulations.
In the following we compute scores per document and accumulate them into overall and genre-based
scores using the number of tokens in each document as weight. Mention detection is done on the Turkish
9

MUC

B3

CEAFe

CEAFm

BLANC

average
best
worst

77.8
92.3
61.8

62.1
89.8
39.6

59.8
88.5
37.2

61.4
90.4
45.2

63.8
88.1
45.0

SVR/GM

average
best
worst

75.0
92.4
60.6

60.6
90.0
36.3

60.5
89.2
39.5

61.3
90.8
42.6

61.7
87.8
38.9

SVR/PM

average
best
worst

32.2
55.0
7.6

23.3
35.7
8.5

17.2
31.2
9.9

24.7
37.9
9.3

18.1
33.5
1.4

Method

Accumulation

SVC/GM

Table 3: Evaluation of our baseline using support vector machine classification (SVC) and regression
(SVR) on gold (GM) and predicted mentions (PM). Scores are F1 scores, averages are again weighted
by document size, best/worst rows are over single documents. SVC on PM is not applicable (see text).
Treebank data and is a deterministic algorithm which does not require learning. Coreference resolution is
done either on gold mentions (GM) or on predicted mentions (PM). Scores are obtained by leave-one-out
cross-validation on all 33 documents of the corpus, yielding 33 folds. All scores are given in percent. For
mention detection we report Precision and Recall, for coreference scores we report only F1.
Table 2 shows baseline results for mention detection and coreference with SVC on gold mentions.
We obtain 83.2% recall for mention detection over the whole Treebank. As expected and as intended,
precision is much worse because we expect the coreference resolution step to eliminate spurious mentions.
Coreference resolution yields a MUC score of 77.8%, while the stricter B3 , CEAF, and BLANC scores
are lower. The worst scores for BLANC, MUC, and CEAFm are obtained from genres Travel and Other,
which is logical because these genres contain only one document each. Hence, in cross-validation, the
training set contains only documents from other genres.
Table 3 shows results for the best and worst document in the corpus, and overall average of scores for
SVC, SVR on gold mentions and SVR on predicted mentions. Average is again weighted by document
size. SVC on gold mentions is the same setup as in Table 2. For comparison we show again the average
score, and additionally the best and worst score obtained from a single document. Using SVR on gold
mentions yields scores similar to SVC except for BLANC where the worst document is 6.1% worse
than with SVM. On predicted mentions, SVC puts nearly all mentions into a single chain, which yields
reasonable MUC score but low other scores. Therefore, we omit SVC results on predicted mentions
from the table, and show only results for SVR which has the systematic advantage of selecting only the
best link. Naturally, coreference prediction on predicted mentions yields significantly worse results than
on gold mentions, with an average MUC score of 32.2% and an average BLANC score of 18.1%. The
document producing the worst BLANC score of 1.4% is a scientific article about the meaning of the word
‘home’, which is a text that contains mainly abstract examples about the animal world, hence the score
is not surprising. The best BLANC score on predicted mentions is obtained from a story about street
children which is about concrete events and individuals.
As this is only a baseline, we did not include more sophisticated features described in (Bengtson and
Roth, 2008). For example, semantic features based on WordNet (Bilgin et al., 2004; Miller, 1995) could
rule out certain predicted mentions as relevant and thus could improve precision of the baseline.

5

Conclusion

The Turkish coreference corpus and the Turkish coreference resolution baseline are a starting point for
further research.
Annotations on the token level have the consequence that coreference between morphemes and other
tokens (or morphemes) cannot be annotated. While such coreference annotations are desirable, we
decided to reduce the complexity of the annotation task by omitting such coreference links and by
presenting tokens without morphological analysis to annotators. For future annotation projects it could
be interesting to extend annotations to include morpheme coreference links. Scoring with the reference
10

scorer tool would require development of a novel CoNLL representation for tokens that are split within
IGs. If this representation is not used for annotation, development of a tool for such annotations is also
necessary. We think this would also require annotators with a higher level of expertise than available for
our study.
As the only pronoun anaphora annotations (Küçük and Yöndem, 2007) that were done on the METUSabanci Turkish Treebank can no longer be found by the authors (personal communication), we have no
possibility of validating the annotations performed in our annotation project.
To improve the mention detection baseline, information about appositives as well as finding a way
to filter out generic mentions could be useful. To improve the coreference resolution baseline, adding
more complex features by integrating Turkish WordNet (Bilgin et al., 2004), Turkish NER (Şeker and
Eryiğit, 2012), and Turkish WSD (İlgen et al., 2012) could be helpful. For a full processing pipeline
from plain text to coreference annotations, we need at least morphological analysis (Sak et al., 2007),
disambiguation (Sak et al., 2007), and dependency parsing (Eryiğit et al., 2008). Available tools are the
ITU-pipeline (Eryiğit, 2014) and the older Zemberek system (Akın and Akın, 2007).

Acknowledgements
We are grateful to Kemal Oflazer and Bilge Say for support about the METU-Sabanci Turkish Treebank,
and to Dilek Küçük and Savaş Yıldırım for support about their papers and datasets.
This work has been supported by The Scientific and Technological Research Council of Turkey
(TUBITAK) under grant agreements 114E430 and 114E777.
Author contributions are as follows (in chronological order): creating documents from Treebank
fragments (B.G.S), evaluating GATE and BRAT annotation software (B.G.S., A.P.), creating the annotation manual (P.S., K.C., F.T., B.G.S.), performing manual adjudication in Phase I (K.C., B.G.S., A.P.,
H.E.K.), annotating mentions for those documents in Phase II that were not annotated in Phase I (K.C.,
H.E.K.), revising the manuscript as an English native-speaker (A.H.K.), writing the baseline software (P.S., F.T., A.P.), managing the project, performing baseline experiments, analyzing data and
result, and writing the manuscript (P.S).

References
Akın, A. A. and Akın, M. D. (2007). Zemberek, An Open Source NLP Framework for Turkic Languages.
Structure, 10:1–5.
Atalay, N. B., Oflazer, K., and Say, B. (2003). The Annotation Process in the Turkish Treebank. In
International Workshop on Linguistically Interpreted Corpora (LINC).
Bagga, A. and Baldwin, B. (1998). Algorithms for Scoring Coreference Chains. In International Conference on Language Resources and Evaluation (LREC), pages 563–566.
Bengtson, E. and Roth, D. (2008). Understanding the value of features for coreference resolution. In
Empirical Methods in Natural Language Processing (EMNLP), pages 294–303.
Bilgin, O., Çetinoğlu, Ö., and Oflazer, K. (2004). Building a Wordnet for Turkish. Romanian Journal
of Information Science and Technology, 7(1-2):163–172.
Broscheit, S., Poesio, M., Ponzetto, P. S., Rodriguez, J. K., Romano, L., Uryupina, O., Versley, Y., and
Zanoli, R. (2010). BART: A Multilingual Anaphora Resolution System. In International Workshop
on Semantic Evaluation (SemEval), pages 104–107.
Buchholz, S. and Marsi, E. (2006). CoNLL-X shared task on Multilingual Dependency Parsing. In
Computational Natural Language Learning (CoNLL), pages 149–164.
Chang, K.-W., Samdani, R., and Roth, D. (2013). A Constrained Latent Variable Model for Coreference
Resolution. In Empirical Methods in Natural Language Processing (EMNLP), pages 601–612.

11

Chen, C. and Ng, V. (2012). Combining the Best of Two Worlds: A Hybrid Approach to Multilingual
Coreference Resolution. In Empirical Methods in Natural Language Processing and Computational
Natural Language Learning, Shared Task, pages 56–63.
Corinna Cortes and Vapnik, V. (1995). Support Vector Networks. Machine Learning, 20(3):273–297.
Culotta, A., Wick, M., and Mccallum, A. (2007). First-Order Probabilistic Models for Coreference Resolution. In Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pages 81–88.
Cunningham, H., Tablan, V., Roberts, A., and Bontcheva, K. (2013). Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLoS computational biology,
9(2):e1002854.
Denis, P. and Baldridge, J. (2009). Global joint models for coreference resolution and named entity
classification. Procesamiento del Lenguaje Natural, 42:87–96.
Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., and Weischedel, R. (2004).
The Automatic Content Extraction (ACE) Program-Tasks, Data, and Evaluation. In International
Conference on Language Resources and Evaluation (LREC), pages 837–840.
Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A., and Vapnik, V. (1997). Support vector regression
machines. Advances in Neural Information Processing Systems, 9:155–161.
Erkan, T. and Akman, V. (1998). Situated Processing of Pronominal Anaphora. In Verarbeitung natürlicher Sprache (KONVENS), pages 369–378. Bilkent University.
Eryiğit, G. (2014). ITU Turkish NLP Web Service. In Conference of the European Chapter of the
Association for Computational Linguistics (EACL) Demonstrations, pages 1–4.
Eryiğit, G., Nivre, J., and Oflazer, K. (2008). Dependency Parsing of Turkish. Computational Linguistics,
34(3):357–389.
Fernandes, E. R., dos Santos, C. N., and Milidiú, R. L. (2012). Latent structure perceptron with feature
induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL: Shared
Task, pages 41–48.
Gaizauskas, R., Cunningham, H., Wilks, Y., Rodgers, P., and Humphreys, K. (1996). GATE: an environment to support research and development in natural language engineering. In International
Conference on Tools with Artificial Intelligence (ICTAI), pages 58–66.
Grishman, R. and Sundheim, B. (1995). Design of the MUC-6 evaluation. In Message Understanding
Conference (MUC), pages 413–422.
Grosz, B. J., Joshi, A. K., and Weinstein, S. (1995). Centering: A Framework for Modelling the Local
Coherence of Discourse. Computational Linguistics, 21(2):203–225.
Hakkani-Tür, D. Z., Oflazer, K., and Tür, G. (2002). Statistical morphological disambiguation for
agglutinative languages. Computers and the Humanities, 36(4):381–410.
Hirst, G. (1981). Anaphora in Natural Language Understanding: A Survey. Springer.
Hobbs, J. R. (1978). Resolving Pronoun References. Lingua, 44(4):311–338.
Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. (2006). OntoNotes: the 90%
solution. In Human Language Technology Conference of the NAACL, Short Papers, pages 57–60.
Krippendorff, K. (1980). Content Analysis: An Introduction to Its Methodology. Sage Publications.
Küçük, D. and Yazıcı, A. (2008). Identification of coreferential chains in video texts for semantic annotation of news videos. In International Symposium on Computer and Information Sciences (ISCIS).

12

Küçük, D. and Yöndem, M. T. (2007). Automatic Identification of Pronominal Anaphora in Turkish
Texts. In International Conference on Computer and Information Science (ICIS).
Kılıçaslan, Y., Güner, E. S., and Yıldırım, S. (2009). Learning-based pronoun resolution for Turkish with
a comparative evaluation. Computer Speech & Language, 23(3):311–331.
Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., and Dan Jurafsky (2013). Deterministic
Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules. Computational Linguistics,
39(4):885–916.
Luo, X. (2005). On coreference resolution performance metrics. In Conference on Human Language
Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 25–32.
Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM, 38(11):39–
41.
Mitkov, R. (2002). Anaphora Resolution. Longman.
Ng, V. (2010). Supervised Noun Phrase Coreference Research: The First Fifteen Years. In Association
for Computational Linguistics (ACL), pages 1396–1411.
Oflazer, K. (1993). Two-level Description of Turkish Morphology. In Conference of the European Chapter
of the Association for Computational Linguistics (EACL), pages 472–472.
Oflazer, K. and Bozşahin, H. C. (1994). Turkish Natural Language Processing Initiative: An Overview.
In Turkish Symposium on Artificial Intelligence and Neural Networks.
Oflazer, K., Göçmen, E., and Bozşahin, C. (1994). An Outline of Turkish Morphology. Technical report,
NATO Science Division SfS III, Brussels.
Oflazer, K., Say, B., Hakkani-Tür, D. Z., and Tür, G. (2003). Building a Turkish Treebank. In Treebanks,
Building and Using Parsed Corpora, pages 261–277. Springer Netherlands.
Passonneau, R., Habash, N., and Rambow, O. (2006). Inter-annotator agreement on a multilingual
semantic annotation task. In International Conference on Language Resources and Evaluation (LREC),
pages 1951–1956.
Passonneau, R. J. (2004). Computing reliability for coreference annotation. In Language Resources and
Evaluation (LREC), pages 1503–1506.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot,
M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning
Research, 12:2825–2830.
Poesio, M., Mehta, R., Maroudas, A., and Hitzeman, J. (2004). Learning to resolve bridging references.
In Annual Meeting on Association for Computational Linguistics (ACL). Paper #143.
Pradhan, S., Luo, X., Recasens, M., Hovy, E., Ng, V., and Strube, M. (2014). Scoring Coreference
Partitions of Predicted Mentions: A Reference Implementation. In Annual Meeting of the Association
for Computational Linguistics (ACL), pages 30–35.
Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012). CoNLL-2012 Shared Task:
Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and
CoNLL: Shared Task, pages 1–40.
Pradhan, S. S., Ramshaw, L., Weischedel, R., MacBride, J., and Micciulla, L. (2007). Unrestricted
Coreference: Identifying Entities and Events in OntoNotes. In International Conference on Semantic
Computing (ICSC 2007), pages 446–453.
Recasens, M. and Hovy, E. (2010). BLANC: Implementing the Rand Index for Coreference Evaluation.
Natural Language Engineering, 17(4):485–510.
13

Recasens, M., Màrquez, L., Sapena, E., Martí, M. A., Taulé, M., Hoste, V., Poesio, M., and Versley,
Y. (2010). SemEval-2010 Task 1: Coreference Resolution in Multiple Languages. In International
Workshop on Semantic Evaluation (SemEval) at ACL, pages 1–8.
Sak, H., Güngör, T., and Saraçlar, M. (2007). Morphological disambiguation of Turkish text with
perceptron algorithm. In Intelligent Text Processing and Computational Linguistics, pages 107–118.
Sapena, E., Padro, L., and Turmo, J. (2012). A Constraint-Based Hypergraph Partitioning Approach to
Coreference Resolution. Computational Linguistics, 39(4):847–884.
Say, B., Zeyrek, D., Oflazer, K., and Özge, U. (2004). Development of a Corpus and a Treebank
for Present day Written Turkish. In İmer, K. and Doğan, G., editors, Current Research in Turkish
Linguistics (Proceedings of the Eleventh International Conference of Turkish Linguistics, 2002), pages
182–192. Eastern Mediterranean University Press.
Schüller, P. (2016). Adjudication of Coreference Annotations via Finding Optimal Repairs of Equivalence
Relations. In International Workshop on Experimental Evaluation of Algorithms for Solving Problems
with Combinatorial Explosion (RCRA), pages 57–71.
Soon, W., Ng, H., and Lim, D. (2001). A machine learning approach to coreference resolution of noun
phrases. Computational linguistics, 27(4):521–544.
Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S., and Tsujii, J. (2012). Brat: a Web-based
Tool for NLP-Assisted Text Annotation. In European Chapter of the Association for Computational
Linguistics (EACL), pages 102–107.
Stoyanov, V. and Eisner, J. (2012). Easy-first Coreference Resolution. In International Conference on
Computational Linguistics (COLING), pages 2519–2534.
Sürmeli, B. G., Cıngıllı, K., Tunçer, F., and Schüller, P. (2016). Turkish coreference annotation manual
(V2). https://tinyurl.com/jovfmua .
Tüfekçi, P. and Kılıçaslan, Y. (2007). A Computational Model for Resolving Pronominal Anaphora in
Turkish Using Hobbs’ Naïve Algorithm. International Journal of Computer, Electrical, Automation,
Control and Information Engineering, 1(5):1402–1406.
Turan, Ü. D. (1996). Null Vs. Overt Subjects in Turkish. PhD thesis, University of Pennsylvania.
van Deemter, K. and Kibble, R. (2000). On Coreferring: Coreference in MUC and Related Annotation
Schemes. Computational Linguistics, 26(4):629–637.
Vilain, M., Burger, J., Aberdeen, J., Connolly, D., and Hirschman, L. (1995). A Model-Theoretic
Coreference Scoring Scheme. In Message Understanding Conference (MUC), pages 45–52.
Weischedel, R., Pradhan, S., Ramshaw, L., Kaufman, J., Franchini, M., El-Bachouti, M., Xue, N.,
Palmer, M., Hwang, J. D., Bonial, C., Choi, J., Mansouri, A., Foster, M., Hawwary, A.-a., Marcus,
M., Taylor, A., Greenberg, C., Hovy, E., Belvin, R., and Houston, A. (2012). OntoNotes Release 5.0.
Yüksel, Ö. and Bozsahin, C. (2002). Contextually appropriate reference generation. Natural Language
Engineering, 8(1):69–89.
Yıldırım, S., Kılıçaslan, Y., and Aykaç, R. E. (2004). A Computational Model for Anaphora Resolution
in Turkish via Centering Theory: an Initial Approach. In International Conference on Computational
Intelligence, pages 124–128.
Zheng, J., Vilnis, L., Singh, S., Choi, J. D., and McCallum, A. (2013). Dynamic Knowledge-Base
Alignment for Coreference Resolution. In Computational Natural Language Learning (CoNLL), pages
153–162.
İlgen, B., Adalı, E., and Tantuğ, A. C. (2012). Building up Lexical Sample Dataset for Turkish Word
Sense Disambiguation. In International Symposium on Innovations in Intelligent Systems and Applications (INISTA), pages 1–5.
14

Şeker, G. A. and Eryiğit, G. (2012). Initial explorations on using CRFs for Turkish Named Entity
Recognition. In International Conference on Computational Linguistics (COLING), pages 2459–2474.

15

