_:n3fbfd1474a1f424b8b21edd4a057e1e7xb0
<http://schema.org/articleBody>
"Artif Intell Law (2014) 22:29–60 DOI 10.1007/s10506-013-9149-8 • • • Oanh Thi Tran Bach Xuan Ngo Minh Le Nguyen Akira Shimazu Published online: 1 December 2013 Springer Science+Business Media Dordrecht 2013 Abstract This paper investigates the task of reference resolution in the legal domain. This is a new interesting task in Legal Engineering research. The goal is to create a system which can automatically detect references and then extracts their referents. Previous work limits itself to detect and resolve references at the docu- ment targets. In this paper, we go a step further in trying to resolve references to sub-document targets. Referents extracted are the smallest fragments of texts in documents, rather than the entire documents that contain the referenced texts. Based on analyzing the characteristics of reference phenomena in legal texts, we propose a four-step framework to deal with the task: mention detection, contextual informa- tion extraction, antecedent candidate extraction, and antecedent determination. We also show how machine learning methods can be exploited in each step. The ﬁnal system achieves 80.06 % in the F1 score for detecting references, 85.61 % accuracy for resolving them, and 67.02 % in the F1 score for the end-to-end setting task on the Japanese National Pension Law corpus. Keywords Reference resolution  Mention detection  Legal texts  JNPL corpus 1 Introduction Legal Engineering (Katayama 2007, 2010; Katayama et al. 2008) is a new research ﬁeld which aims to achieve a trustworthy electronic legal society. The main goals are to help experts make complete and consistent laws and to design an information system which works based on laws. Hence, it is vital to develop a system which can O. T. Tran (&)  B. X. Ngo  M. L. Nguyen  A. Shimazu Laboratory of Natural Language Processing, School of Information Science, Japan Advanced Institute of Science and Technology (JAIST), 1-1 Asahidai, Nomi, Ishikawa 923-1292, Japan e-mail: oanhtt@gmail.com; oanhtt@jaist.ac.jp 123 30 O. T. Tran et al. process legal texts automatically. One of the obstacles to law processing is that, at the discourse level, legal texts contain a lot of reference phenomena. These references usually contain precious information. The law will be difﬁcult to comprehend if we cannot access the referenced items within it. Resolving the reference phenomena, therefore, is an important task in Legal Engineering. Figure 1 shows excerpts from three documents named A12P1, A12P4, and A13P1. These excerpts contains two references (the texts bounded in red angle brackets), i.e. ‘the provision of previous article, para 4’ in the document A13P1 and ‘the notiﬁcation in the provision of para 1’ in the document A12P4. To comprehend the contents of these documents, it is important to know the referenced items. In other words, we need to know to which part of texts (the texts bounded in green square brackets) these references refer. This kind of references is very popular in legal documents because law-makers usually import pieces of available information which have already been introduced in other documents by using briefer expressions. This, as a result, helps to guarantee the soundness as well as the consistency in a law system. We name these briefer expressions ‘references (or mentions), and their referenced items ‘referents (or antecedents)’. We will discuss the characteristics of references as well as their referents later in Sect. 3. There exists some work related to this kind of research, developed for several languages such as Italian (Bolioli et al. 2002; Palmirani et al. 2003), Spanish (Mercedes et al. 2005), and Dutch (Maat et al. 2006). In previous work, authors focus on detecting and resolving so-called normative references to distinguish them from the above references. Normative references are slightly different from the above references. In the above examples, normative references would appear in the forms of ‘para 1’ and ‘previous article, para 4’. In resolving these normative references, authors limit resolvers to identifying only the referred documents but not to which parts of texts in these documents . Normative references are much easier to detect and resolve because most references are correct and unambiguous. They mostly follow some regular expressions. Therefore, knowing the naming rules of the law, we can easily determine the identiﬁcation of the document which is referred to by a reference. In this paper, we study the reference resolution task for a non-Western language, particularly the Japanese language. Different from previous work, which is limited to determine the positions of the referred documents (A12P1 and A12P4), in this research, we go a step further. Our reference resolver tries to extract the smallest fragments of texts that are actually referred to by references (the texts in green The term ‘documents’ corresponds to articles, paragraphs, items, or sub-items according to the naming rules used in the legal domain. These two reference examples are two typical examples of two classes of references, which will be described in more detail later. In this research we derive the term ‘mention’ and ‘antecedent’ from reference resolution in general texts to describe this relationship. We use the term ‘mentions’ to denote references that contain referring texts. The texts that mentions refer to are called ‘antecedents’. With this output, users/law-makers need to read over the referenced document to ﬁnd which part of texts is actually referred to. 123 Automated reference resolution 31 Fig. 1 Examples of reference phenomena in legal texts (in this ﬁgure, references are bounded in red angle brackets (hi) while their referents are bounded in green square brackets ([ ])). Expressions start after a colon (:) or a semicolon (;) in the bounded texts are the identiﬁcation expressions (ID) of these texts (i.e. A12P1-1). A reference and its referent have the same ID. (Color ﬁgure online) square brackets). Resolving this type of references is more difﬁcult because it requires syntactic and semantic understandings of references and its context information as well as of the referenced document that contains the referenced texts. Therefore, the methods used in previous work are not sufﬁcient to resolve these references. Based on the characteristics of reference phenomena in legal texts which will be discussed in Sect. 3, we propose a four-step framework using machine learning approaches which results in the ﬁnal system being automatically trainable from a corpus with a minimal amount of human intervention. In the proposed framework, we recognize references (mentions) in the ﬁrst step—mention detection. For each output reference, we then extract its contextual information in the second step—contextual information extraction. This contextual information will be used in the third step—antecedent candidate extraction—to extract referent (antecedent) candidates of this mention. The fourth step—antecedent determination—will select the best one among its candidates to determine its exact referents. In ﬁnding solutions to each step of the framework, we prefer statistical and machine learning approaches (SML) to rule based approaches. There are several reasons for using SML. Firstly, SML can recognize patterns well by learning from data. Secondly, 123 32 O. T. Tran et al. SML can deal with ambiguities in rule-based approach due to the ability to integrate many kinds of features. Thirdly, using SML, we do not need to design rules. Designing rules requires domain knowledge. This process is also time-consuming. Our main contributions can be summarized as follows: • Introducing the task of reference resolution in the domain of legal texts, in which we detect references and then map them to the smallest fragment of texts that they refer to. • Analyzing the characteristics of references in the legal texts. Based on this analysis, we propose a four-step framework using machine learning approaches to solve the task. • Introducing a new annotated corpus, the Japanese National Pension Law (JNPL) corpus on reference resolution. • Conducting experiments and evaluating our framework on the JNPL corpus. From the experiments, we obtained 80.06 % in the F1 score in detecting references and 85.61 % accuracy in resolving them. In the whole system, we obtained 67.02 % in the F1 score. Building a good reference resolver brings many potential beneﬁts such as providing computer science supports in making, maintaining and validating legal documents; supporting in ﬁnding contradictions in legal texts, which is obtained by reasoning and semantic processing such as semantic parsing and reference resolution; supporting in building a question answering system that allows citizens to have easier access to legal information, etc. The rest of this paper is organized as follows: Sect. 2 reviews some related work. Section 3 introduces some characteristics of reference phenomena in legal texts. Based on those characteristics, Sect. 4 presents our proposed framework to solve this task. Section 5 presents solutions for each step in the proposed framework. Next, we will describe experiments in Sect. 6 In this section, we also analyze the impact of each step on the whole system and present an output example of the ﬁnal system. In addition, we propose a semi-supervised technique to improve the performance of the ﬁnal system. Section 7 provides an analysis of errors in the proposed framework. In Sect. 8, we discuss the performance as well as the versioning problem of the system. Finally, Sect. 9 concludes the paper and proposes future work. 2 Related work This section presents three kinds of related works, and places our work in the scope of research on references. Firstly, we review some typical studies which also consider resolving a fragment of texts to documents or sub-document targets. Secondly, we review some work on reference resolution conducted in general texts. Finally, we focus on studies on detecting and resolving references within the legal domain, which are closest to our work. In this ﬁnal sub-section, we also distinguish how our work is different from the previous work. 123 Automated reference resolution 33 2.1 Studies on resolving a fragment of texts to documents or sub-document targets If we consider each reference as an anchor and its referent as a linked document, our work somehow can be regarded as citation linking which is studied in works such as Entity-linking, Linking-the-Wiki, citation processing, etc. • Entity-Linking is a track in a series of evaluations and workshops organized to promote research in natural language processing and related applications. Given a named entity of a person, organization, etc. and a document containing that name, the goal of this track is to determine the Knowledge Base node (KB) for that named entity, adding a new node for the entity if it is not already in the KB. The reference KB is derived from English Wikipedia, while source documents come from a variety of languages, including English, Chinese, and Spanish. Such systems typically search for candidate entities and then disambiguate them. Some representative works are by Hachey et al. (2013) and Hachey et al. (2011). • Link-the-Wiki aims to produce a standard procedure and metrics for the evaluation of link discovery between documents. Given a new orphan (unlinked) document, the task is to analyze the text and recommend a set of outgoing links from anchors (speciﬁed as passages in the orphan document) to best entry points (BEPs) in existing documents in the collection. The BEP for a link should be the position in the target document from which the reader, having just followed the link, should begin reading. • Citation Processing: A common prerequisite for knowledge discovery is to accurately combine data from multiple, heterogeneous sources into a uniﬁed, mineable database. An important step in creating such a database is record deduplication, consolidating multiple records that refer to the same abstract entity [i.e. Culotta and McCallum (2005)]. 2.2 Studies on reference resolution in general texts Reference resolution (Jurafsky and Martin (2009) is the task of determining which entities are referred to by which linguistic expressions. This task plays an important role in a large number of natural language processing (NLP) applications. Therefore, it has attracted much attention within the NLP community. Among all types of reference phenomena, coreference is the most popular and is the focus of most research on reference resolution in general texts. When the reader must look back to the previous context, reference is called ‘anaphoric reference’. When the reader must look forward, it is termed ‘cataphoric reference’. Many works on various aspects [linguistic features (Haghighi and Klein 2009;Ng 2007), machine learning models (Soon et al. 2001), multiple languages (Recasens et al. 2010), etc.] http://www.nist.gov/tac/2013/KBP/EntityLinking/index.html. http://en.wikipedia.org/wiki/Main_Page. http://www.inex.otago.ac.nz/tracks/wiki-link/wiki-link.asp. 123 34 O. T. Tran et al. of the reference resolution in the general domain have been published. This has boosted the development of robust machine learning models for the task in the general domain. 2.3 Studies on reference resolution within the legal domain This section describes studies on reference resolution of legal texts, which are the most closely related to our work. To the best of our knowledge, there are only a few studies done for Italian, Dutch, and Spanish legal texts. For Italian legal texts, there exists two typical studies. The ﬁrst one is the work of Palmirani et al. (2003), in which the authors conduct a project to work out and implement a model for recognizing, understanding, normalizing the normative references found in legal texts and bringing such references under a set of common standards in order to favour the interoperability between different legal information systems. The second one is the work of Bolioli et al. (2002), in which the authors name references as citations. On the basis of the different writing of the element Act, authors have classiﬁed the various citation formats provided for by the drafting rules into three categories, called: normal, simpliﬁed, and non-paradigmatic citations. They also model each type of citations to extract templates to detect them. The ﬁrst analysis carried out on the laws enacted in the 1990s making up part of the selected legislative corpus conﬁrms that, up until that time, it is possible to identify and extract more than 95 % of the explicit textual legislative citations, conforming to the legislative drafting rules. For Dutch legal texts, Maat et al. (2006) also perform automated detection of reference structures in law. They ﬁrst discuss the type and structure of references in Dutch legal sources, and then propose a grammar containing most of the patterns analyzed to recognize these references. After a reference has been found in a text, they resolve it to the URI of the document that is referred to. They also expand their work to other legal sources. In testing, they achieved a very high accuracy, between 95 and 99 %. For Spanish legal texts, Mercedes et al. (2005) present an application of information extraction. Its goal is to automate the extraction of references from legal documents and the storage of their information in order to facilitate an automatic processing of these information items by services offered in digital libraries. They ﬁrst classify references using four criteria. After that, they extract patterns based on the information about vocabulary used to name legal items and the grammar associated with each type of reference. They use these patterns to extract references for each input document. Each extracted reference is then processed by the solver, that tries to match each reference to some legal items. In previous related works, the authors only consider references in terms of linguistic expressions that identify a speciﬁc act or a text partition referred to. Their purpose is to link the content of laws based on the naming rules used in a speciﬁc legal domain. With the example sentence in Fig. 1, which contains the phrase ‘the notiﬁcation in the provision of paragraph 1’, these works only consider extracting the reference ‘the provision of paragraph 1’, and then resolve them to the entire referenced document (i.e. Paragraph 1 of Article 12). Hence, to understand which 123 Automated reference resolution 35 notiﬁcation is referred to, users need to read that document to get its description. This is somewhat redundant because that document may contain unnecessary information for the comprehension of the input sentence. In this paper, we go a step further. To avoid over-reading these unnecessary texts in the referenced document, we consider a wider range of references. We extract references in the form of which content is referred to and the location containing that content. After that, we resolve them to the exact explanation of that content rather than only the entire document which covers the explanation. In particular, in the example sentence in Fig. 1, we extract the full phrase and resolve it to the smallest fragment of texts that describes the type of the notiﬁcation in Paragraph 1, i.e. ‘notiﬁcation of matters relating to the change of name, address, as well as matters relating to change of type and loss and acquisition of the qualiﬁcation’. Moreover, we propose the use of machine learning approaches (rather than rule- based approaches as used in the previous works) which results in the ﬁnal system being automatically trainable from a corpus with a minimal amount of human intervention. 3 Characteristics of references in legal texts Some characteristics of references in the legal domain are worth consideration when dealing with an automatic extraction and resolution of references. Most types of references in legal texts relate to terms, deﬁnitions or provisions of articles. Although referenced texts can occur before or after a reference, in this research we derive the term ‘mention’ and ‘antecedent’ from reference resolution in general texts to describe this relationship. We use the term ‘mentions’ to denote references that contain referring texts. The texts that mentions refer to are called ‘antecedents’. Mentions in legal texts have their own structures, which are different from mentions in the general domain. They mostly conform to several kinds of patterns. Figure 2 shows that a mention usually consists of two main parts: a position part and a content part. The later part may be a noun or a noun phrase (this noun phrase can be nested). The former part usually conforms to a regular expression. By observing the position parts, we can see that they mostly fall under one of the following types: 1. Single position: concerns a single position of a mention. (a) Well-formed positions: These positions comprise a label, such as an article, a paragraph, or an item combined with a number. They usually start with the broadest scope and end with the narrowest one. These positions contain all the information needed to identify the location of the referred item. (b) Anaphora, indirect positions: They often refer to a position of an earlier mention. These positions are always resolved to one of the former (label In Fig. 2, | means ‘or’, [ ] means ‘optional’, and ? means ‘repeat one or more times’. An example of a mention and its translation into English are also given in Fig. 2. 123 36 O. T. Tran et al. and number). This requires some context information (documents referenced before and near the current mention, in a speciﬁc part of the same texts…) to solve the mention. 2. Coordinated positions: Several antecedents are referred to in the same mention by using some linking expressions such as and, or, from … to, etc. 3. Special cases: These are used when an element in the text contains a list that is preceded by a description of the list, without which the list does not make sense. Another special case is the use of the word each time that refers to sub-items of an article. Figure 3 shows some examples of position parts in mentions of legal texts, which belong to different types as distinguished above. Antecedents usually are deﬁnitions or explanations of related terms or provisions. They can be nouns, noun phrases, sentences, and paragraphs of articles or they can even be whole articles in some cases. They help readers to comprehend the law thoroughly, and also help lawmakers to create legal texts that are concise and easy to understand. 4 A four-step framework to reference resolution in legal texts This section presents our proposed framework for the reference resolver as shown in Fig. 4. In this ﬁgure, oval nodes denote data and box nodes denote processes. In comparison with reference resolution in general texts, this framework has two additional steps (Step 1 and Step 2 that are bounded in the dotted rounded-corner rectangle). These two steps take advantage of the characteristics of references in legal texts for solving the task more effectively. Step 0 Mention detection This step identiﬁes the occurrence of mentions in legal texts. Detecting mentions is very important for the downstream processing of the framework. Each mention extracted in this step will be analyzed via two intermediate steps, viz Step 1 and Step 2. The purpose of Step 1 is to extract the context information of the mention which beneﬁts the extraction of its antecedent candidates in Step 2. Step 1 Context information extraction The goal is to determine the context information, including the locative information that a given mention refers to, documents referenced before and near the current mention, the content part of the mention, and its classiﬁcation. This information will be used later to narrow down the search space of antecedent candidates for that mention. This step is divided into three sub-steps as shown on the right hand side of Fig. 4. • 1.1: Mention splitting—Splits a mention into two parts which are a position part and a content part. These two parts are independent, and therefore should be processed separately. The position part allows to locate the position of the 123 Automated reference resolution 37 Fig. 2 The structure of mentions in legal texts Fig. 3 Some examples of different types of position parts of mentions in legal texts Fig. 4 A four-step framework for resolving references in legal texts 123 38 O. T. Tran et al. document containing the referenced texts while the content part helps in extracting the smallest fragment of texts that this mention refers to. • 1.2: Mention Classiﬁcation—Determines whether a mention refers to an entire document (Class 1) or only a fragment of a document (Class 2). Depending on the classiﬁcation of a mention, different strategies are designed to determine its antecedent. • 1.3: Position Recognition—Locates the scope of antecedents. The goal is to exactly determine which articles, which paragraphs, which items etc., a given mention refers to. Step 2 Antecedent candidate extraction This step uses the position information extracted from Step 1 to determine the scope of the exact document (in which articles, which paragraphs, which items) that a mention refers to. From that document, we extract antecedent candidates for a given mention based on the dependency trees of the antecedent sentences or punctuation marks such as commas and full stops. Step 3 Antecedent determination From a list of candidates extracted in Step 2, we have to determine an exact antecedent for each mention. The exact antecedent is the smallest fragment of texts that this mention refers to. This is the most important and the most difﬁcult step. The strategy to determine the exact antecedent of a mention depends on the classiﬁcation of a mention in the mention classiﬁcation step (Sub-step 1.2). We will describe these strategies in more detail in Sect. 5.5. 5 Solutions to each step of the framework In this section, we will present our proposed solutions for each step of the framework in the previous section. 5.1 Mention detection and mention splitting In the mention detection step, the goal is to extract all mentions appearing in the input string. The input string is a sequence of words and the output is a collection of mentions (or references) contained in the input string. Each extracted mention will be an input for the mention splitting sub-step. This sub-step divides this mention into two parts: the position and the content parts. Similarly to many classical NLP tasks such as text chunking (Pitler et al. 2010) and named entity recognition (Finkel and Manning 2010), we also formulate these two tasks as sequence labeling problems. Sequence labeling is a type of pattern recognition task that involves the algorithmic assignment of a categorical label to each member of a sequence of observed values. A common example of a sequence labeling task is part of speech tagging, which seeks to assign a part of speech to each word in an input sentence or document. Sequence labeling can be treated as a set of 123 Automated reference resolution 39 independent classiﬁcation tasks, one per member of the sequence. However, accuracy is generally improved by making the optimal label for a given element dependent on the choices of nearby elements, using special algorithms to choose the globally best set of labels for the entire sequence at once. In the mention detection step, each word is assigned a label indicating whether it starts a speciﬁc mention, is inside a speciﬁc mention, or is outside any mention. Figure 5 illustrates an example of a Japanese law sentence in the IOB, IOE and FIL notations. In this example, the source sentence contains one mention (red texts) that should be detected. The labels of this sentence using these three notations are described as follows: • In the IOB notation, the ﬁrst element of a mention is tagged with B_M (begin of mention); the remaining elements of the mention are tagged with I_M (inner of mention); all elements outside the mention are tagged with O (Others). • In the IOE notation, the ﬁrst and the intermediate elements of a mention are tagged with I_M (inner of mention); the last element of the mention is tagged with E_M (ending of mention); all elements outside the mention are tagged with O (others). • In the FIL notation, the ﬁrst element of a mention is tagged with F_M (ﬁrst of mention); the intermediate elements of the mention are tagged with I_M (inner of mention); the last element of the mention is tagged with E_M (ending of mention); all elements outside the mention are tagged with O (others). Each detected mention is split into two parts; the position part and the content part. Figure 6 illustrates an example of a mention in the IOB notation: the ﬁrst word of the position part is tagged with B_P (beginning of the position part); the remaining words of the position part are tagged with I_P (inner of the position part); all words of the content part are tagged with O (others). In these two tasks, we use conditional random ﬁelds (CRF) (Lafferty et al. 2001) as the learning method to learn the sequence labeling models. CRFs are Fig. 5 Mention detection: a law sentence in the IOB, IOE and FIL notations. (Color ﬁgure online) Fig. 6 Mention splitting: a law mention in the IOB notation 123 40 O. T. Tran et al. discriminative undirected graphical models, which encode known relationships between observations and construct consistent interpretations. It is often used for labeling or parsing of sequential data. There are three popular reasons for using CRFs; (1) the nature of these two tasks; (2) the advantages of CRFs, which have all the strong points of maximum entropy Markov models (McCallum et al. 2000) but do not suffer from the label bias problem; (3) CRFs have been applied successfully to many NLP tasks such as chunking, named entity recognition, recognizing structures of law sentences Bach et al. (2011), etc. In extracting feature sets, we use a combination of n-gram (n B 3) of words, part- of-speech tags, and chunking information. When performing experiments, we exploit three kinds of label settings which are the IOB, IOE, and FIL notations (Ludtke and Sato 2003). 5.2 Mention classiﬁcation In this sub-step, each extracted mention of the previous step is classiﬁed into one of two predeﬁned classes; Class 1 and Class 2. Figure 7 shows some examples of mentions belonging to these two classes. To achieve the goal of this sub-step, we train a classiﬁer using supervised machine learning methods that have the advantage of combining arbitrary types of information in making a classiﬁcation decision. We use two robust classiﬁers which have demonstrated strong performance in many classiﬁcation tasks especially in statistical NLP [i.e. POS tagging Bach et al. (2013)]. In the ﬁrst classiﬁer, the classiﬁcation is performed with a statistical approach, built around the maximum entropy (MaxEnt) principle (Ratnaparkhi 1997). This principle allows to estimate the conditional probability p(y|x) that a model outputs a label y given a context x as follows: PðyjxÞ¼ exp k f ðx; yÞ i i ZðxÞ where f (x, y) refers to a feature function; k is a parameter of the model; and Z(x)is i i a normalization factor. There are several models that use this method, subject to certain constraints. Among these models, the MaxEnt chooses the model with the ﬂattest probability distribution (corresponding to the highest entropy). This con- strained optimization problem is ﬁrst converted into a dual optimization problem using Lagrange multipliers. The solution is then found by applying the improved iterative scaling method or LBFGS method. In the second classiﬁer, we use support vector machines (SVMs)—a statistical machine learning technique proposed by Vapnik (1998). An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. Besides linear classiﬁcation, SVMs can also deal with non-linear classiﬁcation efﬁciently by using kernel functions which implicitly map its inputs into high-dimensional feature spaces. 123 Automated reference resolution 41 Fig. 7 Some examples of mentions of two classes With each mention, we will extract features based on n-grams (n B 3) of words appearing in the mention. After that, the model will classify which class it belongs to. We set up two experimental settings by using two kinds of words: content words which are the output of the mention splitting step and all words of a mention (not only content words but also position words). 5.3 Position recognition In this sub-step, the position of the antecedent of a given mention is located. The input is a position part, which is the output of the mention splitting sub-step. The goal is to recognize which articles, which paragraphs, which items etc., a given mention refers to. We use regular expressions which are carefully constructed to recognize and determine the scope of antecedent candidates for a given mention. Some examples of position recognition results are given in Fig. 8. The patterns of the position parts were described in Sect. 2. Recognizing the position of the antecedent is usually simple when the position part of a mention is a complete one (i.e. the ﬁrst example in Fig. 8). A position is complete if it completely includes the information of the document it refers to. We can ﬁnd the article in the list. From this article, we will specify the precise location of the antecedent. Using regular expressions, we know where it should be located. For this type, we also use regular expressions to recognize complex positions and then resolve each of their positions. Another group of positions that is a little bit harder to resolve is anaphora. We would like to list some typical cases as follows: • References that refer to the current text: For example, this article, this paragraph, and so on. Recognizing this type of anaphora is feasible to resolve if we know the current location of the reference (i.e. the second example in Fig. 8). • References that refer to an earlier point in the text, such as the previous article. These can be resolved using structure information of the law (i.e. the third example in Fig. 8). • References that refer to an earlier reference, i.e. dou kou (the same paragraph). This reference refers not to the current paragraph, but to a paragraph that was previously mentioned in the text (i.e. the fourth example in Fig. 8). It is usually the most recent documents referenced before and near the current reference in the text. To resolve this type of references, we need to keep a history of the references found so far. 123 42 O. T. Tran et al. Fig. 8 Some examples of the output of the position recognition step 5.4 Antecedent candidate extraction This step extracts a list of candidates in the search space for a given mention. The search space is the output of the position recognition sub-step. We have two strategies to extract candidates based on whether the mention head (or synonyms of the mention head) of a mention appears in the antecedent sentences. 1. If the mention head or its synonyms appear, we extract candidates according to two ways: (a) Based on the dependency trees: We assume that the node containing the mention head or its synonyms is n . head • Type 1: Full subtrees—With each ancestor node of n , we extract all head its descendants as a candidate. • Type 2: Left subtrees—At each ancestor node of n , starting from head the ancestor node’s child that contains the node n , we gradually head scan to the leftmost child of the ancestor node. At each currently scanned child, we extract all descendants of all the scanned children as a candidate. This originates from the fact that, in Japanese, an explanation of a noun usually appears on its left side. • Type 3: We limit the boundary of all candidates of Type 1 and Type 2 which must be ended after the position of the mention head or its synonyms. (b) Based on the punctuation marks (usually a comma): With each candidate sentence, we perform a right-to-left scan from the position of the mention A mention head is the main noun of a mention. It identiﬁes the intellectual entity that this mention contains. 123 Automated reference resolution 43 head or its synonyms. Each time we meet a comma or the beginning of a sentence, we will extract the scanned texts as a candidate. Figure 9 illustrates an example of extracted candidates using strategy 1(a). Type 1 extracts Tree 1 and Tree 2; Type 2 extracts Tree 3, 4, 5, and 6; 0 0 0 0 0 0 Type 3 extracts Trees 1 ,2 from Trees 1, 2 and Trees 3 ,4 ,5 , and 6 from Trees 3, 4, 5, and 6. After that we remove trees that are duplicated from 0 0 previously extracted trees (Trees 1 ,3, 3 and 4 are identical to Tree 1; 0 0 Tree 6 is identical to Tree 2 ) to create unique candidates. To understand more about the ﬁrst strategy of candidate extraction, let us use the texts in Article 12, paragraph 1 (see Fig. 1 in Sect. 1) as an example. We also wish to ﬁnd the candidates for the reference ‘the notiﬁcation in the provision of para 1’ in Article 12, paragraph 4. The parse tree of the sentence in this document is presented in Fig. 10. In this example, the node that contains the mention head ‘notiﬁcation’ is the node 21 (marked red), which has no ancestor node. By using strategy 1a), we extract 14 candidates as shown in Fig. 11. Type 1 (full subtrees) yields one candidate (the ﬁrst one). Type 2 (left sub-strees) yields the candidates from 2 to 7, in which candidate 5 is a true referent. Type 3 yields the candidates from 8 to 14. After that, we remove duplicated candidates, i.e. the candidates 1 and 7; and accordingly the candidates 8 and 14 . In all, we obtained 12 candidates for the given reference. 2. If neither the mention head nor its synonyms appear, we obtain candidates for a mention as described in 1(b). However, the scan point starts from the end to the beginning of a candidate sentence. 5.5 Antecedent determination This step aims at determining the exact antecedent for each mention. Depending on the classiﬁcation of a mention, we have two strategies to determine its exact antecedent. • For a mention of Class 1, using the information about the position of the antecedent, we can exactly determine the antecedent for this mention. • For a mention of Class 2, from its candidates, we use a state-of-the-art model on the coreference resolution task to rank candidates based on the probability that the candidate is the antecedent of a given mention. The 1st-ranked candidate will be selected as the exact antecedent. To determine the exact antecedent for an active mention from a list of candidates, many models have been proposed, including mention-pair models (Ng and Cardie 2002; Ponzetto and Strube 2006; Soon et al. 2001), entity-mention models (Luo et al. 2004; Yang et al. 2004), mention-ranking models (Yang et al. 2003) and These identical trees are marked with the same color in Fig. 9. These two candidates are created from the candidates 1 and 7 respectively. 123 44 O. T. Tran et al. Fig. 9 An example of extracting candidates using strategy 1(a)(n = 17). (Color ﬁgure online) head Fig. 10 An example of parsing the sentence in the document of Article 12, Paragraph 1. (Color ﬁgure online) cluster-ranking models (Rahman and Ng 2009). Among these models, mention- ranking and cluster-ranking models are more robust than the ﬁrst two models in the sense that they can address the limitation of previous ones (Yang et al. 2003; Rahman and Ng 2009;Ng 2010). These ranking models yield a theoretically more adequate and empirically better performance on some public standard corpora of the coreference resolution task. They directly capture the ranking on all the antecedent candidates of an active mention, instead of considering them independently. In this paper, therefore, we choose a method of mention-ranking models to solve the task. rank To train a ranker, we used the SVM ranker-learning algorithm SVM (Joachims 2006) as the learning method. In the training phase, each instance i(m , c ) is created i j based on a mention m and an antecedent candidate c . The feature sets used to i j represent each training instance are grouped into three sets which are described in more detail in Table 1. These linguistic features are supposed to be effective in estimating the probability that a candidate is an antecedent of an active mention. These feature sets are extracted using the n-gram information, the mention head (or 123 Automated reference resolution 45 Fig. 11 Candidates extracted by using the ﬁrst strategy to extract candidates for the reference ‘the notiﬁcation in the provision of para 1’ its synonyms) and the dependency tree of the antecedent sentence. The ﬁrst feature set tries to capture the boundary of the candidate c . The second feature set checks if the candidate c contains the mention head of m and calculates the distance from the j i position of the mention head to the beginning and the end of c . The third feature set captures how good c is a meaningful grammar unit as well as the smallest fragment of texts that supports the understanding of the mention m . The way of assigning class values to each training instance is as follows: Assuming that S is the set of training instances created for a mention m , the class k i value for an instance i(m , c )in S is the rank of m among competing candidates as i j k i determined using the OFFSET value. Why do we need the OFFSET value? Because extracting the candidate that is the gold antecedent of a mention is a quite difﬁcult task. Consequently, in this step, the system is unable to ﬁnd the correct antecedent in some special cases. Moreover, the purpose of resolving mentions in legal texts is to show the referenced texts so that readers can quickly understand more about the rules that they are reading. Therefore, can we loosen the criteria to estimate whether the output of the system is considered to be the 123 46 O. T. Tran et al. Table 1 Feature sets extracted for the training instance i(m , c ) i j No. Types Feature names Notes 1 N-gram Start [1] 1st word of c Start [1–2] 1st and 2nd words of c Start [1–3] 1st,2nd, and 3rd words of c End [1] Last word of c End [–2] Two last words of c End [1–3] Three last words of c 2 Mention ContainMentionHead Does c contain the mention head of m ? j i head DistanceToBeginning The distance from position to the beginning of c head j DistanceToEnding The distance from position to the end of c head j MentionHeadAtBegin Does m appear at the beginning of c ? i j MentionHeadAtEnd Does m appear at the end of c ? i j 3 Dependency CountNodesToMeetingNode The distance from the node containing the mention head to the meeting node n meeting NumberOfLeftChilds At node n , count the number of children from its leftmost child to the child that contains the ﬁrst word of the meeting candidate c NumberOfRightChilds At node n , count the number of children from its rightmost child to the child that contains the last word of the meeting candidate c position : the position of the mention head in the antecedent sentence; n : the meeting node where the concatenation of all of its descendants covers the candidate c head meeting j Automated reference resolution 47 1 if c is the gold antecedent of the mention m j i 0:5 if c covers the gold; and the number of words valuei ðm ; c Þ¼ i j > outside the gold is less than OFFSET 0 otherwise After training, this mention-ranking model will be used to rank the candidates for a given mention. We create test instances for a mention by pairing the latter with each of its antecedent candidates. In the testing phase, the candidate that is assigned the largest value by the ranker is selected as the exact antecedent of that mention. 6 Experiments 6.1 Corpus The Japanese National Pension Law (JNPL) corpus on reference resolution was manually built by law-making experts. In this corpus, all references that refer to the inside scope of the JNPL were marked. However, the JNPL does not include the references that refer to other laws, or refer to ambiguous ranges. The architecture of the JNPL is shown in Fig. 12. The law consists of articles, articles consist of paragraphs, and paragraphs consist of sentences. A sentence may belong to items, sub-items, or sub-sub-items of a paragraph. Below are some statistics about this corpus: • # Articles: 99 articles • # Paragraphs: 931 paragraphs • # Mentions: 748 mentions (586 mentions of Class 1, and 162 mentions of Class 2) 6.2 Experimental setup We divided the JNPL corpus into 10 sets, and conducted tenfold cross-validation tests for all experiments. For the mention detection task which we modeled as a sequence labeling problem, we evaluated the performance of our system by precision, recall, and the F1 score as follows: #correctly detected mentions #correctly detected mentions P ¼ ; R ¼ ; #detected mentions #gold mentions 2  Precision  Recall F1 ¼ : Precision þ Recall For the remaining four tasks (the mention splitting, the mention classiﬁcation, the position recognition and the antecedent determination steps), we conduct Footnote 12 continued same as the gold antecedent? Instead of exactly matching, we allow the output to exceed the boundary of the gold antecedent with the smallest number of words possible. If the output contains the gold ante- cedent, and the total number of words in additional texts is not greater than the OFFSET value, the output is considered to be true. In experiments, we set OFFSET to be equal to 10. 123 48 O. T. Tran et al. Fig. 12 The architecture of the JNPL corpus on reference resolution experiments using the gold mentions of the corpus. With each mention, the system always outputs results for its input. Therefore, to estimate the performance of these sub-systems, we use the accuracy score to evaluate the experimental results. #correctly processed mentions Accuracy ¼ : #gold mentions where the numerator, #correctly processed mentions, corresponds to the number of mentions that are correctly split in the mention splitting task; correctly classiﬁed in the mention classiﬁcation task; correctly position-recognized in the position rec- ognition step and correctly antecedent-determined in the antecedent determination step. To estimate the performance of the whole system, we use precision, recall and the F1 score as follows: 123 Automated reference resolution 49 #correctly detected and resolved mentions P ¼ ; #detected mentions #correctly detected and resolved mentions 2  Precision  Recall R ¼ ; F1 ¼ : #gold mentions Precision þ Recall where the #correctly detected and resolved mentions refers to the total number of mentions that are correctly detected and correctly antecedent-determined. 6.3 Experimental results 6.3.1 Mention detection and mention splitting To learn the models, we used the CRF?? tool written by Kudo. In extracting feature sets, we used a combination of n-gram (n B 3) of words, part-of-speech tags, and chunking information. This information was taken from the output of the Cabocha . Cabocha (Kudo and Matsumoto 2002) is a Japanese dependency structure analyzer based on Support Vector Machines (SVMs) (Vapnik 1998). When performing experiments, we used three kinds of label settings which are the IOB, IOE, and FIL notations (Ludtke and Sato 2003). We also investigated the task using different combinations of feature sets to determine which feature sets yield better performances. Experimental results of the mention detection sub-step are shown in Table 2. We realize that the more feature sets we use, the better the results are. This is reasonable because these feature sets yield more beneﬁts to the model. We obtained the highest performance of 80.14 % in the F1 score using the FIL notation on feature sets which are based on words, part-of-speech tags and chunking information. That is because most mentions begin with some words clearly locating positions such as dai, tsugi, mae, dou, etc; and end with a noun phrase. In some cases, the NPs were nested, which slightly worsened the performance of the IOE notation in comparison with that of the IOB notation. In all settings, the recall was higher than the precision score. This can be attributed to the fact that some detected mentions refer to the outside scope of the corpus and therefore are not annotated. Experimental results of the mention splitting sub-step are given in Table 3. For all three notations, we obtained a very high accuracy of 99.3 % when using words, part-of-speech tags, and chunking features. The reason for this high performance is that most mentions have their position parts ending with particles like ni or no. 6.3.2 Mention classiﬁcation To perform this step using MEM, we used the public MaxEnt classiﬁer tool with the BLMVM optimization algorithm (Benson and More 2001). We also used an http://crfpp.googlecode.com/svn/trunk/doc/index.html. http://code.google.com/p/cabocha/. Downloaded from: http://www-tsujii.is.s.u-tokyo.ac.jp/*tsuruoka/maxent/. 123 50 O. T. Tran et al. Table 2 Experimental results for the mention detection task (%) Notations Word Word ? POS Word ? POS ? chunk RP F1 RP F1 RP F1 IOB 79.05 76.95 77.95 80.6 79.32 79.92 80.58 79.6 80.06 IOE 79.2 77.93 78.53 79.63 78.3 78.92 79.99 79.09 79.51 FIL 79.88 77.83 78.82 80.51 79.49 79.95 80.7 79.66 80.14 Table 3 Experimental results Notations Word Word ? POS Word ? POS ? chunk for the mention splitting sub- step [Accuracy (%)] IOB 99.23 99.17 99.3 IOE 99.17 99.17 99.3 FIL 99.17 99.17 99.3 SVM implementation of LibSVM of Chang and Lin (2011), which is an integrated software for support vector classiﬁcation. As an appropriate baseline model, we chose the model that assigns all mentions to the larger class (Class 1 by default). Experimental results in Table 4 show that using only the content words results in a higher accuracy than using all words of a mention with the two machine learning methods. This is because position words do not usually convey the information about the class of a mention. For this step, SVM performs slightly better than MEM on both settings. The results also show that our system outperforms the baseline model by &8 % in terms of accuracy. 6.3.3 Position recognition By using regular expressions and context information, we were able to locate up to 96.18 % of correct positions of all mentions. The few cases where our system was unable to recognize the correct position can be classiﬁed under some exceptions that do not conform to the regular expressions that were described in Sect. 2. 6.3.4 Antecedent determination In this paper, we chose to run an algorithm proposed by Rahman and Ng (2009). To train a ranker, we used as the learning method the SVM ranker-learning algorithm rank 17 struct SVM (Joachims 2006) , an instance of SVM for efﬁciently training Ranking SVMs (Joachims 2002). Table 5 presents the experimental results of determining a true antecedent among candidates for each mention in the corpus. We conducted experiments by adding feature sets incrementally to observe their effect on the model. The experimental results indicated that the combination of feature sets of n-gram, the mention head Downloaded from http://www.csie.ntu.edu.tw/*cjlin/libsvm/. This software can be downloaded from http://www.cs.cornell.edu/people/tj/svm_light/svm_rank.html. 123 Automated reference resolution 51 Table 4 Experimental results Content words All words of the mention classiﬁcation task [Accuracy (%)] Baseline 78.34 78.34 MEM 87.03 86.54 SVM 87.16 87.23 Table 5 Experimental results No. Types of features Accuracy (%) of the antecedent determination step 1 n-gram 84.76 2 n-gram ? head 85.40 3 n-gram ? head ? dependency 85.61 and the dependency information yields the best performance. We obtained 85.61 % accuracy in determining the antecedents for all mentions in the JNPL corpus. This table also shows that we achieved better performance if we integrate more feature sets into the model. This section also measures whether the output contains the gold antecedent and the length of additional texts in terms of words is greater than the OFFSET value, but not the entire document. The accuracy is 90.63 %. 6.4 Analyzing the impact of each step on the ﬁnal system Table 6 shows the experimental results of analyzing the impact of each step on the ﬁnal system based on the following assumptions: • Experiment 1: assumes that the mention detection step correctly detects all mentions in the corpus. If that occurred, the whole system achieved 79.23 % in the F1 score. • Experiments 2,3, and 4: assume that the mention splitting, the mention classiﬁcation and the position recognition sub-steps perform perfectly respectively. • Experiment 5: assumes that Step 1—Contextual Information Extraction—is correctly performed. With that assumption, the whole system achieved 71.14 % in the F1 score. • Experiment 6: assumes that the ﬁrst two steps perform perfectly. If that occurred, we obtained 86.17 % in the F1 score for the whole system. • Experiment 7: does not assume that any step perform perfectly, and the performance reached 67.02 % in the F1 score. To comprehend the impact of each step on the whole system, it should be noted that the higher the F1 score, the more important the step is. From these results, we can see that the mention detection step plays the most important role in the performance of the whole system. The next important step is the contextual information extraction. Since this step includes 3 sub-steps, the order of the role of each step in importance in the ﬁnal system is as follows: the mention classiﬁcation sub-step, the position recognition sub-step, and the mention splitting sub-step. 123 52 O. T. Tran et al. Table 6 Experimental results of the effect of each step on the ﬁnal system No. Assumptions Precision Recall F1 1 Step 0 (MD) is true 79.73 78.74 79.23 2 Step 1.1 (MS) is true 67.42 66.65 67.03 3 Step 1.2 (MC) is true 71.53 70.71 71.12 4 Step 1.3 (PR) is true 67.48 66.66 67.07 5 Step 1 (MC ? MS ? PR) is true 71.56 70.72 71.14 6 Step 0 and Step 1 are true 86.74 85.61 86.17 7 End-to-end system 67.42 66.63 67.02 MD, MS, MC, and PR stand for the mention detection, mention splitting, mention classiﬁcation and position recognition steps respectively The last experiment is also the end-to-end system in which the input of the current step will be the output of the previous step. In the whole task setting, we obtained a result of 66.63 % recall, 67.42 % precision, and 67.02 % in the F1 score in determining antecedents for mentions needed to be resolved in the corpus. This performance is quite promising. 6.5 Improving the performance of the ﬁnal system This section looks at the limitation of the proposed four-step framework and suggests a solution for overcoming this limitation 6.5.1 Limitation of the framework We proposed a cascade framework, where the output of the previous step was used as the input of the next step. So, the errors of the previous step will be propagated to the next step. By analyzing the effect of each step on the ﬁnal system, we saw that the ﬁrst step—mention detection has the biggest inﬂuence on the performance of the whole system. Experimental results of each step showed that this step together with the mention classiﬁcation step yield the lowest performance. Hence, in this section, we would like to focus on improving these two phases through the consideration of using the Brown clustering algorithm to improve the performance. 6.5.2 A simple semi-supervised method using Brown clustering This sub-section will brieﬂy describe the Brown clustering algorithm and show how to use the Brown word clusters to improve the mention detection step and the mention classiﬁcation sub-step. Among the word representation methods, we chose the Brown clustering algorithm for our work because of its simplicity and efﬁciency. The Brown clustering algorithm is a word clustering algorithm based on the mutual information of bigrams (Brown et al. 1992). The input to the algorithm is a set of words and a text corpus. In the initial step, each word belongs to its own 123 Automated reference resolution 53 individual cluster. The algorithm then gradually groups clusters to build a hierarchical clustering of words. Figure 13 shows an example of a Brown word-cluster hierarchy in a binary tree style. In this tree, each leaf node corresponds to a word, which is uniquely identiﬁed by its path from the root node. This path can be represented by a bit string, as shown on the left side. From the root node, we add bit 0 to the left branch and bit 1 to the right branch. A word-cluster hierarchy is reduced to a depth of n if all words with the same n-bit preﬁx are grouped in one cluster. For example, if the word-cluster hierarchy on the left side of Fig. 13 is reduced to a depth of 2, we will obtain a new hierarchy on the right side of Fig. 13. Features extracted at n-bit depth are binary strings of length n. By reducing the word-cluster tree to different values of depth n, we can group words at various levels, from coarse clusters (small value of n) to ﬁne clusters (large value of n). These features are used as extra word features. Next, we will present how to integrate into the models in order to improve the performance of the mention detection step and the mention classiﬁcation sub-step. Mention detection step with extra word features The main idea of our semi-supervised learning method is to use unsupervised word representations (Brown word clusters) as extra word features of a supervised model. In this framework, unlabeled data is used to produce word clusters. From these word clusters, we extract extra word features, and add these features to a supervised model (labeled data are used to train this model). Figure 14 shows our semi-supervised learning framework. This framework consists of two phase: the unsupervised phase with the Brown clustering algorithm, and the supervised phase with CRFs. To produce word representations, we ﬁrst collected plain text from the Japanese law translation website . This website provides many Japanese law articles in both Japanese and English. The plain text corpus which we downloaded included more than 67,000 sentences of Japanese laws. We ﬁrst performed word segmenting using Cabocha tool, and then conducted the Brown clustering algorithm to cluster words. We used the implementation of Liang (2005), and set the number of clusters at 200. For the mention detection task, we extracted features at 4-bit depth and 6-bit depth. We integrated these features into three models which are the best models of each notation. The results in Table 7 show that using extra words features improves the models on all three kinds of notations. For the best performance, the system increases the F1-score from 80.05 upto 80.82 %. Mention classiﬁcation step with extra word features With the same methodology when integrating extra word features of the mention detection step, we also conducted experiments for the mention classiﬁcation sub- step. We used the same setting when extracting extra word features. The experimental results of the semi-supervised method with extra word features are shown in Table 8. The results showed that using extra words features improves the models on the two learning algorithms, which are SVM and MEM. For the best http://www.japaneselawtranslation.go.jp. 123 54 O. T. Tran et al. Fig. 13 An example of Brown word-cluster hierarchy Fig. 14 Semi-supervised learning framework Table 7 Mention detection: No. Label settings Features Precision Recall F1 experimental results when integrating extra word features 1 IOB - 80.58 79.60 80.06 using Brown clustering ? 80.97 80.13 80.51 information 2 IOE - 79.99 79.09 79.51 ? 80.82 80.9 80.82 -, means that we did not use 3 FIL - 80.70 79.66 80.14 extra word features; ?, means ? 80.55 81.01 80.74 that we used extra word features Table 8 Mention classiﬁcation: Classiﬁers Features Content words All words experimental results when integrating extra word features MEM - 87.03 86.54 using Brown clustering ? 87.86 87.08 information SVM - 87.16 87.23 ? 87.88 87.40 performance, the system increases the F1-score from 87.03 up to 87.86 % on MEM, and fro 87.16 up to 87.88 % on SVM. 6.6 A true example of using the ﬁnal system Figure 15 gives an example of the systems output. It shows the detail process of how the system works. The input is a piece of an article which is in the form of a sequence of characters. This input contains a mention ‘The employer who employs the second type 123 Automated reference resolution 55 of insured person provided in the preceding paragraph’ that should be detected. The ﬁrst mention detection step correctly recognizes this mention and lets this mention go through Step 1. In this step, Sub-step 1.1 splits it into two parts which are a position part (‘in the preceding paragraph’) and a content part (‘The employer who employs the second type of insured person provided’). Based on the content part, Sub-step 1.2 classiﬁes this mention to Class 2. This means that this mention only refers to a fragment of a document. Based on the position part, Sub-step 1.3 recognizes the position that contains the referenced text fragment of this mention, which is in Article 12, Paragraph 6. The outputs from Sub-step 1.2 and 1.3 are then used in combination to extract all possible antecedent candidates for this mention through Step 2. The last step—antecedent determination—determines exactly the true referenced texts (the texts in green) for this mention among its candidates. 7 Error analysis This paper focused on two major problems. The ﬁrst one is to detect all references (mentions) in the legal texts, and the second one is to resolve these mentions to their Fig. 15 An output example of our system 123 56 O. T. Tran et al. correct referenced texts (antecedents). This section analyzes error cases and reasons that contributed to the instance of failures in our system. By observing the output of the ﬁnal system, we realized that in detecting mentions, most of errors can be attributed to the following reasons: • Detected but they are not labeled as mentions in the corpus [Because, these mentions refer to the documents which is beyond the scope of JNPL corpus (i.e. the ﬁrst example in Fig. 16)]. • Mentions are detected beyond the true boundary as labeled in the corpus. This is usually caused by ambiguities of mentions, in which the content parts are usually nested noun phrases (i.e. the second example in Fig. 16). In determining antecedents, most of errors are caused by the following reasons: • Mentions are wrongly split, classiﬁed, and position-recognized in intermediate sub-steps. • The antecedent candidate extraction step did not extract the gold antecedent for mentions. Among all mentions that we considered, there are approximately 90 % of mentions whose extracted candidates contains their gold antecedents. Therefore, it is impossible to ﬁnd the gold antecedents for the remaining 10 % of mentions. • Dependency parsers typically could not deal well with long sentences in the legal domain. So, the information extracted using dependency trees is not highly reliable in some cases. 8 Discussion In this section, we discuss more our system. We ﬁrst discuss the performance of the system in comparison to other related systems. Then, we discuss the versioning problem of our work. 8.1 Comparison with previous work Previous sections (Sects. 1, 2) have distinguished the difference of this work and previous related work. References which are considered in this research consist of two main parts, i.e. position parts and content parts. This form is slightly different from the form of normative references, which are studied in previous work. Normative references only consist of position parts, which are used to determine the location of referenced documents. References in this research are extended to cover not only position parts but also content parts. These content parts are used to determine the smallest fragments of texts in the referenced documents, which are referred to. This is the novel point of this research. Compared with previous work, therefore, our system yields a lower performance . This is understandable because our system aimed at a more complete target, which tries to resolve references to the Previous work reported F scores of more than 90 % for normative reference detection and resolution. 123 Automated reference resolution 57 Fig. 16 Some error examples of the mention detection step smallest fragment of texts rather than only locating the referenced document .This resolution requires deep syntactic and semantic knowledge of the texts. Hence, detecting and resolving this kind of references is much more difﬁcult in comparison with normative references in previous work. 8.2 The versioning problem of laws In this research, we did not take versions of laws into account. This means that our corpus only contains one version of the JNPL law. If the law changes, the system should be changed a little bit to adapt to that change. This needs to be taken into account in Steps 2 and 3 of the framework. Our proposed method is based on supervised learning. Therefore, the method can integrate features related to versioning to deal with this problem. In this case, the information about how laws are changed will be used to help the resolver to extract the expected referents (extract only old referents, or only new referents, or both depending on the real applications). This interesting problem will be considered in future work. 9 Conclusion and future work In this paper, we presented a new interesting task of reference resolution in legal texts. The goal is to create a system which can automatically extract referents for references in real time. In this research, we go a step further in comparison to previous work by extracting the smallest fragments of texts in the referenced documents. This does not only help readers in understanding the law, support law- makers in developing and amending laws, but also support in building an information system which works based on laws, etc. In other research, we also considered to detect and resolve normative references in Japanese language. The experimental results showed that we also obtained more than 90 % in the F1 score on the same Japanese National Pension Law corpus on normative references. 123 58 O. T. Tran et al. Based on the characteristics of reference phenomena in legal texts, we proposed a framework with four steps to solve the task. In comparison to reference resolution in general texts, our framework consists of two extra steps (Step 1 and 2), which take advantage of the reference phenomena in legal texts to help solve the task more effectively. Experiments on the JNPL showed very promising results. We achieved a performance of 80.06 % in the F1 score in the mention detection step; a 99.3 % accuracy in splitting mentions into two parts; a 96.18 % accuracy in locating the positions of antecedents; an 87.03 % accuracy in classifying them. In the ﬁnal antecedent determination step, the system achieved 85.61 % accuracy in determin- ing the antecedents for all mentions in the corpus. Our ﬁnal end-to-end system achieved 67.02 % in the F1 score in the whole task of detecting and determining antecedents for mentions. There is a wide range of open possibilities for improving this important task. For example, by conducting experiments to estimate the effects of each step on the ﬁnal model, we see that an improvement of this framework should imply paying more attention to the mention detection step. In addition, we also should take care of solving error cases analyzed in the error analysis section. In future research, we will exploit the output of this system to ﬁnd contradictions in legal texts. We will also build a question answering system to support citizens in accessing legal documents. Acknowledgments This work was partly supported by Grant-in-Aid for Scientiﬁc Research, Education and Research Center for Trustworthy e-Society, JAIST Research Grants, and JAIST Overseas Training Program for 3D Program Students. We would like to give special thanks to two people who built our corpus. They are a person who graduated from a law school and worked in the government and a student of a law school. References Bach NX, Minh NL, Shimazu A (2011) RRE task: The task of recognition of requisite part and effectuation part in law sentences. Int J Comput Process Lang 23(2):109–130 Bach NX, Hiraishi K, Minh NL, Shimazu A (2013) Dual decomposition for vietnamese part-of-speech tagging. In: Proceedings of the 17th international conference on knowledge-based and intelligent information and engineering systems (KES), procedia computer science, pp 123–131 Benson S, More J (2001) A limited-memory variable-metric method for bound-constrained minimization. In: Preprint ANL/MCS-P909-0901 Bolioli A, Dini L, Mercatali P, Romano F (2002) For the automated mark-up of italian legislative texts in xml. In: Proceedings of international on legal knowledge and information systems (Jurix), pp 21–30 Brown P, deSouza P, Mercer R, Pietra V, Lai J (1992) Class-based n-gram models of natural language. J Comput Linguist 18(4):467–479 Chang C, Lin C (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(27):1–27, software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm/ Culotta A, McCallum A (2005) Joint deduplication of multiple record types in relational data. In: Proceedings of the 14th ACM international conference on information and knowledge management (CIKM), pp 257–258 Finkel J, Manning C (2010) Hierarchical joint learning: improving joint parsing and named entity recognition with non-jointly labeled data. In: Proceedings of the 48th annual meeting of the association for computational linguistics (ACL), pp 720–728 Hachey B, Radford W, Curran J (2011) Graph-based named entity linking with Wikipedia. In: Proceedings of the 12th international conference on web information system engineering (WISE), pp 213–226 123 Automated reference resolution 59 Hachey B, Radford W, Nothman J, Honnibal M, Curran J (2013) Evaluating entity linking with Wikipedia. J Artif Intell 194:130–150 Haghighi A, Klein D (2009) Simple coreference resolution with rich syntactic and semantic features. In: Proceedings of empirical methods in natural language processing (EMNLP), pp 1152–1161 Joachims T (2002) Optimizing search engines using clickthrough data. In: Proceedings of ACM conference on knowledge discovery and data mining (KDD), pp 133–142 Joachims T (2006) Training linear SVMs in linear time. In: Proceedings of ACM conference on knowledge discovery and data mining (KDD), pp 217–226 Jurafsky D, Martin J (2009) Speech and language processing: an introduction to natural language processing, computational linguistics and speech recognition, 2nd edn. Prentice Hall Series in Artiﬁcial Intelligence Katayama T (2007) Legal engineering—an engineering approach to laws in e-society age. In: Proceedings of international workshop on juris-informatics (JURISIN) Katayama T (2010) The curent status of the art of the 21st COE programs in the information sciences ﬁeld. veriﬁable and evolvable e-society—realization of trustworthy e-society by computer science. J Inf Process Soc Jpn 46(5):515–521 (in Japanese) Katayama T, Shimazu A, Tojo S, Futatsugi K, Ochimizu K (2008) e-society and legal engineering. J Japanese Soc Artif Intell 23(4):529–536 (in Japanese) Kudo T, Matsumoto Y (2002) Japanese dependency analysis using cascaded chunking. In: Proceedings of the 6th conference on natural language learning 2002 (COLING 2002 post-conference workshops) (CoNLL 2002), pp 63–69 Lafferty J, McCallum A, Pereira F (2001) Conditional random ﬁelds: probabilistic models for segmenting and labeling sequence data. In: Proceedings of international conference on machine learning (ICML), pp 282–289 Liang P (2005) Semi-supervised learning for natural language. Master’s thesis, Massachusetts Institute of Technology Ludtke D, Sato S (2003) Fast base np chunking with decision trees experiments on different POS tag settings. In: Proceedings of conferences on computational linguistics and natural language processing (CICLing), pp 139–150 Luo X, Ittycheria A, Jing H, Kambhatla N, Roukos S (2004) A mention-synchronous coreference resolution algorithm based on the bell tree. In: Proceedings of annual meeting of the association for computational linguistics (ACL), pp 135–142 Maat E, Winkels R, Engers T (2006) Automated detection of reference structures in law. In: Proceedings of international on legal knowledge and information systems (Jurix), pp 41–50 McCallum A, Freitag D, Pereira F (2000) Maximum entropy Markov models for information extraction and segmentation. In: Proceedings of the 17th international conference on machine learning (ICML), pp 591–598 Mercedes M, Pablo D, Damaso-Javier V (2005) Reference extraction and resolution for legal texts. In: Proceedings of PReMI, pp 218–221 Ng V (2007) Semantic class induction and co-reference resolution. In: Proceedings of annual meeting of the association for computational linguistics (ACL), pp 536–543 Ng V (2010) Supervised noun phrase coreference research: the ﬁrst ﬁfteen years. In: Proceedings of annual meeting of the association for computational linguistics (ACL), pp 1396–1411 Ng V, Cardie V (2002) Improving machine learning approaches to coreference resolution. In: Proceedings of annual meeting of the association for computational linguistics (ACL), pp 104–111 Palmirani M, Brighi R, Massini M (2003) Automated extraction of normative references in legal texts. In: Proceedings of international conference on artiﬁcial intelligence and law (ICAIL), pp 105–106 Pitler E, Bergsma S, Lin D, Church K (2010) Using web-scale n-grams to improve base np parsing performance. In: Proceedings of the 23rd international conference on computational linguistics (Coling 2010), pp 886–894 Ponzetto S, Strube M (2006) Exploiting semantic role labeling, wordnet and Wikipedia for coreference resolution. In: Proceedings of human language technologies: annual conference of the North American chapter of the association for computational linguistics (HLT-NAACL), pp 192–199 Rahman A, Ng V (2009) Supervised models for coreference resolution. In: Proceedings of empirical methods in natural language processing (EMNLP), pp 968–977 Ratnaparkhi A (1997) A simple introduction to maximum entropy models for natural language processing. Technical report, Institute for Research in Cognitive Science, University of Pennsylvania 123 60 O. T. Tran et al. Recasens M, Marquez L, Sapena L, Marti M, Taule M, Hoste V, Poesio M, , Versley Y (2010) Semeval- 2010 task 1: co-reference resolution in multiple languages. In: Proceedings of international workshop on semantic evaluation, pp 1–8 Soon W, Lim D, Ng H (2001) A machine learning approach to co-reference resolution of noun phrases. J Comput Linguist 27(4):521–544 Vapnik V (1998) Statistical learning theory. Wiley, Hoboken Yang X, Zhou G, Su J, Tan C (2003) Coreference resolution using competitive learning approach. In: Proceedings of annual meeting of the association for computational linguistics (ACL), pp 176–183 Yang X, Su J, Zhou G, Tan C (2004) An np-cluster based approach to coreference resolution. In: Proceedings of international conference on computational linguistics(COLING), pp 226–232 "
<https://plosjournal.deepdyve.com/lp/springer-journals/automated-reference-resolution-in-legal-texts-wfQlQgZLR7>