_:n131b972129c94291b96e6efcf9b10be9xb0
<http://schema.org/articleBody>
"Artif Intell Law (2016) 24:347–370 DOI 10.1007/s10506-016-9188-z Cognitive computing and proposed approaches to conceptual organization of case law knowledge bases: a proposed model for information preparation, indexing, and analysis 1 2 3 • • • Amie Taal James A. Sherer Kerri-Ann Bent Emily R. Fedeles Published online: 21 October 2016 Springer Science+Business Media Dordrecht 2016 Abstract Carole Hafner’s scholarship on the conceptual organization of case law knowledge bases (COC) was an original approach to distilling a library’s worth of cases into a manageable subset that any given legal researcher could review. Her approach applied concept indexation and concept search based on an annotation model of three interacting components combined with a system of expert legal reasoning to aid in the retrieval of pertinent case law. Despite the clear value this tripartite approach would afford to researchers in search of cases with similar fact patterns and desired (or undesired) outcomes, this approach has not been applied consistently in the intervening years since its introduction. Speciﬁcally, the con- ceptual representation of domain concepts and the case frames were not pursued by researchers, and they were not applied by the legal case indexing services that came to dominate the electronic case law market. Advances since Hafner’s original scholarship in the form of (1) digitized case law and related materials; (2) computer The views expressed herein are solely those of the authors, should not be attributed to their places of employment, colleagues, or clients, and do not constitute solicitation or the provision of legal or security advice. & Amie Taal amie33_uk@yahoo.co.uk James A. Sherer jsherer@bakerlaw.com Kerri-Ann Bent kbent7@gmail.com Emily R. Fedeles efedeles@bakerlaw.com DeutscheBank AG, New York, NY, USA BakerHostetler, New York, NY, USA Barclays, New York, NY, USA 123 348 A. Taal et al. science analytical protocols; and (3) more advanced forms of artiﬁcial intelligence approaches present the question of whether Hafner’s COC model could move from the hypothetical to the real. Keywords Concept indexation  Case law domain  Case law knowledge bases Digital search and conceptual retrieval 1 Introduction The Xerox Palo Alto Research Center (PARC), incubator for such inventions as early laser printing, personal computers, the Graphical User Interface, and (of course) the computer mouse, has become the poster child for brilliant technological ideas that needed other advances—or other opportunities—before they were commercially or societally viable (Gladwell 2011). Like many of PARC’s ideas, Hafner’s conceptual organization of case law knowledge bases (COC) scholarship was formed in a world where commercial implementation of those ideas was not yet viable. Hafner’s approach to distilling a library’s worth of cases into a manageable subset applied concept indexation and concept search based on an annotation model of three interacting components combined with a system of expert legal reasoning to aid in the retrieval of pertinent case law. The ﬁrst component of Hafner’s COC scholarship is a domain knowledge model that deﬁnes the basic concepts of a case law domain, which could include actors, events, and/or relationships from a particular domain (Wyner 2012). The second component is a legal case frame that represents the properties, roles, and relationships of elements of the case (Wyner 2012). The third component is comprised of Issue/Case Discrimination Trees that represent the signiﬁcance of each case relative to a model of the normative relationships of the legal domain (Wyner 2012). These Issue/Case Discrimination Trees serve the purpose of dealing with ‘‘conﬂicting rules, exceptions, negative queries, and ‘customary factors that may inﬂuence an outcome’ rather than necessary and sufﬁcient factors’’ (Wyner 2012). The Trees have nodes both for both legal issues and for factors that normatively inﬂuence the decision in the case; the nodes are connected in various ways and the cases are linked to the Tree, where they are then associated with the issues and factors and the direction of the decision (Wyner 2012). Following Hafner’s (1987) proposal of the COC model, other researchers have explored even less tangible concepts, such as the spirit of the law; unveiling political and social values; and those inferences that may be drawn from the societal relationships underpinning case-based legal argument (Bench-Capon and Sartor 2003; Bench-Capon et al. 2005, 2013). This exploration of the intangible, originally proposed by Berman and Hafner (1993), considers the concept that ‘‘cases reveal preferences between conﬂicting reasons, and that these preferences are explained in the theory through the priority given to the values promoted by these reasons. Values, in this way, allow us to deduce preferences which have not yet been explicitly tested in any precedent case’’ (Bench-Capon et al. 2005). 123 Cognitive computing and proposed approaches to conceptual… 349 When presented with new cases, theory construction may be utilized ‘‘to produce a theory which provides a sufﬁcient explanation of the previous cases, and which gives an answer in the new case. Competing theories may be constructed, and these need to be critically compared to determine which should be applied to the new case’’ (Bench-Capon et al. 2005). Speciﬁcally, Trevor Bench-Capon and his co- authors explore a case-modeling approach that accounts for the comparative strength of the values advanced by the related arguments, which lends itself to more sophisticated case-based reasoning capabilities (Bench-Capon et al. 2005). While investigating Value-based Argumentation Frameworks (VAFs), the authors argue that, when compared to the standard notion of Argumentation Frameworks, VAFs create results more closely aligned with Hafner’s vision; that is, those that not only support previous case law, but also provide an explanation for newly presented cases (Bench-Capon et al. 2005). In addition to these approaches, the component parts of many of these AI models have advanced. Our proposed model incorporates these advancements, integrating those component parts into a process that presents a unique perspective that differs from these cited models. Likewise, while Watson and AlphaGo represent advances of a different kind (where the component parts are not as readily discernible) their results are indicators of how they might be applied in service of Hafner’s COC model. In support of our proposed model, we hypothesize that the three fundamental changes presented in this study, which have occurred since Hafner’s COC scholarship, may provide a second life for the ideas she presented and propose that the implementation of Hafner’s COC model (Hafner 1987) is possible given present-day technologies and approaches. Hafner’s COC model envisioned an indexing of case law according to an applied concept search, which itself would be based on an annotation model of three interacting components, combined with a system of expert legal reasoning to aid in the retrieval of pertinent case law for practitioners and fact ﬁnders. (Hafner 1987) The ﬁrst of these fundamental changes focuses on case law availability, as the library’s worth of ‘‘thousands of cases’’ (Hafner 1987) has expanded exponentially since Hafner’s research. Not only have legal indexing services progressed in earnest and digitized many prior decisions— which provide legal professionals with an increased ability to carry out their operations effectively—but presently the vast majority of new decisions as well as supporting legal documentation (such as pleadings) are natively digital, much more easily indexed, and readily available. Second, with the rapid advances in technology and computer processing speeds, case law datasets can become extremely large while remaining manageable for the application of advanced statistical modeling. Third, new models of text analytics and cognitive computing work best across exactly these types of large datasets (Big Data) and take full advantage of statistical modeling and artiﬁcial intelligence techniques that could replace the human element originally presented within Hafner’s COC strategy. These new models provide an automated and statistically supported index of the Issue/Case Discrimination Trees (Hafner 1987) that Hafner posited would represent the signiﬁcance of a given case relative to the form of a model case or normative model within both legal scholarship and actual legal practice (Fig. 1). 123 350 A. Taal et al. Old Case Law Information Backfile New Case Preparation Conversion Law Case Law Related Information (Semi- (Articles, structured Briefs) Information) Intelligent Data Character Cleansing Recognition (ICR) Content Named Entity Indexing Entity Extraction Extraction (NER) (CEE) Human- Powered Directories Auto Concept Classification Clustering Auto Pattern Search Categorization Analysis Semantic Parametric/ Search Fielded Queries Concept and Ontologies Fuzzy Search Fig. 1 Proposed model—Information Preparation, Index and Analysis model (IPIA model) 123 Cognitive computing and proposed approaches to conceptual… 351 We further hypothesize that any such analysis of the available case law data would be worthwhile for practitioners by presenting at the very least a different perspective on the cases. And if it performs well, this analysis might lead to further automated tools that could, perhaps, extend such reasoning into the future (given other hypothetical factual inputs) and suggest where case law might be trending. To research these points, we examine current text analytic and cognitive computing models to see how and where they might best be applied to the case law corpus to generate the values Hafner contemplated. This research coalesces into a model called the ‘‘Information Preparation, Index and Analysis Model (IPIA Model)’’ that incorporates existing technologies and approaches and which may perhaps even be capable of automatically assigning domain concepts and case frames and further allocating the cases into sets of ‘‘like’’ matters and creating patterns where none currently exist. Finally, we examine more future- reaching approaches (e.g., the IBM Watson and Google DeepMind’s AlphaGo program) in a nod to Hafner’s conjectures and include our thoughts on their applicability. 2 The proposed model: information preparation, index and analysis model (IPIA model) We examine current resources, present-day information governance practices, and relate generally-applicable analysis tools to determine if Hafner’s original COC strategy is something that existing technology could perform given access to the right information and unfettered use of current processes. We conclude that not only is this performance available, it might be made available with very little additional human input. The model we developed from this examination is composed of three steps: (1) information preparation, using existing information resources; (2) indexing, using a number of technologies devoted to cleansing and parsing the data; and (3) analysis, using present-day analytical tools on the collected and parsed data to ﬁll the ﬁelds Hafner originally envisioned (Hafner 1987; Wyner 2012). Our research also determines that certain types of scholarship that echo Hafner’s model are currently being done on an ad hoc basis (Reidenberg et al. 2013), supporting our contention that this approach is not only feasible but also worth the time and effort it would require for legal practitioners, legal researchers and others with an interest in this space—who might include interested pro se litigants who require access to the law in a fashion that provides value even in those instances where the litigant has very little familiarity with the law or court practices (Dixon 2016; Pelletier and Loubier 2016). Finally, while we examine other, even more novel approaches focusing on Technology Assisted Review (TAR), the IBM Watson cognitive computing model (Watson) and the Google technology (Google), our proposed model does not require more than the basics of some of these approaches, although it would certainly stand to beneﬁt from them. 123 352 A. Taal et al. 2.1 Information preparation 2.1.1 Old case law (pre-modern digitization) The beginning stages of our model look at various data sources and examine available information that can be fed into the system, whether through existing digital case law or additional Old Case Law (comprised of prior United States court decisions that have not yet been digitized or made available to the general public due to restricted availability by courts) (Fig. 2). There is a signiﬁcant volume of such hard copy material being digitized at present, including highly publicized ventures such as the Harvard ‘‘Free the Law’’ Digitization project now known as the ‘‘Caselaw Access Project’’, which seeks to ‘‘scan and digitize’’ the Harvard Law School collection of United States case law reports by mid- 2017, thereby digitizing the ‘‘ofﬁcial print version of all historical U.S. court decisions’’ comprising 40 million pages of law dating back to the 1700s (Dixon 2016). This trend toward online publication of present and past case law has gained momentum through the years, as evidenced in those instances where national bodies are heavily involved in ‘‘providing online access to case law, legislation and quite a rich selection of secondary materials’’ (Pelletier and Loubier 2016). These resources are used for a variety of purposes, including other projects designed to consider trends within case law as a whole, i.e. trends within certain courts and other countries (Alden et al. 2007). Finally, Old Case Law also carries with it more than simply the words in the cases themselves. It also incorporates the framework carried forward from the early National Reporting System’s inherent structural coherence, which remained even when the subject structure itself was modiﬁed going forward (Berring 1987). This type of structure may be useful for those analytical approaches that look to where certain words or phrases are located to help determine their import. 2.1.2 Backﬁle conversion A determination of what should be converted as part of a digitization project in support of the COC model would then lead to a backﬁle conversion project (Ross and Old Case Law Information Backfile New Case Preparation Conversion Law Case Law Related Information (Semi- (Articles, Briefs) structured Information) Fig. 2 IPIA model—Information Preparation Stage 123 Cognitive computing and proposed approaches to conceptual… 353 Ryona 1977)—that is, an examination of other available sources of Old Case Law information that would be scanned or otherwise digitally ingested in order to serve as part of the overall data corpus against which the COC strategy could be deployed through a number of possible technological means. We note that, given the efforts of the programs discussed above, a comprehensive approach to backﬁle conversion that digitizes all case law would certainly advance this portion of the model. Further, this process of backﬁle conversion, while not represented as such in the diagram, is additionally available for application against other related information that could touch upon case law while also giving context to it, such as law review articles contemporaneous to the case law or other scholarly or academic treatments. Backﬁle conversion is an important part of why the present-day application of Hafner’s COC strategy has become more practical, as there have been a number of ‘‘massive backﬁle conversion projects of journal publishers such as Elsevier, Wiley, and Springer’’ and additional investments across the course of a subset of databases that themselves offer up a gold mine of available legal scholarship (Garrett 2006). Reidenberg et al. began their paper by outlining those databases that the team searched for information, indicating the use of ‘‘a comprehensive set of academic databases to assure an accurate canvassing of legal scholarship’’ (2013). These were: • An Index to Legal Periodicals, covering the 500 most signiﬁcant legal periodicals; however, the index only includes titles and did not allow full-text search. • The Lexis database, offering full-text search for virtually every American legal periodical, as well as numerous foreign and international journals, including those otherwise unpublished cases only Lexis publishes (Robel 1989). • The Westlaw database, offering full-text search for virtually every American legal periodical, including those otherwise unpublished cases only Westlaw publishes (Robel 1989). • The Index to Foreign Legal Periodicals, providing an index of foreign (i.e. non- U.S.) legal periodicals published in English. Like the Index to Periodicals, this index includes only publication titles and does not offer full-text search (Reidenberg et al. 2013). At ﬁrst blush, the work by Reidenberg et al. (2013) seemed to pick-and-choose digitized content in order to approach speciﬁc questions. However, we imagine that they, like us, would have happily accepted an even fuller set of legal scholarship materials. And given the opportunity to perform the COC analysis we describe further below, there does not seem to be any immediate qualitative downside to running the analysis across (or within) all available information. Certainly more data makes for a longer analytic cycle, but the beneﬁts should greatly outweigh the additional computing load. 2.1.3 Related information Related Information could include Law Review articles or other legal scholarship and party brieﬁng associated with published opinions—whether written and 123 354 A. Taal et al. submitted in connection with a speciﬁc case, or otherwise citing a case. The authors of Law Review articles and other legal scholarship provide additional connections between case fact patterns that might aid an indexing model in presenting those same connections within the automatic process, further enriching the prepared data set. Party brieﬁng, while making perhaps other (but perhaps fewer) connections between cases, often presents additional facts about the dispute that court decisions omit but which would also serve to enrich the prepared data set. 2.2 New case law and semi-structured information As additional case law is added to the canon and to online repositories, the consistent form of cases and its evolution into both a comprehensive form of publishing as well as the placing of case law information into a ‘‘like format’’ make it even easier to divine what certain words or phrases mean based on where they are located (Berring 1987). This type of page-location-based divination is common in image analysis, where a group of pixels in one location means something entirely different in another, and the same would apply, for example, where the word ‘‘Wood’’ would mean something different if found in the caption of a case (likely a party to the litigation) rather than only within the case itself where it would more often be ‘‘wood from a lumberyard’’ (Luo et al. 2005). 2.2.1 Data cleansing That real world ‘‘data is dirty’’ is a common statement (Herna ´ndez and Stolfo 1998), and certain types of data—including those associated with the type of character recognition issues discussed below—require additional steps to create accurate data repositories (Gruman 2007) (Fig. 3). Data cleansing is the processing of cleaning data to remove corrupted and inaccurate records from a data set and in certain instances conversions and decrypting of data to render them readable. The mechanisms involved work hand- in-hand with such technologies as Intelligent Character Recognition (ICR) which, as Intelligent Character Data Cleansing Recognition (ICR) Content Entity Named Entity Indexing Extraction (CEE) Extraction (NER) Human-Powered Directories Fig. 3 IPIA model—Indexing Stage 123 Cognitive computing and proposed approaches to conceptual… 355 described more fully below, recognizes text from images and handwritten works and converts them to editable and searchable text. Data Cleansing would then incorporate other processes to determine if a speciﬁc entry is an address that might be missing a zero in a zip code or a state name that, while still misspelled, is identiﬁable based on the context in which the word is found and might be automatically corrected by the data cleansing efforts. Data cleansing mechanisms in this model can do more than simply get rid of inaccurate addresses or incomplete forms. They might be employed intentionally to remove information prohibited by copyright or other restrictions (Berring 1987). They could also remove personally identiﬁable information (PII) (Bouzeghoub and Lenzerini 2001) in those instances where party names or other references would be unhelpful for the COC modeling process and could otherwise potentially harm the individuals to which they refer. 2.2.2 Intelligent character recognition ICR is an advanced form of traditional Optical Character Recognition systems that examine text, including handwriting (to varying degrees of efﬁcacy) and indexes that information back into the database (Jain and Lazzerini 1999). These ICR systems may even be used in combination to further improve results (Rahman and Fairhurst 2003), such that the available information for the analytical techniques discussed below is more comprehensive and can produce better recognition rates. 2.2.3 Content entity extraction Content Entity Extraction (CEE) aims to ‘‘extract content from human language automatically’’ (Doddington et al. 2004). That is, the process attempts to recognize proper names and associated data, including the recognition of ‘‘[entities, not just names,’’ as well as ‘‘relations…and the characterization of relations between (pairs of) entities]’’ according to the following: • Role—the role a person plays in an organization, which can be sub-typed as Management, General Staff, Member, Owner, Founder, Client, Afﬁliate-Partner, Citizen-Of, or Other; • Part—i.e., part-whole relationships, sub-typed as Subsidiary, Part-Of, or Other; • At—location relationships, which can be sub-typed Located, Based-In, or Residence; • Near—to identify relative locations; and • Social—sub-typed as Parent, Sibling, Spouse, Grandparent, Other-Relative, Other-Persona, Associate, or Other-Professional (Doddington et al. 2004). This CEE analysis would create a case model within the legal domain as is done in legal ontology model creation (Visser and Bench-Capon 1998). The legal ontology models were ‘‘developed to address the legal domain’’ and relate to representations of, among other things, legal concepts and legal knowledge (Griffo et al. 2015). This development demonstrates how legal ontologies represent a 123 356 A. Taal et al. similar attempt to determine an explicit conceptualization of the knowledge-system domains of the kind that would be useful in creating the type of relationship matrix Hafner’s COC considers (Visser and Bench-Capon 1998). Indeed, Noy and Hafner examined exactly this approach when reviewing the then-current state of the art in Ontology Design (Noy and Hafner 1997). The CEE process would support that explicit conceptualization aim by automatically creating a set of links between and among identiﬁed parties. An aspect of the ‘‘Free the Law’’ project, where the scanned TIFF and JP2 images are transformed into ‘‘machine-readable XML ﬁles’’ with the processing for that project redacting ‘‘copyrighted publisher-created annotations and any other editorial content’’ (Dixon 2016), would be further helpful with this part of the process where content management watermarks or copyright materials (e.g., ‘‘Westlaw Content’’) would be ignored by the CEE process. 2.2.4 Named entity extraction Named Entity Extraction (NER) is a logical extension of the ICR process, where the program automatically identiﬁes speciﬁc names, or employs a ‘‘name spotting system’’ across a corpus of information (Baluja et al. 2000). NER would operate in tandem with the CEE process to further distinguish where relationships between identiﬁed parties actually existed. 2.2.5 Human-powered directories Here, the human element shines, importantly—but as a component of the indexing process rather than a best case. While human efforts may be much more prone to inconsistency (Tarakeswar and Kavitha 2011) and related errors that are altogether too human, there are certainly relationships between and among parties that may present complications that computational algorithms are not yet sophisticated enough to deal with. Further, Human-Powered Directories may also help guard against concerns that too great a reliance on computer analytics absent a human touch may lead to the elimination of ‘‘associative trails’’ (Browne et al. 1945) which ‘‘characterize human thought and sets it apart from machine process’’ (Garrett 2006). 2.3 Analysis 2.3.1 Auto classiﬁcation Analysis is not a panacea; at its best, one author noted that it is, in its most basic form, ‘‘keyword searching as our generation’s answer to the problem of information glut’’ when combined with ‘‘various mechanical sorting and ranking algorithms’’ to create ‘‘the dyke we have erected against the ﬂood of information that the progressive digitization of all texts is creating’’ (Garrett 2006) (Fig. 4). However, it remains a powerful tool and one of the underpinning practices for our proposed approach to assist with further statistical analysis (Lemieux 2012). 123 Cognitive computing and proposed approaches to conceptual… 357 Auto Classification ConceptClustering Auto Pattern Search Categorization Analysis Parametric/ Fielded Semantic Search Queries Concept and Fuzzy Ontologies Search Fig. 4 IPIA model—Analysis Stage 2.3.2 Auto categorization As one author notes, ‘‘the lack of coherence of the results of full-text searching is fundamentally at odds with natural patterns of knowledge acquisition’’ (Garrett 2006). Case law analysis might be split into two main approaches—fact extraction and sentence extraction—where the ‘‘former uses information extraction techniques to ﬁll predeﬁned templates which serve as a summary of the document; the latter compiles summaries by extracting key sentences with some smoothing to increase the coherence between the sentences’’ (Grover et al. 2004). Here, the model would attempt to do both, categorizing documents according to the facts and relationships underlying them, and then creating those summaries—in our pre-determined form— that Hafner’s COC model suggests to us. 2.3.3 Semantic search Semantic Search operates by way of the searcher’s intent (Guha et al. 2003); here, we begin with Hafner’s COC model. This approach uses existing semantic search techniques, such as text mining and document summarization, and leaning away from a ‘‘static dictionary’’ to ‘‘understand the intent of a searcher’s query within a speciﬁc context’’ (Patel 2016). As discussed before, the location of certain words and phrases within captions, in summaries, in footnotes, or within case citations will aid in this work. 2.3.4 Ontologies Ontologies are simply forms of shared conceptualization (Tran et al. 2007); we will use this structure further to classify and learn from the case law and related source corpus as we deﬁne relationships within the Hafner COC model. Case law ontologies upon which we would build this portion of the analysis already exist and 123 358 A. Taal et al. were placed upon published United States case law relatively early on in the form of the ‘‘West American Digest System’’ (Berring 1986). Early forms of this type of imposed structure took the form of the ‘‘universal subject thesaurus’’ where ‘‘language and concepts were normalized as the West editor prepared the headnotes for each case…which helped to make the law comprehensible and lent overall structural coherence’’ (Berring 1987). The current ontological models have increased with sophistication over time, including those used in eDiscovery related products and services (Taal et al. 2015), and should be encompassing enough to incorporate fully the COC approach. 2.3.5 Concept clustering Concept clustering provides a way to cluster objects (or documents) together by deﬁning a concept common to those objects (Zhang and Wu 2008). These may be especially needed in those environments where similar or ‘‘closely related concepts’’ exist within the corpus of data being examined (Fung et al. 2011)— which is certainly the case within the world of legal research, case law, and articles written directly addressing the same. In addition, the uses of technology and metadata have been applied to legal texts since Hafner’s original exploration of the topic, with automatic summarization processes having progressed to a point of technical feasibility where services that rely on statistical classiﬁers can categorize sentences as candidates for text excerpts for summaries. Such approaches incorporate word frequency measures as well as other features that reﬂect the linguistic properties of those candidates (Grover et al. 2003). In fact, the literature distinguishes between indicative and informative summaries where the ‘‘former provides a reference function for selecting documents for more in-depth reading while the latter aims to cover all the salient information in the source at some level of detail’’ (Hachey and Grover 2006). 2.3.6 Pattern search Pattern search is nothing new and incorporates additional methods by which searchers may sift and sort documents into ‘‘basic groups using ﬁltering technologies based on document content’’ (Smallwood 2014). We expect that, like other researchers, we will ﬁnd this process most useful as part of an evolutionary approach as the analysis progresses (Koziel and Michalewicz 1999) and certainly as part of the ‘‘predictive coding’’ or ‘‘technological review process’’ we also consider (Smallwood 2014). 2.3.7 Parametric/ﬁelded queries In the Hafner COC context, parametric and ﬁelded queries will examine structured data tables created through the use of the other processes discussed above, using precision of the ﬁelds to determine how to best sort and categorize available data (Chang et al. 1999). We expect that this point of the analysis will be helpful not only after the data has been cleansed and fully reviewed, but also as part of the iterative 123 Cognitive computing and proposed approaches to conceptual… 359 nature of the entire process and as part of a quality-control approach for how the data corpus looks as a whole. 2.3.8 Concept and fuzzy search We want to conﬁrm that our approach would not be limited exactly to what the words seem to indicate but also would include what they mean given the context and the multiplicity of potential meanings as well as misspellings and other user (or OCR/ICR) errors (Sanchez and Yamanoi 2006). Further, fuzzy search has a special place in the hearts of eDiscovery practitioners, as this type of analysis has been codiﬁed in case law (250 FRD 251) and given rise to an expectation within the ﬁeld such that any approach such as ours should include speciﬁc references in order to support the COC model’s acceptance in whatever manner it might be helpful for practitioners. 3 Further analysis In addition to the above considerations, we examine three relatively well-publicized, modern approaches that could be further utilized in developing the COC model which could aid practitioners and legal decision makers. We considered Technology Assisted Review in general (Grossman and Cormack 2013), IBM’s Watson platform, and Google’s AlphaGo engine (Silver et al. 2016) to determine what additional facets each could provide given the overall composition of our proposed model. 3.1 Technology assisted review Technology assisted review (TAR) is deﬁned as a process for: Prioritizing or Coding a Collection of Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of documents and then extrapolates those judgments to the remaining Document Collection. Some TAR methods use Machine Learning Algorithms to distinguish Relevant from Non-Relevant Documents, based on Training Examples Coded as Relevant or Non-Relevant by the Subject Matter Experts(s), while other TAR methods derive systematic Rules that emulate the expert(s)’ decision-making process. TAR processes generally incorporate Statistical Models and/or Sampling techniques to guide the process and to measure overall system effectiveness (Grossman and Cormack (2013), hereinafter cited as ‘‘TAR Glossary’’). An analysis of TAR may also consider three tasks: search, analysis, and review. Seminal legal researchers in the ﬁeld Grossman and Cormack (2016) posit that the ‘‘objective of search is to ﬁnd enough documents to satisfy an information need, such as the answer to a question or support for a proposition’’ and that the ‘‘objective of analysis is to gain understanding from available data, so as to support 123 360 A. Taal et al. decision-making’’. The objective of ‘‘review’’ speaks speciﬁcally to responsiveness to a request for production in litigation, which is less applicable to our analysis. The model is a good framework to push the proverbial envelope even further into an automated process that would model the Hafner COC approach using limited human resources and intervention. The ﬁrst two considerations seem, at least at face value, to support Hafner’s COC approach if only in a limited sense. That is, we read the COC approach as the process of essentially applying a third-party framework to existing cases in order to create additional data about those cases that may be used to categorize the cases. Addressing analysis ﬁrst, we would use TAR technologies to analyze the case law presented in an automated fashion. This type of analysis could vary from identifying how many parties there were to a speciﬁc action, the location of the case, or the date of the decision. This type of analysis could then go into a second level of detail regarding the language used within the case, beginning with the terms of art related to speciﬁc types of action and then determining what other vocabulary coincided with those terms of art. Using this type of incremental analysis across the digitized corpus of case law, and building up different proﬁles for cases, we could then search across those cases to ﬁnd subsets of cases that are most similar to one another. Using TAR as applied to the case law, we could then ‘‘rank or make afﬁrmative predictions about the relevance or non-relevance of’’ speciﬁc cases based on reviewer evaluation of a subset and then use those predictions to assemble a more robust set of cases from which we could make additional, statistically supported observations about those cases (Grossman and Cormack 2016). In this way, we would automate our evaluation of existing case law according to Hafner’s COC approach without the need to review each case manually and make subjective determinations. Not only would this approach eliminate that need, it might also improve the process by avoiding the human error recognized in earlier studies by researchers Blair and Maron (1985). Another paper examining the history of the Blair and Maron scholarship notes a similar consideration to that which we raise regarding the new availability of technology and its support of new technologies that might assist a feasible approach of Hafner’s COC approach—especially one that does not require expensive human input or that might perform better without it. Regard and Matzen (2013) state that in 1985 ‘‘full-text was scarce, as documents were largely available as paper documents, not as full-text electronic ﬁles. Paper to text conversion via optical character recognition (‘‘OCR’’) was in its infancy at the time and it was estimated to cost 209 as much to type and verify full text documents as it did to type and verify manually assigned index terms’’. The human costs have not disappeared but as computing power has increased, the relative cost has dropped (Rotman 2013), and incorporating the automated process embodied in TAR will likely improve the results and continue to lower the costs. 3.2 The IBM model In 2007, IBM built Watson, a multi-faceted technology platform, which beat the two highest ranked Jeopardy players by mastering open-domain question answering 123 Cognitive computing and proposed approaches to conceptual… 361 (Ferrucci et al. 2013) and garnered a tremendous amount of publicity while doing so. Watson uses natural language processing and machine learning to reveal insights from large amounts of unstructured data. The end result is a multi-disciplinary module that can interpret data in a human-like manner and adapt its output in response to the introduction of new source references (Ferrucci et al. 2013). Watson was initially focused on data compilation for marketing campaigns, but its increasingly expansive cognitive capabilities were directed towards other endeavors, including the Watson-Marchesa collaboration for the 2016 Met Gala (Tadena 2016). Rooted in the belief that color and images can indicate moods and send messages, Marchesa selected ﬁve key human emotions—joy, passion, excitement, encouragement and curiosity—which they wanted their dress to convey. Social media reactions from the gala were fed into Watson’s cognitive color design model and then processed and displayed as color effects onto the fabric of the dress (Rubin 2016). While seemingly disparate, both Watson’s Jeopardy perfor- mance and the Marchesa model in fact illustrate the evolution in conceptual cognitive computing evaluation capability that Hafner felt was missing in (Hafner 1987). In fact, Watson’s legal module ROSS may match up well with the meta- analytical system Hafner considered the ‘‘missing link’’ required to have a robust case-based computational search model that could take into consideration the broader social and jurisprudential contexts which are critical components of case law (Hafner and Berman 2002). 3.2.1 ROSS Berman and Hafner (1993) predicted that case-based analysis would require a teleological component especially as complexity in resources for legal knowledge, court rulings, and laws grew, in turn making simple keyword searches inadequate when addressing the volume, variety, velocity, and veracity of legal data (Sills 2016). Using the IBM platform and its natural-language processing capability, ROSS was designed to ﬁlter through legal documents and understand the law and questions from lawyers as practitioners would ask one another and provide a response ranked on relationships and relevancy. For instance, one can ask ROSS questions in plain English like, ‘‘Can an automatic stay be lifted if a plaintiff in another case requests it?’’ (Cutler 2015). In response, ROSS ‘‘assesses legal precedents, suggests readings and attaches more information about the purpose behind the case paired with citation and conﬁdence ratings,’’ a process which ultimately makes the legal case organization and retrieval system at least a ‘‘little smarter’’ in the manner Hafner hoped. As ROSS is currently in its early stages, the majority of what we know about it comes from marketing media and blog posts. Keeping that in mind, ROSS’s remarkability thus far has been attributed to the fact that it is designed to mimic the human process of reading, identifying patterns in text and provide contextualized answers along with precedents and citations from case law. Hafner highlighted the importance of having this type of case-based legal analytical search structure, because attorneys and courts often tend to perceive and express the purpose behind decisions in varying levels of abstraction (Hafner and Berman 2002). ROSS aims to 123 362 A. Taal et al. take a step in that direction by factoring less tangible concepts like the spirit of the law unveiling political and social values and drawing inferences from the societal relationships underpinning case-based legal arguments. By applying a DeepQA approach, ROSS is learning to understand the law instead of simply translating words and syntax into search results. To achieve this human-like comprehension, ROSS was taught, in part, like any other legal student is also taught: it was loaded with a huge volume of public legal documents (Sills 2016). But unlike a law student, ROSS can sort through more than a billion text documents each second and its cognitive computing capability allows ROSS to learn from each question, incorporate feedback and get smarter over time (Jackson 2015). 3.2.2 The ROSS DeepQA process DeepQA is a standard parallel probabilistic evidence-based architecture (Boguraev et al. 2014) deployed in the ﬁrst stage of data processing that performs a detailed analysis of a question to determine what it is asking for and how best to approach answering it (Ferrucci et al. 2010). The overarching principles in DeepQA are massive parallelism, many experts, pervasive conﬁdence estimation, and integration of shallow and deep knowledge. • Massive parallelism: Exploit massive parallelism in the consideration of multiple interpretations and hypotheses. • Many experts: Facilitate the integration, application, and contextual evaluation of a wide range of loosely coupled probabilistic question and content analytics. • Pervasive conﬁdence estimation: No component commits to an answer; all components produce features and associated conﬁdences, scoring different question and content interpretations. An underlying conﬁdence-processing substrate learns how to stack and combine the scores. • Integrate shallow and deep knowledge: Balance the use of strict semantics and shallow semantics, leveraging [several] loosely formed ontologies (Ferrucci et al. 2010). The question analysis that follows an inquiry uses Watson’s parsing and semantic analysis capabilities: a deep Slot Grammar parser, a NER (as discussed above), a co- reference resolution component, and a relation extraction component or type of auto-classiﬁcation approach (Ferrucci et al. 2010). It applies numerous detection rules and classiﬁers using analytical features to parse the question presented, including: (1) that part of the question that references the answer (the ‘‘focus’’); (2) the part of the question indicating what is being asked for (lexical answer types); (3) a classiﬁcation of the question into broad types; and (4) elements of the question more esoteric in nature (e.g., ‘‘nested sub-questions that must be separately answered’’) (Ferrucci et al. 2010). When combined, these elements provide an end- to-end question answering case-based taxonomy for legal reasoning that is ranked by system accuracy (Wang et al. 2012). Although detecting semantic relations in text is an active challenge in natural- language processing and information retrieval, at the outset of developing the 123 Cognitive computing and proposed approaches to conceptual… 363 DeepQA approach the IBM team observed that they needed system-level advances, rapid integration, and the evaluation of new ideas and components to further their progress (Ferrucci et al. 2010). Detecting new relations to the question text improves both background relational knowledge as well as potential answers, thereby contributing to additional evidence to score supporting data (Wang et al. 2012). While she predicted its usefulness, it is unclear whether Hafner considered the precision capabilities of a cognitive system versus that of a system using a lexical search; however, notable differences exist. In the DeepQA question-answering framework, ROSS uses two primary approaches to broad domain relation and scoring: one based on manual pattern speciﬁcation and the other based on statistical methods for pattern elicitation, which uses a novel transfer learning technique, i.e., relation topics (Wang et al. 2012). Granted, the rule-based approach is more precise and is used by several DeepQA components, but it traditionally requires manual effort, which allows for coverage on only a small targeted set of relations (Chiticariu et al. 2013). Statistical approaches, on the other hand, automatically learn how to extract semantic relations from the data and can be applied to detect a large amount of relations (over 200 times more relations) (Navigli and Ponzetto 2012). Although the precision of the statistical relation detectors is not as high as that of the rule- based approach, their overall impact on the system through passage scoring is statistically signiﬁcant because of their broad coverage of knowledge (Wang et al. 2012). Although it may be open for debate whether this lower precision technique is ideal for a legal system, it certainly is something to consider when utilizing technology to streamline the legal processes. The DeepQA framework also contains a suite of decomposition rules that use predominantly lexico-syntactic features to identify facts within complex questions such as those that have hidden associations or implicit relationships (Chu-Carroll et al. 2012). It also contains a question-rewriting component and a candidate re- ranker, which uses machine learning and heuristic selection strategies to generate a ﬁnal ranked answer list, taking into account answer conﬁdences from the base question answer system (Kalyanpur et al. 2012). The ﬁnal stage in the IBM DeepQA pipeline involves ranking all candidate answers according to their evidence scores and judging the likelihood that each answer is correct. In DeepQA, this is done using a machine learning framework that is phase-based, providing capabilities for manipulating the data and applying machine learning in successive applications (Gondek et al. 2012). This is the ultimate test of applying machine learning for evidence-based hypothesis evaluation as the approach facilitates an agile development environment for DeepQA, in which evidence scoring strategies can be easily introduced, revised, and reconﬁgured without error-prone manual keyword search efforts (Gondek et al. 2012). 3.3 The Google model(s) Google has been involved in a number of algorithmic and related approaches to machine ‘‘learning,’’ including machine learning algorithms which focus on ‘‘[t]ext classiﬁcation and clustering [that] play an important role in many applications’’ such 123 364 A. Taal et al. as ‘‘document retrieval, web search, [and] spam ﬁltering’’ (Le and Mikolov 2014). Google DeepMind’s artiﬁcial intelligence program, AlphaGo, is based on a combination of deep neural networks and tree search that is able to play Go at the level of the strongest human players (Silver et al. 2016). It has been termed ‘‘exemplary’’ by its developers, tackling the many difﬁculties faced by artiﬁcial intelligence: ‘‘a challenging decision-making task, an intractable search space, and an optimal solution so complex it appears infeasible to directly approximate using a policy or value function’’ (Silver et al. 2016). Its ability to develop deep neural networks and apply them to a tree search would certainly allow it support Hafner’s COC approach, at least in a limited sense. AlphaGo was born out of a desire of Google DeepMind developers who set forth to create an artiﬁcial intelligence program that would lay the foundation for ‘‘software that is smart enough to solve humanity’s biggest problems’’ (Simonite 2016). To do this, the developers chose to develop a program with the capability of defeating world-class human players in the game Go, which is known as the ‘‘Holy Grail’’ of artiﬁcial intelligence and a frequent subject of analysis (van der Werf 2005). It is a notoriously difﬁcult game for computers to play because the number of potential legal board positions is greater than the number of atoms in the universe; a number that was only determined in early 2016 (Moyer 2016) despite the game originating in ancient China (Gelly and Wang 2006). Other artiﬁcial intelligence programs have challenged humans in games and emerged victorious, and one such example is the IBM supercomputer called Deep Blue that beat world chess champion Gary Kasparov in the pair’s second of two matches (Campbell et al. 2002). But for all of its complexities, chess does not hold a candle to Go as seen through the ‘‘eyes’’ of an artiﬁcial intelligence program. All games of perfect information—such as chess, tic-tac-toe, and Go—have an ‘‘optimal value function, v*(s), which determines the outcome of the game, from every board position or state s, under perfect play by all players’’ (Arrow 1953). Individuals can solve these games by ‘‘recursively computing the optimal value function in a search tree containing approximately b^d possible sequence of moves, where b is the game’s breadth—number of legal moves per position—and d is its depth—game length’’ (Silver et al. 2016). An exhaustive search is infeasible in large games such as chess (b = 35, d = 80) and Go (b = 250, d = 150) (Newman et al. 2016). However, the effective search space can be reduced in two ways (Silver et al. 2016). First, the search’s depth may be reduced by ‘‘position evaluation’’ (Silver et al. 2016) or truncating the search tree at a certain point and replacing the sub-tree below that point by an approximate value function that will predict the outcome from the point at which the search tree was truncated (Silver et al. 2016). This approach has led to impressive results in such games as chess, checkers, and Othello but posed signiﬁcant issues due to the complexity of Go (Bouzy and Cazenave 2001). In the same vein, it could easily predict straightforward case results but might run into problems with cases relying on more advanced legal theories. Second, the breadth of the search may be reduced by sampling actions from a policy p(a|s), that is a probability distribution over possible moves a in position s (Silver et al. 2016). In this manner, Monte Carlo rollouts are able to ‘‘search to maximum depth without 123 Cognitive computing and proposed approaches to conceptual… 365 branching out at all, by sampling long sequences of actions for both players from a policy p’’; taking the average over such rollouts will provide an effective evaluation of one’s position, which has resulted in ‘‘superhuman performance in backgammon and Scrabble, and weak amateur level play in Go’’ (Silver et al. 2016). Monte Carlo tree search (MCTS) uses Monte Carlo rollouts over time in order to estimate the values of each state in a search tree structure (Browne et al. 1945). The more simulations that are executed, the larger the search tree grows, and the more accurate the relevant values become (Browne et al. 1945). Simultaneously, the policy used to select actions during search is also improved over time as ‘‘children’’ with higher values are selected (Silver et al. 2016). Essentially, this policy ‘‘converges to optimal play’’ while ‘‘the evaluations converge to the optimal value function’’ (Silver et al. 2016) The strongest Go programs were based on MCTS that were enhanced by polices trained to predict human expert moves, but this approach has only risen to the level of strong amateur play (Silver et al. 2016). To create a more successful Go program, the developers at Google DeepMind drew inspiration from other successful convolutional neural networks (CNN or ConvNets) addressing visual domain issues such as image classiﬁcation and facial recognition (Krizhevsky et al. 2012). Those networks use ‘‘many layers of neurons, each arranged in overlapping tiles, to construct increasingly abstract, localized representations of an image’’ (Silver et al. 2016). Similarly, the developers ‘‘pass in the board position as a 19 9 19 image and use convolutional layers to construct a representation of the position’’ (Silver et al. 2016). These neural networks are then used to ‘‘reduce the effective depth and breadth of the search tree: evaluating positions using a value network and sampling actions using a policy network’’ (Silver et al. 2016). These advances, and notably the progression of AI ‘‘game playing’’ in chess and subsequently Go, have been declared ‘‘the ultimate litmus test of the effectiveness of machine learning and artiﬁcial intelligence’’ (Alarie 2016). Go is especially notable as, unlike chess (where IBM’s ‘‘Deep Blue won the match by applying brute force—it searched all possible moves and chose the best ones’’), Go requires a different approach that was long thought to require human intuition (Liu 2016). This, in turn, led Liu to consider (without a ﬁnal determination) whether the AlphaGo success brought the application of AI ‘‘a step closer to an AI lawyer who can provide legal advice.’’ (Liu 2016). The software’ neural networks are trained to achieve victory using a ‘‘pipeline consisting of several stages of machine learning’’ (Silver et al. 2016). First, a supervised learning (SL) policy network is trained directly from expert human moves (Silver et al. 2016). Training a 13-layer policy network from 30 million positions from the KGS Go Server (Silver et al. 2016) provides the software with ‘‘fast, efﬁcient learning updates with immediate feedback and high-quality gradients’’ (Silver et al. 2016). During this stage a less accurate ‘‘fast policy’’ is also trained so that it can ‘‘rapidly sample actions during rollouts’’ (Silver et al. 2016). Next, the engineers train a ‘‘reinforcement learning (RL) policy network,’’ which ‘‘improves the SL policy network by optimizing the ﬁnal outcome of games of self-play’’ (Silver et al. 2016). Essentially, in this stage, software is programmed to explore a new environment and adjust its behavior to increase some kind of virtual reward with the focus on the ‘‘correct goal of winning games, rather than 123 366 A. Taal et al. maximizing predictive accuracy’’ (Silver et al. 2016). This happens as a result of games played between the current policy network and a randomly selected previous iteration of the policy network (Silver et al. 2016). Finally, the engineers train a value network, which ‘‘predicts the winner of games played by the RL policy network against itself’’ (Silver et al. 2016). This stage focuses on position evaluation, estimating a value function ‘‘that predicts the outcome from position s of games played by using policy p for both players’’ (Silver et al. 2016). AlphaGo combines these policy and value networks in an ‘‘MCTS algorithm that selects actions by look ahead search,’’ which selects the most promising moves for the program (Silver et al. 2016). In sum, the developers had created a program that ‘‘learned to discovery new strategies for itself, by playing millions of games between its neural networks, against themselves, and gradually improving’’ (Metz 2016). This approach to game playing is relevant to the current practice of law insofar as courts are relying less on ‘‘bright line’’ or black/white distinctions for decisions and are instead making decisions along the lines of ‘‘relationship tests’’ that examine relationships that fall between such extremes (Alarie 2016). Due to the ‘‘test’’ nature of these determinations, Alarie even posits that such game playing AI computers and systems may be better suited to make them, as they can ‘‘simultaneously take into account every judgment that [one] has been exposed to in training, can construct elaborate models of relationships among the variables accounted for in the case law, and can quantify with precision the result of the algorithms used to evaluate the possibilities’’ just as the AI systems do when playing these games (Alarie 2016). This type of functionality could become part of the COC approach by using the technology to analyze the case law as it is presented to the device automatically. The SL policy network could be trained directly from expert human moves; that is, by ‘‘learning’’ as a researcher sorts case law according to the desired outcome. Second, the RL policy network could be trained by experiencing a higher level analysis with respect to the language used within the case. These connections would be ampliﬁed during the training of the value network, which would allow AlphaGo to predict more focused and accurate outcomes when combing through the case database. 4 Conclusions and future work ROSS is potentially a powerful new approach for applying structure and reasoning over unstructured natural language content while providing a foundation for developing decision support systems that can address many of the cognitive challenges practitioners face as well as some of the weaknesses of prior approaches (Ferrucci et al. 2013). With its ability to employ IBM’s DeepQA query technology, ROSS may become useful to its users over time especially when coupled with its cognitive ability to adapt and evolve (Ferrucci et al. 2010). A speciﬁc challenge to the Hafner COC model, however, is that statutes, regulations, how-to-guides, policies, contracts, and of course case law do not seem to work together especially well, making it challenging for systems like Watson to interpret them (Lippe and 123 Cognitive computing and proposed approaches to conceptual… 367 Katz 2014) and difﬁcult to create further metadata of the type Hafner thought would be so valuable. Therefore, while ROSS/Watson can help keep costs down and lower the point of entry for clients, weighing data at a fairly sophisticated level, drafting documents, making arguments, and creating the types of consistent stories between and among cases that Hafner envisioned will still be left in large part to human lawyers interfacing with this platform (Sills 2016). In many ways, Go embodies ‘‘the difﬁculties faced by artiﬁcial intelligence: a challenging decision-making task, an intractable search space, and an optimal solution so complex it appears infeasible to directly approximate using a policy or value function’’ (Mandziuk 2007). It is easy to picture the challenges faced by Hafner’s COC proposal becoming a reality through a tool with the sophistication of AlphaGo. The team behind Google’s DeepMind unit wanted to create ‘‘general artiﬁcial intelligence’’—something that, like a human, can learn to take on just about any task, envisioning it ‘‘doing things as diverse as advancing medicine by formulating and testing scientiﬁc theories, and bounding around in agile robot bodies’’ (Simonite 2016). The goal was for DeepMind’s researchers to ‘‘learn how to build even smarter software, and push the software to learn how to tackle more difﬁcult decisions and problems’’ (Simonite 2016). The creators behind AlphaGo, then, were intent on creating a tool that might propel Hafner’s COC into reality with minimal effort once the process began. Still, while aspirations are helpful, the approach has yet to produce a model that would provide the Hafner COC model with reliable underpinnings on which practitioners, scholars, and judges could depend. The integration of Technology Assisted Review would also likely provide value, especially given how certain technologies within the eDiscovery space are integrating many of the primary model’s considerations into software platforms and suites. And while we did not identify a one-stop shop that would incorporate the entirety of our proposal into its offering, there may, in fact, be one that would make this analysis even more academic than it already is. Finally, the primary model we discuss above (IPIA Model) is more of a combination approach that would have to be applied against a robust body of case law and then revised. This approach is still theoretical, but it is based on existing— and, in many cases, fairly established—technologies and analytical approaches. We remain optimistic that this approach could work, and that it would provide value to legal scholars as well as practitioners, if only because our research indicated that others are trying to replicate certain parts of this proposed framework in support of their own efforts. References Alarie B (2016) The path of the law: toward legal singularity. Available at Social Science Research Network 2767835. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2767835. Accessed 07 Sept Alden C, Ellis S, Lesnak B, Mueller J, Ryan MC (2007) Trends in federal judicial citations and law review articles. Court of Appeals for the Second Circuit Roundtable Discussion, Benjamin N. Cardozo School of Law Arrow KJ et al (1953) Contributions to the theory of games, II edn. Princeton University Press, Princeton 123 368 A. Taal et al. Baluja S, Mittal VO, Sukthankar R (2000) Applying machine learning for high-performance named-entity extraction. Comput Intell 16(4):586–595 Bench-Capon T, Sartor G (2003) A model of legal reasoning with cases incorporating theories and values. Artif Intell 150(1–2):97–143 Bench-Capon T, Atkinson K, Chorley A (2005) Persuasion and value in legal argument. J Log Comput 15(6):1075–1097 Bench-Capon T, Prakken H, Wyner A, Atkinson K (2013) Argument schemes for reasoning with legal cases using values. In: Proceedings of the fourteenth international conference on artiﬁcial intelligence and law, pp 13–22 Berman D, Hafner C (1993) Representing teleological structure in case-based reasoning: the missing link. In: Proceedings of the fourth international conference on artiﬁcial intelligence and law. ACM, New York, pp 50–59 Berring RC (1986) Full-text databases and legal research: backing into the future. High Technol Law J 1(1):27–60 Berring RC (1987) Legal research and legal concepts: where form molds substance. Calif Law Rev 75:15–19 Blair D, Maron ME (1985) An evaluation of retrieval effectiveness for a full-text document-retrieval system. Commun ACM 28(3):289–299 Boguraev B, Patwardhan S, Kalyanpur A, Jr C-C, Lally A (2014) Parallel and nested decomposition for factoid questions. Nat Lang Eng 20(04):441–468 Bouzeghoub M, Lenzerini M (2001) Introduction to: data extraction, cleaning, and reconciliation a special issue of information systems, an international journal. Inf Syst 26(8):535–536 Bouzy B, Cazenave T (2001) Computer Go: an AI oriented survey. Artif Intell 132(1):390–403 Browne C, Powley E, Whitehouse D, Lucas SM, Cowling PI, Rohlfshagen P, Tavener S, Perez D, Samothrakis S, Bush V (1945) As we may think. Atl Mon 101–108 Campbell M, Hoane AJ, Hsu FH (2002) Deep blue. Artif Intell 134(1):57–83 Chang CCK, Garcia-Molina H, Paepcke A (1999) Predicate rewriting for translating Boolean queries in a heterogeneous information system. ACM Trans Inf Syst (TOIS) 17(1):1–39 Chiticariu L, Li Y, Reiss FR (2013) Rule-based information extraction is dead! Long live rule-based information extraction systems! In Proceedings of the 2013 conference on empirical methods in natural language processing, pp 827–832 Chu-Carroll J, Brown E, Lally A, Murdock JW (2012) Identifying implicit relationships. IBM J Res Dev 56(3.4):12 Colton S (2012) A survey of Monte Carlo tree search methods computational intelligence and AI in games. IEEE Trans Comput Intell AI Games 4(1):1–43 Cutler KM (2015) YC’s ROSS Intelligence Leverages IBM’s Watson to make sense of legal knowledge. TechCrunch. http://techcrunch.com/2015/07/27/ROSS-intelligence. Accessed 07 Sept 2016 Dixon JA (2016) Harvard Launches ‘‘Free the Law’’ Digitization Project. Library Journal. http://lj. libraryjournal.com/2015/12/oa/harvard-launches-free-the-law-digitization-project/. Accessed 07 Sept 2016 Doddington GR, Mitchell A, Przybocki M, Ramshaw L, Strassel S, Weischedel R (2004) The Automatic Content Extraction (ACE) program-tasks, data, and evaluation. In: Conference on Language Resources and Evaluation, vol 2, p 1 Ferrucci D, Brown E, Chu-Carroll J, Fan J, Godek D, Kalyanpur AA, Lally A, Murdock JW, Nyberg E, Prager J, Schlaefer N, Welty C (2010) Building Watson: an overview of the DeepQA Project. AI Magazine 31(3):59–79 Ferrucci D, Levas A, Bagchi S, Gondek D, Mueller ET (2013) Watson: beyond Jeopardy. IBM J Res Dev 199:93–105 Fung ST, Tam V, Lam EY (2011) Enhancing learning paths with concept clustering and rule-based optimization. Advanced Learning Technologies (ICALT). In: 2011 11th IEEE international conference on. advanced learning technologies (ICALT), pp 249–253 Garrett J (2006) KWIC and dirty? Human cognition and the claims of full-text searching. J Electron Publ 9(1). http://quod.lib.umich.edu/j/jep/3336451.0009.106?view=text;rgn=main. Accessed 07 Sept 2016 Gelly S, Wang Y (2006) Exploration exploitation in go: UCT for Monte-Carlo go. In: NIPS: Neural Information Processing Systems Conference On-line trading of Exploration and Exploitation Workshop 123 Cognitive computing and proposed approaches to conceptual… 369 Gladwell M (2011) Creation Myth: Xerox PARC, Apple, and the truth about innovation. The New Yorker. http://www.newyorker.com/magazine/2011/05/16/creation-myth. Accessed 07 Sept 2016 Gondek DC, Lally A, Kalyanpur A, Murdock W, Duboe ´ PA, Zhang L, Pan Y, Qiu ZM, Welty C (2012) A framework for merging and ranking answers in DEEPQA. IBM J Res Dev 56(3.4):14-1 Griffo C, Almeida JP, Guizzardi G (2015) A systematic mapping of the literature on legal core ontologies. In: 7th Brazilian Symposium on Ontology Research Grossman MR, Cormack GV (2013) The Grossman-Cormack glossary of technology-assisted review. Fed Courts Law Rev 7:85 Grossman MR, Cormack GV (2016) A tour of technology-assisted review. Perspectives on Predictive Coding and Other Advanced Search and Review Technologies for the Legal Practitioner (ABA 2016) Grover C, Hachey B, Hughson I, Korycinski C (2003) Automatic summarisation of legal documents. In: International conference on artiﬁcial intelligence and law, pp 243–251 Grover C, Hachey B, Hughson I (2004) The HOLJ Corpus: supporting summarisation of legal texts. In: The 5th International Workshop on Linguistically Interpreted Corpora Gruman G (2007) Treasure hunt. CIO Magazine. http://www.cio.com.au/article/204860/treasure_hunt/ ?pp=3. Accessed 07 Sept 2016 Guha R, McCool R, Miller E (2003) Semantic search. In: Proceedings of the 12th international conference on World Wide Web. ACM, pp 700–709 Hachey B, Grover C (2006) Extractive summarisation of legal texts. Artif Intell Law 14(4):305–345 Hafner C (1987) Conceptual organization of case law knowledge bases. In: Proceedings of the 1st international conference on artiﬁcial intelligence and law. ACM, pp 35–42 Hafner C, Berman D (2002) The role of context in case-based legal reasoning: teleological, temporal, and procedural. Artif Intell Law 10(1–3):19–64 Hernandez MA, Stolfo SJ (1998) Real-world data is dirty: data cleansing and the merge/purge problem. Data Min Knowl Discov 2(1):9–37 Jackson B (2015) Meet ROSS, the Watson-powered super intelligent super attorney itbusiness.ca. www. itbusiness.ca/news/meet-ROSS-the-watson-powered-super-intelligent-attorney/53376. Accessed 07 Sept 2016 Jain LC, Lazzerini B (eds) (1999) Knowledge-based intelligent techniques in character recognition. CRC, London Kalyanpur A, Patwardhan S, Boguraev BK, Lally A, Chu-Carroll J (2012) Fact based question decomposition in DEEPQA. IBM J Res Dev 56(34):13-1 Koziel S, Michalewicz Z (1999) Evolutionary algorithms, homomorphous mappings, and constrained parameter optimization. Evol Comput 7(1):19–44 Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classiﬁcation with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105 Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: ICML Lemieux S (2012) Auto-classiﬁcation: Friend or Foe of taxonomy management?. CMS WiRE. http:// www.cmswire.com/cms/information-management/autoclassiﬁcation-friend-or-foe-of-taxonomy- management-014222.php. Accessed 07 Sept 2016 Lippe P, Martin KD (2014) 10 predictions about how IBM’s Watson will impact the legal profession. ABA Journal. http://www.abajournal.com/legalrebels/article/10_predictions_about_how_ibms_ watson_will_impact/. Accessed 07 Sept 2016 Liu B (2016) Can artiﬁcial intelligence ever give legal advice? Brief 43(6):8 Luo J, Savakis AE, Singhal A (2005) A Bayesian network-based framework for semantic image understanding. Pattern Recognit 38(6):919–934 Mandziuk J (2007) Computational intelligence in mind games. In: Challenges for computational intelligence. Springer, Berlin, pp 407–442 Metz C (2016) In two moves, AlphaGo and Lee Sedol redeﬁne the future. Wired. http://www.wired.com/ 2016/03/two-moves-alphago-lee-sedol-redeﬁned-future/. Accessed 06 Sept 2016 Moyer C (2016) How Google’s AlphaGo Beat a Go World Champion. The Atlantic. http://www. theatlantic.com/technology/archive/2016/03/the-invisible-opponent/475611/. Accessed 06 Sept Navigli R, Ponzetto SP (2012) BabelNet: the automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif Intell 193:217–250 Newman AJ, Richardson CI, Kain SM, Stankiewicz PG, Guseman PR, Schreurs BA, Dunne JA (2016) Reconnaissance blind multi-chess: an experimentation platform for ISR sensor fusion and resource 123 370 A. Taal et al. management. In: SPIE Defense ? Security. International Society for Optics and Photonics, p 984209 Noy N, Hafner C (1997) The state of the art in ontology design: a survey and comparative review. Artif Intell Mag 18(3):53 Patel N (2016) Everything you need to know about semantic search and what it means for your Website. The Daily Egg. https://blog.crazyegg.com/2016/02/02/everything-about-semantic-search/. Accessed 07 Sept 2016 Pelletier F, Loubier S (2016) Collaboration among Legal Information Providers in Quebec: CAIJ and Lexum’s Shared Vision. J Open Access Law 4(1) Rahman AFR, Fairhurst MC (2003) Multiple classiﬁer decision combination strategies for character recognition: a review. Doc Anal Recognit 5(4):166–194 Regard D, Matzen T (2013) A re-examination of Blair & Maron (1985). DESI V Position Paper Reidenberg JR, Debelak J, Kovnot J, Bright M, Russell NC, Alvarado D, Seiderman E, Rosen A (2013) Internet Jurisdiction: A survey of legal scholarship published in English and United States Case Law. Center on Law and Information Policy Robel LK (1989) The myth of the disposable opinion: unpublished opinions and government litigants in the United States Courts of Appeals. Mich Law Rev 87(5):940–962 Ross J, Royan B (1977) Backﬁle conversion today: CIM era or chimera? Program 11(4):156–165 Rotman D (2013) How technology is destroying jobs. MIT Technol Rev 16(4):28–35 Rubin A (2016) Weaving Cognitive into Couture: Watson and Machesa Collaborate for the Met Gala, IBM Blog. (https://www.ibm.com/blogs/think/2016/04/29/watson-and-marchesa). Accessed 07 Sept Sanchez E, Yamanoi T (2006) Fuzzy ontologies for the semantic web. In: Larsen L et al (eds) Flexible query answering systems. Springer, Berlin, pp 691–699 Satoh J (2008) Judicial review in Japan: An overview of the case law and an examination of trends in the Japanese Supreme Court’s Constitutional Oversight. Loyola Los Angeles Law Rev 41:603 Sills A (2016) ROSS and Watson tackle the law. IBM Blog. https://www.ibm.com/blogs/watson/2016/01/ ROSS-and-watson-tackle-the-law. Accessed 07 Sept 2016 Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of Go with deep neural networks and tree search. Nat Int J Wkly Sci 529:484–504 Simonite T (2016) How Google plans to solve artiﬁcial intelligence MIT technology review. https://www. technologyreview.com/s/601139/how-google-plans-to-solve-artiﬁcial-intelligence/. Accessed 07 Sept 2016 Smallwood RF (2014) Information governance: concepts, strategies, and best practices. Wiley, Hoboken Taal A, Le J, Sherer J (2015) A consideration of eDiscovery technologies for internal investigations. In: International conference on global security, safety, and sustainability. Springer, Berlin Tadena N (2016) IBM Watson’s data-crunching capabilities are gaining traction among companies in the marketing space. The Wall Street Journal. http://www.wsj.com/articles/ibm-watsons-data- crunching-gains-traction-with-marketing-ﬁrms-1463569201. Accessed 07 Sept 2016 Tarakeswar K, Kavitha D (2011) Search engines: a study. J Comput Appl (JCA) 4(1):29–33 Tran T, Cimiano P, Rudolph S, Stude R (2007) Ontology-based interpretation of keywords for semantic search. In: Aberer K et al (eds) The semantic web. Springer, Berlin, pp 523–536 van der Werf EDC (2005) AI techniques for the game of Go. UPM, Universitaire Pers Maastricht, The Netherlands Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008) Visser PR, Bench-Capon TJ (1998) A comparison of four ontologies for the design of legal knowledge systems. Artif Intell Law 6(1):27–57 Wang C, Kalyanpur A, Fan J, Boguraev B (2012) Relation extraction and scoring in DEEPQA. IBM J Res Dev 56(3.4):9-1 Wyner A (2012) Carole D. Hafner conceptual organization of case law knowledge bases. In: T Bench- Capon et al (eds) A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law 6-7 Zhang C, Wu D (2008) Concept extraction and clustering for topic digital library construction. In: Proceedings of the 2008 IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology-volume 03 "
<https://plosjournal.deepdyve.com/lp/springer-journals/cognitive-computing-and-proposed-approaches-to-conceptual-organization-7lBCJE7ljC>