abstract: This work investigates legal concepts and their expression in Portuguese, concentrating on the "Order of Attorneys of Brazil" (in Portuguese 'Ordem dos Advogados do Brasil' or OAB) Bar exam and especially on its questions on Ethics, a subset of the multiple questions this national bar exam deals with. Using the corpus formed by a collection of multiple-choice exams, three norms related to the Ethics part of the OAB exam (the law 8906, OAB Ethics Code and OAB's General Regulation of the law 8906) and some language resources (Princeton WordNet and OpenWordNet-PT) and tools (AntConc and Freeling), we began to investigate the concepts missing from our repertory of concepts and words in Portuguese, as expounded in our basic lexical resource, the knowledge base OpenWordNet-PT. We add these concepts to a special glossary and hence obtain a representation of these texts that is mostly "contained" in the lexical knowledge base. These eventually can be used to calculate entailment and contradiction between passages, which would support the answer the questions automatically.
address: FGV/Direito Rio and FGV/EMAp; IBM Research and FGV/EMAp; FGV/EMAp; Nuance Communications
author: Alessandra Cid; Alexandre Rademaker; Bruno Cuconato; Valeria de Paiva
bibliography: references.bib
title: Linguistic Legal Concept Extraction in Portuguese

# Introduction

Becoming a lawyer is a widely varied process around the world. Common to all jurisdictions are requirements of age and competence; some jurisdictions also require documentation of citizenship or immigration status. However, the most varied requirements are those surrounding the obtaining of the license to practice, whether it includes finishing a law degree, passing an exam, or serving an apprenticeship. Basic requirements vary from country to country.

In Brazil, the "Order of Attorneys of Brazil" (in Portuguese 'Ordem dos Advogados do Brasil' or OAB), the Brazilian Bar association, administers a bar examination nationwide three times a year. The exam is divided in two stages – the first consists of 80 multiple choice questions covering several disciplines (e.g. Ethics, Human Rights, Philosophy of Law, Constitutional Law, Administrative Law, Civil Law, etc). The candidate must score at least 40 questions correctly to proceed to the second part of the exam, which requires answering four essay questions and a drafting project (writing a motion, opinion or claim document). Success in the examination allows one to practice in any court or jurisdiction of the country. However only 20% of those who take the exam are successful.

Our working hypothesis is that to answer correctly the multiple choice questions of the OAB Bar examination, candidates need to construct a model of the world, which consists of concepts that they learn when reading the text of the laws and jurisprudence associated with the questions. We would like to develop a computer system capable of mimicking some of these human abilities, using Natural Language Processing (NLP) tools.

In the next section, we describe some NLP tools, including our main lexical resource, the OpenWordNet-PT (abbreviated as OpenWN-PT or simply OWN-PT). Then we discuss extensions to deal with legal domains and the methodology we developed to deal with this kind of specific, very sophisticated domain.

# The OAB Bar Examination

The OAB Bar examination provides a sensible benchmark to evaluate a system attempting to provide question-answering facilities, based on Brazilian laws and regulations. An ideal legal question-answering system would take a question `Q` in natural language and a corpus of all legal documents in a given jurisdiction `LawCorpus`, and would return both a correct answer (easier if using multiple choice) and its legal foundation, i.e., which sections of which norms or laws provide support for the answer and why this is so. As stated this is too broad and too hard: we hope to provide a sample corpus (a subset of `LawCorpus`) with a single detailed law, to see how far we can get the processing to go. We then go on to describe a methodology for continuously improving our processing of the vocabulary of the law.

Previous work on the corpus constructed from multiple choice questions attests to the suitability of the data obtained from the OAB Bar questions. The data from OAB's previous exams and their answer keys were obtained as PDF files from the official source at <http://oab.fgv.br/>, cleaned and prepared for processing with some scripts . In we also described a simple question answering system targeting the exams, based on shallow NLP methods. In we improved the system by incorporating wordnet data to its analysis process, and started doing a very preliminary effort of expanding OWN-PT to the legal domain. However, it is clear from inspection (and previous work) that the Law domain has many concepts and words that are only used within the legal profession. If they are to be used to reason about the Law, these concepts and words need to be added to OWN-PT, our basic lexical resource, described below. This work follows this effort, so that future systems can benefit from a legal lexical semantic database.

# OpenWordNet-PT

The OpenWordnet-PT is an open access wordnet for Portuguese, originally developed by Valeria de Paiva, Alexandre Rademaker and Gerard de Melo as a syntactic projection of Universal WordNet (UWN) . When the project on OWN-PT started there was no open wordnet-like resource for Portuguese.

The process of building the OWN-PT used machine learning to construct relationships between graphs representing information coming from several versions of Wikipedia, as well as from open dictionaries. Starting as a projection at the level of the lemmas in Portuguese and their relationships, the OWN-PT has been constantly improved through linguistically motivated additions, manual and semi-automatic, making use of large corpora. This kind of construction, automatically started, but manually curated and improved, is the hallmark of our work using OWN-PT.

The OWN-PT has been developed since 2010 with the main objective of eventually serving as a lexicon for a proposed NLP system focused on logical reasoning, based on knowledge representations coming from language. The philosophy of OWN-PT is to maintain a close alignment with the original Princeton WordNet (PWN) , but to remove the biggest mistakes created by automated methods, using language tools and skills for this cleaning up task. One positive consequence of the close connection between PWN and OWN-PT is the latter ability to minimize the impact of lexicographical decisions on the separation or grouping of senses in a synset, as these decisions are, for the time being, delegated to PWN, to a large extent. We strive for precision, rather than coverage, as far as OWN-PT is concerned, and precision is surely needed when dealing with the legal domain.

# Analyzing the legal vocabulary

We want to make sure that the basis created with OWN-PT is broad enough to allow us to deal with specific domains such as Law or Geology. Clearly these specific domains have specific vocabulary, both in terms of words that are not part of the everyday vocabulary, but specially in the use of expressions. In Law, there are several expressions that are not, as yet, part of the OWN-PT. Many common nouns are missing. A significant number of these are nominalizations such as *impetração* (a kind of filing), *postulação* (postulation), where the verbs *impetrar* and *postular* (to file, to postulate) are already in the OWN-PT lexicon.

Some of the missing expressions are adjectives, like *fundacional* (foundational) and *constitutivo* (constituent) coming from nouns *fundação* (foundation) and *constituição* (constitution), where sometimes PWN prefers not to list all the morphologically derived expressions. Still others are nouns that are nominalizations of adjectives, like *nulidade* (nullity), derived from *nulo* (null), where, again, morphology could play am important role.

Expressions such as the name of a law, e.g. *Estatuto da Advocacia* or the name of the professional association of lawyers in Brazil, the "Ordem dos Advogados do Brasil" (OAB) are essential synsets that needed to be created. These are expected, since the named Brazilian legal entities are clearly different from the American ones. We need synsets corresponding to the ones for *President of the United States* and *U.S. Congress*, for instance.

There are also other, more general legal expressions, called *multiword expressions* (MWEs), that really describe the field, but are harder to deal with. Some of these are in Latin, such as *habeas corpus* or *data venia*. But most others are simply common Portuguese words, used in fixed expressions, which have more specific meanings. For example the expression *defensor público* could be used for someone who defends the public or someone who defends something in public, but it is mostly used to describe the attorney, appointed by the Estate to defend the interests of poor citizens, who are not able to pay for a lawyer.

Some recent work on these multiword expressions, especially on English noun compounds , makes the point that multiword expressions can be compositional or non-compositional, conventionalized and not conventionalized. In , it is said in the introduction:

> The lack of practical data sets that can be used in the training and evaluation of multiword expression (MWE) related systems is a notorious problem . It is partly due to the heterogeneous nature of MWEs, partly due to their frequency, and partly due to the unclear boundaries between MWEs and regular phrases. These issues have made the compilation of useful MWE data sets challenging, and any effort to create them invaluable.

It is abundantly clear that specific domains like Law require a bigger set of MWEs, both compositional (or not) and conventionalized (or not). Briefly we can say that *semantic non-compositionality* is the property of a compound whose meaning can not be readily interpreted from the meanings of its components. *Conventionalization* refers to the situation where a sequence of words that refer to a particular concept is commonly accepted in such a way that its constituents cannot be easily substituted for near-synonyms, because of some cultural or historical conventions. A large fraction of compounds are to some extent conventionalized, however we are interested in only clear and well-known conventionalizations, which refer to as "marked conventionalization". We assume that non-compositional compounds are by definition conventionalized, hence it only makes sense to consider conventionalization (or not) of compositional compounds.

It is also clear that non-compositional MWEs are easier to spot: for example *má fé* (bad faith) has nothing to do with *fé* (faith) in its most used meaning of 'religious belief'. It simply means "in a deceiving way" and it is not specific to Law, but common currency both in Portuguese and in English, where the synset {`00753240-n`: *for double-dealing, duplicity*} has a gloss which reads *acting in bad faith; deception by pretending to entertain one set of intentions while acting under the influence of another*.

# Experiments

In order to identify relevant legal terms and multiword expressions and to analyze how this legal vocabulary can be incorporated to the OWN-PT, we describe three small experiments with legal texts and the OWN-PT.

In our first experiment, we investigated some of the English terms in the PWN synsets that were classified by the synset[^1] {`08441203-n`: *jurisprudence, law*} in PWN/OWN-PT.[^2] Our hypothesis was that, by translating the English terms that were already classified as legal vocabulary, we would incorporate important legal terms in Portuguese to the OWN-PT. We analyzed some of the terms that were on the topic, verifying the quality of the translations of the terms that were already translated and seeing if we could translate the ones that were not. We reached the conclusion that the synsets related to {`08441203-n`: *jurisprudence, law*} were very specific to American Law and that by adding their translations to the OWN-PT we were not expanding it with relevant words for legal vocabulary in Portuguese. For example, we found several expressions for specific types of laws in English, such as "Gag Law" or "Blue sky law", that are not used in Portuguese. Given this situation, we moved on to our second experiment.

Our second experiment deals with a wholesale construction of a glossary of legal terms extracted from the OAB questions and from the three norms that are the base of the Ethics questions of the OAB exam. These last three norms are: the law 8906 of July of 1994, the 'Código de Ética da OAB' (Ethics Code of the OAB) and the 'Regulamento Geral da OAB' (OAB's General Regulation). We analyzed these documents using AntConc , a corpus analysis toolkit for concordancing and text analysis. Using AntConc, we obtained a list of 6,890 bi-grams and tri-grams on the texts that occur more than 9 times. Since AntConc works over the raw text, without using any linguistic annotation, we had to filter n-grams that were clearly not MWEs. Two annotators filtered the list independently and we combined the results ending up with 430 candidates of MWEs.

Following , instead of deciding which n-grams are true MWEs as opposed to simple collections of words that happen to occur together, we used a simple test to classify each candidate as compositional or non-compositional and conventional or non-conventional. Is the meaning of this expression explained by the meanings of its parts? If not, then we think we have a non-compositional MWE. If the meaning of the expression is compositional, is it a title of an article in the Portuguese Wikipedia?[^3] If yes, we reckon this is sufficient evidence to characterize a conventional MWE. If it is not a Wikipedia title, it may be that Wikipedia should have one such page and is missing it. Therefore, our process is probably an oversimplification that could be improved in the future. Finally, for the top 50 most frequent MWEs, we identified the possible head words from expressions. We assigned the nouns to their respective synset on OWN-PT and the expressions to their hyponym.[^4]

In our third experiment, we investigated the lexical units of the law 8906, one of the norms used in the second experiment. Law 8906 describes the rights and obligations of lawyers and how they can advocate for their clients. Since the Ethics part of the Bar examination is one of the most straightforward sections of the exam, it makes sense for us to make sure that the whole law is processed correctly and that all the required vocabulary is in place, before starting to analyze the law itself and before trying to relate the OAB ethics questions to their answers.

The experiment was carried out using Freeling , an NLP library able to process Brazilian Portuguese. We processed the Law 8906, investigating the results of the tokenization, lemmatization, part-of-speech (PoS) tagging and word sense disambiguation, that is, if they were correctly assigned to OWN-PT senses in the context of the articles of the law. This allowed us to evaluate how Freeling's modules could be adapted to process the law more accurately and enabled us to measure how many words belonging to the legal vocabulary were already on OWN-PT or needed to be added.

Some of Freeling's results after processing the law were expected. Since OWN-PT, just as Princeton WordNet, does not cater for pronouns, determiners or prepositions, it did not have a meaning assignment for these cases. Freeling lemmatization and PoS tagging modules are driven by a dictionary of word forms. The words that are not in Freeling's dictionary must have the lemma and part-of-speech tag guessed which introduce some errors. For example, the Portuguese word *juizado* (court) was not in the dictionary, so its lemmatization was wrongly ascribed as *juizados*. This was evidence that FreeLing's dictionary did not have it and we simply added it. The multiword expressions identified and added to OWN-PT must also be added to the Freeling locutions file, so that tokens that are part of an MWE are joined enabling the word sense disambiguation module to associate the whole expression to an OWN-PT synset. Other bugs are still under investigation, but the results we obtained so far are summarized in Table , where we present basic statistics of Freeling's analysis of Law 8906. To obtain the unique totals we considered the pair (lemma,PoS tag), and we only considered that a word was missing a sense if it was tagged as the right PoS. Law 8906 comprise 87 articles summing up 231 sentences and 10,242 tokens (1,508 unique types/words). Table  shows in the last column that we are still missing some words in OWN-PT.

As a way of completing OWN-PT, one can discover easy synsets that needed completion. For example, the synsets {`01987341-a`: *reserved (marked by self-restraint and reticence; "was habitually reserved in speech, withholding her opinion")*} and *01988324-a*reserved (set aside for the use of a particular person or party), are very easy to complete. The word *reservado* is almost exactly the same as the English one and has the associated adverb *reservadamente* corresponding to the also empty synset {`00441649-r`: *reservedly (with reserve; in a reserved manner)*}. There are plenty of occasions where English and Portuguese conceptualize things differently and these require plenty of effort. Finding the easy synsets, where not only concepts, but even words are almost the same and making sure that these are complete in the lexical base is one of our main goals for OWN-PT.

|            | total | unique | no sense |
|:-----------|------:|-------:|---------:|
| Nouns      |  2629 |    727 |      190 |
| Adjectives |   634 |    234 |       60 |
| Verbs      |  1167 |    330 |       16 |
| Adverbs    |   268 |     77 |       32 |

Analysis of Law 8906 by Freeling

# Conclusions

This preliminary work investigates legal concepts and their expression in Portuguese, concentrating on the OAB Bar exam and on one of the Ethics' laws, that a subset of the exam questions are based on. Using the corpus formed by the collection of multiple-choice questions in the exams, this one specific law and some language resources (PWN and OWN-PT) and NLP tools (AntConc and Freeling) we began to investigate the concepts missing from our repertory of concepts and words in Portuguese, the knowledge base OWN-PT.

This initial work does not require a huge effort of evaluation, since we are mostly discovering word forms and senses that the lexical resource does not have, yet. We can count how many word forms were added to synsets, how many MWEs we had to create, but we have not found baselines to compare our work to, so far.

As maintainers and curators of OWN-PT this kind of work provides a way of dicing up a nice subset of synsets to look up and correct. Most of the work we have done so far on improving OWN-PT has been based on grammatical functions: we had a push to improve the verb lexicon , another push to provide nominalizations and their links , an effort to increase demonyms and gentilics , which was meant to break down the huge class of adjectives into smaller subsets. We have not done as much in terms of topics or semantic domains. We did a preliminary study of Geological Eras , but this was mostly to check the feasibility of merging an external ontology to the subjacent hierarchy of OWN-PT noun synsets. We also did a preliminary assessment of temporal related concepts , using HeidelTime . But this is our first time tackling a sophisticated and specialized field like Law.

We are aware of several difficulties ahead. Firstly the judicial system in Brazil (based on Roman law) is very different from the 'Common Law' system in use in the US and UK, where most of the lexical resources we want to make use of, originate. This difficulty is discussed in , who wanted to use FrameNet to create a corpus encompassing the whole judicial system in Brazil. They say that "This\[their proposed\] corpus is being planned to be representative of the entire legal production of Brazilian courts and Brazilian legislative houses, such as the laws published by the Brazilian Senate and the judicial decisions of the Federal Courts". We are less ambitious, we hope to produce a corpus of laws and regulations that allow us to answer the Ethics questions of our collection of OAB exams, at least to begin with.

Secondly we would like to produce a large glossary of legal terms that could be used for students actively taking the OAB exam, focusing on the multiple choice questions. There are several good juridical dictionaries in Portuguese and in English, but it seems to us that an open-source one, with relations of synonymy and antonymy, mostly based on the past OAB entrance examinations would be useful to students and professors of Law. Thirdly, we reckon that this project, although ambitious, would allow us to push on the direction we really want to work on, that is, the direction of reasoning with the contents of the legal texts. We plan to start using shallow methods, but also want to try our hand at deep logical representations, hybridized together with learning approaches, to try and detect entailment and contradiction between pieces of legal text.

As for future work, we need to complete the glossaries that we started constructing. When the mappings are consistently investigated, we need to establish a process to make sure that newer changes do not undermine the previous work, i.e. we need to establish test suites and regression tests, as described for instance in . We also need to expand and complete our glossary of juridical terms, making sure that they fit in with what is provided by open source juridic dictionaries, such as the Legal Dictionary of 'The Free Dictionary by Farlex'.[^5] Finally we would like also to design and implement our own system for computing "entailment and contradiction detection" between the OAB examination questions and their answers.

[^1]: A wordnet is a lexical database that groups nouns, verbs, adjectives and adverbs into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations.

[^2]: PWN contains many semantic relations between synsets, besides the most well-known `hyponym`, `hypernym`, and `antonym`, we also have the relation `classifiedByTopic` for grouping synsets into domains.

[^3]: We obtained a list of titles of all Portuguese pages from Wikipedia at <https://dumps.wikimedia.org/other/>.

[^4]: The list of MWEs and all data from the experiments will be made available upon acceptance.

[^5]: <https://legal-dictionary.thefreedictionary.com/>


abstract: I briefly review the "decohering histories" or "consistent histories" formulation of quantum theory, due to Griffiths, Omnès, and Gell-Mann and Hartle (and the subject of my graduate work with George Sudarshan). I also sift through the many meanings that have been attached to decohering histories, with an emphasis on the most basic one: Decoherence of appropriate histories is needed to establish that quantum mechanics has the correct classical limit. Then I will describe efforts to find physical mechanisms that do this. Since most work has focused on density matrix versions of decoherence, I'll consider the relation between the two formulations, which historically has not been straightforward. Finally, I'll suggest a line of research that would use recent results by Sudarshan to illuminate this aspect of the classical limit of quantum theory.
address: Theoretical Division, Los Alamos National Laboratory, Los Alamos, NM  87545
author: Eric D. Chisolm
title: Decohering histories and open quantum systems

# Introduction

Many of the most interesting developments in quantum theory since Bell's Theorem have centered on the notion of "decoherence," understood most generally as the absence of interference effects. Decoherence is to be avoided at all costs when constructing mesoscopic or macroscopic quantum devices, such as SQUIDs or quantum computers, while its unavoidability in certain domains is said to be an important part of why large parts of the world appear classical. One area where decoherence has played an important role is the so-called "decohering histories" or "consistent histories" formulation of quantum theory, due to Griffiths , Omnès , and Gell-Mann and Hartle . While it has not been adopted universally as an interpretive framework for quantum mechanics, which is what it was originally proposed for, it is certainly a useful tool for clarifying certain aspects of the classical limit of quantum theory, which remains a challenging problem despite the enormous progress made in the last few decades. In this article I will review the basic notions underlying decohering histories, and I will also spend a little time discussing various interpretations of this formalism. Having settled for application to the classical limit, leaving questions of interpretation aside, I will then compare the decohering histories approach to decoherence as seen in the time evolution of reduced density matrices. Since I argue that demonstrating decoherence of appropriate histories is needed to properly establish the classical limit of quantum theory, connecting these two approaches is of some importance. I will run across a problem which Sudarshan and I discussed at some length near the end of my time with him, without our coming to any good solution. Finally, I will apply recent results due to Sudarshan and coworkers to suggest a potentially very useful resolution of the issue, one that could illuminate not just this area but other aspects of the classical limit of quantum mechanics.

# Histories and decoherence

The formalism of quantum mechanics is related to empirical results by the *Born rule*, which states the following: If a system is in state $|\Psi\rangle$ at time $0$, then the probability at time $t$ that the system will be found in state $|\alpha\rangle$ is given by $$p(\alpha) = \left| \langle \alpha | U(t) | \Psi \rangle \right|^2
\label{pureBorn}$$ where $U(t)$ is the time development operator which evolves the state from time $0$ to time $t$. This expression can be rewritten $$\begin{aligned}
p(\alpha) & = & \Tr[P_\alpha(t) \rho] \nonumber \\
          & = & \Tr[P_\alpha(t) \rho P_\alpha(t)],
\label{genBorn}
\end{aligned}$$ where $\rho = | \Psi \rangle \langle \Psi |$ is the initial density operator of the system, $P_\alpha$ is the projection operator onto state $| \alpha \rangle$, and I have switched to the Heisenberg picture, in which observables evolve in time but states do not. The second line follows from the first because projection operators are idempotent and the trace is cyclic in its arguments. This expression is actually more general than Eq. () because it remains valid when $\rho$ is a general density operator, representing an incoherent statistical mixture of pure states given by projectors.

An obvious generalization of this rule gives the probability not for a single measurement result, but for a series of measurement results at a succession of times: $$p(\alpha_1, \alpha_2, \ldots, \alpha_n) = \Tr[P_{\alpha_n}(t_n) \cdots
              P_{\alpha_2}(t_2) P_{\alpha_1}(t_1) \rho P_{\alpha_1}(t_1)
              P_{\alpha_2}(t_2) \cdots P_{\alpha_n}(t_n)].
\label{manyeventBorn}$$ This rule was given by Wigner , and Aharanov, Bergmann, and Lebowitz discussed both this rule and its time-symmetric generalization , but it is best known now because of its use in describing decohering histories. Before I get to decoherence, though, let me introduce some simplifying notation. Let $\alpha$ denote the sequence $\alpha_1, \alpha_2, \ldots, \alpha_n$, and given a sequence of measurements as above I define the *history* $C_\alpha$ by $$C_\alpha = P_{\alpha_1}(t_1) P_{\alpha_2}(t_2) \cdots P_{\alpha_n}(t_n),$$ in terms of which Eq. () becomes the much tidier result $$p(\alpha) = \Tr(C_\alpha^\dag \rho C_\alpha).
\label{histprob}$$ This expression allows for much more conceptual contact between quantum and classical theory, as a history is the quantum mechanical analog of a trajectory of a classical particle. We routinely speak of such a particle occupying a succession of positions at a series of times, perhaps as it orbits another body, and in standard expositions of quantum theory it is not obvious what the corresponding quantum object is. This immediately suggests that histories may be of use in elucidating the classical limit of quantum mechanics, and that hope will be borne out below.

Since histories are products of projection operators, they inherit some (but not all) of the properties of their factors. Projection operators come in sets that satisfy $$\begin{aligned}
P_\alpha P_\beta & = & P_\alpha \delta_{\alpha \beta} \nonumber \\
\sum P_\alpha & = & \mathbb{I}
\label{projcomplete}
\end{aligned}$$ where $\mathbb{I}$ is the identity operator. The first condition is that the set $\{P_\alpha\}$ is *exclusive*; measurement results corresponding to two different $P_\alpha$ cannot occur simultaneously. \[This restriction is lifted when one replaces projection operators with positive operator valued measures (POVM), but as I'll show in a moment exclusivity is important for defining decoherence, so how one would construct decohering histories using POVM is not clear.\] The second condition guarantees that $\{P_\alpha\}$ is *exhaustive*, so every possible measurement result is included. Given a set of projection operators $\{P_{\alpha_i}\}$ at each of a predetermined set of times $t_1, t_2, \ldots, t_n$, one can construct a set of histories $\{C_\alpha\}$ that satisfy $$\sum C_\alpha = \mathbb{I}, \label{histcomplete}$$ so the histories are also exhaustive. Since a given set of histories defines a complete statistical sample space for a system, it makes sense to ask whether the probabilities associated with the histories by Eq. () satisfy the classical Kolmogorov axioms of probability:

1.  $p(\alpha) \geq 0$ for all $\alpha$.

2.  $\sum_\alpha p(\alpha) = 1$.

3.  $p(\alpha \mbox{\ or } \beta) = p(\alpha) + p(\beta)$ for all $\alpha \neq \beta$.

Axiom $1$ is clearly satisfied by Eq. (), and Axiom $2$ is satisfied thanks to Eqs. () and (), but we're in trouble with Axiom $3$. This is easy to see if we recall that composite measurements in quantum theory are defined by taking the sum of the corresponding projection operators; e.g. if $P_1$ represents the spin of a spin-$1$ system being found up in the $z$ direction, and $P_{-1}$ represents the spin being found down, then $$P = P_1 + P_{-1} \label{addproj}$$ represents the spin being found to be nonzero. If we apply this same logic to histories then the operator representing either history $\alpha$ or history $\beta$ is given by $$C_{\alpha \mbox{\ or } \beta} = C_\alpha + C_\beta.$$ (Notice that this is obviously correct if the two histories have the same events at all times but one, by the remarks that led to Eq. ().) Therefore $$\begin{aligned}
p(\alpha \mbox{\ or } \beta) & = & \Tr(C_{\alpha \mbox{\ or } \beta}^\dag \, \rho \,
                                       C_{\alpha \mbox{\ or } \beta}) \nonumber \\
       & = & \Tr(C_\alpha^\dag \rho C_\alpha) + \Tr(C_\beta^\dag \rho C_\beta) +
             \Tr(C_\alpha^\dag \rho C_\beta) + \Tr(C_\beta^\dag \rho C_\alpha) \nonumber \\
       & = & p(\alpha) + p(\beta) + 2 \mbox{Re}\Tr(C_\alpha^\dag \rho C_\beta),
\end{aligned}$$ and Axiom $3$ holds for all histories if and only if the final term vanishes for any $\alpha \neq \beta$. Now we recognize this term; it is nothing more than a generalization of the *interference* terms we see ubiquitously in quantum theory. In fact, if we were to take a two-slit interferometer as our system and consider two-event histories that consisted of a particle passing through one of the two slits and then striking the screen, this term would be exactly the usual interference term. Thus we have generalized not only the Born rule but the expression for interference, and we have seen that in general it is the presence of interference that prevents quantum mechanical probabilities from satisfying the classical axioms of probability.

All of this inspires some new definitions. Given a complete set of histories, I define the *decoherence functional* between any two of them to be $$D(\alpha, \beta) = \Tr(C_\alpha^\dag \rho C_\beta).$$ Now $D(\alpha, \alpha)$ is the probability that history $\alpha$ occurs (cf. Eq. ()), and if $\alpha \neq \beta$ then the real part of $D(\alpha, \beta)$ is the interference term between histories $\alpha$ and $\beta$. Thus a set of histories obeys the Kolmogorov rules of probability if and only if $$\mbox{Re}D(\alpha, \beta) = p(\alpha) \delta_{\alpha\beta}.
\label{weakdeco}$$ This condition is called *weak decoherence*. In many practical calculations the stronger condition $$D(\alpha, \beta) = p(\alpha) \delta_{\alpha\beta}.
\label{meddeco}$$ is actually achieved; this is called *medium decoherence*. (These names suggest that there ought to be some notion of strong decoherence, and several definitions have been proposed, but none have caught on, so I won't bother describing them.) Exactly what decoherence means physically is the subject of the next section.

# Interpretation

When decoherence of histories was introduced by Griffiths , who called it *consistency* of histories, his goal was to develop a language for speaking meaningfully about the dynamics of a *closed* quantum system, for which the Copenhagen interpretation has nothing to say because a closed system by definition is not being measured. His idea was to allow the mathematical formalism of the theory tell him how to speak meaningfully about its dynamics; he derived Eq. () for the probability of a history, and he concluded that the formalism allows only those sets of histories which obey the Kolmogorov axioms to be assigned probabilities in a meaningful way. Thus he was led to Eq. (), and he argued that if a set of histories of a closed quantum system exhibits weak decoherence, then it is meaningful to say that one of the histories in question actually happens in the system, with probabilities given by Eq. (). Soon after, Omnès arrived at similar expressions but with more of a focus on the logic of quantum theory, and he later integrated them into his own interpretation of quantum mechanics, which he describes as "a consistent and complete reformulation of the Copenhagen interpretation" which lacks Copenhagen's flaws . In a slightly different vein, Gell-Mann and Hartle had their sights set on quantum cosmology, where it had been realized that the Copenhagen interpretation was singularly inappropriate, the universe as a whole being the very model of a system with no outside observers. They have developed a quantum theory of closed systems, in which decoherence (they first applied this term to histories) is the sieve that determines à la Griffiths which histories are candidates for reality and which are not. Various objections have been raised to the use of decohering histories in this fashion, most famously by Dowker and Kent , and less famously by Sudarshan, Jordan, and me at about the same time , which elicited thorough responses from Griffiths that were further elaborated in his extensive development of his program in .

In contrast, in this paper I would like to focus less on interpretation and more on the relevance of decohering histories to the classical limit. Roughly speaking, the classical limit of quantum mechanics consists of two components: Classical statistics and classical dynamics. By classical statistics, I mean that the probabilities associated with histories in classical physics obey the Kolmogorov rules of probability given above; Axiom 3 in particular is an essential part of our classical intuition, from coin tossing to weather prediction. By classical dynamics, I mean that the probabilities assigned by theory are strongly peaked around histories that correspond to solutions of the classical equations of motion; it is this aspect of the classical limit that received the lion's share of attention until about 30 years ago, when decoherence in various forms made its appearance. Both components, however, are equally necessary, which means that decoherence of an appropriately chosen set of histories is a precondition for a quantum system to have a sensible classical limit, and it is in precisely those histories that classical behavior will be seen. Thus even if all we care about is establishing classical behavior for all practical purposes (precisely John Bell's famous FAPP ), decoherence of appropriate histories must be demonstrated. Now I'll describe how this has been attempted.

# Mechanisms of decoherence

By far the most popular option in the literature for achieving decoherence is to separate the quantum system under consideration into a preferred subsystem and the remainder, often called the *environment*. Such a separation corresponds to an expression of the system's Hilbert space $\mathcal{H}$ as $$\mathcal{H} = \mathcal{H^S} \otimes \mathcal{H^E},$$ where $\mathcal{H^S}$ is the Hilbert space of the subsystem and $\mathcal{H^E}$ describes the environment. This decomposition is actually quite general, and it corresponds to the expression of the configuration space of a classical system as the Cartesian product of subspaces carrying different degrees of freedom; for example, a single particle with spin moving in space can be described as the product of two subsystems, namely the spatial and spin degrees of freedom, each with its own Hilbert space. Once this separation is defined, decoherence is achieved by considering only histories composed of projections $P^\mathcal{S}$ onto the subsystem and by allowing the subsystem and environment to interact, which generally greatly accelerates the process of decoherence. This second factor leads to the use of open systems theory for decoherence, as the time evolution of the subsystem is normally far from Hamiltonian even if a Hamiltonian describes the full system.

Some of the most popular model systems for illustrating decoherence, as described by Stamp , can be classified as follows. The first and most famous are harmonic oscillators coupled linearly to baths of similar oscillators, typically in a thermal state with some simplified spectrum of frequencies; the preferred oscillator exhibits decoherence in its position basis due to interactions with the bath. The influence functional technique, due to Feynman and Vernon , was applied to this system by Caldeira and Leggett , and their calculation is still widely cited as an illustration of the efficiency with which environmental interaction produces decoherence. Caldeira and Leggett were at some pains to argue in their work that this model was considerably more generic than it would seem at first sight; the need for such arguments should be clear, given that their intent is to illustrate what should be one of the most widespread phenomena in nature (responsible for the entire classical world, after all). A second set of calculations has focused on finite-dimensional quantum systems coupled to spin baths ; the rationale was that such systems should do a good job of representing the low-energy dynamics of condensed matter systems, in which generally only a few low-lying energy levels should be excited. A third mechanism is "third-party decoherence" , in which correlations between subsystems which suppress interference arise not because of any interaction between them, but due to their mutual interactions with a third subsystem. (The particular example Stamp gives is of a particle passing through a two-slit interferometer which loses spatial interference because its spin interacts with the slits, even though its spin and spatial degrees of freedom are uncoupled.)

These sorts of calculations have two features in common which are worth mentioning. First, almost all of them are performed not on decoherence functionals, but on *reduced density matrices* describing only the preferred subsystem; for example, Caldeira and Leggett actually showed that the reduced density matrix of the preferred oscillator diagonalizes rapidly in the position representation for a variety of initial states, while its diagonal elements remain relatively unchanged. There have been exceptions, such as , but the majority of calculations in the literature have considered not histories but reduced density matrices or related objects. Given that I have argued for the importance of decoherence functionals in establishing classicality, it is worth asking how much light density matrix calculations shed on decohering histories. Second, a large fraction of the calculations have either assumed, or made other simplifying assumptions to guarantee, that the subsystem's evolution is described by a master equation of some sort, often Markovian. When Sudarshan and I discussed this state of affairs, he expressed his concern that such approximations were far more restrictive than these authors perhaps realized, and that they cast some doubt on their claims to have illustrated generic features that give rise to the classical world. These issues are the subject of the next two sections.

# Decoherence functionals vs. reduced density matrices

Particularly since the histories approach to decoherence puts so much emphasis on diagonalizing a functional, it is easy to misunderstand the importance of diagonalizing reduced density matrices in that approach (at least it was easy for me to misunderstand it). Every density matrix, reduced or not, is of course always diagonal in some basis simply because it is self-adjoint. The goal of the reduced density matrix approach is this: Given a full system with initial state $\rho$, reduced initial state $\Tr_\mathcal{E}\rho$, and full time evolution operator $U(t)$, the time evolution of the reduced density matrix $$\Tr_\mathcal{E}\rho \rightarrow \Tr_\mathcal{E}[U(t) \rho U^\dag(t)]
\label{redtimeevolve}$$ sometimes has the property that the result diagonalizes rapidly in a particular basis, far more rapidly than the diagonal elements in that basis undergo change, and it stays diagonal over time. Further, this behavior is largely independent of the initial state $\rho$. The states making up such a basis, possibly with additional properties, are called "pointer states" , and the idea behind the reduced density matrix approach to decoherence is that such bases serve to define a preferred class of observables in which classical behavior emerges in appropriate systems. This is actually a very sensible idea, and in fact it deals directly with the fundamental issue raised by Dowker and Kent , which is that in general decoherence of histories depends far too sensitively on the exact choice of projection operators in the histories and the times at which they are placed. If anything is obvious about classicality, it is that it's robust; one does not have to look at the positions of a planet orbiting the sun at a carefully orchested sequence of times in order to see different alternate trajectories failing to interfere. Thus the obvious question to ask is whether histories composed of projections onto pointer states decohere. To see whether this is so, let me consider a representative decoherence functional $$D(\alpha,\beta) = \Tr\left[P^\mathcal{S}_{\alpha_n}(t_n) \cdots
                  P^\mathcal{S}_{\alpha_1}(t_1) \rho P^\mathcal{S}_{\beta_1}(t_1)
                  \cdots P^\mathcal{S}_{\beta_n}(t_n)\right].$$ Let me also make this expression a bit more explicit by expressing the trace as the composition of two partial traces, over the environment and subsystem, and let me also leave the Heisenberg picture and write the time evolutions explicitly, with the result $$D(\alpha, \beta) = \Tr_\mathcal{S}\Tr_\mathcal{E}\big[
                   P^\mathcal{S}_{\alpha_n} U(t_n,t_{n-1}) \cdots
                   P^\mathcal{S}_{\alpha_1} U(t_1,t_0) \rho
                   U(t_0,t_1) P^\mathcal{S}_{\beta_1} \cdots U(t_{n-1},t_n)
                   P^\mathcal{S}_{\beta_n}\big].
\label{opendecofunc}$$ Now let us suppose that the time evolution in Eq. () does in fact diagonalize the reduced density matrix in some basis, and further that the projections $P^\mathcal{S}_{\alpha_i}$ project onto that very basis. Can we conclude that this functional (or at least its real part) vanishes if any $\alpha_i \neq \beta_i$? Not obviously, for the simple reason that the time evolution is repeatedly interrupted by projections before the partial trace over the environment is taken, and while the partial trace commutes with those projections, it does not commute with the time evolution. This is all obvious, of course, and this problem is discussed in some detail by Kiefer in Chapter 5 of . An important reference in that book is work by Paz and Zurek , whose solution, with notation modified for consistency with other work shown below, is as follows. Suppose there exist operators $M(t_2,t_1)$ such that for any operator $A$ acting on the full system, $$\Tr_\mathcal{E}[U(t_2,t_1) A U(t_1,t_2)] = M(t_2,t_1) \{\Tr_\mathcal{E} A\};
\label{evolveA}$$ in other words, suppose that the time evolution of Eq. () can be expressed as an operator acting on the reduced state only. In that case Eq. () can be written $$D(\alpha,\beta) = \Tr_\mathcal{S}\big[P^\mathcal{S}_{\alpha_n}
                  M(t_n,t_{n-1})\big\{ \cdots P^\mathcal{S}_{\alpha_1}
                  M(t_1,t_0) \{ \Tr_\mathcal{E} \rho \} P^\mathcal{S}_{\beta_1}
                  \cdots \big\} P^\mathcal{S}_{\beta_n}\big].$$ Now it's obvious that if the times $t_i$ are sufficiently well separated for the decoherence mechanism to do its work, the result of each $M(t_i, t_{i-1})$ will be diagonal in the pointer basis and the $\alpha_i \neq \beta_i$ terms will vanish. But is it reasonable to hypothesize such an operator $M$? For some time I struggled with this issue, because the justification offered by Paz and Zurek seemed to rely excessively on special assumptions, such as a Markovian master equation, of the type with which Sudarshan expressed such concern. However, recent work by Sudarshan and coworkers has resolved this issue rather neatly, which brings me to my final topic.

# Decoherence functionals and the theory of open systems

Despite all my concerns with master equations and such, it was shown recently by Jordan, Shaji, and Sudarshan that time evolution of operators in a subsystem of a full quantum system experiencing unitary evolution can always be expressed in a form very similiar to Eq. () with no assumptions whatsoever. Suppressing the time indices for the moment for convenience, the expression is $$M(\Tr_\mathcal{E}A) = L(\Tr_\mathcal{E}A) + K$$ where $$L(\Tr_\mathcal{E}A) = \Tr_\mathcal{E}\left[U \left(\Tr_\mathcal{E}A
                      \otimes \frac{\mathbb{I}^\mathcal{E}}
                      {\dim \mathcal{H^E}} \right) U^\dag\right]
\label{linpart}$$ and $$K = \Tr_\mathcal{E}\left[U\left(A - \Tr_\mathcal{E}A \otimes
    \frac{\mathbb{I}^\mathcal{E}}{\dim \mathcal{H^E}}\right)U^\dag\right],
\label{affpart}$$ where I have reexpressed their results using my own notation. Several comments are in order concerning this form. First, the whole thing may look like simple sleight of hand (after all, this is simply Eq. () with Eq. () added and subtracted), and it would be if this form had no advantages; but Jordan *et al.* prove several significant results about this form and develop useful expressions for calculating with it in , so its apparent simplicity is deceptive. Second, although Sudarshan has always emphasized that the most general time evolution of a density operator can always be cast in linear form, the form shown here is instead affine, and that is the source of some of its charming features. Notice that the linear term depends only on the time evolution operator $U$ and not on any correlations between the subsystem and the environment, while the contribution from the correlations has been isolated in the affine term. Third, this means that in fact $M$ is not a function of only $\Tr_\mathcal{E} A$, but $A$ in its entirety, as of course it must be; the idea here is not to neglect the correlations but to put them in a more tractable form. Finally, since this expression is available generally, it clearly follows that the $M(t_i, t_{i-1})$, while always defined, will not in general form a semigroup (as this would imply a master equation), and forming a group is almost always out of the question, as that would imply Hamiltonian evolution of the subsystem.

Casting $M(t_i,t_{i-1})$ in this form makes one point particularly clear. While it is typically assumed that the time evolution of the preferred subsystem obeys a master equation, is Markovian, or is otherwise special, all of those assumptions are for computational simplicity only; none of them are needed to express decoherence of histories. Stamp forcefully expresses a concern, hinted at previously in this paper, that the work which has been done up to this point on decoherence has not considered sufficiently general situations to convince the general population of physicists that truly generic mechanisms for producing the classical world have in fact been unearthed ; and while I don't wish to endorse Stamp's position fully, I do agree that the model calculations performed up to this point don't quite convey the flavor of a universal phenomenon. This leads me to the fundamental point of this paper: I suggest that a study of decoherence functionals using the formalism of Jordan *et al.* would give a broader and more general picture of decoherence as a phenomenon than the sorts of calculations largely performed up to this point have done. Jordan *et al.* state that the aim of their paper is "to simply describe the Schrödinger picture before making approximations to it" (such as the introduction of master equations), and I suggest that such a perspective would benefit the study of decoherence, which in many ways is so much more mature than it was a decade ago.

This maturity shows itself particularly in the research program of Zurek and coworkers, recently summarized in , in which it is recognized that decoherence is only one of many ingredients that characterize classicality. The larger picture is that the classical world is characterized by particular observables that carry several properties:

-   The corresponding eigenstates are highly resistant to environmental dephasing, while their superpositions are not.

-   Thus they are selected by the details of dynamics, not our arbitrary choices; it is dynamics that tells us what we can potentially observe.

-   These observables imprint their values highly redundantly on the environment, so it is possible to deduce their values by sampling only a small fraction of the environment (knowing the position of an object after sampling only very few of the photons reflected from it is an obvious example).

-   Finally, histories composed of projections onto eigenstates of these observables exhibit decoherence.

Many of these issues have been studied in explicit models in a master equation context, and I suggest that further study using this more general open systems perspective would be of great value.

However, the results of Jordan *et al.* require extension in certain directions to be useful here. The most obvious is that Eqs. () and () require the dimension of the environment's Hilbert space to be finite, as they involve the introduction of a "state of complete ignorance" of the environment, which is defined only in the finite-dimensional case. While this is adequate for spin bath calculations, an infinite-dimensional version is clearly required for other applications. Perhaps this is easily done by replacing the state of complete ignorance with a thermal equilibrium state, but that remains to be demonstrated.

# Conclusions

Establishing that quantum mechanics has the correct classical limit has proved to be a surprisingly challenging problem; some time passed before the full extent of the problem was even clear. Since the recognition that decoherence was an important contributor, however, various research programs have made progress on this issue. Here I have summarized results due to Griffiths, Omnès, and Gell-Mann and Hartle, and I have argued that "decoherence" or "consistency" in their sense is necessary for establishing one aspect of the classical limit, namely the validity of classical statistics. I have also pointed out that decoherence is actually only one ingredient among many for demonstrating this validity in certain regimes. After describing their formalism, I tried to bring it into contact with approaches to decoherence that consider not histories but reduced density matrices of selected subsystems, and I found that the connection seemed to depend on making certain approximations in the quantum theory of open systems, such as the existence of Markovian master equations. I also suggested that this might cast some doubts on claims that truly universal mechanisms explaining the emergence of classicality have been found. Then I described a way to analyze histories based on recent work by Jordan, Shaji, and Sudarshan that seems to justify this connection while avoiding the usual approximations, thus offering the possibility of demonstrating decoherence in a much broader range of situations. I suggest that this work offers a new path forward for performing calculations that can illuminate many different aspects of the classical limit, including but certainly not limited to decoherence of histories, with considerably greater generality.

Clearly I am indebted first and foremost to E. C. George Sudarshan, to whom this paper is dedicated. Knowing my interest in foundational issues in quantum theory, he introduced me to Gell-Mann and Hartle's early articles on decoherence, and our analysis of their work and others' has influenced my understanding of these issues long after I moved on to other topics in my day job. I'd also like to acknowledge Thomas Jordan, with whom Sudarshan and I wrote ; he has written many clear and useful articles on foundational issues in quantum theory, and he continues to do so to this day. I also appreciate very helpful conversations with Wojciech Zurek, who changed my views on some of the issues described here and also pointed me to some important current references. Finally, I should also mention that this work was supported by the U. S. Department of Energy under contract DE-AC52-06NA25396.

# References


abstract: Paragraph Vectors has been recently proposed as an unsupervised method for learning distributed representations for pieces of texts. In their work, the authors showed that the method can learn an embedding of movie review texts which can be leveraged for sentiment analysis. That proof of concept, while encouraging, was rather narrow. Here we consider tasks other than sentiment analysis, provide a more thorough comparison of Paragraph Vectors to other document modelling algorithms such as Latent Dirichlet Allocation, and evaluate performance of the method as we vary the dimensionality of the learned representation. We benchmarked the models on two document similarity data sets, one from Wikipedia, one from arXiv. We observe that the Paragraph Vector method performs significantly better than other methods, and propose a simple improvement to enhance embedding quality. Somewhat surprisingly, we also show that much like word embeddings, vector operations on Paragraph Vectors can perform useful semantic results.
author: Andrew M. Dai
Google
`firstname.lastname@example.com`
Christopher Olah
Google
`email@example.com`
Quoc V. Le
Google
`email@example.com`
bibliography: translate.bib
title: Document Embedding with Paragraph Vectors

# Introduction

Central to many language understanding problems is the question of knowledge representation: How to capture the essential meaning of a document in a machine-understandable format (or "representation"). Despite much work going on in this area, the most established format is perhaps the bag of words (or bag of n-gram) representations . Latent Dirichlet Allocation (LDA)  is another widely adopted representation.

A recent paradigm in machine intelligence is to use a distributed representation for words  and documents . The interesting part is that even though these representations are less human-interpretable than previous representations, they seem to work well in practice. In particular, Le and Mikolov  show that their method, Paragraph Vectors, capture many document semantics in dense vectors and that they can be used in classifying movie reviews or retrieving web pages.

Despite their success, little is known about how well the model works compared to Bag-of-Words or LDA for other unsupervised applications and how sensitive the model is when we change the hyperparameters.

In this paper, we make an attempt to compare Paragraph Vectors with other baselines in two tasks that have significant practical implications. First, we benchmark Paragraph Vectors on the task of Wikipedia browsing: given a Wikipedia article, what are the nearest articles that the audience should browse next. We also test Paragraph Vectors on the task of finding related articles on arXiv. In both of these tasks, we find that Paragraph Vectors allow for finding documents of interest via simple and intuitive vector operations. For example, we can find the Japanese equivalence of "Lady Gaga.".

The goal of the paper is beyond benchmarking: The positive results on Wikipedia and arXiv datasets confirm that having good representations for texts can be powerful when it comes to language understanding. The success in these tasks shows that it is possible to use Paragraph Vectors for local and non-local browsing of large corpora.

We also show a simple yet effective trick to improve Paragraph Vector. In particular, we observe that by jointly training word embeddings, as in the skip gram model, the quality of the paragraph vectors is improved.

# Model

The Paragraph Vector model is first proposed in . The model inserts a memory vector to the standard language model which aims at capturing the topics of the document. The authors named this model "Distributed Memory":

As suggested by the figure above, the paragraph vector is concatenated or averaged with local context word vectors to predict the next word. The prediction task changes the word vectors and the paragraph vector.

The paragraph vector can be further simplified when we use no local context in the prediction task. We can arrive at the following "Distributed Bag of Words" model:

At inference time, the parameters of the classifier and the word vectors are not needed and backpropagation is used to tune the paragraph vectors.

As the distributed bag of words model is more efficient, the experiments in this paper focuses on this implementation of the Paragraph Vector. In the following sections, we will explore the use of Paragraph Vectors in different applications in document understanding.

# Experiments

We conducted experiments with two different publicly available corpora: a corpus from the repository of electronic preprints (arXiv), and a corpus from the online encyclopaedia (Wikipedia).

In each case, all words were lower-cased before the datasets were used. We also jointly trained word embeddings with the paragraph vectors since preliminary experiments showed that this can improve the quality of the paragraph vectors. Preliminary results also showed that training with both unigrams and bigrams does not improve the quality of the final vectors. We present a range of qualitative and quantitative results. We give some examples of nearest neighbours to some Wikipedia articles and arXiv papers as well as a visualisation of the space of Wikipedia articles. We also show some examples of nearest neighbours after performing vector operations.

For the quantitative evaluation, we attempt to measure how well paragraph vectors represent semantic similarity of related articles. We do this by constructing (both automatically and by hand) triplets, where each triplet consists of a pair of items that are close to each other and one item that is unrelated.

For the publicly available corpora we trained paragraph vectors over at least 10 epochs of the data and use a hierarchical softmax constructed as a Huffman tree as the classifier. We use cosine similarity as the metric. We also applied LDA with Gibbs sampling and 500 iterations with varying numbers of topics. For LDA, we set $\alpha$ to 0.1 and used values of $\beta$ between 0.01 and 0.000001. We used the posterior topic proportions for each paper with Hellinger distance to compute the similarity between pairs of documents. For completeness, we also include the results of averaging the word embeddings for each word in a paper and using that as the paragraph vector. Finally, we consider the classical bag of words model where each word is represented as a one-hot vector weighted by TF-IDF and the document is represented by that vector, with comparisons done using cosine similarity.

## Performance of Paragraph Vectors on Wikipedia entries

We extracted the main body text of 4,490,000 Wikipedia articles from the English site. We removed all links and applied a frequency cutoff to obtain a vocabulary of 915,715 words. We trained paragraph vectors on these Wikipedia articles and visualized them in Figure  using t-SNE . The visualization confirms that articles having the same category are grouped together. There is a wide range of sport descriptions on wikipedia, which explains why the sports are less concentrated.

We also qualitatively look at the nearest neighbours of Wikipedia articles and compare Paragraph Vectors and LDA. For example, the nearest neighbours for the Wikipedia article "Machine learning" are shown in Table . We find that overall Paragraph Vectors have better nearest neighbours than LDA.

| **LDA**                       | **Paragraph Vectors**               |
|:------------------------------|:------------------------------------|
| Artificial neural network     | Artificial neural network           |
| Predictive analytics          | Types of artificial neural networks |
| Structured prediction         | Unsupervised learning               |
| **Mathematical geophysics**   | Feature learning                    |
| Supervised learning           | Predictive analytics                |
| Constrained conditional model | Pattern recognition                 |
| Sensitivity analysis          | Statistical classification          |
| **SXML**                      | Structured prediction               |
| Feature scaling               | Training set                        |
| Boosting (machine learning)   | Meta learning (computer science)    |
| Prior probability             | Kernel method                       |
| Curse of dimensionality       | Supervised learning                 |
| **Scientific evidence**       | Generalization error                |
| Online machine learning       | Overfitting                         |
| N-gram                        | Multi-task learning                 |
| Cluster analysis              | Generative model                    |
| Dimensionality reduction      | Computational learning theory       |
| **Functional decomposition**  | Inductive bias                      |
| Bayesian network              | Semi-supervised learning            |

Nearest neighbours to "Machine learning." Bold face texts are articles we found unrelated to "Machine learning." We use Hellinger distance for LDA and cosine distance for Paragraph Vectors as they work the best for each model.

We can perform vector operations on paragraph vectors for local and non-local browsing of Wikipedia. In Table  and Table , we show results of two experiments. The first experiment is to find related articles to "Lady Gaga." The second experiment is to find the Japanese equivalence of "Lady Gaga." This can be achieved by vector operations: $pv$("Lady Gaga") - $wv$("American") + $wv$("Japanese") where $pv$ is paragraph vectors and $wv$ is word vectors. Both sets of results show that Paragraph Vectors can achieve the same kind of analogies like Word Vectors .

```latex
\subfloat[Wikipedia nearest neighbours to ``Lady Gaga'' using Paragraph
  Vectors. All articles are relevant.] {
\label{wiki-nn}
\begin{tabular}{lc}
  \toprule
\multicolumn{1}{c}{\bf Article} &\multicolumn{1}{c}{\bf Cosine}\\
\multicolumn{1}{c}{\bf} &\multicolumn{1}{c}{\bf Similarity}
\\ \midrule
Christina Aguilera &    0.674 \\
Beyonce & 0.645 \\
Madonna (entertainer)   & 0.643 \\
Artpop   & 0.640 \\
Britney Spears  & 0.640 \\
Cyndi Lauper    & 0.632 \\
Rihanna & 0.631 \\
Pink (singer)   & 0.628 \\
Born This Way   & 0.627 \\
The Monster Ball Tour   & 0.620  \\
\bottomrule
\end{tabular}
}
```
```latex
\subfloat[Wikipedia nearest neighbours to ``Lady Gaga'' - ``American''
  + ``Japanese'' using Paragraph Vectors. Note that Ayumi Hamasaki is
  one of the most famous singers, and one of the best selling artists
  in Japan. She also has an album called ``Poker Face'' in 1998. ] {
\label{wiki-vec}
\begin{tabular}{lc}
  \toprule
\multicolumn{1}{c}{\bf Article} &\multicolumn{1}{c}{\bf Cosine}\\
\multicolumn{1}{c}{\bf } &\multicolumn{1}{c}{\bf Similarity}
\\ \midrule
Ayumi Hamasaki  & 0.539 \\
Shoko Nakagawa  & 0.531 \\
Izumi Sakai & 0.512 \\
Urbangarde  & 0.505 \\
Ringo Sheena    & 0.503 \\
Toshiaki Kasuga & 0.492 \\
Chihiro Onitsuka    & 0.487 \\
Namie Amuro & 0.485 \\
Yakuza (video game) & 0.485 \\
Nozomi Sasaki (model)   & 0.485 \\
\bottomrule
\end{tabular}
}
```

To quantitatively compare these methods, we constructed two datasets for triplet evaluation. The first consists of 172 triplets of articles we knew were related because of our domain knowledge. Some examples are: "Deep learning" is closer to "Machine learning" than "Computer network" or "Google" is closer to "Facebook" than "Walmart" etc. Some examples are hard and probably require some deep understanding of the content such as "San Diego" is closer to "Los Angeles" than "San Jose."

The second dataset consists of 19,876 triplets in which two articles are closer because they are listed in the same category by Wikipedia, and the last article is not in the same category, but may be in a sibling category. For example, the articles for "Barack Obama" are closer to "Britney Spears" than "China." These triplets are generated randomly.[^1]

We will benchmark document embedding methods, such as LDA, bag of words, Paragraph Vector, to see how well these models capture the semantic of the documents. The results are reported in Table  and Table . For each of the methods, we also vary the number of embedding dimensions.

|                          |                       |              |
|:-------------------------|:---------------------:|:------------:|
| **Model**                |     **Embedding**     | **Accuracy** |
|                          | **dimensions/topics** |              |
| Paragraph vectors        |         10000         |    93.0%     |
| LDA                      |         5000          |     82%      |
| Averaged word embeddings |         3000          |    84.9%     |
| Bag of words             |                       |    86.0%     |

Performances of different methods on hand-built triplets of Wikipedia articles on the best performing dimensionality.

|                          |                       |              |
|:-------------------------|:---------------------:|:------------:|
| **Model**                |     **Embedding**     | **Accuracy** |
|                          | **dimensions/topics** |              |
| Paragraph vectors        |         10000         |    78.8%     |
| LDA                      |         5000          |    67.7%     |
| Averaged word embeddings |         3000          |     74%      |
| Bag of words             |                       |    78.3%     |

Performances of different methods on dataset with generated Wikipedia triplets on the best performing dimensionality.

From the results in Table  and , it can be seen that paragraph vectors perform better than LDA. We also see a peak in paragraph vector performance at 10,000 dimensions. Both paragraph vectors and averaging word embeddings perform better than the LDA model. For LDA, we found that TF-IDF weighting of words and their inferred topic allocations did not affect the performance. From these results, we can also see that joint training of word vectors improves the final quality of the paragraph vectors.

## Performance of Paragraph Vectors on arXiv articles

We extracted text from the PDF versions of 886,000 full arXiv papers. In each case we only used the latest revision available. We applied a minimum frequency cutoff to the vocabulary so that our final vocabulary was 969,894 words.

We performed experiments to find related articles using Paragraph Vectors. In Table  and Table , we show the nearest neighbours of the original Paragraph Vector paper "Distributed Representations of Sentences and Documents" and the current paper. In Table , we want to find the Bayesian equivalence of the Paragraph Vector paper. This can be achieved by vector operations: $pv$("Distributed Representations of Sentences and Documents") - $wv$("neural") + $wv$("Bayesian") where $pv$ are paragraph vectors and $wv$ are word vectors learned during the training of paragraph vectors. The results suggest that Paragraph Vector works well in these two tasks.

|  |  |
|:---|:--:|
| **Title** | **Cosine** |
|  | **Similarity** |
| Evaluating Neural Word Representations in Tensor-Based Compositional Settings | 0.771 |
| Polyglot: Distributed Word Representations for Multilingual NLP | 0.764 |
| Lexicon Infused Phrase Embeddings for Named Entity Resolution | 0.757 |
| A Convolutional Neural Network for Modelling Sentences | 0.747 |
| Distributed Representations of Words and Phrases and their Compositionality | 0.740 |
| Convolutional Neural Networks for Sentence Classification | 0.735 |
| SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation | 0.735 |
| Exploiting Similarities among Languages for Machine Translation | 0.731 |
| Efficient Estimation of Word Representations in Vector Space | 0.727 |
| Multilingual Distributed Representations without Word Alignment | 0.721 |

arXiv nearest neighbours to "Distributed Representations of Sentences and Documents" using Paragraph Vectors.

|  |  |
|:---|:--:|
| **Title** | **Cosine** |
|  | **Similarity** |
| Distributed Representations of Sentences and Documents | 0.681 |
| Efficient Estimation of Word Representations in Vector Space | 0.680 |
| Thumbs up? Sentiment Classification using Machine Learning Techniques | 0.642 |
| Distributed Representations of Words and Phrases and their Compositionality | 0.624 |
| KNET: A General Framework for Learning Word Embedding using | 0.622 |
|        Morphological Knowledge |  |
| Japanese-Spanish Thesaurus Construction Using English as a Pivot | 0.614 |
| Multilingual Distributed Representations without Word Alignment | 0.614 |
| Catching the Drift: Probabilistic Content Models, with Applications | 0.613 |
|        to Generation and Summarization |  |
| Exploiting Similarities among Languages for Machine Translation | 0.612 |
| A Survey on Information Retrieval, Text Categorization, and Web Crawling | 0.610 |

arXiv nearest neighbours to the current paper using Paragraph Vectors.

|  |  |
|:---|:--:|
| **Title** | **Cosine** |
|  | **Similarity** |
| Content Modeling Using Latent Permutations | 0.629 |
| SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation | 0.611 |
| Probabilistic Topic and Syntax Modeling with Part-of-Speech LDA | 0.579 |
| Evaluating Neural Word Representations in Tensor-Based Compositional Settings | 0.572 |
| Syntactic Topic Models | 0.548 |
| Training Restricted Boltzmann Machines on Word Observations | 0.548 |
| Discrete Component Analysis | 0.547 |
| Resolving Lexical Ambiguity in Tensor Regression Models of Meaning | 0.546 |
| Measuring political sentiment on Twitter: factor-optimal design for | 0.544 |
|        multinomial inverse regression |  |
| Scalable Probabilistic Entity-Topic Modeling | 0.541 |

arXiv nearest neighbours to "Distributed Representations of Sentences and Documents" - "neural" + "Bayesian". I.e., the Bayesian equivalence of the Paragraph Vector paper.

To measure the performance of different models on this task, we picked pairs of papers that had at least one shared subject, the unrelated paper was chosen at random from papers with no shared subjects with the first paper. We produced a dataset of 20,000 triplets by this method.

| **Model**                | **Embedding dimensions/topics** | **Accuracy** |
|:-------------------------|:-------------------------------:|:------------:|
| Paragraph vectors        |               100               |    85.0%     |
| LDA                      |               100               |    85.0%     |
| Averaged word embeddings |               300               |    81.1%     |
| Bag of words             |                                 |    80.4%     |

Performances of different methods at the best dimensionality on the arXiv article triplets.

From the results in Table , it can be seen that paragraph vectors perform on par than the best performing number of topics for LDA. Paragraph Vectors are also less sensitive to differences in embedding size than LDA is to the number of topics. We also see a peak in paragraph vector performance at 100 dimensions. Both models perform better than the vector space model. For LDA, we found that TF-IDF weighting of words and their inferred topic allocations did not affect performance.

# Discussion

We described a new set of results on Paragraph Vectors showing they can effectively be used for measuring semantic similarity between long pieces of texts. Our experiments show that Paragraph Vectors are superior to LDA for measuring semantic similarity on Wikipedia articles across all sizes of Paragraph Vectors. Paragraph Vectors also perform on par with LDA's best performing number of topics on arXiv papers and perform consistently relative to the embedding size. Also surprisingly, vector operations can be performed on them similarly to word vectors. This can provide interesting new techniques for a wide range of applications: local and nonlocal corpus navigation, dataset exploration, book recommendation and reviewer allocation.

[^1]: The datasets are available at <http://cs.stanford.edu/~quocle/triplets-data.tar.gz>