{'fraction_non_alphanumeric': 0.031681987392432664, 'fraction_numerical': 0.012947682788760349, 'mean_word_length': 4.854719039825423, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Neural Methods for Effective, Efficient, and Exposure-Aware Information Retrieval

Neural networks with deep architectures have demonstrated significant performance improvements in computer vision, speech recognition, and natural language processing. The challenges in information retrieval (IR), however, are different from these other application areas. A common form of IR involves ranking of documents -- or short passages -- in response to keyword-based queries. Effective IR systems must deal with query-document vocabulary mismatch problem, by modeling relationships between different query and document terms and how they indicate relevance. Models should also consider lexical matches when the query contains rare terms -- such as a person's name or a product model number -- not seen during training, and to avoid retrieving semantically related but irrelevant results. In many real-life IR tasks, the retrieval involves extremely large collections -- such as the document index of a commercial Web search engine -- containing billions of documents. Efficient IR methods should take advantage of specialized IR data structures, such as inverted index, to efficiently retrieve from large collections. Given an information need, the IR system also mediates how much exposure an information artifact receives by deciding whether it should be displayed, and where it should be positioned, among other results. Exposure-aware IR systems may optimize for additional objectives, besides relevance, such as parity of exposure for retrieved items and content publishers. In this thesis, we present novel neural architectures and methods motivated by the specific needs and challenges of IR tasks.


Impact Statement
The research presented in this thesis was conducted while the author was employed at Bing, a commercial web search engine. Many of the research questions investigated here have consequently been motivated by real world challenges in building large scale retrieval systems. The insights gained from these studies have informed, and continues to influence, the design of retrieval models in both industry and academia.
Duet was the first to demonstrate the usefulness of deep representation learning models for document ranking. Since then, the academic community has continued to build on those early results, culminating in large improvements in retrieval quality from neural methods over traditional IR approaches at the TREC 2019 Deep Learning track. Many of these deep models have subsequently been deployed at the likes of Bing and Google. Similarly, our foray into efficient neural architectures is both of academic interest and critical to increasing the scope of impact of these computation-heavy models.
Search systems do not just exist in laboratory environments but are inherently sociotechnical instruments that mediate what information is accessible and consumed. This presents enormous responsibility on these systems to ensure that retrieved results are representative of the collections being searched, and that the retrieval is performed in a manner fair to both content producers and consumers.
Lack of attention to these facets may lead to serious negative consequences-e.g., the formation of information filter bubbles or visible demographic bias in search results. The capability to directly optimize for expected exposure may be key to addressing some of these concerns.
Finally, to accurately qualify the impact of our research, we must look beyond published empirical and theoretical results for contributions. Science is not an individual endeavour, and therefore any evaluation of research must also encompass the impact of the artifacts produced by said research on its immediate field of study and the academic community around it. A more meaningful evaluation of our contributions, therefore, requires juxtaposition of how the field of neural information retrieval has evolved during the course of this research and have been supported directly by our work. In 2016, when neural IR was still an emerging area, we organized the first workshop focused on this topic at the ACM SIGIR conference.
That year, approximately 8% of the published papers at SIGIR were related to this topic. In contrast, this year at the same conference about 79% of the publications employed neural methods. Our research has directly contributed to this momentum in several ways-including, building standard task definition and benchmarks in the form of MS MARCO and the TREC Deep Learning track, which has been widely adopted as the primary benchmark by the community working on deep learning for search. More recently we have also released a large user behavior dataset, called ORCAS, that may enable training even larger and more sophisticated neural retrieval models. We have organized several workshops and tutorials to bring together researchers whose work cuts across information retrieval, natural language processing, and machine learning domains to build a community around this topic.
Our early survey of neural IR methods also resulted in an instructive manuscript that both summarized important progress and charted out key directions for the field.

Acknowledgements
It takes a proverbial village to teach the skills necessary to do good science and become an independent researcher. So, when I pause and look back at my academic journey thus far, I am filled with gratitude towards every person who professionally and personally supported, mentored, and inspired me on this journey. This thesis is as much a product of the last four years of my own research as it is the fruit of the time and labor that others invested in me on the way.
I am grateful to my supervisor Emine Yilmaz not just for her mentorship and guidance during my PhD but also for her initial encouragement to pursue a doctorate. I remember my apprehension about pursuing a part-time degree while employed full-time. Emine did not just believe that I could succeed in this endeavour, but also helped me find the confidence to pursue it. My PhD journey would not even have started without her support and guidance. I am grateful to her for placing that enormous trust in me and then guiding me with patience and kindness. I thank her for the countless technical discussions and insights, and for all the close collaboration including organizing the TREC Deep Learning track.
A few years before the start of my PhD journey, while I was still a ranking engineer at Bing, I walked into Nick Craswell's office one day and expressed my interest to join his applied science team in Cambridge (UK) to "learn the ropes" for how to do research. The trust Nick placed in me that day when he encouraged me to pursue that dream, altered the course of my professional career. Seven years later, I am still proud to be part of Nick's team and grateful for having grown under his mentorship. Nick has constantly encouraged and supported, and often gone out of his way to do so, my dreams of pursuing a career in research. I co-authored my first book with Nick. We co-organized the first deep learning workshop at SIGIR together. We have collaborated on numerous projects and papers over the years.
His insightful comments and thoughtful feedback have always served as important lessons for me on how to do good science. as a mentor and a collaborator, and hope for continued collaborations in the future. David Hawking deserves a special mention in this thesis. I was fortunate to receive David's mentorship around the time when I was still juggling personal doubts on whether (and how) to do a PhD. His recommendations and guidance had a strong influence on my eventual plans for the doctorate and even how I structure this thesis. David has made no secret of his eagerness for the day when he can finally congratulate me on a successful completion of my PhD. In fact, he has expressed that excitement since the day University College London (UCL) accepted my PhD application. So, not only am I filled with immense gratitude for David's guidance in helping me realize this important professional milestone, but I am also excited about finally celebrating this day with him that has been four years in the making.
Over the years, I have been privileged to collaborate with and receive coaching from many other incredible members of the research community. Susan Dumais is a pillar of the computer science community and I have long considered her both a mentor and a role model. Rich Caruana is another incredible mentor and role model whose passion, be it for science or sailing, is highly contagious. Over the years, I have also benefited from valuable feedback and advice from Peter Bailey  These are uncertain times but also a historic moment when people around the world are coming together in unprecedented numbers fighting to build a kinder and more just world for everyone, and for that I am most grateful.

Introduction
Over the last decade, there have been dramatic improvements in performance in computer vision, speech recognition, and machine translation tasks, witnessed in research and in real-world applications [1][2][3][4][5]. These breakthroughs were largely fuelled by recent advances in neural network models, usually with multiple hidden layers, known as deep architectures combined with the availability of large datasets [6] and cheap compute power for model training. Exciting novel applications, such as conversational agents [7,8], have also emerged, as well as game-playing agents with human-level performance [9,10]. Work has now begun in the information retrieval (IR) community to apply these neural methods, leading to the possibility of advancing the state of the art or even achieving breakthrough performance as in these other fields.
Retrieval of information can take many forms [11]. Users can express their information need in the form of a text query-by typing on a keyboard, by selecting a query suggestion, or by voice recognition-or the query can be in the form of an image, or in some cases the need can be implicit. Retrieval can involve ranking existing pieces of content, such as documents or short-text answers, or composing new responses incorporating retrieved information. Both the information need and the retrieved results may use the same modality (e.g., retrieving text documents in response to keyword queries), or be different (e.g., image search using text queries).
The information within the document text may be semi-structured, and the organization scheme may be shared between groups of documents in the collection-e.g., web pages from the same domain [12]. If the query is ambiguous, retrieval system may consider user history, physical location, temporal changes in information, or other context when ranking results. IR systems may also help users formulate their intent (e.g., via query auto-completion or query suggestion) and can extract succinct summaries of results that take the user's query into account. Neural IR refers to the application of shallow or deep neural networks to these retrieval tasks.
We note that many natural language processing (NLP) tasks exist that are not IR. Machine translation of text from one human language to another is not an IR task, because translating language and searching a corpus to satisfy a user's information need are different. However, translation could be used in an IR system, to enable cross-language retrieval on a multilingual corpus [13]. Inferring attributes of a named entity [14], from text or graph-structured data, is not an IR task in itself.
However, an IR system could use inferred entity attributes to enhance its performance on IR tasks. In general, many NLP tasks do not involve information access and retrieval, so are not IR tasks, but some can still be useful as part of a larger IR system.
In this thesis, we focus on neural methods that employ deep architectures to retrieve and rank documents in response to a query, an important IR task. A search query may typically contain a few terms, while the document length, depending on the scenario, may range from a few terms to hundreds of sentences or more. Neural models for IR use vector representations of text, and usually contain a large number of parameters that need to be tuned. ML models with large set of parameters typically benefit from large quantity of training data [15][16][17][18][19]. Unlike traditional learning to rank (LTR) approaches [20] that train ML models over a set of handcrafted features, recent neural models for IR typically accept the raw text of a query and document as input. Learning suitable representations of text also demands largescale datasets for training [21]. Therefore, unlike classical IR models, these neural approaches tend to be data hungry, with performance that improves with more training data.
In other fields, the design of neural network models has been informed by characteristics of the application and data. For example, the datasets and successful architectures are quite different in visual object recognition, speech recognition, and game playing agents. While IR shares some common attributes with the field of NLP, it also comes with its own set of unique challenges. Effective IR systems must deal with query-document vocabulary mismatch problem, by modeling relationships between different query and document terms and how they indicate relevance. Models should also consider lexical matches when the query contains rare terms-such as a person's name or a product model number-not seen during training, and to avoid retrieving semantically related but irrelevant results. In many real-life IR tasks, the retrieval involves extremely large collections-such as the document index of a commercial Web search engine-containing billions of documents. Efficient IR methods should take advantage of specialized IR data structures, such as inverted index, to efficiently retrieve from large collections. Given an information need, the IR system also mediates how much exposure an information artifact receives by deciding whether it should be displayed, and where it should be positioned, among other results. Exposure-aware IR systems may optimize for additional objectives, besides relevance, such as parity of exposure for retrieved items and content publishers.
In our work, we focus on methods using deep neural networks for document ranking, and to a lesser extent other retrieval tasks. We identify key challenges and principles which motivate our design of novel neural approaches to ranking. We study these proposed methods with respect to retrieval quality, query response time, and exposure disparity. In Section 1.1, we summarize our key contributions. The remainder of this chapter is dedicated to describing the problem formulation. In Section 1.2, we provide an overview of the IR tasks that we use for evaluation. In Section 1.3, we describe a set of common notations that we use in the remainder of this thesis. Finally, we describe relevant IR metrics in Section 1.4.

Contributions
In this thesis, we explore neural network based approaches to IR. This section summarizes the key research contributions of this thesis by chapter. Where appropriate, we also cite the publications that forms the basis of that chapter.
• Chapter 1-3 are based on the book by Mitra and Craswell [22]-and corresponding tutorials [23][24][25][26][27]. The current chapter introduces key IR tasks, evaluation metrics, and mathematical notations that are referenced throughout in this thesis. Chapter 2 presents our motivation for exploring neural IR methods.
Chapter 3 provides a survey of existing literature on neural and traditional non-neural approaches to IR. Key concepts related to IR models and neural representation learning are explained. These three chapters have no novel theoretical or algorithmic contributions, but provides a detailed overview of the field that also serves as the background for the remaining sections.
• Chapter 4 is based on [21,[28][29][30] and emphasizes the importance of incorporating evidence based on both patterns of exact query term matches in the document as well as the similarity between query and document text based on learned latent representations for retrieval. We operationalize this principle by proposing a deep neural network architecture, called Duet, that jointly learns two deep neural networks focused on matching using lexical and latent representations of text, respectively. We benchmark the proposed model on: (i) Bing document ranking task, (ii) TREC Complex Answer Retrieval task, (iii) MS MARCO passage ranking task, and (iv) TREC 2019 Deep Learning track document and passage ranking tasks and demonstrate that estimating relevance by inspecting both lexical and latent matches performs better than considering only one of those aspects for retrieval.
• Chapter 5 is based on [31] and studies neural methods in the context of retrieval from the full collection, instead of just reranking. In particular, we study the impact of incorporating the query term independence (QTI) assumption in neural architectures. We find that incorporating QTI assumption in several deep neural ranking models results in minimal (or no) degradation in ranking effectiveness. However, under the QTI assumption, the learned ranking functions can be combined with specialised IR data structures, e.g., inverted index, for fast and scalable candidate generation in the first stage of retrieval. We benchmark on: (i) MS MARCO passage ranking task and (ii) TREC 2019 Deep Learning track to demonstrate that neural methods can be employed for more effective but also efficient candidate generation.
• Chapter 6 is based on [32] and studies learning to rank with neural networks in the context of stochastic ranking. Due to presentation bias, a static ranked list of results may cause large difference in exposure of items with similar relevance. We present a stochastic ranking framework that can optimize towards exposure targets under different constraints-e.g., individual and group exposure parity. While the original study [32] is a collaborative project focusing on the expected exposure metric, this chapter summarizes our key contributions related to the framework of model optimization for individual and groupwise parity of expected exposure.
• Chapter 7, based on [33,34], looks at the application of neural IR beyond ad hoc retrieval-to the query auto-completion (QAC) task. The ranking task, in case of QAC, poses challenges that are different from those in querydocument or query-passage matching. In this chapter, we study two applications of deep models for QAC: (i) Recommending completions for rare query prefixes and (ii) modeling query reformulations for session context-aware QAC.
• Chapter 8 summarizes findings from our recent efforts on large-scale benchmarking of deep neural IR methods at TREC [35]. The TREC Deep Learning track [35,36] provides a strict blind evaluation for IR methods that take advantage of large supervised training datasets, and have been instrumental in demonstrating the superior retrieval quality for many recent neural methods proposed by the research community.
• Finally, in Chapter 9, we conclude with a discussion on the future of neural IR research. In this chapter, we reflect on the progress we have already made as a field and provide some personal perspectives on the road ahead.

Evaluation tasks
We focus on text retrieval in IR, where the user enters a text query and the system returns a ranked list of search results. Search results may be passages of text or full text documents. The system's goal is to rank the user's preferred search results at the top. This problem is a central one in the IR literature, with well-understood challenges and solutions.
Text retrieval methods for full text documents and for short text passages have application in ad hoc retrieval systems and question answering systems, respectively.
We describe these two tasks in this section.

Ad hoc retrieval
Ranked document retrieval is a classic problem in information retrieval, as in the main task of the Text Retrieval Conference [37], and performed by commercial search engines such as Google, Bing, Baidu, and Yandex. TREC tasks may offer a choice of query length, ranging from a few terms to a few sentences, whereas search engine queries tend to be at the shorter end of the range. In an operational search engine, the retrieval system uses specialized index structures to search potentially billions of documents. The results ranking is presented in a search engine results page (SERP), with each result appearing as a summary and a hyperlink.

Question-answering
Question-answering tasks may range from choosing between multiple choices (typically entities or binary true-or-false decisions) [39][40][41][42] to ranking spans of text or passages [43][44][45][46][47], and may even include synthesizing textual responses by gathering evidence from one or more sources [48,49]. TREC question-answering experiments [43] has participating IR systems retrieve spans of text, rather than documents, in response to questions. IBM's DeepQA [47] system-behind the Watson project that famously demonstrated human-level performance on the American TV quiz show, "Jeopardy!"-also has a primary search phase, whose goal is to find as many potentially answer-bearing passages of text as possible. With respect to the question-answering task, the scope of this thesis is limited to ranking answer containing passages in response to natural language questions or short query texts.
Retrieving short spans of text pose different challenges than ranking docu- Therefore, IR systems that focus on the question-answering task need to model the patterns expected in the answer passage based on the intent of the question.
The focus of this thesis is on ad hoc retrieval, and to a lesser extent on questionanswering. However, neural approaches have shown interesting applications to other existing retrieval scenarios, including query recommendation [50], modelling diversity [51], modelling user click behaviours [52], entity ranking [53,54], knowledge-based IR [55], and even optimizing for multiple IR tasks [56]. In addition, recent trends suggest that advancements in deep neural networks methods are also fuelling emerging IR scenarios such as proactive recommendations [57][58][59], conversational IR [60,61], and multi-modal retrieval [62]. Neural methods may have an even bigger impact on some of these other IR tasks. To demonstrate that neural methods are useful in IR-beyond the document and passage ranking taskswe also present, in this thesis, a brief study on employing deep models for the QAC task in Chapter 7.

Notation
We adopt some common notation for this thesis shown in Table 1. 1. We use lowercase to denote vectors (e.g., ⃗ x) and upper-case for tensors of higher dimensions (e.g., X). The ground truth rel q (d) in Table 1.1 may be based on either manual relevance annotations or be implicitly derived from user behaviour on SERP (e.g., from clicks).

Metrics
A large number of IR studies [63][64][65][66][67][68][69][70] have demonstrated that users of retrieval systems tend to pay attention mostly to top-ranked results. IR metrics, therefore, focus on rank-based comparisons of the retrieved result set R to an ideal ranking of documents, as determined by manual judgments or implicit feedback from user behaviour data. These metrics are typically computed at a rank position, say k, and then averaged over all queries in the test set. Unless otherwise specified, R refers to the top-k results retrieved by the model. Next, we describe a few standard metrics used in IR evaluations.
Precision and recall Precision and recall both compute the fraction of relevant documents retrieved for a query q, but with respect to the total number of documents in the retrieved set R q and the total number of relevant documents in the collection D, respectively. Both metrics assume that the relevance labels are binary.
Mean reciprocal rank (MRR) Mean reciprocal rank [71] is also computed over binary relevance judgments. It is given as the reciprocal rank of the first relevant document averaged over all queries.
Mean average precision (MAP) The average precision [72] for a ranked list of documents R is given by, Where, Precision q,i is the precision computed at rank i for the query q. The average precision metric is generally used when relevance judgments are binary, although variants using graded judgments have also been proposed [73]. The mean of the average precision over all queries gives the MAP score for the whole set.
Normalized discounted cumulative gain (NDCG) There are a few different variants of the discounted cumulative gain (DCG q ) metric [74] which can be used when graded relevance judgments are available for a query q-say, on a five-point scale between zero to four. One incarnation of this metric is as follows.
The ideal DCG (IDCG q ) is computed the same way but by assuming an ideal rank order for the documents up to rank k. The normalized DCG (NDCG q ) is then given by, Normalized cumulative gain (NCG) A metric related to NDCG but suitable for evaluating the quality of retrieval for first stage candidate generation methods is NCG-i.e., NDCG without the position discounting.
NCG has been employed in the literature [35,36,75] to measure how much relevant items are recalled as part of candidate generation without paying attention to the exact order in which the candidates appear in the retrieved set.

Chapter 2 Motivation
We expect a good retrieval system to exhibit certain general attributes. We highlight some of them in this chapter. The design of any neural methods for IR should be informed by these desired properties. We operationalize these intuitions later in Chapters 4-7. In this chapter, we also introduce a general taxonomy of neural approaches for document ranking by categorizing them based on the step of the retrieval process they influence. This discussion on a general taxonomy should serve as a common lens through which we can inspect both existing neural IR approaches as well as the new deep neural ranking models described in the rest of this thesis.

Desiderata of IR models
For any IR system, the relevance of the retrieved items to the input query is of foremost importance. But to evaluate the effectiveness of an IR system in isolation without considering critical dimensions, such as the efficiency of the system or its robustness to collections with different properties, can be tantamount to a theoretical exercise without practical usefulness. An IR system mediates what information its users are exposed to and consume. It is, therefore, also important to quantify and limit any systematic disparity that the retrieval system may inadvertently cause with respect to exposure of information artifacts of similar relevance, or their publishers.
These concerns not only serve as yard sticks for comparing the different neural and non-neural approaches but also guide our model designs. Where appropriate, we connect these motivations to our contributions in this area, some of which form the basis for subsequent chapters in this thesis.

Semantic matching
Most traditional approaches to ad hoc retrieval count repetitions of the query terms in the document text. Exact term matching between query and document text, while simple, serves as a foundation for many IR systems. Different weighting and normalization schemes over these counts leads to a variety of TF-IDF models, such as BM25 [76]. However, by only inspecting the query terms the IR model ignores all the evidence of aboutness from the rest of the document. So, when ranking for the query "Australia" only the occurrences of "Australia" in the document are considered-although the frequency of other terms like "Sydney" or "kangaroo" may be highly informative. In the case of the query "what channel are the seahawks on today", the query term "channel" provides hints to the IR model to pay attention to the occurrences of "ESPN" or "Sky Sports" in the document text-none of which appears in the query itself.
For IR tasks, such as QAC, the lexical similarity between the input (e.g., the query prefix) and candidate items (e.g., the possible completions) is minimal. In such scenarios, understanding the relationship between the query prefix and suffix requires going beyond inspecting lexical overlap.
Semantic understanding, however, goes further than mapping query terms to document terms [77]. A good IR model may consider the terms "hot" and "warm" related, as well as the terms "dog" and "puppy"-but must also distinguish that a user who submits the query "hot dog" is not looking for a "warm puppy" [78].
At the more ambitious end of the spectrum, semantic understanding would involve logical reasoning by the IR system-so for the query "concerts during SIGIR" it associates a specific edition of the conference (the upcoming one) and considers both its location and dates when recommending concerts nearby during the correct week.
These examples motivate that IR models should have some latent representations of intent as expressed by the query and of the different topics in the document text-so that inexact matching can be performed that goes beyond lexical term counting. (a) Distribution of query impressions   [79]. The plots highlight that these quantities follow a Zipfian distribution.

Robustness to rare inputs
Query frequencies in most IR tasks follow a Zipfian distribution [80] (see Fig-ure 2.1). In the publicly available AOL query logs [79], for example, more than 70% of the distinct queries are seen only once in the period of three months from which the queries are sampled. In the same dataset, more than 50% of the distinct documents are clicked only once. A good IR method must be able to retrieve these infrequently searched-for documents and perform reasonably well on queries containing terms that appear extremely rarely, if ever, in its historical logs.
Many IR models that learn latent representations of text from data often naively assume a fixed size vocabulary. These models perform poorly when the query consists of terms rarely (or never) seen during training. Even if the model does not assume a fixed vocabulary, the quality of the latent representations may depend heavily on how often the terms under consideration appear in the training dataset.
Exact matching models, like BM25 [76], on the other hand can precisely retrieve documents containing rare terms.
Semantic understanding in an IR model cannot come at the cost of poor retrieval performance on queries containing rare terms. When dealing with a query such as "pekarovic land company" the IR model will benefit from considering exact matches of the rare term "pekarovic". In practice an IR model may need to effectively trade-off exact and inexact matching for a query term. However, the decision of when to perform exact matching can itself be informed by semantic understanding of the context in which the terms appear in addition to the terms themselves.

Robustness to variable length text
Depending on the task, the IR system may be expected to retrieve documents, passages, or even short sequences consisting of only a few terms. The design of a retrieval model for long documents is likely to share some similarities to a passage or short text retrieval system, but also be different to accommodate distinct challenges associated with retrieving long text. For example, the challenge of vocabulary mismatch, and hence the importance of semantic matching, may be amplified when retrieving shorter text. On the other hand, when matching the query against longer text, it may be more informative to consider the positions of the matches, than in the case of short text matching. When specifically dealing with long text, the model's compute and memory requirements may be significantly higher and require careful design choices for mitigation.
Typical text collections contain documents of varied lengths (see Figure 2.2).
Even when constrained to document retrieval, a good IR system must be able to deal with documents of different lengths without over-retrieving either long or short documents. Relevant documents may also contain irrelevant sections, and the relevant content may either be localized, or spread over multiple sections in the document.
Document length normalization is well-studied in the context of IR models (e.g., pivoted length normalization [81]), and this existing research should inform the design of any new IR models.

Efficiency
Efficiency is one of the salient points of any retrieval system. A typical commercial Web search engine may deal with tens of thousands of queries per second 1retrieving results for each query from an index containing billions of documents.
Search engines typically involve specialised data structures, such as inverted index, and large multi-tier architectures-and the retrieval process generally consists 1 http://www.internetlivestats.com/one-second/#google-band

Parity of exposure
IR systems mediate what information its users are exposed to. Under presentation bias, a static ranking may disproportionately distribute exposure between items of similar relevance. Exposure optimization has, therefore, been proposed as a means of achieving fairness in ranking for individuals [84] or groups defined by sensitive attributes such as gender or race [85]. Stochastic ranking policies that optimize for individual or group parity of exposure in expectation may be more appropriate under these settings.

Sensitivity to context
Retrieval in the wild can leverage many implicit and explicit context information.
The query "weather" may refer to the weather in Seattle or in London depending on where the user is located. An IR model may retrieve different results for the query "decorations" depending on the current season. The query "giants match highlights" may be better disambiguated if the system knows whether the user is a fan of baseball or American football, whether she is located on the East or the West coast of USA, or if the model has knowledge of recent sport fixtures. In conversational IR systems, the correct response to the question "When did she become the prime minister?" would depend on disambiguating the correct entity based on the context of references made in the previous turns of the conversation. In proactive retrieval scenarios [86][87][88][89], the retrieval can even be triggered based solely on implicit context without any explicit query submission from the user. Relevance in many applications is, therefore, situated in the user and task context, and is an important consideration in the design of IR systems.

Robustness to corpus variance
An interesting consideration for IR models is how well they perform on corpora whose distributions are different from the data that the model was trained on. While this list of desired attributes of an IR model is in no way complete, it serves as a reference for comparing many of the neural and non-neural approaches described in the rest of this thesis.

Designing neural models for IR
In the previous section, we discuss several important desiderata of IR models. These expectations inform the design of neural architectures described in this thesis.
Machine learning models-including neural networks-are employed for learning to rank [20] in IR, as we discuss in Section 3. 4

Traditional IR models
In this section, we introduce a few of the traditional IR approaches. The decades of insights from these IR models not only inform the design of our new neural based approaches, but these models also serve as important baselines for comparison. They also highlight the various desiderata that we expect the neural IR models to incorporate.

BM25
There is a broad family of statistical functions in IR that consider the number of occurrences of each query term in the document-i.e., term-frequency (TF)-and the corresponding inverse document frequency (IDF) of the same terms in the full collection (as an indicator of the informativeness of the term). One theoretical basis for such formulations is the probabilistic model of IR that yielded the BM25 [76] ranking function.
Where, avgdl is the average length of documents in the collection D, and k 1 and b are parameters that are usually tuned on a validation dataset. In practice, k 1 is sometimes set to some default value in the range [1.2, 2.0] and b as 0. 75. The id f (t) is computed as, BM25 aggregates the contributions from individual terms but ignores any phrasal or proximity signals between the occurrences of the different query terms in the document. A variant of BM25 [98,99] also considers documents as composed of several fields (such as, title, body, and anchor texts).
p(E ) is the maximum likelihood estimate (MLE) of the probability of event E , and p(q|d) indicates the probability of generating query q by randomly sampling terms from document d. In its simplest form, we can estimate p(t q |d) by, However, most formulations of language modelling based retrieval typically employ some form of smoothing [102] by sampling terms from both the document d and the full collection D. The two common smoothing methods are: Jelinek-Mercer smoothing [103] Both TF-IDF and language modelling based approaches estimate document relevance based on the count of only the query terms in the document. The position of these occurrences and the relationship with other terms in the document are ignored.
Translation models Berger and Lafferty [105] proposed an alternative method to estimate p(t q |d) in the language modelling based IR approach (Equation 3.6), by assuming that the query q is being generated via a "translation" process from the document d.
The p(t q |t d ) component allows the model to garner evidence of relevance from nonquery terms in the document. Berger and Lafferty [105] propose to estimate p(t q |t d ) from query-document paired data similar to techniques in statistical machine translation [106,107]-but other approaches for estimation have also been explored [108].
Dependence model None of the three IR models described so far consider proximity between query terms. To address this, Metzler and Croft [109] proposed a linear model over proximity-based features. (3.11) Where, ow(q) and uw(q) are the set of all contiguous n-grams (or phrases) and the set of all bags of terms that can be generated from query q. t f #1 and t f #uwN are the ordered-window and unordered-window operators from Indri [110]. Finally, λ ow and λ uw are the tuneable parameters of the model.
Pseudo relevance feedback (PRF) PRF-based methods-e.g., Relevance Models (RM) [111,112]-typically demonstrate strong performance at the cost of executing an additional round of retrieval. The set of ranked documents R 1 from the first round of retrieval is used to select expansion terms to augment the query which is used to retrieve a new ranked set of documents R 2 that is presented to the user.
The underlying approach to scoring a document in RM is by computing the KL divergence [113] between the query language model θ q and the document language  But under the RM3 [114] formulation the new query language modelθ q is estimated by, Besides language models, PRF based query expansion has also been explored in the context of other retrieval approaches (e.g., [115,116]  vance. All existing neural approaches to IR can be broadly categorized based on whether they influence the query representation, the document representation, or in estimating relevance. A neural approach may impact one or more of these stages shown in Figure 3. 1. Neural networks are useful as learning to rank models as we will discuss in Section 3. 4. In these models, a joint representation of query and document is generated using manually designed features and the neural network is used only at the point of match to estimate relevance, as shown in Figure 3.2a. In Section 3.6.4, we will discuss DNN models, such as [21,117], that estimate relevance based on patterns of exact query term matches in the document. Unlike traditional learning to rank models, however, these architectures (shown in Figure 3.2b) depend less on manual feature engineering and more on automatically detecting regularities in good matching patterns. More recent deep learning methods, such as [121], consume query and document as single concatenated sequence of terms, instead of representing them as separate term vectors.
In contrast, many (shallow and deep) neural IR models depend on learning useful low-dimensional vector representations-or embeddings-of query and document text, and using them within traditional IR models or in conjunction with simple similarity metrics (e.g., cosine similarity). These models shown in Figure 3.2c may learn the embeddings by optimizing directly for the IR task (e.g., [118]), or in an unsupervised setting (e.g., [119]). Finally, Figure 3.2d shows IR approaches where the neural models are used for query expansion [91,120].
While the taxonomy of neural approaches described in this section is rather simple, it does provide an intuitive framework for comparing the different neural approaches in IR and highlights the similarities and distinctions between these different techniques.  Under local representations the terms "banana", "mango", and "dog" are distinct items. But distributed vector representations may recognize that "banana" and "mango" are both fruits, but "dog" is different.

A tale of two representations
Vector representations are fundamental to both information retrieval and machine learning. In IR, terms are typically the smallest unit of representation for indexing and retrieval. Therefore, many IR models-both non-neural and neural-focus is given by a vector that has the value one in the position corresponding to "banana" and zero everywhere else. Similarly, the terms "mango" and "dog" are represented by setting different positions in the vector to one. Figure 3.3a highlights that under this scheme each term is a unique entity, and "banana" is as distinct from "dog" as  to the term. The representation scheme in (a) depends on the documents containing the term while the scheme shown in (b) and (c) depends on other terms that appears in its neighbourhood. The scheme (b) ignores inter-term distances. Therefore, in the sentence "Time flies like an arrow; fruit flies like a banana", the feature "fruit" describes both the terms "banana" and "arrow". However, in the representation scheme of (c) the feature "fruit −4 " is positive for "banana", and the feature "fruit +1 " for "arrow".
it is from "mango". Terms outside of the vocabulary either have no representation or are denoted by a special "UNK" symbol under this scheme.
Distributed representations Under distributed representations every term is represented by a vector ⃗ v ∈ R |k| . ⃗ v can be a sparse or a dense vector-a vector of hand-crafted features or a latent representation in which the individual dimensions are not interpretable in isolation. The key underlying hypothesis for any distributed representation scheme, however, is that by representing a term by its attributes allows for defining some notion of similarity between the different terms based on the chosen properties. For example, in Figure 3.3b "banana" is more similar to "mango" than "dog" because they are both fruits, but yet different because of other properties that are not shared between the two, such as shape.
A key consideration in any feature based distributed representation is the choice of the features themselves. One approach involves representing terms by features that capture their distributional properties. This is motivated by the distributional hypothesis [122] that states that terms that are used (or occur) in similar context tend to be semantically similar. Firth [123] famously purported this idea of distributional semantics 1 by stating "a word is characterized by the company it keeps". However, the distribution of different types of context may model different semantics of a term. Figure 3.4 shows three different sparse vector representations of the term "banana" corresponding to different distributional feature spaces-documents containing the term (e.g., LSA [124]), neighbouring terms in a window (e.g., HAL [125], COALS [126], and [127]), and neighbouring terms with distance (e.g., [128]). Finally, Figure 3.4d shows a vector representation of "banana" based on the character trigraphs in the term itself-instead of external contexts in which the term occurs. In Section 3.2.2 we will discuss how choosing different distributional features for term representation leads to different nuanced notions of semantic similarity between them. When the vectors are high-dimensional, sparse, and based on observable features we refer to them as observed (or explicit) vector representations [128]. When the vectors are dense, small (k ≪ |T |), and learnt from data then we instead refer to them as latent vector spaces, or embeddings. In both observed and latent vector spaces, several distance metrics can be used to define the similarity between terms, although cosine similarity is commonly used.
Most embeddings are learnt from observed features, and hence the discussions in In the context of neural models, distributed representations generally 1 Readers should take note that while many distributed representations take advantage of distributional properties, the two concepts are not synonymous. A term can have a distributed representation based on non-distributional features-e.g., parts of speech classification and character trigraphs in the term.

Notions of similarity
Any vector representation inherently defines some notion of relatedness between terms. Is "Seattle" closer to "Sydney" or to "Seahawks"? The answer depends on the type of relationship we are interested in. If we want terms of similar type to be closer, then "Sydney" is more similar to "Seattle" because they are both cities.
However, if we are interested to find terms that co-occur in the same document or passage, then "Seahawks"-Seattle's football team-should be closer. The former represents a typical, or type-based notion of similarity while the latter exhibits a If we want to compare "Seattle" with "Sydney" and "Seahawks based on their respective vector representations, then the underlying feature space needs to align with the notion of similarity that we are interested in. It is, therefore, important for the readers to build an intuition about the choice of features and the notion of similarity they encompass. This can be demonstrated by using a toy corpus, such as the one in Table 3. 1. Figure 3.6a shows that the "in documents" features naturally lend to a topical sense of similarity between the terms, while the "neighbouring terms with distances" features in Figure 3.6c gives rise to a more typical notion of relatedness. Using "neighbouring terms" without the inter-term distances as features, however, produces a mixture of topical and typical relationships. This is because when the term distances (denoted as superscripts) are considered in the feature definition then the document "Seattle Seahawks Wilson" produces the bag-of- and "Seahawks", respectively. The overlap increases when a larger window-size over the neighbouring terms is employed pushing the notion of similarity closer to a topical definition. This effect of the windows size on the latent vector space was reported by Levy and Goldberg [130] in the context of term embeddings.
Readers should note that the set of all inter-term relationships goes beyond the two notions of typical and topical that we discuss in this section. For example, vector representations could cluster terms closer based on linguistic styles-e. g

Observed feature spaces
Observed feature space representations can be broadly categorized based on their choice of distributional features (e.g., in documents, neighbouring terms with or without distances, etc.) and different weighting schemes (e.g., TF-IDF, positive pointwise mutual information, etc.) applied over the raw counts. We direct the readers to [140,141] which are good surveys of many existing observed vector representation schemes.
Levy et al. [128] demonstrated that explicit vector representations are amenable to the term analogy task using simple vector operations. A term analogy task involves answering questions of the form "man is to woman as king is to ____?"-the correct answer to which in this case happens to be "queen". In NLP, term analogies are typically performed by simple vector operations of the following form followed by a nearest-neighbour search,  The figure shows different distributed representations for the four terms-"Seattle", "Seahawks", "Denver", and "Broncos"-based on the toy corpus in Table 3. 1. Shaded circles indicate non-zero values in the vectors-the darker shade highlights the vector dimensions where more than one vector has a nonzero value. When the representation is based on the documents that the terms occur in then "Seattle" is more similar to "Seahawks" than to "Denver". The representation scheme in (a) is, therefore, more aligned with a topical notion of similarity. In contrast, in (c) each term is represented by a vector of neighbouring terms-where the distances between the terms are taken into consideration-which puts "Seattle" closer to "Denver" demonstrating a typical, or type-based, similarity. When the inter-term distances are ignored, as in (b), a mix of typical and topical similarities is observed. Finally, it is worth noting that neighbouring-terms based vector representations leads to similarities between terms that do not necessarily occur in the same document, and hence the term-term relationships are less sparse than when only in-document features are considered. We present a visual intuition of why this works in practice in Figure 3.7, but we refer the readers to [128,142] for a more rigorous mathematical handling of this subject.

Embeddings
While observed vector spaces based on distributional features can capture interesting relationships between terms, they have one big drawback-the resultant representations are highly sparse and high-dimensional. The number of dimensions, for example, may be the same as the vocabulary size, which is unwieldy for most practical tasks. An alternative is to learn lower dimensional representations that retains useful attributes from the observed feature spaces.
An embedding is a representation of items in a new space such that the properties of-and the relationships between-the items are preserved. Goodfellow et al. [143] articulate that the goal of an embedding is to generate a simpler representation-where simplification may mean a reduction in the number of dimensions, a decrease in the sparseness of the representation, disentangling the principle components of the vector space, or a combination of these goals. In the context of term embeddings, the explicit feature vectors-like those discussed in Common approaches for learning embeddings include either factorizing the term-feature matrix (e.g.LSA [124]) or using gradient descent based methods that try to predict the features given the term (e.g., [144,145]). Baroni et al. [146] empirically demonstrate that these feature-predicting models that learn lower dimensional representations, in fact, also perform better than explicit counting based models on different tasks-possibly due to better generalization across terms-although some counter evidence the claim of better performances from embedding models have also been reported in the literature [147].
The sparse feature spaces of Section 3.2.3 are easier to visualize and leads to more intuitive explanations-while their latent counterparts may be more practically useful. Therefore, it may be useful to think sparse, but act dense in many scenarios. In the rest of this section, we will describe some of these neural and non-neural latent space models.
Latent Semantic Analysis (LSA) LSA [124] involves performing singular value decomposition (SVD) [148] on a term-document (or term-passage) matrix X to obtain its low-rank approximation [149]. SVD on X involves solving X = UΣV T , where U and V are orthogonal matrices and Σ is a diagonal matrix. 3 . . , σ l , ⃗ u 1 , . . . ,⃗ u l , and ⃗ v 1 , . . . ,⃗ v l are the singular values, and the left and the right singular vectors, respectively. The k largest singular values-and corresponding singular vectors from U and V -is the rank k approximation of X (X k = U k Σ k V T k ) and Σ k ⃗ t i is the embedding for the i th term.
While LSA operate on a term-document matrix, matrix factorization based approaches can also be applied to term-term matrices [126,150,151]. The training data may contain many instances of the same term-feature pair proportional to their frequency in the corpus (e.g., word2vec [145]), or their counts can be pre-aggregated (e.g., GloVe [156]). The architecture is a neural network with a single hidden layer whose size is much smaller than that of the input and the output layers. Both models use one-hot representations of terms in the input and the output. The learnable parameters of the model comprise of the two weight matrices W in and W out that corresponds to the embeddings the model learns for the input and the output terms, respectively. The skip-gram model trains by minimizing the error in predicting a term given one of its neighbours. The CBOW model, in contrast, predicts a term from a bag of its neighbouring terms.

W in W out
Word2vec For word2vec [145,[157][158][159][160], the features for a term are made up of its neighbours within a fixed size window over the text. The skip-gram architecture (see Figure 3.8a) is a simple one hidden layer neural network. Both the input and the output of the model are one-hot vectors and the loss function is as follows, where, S is the set of all windows over the training text and c is the number of neighbours we want to predict on either side of the term t i . The denominator for the softmax function for computing p(t i+ j |t i ) sums over all the terms in the vocabulary. This is prohibitively costly and in practice either hierarchical-softmax [161] or negative sampling is employed, which we discuss more in Section 3. 4 The continuous bag-of-words (CBOW) architecture (see Figure 3.8b) is similar to the skip-gram model, except that the task is to predict the middle term given all the neighbouring terms in the window. The CBOW model creates a single training sample with the sum of the one-hot vectors of the neighbouring terms as input and the one-hot vector ⃗ v t i -corresponding to the middle term-as the expected output.
Contrast this with the skip-gram model that creates 2 × c samples by individually pairing each neighbouring term with the middle term. During training, the skipgram model trains slower than the CBOW model [145] because it creates more training samples from the same windows of text.
Word2vec gained particular popularity for its ability to perform term analogies using simple vector algebra, similar to what we discussed in Section 3. 2

.3. For domains
where the interpretability of the embeddings is important, Sun et al. [162] introduced an additional constraint in the loss function to encourage more sparseness in the learnt representations.
GloVe The skip-gram model trains on individual term-neighbour pairs. If we aggregate all the training samples such that x i j is the frequency of the pair ⟨t i ,t j ⟩ in the training data, then the loss function changes to, is the cross-entropy error between the actual co-occurrence probabilitȳ p(t j |t i ) and the one predicted by the model p(t j |t i ). This is similar to the loss function for GloVe [156] if we replace the cross-entropy error with a squared-error and apply a saturation function f (. . . ) over the actual co-occurrence frequencies. where, GloVe is trained using AdaGrad [163]. Similar to word2vec, GloVe also generates two different (IN and OUT) embeddings, but unlike word2vec it generally uses the sum of the IN and the OUT vectors as the embedding for each term in the vocabulary.

Term embeddings for IR
Traditional IR models use local representations of terms for query-document matching. The most straight-forward use case for term embeddings in IR is to enable The paragraph2vec architecture as proposed by Le and Mikolov [164] trains by predicting a term given a document (or passage) ID containing the term. By trying to minimize the prediction error, the model learns an embedding for the term as well as for the document. In some variants of the architecture, optionally the neighbouring terms are also provided as input-as shown in the dotted box.
inexact matching in the embedding space. In Section 2.1, we argued the importance of inspecting non-query terms in the document for garnering evidence of relevance.
For example, even from a shallow manual inspection, it is possible to conclude that the passage in Figure 3.10a is about Albuquerque because it contains "metropolitan", "population", and "area" among other informative terms. On the other hand, the passage in Figure 3.10b contains "simulator", "interpreter", and "Altair" which suggest that the passage is instead more likely related to computers and technology.
In traditional term counting based IR approaches these signals are often ignored.
Unsupervised term embeddings can be incorporated into existing IR approaches for inexact matching. These approaches can be broadly categorized as those that compare the query with the document directly in the embedding space; Albuquerque is the most populous city in the U.S. state of New Mexico. The high-altitude city serves as the county seat of Bernalillo County, and it is situated in the central part of the state, straddling the Rio Grande. The city population is 557,169 as of the July 1, 2014 population estimate from the United States Census Bureau, and ranks as the 32nd-largest city in the U.S. The Albuquerque metropolitan statistical area (or MSA) has a population of 907,301 according to the United States Census Bureau's most recently available estimate for 2015.
(a) About Albuquerque Allen suggested that they could program a BASIC interpreter for the device; after a call from Gates claiming to have a working interpreter, MITS requested a demonstration. Since they didn't actually have one, Allen worked on a simulator for the Altair while Gates developed the interpreter. Although they developed the interpreter on a simulator and not the actual device, the interpreter worked flawlessly when they demonstrated the interpreter to MITS in Albuquerque, New Mexico in March 1975; MITS agreed to distribute it, marketing it as Altair BASIC.
(b) Not about Albuquerque Figure 3.10: Two passages both containing exactly a single occurrence of the query term "Albuquerque". However, the passage in (a) contains other terms such as "population" and "area" that are relevant to a description of the city. In contrast, the terms in passage (b) suggest that it is unlikely to be about the city, and only mentions the city potentially in a different context. and those that use embeddings to generate suitable query expansion candidates from a global vocabulary and then perform retrieval based on the expanded query. We discuss both these classes of approaches in the remainder of this section.

Query-document matching
One strategy for using term embeddings in IR involves deriving a dense vector representation for the query and the document from the embeddings of the individual terms in the corresponding texts. The term embeddings can be aggregated in different ways, although using the average word (or term) embeddings (AWE) is quite common [119,164,[170][171][172][173][174]. Non-linear combinations of term vectorssuch as using Fisher Kernel Framework [175]-have also been explored, as well as other families of aggregate functions of which AWE has been shown to be a special case [176].
The query and the document embeddings themselves can be compared using a variety of similarity metrics, such as cosine similarity or dot-product. For example, An important consideration here is the choice of the term embeddings that is appropriate for the retrieval scenario. While, LSA [124], word2vec [157], and GloVe [156] are commonly used-it is important to understand how the notion of inter-term similarity modelled by a specific vector space may influence its performance on a retrieval task. In the example in Figure 3.10, we want to rank documents that contains related terms-such as "population" or "area"-higher. These terms are topically similar to the query term "Albuquerque". Intuitively, a document about "Tucson"-which is typically similar to "Albuquerque"-is unlikely to satisfy the user intent. The discussion in Section 3.2.2 on how input features influence the notion of similarity in the learnt vector space is relevant here.
Models, such as LSA [124] and Paragraph2vec [164], that consider termdocument pairs generally capture topical similarities in the learnt vector space.
On the other hand, word2vec [157] and GloVe [156] embeddings may incorporate a mixture of topical and typical notions of relatedness. The inter-term relationships modelled in these latent spaces may be closer to type-based similarities when trained with short window sizes or on short text, such as on keyword queries [119,130].
In Section 3.2.4, we note that the word2vec model learns two different embeddings-IN and OUT-corresponding to the input and the output terms. In retrieval, if a query contains a term t i then-in addition to the frequency of occurrences of t i in the document-we may also consider the presence of a different term t j in the document to be a supporting evidence of relevance if the pair of terms ⟨t i ,t j ⟩ frequently co-occurs in the collection. As shown in Equation 3.19,  [119,172] where the word2vec embeddings are trained on search queries. Training on short query text, however, makes the inter-term similarity more pronouncedly typical (where, "Yale" is closer to "Harvard" and "NYU") when both terms are represented using their IN vectors. In contrast, the IN-OUT similarity (where, "Yale" is closer to "faculty" and "alumni") mirrors more the topical notions of relatedness. in the skip-gram model this probability of co-occurrence p(t j |t i ) is proportional to , the dot product between the IN embeddings of t i and the OUT embeddings of t j . Therefore, Nalisnick et al. [172] point out that when using word2vec embeddings for estimating the relevance of a document to a query, it is more appropriate to compute the IN-OUT similarity between the query and the document terms. In other words, the query terms should be represented using the IN embeddings and the document terms using the OUT embeddings.
An alternative to representing queries and documents as an aggregate of their term embeddings is to incorporate the term representations into existing IR models, such as the ones we discussed in Section 3.1. Zuccon et al. [108] proposed the Neural Translation Language Model (NTLM) that uses the similarity between term embeddings as a measure for term-term translation probability p(t q |t d ) in Equation 3. 11.
On similar lines, Ganguly et al. [177] proposed the Generalized Language Model (GLM) which extends the Language Model based approach in Equation 3.9 to, Where, N t is the set of nearest-neighbours of term t. Ai et al. [168] incorporate paragraph vectors [164] into the query-likelihood model [100].
Another approach, based on the Earth Mover's Distance (EMD) [178], involves estimating similarity between pairs of documents by computing the minimum distance in the embedding space that each term in the first document needs to travel to reach the terms in the second document. This measure, commonly referred to as the Word Mover's Distance (WMD), was originally proposed by Wan et al. [179,180], but used WordNet and topic categories instead of embeddings for defining the distance between terms. Term embeddings were later incorporated into the model by Kusner et al. [181,182]. Finally, Guo et al. [183] incorporated similar notion of distance into the Non-linear Word Transportation (NWT) model that estimates relevance between a a query and a document. The NWT model involves solving the following constrained optimization problem, is the set of all unique terms in document d, and b is a constant.
Another term-alignment based distance metric was proposed by Kenter and de Rijke [184] for computing short-text similarity. The design of the saliencyweighted semantic network (SWSN) is motivated by the BM25 [76] formulation.
Here s s is the shorter of the two sentences to be compared, and s l the longer sentence. However, the model is easily duped by a non-relevant passage that has been artificially injected with the term "Cambridge" (Figure 3.11c). The embedding space based matching, on the other hand, can spot that the other terms in the passage pro-the city of cambridge is a university city and the county town of cambridgeshire , england . it lies in east anglia , on the river cam , about 50 miles ( 80 km ) north of london . according to the united kingdom census 2011 , its population was 123867 ( including 24488 students ) . this makes cambridge the second largest city in cambridgeshire after peterborough , and the 54th largest in the united kingdom . there is archaeological evidence of settlement in the area during the bronze age and roman times ; under viking rule cambridge became an important trading centre . the first town charters were granted in the 12th century , although city status was not conferred until 1951 .
(a) Passage about the city of Cambridge oxford is a city in the south east region of england and the county town of oxfordshire . with a population of 159994 it is the 52nd largest city in the united kingdom , and one of the fastest growing and most ethnically diverse . oxford has a broad economic base . its industries include motor manufacturing , education , publishing and a large number of information technology and sciencebased businesses , some being academic offshoots . the city is known worldwide as the home of the university of oxford , the oldest university in the englishspeaking world . buildings in oxford demonstrate examples of every english architectural period since the arrival of the saxons , including the mid18thcentury radcliffe camera . oxford is known as the city of dreaming spires , a term coined by poet matthew arnold .
(b) Passage about the city of Oxford the cambridge ( giraffa camelopardalis ) is an african eventoed ungulate mammal , the tallest living terrestrial animal and the largest ruminant . its species name refers to its camellike shape and its leopardlike colouring . its chief distinguishing characteristics are its extremely long neck and legs , its hornlike ossicones , and its distinctive coat patterns . it is classified under the family giraffidae , along with its closest extant relative , the okapi . the nine subspecies are distinguished by their coat patterns . the scattered range of giraffes extends from chad in the north to south africa in the south , and from niger in the west to somalia in the east . giraffes usually inhabit savannas , grasslands , and open woodlands .
(c) Passage about giraffes, but 'giraffe' is replaced by 'Cambridge' Figure 3.11: A visualization of IN-OUT similarities between terms in different passages with the query term "Cambridge". The visualization reveals that, besides the term "Cambridge", many other terms in the passages about both Cambridge and Oxford have high similarity to the query term. The passage (c) is adapted from a passage on giraffes by replacing all the occurrences of the term "giraffe" with "cambridge". However, none of the other terms in (c) are found to be relevant to the query term. An embedding based approach may be able to determine that passage (c) is non-relevant to the query "Cambridge", but fail to realize that passage (b) is also non-relevant. A term counting-based model, on the other hand, can easily identify that passage (b) is non-relevant but may rank passage (c) incorrectly high.  Embedding based models often perform poorly when the retrieval is performed over the full document collection [119]. However, as seen in the example of Figure 3.11, the errors made by embedding based models and exact matching models may be different-and the combination of the two is often pref-

Query expansion
Instead of comparing the query and the document directly in the embedding space, an alternative approach is to use term embeddings to find good expansion candidates from a global vocabulary, and then retrieving documents using the expanded query.
Different functions [91,120,186] have been proposed for estimating the relevance of candidate terms to the query-all of them involves comparing the candidate term individually to every query term using their vector representations, and then aggregating the scores. For example, [91,120] estimate the relevance of candidate term t c as, Term embedding based query expansion on its own performs worse than pseudorelevance feedback [120]. But like the models in the previous section, shows better performances when used in combination with PRF [186].
Diaz et al. [91] explored the idea of query-specific term embeddings and found that they are more effective in identifying good expansion terms than a global representation (see Figure 3.12). The local model proposed by Diaz et al. [91] incorporate relevance feedback in the process of learning the term embeddings-a set of documents is retrieved for the query and a query-specific term embedding model is trained. This local embedding model is then employed for identifying expansion candidates for the query for a second round of document retrieval.
Term embeddings have also been explored for re-weighting query terms [187] and finding relevant query re-writes [165], as well as in the context of other IR tasks such as cross-lingual retrieval [171] and entity retrieval [53,54]. In Section 3.6, we will discuss neural network models with deeper architectures and their applications to retrieval.

Supervised learning to rank
Learning to rank (LTR) for IR uses training data rel q (d), such as human relevance labels and click data, to train towards an IR objective. Unlike traditional IR approaches, these models typically have large number of learnable parameters that require many training samples to be tuned [16]. LTR models represent a rankable item-e.g., a query-document pair-as a feature vector ⃗ x ∈ R n . The ranking model f : ⃗ x → R is trained to map the vector to a real-valued score such that for a given query more relevant documents are scored higher and some chosen rank-based metric is maximized. The model training is said to be end-to-end if the parameters of f are learned all at once rather than in parts, and if the vector ⃗ x contains simple features rather than models. Liu [20] categorizes the different LTR approaches based on their training objectives.
• In the pointwise approach, the relevance information rel q (d) is in the form of a numerical value associated with every query-document pair with input vec- The numerical relevance label can be derived from binary or graded relevance judgments or from implicit user feedback, such as a clickthrough rate. A regression model is typically trained on the data to predict the numerical value rel q (d) given ⃗ x q,d .
• In the pairwise approach, the relevance information is in the form of preferences between pairs of documents with respect to individual queries (e.g., . The ranking problem in this case reduces to that of a binary classification to predict the more relevant document. • Finally, the listwise approach involves directly optimizing for a rank-based metric such as NDCG-which is more challenging because these metrics are often not continuous (and hence not differentiable) with respect to the model parameters.
Many machine learning models-including support vector machines [188], neural networks [189], and boosted decision trees [190]-have been employed over the years for the LTR task, and a correspondingly large number of different loss functions have been explored.

Input features
Traditional LTR models employ hand-crafted features [20] for representing querydocument pairs in ⃗ x. The design of these features typically encodes key IR insights and belong to one of the three categories.
• Query-independent or static features (e.g., incoming link count and document length) • Query-dependent or dynamic features (e.g., BM25) • Query-level features (e.g., query length) In contrast, in recently proposed neural LTR models the deep architecture is responsible for feature learning 5 from simple vector representations of the input which may resemble the schemes described in Section 3.5.1 (e.g., [118]) or the interaction-based representations that we discuss later in Section 3. 6.3 (e.g., [21,192]). These features, learnt from the query and document texts, can be combined with other features that may not be possible to infer from the content, such as document popularity [193].

Loss functions
In ad hoc retrieval, the LTR model needs to rank the documents in a collection D in response to a query. When training a neural model for this task, the ideal ranking of documents for a query q from the training dataset can be determined based on the relevance labels rel q (d) associated with each document d ∈ D. In the pointwise approach, the neural model is trained to directly estimate rel q (d), which can be a numeric value or a categorical label.

Regression loss
Given ⃗ x q,d , the task of estimating the relevance label rel q (d) can be cast as a regression problem, and a standard loss function-such as the square loss-can be employed.
Where, s(⃗ x q,d ) is the score predicted by the model and rel q (d) can either be the value of the relevance label [194] or the one-hot representation when the label is categorical [195].
Classification loss When the relevance labels in the training data are categorical, it makes more sense to treat the label prediction problem as a multiclass classification.
The neural model under this setting, estimates the probability of a label y given ⃗ x q,d .
The probability of the correct label y q,d (= rel q (d)) can be obtained by the softmax function, The softmax function normalizes the score of the correct label against the set of all possible labels Y . The cross-entropy loss can then be applied [196] as follows, However, a ranking model does not need to estimate the true relevance label accurately as long as it ranks the relevant documents D + over all the other candidates in D. Typically, only a few documents from D are relevant to q. If we assume a binary notion of relevance, then the problem is similar to multi-label classification-or, multiclass classification if we assume a single relevant document d + per query-where the candidate documents are the classes. Next, we discuss loss functions for LTR models that tries to predict the relevant document by maximizing p(d + |q).
Note that this is different from the classification loss in Equation 3.46 which maxi- Contrastive loss In representation learning models, a relevant document should be closer to the query representation than a non-relevant document. The contrastive loss [197,198]-common in image retrieval-learns the model parameters by minimizing the distance between a relevant pair, while increasing the distance between dissimilar items.
Contrastive loss assumes that the relevance label y q,d ∈ {0, 1} is binary. For each training sample, either L pos or L neg is applied over the distance dist q,d as predicted by the model. In particular, Hadsell et al. [198] use the following formulation of this loss function.
Where, m is a margin.

Cross-Entropy loss over documents
The probability of ranking d + over all the other documents in the collection D is given by the softmax function, The cross-entropy (CE) loss then maximizes the difference between scores generated by the model for relevant and less relevant documents.
However, when D is the full collection then computing the softmax (i.e. the denominator in Equation 3.53) is prohibitively expensive. Coincidentally, the CE loss is also useful for non-IR tasks, such as language modelling [144,145], where the model needs to predict a single term from a large vocabulary given its neighbours as input. Several different approaches have been proposed in the LM literature to address this computational complexity that is relevant to our discussion. We briefly describe some of these strategies here.
Hierarchical softmax Instead of computing p(d + |q) directly, Goodman [199] groups the candidates D into a set of classes C, and then predicts the correct class c + given q followed by predicting d + given ⟨c + , q⟩.
The computational cost in this modified approach is a function of |C|+|c + | which is typically much smaller than |D|. Further computational efficiency can be achieved by employing a hierarchy of such classes [161,200]. The hierarchy of classes is typically based on either similarity between candidates [145,201,202], or frequency binning [203]. Zweig and Makarychev [204] and Grave et al. [205] have explored strategies for building the hierarchy that directly minimizes the computational complexity.
Importance sampling (IS) An alternative to computing the exact softmax, is to approximately estimate it using sampling based approaches. Note, that we can re-write Equation 3.53 as follows, To train a neural model using back-propagation, we need to compute the gradient ∇ θ of the loss L CE with respect to the model parameters θ , As Senécal and Bengio [206]

Noise Contrastive Estimation (NCE)
In NCE [211][212][213], the task is modified to that of a binary classification. The model is trained to distinguish a sample drawn from a true distribution p(d|q) from a sample drawn from a noisy distributionp(d).
The training data contains k noisy samples for every true sample. Let, E andĒ indicate that a sample is drawn from the true and the noisy distributions, respectively. Then, We want our model to learn the true distribution p(d|q). Remember, that according to our model, A key efficiency trick involves setting z(q) to 1 [212][213][214]. Therefore, Putting Equation 3.67 back in Equation 3.63 and 3. 64.
Finally, the NCE loss is given by, Note, that the outer summation iterates over all the positive ⟨x, d + ⟩ pairs in the training data.
which changes the NCE loss to the NEG loss.
BlackOut Related to both IS and NCE, is BlackOut [215]. It is an extension of the DropOut [216] method that is often employed to avoid over-fitting in neural models with large number of parameters. DropOut is typically applied to the input or hidden layers of the network and involves randomly dropping a subset of the neural units and their corresponding connections. BlackOut applies the same idea to the output layer of the network for efficiently computing the loss. We refer readers to [215] for more rigorous discussions on the relationship between IS, NCE, and DropOut.
For document retrieval Huang et al. [118] approximate the cross-entropy loss of additional notes on these approaches, we refer the readers to [217][218][219].
In a typical retrieval scenario, however, multiple documents may be relevant to the same query q, and the notion of relevance among this set of documents D + may be further graded. Some LTR approaches consider pairs of documents for the same query and minimize the average number of inversions in ranking-i.e., d i ≻ q d j but d j is ranked higher than d i . The pairwise loss employed in these approaches has the following form [220], where, some possible choices for ϕ include, • Hinge function ϕ (z) = max(0, 1 − z) [221,222] • Exponential function ϕ (z) = e −z [223] • Logistic function ϕ (z) = log(1 + e −z ) [189] RankNet loss RankNet [189] is a pairwise loss function that has been a common choice for training neural LTR models and was also for many years an industry favourite, such as at the commercial Web search engine Bing. 6  R n → R, typically a neural network but can also be any other machine learning model whose output is differentiable with respect to its parameters, computes the Given the scores ⟨s i , s j ⟩, the probability that d i would be ranked higher than d j is given by, Where, the constant σ determines the shape of the sigmoid. During training, the probability of ranking d i higher than d j for q is maximized.
be the true preference label between d i and d j for the training sample-denoting d i is less, equal, or more relevant than d j , respectively. Then the desired probability of The cross-entropy loss L between the desired probabilityp i j and the predicted probability p i j is given by, Note that L is differentiable with respect to the model output s i and hence the model can be trained using gradient descent. We direct the interested reader to [224] for more detailed derivations for computing the gradients for RankNet.
Readers should note the obvious connection between the CE loss described previously and the RankNet loss. If in the denominator of Equation 3.53, we only sum over a pair of relevant and non-relevant documents then it reduces to the logisticloss function of RankNet described in Equation 3. 80 This formulation of LambdaRank can optimize directly for NDCG [226,227], and any other IR measure by incorporating the corresponding delta change in Equation 3. 81.

ListNet and ListMLE loss
The probability of observing a particular rank order can be estimated from the individual document scores using different models [228][229][230].
For example, according to the Luce model [228], given four items the probability of observing a particular rank-order, say , is given by: Where, π is a particular permutation and ϕ is a transformation (e.g., linear, exponential, or sigmoid) over the score s i corresponding to item d i . Using this model, we can compute the probability distribution over all possible permutations based on the model scores and the ground truth labels. The K-L divergence between these two distributions gives us the ListNet loss [231].
However, computing the probability distribution over all possible permutations is computationally expensive, even when restricted to only the top-K items. The ListMLE loss [232] instead computes the probability of the ideal permutation based on the ground truth. However, with categorical labels more than one ideal permutation may be possible which should be handled appropriately.
Many of the challenges discussed in this section are common to both retrieval tasks as well as multiclass and multilabel classification with extremely large number of classes-often referred to as extreme classification [233][234][235]. Ad hoc retrieval can be posed as an extreme classification task under a binary notion of relevance and a fixed collection constraint. New loss functions (e.g. the spherical loss family [236][237][238]) have been explored for these large scale classification tasks which may be relevant for neural retrieval research. The problem of learning from sparse biased labels [239,240] is also an important challenge in these frameworks. Finally, deep neural models for LTR with large number of parameters may require large training data for supervised learning. Alternative training schemes-e.g., using weak supervision signals [241,242] or adversarial learning [96,243]-are emerging.   For an input vector ⃗ x, the model produces the output ⃗ y as follows,

Deep neural networks
The model training involves tuning the parameters W 1 , ⃗ b 1 , W 2 , and ⃗ b 2 to minimize the loss between the expected output and the output predicted by the final layer.  Figure 3.14: Consider a toy binary classification task on a corpus of four short texts-"surface book", "kerberos library", "library book", and "kerberos surface"where the model needs to predict if the text is related to computers. The first two texts-"Surface Book" and "kerberos library"-are positive under this classification, and the latter two negative. The input feature space consists of four binary features that indicate whether each of the four terms from the vocabulary is present in the text. The table shows that the specified classes are not linearly separable with respect to the input feature space. However, if we add couple of hidden nodes, as shown in the diagram, then the classes can be linearly separated with respect to the output of the hidden layer.
The parameters are usually trained discriminatively using backpropagation [247][248][249]. During forward-pass each layer generates an output conditioned on its input, and during backward pass each layer computes the error gradient with respect to its parameters and its inputs.
The design of a DNN typically involves many choices of architectures and hyper-parameters. Neural networks with as few a single hidden layer-but with sufficient number of hidden nodes-can theoretically approximate any function [250].
In practice, however, deeper architectures-sometimes with as many as 1000 layers [251]-have been shown to perform significantly better than shallower networks.
For readers who are less familiar with neural network models, we present a simple example in Figure 3.14 to illustrate how hidden layers enable these models to capture non-linear relationships. We direct readers to [252] for further discussions on how additional hidden layers help.
The rest of this section is dedicated to the discussion of input representations and standard architectures for deep neural models.

Input text representations
Neural models that learn representations of text take raw text as input. A key consideration is how the text should be represented at the input layer of the model. The input text can also be pre-tokenized into terms-where each term is represented by either a sparse vector or using pre-trained term embeddings (Figure 3.15d).
Terms may have a one-hot (or local) representation where each term has an unique ID ( Figure 3.15b), or the term vector can be derived by aggregating one-hot vectors of its constituting characters (or character n-graphs) as shown in Figure 3.15c. If pre-trained embeddings are used for term representation, then the embedding vectors can be further tuned during training or kept fixed.
Similar to character-level models, the term vectors are further aggregated (by concatenation or sum) to obtain the representation of longer chunks of text, such as sentences. While one-hot representations of terms (Figure 3.15b) are common in many NLP tasks, historically pre-trained embeddings (e.g., [256,257]) and character n-graph based representations (e.g., [21,118]) are more commonplace in IR.

Architectures
In this section, we describe few standard neural architectures commonly used in IR. For broader overview of neural architectures and design patterns please refer to [2,143,247].  Shift-invariant neural operations Convolutional [1,246,258,259] and recurrent [260][261][262][263] architectures are commonplace in many deep learning applications.
These neural operations are part of a broader family of shift-invariant architectures.
The key intuition behind these architectures stem from the natural regularities ob-   In convolution or pooling, each window is applied independently. In con-  trast, in the recurrent architecture of Figure 3.16e the cell not only considers the input window but also the output of the previous instance of the cell as its input.
Many different cell architectures have been explored for recurrent neural networks (RNN)-although Elman network [264], Long Short-Term Memory (LSTM) [263], and Gated Recurrent Unit (GRU) [265,266] are commonly used. RNNs are typically applied to sequences but can also be useful for two (and higher) dimensional inputs [267].
One consideration when using convolutional or recurrent layers is how the window outputs are aggregated. Convolutional layers are typically followed by pooling or fully-connected layers that perform a global aggregation over all the window instances. While a fully-connected layer is aware of each window position, a global pooling layer is typically agnostic to it. However, unlike a fully-connected layer, a global max-pooling operation can be applied to a variable size input. Where a global aggregation strategy may be less appropriate (e.g., long sequences), recurrent networks with memory [268][269][270] and/or attention [40,[271][272][273][274] may be useful. Figure 3.16e shows tree-structured (or recursive) neural networks [275][276][277][278][279] where the same cell is applied at multiple levels in a tree-like hierarchical fashion resulting in a recursive aggregation strategy.
Finally, attention mechanisms-in particular, self-attention [280]-have demonstrated remarkable usefulness for many NLP and IR tasks. In a typical attention setting, we have a set of n items that we can attend over and an input context, and we produce a probability distribution {a 1 , . . . , a i , . . . , a n } of attending to each item as a function of similarity between a learned representation q of the context and learned representations k i of the items. The final output o is the aggregate of learned value v i corresponding to each item weighted by their attention In self-attention, we repeat the above process n times treating one of the n items themselves as the context in each case. Self-attention layers have been operationalized in Transformer-based [280] architectures, e.g., BERT [281].

Auto-encoders
The autoencoder architecture [248,282,283] is based on the information bottleneck method [155]. The goal is to learn a compressed representation The model has an hour-glass shape as shown in Figure 3.
Siamese networks Siamese networks were originally proposed for comparing fingerprints [284] and signatures [285]. Yih et al. [286] later adapted the same architecture for comparing short texts. The siamese network, as seen in Figure 3.17b, resembles the autoencoder architecture (if you squint hard enough)-but unlike the latter is trained on pairs of inputs ⟨input 1 , input 2 ⟩. The architecture consists of two models (model 1 and model 2 ) that project input 1 and input 2 , respectively, to ⃗ v 1 and ⃗ v 2 in a common latent space. A pre-defined metric (e.g., cosine similarity) is used to then compute the similarity between ⃗ v 1 and ⃗ v 2 . The model parameters are optimized such that ⃗ v 1 and ⃗ v 2 are closer when the two inputs are expected to be similar, and further away otherwise.
One possible loss function is the logistic loss. If each training sample consist then we minimize,  In an autoencoder the embeddings are learnt by minimizing the self-reconstruction error, whereas a Siamese network focuses on retaining the information that is necessary for determining the similarity between a pair of items (say, a query and a document).
Where, γ is a constant that is often set to 10. Typically, both the models-model 1 and model 2 -share identical architectures, but can also choose to share the same parameters. In image retrieval, the contrastive loss [197,198] is also used for training Siamese networks.
It is important to note that, unlike the autoencoder, the minimal sufficient statistics retained by a Siamese network is dictated by which information it deems important for determining the similarity between the paired items.
Variational autoencoders (VAE) In Variational autoencoders [287,288], the encoder part of the network generates two separate vectors-the vector of means µ and the vector of standard deviations σ . The latent representation ⃗ x of the input is then generated by sampling a random variable x i with mean µ i and standard deviation σ i along each of the k latent dimensions.
By sampling the latent representation, we expose the decoder to a certain degree of local variations in its input that should force the model to learn a smoother continuous latent space. The VAE is trained by jointly minimizing the reconstruction losssimilar to vanilla autoencoders-and an additional component to the loss function which is the KL-divergence between the latent variable x i and a unit gaussian.
Without the L KL−divergence component the model can learn very different µ for different classes of inputs and minimize the λ to be arbitrarily small such that the learnt latent space is no longer smooth or continuous. Readers should note that the sampling step is non-differentiable, but the model can be trained using the "reparameterization trick" proposed by Kingma and Welling [287].
An important application of VAE is for the synthesis of new items (e.g., images [289] or text [290]) not observed in the training collection. Another class of techniques for synthesis includes the Generative Adversarial Networks.
Generative Adversarial Networks (GAN) Goodfellow et al. [291] proposed a framework for training generative models under an adversarial setting. GANs typically consist of two separate neural networks-a generator network and a discriminator network. The goal of the generator network is to synthesize new (fake) items that mimic similar distributions as items that exist in the training collection. The goal of the discriminator network is to correctly distinguish between a true item and an item produced by the generator. The generator is trained to maximize the probability of the discriminator wrongly classifying the true and the generated item-which corresponds to a minimax two-player game.

Neural toolkits
In recent years, the advent of numerous flexible toolkits [292][293][294][295][296][297][298][299]  • computing the output of the node given the input (forward-pass) • computing the gradient of the loss with respect to the inputs, given the gradient of the loss with respect to the output (backward-pass) • computing the gradient of the loss with respect to its parameters, given the gradient of the loss with respect to the output (backward-pass) A deep neural network, such as the one in Figure 3.13 or ones with much more complex architectures (e.g., [251,300,301]), can then be specified by chaining instances of these available computation nodes, and trained end-to-end on large datasets using backpropagation over GPUs or CPUs. In IR, various application interfaces [302,303] bind these neural toolkits with existing retrieval/indexing frameworks, such as Indri [110]. Refer to [304] for a comparison of different neural toolkits based on their speed of training using standard performance benchmarks.

Deep neural models for IR
Traditionally, deep neural network models have much larger number of learnable parameters than their shallower counterparts. A DNN with a large set of parameters can easily overfit to smaller training datasets [305]. Therefore, during model design it is typical to strike a balance between the number of model parameters and the size of the data available for training. Data for ad hoc retrieval mainly consists of, • Corpus of search queries • Corpus of candidate documents • Ground truth-in the form of either explicit human relevance judgments or implicit labels (e.g., from clicks)-for query-document pairs While both large scale corpora of search queries [79,306] and documents [307][308][309] are publicly available for IR research, the amount of relevance judgments that can be associated with them are often limited outside of large industrial research labs-mostly due to user privacy concerns. We note that we are interested in datasets where the raw text of the query and the document is available. Therefore, this excludes large scale public labelled datasets for learning-to-rank (e.g., [310]) that don't contain the textual contents.
The proportion of labelled and unlabelled data that is available influences the level of supervision that can be employed for training these deep models. Most of the models we covered in Section 3.3 operate under the data regime where large corpus of documents or queries is available, but limited (or no) labelled data. Under such settings where no direct supervision or relevance judgments is provided, typically an unsupervised approach is employed (e.g., using auto-encoding [311] or masked language modeling [281]). The unlabelled document (or query) corpus is used to learn good text representations, and then these learnt representations are incorporated into an existing retrieval model or a query-document similarity metric.
If small amounts of labelled data are available, then that can be leveraged to train a retrieval model with few parameters that in turn uses text representations that is pre-trained on larger unlabelled corpus. Examples of such semi-supervised training includes models such as [117,192,256]. In contrast, fully-supervised modelse.g., [21,28,118,312,313]-optimize directly for the target task by training on large number of labelled query-document pairs.
It is also useful to distinguish between deep neural models that focus on ranking long documents, from those that rank short texts (e.g., for the questionanswering task, or for document ranking where the document representation is based on a short text field like title). The challenges in short text ranking are somewhat distinct from those involved in the ad hoc retrieval task [314]. When computing similarity between pairs of short-texts, vocabulary mismatches are more likely than when the retrieved items contain long text descriptions [315]. Neural models that perform matching in a latent space tend to be more robust towards the vocabulary mismatch problem compared to lexical term-based matching models.
On the other hand, documents with long body texts may contain mixture of many topics and the query matches may be spread over the whole document. A neural document ranking model must effectively aggregate the relevant matches from different parts of a long document. In the rest of this section, we discuss several neural architectures and approaches to document ranking.

Document auto-encoders
Salakhutdinov and Hinton [311] proposed Semantic Hashing-one of the earliest deep neural models for ad hoc retrieval.
While, DSSM [118] employs deep fully-connected architecture for the query and the document models, more sophisticated architectures involving convolutional layers [185,257,317,318], recurrent layers [319,320], and tree-structured networks [278] have also been explored. The similarity function can also be parameter- ized and implemented as additional layers of the neural network as in [312]. Most of these models have been evaluated on the short text matching task, but Mitra et al. [21] recently reported meaningful performances on the long document ranking task from models like DSSM [118] and CDSSM [185] under telescoping evaluation. Mitra et al. [21] also show that sampling the negative documents uniformly from the collection is less effective to using documents that are closer to the query intent but judged as non-relelvant by human annotators in similar evaluation settings.
Notions of similarity It is important to emphasize that our earlier discussion in Section 3.2.2 on different notions of similarity between terms that can be learnt by shallow embedding models is also relevant in the context of these deeper architectures.
In the case of Siamese networks, such as the convolutional-DSSM (CDSSM) [185], the notion of similarity being modelled depends on the choice of the paired data that the model is trained on. When the CDSSM is trained on query and document title pairs [185] then the notion of similarity is more topical in nature. Mitra and Craswell [34] trained the same CDSSM architecture on query prefix-suffix pairs which, in contrast, captures a more typical notion of similarity, as shown in Table 7. 2. In a related work, Mitra [33] demonstrated that the CDSSM model when trained on session-query pairs is amenable to vector-based text analogies. ⃗ By modelling different notions of similarity these deep neural models tend to be more suitable for other IR tasks, such as query auto-completion [34] or sessionbased personalization [33].

Interaction-based networks
Siamese networks represent both the query and the document using single embedding vectors. Alternatively, we can individually compare different parts of the query with different parts of the document, and then aggregate these partial evidences of relevance. Especially, when dealing with long documents-that may contain a mixture of many topics-such a strategy may be more effective than trying to represent the full document as a single low-dimensional vector. Typically, in these approaches a sliding window is moved over both the query and the document text and each instance of the window over the query is compared (or "interacts") against   The interaction matrix based approach have been explored both for short text matching [256,257,[321][322][323][324], as well as for ranking long documents [21,192,325,326].

Lexical matching networks
Much of the explorations in neural IR models have focused on learning good representations of text. However, these representation learning models tend to perform poorly when dealing with rare terms and search intents. In Section 2. 1

Conclusion
We surveyed a large body work in this section. We introduced the fundamentals of traditional IR models and representation learning with neural networks. We presented some of the recent (shallow and deep) neural approaches for document ranking and question-answer matching. Readers should note that this is an active area for research, and new architectures and learning methods are continuously emerging. So, it is likely that by the time this thesis is published, many of the methods described here may have already been superseded by more recent and advanced methods. In the subsequent chapters of this thesis, we will cover our contributions in the form of new neural models and approaches for some of these IR tasks.

Chapter 4
Learning to rank with Duet networks In traditional Web search, the query consists of only few terms but the body text of the documents may typically have tens or hundreds of sentences. In the absence of click information, such as for newly-published or infrequently-visited documents, the body text can be a useful signal to determine the relevance of the document for the query. Therefore, extending existing neural text representation learning approaches to long body text for document ranking is an important challenge in IR.
However, as was noted previously [328], despite the recent surge in interests towards applying deep neural networks (DNN) for retrieval, their success on ad hoc retrieval tasks has been rather limited. Some papers [120,192] report worse performance of neural embedding models when compared to traditional term-based approaches, such as BM25 [76].
Traditional IR approaches consider terms as discrete entities. The relevance of the document to the query is estimated based on, amongst other factors, the number of matches of query terms in the document, the parts of the document in which the matches occur, and the proximity between the matches. In contrast, latent semantic analysis (LSA) [124], probabilistic latent semantic analysis (PLSA) [152] and latent Dirichlet allocation (LDA) [154,329]    for short-text matching as either interaction-focused [256,257,321] or representation-focused [118,185,257,312,317]. They claim that IR tasks are different from NLP tasks, and that it is more important to focus on exact matching for the former and on learning text embeddings for the latter. Mitra et al. [119], on the other hand, claim that models that compare the query and the document in the latent semantic space capture a different sense of relevance than models that focus on exact term matches, and therefore the combination of the two is more favourable. Our work is motivated by the latter intuition that it is important to match the query and the document using both local and distributed representations of text. We propose a novel ranking model comprised of two separate DNNs that model query-document relevance using local and distributed representations, respectively. The two DNNs, referred to henceforth as the local subnetwork and the distributed subnetwork, are jointly trained as part of a single model, that we name as the Duet network because the two subnetworks co-operate to achieve a common goal. In this chapter, we show that the combination of the two DNNs not only outperforms the individual subnetworks, but also demonstrates large improvements over traditional baselines and other previously proposed models based on DNNs on the document ranking task. Unlike other previous work [120,192], our model significantly outperforms classic IR approaches by using a DNN to learn text representation.
Deep neural network models are known to benefit from large training data, achieving state-of-the-art performance in areas where large scale training corpora are available [2,210]. Some of the lack of positive results from neural models in ad hoc retrieval is likely due to the scarce public availability of large quantity of training data necessary to learn effective representations of text. In Section 4.5, we will present some analysis on the effect of training data on the performance of these DNN models. In particular, we found that-unsurprisingly-the performance of the To summarize, the key contributions of this chapter are: 1. We propose a novel Duet network that jointly learns two deep neural networks that match query and document based on their lexical similarity and similarity in their learned latent representations, respectively.

2.
We demonstrate that Duet out-performs previous state-of-the-art neural and traditional non-neural baselines. 3. We demonstrate that training with documents judged as non-relevant as the negative examples is more effective than randomly sampling them from corpus. 4. We report additional improvement to the original Duet network evaluated on two recently released public benchmarks with sufficiently large training data.  the number of terms in the vocabulary for the local subnetwork). The query q and the document d is, in turn, represented by the matrices X q and X d , respectively.

The Duet network
We fix the length of the inputs across all the queries and the documents such that we consider only the first n q terms in the query and the first n d terms in the document.
If either the query or the document is shorter than these target dimensions, then the input vectors are padded with zeros. The truncation of the document body text to the first n d terms is performed only for our subnetwork and its variants, but not for the baseline models. For all the neural and the non-neural baseline models we consider the full body text.   The interaction matrix X is first passed through a convolutional layer with n hidden filters, a kernel size of n d × 1, and a stride of 1. The output Z i corresponding to the i th convolutional window over X is a function of the match between the t q i term against all the terms in the document, Where X i is the row i of X, tanh is performed elementwise, and the n d × n hidden matrix W contains the learnable parameters of the convolutional layer. The output Z of the convolutional layer is a matrix of dimension n hidden × n q . The output of the convolutional layer is then passed through two fully-connected layers, a dropout layer, and a final fully-connected layer that produces a single real-valued output.
All the nodes in the local subnetwork uses the hyperbolic tangent function for nonlinearity.

Distributed subnetwork
The distributed subnetwork learns dense lower-dimensional vector representations of the query and the document text, and then computes the positional similarity between them in the learnt embedding space. Instead of one-hot encoding of terms, as in the local subnetwork, we use a character n-graph based representation of each term in the query and document. Our n-graph based input encoding is motivated by the trigraph encoding proposed by Huang et al. [118], but unlike their approach we don't limit our input representation to n-graphs of a fixed length. For each term, we count all the n-graphs present for 1 ≤ n ≤ n maxgraph . We then use this n-graph frequency vector of length m distrib to represent the term.
Instead of directly computing the interaction between the m distrib × n q matrix X q and the m distrib ×n d matrix X d , we first learn a series of nonlinear transformations to the character-based input. For both the query and the document, the first step is convolution. The m distrib × n window convolution window has filter size of n hidden .
It projects n window consecutive terms to a n hidden -dimensional vector, then takes a stride by 1 position, and projects the next n window terms, and so on. For the query, the convolution step generates a tensor of dimensions n hidden × (n q − n window + 1).
For the document, it generates one of dimensions n hidden × (n d − n window + 1).
Following this, we conduct a max-pooling step. For the query the pooling kernel dimensions are 1 × (n q − n window + 1). For the document, it is 1 × n pool .
Thus, we get one n hidden -dimensional embedding ⃗ v q for the query and a n hidden × To perform the matching, we conduct the element-wise or Hadamard product between the embedded document matrix and the extended or broadcasted query After this, we pass the matrix through fully connected layers, and a dropout layer until we arrive at a single score. Like the local subnetwork, we use hyperbolic tangent function here for non-linearity.

Optimization
Each training sample consists of a query q, a relevant document d + and a set of non- . . , d n neg }. We use a softmax function to compute the posterior probability of the positive document given a query based on the score.
We maximize the log likelihood log p(d + |q) using stochastic gradient descent.

Experiments
We conduct three experiments on a document ranking task to test: (1)  In addition, we also evaluate the effectiveness of the Duet model on the TREC Complex Answer Retrieval (TREC CAR) task [331]. In this section, we detail both the experiment setup and the corresponding baseline implementations.   Because all of our datasets are derived from sampling real query logs and because queries naturally repeat, there is some overlap in queries between the training and testing sets. Specifically, 14% of the testing queries in the weighted set occurr in the training set, whereas only 0.04% of the testing queries in the unweighted set occurr in the training set. We present both results for those who may be in environments with repeated queries (as is common in production search engines) and for those who may be more interested in cold start situations or tail queries.

TREC Complex Answer Retrieval task
The goal of TREC CAR task is to, given a document title and a section heading from the same document as a query, retrieve and rank passages from a provided collection. In order to support this task, the TREC CAR organizers present a large training set derived from English Wikipedia.
The mediawiki format of articles is parsed to extract the title, the section headings, and the corresponding passages. The collection is filtered to exclude pages which belong to frequent categories, such as people and events, and articles with less than five sections are discarded.
For each heading, we construct a set that includes all passages from the page (in random order) as well as the same amount of passages randomly drawn from other pages. On average this process yields a mean of 35 passages per section which includes: (1) passages from the correct section, (2) passages from the same page, but from different sections, or (3) passages from other pages. The retrieval involves ranking the correct passages (1) higher than the passages from the wrong section or article (2 and 3). We split the dataset for training and testing at a 4:1 ratio.

Training
Besides the architecture (Figure 4.2), our model has the following free parameters: (1) the maximum order of the character-based representation for the distributed sub- For both the document ranking and the TREC CAR tasks we limit the maximum number of query terms n q to 10, and fix the window size of the convolution n window to 3. The dropout rate is also set to 0.20 for both.
For the document ranking task, we consider the first 1000 terms in the document. Correspondingly, the max-pooling window size n pool is fixed at 100, and n hidden is set to 300 tees that both models observed the exact same pairs of positive and negative documents during training. We adopted the same loss function as proposed by Guo et al. [117].

TREC Complex Answer Retrieval task
We rank results using Okapi BM25 [76] with k 1 =1.2 and b=0. 75. Porter stemming is applied to a Lucene 6.4.1 index and the query.
In addition, we experiment with three different query expansion approaches (terms, entities, and passages) and three vector space representations of queries and documents (tf-idf, GloVe embeddings, and RDF2Vec embeddings). Each of the possible combinations (e.g., term-expansion + tf-idf vectors, or passage-expansion + term embedding vectors) defines a query representation. Results are ranked according to the cosine similarity between the vector representations of the query and the document.
We experiment with three different query expansion approaches: • Expansion terms (RM). Feedback terms are derived using pseudo relevance feedback and the relevance model [112]. We use Galago's implementation 5 which is based on a Dirichlet smoothed language model for the feedback run.
We achieve the best performance by expanding the query with top 10 terms extracted from the top 10 feedback documents.
• Expansion entities. We also expand the query using supporting entities retrieved by a search for the query. Best performance is achieved using 10 entities for expansion.
• Passage Rocchio. Inspired by the work of Banerjee and Mitra [332], we retrieve other passages, which have identical section heading to the heading part of our query, from the portion of the dataset reserved for training. For example, given a query such as "United States demographic", with respect to the entity United States, we collect supporting passages from the pages of other entities (e.g." United Kingdom), that fall under the section titled "Demographics". Headings are processed with tokenisation, stopword and digit removal, and stemming. We are able to retrieve at least one supporting passage for one-third of our queries. We obtain best performance from expanding the query with 5 passages.
We investigate three vector representation schemes for the query and the passage: where, u(d) is the set of unique words in query q.
• Entity Embeddings. Queries and documents are represented as their mentioned DBpedia entities, using the entity linker TagMe [333]-with default parameters. We obtain latent vector representations ⃗ v e of each linked entity e using pre-computed RDF2Vec entity embeddings [334]. Query and passage representation is obtained from weighted average of these entity vectors.
Entity vectors are weighted based on inlink and outlink statistics from the 2015-04 DBpedia Data Set [335].
where, ent(d) is the set of entities in query q.
Additionally, we combine the ranking-score of these different baselines with supervised machine learning [336]. We train a linear model using RankLib 6 optimized for MAP, trained with coordinate ascent.

Evaluation
For the document ranking task, we report the normalized discounted cumulative gain (NDCG) metric computed at positions one and ten. All performance metrics are averaged over queries for each run. Whenever testing for significant differences in performance, we used the paired t-test with a Bonferroni correction. For the TREC CAR task, we report MRR, R-Prec, and MAP numbers for all the models. This supports our underlying hypothesis that matching in a latent semantic space can complement exact term matches in a document ranking task, and hence a combination of the two is more appropriate. Note that the NDCG numbers for the local and the distributed subnetworks correspond to when these DNNs are trained individually, but for Duet the two DNNs are trained together as part of a single neural network.
Among the baseline models, including both traditional and neural network based models, CDSSM and DESM achieve the highest NDCG at position one and ten, respectively, on the weighted test set. On the unweighted test set DRMM is our best baseline model at both rank positions. Duet demonstrates significant improvements over all these baseline models on both test sets and at both NDCG positions.  We also test our independent local and distributed models against their conceptually closest baselines. Because our local model captures both matching and proximity, we compared performance to dependence models (DM). While the performance in terms of NDCG@1 is statistically indistinguishable, both NDCG@10 results are statistically significant (p < 0.05). We compared our distributed model to the best neural model for each test set and metric. We found no statistically significant difference except for NDCG@10 for the weighted set.
We were interested in testing our hypotheses that training with labeled negative documents is superior to training with randomly sampled documents presumed to be negative. We conducted an experiment training with negative documents following each of the two protocols.  over the BM25 baseline. The most promising results-among the baseline methods which employ cosine similarity as a ranking function-are obtained when the query is expanded with supporting textual paragraphs. This is an interesting finding that reconfirms the results of previous work on the automatic generation of Wikipedia articles based on its structural information [332,341]. The learning to rank model is our best performing baseline. Duet yields a substantial improvement over all presented approaches, including a 47% improvement in MAP over the BM25 baseline and a 14% improvement over the learning to rank model.

Further improvements
In follow up work, we explore several additional modifications to the original Duet architecture and demonstrate through an ablation study that incorporating these changes results in significant improvements on passage ranking. We evaluate the modified Duet model on the MS MARCO passage ranking task [48] and the TREC Deep Learning track [35]. In the context of the document ranking task at TREC, we further modify the architecture to incorporate multiple-field representation of documents.

Duet on MS MARCO
In this section, we briefly describe several modifications to the Duet architecture in the context of passage ranking. A public implementation of the updated Duet model using PyTorch [342] is available online 7 .
1. Word embeddings. We replace the character level n-graph encoding in the input of the distributed subnetwork with word embeddings. We see significant reduction in training time given a fixed number of minibatches and a fixed minibatch size. This change primarily helps us to train on a significantly larger amount of data under fixed training time constraints. We initialize the word embeddings using pre-trained GloVe [156] embeddings before training Duet.

2.
Inverse document frequency weighting. In contrast to some of the other datasets on which Duet has been previously evaluated [21,28], the MS MARCO dataset contains a relatively larger percentage of natural language queries and the queries are considerably longer on average. In traditional IR models, the inverse document frequency (IDF) [343] of a query term provides an effective mechanism for weighting the query terms by their discriminative power. In the original Duet architecture, the input to the local subnetwork corresponding to a query q and a document d is a binary interaction matrix X ∈ R |q|×|d| defined as follows: We incorporate IDF in Duet by weighting the interaction matrix by the IDF of the matched terms. We adopt the Robertson-Walker definition of IDF [344] normalized to the range [0, 1].
Where, N is the total number of passages in the collection and n t is the number of passages in which the term t appears at least once. 3. Non-linear combination of local and distributed subnetworks. Zamani et al. [345] show that when combining different subnetworks in a neural ranking model, it is more effective if each subnetwork produce a vector output that are further combined by additional multi-layer perceptrons (MLP). In the original Duet, the local and the distributed subnetwork produce a single score that are linearly combined. In our updated architecture, both subnetworks produce a vector that are further combined by an MLP-with two hidden layers-to generate the estimated relevance score. where q is a query and p + and p − are a pair of passages, with p + being more relevant to q than p − . Similar to the original Duet, we employ the cross-entropy with softmax loss to learn the parameters of our network M :

Rectifier Linear Units (ReLU
where, ℓ(∆) = log(1 + e −σ ·∆ ) (4.11) Where, M q,p is the relevance score for the pair ⟨q, p⟩ as estimated by the model M . Note, that by considering a single negative passage per sample, our loss is equivalent to the RankNet loss [189].
We use the Adam optimizer with default parameters and a learning rate of 0.001. We set σ in Equation 5.8 to 0.1 and dropout rate for the model to 0. 5

Duet on TREC Deep Learning track
The deep learning track at TREC 2019 makes large training datasets-suitable for traininig deep models with large number of learnable parameters-publicly available in the context of a document ranking and a passage ranking tasks. We benchmark Duet on both tasks.
In the context of the document ranking task, we adapt Duet to ingest a "multiple field" view of documents, based on findings from Zamani et al. [345]. We refer to this new architecture as Duet with Multiple Fields (DuetMF). We also combine the relevance estimates from DuetMF with several other traditional and neural retrieval methods in a learning-to-rank (LTR) [20]  We incorporate all of these ideas in the updated Duet network as shown in Fig. 4 In light of the recent success of large pretrained language models-  We add couple of features based on query length and domain quality-where the latter is defined simply as a ratio between how often documents from a given domain appear in the positively labeled training data and in the overall document collection.
Finally, for the passage ranking task, we adopt the exact same model and training procedure from Section 4. 4

.1. Our final submission is an ensemble of eight
Duet networks.   NDCG performance of different models by length of query and how rare the rarest query term is in the training data. For the rare term analysis, we place all query terms into one of five categories based on their occurrence counts in the training data. Then we then categorize each query in the test dataset based on the frequency of the rarest term belongs in the query. We include a category for queries with at least one term which has no occurrences in the training data.

Discussion
Our results demonstrated that our joint optimization of local and distributed subnetworks provides substantial improvement over several state-of-the-art baselines. Although the independent models were competitive with existing baselines, the combination provided a significant boost.
We also confirm that using judged negative documents should be used when available. We speculate that training with topically-similar (but non-relevant) documents allows the model to better discriminate between the documents provided by an earlier retrieval stage that are closer to each other w.r.t. relevance. This sort of staged ranking, first proposed by Cambazoglu et al. [353], is now a common web search engine architecture.
In Section 4.2.3 we described our baseline models according to which of the properties of effective retrieval systems they incorporate. It is reasonable to expect that models with certain properties are better suited to deal with certain segments of queries. For example, the relevant Web page for the query "what channel are the seahawks on today" may contain the name of the actual channel (e.g." "ESPN" or "FOX") and the actual date for the game, instead of the terms "channel" or "today". A retrieval model that only counts repetitions of query terms is likely to retrieve less relevant documents for this query -compared to a model that con- siders "ESPN" and "FOX" to be relevant document terms. In contrast, the query "pekarovic land company", which may be considered as a tail navigational intent, is likely to be better served by a retrieval model that simply retrieves documents containing many matches for the term "pekarovic". A representation learning model is unlikely to have a good representation for this rare term, and therefore may be less To better understand the relationship of our models to existing baselines, we compared the per-query performance amongst all models. We conjecture that similar models should perform similarly for the same queries. We represented a retrieval model as a vector where each position of the vector contains the performance of the model on a different query. We randomly sample two thousand queries from our weighted test set and represent all ranking models as vectors of their NDCG values against these two thousand queries. We visualized the similarity between models by projecting using principal component analysis on the set of performance vectors.
The two-dimensional projection of this analysis is presented in Figure 4. 7  A last consideration when comparing these models is runtime efficiency.

Conclusion
We propose a novel ranking model composed of two separate deep subnetworks, one that matches using local representation of text, and another that learns distributed representation before matching. The Duet of these two subnetworks achieve better performance compared to the sub-models individually on the document ranking and passage ranking tasks-as well as significant improvements over other neural and traditional non-neural baselines. Our analysis indicate that the improvements over traditional methods are more substantial in the presence of larger training datasets.

Chapter 5
Retrieve, not just rerank, using deep neural networks In response to short text queries, search engines attempt to retrieve the top few relevant results by searching through collections containing billions of documents [354], often under a second [355]. Response time is a key consideration in web search. Even a 100ms latency has been shown to invoke negative user reactions [356,357]. To achieve such short response times, these systems typically distribute the collection over multiple machines that can be searched in parallel [358]. Specialized data structures-such as inverted indexes [359,360]-are used to dramatically cut down the number of documents required to be evaluated for any specific query.
The index organization and query evaluation strategies, in particular, trade-off retrieval effectiveness and efficiency during the candidate generation stage. However, unlike in late stage re-ranking where machine learning (ML) models are commonplace [20,361], the candidate generation frequently employs traditional retrieval models with few learnable parameters.
Query evaluation using state-of-the-art deep neural ranking models require time and resource intensive computations. Typically these models also require both query and document as input to inspect the interactions between query and document terms. The study of these neural ranking methods have, therefore, been largely limited to late stage re-ranking. Efficient retrieval using these complex machine learned relevance estimators is an important challenge [362].
Recently, a few different attempts [363][364][365] have been made to leverage neural methods for retrieval over large collections. All of these studies focus on neural methods that compute the latent representations of documents independent of the query. This allows the document embeddings to be precomputed. At query evaluation time, only the query embedding is computed by evaluating the corresponding portion of the deep neural model. This is followed by an approximate nearest-neighbour search over the collection-using the precomputed document embeddings. This approaches typically achieve significantly poorer retrieval performance compared to traditional IR methods-and generally need to be combined with classical IR functions [363,364].
In this chapter, we describe a different approach-that assumes query term independence-to leverage state-of-the-art neural ranking models for retrieval over the full collection. Based on our initial study, we posit there the is significant opportunity to use neural methods in combination with impact-ordered inverted index [366][367][368]. These data structures employ score quantization for efficient retrieval.
In the second half of this chapter, we propose a method to learn appropriate quantization schemes that optimize for retrieval effectiveness.
In this study, we incorporate the QTI assumption into three state-of-the-art neural ranking models-BERT, Duet, and CKNRM-and evaluate their effectiveness on the MS MARCO passage ranking task [48]. We surprisingly find that the two of the models suffer no statistically significant adverse affect w.r.t. ranking effectiveness on this task under the query term independence assumption. While the performance of BERT degrades under the strong query term independence assumptionthe drop in MRR is reasonably small and the model maintains a significant performance gap compared to other non-BERT based approaches. We conclude that at least for a certain class of existing neural IR models, incorporating query term independence assumption may result in significant efficiency gains in query evaluation at minimal (or no) cost to retrieval effectiveness.

Related work
Several neural IR methods-e.g., [117,172,177,184]-already operate under query term independence assumption. However, recent performance breakthroughs on many IR tasks have been achieved by neural models [21,121,256,257,348] that learn latent representations of the query or inspect interaction patterns between query and document terms. In this work, we demonstrate the potential to incorporate query term independence assumption in these recent representation learning and interaction focused models.
Some neural IR models [118,311,372] learn (dense low-dimensional or sparse high-dimensional) vector representations of document that can be computed independently of the query. The query-document relevance is then estimated as simple similarity functions (e.g., cosine or dot-product) of the learned representations.
These models are also amenable to precomputation of document representations and fast retrieval using approximate nearest neighbor search [373]-or even traditional IR data structures [365]. However, these approaches do not work when the model architecture incorporates early interactions between query and document representations-e.g., [21,121,192,348]. The approach proposed in this study allows for interactions between individual query terms and documents.

Model
IR functions that assume QTI observe the following general form: is the set of positive real-valued scores as estimated by the relevance model corresponding to documents d ∈ C in collection C w.r.t. to terms t ∈ V in vocabulary V -and S q,d denotes the aggregated score of document d w.r.t.
to query q. For example, in case of BM25 [76]: Where, tf and idf denote term-frequency and inverse document frequency, respectively-and k 1 and b are the free parameters of the BM25 model.
Deep neural models for ranking, in contrast, do not typically assume QTI. Instead, they learn complex matching functions to compare the candidate document to the full query. The parameters of such a model ϕ is typically learned discriminatively by minimizing a loss function of the following form: We use d + and d − to denote a pair of relevant and non-relevant documents, respectively, w.r.t. query q. The instance loss ℓ in Equation 5.8 can take different forms-e.g., ranknet [189] or hinge [221].
Given a neural ranking model ϕ , we define Φ-the corresponding model under the QTI assumption-as: The new model Φ preserves the same architecture as ϕ but estimates the relevance of a document independently w.r.t. each query term, as shown in Figure 5. 1.
The parameters of Φ are learned using the modified loss:  Given collection C and vocabulary V , we precompute ϕ t,d for all t ∈ V and d ∈ C. In practice, the total number of combinations of t and d may be large but we can enforce additional constraints on which ⟨t, d⟩ pairs to evaluate, and assume no contributions from remaining pairs. During query evaluation, we can lookup the precomputed score ϕ t,d without dedicating any additional time and resource to evaluate the deep ranking model. We employ an inverted index, in combination with the precomputed scores, to perform retrieval from the full collection using the learned relevance function Φ. We note that several IR data structures assume that ϕ t,d be always positive which may not hold for any arbitrary neural architecture.
But this can be rectified 1 by applying a rectified linear unit [346] activation on the model's output.

Task description
We study the effect of the QTI assumption on deep neural IR models in the context of the MS MARCO passage ranking task [48]. We find this ranking task to be suit- Conveniently, several of these high-performing models include public implementations for the ease of reproducibility.
Two comparable benchmarks include the TREC CAR [28,374]  Note that the thousand passages per query were originally retrieved using BM25 from a collection that is provided as part of the MS MARCO dataset. This allows us to also use this dataset in a retrieval setting-in addition to the re-ranking setting used for the official challenge. We take advantage of this in our study.

Baseline models
We begin by identifying models listed on the MS MARCO leaderboard that can serve as baselines for our work. We only consider the models with public implementations. We find that a number of top performing entries-e.g., [121]-are based on recently released large scale language model called BERT [281]. The BERT based entries are followed in ranking by Duet and CKNRM. Therefore, we limit this study to BERT, Duet, and CKNRM.
BERT Nogueira and Cho [121] report state-of-the-art retrieval performance on the MS MARCO passage re-ranking task by fine tuning BERT [281] pretrained models.
In this study, we reproduce the results from their paper corresponding to the BERT Base model and use it as our baseline. Under the term independence assumption, we evaluate the BERT model once per query term-wherein we input the query term as sentence A and the passage as sentence B.
Duet We employ the Duet variant described in Section 4. 4

.1 for this study.
CKNRM The CKNRM model [348] combines kernel pooling based soft matching [349] with a convolutional architecture for comparing n-grams. CKNRM uses kernel pooling to extract ranking signals from interaction matrices of query and passage n-grams. Under the query term independence assumption, the model considers one query term at a time-i.e., the interactions between individual query unigrams and passage n-grams. We use a public implementation 3 of the model in our study.  Note that all three models emphasize on early interactions between query and document representations-unlike other prior work [118,372] where the interaction is limited to the final stage. Under the QTI assumption, we allow early interaction between individual query terms and document, but delay the full query-document interaction till the end. Our observation that delaying the query-document interaction has no significant impact on effectiveness of these interaction-based models is

Results
a key finding of this study.
We posit that models with query term independence assumption-even when slightly less effective compared to their full counterparts-are likely to retrieve better candidate sets for re-ranking. To substantiate this claim, we conduct a smallscale retrieval experiment based on a random sample of 395 queries from the test set. We use the Duet model with the query term independence assumption to precompute the term-passage scores constrained to (i) the term appears at least once in the passage, and (ii) the term does not appear in more than 5% of the passage collection. Table 5.2 compares Duet and BM25 on their effectiveness as a first stage retrieval method in a potential telescoping setting [82]. We observe a 6.25% improvement in recall@1000 from Duet over the BM25 baseline. To perform similar retrieval from the full collection using the full Duet model, unlike its query-termindependent counterpart, is prohibitive because it involves evaluating the model on every passage in the collection against every incoming query.

Conclusion
The emergence of compute intensive ranking models, such as BERT, motivates rethinking how these models should be evaluated in large scale IR systems. The approach proposed in this paper moves the burden of model evaluation from the query evaluation stage to the document indexing stage. This may have further consequences on computational efficiency by allowing batched model evaluation that more effectively leverages GPU (or TPU) parallelization.
This preliminary study is based on three state-of-the-art deep neural models on a public passage ranking benchmark. The original design of all three models-BERT, Duet, and CKNRM-emphasize on early interactions between query and passage representations. However, we observe that limiting the interactions to passage and individual query terms has reasonably small impact on their effectiveness.
These results are promising as they support the possibility of dramatically speeding up query evaluation for some deep neural models, and even employing them to retrieve from the full collection. The ability to retrieve-and not just re-rankusing deep models has significant implications for neural IR research. Any loss in retrieval effectiveness due to incorporating strong query term independence assumptions may be further recovered by additional stages of re-ranking in a telescoping approach [82].
This study is focused on the passage ranking task. The trade-off between effectiveness and efficiency may be different for document retrieval and other IR tasks.
Traditional IR methods in more complex retrieval settings-e.g., when the document is represented by multiple fields [99]-also observe the query term independence assumption. So, studying the query term independence assumption in the context of corresponding neural models-e.g., [345]-may also be appropriate. We note these as important future directions for our research.
The findings from this study may also be interpreted as pointing to a gap in our current state-of-the-art neural IR models that do not take adequate advantage of term proximity signals for matching. This is another finding that may hold interesting clues for IR researchers who want to extract more retrieval effectiveness from deep neural methods.

Chapter 6
Stochastic learning to rank for target exposure Retrieval systems mediate what information users are exposed to and consume. A typical large collection may contain several documents that are relevant, albeit to varying degrees, to a user's query. Because users rarely inspect all retrieved results exhaustively, the IR system must prioritize what documents are exposed more than others to maximize the chances of user satisfaction. The need for this prioritization is often operationalized by formulating retrieval as a ranking task, as we have also assumed in previous chapters. Consequently, this assumption that the system produces a ranked list of results is often also baked into the design of many information access interfaces. A common form of presentation involves displaying a vertical (or sometimes horizontal) result list. Even sophisticated visual interfaces, such as grid layouts, or non-visual interaction modes, as in the case of voice-based search, may assume that the backend retrieval system returns a ranked list of results which determines how prominently they should be displayed. Across these different modalities, the probability that the user inspects a certain result depends on its display position [69,376] and size [377] among other factors, which in turn may be determined by the document's rank in the results list.
A static ordering by estimated relevance makes sense if we assume: (i) the IR system is only concerned about satisfying the user performing the search, and (ii) all relevant documents are equivalent from the user's perspective and therefore the user's interests are best served by ordering retrieved documents strictly by their estimated relevance. In many real-life IR scenarios, however, the system must also care about document and producer-side fairness [84,85,378]. For example, in web search we may want the IR system to give equal exposure to documents of comparable relevance-which may directly impact their monetization and other value that producers can extract from content exposure. When documents correspond to different demographics like gender or race-e.g., candidate profiles on a job application website-parity of exposure across demographics may be important from fairness and legal concerns. In scenarios where the system produces a ranking of service providers, such as booking a hotel or hailing a ride [379], distributing exposure across multiple providers may be necessary to avoid producer starvation or overload. When retrieved documents have comparable relevance but contain different information, then balanced exposure may increase diversity of consumption and help mitigate phenomenon like filter bubbles [380].
In these scenarios, a single fixed ranking makes less sense. Instead, it may be more meaningful for the system to present different randomized permutations of comparably relevant documents to distribute exposure more fairly among them.
Such stochastic ranking policies provide a framework for optimizing how exposure is distributed in expectation. In Chapters 4 and 5, we adopted the narrow view that it is sufficient to learn a relevance model whose estimates are appropriate for generating single static ordering of results. In contrast, in this chapter we shift our focus to optimizing models that produce relevance estimates appropriate for generating different permutations of results that minimizes the deviation of expected exposure from a specified target distribution. Our main contribution here is to adapt the learning to rank [20] framework for direct optimization towards target exposure.

Related work
In the learning to rank [20] literature, several optimization objectives have been proposed that can be broadly categorized into: (i) pointwise, (ii) pairwise, and (iii) listwise loss functions. Because the exposure of a document is a function of its rank in the result list, our optimization goals are better served by the listwise formulation.
Several listwise loss functions [231,232] operationalize the idea of deriving the probability of a rank ordering given the score distribution over documents using the Plackett-Luce model [228,229]. It is noteworthy, that enumerating all distinct document permutations can be computationally challenging even for a moderately sized set of candidates. More recently, Bruch et al. [381] demonstrated a mechanism for sampling rankings from the Plackett-Luce distribution using the reparameterization trick [351] that is amenable to gradient-based optimization. Their approach involves adding independently drawn noise samples from the Gumbel distribution [382] and then deriving the approximate rank of the document following the method proposed by Qin et al. [383] and Wu et al. [384]. While not developed in the context of deploying stochastic ranking models, we adopt a similar methodology herein in our framework.
Our work is at the intersection of learning to rank optimization and expected exposure metrics. For the latter, we operationalize the framework proposed by Diaz et al. [32]. In Section 6.2, we provide a brief primer on this topic.

Expected exposure metrics
Given an information need q, Diaz et al. [32] define the expected exposure ε d of document d as: Where, σ is a ranking of documents in the collection, sampled from π q , a probability distribution over all possible permutations of documents conditioned on q. We use µ(d|σ ) to denote the conditional probability of exposure of document d given ranking σ . To compute µ(d|σ ), we can adopt any arbitrary user behavior model [385] that defines how the user interacts with the presented rank list. For example, the rank-biased precision (RBP) [386] metric assumes that a user's probability of visiting a position decreases exponentially with rank.
Where, ρ σ ,d is the rank of the document d in σ -and γ is the patience parameter that controls how deep in the ranking the user is likely inspect. We adopt this RBP user behavior model in this study but note that this analysis can be easily extended to more elaborate browsing models like the cascade model [387].
Plugging in the RBP user-model into Equation 6.1 we get: Diaz et al. [32] further define a metric that quantifies the deviation between the expected exposure vector ε corresponding to all documents in the collection under The target exposure ε * specifies the ideal behaviour we desire from our retrieval system. One way to compute this is by assuming some oracle ranking policy.
For example, in this work we adopt the principle of equal expected exposure defined by Diaz et al. [32] as: Given a fixed information need, no item should be exposed (in expectation) more or less than any other item of the same relevance.
Under this ideal policy, documents always appear in ranking above other documents of lower grades, and documents in the same grade are permuted with uniform probability. Let m g be the number of documents with relevance grade g and m >g the number of documents with relevant grade strictly larger than g. Given an RBP browsing model, the optimal exposure for a document d with grade g is, We refer the readers to the original paper for more detailed derivation and discussion of this individual exposure parity target.
If we associate the documents in our collection with a set A of k attributes, then we can also define a group notion of exposure parity. These attributes may reflect, for example, demographic information about the content producer or some topical grouping by content. Let A be a n × k binary matrix mapping each of the n documents in the collection to their group identity. We can then compute the total exposure for all documents with an attribute as ξ = A ⊺ ε. We recover equal exposure across groups, by enforcing ξ to be uniform. We can replace ε with ξ in Equation 6.4 to define as a measure of equal exposure across groups. Other notions of demographic and group fairness can be similarly derived.

Optimizing for target exposure
Following the Plackett-Luce model [228,229], given some arbitrary score distribution y over documents, we can sample different rankings by iteratively sampling documents without replacement based on the following softmax distribution.
Under the assumption of binary relevance and a perfect relevance estimator, Plackett-Luce randomization should perform optimally. However, learning to rank models are not perfect estimators of relevance. Therefore, we believe there should be some advantage to optimizing directly for expected exposure. We leverage recent results in optimization of relevance-based objectives computed over a distribution over rankings [381]. Our results can be seen as an extension of this framework to individual and group exposure objectives.
We focus on a shared model architecture with varying loss functions in order to measure differences due to the objective alone, instead of artifacts resulting from the functional form of the models. We begin by describing how we optimize for expected exposure before proceeding to our experiment design and empirical results.

Individual exposure parity
Although optimizing for pointwise or pairwise loss has been well-studied in the information retrieval community, directly optimizing for a metric based on a distribution over rankings has received less attention.
We begin by defining an appropriate loss function for our model. Turning to Equation 6.5, we can drop the constant term and add a hyperparameter to trade-off disparity and relevance, Where ε * is based on graded relevance.
Let f θ : D → R be a document scoring function parameterized by θ . Given a query, y is a n × 1 vector of document scores for the entire collection such that, Using a Plackett-Luce model, we can translate the raw scores into sampling probabilities as in Equation 6. 8. This allows us to construct a ranking σ by sampling documents sequentially. Unfortunately, this sampling process is nondifferentiable and, therefore, prohibitive to a large class of models, including those that learn by gradient descent. We address this by adopting the method proposed by Bruch et al. [381]. To construct a sampled ranking σ , we reparameterize the probability distribution by adding independently drawn noise samples G from the Gumbel distribution [382] to y and sorting documents by the "noisy" probability distributionp,p Where G i is sample from the Gumbel distribution.
U ∼ Uniform(0, 1) ( 6.12) Given the perturbed probability distributionp, we compute each document's smooth rank [383,384] as, The smooth rank is sensitive to the temperature τ [388]. At high temperatures the smooth rank is a poor approximation of the true rank and at low temperatures may result in vanishing gradients. To rectify this issue, we employ the straightthrough estimator [389] to compute the true ranks in forward pass but differentiate the gradients with respect to the smooth ranks during backpropagation.
Using the estimated ranks and a specified user model we compute the exposure for each document. For example, assuming RBP as the user model the exposure of document d from a single ranking σ is given by ε d = γ (ρ σ ,d −1) . We compute expected exposure by averaging over n train different rankings-each generated by independently sampling different Gumbel noise in Equation 6. 10.
We use this expected exposure vector ε in Equation 6.9 to compute the loss that we minimize through gradient descent, as shown in Figure 6.1. The relevance grades are not used for training beyond computing target exposure. We set τ in Equation 6.13 to 0.1.

Group exposure parity
We can also adapt this model to optimize fpr group-level exposure parity. To do so, we replace ∥ε∥ 2 2 with ∥ξ ∥ 2 2 in Equation 6.9 to define an optimization objective that trades-off relevance and group parity.
This loss function assumes that the ideal policy distributes exposure equally across all groups. Optimization objectives corresponding to other group exposure criteria can be derived similarly in future work.

Models
We restrict our choice of baselines to neural networks so that the exposure-based optimization can be compared to baseline ranking loss functions with respect to the To sample multiple rankings proportional to the softmax distribution over document scores, we first add independently sampled noise from the Gumbel distribution to the scores and then estimate corresponding smooth rank values. We then compute exposure of each document for a given ranking based on a preselected user model. Next, we estimate the expected exposure of a document by averaging across multiple rankings. Finally, we compute the loss between the predicted and the target expected exposure vectors, which can be then minimized using gradient-based methods as every step in the above process is differentiable.
are derived by employing Plackett-Luce sampling over two deterministic policies (pointwise and pairwise models) with varying softmax temperatures to obtain different trade-off points between disparity and relevance. We set n train to 20 for our model and n test to 50 for all models.
We consider three training objectives in our experiments. The pointwise model [194] minimizes the squared error between the model prediction and true relevance. The pairwise model [189] minimizes misclassified preferences using a cross-entropy loss. The expected exposure model minimizes the loss in Equation 6.9 and, in our group parity experiments, Equation 6.14.

Data
Our experiments use the MSLR-WEB10k dataset [390], a learning-to-rank dataset containing ten thousand queries. We perform five-fold cross validation (60/20/20 split between training, validation, and testing sets). Each query-document pair is represented by a 136-dimensional feature vector and graded according to relevance on a five-point scale.
For the group parity experiments, as there are no obvious appropriate group attributes in the MSLR-WEB10k dataset, we discretize the PageRank feature in the ranges <1000, 1000-10000, and ≥10000 and treat it as a group attribute. The choice of using discretized PageRank as a group attribute is rather arbitrary, but we confirm that this discretization scheme is reasonable as roughly 70% of the queries have at least one document corresponding to each group with a relevance grade greater than one.

Evaluation
We use a γ = 0.50 for all of our experiments, as consistent with standard TREC evaluation protocol. RBP is evaluated at depth 20.

Results
We present the results of our experiments in Table 6. 1. In terms of expected exposure, we do not observe a difference in performance between pointwise and pairwise Table 6.1: Results for optimizing towards individual and group parity using different ranking objectives. We report average EE-AUC for both tasks and highlight the best performance for each in bold. Optimizing directly for individual and group parity using our proposed method achieves best performance in both cases.
Loss function AUC Individual parity Group parity Pointwise loss 0.229 0.112 Pairwise loss 0.229 0.108 Our methods Expected exposure loss (Eqn. 6.9) 0.238 0.141 Group parity loss (Eqn. 6.14) 0.178 models. However, directly optimizing for expected exposure resulted in a 3.9% improvement in EE-AUC over the pointwise and pairwise models. We confirm that the difference in EE-AUC follows a normal distribution and accordingly perform a paired student's t-test to check their statistical significance. The EE-AUC differences between our proposed method and the baselines are statistically significant In terms of group parity, we observe a difference in performance between pointwise and pairwise models. Moreover, directly optimizing for expected exposure results in improved performance while directly optimizing for group parity further boosts performance. The gap in EE-AUC between all pairs of models are statistically significant (p < 0.01).
These results, while based on a limited study, indicate that direct optimization for expected exposure metrics is both viable in the learning to rank framework as well as useful for optimization under fairness constraints.

Conclusion
An exposure-based view of retrieval explicitly codifies the role that IR systems play as intermediary in two-sided marketplaces consisting of users seeking information and documents (or their producers). Stochastic ranking policies allow for more balanced distribution of exposure over multiple rankings. In this work, we demonstrate that these policies can be directly optimized to reduce deviation from specified target exposure distribution. While our work is grounded in parity of individual and group exposure, the framework described is flexible enough to incorporate any arbitrary target exposure policy beyond fairness constraints-e.g., based on topical diversity considerations or to maximize monetization in the context of paid search.
Our definition of target exposure in this work is based on a universal notion of relevance. If the relevance of a document instead changes based on the searcher (i.e., personalization) or other context (e.g., location), then our framework needs to be appropriately extended. Exposure can also be nuanced by user attributes. For example in commercial searches, exposure to users with an intent to purchase may be weighted differently than to users who may be casually browsing. We believe that there is a rich space for exploring different extensions of our proposed framework.
Deploying stochastic ranking policies may also come with its own unique challenges. For example, randomized rankings may have unintended consequences on the system's caching mechanisms. It may also make it harder for users to re-find information [391] they have previously discovered for a query. More detailed studies are also necessary to understand the differential impact of stochastic policies on queries of varying difficulty, especially on queries for which the model's relevance estimates are highly uncertain.

Chapter 7
Learning to Rank for Query

Auto-Completion
In this chapter, we discuss the application of deep architectures to the query autocompletion task that presents different challenges than ad hoc retrieval. Query autocompletion helps the user of a search system to formulate their information request by recommending queries based on their partially typed query. The query autocompletion system typically considers the user history, the task context the location and temporal context, and other information to make more relevant recommendation. The ranking task, in case of query auto-completion, therefore, involves ranking either query suffixes or full query candidates in response to a query prefix. In this chapter, we discuss work in which we employ deep neural networks for that ranking task.

Query Auto-Completion for Rare Prefixes
As users enter their query into the search box, most modern search engines provide a ranked list of query suggestions based on the current prefix already typed by the user. In a typical approach used by many query auto-completion (QAC) systems, candidate queries are identified by doing an exact prefix lookup against a fixed set of popular queries, using a data structure such as a prefix tree [95]. The candidates are then ranked by their expected likelihood, which is typically computed as a function of its past popularity (commonly referred to as the MostPopularCompletion Table 7.1: Synthetic QAC candidates generated by the suffix-based approach and ranked using only the CDSSM similarity feature. The CDSSM model projects both the prefix and the suffix to a common 128-dimensional space allowing us to rank according to prefix-suffix cosine similarity. One of the lower quality synthetic candidates "cheapest flights from seattle to airport" is ranked seventh in the second list. (MPC) model [392]). Such a system can only suggest queries with enough historic popularity to make it into the prefix tree.
We propose an additional candidate generation strategy for QAC by mining popular query suffixes. Candidate suffixes are popular n-grams that appear at the ends of queries. By appending such n-grams suffixes to a user's query prefix we can generate synthetic suggestion candidates that have never been observed in the historical query logs.
pc returns the set of queries that qualify as completions for the query q and lf is the frequency of the query in the search logs.
Language modelling based approaches for sentence completion have been studied in the context of e-mail and document authoring [393][394][395][396]. In Web search, White and Marchionini [397] and Fan et al. [398] proposed models for term recommendations to aid users in their query formulation process. Bhatia et al. [399] extracted frequently occurring phrases from document corpus and used them to gen-

Model
For document retrieval, Shen et al. [317] demonstrated that discriminatively training a deep neural network model with a convolutional-pooling structure on clickthrough data can be effective for modelling query-document relevance. We adopt the CDSSM by training on a prefix-suffix pairs dataset (instead of query-document titles). The training data for the CDSSM is generated by sampling queries from the search logs and splitting each query at every possible word boundary. For example, from the query "breaking bad cast" we generate the two pairs ("breaking", "bad cast") and ("breaking bad", "cast"). The architecture shown in Figure 7.1 is used on both the prefix and the suffix side of the CDSSM model.
It is important to emphasize that our earlier discussion in Section 3.2.2 on different notions of similarity between terms that can be learnt by shallow embedding models is also relevant in the context of these deeper architectures. In the case of CDSSM [185], the notion of similarity being modelled depends on the choice of

Method
We propose two key ideas in this section. Firstly, we generate synthetic query suggestion candidates for QAC using popular query suffixes. We then introduce n-gram and CDSSM based features in a supervised learning setting to rank these synthetic suggestions alongside the full-query suggestion candidates.
Candidate Generation From every query in the search engine logs we generate all possible n-grams from the end of the query. For example, from the query "bank of america" we generate the suffixes "america", "of america" and "bank of america".
By aggregating across all queries we identify the most popular suffixes. Table 7.3 shows the most frequently observed query suffixes in the publicly available AOL logs [79].
Next, for a given prefix we extract the end-term as shown in Figure 7. 2. We match all the suffixes that start with the end-term from our precomputed set. These selected suffixes are appended to the prefix to generate synthetic suggestion candidates. For example, the prefix "cheap flights fro" is matched with the suffix "from cheapest flight fro End-term: "fro" cheapest flight from End-term: "from" cheapest flight from End-term: "from " cheapest flight from n End-term: "n" seattle" to generate the candidate "cheap flights from seattle". Note that many of these synthetic suggestion candidates are likely to not have been observed by the search engine before.
We merge these synthetic suggestions with the set of candidates selected from the list of historically popular queries. This combined set of candidates is used for ranking as we will describe in Sec 7. 1.4.

Ranking Features
For every prefix and suggestion candidate (synthetic or previously observed), we compute a set of common features for the supervised ranking model. We describe these features in this section, focusing on the n-gram and CDSSM features that we propose for this setting.
• N-gram based features. Given a candidate suggestion q, we compute features based on the frequency of n-grams f ngram i of different lengths (for i = 1 to 6).
where, ng i (q) is the set of all n-grams in query q of length i. lf(g) is the observed frequency of the n-gram g in the historical query logs. These ngram features model the likelihood that the candidate suggestion is generated by the same language model as the queries in the search logs.
• CDSSM based features. Given a prefix p and a suggestion candidate c, we extract a normalized prefixp by removing the end-term from the prefix. Then a normalized suffixs is extracted by removingp from the query c. Then we use the trained CDSSM model to project the normalized prefix and the normalized suffix to a common 128-dimensional space and compute a f cdssm feature.
where ⃗ vp and ⃗ vs are the CDSSM vector outputs corresponding top ands, respectively.

Experiments
Our experiment setup is based on the learning to rank framework proposed by Shokouhi [402]. We generate all possible prefixes 1 from each query impression to use for training, validation and testing. For each prefix we identify the set of candidate suggestions as described in Section 7.1. 3. We associate a positive relevance judgment with the candidate that matches the original query from which the prefix was extracted. To accurately measure the coverage impact of our approach we retain all prefix impressions where the submitted query is not in the list of candidates available for ranking.
We train LambdaMART [403] models for ranking the suggestions using features described in Section 7.1. 3. We limit our ranking task to instances where the prefix contains at least one complete word, since completions with very short prefixes is already well solved by our popularity-based features and we are focusing on rare prefixes. We always train 300 trees (with early stopping using a validation set) and evaluate the model performances on the test set using the mean reciprocal rank (MRR) metric.
We conduct all our experiments on the publicly available AOL query logs [79] and reproduce the same results on the large-scale query logs of the Bing search engine. We refer to these two datasets hereafter as the AOL testbed and the Bing testbed, respectively.   are trained with the suffix-based candidates in the training data.

Results
The models with the f cdssm feature perform better than the corresponding models without the feature across all experiments. However, in general the f ngram i features seems to be showing higher improvements compared to the CDSSM based feature. We hypothesize that the f cdssm feature is less precise than the f ngram i features. For example, we can see in Table 7.1 that the CDSSM based feature ranks a suffix highly that generates a semantically meaningless query suggestion "cheapest flight from seattle to airport". While "airport" is a location that you can take a flight to, in the context of the given prefix it is clearly an inappropriate suggestion. It is possible that the prefix-suffix pairs based training of the CDSSM can be further improved. We believe that this is an important area for future investigations given that the CDSSM holds certain other advantages over n-gram models. For example, the  both models are higher for rare prefixes than for the frequent ones. One factor in this is that rare prefixes tend to be longer and therefore more specific, giving fewer candidates to rank and making it easier to achieve good MRR.

Conclusion
We proposed a novel candidate generation technique for query auto-completion by mining and ranking popular query suffixes. Our empirical study shows that this is an effective strategy for significantly improving MRR for rare and unseen prefixes.
The supervised ranking framework proposed in this paper is generic and can be employed in any QAC system that combines multiple sources of candidates. We described features based on n-gram language models and convolutional neural networks with demonstrable improvements.
While we have shown significant improvements in MRR using synthetic candidate generation, we have not measured how often this approach generates semantically meaningless synthetic suggestions and have not quantified the effect of showing synthetic suggestions to search users. A user study on this aspect is left as future work. There is also further scope for exploring other language models (such as recurrent neural networks) in the context of this task.

Session Context Modelling for Query Auto-Completion
Short-term user history provides useful cues about the user intent that an IR system can consider to improve the relevance of retrieved results [404]. In QAC systems, in particular, when only a few characters have been entered the search engine has little understanding of the actual information need of the user and the generic suggestions provided by a non-contextual QAC system typically perform poorly [392].
The high ambiguity associated with short prefixes makes QAC a particularly interesting candidate for leveraging any additional information available about the user's current task. The same study also showed that 49% of Web searches are preceded by a different search which can be used to gain additional insights into the user's current information need.
The majority of previous work [405,406] on using short-term user history for search personalization has been focused on modelling the topical relevance of the candidate results (documents or query suggestions) to the previous queries and viewed documents in the same search session. Using such implicit feedback has been shown to be a very attractive strategy for improving retrieval performance when the user intent is ambiguous. For example, knowing that the user's previous query was "guardians of the galaxy" can help to inform a QAC system to promote the query "imdb" in ranking over "instagram" when the user has just typed "i" in the search box. Query reformulation behaviours within search sessions have also been studied but are mostly limited to taxonomy based classifications [407,408] and models based on syntactic changes [409]. A quick study of a sample of Bing's search engine logs reveal that users frequently search for "san francisco 49ers" and "san francisco weather" immediately after searching for "san francisco". Similarly, the query "detroit" is often followed by the queries "detroit lions" and "detroit weather".
Intuitively, "san francisco" → "san francisco 49ers" represents a similar shift in user's intent as "detroit" → "detroit lions". We can see many such frequently occurring patterns of reformulations in large scale search logs. Modelling these reformulations using lexical matching alone is difficult. For example, we understand that "movies" → "new movies" is not the same intent shift as "york" → "new york" even though in both cases the same term was added to both the queries by the user. On the other hand, "london" → "things to do in london" and "new york" → "new york tourist attractions" are semantically similar although the two reformulations involve the addition of completely disjoint sets of new terms to the queries.
In text processing, Mikolov et al. [158]  context features based on these vector representations can significantly improve the QAC ranking over the supervised ranking baseline proposed by Shokouhi [402].
The main contributions of the work described in this section are, • Demonstrating that query reformulations can be represented as lowdimensional vectors which map syntactically and semantically similar query changes close together in the embedding space.
• Using features based on the distributed representations of queries and reformulations to improve upon a supervised ranking baseline for session contextaware QAC ranking. Our experiments on the large-scale query logs of the Bing search engine and the publicly available AOL query logs [79] show that these features can improve MRR by more than 10% on these testbeds.
• Demonstrating that CDSSM trained on session query pairs performs significantly better for the contextual QAC ranking task compared to the CDSSM model trained on clicked query-document pairs.
Next, we review related work that are relevant to this study.

Related work
In Web search, Bennett et al. [404] investigated the impact of short-term and longterm user behaviour on relevance prediction, and showed that short-term user history becomes more important as the session progresses. Li  Yan et al. [411] proposed an approach that maps queries and clicks to latent search intents represented using Open Directory Project 3 categories for making context-aware query recommendations. Cao et al. [406] and Liao et al. [412] have explored session context using latent concept clusters from click-through bipartite graphs, while Guo et al. [413] represented the user's previous queries using a regularized topic model. Zhang  Previous studies on the relationships between neighbouring queries from a search session have been mostly focused on categorizing the reformulations based on broad manually defined taxonomies (e.g., generalization, specialization, error correction and parallel move) [417] or understanding the user goals behind common actions (e.g., addition, removal or substitution of terms) [418]. Motivated by the broad manually identified reformulation categories Xiang et al. [419] and Jiang et al. [420] designed simple features for supervised retrieval models. Finally, Guan et al. [409] use reinforcement learning for modifying term weights in response to the observed modifications made to the query by the user.
While clearly using session context for Web search is a well-studied topic, context-sensitive query auto-completion has been discussed less thoroughly in the literature. Weber and Castillo [421] and Shokouhi [402] showed how query distributions change across different user demographics and argued that QAC systems based on personalization features can significantly outperform popularity-based baselines. Ranking suggestions based on temporal context has also been explored [422,423].
The two QAC related studies most relevant to our work have been done by Shokouhi [402] and Kharitonov et al. [424]. To capture short-term context, Shokouhi [402] relied on letter n-gram matches between the previous queries and the candidates, and trained a supervised ranking model for combining them with MPC and other non-contextual and user demographic features. Kharitonov et al. [424] proposed a unified framework for contextualizing and diversifying the ranking of QAC suggestions. Their empirical evaluations show that by considering the user's previous query alone more than 96% of the improvements can be achieved, as compared to additionally considering the document examination history and diversification context. Given the previous query, their proposed model computes the expected probability of a given completion as follows, Where c is an indicator variable whose value is 1 if the user continues the current task, and 0 otherwise. The two primary components of the above equation are P(q 1 ) and P(q 1 |c = 1, q 0 ), which correspond to the probability of observing the query q 1 globally and in the context of the query q 0 , respectively, in the query logs.
For our evaluation, we implement the supervised ranking framework proposed by Shokouhi and include the n-gram similarity, the query frequency and the query pairwise frequency features among others as described in Section 7.2.3.

Model
We adopt the CDSSM architecture proposed by Shen et al. [317] for our study.
Unless specified otherwise, for all models in this paper the window size for the convolutional layer is set to three and the dimensions of the output vector to 32. The training data for the CDSSM models consists of source-target text pairs.
The original DSSM [118] and CDSSM [317] models were trained on clickthrough data which consists of pairs of queries and document titles, corresponding to clicked results.
In addition to clickthrough data, we also train the CDSSM models on sampled pairs of queries from search logs that were observed in succession during user sessions. In the rest of this paper, we refer to this as the session pairs dataset. For a pair of observed queries q 1 and q 2 , if the dataset includes both the ordering q 1 → q 2 and q 2 → q 1 then we refer to it as the symmetric session pairs dataset, otherwise as asymmetric. The symmetric session pairs data is further randomly sub-sampled by half to keep the count of the training pairs in both the datasets comparable.
The session pairs datasets are extracted from the exact same user sessions from which the clickthrough data is generated. While this does not imply that the actual count of training pairs in these two types of datasets are equal, it does make the comparison more meaningful as it assumes the same amount of raw log data is examined for training both the types of models. In practice, however, we did observe the data sizes to be comparable across all three datasets during this study.
All the CDSSM models in this study are trained using mini-batch based stochastic gradient descent, as described by Shen et al. [317]. Each mini-batch consists of 1024 training samples (source-target pairs) and for each positive pair 100 negative targets are randomly sampled from the data for that source that were The CDSSM models project the queries to an embedding space with fixed number of dimensions. The semantic similarity between two queries q 1 and q 2 in this semantic space is defined by, Where ⃗ v q 1 and ⃗ v q 2 are the CDSSM vector outputs corresponding to the two queries,  [145] on continuous space language models for text processing, and gives us an intuitively understandable representation of query reformulations as their offset vectors in the embedding space. More specifically, we define the reformulation from query q 1 to q 2 as, Where ⃗ v q 1 and ⃗ v q 2 are the CDSSM vector embeddings of the two queries, respectively. This explicit vector representation provides a framework for studying frequently occurring query reformulation patterns. To illustrate this, we randomly sample approximately 65K pairs of queries that were observed in succession in Bing's logs. For each pair, we compute the offset vector using a CDSSM model.
We then run a simple k-means clustering (k = 100) and examine the top clusters.
Example reformulations from five of the biggest clusters are shown in Table 7. 5.
A further study of these reformulation vectors can reveal important insights about user behaviour, such as the popularity of certain reformulation patterns. For example, we randomly sampled 100,000 adjacent pairs of queries from Bing's logs that were observed in search sessions. Our analysis show that there are more pairs similar to the narrowing reformulation "new york" → "things to do in new york" in the sampled set, than its inverse. Similarly, the misspelling "fcebook" followed by "facebook" is a more commonly observed pattern than the other way around, as illustrated in Figure 7.5.
Next, we list qualitative examples in Table 7.6 to demonstrate the predictive aspect of these reformulation vectors. Similar to the analogy based test proposed by Mikolov et al. [145], these examples show that we can obtain intuitively understandable results by performing simple algebraic operations in the embedding space. For example, we compute the vector sum of the projections (normalized to their unit norm) of the queries "new york" and "newspaper".
Then from a fixed set of candidates we find the query whose embedding has the highest cosine similarity with ⃗ v target . For our analysis we picked the top one mil-

Experiments
Our empirical evaluations are based on the learning to rank framework proposed by Shokouhi [402] for personalized query auto-completions. In this setup, we learn a supervised ranking model based on training data generated from implicit user feedback. The output of the CDSSM models, described in the previous section, are used to generate additional features for this supervised ranking model. The baseline ranking model (henceforth referred to simply as the baseline model) contains both the non-contextual and the (non-CDSSM based) contextual features. We compare all models using the MRR metric, and the study is repeated on two different testbeds to further confirm the validity of the results.
Testbeds We conduct our experiments on a large scale search query dataset sampled from the logs of the Bing search engine. We also reproduce our results using the publicly available AOL query logs [79]. In the rest of this paper we refer to these two datasets as the Bing testbed and the AOL testbed, respectively.
• Learning to rank To generate the training, the validation and the test sets we sample query impressions from the corresponding portions of the logs. For each query impression, a prefix is generated by splitting the query at a randomly selected position 5 . For each prefix a positive relevance judgment is assigned to the suggestion candidate that matches the final submitted query and all the others are labelled as irrelevant.
The training data collected in the above process consists of labelled prefixquery pairs. With respect to the choice of learning-to-rank algorithms, we chose LambdaMART [403], a boosted tree version of LambdaRank [425], that won the Yahoo! Learning to Rank Challenge (2010) [426] and is considered as one of the For our final evaluation we report the Mean Reciprocal Rank of the submitted query averaged over all sampled impressions on each of the two testbeds. 5 The prefixes in our study are strictly shorter than the original query and limited to no more than 30 characters in length.

Features
The baseline contextual and non-contextual features, as well as the features based on the CDSSM outputs are described in this section.
• Non-contextual features The MostPopularCompletion (MPC) model is one of the baselines for our study. We also use the output of this model as a feature for the supervised ranking model. Other non-contextual features include the prefix length (in characters), the suggestion length (in both characters and words), the vowels to alphabets ratio in the suggestion, and a boolean feature indicating whether the suggestion contains numeric characters.
• N-gram similarity features We compute the character n-gram similarity (n=3) between the suggestion candidate and the previous queries from the same user session. This is an implementation of the short history features described by Shokouhi [402]. A maximum of 10 previous queries are considered.
• Pairwise frequency feature From the background data, we generate the top 10 million most popular adjacent pairs of queries observed in search sessions.
For a given impression, the previous query and the suggestion candidate pair is matched against this dataset and the corresponding frequency count is used as the feature value. If no matches are found, then the feature value is set to zero.
• CDSSM topical similarity features The CDSSM models are trained as de-   We also compare the improvements from the different models based on the length of the input prefixes. Bar-Yossef and Kraus [392] have previously reported that non-contextual QAC systems generally perform poorly when the user has typed only a few characters due to the obvious ambiguity in user intent. Figure 7.6 illus-

Discussion
We demonstrated significant improvements in the query auto-completion ranking task using the CDSSM based session context features. We now discuss potential implications of these vector representations on session modelling and list some of the assumptions and limitations of the evaluation framework used in this study. prediction accuracy for such session classification tasks.

Assumptions and limitations
We have based our empirical study on the supervised ranking framework proposed by Shokouhi [402]. In doing so, we inherit some of the assumptions in the designs of that framework. Firstly, we assume that the user has a pre-determined query in mind for input and would be satisfied if it appears in the QAC suggestions list. However Hofmann et al. [69] have shown that due to the high examination bias towards top-ranked results, sub-optimal QAC ranking can negatively affect the quality of the query submitted by the user. As many Web search engines implement some form of an auto-completion feature, it is likely that The generation of the prefixes also assumes that each query was typed completely by the user in a strictly left-to-right progression and the user is equally likely to examine and engage with the QAC system after each character is typed. In practice, however, users are often aided in the query formulation process (partially or completely) by various features of the search engine, such as QAC or related query recommendations. Users also often correct already entered text during the query formulation process. In these cases the generation of all possible prefixes from the submitted query does not accurately reflect the actual prefixes typed by the user.
Li et al. [431] and Mitra et al. [68] have also shown that user engagement with Lastly, Shokouhi [402] generates all the possible prefixes of each query in the log data. This results in an obvious over-representation of long prefixes in the generated datasets. To avoid this issue we extract a single prefix per query by splitting at a random position within the query.
Despite the different underlying assumptions, the framework proposed by Shokouhi [402] provides a reasonable setup to learn a baseline context-aware ranking model for QAC, and hence we adopt it for this study.

Conclusion
We have demonstrated that the distributed representation of queries by the CDSSM holds useful information about inter-query relationships. The reformulation vectors exhibit regularities that makes them interesting for modelling session context for query suggestion tasks. Our experiments show that using features based on the reformulation vectors improves MRR for QAC ranking over using features based on the query vectors alone. The best improvements, however, are achieved by the combination of features based on both these vector representations. We have also demonstrated that training the latent semantic models on session query pairs produces further improvements over the model trained on query-document pairs.
While the biggest improvements are observed on short prefixes, the ranking model containing the CDSSM based features perform better than the supervised ranking baseline on all the prefix length based segments. We have also studied the effects of considering different number of previous queries within the session for context and the number of dimensions used to represent the query and reformulation vectors on the model performance. While we evaluate these models on the query autocompletion ranking task, the features we described in this paper may also be useful for generating context sensitive related query recommendations and query rewriting.
Furthermore, by projecting documents to this same embedding space, future studies may be able to extend these contextual features to document ranking in Web search.
Lastly, the reformulation vectors provide an interesting framework for studying sessions and intent progressions. We anticipate that these distributed representations of queries, documents and reformulations will become more frequently used as tools for future studies on search personalization and session search.

TREC Deep Learning track
The TREC 2019 Deep Learning Track has two tasks: Document retrieval and passage retrieval. Each task has a dataset that is new to TREC, although the passage task is similar to the MS MARCO passage ranking leaderboard [48], but with a new test set in the TREC version with more comprehensive labeling. Both tasks are ad-hoc retrieval, meaning that there is a fixed document set, and the goal of the information retrieval system is to respond to each new query with results that would satisfy the querying user's information need. Ad-hoc retrieval is a very common scenario in real-world search applications and in TREC. Below we provide more detailed information about the document retrieval and passage retrieval tasks, as well as the datasets provided as part of these tasks.
Document retrieval task The first task focuses on document retrieval-with two subtasks: (i) Full retrieval and (ii) top-100 reranking.
In the full retrieval subtask, the runs are expected to rank documents based on their relevance to the query, where documents can be retrieved from the full document collection provided. This subtask models the end-to-end retrieval scenario.
Note, although most full retrieval runs had 1000 results per query, the reranking runs had 100, so to make the MAP and MRR results more comparable across subtasks we truncated full retrieval runs by taking the top-100 results per query by score.
In the reranking subtask, participants were provided with an initial ranking of 100 documents, giving all participants the same starting point. The 100 were retrieved using Indri [110] on the full corpus with Krovetz stemming and stopwords eliminated. Participants were expected to rerank the candidates w.r.t. their estimated relevance to the query. This is a common scenario in many real-world retrieval systems that employ a telescoping architecture [82,83]. The reranking subtask allows participants to focus on learning an effective relevance estimator, without the need for implementing an end-to-end retrieval system. It also makes the reranking runs more comparable, because they all rerank the same set of 100 candidates.
For judging, NIST's pooling was across both subtasks, and they also identified additional documents for judging via classifier. Further, for queries with many relevant documents, additional documents were judged. These steps were carried out to identify a sufficiently comprehensive set of relevant results, to allow reliable future dataset reuse. Judgments were on a four-point scale: [3] Perfectly relevant: Document is dedicated to the query, it is worthy of being a top result in a search engine.
[2] Highly relevant: The content of this document provides substantial information on the query.
[1] Relevant: Document provides some information relevant to the query, which may be minimal.
[0] Irrelevant: Document does not provide any useful information about the query.
Passage retrieval task Similar to the document retrieval task, the passage retrieval task includes (i) a full retrieval and (ii) a top-1000 reranking tasks.
In the full retrieval subtask, given a query, the participants were expected to retrieve a ranked list of passages from the full collection based on their estimated likelihood of containing an answer to the question. Participants could submit up to 1000 passages per query for this end-to-end retrieval task.
In the top-1000 reranking subtask, 1000 passages per query query were provided to participants, giving all participants the same starting point. The sets of 1000 were generated based on BM25 retrieval with no stemming as applied to the full collection. Participants were expected to rerank the 1000 passages based on their estimated likelihood of containing an answer to the query. In this subtask, we can compare different reranking methods based on the same initial set of 1000 candidates, with the same rationale as described for the document reranking subtask.
For judging, NIST's pooling was across both subtasks, and they also identified additional passages for judging via classifier. Further, for queries with many rel-evant passages, additional passages were judged. These steps were carried out to identify a sufficiently comprehensive set of relevant results, to allow reliable future dataset reuse. Judgments were on a four-point scale: [3] Perfectly relevant: The passage is dedicated to the query and contains the exact answer.
[2] Highly relevant: The passage has some answer for the query, but the answer may be a bit unclear, or hidden amongst extraneous information.
[1] Related: The passage seems related to the query but does not answer it.
[0] Irrelevant: The passage has nothing to do with the query.

Datasets
Both tasks have large training sets based on human relevance assessments, derived from MS MARCO. These are sparse, with no negative labels and often only one positive label per query, analogous to some real-world training data such as click logs.
In the case of passage retrieval, the positive label indicates that the passage contains an answer to a query. In the case of document retrieval, we transferred the passage-level label to the corresponding source document that contained the passage. We do this under the assumption that a document with a relevant passage is a relevant document, although we note that our document snapshot was generated at a different time from the passage dataset, so there can be some mismatch. Despite this, in the document retrieval task machine learning models seem to benefit from using the labels, when evaluated using NIST's non-sparse, non-transferred labels.
This suggests the transferred document labels are meaningful for our TREC task.
The passage corpus is the same as in MS MARCO passage retrieval leaderboard. The document corpus is newly released for use in TREC. Each document has three fields: (i) URL, (ii) title, and (iii) body text.    28 26 Learning Track website 1 . Interested readers are also encouraged to refer to [48] for details on the original MS MARCO dataset.

Results and analysis
Submitted runs A total of 15 groups participated in the TREC 2019 Deep Learning Track, with an aggregate of 75 runs submitted across both tasks.
Based run submission surveys, we classify each run into one of three categories: • nnlm: if the run employs large scale pre-trained neural language models, such as BERT [281] or XLNet [441] • nn: if the run employs some form of neural network based approach-e.g., Duet or using word embeddings [350]-but does not fall into the "nnlm" category • trad: if the run exclusively uses traditional IR methods like BM25 [76] and RM3 [114].
We further categorize runs based on subtask: • rerank: if the run reranks the provided top-k candidates, or • fullrank: if the run employs their own phase 1 retrieval system.
We find that only 21 (28%) submissions fall under the "rerank" category-while the remaining 54 (72%) are "fullrank". We also encouraged some participants to run strong traditional IR baselines, and submit them as additional runs under the "BASELINE" group.
Overall results Our main metric in both tasks is Normalized Discounted Cumulative Gain (NDCG)-specifically, NDCG@10, since it makes use of our 4-level judgments and focuses on the first results that users will see. To analyse if any of the fullrank runs recall more relevant candidates in phase 1 compared to those provided for the reranking subtask, we also report Normalized Cumulative Gain (NCG) at rank 100 and 1000 for the document and passage ranking tasks, respectively. We choose to report NCG because it discriminates between recalling documents with different positive relevance grades and is a natural complement to NDCG, our main metric. Although NCG is not officially supported by trec_eval, we confirm that it correlates strongly with the recall metric for these analysed runs.
Deep learning vs. traditional ranking methods An important goal of this track is to compare the performance of different types of model, using large human-labeled training sets, for the core IR task of ad-hoc search. Indeed this is the first time a TREC-style blind evaluation has been carried out to compare state-of-the-art neural and traditional IR methods.  NDCG@10 results, broken down by run type. Runs of type "nnlm", meaning they use language models such as BERT, performed best on both tasks. Other neural network models "nn" and non-neural models "trad" had relatively lower performance. More iterations of evaluation and analysis would be needed to determine if this is a general result, but it is a strong start for the argument that deep learning methods may take over from traditional methods in IR applications. Figure 8.2a plots the NDCG@10 performance of the different runs for the document retrieval task, broken down by model type. In general, runs in the category "nnlm" outperform the "nn" runs, which outperform the "trad" runs. The best performing run of each category is indicated, with the best "nnlm" and "nn" models outperforming the best "trad" model by 29.4% and 14.8% respectively.
The passage retrieval task reveals similar pattern. In Figure 8.2b, the gap between the best "nnlm" and "nn" runs and the best "trad" run is larger, at 37.4% and 23.7% respectively. One explanation for this could be that vocabulary mismatch between queries and relevant results is more likely in short text, so neural methods that can overcome such mismatch have a relatively greater advantage in passage retrieval. Another explanation could be that there is already a public leaderboard, albeit without test labels from NIST, for the passage task. Some TREC participants may have submitted neural models multiple times to the public leaderboard, and are well practiced for the passage ranking task.
In query-level win-loss analysis for the document retrieval task (Figure 8.3) the best "nnlm" model outperforms the best "trad" run on 36 out of 43 test queries (i.e., 83.7%). Passage retrieval shows a similar pattern in Figure 8. 4. Neither task has a large class of queries where the "nnlm" model performs worse. However, more iterations of rigorous blind evaluation with strong "trad" baselines, plus more scrutiny of the benchmarking methods, would be required to convince us that this is true in general.
Next, we analyze the runs by representing each run as a vector of 43 NDCG@10 scores. In this vector space, two runs are similar if their NDCG vectors are similar, meaning they performed well and badly on the same queries. Using t-SNE [442] we then plot the runs in two dimensions, which gives us a visualization where similar runs will be closer together and dissimilar results further apart. This method of visualizing inter-model similarity was first proposed by Mitra et al. [21] and we employ it to generate the plots in Figure 8. 5.
On both document and passage retrieval tasks, the runs appear to be first clustered by group-see Figures 8.5b and 8.5d. This is expected, as different runs from the same group are likely to employ variations of the same approach. In Figures 8.5a and 8.5c, runs also cluster together based on their categorization as "nnlm", "nn", and "trad".
End-to-end retrieval vs. reranking. Our datasets include top-k candidate result lists, with 100 candidates per query for document retrieval and 1000 candidates per query for passage retrieval. Runs that simply rerank the provided candidates are "rerank" runs, whereas runs that perform end-to-end retrieval against the corpus, with millions of potential results, are "fullrank" runs. We would expect that a "fullrank" run should be able to find a greater number of relevant candidates than we provided, achieving higher NCG@k. A multi-stage "fullrank" run should also be able to optimize the stages jointly, such that early stages produce candidates that later stages are good at handling.
According to Figure 8.6, "fullrank" did not achieve much better NDCG@10 performance than "rerank" runs. While it was possible for "fullrank" to achieve better NCG@k, it was also possible to make NCG@k worse, and achieving significantly higher NCG@k does not seem necessary to achieve good NDCG@10.
Specifically, for the document retrieval task, the best "fullrank" run achieves only 0.9% higher NDCG@10 over the best "rerank' run. For the passage retrieval  Figure 8.3: Comparison of the best "nnlm" and "trad" runs on individual test queries for the document retrieval task. Queries are sorted by difference in mean performance between "nnlm" and "trad"runs. Queries on which "nnlm" wins with large margin are at the top.  Figure 8.4: Comparison of the best "nnlm" and "trad" runs on individual test queries for the passage retrieval task. Queries are sorted by difference in mean performance between "nnlm" and "trad"runs. Queries on which "nnlm" wins with large margin are at the top. (d) By group name on passage retrieval task Figure 8.5: Visualizing inter-run similarity using t-SNE. Each run is represented by a 43-dimensional vector of NDCG@10 performance on corresponding 43 test queries. The 43-dimensional vector is then reduced to two-dimensions and plotted using t-SNE. Runs that are submitted by the same group generally cluster together. Similarly, "nnlm", "nn", and "trad" runs also demonstrate similarities. task, the difference is 3.6%.
The best NCG@100 for the document retrieval task is achieved by a welltuned combination of BM25 [76] and RM3 [114] on top of document expansion using doc2query [443]-which improves by 22.9% on the metric relative to the set of 100 candidates provided for the reranking task. For the passage retrieval task, the best NCG@1000 is 20.7% higher than that of the provided reranking candidate set.
Given this was the first ever Deep Learning Track at TREC, we are not yet seeing a strong advantage of "fullrank" over "rerank". However, we hope that as the body of literature on neural methods for phase 1 retrieval (e.g., [31,345,363,443]) (d) NCG@1000 for runs on the passage retrieval task Figure 8.6: Analyzing the impact of "fullrank" vs. "rerank" settings on retrieval performance. Figure (a) and (b) show the performance of different runs on the document and passage retrieval tasks, respectively. Figure (c) and (d) plot the NCG@100 and NCG@1000 metrics for the same runs for the two tasks, respectively. The runs are ordered by their NDCG@10 performance along the x-axis in all four plots. We observe, that the best run under the "fullrank" setting outperforms the same under the "rerank" setting for both document and passage retrieval tasks-although the gaps are relatively smaller compared to those in Figure 8 NIST labels vs. Sparse MS MARCO labels. Our baseline human labels from MS MARCO often have one known positive result per query. We use these labels for training, but they are also available for test queries. Although our official evaluation uses NDCG@10 with NIST labels, we now compare this with reciprocal rank (RR) using MS MARCO labels, and MRR using NIST labels. Our goal is to understand how changing the labeling scheme and metric affects the overall results of the track,  but if there is any disagreement we believe the NDCG results are more valid, since they evaluate the ranking more comprehensively and a ranker that can only perform well on labels with exactly the same distribution as the training set is not robust enough for use in real-world applications, where real users will have opinions that are not necessarily identical to the preferences encoded in sparse training labels. In

Conclusion
The TREC 2019 Deep Learning Track introduced two large training datasets, for a document retrieval task and a passage retrieval task, generating two ad hoc test collections with good reusability. For both tasks, in the presence of large training data, non-neural network runs were outperformed by neural network runs. Among the neural approaches, the best-performing runs tended to use transfer learning, employing a pretrained language model such as BERT. In future it will be interesting to confirm and extend these results, understanding what mix of data and multi-stage training lead to the best overall performance.
We compared reranking approaches to end-to-end retrieval approaches, and there was not a huge difference, with some runs performing well in both regimes.
This is another result that would be interesting to track in future, since we would expect that end-to-end retrieval should perform better if it can recall documents that are unavailable in a reranking subtask. Unlike traditional IR methods, where relevance is estimated largely by counting occurrences of query terms in document text, the neural methods described in this thesis focus on learning useful text representations guided by optimization objectives that correspond to tasks such as ranking and language modeling. Based on the empirical evidence presented in this thesis-and the substantial body of neural IR literature that has been emerging over the recent years-it is safe to conclude that these representation learning methods are able to demonstrate sizeable improvements over traditional IR methods in the presence of large training corpora.
Ongoing new research efforts in this area may be concerned with further improving result quality (effectiveness) while lowering compute and memory costs (efficiency), and even coming up with more elaborate measures of successful retrieval outcomes (e.g., exposure-based metrics) that these models can be optimized towards.
However, this emerging family of neural methods may be causing more fundamental shifts in the field of IR. For example, we argue that after at least two decades of largely unsuccessful attempts at leveraging models and artifacts from NLP to improve IR tasks [444][445][446][447][448], we are now witnessing surprisingly huge benefits from applications of deep NLP models in retrieval. These new NLP artifacts, however, are not in the form thesauri or parts of speech tags, but rather in the form of pretrained language models and latent text representations. While, these black box language models may pick up certain linguistic regularities from training on large corpora, it is also possible, if not likely, that these learned latent representations encode relationships and attributes that are very different to our own notion of linguistic properties. By simply modeling observed regularities in unlabeled corpora, a language model may in fact learn that "duck" and "porcupine" are similar given they appear in similar contexts-such as, "how much does a duck weigh?" and "how much does a porcupine weigh?". If our goal is to maximize some averaged relevance metrics for a query autocompletion task, it may indeed be reasonable that "duck" and "porcupine" have similar latent representations. Similarly, the latent space may be able to encode seemingly nonsensical concepts such as a "duckupine" even if it has no meaningful counterpart in the real world, except may be in literary fiction (see Figure 9.1).
This poses an interesting challenge for the research community. While, we are reasonably good at measuring how effective these black box models are at improving retrieval, it is significantly harder to articulate exactly what knowledge and world view these models encode (and do not encode), and even more difficult to quantify the progress the IR community is making with regards to better understanding of retrieval tasks from the application of these models. This is not to imply that the learned latent representations must be perfectly interpretable to qualify as scientific progress, but rather we are making a case for viewing the contributions of neural IR through a much broader lens that encourages its usage to aid the development of new IR theory and improved understanding of retrieval tasks.
On that note, we conclude this thesis by summarize the contribution of our own work, as described in the earlier chapters, in Section 9.1, and identifying key future challenges and opportunities for the field in Section 9.2.

A summary of our contributions
This thesis summarizes a substantial body of work on neural methods for text retrieval. We ground our contributions by presenting a thorough survey of the field.
We highlight the challenges that are unique to IR and use them to motivate novel learning approaches and model architectures.
We begin with Duet-a neural model that gathers evidence of a document's relevance to a query by inspecting patterns of query term matches in the document as well as learning latent query and document representation for matching. The The scope of impact of neural IR models is limited, if restricted only to late stage re-ranking. Therefore, we incorporate a query term independence assumption to re-design the Duet model. The re-architected model is amenable to full precomputation while retaining all the effectiveness of the original Duet architecture.
This opens the opportunity to employ deep neural models, like Duet and BERTbased ranking, for efficient retrieval from the full collection.
While, learning to rank methods traditionally focus on producing a static rank-ing, we also explore an optimization strategy for stochastic ranking. We argue that in real world retrieval systems, it makes sense to measure and optimize towards expected exposure of retrieved items, in the pursuit of fairness and diversity related outcomes.
We demonstrate the usefulness of deep neural network based approaches to IR tasks beyond document and passage retrieval, such as query auto-completion and session modeling. Finally, we initiate a large-scale benchmarking effort for neural IR methods at TREC and report our key findings.
The body of work described in this thesis was not conducted in isolation. We conducted several other studies, in collaboration, focused on neural IR that we do not describe here. These efforts focused on exploring schemes for explicit regularization [96,97] during model training, studying reinforcement learning based approaches to retrieval [75], designing neural ranking models for structured documents [345], prototyping proactive retrieval systems [58], and even contributing to general purpose neural toolkits [293].

The Future of neural IR
An ideal IR model would be able to infer the meaning of a query from context.
Given a query about the Prime Minister of UK, for example, it may be obvious from context whether it refers to John Major or Teresa May-perhaps due to the time period of the corpus, or it may need to be disambiguated based on other context such as the other query terms or the user's short or long-term history. If the model learns a representation that encodes this context, perhaps making Prime Minister close to Teresa May in a latent space, it is like a library. To scale to a large corpus, this memorization would need to cover a massive number of connections between entities and contexts, which could potentially be limited by model capacity. Memorization could also cause update problems, for example if there is a new Prime Minister but the model and most documents still refer to the old one. To avoid these problems, another design could avoid memorizing connections in the corpus, and instead perform some per-query process that reads the corpus and perhaps even reasons about the content, like a librarian.
Many of the breakthroughs in deep learning have been motivated by the needs of specific application areas. Convolutional neural networks, for example, are commonly employed by the vision community, whereas recurrent architectures find more applications in speech recognition and NLP. It is likely that the specific nature of IR tasks and data will inform our choice of neural architectures and drive us towards new designs. Future IR explorations may also be motivated by developments in related areas, such as NLP. Neural architectures that have been evaluated on non-IR tasks [449][450][451][452][453] can be investigated in the retrieval context. New methods for training neural IR models-e.g., using reinforcement [75,454,455] or adversarial learning [96,243]-may also emerge as important directions for future explorations. In particular, large scale unsupervised training of language models-e.g., BERT [281]-have already demonstrated significant jump in retrieval performance on public benchmarks [35].
However, given the pace at which the area of deep learning is growing, in terms of the number of new architectures and training regimes, we should be wary of the combinatorial explosion of trying every model on every IR task. We should not disproportionately focus on maximizing quantitative improvements and in the process, neglect theoretical understanding and qualitative insights. It would be a bad outcome for the field if these explorations do not grow our understanding of the fundamental principles of machine learning and information retrieval. Neural models should not be the hammer that we try on every IR task, or we may risk reducing every IR task to a nail. 1 Rather, these new models should also be the lens through which researchers gain new insights into the underlying principles of IR tasks. This may imply that sometimes we prefer neural models that, if not interpretable, then at least are amenable to analysis and interrogation. We may elicit more insights from simpler models while more sophisticated models may achieve state-of-the-art performances. As a community, we may need to focus on both to achieve results that are both impactful as well as insightful.
IR also has a role in the context of the ambitions of the machine learning community. Retrieval is key to many one-shot learning approaches [456,457].
Ghazvininejad et al. [458] proposed to "search" external information sources in the process of solving complex tasks using neural networks. The idea of learning local representations proposed by Diaz et al. [91] may be applicable to non-IR tasks.
While we look at applying neural methods to IR, we should also look for opportunities to leverage IR techniques as part of-or in combination with-neural and other machine learning models.
We must also renew our focus on the fundamentals, including benchmarking and reproducibility. An important prerequisite to enable the "neural IR train" to steam forward is to build shared public resources-e.g., large scale datasets for training and evaluation, and repository of shared model implementations-and to ensure that appropriate bindings exist (e.g., [302,303]) between common IR frameworks and toolkits from the neural network community. At the time of writing this thesis, we just concluded the first ever TREC track on deep learning based approaches to IR using the MS MARCO [48] dataset. We hope that these datasets and others would assume the same critical role in fueling neural IR progress as the ImageNet database [459] in the computer vision community.
The emergence of new IR tasks also demands rethinking many of our existing metrics. The metrics that may be appropriate for evaluating document ranking systems may be inadequate when the system generates textual answers in response to information seeking questions. In the latter scenario, the metric should distinguish between whether the response differs from the ground truth in the information content or in phrasing of the answer [49,460,461]. As multi-turn interactions with retrieval systems become more common, the definition of task success will also need to evolve accordingly. A good Neural IR research agenda should not only focus on novel techniques, but also encompass all these other aspects.