author: Alex Graves
`email@example.com`
Greg Wayne
`email@example.com`
Ivo Danihelka
`firstname.lastname@example.com`

Google DeepMind, London, UK
bibliography: ntm_arxiv.bib
title:
  .
  ------------------------------------------------------------------------
  .
  **Neural Turing Machines**
  .
  .
  ------------------------------------------------------------------------

**Abstract**

We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that *Neural Turing Machines* can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.

# Introduction

Computer programs make use of three fundamental mechanisms: elementary operations (e.g., arithmetic operations), logical flow control (branching), and external memory, which can be written to and read from in the course of computation . Despite its wide-ranging success in modelling complicated data, modern machine learning has largely neglected the use of logical flow control and external memory.

Recurrent neural networks (RNNs) stand out from other machine learning methods for their ability to learn and carry out complicated transformations of data over extended periods of time. Moreover, it is known that RNNs are Turing-Complete , and therefore have the capacity to simulate arbitrary procedures, *if* properly wired. Yet what is possible in principle is not always what is simple in practice. We therefore enrich the capabilities of standard recurrent networks to simplify the solution of algorithmic tasks. This enrichment is primarily via a large, addressable memory, so, by analogy to Turing's enrichment of finite-state machines by an infinite memory tape, we dub our device a "Neural Turing Machine" (NTM). Unlike a Turing machine, an NTM is a differentiable computer that can be trained by gradient descent, yielding a practical mechanism for learning programs.

In human cognition, the process that shares the most similarity to algorithmic operation is known as "working memory." While the mechanisms of working memory remain somewhat obscure at the level of neurophysiology, the verbal definition is understood to mean a capacity for short-term storage of information and its rule-based manipulation . In computational terms, these rules are simple programs, and the stored information constitutes the arguments of these programs. Therefore, an NTM resembles a working memory system, as it is designed to solve tasks that require the application of approximate rules to "rapidly-created variables." Rapidly-created variables are data that are quickly bound to memory slots, in the same way that the number 3 and the number 4 are put inside registers in a conventional computer and added to make 7 . An NTM bears another close resemblance to models of working memory since the NTM architecture uses an attentional process to read from and write to memory selectively. In contrast to most models of working memory, our architecture can learn to use its working memory instead of deploying a fixed set of procedures over symbolic data.

The organisation of this report begins with a brief review of germane research on working memory in psychology, linguistics, and neuroscience, along with related research in artificial intelligence and neural networks. We then describe our basic contribution, a memory architecture and attentional controller that we believe is well-suited to the performance of tasks that require the induction and execution of simple programs. To test this architecture, we have constructed a battery of problems, and we present their precise descriptions along with our results. We conclude by summarising the strengths of the architecture.

# Foundational Research

## Psychology and Neuroscience

The concept of working memory has been most heavily developed in psychology to explain the performance of tasks involving the short-term manipulation of information. The broad picture is that a "central executive" focuses attention and performs operations on data in a memory buffer . Psychologists have extensively studied the capacity limitations of working memory, which is often quantified by the number of "chunks" of information that can be readily recalled .[^1] These capacity limitations lead toward an understanding of structural constraints in the human working memory system, but in our own work we are happy to exceed them.

In neuroscience, the working memory process has been ascribed to the functioning of a system composed of the prefrontal cortex and basal ganglia . Typical experiments involve recording from a single neuron or group of neurons in prefrontal cortex while a monkey is performing a task that involves observing a transient cue, waiting through a "delay period," then responding in a manner dependent on the cue. Certain tasks elicit persistent firing from individual neurons during the delay period or more complicated neural dynamics. A recent study quantified delay period activity in prefrontal cortex for a complex, context-dependent task based on measures of "dimensionality" of the population code and showed that it predicted memory performance .

Modeling studies of working memory range from those that consider how biophysical circuits could implement persistent neuronal firing to those that try to solve explicit tasks . Of these, Hazy et al.'s model is the most relevant to our work, as it is itself analogous to the Long Short-Term Memory architecture, which we have modified ourselves. As in our architecture, Hazy et al.'s has mechanisms to gate information into memory slots, which they use to solve a memory task constructed of nested rules. In contrast to our work, the authors include no sophisticated notion of memory addressing, which limits the system to storage and recall of relatively simple, atomic data. Addressing, fundamental to our work, is usually left out from computational models in neuroscience, though it deserves to be mentioned that Gallistel and King and Marcus have argued that addressing must be implicated in the operation of the brain.

## Cognitive Science and Linguistics

Historically, cognitive science and linguistics emerged as fields at roughly the same time as artificial intelligence, all deeply influenced by the advent of the computer . Their intentions were to explain human mental behaviour based on information or symbol-processing metaphors. In the early 1980s, both fields considered recursive or procedural (rule-based) symbol-processing to be the highest mark of cognition. The Parallel Distributed Processing (PDP) or connectionist revolution cast aside the symbol-processing metaphor in favour of a so-called "sub-symbolic" description of thought processes .

Fodor and Pylyshyn famously made two barbed claims about the limitations of neural networks for cognitive modeling. They first objected that connectionist theories were incapable of *variable-binding*, or the assignment of a particular datum to a particular slot in a data structure. In language, variable-binding is ubiquitous; for example, when one produces or interprets a sentence of the form, "Mary spoke to John," one has assigned "Mary" the role of subject, "John" the role of object, and "spoke to" the role of the transitive verb. Fodor and Pylyshyn also argued that neural networks with fixed-length input domains could not reproduce human capabilities in tasks that involve processing *variable-length structures*. In response to this criticism, neural network researchers including Hinton , Smolensky , Touretzky , Pollack , Plate , and Kanerva investigated specific mechanisms that could support both variable-binding and variable-length structure within a connectionist framework. Our architecture draws on and potentiates this work.

Recursive processing of variable-length structures continues to be regarded as a hallmark of human cognition. In the last decade, a firefight in the linguistics community staked several leaders of the field against one another. At issue was whether recursive processing is the "uniquely human" evolutionary innovation that enables language and is specialized to language, a view supported by Fitch, Hauser, and Chomsky , or whether multiple new adaptations are responsible for human language evolution and recursive processing predates language . Regardless of recursive processing's evolutionary origins, all agreed that it is essential to human cognitive flexibility.

## Recurrent Neural Networks

Recurrent neural networks constitute a broad class of machines with dynamic state; that is, they have state whose evolution depends both on the input to the system and on the current state. In comparison to hidden Markov models, which also contain dynamic state, RNNs have a distributed state and therefore have significantly larger and richer memory and computational capacity. Dynamic state is crucial because it affords the possibility of context-dependent computation; a signal entering at a given moment can alter the behaviour of the network at a much later moment.

A crucial innovation to recurrent networks was the Long Short-Term Memory (LSTM) . This very general architecture was developed for a specific purpose, to address the "vanishing and exploding gradient" problem , which we might relabel the problem of "vanishing and exploding sensitivity." LSTM ameliorates the problem by embedding perfect integrators for memory storage in the network. The simplest example of a perfect integrator is the equation $\mathbf{x}(t+1) = \mathbf{x}(t) + \mathbf{i}(t)$, where $\mathbf{i}(t)$ is an input to the system. The implicit identity matrix $I \mathbf{x}(t)$ means that signals do not dynamically vanish or explode. If we attach a mechanism to this integrator that allows an enclosing network to choose when the integrator listens to inputs, namely, a programmable gate depending on context, we have an equation of the form $\mathbf{x}(t+1) = \mathbf{x}(t) + g(\text{context}) \mathbf{i}(t)$. We can now selectively store information for an indefinite length of time.

Recurrent networks readily process variable-length structures without modification. In sequential problems, inputs to the network arrive at different times, allowing variable-length or composite structures to be processed over multiple steps. Because they natively handle variable-length structures, they have recently been used in a variety of cognitive problems, including speech recognition , text generation , handwriting generation and machine translation . Considering this property, we do not feel that it is urgent or even necessarily valuable to build explicit parse trees to merge composite structures greedily .

Other important precursors to our work include differentiable models of attention and program search , constructed with recurrent neural networks.

# Neural Turing Machines

A Neural Turing Machine (NTM) architecture contains two basic components: a neural network *controller* and a memory bank. presents a high-level diagram of the NTM architecture. Like most neural networks, the controller interacts with the external world via input and output vectors. Unlike a standard network, it also interacts with a memory matrix using selective read and write operations. By analogy to the Turing machine we refer to the network outputs that parametrise these operations as "heads."

Crucially, every component of the architecture is differentiable, making it straightforward to train with gradient descent. We achieved this by defining 'blurry' read and write operations that interact to a greater or lesser degree with all the elements in memory (rather than addressing a single element, as in a normal Turing machine or digital computer). The degree of blurriness is determined by an attentional "focus" mechanism that constrains each read and write operation to interact with a small portion of the memory, while ignoring the rest. Because interaction with the memory is highly sparse, the NTM is biased towards storing data without interference. The memory location brought into attentional focus is determined by specialised outputs emitted by the heads. These outputs define a normalised weighting over the rows in the memory matrix (referred to as memory "locations"). Each weighting, one per read or write head, defines the degree to which the head reads or writes at each location. A head can thereby attend sharply to the memory at a single location or weakly to the memory at many locations.

## Reading

Let $\mathbf{M}_t$ be the contents of the $N \times M$ memory matrix at time t, where $N$ is the number of memory locations, and $M$ is the vector size at each location. Let $\mathbf{w}_t$ be a vector of weightings over the $N$ locations emitted by a read head at time $t$. Since all weightings are normalised, the $N$ elements $w_t(i)$ of $\mathbf{w}_t$ obey the following constraints: $$\sum_i w_t(i) = 1, \qquad 0 \leq w_t(i) \leq 1, \, \forall i.$$ The length $M$ *read vector* $\mathbf{r}_t$ returned by the head is defined as a convex combination of the row-vectors $\mathbf{M}_t(i)$ in memory: $$\mathbf{r}_t \longleftarrow \sum_i{w_t(i) \mathbf{M}_t(i)},$$ which is clearly differentiable with respect to both the memory and the weighting.

## Writing

Taking inspiration from the input and forget gates in LSTM, we decompose each write into two parts: an *erase* followed by an *add*.

Given a weighting $\mathbf{w}_t$ emitted by a write head at time $t$, along with an *erase vector* $\mathbf{e}_t$ whose $M$ elements all lie in the range $(0, 1)$, the memory vectors $\mathbf{M}_{t-1}(i)$ from the previous time-step are modified as follows: $$\mathbf{\tilde{M}}_t(i) \longleftarrow \mathbf{M}_{t-1}(i) \left[\mathbf{1}-w_t(i) \mathbf{e}_t\right],$$ where $\mathbf{1}$ is a row-vector of all $1$-s, and the multiplication against the memory location acts point-wise. Therefore, the elements of a memory location are reset to zero only if both the weighting at the location and the erase element are one; if either the weighting or the erase is zero, the memory is left unchanged. When multiple write heads are present, the erasures can be performed in any order, as multiplication is commutative.

Each write head also produces a length $M$ *add vector* $\mathbf{a}_t$, which is added to the memory after the erase step has been performed: $$\mathbf{M}_t(i) \longleftarrow \mathbf{\tilde{M}}_t(i) + w_t(i)\, \mathbf{a}_t.$$ Once again, the order in which the adds are performed by multiple heads is irrelevant. The combined erase and add operations of all the write heads produces the final content of the memory at time $t$. Since both erase and add are differentiable, the composite write operation is differentiable too. Note that both the erase and add vectors have $M$ independent components, allowing fine-grained control over which elements in each memory location are modified.

## Addressing Mechanisms

Although we have now shown the equations of reading and writing, we have not described how the weightings are produced. These weightings arise by combining two addressing mechanisms with complementary facilities. The first mechanism, "content-based addressing," focuses attention on locations based on the similarity between their current values and values emitted by the controller. This is related to the content-addressing of Hopfield networks . The advantage of content-based addressing is that retrieval is simple, merely requiring the controller to produce an approximation to a part of the stored data, which is then compared to memory to yield the exact stored value.

However, not all problems are well-suited to content-based addressing. In certain tasks the content of a variable is arbitrary, but the variable still needs a recognisable name or address. Arithmetic problems fall into this category: the variable $x$ and the variable $y$ can take on any two values, but the procedure $f(x,y) = x \times y$ should still be defined. A controller for this task could take the values of the variables $x$ and $y$, store them in different addresses, then retrieve them and perform a multiplication algorithm. In this case, the variables are addressed by location, not by content. We call this form of addressing "location-based addressing." Content-based addressing is strictly more general than location-based addressing as the content of a memory location could include location information inside it. In our experiments however, providing location-based addressing as a primitive operation proved essential for some forms of generalisation, so we employ both mechanisms concurrently.

presents a flow diagram of the entire addressing system that shows the order of operations for constructing a weighting vector when reading or writing.

### Focusing by Content

For content-addressing, each head (whether employed for reading or writing) first produces a length $M$ *key vector* $\mathbf{k}_t$ that is compared to each vector $\mathbf{M}_t(i)$ by a similarity measure $K\big[\cdot, \cdot\big]$. The content-based system produces a normalised weighting $w_t^c$ based on the similarity and a positive *key strength*, $\beta_t$, which can amplify or attenuate the precision of the focus: $$\begin{aligned}
 \label{content_focus}
w^c_t(i) & \longleftarrow &\frac{\exp\bigg(\beta_t K\big[\mathbf{k}_t, \mathbf{M}_t(i)\big]\bigg)}{\sum_j \exp\bigg(\beta_t K\big[\mathbf{k}_t, \mathbf{M}_t(j)\big]\bigg)}.
\end{aligned}$$ In our current implementation, the similarity measure is cosine similarity: $$\begin{aligned}
 \label{cosine}
K \big[ \mathbf{u}, \mathbf{v} \big] & = & \frac{\mathbf{u} \cdot \mathbf{v}}{||\mathbf{u}|| \cdot ||\mathbf{v}||}.
\end{aligned}$$

### Focusing by Location

The location-based addressing mechanism is designed to facilitate both simple iteration across the locations of the memory and random-access jumps. It does so by implementing a rotational shift of a weighting. For example, if the current weighting focuses entirely on a single location, a rotation of $1$ would shift the focus to the next location. A negative shift would move the weighting in the opposite direction.

Prior to rotation, each head emits a scalar *interpolation gate* $g_t$ in the range $(0, 1)$. The value of $g$ is used to blend between the weighting $\mathbf{w}_{t-1}$ produced by the head at the previous time-step and the weighting $\mathbf{w}^c_{t}$ produced by the content system at the current time-step, yielding the *gated weighting* $\mathbf{w}^g_t$: $$\label{eq:interpolation}
\mathbf{w}^g_t \longleftarrow g_t \mathbf{w}^c_{t}  + (1-g_t) \mathbf{w}_{t-1}.$$ If the gate is zero, then the content weighting is entirely ignored, and the weighting from the previous time step is used. Conversely, if the gate is one, the weighting from the previous iteration is ignored, and the system applies content-based addressing.

After interpolation, each head emits a *shift weighting* $\mathbf{s}_t$ that defines a normalised distribution over the allowed integer shifts. For example, if shifts between -1 and 1 are allowed, $\mathbf{s}_t$ has three elements corresponding to the degree to which shifts of -1, 0 and 1 are performed. The simplest way to define the shift weightings is to use a softmax layer of the appropriate size attached to the controller. We also experimented with another technique, where the controller emits a single scalar that is interpreted as the lower bound of a width one uniform distribution over shifts. For example, if the shift scalar is 6.7, then $s_t(6) = 0.3$, $s_t(7) = 0.7$, and the rest of $\mathbf{s}_t$ is zero.

If we index the $N$ memory locations from $0$ to $N-1$, the rotation applied to $\mathbf{w}^g_t$ by $\mathbf{s}_t$ can be expressed as the following circular convolution: $$\begin{aligned}
\label{eq:rotation}
\tilde{w}_t(i)
% &= \big( \mathbf{w}^g_t(\cdot) \circledast \mathbf{s}_t(\cdot) \big)(i) \nonumber \\
&\longleftarrow \sum_{j=0}^{N-1} w^g_t(j)\, s_t(i-j)
\end{aligned}$$ where all index arithmetic is computed modulo $N$. The convolution operation in can cause leakage or dispersion of weightings over time if the shift weighting is not sharp. For example, if shifts of -1, 0 and 1 are given weights of 0.1, 0.8 and 0.1, the rotation will transform a weighting focused at a single point into one slightly blurred over three points. To combat this, each head emits one further scalar $\gamma_t \ge 1$ whose effect is to sharpen the final weighting as follows: $$\label{eq:sharpen}
w_t(i) \longleftarrow \frac{\tilde{w}_t(i)^{\gamma_t}}{\sum_j \tilde{w}_t(j)^{\gamma_t}}$$

The combined addressing system of weighting interpolation and content and location-based addressing can operate in three complementary modes. One, a weighting can be chosen by the content system without any modification by the location system. Two, a weighting produced by the content addressing system can be chosen and then shifted. This allows the focus to jump to a location next to, but not on, an address accessed by content; in computational terms this allows a head to find a contiguous block of data, then access a particular element within that block. Three, a weighting from the previous time step can be rotated without any input from the content-based addressing system. This allows the weighting to iterate through a sequence of addresses by advancing the same distance at each time-step.

## Controller Network

The NTM architecture architecture described above has several free parameters, including the size of the memory, the number of read and write heads, and the range of allowed location shifts. But perhaps the most significant architectural choice is the type of neural network used as the controller. In particular, one has to decide whether to use a recurrent or feedforward network. A recurrent controller such as LSTM has its own internal memory that can complement the larger memory in the matrix. If one compares the controller to the central processing unit in a digital computer (albeit with adaptive rather than predefined instructions) and the memory matrix to RAM, then the hidden activations of the recurrent controller are akin to the registers in the processor. They allow the controller to mix information across multiple time steps of operation. On the other hand a feedforward controller can mimic a recurrent network by reading and writing at the same location in memory at every step. Furthermore, feedforward controllers often confer greater transparency to the network's operation because the pattern of reading from and writing to the memory matrix is usually easier to interpret than the internal state of an RNN. However, one limitation of a feedforward controller is that the number of concurrent read and write heads imposes a bottleneck on the type of computation the NTM can perform. With a single read head, it can perform only a unary transform on a single memory vector at each time-step, with two read heads it can perform binary vector transforms, and so on. Recurrent controllers can internally store read vectors from previous time-steps, so do not suffer from this limitation.

# Experiments

This section presents preliminary experiments on a set of simple algorithmic tasks such as copying and sorting data sequences. The goal was not only to establish that NTM is able to solve the problems, but also that it is able to do so by learning compact internal programs. The hallmark of such solutions is that they generalise well beyond the range of the training data. For example, we were curious to see if a network that had been trained to copy sequences of length up to 20 could copy a sequence of length 100 with no further training.

For all the experiments we compared three architectures: NTM with a feedforward controller, NTM with an LSTM controller, and a standard LSTM network. Because all the tasks were episodic, we reset the dynamic state of the networks at the start of each input sequence. For the LSTM networks, this meant setting the previous hidden state equal to a learned bias vector. For NTM the previous state of the controller, the value of the previous read vectors, and the contents of the memory were all reset to bias values. All the tasks were supervised learning problems with binary targets; all networks had logistic sigmoid output layers and were trained with the cross-entropy objective function. Sequence prediction errors are reported in bits-per-sequence. For more details about the experimental parameters see .

## Copy

The copy task tests whether NTM can store and recall a long sequence of arbitrary information. The network is presented with an input sequence of random binary vectors followed by a delimiter flag. Storage and access of information over long time periods has always been problematic for RNNs and other dynamic architectures. We were particularly interested to see if an NTM is able to bridge longer time delays than LSTM.

The networks were trained to copy sequences of eight bit random vectors, where the sequence lengths were randomised between 1 and 20. The target sequence was simply a copy of the input sequence (without the delimiter flag). Note that no inputs were presented to the network while it receives the targets, to ensure that it recalls the entire sequence with no intermediate assistance.

As can be seen from , NTM (with either a feedforward or LSTM controller) learned much faster than LSTM alone, and converged to a lower cost. The disparity between the NTM and LSTM learning curves is dramatic enough to suggest a qualitative, rather than quantitative, difference in the way the two models solve the problem.

We also studied the ability of the networks to generalise to longer sequences than seen during training (that they can generalise to novel vectors is clear from the training error). demonstrate that the behaviour of LSTM and NTM in this regime is radically different. NTM continues to copy as the length increases[^2], while LSTM rapidly degrades beyond length 20.

The preceding analysis suggests that NTM, unlike LSTM, has learned some form of copy algorithm. To determine what this algorithm is, we examined the interaction between the controller and the memory (). We believe that the sequence of operations performed by the network can be summarised by the following pseudocode:

This is essentially how a human programmer would perform the same task in a low-level programming language. In terms of data structures, we could say that NTM has learned how to create and iterate through arrays. Note that the algorithm combines both content-based addressing (to jump to start of the sequence) and location-based addressing (to move along the sequence). Also note that the iteration would not generalise to long sequences without the ability to use relative shifts from the *previous* read and write weightings (Equation ), and that without the focus-sharpening mechanism (Equation ) the weightings would probably lose precision over time.

## Repeat Copy

The repeat copy task extends copy by requiring the network to output the copied sequence a specified number of times and then emit an end-of-sequence marker. The main motivation was to see if the NTM could learn a simple nested function. Ideally, we would like it to be able to execute a "for loop" containing any subroutine it has already learned.

The network receives random-length sequences of random binary vectors, followed by a scalar value indicating the desired number of copies, which appears on a separate input channel. To emit the end marker at the correct time the network must be both able to interpret the extra input and keep count of the number of copies it has performed so far. As with the copy task, no inputs are provided to the network after the initial sequence and repeat number. The networks were trained to reproduce sequences of size eight random binary vectors, where both the sequence length and the number of repetitions were chosen randomly from one to ten. The input representing the repeat number was normalised to have mean zero and variance one.

shows that NTM learns the task much faster than LSTM, but both were able to solve it perfectly.[^3] The difference between the two architectures only becomes clear when they are asked to generalise beyond the training data. In this case we were interested in generalisation along two dimensions: sequence length and number of repetitions. illustrates the effect of doubling first one, then the other, for both LSTM and NTM. Whereas LSTM fails both tests, NTM succeeds with longer sequences and is able to perform more than ten repetitions; however it is unable to keep count of of how many repeats it has completed, and does not predict the end marker correctly. This is probably a consequence of representing the number of repetitions numerically, which does not easily generalise beyond a fixed range.

suggests that NTM learns a simple extension of the copy algorithm in the previous section, where the sequential read is repeated as many times as necessary.

## Associative Recall

The previous tasks show that the NTM can apply algorithms to relatively simple, linear data structures. The next order of complexity in organising data arises from "indirection"—that is, when one data item points to another. We test the NTM's capability for learning an instance of this more interesting class by constructing a list of items so that querying with one of the items demands that the network return the subsequent item. More specifically, we define an item as a sequence of binary vectors that is bounded on the left and right by delimiter symbols. After several items have been propagated to the network, we query by showing a random item, and we ask the network to produce the next item. In our experiments, each item consisted of three six-bit binary vectors (giving a total of 18 bits per item). During training, we used a minimum of 2 items and a maximum of 6 items in a single episode.

shows that NTM learns this task significantly faster than LSTM, terminating at near zero cost within approximately $30,000$ episodes, whereas LSTM does not reach zero cost after a million episodes. Additionally, NTM with a feedforward controller learns faster than NTM with an LSTM controller. These two results suggest that NTM's external memory is a more effective way of maintaining the data structure than LSTM's internal state. NTM also generalises much better to longer sequences than LSTM, as can be seen in . NTM with a feedforward controller is nearly perfect for sequences of up to 12 items (twice the maximum length used in training), and still has an average cost below 1 bit per sequence for sequences of 15 items.

In , we show the operation of the NTM memory, controlled by an LSTM with one head, on a single test episode. In "Inputs," we see that the input denotes item delimiters as single bits in row 7. After the sequence of items has been propagated, a delimiter in row 8 prepares the network to receive a query item. In this case, the query item corresponds to the second item in the sequence (contained in the green box). In "Outputs," we see that the network crisply outputs item 3 in the sequence (from the red box). In "Read Weightings," on the last three time steps, we see that the controller reads from contiguous locations that each store the time slices of item 3. This is curious because it appears that the network has jumped directly to the correct location storing item 3. However we can explain this behaviour by looking at "Write Weightings." Here we see that the memory is written to even when the input presents a delimiter symbol between items. One can confirm in "Adds" that data are indeed written to memory when the delimiters are presented (e.g., the data within the black box); furthermore, each time a delimiter is presented, the vector added to memory is different. Further analysis of the memory reveals that the network accesses the location it reads after the query by using a content-based lookup that produces a weighting that is shifted by one. Additionally, the key used for content-lookup corresponds to the vector that was added in the black box. This implies the following memory-access algorithm: when each item delimiter is presented, the controller writes a compressed representation of the previous three time slices of the item. After the query arrives, the controller recomputes the same compressed representation of the query item, uses a content-based lookup to find the location where it wrote the first representation, and then shifts by one to produce the subsequent item in the sequence (thereby combining content-based lookup with location-based offsetting).

## Dynamic N-Grams

The goal of the dynamic N-Grams task was to test whether NTM could rapidly adapt to new predictive distributions. In particular we were interested to see if it were able to use its memory as a re-writable table that it could use to keep count of transition statistics, thereby emulating a conventional N-Gram model.

We considered the set of all possible 6-Gram distributions over binary sequences. Each 6-Gram distribution can be expressed as a table of $2^5=32$ numbers, specifying the probability that the next bit will be one, given all possible length five binary histories. For each training example, we first generated random 6-Gram probabilities by independently drawing all $32$ probabilities from the $Beta(\frac{1}{2}, \frac{1}{2})$ distribution.

We then generated a particular training sequence by drawing 200 successive bits using the current lookup table.[^4] The network observes the sequence one bit at a time and is then asked to predict the next bit. The optimal estimator for the problem can be determined by Bayesian analysis : $$P(B=1|N_1, N_0, \mathbf{c}) = \frac{N_1 + \frac{1}{2}}{N_1 + N_0 + 1}$$ where $\mathbf{c}$ is the five bit previous context, $B$ is the value of the next bit and $N_0$ and $N_1$ are respectively the number of zeros and ones observed after $\mathbf{c}$ so far in the sequence. We can therefore compare NTM to the optimal predictor as well as LSTM. To assess performance we used a validation set of 1000 length 200 sequences sampled from the same distribution as the training data. As shown in , NTM achieves a small, but significant performance advantage over LSTM, but never quite reaches the optimum cost.

The evolution of the two architecture's predictions as they observe new inputs is shown in , along with the optimal predictions. Close analysis of NTM's memory usage () suggests that the controller uses the memory to count how many ones and zeros it has observed in different contexts, allowing it to implement an algorithm similar to the optimal estimator.

## Priority Sort

This task tests whether the NTM can sort data—an important elementary algorithm. A sequence of random binary vectors is input to the network along with a scalar priority rating for each vector. The priority is drawn uniformly from the range \[-1, 1\]. The target sequence contains the binary vectors sorted according to their priorities, as depicted in .

Each input sequence contained 20 binary vectors with corresponding priorities, and each target sequence was the 16 highest-priority vectors in the input.[^5] Inspection of NTM's memory use led us to hypothesise that it uses the priorities to determine the relative location of each write. To test this hypothesis we fitted a linear function of the priority to the observed write locations. shows that the locations returned by the linear function closely match the observed write locations. It also shows that the network reads from the memory locations in increasing order, thereby traversing the sorted sequence.

The learning curves in demonstrate that NTM with both feedforward and LSTM controllers substantially outperform LSTM on this task. Note that eight parallel read and write heads were needed for best performance with a feedforward controller on this task; this may reflect the difficulty of sorting vectors using only unary vector operations (see ).

## Experimental Details

For all experiments, the *RMSProp* algorithm was used for training in the form described in with momentum of 0.9. give details about the network configurations and learning rates used in the experiments. All LSTM networks had three stacked hidden layers. Note that the number of LSTM parameters grows quadratically with the number of hidden units (due to the recurrent connections in the hidden layers). This contrasts with NTM, where the number of parameters does not increase with the number of memory locations. During the training backward pass, all gradient components are clipped elementwise to the range (-10, 10).

```latex
\begin{small}
\centering % used for centering table
\begin{tabular}{l c c c c r} % centered columns (4 columns)
\\[-2.2ex]\hline\\[-2.1ex]
Task & \#Heads & Controller Size & Memory Size & Learning Rate & \#Parameters \\ % inserts table
%heading
\\[-2.2ex]\hline\\[-2.1ex]% inserts single horizontal line
Copy & 1 & 100 & 128 $\times$ 20 & $10^{-4}$ & $17,162$ \\ % inserting body of the table
Repeat Copy & 1 & 100 & 128 $\times$ 20 & $10^{-4}$ & $16,712$ \\
Associative & 4 & 256 & 128 $\times$ 20 & $10^{-4}$ & $146,845$ \\
N-Grams & 1 & 100 & 128 $\times$ 20 & $3 \times 10^{-5}$ & $14,656$ \\
Priority Sort & 8 & 512 & 128 $\times$ 20  & $3 \times 10^{-5}$ & $508,305$ \\  % [1ex] adds vertical space
\\[-2.2ex]\hline\\[-2.1ex]%inserts single line
\end{tabular}
\caption{\small{\textbf{NTM with Feedforward Controller Experimental Settings}}} % title of Table
\label{tab:details_ntm_ff} % is used to refer this table in the text
\end{small}
```
```latex
\begin{small}
\centering % used for centering table
\begin{tabular}{l c c c c r} % centered columns (4 columns)
\\[-2.2ex]\hline\\[-2.1ex]%inserts double horizontal lines
Task & \#Heads & Controller Size & Memory Size & Learning Rate & \#Parameters \\ % inserts table
%heading
\\[-2.2ex]\hline\\[-2.1ex]% inserts single horizontal line
Copy & 1 & 100 & 128 $\times$ 20 & $10^{-4}$ & $67,561$ \\ % inserting body of the table
Repeat Copy & 1 & 100 & 128 $\times$ 20 & $10^{-4}$ & $66,111$ \\
Associative & 1 & 100 & 128 $\times$ 20 & $10^{-4}$ & $70,330$ \\
N-Grams & 1 & 100 & 128 $\times$ 20 & $3 \times 10^{-5}$ & $61,749$ \\
Priority Sort & 5 & $2 \times 100$ & 128 $\times$ 20  & $3 \times 10^{-5}$ & $269,038$ \\ % [1ex] adds vertical space
\\[-2.2ex]\hline\\[-2.1ex]%inserts single line
\end{tabular}
\caption{\small{\textbf{NTM with LSTM Controller Experimental Settings}}} % title of Table
\label{tab:details_ntm_lstm} % is used to refer this table in the text
\end{small}
```
```latex
\begin{small}
\centering % used for centering table
\begin{tabular}{l c c r} % centered columns (4 columns)
\\[-2.2ex]\hline\\[-2.1ex]
% \hline %inserts double horizontal lines
Task & Network Size & Learning Rate & \#Parameters \\  % inserts table
%heading
\\[-2.2ex]\hline\\[-2.1ex]% inserts single horizontal line
Copy & $3 \times 256$ & $3 \times 10^{-5}$ & $1,352,969$ \\ % inserting body of the table
Repeat Copy & $3 \times 512$ & $3 \times 10^{-5}$ & $5,312,007$ \\
Associative & $3 \times 256$ & $10^{-4}$ & $1,344,518$ \\
N-Grams & $3 \times 128$ & $10^{-4}$ & $331,905$ \\
Priority Sort & $3 \times 128$ & $3 \times 10^{-5}$ & $384,424$ \\  % [1ex] adds vertical space
\\[-2.2ex]\hline\\[-2.1ex]%inserts single line
\end{tabular}
\caption{\small{\textbf{LSTM Network Experimental Settings}}} % title of Table
\label{tab:details_lstm} % is used to refer this table in the text
\end{small}
```

# Conclusion

We have introduced the Neural Turing Machine, a neural network architecture that takes inspiration from both models of biological working memory and the design of digital computers. Like conventional neural networks, the architecture is differentiable end-to-end and can be trained with gradient descent. Our experiments demonstrate that it is capable of learning simple algorithms from example data and of using these algorithms to generalise well outside its training regime.

# Acknowledgments

Many have offered thoughtful insights, but we would especially like to thank Daan Wierstra, Peter Dayan, Ilya Sutskever, Charles Blundell, Joel Veness, Koray Kavukcuoglu, Dharshan Kumaran, Georg Ostrovski, Chris Summerfield, Jeff Dean, Geoffrey Hinton, and Demis Hassabis.

[^1]: There remains vigorous debate about how best to characterise capacity limitations .

[^2]: The limiting factor was the size of the memory (128 locations), after which the cyclical shifts wrapped around and previous writes were overwritten.

[^3]: It surprised us that LSTM performed better here than on the copy problem. The likely reasons are that the sequences were shorter (up to length 10 instead of up to 20), and the LSTM network was larger and therefore had more memory capacity.

[^4]: The first 5 bits, for which insufficient context exists to sample from the table, are drawn i.i.d. from a Bernoulli distribution with $p=0.5$.

[^5]: We limited the sort to size 16 because we were interested to see if NTM would solve the task using a binary heap sort of depth 4.


April 2012 BI-TP 2013/06

**Calibrating the In-Medium Behavior of Quarkonia**

**Helmut Satz**

Fakultät für Physik, Universität Bielefeld

D-33501 Bielefeld, Germany

firstname.lastname@example.com

**Abstract**

Quarkonium production has been considered as a tool to study the medium formed in high energy nuclear collisions, assuming that the formation of a hot and dense environment modifies the production pattern observed in elementary collisions. The basic features measured there are the relative fractions of hidden to open heavy flavor and the relative fractions of the different hidden heavy flavor states. Hence the essential question is if and how these quantities are modified in nuclear collisions. We show how the relevant data must be calibrated, i.e., what reference has to be used, in order to determine this in a model-independent way.

The original suggestion for charmonium production as a means to test the formation of a deconfined medium in high energy nuclear collisions was based on the idea that color screening in such a medium would prevent the binding of charm quarks to a color neutral $J/\psi$ . The $J/\psi$ production process in elementary hadronic collisions (taking $pp$ as example) begins with the formation of a $c{\bar c}$ pair; this pair can then either lead to open charm production (about 90 %) or subsequently bind to form a charmonium state (about 10 % for all charmonia). A schematic illutration (Fig. ) shows the dominant high energy reaction through gluon fusion.

The initial $c{\bar c}$ production can be calculated in terms of the parton distribution functions $f_p$ of the relevant hadrons and the pertubative partonic cross section. The full description of charmonium binding has so far resisted various theoretical attempts; on the other hand, the process is in good approximation independent of the incident hadronic collision energy . This is a consequence of the fact that the heavy quark propagator in the reaction $gg \to c{\bar c}$ strongly dampens the mass variation of the $c{\bar c}$ pair with incident energy. Thus the fractions of the produced $c{\bar c}$ system into hidden vs. open charm as well as those for the different charmonium states are approximately constant; once determined at one energy, they remain the same also for different energies. As a result, the phenomenological color evaporation model provides a good description of charmonium production through the form $$\sigma_{hh\to J/\psi}(s) = g_{c{\bar c}\to J/\psi}~\!\sigma_{hh \to c{\bar c}}(s),
\label{cem}$$ and correspondingly for the other charmonium states. Here the constant $g_{c{\bar c}\to J/\psi}$ specifies what fraction of the total $c{\bar c}$ production cross section goes into $J/\psi$ production; in $pp$ collisions it is typically about 2 %. The set of the different constants $g_{c{\bar c}\to i}$ for the different charmonium states $i$ thus effectively characterizes charmonium production in the absence of a medium.

A further important aspect of quarkonium production in elementary collisions is that the observed (1S) ground states $J/\psi$ and $\Upsilon$ are in both cases partially produced through feed-down from higher excited states . Of the observed $J/\psi$ rates, only some 60 % is a directly produced $J/\psi(1S)$ state; about 30 % comes from $\chi_c(1P)$ and 10 % from $\psi'(2S)$ decay. Because of the narrow width of the excited states, their decay occurs well outside any interaction region.

The features we have here summarized for charm and charmonium production are readily extended to that of bottom and bottomonium. To simplify the discussion, we shall continue referring to the charmonium case, keeping in mind that all arguments apply as well to bottomonia. Given the patterns observed in elementary collisions, we want to see how they are modified in the presence of a medium, as provided by nuclear collisions. From the point of view of production dynamics, one way such modifications can arise is as *initial state effects*, which take place before the $c{\bar c}$ pair is produced. The main possibilities considered so far are nuclear modifications of the parton distribution functions (shadowing or antishadowing) and a possible energy loss of the partons passing through the nuclear medium to produce the $c{\bar c}$. Once produced, the pair can encounter *final state effects*, either in the form of a phase space shift already of the $c{\bar c}$, e.g., through an energy loss of the unbound charm quarks, or through effects on the nascent or fully formed charmonium state. Such effects may arise from the passage through the cold nuclear medium, or because of the presence of the medium newly produced in the nuclear collision. The latter is evidently what we have in mind when we want to use quarkonia to study quark-gluon plasma production. The difficulties encountered over the past years in arriving at a conclusive analysis of the relevant nuclear collision data on $J/\psi$ production are largely due to the problem of parametrizing the different effects and then constructing a convincing model correctly incorporating all of them. We want to show here that today experimental means have become available which allow us to calibrate the measured results in a way which avoids these difficulties.

Before we turn to the problem of determining medium effects on quarkonium production, we recall the two main conceptual approaches on what may happen; the two can perhaps best be labelled *suppression* and *enhancement*. Color screening in a quark-gluon plasma will decrease the quarkonium binding, both in strength and in its spatial range, and this should for sufficiently energetic nucleus-nucleus collisions lead to quarkonium dissociation or melting. Since the larger and less tightly bound states will melt at lower temperature or energy density than the ground states, color screening will produce *sequential suppression* . We illustrate this for the $J/\psi$. After an initial threshold melting the $\psi'$ and hence removing its feed-down component for $J/\psi$ production, there will be a second threshold for $\chi_c$ melting and then finally a third, at which the direct $J/\psi(1s)$ is dissociated. The resulting pattern is illustrated in Fig. . We have here introduced something denoted as $J/\psi$ survival probability. Theoretically, this is the chance of a $J/\psi$ to persist as a bound state in a deconfined medium. How to properly define this quantity as a useful observable in experimental studies is the subject of this paper.

The other alternative, $J/\psi$ enhancement, assumes, in accord with color screening and the resulting suppression, that at sufficiently high energies there is an almost complete dissociation of the $J/\psi$'s produced in primary nucleon-nucleon interactions. On the other hand, at such collision energies, these interactions lead to abundant $c{\bar c}$ production; the rate for this process grows faster than that for the production of light quarks, and if the $c{\bar c}$ pairs remain present in the evolution of the medium, the system will at the hadronisation point show an oversaturation of charm, compared to the predicted thermal abundance. If these charm quarks have become part of an equilibrated medium and as such undergo hadronisation in the form of statistical combination, then such secondary charmonium formation can convert more $c{\bar c}$ pairs into $J/\psi$'s than the dynamical primary production mechanism, thus leading to an effective $J/\psi$ enhancement , as shown in Fig. . We note that this scenario invokes a new and so far unknown binding dynamics to form charmonia; the outcome is assumed to be determined simply by the relative abundance of charm quarks .

We now come to our central question: how to calibrate the $J/\psi$ survival? Since we are interested in using quarkonium production as a tool to study the medium produced in nuclear collisions, our primary concern is not if such collisions produce more or fewer $c{\bar c}$ pairs than proton-proton collisions, but rather if the presence of the medium modifies the fraction of produced $c{\bar c}$ pairs going into charmonium formation. In other words, the crucial quantitity is the amount of charmonium production relative to that of open charm . To illustrate: in $pp$ collisions, about 2 % of the total $c{\bar c}$ production goes into $J/\psi$'s. If in high energy nuclear collisions the total $c{\bar c}$ production rate were reduced by a factor two, but we still have 2 % of these going into $J/\psi$'s, then evidently $AA$ collisions do not modify $J/\psi$ binding. This is, of course, strictly true only for fully integrated rates, which is difficult to check; we return to the issue of phase space variations shortly. But we note here already that in the example just given, the number of $J/\psi$'s produced in $AA$ collisions would be half of that obtained by scaling the results from $pp$ interactions. That is, however, not an indication of any $J/\psi$ suppression; it is just the consequence of having fewer $c{\bar c}$ pairs to start with – the formation probability of charmonia has remained the same as in $pp$ collisions, 2 %. It is thus crucial to correctly calibrate the survival in nuclear collisions.

To achieve that, we recall: both sequential suppression and statistical enhancement state that the total number of $c{\bar c}$ pairs produced in nuclear collisions is distributed among hidden and open charm differently than it is in proton-proton collisions. The essential observable is thus the ratio of charmonium states to open charm in nuclear collisions, compared to that in $pp$ interactions. Sequential suppression predicts it to decrease with centrality and eventually vanish, statistical enhancement has it increasing with centrality, in many studies beyond the $pp$ value. Hence the relevant observable is the fraction of charmonia to open charm, or more generally, that of quarkonia to the relevant open heavy flavor production . In this quantity, if measured over the entire phase space, the effects of possible initial state nuclear modifications – shadowing/antishadowing, parton energy loss – cancel out, so that whatever changes it shows relative to the $pp$ pattern is due to final state effects.

It should be noted here that in actual applications, it will most likely not be necessary (nor generally possible) to measure the total open charm production rate. The relative abundances of the different open charm states produced in high energy collisions are so far found to be in good agreement with the predictions of the statistical hadronization model , with a universal hadronisation temperature of some 150 - 170 MeV. For open charm production, this model assumes an initial dynamical (perturbative) $c{\bar c}$ production; subsequently, at the hadronisation point, the charm quarks thus produced form open charm hadrons according to their statistical abundances. The relative production rates are thus totally determined in terms of the mass of the open charm state and the universal hadronisation temperature. Such a description is found to hold very well for open charm production in high energy interactions from $e^+e^-$ annihilation through $pp$ and $pA$ collisions; studies of the $AA$ behavior are underway. However, the model does not work for charmonium production, neither in elementary ($e^+e^-$, $pp$) nor in nuclear interactions, since the binding mechanism forming charmonia is evidently of dynamical and not of statistical origin. In nuclear collisions, there are in addition medium effects on the binding.

As a consequence, we expect that the *relative abundances* of the different open charm states will not depend on the collision energy and remain essentially the same for elementary and nuclear collisions. Analogous to eq. (), the production rate $N(hh \to D_i)$ for a specific open charm state $D_i$ is then a constant fraction of the total open charm rate $N(hh \to c{\bar c})$, $$N(hh \to D_i) \simeq {\bar g}_{c{\bar c}\to D_i} N(hh \to c{\bar c}).
\label{open}$$ The crucial observable, the rate of ($J/\psi$)/open charm, can thus in good approximation be taken as the rate of ($J/\psi$)/$D_i$ for a specific $D$-meson state. This will, of course, greatly facilitate the analyis.

The final state effect of interest to us is that caused by the newly produced medium. We therefore have to check to what extent the nascent or fully formed $J/\psi$ is already dissociated by the cold nuclear matter of the target or the projectile. This can be done by studying the ratio hidden to open charm in high energy $pA$ collisions. It is in fact known that in such interactions there exist reductions of $J/\psi$ production beyond the scaled $pp$ results, but the origin of these is not unambiguously clarified. They could arise largely from initial state effects; but if there is a stronger suppression of $\psi'$ than of $J/\psi$ at energies for which the state is formed inside the nuclear medium, this would indicate that there the interaction depends on its physical size. The different fates of the different quarkonium states in *cold* nuclear matter are due to their different sizes, in a *hot* medium due to the different binding energies. And dissociation by color screening should start with a threshold, while the break-up in nuclear matter is presumably continuous. Measurements of charmonia relative to open charm in $pA$ up to highest energies (RHIC, LHC) are therefore of great importance.

First applications of the in-medium charmonium study based on the relative survival of charmonia vs. open charm were started last year, using LHC data from ALICE and CMS . In Fig. (a), we show mid-rapidity ALICE data for $J/\psi$ production at intermediate transverse momenta, compared to open charm production in a similar kinematic region. In Fig. (b), the comparison is extended to larger transverse momenta, using CMS data for $J/\psi$ production. In both cases, $J/\psi$ production relative to $pp$ results, scaled by the number of collisions, decreases with increasing centrality, as seen by the corresponding $R_{AA}$ values.

This decrease has at times been considered as suppressed $J/\psi$ production. However, that is incorrect: the corresponding $R_{AA}$ for open charm production, as determined through $D$ measurements, shows within errors the same behavior. In other words, the reduction of the $J/\psi$ is in complete agreement with that of open charm; there is neither suppression nor enhancement, the fraction of the produced $c{\bar c}$ pairs going into $J/\psi$ production has remained in the $AA$ collisions considered here the same as in the corresponding $pp$ interactions: $$R_{AA}(J/\psi) = {N_{AA}(J/\psi) \over n_c N_{pp}(J/\psi)} =
{N_{AA}(c{\bar c}) \over n_c N_{pp}(c{\bar c})} = R_{AA}(c{\bar c}),$$ with $n_c$ denoting the scaling factor for the number of collisions at the corresponding centrality. We therefore have $${N_{AA}(J/\psi) \over N_{AA}(c{\bar c})} =
{N_{pp}(J/\psi) \over N_{pp}(c{\bar c})} = g_{c{\bar c}\to J/\psi}.$$ at all centralities, in the kinematic regime indicated. If we divide the $J/\psi$ rates by the open charm rates in the same kinematic region (large $P_T$), this ratio becomes centrality independent, and if we normalize it to the corresponding value from $pp$ collisions, it becomes unity. The correct and model-independent $J/\psi$ survival probability for experimental study is thus $$S_{J/\psi} = \left({N_{AA}(J/\psi) \over N_{AA}(c{\bar c})}\right)
/ \left({N_{pp}(J/\psi) \over N_{pp}(c{\bar c})}\right) = {1 \over g_{c{\bar c}\to J/\psi}}
\left({N_{AA}(J/\psi) \over N_{AA}(c{\bar c})}\right).$$ In the kinematic regime considered so far, it is indeed unity, there is neither suppression nor enhancement.

At the LHC, corresponding data for low $P_T$ open charm production is not yet available. At RHIC, however, it is provided by the PHENIX and STAR collaborations , and the relevant comparisons of $J/\psi$ vs. open charm production are shown in Fig. . At high $P_T$ we have a similar behavior as at the LHC, no change of $J/\psi$ production relative to open charm. At low $P_T$, however, the $R_{AA}(c{\bar c})$ of open charm is within errors unity over the whole centrality range; in contrast, $R_{AA}(J/\psi)$ decreases strongly and thus here gives the correct $J/\psi$ survival probability. We see that now with increasing centrality, a smaller and smaller fraction of $c{\bar c}$ pairs go into $J/\psi$ production, with a suppression of up to 75% for the most central collisions. The $J/\psi$ production finally surviving could conceivably be largely due to corona interactions .

In this context it seems of interest to note that a decrease of the open charm production rates between central and forward rapidity is observed in both $pA$ and $AA$ collisions relative to $pp$ collisions, at Fermilab (pA) and at RHIC (Cu-Cu). We note in particular in the Fermilab data based on 800 GeV $pA$ collisions (see Fig. ) that at forward rapidity there is a suppression of open charm production, relative to scaled $pp$ rates , while at midrapidity there is not. A similar effect is observed in $Cu-Cu$ collisions with $\sqrt s = 200$ GeV at RHIC . If this is passed on to charmonium production, one expects for RHIC and LHC data a smaller $R_{AA}(J/\psi)$ at forward than at mid-rapidity, simply because there are fewer $c{\bar c}$ pairs there to form charmonia, no matter how. Such an effect was indeed observed at RHIC and has often been considered a puzzle .

The aim of this work was to show how quarkonium production has to be calibrated in order to obtain experiment-based, model-independent information on suppression or enhancement due to the hot medium formed in nuclear collisions. The data cited are given for illustration, to make the point. In a more complete study, the transverse momentum range of $J/\psi$ and open charm should be correlated more precisely. The crucial question now remains, of course, how the open charm rates for low $P_T$ production at the LHC will behave. That kinematic region is responsible for the bulk of $c{\bar c}$ production, and the issue to be decided is if nuclear collisions do or do not lead to an overall reduction. At RHIC, that is not the case, as Fig. (b) shows.

**Acknowledgements**

It is a pleasure to thank Zaida Conesa del Valle, Torsten Dahms, Louis Kluberg, Mike Leitch, Carlos Lourenço and Jürgen Schukraft for helpful discussions. Particular thanks go to Torsten Dahms also for pointing out the applicability of RHIC data for the analysis and for providing me with the corresponding Figures.


abstract: Pushing the precision frontier for gravitational waves is one of the most urgent tasks in theoretical physics today, in light of the increasing influx of data from a rapidly growing worldwide network of gravitational-wave detectors. Levi will analytically predict gravitational radiation from such compact binaries and further develop the use of quantum field theory advances to study gravity. She seeks to uncover profound duality relations between gauge and gravity.
  .
  International Editor Clifford Holt spoke to Levi about her research, the importance of analytical high-precision predictions for real-world gravitational-wave data, and potential challenges as she works to develop a better idea of universal commonalities across classical and quantum field theories.
author: Michèle Levi[^1]
date: August 6, 2021
title: Pushing the precision frontier for
  gravitational waves

**Firstly, congratulations on being awarded the Ernest Rutherford Fellowship. How important are funding mechanisms such as these, and what will the Fellowship mean for your own research and career?**

Thank you! A funding mechanism like this prestigious early-career award from the STFC (one of the seven UK research councils under UKRI) is very important and unique. This is the most competitive scheme in the UK for early-career researchers in my domain, devised to ensure that those 10 carefully-selected awardees, who are recognised as the very top international talent and leadership 'material' in science early on in their career, will be integrated into top academic positions in the leading research institutions in the UK.

This will enable the awardees to play a key role in the coming decades in the scientific and transformative leadership at the national level in the UK and will also represent the UK at the international level, solidifying its position as a scientific world-leader for the next generations. As a public funder, UKRI selects these future leaders to shape strategic UK policies, to bolster the UK's scientific place in the world, and to bring about a radical shift in the current research culture in terms of the way that science is conducted, namely its standards and its values, but also its role and commitment within society.

Currently, many academics have sadly forgotten that their raison d'etre is to serve the public, that they have the privilege of expanding the frontiers of knowledge mainly thanks to the public money that is funding their research. We must insist that science that is peer-reviewed and approved for publication and funding lives up to the standards of FAIR (Findabilty, Accessibilty, Interoperabilty, Reusabilty) principles, namely of responsible research of integrity, that is publicly available, widely accessible, and reproducible. We should also advocate for equality, diversity, and inclusion in Science, which should properly represent Society, and engage with the public. It is very clear that UKRI as a public funder has a strong commitment to realising these aspirations, which are unfortunately still overlooked by the majority of private funders.

I am excited and honoured to transfer and base my cutting-edge research programme in the UK, and to be part of making the UK a world-leading force in the timely domain of gravitational waves with the most innovative theoretical approaches and computational tools applied to gravity. I am also incredibly aligned with the aspirations of UKRI to transform the current research culture, and as I myself belong to an overlap of under-represented and marginalised groups in science, I am particularly aware to how systemic inequalities affect the ability of students and researchers to truly realise their potential and excel, or even to just conform to the common skewed metrics of academic merit.

I adamantly champion and pursue equality, diversity, and inclusion in every aspect of my everyday scientific activities. I believe and I see that it is only through these consistent everyday actions and through my being a role model that I can ultimately translate my research work to further concrete scientific and societal impact. I also adamantly advance the broad communication of science across the research community and through public outreach, which I regard as a social imperative. I also find it extremely rewarding to share my knowledge with as many people as possible.

**Your project aims to deliver analytical high-precision predictions for real world gravitational wave data. Can you tell me more about this and why it is important?**

Gravitational radiation is a probing prediction of any complete theory of gravity. Currently, we still do not have a complete theory of gravity and this is true for both ends of the scale, since we do not know how to describe gravity on quantum scales, which are the smallest scales, and we do not have a theory that captures gravity on the largest scales – cosmological scales, since we do not know how to explain the puzzle of dark matter or the accelerated expansion of the Universe that was discovered in 1998. Understanding how the force of gravity works on these more 'extreme' scales is possibly the deepest and most fundamental longstanding open problem in physics.

The first Earth-shaking gravitational-wave (GW) detection announced in 2016 launched a new era of gravitational-wave astronomy and precision gravity, which is only in its infancy, with vast future impacts in astrophysics and cosmology and, most notably, in fundamental physics. Since this first milestone detection, ground-based detectors keep multiplying, and many more are planned for the coming years. They are designed to measure various frequency ranges, which will enable a richer array of sources and events that generate gravitational radiation to be captured.

The technology of measurements is constantly upgraded, either in currently-operating detectors like LIGO or VIRGO, or in upcoming and planned ones, like KAGRA in Japan, and others in the US, Europe, India, and Korea. Furthermore, there are also concrete planned space-based experiments from Europe, China, and Japan that will also target smaller frequencies and other types of sources. All in all, in the coming years and decades we are headed towards a growing influx of gravitational-wave data that will have higher and higher quality.

Let me stress what, I believe, much of the community in Europe and the US does not fully recognise yet: that Asia is about to emerge as possibly the major player in gravitational-wave astronomy in the coming years: in Japan, KAGRA is already operational; in India, IndiGO will become operational in just a few years; and in Korea, SOGRO is planned to uniquely target mid-range frequencies from Earth, while China and Japan are also planning space-borne detectors, TianQin and DECIGO, respectively.

**How will your research help to inform the activities of detectors such as VIRGO and LIGO, as well as perhaps LISA? How could your theoretical work come to complement future observations/detections?**

My research generates theoretical data from which a huge bank of templates is created to be matched in the detectors by the real data that is measured. The GW signal is weak and is covered by many sources of noise so, in order to identify it the matched filtering technique is used for detection, which requires the most accurate theoretical signal templates possible. When a match is found, a detection is announced. Yet, lots of real-world data, even if is great data is just data without interpretation. What transforms data into knowledge is theory and analysis. This is especially true for the gravitational-wave signal since it is so weak, so it is really important to separate the noise from the true signal.

For the most part, the observable differences among the plethora of candidate theories for gravity suggested along the years have been insanely tiny. In most cases, all of these theories seem grossly similar within gravitational-wave measurements. So it is only the theoretical resolution into various physical effects at a very high-precision, such as those that have been computed in my research since my graduate studies, that will enable us to really distinguish among the various suggested theories of gravity, and to close in on which is the viable one.

Another feature which is rather unique to my own theoretical framework is that it provides predictions for generic sources of compact binaries, not only for binaries of two black holes. As the technology in upcoming detectors advances, and as the sensitivity improves, we are able to observe more and more neutron stars, for example, in the radiating binaries. Whereas from black holes we can 'only' learn about gravity theories, neutron stars also tell us about the theory of QCD (Quantum Chromodynamics) in extreme conditions that no human-built lab could produce. With future detectors, we also expect to observe white dwarfs, and learn more about them.

**Have you yet been able to identify any potential challenges that you expect to have to overcome as you work to develop a better idea of universal commonalities across classical and quantum field theories?**

Right now, we see some analogies between interactions of classical extended gravitating objects like the huge non-rotating black holes in radiating binaries, and elementary particles of quantum nature, which we call 'scalar particles' – these are boson particles of zero quantum spin. But the precise mapping between them, and more importantly its generalisations for all types of elementary particle, is poorly understood. In particular, my theoretical work pushed the understanding of spinning gravitating objects, which would also be analogous to higher-spin particles. Currently, the highest-spin particle that has been observed in nature is of spin one, and we also believe there could be particles of spin three/two according to supersymmetric theories, as well as our theoretically predicted graviton – a massless particle of spin two. But what about particles with spins larger than two? Do they exist? Could they be elementary? Or only show up as what we call 'composite' particles, which must be composed of elementary ones?

At the moment, we cannot even manage to give well-defined predictions of observables of quantum field theories that capture such higher-spin particles, and we are certainly not able to write down such quantum field theories explicitly. Furthermore, we are not certain whether this fundamental issue exists already at the classical level, i.e. with the type of two-body interactions that my work is tackling for spinning gravitating objects. The big challenge for my research programme, therefore, would be to more rigorously and generally link the classical and quantum sides of these analogies and to clarify whether there is indeed a fundamental issue with higher-spin – and, if so, whether it originates in the quantisation of the theory, or if it already exists at the classical level.

This all relates to possibly the biggest open question in theoretical physics of how to complete the theory of gravity at high energies, and therefore it would indeed be a breakthrough for my upcoming research programme to be able to provide any further insights on that.

**What are your short/long term hopes and ambitions?**

Since my graduate studies, I have had the privilege of pioneering and leading innovative analytical work and high-precision computation on the highly intricate problem of compact binaries that source gravitational radiation. At the time, I did not believe that I would live to see actual detections of gravitational waves from these sources, since at that time the detectors had been operational for more than a decade without any successful detection.

Yet, the detection came in February 2016, and I can still remember it as though it were yesterday; I was overwhelmed with excitement and wonder for what was to come when the first observation of a gravitational-wave signal by LIGO was announced that day. Since then, the scientific prospects of gravitational-wave science have been continuously improving and exceeding all expectations. The future truly looks symphonic, filled with gravitational-wave echoes.

With this increasing influx of data of an ever improving quality, we have a chance to really push theoretical predictions and to test so much more theory in gravity and even in QCD. My research programme is exactly aimed at maximising the potential of what we can learn from gravitational-wave measurements on the theory of gravity across all scales.

I hope to engage the broader scientific community with this exciting new domain of research, such that it really crosses over all sub-fields of theoretical physics, especially since this is a great scientific challenge that can benefit from so many different specific perspectives. I believe that every one of these perspectives can bring in something fresh and interesting to the mix. I also believe that, to some extent, I have already contributed to the widespread attention that this type of research obtained in the traditional high energy physics community. In the UK, I hope to engage more people and institutions with the research in this field as the UK really has the potential to become a superpower in this domain, especially in the type of analytical work that I am involved in, which has been traditionally dominated by France.

I am also a big believer in open code and open data, and I really want to see individual researchers, big experimental collaborations, scientific journals, and academic institutions all persistently moving towards this in the coming years. The scientific challenges at hand are too big, and we need to focus on the science and make our best joint efforts to make breakthroughs, instead of yielding to the old zero-sum game mentality that keeps theoretical physics stagnated.

Simply put, I just hope that, together, we can all increase our knowledge of the world around us. Isn't that the greatest privilege of our society?

# Acknowledgements

ML received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant 847523.
Her upcoming research program will be supported by the STFC Rutherford grant ST/V003895/1 *"Harnessing QFT for Gravity"*.

[^1]: <https://sites.google.com/view/levim>


abstract: In this letter, we study the possibility that Kaluza-Klein dark matter in a model with one universal extra dimension is responsible for the recent observations of the PAMELA and ATIC experiments. In this model, the dark matter particles annihilate largely to charged leptons, which enables them to produce a spectrum of cosmic ray electrons and positrons consistent with the PAMELA and ATIC measurements. To normalize to the observed signal, however, large boost factors ($\sim$ $10^3$) are required. Despite these large boost factors and significant annihilation to hadronic modes (35%), we find that the constraints from cosmic ray antiproton measurements can be satisfied. Relic abundance considerations in this model force us to consider a rather specific range of masses (approximately 600-900 GeV) which is very similar to the range required to generate the ATIC spectral feature. The results presented here can also be used as a benchmark for model-independent constraints on dark matter annihilation to hadronic modes.
address: $^1$Particle Astrophysics Center, Fermi National Accelerator Laboratory, Batavia, IL 60510
  $^2$Department of Astronomy and Astrophysics, The University of Chicago, Chicago, IL 60637
  $^3$Department of Physics, University of Michigan, Ann Arbor, MI 48109
author: Dan Hooper$^{1,2}$ and Kathryn M. Zurek$^{1,3}$
date: 2024-10-02
title: The PAMELA and ATIC Signals From Kaluza-Klein Dark Matter

Recently, the PAMELA collaboration has published results which show an excess of cosmic ray positrons relative to electrons above approximately 10 GeV , confirming previous indications from HEAT  and AMS-01 . Furthermore, the ATIC experiment has reported a surprising feature in the cosmic ray electron (plus positron) spectrum between approximately 300 and 800 GeV . These observations collectively indicate the presence of a bright source of very high energy electrons and positrons within a few kiloparsecs of the Solar System.

The leading astrophysical hypothesis for the origin of these particles is a nearby and relatively young pulsar (or pulsars), such as B0656+14 or Geminga . A more exciting possibility, however, is that the signals reported by ATIC and PAMELA are the result of dark matter annihilations taking place in the halo of the Milky Way. In order for dark matter to generate these signals, however, care must be taken to avoid a number of potential problems. In particular, the spectrum of electrons and positrons predicted to be generated in the annihilations of most dark matter candidates is much too soft to fit the observations of PAMELA and ATIC . Furthermore, if the annihilation rate throughout the halo of the Milky Way is normalized to account for the PAMELA and ATIC signals, most dark matter candidates will also generate an unacceptably large flux of cosmic ray antiprotons . Possible solutions to these problems include WIMPs which annihilate mostly to charged leptons  or WIMPs which are distributed preferentially in the local neighborhood of the galaxy .

In this letter, we consider Kaluza-Klein dark matter within the context of a model with one extra spatial dimension. In this scenario, the extra dimension is universal, meaning that all of the Standard Model fields are free to propagate through it . Standard Model fields with momentum in the direction of the extra dimension appear as heavy particles known as Kaluza-Klein (KK) states. In particular, we assume that the extra dimension is compactified with an approximate size of $R \sim {\rm TeV}^{-1}$, leading to a complete KK copy of the Standard Model with masses at or around the TeV scale.

Although KK-number conservation is broken in phenomenologically viable scenarios, KK-parity can naturally be conserved in this model, leading to the stability of the lightest KK particle (LKP). An attractive choice for the LKP is the first KK excitation of the hypercharge boson, $B^{(1)}$. This state is stable, colorless, electrically neutral, and constitutes a viable candidate for dark matter  (for a review, see Ref. ). $B^{(1)}$ pairs annihilate to (zero mode) fermion pairs through the $t$-channel exchange of a KK-fermion with a total cross section approximately given by $\sigma v \approx 1.7\times 10^{-26}\, {\rm cm}^3/{\rm s} \times  (1 \,{\rm TeV}/m_{B^{(1)}})^2$. If all other first level KK states are neglected, this cross section leads to a thermal relic abundance of KK dark matter equal to the measured density of dark matter for the choice of $m_{B^{(1)}} \approx 800$ GeV. If KK leptons and other KK states are included in the calculation, LKPs with masses in the range of approximately 600 to 900 GeV can easily be produced with the desired abundance .

The $B^{(1)}-f^{(1)}-f^{(0)}$ couplings which go into the determination of the LKP's annihilation cross section each scale with the fermion's hypercharge, and thus annihilations to charged leptons are preferred over other possible final states. Furthermore, unlike neutralinos (or other Majorana fermions) the annihilation of KK dark matter to light fermions is not helicity suppressed. For a largely degenerate first level KK spectrum, LKPs annihilate to $e^+ e^-$, $\mu^+ \mu^-$ and $\tau^+ \tau^-$ 20% of the time each. Of the remaining annihiations, most (approximately 35%) produce quark pairs. The remaining fraction produce neutrinos and higgs bosons. As we will show, the annihilation modes of KK dark matter lead to very distinctive features in the cosmic ray positron fraction  and cosmic ray electron spectrum . Similar signatures are expected from other dark matter candidates which annihilate mostly to charged leptons.

To calculate the spectrum of cosmic ray electrons and positrons produced in the annihilations of KK dark matter, we solve the propagation equation : $$\begin{aligned}
\frac{\partial}{\partial t}\frac{dN_{e}}{dE_{e}} &=& \vec{\bigtriangledown} \cdot \bigg[K(E_{e})  \vec{\bigtriangledown} \frac{dN_{e}}{dE_{e}} \bigg]
+ \frac{\partial}{\partial E_{e}} \bigg[b(E_{e})\frac{dN_{e}}{dE_{e}}  \bigg] \nonumber \\ &+& Q(E_{e},\vec{x}),
\label{dif}
\end{aligned}$$ where $dN_{e}/dE_{e}$ is the number density of electrons/positrons per unit energy, $K(E_{e})$ is the diffusion coefficient, and $b(E_{e}) \approx 10^{-16} \, ({E_e} / 1 \, \rm{GeV})^2 \,\, \rm{s}^{-1}$ is the electron/positron energy loss rate. As we expect to be in the steady state limit, we set the left hand side of Eq.  to zero. The source term, $Q(E_e, \vec{x})$, reflects both the distribution of dark matter in the Galaxy, and the mass, annihilation rate, and dominant annihilation channels of the dark matter. We also must select boundary conditions for our diffusion zone. In particular, we consider a cylinder of half-thickness $L$, beyond which cosmic rays are able to escape the Galactic Magnetic Field. For this calculation, we have used the package DarkSUSY .

We consider two sets of diffusion parameters in our study which provide good fits to the measured B/C, sub-Fe/Fe, and Be$^{10}$/Be$^{9}$ ratios in the cosmic ray spectrum. We will refer to these two parameter sets as propagation models A and B: $$\begin{aligned}
\mbox{A: $K(E_{e}) =5.3\times10^{28}\, (E_{e} / 4 \, \rm{GeV})^{0.43} \,{\rm cm^2/s}$, L=4\, {\rm kpc}  } \nonumber \\
\mbox{B: $K(E_{e}) =1.4\times10^{28}\, (E_{e} / 4 \, \rm{GeV})^{0.43} \,{\rm cm^2/s}$, L=1\, {\rm kpc}  }. \nonumber
\end{aligned}$$

For the source term, we adopt a dark matter distribution which follows the Navarro-Frenk-White (NFW) halo profile . We allow the normalization of the dark matter annihilation rate to be adjusted freely to fit to the observed PAMELA and ATIC features.

In Fig. , the positron fraction including the contribution from annihilating KK dark matter is compared to the measurements of PAMELA. In computing these results, we have adopted the cosmic ray electron and positron backgrounds as described in Ref. , but allow the spectral slope of the primary electron spectrum to vary within the current measurement errors. In particular, we adopt background electron spectral slopes of $dN_e/dE_e \propto E_e^{-3.25}$ and $dN_e/dE_e \propto E_e^{-3.4}$ for propagation models A and B, respectively. We also allowed the normalization of the dark matter contribution (*ie.* the annihilation rate) to take on the value that best matches the data. In each case, we find excellent fits to the observations, although very high annihilation rates are required. We quantify this by introducing a boost factor (BF) which denotes the enhancement of the annihilation rate relative to a smooth halo with a local dark matter density of 0.3 GeV/cm$^3$. We find that boost factors in the range of approximately 400 to 2000 are required to produce the positron fraction observed by PAMELA. Although such values are considerably larger than those generally predicted based on the results of N-body simulations , such simulations do not have the resolution to study the small scale structure of the galactic halo distribution and rely on extrapolations to estimate annihilation boost factors. For this reason, we do not dismiss the possibility that small scale inhomogeneities in the dark matter distribution could significantly boost the annihilation rate.

In Fig. , we show the cosmic ray electron plus positron spectrum, with contributions from the same dark matter model and parameters shown in Fig. , compared to the measurements of ATIC . Here, we have attempted to account for possible systematic errors (which are not quantified in Ref. ) by including 5% systematic errors (in addition to the statistical errors shown) in the calculation of the $\chi^2$, and by allowing for an overall shift in the normalization/exposure of up to 25%. Although very limited information is available regarding the systematic errors involved in these observations, we feel that we have made reasonable estimates of these quantities.

Based on the result shown in Figs.  and , we conclude that KK dark matter is capable of providing a good fit to the combined observations of PAMELA and ATIC. In particular, we find acceptable values for the $\chi^2$ per degree of freedom for each mass and propagation model we have considered (although propagation model B provides somewhat better fits than model A). We would like to emphasize that the mass of the dark matter particle in this model is quite constrained by relic abundance considerations, and could not easily have been very different from the values we have considered here. In particular, to obtain a thermal relic abundance of KK dark matter, masses in the approximate range of 600 to 900 GeV are generally required . Furthermore, KK masses lighter than about 300-400 GeV are inconsistent with electroweak precision measurements .

Before we can conclude that KK dark matter in this model is successful in producing the signals of PAMELA and ATIC, we must consider constraints from measurements of cosmic ray antiprotons, gamma rays, and synchrotron emission. In particular, dark matter annihilations which produce mostly quarks or gauge bosons are typically expected to produce more cosmic ray antiprotons than are observed by PAMELA  if the overall annihilation rate is normalized to generate the PAMELA and ATIC positron/electron excesses. This problem is somewhat mitigated in the case of KK dark matter, however, as most of the annihilations proceed to charged leptons (which efficiently generate electrons and positrons, while not contributing to the antiproton flux). In Fig. , we show the contribution to the cosmic ray antiproton-to-proton ratio from KK dark matter, for the parameters (annihilation rates, masses, and propagation models) used in Figs.  and . The upper two lines in this figure correspond to the results found using propagation model A, and clearly exceed the ratio measured by PAMELA . The lower two lines, in contrast, denote the results using propagation model B and are consistent with PAMELA's antiproton measurement. We thus conclude that in order for KK dark matter to produce the PAMELA and ATIC signals without overproducing antiprotons, a propagation model with a rather narrow diffusion zone ($L\sim 1$ kpc) must be adopted. We remind the reader that such a propagation model is completely consistent with all current cosmic ray data.

Other constraints on dark matter annihilations in the Milky Way include those obtained by gamma ray, radio, and microwave observations of the Galactic Center region. If an NFW-like halo profile is adopted and the annihilation rate throughout the halo is enhanced by a common boost factor (normalized to the PAMELA and ATIC signals), these constraints are likely to be exceeded . As baryonic effects are anticipated to modify the dark matter distribution within the inner kiloparsecs (or parsecs) of the galaxy, however, it is difficult to arrive at any strong conclusions. In particular, the boost factors which enhances the dark matter annihilation rate could potentially vary with location, and could be considerably lower in the inner kiloparsecs of the Milky Way than in the vicinity of the Solar System. For example, if large boost factors exist, it would appear to imply the presence of a very large number of small sub-halos within the Milky Way. In the regions of the galaxy with the most stars (*ie.* the inner galaxy), tidal interactions are the most likely to destroy a large fraction of the sub-halos, leading to smaller boost factors relative to those further the Galactic Center .

Instead of limiting our discussion to the specific KK dark matter candidate described here, we could retain much of the same phenomenology while considering a generic class of dark matter candidates which annihilate through couplings to hypercharge without chirality suppression and, thus, efficiently produce electrons and other charged leptons. In this more general context, we can imagine ways in which the need for a large boost factors could be relaxed. In particular, dark matter particles with considerably larger cross sections than those implied by naive thermal abundance arguments are possible. For example, the dark matter particles may have been produced through a non-thermal mechanism in the early universe. Alternatively, the annihilation cross section could be strongly enhanced at low velocities due to non-perturbative processes , leading to a very high annihilation rate in the galactic halo. The results found for the case of KK dark matter in this letter can be applied to this broader classes of dark matter candidates, enabling the positron fraction and electron spectrum reported by PAMELA and ATIC to be generated without overproducing cosmic ray antiprotons, and potentially without the need of large boost factors.

In conclusion, we have shown here that Kaluza-Klein dark matter in a model with one universal extra dimension is capable of generating a spectrum of cosmic ray positrons and electrons consistent with the recent observations of PAMELA and ATIC. We have also shown that this can be accomplished without exceeding the measured quantity of cosmic ray antiprotons.

It is interesting to note that the mass of the Kaluza-Klein dark matter particle required to generate the observed dark matter abundance in this model is very similar to the value required to produce the feature in the cosmic ray electron spectrum reported by ATIC. Despite the overall lack of freedom in this model (the approximate mass and dominant annihilation channels are determined by the model), the PAMELA and ATIC signals can each be naturally accommodated. The least attractive feature of this scenario is the high annihilation rate (*ie.* large boost factor) that is required to normalize the overall signal to the PAMELA and ATIC data.

We would like to thank Joakim Edsjo for him help with DarkSUSY. This work has been supported by the US Department of Energy, including grant DE-FG02-95ER40896, and by NASA grant NAG5-10842.