Question: Represent the article for finding a single-sentence TLDR

Question: Over the passage of time Unmanned Autonomous Vehicles (UAVs), especially Autonomous flying drones grabbed a lot of attention in Artificial Intelligence. Since electronic technology is getting smaller, cheaper and more efficient, huge advancement in the study of UAVs has been observed recently. From monitoring floods, discerning the spread of algae in water bodies to detecting forest trail, their application is far and wide. Our work is mainly focused on autonomous flying drones where we establish a case study towards efficiency, robustness and accuracy of UAVs where we showed our well supported through experiments. We provide details of the software and hardware architecture used in the study. We further discuss about our implementation algorithms and present experiments that provide a comparison between three different state-of-the-art algorithms namely TrailNet, InceptionResnet and MobileNet in terms of accuracy, robustness, power consumption and inference time. In our study, we have shown that MobileNet has produced better with very less computational requirement and power consumption. We have also reported the challenges we have faced during our work as well as a brief discussion on our future work to improve safety features and performance. In modern era, UAVs have become very popular and have basic intelligence of being driven autonomously. Talking about ground traffic, these vehicles have limitations of physical paths and barriers. However, such is not the case with flying objects like drones as they do not suffer from such physical limitations. Autonomous Flying objects are in much of discussion these days and are striding all across the realm -traffic monitoring BID9, agriculture BID11, inventory management BID4, surveillance BID15, data mining, disaster response BID10, etc. As their areas of application increase, it becomes more important to find algorithms well suited for these kind of vehicles. Some applications may not require the drone to be extremely accurate, but may require it to work for longer durations e.g. in surveillance applications while others may require it to be very precise but may not require it to work for long duration e.g. in delivery of items. In the last decade, significant changes have been observed in the field of autonomous motion planning of vehicles, UAVs in particular. The motion planning of UAVs are distinctly difficult because of several complexities that comes with aerial vehicles. The salience of differential constraints, uncertainty in the vehicle state and limited knowledge about the environment makes it impossible to have a precise pre-computed plan to follow through. These differences considerably gave rise to various approaches and techniques for planning the motion of these unmanned autonomous vehicles. The ambiguity of different algorithms and its inherent adversity pose an intriguing scope for a bench-marking study to enhance accuracy, robustness, power consumption, safety and inference time and fine tuning it further. Throughout this paper, we bear in mind some of the generic characteristics and prerequisites relating to UAVs. The basic design of a UAV is modelled to have acceleration and velocity constraints. Furthermore, the higher-order differential constraints also associate themselves with the equation of motion of a drone. However, the coherent objective involved in all UAVs is to guide the vehicle towards a goal. In this paper, we introduce to the best of our knowledge, a very first comparative study of three algorithms in order to find a better motion control of a drone for detecting a trail. In order to be able to compare a set of algorithms in a meticulous way, it is necessary to establish their precision and robustness and to evaluate its power consumption as well as inference time. Along with these metrics established for a particular algorithm, it is also necessary to consider the distinct areas of its application. Only then, based on the requirements called for by a particular application, a reasonable opinion about an algorithm is to be formed. Our study covers recent developments and algorithms used in the area of trail detection by UAVs and runs down as comprehensively as possible what has been already upheld regarding these algorithms. We briefly review the recent applications of drones and Unmanned Aerial Vehicles(UAVs) and their challenges. Our work is partly inspired by the research of BID13 on a MAV(Micro Aerial Vehicles) system for autonomous trail. They introduced and trained a deep neural network named Trailnet to estimate the view orientation and lateral offset of the MAV with respect to the trail center. They ran all their vision systems in real time on NVIDIA Jetson TX1. We have presented a detailed study on how we can increase their accuracy and lower their training time. We selected InceptionResnet BID14 and over their proposed.Inception architecture BID14 has been shown to achieve very good performance at relatively low computational cost. Recently, the introduction of residual connections in conjunction with a more traditional architecture has yielded state-of-art performance in the 2015 ILSVRC challenge. Its performance was similar to the latest generation Inception-v3 network. There is also some evidence of residual Inception networks outperforming similarly expensive Inception networks without residual connections by a thin margin. In BID7 a class of efficient models called MobileNets was first presented for embedded vision applications. They are based on a streamlined architecture which uses depth-wise separable convolutions to build light weight deep neural networks. Two simple global hyper-parameters introduced efficiently trade off between latency and accuracy. MobileNets are built primarily from depth-wise separable convolutions initially introduced in Sif and subsequently used in Inception models BID8 ) to reduce the computation in the first few layers. In this paper, we have tried to put forward a comparative study to find an algorithm well suited for UAVs. For this task, we selected 3 algorithms from state-of-the-art namely Trailnet, InceptionResnet and Mobilenet. We have chosen Inception-ResNet and MobileNet for our experiments as they were proven to perform very well on classification tasks. Our aim is to train these algorithms offline, implement them on the drone and use their output commands to correctly navigate the drone through trails. We have used Udacity simulator to simulate the path generated by the output of these algorithms and to compare their robustness during long trails. An important goal of our method is to demonstrate the effectiveness of low cost systems for the complex task of flying an autonomous drone. In this section, we want to put forward the architecture (both hardware and software) that we have used for our research in order to achieve our . We will further discuss about the basis of choosing Jetson TX2 for computing and how it has shaped our upshot. Also, we will talk about using ROS together with Ubuntu L4T and the hardware configuration of our drone. ).We have also used a Windows 10, Intel(R) Xeon(R) Silver 4114 CPU @2.20GHz(20 CPUs), 2.2GHz, 32GB RAM with Nvidia Titan Xp GPU for training purposes. 3. ZED Camera: We have used ZED stereo camera, which is a standard 3D USB camera for object detection and image capturing. It adds depth perception, positional tracking and 3D mapping using advanced sensing technology based on human stereo vision. 4. Battery: We use Venom 35C, 14.8V, 5000 mAh lithium polymer battery which is best suitable for our drone 5. GPS: We use Holybro Ublox Neo-M8N GPS module which provides high sensitivity and minimal acquisition time while maintaining low system power. 1. Jetpack -Jetpack is an SDK released by NVIDIA and is the most comprehensive solution for building artificially intelligent applications in recent times jet. Our Jetson Developer Kit (TX2) on the drone was first flashed with the latest OS image -Ubuntu 16.04 (Xenial). We further used JetPack installer to install developer tools for both Host and the Drone (the developer kit on the drone). All the libraries, TensorRT, cuDNN, CUDA, OpenCV to name some were installed to jumpstart our development environment. For our case study, we used JetPack-L4T 3.2. 2. ROS -ROS or Robot Operating System is a collection of Linux based frameworks acting as robotics middleware. We have used ROS Kinetic Kame kin for hardware abstraction of Joystick and the Controller present on the drone. ROS also plays an important part in controlling devices at a very low-level as well as for transferring data. A ROS system typically consists of a number of independent nodes. For our environment, these nodes are MAVROS, Control Node, DNN, Camera and Joystick -each of which is able to communicate with each other using subscribe or publish messaging model. All nodes are registered with the master node (MAVROS for our case) which in turn helps them to find and communicate with each other. The MAVROS enables MAVLink (Micro Air Vehicle Link) mav protocol to communicate with the PX4 (FCU) on-board.3. Communication -The on-board Jetson TX2 had wifi access point enabled by us before installing it on the drone. As a , the host PC could connect to the Jetson TX2 wirelessly and was able to access it remotely. By sending commands through the terminal, the host was able to control the drone by Secure Socket Shell (SSH) network protocol. 4. Udacity Simulator -Learning about UAVs comes with a lot of risks considering the constant fright of crashing. From testing new hardwares, learning to control the flight controller unit of the drone to further fine tuning of our algorithms, the ing failures can be really expensive. To overcome this we have used a simulator offered by Udacity in its Flying Car nanodegree program. FIG2 demonstrates the state transition diagram of the drone using this simulator. We use standard UART(Universal Asynchronous Receiver/Transmitter) communications for controls. The transmitting UART converts parallel data from a controlling device like CPU into serial form, transmits it in serial form to receiving UART, which then converts the serial data back to parallel for the receiving device. FIG4 shows the workflow involving all the modules and communication between them. For our experiments, we use IDSIA trail dataset BID6 which were collected on Swiss Alps forest trails. It consists of approximately 15GB of image data acquired using different cameras. The entire dataset is divided into 15 parts, in the form of folders numbered from 000 till 014. We have used folders 001, 002, 004, 005, 006, 007 and 009 for training our model and used folders 003, 008 and 010 for validation. We ran our tests on folder 012. We had to discard folders 000, 013 and 014 as they comprise of only preliminary test data. In this section we first go through the model selections that we made by considering their performances on ImageNet Challenge BID12. ImageNet project in an ongoing large visual database designed for use in visual object recognition software research. Since 2010, ImageNet has been running an annual competition in visual recognition where participants are provided with 1.2 Million images belonging to 1000 different classes. Several deep learning architectures have been proposed since then and we have considered two of them for our experiments. We have chosen the models based on their performance on Imagenet Challenge BID12 considering accuracy and computation as major metrics. Accuracy is one of the critical aspect in building deep learning models. It depends on network architecture and the amount of data available for training. Also most of the convolutional neural networks (ConvNets) have huge memory and computational requirements especially during training. This is an important concern specifically for our purpose as memory and space footprint is limited in embedded AI devices such as drones. Also size of the final trained model becomes important to consider as we will be deploying the models to run locally on drone. In general, more computationally intensive networks usually tends to produce more accuracy. Hence, there is always a trade-off between accuracy and computational cost. Apart from these, there are many other factors which are important in selecting models such as training time, ability of a network to generalize well, inference time etc. Considering all of these factors, we have chosen MobileNet BID7 Transfer learning is a Machine Learning technique to store knowledge gained on solving one problem and applying it to a different related problem. The three major transfer learning scenarios are using ConvNet as fixed feature extractor, fine-tuning the ConvNet and using pre-trained models. The two most important factors that help us to decide on what type of transfer learning we should perform on new dataset are size of the dataset and similarity to original dataset. Since our dataset is large enough and different from dataset, we have decided to fine-tune our models. Fine-tuning is a transfer learning strategy to replace the final layers of the ConvNets and tweak them so that they can learn more robust features relevant to our problem. We then retrain the classifier on the new dataset and fine-tune the weights of the pre-trained network by continuing the back propagation. Usually the earlier features of a ConvNet contain more generic features (like edge detectors, color detectors etc.) which are useful for many image detecting tasks. Later layers of ConvNets extract minute detailed features specific to the classes contained in our problem. We have fine-tuned both InceptionResnet and MobileNet by initializing them with their weights pretrained on ImageNet BID5 and retrain the model from scratch. In this section we discuss the of our experiments with comparisons in terms of size of the models, accuracy, inference time and power consumption. Table 2 compares the architectural complexity, training time and accuracy of TrailNet, InceptionResnet and MobileNet on our dataset. It is evident from our observations that both InceptionResnet and MobileNet have an accuracy better than TrailNet. Also the training time and computational cost involved for MobileNet is much less than that of other two models because of its less complexity, even though its accuracy is on par with InceptionResnet model. We have performed testing of our models by running them on Udacity simulator. We have used a dataset consisting of 2000 images pertaining to a trail path of approximately 100 meters and simulated the drone by the commands given by output of the algorithms. FIG5 shows the path traversed by the drone autonomously on the test data environment, where a) is the path traversed by drone manually controlled using simulator, b) is the path traversed when the simulator is controlled with output of Inception-Resnet and c) is the path traversed by drone when controlled with output of MobileNet. It can be seen that the drone is more robust and pretty much followed the ground truth path using both the models. We have measured the inference time (i.e., time taken by the model to predict output of test dataset) on Jetson TX2 machine and the following table shows the comparison. From our observations in TAB2, MobileNet has a very less inference time, which means the cameras with high frame rate can be operated using the MobileNet model for more time and with less energy consumption compared to other two models. We have calculated our power draw values on Nvidia Titan Xp GPU with a maximum possible power draw of 300W. Also the average power drawn with idle GPU is 15W. We can see that MobileNet draws very less power compared to other two models which in long battery life of the drone. We have taken a batch size of 32 for inference power draw calculation. Table 2, MobileNet has a very less inference time, which means the cameras with high frame rate can be operated using the MobileNet model for more time and with less energy consumption compared to other two models. Note that we have not trained TrailNet model from scratch and so we have not reported the peak power consumed in this case. While producing this work, we have encountered several challenges. Few of these challenges are listed below:1. The major challenge encountered was to run our DNN models on the physical drone in real time due to a hardware bug we were facing with the FCU.2. We could not train our models from scratch due to lack of significant amount of dataset. Additionally, we handled a lot of issues to make our models more stable and robust. Since in each trial, the number of images in each class (left, straight and right) were different, there was a lot of data imbalance which we solved by upsampling and downsampling the dataset.3. Due to the large number of training parameters at the beginning, our models were overfitted. We eliminated over-fitting by introducing several data augmentation techniques (random flipping, random rotation, random contrast and transition etc.). We further included regularization (especially dropout layers) in order to reduce network complexity.4. Power is one of the important factors especially in mobile embedded devices with small size and computational power. Typically, deep learning algorithms consume more power specifically for the real time inference. We have made an estimate of the power consumption of each of our model by calculating the GPU power drawn by them but we could not test how long our drone would run implementing each of these models due to the hardware bug mentioned before. In this paper, we have presented a comparison between 3 algorithms -TrailNet, InceptionResnet and MobileNet in terms of accuracy, computational cost, power consumption, inference time and robustness. The choice of algorithm for UAVs varies on the basis of several factors. In our work, we have worked with some of the factors which we thought would be pivotal in algorithm selection considering reasonable comparisons. We observed in our study that MobileNet outperformed others with very less computational requirement and power consumption. Hence in our opinion, MobileNet is more befitting for drones and other embedded devices compared to TrailNet and InceptionResnet. Safety is another major concern in terms of drones. There can be many situations such as collision with the objects, external disturbances like winds, chances of drone moving out of manual controller zone, battery issues, chances of getting stolen and other safety hazards. We will be implementing these drone related safety features in our future work.

Represent the TLDR for finding the corresponding scientific article

case study on optimal deep learning model for UAVs

Data stream algorithms can be improved using deep learning, while retaining performance guarantees.


Question: Represent the scientific article to find a short TLDR!

Question: Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer , a sequence model based on self-attention, has achieved compelling in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance . This is impractical for long sequences such as musical compositions since their memory complexity is quadratic in the sequence length. We propose an algorithm that reduces the intermediate memory requirements to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long (thousands of steps) compositions with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-competition, and obtain state-of-the-art on the latter. A musical piece often consists of recurring elements at various levels, from motifs to phrases to sections such as verse-chorus. To generate a coherent piece, a model needs to reference elements that came before, sometimes in the distant past, and then repeat, vary, and further develop them to create contrast and surprise. Intuitively, self-attention could be a good match for this task. Self-attention over its own previous outputs allows an autoregressive model to access any part of the previously generated output at every step of generation. By contrast, recurrent neural networks have to learn to proactively store elements to be referenced in a fixed size state or memory, making training potentially much more difficult. We believe that repeating self-attention in multiple, successive layers of a Transformer decoder BID17 can help capture the multiple levels at which self-referential phenomena exist in music. In its original formulation, the Transformer relies on absolute position representations, using either positional sinusoids or learned position embeddings that are added to the per-position input representations. Recurrent and convolutional neural networks instead model position in relative terms: RNNs through their recurrence over the positions in their input, and CNNs by applying kernels that effectively choose which parameters to apply based on the relative position of the covered input representations. Music has multiple dimensions along which relative differences arguably matter more than their absolute values; the two most prominent are timing and pitch. To capture such pairwise relations between representations, BID13 introduce a relation-aware version of self-attention which they use successfully to modulate self-attention by the distance between two positions. We extend this approach to capture relative timing and optionally also pitch, which yields improvement in both sample quality and perplexity for the JSB Chorales dataset. As opposed to the original Transformer, samples from a Transformer with our relative attention mechanism maintain the regular timing grid present in this dataset. The model furthermore captures global timing, giving rise to regular phrases. The original formulation of relative attention BID13 requires O(L 2 D) memory where L is the sequence length and D is the dimension of the model's hidden state. This is prohibitive for long sequences such as those found in the Maestro dataset of human-performed virtuosic, classical piano music BID7. In Section 3.4, we show how to reduce the memory requirements to O(LD), making it practical to apply relative attention to long sequences. The Maestro dataset consists of MIDI recorded from performances of competition participants, bearing expressive dynamics and timing on a less than 10-millisecond granularity. Discretizing time in a fixed grid on such a resolution would yield unnecessarily long sequences as not all events change on the same timescale. We hence adopt a sparse, MIDI-like, event-based representation from , allowing a minute of music with a 10-millisecond resolution to be represented at lengths around 2K. This is in contrast to a 6K to 18K length that would be needed on a serialized multi-attribute fixed-grid representation. As position in sequence no longer corresponds to time, a priori it is not obvious that relative attention should work as well with such a representation. However, we will show in Section 4.2 that it does improve perplexity and sample quality over strong baselines. We speculate that idiomatic piano gestures such as scales, arpeggios and other motifs all exhibit a certain grammar and recur periodically, hence knowing their relative positional distances makes it easier to model this regularity. This inductive bias towards learning relational information, as opposed to patterns based on absolute position, suggests that the Transformer with relative attention could generalize beyond the lengths it was trained on, which our experiments in Section 4.2.1 confirm. We show the first successful use of Transformers in generating music that exhibits long-term structure. Before our work, LSTMs were used at timescales of 15s (~500 tokens) of piano performances . Our work demonstrates that Transformers not only achieve state-of-the-art perplexity on modeling these complex expressive piano performances, but can also generate them at the scale of minutes (thousands of tokens) with remarkable internal consistency. Our relative self-attention formulation is essential to the model's quality. In listening tests (see Section 4.2.3), samples from models with relative self-attention were perceived as more coherent than the baseline Transformer model BID17. Relative attention not only enables Transformers to generate continuations that elaborate on a given motif, but also to generalize and generate in consistent fashion beyond the length it was trained on (see Section 4.2.1). In a seq2seq setup, Transformers can generate accompaniments conditioned on melodies, enabling users to interact with the model. The space complexity of the relative self-attention mechanism in its original formulation BID13 made it infeasible to train on sequences of sufficient length to capture long-range structure in longer musical compositions. To address this, we present a crucial algorithmic improvement to the relative self-attention mechanism, dramatically reducing its memory requirements from O(L 2 D) to O(LD). For example, the memory consumption per layer is reduced from 8.5 GB to 4.2 MB (per head from 1.1 GB to 0.52 MB) for a sequence of length L = 2048 and hidden-state size D = 512 (per head D h = D H = 64, where number of heads is H = 8) (see TAB1), allowing us to use GPUs to train the relative self-attention Transformer on long sequences. Sequence models have been the canonical choice for modeling music, from Hidden Markov Models to RNNs and Long Short Term Memory networks (e.g., BID3 ;), to bidirectional LSTMs (e.g., BID6 . Successful application of sequential models to polyphonic music often requires serializing the musical score or performance into a single sequence, for example by interleaving different instruments or voices. Alternatively, a 2D pianoroll-like representation (see A.1 for more details) can be decomposed into a sequence of multi-hot pitch vectors, and their joint probability distributions can be captured using Restricted Boltzmann Machines BID14 BID8 or Neural Autoregressive Distribution Estimators (NADE; BID10 . Pianorolls are also image-like and can be modeled by CNNs trained either as generative adversarial networks (GAN; BID4 . (e.g., BID2 or as orderless NADEs BID15 BID19) (e.g., BID9 . use self-similarity in style-transfer fashion, where the self-similarity structure of a piece serves as a template objective for gradient descent to impose similar repetition structure on an input score. Self-attention can be seen as a generalization of self-similarity; the former maps the input through different projections to queries and keys, and the latter uses the same projection for both. Dot-product self-attention is the mechanism at the core of the Transformer, and several recent works have focused on applying and improving it for image generation, speech, and summarization BID12). A key challenge encountered by each of these efforts is scaling attention computationally to long sequences. This is because the time and space complexity of self-attention grows quadratically in the sequence length. For relative self-attention BID13 this is particularly problematic as the space complexity also grows linearly in the dimension, or depth, of the per-position representations. 3.1 DATA REPRESENTATION We take a language-modeling approach to training generative models for symbolic music. Hence we represent music as a sequence of discrete tokens, with the vocabulary determined by the dataset. Datasets in different genres call for different ways of serializing polyphonic music into a single stream and also discretizing time. The JSB Chorale dataset consists of four-part scored choral music, which can be represented as a matrix where rows correspond to voices and columns to time discretized to sixteenth notes. The matrix's entries are integers that denote which pitch is being played. This matrix can than be serialized in raster-scan fashion by first going down the rows and then moving right through the columns (see A.1 for more details). Compared to JSB Chorale, the piano performance data in the Maestro dataset includes expressive timing information at much finer granularity and more voices. For the Maestro dataset we therefore use the performance encoding proposed by which consists of a vocabulary of 128 NOTE_ON events, 128 NOTE_OFFs, 100 TIME_SHIFTs allowing for expressive timing at 10ms and 32 VELOCITY bins for expressive dynamics (see A.2 for more details). The Transformer decoder is a autoregressive generative model that uses primarily self-attention mechanisms, and learned or sinusoidal position information. Each layer consists of a self-attention sub-layer followed by a feedforward sub-layer. DISPLAYFORM0 DISPLAYFORM1 The attention outputs for each head are concatenated and linearly transformed to get Z, a L by D dimensional matrix. A upper triangular mask ensures that queries cannot attend to keys later in the sequence. For other details of the Transfomer model, such as residual connections and learning rates, the reader can refer BID17. The feedforward (FF) sub-layer then takes the output Z from the previous attention sub-layer, and performs two layers of point-wise dense layers on the depth D dimension, as shown in Equation 2. W 1, W 2, b 1, b 2 are weights and biases of those two layers. DISPLAYFORM2 3.3 RELATIVE POSITIONAL SELF-ATTENTION As the Transformer model relies solely on positional sinusoids to represent timing information, BID13 introduced relative position representations to allow attention to be informed by how far two positions are apart in a sequence. This involves learning a separate relative position embedding E r of shape (H, L, D h), which has an embedding for each possible pairwise distance r = j k − i q between a query and key in position i q and j k respectively. The embeddings are ordered from distance −L + 1 to 0, and are learned separately for each head. In BID13, the relative embeddings interact with queries and give rise to a S rel, an L × L dimensional logits matrix which modulates the attention probabilities for each head as: DISPLAYFORM3 We dropped head indices for clarity. Our work uses the same approach to infuse relative distance information in the attention computation, while significantly improving upon the memory footprint for computing S rel. For each head, BID13 DISPLAYFORM4, containing the embeddings that correspond to the relative distances between all keys and queries. Q is then reshaped to an (L, 1, D h) tensor, and S rel = QR. 2 This incurs a total space complexity of O(L 2 D), restricting its application to long sequences. We improve the implementation of relative attention by reducing its intermediate memory requirement DISPLAYFORM0, with example lengths shown in TAB1. We observe that all of the terms we need from QR are already available if we directly multiply Q with E r, the relative position embedding. After we compute QE r, its (i q, r) entry contains the dot product of the query in position i q with the embedding of relative distance r. However, each relative logit (i q, j k) in the matrix S rel from Equation 3 should be the dot product of the query in position i q and the embedding of the relative distance j k − i q, to match up with the indexing in QK. We therefore need to "skew" QE r so as to move the relative logits to their correct positions, hence S rel = Skew(QE r). The "skewing" procedure is illustrated in FIG0 and will be detailed in the next section. The time complexity for both methods are O(L 2 D), while in practice our method is 6x faster at length 650 as prior work still requires manipulating larger tensors. Hence, we propose a "skewing" procedure to transform an absolute-by-relative (i q, r) indexed matrix into an absolute-by-absolute (i q, j k) indexed matrix. The row indices i q stay the same while the columns indices are shifted according to the following equation: FIG0 the upper right green dot in position of QE r after skewing has a column index of 2 − (3 − 1) + 0 = 0, ing in a position of in S rel. DISPLAYFORM1 We outline the steps illustrated in FIG0 below.1. Pad a dummy column vector of length L before the leftmost column.2. Reshape the matrix to have shape (L+1, L). (This step assumes NumPy-style row-major ordering.)3. Slice that matrix to retain only the last l rows and all the columns, ing in a (L, L) matrix again, but now absolute-by-absolute indexed, which is the S rel that we need. For very long sequences, the quadratic memory requirement of even baseline Transformer is impractical. Local attention has been used for example in Wikipedia and image generation (; by chunking the input sequence into non-overlapping blocks. Each block then attends to itself and the one before, as shown by the smaller thumbnail on the top right corner of FIG2 .To extend relative attention to the local case, we first note that the right block has the same configuration as in the global case (see FIG0) but much smaller: DISPLAYFORM0 where M is the number of blocks, and N be the ing block length) as opposed to L 2. The left block is unmasked with relative indices running from -1 (top right) to -2N + 1 (bottom left). Hence, the learned E r for the local case has shape (2N − 1, N).Similar to the global case, we first compute QE r and then use the following procedure to skew it to have the same indexing as QK, as illustrated in FIG2 1. Pad a dummy column vector of length N after the rightmost column.2. Flatten the matrix and then pad with a dummy row of length N − 1. 4.1 J.S. BACH CHORALES J.S. Bach Chorales is a canonical dataset used for evaluating generative models for music 3 (e.g., BID0 BID1 ; BID5 BID9 . It consists of score-based four-part chorales. We first discretize the scores onto a 16th-note grid, and then serialize them by iterating through all the voices within a time step and then advancing time (see A.1 for more details). As there is a direct correspondence between position in sequence and position on the timing/instrument grid in a piece, adding relative position representations could make it easier to learn this grammar. We indeed see relative attention drastically improve negative log-likelihood (NLL) over baseline Transformer TAB3 ). This improvement is also reflected in sample quality. The samples now maintain the necessary timing/instrument grid, always advancing four steps before advancing in time. As local timing is maintained, the model is able to capture timing on a more global level, giving rise to regular phrasing, as shown in the right score of FIG1. In addition to relative attention, we explored enhancing absolute timing through concatenating instead of adding the sinusoids to the input embeddings. This allows the model to more directly learn its absolute positional mapping. This further improves performance for both the baseline and relative transformer TAB3. We compare against COCONET as it is one of the best-performing models that has also been evaluated on the 16-note grid using the canonical dataset split. To directly compare, we re-evaluated COCONET to obtain note-wise losses on the validation set 4. For the Transformer models (abbreviated as TF), we implemented our attention mechanisms in the Tensor2Tensor framework. We use 8 heads, and keep the query, key (att) and value hidden size (hs) fixed within a config. We tuned number of layers (L in {4,5,6}), attention hidden size (att in {256, 512}) and pointwise feedforward hidden size (ff in {512, 1024}). A musical event bears multiple attributes, such as timing, pitch, instrument etc. To capture more relational information, we extend relative attention to capture pairwise distances on additional attributes. We learn separate relative embeddings for timing E t and also pitch E p. E t has entries corresponding to how many sixteenth notes apart are two positions in time, while E p embeds the pairwise pitch interval. However this approach is not directly scalable beyond J.S. Bach Chorales because it involves explicitly gathering relative embeddings for R t and R p, ing in a memory complexity of O(L 2 D) as in BID13. This is due to relative information being computed based on content as opposed to content-invariant information such as position in sequence. It was sufficient to add these extra relational information to the first layer, perhaps because it is closest to the raw input content. Here, the relative logits are computed from three terms, We train on random crops of 2048-token sequences and employ two kinds of data augmentation: pitch transpositions uniformly sampled from {−3, −2, . . ., 2, 3} half-steps, and time stretches uniformly sampled from the set {0.95, 0.975, 1.0, 1.025, 1.05}. For evaluation, we segment each sequence sequentially into 2048 length subsequences and also keep the last subsequences that are of shorter lengths. This in 1128 and 1183 subsequences in the validation and test set respectively. Each subsequence is then evaluated by running the model forward once with teaching forcing. As the subsequences vary in length, the overall negative loglikelihood (NLL) is averaged entirely on the token level. We compare our to PerformanceRNN (LSTM, which first used this dataset) and LookBack RNN (LSTM with attention) BID19. LookBack RNN uses an input representation that requires monophonic music with barlines which is information that is not present 5 Maestro dataset: urlhttps://magenta.tensorflow.org/datasets/maestro 6 Piano-e-Competition: http://www.piano-e-competition.com/ 7 COCONET is an instance of OrderlessNADE, which approximates a mixture model over orderings where orderings are assumed to be uniformly distributed. Hence, its loss is computed by averaging losses over multiple random orderings. The current row in the table reports this loss. In contrast, the row above corresponds to evaluating Coconet as an autoregressive model under the chronological ordering. BID9 show that sample quality is better when using Gibbs sampling (which uses conditionals from multiple orderings) as opposed to autoregressive generation (which only uses conditionals from one ordering). DISPLAYFORM0 in performed polyphonic music data, hence we simply adopt their architecture. TAB4 shows that Transformer-based architectures fits this dataset better than LSTM-based models. We implemented our attention mechanisms in the Tensor2Tensor framework, and used the default hyperparameters for training, with 0.1 learning rate and early stopping. We compare four architectures, varying on two axes: global versus local, and regular versus relative attention. We found that reducing the query and key channel size (att) to three forth of the hidden size (hs) works well for this dataset and used this setup for all of the models, while tuning on number of layers (L) and dropout rate (d). We use block size (bs) 512 for local attention (; . For relative global attention, the maximum relative distance to consider is set to half the training sequence length. For relative local attention, it is set to the full memory length which is two blocks long. TAB4 shows that our memory-efficient relative attention formulations outperform regular attention in both the global and the local case. When looking at the other axes, we see global attention outperforming local attention in both the relative and regular case. Global attention may have the advantage of being able to directly look back for repeating motifs. With a larger dataset, local attention may fare well as it allows for much deeper models and longer sequences, as seen in text and image generation work . In turn, both domains could benefit from the use of relative local attention. When primed with an initial motif (Chopin's Étude Op. 10, No. 5) shown in the top left corner of FIG4, we see the models perform qualitatively differently. Transformer with relative attention elaborates the motif and creates phrases with clear contour which are repeated and varied. Baseline Transformer uses the motif in a more uniform fashion, while LSTM uses the motif initially but soon drifts off to other material. Note that the generated samples are twice as long as the training sequences. Relative attention was able to generalize to lengths longer than trained but baseline Transformer deteriorates beyond its training length. See Appendix C for visualizations of how our Relative Transformer attends to past motifs. To explore the sequence-to-sequence setup of Transformers, we experimented with a conditioned generation task where the encoder takes in a given melody and the decoder has to realize the entire performance, i.e. melody plus accompaniment. The melody is encoded as a sequence of tokens as in BID19, quantized to a 100ms grid, while the decoder uses the performance encoding described in Section 3.1 (and further illustrated in A.2). We use relative attention on the decoder side and show in TAB5 that it also improves performance. To compare the perceived sample quality of models trained on the Maestro dataset, and their ability to generate a continuation for a priming sequence, we carried out a listening test study comparing the baseline Transformer, our Transformer with relative-attention, PerformanceRNN (LSTM), and the validation set. Participants were presented with two musical excerpts (from two different models that were given the same priming sequence) and asked to rate which one is more musical on a Likert scale. For each model, we generated 10 samples each with a different prime, and compared them to three other models, ing in 60 pairwise comparisons. Each pair was rated by 3 different participants, yielding a total of 180 comparisons. Figure 5 shows the number of comparisons in which an excerpt from each model was selected as more musical. The improvement in sample quality from using relative attention over the baseline Transformer model was statistically significant (see Appendix B for the analysis), both in aggregate and between the pair. Even though in aggregate LSTMs performed better in the study than the Transformer, despite having higher perplexity, but when compared against each other head to head, the were not statistically significant (see TAB6 in Appendix B).(Ours) Figure 5: Samples from the Transformer with our efficient relative attention were rated more musical more times than LSTM and baseline Transformer. Error bars show standard deviation of the mean. In this work we demonstrated that the Transformer equipped with relative attention is very well-suited for generative modeling of symbolic music. The compelling long-term structure in the samples from our model leaves us enthusiastic about this direction of research. Moreover, the ability to expand upon a prime, in particular, suggests potential applications as creative tool. The significant improvement from relative attention highlights a shortcoming of the original Transformer that might also limit its performance in other domains. Improving the Transformer's ability to capture periodicity at various time scales, for instance, or relations between scalar features akin to pitch could improve time-series models. Our memory-efficient implementation enables the application of relative attention to much longer sequences such as long texts or even audio waveforms, which significantly broadens the range of problems to which it could be applied. Adapting sequence models for music requires making decisions on how to serialize a polyphonic texture. The data type, whether score or performance, makes certain representations more natural for encoding all the information needed while still ing in reasonable sequence lengths. A.1 SERIALIZED INSTRUMENT/TIME GRID (USED FOR THE J.S.BACH CHORALES DATASET) The first dataset, J.S. Bach Chorales, consists of four-part score-based choral music. The time resolution is sixteenth notes, making it possible to use a serialized grid-like representation. Figure 6 shows how a pianoroll (left) can be represented as a grid (right), following BID9. The rows show the MIDI pitch number of each of the four voices, from top to bottom being soprano (S), alto (A), tenor (T) and bass (B), while the columns is discretized time, advancing in sixteenth notes. Here longer notes such as quarter notes are broken down into multiple repetitions. To serialize the grid into a sequence, we interleave the parts by first iterating through all the voices at time step 1, and then move to the next column, and then iterate again from top to bottom, and so on. The ing sequence is S 1 A 1 T 1 B 1 S 2 A 2 T 2 B 2..., where the subscript gives the time step. After serialization, the most common sequence length is 1024. Each token is represented as onehot in pitch. S: 67, 67, 67, 67 A: 62, 62, 62, 62 T: 59, 59, 57, 57 B: 43, 43, 45, 45 Figure 6: The opening measure of BWV 428 is visualized as a pianoroll (left, where the x-axis is discretized time and y-axis is MIDI pitch number), and encoded in grid representation with sixteenth note resolution (right). The soprano and alto voices have quarter notes at pitches G4 and D4, the tenor has eighth notes at pitches B3 and A3, and the bass has eighth notes at pitches A2 and G2. The second dataset, Maestro BID7, consists of polyphonic piano performances with expressive timing and dynamics. The time resolution here is on the millisecond level, so a grid representation would in sequences that are too long. Instead, the polyphonic performance is serialized into a sequence of one hot encoded events as proposed in.First, the input MIDI files are preprocessed to extend note durations based on sustain pedal control events. The sustain pedal is considered to be down whenever a sustain control change is encountered with a value >= 64; the sustain pedal is then considered up after a control change with a value < 64. Within a period where the sustain pedal is down, the duration of each note is extended to either the beginning of the next note of the same pitch or the end of the sustain period, whichever happens first. If the original duration extends beyond the time when the sustain pedal is down, that original duration is used. Next, the MIDI note events are converted into a sequence from the following set of vocabulary: 128 NOTE_ON events for starting a note of with one of the 128 MIDI pitches, 128 NOTE_OFF events for ending a note with one of the 128 MIDI pitches, 100 TIME_SHIFT events representing forward time shifts in 10ms increments from 10ms to 1s, and 32 SET_VELOCITY events representing the velocity for future NOTE_ON events in the form of the 128 possible MIDI velocities quantized into 32 bins. An example performance encoding is illustrated in Figure 7. TAB1 TIME_SHIFT<500>, NOTE_OFF<65> Figure 7: A snippet of a piano performance visualized as a pianoroll (left) and encoded as performance events (right, serialized from left to right and then down the rows). A C Major chord is arpeggiated with the sustain pedal active. At the 2-second mark, the pedal is released, ending all of the notes. At the 3-second mark, an F is played for.5 seconds. The C chord is played at velocity 80 and the F is played at velocity 100. Participants were presented with two musical excerpts that shared a common priming sequence. For each excerpt, the priming sequence was played, followed by 2.5 seconds of silence, followed by the priming sequence again and a continuation of that sequence. The continuations were either sampled from one of the models or extracted from our validation set. We evaluated all possible pairs in the space of data and model samples, except from the same model. Each continuation had a length of 512 events using the encoding described in Section A.2. This corresponds to the length the models were trained on to remove the deteriorating effect that happens with baseline Transformer when asked to generate beyond the length it was trained on. Participants were asked which excerpt they thought was more musical on a Likert scale of 1 to 5. The pair is laid out left versus right, with 1 indicating the left is much more musical, 2 the left is slightly more musical, 3 being a tie, 4 being the right is slightly more musical, and 5 the right is much more musical. For each model, we generated 10 samples each with a different prime, and compared them to three other models, ing in 60 pairwise comparisons. Each pair was rated by 3 different participants, yielding a total of 180 comparisons. A Kruskal-Wallis H test of the ratings showed that there was a statistically significant difference between the models: χ 2 = 63.84, p = 8.86e-14< 0.01. TAB6 show a post-hoc analysis on the comparisons within each pair, using the Wilcoxon signed-rank test for matched samples. TAB7 shows a post-hoc analysis of how well each model performed when compared to all pairs, and compares each model's aggregate against each other, using the Mann-Whitney U test for independent samples. We use a Bonferroni correction on both to correct for multiple comparisons. The win and loss counts bucket scores 4, 5 and scores 1, 2 respectively, while the tieing score is 3.Both within pairs and between aggregates, participants rated samples from our relative Transformer as more musical than the baseline Transformer with p < 0.01/6.For within pairs, we did not observe a consistent statistically significant difference between the other model pairs, baseline transformer versus LSTM and LSTM versus relative Transformer. When comparing between aggregates, LSTM was overall perceived as more musical than baseline Transformer. Relative Transformer came a bit close to outperforming LSTM with p = 0.018. When we listen to the samples from the two, they do sound qualitatively different. Relative Transformer often exhibits much more structure (as shown in FIG4), but the effects were probably less pronounced in the listening test because we used samples around 10s to 15s, which is half the length of those shown in FIG4 to prevent the baseline Transformer from deteriorating. This weakens the comparison on long-term structure. When compared to real music from the validation set, we see that in aggregates, real music was better than LSTM and baseline Transformer. There was no statistical significant difference between real music and relative Transformer. This is probably again due to the samples being too short as real music is definitely still better. One advantage of attention-based models is that we can visualize its attention distribution 3. This gives us a glimpse of how the model might be building up recurring structures and how far it is attending back. The pianorolls in the visualizations below is a sample generated from Transformer with relative attention. Each figure shows a query (the source of all the attention lines) and previous memories being attended to (the notes that are receiving more softmax probabiliy is highlighted in). The coloring of the attention lines correspond to different heads and the width to the weight of the softmax probability. Figure 8: This piece has a recurring triangular contour. The query is at one of the latter peaks and it attends to all of the previous high notes on the peak, all the way to beginning of the piece. Steps 2,3:Figure 10: Relative global attention: Steps (from left to right) for "skewing" an absolute-by-relative (i q, r) indexed matrix into absolute-by-absolute (i q, j k). Grey indicates self-attention masks or entries introduced by the skewing procedure. Positions with relative distance zero are marked. Entries outlined by purple are removed in step 3. Steps 1, 2Steps 3 Steps 4Figure 11: Relative local attention: Steps (from left to right) for "skewing" an (i q, r) indexed matrix with 2N − 1 ranged relative indices r into (i q, j k indexed. Shapes are indicated above the boxes, while indices in the boxes give relative distances.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show the first successful use of Transformer in generating music that exhibits long-term structure.

Represent the short summary for finding the corresponding scientific article (long)

We introduce MIST RNNs, which a) exhibit superior vanishing-gradient properties in comparison to LSTM; b) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies; and c) are much more efficient than previously-proposed NARX RNNs, with even fewer parameters and operations than LSTM.


Question: represent the article for finding a single-sentence tldr

Question: Sequential decision problems for real-world applications often need to be solved in real-time, requiring algorithms to perform well with a restricted computational budget. Width-based lookaheads have shown state-of-the-art performance in classical planning problems as well as over the Atari games with tight budgets. In this work we investigate width-based lookaheads over Stochastic Shortest paths (SSP). We analyse why width-based algorithms perform poorly over SSP problems, and overcome these pitfalls proposing a method to estimate costs-to-go. We formalize width-based lookaheads as an instance of the rollout algorithm, give a definition of width for SSP problems and explain its sample complexity. Our experimental over a variety of SSP benchmarks show the algorithm to outperform other state-of-the-art rollout algorithms such as UCT and RTDP. Model-based lookahead algorithms provide the ability to autonomously solve a large variety of sequential decision making problems. Lookaheads search for solutions by considering sequences of actions that can be made from the current state up to a certain time into the future. For realworld applications decisions often need to be computed in real-time, requiring algorithms to perform with a restricted computational budget. Limiting search in this way can in considering states and trajectories which do not provide useful information. To address this, lookaheads can be augmented with heuristics that estimate costs-to-go to prioritise states and trajectories, and have been shown to perform well where computation budgets are restricted BID8.This paper is concerned with Stochastic Shortest Path (SSP) problems which are often used to compare and evaluate search algorithms. We consider the width-based family of planning algorithms, first introduced by BID15, which aim to prioritise the exploration of novel areas of the state space. Two width-based planners, Lipovetzky and Geffner's breadth-first search, IW, and the depth-first search, Rollout-IW BID1, are investigated on SSP problems. We first provide the necessary for SSP problems and width-based algorithms, while also formalising width-based algorithms as instances of the rollout algorithm BID4. We then show the motive to augment width-based lookaheads with cost estimates on SSP problems, define the width of SSP problems and propose a novel width-based algorithm that estimates costs-to-go by simulating a general base policy. Our experimental study shows that the algorithm compares favourably to the original Rollout-IW algorithm and to other state-of-the-art instances of the rollout algorithm. We concern ourselves with the problem of decision under stochastic uncertainty over a finite number of stages, which we characterise following closely the presentation of BID4. We are given a discrete-time dynamic system x k+1 = f k (x k, u k, w k), k = 0, 1,..., N − 1 where the state x k is an element of a space S k ⊂ R d, the control u k is an element of space C k ⊂ N, and the random disturbance w k is an element of a space D k ⊂ R m 1. The control u k is constrained to take values in a given non-empty subset U (x k) ⊂ C k, which depends on the current state x k, so that u k ∈ U k (x k) for all x k ∈ S k and k. The random disturbance w k is characterised by a probability distribution P k (·|x k, u k) that may depend explicitly on x k and u k but not on the values of previous disturbances w k−1,..., w 0. We consider the class of policies, or control laws, corresponding to the sequence of functions DISPLAYFORM0 where µ k maps states x k into controls u k = µ k (x k) and is such that µ k (x k) ∈ U (x k) for all x k ∈ S k. Such policies will be called admissible. Given an initial state x 0 and admissible policy π, the states x k and disturbances w k are random variables with distributions defined through the system equation DISPLAYFORM1 Thus, for given functions g f (terminal cost) and g the expected cost of π starting at x 0 is DISPLAYFORM2 where the expectation is taken over the random variables w k and x k. An optimal policy π * is one that minimises this cost DISPLAYFORM3 where Π is the set of all admissible policies. The optimal cost J * (x 0) depends on x 0 and is equal to J π * (x 0). We will refer to J * as the optimal cost or optimal value function that assigns to each initial state x 0 the cost J * (x 0). We use BID4 definition, that formulates Stochastic Shortest Path (SSP) problems as the class of optimal control problems where we try to minimize DISPLAYFORM0 with α set to 1 and we assume there is a cost-free termination state t which ensures that J π (x 0) is finite. Once the system reaches that state, it remains there at no further cost, that is, f (t, u, w) = t with probability 1 and g(t, u, w) = 0 for all u ∈ U (t). We note that the optimal control problem defined at the beginning of this section is a special case where states are pairs (x k, k) and all pairs (x N, N) are lumped into termination state t. In order to guarantee termination with probability 1, we will assume that there exists an integer m such that there is a positive probability that t will be reached in m stages or less, regardless of what π is being used and the initial state x 0. That is, for all admissible policies and i = 1,..., m it holds DISPLAYFORM1 A policy π will be proper if the condition above is satisfied for some m, and improper otherwise. A particularly effective on-line approach to obtain suboptimal controls is rollout, where the optimal cost-to-go from current state x k is approximated by the cost of some suboptimal policy and a d-step lookahead strategy. The seminal RTDP BID2 algorithm, is an instance of the rollout strategy where the lookahead is uniform, d = 1, and controlsμ(x k) selected at stage k and for state x k are those that attain the minimum DISPLAYFORM0 whereJ k+1 is an approximation on the optimal cost-to-go J * k+1. If the approximation is from below, we will refer to it as a base heuristic, and can either be problem specific BID8, domain independent BID5 BID22 or learnt from interacting with a simulator BID18. Alternatively, J k+1 can be defined as approximating the cost-to-go of a given suboptimal policy π, referred to as a base policy, where estimates are obtained via simulation BID19. We will denote the ing estimate of cost-to-go as H k (x k) 2. The of combining the lookahead strategy and the base policy or heuristic is the rollout policy,π {μ 0, µ 1, . . .,μ N −1} with associated costJ(x k). Such policies have the property that for all x k and k DISPLAYFORM1 when H k is approximating from above the cost-to-go of a policy, as shown by BID4 from the DP algorithm that defines the costs of both the base and the rollout policy. To compute at time k the rollout controlμ(x k), we compute and minimize over the values of the Q-factors of state and control pairs (x l, u l), DISPLAYFORM2 for admissible controls u l ∈ U (x l), l = k + i, with i = 0,..., d − 1, and DISPLAYFORM3 for l = k +d. In this paper we make a number of assumptions to ensure the viability of lookaheads with d > 1. We will assume that we can simulate the system in Eq. 3 under the base policy, so we can generate sample system trajectories and corresponding costs consistent with probabilistic data of the problem. We further assume that we can reset the simulator of the system to an arbitrary state. Performing the simulation and calculating the rollout control still needs to be possible within the real-time constraints of the application, which is challenging as the number of Q-factors to estimate and minimizations to perform in Equations 9-10 is exponential on the average number of controls available per stage and d, the maximum depth of the lookahead. We avoid the blowup of the size of the lookahead by cutting the recursion in Equation 9 and replacing the right hand side by that of Equation 10. As detailed in the next section, we will do this when reaching states x l that are deemed not to be novel according to the notion of structural width by BID15. This in a selective strategy alternative to the upper confidence bounds BID0 used in popular instances of Monte-Carlo Tree Search (MCTS) algorithms like UCT, that also are instances of the rollout algorithm BID4 ). We instantiate the rollout algorithm with an l-step, depth-selective lookahead policy using Width-based Search BID15. These algorithms both focus the lookahead and have good any-time behaviour. When it comes to prioritisation of expanding states, width-based methods select first states with novel valuations of features defined over the states BID16 BID9 DISPLAYFORM0 is the domain of variable x i, has not been seen before in the current search. Note that novel states are independent of the objective function used, as the estimated cost-to-go J is not used to define the novelty of the states. IW has recently been integrated as an instance of a rollout algorithm, and has been shown to perform well with respect to learning approaches with almost real-time computation budgets over the Atari games BID1. The breadth-first search strategy underlying IW ensures a state variable x i is seen for the first time through the shortest sequence of control steps, i.e. the shortest path assuming uniform costs g(x, u, w).4 On the other hand, depth-first rollout algorithms cannot guarantee this property in general. Rollout IW (RIW) changes the underlying search of IW into a depth-first rollout. In order to ensure that RIW considers a state to be novel iff it reaches at least one value of a state variable x i l through a shortest path, we need to adapt the definition of novelty. Intuitively, we need to define a set of state features to emulate the property of the breadth-first search strategy. Let d(x i, v) be the best upper bound known so far on the shortest path to reach each value v ∈ D(x i) of a state variable from the root state x k. Initially d(x i, v) = N for all state variables, where N is the horizon which is the maximum search depth allowed for the lookahead, thus denoting no knowledge initially. When a state x l is generated, d(x i, v) is set to l for all state variables where l < d(x i, v). Since RIW always starts each new rollout from the current state x k, in order to prove a state x l to be novel we have to distinguish between x l being already in the lookahead tree and x l being new. If x l is new in the tree, to conclude it is novel, it is sufficient to show that there exists a state variable x i whose known shortest path value d(x i, v) > l. If x l is already in the tree, we have to prove the state contains at least one state variable value x i whose shortest path is l = d(x i, v), i.e. state x l is still novel and on the shortest path to x i. Otherwise the rollout is terminated. In order to ensure the termination of RIW, non-novel states are marked with a solved label. The label is backpropagated from a state x l+1 to x l if all the admissible control inputs u ∈ U (x l) yield states x l+1 = f l (x l, u, w l) already labeled as solved. RIW terminates once the root state is labeled as solved BID1. Nonnovel states x l are treated as terminals and their cost-to-go is set to 0. This can induce a bias towards non-novel states rather than true terminal states. In the next section we investigate how to overcome the ill-behaviour of RIW when a state x l is non-novel. We discuss the importance of estimating upper-bounds on the cost-to-go H l (x l) instead of assigning termination costs. This turns out to be essential for RIW over SSPs. Despite the successes of width-based algorithms on a variety of domains including the Atari-2600 games BID16 BID1, the algorithms, as will be shown, have poor performance on SSP problems. We illustrate this with two scenarios. First, width-based lookaheads prefer trajectories leading to non-novel states over longer ones that reach a goal. Second, and specific to depth-first width-based lookaheads, we show that useful information is ignored. We can demonstrate these scenarios using a simple SSP problem with uniform and unitary action costs, shown in FIG0. The task is to navigate to a goal location using the least number of left, right, up or down actions. Any action that would in the agent moving outside of the grid produces no change in its position. The features used by the width-based planners are the coordinates for the current agent position. Both IW and RIW algorithms, given a sufficient budget, would in the lookahead represented by yellow lines in FIG0. As expected, both lookaheads contain the shortest paths to make each feature of the problem true. For both IW and RIW, we back up the costs found in the lookahead starting from terminal and non-novel states. In this instance a move down or left from the agent's initial state has no effect, thus immediately producing a non-novel state. When backing up values, down and left have an expected cost of 1, which is less than the optimal cost of 2 for up, the action that leads to the top left goal state. This prevents both IW and RIW from ever achieving the goal, as they keep selecting those useless actions. Furthermore, if the goal is the top right location in FIG0, RIW's random action selection can generate a trajectory that reaches the goal. Yet, trajectories leading to the goal are pruned away, as non-novel states in later considered trajectories are treated as terminals, again ing in the lookahead represented by the yellow lines in FIG0. BID1 introduced the algorithm RIW in the context of deterministic transition functions. In this section we discuss its properties in the context of SSPs. The set of features used to evaluate the novelty of a state is DISPLAYFORM0 is the domain of variable x i, and d is a possible shortest path distance. Note that the horizon N is the upper-bound of d. The maximum number of novel states is O(|F |), as the maximum number of shortest paths for a feature (v, i, ·) ∈ F is N. That is, in the worst case we can improve the shortest path for (v, i, ·) by one control input at a time. The labeling of nodes ensures the number of rollouts from the initial state in RIW is at most O(|F | × b), where b = max x l |U (x l)| is the maximum number of applicable control variables in a state, i.e. maximum branching factor. When the labeling is applied to stochastic shortest path problems, the ing lookahead tree is a relaxation of the original SSP, as it allows just one possible outcome of a control input. Alternatively, one can back-propagate the label solved to a state x l iff 1) all admissible control inputs u ∈ U (x l) have been applied ing in states labeled as solved, and 2) the tree contains all the possible ing states of each control input u ∈ U (x l). We refer to this new strategy to backpropagate labels as λ-labeling. We denote as λ the maximum number of states that can from applying u ∈ U (x l−1) in a state x l. That is, λ = max x,u,w |f (x, u, w)|. RIW with λ-labeling will terminate after at most DISPLAYFORM1 Furthermore, we can reconcile the notion of width over classical planning problems BID15 with SSPs. A terminal state t made of features f ∈ F has width 1 iff there is a trajectory x 0, u 0,..., u n−1, x n for n ≤ N where x n = t, such that for each x j in the trajectory 1) the prefix x 0, u 0,..., u j−1, x j reaches at least one feature DISPLAYFORM2, it is a shortest path possible to reach a value in x i, 2) any shortest path to f j can be extended with a single control input u into a shortest path for a feature f j+1 complying with property 1) in state x j+1, and 3) the shortest path for f n is also a shortest path for termination state t. RIW with the new labeling strategy is guaranteed to reach all width 1 terminal states t. Theorem 1. Rollout IW with λ-labeling is guaranteed to reach every width 1 terminal state t in polynomial time in the number of features F if λ = 1. If λ = ∞, RIW will not propagate any solved label, and terminate only when the computational budget is exhausted. For simplicity, we assumed shortest paths are equivalent to the shortest sequence of control inputs. To generalize to positive non-uniform costs, the distance d in the features should keep track of the cost of a path i g(x i, u i, w i) instead of its length, and the horizon be applied to the cost of the path. The most successful methods for obtaining cost-to-go approximations have revolved around the idea of running a number of Monte Carlo simulations of a suboptimal base policy π BID10 BID7. This amounts to generating a given number of samples for the expression minimized in Equation 7 starting from the states x l over the set of admissible controls u l ∈ U (x l) in Equation 10, averaging the observed costs. Three main drawbacks BID4 follow from this strategy. First, the costs associated with the generated trajectories may be wildly overestimating J * (x l) yet such trajectories can be very rare events for the given policy. Second, some of the controls u l may be clearly dominated by the rest, not warranting the same level of sampling effort. Third, initially promising controls may turn out to be quite bad later on. MCTS algorithms aim at combining lookaheads with stochastic simulations of policies and aim at trading off computational economy with a small risk of degrading performance. We add two new methods to the MCTS family, by combining the multi-step, width-based lookahead strategy discussed in the previous section with two simulation-based cost-to-go approximations subject to a computational budget that limits the number of states visited by both the lookahead and base policy simulation. The first method, which we call RIW-RW, uses as the base policy a random walk, a stochastic policy that assigns the same probability to each of the controls u admissible for state x, and is generally regarded as the default choice when no domain knowledge is readily available. A rolling horizon H is set for the rollout algorithm that follows from combining the RIW lookahead with the simulation of random walks. The maximal length of the latter is set to H − l, where l is the depth of the non-novel state. Both simulations and the unrolling of the lookahead are interrupted if the computational budget is exhausted. While this can in trajectories sometimes falling short from a terminal, it keeps a lid on the possibility of obtaining extremely long trajectories that eat into the computational budget allowed and preclude from further extending the lookahead or sampling other potentially more useful leaf states x l. One of the most striking properties of rollout algorithms is the cost improvement property in Equation 8, suggesting that upper bounds on costs-to-go can be used effectively to approximate the optimal costs J *. Inspired by this, the second width-based MCTS method we discuss leverages the sampling techniques known as stochastic enumeration (SE) BID19 to obtain an unbiased estimator for upper bounds on costs-to-go, or in other words, estimates the maximal costs a stochastic rollout algorithm with a large depth lookahead can incur. SE methods are inspired by a classic algorithm by D. E. Knuth to estimate the maximum search effort by backtracking search. Knuth's algorithm estimates the total cost of a tree T with root u keeping track of two quantities, C the estimate of the total cost, and D the expectation on the number of nodes in T at any given level of the tree, and the number of terminal nodes once the algorithm terminates. Starting with the root vertex u and D ← 1, the algorithm proceeds by updating D to be D ← |S(u)|D and choosing randomly a vertex v from the set of successors S(u) of u. The estimate C is then updated C ← C + c(u, v)D using the cost of the edge between vertices u and v. These steps are then iterated until a vertex v is selected s.t. S(v) = 0. We observe that Knuth's C quantity would correspond to the worst-case cost-to-goJ(x) k of a rollout algorithm using a lookahead strategy with d set to the rolling horizon H and the trivial base heuristic that assigns 0 to every leaf state. Furthermore, we assume that the algorithm either does not find any terminals within the limits imposed by the computational budget assigned, or if it finds one such state, it is too the very last one being visited. Lookaheads define trees over states connected by controls, edge costs c(u, v) correspond directly with realisations of the random variable g(x, u, w) and the set of successors S(v) of a vertex corresponds with the set of admissible controls U (x). While Knuth's algorithm estimates are an unbiased estimator, the variance of this estimator can be exponential on the horizon, as the accuracy of the estimator lies on the assumption that costs are evenly distributed throughout the tree BID19. In the experiments discussed next, we use Knuth's algorithm directly to provide H k (x k), adding the stopping conditions to enforce the computational budget and limiting the length of trajectories to H − l as above. In comparison with simulating the random walk policy, the only overhead incurred is keeping up-to-date quantities C and D with two multiplications and an addition. To evaluate the different methods we use a number of GridWorld (Sutton and Barto 2018) domains, an instance of a SSP problem. The goal in GridWorld is to move from an initial position in a grid to a goal position. In each state 4 actions are available: to move up, down, left or right. Any action that causes a move outside of the grid in no change to the agent's position. Actions have a cost of 1, with the exception of actions that in reaching the goal state, that have a cost of 0. The complexity of GridWorld can be scaled through the size of the grid and the location and number of goals. GridWorld also allows for extensions, which we use to have domains with a stationary goal, moving goals, obstacles and partial observability. For each instance of the GridWorld domain we have a d 0 × d 1 grid, and the state is the current location of the agent, x = (a 0, a 1) where a i is the agent's position in dimension i. The transition function is formalised as DISPLAYFORM0 where, ef specifies the change in the agent's position for each action, T k ⊂ S k is the set of goal states and x k+1 = x k where the condition in FIG0 is not met. The cost of a transition is defined as DISPLAYFORM1 For the stationary goal setting we have a single goal state which is positioned in the middle of the grid by dividing 223.2 ± 10.4 199.9 ± 13.6 145.5 ± 12.9 DISPLAYFORM2 where δ t k gives the relative change of the goal state t k for the time step k + 1 and DISPLAYFORM3 Resulting in two goals starting at opposite corners of the grid moving back and forth on the same diagonal. The obstacles setting, uses the stationary goal, but modifies S k such that, DISPLAYFORM4 where O ⊂ N 2 and is the set of obstacles, that is grid cells in which the agent is not allowed. Having partially observable obstacles in GridWorld provides an instance of the stochastic Canadian Traveller Problem (CTP) (Papadimitriou and Yannakakis 1991). The objective in CTP is to find the shortest path between one location in a road map to another, however, there is a known probability for each road in the map that due to weather conditions the road is blocked. A road in CTP can only be observed as being blocked or unblocked by visiting a location connected to it, and once a road status is observed the status remains unchanged. In terms of the GridWorld problem, each grid cell has a known probability of being a member of the obstacle set, O. The agent can only observe cells as being obstacles or not when it is in a neighbouring cell. Once a grid cell is observed it is then known that it is either an obstacle or not for the remaining duration of the episode. John Langford designed two MDP problems 5 described as Reinforcement Learning (RL) "Acid" intended to be RIW NA 212.9 ± 11.3 215.9 ± 10.8 223.5 ± 9.8 Rnd.236.2 ± 6.5 200.4 ± 11.4 110.6 ± 11.8 Table 2: Same experimental setting as TAB2 over GridWorld with a moving goal.difficult to solve using common RL algorithms, such as Qlearning. Langford's two problems allow two actions from every state. The state space for the problems is S k = {i | 0 ≤ i < N} where the number of states value, N, allows the complexity of the problem to be controlled. Langford originally specified the problems as reward-based, here we modify them to be SSP cost-based problems. Reward shaping is commonly used to make Reinforcement Learning easier by encouraging actions, through higher rewards, towards a goal state or states. The first of Langford's problems is named Antishaping and uses reward shaping to encourage actions away from the goal state. Antishaping has the transition function DISPLAYFORM5 otherwise, the state remains unchanged, x k+1 = x k. The set containing the goal state is T k = {N − 1}, which can be achieved by continuously selecting u k = 0. The cost of each transition in Antishaping is 0.25 divided by N -x k+1, except when x k+1 = N − 1 where the cost is 0. The problem becomes a large plateau where longer paths become more costly at larger rates. The motivation behind Langford's second problem, Combolock, is if many actions lead back to a start state, random exploration is inadequate. The Combolock problem has the transition function DISPLAYFORM6 otherwise x k+1 is equal to the initial position of 0. The goal state is T k = {N − 1} and sol x k is either 0 or 1 assigned to state x k which remains constant. For each state x ∈ S, sol x has an equal chance of either being 0 or 1. The cost of each transition in Combolock is 1 except for the transition that leads to the terminal state N − 1 where the cost is 0. While common Reinforcement Learning algorithms such as Q-Learning methods will struggle to solve these domains, it is claimed by Langford that the E 3 (Kearns and Singh 2002) family of algorithms, whose exploration do not rely solely on random policies or reward feedback but on exploring the maximum number of states, will perform well. 43.9 ± 1.9 35.9 ± 2.5 25.3 ± 2.4 UCT Man. 37.0 ± 2.9 36.4 ± 2.9 36.4 ± 2.9 Rnd.43.8 ± 1.9 38.5 ± 2.5 25.9 ± 1.9 RIW Man. 36.4 ± 2.9 36.4 ± 2.9 36.4 ± 2.9 NA 49.8 ± 0.4 48.8 ± 1.0 49.3 ± 0.8 Rnd.44.9 ± 1.7 34.5 ± 2.8 19.3 ± 2.1 20 1Stp Man. 76.7 ± 5.5 77.1 ± 5.4 76.7 ± 5.5 Rnd.97.9 ± 1.6 88.0 ± 3.4 62.7 ± 4.8 UCT Man. 78.0 ± 5.4 78.4 ± 5.3 73.4 ± 5.7 Rnd.98.7 ± 1.2 96.4 ± 1.9 77.2 ± 4.2 RIW Man. 79.7 ± 4.9 76.7 ± 5.5 76.7 ± 5.5 NA 100.0 ± 0.0 100.0 ± 0.0 99.6 ± 0.8 Rnd.98.5 ± 1.3 88.0 ± 3.4 29.3 ± 3.1 Man.194.6 ± 13.4 191.5 ± 13.6 196.6 ± 13.2 Rnd.249.2 ± 1.1 244.4 ± 3.7 216.8 ± 9.3 UCT Man.194.6 ± 13.4 195.6 ± 13.3 184.4 ± 13.9 Rnd.249.0 ± 1.9 243.1 ± 4.3 231.6 ± 7.9 RIW Man. 208.6 ± 10.7 210.6 ± 11.1 193.5 ± 13.4 NA 250.0 ± 0.0 250.0 ± 0.0 250.0 ± 0.0 Rnd.247.9 ± 2.6 242.9 ± 4.3 196.1 ± 11.3 Table 3: Same settings as TAB2 over GridWorld with fully observable obstacles and a stationary goal. We evaluate the depth-first width-based rollout algorithm, RIW, with and without being augmented using base policies. λ = 1 is used for the labels back-propagation. We did not observe statistically significant changes with λ = ∞. For the GridWorld domain we define the features on which RIW plans over as F = {(a, i, d) | a ∈ D(x i)} where d is the length of the control input path from the initial state, a is the agent's position in the grid in dimension i and D(x i) is the domain of the agent's position, a, in dimension i. For Antishaping and Combolock the feature set will be F = {(i, d) | i ∈ N )} where i is the state number the agent is in and N is the number of states of the domain. Two additional rollout algorithms are also considered, the one-step rollout algorithm, RTDP BID2 and the multi-step, selective, regret minimisation, rollout algorithm Upper Confidence bounds applied to Trees (UCT) BID14. The exploration parameter of UCT is set to 1.0 for all experiments. For all the methods that use a base policies the maximum depth of a simulated trajectory is equal to H − l, where l is the depth at which the simulated trajectory began and H is the horizon value of the lookahead. Also, a single, as opposed to multiple, simulated trajectory for the cost-to-go approximation is used, as initial indicated it is favourable. We also report the algorithms using a Manhattan distance heuristic for the GridWorld domains that use obstacles. Using the Manhattan distance for the GridWorld problems with obstacles provides a lower bound on the cost-to-go. Each method on the domains is evaluated at different levels of complexity by varying the number of states. The methods are evaluated using different simulator budgets. The simulator budgets are the maximum simulator calls allowed for the evaluation at each time step. For each algorithm and domain Dim. Heu. Simulator Budget 100 1000 10000 10 1Stp Man. 36.6 ± 2.9 35.0 ± 3.0 35.9 ± 2.9 Rnd.40.4 ± 2.1 27.2 ± 2.5 15.4 ± 1.6 UCT Man. 28.4 ± 2.9 29.5 ± 3.1 18.5 ± 2.7 Rnd.41.5 ± 2.0 36.1 ± 2.3 22.2 ± 2.0 RIW Man. 28.1 ± 3.0 28.3 ± 3.0 26.5 ± 3.0 NA 49.5 ± 0.7 49.1 ± 0.9 49.8 ± 0.4 Rnd.43.5 ± 1.9 23.4 ± 2.6 11.1 ± 1.3 Man. 71.8 ± 5.8 74.0 ± 5.7 71.9 ± 5.8 Rnd.97.0 ± 1.7 82.3 ± 3.9 49.8 ± 4.6 UCT Man. 53.8 ± 5.7 60.9 ± 6.2 38.0 ± 5.5 Rnd.98.0 ± 1.3 87.4 ± 4.0 63.0 ± 4.8 RIW Man. 53.5 ± 5.6 44.1 ± 5.5 44.1 ± 5.6 NA 100.0 ± 0.0 99.5 ± 1.0 99.5 ± 1.0 Rnd.97.6 ± 1.5 79.7 ± 4.4 20.7 ± 1.9 Results are also presented for the Atari-2600 game Skiing, which is a SSP problem. We use the OpenAI gym's BID6 interface of the Arcade Learning Environment (ALE) BID3 ) and use the slalom game mode of Skiing. In the slalom mode the aim is to ski down the course as fast as possible while going through all the gates. Once the finish line is reached, a 5 second time penalty is applied for each gate that is missed. The reward values provided by ALE for Skiing is the negative value of the total time taken plus any time penalties in centiseconds, which we use as a cost. We use the environment settings as described by BID17 with a frame skip of 5 and a probability of 0.25 of repeating the previous action sent to environment instead of the current one, which Machado et al. call sticky actions. For evaluation we use a simulator budget of 100 and partial caching as described by BID1, in that we cache simulator state-action transitions, thus assuming determinism, but clear the cached transitions when executing an action in the environment. However, the lookahead tree itself is not cleared when executing an action in the environment as is done for the other domains trialed. The maximum episode length is capped at 18,000 frames with a frame skip of 5 this equals 3,600 actions. Using a simulation based cost-to-go approximation is infeasible with a simulator budget of 100 and the maximum episode length of 3,600 actions. Therefore we report the algorithms using a heuristic cost-to-go estimate, which is the the number of gates that have either been missed or are still left times the time penalty of 500 centiseconds. For the RIW algorithms we use the pixel values from the current gray scaled screen at full resolution, that is 180 by 210 pixels, as features. All experiments were run within the OpenAI gym frame- 1.7 ± 0.1 1.3 ± 0.1 1.1 ± 0.1 work BID6 ) and the code used for the algorithms and domains is available through GitHub 6. The different H functions reported here are H NA = 0, the random policy H Rnd, and the Manhattan distance H Man. The algorithms were also evaluated using Knuth's algorithm with a different range of rollouts for the cost-to-go estimate, however, the are not reported here as they are either statistically indifferent or dominated by the using H Rnd with a single rollout. BID4 suggests that MCTS algorithms should readily benefit from stronger algorithms to estimate costs-to-go by simulation of stochastic policies. Our experiments showed that if synergies exist these do not manifest when using off-the-shelf stochastic estimation techniques like the ones discussed by BID19. TAB2, 2 and 3 report the of the different lookahead algorithms on the GridWorld domain variants with a stationary goal, moving goals and obstacles respectively. For these three domains, were also collected for a 100x100 grid, however, the were omitted from the tables as the simulator budgets used were not sufficient to find anything meaningful. The on the stationary goal GridWorld domain shown in TAB2 provide a number of insights about the rollout algorithms reported. First, we can see RIW benefits from using H Rnd rather than H NA where the larger simulator budgets are used. As the simulator budget increases, as could be expected, so does the performance of all the methods using H Rnd. On the contrary, with H NA RIW's performance remains constant across the different budgets. The explanation for this can be found in the motivating example we gave previously in this paper with the agent preferring the shorter trajectory of driving into the boundary of the grid. TAB2 also shows that given the largest budget and H Rnd, RIW statistically outperforms the other algorithms on the three domains of different size. Table 2 for GridWorld with moving goals has similar patterns as the stationary goal domain in that RIW with H Rnd dominates performance for the largest budget. Also, the majority of performances for the smaller budgets, excluding RIW 199.0 ± 1.9 193.1 ± 5.0 190.2 ± 6.0 Table 6: Same settings as TAB6 over Combolock. GridWorld with a stationary goal and obstacles displayed in Table 3 continues the trend of . Using the largest budget RIW with H Rnd outperforms all methods on the 10x10 and 20x20 domains. For the 50x50 a number of are statistically indifferent. For this domain the algorithms using H Man as the base heuristic are also reported. While using the largest budget on the 10x10 grid H Rnd dominates H Man, for the larger 50x50 we see H Man dominates H Rnd for UCT, and is competitive for the other methods. For the smallest simulator budget on CTP reported in Table 4 using H Man with RIW and UCT are the dominate methods. For the largest simulator budget RIW using H Rnd is dominant over all other methods for both sized domains. We also see that in most cases for the two smaller budgets H Man dominates the H Rnd methods. TAB6 and 6 show on the smaller 10 state domains RIW with H Rnd is statistically dominant over all other methods on Antishaping and Combolock for the 500 and 1000 simulator budgets. However, for the more complex 50 state domains, the of all algorithms using H Rnd are statistically indifferent. It can be observed that using H Rnd with RIW does improve its performance compared with H NA across all the domain settings with simulator budgets of 500 and 1000, besides Antishaping with 50 states. For the Skiing Atari-2600 game in Table 7 H Heu is the heuristic value based on the number of gates missed and remaining as described in the previous section. RIW using H Heu dominates all other methods. Comparing RIW using H Heu with those reported by BID17, it has similar performance to the DQN algorithm BID18 after 100 million frames of training. Since the simulation budget per action we use here is equivalent to 500 frames, and given that the maximum episode duration spans 3,600 actions, RIW achieves the performance in Table 7 considering only 1.8 million frames. considers AlphaGo Zero BID20 to be state-of-the-art in MCTS algorithms. It combines the reasoning over confidence intervals first introduced with UCT BID14 and the classic simulation of base policies BID10, adding to both supervised learning algorithms to obtain, offline, parametric representations of costs-to-go which are efficient to evaluate. The ing algorithm achieves super-human performance at the game of Go, long considered too hard for AI agents. Rather Table 7: Average and 95% confidence interval for the cost on the Atari-2600 Skiing game over 100 episodes.than using descriptions of game states directly as we do, AlphaZero uses a CNN to extract automatically features that describe spatial relations between game pieces. Like us, AlphaZero's lookahead uses a stochastic policy to select what paths to expand, but rather than Q-factors, uses estimated win probabilities to prioritise controls, and simulates the opponent strategy via self-play to generate successor states. Our simulators are always given and remain unchanged, rather than being dynamic as is the case for AlphaZero. BID11 have recently presented a hybrid planning and learning approach that integrates BID1 rollout, with a deep neural network. Similarly to AlphaZero, they use it to both guide the search, and also to extract automatically the set of state features F. Interestingly, Junyent et al.'s work does not use deep neural networks to approximate costs-to-go as AlphaZero does. A significant improvement in performance over Bandres et al. original rollout algorithm is reported with policies learnt after 40 million interactions with the simulator, using an overall computational budget much bigger than the one used by Bandres et al. MCTS approaches typically combine lookaheads and cost-togo approximations, along with statistical tests to determine what are the most promising directions and focus their sampling effort. The width-based methods described in this paper do so too, but in ways which are, at first sight, orthogonal to existing strategies. It remains an area of active research to map out exactly how the width-based methods described in this paper, and those elsewhere by BID11 too, provide alternatives to the limitations of existing MCTS approaches. Having said this, there is no general theory guiding the design of MCTS algorithms BID4, and to avoid generating ad-hoc, problem dependent solutions involuntarily it is important to follow strict protocols that alert of potential lack of statistical significance in , while relying on a diverse set of benchmarks that can be both easily understood, and highlight limitations of existing state-of-theart methods and overcome them.

We propose a new Monte Carlo Tree Search / rollout algorithm that relies on width-based search to construct a lookahead.

Represent the TLDR for finding the corresponding scientific article

In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.