Question: Represent the article for finding a single-sentence TLDR.

Question: Decades of research on the neural code underlying spatial navigation have revealed a diverse set of neural response properties. The Entorhinal Cortex (EC) of the mammalian brain contains a rich set of spatial correlates, including grid cells which encode space using tessellating patterns. However, the mechanisms and functional significance of these spatial representations remain largely mysterious. As a new way to understand these neural representations, we trained recurrent neural networks (RNNs) to perform navigation tasks in 2D arenas based on velocity inputs. Surprisingly, we find that grid-like spatial response patterns emerge in trained networks, along with units that exhibit other spatial correlates, including border cells and band-like cells. All these different functional types of neurons have been observed experimentally. The order of the emergence of grid-like and border cells is also consistent with observations from developmental studies. Together, our suggest that grid cells, border cells and others as observed in EC may be a natural solution for representing space efficiently given the predominant recurrent connections in the neural circuits. Understanding the neural code in the brain has long been driven by studying feed-forward architectures, starting from Hubel and Wiesel's famous proposal on the origin of orientation selectivity in primary visual cortex BID19. Inspired by the recent development in deep learning BID25 BID30 BID18 BID39, there has been a burst of interest in applying deep feedforward models, in particular convolutional neural networks (CNN) BID29, to study the sensory systems, which hierarchically extract useful features from sensory inputs (see e.g., BID61 ; BID24 ; BID22 ; BID60).For more cognitive tasks, neural systems often need to maintain certain internal representations of relevant variables in the absence of external stimuli-a process that requires more than feature extraction. We will focus on spatial navigation, which typically requires the brain to maintain a representation of self-location and update it according to the animal's movements and landmarks of the environment. Physiological studies done in rodents and other mammals (including humans, non-human primates and bats) have revealed a variety of neural correlates of space in Hippocampus and Entorhinal Cortex (EC), including place cells BID41, grid cells BID10 BID15 BID11 BID62 BID23 BID20, along with border cells BID49, band-like cells BID27 and others (see FIG0). In particular, each grid cell only fires when the animal occupies a distinct set of physical locations, and strikingly these locations lie on a lattice. The study of the neural underpinning of spatial cognition has provided an important window into how high-level cognitive functions are supported in the brain BID0.How might the spatial navigation task be solved using a network of neurons? Recurrent neural networks (RNNs) BID18 BID12 BID43 BID54 BID13 BID53 seem particularly useful for these tasks. Indeed, recurrent-based continuous attractor networks have been one popular type of models proposed for the formation of grid cells BID4 BID5 and place cells BID45. Such models have provided valuable insights into one set of possible mechanisms that could support the formation of the grids. However, these models typically rely on fine-tuned connectivity patterns, in particular the models need a subtle yet systematic asymmetry in the connectivity pattern to move the attractor state according to the animal's own movement. The existence of such a specific 2D connectivity in rodent EC remains unclear. Additionally, previous models have mainly focused on grid cells, while other types of responses that co-exist in the Entorhinal Cortex have been largely ignored. It would be useful to have a unified model that can simultaneously explain different types of neural responses in EC.Motivated by these considerations, here we present an alternative modeling approach for understanding the representation of space in the neural system. Specifically, we trained a RNN to perform some spatial navigation tasks. By leveraging the recent development in RNN training and knowledge of the navigation system in the brain, we show that training a RNN with biologically relevant constraints naturally gives rise to a variety of spatial response profiles as observed in EC, including grid-like responses. To our knowledge, this is the first study to show that grid-like responses could emerge from training a RNN to perform navigation. Our implies that the neural representation in EC may be seen as a natural way for the brain to solve the navigation task efficiently BID55. More generally, it suggests that RNNs can be a powerful tool for understanding the neural mechanisms of certain high-level cognitive functions. recorded when an animal navigates in a square environment, replotted from BID27, with the heat map representing the firing rate of this neuron as a function of the animal's location (red corresponds to high firing rate); a "band-like" cell from BID27; a border cell from BID49; an irregular spatially tuned cell from BID7; a "speed cell" from BID26, which exhibits roughly linear dependence on the rodent's running speed; a "heading direction cell" from BID46, which shows systematic change of firing rate depending on animal's heading direction. b) The network consists of N = 100 recurrently connected units (or neurons) which receive two external inputs, representing the animal's speed and heading direction. The two outputs linearly weight the neurons in the RNN. The goal of training is to make the responses of the two output neurons accurately represent the animal's physical location. c) Typical trajectory after training. As shown, the output of the RNN can accurately, though not perfectly, track the animal's location during navigation. Our network model consists of a set of recurrently connected units (N = 100). The dynamics of each unit in the network u i (t) is governed by the standard continuous-time RNN equation: DISPLAYFORM0 for i = 1,..., N. The activity of each unit, u i (t), is related to the activation of that unit, x i (t), through a nonlinearity which in this study we take to be u i (t) = tanh(x i (t)). Each unit receives input from other units through the recurrent weight matrix W rec and also receives external input, I(t), that enters the network through the weight matrix W in. Each unit has two sources of bias, b i which is learned and ξ i (t) which represents noise intrinsic to the network and is taken to be Gaussian with zero mean and constant variance. The network was simulated using the Euler method for T = 500 timesteps of duration τ /10.To perform a 2D navigation task with the RNN, we linearly combine the firing rates of units in the network to estimate the current location of the animal. The responses of the two linear readout neurons, y 1 (t) and y 2 (t), are given by the following equation: DISPLAYFORM1 The network inputs and outputs were inspired by simple spatial navigation tasks in 2D open environments. The task resembles dead-reckoning (sometimes referred to as path integration), which is ethologically relevant for many animal species BID6 BID38 BID9. To be more specific, the inputs to the network were the animal's speed and direction at each time step. Experimentally, it has been shown that the velocity signals exist in EC BID46 BID26 BID17, and there is also evidence that such signals are necessary for grid formation BID58.Throughout the paper, we adopt the common assumption that the head direction of the animal coincides with the actual moving direction. The outputs were the x-and y-coordinates of the integrated position. The direction of the animal is modeled by modified Brownian motion to increase the probability of straight-runs, in order to be consistent with the typical rodent's behavior in an open environment. The usage of such simple movement statistics has the advantage of having full control of the simulated trajectories. However, for future work it would be very interesting to test the model using different animals' real movement trajectories to see how the might change. Special care is taken when the animal is close to the boundary. The boundary of the environment will affect the statistics of the movement, as the animal cannot cross the boundary. This fact was reflected in the model by re-sampling the angular input variable until the input angle did not lead the animal outside the boundary. In the simulations shown below, the animal always starts from the center of the arena, but we verified that the are insensitive to the starting locations. We optimized the network parameters W rec, W in, b and W out to minimize the squared error in equation between target x-and y-coordinates from a two dimensional navigation task (performed in rectangular, hexagonal, and triangular arenas) and the network outputs generated according to equation. DISPLAYFORM0 Parameters were updated with the Hessian-free algorithm BID35 ) using minibatches of size M = 500 trials. In addition to minimizing the error function in equation we regularized the input and output weights according to equation and the squared firing rates of the units (referred to as metabolic cost) according to equation. In sum, the training aims to minimize a loss function, that consists of the error of the animal, the metabolic cost, and a penalty for large network parameters. DISPLAYFORM1 We find that the are qualitatively insensitive to the initialization schemes used for the recurrent weight matrix W rec. For the presented in this paper, simulations in the hexagonal environment were obtained by initializing the elements of W rec to be zero mean Gaussian random variables with variance 1.5 2 /N, and simulations in the square and triangular environments were initialized with an orthogonal W rec BID48. We initialized the bias b and output weights W out to be zero. The elements of W in were zero mean Gaussian variables with variance 1/N in. We run simulation experiments in arenas with different boundary shapes, including square, triangular and hexagonal. FIG0 shows a typical example of the model performance after training; the network (red trace) accurately tracks the animal's actual path (black). We are mostly interested in what kind of representation the RNN has learned to solve this navigation task, and whether such a representation resembles the response properties of neurons in EC. To test whether the trained RNN developed location-selective representations, we plot individual neurons' mean activity level as a function of the animal's location during spatial exploration. Note that these average response profiles should not be confused with the linear filters typically shown in feedforward networks. Surprisingly, we find neurons in the trained RNN show a range of interesting spatial response profiles. Examination of these response profiles suggests they can be classified into distinct functional types. Importantly, as we will show, these distinct spatial response profiles can be mapped naturally to known physiology in EC. The spatial responses of all units in trained networks are shown in the Appendix. Grid-like responses Most interestingly, we find some of the units in the RNN exhibit clear grid-like responses FIG1 ). These firing patterns typically exhibit multiple firing fields, with each firing field exhibiting roughly circular symmetric or ellipse shape. Furthermore, the firing fields are highly structured, i.e., when combined, are arranged on a regular lattice. Furthermore, the structure of the response lattice depends on the shape of the boundary. In particular, training the network to perform self-localization in a square environment tends to give rectangular grids. In hexagonal and triangular environments, the grids are closer to triangular. Experimentally, it is shown that (medial) EC contains so-called grid cells which exhibit multiple firing fields that lie on a regular grid BID10 BID15. The grid-like firing patterns in our simulation are reminiscent of the grid cells in rodents and other mammals. However, we also notice that the the grid-like model responses typically exhibit few periods, not as many as experimental data (see FIG0 . It is possible that using a larger network might reveal finer grid-patterns in our model. Nonetheless, it is surprising that the gird-like spatial representations can develop in our model, given there is no periodicity in the input. Another potential concern is that, experimentally it is reported that the grids are often on the corners of a triangular lattice BID15 even in square environments (see FIG0, though the grids are somewhat influenced by the shape of the environment. However, the rats in these experiments presumable had spatial experience in other environments with various boundary shapes. Experimentally, it would be interesting to see if grid cells would lie on a square lattice instead if the rats are raised in a single square environment -a situation we are simulating here. Border responses Many neurons in the RNN exhibit selectivity to the boundary FIG1). Typically, they only encode a portion of the boundary, e.g. one piece of wall in a square shaped environment. Such properties are similar to the border cells discovered in rodent EC BID49 BID47 BID31 ). Experimentally, border cells mainly fire along one piece of wall, although some have been observed to fire along multiple borders or along the whole boundary of the environment; interestingly, these multi-border responses were also observed in some RNN models. Currently, it is unclear how the boundary-like response profiles emerge BID49 BID47 BID31. Our model points to the possibility that the border cells may emerge without the presence of tactile cues. Furthermore, it suggests that border cell formation may be related to the movement statistics of the animals, i.e. due to the asymmetry of the movement statistics along the boundary. Band-like responses Interestingly, some neurons in the RNN exhibit band-like responses (FIG1). In most of our simulations, these bands tend to be parallel to one of the boundaries. For some of the units, one of the bands overlaps the boundary, but for others, that is not the case. Experimentally, neurons with periodic-like firing patterns have been recently reported in rodent EC. In one study, it has been reported that a substantial portion of cells in EC exhibit band-like firing characteristics BID27. However, we note that based on the reported data in BID27, the band pattern is not as clear as in our model. Spatially-stable but non-regular responses Besides the units described above, most of the remaining units also exhibit stable spatial responses, but they do not belong to the above categories. These response profiles can exhibit either one large irregular firing field; or multiple circular firing fields, but these firing fields do not show a regular pattern. Experimentally these types of cells have also been observed. In fact, it is recently reported that the non-grid spatial cells constitute a large portion of the neurons in Layer II and III of rodent EC BID7. shown in Figure 3. Interestingly, we observe that the model border cells tend to have almost zero speed-tuning (e.g., see Figure 3g,h).Head direction tuning A substantial portion of the model neurons show direction tuning. There are a diversity of direction tuning profiles, both in terms of the strength of the tuning and their preferred direction. Example tuning curves are shown in Figure 3, and the direction tuning curves of a complete population are shown in the Appendix. Interestingly, in general model neurons which show the strongest head direction tuning do not show a clear spatial firing pattern (see Figure 3a,b,c). This suggests that there are a group of neurons which are mostly responsible for encoding the direction. We also notice that neurons with clear grid-like firing can exhibit a variety of direction tuning strengths, from weak to strong (Figure 3d,e,f). In the Appendix, we quantify the relation between these different tuning properties at the whole population level, which show somewhat complex dependence. Experimentally, the heading direction tuning in EC is well-known, e.g., BID46. Both the grid and non-grid cells in EC exhibit head direction tuning BID46. Furthermore, the linear speed dependence of the model neurons is similar to the properties of speed cells reported recently in EC BID26. Our is also consistent with another recent study reporting that the majority of neurons in EC exhibit some amount of speed tuning BID17.It remains an open question experimentally, at a population level, how different types of tuning characteristics in EC relate to each other. We next investigate how the spatial response profiles evolve as learning/training progresses. We report two main observations. First, neurons that fire selectively along the boundary typically emerge first. Second, the grid-like responses with finer spatial tuning patterns only emerge later in training. For visualization, we perform dimensionality reduction using the t-SNE algorithm BID33. This algorithm embeds 100 model neurons during three phases of training (early, intermediate, and late) into a two-dimensional space according to the similarity of their temporal responses. Here the similarity metric is taken to be firing rate correlation. In this 2D space as shown in FIG3, border cell representations appear early and stably persist through the end of training. Furthermore, early during training all responses are similar to the border related responses. In contrast, grid-like cells typically undergo a substantial change in firing pattern during training before settling into their final grid-like representation FIG3 ).The developmental time line of the grid-like cells and border cells is roughly consistent with developmental studies in rodents. Experimentally, it is known that border cells emerge earlier in development, and they exist at about 2 weeks after the rat is born BID3. The grid cells mature only at about 4 weeks after birth BID28 BID57 BID3. Furthermore, our simulations suggest the reason why border cells emerge earlier in development may be that computationally it is easier to wire-up a network that gives rise to border cell responses.. These cells' response profiles appear early and stably persist through training, illustrated by the short distance they travel in this space. In b), we show that the neurons which eventually become grid cells initially have tuning profiles similar to the border cells but then change their tuning substantially during learning. As a natural consequence, they need to travel a long distance in this space between the early and late phase of the training. Spatial responses are shown for four of these grid-like cells during the late phase of training. We find appropriate regularizations of the RNN to be crucial for the emergence of grid-like representations. We only observed grid-like representations when the network was encouraged to store information while perturbed by noise. This was accomplished by setting the speed input to zero, e.g. zero speed 90% of the time, and adding Gaussian noise to the network (ξ i (t) in equation FORMULA0 ); the precise method for setting the speed input to zero and the value of the noise variance is not crucial for our simulations to develop grid-like representations. The cost function which aims to capture the penalization on the metabolic cost of the neural activity also acts as an important regularization. Our simulations show that the grid-like representation did not emerge without this metabolic cost. In FIG4, we show typical simulation for a square environment, with and without proper metabolic regularization. In the Appendix, we illustrate the effect of regularization further, in particular the role of injecting noise into the RNN units. Our are consistent with the general notion on the importance of incorporating proper constraint for learning useful representations in neural networks BID2. Furthermore, it suggests that, to learn a model with response properties similar to neural systems it may be necessary to incorporate the relevant constraints, e.g., noise and metabolic cost. One natural question is whether the trained RNNs are able to perform localization when the path length exceeds the typical length used during training (500 steps), in particular given that noise in a b Figure 6: Error-correction happens at the boundary and the error is stable over time. At the boundary, the direction is resampled to avoid input velocities that lead to a path extending beyond the boundary of the environment. These changing input statistics at the boundary, termed a boundary interaction, are the only cue the RNN receives about the boundary. We find that the RNN uses the boundary interactions to correct the accumulated error between the true integrated input and its prediction based on the linear readout of equation FORMULA1. Panel a), the mean squared error increases when there are no boundary interactions, but then decreases after a boundary interaction, with more boundary interactions leading to greater error reduction. In the absence of further boundary interaction, the squared error would gradually increase again (blue curve) at roughly a constant rate. b) The network was trained using mini-batches of 500 timesteps but has stable error over a duration at least four orders of magnitude larger. The error of the RNN output (mean and standard deviation shown in black, computed based on 10000 timesteps) is compared to the error that would be achieved by an RNN outputting the best constant values (red).the network would gradually accumulate, leading to a decrease in localization performance. We test this by simulating paths that are several orders of magnitude longer. Somewhat surprisingly, we find the RNNs still perform well (Figure 6b). In fact, the squared error (averaged over every 10000 steps) is stable. The spatial response profiles of individual units also remain stable. This implies that the RNNs have acquired intrinsic error-correction mechanisms during training. As shown earlier, during training some of the RNN units develop boundary-related firing FIG1 ), presumably by exploiting the change of input statistics around the boundary. We hypothesize that boundary interactions may enable error-correction through signals based on these boundary-related activities. Indeed, we find that boundary interactions can dramatically reduce the accumulated error (Figure 6a). Figure 6a shows that, without boundary interactions, on average the squared error grows roughly linearly as expected, however, interactions with the boundaries substantially reduce the error, and more frequent boundary interactions can reduce the error further. Error-correction on grid cells via boundary interactions has been proposed BID16 BID44, however, we emphasize that the model proposed here develops the grid-like responses, boundary responses and the error-correction mechanisms all within the same neural network, thus potentially providing a unifying account of a diverse set of phenomena. In this paper, we trained RNNs to perform path integration (dead-reckoning) in 2D arenas. We found that after training RNNs with appropriate regularization, the model neurons exhibit a variety of spatial and velocity tuning profiles that match neurophysiology in EC. What's more, there is also similarity in terms of when these distinct neuron types emerge during training/development. The EC has long been thought to be involved in path integration and localization of the animal's location. The general agreement between the different response properties in our model and the neurophysiology provide strong evidence supporting the hypothesis that the neural population in EC may provide an efficient code for representation self-locations based on the velocity input. Recently, there has been increased interest in using complex neural network models to understand the neural code. But the focus has been on using feedforward architectures, in particular CNNs BID29. Given the abundant recurrent connections in the brain, it seems a particularly fruitful avenue to take advantage of the recent development in RNNs to help with neuroscience questions BID34 BID50 BID37 BID53. Here, we only show one instance following this approach. However, the insight from this work could be general, and potentially useful for other cognitive functions as well. The finding that metabolic constraints lead to the emergence of grid-like responses may be seen as conceptually related to the efficient coding hypothesis in visual processing BID1, in particular the seminal work on the emergence of the V1-like Gabor filters in a sparse coding model by BID42. Indeed, our work is partly inspired by these . While there are conceptual similarities, however, we should also note there are differences between the sparse coding work and ours. First, the sparsity constraint in sparse coding can be naturally viewed as a particular prior while in the context of the recurrent network, it is difficult to interpret that way. Second, the grid-like responses are not the most sparse solution one could imagine. In fact, they are still quite dense compared to a more spatially localized representation. Third, the grid-like patterns that emerged in our network are not filters based on the raw input, rather the velocity inputs need to be integrated first in order to encode spatial locations. Our work is also inspired by recent work using the efficient coding idea to explain the functional architecture of the grid cells BID55. It has been shown that efficient coding considerations could explain the particular set of grid scales observed in rodents BID52. However, in that work, the firing patterns of the neurons are assumed to have a lattice structure to start with. Furthermore, our work is related to the study by Sussillo and others BID53, in which they show that regularization of RNN models are important for generating solutions that are similar to the neural activity observed in motor cortex. In Sussillo et al., a smoothness constraint together with others lead to simple oscillatory neural dynamics that well matches the neural data. We have not incorporated a smoothness constraint into our network. Additionally, we note that there are a few recent studies which use place cells as the input to generate grid cells BID8 BID51, which are fundamentally different from our work. In these feedforward network models, the grid cells essentially perform dimensionality reduction based on the spatial input from place cells. However, the main issue with these models is that, it is unclear how place cells acquire spatial tuning in the first place. To the contrary, our model takes the animal's velocity as the input, and addresses the question of how the spatial tuning can be generated from such input, which are known to exist in EC BID46 BID26. In another related study BID21, the authors train a RNN with LSTM units BID18 to perform different navigation tasks. However, no grid-like spatial firing patterns are reported. Although our model shows a qualitative match to the neural responses observed in the EC, nonetheless it has several major limitations, with each offering interesting future research directions. First, the learning rule we use seems to be biologically implausible. We are interested in exploring how a more biologically plausible learning rule could give rise to similar BID32 BID37 BID14. Second, the simulation do not show a variety of spatial scales in grid-like cells. Experimentally, it is known that grid cells have multiple spatial scales, that scale geometrically with a ratio 1.4 BID52, and this particular scale ratio is predicted by efficient coding of space BID55. We are investigating how to modify the model to get a hierarchy of spatial scales, perhaps by incorporating more neurons or modifying the regularization. Last but not least, we have focused on the representation produced by the trained RNN. An equally important set of questions concern how the networks actually support the generation of such a representation. As a preliminary effort, we have examined the connectivity patterns of the trained network, and they do not seem to resemble the connectivity patterns required by standard attractor network models. Maybe this should not be seen as too surprising. After all, the trained networks can produce a diverse set of neural responses, while the previous models only led to grid responses. It would be interesting for future work to systematically examine the questions related to the underlying mechanisms. To quantify the speed selectivity of each unit we first fit a line to the tuning curve of unit activity as a function of speed. The speed selectivity is the absolute value of the slope. If the unit activity is not modulated by speed then the speed selectivity is 0. To quantify the direction selectivity of each unit we calculated the average unit activity as a function of direction input and then took the maximum minus minimum of this tuning curve. If the unit activity is not modulated by direction then the direction selectivity is 0. To quantify the spatial selectivity we used lifetime sparseness BID56. If the unit activity is not modulated by spatial location then the spatial selectivity is 0. Each dot in the figures below show the selectivity for a single unit. Direc$on selec$vity

To our knowledge, this is the first study to show how neural representations of space, including grid-like cells and border cells as observed in the brain, could emerge from training a recurrent neural network to perform navigation tasks.

Represent the short summary for finding the corresponding scientific article (long)

We propose a continual version of Equilibrium Propagation, where neuron and synapse dynamics occur simultaneously throughout the second phase, with theoretical guarantees and numerical simulations.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song. Such components include voice, bass, drums and any other accompaniments. While end-to-end models that directly generate the waveform are state-of-the-art in many audio synthesis problems, the best multi-instrument source separation models generate masks on the magnitude spectrum and achieve performances far above current end-to-end, waveform-to-waveform models. We present an in-depth analysis of a new architecture, which we will refer to as Demucs, based on a (transposed) convolutional autoencoder, with a bidirectional LSTM at the bottleneck layer and skip-connections as in U-Networks . Compared to the state-of-the-art waveform-to-waveform model, Wave-U-Net , the main features of our approach in addition of the bi-LSTM are the use of trans-posed convolution layers instead of upsampling-convolution blocks, the use of gated linear units, exponentially growing the number of channels with depth and a new careful initialization of the weights. Results on the MusDB dataset show that our architecture achieves a signal-to-distortion ratio (SDR) nearly 2.2 points higher than the best waveform-to-waveform competitor (from 3.2 to 5.4 SDR). This makes our model match the state-of-the-art performances on this dataset, bridging the performance gap between models that operate on the spectrogram and end-to-end approaches. Cherry first noticed the "cocktail party effect" : how the human brain is able to separate a single conversation out of a surrounding noise from a room full of people chatting. Bregman later tried to understand how the brain was able to analyse a complex auditory signal and segment it into higher level streams. His framework for auditory scene analysis spawned its computational counterpart, trying to reproduce or model accomplishments of the brains with algorithmic means , in particular regarding source separation capabilities. When producing music, recordings of individual instruments called stems are arranged together and mastered into the final song. The goal of source separation is to recover those individual stems from the mixed signal. Unlike the cocktail party problem, there is not a single source of interest to differentiate from an unrelated noise, but instead a wide variety of tones and timbres playing in a coordinated way. In the SiSec Mus evaluation campaign for music separation (Stöter et al., 2018), those individual stems were grouped into 4 broad categories: drums, bass, other, vocals. Given a music track which is a mixture of these four sources, also called the mix, the goal is to generate four waveforms that correspond to each of the original sources. We consider here the case of supervised source separation, where the training data contain music tracks (i.e., mixtures), together with the ground truth waveform for each of the sources. State-of-the-art approaches in music source separation still operate on the spectrograms generated by the short-time Fourier transform (STFT). They produce a mask on the magnitude spectrums for each frame and each source, and the output audio is generated by running an inverse STFT on the masked spectrograms reusing the input mixture phase ). Several architectures trained end-to-end to directly synthesize the waveforms have been proposed (Lluís et al., 2018;), but their performances are far below the state-of-the-art: in Figure 1: Mel-spectrogram for a 0.8 seconds extract of the bass source from the track "Stich Up" of the MusDB test. From left to right: ground truth, Conv-Tasnet estimate and Demucs estimate. We observe that Conv-Tasnet missed one note entirely. the last SiSec Mus evaluation campaign (Stöter et al., 2018), the best model that directly predicts waveforms achieves an average signal-to-noise ratio (SDR) over all four sources of 3.2, against 5.3 for the best approach that predicts spectrograms masks (also see Table 1 in Section 6). An upper bound on the performance of all methods relying on masking spectrograms is given by the SDR obtained when using a mask computed using the ground truth sources spectrograms, for instance the Ideal Ratio Mask (IRM) or the Ideal Binary Mask (IBM) oracles. For speech source separation, proposed Conv-Tasnet, a model that reuses the masking approach of spectrogram methods but learns the masks jointly with a convolutional front-end, operating directly in the waveform domain for both the inputs and outputs. Conv-Tasnet surpasses both the IRM and IBM oracles. Our first contribution is to adapt the Conv-Tasnet architecture, originally designed for monophonic speech separation and audio sampled at 8 kHz, to the task of sterephonic music source separation for audio sampled at 44.1 kHz. Our experiments show that Conv-Tasnet outperforms all previous methods by a large margin, with an SDR of 5.7, but still under the SDR of the IRM oracle at 8.2 (Stöter et al., 2018). However, while Conv-Tasnet separates with a high accuracy the different sources, we observed artifacts when listening to the generated audio: a constant broadband noise, hollow instruments attacks or even missing parts. They are especially noticeable on the drums and bass sources and we give one such example on Figure 1. Conv-Tasnet uses an over-complete linear representation on which it applies a mask obtained from a deep convolutional network. Because both the encoder and decoder are linear, the masking operation cannot synthesize new sounds. We conjecture that the overlap of multiples instruments sometimes lead to a loss of information that is not reversible by a masking operation. To overcome the limitations of Conv-Tasnet, our second contribution is to propose Demucs, a new architecture for music source separation. Similarly to Conv-Tasnet, Demucs is a deep learning model that directly operates on the raw input waveform and generates a waveform for each source. Demucs is inspired by models for music synthesis rather than masking approaches. It is a U-net architecture with a convolutional encoder and a decoder based on wide transposed convolutions with large strides inspired by recent work on music synthesis (Défossez et al., 2018). The other critical features of the approach are a bidirectional LSTM between the encoder and the decoder, increasing the number of channels exponentially with depth, gated linear units as activation function which also allow for masking, and a new initialization scheme. We present experiments on the MusDB benchmark, which first show that both Conv-Tasnet and Demucs achieve performances significantly better than the best methods that operate on the spectrogram, with Conv-Tasnet being better than Demucs in terms of SDR. We also perform human evaluations that compare Conv-Tasnet and our Demucs, which show that Demucs has significantly better perceived quality. The smaller SDR of Demucs is explained by more contamination from other sources. We also conduct an in-depth ablation study of the Demucs architecture to demonstrate the impact of the various design decisions. Finally, we carry out additional experiments by adding 150 songs to the training set. In this experiment, Demucs and TasNet both achieve an SDR of 6.3, suggesting that the gap in terms of SDR between the two models diminishes with more data, making the Demucs approach promising. The 6.3 points of SDR also set a new state-of-the-art, since it improves on the best previous of 6.0 on the MusDB test set obtained by training with 800 additional songs. We discuss in more detail the related work in the next Section. We then describe the original ConvTasnet model of and its adaptation to music source separation. Our Demucs architecture is detailed in Section 4. We present the experimental protocol in Section 5, and the experimental compared to the state-of-the-art in Section 6. Finally, we describe the of the human evaluation and the ablation study. A first category of methods for supervised music source separation work on time-frequency representations. They predict a power spectrogram for each source and reuse the phase from the input mixture to synthesise individual waveforms. Traditional methods have mostly focused on blind (unsupervised) source separation. Non-negative matrix factorization techniques model the power spectrum as a weighted sum of a learnt spectral dictionary, whose elements are grouped into individual sources. Independent component analysis (Hyvärinen et al., 2004) relies on independence assumptions and multiple microphones to separate the sources. Learning a soft/binary mask over power spectrograms has been done using either HMM-based prediction or segmentation techniques . With the development of deep learning, fully supervised methods have gained momentum. Initial work was performed on speech source separation , followed by works on music using simple fully connected networks over few spectrogram frames , LSTMs , or multi scale convolutional/recurrent networks (; . showed that Wiener filtering is an efficient post-processing step for spectrogram-based models and it is now used by all top performing models in this category. Those methods have performed the best during the last SiSec 2018 evaluation (Stöter et al., 2018) for source separation on the MusDB dataset. After the evaluation, a reproducible baseline called Open Unmix has been released by Stöter et al. and matches the top submissions trained only on MusDB. MMDenseLSTM, a model proposed by and trained on 807 unreleased songs currently holds the absolute record of SDR in the SiSec campaign. Both Demucs and Conv-Tasnet obtain significantly higher SDR. More recently, models operating in the waveform domain have been developed, so far with worse performance than those operating in the spectrogram domain. A convolutional network with a U-Net structure called Wave-U-Net was used first on spectrograms and then adapted to the waveform domain . Wave-U-Net was submitted to the SiSec 2018 evaluation campaign with a performance inferior to that of most spectrogram domain models by a large margin. A Wavenet-inspired, although using a regression loss and not auto-regressive, was first used for speech denoising and then adapted to source separation (Lluís et al., 2018). Our model significantly outperforms Wave-U-Net. Given that the Wavenet inspired model performed worse than Wave-U-Net, we did not consider it for comparison. In the field of monophonic speech source separation, spectrogram masking methods have enjoyed good performance . developed a waveform domain methods using masking over a learnable front-end obtained from a LSTM that reached the same accuracy. Improvements were obtained by for spectrogram methods using the unfolding of a few iterations of a phase reconstruction algorithm in the training loss. In the mean time, refined their approach, replacing the LSTM with a superposition of dilated convolutions, which improved the SDR and definitely surpassed spectrogram based approaches, including oracles that use the ground truth sources such as the ideal ratio mask (IRM) or the ideal binary mask (IBM). We show in this paper that Conv-Tasnet also outperforms all known methods for music source separation. However it suffers from significantly more artifacts than the Demucs architecture we introduce in this paper, as measured by mean opinion score. We describe in this section the Conv-Tasnet architecture of and give the details of how we adapted the architecture to fit the setting of the MusDB dataset. Overall framework Each source s is represented by a waveform x s ∈ R C,T where C is the number of channels (1 for mono, 2 for stereo) and T the number of samples of the waveform. The mixture (i.e., music track) is the sum of all sources x:= S s=1 x s. We aim at training a model g parameterized by θ, such that g(x) = (g s (x; θ)) S s=1, where g s (x; θ) is the predicted waveform for source s given x, that minimizes for some dataset D and reconstruction error L. The original Conv-Tasnet was trained using a loss called scale-invariant source-to-noise ratio (SI-SNR), similar to the SDR loss described in Section 5. We instead use a simple L1 loss between the estimated and ground truth sources. We discuss in more details regression losses in the context of our Demucs architecture in Section 4.2. The original Conv-Tasnet architecture Conv-Tasnet is composed of a learnt front-end that transforms back and forth between the input monophonic mixture waveform sampled at 8 kHz and a 128 channels over-complete representation sampled at 1 kHz using a convolution as the encoder and a transposed convolution as the decoder, both with a kernel size of 16 and stride of 8. The high dimensional representation is masked through a separation network composed of stacked residual blocks. Each block is composed of a a 1x1 convolution, a PReLU (b) non linearity, a layer wise normalization over all channels jointly , a depth-wise separable convolution with a kernel size of 3, a stride of 1 and a dilation of 2 n mod N, with n the 0-based index of the block and N an hyper-parameter, and another PReLU and normalization. The output of each block participates to the final mask estimation through a skip connection, preceded by a 1x1 convolution. The original Conv-Tasnet counted 3 × N blocks with N = 8. The mask is obtained summing the output of all blocks and then applying ReLU. The output of the encoder is multiplied by the mask and before going through the decoder. Conv-Tasnet for music source separation We adapted their architecture to the task of stereophonic music source separation: the original Conv-Tasnet has a receptive field of 1.5 seconds for audio sampled at 8 kHz, we take N = 10 and increased the kernel size (resp. stride) of the encoder/decoder from 16 (resp. 8) to 20 (resp. 10), leading to the same receptive field at 44.1 kHz. We observed better using 4 × N blocks instead of 3 × N and 256 channels for the encoder/decoder instead of 128. With those changes, Conv-Tasnet obtained state-of-the-art performance on the MusDB dataset, surpassing all known spectrogram based methods by a large margin as shown in Section 6. Separating entire songs The original Conv-Tasnet model was designed for short sentences of a few seconds at most. When evaluating it on an entire track, we obtained the best performance by first splitting the input track into chunks of 8 seconds each. We believe this is because of the global layer normalization. During training, only small audio extracts are given, so that a quiet part or a loud part would be scaled back to an average volume. However, when using entire songs as input, it will most likely contain both quiet and loud parts. The normalization will not map both to the same volume, leading to a difference between training and evaluation. We did not observe any side effects when going from one chunk to the next, so we did not look into fancier overlap-add methods. The architecture we propose, which we call Demucs, is described in the next few subsections, and the reconstruction loss is discussed in Section 4.2. Demucs takes a stereo mixture as input and outputs a stereo estimate for each source (C = 2). It is an encoder/decoder architecture composed of a convolutional encoder, a bidirectional LSTM, and a convolutional decoder, with the encoder and decoder linked with skip U-Net connections. Similarly to other work in generation in both image (; 2017) and sound (Défossez et al., 2018), we do not use batch normalization as our early experiments showed that it was detrimental to the model performance. The overall architecture is depicted in Figure 2a. Encoder The encoder is composed of L:= 6 stacked convolutional blocks numbered from 1 to L. Each block i is composed of a convolution with kernel size K:= 8, stride S:= 4, C i−1 input channels, C i output channels and ReLU activation, followed by a convolution with kernel size 1, 2C i output channels and gated linear units (GLU) as activation function . Since GLUs halve the number of channels, the final output of block i has C i output channels. A block is described in Figure 2b. Convolutions with kernel width 1 increase the depth and expressivity of the model at low computational cost. As we show in our ablation study 6.2, the usage of GLU activations after these convolutions significantly boost performance. The number of channels in the input mixture is C 0 = C = 2, while we use C 1:= 100 as the number of output channels for the first encoder block. The number of channels is then doubled at each subsequent block, i.e., C i:= 2C i−1 for i = 2..L, so the final number of channels is C L = 3200. We then use a bidirectional LSTM with 2 layers and a hidden size C L. The LSTM outputs 2C L channels per time position. We use a linear layer to take that number down to C L. Decoder The decoder is mostly the inverse of the encoder. It is composed of L blocks numbered in reverse order from L to 1. The i-th blocks starts with a convolution with stride 1 and kernel width 3 to provide context about adjacent time steps, input/output channels C i and a ReLU activation. Finally, we use a transposed convolution with kernel width 8 and stride 4, C i−1 outputs and ReLU activation. The S sources are synthesized at the final layer only, after all decoder blocks. The final layer is linear with S · C 0 output channels, one for each source (4 stereo channels in our case), without any additional activation function. Each of these channels directly generate the corresponding waveform. U-network structure Similarly to Wave-U-Net , there are skip connections between the encoder and decoder blocks with the same index, as originally proposed in U-networks . While the main motivation comes from empirical performances, an advantage of the skip connections is to give a direct access to the original signal, and in particular allows to directly transfers the phase of the input signal to the output, as discussed in Section 4.2. Unlike Wave-U-Net, we use transposed convolutions rather than linear interpolation followed by a convolution with a stride of 1. For the same increase in the receptive field, transposed convolutions require 4 times less operations and memory. This limits the overall number of channels that can be used before running out of memory. As we observed that a large number of channels was key to obtaining good , we favored the use of transposed convolutions, as explained in Section 6. Motivation: synthesis vs masking The approach we follow uses the U-Network architecture (; ;), and builds on transposed convolutions with large number of channels and large strides inspired by the approach to the synthesis of music notes of Défossez et al.. The U-Net skip connections and the gating performed by GLUs imply that this architecture is expressive enough to represent masks on a learnt representation of the input signal, in a similar fashion to Conv-Tasnet. The Demucs approach is then more expressive than Conv-Tasnet, and its main advantages are the multi-scale representations of the input and the non-linear transformations to and from the waveform domain. For the reconstruction loss L(g s (x; θ), x s ) in equation 1, we either use the average mean square error or average absolute error between waveforms: for a waveform x s containing T samples and corresponding to source s, a predicted waveformx s and denoting with a subscript t the t-th sample of a waveform, we use one of L 1 or L 2: In generative models for audio, direct reconstruction losses on waveforms can pose difficulties because they are sensitive to the initial phases of the signals: two signals whose only difference is a shift in the initial phase are perceptually the same, but can have arbitrarily high L 1 or L 2 losses. It can be a problem in pure generation tasks because the initial phase of the signal is unknown, and losses on power/magnitude spectrograms are alternative that do not suffer from this lack of specification of the output. Approaches that follow this line either generate spectrograms (e.g.,, or use a loss that compares power spectrograms of target/generated waveforms (Défossez et al., 2018). The problem of invariance to a shift of phase is not as severe in source separation as it is in unconditional generation, because the model has access to the original phase of the signal. The pase can easily be recovered from the skip connections in U-net-style architectures for separation, and is directly used as input of the inverse STFT for methods that generate masks on power spectrograms. As such, losses such as L1/L2 are totally valid for source separation. Early experiments with an additional term including the loss of Défossez et al. did not suggest that it boosts performance, so we did not pursue this direction any further. Most our experiments use L1 loss, and the ablation study presented in Section 6.2 suggests that there is no significant difference between L1 and L2. The initialization of deep neural networks is known to have a critical impact on the overall performances (; a), up to the point that showed that with a different initialization called fixup, very deep residual networks and transformers can be trained without batch normalization. While Fixup is not designed for U-Net-style skip connections, we observed that the following different initialisation scheme had great positive impact on performances compared to the standard initialization of He et al. (2015a) used in U-Networks. Considering the so-called Kaiming initialization (a) as a baseline, let us look at a single convolution layer for which we denote w the weights after the first initialization. We take α:= std(w)/a, where a is a reference scale, and replace w by w = w/ √ α. Since the original weights have element-wise order of magnitude (KC in) −1/2 where K is the kernel width and C in the number of output channels, it means that our initialization scheme produces weights of order of magnitude (KC in) −1/4, together with a non-trivial scale. Based a search over the values [0.01, 0.05, 0.1], we select a = 0.1 for all the regular and transposed convolutions, see Section 6 for more details. We experimentally observed that on a randomly initialized model applied to an audio extract, it kept the standard deviation of the features along the layers of the same order of magnitude. Without initial rescaling, the output the last layer has a magnitude 20 times smaller than the first. A perfect source separation model is time equivariant, i.e. shifting the input mixture by X samples will shift the output Y by the exact same amount. Thanks to its dilated convolutions with a stride of 1, the mask predictor of Conv-Tasnet is naturally time equivariant and even if the encoder/decoder is not strictly equivariant, Conv-Tasnet still verifies this property experimentally . Spectrogram based method will also verify approximately this property. Shifting the input by a small amount will only reflect in the phase of the spectrogram. As the mask is computed only from the magnitude, and the input mixture phase is reused, the output will naturally be shifted by the same amount. On the other hand, we noticed that our architecture did not naturally satisfy this property. We propose a simple workaround called randomized equivariant stabilization, where we sample S random shifts of an input mixture x and average the predictions of our model for each, after having applied the opposite shift. This technique does not require changing the training procedure or network architecture. Using S = 10, we obtained a 0.3 SDR gain, see Section 6.2 for more details. It does make evaluation of the model S times slower, however, on a V100 GPU, separating 1 minute of audio at 44.1 kHz with Demucs takes only 0.8 second. With this technique, separation of 1 minute takes 8 seconds which is still more than 7 times faster than real time. MusDB and additional data We use the MusDB dataset , which is composed of 150 songs with full supervision in stereo and sampled at 44100Hz. For each song, we have the exact waveform of the drums, bass, other and vocals parts, i.e. each of the sources. The actual song, the mixture, is the sum of those four parts. The first 84 songs form the train set, the next 16 songs form the valid set (the exact split is defined in the musdb python package) while the remaining 50 are kept for the test set. We collected raw stems for 150 tracks, i.e., individual instrument recordings used in music production software to make a song. We manually assigned each instrument to one of the sources in MusDB. We call this extra supervised data the stem set. We also report the performances of Tasnet and Demucs trained using these 150 songs in addition to the 84 from MusDB, to anaylze the effect of adding more training data. Source separation metrics Measurements of the performance of source separation models was developed by Vincent et al. for blind source separation and reused for supervised source separation in the SiSec Mus evaluation campaign (Stöter et al., 2018). Similarly to previous work , we focus on the SDR (Signal to Distortion Ratio) which measures the log ratio between the volume of the estimated source projection onto the ground truth, and the volume of what is left out of this projection, typically contamination by other sources or artifacts. Other metrics can be defined (SIR and SAR) and we present them in the supplementary material. We used the python package museval which provide a reference implementation for the SiSec Mus 2018 evaluation campaign. As done in the SiSec Mus competition, we report the median over all tracks of the median of the metric over each track computed using the museval package. As baselines, we selected Open Unmix (Stöter et al., 2019), a 3-layer BiLSTM model with encoding and decoding fully connected layers on spectrogram frames. It was released by the organizers of the SiSec 2018 to act as a strong reproducible baseline and matches the performances of the best candidates trained only on MusDB. We also selected MMDenseLSTM , a multi-band dense net with LSTMs at different scales of the encoder and decoder. This model was submitted as TAK2 and trained with 804 extra labeled songs 1. Both MMDenseLSTM and Open Unmix use Wiener filtering as a last post processing step. The only waveform based method submitted to the evaluation campaign is Wave-U-Net with the identifier STL2. Metrics were downloaded from the SiSec submission repository. for Wave-U-Net and MMDenseLSTM. For Open Unmix they were provided by their authors 2. We also provide the metrics for the Ideal Ratio Mask oracle (IRM), which computes the best possible mask using the ground truth sources and is the topline of spectrogram based method (Stöter et al., 2018). Table 1: Comparison of Conv-Tasnet and Demucs to state-of-the-art models that operate on the waveform (Wave-U-Net) and on spectrograms (Open-Unmix without extra data, MMDenseLSTM with extra data) as well as the IRM oracle on the MusDB test set. The Extra? indicates the number of extra training songs used. We report the median over all tracks of the median SDR over each track, as done in the SiSec Mus evaluation campaign (Stöter et al., 2018). The All column reports the average over all sources. Demucs metrics are averaged over 5 runs, the confidence interval is the standard deviation over √ 5. In bold are the values that are statistically state-of-the-art either with or without extra training data. Epoch definition and augmentation We define one epoch over the dataset as a pass over all 11-second extracts with a stride of 1 second. We use a random audio shift between 0 and 1 second and keep 10 seconds of audio from there as a training example. We perform the following data augmentation , also used by Open Unmix and MMDenseLSTM: shuffling sources within one batch to generate one new mix, randomly swapping channels. We additionally multiply each source by ±1 . All Demucs models were trained over 240 epochs. Conv-Tasnet was trained for 360 epochs when trained only on MusDB and 240 when trained with extra data and using only 2-seconds audio extracts. Training setup and hyper-parameters All models are trained with 16 V100 GPUs with 32GB of RAM. We use a batch size of 64, the Adam optimizer with a learning rate was chosen among [3e-4, 5e-4] and the initial number of channels was chosen in based on the L1 loss on the validation set. We obtained best performance with a learning rate of 3e − 4 and 100 channels. We then tried 3 different values for the initial weight rescaling reference level described in Section 4.3, [0.01, 0.05, 0.1] and selected 0.1. We computed confidence intervals using 5 random seeds in Table 1. For the ablation study on Table 4, we provide metrics for a single run. In this section, we first provide experimental on the MusDB dataset for Conv-Tasnet and Demucs compared with state-of-the-art baselines. We then dive into the ablation study of Demucs. We provide a comparison the state-of-the-art baselines on Table 1. The models on the top half were trained without any extra data while the lower half used unreleased training songs. As no previous work included confidence intervals, we considered the single metric provided by for the baselines as the exact estimate of their mean performance. Quality of the separation We first observe that Demucs and Conv-Tasnet outperforms all previous methods for music source separation. Conv-Tasnet has significantly higher SDR with 5.73, improving by 0.4 over Open-Unmix. Our proposed Demucs architecture has worse overall performance but , all methods are still far below the IRM oracle, leaving room for future improvements. We provide for the other metrics (SIR and SAR) as well as box plots with quantiles over the test set tracks in Appendix B. Audio samples for Demucs, Conv-Tasnet and all baselines are provided in the ICLR link code, with more details given in Appendix A. Human evaluations We noticed strong artifacts on the audio separated by Conv-Tasnet, especially for the drums and bass sources: static noise between 1 and 2 kHz, hollow instrument attacks or missing notes as illustrated on Figure 1. In order to confirm this observation, we organized a mean opinion score survey. We separated 8 seconds extracts from all of the 50 test set tracks for Conv-Tasnet, Demucs and Open-Unmix. We asked 38 participants to rate 20 samples each, randomly taken from one of the 3 models or the ground truth. For each one, they were required to provide 2 ratings on a scale of 1 to 5. The first one evaluated the quality and absence of artifacts (1: many artifacts and distortion, content is hardly recognizable, 5: perfect quality, no artifacts) and the second one evaluated contamination by other sources (1: contamination if frequent and loud, 5: no contamination). We show the on Tables 2 and 3. We confirmed that the presence of artifacts in the output of Conv-Tasnet degrades the user experience, with a rating of 2.85±.08 against 3.22 ±.09 for Demucs. On the other hand, Conv-Tasnet samples had less contamination by other sources than Open-Unmix or Demucs, although by a small margin, with a rating of 3.42 ±.09 against 3.30 ±.10 for Demucs and 3.27 ±.11 for Open-Unmix. Training speed We measured the time taken to process a single batch of size 16 with 10 seconds of audio at 44.1kHz (the original Wave-U-Net being only trained on 22 kHz audio, we double the time for fairness), ignoring data loading and using torch.cuda.synchronize to wait on all kernels to be completed. MMDenseLSTM does not provide a reference implementation. Wave-U-Net takes 1.2 seconds per batch, Open Unmix 0.2 seconds per batch and Demucs 1.6 seconds per batch. Conv-Tasnet cannot be trained with such a large sample size, however a single iteration over 2 seconds of audio with a batch size of 4 takes 0.7 seconds. We provide an ablation study of the main design decisions for Demucs in Table 4. Given the cost of training a single model, we did not compute confidence intervals for each variation. Yet, any difference inferior to.06, which is the standard deviation observed over 5 repetitions of the Reference model, could be attributed to noise. We observe a small but not significant improvement when using the L1 loss instead of the MSE loss. Adding a BiLSTM and using the initial weight rescaling described in Section 4.3 provides significant gain, with an extra 0.48 SDR for the first and 0.64 for the second. We observe that using randomized equivariant stabilization as described in Section 4 gives a gain of almost 0.3 SDR. We did not report the validation loss as we only use the stabilization when computing the SDR over the test set. We applied the randomized stabilization to Open-Unmix and Conv-Tasnet with no gain, since, as explained in Section 4.4, both are naturally equivariant with respect to initial time shifts. We introduced extra convolutions in the encoder and decoder, as described in Sections 4.1. The two proved useful, improving the expressivity of the model, especially when combined with GLU activation. Using a kernel size of 3 instead of 1 in the decoder further improves performance. We conjecture that the context from adjacent time steps helps the output of the transposed convolutions to be consistent through time and reduces potential artifacts arising from using a stride of 4. We showed that Conv-Tasnet, a state-of-the-art architecture for speech source separation that predicts masks on a learnt front-end over the waveform domain, achieves state-of-the-art performance for music source separation, improving over all previous spectrogram or waveform domain methods by 0.4 SDR. While Conv-Tasnet has excellent performance to separate sources, it suffers from noticeable artifacts as confirmed by human evaluations. We developed an alternative approach, Demucs, that combines the ability to mask over a learnt representation with stronger decoder capacity that allows for audio synthesis. We conjecture that this can be useful when information is lost in the mix of instruments and cannot simply be recovered by masking. We show that our approach produces audio of significantly higher quality as measures by mean opinion scores and matches the SDR of Conv-Tasnet when trained with 150 extra tracks. We believe those make it a promising alternative to methods based on masking only.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We match the performance of spectrogram based model with a model trained end-to-end in the waveform domain

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the article for finding a single-sentence TLDR

Question: Although challenging, strategy profile evaluation in large connected learner networks is crucial for enabling the next wave of machine learning applications. Recently, $\alpha$-Rank, an evolutionary algorithm, has been proposed as a solution for ranking joint policy profiles in multi-agent systems. $\alpha$-Rank claimed scalability through a polynomial time implementation with respect to the total number of pure strategy profiles. In this paper, we formally prove that such a claim is not grounded. In fact, we show that $\alpha$-Rank exhibits an exponential complexity in number of agents, hindering its application beyond a small finite number of joint profiles. Realizing such a limitation, we contribute by proposing a scalable evaluation protocol that we title $\alpha^{\alpha}$-Rank. Our method combines evolutionary dynamics with stochastic optimization and double oracles for \emph{truly} scalable ranking with linear (in number of agents) time and memory complexities. Our contributions allow us, for the first time, to conduct large-scale evaluation experiments of multi-agent systems, where we show successful on large joint strategy profiles with sizes in the order of $\mathcal{O}(2^{25})$ (i.e., $\approx \text{$33$ million strategies}$) -- a setting not evaluable using current techniques. Scalable policy evaluation and learning have been long-standing challenges in multi-agent reinforcement learning (MARL) with two difficulties obstructing progress. First, joint-strategy spaces exponentially explode when a large number of strategic decision-makers is considered, and second, the underlying game dynamics may exhibit cyclic behavior (e.g. the game of Rock-Paper-Scissor) rendering an appropriate evaluation criteria non-trivial. Focusing on the second challenge, much work in multi-agent systems followed a game-theoretic treatment proposing fixed-points, e.g., Nash equilibrium, as potentially valid evaluation metrics. Though appealing, such measures are normative only when prescribing behaviors of perfectly rational agents -an assumption rarely met in reality;. In fact, many game dynamics have been proven not converge to any fixed-point equilibria , but rather to limit cycles . Apart from these aforementioned inconsistencies, solving for a Nash equilibrium even for "simple" settings, e.g. two-player games is known to be PPAD-complete -a demanding complexity class when it comes to computational requirements. To address some of the above limitations, recently proposed α-Rank as a graph-based game-theoretic solution to multi-agent evaluation. α-Rank adopts Markov Conley Chains to highlight the presence of cycles in game dynamics, and attempts to compute stationary distributions as a mean for strategy profile ranking. Though successful in small-scale applications, α-Rank severely suffers in scalability contrary to polynomial time claims made in. In fact, we show that α-Rank exhibits exponential time and memory complexities shedding light on the small-scale empirical study conducted in, whereby the largest reported game included only four agents with four available strategies each. In this work, we put forward α α -Rank as a scalable alternative for multi-agent evaluation with linear time and memory demands. Our method combines numerical optimization with evolutionary game theory for a scalable solver capable of handling large joint spaces with millions of strategy profiles. To handle even larger profiles, e.g., tens to hundreds of millions, we further introduce an oracle Figure 1: Example of population based evaluation on N = 3 learners each with 3 strategies and 5 copies. a) Each population obtains a fitness value P i depending on the strategies chosen, b) mutation strategy (red star), and c) population either selecting original strategy, or adopting the novel strategy. mechanism transforming joint evaluation into a sequence of incremental sub-games with varying sizes. Given our algorithmic advancements, we justify our claims in a largescale empirical study involving systems with O(2 25) possible strategy profiles. We first demonstrate the computation advantages of α α -Rank on varying size stochastic matrices against other implementations in Numpy, PyTorch, and OpenSpiel. With these successes, we then consider experiments unsolvable by current techniques. Precisely, we evaluate multi-agent systems in self-driving and Ising model scenarios each exhibiting a prohibitively-large strategy space (i.e., order of thousands for the former, and tens of millions for the latter). Here, we again show that α α -Rank is capable of recovering correct strategy ranking in such complex domains. In α-Rank, strategy profiles of N agents are evaluated through an evolutionary process of mutation and selection. Initially, agent populations are constructed by creating multiple copies of each learner i ∈ {1, . . ., N} assuming that all agents (in one population) execute the same unified policy. With this, α-Rank then simulates a multi-agent game played by randomly sampled learners from each population. Upon game termination, each participating agent receives a payoff to be used in policy mutation and selection after its return to the population. Here, the agent is faced with a probabilistic choice between switching to the mutation policy, continuing to follow its current policy, or randomly selecting a novel policy (other than the previous two) from the pool. This process repeats with the goal of determining an evolutionary strong profile that spreads across the population of agents. Each of the above three phases is demonstrated in Fig. 1 on a simple example of three agents -depicted by different symbols -each equipped with three strategies -depicted by the colors. Mathematical Formalisation, Notation, and Definitions: We next formalize the process posed by α-Rank, which will lead to its limitations, and also pave the way for our own proposed solution. We consider N agents with each agent i having access to a set of strategies of size s i. At round k of the evaluation process, we denote the strategy profile for agent i by S th allowed policy of the learner. X represents the set of states and A i is the set of actions for agent i. With this, we define a joint strategy profile for all participating agents as policies belonging to the joint strategy pool, i and j i ∈ {1, . . ., s i}. To evaluate performance, we assume each agent is additionally equipped with a payoff (reward) function P is the pool of joint strategies so to accommodate the effect of other learners on the i th player performance further complicating the evaluation process. Finally, given a joint profile π joint, we define the corresponding joint payoff to be the collection of all individual payoff functions, i.e., P After attaining rewards from the environment, each agent returns to its population and faces a choice between switching to a mutation policy, exploring a novel policy, or sticking to the current one Such a choice is probabilistic and defined proportional to rewards. Precisely, agent i adopts i,c, π with µ ∈ R + denoting an exploration parameter 1, π i representing policies followed by other agents at round k, and α ∈ R + an intensity ranking parameter. As noted in, one can relate the above switching process to a random walk on a Markov chain with states defined as elements in S [k] joint and transition probabilities through payoff functions. In particular, each entry in the transition probability matrix T ∈ R refers to the probability of one agent switching from one policy in a relation to attained payoffs. Precisely, consider any two joint strategy profiles π [k] joint andπ [k] joint that differ in only one individual strategy for the i th agent, i.e., there exists a unique agent such that π −i defining the probability that one copy of agent i with strategy π [k] i,a invades the population with all other agents (in that population) playingπ −i, such a probability is formalized as:, and 1 /m otherwise, with m being the size of the population. So far, we presented relevant derivations for the (π joint) entry of the state transition matrix when exactly the i th agent differs in exactly one strategy. Having one policy change, however, only represents a subset of allowed variations, where two more cases need to be considered. Now we restrict out attention to variations in joint policies involving more than two individual strategies, i.e., π joint,π [k] joint = 0. Consequently, the remaining event of self-transitions can be thus written as. Summarising the above three cases, we can then write the (π joint)'s entry of the Markov chain's transition matrix as: The goal in α-Rank is to establish an ordering in policy profiles dependent on evolutionary stability of each joint strategy. In other words, higher ranked strategies are these that are prevalent in populations with higher average times. Formally, such a notion can be easily derived as the limiting vector v 0 of our Markov chain when evolving from an initial distribution v 0. Knowing that the limiting vector is a stationary distribution, one can calculate strategy rankings as the solution to the following eigenvector problem: Limitations of α-Rank: Though the work in seeks to determine a solution to the above problem, it is worth mentioning that α-Rank suffers from one major drawback-the scalability-that we remedy in this paper. We note that the solution methodology in α-Rank is in fact unscalable to settings involving more than a hand-full of agents. Particularly, authors claim polynomial complexities of their solution to the problem in Eqn. 3. Though polynomial, such a complexity, however, is polynomial in an exponential search space, i.e., the space of joint strategy profiles. As such, the polynomial complexity claim is not grounded, and need to be investigated. In short, α-Rank exhibits an exponential (in terms of the number of agents) complexity for determining a ranking, thus rendering it inapplicable to settings involving more than a small amount of agents. In what comes next, we first discuss traditional approaches that could help solve the Eqn. 3; soon we realize an off-the-shelve solution is unavailable. Hence, we commence to propose an efficient evaluation algorithm, i.e., α α -Rank, based on stochastic optimization with suitable complexities and rigorous theoretical guarantees. At the end, we propose a search heuristic to further scale up our method by introducing oracles and we name it by α α -Oracle. The problem of computing stationary distributions is a long-standing classical problem from linear algebra. Various techniques including power method, PageRank, eigenvalue decomposition, and mirror descent can be utilized for solving the problem in Eqn. 3. As we demonstrate next, any such implementation scales exponentially in the number of learners, as we summarize in Table 1. Power Method. One of the most common approaches to computing the solution in Eqn. 3 is the power method. Power method computes the stationary vector v k by constructing a sequence {v j} j≥0 from a non-zero initial vector v 0 by applying Though viable, we first note that the power method exhibits an exponential memory complexity in terms of the number of agents. To formally derive the bound, define n to represent the total number of joint strategy profiles (i.e., n = |S Analyzing its time complexity, on the other hand, requires a careful consideration that links convergence rates with the ing graph topology of the Markov chain. Precisely, the convergence rate of the power method is dictated by the second-smallest eigenvalue of the normalized Laplacian, L G, of the graph, G, associated to the Markov chain in Section 2, i.e., v joint and transition probability matrix T [k]. The second-smallest eigenvalue of the normalized Laplacian of the graph associated with the Markov chain is given by:, with s i denoting the number of strategies of agent i. Due to space constraints, the full proof of the above lemma is refrained to Appendix A.1. The importance of Lemma A.1 is that the ant time complexity of the power method is also exponential of the form O n × log (PageRank. Inspired by ranking web-pages on the internet, one can consider PageRank for computing the solution to the eigenvalue problem in Eqn. 3. Applied to our setting, we first realize that the memory is analogous to the power method that is O(m size) = O n Eigenvalue Decomposition. Apart from the above, we can also consider the problem as a standard eigenvalue decomposition task (also what the original α-Rank is implemented according to) and adopt the method in to compute the stationary distribution. Unfortunately, state-of-the-art techniques for eigenvalue decomposition also require exponential memory and exhibit a time complexity of the form Clearly, these bounds restrict α-Rank to small number of agents N. Mirror Descent. The ordered subsets mirror descent requires at each iteration a projection on standard n−dimensional simplex: As stated in the paper, the computing of this projection requires O(n log n) time. In our setting, n = N i=1 s i is the total number of joint strategy profiles. Hence, the projection step is exponential in the number of agents N. This makes mirror descent inapplicable for α-Rank when N is large. Rather than seeking an exact solution to the problem in Eqn. 3, one can consider approximate solvers by defining a constraint optimization objective: The constrained objective in Eqn. 4 simply seeks a vector x minimizing the distance between x, itself, and while ensuring that x lies on an ndimensional simplex (i.e., x T 1 = 1, and x 0). Due to time and memory complexities required for computing exact solutions, we focus on determining an approximate vectorx defined to be the solution to the following relaxed problem of Eqn. 4: The optimization problem in Eqn. 5 can be solved using a barrier-like technique that we detail below. Before that, it is instructive to clarify the connection between the original and the relaxed problems Proposition: [Connections to Markov Chain] Letx be a solution to the relaxed optimization problem in Eqn. 5. Then,x/||x||1 = v [k] is the stationary distribution to the Markov chain in Section 2. Importantly, the above proposition allows us to focus on solving the problem in Eqn. 5 which only exhibits inequality constraints. Problems of this nature can be solved by considering a barrier function leading to an unconstrained finite sum minimization problem. To do so, denoting b [k] i to be the i th row of T [k],T − I, we can write. Introducing logarithmic barrier-functions, with λ > 0 being a penalty parameter, we arrive at Eqn. 6 is a standard finite minimization problem that can be solved using any off-the-shelve stochastic optimization algorithm, e.g., stochastic gradients, ADAM among others. A stochastic gradient execution involves sampling a strategy profile i t ∼ [1, . . ., n] at iteration t, and then executing a descent step:, with ∇ x f it (x t) being a sub-sampled gradient of Eqn. 6, and λ being a scheduled penalty parameter with λ t+1 = λt /γ for some γ > 1, See Phase I in Algorithm 1 for the pseudo-code. We can further derive a convergence theorem of: Theorem: [Convergence of Barrier Method] Letx λ be the output of a gradient algorithm descending in the objective in Eqn. 6, after T iterations, then where expectation is taken w.r.t. all randomness of a stochastic gradient implementation, and γ > 1 is a decay-rate for λ, i.e., λ, penalty parameter λ ≈ O, λ decay rate γ > 1, total number of joint strategy profiles n, and a constraint relaxation term δ. Oracle Parameters: initialize a subset of strategy pools for all agents {S i } by randomly sampling from {S i} 2: Set outer iteration count k = 0 3: while stopping criteria do: Phase I: Scalable Policy Evaluation (Section 3.2): for t = 0 → T − 1 do: Uniformly sample one strategy profile i Update solution x 10: Phase II (if turned on): Scalable Policy Evaluation with Oracle (Section 3.3): for each agent i do: 12: Compute the best-response strategy π * i by solving Eqn. 8. Update strategy pools for each agent i as S Set k = k + 1 15: Return: Best performing strategy profile π joint, across all agents. The proof of the above theorem (see the full proof in Appendix A.2) is interesting by itself, a more important aspect is the memory and time complexity implications posed by our algorithm. Theorem A.2 implies that after T = O(1 / 2) iterations with being a precision parameter, our algorithm outputs a vectorx λ 0 such that Moreover, one can easily see 3 that after T steps, the overall time and memory complexities of our update rules are given by eventually leads to a memory complexity of Table. 1). Hence, our algorithm is able to achieve an exponential reduction, in terms of number of agents, in both memory and time complexities. So far, we have presented scalable multi-agent evaluations through stochastic optimization. We can further boost scalability (to tens of millions of joint profiles) of our method by introducing an oracle mechanism. The heuristic of oracles was first introduced in solving large-scale zero-sum matrix games . The idea is to first create a restricted sub-game in which all players are only allowed to play a restricted number of strategies, which are then expanded by adding incorporating each of the players' best-responses to opponents; the sub-game will be replayed with agents' augmented strategy pools before a new round of best responses is found. The worse-case scenario of introducing oracles would be to solve the original evaluation problem in full size. The best response is assumed to be given by an oracle that can be simply implemented by a grid search. Precisely, given the top-rank profile π at iteration k, the goal for agent i is to select 4 the optimal π * i from the pre-defined strategy pool S i to maximize the reward with x [k] h denoting the state, u −i,h ) denoting the actions from agent i and the opponents, respectively. The heuristic of solving the full game from restricted sub-games is crucial especially when it is prohibitively expensive to list all joint-strategy profiles, e.g., in scenarios involving tens-of-millions of joint profiles. For a complete exposition, we summarize the pseudo-code in Algorithm 1. In the first phase, vanilla α α -Rank is executed (lines 4-9), while in the second (lines 11 -13), α α -Rank with Oracle (if turned on) is computed. To avoid any confusion, we refer to the latter as α α -Oracle. Note that even though in the two-player zero-sum games, the oracle algorithm is guaranteed to converge to the minimax equilibrium. Providing valid convergence guarantees for α α -Oracle is an interesting direction for future work. In this paper, we rather demonstrate the effectiveness of such an approach in a large-scale empirical study as shown in Section 4. In this section, we evaluate the scalability properties of α α -Rank 5. Precisely, we demonstrate that our method is capable of successfully recovering optimal policies in self-driving car simulations and in the Ising model where strategy spaces are in the order of up to tens-of-millions of possible strategies. We note that these sizes are well beyond the capabilities of state-of-the-art methods, e.g., α-Rank ) that considers at maximum four agents with four strategies, or AlphaStar which handles about 600 strategies as detailed in. Sparsity Data Structures. During the implementation phase, we realised that the transition probability, T [k], of the Markov chain induces a sparsity pattern (each row and column in 2) that if exploited can lead to significant speed-ups. To fully leverage such sparsity, we tailored a novel data structure for sparse storage and computations needed by Algorithm 1. More details can be found in Appendix B.1. Correctness of Ranking Results. Before conducting large-scale sophisticated experiments, it is instructive to validate the correctness of our on the simple cases especially those reported by. We therefore test on three normal-form games. Due to space constraints, we refrain the full description of these tasks to Appendix B.2. Fig. 2 shows that, in fact, generated by α α -Rank, the Phase I of Algorithm 1, are consistent with α-Rank's . Complexity Results on Random Matrices. We measured the time and memory needed by our method for computing the stationary distribution with varying sizes of simulated random matrices. Baselines includes eigenvalue decomposition from Numpy, optimization tools in PyTorch, and α-Rank from OpenSpiel. For our algorithm we terminated execution with gradient norms being below a predefined threshold of 0.01. According to Fig. 3, α α -Rank can achieve three orders of magnitude reduction compared to eigenvalue decomposition in terms of time. Most importantly, the performance gap keeps developing with the increasing matrix size. Autonomous Driving on Highway: High-way provides an environment for simulating self-driving scenarios with social vehicles designed to mimic real-world traffic flow as strategy pools. We conducted a ranking experiment involving 5 agents each with 5 strategies, i.e. a strategy space in the order of O(5 5) (3125 possible strategy profiles). Agent strategies varied between "rational" and "dangerous" drivers, which we encoded using different reward functions during training (complete details of defining reward functions can be found in Appendix C.2). Under this setting, we know, upfront, that optimal profile corresponds to all agents is five rational drivers. Cars trained using value-iteration and rewards averaged from 200 test trails were reported. Due to the size of the strategy space, we considered both α α -Rank and α α -Oracle. We set α α -Oracle to run 200 iterations of gradient updates in solving the top-rank strategy profile (Phase I in Algorithm 1). Results depicted in Fig. 4 (a) clearly demonstrate that both our implementations are capable of recovering the correct highest ranking strategy profile. We also note that though such sizes are feasible using α-Rank and the power-method, our achieve 4 orders of magnitude reduction in total number of iterations. Ising Model Experiment: The Ising model is the model for describing ferromagnetism in statistical mechanics. It assumes a system of magnetic spins, where each spin a j is either an upspin, ↑, or down-spin, ↓. The system energy is defined by E(a, h) with h j and λ being constant coefficients. The probability of one set of spin configuration is P (a) = exp(−E(a,h)/τ ) a exp(−E(a,h)/τ ) where τ is the environmental temperature. Finding the equilibrium of the system is notoriously hard because it is needed to enumerate all possible configurations in computing P (a).Traditional approaches include Markov Chain Monte Carlo (MCMC). An interesting phenomenon is the phase change, i.e., the spins will reach an equilibrium in the low temperatures, with the increasing τ, such equilibrium will suddenly break and the system becomes chaotic. Here we try to observe the phase change through multi-agent evaluation methods. We assume each spins as an agent, and the reward to be r j = h j a j + λ 2 k =j a j a k, and set α = 1 τ to build the link between Eqn. 1 and P (a). We consider the top-rank strategy profile from α α -Oracle as the system equilibrium and compare it against the ground truth from MCMC. We consider a five-by-five 2D model which induces a prohibitively-large strategy space of size 2 25 (tens of millions) to which the existing baselines, including α α -Rank on the single machine, are inapplicable. Fig. 4(b) illustrates that our method identifies the same phase change as what MCMC suggests. We show an example of how α α -Oracle's top-ranked profile finds the system equilibrium in Fig. 4 (c) at τ = 1. In this paper, we demonstrated that the approach in exhibits exponential time and memory complexities. We then proposed α α -Rank as a scalable solution for multi-agent evaluation with linear time and memory demands. In a set of experiments, we demonstrated that our method is truly scalable capable of handling large strategy spaces. There are a lot of interesting avenues for future research. First, we plan to theoretically analyze convergence properties of the ing oracle algorithm, and further introduce policy learning through oracles. Second, we plan take our method to the real-world by conducting multi-robot experiments. joint and transition probability matrix T [k]. The second-smallest eigenvalue of the normalized Laplacian of the graph associated with the Markov chain is given by:, with s i denoting the number of strategies of agent i. Proof: For simplicity we drop round index k in the below derivation. Notice, the underlying graph for the constructed Markov Chain can be represented as a Cartesian product of N complete graphs Indeed, two vertices π [k],π [k] ∈ G are connected by the edge if and if only these joint strategy profiles differ in at most one individual strategy, i.e ∃!i ∈ {1, . . ., N}: −i }.Hence, the spectral properties of G can be described in terms of spectral properties of K si as follows : ) is the i th eigenvalue of the unnormalized Laplacian of the complete graph K sj and ϑ i,j is the corresponding eigenvector 7. The spectrum of unnormalized Laplacian of the complete graph K si is given by Spectr(K si) = {0, s i − 1} and the only eigenvector corresponding to zero eigenvalue is 1 ∈ R si. Therefore, the minimum non-zero eigenvalue of unnormalized Laplacian of G is given by min i s i − 1. Finally, due to the fact that G is a regular graph (with degree of each node is equal to N i=1 s i − N + 1), the smallest non-zero eigenvalue of the normalized Laplacian of G is given by Giving this , the overall time complexity of Power Method is bounded by O n × log = O (log n). As for the memory complexity, Power Method requires has the same requirements as PageRank algorithm. 8 These imply that Power Method scales exponentially with number of agents N, and therefore, inapplicable when N is large. Theorem: [Convergence of Barrier Method] Letx λ be the output of a gradient algorithm descending in the objective in Eqn. 6, after T iterations, then where expectation is taken w.r.t. all randomness of a stochastic gradient implementation, and γ > 1 is a decay-rate for λ, i.e., λ Sample i t ∼ [1, . . ., n] and compute: The above implies given precision parameter > 0, after T = O 1 2 iterations, Algorithm 2 outputs vectorx λ 0 such that: Hence, by tuning parameters δ and one can approximate a stationary distribution vector ν [k]. Algorithm 2 starts with uniform distribution vector x 0 = 1 n 1 and at step t it updates the previous iterative x t by a rule given in line 6. Let 9. Then, for j / ∈ N t all entries [x t+1] j are equal to each other and updated as: Given value x T t 1, the above computation takes O time and space. For entries j ∈ N t, all entries [x t+1] j might be different (worst case) and, therefore, update The transitional probability matrix T in α α -Rank is sparse; each row and column in T [k] contains N i=1 s i − N + 1 non-zero elements (see Section 3.2). To fully leverage such sparsity, we design a new data structure (see Fig. 5) for the storage and computation. Compared to standard techniques (e.g., COO, CSR, and CRS 10) that store (row, column, value) of a sparse vector, our data structure adopts a more efficient protocol that stores (defaults, positions, biases) leading to improvements in computational efficiency, which gives us additional advantages in computational efficiency. We reload the operations for such data structure including addition, scalar multiplication, dot product, element-wise square root, L1 norm. We show the example of addition in Fig. 5. Our algorithm provides the expected ranking in all three normal-form games shown in Fig. 6, which is consistent with the in α-Rank. Battle of sexes. Battle of sexes is an asymmetric game R OM = [. As it is a single-population game, we adopt the transitional probability matrix of Eqn. 11 in . Such game has the inherent structure that Rock/Paper/Scissor is equally likely to be invaded by a mutant, e.g., the scissor population will always be fixated by the rock population, therefore, our method suggests the long-term survival rate for all three strategies are the same ( For all of our experiments, the gradient updates include two phases: warm-up phase and Adam phase. In the warm-up phase, we used standard stochastic gradient descent; after that, we replace SGD with Adam till the convergence. In practice, we find this yields faster convergence than normal stochastic gradient descent. As our algorithm does column sampling for the stochastic matrix (i.e. batch size equals to one), adding momentum term intuitively help stabilize the learning. We also implement infinite α, when calculating transition matrix (or its column), where our noise term is set to be 0.01. For most of our experiments that involve α α -rank, we set the terminating condition to be, when the gradient norm is less than 10 −9. However, for Random Matrix experiment, we set the terminating gradient norm to be 10 −2 • Learning rate to be in between 15 -17 • Alpha (ranking intensity) to be in between 1 -2.5 • Number of Population to be between 25 -55 (in integer) For all of the Adam experiments, after the warmup-step we chooses to decay δ and λ by 0.999 for each time steps, where we have δ to always be 0.1. Similarly, λ starts at the value 0.5. However, in speed and memory experiment, we chooses the decay to be 0.9 Collision Reward is calculated when agent collided with either social car or other agents. All of our value iteration agents are based on environment discretization, which represents the environment in terms of time to collision MDP, taking into account that the other agents are moving in constant speed. For all experiments, we run value-iteration for 200 steps with the discounting factor of 0.99. For each controllable cars, the default speed is randomized to be between 10 to 25, while the social cars, the speed are randomized to be between 23 to 25. We define five types of driving behaviors (one rational + four dangerous) by letting each controlled car have a different ego reward function during training (though the reward we report is the environmental reward which cannot be changed). By setting this, we can make sure, at upfront, the best joint-strategy strategy should be all cars to drive rationally.

Represent the TLDR for finding the corresponding scientific article

We provide a scalable solution to multi-agent evaluation with linear rate complexity in both time and memory in terms of number of agents

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper, a new method we call Centered Initial Attack (CIA) is provided. It insures by construction the maximum perturbation to be smaller than a threshold fixed beforehand, without the clipping process.