Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Reinforcement learning (RL) is a powerful framework for solving problems by exploring and learning from mistakes. However, in the context of autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world. For this reason, AV RL is generally only viable in simulation. Because these simulations have imperfect representations, particularly with respect to graphics, physics, and human interaction, we find motivation for a framework similar to RL, suitable to the real world. To this end, we formulate a learning framework that learns from restricted exploration by having a human demonstrator do the exploration. Existing work on learning from demonstration typically either assumes the collected data is performed by an optimal expert, or requires potentially dangerous exploration to find the optimal policy. We propose an alternative framework that learns continuous control from only safe behavior. One of our key insights is that the problem becomes tractable if the feedback score that rates the demonstration applies to the atomic action, as opposed to the entire sequence of actions. We use human experts to collect driving data as well as to label the driving data through a framework we call ``Backseat Driver'', giving us state-action pairs matched with scalar values representing the score for the action. We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples. We empirically validate several models in the ReNeg framework, testing on lane-following with limited data. We find that the best solution in this context outperforms behavioral cloning has strong connections to stochastic policy gradient approaches. We seek a way learn autonomous vehicle control using only RGB input from a front-facing camera. (Example input image in the appendix.) Specifically, we want to produce a policy network to map states to actions in order to follow a lane as close to center as possible. We would like the agent to behave optimally in dangerous situations, and recover to safe situations. For this to happen, the data the agent trains on must include these sub-optimal states. However, we do not want to leave the safety determination up to any machine. Thus, we require that a human explorer take all the actions. This is referred to as "Learning from Demonstration" or LfD. An expert human does the exploration and thus can limit the danger by controlling how dangerous the negative examples are (i.e. the human can swerve to show suboptimal driving but never drive off the road, as an RL agent likely would). Still, in order to get into sub-optimal states, the explorer will need to take sub-optimal actions. Ideally we could replace these sub-optimal actions with labels corresponding to the correct and optimal actions to take so we could perform supervised learning, which provides the strongest signal. However, it is notoriously difficult to assign supervised labels to driving data because it is hard to know the correct steering angle to control a vehicle when your steering has no effect on what happens to the vehicle BID11. Instead of trying to enable humans to assign supervised labels by somehow showing the consequences of their actions, we focus on letting humans assign feedback that evaluates the sub-optimal actions, without actually representing the optimal action. Thus our goal is to learn a deterministic continuous control policy from demonstration, including both good and bad actions and continuous scores representing this valence. Our problem statement is somewhere in between supervised learning and RL: we focus on a general approach that is capable of mapping continuous sensor input to an arbitrary (differentiable) continuous policy output and capable of using feedback for singular predetermined actions. Our problem falls within the supervised setting since our agent cannot chose want actions to take, in order to avoid the agent exploring dangerous states, and all data collection is done prior to training. However, we also wish to incorporate evaluative scalar feedback given to each data point, which traditionally falls under the RL framework. We refer to this problem setting as the ReNeg framework, since we are essentially performing a regression with scalar weights attached to each data point, that can be positive and negative. For the demonstration, a human demonstrator drives, yielding state-action pairs the agent can learn from. In order to teach the agent which actions are good or bad, an expert critic, or "backseat driver," labels the actions with a continuous scalar. For a viable solution to this problem, we need to define a loss function that induces an effective and robust policy. We also need to choose what driving data to collect to expose the agent to a variety of good and bad states, and in particular show the agent how to get out of bad states where an accident is imminent. What data to collect is non-obvious: we want to explore a range of good and bad states so the agent learns a reasonable policy for how to act in all kinds of states. However, to make this feasible in the real world, we want to avoid exploring dangerous states. Finally, we need to carefully choose a way to collect human feedback that contains signal from which the agent can learn, but is not too hard to collect. We will discuss the choices we made for these three parts of the problem in the Loss Function, Driving Data, and Feedback sections respectively. Although we validate in simulation, specifically Unity, for the sake of ease, the algorithms we test could be easily trained in the real world. Learning from demonstration has mostly been studied in the supervised learning framework and the Markov Decision Process (MDP) framework. In the former, it is generally known as imitation learning (or behavioral cloning in the AV setting), and in the latter, it is generally known as apprenticeship learning. There are other relevant RL algorithms, but none for LfD in our problem setting, as we will show. The supervised learning in this area has focused on a least squares regression that maps from an input state to action. The easiest approach to take here is to have an expert perform an optimal demonstration, and then simply use that as training data. The main issue here is that the runtime and training distributions can be vastly different. That is, once the trained agent is acting on its own after training and encounters states it has not seen, it does not know how to act, and strays further from the intended behavior. This problem is known as the Covariate Shift and the widely accepted solution generally follow the approach laid out in DAgger BID10. DAgger allows the agent to explore and then uses the expert to label the new dataset, then training on all of the combined data. Such an approach has even been improved upon both to address scalar feeddback by incorporating experts that can label Q values with the AggreVaTeD algorithm BID9, and to and to address deep neural networks by finding a policy gradient with the Deeply AggreVaTeD algorithm BID15. However, these DAgger based policies require the agent to ex-plore and make mistakes. The MDP research on apprenticeship learning has largely been on inverse reinforcement learning, or IRL BID0, in which a reward function is estimated given the demonstration of an expert. However, often in this framework, the reward function is restricted to the class of linear combinations of the discrete features of the state, and the framework only allows for positive examples. In addition, there has been work on inverse reinforcement learning from failure , which allows for a sequence of positive or a sequence of negative examples. There is also distance minimization for reward learning from scored trajectories BID2, which allows for gradation in the scores, but does not allow for an arbitrary reward function on continuous inputs or labels for atomic actions as opposed to a trajectory or sequence of actions. Moreover, these IRL methods are not a candidate for our problem, since they require an exact MDP solution with a tractable transition function and exploration to find the optimal policy. The issue we have is not that we don't have the reward function, but that even with the more informative feedback, we cannot use exploration to learn the optimal policy. It is interesting to note that there are off-policy RL algorithms as well, but we would like to highlight that this is not the same thing as LfD. LfD, as we use it, means that we have collected all of our data before training. This could be thought of as on-policy only if our policy and start state never changes. Whereas, off-policy RL (e.g. Off-Policy Actor-Critic BID3, Q-Learning BID16, Retrace BID7) generally requires agents to have a non-zero probability of choosing any action in any state, in order for the algorithm to converge. (Moreover, it is the somewhat parenthetical opinion of at least one of authors of this paper that in the off-policy policy gradient RL framework, using importance sampling to calculate an expectation over a different action distribution is fine, but changing the objective function from an expectation over the learned policy state-visitation distribution to an expectation over the behavior (exploratory) state-visitation distribution, as in BID3, is an unsatisfactory answer that does not give the optimal policy for the agent when running on the learned policy, as we will want to do.) Normalized Actor Critic (NAC) does attempt to bridge the gap between off-policy and LfD, and works with bad as well as good demonstration, however, NAC does not allow for restricted exploration either, since it adds entropy to the objective function to encourage exploration BID4. NAC has also only been done for discrete action control, not continuous as we want to do. Finally, there are many RL algorithms that use human evaluative feedback, but none for LfD. One RL algorithm with human feedback of note is COACH BID6, which is based on the stochastic policy gradient. COACH is an on-policy RL algorithm that uses human-feedback to label the agent's actions while exploring. COACH's view on human feedback helps us to draw connections to RL, as we will discuss later. However, COACH was designed for on-policy exploration and uses discrete feedback values of 1 and -1, whereas we generalize to continuous values in [−1, 1]. We cannot use a stochastic policy gradient in a justified manner, since we do not explore with a stochastic policy. Out of all the LfD work in the AV context, the most notable has either been on behavioral cloning BID1 BID8 or using IRL to solve sub-tasks such as driving preferences that act on top of a safely functioning trajectory planner BID5. To the best of our knowledge, no research so far has focused on using any kind of evaluative scalar feedback provided by a human in the context of AV control with LfD. That is, no one has solved how to take states, actions, and continuous feedback with respect to those actions, and convert them into a control policy for an AV, without having the AV explore. We believe that this is a major oversight: many AV research groups are investing huge amounts of time into collecting driving data; if they used our model, they could improve performance simply by having an expert labeler sit in the car with the driver for no additional real time. That is, when f < 0, DISPLAYFORM0 3. The rate at which the loss is minimized should be determined by the magnitude of the feedback. That is, when f > 0, ∂|Loss| ∂f > 0, and when f < 0, ∂|Loss| ∂f < 0.These three properties together ensure that the network avoids the worst negative examples as much as possible, while seeking the best examples. Given an input state s, the first loss function that comes to mind is what we term "scalar loss": DISPLAYFORM1 This loss function is notable for several reasons. First, it is a generalization of mean squared error, the standard behavioral cloning loss function: DISPLAYFORM2 Mean squared error is a well-principled loss function if you assume Gaussian noise in your training data. That is, you assume the probability of your data can be given by Gaussian noise around some mean and you learn to predictθ as that mean. Given this assumption, you can derive MSE as the loss that produces a maximum likelihood estimate for your parameters. Let the parameters of the model be represented by p and probability be P r.θ is parameterized by p and will be used interchangeably withθ p, when clarity is needed. Please note that θ refers to the angle label, and not the model parameters: DISPLAYFORM3 Generally, log(√ 2πσ 2) is left out since it is a constant w.r.t. our parameters and so will go away when the gradient is taken, leaving: DISPLAYFORM4 2 is also left out, since it only acts to scale the gradient, and can be accounted for by adjusting the learning rate of gradient descent, leaving: DISPLAYFORM5 However, we note that, if we interpret |f |, the magnitude of our feedback, as 1 2σ 2, we can view |f | as a measure of certainty. This certainly applies to a Gaussian distribution with a variance of at least 1 2, since f ∈ for positive examples. For negative examples, we generalize further by removing the magnitude calculation, and allowing our feedback to be negative. This enforces that we minimize the probability of negative data: DISPLAYFORM6 To be able to easily recover behavioral cloning, we introduce two hyperparameters. The first such parameter is the ability to threshold feedback values. If we threshold, we simply replace every f with sign(f). Thresholding eliminates gradations in positive and negative data. Additionally, we introduced the parameter α, which scales down all our negative examples' feedback: f:= max(f, αf). This trades off between behavioral cloning and avoiding negative examples. We apply max(f, αf) after we threshold, so if we threshold with and set α to 0.0, we recover behavioral cloning. Our scalar loss is also notable since it closely resembles a loss that induces a stochastic policy gradient. In a standard RL policy network such as REINFORCE, the gradient would be ∇ p = ∇ p (Q π (θ) * −log(P r(θ))) BID17. The loss then, that would induce this gradient is Loss = Q π (θ) * −log(P r(θ)). R, the return, or a sample contributing to Q π (θ), is generally used. In continuous control, one could instead predict a meanθ for a normal distribution and then sample your action θ from that normal distribution. As demonstrated above, if you replace log(P r(θ)) with the probability density function for a normal distribution, the loss you wind up with is precisely MSE scaled by R. Substituting this scalar into the derivation above at every step, you get: DISPLAYFORM7 A full derivation of this loss given the stochastic policy gradient and Gaussian policy can be found in the appendix. Clearly if we view f θ, our feedback, as Rθ and assume a Gaussian policy, we get our scalar loss function. COACH in fact points out that you can view online feedback given for the current policy, fθ as a sample from Q π (θ), or A π (θ), the advantage function, and empirically verifies that this works for on-line RL with discrete feedback BID6. This similarity was useful for inspiration and ideation, but actually falls short of rigorous justification for two reasons. Since we are training off-policy, and, more specifically, on a pre-determined policy that does not explore stochastically and does not vary depending on the current predicted policy, we run into major issues justifying our loss this way. First, the main "off-policy" issue here is that for the stochastic policy gradient to hold, the exploration must be stochastic. However, in our case, the data is drawn from a pre-determined, deterministic policy. We can illuminate the intuition for why the stochastic policy gradient no longer works by considering a simple example. Consider the network attempting to learn the correct predictedθ for a given state. Consider that there is only one demonstrated θ, -1, with a feedback of -1. Now, no matter whatθ the network predicts, the θ action that is taken during training will always be -1, and the feedback will always be -1. Moreover, consider that the actual optimum isθ = 1, and the network is currently predictingθ = −2. Using our scalar loss, the network will increase distance between θ andθ by decreasingθ, making the policy worse and worse. This would not happen when using a stochastic RL policy, since the network can explore states around the current predictedθ by choosing appropriate actions. Using a stochastic RL policy, given enough samples on either side ofθ, the network will have larger and larger gradients, the more negativeθ is. But these gradients will not keeping "pushing" the prediction to the left of -1, but rather will randomly causeθ to move around, gradually moving to the right as it finds better feedback, and eventually converging at the optimum ofθ = 1. The network will move less "violently" and more "stably" the closerθ gets to 0. And when the network eventually reaches the positive numbers, it might get "pulled" to the left a bit when it happens to sample a worse action, but it will not get pulled as strongly as when it samples a better action to the right. We can now intuitively see the issue: the neural network cannot not influence the probability of seeing an example again, which can lead to problems with learning the policy. In RL, a policy network can try a bad action, and then move the policy away from that action and not revisit it. On the other hand, if we have a bad example in our training set for a given state, on every epoch of training, our neural net will encounter this example and take a step away from it, thus pushing our network as far away from it as possible. Taking these steps is not necessarily helpful since the network may not have favored taking the bad action before. We could use some sort of importance sampling BID14 BID3, as is done with stochastic off-policy exploration, to scale down the loss for examples we are far away from. However, this would make our update have almost no effect when we are far away from positive examples, and with the deterministic exploration of ReNeg, the distinction between positive and negative examples now matters. We can't have it both ways just by multiplying by the probability of θ given our model. (This happens since the Gaussian PDF decreases exponentially with difference |θ −θ|, but the loss only increases quadratically with the difference, due to the logarithm. Thus the gradient tends toward 0 due to the differing rates of growth, as the difference gets large.) Moreover, importance sampling consistently reduced performance for learning from demonstration in the NAC paper BID4.Another, perhaps less significant issue, is that our feedback represents Q *, and not Qπ. Even if the human critic could re-assigned labels as the agent trains, he/she would have no way to sample the return from Qπ, without letting the agent explore freely. This is significant because an action that is optimal for the expert may be a dangerous and poor decision for a non-optimal policy. This is perhaps less significant than the other issue, since there are no actions in our data that could be considered both good and risky. What is good for one policy (i.e. steering back to the middle of the road) is generally good for all policies, and so can act rather greedily with respect to Q *, even though it will not always be following the optimal policy. For these reasons, we find the comparison to stochastic policy gradients useful, but ultimately uncompelling. Applying RL losses to supervised learning does not provide the mathematical justification we need. The stochastic policy gradient is no longer computing the gradient of the current policy at all. Thus, we focus primarily on extending and generalizing MSE. Yet, we can still learn from the policy gradient comparison. In particular, we acknowledge that the sign of our feedback is far more significant than it was in the RL context (which is one of the reasons we introduce the α parameter, which scales down the importance of negative examples, in the event that we collect too many negative examples). In RL, using the stochastic policy gradient, it did not matter if negative examples "pulled"θ closer to them, so long as the positive examples pulledθ more, since you would eventually try one of those actions, if it is a nearby optimum. However, since actions are no longer sampled dependent on the current policy, suddenly the sign matters very much. In fact, even if we have a positive example for a state, if we have more negative examples than positive examples, we may wind up ignoring our positive examples entirely in an effort to get away from our negative examples. This case highlights the trouble inherent in using negative examples: It is hard to know how and when to take into account the negative examples. FIG1, if we perform a regression on positive and negative examples with more negative examples than positive in one state, we may wind up in a case where our loss is minimized by a prediction of positive or negative infinity, and thus our regression is "overwhelmed" by the negative examples. This led us to our second "exponential" loss: DISPLAYFORM8 Using this loss, negative examples will have infinite loss at distance 0, and then drop off exponentially with distance. We hope that this will create regressions more akin to the second image in FIG1. In this image, adding more negative points will still nudge the regression away more and more, but one positive point not too close by should be enough to prevent it from diverging to positive or negative infinity. It should be noted that the loss in a particular state still could only have negative examples, especially in a continuous state-space like ours where states are unlikely to be revisited. However, the reduction in loss caused by diverging to infinity would be so small that it should not happen simply due to continuity with nearby states enforced by the structure of the network. In addition, one concern with this loss could be that for positive fractional differences, and negative non-fractional differences, the desired property 3) of loss functions no longer holds. That is, our positive loss will not grow with f if the difference being exponentiated is a fraction. And for negative exponents, the loss will only grow if the difference is a fraction that shrinks as it is raised to increasing powers of f. However, we hope that for negative examples, distances that are more than 1 unit away will not occur often (since 1 unit is half the distance range). We discuss a potential future solution in the appendix to patch this loss function. Our final loss function should produce regressions more like the final image in FIG1: We propose directly modelling the feedback with another neural network (which we call the FNet) for use as a loss function for our PNet. If this FNet is correctly able to learn to copy how we label data with feedback, it could be used as a loss function for regression with our PNet. Thus, in order to maximize feedback, our loss function would be as follows: DISPLAYFORM9 After learning this FNet, we can either use it as a loss function to train a policy network or, every time we want to run inference, we can run a computationally expensive gradient descent optimization to pick the best action. Because the latter does not depend on the training distribution (so we do not have the issue of the runtime and training distributions being different), and it is more efficient, we choose an even easier version of the latter: we pick the best action out of a list of discrete options according to the FNet's predictions. One feature of the FNet is that adding more negative points will not "push" our regression further away from this point, but rather just make our FNet more confident of the negative feedback there. This may not be the desired effect for all applications. Moreover, the FNet cannot operate on purely positive points with no gradation. That is, behavioral cloning cannot be recovered from it. We recorded 20 minutes of optimal driving and labeled all of this data with a feedback of 1.0. Choosing the suboptimal data and how to label it was a bit more tricky. One reason that we are not using reinforcement learning is that letting the car explore actions is dangerous. In this vein, we wanted to collect data only on "safe" driving. However, the neural network needs data that will teach it about bad actions well as good actions that recover the car from bad states. In order to explore these types of states and actions, we collected two types of bad driving: "swerving" and "lane changing". The first image in FIG2 is swerving. In swerving, the car was driving in a "sine wave" pattern on either side of the road. We collected 10 minutes of this data on the right side of the road and 10 minutes on the left. The second image is lane changing. For this, we drove to the right side of the road, straightened out, stayed there, and then returned to the middle. We repeated this for 10 minutes, and then collected 10 minutes on the left-hand side as well. Backseat Driver is the framework we use to compute and collect feedback in the AV context. Our feedback includes much more information than just a reward (as is used in RL): we take our label to directly measure how "good" an action is relative to other possible actions. We use this approach instead of labeling rewards for actions both because we found it an easy way to label data with feedback, and because it contains more signal. How exactly to label the data with feedback, however, is non-obvious. At first, we considered labeling using a slider from -1 to 1. However, using a slider can be non-intuitive in many cases and there would be discontinuities in feedback you would want to give. For example, if the demonstrator is driving straight off the road and then starts to turn left back onto the road, there would be a large discontinuity in the very negative and then slightly positive feedback. In order to circumvent these issues, and to make the labeling process more intuitive, we decided to collect feedback using the steering wheel. We found that it is easier for people to focus on the steering angle, since that is how we are used to controlling cars. Our first thought was to just turn the steering wheel to the correct angle. However, this is very difficult to estimate, especially on turns, when you cannot see the actual effects your steering is having on the car. (Note that if we did this, our algorithm would turn into behavioral cloning.) Instead, we decided to label the differential. That is, we turned the wheel to the left if the car should steer more left. This signal shows where the error is (i.e. "You should be turning more" or "You're turning the wrong way"). Note that the label does not need to be the exact correct angle; it just needs to show in which direction the current action is erring, and proportionally how much. We call this method of human feedback collection "Backseat Driver." In order to process the angle labels into a feedback value in [−1, 1], we used the equation below: DISPLAYFORM0 Note: We first normalize all of our corrections by dividing by the greatest collected correction c, so all of our c values fall in [−1, 1]. In line 1 above, if we are turning the steering wheel in the same direction as the car (with some of error), then the feedback should be positive. (We set epsilon to 5 θmax so that it allows up to 5 degrees of tolerance.) Since c represents a delta in steering, a greater delta should in a less positive signal. Therefore, the feedback should be proportional to −|c|.We add 1 to ensure all c in the same direction as θ are positive. If c is in a different direction than we were steering (line 3), then the feedback should be negative, so we just return −|c| as the feedback. Thus the greater the delta, the more negative the feedback will be. If c is in the same direction, on the other hand, we chose to scale these feedbacks up so that the feedback is positive, but less positive for a greater differential. This makes sense since if, for example, the car is steering left and we tell it to steer more left, this is not as bad as if the car is steering the wrong way. Thus, slow actions back to the center of the road will be rewarded less than quick actions back to the center of the road. We chose to use only an hour of data because we wanted to see how far we could get with limited data. While our feedback is relatively easy to collect compared to other options, it still takes up human hours, so we would like to limit its necessity. We sampled states at a rate of approximately two frames per second, since states that are close in time tend to look very similar. We augmented our data by flipping it left-right, inverting the angle label, and leaving the feedback the same. After this augmentation, we had 17,918 training images and 3,162 validation images (a 85:15 split).We chose to learn our policy with an end-to-end optimization of a neural network to approximate a continuous control function. Such networks are capable of tackling a wide array of general problems and end-to-end learning has the potential to better optimize all aspects of the pipeline for the task at hand. Given the option to use separate (differentiable) modules for sub-tasks such as computer vision, or to connect these modules in a way that is differentiable for the specific task, the latter will always perform better, since the end-to-end model always has the ability to simply not update the sub-module if it will not help training loss. We used transfer learning to help bootstrap learning with limited data for both the PNet and Fnet. We decided to start with a pretrained Inception v3 since it has relatively few parameters, but has been trained to have strong performance on ImageNet, giving us a head start on any related computer vision task. We kept some early layers of Inception and added several of our own, followed by a tanh activation for the output. We tried splitting the network at various layers and found that one about halfway through the network (called mixed 2) worked best. The layers we added after the split were fully connected layers of sizes 100, 300, and 20. For the FNet, the angle input is concatenated onto the first fully-connected layer. (Find an architecture diagram in the appendix.) We tested our trained models by running them in our Unity simulator and recording the time until the car crashed or all four tires left the road. (Note: We designed the ReNeg framework so it need not be run in a simulator. This would work just as well in the real world. We only used a simulator because it was all we had access to.) We first tested our PNet with the scalar loss, our PNet with the exponential loss, and our FNet. We plotted their mean times over the eight runs with standard deviation. See Figure 3 below. (All of these were on the default hyperparameters listed in the appendix, except for the exponential loss model, for which we set α to 0.1 since otherwise we could not get it to converge.) Based on the predicted angles, the FNet seemed to primarily predict feedback based on state, not angle; this makes sense given that the feedback in "bad" states is generally "bad", except for the split second when the "good" action takes place. Average Time Lasted (Given that the scalar loss performed best (and was training correctly), we spent more time tuning the hyperparameters for this model. This can be seen in Figure 4. We note several interesting things from figure 4 exploring the scalar loss. First, the pink group is identical to the solid blue group except that all the α values are 0, meaning that negative-feedback examples are zeroed out and effectively ignored in training. The blue bars are much higher than their pink counterparts, indicating that the negative data is useful. Second, we can see that thresholding the feedback to -1 and 1 (the blue crosshatch pattern) increased the scalar performance to about 30 seconds (compared to 6 seconds with the default hyperparameters). At first this could be taken to indicate that having gradations in positive and negative data could be harmful to training. However, we see that the same performance is achieved (and surpassed) by increasing the learning rate instead of thresholding, by looking to the 2 rightmost blue bars. The reason for this is likely that we tuned the learning rate to work well on thresholded data, and so, when we don't threshold/clone our data, the scalars on the loss drop significantly, forcing the network to take smaller steps, and effectively decreasing the learning rate. Increasing the learning rate instead of thresholding yields much better performance, indicating that gradations in the data (with a high learning rate) do help training. Note, α can in fact be in [0, ∞], however we focused on α = 0 and α = 1, since this corresponds to no negative examples and an equal weighting between positive and negative examples. It it difficult to tune independently due its relationship with the learning rate. Moreover, all we are trying to show is that using negative examples, with some relative importance to the positive examples, can be beneficial. After selecting the best learning rate for each model, we then trained new versions of each network 2 more times, for a total of 3 models each, to account for stochasticity in SGD. (The best learning rate for both turned out to be 1e-5; see the appendix for more details on tuning.) Each time, we let the model drive the car for 8 trials and calculated the performance as the mean time before crashing over these 8 trials. We then calculated the mean performance for each over the 3 training sessions. FIG4 shows the . We hypothesized that for the task of learning lane following for autonomous vehicles from demonstration, adding in negative examples would improve model performance. Our scalar loss model performed over 1.5 times as well as the behavioral cloning baseline, showing our hypothesis to be true. The specific method of regression with negative examples we used allows for learning deterministic continuous control problems from demonstration from any range of good and bad behavior. Moreover, the loss function that empirically worked the best in this domain does not require an additional neural network to model it, and it induces a stochastic policy gradient that could be used for fine-tuning with RL. We also introduced a novel way of collecting continuous human feedback for autonomous vehicles intuitively and efficiently, called Backseat Driver. We thus believe our work could be extremely useful in the autonomous control industry: with no additional real world time, we can increase performance over supervised learning by simply having a backseat driver.6 APPENDIX 6.1 EXAMPLE INPUT The architecture of our model, based on Inception v3. After the split, the branch on the right is the branch used for our PNet or FNet, while the left branch is ignored. The new layers we added were 3 fully connected layers of sizes 100, 300, and 20. The final activation is Tanh to ensure the output range of (-1,1). For the FNet, the angle input is concatenated onto the first fully-connected layer. Our batch size was 100 and we trained for 5 epochs. Unless otherwise specified for a given model in the experiments, we used an α value of 1.0, we did not threshold, and we used a learning rate of 1e-6. As in the Inception model we were using, our input was bilinearly sampled to match the resolution 299x299. Likewise, we subtracted off an assumed mean of 256.0/2.0 and divided by an assumed standard deviation of 256.0/2.0. During training, we kept track of two validation metrics: the loss for the model being trained, and the average absolute error on just the positive data multiplied by 50. The first we refer to as "loss" and the second we refer to as "cloning error" (since it is the 50 times the square root of the cloning error) or just "error". The reason we multiplied by 50 is that this is how Unity converts the -1 to 1 number to a steering angle, so the error is the average angle our model is off by on the positive data.(This is true with the maximum angle set to 50.)During training, these two metrics generally behaved very similarly, however, in the models for which we increased the learning rate, these eventually start to diverge. In this case, the error on the positive data started to increase, but the loss was still decreasing. For this reason, we tried varying the learning rate on several models, to see if the loss was more important than the "cloning" error. It is clear that the behavioral cloning models (thresholded with α = 0.0) should in general do better on the "cloning" error, since they are very closely related. Whereas for non-thresholded data, it was trained with examples weighted differently. And for the negative data, it was trained to also get "away" from negative examples. We hope that even though the cloning error may increase, this means that it is because the model is choosing something better than (yet further away from) the positive examples. We still use the cloning error, however, because it is a useful intuitive metric for training and comparison. We tried several learning rates for both behavioral cloning and ReNeg. We compared the performance, shown in figures 6 and 7, and found that 1e-5 worked best for both. Average Time Lasted (sec) 0 50 100 150 5e-6 1e-5 1.5e-5 2e-5 Figure 6: The scalar loss performed best with a learning rate of 1e-5. Here is the derivation from the stochastic policy gradient to the loss that induces it, which is very similar to our scalar loss: ∇Loss = ∇(R * −log(P r(θ))) DISPLAYFORM0 DISPLAYFORM1 Future research involving human feedback should focus on 2 things: the loss function and enforcing continuity. These will be briefly explored in the next two sections. Note, before we discuss potential extensions of this work, that fine-tuning the policy once it is acceptably safe is a separate but also interesting problem. Supervised approaches involving a safety driver taking over control, and retraining on this data a la DAgger, should probably be explored. Additionally, instead of aggregating the new data with the old, active learning approaches could be explored, where the model is not entirely retrained. we point the reader to BID8 for an AV application of DAgger. Immediate next steps should likely focus on alterations to the loss function. Here we introduce a fourth desired property that can ensure our negative examples have less or the same "impact" as they get farther away, and positive examples have more of an impact as they get farther away. In other words, for positive examples, as D increases, the update (derivative) is always the same or greater in magnitude, and for negative examples, the same is true as D decreases. The loss is concave up with respect to D i.e. We propose two loss functions that meet properties 1-4:1. We can accomplish this exponential decay by modifying our scalar function in a very easy way: Move the "sign" of f into the exponent:Loss inverse = |f | * (θ(s) −θ(s)) 2 * sign(f)Using this loss function we have all three properties satisfied. That is, positive examples encourage moving towards them, negative examples encourage moving away from them, and the amount of this movement increases with the magnitude of f. Moreover, we also have the property that, in negative examples, loss drops off exponentially with the distance from the negative example (because we are dividing by it).2. If we want our scalar loss function to have neither an exponential decay nor an exponential increase with the distance from the negative points, we can simply use the following loss: DISPLAYFORM0 This has the not-so-nice property that, in the positive example, it allows outliers much more easily than the traditional squared loss. However, it has the very nice property that, given a single state input, as long as you have more positive examples than negative examples, your loss will always be minimized in that state by a value between your positive examples. This is because, as soon as you get to your greatest or least positive example, every step away from your positive examples will cost you 1 loss, for each positive example you have, and you will only lose 1 loss for each negative example you have. (Note, if you are not thresholding, then this translates to more total |f | for positive examples than negative examples.) 6.7.2 CONTINUITY Because in both our scalar and exponential loss, our loss function at a given state with just a negative example is minimized by moving away from the negative example, our regression in that state will tend toward positive or negative infinity. Certainly having a cost on negative examples that drops of exponentially will help, but it may not be enough. Moreover, we may not want to rely on the structure of neural networks to discourage this discontinuity. Therefore, research could be done on adding a regularization term to the loss that penalizes discontinuity. That is, we would add some small loss based on how dissimilar the answers for nearby states are. Of course, this implies a distance metric over states, but using consecutive frames may suffice.

We introduce a novel framework for learning from demonstration that uses continuous human feedback; we evaluate this framework on continuous control for autonomous vehicles.

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the scientific article to find a short TLDR

Question: We explore the use of Vector Quantized Variational AutoEncoder (VQ-VAE) models for large scale image generation. To this end, we scale and enhance the autoregressive priors used in VQ-VAE to generate synthetic samples of much higher coherence and fidelity than possible before. We use simple feed-forward encoder and decoder networks, thus our model is an attractive candidate for applications where the encoding and decoding speed is critical. Additionally, this allows us to only sample autoregressively in the compressed latent space, which is an order of magnitude faster than sampling in the pixel space, especially for large images. We demonstrate that a multi-scale hierarchical organization of VQ-VAE, augmented with powerful priors over the latent codes, is able to generate samples with quality that rivals that of state of the art Generative Adversarial Networks on multifaceted datasets such as ImageNet, while not suffering from GAN's known shortcomings such as mode collapse and lack of diversity. Deep generative models have significantly improved in the past few years [1; 18; 17]. This is, in part, thanks to architectural innovations as well as computation advances that allows training them at larger scale in both amount of data and model size. The samples generated from these models are hard to distinguish from real data without close inspection, and their applications range from super resolution BID14 to domain editing BID31, artistic manipulation BID23, or text-to-speech and music generation BID16.We distinguish two main types of generative models: maximum likelihood based models, which include VAEs [11; 21], flow based [5; 20; 6; 12] and autoregressive models [14; 27]; and implicit generative models such as Generative Adversarial Networks (GANs) BID7. Each of these models offer several trade-offs such as sample quality, diversity, speed, etc. GANs optimize a minimax objective with a generator neural network producing images by mapping random noise onto an image, and a discriminator defining the generators' loss function by classifying its samples as real or fake. Larger scale GAN models can now generate high-quality and highresolution images [1; 10]. However, it is well known that samples from these models do not fully capture the diversity of the true distribution. Furthermore, GANs are challenging to evaluate, and a satisfactory generalization measure on a test set to assess overfitting does not yet exist. For model comparison and selection, researchers have used image samples or proxy measures of image quality such as Inception Score (IS) BID22 and Fréchet Inception Distance (FID) BID8.In contrast, likelihood based methods optimize negative log-likelihood (NLL) of the training data. This objective allows model-comparison and measuring generalization to unseen data. Additionally, since the probability that the model assigns to all examples in the training set is maximized, likelihood based models, in principle, cover all modes of the data, and do not suffer from the problems of mode collapse and lack of diversity as seen in GANs. In spite of these advantages, directly maximizing likelihood in the pixel space can be challenging. First, NLL in pixel space is not always a good measure of sample quality BID24, and cannot reliably be used to make comparison between different model classes. There is no intrinsic incentive for these models to focus on, for example, global structure. Some of these issues are alleviated by introducing inductive biases such as multi-scale [26; 27; 19; 16] or by modeling the dominant bit planes in an image [13; 12].In this paper we use ideas from lossy compression to relieve the generative model from modeling negligible information. Indeed, techniques such as JPEG BID30 have shown that it is often possible to remove more than 80% of the data without noticeably changing the perceived image quality. As proposed by BID28, we compress images into a discrete latent space by vector-quantizing intermediate representations of an autoencoder. These representations are over 30x smaller than the original image, but still allow the decoder to reconstruct the images with little distortion. The prior over these discrete representations can be modeled with a state of the art PixelCNN [27; 28] with selfattention BID29, called PixelSnail BID2. When sampling from this prior, the decoded images also exhibit the same high quality and coherence of the reconstructions (see FIG0). Furthermore, the training and sampling of this generative model over the discrete latent space is also 30x faster than when directly applied to the pixels, allowing us to train on much higher resolution images. Finally, the encoder and decoder used in this work retains the simplicity and speed of the original VQ-VAE, which means that the proposed method is an attractive solution for situations in which fast, low-overhead encoding and decoding of large images are required. The VQ-VAE model BID28 can be thought of as a communication system. It consists of an encoder that maps observations onto a sequence of discrete latent variables, and a decoder that reconstructs the observations from the discrete code. Both use a shared codebook. The encoder is a non-linear mapping from the input space, x, to a vector in an embedding space, E(x). The ing vector is then quantized based on its distance to the prototype vectors in the codebook c k, k ∈ 1... K such that each vector E(x) is replaced by the index of the nearest prototype vector in the codebook and is transmitted to the decoder (note that this process can be lossy). The decoder maps back the received indices to their corresponding vectors in the codebook, from which it reconstructs the data via a non-linear function. To learn these mappings, the gradient of the reconstruction error is then back-propagated to the decoder, and to the encoder using the straight-through gradient estimator. The VQ-VAE model incorporates two additional terms in its objective to align the vector space of the codebook with the output of the encoder. The codebook loss, which only applies to the codebook variables, brings the selected codebook c close to the output of the encoder, E(x). The commitment loss, which only applies to the encoder weights, encourages the output of the encoder to stay close to the chosen codebook vector to prevent it from fluctuating too frequently from one code vector to another. The overall objective is described in equation 1, where c is the quantized code for the training example x, E is the encoder function and D is the decoder function. The operator sg refers to a stop-gradient operation that blocks gradients from flowing into its argument, and β is a hyperparameter which controls the reluctance to change the code corresponding to the encoder output. DISPLAYFORM0 As proposed in BID28, for the codebook loss (the second term in equation 1) we use the exponential moving average updates for the codebook, as a replacement for the second loss term in Equation 1: DISPLAYFORM1 where DISPLAYFORM2 i is the number of vectors in E(x) in the mini-batch that will be quantized to codebook item e i and γ is a decay parameter with a value between 0 and 1. We used the default γ = 0.99 in all our experiments. We use the released VQ-VAE implementation in the Sonnet library 1 2. The proposed method follows a two-stage approach: first, we train a hierarchical VQ-VAE (see FIG1) to encode images onto a discrete latent space, and then we fit a powerful PixelCNN prior over the discrete latent space induced by all the data. The input to the model is a 256 × 256 image that is compressed to quantized latent maps of size 64 × 64 and 32 × 32 for the bottom and top levels, respectively. The decoder reconstructs the image from the two latent maps. As opposed to vanilla VQ-VAE, in this work we use a hierarchy of vector quantized codes to model large images. The main motivation behind this is to model local information, such as texture, separately from structural global information such as shape and geometry of objects. The prior model over each level can thus be tailored to capture the specific correlations that exist in that level. More specifically, the prior over the latent map responsible for structural global information, which we refer to as the top prior (see FIG1), can benefit from a larger receptive field of multi-headed selfattention layers to capture correlations in spatial locations that are far apart in the image. In contrast, the conditional prior model, referred to as the bottom prior, over latents that encode local information must have much larger resolution. As such, using as many self-attention layers as in the top-level prior is neither necessary nor practical due to memory constraints. For the prior over local information, we thus find that using a larger conditioning stack (coming from the global information code) yields more significant improvements. The hierarchical factorization also allows us to train larger models: we train each prior separately, thereby leveraging all the available compute and memory on hardware accelerators for each prior. DISPLAYFORM0 The structure of our multi-scale hierarchical encoder is illustrated in FIG1. We note that if dependencies between latent maps are such that they are strictly a compressed version of the quantized latent maps they depend on, then they would encode only redundant information that already exists in the preceding latent maps. We therefore allow each level in the hierarchy to separately depend on pixels, which encourages encoding complementary information in each latent map that can contribute to reducing the reconstruction error in the decoder. For 256×256 images, we use a two level latent hierarchy. As depicted in FIG1, the encoder network first transforms and downsamples the image by a factor of 4 to a 64 × 64 representation which is quantized to our bottom level latent map. Another stack of residual blocks then further scales down the representations by a factor of two, yielding a top-level 32 × 32 latent map after quantization. In order to further compress the image, and to be able to sample from the model learned during stage 1, we learn a prior over the latent codes. Fitting prior distributions using neural networks from training data has become common practice, as it can significantly improve the performance of latent variable models BID1. This procedure also reduces the gap between the marginal posterior and the prior. Thus, latent variables sampled from the learned prior at test time are close to what the decoder network has observed during training which in more coherent outputs. From an information theoretic point of view, the process of fitting a prior to the learned posterior can be considered as lossless compression of the latent space by re-encoding the latent variables with a distribution that is a better approximation of their true distribution, and thus in bit rates closer to Shannon's entropy. Therefore the lower the gap between the true entropy and the negative log-likelihood of the learned prior, the more realistic image samples one can expect from decoding the latent samples. In the VQ-VAE framework, this auxiliary prior is modeled with a powerful, autoregressive neural network such as PixelCNN in a post-hoc second stage. More specifically, we use self-attention layers, interspersed with masked convolution blocks as proposed by BID2, to model each level of the latent hierarchy as shown in FIG3. The top-level uses an unconditional network, and the downstream latent layers are modeled using a conditional stack that transforms the latent dependencies into spatial conditioning representations. Both codes are modeled with a PixelCNN, the first is conditioned on the class label, the second stage PixelCNN is conditioned on the class label and the generated codes of the first. The decoder is feed-forward, so producing the image given the two latent codes is very fast. (The example image with a parrot is generated with this model).Our top-level prior network models 32 × 32 latent variables. The residual gated convolution layers of PixelCNN are interspersed with causal multi-headed attention every five layers. To regularize the model, we incorporate dropout after each residual block as well as dropout on the logits of each attention matrix. We found that adding deep residual networks consisting of 1 × 1 convolutions on top of the PixelSnail stack further improves likelihood without slowing down training or increasing memory footprint too much. Our bottom-level conditional prior operates on latents with 64×64 spatial dimension. This is significantly more expensive in terms of required memory and computation cost. Fortunately, as described in Sect. 3.1, the information encoded in this level of the hierarchy mostly corresponds to local features, which do not require large receptive fields as they are conditioned on the top-level prior. Therefore, we use a less powerful network with no attention layers. We also found that using a deep residual conditioning stack significantly helps at this level. The foundation of our work is the VQ-VAE framework of BID28. Our prior network is based on Gated PixelCNN BID27 augmented with self-attention BID29, as proposed in BID2. BigGAN BID0 is currently state-of-the-art in FID and Inception scores, and produces high quality high-resolution images. The improvements in BigGAN were due to incorporating architectural advances such as self-attention, better stabilization methods, scaling up the model on TPUs and a mechanism to trade-off sample diversity with sample quality. In our work we also investigated how the addition of some of these elements, in particular self-attention and compute scale, improve the quality of samples of VQ-VAE models. Recent attempts to generate high resolution images with likelihood based models include Subscale Pixel Networks of BID15. Similar to the parallel multi-scale model introduced in BID18, SPN imposes a partitioning on the spatial dimensions, but unlike BID18 SPN does not make the corresponding independence assumptions, whereby it trades sampling speed with density estimation performance and sample quality. Hierarchical latent variables have been proposed in e.g. BID20. Specifically for VQ-VAE, BID3 uses a hierarchy of latent codes for modeling and generating music using a WaveNet decoder. The specifics of the encoding is however different from ours: in our work the higher levels of hierarchy do not exclusively refine the information encoded in the lower levels, but they extract complementary information at each level, as discussed in Sect. 3.1. Because we are using simple, feed-forward decoders and optimizing mean squared error in the pixels, our model does not suffer from, and thus needs no mitigation for, the hierarchy collapse problems detailed in BID3. Concurrent to our work, BID6 extends BID3 for generating high-resolution images. The primary difference to our work is the use of autoregressive decoders in the pixel space. In contrast, for reasons detailed in Sect. 3, we use autoregressive models exclusively as priors in the compressed latent space. Additionally, the same differences with BID3 outlined above also exists between our method and BID6. Objective evaluation and comparison of generative models, specially across model families, remains a challenge BID24. Current image generation models trade-off sample quality and diversity (or precision vs recall BID21). In this section, we present qualitative of our model trained on ImageNet 256×256. The samples look sharp and diverse across several representative classes as can be seen in the class conditional samples provided in Fig. 6. For comparing diversity, we provide samples from our model juxtaposed with those of BigGAN-deep BID0, the state of the art GAN model 3 in Fig. 5. As can be seen in these side-by-side comparisons, VQ-VAE is able to provide samples of comparable fidelity yet with much higher diversity. As mentioned previously, an important advantage of likelihood based models is that it allows assessing overfitting by comparing NLL values between training and validation sets. The NLL values reported in Table 1 We propose a simple method for generating diverse high resolution images using VQ-VAE, combining a vector quantized neural representation learning technique inspired by ideas from lossy compression with powerful autoregressive models as priors. Our encoder and decoder architectures are kept simple and light-weight as in the original VQ-VAE, with the only difference that we propose using hierarchical multi-scale latent maps for larger images. The improvements seen in the quality of the samples are largely due to the architectural advances in the PixelCNN style priors that more accurately estimate the distribution over the latent space. In particular, using self-attention seems to be a crucial component for accurately capturing the structure and geometry of objects encoded in the top-level latent map. We also observe that the quality of our samples is correlated with the improvements in the negative log-likelihood of the model in the latent space, where small gains in likelihood often translate to dramatic improvements in sample quality. The fidelity of our best class conditional samples are competitive with the state of the art Generative Adversarial Networks, while we see dramatically broader diversity in several classes, contrasting our method against the known limitations of GANs. We believe our experiments vindicate maximum likelihood in the latent space as a simple and effective objective for learning large scale generative models that do not suffer from the shortcomings of adversarial training. We here present additional samples from our model trained on ImageNet. All these samples are taken without any cherry-picking.

scale and enhance VQ-VAE with powerful priors to generate near realistic images.

Represent the TLDR for finding the corresponding scientific article.

In this paper we introduce a training method, called look-up table quantization (LUT-Q), which learns a dictionary and assigns each weight to one of the dictionary's values


Question: We present a graph neural network assisted Monte Carlo Tree Search approach for the classical traveling salesman problem (TSP). We adopt a greedy algorithm framework to construct the optimal solution to TSP by adding the nodes successively. A graph neural network (GNN) is trained to capture the local and global graph structure and give the prior probability of selecting each vertex every step. The prior probability provides a heuristics for MCTS, and the MCTS output is an improved probability for selecting the successive vertex, as it is the feedback information by fusing the prior with the scouting procedure. Experimental on TSP up to 100 nodes demonstrate that the proposed method obtains shorter tours than other learning-based methods. Traveling Salesman Problem (TSP) is a classical combinatorial optimization problem and has many practical applications in real life, such as planning, manufacturing, genetics (b). The goal of TSP is to find the shortest route that visits each city once and ends in the origin city, which is well-known as an NP-hard problem . In the literature, approximation algorithms were proposed to solve TSP . In particular, many heuristic search algorithms were made to find a satisfactory solution within a reasonable time. However, the performance of heuristic algorithms depends on handcrafted heuristics to guide the search procedure to find competitive tours efficiently, and the design of heuristics usually requires substantial expertise of the problem . Recent advances in deep learning provide a powerful way of learning effective representations from data, leading to breakthroughs in many fields such as speech recognition . Efforts of the deep learning approach to tackling TSP has been made under the supervised learning and reinforcement learning frameworks. Vinyals et al. introduced a pointer network based on the Recurrent Neural Network (RNN) to model the stochastic policy that assigns high probabilities to short tours given an input set of coordinates of vertices. Dai et al. tackled the difficulty of designing heuristics by Deep Q-Network (DQN) based on structure2vec (b), and a TSP solution was constructed incrementally by the learned greedy policy. Most recently, Kool et al. used Transformer-Pointer Network to learn heuristics efficiently and got close to the optimal TSP solution for up to 100 vertices. These efforts made it possible to solve TSP by an end-to-end heuristic algorithm without special expert skills and complicated feature design. In this paper, we present a new approach to solving TSP. Our approach combines the deep neural network with the Monte Carlo Tree Search (MCTS), so that takes advantage of the powerful feature representation and scouting exploration. A graph neural network (GNN) is trained to capture the local and global graph structure and predict the prior probability, for each vertex, of whether this vertex belongs to the partial tour. Besides node features, we integrate edge information into each update-layer in order to extract features efficiently from the problem whose solution relies on the edge weight. Similar to above-learned heuristic approaches, we could greedily select the vertex according to the biggest prior probability and yet the algorithm may fall into the local optimum because the algorithm has only one shot to compute the optimal tour and never goes back and reverses the decision. To overcome this problem, we introduce a graph neural network assisted Monte Carlo Tree Search (GNN-MCTS) to make the decision more reliable by a large number of scouting simulations. The trained GNN is used to guide the MCTS procedure that effectively reduces the complexity of the search space and MCTS provides a more reliable policy to avoid stuck in a local optimum. Experimental on TSP up to 100 vertices demonstrate that the proposed method obtains shorter tours than other learning-based methods. The remainder of the paper is organized as follows: After reviewing related work in Section 2, we briefly give a preliminary introduction to TSP in Section 3. Our approach is formulated in Section 4. Experimental are given in Section 5, followed by the in Section 6. The TSP is a well studied combinatorial optimization problem, and many learning-based algorithms have been proposed. In 1985, Hopfield et al. proposed a neural network to solve the TSP . This is the first time that researchers attempted to use neural networks to solve combinatorial optimization problems. Since the impressive produced by this approach, many researchers have made efforts on improving the performance . Many shallow network architectures were also proposed to solve the combinatorial optimization problem (; ; ;). Recent years, deep neural networks have been adopted to solve the TSP and many works have achieved remarkable . We summarize the existing learning-base methods from the following aspects. Vinylas et al. proposed a neural architecture called Pointer Net (Ptr-Net) to learn the conditional probability of a tour using a mechanism of the neural attention. Instead of using attention to blend hidden units of an encoder to a context vector, they used attention as pointers to the input vertices. The parameters of the model are learned by maximizing the conditional probabilities for the training examples in a supervised way. Upon test time, they used a beam search procedure to find the best possible tour. Two flaws exist in the method. First, Ptr-Net can only be applied to solve problems of a small scale (n ≤ 50). Second, the beam search procedure might generate invalid routes. Bello et al. proposed a framework to tackle TSP using neural networks and reinforcement learning. Similar to Vinylas et al., they employed the approach of Ptr-Net as a policy model to learn a stochastic policy over tours. Furthermore, they masked the visited vertices to avoid deriving invalid routes and added a glimpse which aggregates different parts of the input sequence to improve the performance. Instead of training the model in a supervised way, they introduced an Actor-Critic algorithm to learn the parameters of the model and empirically demonstrated that the generalization is better compared to optimizing a supervised mapping of labeled data. The algorithm significantly outperformed the supervised learning approach with up to 100 vertices. Kool et al. introduced an efficient model and training method for TSP and other routing problems. Compared to , they removed the influence on the input order of the vertices by replacing recurrence (LSTMs) with attention layers. The model can include valuable information about the vertices by multi-head attention mechanism which plays an important role in the setting where decisions relate directly to the vertices in a graph. Similar to , they applied a reinforcement learning method to train the model. Instead of learning a value function as a baseline, they introduced a greedy rollout policy to generate baseline and empirically showed that the greedy rollout baseline can improve the quality and convergence speed for the approach. They improved the state-of-art performance among 20, 50, and 100 vertices. Independent of the work of Kool et al., Deudon et al. also proposed a framework which uses attention layers and reinforcement learning algorithm (Actor-Critic) to learn a stochastic policy. Guided Tree Search Neural Network Choose Best Vertex Figure 1: Approach overview. First, the graph is fed into the graph neural network, which captures global and local graph structure and generates a prior probability that indicates how likely each vertex is in the tour sequence. Then, with the help of the graph neural network, a developed MCTS outputs an improved probability by scouting simulations. Lastly, we visit the best vertex among unvisited vertices according to the improved probability. The above process will loop until all vertices are visited. They combined the machine learning methods with an existing heuristic algorithm, i.e., 2-opt to enhance the performance of the framework. Dai et al. proposed a framework, which combines reinforcement learning with graph embedding neural network, to construct solutions incrementally for TSP and other combinatorial optimization problems. Instead of using a separate encoder and decoder, they introduced a graph embedding network based on the structure2vec (a) to capture the current state of the solution and the structure of a graph. Furthermore, they used Q-learning parameterized by the graph embedding network to learn a greedy policy that outputs which vertex being inserted into the partial tour. They adopt the farthest strategy to get the best insertion position of the partial tour. Nowak et al. propose a supervised manner to directly output a tour as an adjacency matrix based on a Graph Neural Network and then convert the matrix into a feasible solution by beam search. The author only reports an optimality gap of 2.7% for n = 20 and slightly worse than the auto-regressive data-driven model . The performance of the above-mentioned methods was suffered due to the greedy policy which selects the vertex according to the biggest prior probability or the value. In this paper, we introduce a new Monte Carlo Tree Search-based algorithm to overcome this problem. Let G(V, E, w) denotes a weighted graph, where V is the set of vertices, E the set of edges and w: E → R + the edge weight function, i.e., w(u, v) is the weight of edge (u, v) ∈ E. We use S = {v 1, v 2, ..., v i} to represent an ordered tour sequence that starts with v 1 and ends with v i, and S = V \ S the set of candidate vertices for addition, condition on S. The target of TSP is to find a tour sequence with the lowest cost, i.e., c(G, S) = For a graph, our goal is to construct a tour solution by adding vertices successively. A natural approach is to train a deep neural network of some form to decide which vertex being added to the partial tour at each step. That is, a neural network f would take the graph G and the partial tour sequence S as input, and the output f (G|S) would be a prior probability that indices how likely each vertex to be selected. Intuitively, we can use the prior probability in a greedy way, i.e., selecting vertex with the biggest probability, to generate the tour sequence incrementally. However, deriving tours in this way might fall into the local optimum because the algorithm has only one shot to compute the optimal tour and never goes back and reverses the decision. To overcome this problem, we enhance the policy-decisions by MCTS assisted with the deep neural network. We begin in Section 4.1 by introducing how to transform TSP into a Markov Decision Process (MDP). Then in Section 4.2, we describe the GNN architecture for parameterizing f (G|S). Finally, Section 4.3 describes GNN-MCTS for combinatorial optimization problems, especially the TSP. The overall approach is illustrated in Figure 1. We present TSP as a MDP as follows, • States: a state s is an ordered sequence of visited vertices on a graph G and the terminal state is that all vertices have been visited. • Transition: transition is deterministic in the TSP, and corresponds to adding one vertex v ∈S to S. • Actions: an action a is selecting a vertex of G from the vertices candidate setS. • Rewards: the reward function r(s, a) at state s is defined as the change of cost after taking action a and transitioning to a new state s, i.e., r(s, a) = −w(v m, v n), where v m and v n are the last vertex in partial tour sequence S and S respectively. • Policy: based on the improved probabilityP generated by the GNN-MCTS, a deterministic greedy policy π(v|S):= arg max v ∈SP (S, v) is used. To compute a good policy, information about the global structure of the graph and the current constructed tour sequence S = {v 1, ..., v i} is required. We tag the nodes which have been visited as x v = 1. Intuitively, f (G|S) should summarize the state of such a "tagged" graph and generate the prior probability that indicates how likely each vertex is to belong to S. It is challenging to design a neural network f (G|S) to capture local and global graph structure. In order to represent such a complicated context, we propose a new deep learning architecture based on graph neural networks (GNN) to parameterize f (G|S). Similar to the basic GNN, we design the neural network f (G|S; Θ) to compute a l-dimensional feature H v for each vertex of a "tagged" graph. We use H t v to denote the real-valued feature vector associated with v after the computation by the layer t. A GNN model consists of a stack of T neural network layers, where each layer aggregates local neighborhood information, i.e., features of neighbors around each node, and then passes this aggregated information on to the next layer. Specifically, the basic GNN model can be implemented as follows. In each layer t ∈ [0, T], a new feature is computed as: where N (v) is the set of neighbors of vertex v, W t 1 and W t 2 are parameter matrices for the layer t, and σ denotes a component-wise non-linear function, e.g., a sigmoid or a ReLU. For t = 0, H 0 v denotes the feature initialization at the input layer. The above GNN architecture has been demonstrated to perform well on combinatorial optimizations problems such as Maximal Independent Set (MIS), Minimum Vector Cover (MVC), etc. . As observed from the Equation 1, the edge information is not taken into account for MIS, MVC, but, for TSP, edge information cannot be ignored, because the object of TSP is computed based on the edge cost, i.e., the distance between the two vertices. We integrate edge information ) is used to compute the prior probability map that indicates how likely each vertex is in the tour sequence. Firstly, the "tagged" graph is fed into the GNN to generate new feature expressions for each vertex. Then all new node feature is concentrated into a long vector that denotes the context of the "tagged" graph. Lastly, the vector is fed into a multilayer perceptron to output the prior probability. The picture on the right (B) depicts the mechanism of computing a new feature of the vertex in one update-layer. into the new node feature H as follows, where e(v, u) is the distance 1 between two vertices and W t 3 are parameter matrices for the layer t. Dai et al. proposed a graph embedding networks (GEN) based on structure2vec to compute new node feature µ as follows, where θ 1 ∈ R l, θ 2, θ 3 ∈ R l×l and θ 4 ∈ R l are model parameters. Compared with GEN, the key improvements are: 1) Our GNN replaces x v in Equation 3 with H v so that the our GNN could integrate the latest feature of the node itself directly in each update procedure. 2) One can regard each update process in the GEN as one update layer of the our GNN, i.e., each calculation is equivalent to going one layer forward, and counting T times is the T layers. Parameters of each layer in our GNN are independent, while parameters are shared between different update processes in GEN which limits the ability of the neural network. 3) Instead of aggregating edge weight by "sum" operation, we use "average" operation to balance the weight of node and edge feature. Experimental show that the above improvements enhance the performance of the neural network. We initialize the node feature H 0 as follows. Each vertex has a feature tag which is a 3-dimensional vector. The first element is binary and equal to 1 if the partial tour sequence S contains the vertex. The second and third elements of the feature tag are the coordinates of the vertex. When a partial tour has been constructed, it can not be changed, and the remaining problem is to find a path from the last vertex, through all unvisited vertices, to the first vertex. To know the first and the last vertex in partial tour sequence S, besides basic feature tags described above, we extend the node feature H 0 by adding feature tags of the first and last vertex in partial tour sequence S (see in Figure 2). Once feature for each vertex is computed after T iterations, and we use the new feature of vertices to define the f (G|S; Θ), which outputs the prior probability indicating how likely each vertex is to belong to partial tour sequence S. More specifically, we fuse all vertex feature H T v as the current state representation of the graph and parameterize f (G|S; Θ) as follows, where sum denotes summation operator. During training, we minimize the cross-entropy loss for each training sample (G i, S i): where S i is a tour sequence which is a permutation of the vertices over graph G i and y j is a one-hot vector whose length is N and S(j)-th position is 1. The architecture of the deep neural networks is illustrated in Figure 2. Similar to the implementation in , the GNN-MCTS uses deep neural networks as a guide. Each node s in the search tree contains edges (s, a) for all legal actions a ∈ A(s). Each edge stores a set of statistics, {N (s, a), Q(s, a), P (s, a)} where N (s, a) is the visit count, Q(s, a) is the action value and P (s, a) is the prior probability of selecting that edge. To be mentioned, three biggest differences between GNN-MCTS and AlphaGo are: • When playing the game of Go, the branch with a high average rate of winning indicates that the route is strong. While TSP is interested in finding the extreme, the average value makes no sense if several suboptimal routes surround the extreme route. Instead of recording the average action value, we propose to track the best action value found under each node's subtree for determining its exploitation value. • In the game of Go, it is common to use {0, 0.5, 1} to denote the of a game composed of loss, draw, and win. Not only is this convenient, but it also meets the requirements of UCT (Kocsis & Szepesvári, 2006) for rewards to lie in the range. In TSP, an arbitrary tour length can be achieve that does not fall into the predefined interval. One can solve this issue by adjusting the parameter c puct of UCT in such a way that it is feasible for a specified interval. It requires substantial trial-and-error on adjusting c puct due to the change in the number of cities. Instead, we address this problem by normalizing the action value of each node n whose parent is node p to as follows, where b p and w p are, respectively, the best (maximum) and the worst (minimum) action value under p, and Q n is the action value of n. The best action value under p is normalized to the value of 1, the worst action value is normalized to 0, and all other are normalized to. • AlphaGo used a learned value function (critic) v(s, θ) to estimate the probability of the current player winning from position s, where the parameters θ are learned from the observations (s, π). However, getting such algorithms to work is non-trivial. Instead, we design a value function h(s) that combines the GNN and beam search to evaluate the possible tour length from the current state to the end state. Guided by the output of GNN, the value function executes beam search from the state corresponding to the leaf node l until reaching an end state. We compute the value of leaf node V l according to the partial tour sequence S corresponding to the end state as follows, The value function is described in algorithm 1. The GNN-MCTS proceeds by iterating over the four phases and then selects a move to play. SET \ {state} for state in SET do if state is end then return state in SET with biggest value 15: The first in-tree phase of each rollouts begins at the root of node s 0 of the search tree and finishes when the rollouts reaches a leaf node s l at time step l. At each of these time steps, t<l, we use a variant of PUCT to balance exploration(i.e., visiting states suggested by the prior policy) and exploitation(i.e., visiting states that have the best value) according to the statistics in the search tree. where c puct is a constant to trading off between exploration and exploitation. Expansion Strategy. When a leaf node l is reached, the corresponding state s l is evaluated by the deep neural network to obtain the prior probability p of its child nodes. The leaf node is expanded and the statistic of each edge Simulation Strategy. Rather than using a random strategy, we use value function h(s) to evaluate the length of the tour that may be generated from the leaf node s l. Back-Propagation Strategy. For each step t < L, the edge statistics are updated in a backward process. The visit counts are increased, N (s t, a t) = N (s t, a t) + 1, and the action value is updated to best value, Q(s t, a t) = max(Q(s t, a t), V l ). Play. At the end of several rollouts, we select node with the biggestP (a|s 0) = 1 − Q(s0,a) b Q(s0,b) as the next move a in the root position s 0. The search tree will be reused at subsequent time steps: the child node relating to the selected node becomes the new root node, and all the statistics of sub-tree below this child node is retained. . We compare against Nearest, Random and Farthest Insertion, as well as Nearest Neighbor, which are non-learned baseline algorithms that also derive a tour by adding vertices successively. Additionally, we compare against excellent deep learning-based methods based on the greedy framework as mentioned in Section 2, most importantly Vinyals et al. , Bello et al. , Kool et al. , and Dai et al. . We generate 50,000 instances (see in appendix A) for TSP20, TSP50, and TSP100, respectively, to train GNN (settings are in appendix B). We use state-of-art solvers (Gurobi and Concorde) to obtain the optimal tour sequence for each instance. Then we generate N samples for each instance according to the optimal tour sequence. We divide the dataset into a training set, a validation set, and a test set according to the ratio of 8: 1: 1. We use Adam with 128 mini-batches and learning rate 10 −3. Training proceeds for 30 epochs on a machine with 2080ti GPU. After training models for TSP20, TSP50, and TSP100, respectively, we use pre-trained GNN to guide GNN-MCTS. During testing, we randomly generate 1000 instances for the above three problems. The parameter settings of the GNN-MCTS used in our experiments are as follows: we set c puct = 1.3 and beam width = 1 for three problems; we set rollouts = 800, 800 and 1200 respectively for TSP20, TSP50, and TSP100. Besides non-learned algorithms, we mainly compare our method with excellent deep learning-based works that derive tours on the greedy mechanism. We implement and train a Pointer network with supervised learning, but we find that our supervised learning are not as good as those reported by . Results of Pointer network on the random instances are from the optimality gaps they report on 20, 50 vertex graphs. For other deep learning-based methods, we use experimental settings suggested by authors to train and get the same performance as reported. Running times are important but hard to compare because they can vary by two orders of magnitude as a of implementation (Python or C++) and hardware (CPU or GPU). Our method is slower than other learning-based methods due to the look-ahead search. Our code is written by Python and we note that the MCTS procedure can speed up by rewritten code to C++. We test our algorithm, Gurobi and learning-based methods on a machine with 32 virtual CPU system (2 * Xeon(R) E5-2620)) and 8 * 2080ti. At each epoch, we test 32 instances in parallel and after 10 epochs, we report the time it takes to solve on each test instance (in Table 2). In order to explore the generalization of our method, we train the GNN on TSP100 random instances and test our method on random instances including TSP200, TSP300 and TSP500. We mainly compare the learning-based methods proposed by Kool et al. and Dai et al. which made the best performance before our work respectively in Encoder-Decoder and Graph Embedding framework. The (in Table 3) show that our algorithm could generalize to larger problems well than other learning-based algorithms even if trained in the small-scale instances. We analyze the effect of different strategies used in the GNN-MCTS procedure. The comparison of different strategies are 1) best. Different from AlphaGo, we track the best action value found under each node's subtree for determining its exploitation value. At the end of several rollouts, we select the node with the best (biggest) action value as the next move in the root position. 2) average. As with the strategy used in AlphaGo, which is common in a two-player game, we track the average action value found under each node's subtree as exploitation value. Rather than selecting the node with the best (biggest) action value, we select the most visited node as the next move in the root position. Table 1 shows the gap between solutions of our approach with two strategies and the best-known solution for TSP20, TSP50, and TSP100. We refer GNN-MCTS to denote "best" strategy and refer GNN-MCTS ave. to denote "average" strategy. The empirical show that using the "best" strategy is far better than using the "average" strategy for TSP. We conduct a controlled experiment on the TSP test set to analyze how each component contributes to the presented approach. First, we use our GNN to generate solutions in a greedy way, i.e., selecting the vertex with the biggest prior probability at each step; we refer to this version as GNN-MCTS-t. Then we use a GNN-MCTS which replaces the value function h(s) (see in algorithm Table 5: Time cost of different beam width w=1 w=5 w=10 TSP20 55ms 265ms 534ms TSP50 147ms 730ms 1461ms TSP100 323ms 1639ms 3338ms 1) with random rollout to generate tours; we refer to this version as GNN-MCTS-v. Furthermore, we take the GNN prior out of the picture and initialize prior probability to 1 for newly expanded nodes; we refer to this version as GNN-MCTS-p. Lastly, a pure MCTS which removes GNN prior and value function is listed for comparison; we refer this version as MCTS. Table 1 shows the gap between the solution of each approach and the best-known solution on different TSP problems. The from GNN-MCTS-p and GNN-MCTS show that GNN prior could help MCTS to effectively reduce the search space so that MCTS can allocate more computing resources to nodes with high value. Furthermore, the from GNN-MCTS-v and GNN-MCTS show that value function h(s) can estimate the path length from the leaf node well and the MCTS, which uses a suitable value function, can perform better than using a random rollout. Lastly, the gap of performance between GNN-MCTS-t and GNN-MCTS shows that the developed MCTS can efficiently avoid algorithm falling into local optimal and plays an important role in enhancing the performance of our method. We conduct experiments to explore the effects of different widths on the performance of value function. Since the beam width mainly affects the accuracy of value function, we use the of value function as a measure and report the Gap as defined in Table 1. Specifically, we set beam width to 1, 5, 10 and test performance of the value function on random instances including TSP20, TSP50, and TSP100. We also count the time cost of the different settings of the beam width. The experimental of Table 4 and Table 5 show that as the beam width increases, the performance of the value function will get better while the time cost will become larger. We need to make a trade-off between accuracy and time cost. Compared with the basic GNN, our GNN integrates edge information for computing new node feature, and it should extract more information and perform well than basic GNN. To support this statement, we compare the performance of basic GNN and our GNN on random instances, including TSP20, TSP50, and TSP100. We generate tour sequences by using the neural network in a greedy way, i.e., selecting vertex with the biggest prior probability at each step. The performance of two GNN is reported in Table 8 (see in appendix D). We also compare the performance of GEN and our GNN to support the key improvements made by our GNN. Similar to the above comparison experiment, we generate tour sequences by using neural network in a greedy way. The performance of GEN and our GNN is reported in Table 8. Furthermore, we use GEN and our GNN to guide MCTS separately on "random" and "clustered" instances, including TSP20, TSP50 and TSP100. We refer to MCTS with GEN as GEN-MCTS. Table 1 reports the quality of solutions and shows that MCTS can get shorter tours when guided by our GNN. We proposed a graph neural network assisted Monte Carlo Tree Search (GNN-MCTS) for the classical traveling salesman problem. The core idea of our approach lies in converting the TSP into a tree search problem. To capture the local and global graph structure, we train a graph neural network (GNN) which integrates node feature and edge weight into the feature update process. Instead of using the prior probability output by GNN in a greedy way, we designed a GNN-MCTS to provide scouting simulation so that the algorithm could avoid being stuck into the local optimum. The exper-imental show that the proposed approach can obtain shorter tours than other learning-based methods. We see the presented work as a step towards a new family of solvers for NP-hard problems that leverage both deep learning and classic heuristics. We will release code to support future progress in this direction. To evaluate our method against other approximation algorithms and deep learning-based approaches, we use an instance generator from the DIMACS TSP Challenge to generate two types of Euclidean instances: "random" instances consist of n points scattered uniformly at random in the square; "clustered" instances include n points that are clustered into n/100 clusters. We consider three benchmark tasks, Euclidean TSP20, 50 and 100, for which we generate a train set of 50,000 instances and a test set of 1,000 instances. Our GNN has T = 3 node-update layers, which is deep enough for nodes to aggregate information associated with their neighbor vertices. Since the input is a "tagged" graph with 9-dimensional feature on vertices, the input contains vectors of size H 0 = 9. The width of other layers are identical: H t = 64 for t = 1, 2. The proposed GNN has a deep architecture that consists of several node-update layers. Therefore, as the model gets deeper with more layers, the more information can be aggregated by nodes. We train proposed GNN with the different number of layers on random instance from TSP20. We greedily use the prior probability, i.e., selecting the vertex with the biggest prior probability, to derive tour sequence. We report Gap as defined in Table 1. The of Table 6 show that the performance of GNN will become better as the number of network layers increases.

Represent the short summary for finding the corresponding scientific article (long)

A Graph Neural Network Assisted Monte Carlo Tree Search Approach to Traveling Salesman Problem

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.