Question:
represent the scientific article to find a short tldr
Examples:
Provided: Since deep neural networks are over-parameterized, they can memorize noisy examples. We address such memorizing issue in the presence of annotation noise. From the fact that deep neural networks cannot generalize neighborhoods of the features acquired via memorization, we hypothesize that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting noisy examples by eliminating them using the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on noisy MNIST, CIFAR-10, and CIFAR-100 in an efficient manner. Deep neural networks (DNNs) have shown excellent performance on visual recognition datasets . However, it is difficult to obtain highquality labeled datasets in practice (a). Even worse, DNNs could not generalize the training data in the presence of noisy examples. Therefore, there is an increasing demand for robust training methods. In general, DNNs optimized with SGD first generalize clean examples under label noise. Based on this, recent studies consider examples that incur small losses on the network that does not overfit noisy examples as being clean . However, such small-loss examples may be corrupted, particularly under a high level of noise. Hence, choosing safe examples from the noisy dataset with small-loss criteria may be impractical. To address this, we find the method of screening out noisy examples among small-loss examples by focusing on well-known observations: (i) noisy examples are learned via memorization rather than via generalization and (ii) under a certain perturbation, network predictions for memorized features easily fluctuate, while those for generalized features do not. Based on these two observations, we hypothesize that out of small-loss examples, training losses of noisy examples would increase by injecting certain perturbation to network parameters, while those of clean examples would not. This suggests that examples that consistently incur small losses under multiple perturbations can be considered as being clean. Since this idea comes from an artifact of SGD optimization, it can be applied to any architecture optimized with SGD. In this work, we introduce a method of perturbing parameters to filter noisy examples out of smallloss examples. By embedding the filtering into training, we propose a new robust training scheme termed learning with ensemble consensus (LEC). In LEC, the network is first trained on the entire training set for a while and then trained on the intersection of small-loss examples of the ensemble of perturbed networks. We present three LECs with different perturbations and evaluate their effectiveness on three benchmark datasets with random label noise , open-set noise (b), and semantic noise. The proposed LEC outperforms existing robust training methods by efficiently removing noisy examples from training batches. Generalization of DNNs. Although DNNs are over-parameterized, they have impressive generalization ability . Some studies argue that gradient-based optimization plays an important role in regularizing DNNs (; . show that DNNs optimized with gradient-based methods generalize clean examples in the early stage of training. Since mislabeling reduces the correlation with other training examples, it is likely that noisy examples are learned via memorization. Therefore, we analyze the difference between generalized and memorized features to discriminate clean and noisy examples. Training DNNs with Noisy datasets. Label noise issues can be addressed by reducing negative impact of noisy examples. One direction is to train with a modified loss function based on the noise distribution. Most studies of this direction estimate the noise distribution prior to training as it is not accessible in general (; ; ;). Another direction is to train with modified labels using the current model prediction . Aside from these directions, recent work suggests a method of exploiting small-loss examples (; ; ;) based on the generalization ability of DNNs. However, it is still hard to find clean examples by relying on training losses. This study presents a simple method to overcome such a problem of small-loss criteria. Suppose that % of examples in a dataset D:= D clean ∪ D noisy are noisy. Let S,D,θ denote the set of (100-)% small-loss examples of the network f parameterized by θ out of examples in D. Since it is generally hard to learn only all clean examples especially on the highly corrupted training set, it is problematic to regard all examples in S,D,θ as being clean. To mitigate this, we suggest a simple idea: to find noisy examples among examples in S,D,θ. Since noisy examples are little correlated with other training examples, they are likely to be learned via memorization. However, DNNs cannot generalize neighborhoods of the memorized features. This means that even if training losses of noisy examples are small, they can be easily increased under a certain perturbation δ, i.e., for (x, y) ∈ D noisy, Unlike noisy examples, the network f trained on the entire set D can generalize some clean examples in the early stage of training. Thus, their training losses are consistently small in the presence of the perturbation δ, i.e., for (x, y) ∈ D clean, This suggests that noisy examples can be identified from the inconsistency of losses under certain perturbation δ. Based on this, we regard examples in the intersection of (100-)% small-loss examples of an ensemble of M networks generated by adding perturbations δ 1, δ 2,..., δ M to θ, i.e., as being clean. We call it ensemble consensus filtering because examples are selected via ensemble consensus. With this filtering, we propose a new robust training method termed learning with ensemble consensus (LEC) described in Algorithms 1 and 2. Both algorithms consist of warmingup and filtering processes. The difference between these two lies in the filtering process. During the filtering process of Algorithm 1, the network is trained on the intersection of (100-)% small-loss examples of M networks within a mini batch B, thus the number of examples updated at once is changing. We can encourage more stable training with a fixed number of examples to be updated at once as described in Algorithm 2. During the filtering process of Algorithm 2, we first obtain the intersection of small-loss examples of M networks within a full batch D at each epoch. We then sample a subset of batchsize from the intersection and train them at each update like a normal SGD. Require: noisy dataset D with noise ratio %, duration of warmingup Tw, # of networks used for filtering M, perturbation δ 1: Initialize θ randomly 2: for epoch t = 1: Tw do Warming-up process 3: for mini-batch index b = 1: Sample a subset of batchsize B b from a full batch D 5: Ensemble consensus filtering 16: for mini-batch index b = 1: Sample a subset of batchsize B b from a full batch D 5: Ensemble consensus filtering 14: for mini-batch index b = 1: Sample a subset of batchsize B b from D t 16: end for 18: end for Now the goal is to find a perturbation δ to be injected to distinguish between generalized and memorized features. We present three LECs with different perturbations in the following. The pseudocodes can be found in Section A.1.3. • Network-Ensemble Consensus (LNEC): Inspired by the observation that an ensemble of networks with the same architecture is correlated during generalization and is decorrelated during memorization , the perturbation δ comes from the difference between M networks. During the warming-up process, M networks are trained independently. During the filtering process, M networks are trained on the intersection of (100-)% small-loss examples of M networks. • Self-Ensemble Consensus (LSEC): We focus on the relationship between and: network predictions for memorized features are uncertain and those for generalized features are certain. Since the uncertainty of predictions also can be captured by multiple stochastic predictions , the perturbation δ comes from the difference between M stochastic predictions of a single network. 1 During the filtering process, the network is trained on the intersection of (100-)% small-loss examples obtained with M stochastic predictions. • Temporal-Ensemble Consensus (LTEC): Inspired by the observation that during training, atypical features are more easily forgetful compared to typical features , the perturbation δ comes from the difference between networks at current and preceding epochs. During the filtering process, the network is trained on the intersection of (100-)% small-loss examples at the current epoch t and preceding min(M − 1, t − 1) epochs. We collect (100-)% small-loss examples at the preceding epochs, rather than network parameters to reduce memory usage. In this section, we show (i) the effectiveness of three perturbations at removing noisy examples from small-loss examples and (ii) the comparison of LEC and other existing methods under various annotation noises. Annotation noise. We study random label noise , open-set noise (b), and semantic noise. To generate these noises, we use MNIST , CIFAR-10/100 that are commonly used to assess the robustness. For each benchmark dataset, we only corrupt its training set, while leaving its test set intact for testing. The details can be found in Section A.1.1. • Random label noise. Annotation issues can happen in easy images as well as hard images (a). This is simulated in two ways: sym-% and asym-%. For sym-%, % of the entire set are randomly mislabeled to one of the other labels and for asym-%, each label i of % of the entire set is changed to i + 1. We study four types: sym-20% and asym-20% to simulate a low level of noise, and sym-60% and asym-40% to simulate a high level of noise. • Open-set noise. In reality, annotated datasets may contain out-of-distribution (OOD) examples. As in , to make OOD examples, images of % examples randomly sampled from the original dataset are replaced with images from another dataset, while labels are left intact. SVHN is used to make open-set noise of CIFAR-100, and ImageNet-32 and CIFAR-100 are used to make open-set noise of CIFAR-10. We study two types: 20% and 40% open-set noise. • Semantic noise. In general, images with easy patterns are correctly labeled, while images with ambiguous patterns are obscurely mislabeled. To simulate this, we select the top % most uncertain images and then flip their labels to the confusing ones. The uncertainty of each image is computed by the amount of disagreement between predictions of networks trained with clean dataset as in. 2 Then, the label of each image is assigned to the label with the highest value of averaged softmax outputs of the networks trained with a clean dataset except for its ground-truth label. We study two types: 20% and 40% semantic noise. Architecture and optimization. Unless otherwise specified, we use a variant of 9-convolutional layer architecture . All parameters are trained for 200 epochs with Adam with a batch size of 128. The details can be found in Section A.1.2. Hyperparameter. The proposed LEC involves three hyperparameters: duration of warming-up T w, noise ratio %, and the number of networks used for filtering M. Unless otherwise specified, T w is set to 10, and M is set to 5 for random label noise and open-set noise, and 10 for semantic noise. We assume that a noise ratio of % is given. Further study can be found in Section 5.2. Evaluation. We use two metrics: test accuracy and label precision . At the end of each epoch, test accuracy is measured as the ratio of correctly predicted test examples to all test examples, and label precision is measured as the ratio of clean examples used for training to examples used for training. Thus, for both metrics, higher is better. For methods with multiple networks, the averaged values are reported. We report peak as well as final accuracy because a small validation set may be available in reality. For each noise type, every method is run four times with four random seeds, e.g., four runs of Standard on CIFAR-10 with sym-20%. A noisy dataset is randomly generated and initial network parameters are randomized for each run of both random label noise and open-set noise. Note that four noisy datasets generated in four runs are the same for all methods. On the other hand, semantic noise is generated in a deterministic way. Thus, only initial network parameters are randomized for each run of semantic noise. 2 The uncertainty of image x is defined by f (x; θn)) where f (; θ) denotes softmax output of network parameterized by θ. Here, N is set to 5 as in. CIFAR-10, asym-40% Figure 2: Label precision (%) of small-loss examples of the current network (in green) and the intersection of small-loss examples of the current and preceding networks (in red) during running LTEC on CIFAR-10 with random label noise. We report the precision from epoch 11 when the filtering process starts. Comparison with Self-training. In Section 3.1, we argue that (100-)% small-loss examples may be corrupted. To show this, we run LEC with M = 1, which is a method of training on (100-)% small-loss examples. Note that this method is similar to the idea of;. We call it Self-training for simplicity. Figure 1 shows the label precision of Selftraining is low especially under the high level of noise, i.e., sym-60%. Compared to Self-training, three LECs are trained on higher precision data, achieving higher test accuracy as shown in Table 1. Out of these three, LTEC performs the best in both label precision and test accuracy. Noisy examples are removed through ensemble consensus filtering. In LTEC, at every batch update, we first obtain (100-)% small-loss examples of the current network and then train on the intersection of small-loss examples of the current and preceding networks. We plot label precisions of small-loss examples of the current network (in green) and the intersection (in red) during running LTEC on CIFAR-10 with random noise in Figure 2. We observe that label precision of the intersection is always higher, indicating that noisy examples are removed through ensemble consensus filtering. Competing methods. The competing methods include a regular training method: Standard, a method of training with corrected labels: D2L , a method of training with modified loss function based on the noise distribution: Forward , and a method of exploiting small-loss examples: Co-teaching . We tune all the methods individually as described in Section A.1.4. Results on MNIST/CIFAR with random label noise. The overall can be found in Figures 3 and 4, and Table 2. We plot the average as a solid line and the standard deviation as a shadow around the line. Figure 3 states that the test accuracy of D2L increases at the low level of label noise as training progresses, but it does not increase at the high level of label noise. This is because D2L puts large weights on given labels in the early stage of training even under the high level of noise. Forward shows its strength only in limited scenarios such as MNIST. Co-teaching does not work well on CIFAR-100 with asym-40%, indicating that its cross-training scheme is vulnerable to smallloss examples of a low label precision (see Figure 4). Unlike Co-teaching, our methods attempt to remove noisy examples in small-loss examples. Thus, on CIFAR-100 with asym-40% noise, both LTEC and LTEC-full surpass Co-teaching by a wide margin of about 6% and 5%, respectively. Results on CIFAR with open-set noise. The overall can be found in Table 3. All the methods including LTEC and LTEC-full perform well under open-set noise. We speculate that this is due to a low correlation between open-set noisy examples. This is supported by the on CIFAR-10, i.e., all the methods perform better on ImageNet-32 noise than on CIFAR-100 noise, as ImageNet-32 has more classes than CIFAR-100. Similar to poorly annotated examples, out-ofdistribution examples are difficult to be generalized during the warming-up process. Therefore, they can be removed from training batches through ensemble consensus filtering. Results on CIFAR with semantic noise. The overall can be found in Table 4. The semantically generated noisy examples are highly correlated with each other, making it difficult to filter out those examples through ensemble consensus. We use 10 as the value of M for semantic noise because ensemble consensus with a bigger M is more conservative. On CIFAR with semantic noise, LTEC and LTEC-full perform comparably or best, compared to the other methods. Of the two, LTEC-full performs better on 40% semantic noise due to its training stability. It is hard to learn all clean examples during the warming-up process. Therefore, clean examples with large losses may be excluded from training batches during the filtering process. However, we expect that the number of clean examples used for training would increase gradually as training proceeds since LEC allows the network to generalize clean examples without overfitting. To confirm this, we measure recall defined by the ratio of clean examples used for training to all clean examples at the end of each epoch during running LTEC and LTEC-full. As expected, recalls of both LTEC and LTEC-full sharply increase in the first 50 epochs as described in Figure 5. Pre-training prior to the filtering process may help to prevent the removal of clean examples from training batches. The number of networks used for filtering. During the filtering process of LEC, we use only the intersection of small-loss examples of M perturbed networks for training. This means that the number of examples used for training highly depends on M. To understand the effect of M, we run LTEC with varying M on CIFAR-10 with random label noise. In particular, the range of M is {1, 3, 5, 7, 9, ∞}. Table 5 shows that a larger M is not always lead to better performance. This is because too many examples may be removed from training batches as M increases. Indeed, the total number of examples used for training is critical for the robustness as claimed in;. Noise ratio. In reality, only a poorly estimated noise ratio may be accessible. To study the effect of poor noise estimates, we run LTEC on CIFAR-10 with random label noise using a bit lower and higher values than the actual noise ratio as in. We also run Co-teaching that requires the noise ratio for comparison. The overall can be found in Table 6. Since it is generally difficult to learn all clean examples, training on small-loss examples selected using the over-estimated ratio (i.e., 1.1) is often helpful in both Co-teaching and LTEC. In contrast, smallloss examples selected using the under-estimated ratio may be highly corrupted. In this case, LTEC is robust to the estimation error of noise ratio, while Co-teaching is not. Such robustness of LTEC against noise estimation error comes from ensemble consensus filtering. Applicability to different architecture. The key idea of LEC is rooted in the difference between generalizaton and memorization, i.e., the ways of learning clean examples and noisy examples in the early SGD optimization. Therefore, we expect that LEC would be applicable to any architecture optimized with SGD. To support this, we run Standard and LTEC with ResNet-20 . The architecture is optimized based on , achieving the final test accuracy of 90.67% on clean CIFAR-10. Here, T w is set to 30 for the optimization details. Table 7 shows LTEC (ResNet) beats Standard (ResNet) in both peak and final accuracies, as expected. This work presents the method of generating and using the ensemble for robust training. We explore three simple perturbation methods to generate the ensemble and then develop the way of identifying noisy examples through ensemble consensus on small-loss examples. Along with growing attention to the use of small-loss examples for robust training, we expect that our ensemble method will be useful for such training methods. A.1.1 ANNOTATION NOISES • Random label noise: For sym-%, % of the entire set are randomly mislabeled to one of the other labels and for asym-%, each label i of % of the entire set is changed to i + 1. The corruption matrices of sym-% and asym-% are described in Figures A1a and A1b, respectively. • Open-set noise: For % open-set noise, images of % examples randomly sampled from the original dataset are replaced with images from external sources, while labels are left intact. For CIFAR-10 with open-set noise, we sample images from 75 classes of CIFAR-100 and 748 classes of ImageNet to avoid sampling similar images with CIFAR-10. • Semantic noise: For semantic noise, we choose uncertain images and then mislabel them ambiguously. In Figure A2, we see that clean examples have simple and easy images, while noisy examples have not. Also, its corruption matrix (see Figure A1c) describes the similarity between classes, e.g., cat and dog, car and truck, etc. The 9-convolutional layer architecture used in this study can be found in Table A1. The network is optimized with Adam with a batchsize of 128 for 200 epochs. The initial learning rate α is set to 0.1. The learning rate is linearly annealed to zero during the last 120 epochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100. The momentum parameters β 1 and β 2 are set to 0.9 and 0.999, respectively. β 1 is linearly annealed to 0.1 during the last 120 epochs for MNIST and CIFAR-10, and during the last 100 epochs for CIFAR-100. The images of CIFAR are divided by 255 and are whitened with ZCA. Additional regularizations such as data augmentation are not applied. The on clean MNIST, CIFAR-10, and CIFAR-100 can be found in Table A2. for mini-batch index b = 1: Sample a subset of batchsize B b from a full batch D S,B b,θ:= (100 −)% small-loss examples of f θ within B b Pt ← Pt ∪ S,B b,θ if t < Tw + 1 then Warming-up process 9: 10: else Filtering process 11: if t = 1 then 12: 13: 14: else 16: 19: 5: end for 6: for epoch t = 2: T end do 7: Pt:= (100 −)% small-loss examples of f θ within D Small-loss examples are computed from the 2nd epoch 8: if t < Tw + 1 then Warming-up process 9: for mini-batch index b = 1: Sample a subset of batchsize B b from a full batch D 11: end for else Filtering process if t < M + 1 then 15: else 17: for mini-batch index b = 1: Sample a subset of batchsize B b from D t 21: The competing methods include a regular training method: Standard, a method of training with corrected labels: D2L , a method of training with modified loss function based on the noise distribution: Forward , and a method of exploiting small-loss examples: Co-teaching . We tune all the methods individually as follows: • Standard: The network is trained using the cross-entropy loss. • D2L: The input vector of a fully connected layer in the architecture is used to measure the LID estimates. The parameter involved with identifying the turning point, window size W is set to 12. The network is trained using original labels until the turning point is found and then trained using the bootstrapping target with adaptively tunable mixing coefficient. • Forward: Prior to training, the corruption matrix C where C ji = P(y = i|y true = j) is estimated based on the 97th percentile of probabilities for each class on MNIST and CIFAR-10, and the 100th percentile of probabilities for each class on CIFAR-100 as in. The network is then trained using the corrected labels for 200 epochs. • Co-teaching: Two networks are employed. At every update, they select their small-loss examples within a minibatch and then provide them to each other. The ratio of selected examples based on training losses is linearly annealed from 100% to (100-)% over the first 10 epochs. We compute space complexity as the number of network parameters and computational complexity as the number of forward and backward passes. Here we assume that early stopping is not used and the noise ratio of % is given. Note that the computational complexity of each method depends on its hyperparameter values, e.g., the duration of the warming-up process T w and the noise ratio %. The analysis is reported in Table A3. Our proposed LTEC is the most efficient because it can be implemented with a single network based on Section A.1.3 and only a subset of the entire training set is updated after the warming-up process. Computational complexity # of forward passes n n 2n M n M n n # of backward passes n ≤ n ≤ 2n ≤ M n ≤ n ≤ n A.3.1 OF LTEC WITH M = ∞ Figure A3 shows that ensemble consensus filtering with too large M removes clean examples from training batches in the early stage of the filtering process. Unlike LTEC with M = 5, the recall of LTEC with M = ∞ does not increase as training proceeds, suggesting that its generalization performance is not enhanced. This shows that a larger M does not always lead to better performance. We expect that pre-training prior to the filtering process helps to reduce the number of clean examples removed by ensemble consensus filtering regardless of M. Figure A3: Recall (%) of LTECs with varying M on CIFAR-10 with random label noise. Match: This work presents a method of generating and using ensembles effectively to identify noisy examples in the presence of annotation noise.  Hard Negative: We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.

Question: Inverse reinforcement learning (IRL) is used to infer the reward function from the actions of an expert running a Markov Decision Process (MDP). A novel approach using variational inference for learning the reward function is proposed in this research. Using this technique, the intractable posterior distribution of the continuous latent variable (the reward function in this case) is analytically approximated to appear to be as close to the prior belief while trying to reconstruct the future state conditioned on the current state and action. The reward function is derived using a well-known deep generative model known as Conditional Variational Auto-encoder (CVAE) with Wasserstein loss function, thus referred to as Conditional Wasserstein Auto-encoder-IRL (CWAE-IRL), which can be analyzed as a combination of the backward and forward inference. This can then form an efficient alternative to the previous approaches to IRL while having no knowledge of the system dynamics of the agent. Experimental on standard benchmarks such as objectworld and pendulum show that the proposed algorithm can effectively learn the latent reward function in complex, high-dimensional environments. Reinforcement learning, formalized as Markov decision process (MDP), provides a general solution to sequential decision making, where given a state, the agent takes an optimal action by maximizing the long-term reward from the environment. However, in practice, defining a reward function that weighs the features of the state correctly can be challenging, and techniques like reward shaping are often used to solve complex real-world problems. The process of inferring the reward function given the demonstrations by an expert is defined as inverse reinforcement learning (IRL) or apprenticeship learning;. The fundamental problem with IRL lies in the fact that the algorithm is under defined and infinitely different reward functions can yield the same policy. Previous approaches have used preferences on the reward function to address the non-uniqueness. suggested reward function that maximizes the difference in the values of the expert's policy and the second best policy. adopted the principle of maximum entropy for learning the policy whose feature expectations are constrained to match those of the expert's. applied the structured max-margin optimization to IRL and proposed a method for finding the reward function that maximizes the margin between expert's policy and all other policies. Neu & Szepesvári unified a direct method that minimizes deviation from the expert's behavior and an indirect method that finds an optimal policy from the learned reward function using IRL. used a game-theoretic framework to find a policy that improves with respect to an expert's. Another challenge for IRL is that some variant of the forward reinforcement learning problem needs to be solved in a tightly coupled manner to obtain the corresponding policy, and then compare this policy to the demonstrated actions. Most early IRL algorithms proposed solving an MDP in the inner loop;;. This requires perfect knowledge of the expert's dynamics which are almost always impossible to have. Several works have proposed to relax this requirement, for example by learning a value function instead of a cost , solving an approximate local control problem or generating a discrete graph of states. However, all these methods still require some partial knowledge of the system dynamics. Most of the early research in this field has expressed the reward function as a weighted linear combination of hand selected features;;. Non-parametric methods such as Gaussian Processes (GPs) have also been used for potentially complex, nonlinear reward functions. While in principle this helps extend the IRL paradigm to flexibly account for non-linear reward approximation; the use of kernels simultaneously leads to higher sample size requirements. Universal function approximators such as non-linear deep neural network have been proposed recently;. This moves away from using hand-crafted features and helps in learning highly non-linear reward functions but they still need the agent in the loop to generate new samples to "guide" the cost to the optimal reward function. has recently proposed deriving an adversarial reward learning formulation which disentangles the reward learning process by a discriminator trained via binary regression data and uses policy gradient algorithms to learn the policy as well. The Bayesian IRL (BIRL) algorithm proposed by uses the expert's actions as evidence to update the prior on reward functions. The reward learning and apprenticeship learning steps are solved by performing the inference using a modified Markov Chain Monte Carlo (MCMC) algorithm. described an expectation-maximization (EM) approach for solving the BIRL problem, referring to it as the Robust BIRL (RBIRL). Variational Inference (VI) has been used as an efficient and alternative strategy to MCMC sampling for approximating posterior densities;. Variational Auto-encoder (VAE) was proposed by as a neural network version of the approximate inference model. The loss function of the VAE is given in such a way that it automatically tries to maximize the likelihood of the data given the current latent variables (reconstruction loss), while encouraging the latent variables to be close to our prior belief of how the variables should look like (KullbeckLiebler divergence loss). This can be seen as an generalization of EM from maximum a-posteriori (MAP) estimation of the single parameter to an approximation of complete posterior distribution. Conditional VAE (CVAE) has been proposed by to develop a deep conditional generative model for structured output prediction using Gaussian latent variables. Wasserstein AutoEncoder (WAE) has been proposed by to utilize Wasserstein loss function in place of KL divergence loss for robustly estimating the loss in case of small samples, where VAE fails. This research is motivated by the observation that IRL can be formulated as a supervised learning problem with latent variable modelling. This intuition is not unique. It has been proposed by using the Cascaded Supervised IRL (CSI) approach. However, CSI uses non-generalizable heuristics to classify the dataset and find the decision rule to estimate the reward function. Here, I propose to utilize the CVAE framework with Wasserstein loss function to determine the non-linear, continuous reward function utilizing the expert trajectories without the need for system dynamics. The encoder step of the CVAE is used to learn the original reward function from the next state conditioned on the current state and action. The decoder step is used to recover the next state given the current state, action and the latent reward function. The likelihood loss, composed of the reconstruction error and the Wasserstein loss, is then fed to optimize the CVAE network. The Gaussian distribution is used here as the prior distribution; however, has described various other prior distributions which can be used based on the class of problem being solved. Since, the states chosen are supplied by the expert's trajectories, the CWAE-IRL algorithm is run only on those states without the need to run an MDP or have the agent in the loop. Two novel contributions are made in this paper: • Proposing a generative model such as an auto-encoder for estimating the reward function leads to a more effective and efficient algorithm with locally optimal, analytically approximate solution. • Using only the expert's state-action trajectories provides a robust generative solution without any knowledge of system dynamics. Section 2 gives the on the concepts used to build our model; Section 3 describes the proposed methodology; Section 4 gives the and Section 5 provides the discussion and . In the reinforcement learning problem, at time t, the agent observes a state, s t ∈ S, and takes an action, a t ∈ A; thereby receiving an immediate scalar reward r t and moving to a new state s t+1. The model's dynamics are characterized by state transition probabilities p(s t+1 |s t, a t). This can be formally stated as a Markov Decision Process (MDP) where the next state can be completely defined by the previous state and action (Markov property) and the agent receives a scalar reward for executing the action. The goal of the agent is to maximize the cumulative reward (discounted sum of rewards) or value function: where 0 ≤ γ ≤ 1 is the discount factor and r t is the reward at time-step t. In terms of a policy π: S → A, the value function can be given by Bellman equation as: Using Bellman's optimality equation, we can define, for any MDP, a policy π is greater than or equal to any other policy π if value function v π (s t) ≤ v π (s t) for all s t ∈ S. This policy is known as an optimal policy (π *) and its value function is known as optimal value function (v *). The bayesian approach to IRL was proposed by by encoding the reward function preference as a prior and optimal confidence of the behavior data as the likelihood. Considering the expert χ is executing an MDP M = (S, A, p, γ), the reward for χ is assumed to be sampled from a prior (known) distribution P R defined as: The distribution to be used as a prior depends on the type of problem. The expert's goal of maximizing accumulated reward is equivalent to finding the optimal action of each state. The likelihood thus defines our confidence in χ's ability to select the optimal action. This is modeled as a exponential distribution for the likelihood of trajectory τ with Q * as: where α χ is a parameter representing the degree of confidence in χ's ability. The posterior probability of the reward function R is computed using Bayes theorem, BIRL uses MCMC sampling to compute the posterior mean of the reward function. For observations x = x 1:n and latent variables z = z 1:m, the joint density can be written as: The latent variables are drawn from a prior distribution p(z) and they are then related to the observations through the likelihood p(x|z). Inference in a bayesian framework amounts to conditioning on data and computing the posterior p(z|x). In lot of cases, this posterior is intractable and requires approximate inference. Variational inference has been proposed in the recent years as an alternative to MCMC sampling by using optimization instead of sampling. For a family of approximate densities ζ over the latent variables, we try to find a member of the family that minimizes the Kullback-Leibler (KL) divergence to the exact posterior The posterior is then approximated with the optimized member of the family q * (z). The KL divergence is then given by Since, the divergence cannot be computed, an alternate objective is optimized in VAE called evidence lower bound (ELBO) that is equivalent, This can be defined as a sum of two separate losses: where L lk is the loss related to the log-likelihood and L div is the loss related to the divergence. CVAE is used to perform probabilistic inference and predict diversely for structured outputs. The loss function is slightly altered with the introduction of class labels c: Wasserstein distance, also known as Kantorovich-Rubenstein distance or earth mover's distance (EMD) , provides a natural distance over probability distributions in the metric space. It is a formulation of optimal transport problem where the Wasserstein distance is the minimum cost required to move a pile of earth (an arbitrary distribution) to another. The mathematical formulation given by is: where c(X, Y) is the cost function, X and Y are random variables with marginal distributions P X and P Y respectively. EMD has been utilized in various practical applications in computer science such as pattern recognition in images. Wasserstein GAN (WGAN) has been proposed by to minimize the EMD between the generative distribution and the data distribution. proposed Wasserstein Auto-encoder (WAE) where the divergence loss has been calculated using the EMD instead of KL-divergence and has been shown to be robust in presence of noise and smaller samples. In this paper, my primary argument is that the inverse reinforcement learning problem can be devised as a supervised learning problem with learning of latent variable. The reward function, r(s t, a t, s t+1), can be formulated as a latent function which is dependent on the state at time t, s t, action at time t, a t, and state at time (t + 1), s t+1. In the CVAE framework, using the state and action pair as the class label c and rewriting the CVAE loss in Equation 17 with s t+1 as x and reward at time t, r t as z, we get: The first part of Equation 20 provides the log likelihood of transition probability of an MDP and the second part gives the KL-divergence of the encoded reward function to the prior gaussian belief. Thus, the proposed method tries to recover the next state from the current state and current action by encoding the reward function as the latent variable and constraining the reward function to lie as close to the gaussian function. The network structure of the method is given in Figure 1. The encoder is a neural network which inputs the current state, current action and next state and generates a probability distribution q(r t |s t+1, s t, a t), assumed to be isotropic gaussian. Since a nearoptimal policy is inputted into the IRL framework, minibatches of randomly sampled (s t+1, s t, a t) are introduced into the network. Two hidden layers with dropout are used to encode the input data into a latent space, giving two outputs corresponding to the mean and log-variance of the distribution. This step is similar to the backward inference problem in IRL methodology where the reward function is constructed from the sampled trajectories for a near-optimal agent. Given the current state and action, the decoder maps the latent space into the state space and reconstructs the next stateŝ t+1 from a sampled r t (from the normal distribution). Similar to the VAE formulation, samples are generated from a standard normal distribution and reparameterized using the mean and log-variance computed in the encoder step. This step resembles the forward inference problem of an MDP where given a state, action and reward distribution, we estimate the next state that the agent gets to. Two hidden layers with dropout are used similar to the encoder. Even though the KL-divergence should be able to provide for the loss theoretically, it does not converge in practice and indeed gives really large values in case of small samples such as in our formulation. provides a Maximum Mean Discrepancy (MMD) measure based on Wasserstein metric for a positive-definite reproducing kernel k(·, ·) such as the Radial Basis Function (RBF) kernel: where H k is the Reproducing Kernel Hilbert Space (RKHS) mapping z: Z → R. The divergence loss can then be written as: where c is the cost between the input, x and the output,z, of the decoder, D, using the sampled latent variable, z (given as mean squared error), The ing CWAE-IRL loss function is given as: In this section, I present the of CWAE-IRL on two simulated tasks, objectworld and pendulum. Objectworld is a generalization of gridworld , described in. It contains NxN grid of states with five actions per state, corresponding to steps in each direction and staying in place. Each action has a 30% chance of moving in a different random direction. There are randomly assigned objects, having one of 2 inner and outer colors chosen, red and green. There are 4 continuous features for each of the grids, each giving the Euclidean distance to the nearest object with a specific inner or outer color. The true reward is +1 in states within 3 cells of outer red and 2 cells of outer green, -1 within 3 cells of outer red, and zero otherwise. Inner colors serve as distractors. The expert trajectories fed have a sample length of 16. The algorithms for objectworld, Maximum Entropy IRL and Deep Maximum Entropy IRL are used from the GitHub implementation of without any modifications. Only continuous features are used for all implementations. CWAE-IRL is compared with prior IRL methods, described in the previous sections. Among prior methods chosen, CWAE-IRL is compared to , and. Only the Maximum Entropy IRL uses the reward as a linear combination of features while the others describe it in a non-linear fashion. The learnt rewards for all the algorithms are shown in Figure 2 with an objectworld of grid size 10. CWAE-IRL can recover the original reward distribution while the Deep Maximum Entropy IRL overestimates the reward in various places. Maximum Entropy IRL and BIRL completely fail to learn the rewards. Deep Maximum Entropy tends to give negative rewards to state spaces which are not well traversed in the example trajectories. However, CWAE-IRL generalizes over the spaces even though they have not been visited frequently. Also, due to the constraint of being close to the prior gaussian belief, the scale of rewards are best captured by the proposed algorithm as compared to the other algorithms which tend to overscale the rewards. The pendulum environment is an well-known problem in the control literature in which a pendulum starts from a random position and the goal is to keep it upright while applying the minimum amount of force. The state vector is composed of the cosine (and sine) of the angle of the pendulum, and the derivative of the angle. The action is the joint effort as 11 discrete actions linearly spaced within the [−2, 2] range. The reward is where w 1, w 2 and w 3 are the reward weights for the angle θ, derivative of angleθ and action a respectively. The optimal reward weights given by OpenAI are [1, 0.1, 0.001] respectively. An episode is limited to 1000 timesteps. A deep Q-network (DQN) has been proposed by that combines deep neural networks with RL to solve continuous state discrete action problems. DQN uses a neural network with gives the Q-values for every action and uses a buffer to store old states and actions to sample from to help stabilize training. Using a continuous state space makes it impossible to have all states visited during training. This also makes it very difficult for the comparison of recovered reward with the actual reward. The DQN is trained for 50,000 episodes. The CWAE-IRL is trained using 25 trajectories and the reward is predicted for 5 trajectories. The error plot between the reward recovered and the actual reward is given in Figure 3. The mean error hovers around 0 showing that under for majority of the states and actions, the proposed method is able to recover the correct reward.

Using a supervised latent variable modeling framework to determine reward in inverse reinforcement learning task

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We view exploration in RL as a problem of matching a marginal distribution over states.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: The travelling salesman problem (TSP) is a well-known combinatorial optimization problem with a variety of real-life applications. We tackle TSP by incorporating machine learning methodology and leveraging the variable neighborhood search strategy. More precisely, the search process is considered as a Markov decision process (MDP), where a 2-opt local search is used to search within a small neighborhood, while a Monte Carlo tree search (MCTS) method (which iterates through simulation, selection and back-propagation steps), is used to sample a number of targeted actions within an enlarged neighborhood. This new paradigm clearly distinguishes itself from the existing machine learning (ML) based paradigms for solving the TSP, which either uses an end-to-end ML model, or simply applies traditional techniques after ML for post optimization. Experiments based on two public data sets show that, our approach clearly dominates all the existing learning based TSP algorithms in terms of performance, demonstrating its high potential on the TSP. More importantly, as a general framework without complicated hand-crafted rules, it can be readily extended to many other combinatorial optimization problems. The travelling salesman problem (TSP) is a well-known combinatorial optimization problem with various real-life applications, such as transportation, logistics, biology, circuit design. Given n cities as well as the distance d ij between each pair of cities i and j, the TSP aims to find a cheapest tour which starts from a beginning city (arbitrarily chosen), visits each city exactly once, and finally returns to the beginning city. This problem is NP-hard, thus being extremely difficult from the viewpoint of theoretical computer science. Due to its importance in both theory and practice, many algorithms have been developed for the TSP, mostly based on traditional operations research (OR) methods. Among the existing TSP algorithms, the best exact solver Concorde succeeded in demonstrating optimality of an Euclidean TSP instance with 85,900 cities, while the leading heuristics and are capable of obtaining near-optimal solutions for instances with millions of cities. However, these algorithms are very complicated, which generally consist of many hand-crafted rules and heavily rely on expert knowledge, thus being difficult to generalize to other combinatorial optimization problems. To overcome those limitations, recent years have seen a number of ML based algorithms being proposed for the TSP (briefly reviewed in the next section), which attempt to automate the search process by learning mechanisms. This type of methods do not rely on expert knowledge, can be easily generalized to various combinatorial optimization problems, thus become promising research direction at the intersection of ML and OR. For the TSP, existing ML based algorithms can be roughly classified into two paradigms, i.e.: End-to-end ML paradigm which uses a ML model alone to directly convert the input instance to a solution. ML followed by OR paradigm which applies ML at first to provide some additional information, to guide the following OR procedure towards promising regions. Despite its high potential, for the TSP, existing ML based methods are still in its infancy, struggling to solve instances with more than 100 cities, leaving much room for further improvement compared with traditional methods. To this end, we propose a novel framework by combining Monte Carlo tree search (MCTS) with a basic OR method (2-opt based local search) using variable neighborhood strategy to solve the TSP. The main contributions are summarized as follows. • Framework: We propose a new paradigm which combines OR and ML using variable neighborhood strategy. Starting from an initial state, a basic 2-opt based local search is firstly used to search within a small neighborhood. When no improvement is possible within the small neighborhood, the search turns into an enlarged neighborhood, where a reinforcement learning (RL) based method is used to identify a sample of promising actions, and iteratively choose one action to apply. Under this new paradigm, OR and ML respectively work within disjoint space, being flexible and targeted, and clearly different from the two paradigms mentioned above. More importantly, as a general framework without complicated hand-crafted rules, this framework could not only be applied to the TSP, but also be easily extended to many other combinatorial optimization problems. • Methodology: When we search within an enlarged neighborhood, it is infeasible to enumerate all the actions. We then seek to sample a number of promising actions. To do this automatically, we implement a MCTS method which iterates through simulation, selection and back-propagation steps, to collect useful information that guides the sampling process. To the best of our knowledge, there is only one existing paper which also uses MCTS to solve the TSP. However, their method is a constructive approach, where each state is a partial TSP tour, and each action adds a city to increase the partial tour, until forming a complete tour. By contrast, our MCTS method is a conversion based approach, where each state is a complete TSP tour, and each action converts the original state to a new state (also a complete TSP tour). The methodology and implementation details of our MCTS are very different from the MCTS method developed in . • Results: We carry out experiments on two sets of public TSP instances. Experimental (detailed in Section 4) show that, on both data sets our MCTS algorithm obtains (within reasonable time) statistically much better with respect to all the existing learning based algorithms. These clearly indicate the potential of our new method for solving the TSP. The rest of this paper is organized as follows: Section 2 briefly reviews the existing learning based methods on the TSP. Section 3 describes in detail the new paradigm and the MCTS method. Section 4 provides and analyzes the experimental . Finally Section 5 concludes this paper. In this section, we briefly review the existing ML based algorithms on the TSP, and then extend to several other highly related problems. Non-learned methods are omitted, and interested readers please find in , , and for an overlook of the leading TSP algorithms. The idea of applying ML to solve the TSP is not new, dated back to several decades ago. proposed a Hopfield-network, which achieved the best TSP solutions at that time. Encouraged by this progress, neural networks were subsequently applied on many related problems (surveyed by). However, these early attempts only achieved limited success, with respect to other state-of-the-art algorithms, possibly due to the lack of high-performance hardware and training data. In recent years, benefited from the rapidly improving hardware and exponentially increasing data, ML (especially deep learning) achieved great successes in the field of artificial intelligence. Motivated by these successes, ML becomes again a hot and promising topic for combinatorial optimization, especially for the TSP. A number of ML based algorithms have been developed for the TSP, which can be roughly classified into two paradigms (possibly with overlaps). End-to-end ML paradigm: introduced a pointer network which consists of an encoder and a decoder, both using recurrent neural network (RNN). The encoder parses each TSP city into an embedding, and then the decoder uses an attention model to predict the probability distribution over the candidate (unvisited) cities. This process is repeated to choose a city one by one, until forming a complete TSP tour. The biggest advantage of the pointer network is its ability of processing graphs with different sizes. However, as a supervised learning (SL) method, it requires a large number of pre-computed optimal (at least high-quality) TSP solutions, being unaffordable for large-scale instances. To overcome this drawback, several successors chose reinforcement learning (RL) instead of SL, thus avoiding the requirement of pre-computed solutions. For example, implemented an actor-critic RL architecture, which uses the solution quality (tour length) as a reward signal, to guide the search towards promising area. proposed a framework which maintains a partial tour and repeatedly calls a RL based model to select the most relevant city to add to the partial tour, until forming a complete tour. also implemented an actor-critic neural network, and chose Sinkhorn policy gradient to learn policies by approximating a double stochastic matrix. and both proposed a graph attention network (GAN), which incorporates attention mechanism with RL to auto-regressively improve the quality of the obtained solution. proposed a MCTS algorithm for the TSP, which also belongs to this paradigm. As explained in the introduction section, this existing method is clearly different from our MCTS algorithm (detailed in Section 3.5). ML followed by OR paradigm: Applying ML alone is difficult to achieve satisfactory performance, thus it is recommended to combine ML and OR to form hybrid algorithms . Following this idea, proposed a supervised approach, which trains a graph neural network (GNN) to predict an adjacency matrix (heat map) over the cities, and then attempts to convert the adjacency matrix to a feasible TSP tour by beam search (OR based method). followed this framework, but chose deep graph convolutional networks (GCN) instead of GNN to build heat map, and then tried to construct tours via highly parallelized beam search. Additionally, several algorithms belonging to above paradigm were further enhanced with OR based algorithms, thus also belonging to this paradigm. For example, the solution obtained by ML is post-optimized by sampling in and , or by 2-opt based local search in . Overall, this hybrid paradigm performs statistically much better than the end-to-end ML paradigm, showing the advantage of combining ML with OR. In addition to the works focused on the classic TSP, there are several ML based methods recently proposed for solving other related problems, such as the decision TSP , the multiple TSP , and the vehicle routing problem , etc. Finally, for an overall survey about machine learning for combinatorial optimization, please refer to and . 3.1 FRAMEWORK The proposed new paradigm for combining OR and ML to solve the TSP is outlined in Fig. 1. Specifically, the search process is considered as a Markov Decision Process (MDP), which starts from an initial state π, and iteratively applies an action a to reach a new state π *. At first, the MDP explores within a small neighborhood, and tries to improve the current state by applying 2-opt based local search. When no further improvement is possible within the small neighborhood, the MDP turns into an enlarged neighborhood, which consists of a large number of possible actions, being infeasible to enumerate one by one. To improve search efficiency, MCTS is launched to iteratively sample a number of promising actions and choose an improving action to apply. When MCTS fails to find an improving action, the MDP jumps into an unexplored search space, and launches a new round of 2-opt local search and MCTS again. This process is repeated until the termination condition is met, then the best found state is returned as the final solution. In our implementation, each state corresponds to a complete TSP solution, i.e., a permutation π = (π 1, π 2, . . ., π n) of all the cities. Each action a is a transformation which converts a given state π to a new state π *. Since each TSP solution consists of a subset of n edges, each action could be viewed as a k-opt (2 ≤ k ≤ n) transformation, which deletes k edges at first, and then adds k different edges to form a new tour. Obviously, each action can be represented as a series of 2k sub-decisions (k edges to delete and k edges to add). This representation method is straightforward, but seems a bit redundant, since the deleted edges and added edges are highly relevant, while arbitrarily deleting k edges and adding k edges may in an unfeasible solution. To overcome this drawback, we develop a compact method to represent an action, which consists of only k sub-decisions, as exemplified in 1, 2, 3, 4, 5, 6, 7, 8). To determine an action, we at first decide a city a 1 and delete the edge between a 1 and its previous city b 1. Without loss of generality, suppose a 1 = 4, then b 1 = 3 and edge is deleted, ing in a temporary status shown in sub-figure (b) (for the sake of clarity, drawn as a line which starts from a 1 and ends at b 1). Furthermore, we decide a city a 2 to connect city b 1, generally ing in an unfeasible solution containing an inner cycle (unless a 2 = a 1). For example, suppose a 2 = 6 and connect it to city 3, the ing temporary status is shown in sub-figure (c), where an inner cycle occurs and the degree of city a 2 increases to 3. To break inner cycle and reduce the degree of a 2 to 2, the edge between city a 2 and city b 2 = 7 should be deleted, ing in a temporary status shown in sub-figure (d). This process is repeated, to get a series of cities a k and b k (k ≥ 2). In this example, a 3 = 3 and b 3 = 2, respectively corresponding to sub-figures (e) and (f). Once a k = a 1, the loop closes and reaches a new state (feasible TSP solution). For example, if let a 4 = a 1 = 4 and connect a 4 to b 3, the ing new state is shown in sub-figure (g), which is redrawn as a cycle in sub-figure (h). Formally, an action can be represented as a = (a 1, b 1, a 2, b 2, . . ., a k, b k, a k+1), where k is a variable and the begin city must coincide with the final city, i.e. a k+1 = a 1. Each action corresponds to a k-opt transformation, which deletes k edges, i.e., (a i, b i), 1 ≤ i ≤ k, and adds k edges, i.e., (b i, a i+1), 1 ≤ i ≤ k, to reach a new state. Notice that not all these elements are optional, since once a i is known, b i can be uniquely determined without any optional choice. Therefore, to determine an action we should only decide a series of k sub-decisions, i.e., the k cities a i, 1 ≤ i ≤ k. Intuitively, this compact representation method brings advantages in two-folds: fewer (only k, not 2k) subdecisions need to be made; the ing states are necessarily feasible solutions. Furthermore, if a i+1 does not belong to the top 10 nearest neighbors of b i, edge (b i, a i+1) is marked as an unpromising edge, and any action involving (b i, a i+1) is marked as an unpromising action. All the unpromising actions are eliminated directly, to reduce the scale of search space. Let L(π) denote the tour length corresponding to state π, then corresponding to each action a = (a 1, b 1, a 2, b 2, . . ., a k, b k, a k+1) which converts π to a new state π *, the difference could be calculated as follows: If ∆(π, π *) < 0, π * is better (with shorter tour length) than π. For state initialization, we choose a simple constructive procedure which starts from an arbitrarily chosen begin city π 1, and iteratively selects a city π i, 2 ≤ i ≤ n among the candidate (unvisited) cities (added to the end of the partial tour), until forming a complete tour π = (π 1, π 2, . . ., π n) which serves as the starting state. More precisely, if there are m > 1 candidate cities at the ith iteration, each candidate city is chosen with probability 1 m. Using this method, each possible state is chosen as the starting state with an equal probability. To maintain the generalization ability of our approach, we avoid to use complex OR techniques, such as the α-nearness criterion for selecting candidate edges and the partition and merge method for tackling large TSP instances , which have proven to be highly effective on the TSP, but heavily depend on expert knowledge. Instead, we choose a straightforward method to search within a small neighborhood. More precisely, the method examines one by one the promising actions with k = 2, and iteratively applies the firstmet improving action which leads to a better state, until no improving action with k = 2 is found. This method is equivalent to the well-known 2-opt based local search procedure, which is able to efficiently and robustly converge to a local optimal state. Once no improving action is found within the small neighborhood, we close the basic 2-opt local search, and switch to an enlarged neighborhood which consists of the actions with k > 2. This method is a generalization of the variable neighborhood search method (Mladenović &), which has been successfully applied to many combinatorial optimization problems. Unfortunately, there are generally a huge number of actions within the enlarged neighborhood (even after eliminating the unpromising ones), being impossible to enumerate them one by one. Therefore, we choose to sample a subset of promising actions (guided by RL) and iteratively select an action to apply, to reach a new state. Following this idea, we choose the MCTS as our learning framework. Inspired by the works in , , and , our MCTS procedure (outlined in Fig. 3) consists of four steps, i.e., Initialization, Simulation, Selection, and Back-propagation, which are respectively designed as follows. Initialization: We define two n × n symmetric matrices, i.e., a weight matrix W whose element W ij (all initialized to 1) controls the probability of choosing city j after city i, and an access matrix Q whose element Q ij (all initialized to 0) records the times that edge (i, j) is chosen during simulations. Additionally, a variable M (initialized to 0) is used to record the total number of actions already simulated. Notice that this initialization step should be executed only once at the beginning of the whole process of MDP. Simulation: Given a state π, we use the simulation process to probabilistically generate a number of actions. As explained in Section 3.2, each action is represented as a = (a 1, b 1, a 2, b 2, . . ., a k, b k, a k+1), containing a series of sub-decisions a i, 1 ≤ i ≤ k (k is also a variable, and a k+1 = a 1), while b i could be determined uniquely once a i is known. Once b i is determined, for each edge (b i, j), j = b i, we use the following formula to estimate its potential Z bij (the higher the value of Z bij, the larger the opportunity of edge (b i, j) to be chosen): Where denotes the averaged W bij value of all the edges relative to city b i. In this formula, the left part emphasizes the importance of the edges with high W bij values (to enhance the intensification feature), while the right part ln (M + 1) Q bij + 1 prefers the rarely examined edges (to enhance the diversification feature). α is a parameter used to achieve a balance between intensification and diversification, and the term "+1" is used to avoid a minus numerator or a zero denominator. To make the sub-decisions sequently, we at first choose a 1 randomly, and determine b 1 subsequently. Recursively, once a i and b i are known, a i+1 is decided as follows: if closing the loop (connecting a 1 to b i) would lead to an improving action, or i ≥ 10, let a i+1 = a 1. otherwise, consider the top 10-nearest neighbors of b i with Z bil ≥ 1 as candidate cities, forming a set X (excluding a 1 and the city already connected to b i). Then, among X each city j is selected as a i+1 with probability p j, which is determined as follows: Once a i+1 = a 1, we close the loop to obtain an action. Similarly, more actions are generated (forming a sampling pool), until meeting an improving action which leads to a better state, or the number of actions reaches its upper bound (controlled by a parameter H). Selection: During above simulation process, if an improving action is met, it is selected and applied to the current state π, to get a new state π new. Otherwise, if no such action exists in the sampling pool, it seems difficult to gain further improvement within the current search area. At this time, the MDP jumps to a random state (using the method described in Section 3.3), which serves as a new starting state. The value of M as well as the elements of matrices W and Q are updated by back propagation as follows. At first, whenever an action is examined, M is increased by 1. Then, for each edge (b i, a i+1) which appears in an examined action, let Q biai+1 increase by 1. Finally, whenever a state π is converted to a better state π new by applying action a = (a 1, b 1, a 2, b 2, . . ., a k, b k, a k+1), for each edge (b i, a i+1), 1 ≤ i ≤ k, let: Where β is a parameter used to control the increasing rate of W biai+1. Notice that we update W biai+1 only when meeting a better state, since we want to avoid wrong estimations (even in a bad action which leads to a worse state, there may exist some good edges (b i, a i+1)). With this backpropagation process, the weight of the good edges would be increased to enhance its opportunity of being selected, thus the sampling process would be more and more targeted. W and Q are symmetric matrices, thus let W ai+1bi = W biai+1 and Q ai+1bi = Q biai+1 always. The MCTS iterates through the simulation, selection and back-propagation steps, until no improving action exists among the sampling pool. Then, the MDP jumps to a new state, and launches a new round of 2-opt local search and MCTS again. This process is repeated, until the allowed time (controlled by a parameter T) has been elapsed. Then, the best found state is returned as the final solution. To evaluate the performance of our MCTS algorithm, we program it in C language 1, and carry out experiments on a large number of public TSP instances. Notice that the reference algorithms are executed on different platforms (detailed below), being extremely difficult to fairly compare the run times. Therefore, we mainly make comparisons in terms of solution quality (achieved within reasonable runtime), and just list the run-times to roughly evaluate the efficiency of each algorithm. Currently there are two data sets widely used by the existing learning based TSP algorithms, i.e., Set 1 2, which is divided into three subsets, each containing 10,000 automatically generated 2D-Euclidean TSP instances, respectively with n = 20, 50, 100. Set 2 3: which contains 38 instances (with 51 ≤ n ≤ 318) extracted from the famous TSPLIB library . We also use these two data sets as benchmarks to evaluate our MCTS algorithm. As described in Section 3, MCTS relies on four hyper parameters (α, β, H and T). We choose α = 1, β = 10, H = 10n (n is the number of cities) as the default settings of the first three parameters. For parameter T used to control the termination condition, we set T = 75n milliseconds for each instance of set 1, and set T = n seconds for each instance of set 2, to ensure that the total time elapsed by MCTS remains reasonable w.r.t. the existing learning based algorithms. 4.3 ON DATA SET 1 Table 1 presents the obtained by MCTS on data set 1, with respect to the existing non-learned and learning-based algorithms. In the table, each line corresponds to an algorithm. Respectively, the first nine lines are non-learned algorithms, among which Concorde and Gurobi are two exact solvers, LKH3 is the currently best heuristic, and OR tools are optimization tools released by Google company . The following six lines are end-to-end ML models (the MCTS algorithm in did not provide detailed , thus being omitted), and the final eight lines are hybrid algorithms which use OR method after ML for post-optimization. For the columns, column 1 indicates the methods, while column 2 indicates the type of each algorithm (explained at the bottom of the table), columns 3-5 respectively give the average tour length, average optimality gap in percentage w.r.t. Concorde, and the total clock time used by each algorithm on all the 10,000 instances with n = 20. Similarly, columns 6-8, 9-11 respectively give the same information on the 10,000 instances with n = 50 and n = 100. All the except ours are directly taken from table 1 of (the order of the algorithms is slightly changed), while unavailable items are marked as "-". Notice that in the latest learning based algorithms and , the experiments were carried out either on a single GPU (Nvidia 1080Ti) or 32 instances in parallel on a 32 virtual CPU system (2 × Xeon E5-2630), and the run-time was recorded as the wall clock time used to solve the 10,000 test instances. Similarly, we also run 32 instances in parallel on a 32 virtual CPU system (2 × Intel Xeon Silver 4110 2.1GHz processor, each with eight cores), and report the wall clock time used to solve the 10,000 test instances. As shown in Table 1, the exact solvers and LKH3 obtain good on all the test instances, while the remaining six non-learned algorithms perform overall poorly. Among the learning based algorithms, in terms of solution quality the hybrid algorithms which combines ML and OR perform clearly much better than the end-to-end ML models, although much more computational times are required. Finally, compared to these existing methods, our MCTS algorithm performs quite well, which succeeds in matching or improving the best known solutions (reported by Concorde) on most of these instances, corresponding to an average gap of -0.0075%, -0.0212%, -0.0178% 4 respectively on the three subsets with n = 20, 50, 100. The computation times elapsed by MCTS remains reasonable (40 minutes for the 10,000 instances with n = 100), close to the latest (currently best) learning based algorithm . This set contains 38 instances (51 ≤ n ≤ 318) extracted from the TSPLIB. For each of these instances, we list in Table 2 the optimal (reported by Concorde within 1 hour for each instance), the reported by S2V-DQN and our MCTS algorithm (indicated in bold if reaching optimality). Notice that S2V-DQN did not clearly report the computational time, while n seconds is allowed by our MCTS to solve each instance with n cities (the 38 instances are executed sequentially, occupying a core of an Intel Xeon Silver 4110 2.1GHz processor). As shown in the table, S2V-DQN reaches optimality on only one instance (berlin52), corresponding to a large average gap (w.r.t. the optimal solutions) of 4.75%. For comparison, our MCTS succeeds in matching the optimal solutions on 28 instances, corresponding to a much smaller average gap (0.21%). Furthermore, MCTS dominates S2V-DQN on all these instances only except two instances pr226 (worse than S2V-DQN) and berlin52 (with equal ), clearly demonstrating its superiority over S2V-DQN. Table 2: Performance of MCTS on data set 2 compared to S2V-DQN ( The average gap of S2V-DQN vs. OPT is 4.75%, while the average gap of MCTS vs. OPT is 0.21%. Overall, we think the experimental on above two data sets clearly show the potential of our MCTS on the TSP. This paper newly develops a novel flexible paradigm for solving TSP, which combines OR and ML in a variable neighborhood search strategy, and achieves highly competitive performance with respect to the existing learning based TSP algorithms. However, how to combine ML and OR reasonably is still an open question, which deserves continuous investigations. In the future, we would try more new paradigms to better answer this question, and extend the work to other combinatorial optimization problems.

Represent the short summary for finding the corresponding scientific article (long)

This paper combines Monte Carlo tree search with 2-opt local search in a variable neighborhood mode to solve the TSP effectively.

represent the tldr for finding the corresponding scientific article

We propose a new combination of evolution strategy and deep reinforcement learning which takes the best of both worlds