Question: Represent this text

Question: For autonomous agents to successfully operate in the real world, the ability to anticipate future scene states is a key competence. In real-world scenarios, future states become increasingly uncertain and multi-modal, particularly on long time horizons. Dropout based Bayesian inference provides a computationally tractable, theoretically well grounded approach to learn different hypotheses/models to deal with uncertain futures and make predictions that correspond well to observations -- are well calibrated. However, it turns out that such approaches fall short to capture complex real-world scenes, even falling behind in accuracy when compared to the plain deterministic approaches. This is because the used log-likelihood estimate discourages diversity. In this work, we propose a novel Bayesian formulation for anticipating future scene states which leverages synthetic likelihoods that encourage the learning of diverse models to accurately capture the multi-modal nature of future scene states. We show that our approach achieves accurate state-of-the-art predictions and calibrated probabilities through extensive experiments for scene anticipation on Cityscapes dataset. Moreover, we show that our approach generalizes across diverse tasks such as digit generation and precipitation forecasting. The ability to anticipate future scene states which involves mapping one scene state to likely future states under uncertainty is key for autonomous agents to successfully operate in the real world e.g., to anticipate the movements of pedestrians and vehicles for autonomous vehicles. The future states of street scenes are inherently uncertain and the distribution of outcomes is often multi-modal. This is especially true for important classes like pedestrians. Recent works on anticipating street scenes BID13 BID9 BID23 do not systematically consider uncertainty. Bayesian inference provides a theoretically well founded approach to capture both model and observation uncertainty but with considerable computational overhead. A recently proposed approach BID6 BID10 uses dropout to represent the posterior distribution of models and capture model uncertainty. This approach has enabled Bayesian inference with deep neural networks without additional computational overhead. Moreover, it allows the use of any existing deep neural network architecture with minor changes. However, when the underlying data distribution is multimodal and the model set under consideration do not have explicit latent state/variables (as most popular deep deep neural network architectures), the approach of BID6; BID10 is unable to recover the true model uncertainty (see FIG0 and BID19). This is because this approach is known to conflate risk and uncertainty BID19. This limits the accuracy of the models over a plain deterministic (non-Bayesian) approach. The main cause is the data log-likelihood maximization step during optimization -for every data point the average likelihood assigned by all models is maximized. This forces every model to explain every data point well, pushing every model in the distribution to the mean. We address this problem through an objective leveraging synthetic likelihoods BID26 BID21 which relaxes the constraint on every model to explain every data point, thus encouraging diversity in the learned models to deal with multi-modality. In this work: 1. We develop the first Bayesian approach to anticipate the multi-modal future of street scenes and demonstrate state-of-the-art accuracy on the diverse Cityscapes dataset without compromising on calibrated probabilities, 2. We propose a novel optimization scheme for dropout based Bayesian inference using synthetic likelihoods to encourage diversity and accurately capture model uncertainty, 3. Finally, we show that our approach is not limited to street scenes and generalizes across diverse tasks such as digit generation and precipitation forecasting. Bayesian deep learning. Most popular deep learning models do not model uncertainty, only a mean model is learned. Bayesian methods BID15 BID18 on the other hand learn the posterior distribution of likely models. However, inference of the model posterior is computationally expensive. In BID6 this problem is tackled using variational inference with an approximate Bernoulli distribution on the weights and the equivalence to dropout training is shown. This method is further extended to convolutional neural networks in BID5. In BID10 this method is extended to tackle both model and observation uncertainty through heteroscedastic regression. The proposed method achieves state of the art on segmentation estimation and depth regression tasks. This framework is used in BID2 to estimate future pedestrian trajectories. In contrast, BID22 propose a (unconditional) Bayesian GAN framework for image generation using Hamiltonian Monte-Carlo based optimization with limited success. Moreover, conditional variants of GANs BID17 are known to be especially prone to mode collapse. Therefore, we choose a dropout based Bayesian scheme and improve upon it through the use of synthetic likelihoods to tackle the issues with model uncertainty mentioned in the introduction. Structured output prediction. Stochastic feedforward neural networks (SFNN) and conditional variational autoencoders (CVAE) have also shown success in modeling multimodal conditional distributions. SFNNs are difficult to optimize on large datasets BID25 due to the binary stochastic variables. Although there has been significant effort in improving training efficiency BID20 BID7, success has been partial. In contrast, CVAEs BID24 assume Gaussian stochastic variables, which are easier to optimize on large datasets using the re-parameterization trick. CVAEs have been successfully applied on a large variety of tasks, include conditional image generation BID1, next frame synthesis BID28, video generation BID0 BID4, trajectory prediction BID12 among others. The basic CVAE framework is improved upon in BID3 through the use of a multiple-sample objective. However, in comparison to Bayesian methods, careful architecture selection is required and experimental evidence of uncertainty calibration is missing. Calibrated uncertainties are important for autonomous/assisted driving, as users need to be able to express trust in the predictions for effective decision making. Therefore, we also adopt a Bayesian approach over SFNN or CVAE approaches. Anticipation future scene scenes. In BID13 ) the first method for predicting future scene segmentations has been proposed. Their model is fully convolutional with prediction at multiple scales and is trained auto-regressively. BID9 improves upon this through the joint prediction of future scene segmentation and optical flow. Similar to BID13 a fully convolutional model is proposed, but the proposed model is based on the Resnet-101 BID8 and has a single prediction scale. More recently, BID14 has extended the model of BID13 to the related task of future instance segmentation prediction. These methods achieve promising and establish the competence of fully convolutional models. In BID23 a Convolutional LSTM based model is proposed, further improving short-term over BID9. However, fully convolutional architectures have performed well at a variety of related tasks, including segmentation estimation BID29 BID30, RGB frame prediction BID16 BID0 among others. Therefore, we adopt a standard ResNet based fully-convolutional architecture, while providing a full Bayesian treatment. We phrase our models in a Bayesian framework, to jointly capture model (epistemic) and observation (aleatoric) uncertainty BID10. We begin with model uncertainty. Let x ∈ X be the input (past) and y ∈ Y be the corresponding outcomes. Consider f: x → y, we capture model uncertainty by learning the distribution p(f |X, Y) of generative models f, likely to have generated our data {X, Y}. The complete predictive distribution of outcomes y is obtained by marginalizing over the posterior distribution, DISPLAYFORM0 However, the integral in FORMULA0 is intractable. But, we can approximate it in two steps BID6. First, we assume that our models can be described by a finite set of variables ω. Thus, we constrain the set of possible models to ones that can be described with ω. Now, is equivalently, DISPLAYFORM1 Second, we assume an approximating variational distribution q(ω) of models which allows for efficient sampling. This in the approximate distribution, DISPLAYFORM2 For convolutional models, BID5 proposed a Bernoulli variational distribution defined over each convolutional patch. The number of possible models is exponential in the number of patches. This number could be very large, making it difficult optimize over this very large set of models. In contrast, in our approach, the number possible models is exponential in the number of weight parameters, a much smaller number. In detail, we choose the set of convolutional kernels and the biases {( DISPLAYFORM3 of our model as the set of variables ω. Then, we define the following novel approximating Bernoulli variational distribution q(ω) independently over each element w i,j k,k (correspondingly b k) of the kernels and the biases at spatial locations {i, j}, DISPLAYFORM4 Note, denotes the hadamard product, M k are tuneable variational parameters, z i,j k,k ∈ Z K are the independent Bernoulli variables, p K is a probability tensor equal to the size of the (bias) layer, |K| (|K |) is the number of kernels in the current (previous) layer. Here, p K is chosen manually. Moreover, in contrast to BID5, the same (sampled) kernel is applied at each spatial location leading to the detection of the same features at varying spatial locations. Next, we describe how we capture observation uncertainty. Observation uncertainty can be captured by assuming an appropriate distribution of observation noise and predicting the sufficient statistics of the distribution BID10. Here, we assume a Gaussian distribution with diagonal covariance matrix at each pixel and predict the mean vector µ i,j and co-variance matrix σ i,j of the distribution. In detail, the predictive distribution of a generative model draw fromω ∼ q(ω) at a pixel position {i, j} is, DISPLAYFORM0 We can sample from the predictive distribution p(y|x) by first sampling the weight matrices ω from and then sampling from the Gaussian distribution in. We perform the last step by the linear transformation of a zero mean unit diagonal variance Gaussian, ensuring differentiability, DISPLAYFORM1 where,ŷ i,j is the sample drawn at a pixel position {i, j} through the liner transformation of z (a vector) with the predicted mean µ i,j and variance σ i,j. In case of street scenes, y i,j is a class-confidence vector and sample of final class probabilities is obtained by pushingŷ i,j through a softmax. For a good variational approximation, our approximating variational distribution of generative models q(ω) should be close to the true posterior p(ω|X, Y). Therefore, we minimize the KL divergence between these two distributions. As shown in BID6 a); BID10 the KL divergence is given by (over i.i.d data points), DISPLAYFORM0 The log-likelihood term at the right of FORMULA7 considers every model for every data point. This imposes the constraint that every data point must be explained well by every model. However, if the data distribution (x, y) is multi-modal, this would push every model to the mean of the multi-modal distribution (as in FIG0 where only way for models to explain both modes is to converge to the mean). This discourages diversity in the learned modes. In case of multi-modal data, we would not be able to recover all likely models, thus hindering our ability to fully capture model uncertainty. The models would be forced to explain the data variation as observation noise BID19, thus conflating model and observation uncertainty. We propose to mitigate this problem through the use of an approximate objective using synthetic likelihoods BID26 BID21 ) -obtained from a classifier. The classifier estimates the likelihood based on whether the modelsω ∼ q(ω) explain (generate) data samples likely under the true data distribution p(y|x). This removes the constraint on models to explain every data point -it only requires the explained (generated) data points to be likely under the data distribution. Thus, this allows modelsω ∼ q(ω) to be diverse and deal with multi-modality. Next, we reformulate the KL divergence estimate of to a likelihood ratio form which allows us to use a classifier to estimate (synthetic) likelihoods, (also see Appendix), DISPLAYFORM1 In the second step of FORMULA8, we divide and multiply the probability assigned to a data sample by a model p(y|x, ω) by the true conditional probability p(y|x) to obtain a likelihood ratio. We can estimate the KL divergence by equivalently estimating this ratio rather than the true likelihood. In order to (synthetically) estimate this likelihood ratio, let us introduce the variable θ to denote, p(y|x, θ = 1) the probability assigned by our model ω to a data sample (x, y) and p(y|x, θ = 0) the true probability of the sample. Therefore, the ratio in the last term of FORMULA8 is, DISPLAYFORM2 In the last step of FORMULA9 we use the fact that the events θ = 1 and θ = 0 are mutually exclusive. We can approximate the ratio p(θ=1|x,y)1−p(θ=1|x,y) by jointly learning a discriminator D(x,ŷ) that can distinguish between samples of the true data distribution and samples (x,ŷ) generated by the model ω, which provides a synthetic estimate of the likelihood, and equivalently integrating directly over (x,ŷ), DISPLAYFORM3 Note that the synthetic likelihood DISPLAYFORM4 is independent of any specific pair (x, y) of the true data distribution (unlike the log-likelihood term in FORMULA7), its value depends only upon whether the generated data point (x,ŷ) by the model ω is likely under the true data distribution p(y|x). Therefore, the models ω have to only generate samples (x,ŷ) likely under the true data distribution. The models need not explain every data point equally well. Therefore, we do not push the models ω to the mean, thus allowing them to be diverse and allowing us to better capture uncertainty. Empirically, we observe that a hybrid log-likelihood term using both the log-likelihood terms of FORMULA0 and FORMULA7 with regularization parameters α and β (with α ≥ β) stabilizes the training process, DISPLAYFORM5 Note that, although we do not explicitly require the posterior model distribution to explain all data points, due to the exponential number of models afforded by dropout and the joint optimization (min-max game) of the discriminator, empirically we see very diverse models explaining most data points. Moreover, empirically we also see that predicted probabilities remain calibrated. Next, we describe the architecture details of our generative models ω and the discriminator D(x,ŷ). The architecture of our ResNet based generative models in our model distribution q(ω) is shown in FIG1. The generative model takes as input a sequence of past segmentation class-confidences s p, the past and future vehicle odometry o p, o f (x = {s p, o p, o f}) and produces the class-confidences at the next time-step as output. The additional conditioning on vehicle odometry is because the sequences are recorded in frame of reference of a moving vehicle and therefore the future observed sequence is dependent upon the vehicle trajectory. We use recursion to efficiently predict a sequence of future scene segmentations y = {s f}. The discriminator takes as input s f and classifies whether it was produced by our model or is from the true data distribution. In detail, generative model architecture consists of a fully convolutional encoder-decoder pair. This architecture builds upon prior work of BID13 BID9, however with key differences. In BID13, each of the two levels of the model architecture consists of only five convolutional layers. In contrast, our model consists of one level with five convolutaional blocks. The encoder contains three residual blocks with max-pooling in between and the decoder consists of a residual and a convoluational block with up-sampling in between. We double the size of the blocks following max-pooling in order to preserve resolution. This leads to a much deeper model with fifteen convolutional layers, with constant spatial convolutional kernel sizes. This deep model with pooling creates a wide receptive field and helps better capture spatio-temporal dependencies. The residual connections help in the optimization of such a deep model. Computational resources allowing, it is possible to add more levels to our model. In BID9 a model is considered which uses a Res101-FCN as an encoder. Although this model has significantly more layers, it also introduces a large amount of pooling. This leads to loss of resolution and spatial information, hence degrading performance. Our discriminator model consists of six convolutional layers with max-pooling layers in-between, followed by two fully connected layers. Finally, in Appendix E we provide layer-wise details and discuss the reduction of number of models in q(ω) through the use of Weight Dropout for our architecture of generators. Next, we evaluate our approach on MNIST digit generation and street scene anticipation on Cityscapes. We further evaluate our model on 2D data FIG0 ) and precipitation forecasting in the Appendix. Here, we aim to generate the full MNIST digit given only the lower left quarter of the digit. This task serves as an ideal starting point as in many cases there are multiple likely completions given the lower left quarter digit, e.g. 5 and 3. Therefore, the learned model distribution q(ω) should contain likely models corresponding to these completions. We use a fully connected generator with 6000-4000-2000 hidden units with 50% dropout probability. The discriminator has 1000-1000 hidden units with leaky ReLU non-linearities. We set β = 10 −4 for the first 4 epochs and then reduce it to 0, to provide stability during the initial epochs. We compare our synthetic likelihood based approach (Bayes-SL) with, 1. A non-Bayesian mean model, 2. A standard Bayesian approach (Bayes-S), 3. A Conditional Variational Autoencoder (CVAE) (architecture as in BID24). As evaluation metric we consider (oracle) Top-k% accuracy BID12. We use a standard Alex-Net based classifier to measure if the best prediction corresponds to the ground-truth class -identifies the correct mode -in TAB1 (right) over 10 splits of the MNIST test-set. We sample 10 models from our learned distribution and consider the best model. We see that our Bayes-SL performs best, even outperforming the CVAE model. In the qualitative examples in TAB1 (left), we see that generations from modelsω ∼ q(ω) sampled from our learned model distribution corresponds to clearly defined digits (also in comparision to FIG2 in BID24). In contrast, we see that the Bayes-S model produces blurry digits. All sampled models have been pushed to the mean and shows little advantage over a mean model. Next, we evaluate our apporach on the Cityscapes dataset -anticipating scenes more than 0.5 seconds into the future. The street scenes already display considerable multi-modality at this time-horizon. Evaluation metrics and baselines. We use PSPNet BID30 to segment the full training sequences as only the 20 th frame has groundtruth annotations. We always use the annotated 20 th frame of the validation sequences for evaluation using the standard mean Intersection-over-Union (mIoU) and the per-pixel (negative) conditional log-likelihood (CLL) metrics. We consider the following baselines for comparison to our Resnet based (architecture in FIG1) Bayesian (Bayes-WD-SL) model with weight dropout and trained using synthetic likelihoods: 1. Copying the last seen input; 2. A non-Bayesian (ResG-Mean) version; 3. A Bayesian version with standard patch dropout (Bayes-S); 4. A Bayesian version with our weight dropout (Bayes-WD). Note that, combination of ResG-Mean with an adversarial loss did not lead to improved (similar observations made in BID13). We use grid search to set the dropout rate (in) to 0.15 for the Bayes-S and 0.20 for Bayes-WD(-SL) models. We set α, β = 1 for our Bayes-WD-SL model. We train all models using Adam BID11 for 50 epochs with batch size 8. We use one sample to train the Bayesian methods as in BID5 and use 100 samples during evaluation. Comparison to state of the art. We begin by comparing our Bayesian models to state-of-the-art methods BID13; BID23 in TAB0. We use the mIoU metric and for a fair comparison consider the mean (of all samples) prediction of our Bayesian models. We alwyas compare to the groundtruth segmentations of the validation set. However, as all three methods use a slightly different semantic segmentation algorithm (Table 2) to generate training and input test data, we include the mIoU achieved by the Last Input of all three methods (see Appendix C for using Dialation 10). Similar to Luc et al. FORMULA0 we fine-tune (ft) to predict at 3 frame intervals for better performance at +0.54sec. Our Bayes-WD-SL model outperforms baselines and improves on prior work by 2.8 mIoU at +0.06sec and 4.8 mIoU/3.4 mIoU at +0.18sec/+0.54sec respectively. Our Bayes-WD-SL model also obtains higher relative gains in comparison to BID13 with respect the Last Input Baseline. These validate our choice of model architecture and show that our novel approach clearly outperforms the state-of-the-art. The performance advantage of Bayes-WD-SL over Bayes-S shows that the ability to better model uncertainty does not come at the cost of lower mean performance. However, at larger time-steps as the future becomes increasingly uncertain, mean predictions (mean of all likely futures) drift further from the ground-truth. Therefore, next we evaluate the models on their (more important) ability to capture the uncertainty of the future. Evaluation of predicted uncertainty. Next, we evaluate whether our Bayesian models are able to accurately capture uncertainity and deal with multi-modal futures, upto t + 10 frames (0.6 seconds) in TAB1. We consider the mean of (oracle) best 5% of predictions BID12 ) of our Bayesian models to evaluate whether the learned model distribution q(ω) contains likely models corresponding to the groundtruth. We see that the best predictions considerably improve over the mean predictionsshowing that our Bayesian models learns to capture uncertainity and deal with multi-modal futures. Quantitatively, we see that the Bayes-S model performs worst, demonstrating again that standard dropout BID10 struggles to recover the true model uncertainity. The use of weight dropout improves the performance to the level of the ResG-Mean model. Finally, we see that our Bayes-WD-SL model performs best. In fact, it is the only Bayesian model whose (best) performance exceeds that of the ResG-Mean model (also outperforming state-of-the-art), demonstrating the effectiveness of synthetic likelihoods during training. In FIG3 we show examples comparing the best prediction of our Bayes-WD-SL model and ResG-Mean at t + 9. The last row highlights the differences between the predictions -cyan shows areas where our Bayes-WD-SL is correct and ResG-Mean is wrong, red shows the opposite. We see that our Bayes-WD-SL performs better at classes like cars and pedestrians which are harder to predict (also in comparison to TAB3 in BID13). In FIG4, we show samples from randomly sampled modelsω ∼ q(ω), which shows correspondence to the range of possible movements of bicyclists/pedestrians. Next, we further evaluate the models with the CLL metric in TAB1. We consider the mean predictive distributions up to t + 10 frames. We see that the Bayesian models outperform the ResG-Mean model significantly. In particular, we see that our Bayes-WD-SL model performs the best, demonstrating that the learned model and observation uncertainty corresponds to the variation in the data. Comparison to a CVAE baseline. As there exists no CVAE BID24 based model for future segmentation prediction, we construct a baseline as close as possible to our Bayesian models Groundtruth, t + 9 ResG-Mean, t + 9 Bayes-WD-SL, t + 9 Comparison Sample #1, t + 9 Sample #2, t + 9 Sample #3, t + 9 Sample #4, t + 9 based on existing CVAE based models for related tasks BID0 BID28.Existing CVAE based models BID0 BID28 ) contain a few layers with Gaussian input noise. Therefore, for a fair comparison we first conduct a study in TAB2 to find the layers which are most effective at capturing data variation. We consider Gaussian input noise applied in the first, middle or last convolutional blocks. The noise is input dependent during training, sampled from a recognition network (see Appendix). We observe that noise in the last layers can better capture data variation. This is because the last layers capture semantically higher level scene features. Overall, our Bayesian approach (Bayes-WD-SL) performs the best. This shows that the CVAE model is not able to effectively leverage Gaussian noise to match the data variation. Uncertainty calibration. We further evaluate predicted uncertainties by measuring their calibration -the correspondence between the predicted probability of a class and the frequency of its occurrence in the data. As in BID10, we discretize the output probabilities of the mean predicted distribution into bins and measure the frequency of correct predictions for each bin. We report the at t + 10 frames in FIG5. We observe that all Bayesian approaches outperform the ResG-Mean and CVAE versions. This again demonstrates the effectiveness of the Bayesian approaches in capturing uncertainty. We propose a novel approach for predicting real-world semantic segmentations into the future that casts a convolutional deep learning approach into a Bayesian formulation. One of the key contributions is a novel optimization scheme that uses synthetic likelihoods to encourage diversity and deal with multi-modal futures. Our proposed method shows state of the art performance in challenging street scenes. More importantly, we show that the probabilistic output of our deep learning architecture captures uncertainty and multi-modality inherent to this task. Furthermore, we show that the developed methodology goes beyond just street scene anticipation and creates new opportunities to enhance high performance deep learning architectures with principled formulations of Bayesian inference. KL divergence estimate. Here, we provide a detailed derivation of. Starting from, we have, DISPLAYFORM0 Multiplying and dividing by p(y|x), the true probability of occurance, DISPLAYFORM1 Using q(ω) dω = 1, DISPLAYFORM2 As log p(y|x)d(x, y) is independent of ω, the variables we are optmizing over, we have, DISPLAYFORM3 APPENDIX B. ON SIMPLE MULTI-MODAL 2D DATA. We show on simple multi-modal 2d data as in the motivating example in the introduction. The data consists of two parts: x ∈ [−10, 0] we have y = 0 and x ∈ we have y = (−0.3, 0.3).The set of models under consideration is a two hidden layer neural network with 256-128 neurons with 50% dropout. We show 10 randomly sampled models fromω ∼ q(ω) learned by the Bayes-S approach in FIG6 and our Bayes-SL approach in FIG7 (with α = 1, β = 0). We assume constant observation uncertainty (=1). We clearly see that our Bayes-SL learns models which cover both modes, while all the models learned by Bayes-S fit to the mean. Clearly showing that our approach can better capture model uncertainty. First, we provide additional training details of our Bayes-WD-SL in TAB3. Generator learning rate 1 × 10 DISPLAYFORM0 Discriminator learning rate 1 × 10 DISPLAYFORM1 # Generator updates per iteration 1 # Discriminator updates per iteration 1 Table 6: Additional Comparison to BID13 using the same Dialation 10 approach to generate training segmentations. Note: Fine Tuned (ft) means both approaches are trained to predict at intervals of three frames (0.18 seconds).Second, we provide additional evaluation on street scenes. In Section 4.2 TAB0 we use a PSPNet to generate training segmentations for our Bayes-WD-SL model to ensure fair comparison with the state-of-the-art BID23. However, the method of BID13 uses a weaker Dialation 10 approach to generate training segmentations. Note that our Bayes-WD-SL model already obtains higher gains in comparison to BID13 with respect the Last Input Baseline, e.g. at +0.54sec, 47.8 -36.9 = 10.9 mIoU translating to 29.5% gain over the Last Input Baseline of BID13 versus 51.2 -38.3 = 12.9 mIoU translating to 33.6% gain over the Last Input Baseline of our Bayes-WD-SL model in TAB0. But for fairness, here we additionally include in Table 6 using the same Dialation 10 approach to generate training segmentations. We observe that our Bayes-WD-SL model beats the model of BID13 in both short-term (+0.18 sec) and long-term predictions (+0.54 sec). Furthermore, we see that the mean of the Top 5% of the predictions of Bayes-WD-SL leads to much improved over mean predictions. This again confirms the ability of our Bayes-WD-SL model to capture uncertainty and deal with multi-modal futures. APPENDIX D. ON HKO PRECIPITATION FORECASTING DATA.The HKO radar echo dataset consists of weather radar intensity images. We use the train/test split used in Xingjian et al. FORMULA0; BID3. Each sequence consists of 20 frames. We use 5 frames as input and 15 for prediction. Each frame is recorded at an interval of 6 minutes. Therefore, they display considerable uncertainty. We use the same network architecture as used for street scene segmentation Bayes-WD-SL FIG1 and with α = 5, β = 1), but with half the convolutional filters at each level. We compare to the following baselines: 1. A deterministic model (ResG-Mean), 2. A Bayesian model with weight dropout. We report the (oracle) Top-10% scores (best 1 of 10), over the following metrics BID27 BID3 ), 1. Rainfall-MSE: Rainfall mean squared error, 2. CSI: Critical success index, 3. FAR: False alarm rate, 4. POD: Probability of detection, and 5. Correlation, in Table 7, Note, that; BID3 reports only scores over mean of all samples. Our ResG-Mean model outperforms these state of the art methods, showing the versatility of our model architecture. Our Bayes-WD-SL can outperform the strong ResG-Mean baseline again showing that it learns to capture uncertainty (see FIG0). In comparison, the Bayes-WD baseline struggles to outperform the ResG-Mean baseline. We further compare the calibration our Bayes-SL model to the ResG-Mean model in FIG8. We plot the predicted intensity to the true mean observed intensity. Table 7: Evaluation on HKO radar image sequences. is stark in the high intensity region. The RegG-Mean model deviates strongly from the diagonal in this region -it overestimates the radar intensity. In comparison, we see that our Bayes-WD-SL approach stays closer to the diagonal. These again show that our synthetic likelihood based approach leads to more accurate predictions while not compromising on calibration. Observation Groundtruth Observation Prediction t − 5 t − 3 t − 1 t t + 2 t + 4 t + 6 t + 8 t + 10 t + 12 t + 13 t + 14 Here, we provide layer-wise details of our generative and discriminative models in TAB6. We provide layer-wise details of the recognition network of the CVAE baseline used in TAB2 (in the main paper). Finally, in TAB0 we show the difference in the number of possible models using our weight based variational distribution 4 (weight dropout) versus the patch based variational distribution (patch dropout) proposed in BID5. The number of patches is calculated using the formula, DISPLAYFORM0 because we use convolutional stride 1, padding to ensure same output resolution and each patch is dropped out (in BID5) independently for each convolutional filter. The number of weight parameters is given by the formula,Filter size × # Input Convolutional Filters × # Output Convolutional Filters + # Bias. TAB0 shows that our weight dropout scheme in significantly lower number of parameters compared to patch dropout BID5.Details of our generative model. We show the layer wise details in TAB6 Details of our discriminator model. We show the layer wise details in TAB8.Details of the recognition model used in the CVAE baseline. We show the layer wise details in TAB0. 36,700,406 11,699,192 36,700,406 11,699,192 Table 11: The difference in the number of possible models using our Weight dropout scheme versus patch dropout BID5 (Appendix D) 18,350,203 5,849,596 68.1% TAB0: Overview of the variational parameters using our Weight dropout scheme versus patch dropout BID5 of both architectures for street scene and precipitation forecasting

Dropout based Bayesian inference is extended to deal with multi-modality and is evaluated on scene anticipation tasks.

represent the short summary for finding the corresponding scientific article (long)

Given A DQN and DDPG hybrid algorithm is proposed to deal with the discrete-continuous hybrid action space., a positive would be Most existing deep reinforcement learning (DRL) frameworks consider action spaces that are either discrete or continuous space. Motivated by the project of design Game AI for King of Glory (KOG), one the world’s most popular mobile game, we consider the scenario with the discrete-continuous hybrid action space. To directly apply existing DLR frameworks, existing approaches either approximate the hybrid space by a discrete set or relaxing it into a continuous set, which is usually less efficient and robust. In this paper, we propose a parametrized deep Q-network (P-DQN) for the hybrid action space without approximation or relaxation. Our algorithm combines DQN and DDPG and can be viewed as an extension of the DQN to hybrid actions. The empirical study on the game KOG validates the efficiency and effectiveness of our method. In recent years, the exciting field of deep reinforcement learning (DRL) have witnessed striking empirical achievements in complicated sequential decision making problems that are once believed unsolvable. One active area of the application of DRL methods is to design artificial intelligence (AI) for games. The success of DRL in the game of Go provides a promising methodology for game AI. In addition to the game of Go, DRL has been widely used in other games such as Atari BID19, Robot Soccer BID8 BID17, and Torcs ) to achieve super-human performances. However, most existing DRL methods only handle the environments with actions chosen from a set which is either finite and discrete (e.g., Go and Atari) or continuous (e.g. MuJoCo and Torcs) For example, the algorithms for discrete action space include deep Q-network (DQN) BID18, Double DQN , A3C BID20; the algorithms for continuous action space include deterministic policy gradients (DPG) BID29 and its deep version DDPG.Motivated by the applications in Real Time Strategic (RTS) games, we consider the reinforcement learning problem with a discrete-continuous hybrid action space. Different from completely discrete or continuous actions that are widely studied in the existing literature, in our setting, the action is defined by the following hierarchical structure. We first choose a high level action k from a discrete set {1, 2, · · ·, K}; upon choosing k, we further choose a low level parameter x k ∈ X k which is associated with the k-th high level action. Here X k is a continuous set for all k ∈ {1, . . ., K}.1 Therefore, we focus on a discrete-continuous hybrid action space A = (k, x k) x k ∈ X k for all 1 ≤ k ≤ K.To apply existing DRL approaches on this hybrid action space, two straightforward ideas include:• Approximate A by an finite discrete set. We could approximate each X k by a discrete subset, which, however, might lose the natural structure of X k. Moreover, when X k is a region in the Euclidean space, establishing a good approximation usually requires a huge number discrete actions.• Relax A into a continuous set. To apply existing DRL framework with continuous action spaces, BID8 define the following approximate space DISPLAYFORM0 where F k ⊆ R. Here f 1, f 2,..., f K is used to select the discrete action either deterministically (by picking arg max i f i) or randomly (with probability softmax(f)). Compared with the original action space A, A might significantly increases the complexity of the action space. Furthermore, continuous relaxation can also lead to unnecessary confusion by over-parametrization. For example, (1, 0, · · ·, 0, x 1, x 2, x 3, · · ·, x K) ∈ A and (1, 0, · · ·, 0, x 1, x 2, x 3, · · ·, x K) ∈ A indeed represent the same action (1, x 1) in the original space A.In this paper, we propose a novel DRL framework, namely parametrized deep Q-network learning (P-DQN), which directly work on the discrete-continuous hybrid action space without approximation or relaxation. Our method can be viewed as an extension of the famous DQN algorithm to hybrid action spaces. Similar to deterministic policy gradient methods, to handle the continuous parameters within actions, we first define a deterministic function which maps the state and each discrete action to its corresponding continuous parameter. Then we define a action-value function which maps the state and finite hybrid actions to real values, where the continuous parameters are obtained from the deterministic function in the first step. With the merits of both DQN and DDPG, we expect our algorithm to find the optimal discrete action as well as avoid exhaustive search over continuous action parameters. To evaluate the empirical performances, we apply our algorithm to King of Glory (KOG), which is one of the most popular online games worldwide, with over 200 million active users per month. KOG is a multi-agent online battle arena (MOBA) game on mobile devices, which requires players to take hybrid actions to interact with other players in real-time. Empirical study indicates that P-DQN is more efficient and robust than BID8's method that relaxes A into a continuous set and applies DDPG. In reinforcement learning, the environment is usually modeled by a Markov decision process (MDP) M = {S, A, p, p 0, γ, r}, where S is the state space, A is the action space, p is the Markov transition probability distribution, p 0 is the probability distribution of the initial state, r(s, a) is the reward function, and γ ∈ is the discount factor. An agent interacts with the MDP sequentially as follows. At the t-th step, suppose the MDP is at state s t ∈ S and the agent selects an action a t ∈ A, then the agent observe an immediate reward r(s t, a t) and the next state s t+1 ∼ p(s t+1 |s t, a t). A stochastic policy π maps each state to a probability distribution over A, that is, π(a|s) is defined as the probability of selecting action a at state s. Whereas a deterministic µ: S → A maps each state to a particular action in A. Let R t = j≥t γ j−t r(s j, a j) be the cumulative discounted reward starting from time-step t. We define the state-value function and the action-value function of policy π as V π = E(R t |S t = s; π) and Q π (s, a) = E(R t |S 0 = s, A 0 = a; π), respectively. Moreover, we define the optimal state-and action-value functions as V π = sup π V π and Q * = sup π Q π, respectively, where the supremum is taken over all possible policies. The goal of the agent is to find a policy the maximizes the expected total discounted reward J(π) = E(R 0 |π), which is can be achieved by estimating Q *. Broadly speaking, reinforcement learning algorithms can be categorized into two classes: value-based methods and policy-based methods. Value-based methods first estimate Q * and then output the greedy policy with respect to that estimate. Whereas policy-based methods directly optimizes J(π) as a functional of π. The Q-learning algorithm BID36 ) is based on the Bellman equation DISPLAYFORM0 which has Q * as the unique solution. In the tabular setting, the algorithm updates the Q-function by iteratively applying the sample counterpart of the Bellman equation DISPLAYFORM1 where α > 0 is the stepsize and s is the next state observed given the current state s and action a. However, when the state space S is so large that it is impossible to store all the states in memory, function approximation for Q * is applied. Deep Q-Networks (DQN) BID19 approximates Q * using a neural network Q(s, a; w) ≈ Q(s, a), where w is the network weights. In the t-th iteration, the DQN updates the parameter using the gradient of the least squares loss function DISPLAYFORM2 In practice, DQN is trained with techniques such as experience replay and asynchronous stochastic gradient descent methods BID20 ) which enjoy great empirical success. In addition to the value-based methods, the policy-based methods directly models the optimal policy. In specific, let π be any policy. We write p t (·|s; π) as the distribution of S t given S 1 = s with actions executed according to policy π. We define the discounted probability distribution ρ π by DISPLAYFORM3 Then the objective of policy-based methods is to find a policy that maximizes the expected reward DISPLAYFORM4 Let π θ be a stochastic policy parametrized by θ ∈ Θ. For example, π θ could be a neural network in which the last layer is a softmax layer with |A| neurons. The stochastic gradient methods aims at finding a parameter θ that maximizes J(π θ) via gradient descent. The stochastic policy gradient theorem BID33 states that DISPLAYFORM5 3)The policy gradient algorithm iteratively updates θ using estimates of (2.3). For example, the REINFORCE algorithm BID37 updates θ using ∇ θ log π θ (a t |s t) · r t. Moreover, the actor-critic methods use another neural network Q(s, a; w) to estimate the value function Q π θ (s, a) associated to policy π θ. This algorithm combines the value-based and policy-based perspectives together, and is recently used to achieve superhuman performance in the game of Go BID31. When the action space is continuous, value-based methods will no longer be computationally tractable because of taking maximum over the action space A in (2.2), which in general cannot be computed efficiently. The reason is that the neural network Q(s, a; w) is nonconvex when viewed as a function of a; max a∈A Q(s, a; w) is the global minima of a nonconvex function, which is NP-hard to obtain in the worst case. To resolve this issue, the continuous Q-learning BID6 rewrite the action value function as Q(s, a) = V (s) + A(s, a), where V (s) is the state value function and A(s, a) is the advantage function that encodes the relative advantage of each action. These functions are approximated by neural networks V (s; θ V) and A(s, a; θ A), respectively, where θ V and θ A are network weights. The action value function is given by DISPLAYFORM0 Then in the t-th iteration, the continuous Q-learning updates θ v and θ a by taking a gradient step using the least squares loss function DISPLAYFORM1 Moreover, it is also possible to adapt policy-based methods to continuous action spaces by considering deterministic policies. Let µ θ: S → A be a deterministic policy. Similar to (2.3), the deterministic policy gradient (DPG) theorem BID29 states that DISPLAYFORM2 Furthermore, this deterministic version of the policy gradient theorem can be viewed as the limit of (2.3) with the variance of π θ going to zero. Based on (2.4), the DPG algorithm BID29 and the deep deterministic policy gradient (DDPG) algorithm are proposed. General reinforcement learning There is a huge body of literature in reinforcement learning, we refer readers to textbooks by; Szepesvári FORMULA6 for detailed introduction. Combined with the recent advancement of deep learning BID5, deep reinforcement learning becomes a blossoming field of research with a plethora of new algorithms which achieve surprising empirical success in a variety of applications that are previously considered extremely difficult and challenging. Finite discrete action space methods For reinforcement learning problems with finite action spaces, BID18 propose the DQN algorithm, which first combines the deep neural networks with the classical Q-learning algorithm BID36. A variety of extensions are proposed to improve DQN, including Double DQN, dueling DQN BID35, bootstrap DQN BID23, asynchronous DQN BID20, and averaged-.In terms of policy-based methods, BID33 propose the REINFORCE algorithm, which is the basic form of policy gradient. An important extension is the actor-critic method BID13, whose asynchronous deep version A3C BID20 produces the stateof-the-art performances on the Arcade Learning Environment (ALE) benchmark BID1.Continuous action space methods Moreover, for DRL on continuous action spaces, BID29 proposes the deterministic policy gradient algorithm and deterministic actor-critic algorithms. This work is further extended by, which propose the DDPG algorithm, which is an model-free actor critic algorithm using deep neural networks to parametrize the policies. A related line of work is policy optimization methods, which improve the policy gradient method using novel optimization techniques. These methods include natural gradient descent BID12, trust region optimization BID27, proximal gradient descent BID28, mirror descent BID21, and entropy regularization BID22.Hybrid actions A related body of literature is the recent work on reinforcement learning with a structured action space, which contains finite actions each parametrized by a continuous parameter. To handle such parametrized actions, BID8 applies the DDPG algorithm on the relaxed action space directly, and BID17 proposes a learning framework updating the parameters for discrete actions and continuous parameters alternately. Game AI Recently remarkable advances have been made in building AI bots for computer games using deep reinforcement learning. These games include Atari Games, a collection of video games, Texas Hold'em, a multi-player poker game, and Doom, a first-person shooter game. See BID18; BID9; BID15; BID2 for details and see BID11 for a comprehensive survey. More notably, the computer Go agent AlphaGo achieves super-human performances by defeating the human world champion Lee Sedol. Two more complicated class of games are the real-time strategy (RTS) games and MOBA games. These are multi-agent games which involves searching within huge state and action spaces that are possibly continuous. Due to the difficulty of these problems, current research for these games are rather inadequate with most existing work consider specific scenarios instead of the full-fledged RTS or MOBA games. See, e.g., BID4; BID24 for an recent attempt on applying DRL methods to RTS games. This section introduces the proposed framework to handle the application with hybrid discretecontinuous action space. We consider a MDP with a parametrized action space A, which consists of K discrete actions each associated with a continuous parameter. In specific, we assume that any action a ∈ A can be written as a = (k, x k), where k ∈ {1, . . ., K} is the discrete action, and x k ∈ X k is a continuous parameter associated with the k-th discrete action. Thus action a is a hybrid of discrete and continuous components with the value of the continuous action determined after the discrete action is chosen. Then the parametrized action space A can be written as DISPLAYFORM0 In the sequel, we denote {1, . . ., K} by [K] for short. For the action space A in (4.1), we denote the action value function by Q(s, a) = Q(s, k, x k) where s ∈ S, 1 ≤ k ≤ K, and x k ∈ X k. Let k t be the discrete action selected at time t and let x kt be the associated continuous parameter. Then the Bellman equation becomes DISPLAYFORM1 Here inside the conditional expectation on the right-hand side of (4.2), we first solve DISPLAYFORM2, and then take the largest Q(s t+1, k, x * k). Note that taking supremum over continuous space X k is computationally intractable. However, the right-hand side of (4.2) can be evaluated efficiently providing x * k is given. To elaborate this idea, first note that, when the function Q is fixed, for any s ∈ S and k ∈ [K], we can view x Q k (s) = argsup DISPLAYFORM3 as a function of state s. That is, we identify (4.3) as a function x Q k: S → X k. Then we can rewrite the Bellman equation in (4.2) as DISPLAYFORM4 Note that this new Bellman equation resembles the classical Bellman equation in (2.1) with A = [K]. Similar to the deep Q-networks, we use a deep neural network Q(s, k, x k ; ω) to approximate Q(s, k, x k), where ω denotes the network weights. Moreover, for such a Q(s, k, x k ; ω), we approximate x Q k (s) in (4.3) with a deterministic policy network x k (·; θ): S → X k, where θ denotes the network weights of the policy network. That is, when ω is fixed, we want to find θ such that Q s, k, x k (s; θ); ω ≈ sup DISPLAYFORM5 Remark 4.1. Readers who are familiar with the work by BID8, that also claims to handle discrete-continuous hybrid action spaces, may be curious of its difference from the proposed P-DQN. The key differences are as follows.• In BID8, the discrete action types are parametrized as some continuous values, say f. And the discrete action that is actually executed is chosen via k = arg max i f (i). Such a trick actually turns the hybrid action space into a continuous action space, upon which the classical DDPG algorithm can be applied. However, in our framework, the discrete action type is chosen directly by maximizing the action's Q value explicitly.• The Q network in BID8 uses the artificial parameters f as input, which makes it an action-value function estimator of current policy (Q π). While in our framework, the Q network is actually an approximate estimator of the optimal policy's action-value function (Q).• We note that P-DQN is an off-policy method that can use historical data, while it is hard to use historical data in BID8 because there is only discrete action k without parameters f.(a) Network of P-DQN (b) Network of DDPG Figure 1: Illustration of the networks of P-DQN and DDPG BID8. P-DQN selects the discrete action type by maximizing Q values explicitly; while in DDPG, the discrete action with largest f, which can be seen as a continuous parameterization of K discrete action types, is chosen. Also in P-DQN the state and action parameters are feed into the Q-network which outputs K action values for each action type; while in DDPG, the continuous parameterization f, instead of the actual action k taken, is feed into the Q-network. Suppose that θ satisfies (4.4), then similar to DQN, we could estimate ω by minimizing the meansquared Bellman error via gradient descent. In specific, in the t-th step, let ω t and θ t be the weights of the value network and the deterministic policy network, respectively. To incorporate multi-step algorithms, for a fixed n ≥ 1, we define the n-step target y t by DISPLAYFORM0 We define the least squares loss function for ω by DISPLAYFORM1 Moreover, since we aim to find θ that minimizes Q[s, k, x k (s; θ); ω] with ω fixed, we define the loss function for θ by DISPLAYFORM2 Then we update ω t and θ t by gradient-based optimization methods. Moreover, the gradients are given by DISPLAYFORM3 Here ∇ x Q(s, k, x k ; ω) and ∇ ω Q(s, k, x k ; ω) are the gradients of the Q-network with respect to its third argument and fourth argument, respectively. By (5.5) and (5.4) we update the parameters using stochastic gradient methods. In addition, note that in the ideal case, we would minimize the loss function Θ t (θ) in (5.3) when ω t is fixed. From the in stochastic approximation methods BID14, we could approximately achieve such a goal in an online fashion via a two-timescale update rule BID3. In specific, we update ω with a stepsize α t that is asymptotically negligible compared with the stepsize β t for θ. In addition, for the validity of Input: Stepsizes {αt, βt} t≥0, exploration parameter, minibatch size B, the replay memory D, and a probability distribution µ over the action space A for exploration. Initialize network weights ω1 and θ1. for t = 1, 2,..., T doCompute action parameters x k ← x k (s, θt). Select action at = (kt, x k t) according to the -greedy policy at = a sample from distribution µ with probability, (kt, x k t) such that kt = arg max k∈[K] Q(s, k, x k ; ωt) with probability 1 −.Take action at, observe reward rt and the next state st+1. DISPLAYFORM0 Use data {y b, s b, a b} b∈ [B] to compute the stochastic gradient ∇ω Q t (ω) and ∇ θ Θ t (θ) defined in (5.5) and (5.4). Update the parameters by ωt+1 ← ωt − αt · ∇ω Q t (ωt) and θt+1 ← θt − βt · ∇ θ Θ t (θt). end for stochastic approximation, we require {α t, β t} to satisfy the Robbins-Moron condition BID25. We present the P-DQN algorithm with experienced replay in Algorithm 1.Note that this algorithm requires a distribution µ defined on the action space A for exploration. In each step, with probability, the agent sample an random action from µ; otherwise, it takes the greedy action with respect to the current value function. In practice, if each X k is a compact set in the Euclidean space (as in our case), µ could be defined as the uniform distribution over A. In addition, as in the DDPG algorithm , we can also add additive noise to the continuous part of the actions for exploration. Moreover, we use experience replay BID18 to reduce the dependencies among the samples, which can be replaced by more sample-efficient methods such as prioritized replay.Moreover, we note that our P-DQN algorithm can easily incorporate asynchronous gradient descent to speed up the training process. Similar to the asynchronous n-step DQN in BID20, we consider a centralized distributed training framework where each process can compute its local gradient and synchronize with a global parameter server. In specific, each local process runs an independent game environment to generate transition trajectories and use its own transitions to compute gradients with respect to ω and θ. These local gradients are then aggregated across multiple processes to update the global parameters. Note that these local stochastic gradients are independent. Thus tricks such as experience replay can be avoided in the distributed setting. Moreover, aggregating independent stochastic gradient decrease the variance of gradient estimation, which yields better algorithmic stability. We present the asynchronous P-DQN algorithm in Algorithm 2. For simplicity, here we only lay out the algorithm for each local process, which fetches ω and θ from the parameter server and computes the gradient. The parameter server stores the global parameters ω, θ. It updates the global parameters using the gradients sent from the local processes. In addition we use the RMSProp BID10 to update the network parameters, which is shown to be more stable in practice. The game King of Glory is a MOBA game, which is a special form of the RTS game where the players are divided into two opposing teams fighting against each other. Each team has a team base located in either the bottom-left or the top-right corner which are guarded by three towers on each of the three lanes. The towers can attack the enemies when they are within its attack range. Each player controls one hero, which is a powerful unit that is able to move, kill, perform skills, and purchase Input: exploration parameter, a probability distribution µ over the action space A for exploration, the max length of multi step return tmax, and maximum number of iterations Nstep. Initialize global shared parameter ω and θ Set global shared counter Nstep = 0 Initialize local step counter t ← 1. repeat Clear local gradients dω ← 0, dθ ← 0. tstart ← t Synchronize local parameters ω ← ω and θ ← θ from the parameter server. repeat Observe state st and let x k ← x k (st, θ) Select action at = (kt, x k t) according to the -greedy policy at = a sample from distribution µ with probability, (kt, x k t) such that kt = arg max k∈[K] Q(st, k, x k ; ω) with probability 1 −.Take action at, observe reward rt and the next state st+1. t ← t + 1 Nstep ← Nstep + 1 until st is the terminal state or t − tstart = tmax Define the target y = 0 for terminal st DISPLAYFORM0 Update global θ and ω using dθ and dω with RMSProp BID10 ). until Nstep > Nmax equipments. The goal of the heroes is to destroy the base of the opposing team. In addition, for both teams, there are computer-controlled units spawned periodically that march towards the opposing base in all the three lanes. These units can attack the enemies but cannot perform skills or purchase equipments. An illustration of the map is in FIG2, where the blue or red circles on each lane are the towers. During game play, the heroes advance their levels and obtain gold by killing units and destroying the towers. With gold, the heros are able to purchase equipments such as weapons and armors to enhance their power. In addition, by upgrading to the new level, a hero is able to improve its unique skills. Whereas when a hero is killed by the enemy, it will wait for some time to reborn. In this game, each team contains one, three, or five players. The five-versus-five model is the most complicated mode which requires strategic collaboration among the five players. In contrast, the one-versus-one mode, which is called solo, only depends on the player's control of a single hero. In a solo game, only the middle lane is active; both the two players move along the middle lane to fight against each other. The map and a screenshot of a solo game are given in FIG2 -(b) and (c), respectively. In our experiments, we play focus on the solo mode. We emphasize that a typical solo game lasts about 10 to 20 minutes where each player must make instantaneous decisions. Moreover, the players have to make different types of actions including attack, move and purchasing. Thus, as a reinforcement learning problem, it has four main difficulties: first, the state space has huge capacity; second, since there are various kinds of actions, the action space is complicated; third, the reward function is not well defined; and fourth, heuristic search algorithms are not feasible since the game is in real-time. Therefore, although we consider the simplest mode of King of Glory, it is still a challenging game for artificial intelligence. In this section, we applied the P-DQN algorithm to the solo mode of King of Glory. In our experiments, we play against the default AI hero Lu Ban provided by the game, which is a shooter with long attack range. To evaluate the performances, we compared our algorithm with the DDPG algorithm BID8 under fair condition. In our experiment, the state of the game is represented by a 179-dimensional feature vector which is manually constructed using the output from the game engine. These features consist of two parts. The first part is the basic attributes of the two heroes, the computer-controlled units, and buildings such as the towers and the bases of the two teams. For example, the attributes of the heroes include Health Point, Magic Point, Attack Damage, Armor, Magic Power, Physical Penetration/Resistance, and Magic Penetration/Resistance, and the attributes of the towers include Health Point and Attack Damage. The second component of the features is the relative positions of other units and buildings with respect to the hero controlled by the P-DQN player as well as the attacking relations between other units. We note that these features are directly extracted from the game engine without sophisticated feature engineering. We conjecture that the overall performances could be improved with a more careful engineered set of features. We simplify the actions of a hero into K = 6 discrete action types: Move, Attack, UseSkill1, UseSkill2, UseSkill3, and Retreat. Some of the actions may have additional continuous parameters to specify the precise behavior. For example, when the action type is k = Move, the direction of movement is given by the parameter x k = α, where α ∈ [0, 2π]. Recall that each hero's skills are unique. For Lu Ban, the first skill is to throw a grenade at some specified location, the second skill is to launch a missile in a particular direction, and the last skill is to call an airship to fly in a specified direction. A complete list of actions as well as the associated parameters are given in TAB0. The ultimate goal of a solo game is to destroy the opponent's base. However, the final is only available when the game terminates. Using such kind of information as the reward for training might not be very effective, as it is very sparse and delayed. In practice, we manually design the rewards using information from each frame. Specifically, we define a variety of statistics as follows. (In the sequel, we use subscript 0 to represent the attributes of our side and 1 to represent those of the opponent.)• Gold difference GD = Gold 0 − Gold 1. This statistic measures the difference of gold gained from killing hero, soldiers and destroying towers of the opposing team. The gold can be used to buy weapons and armors, which enhance the offending and defending attributes of the hero. Using this value as the reward encourages the hero to gain more gold.• Health Point difference (HPD = HeroRelativeHP 0 − HeroRelativeHP 1): This statistic measures the difference of Health Point of the two competing heroes. A hero with higher Health Point can bear more severe damages while hero with lower Health Point is more likely to be killed. Using this value as the reward encourages the hero to avoid attacks and last longer before being killed by the enemy.• Kill/Death KD = Kills 0 − Kills 1. This statistic measures the historical performance of the two heroes. If a hero is killed multiple times, it is usually considered more likely to lose the game. Using this value as the reward can encourage the hero to kill the opponent and avoid death.• Tower/Base HP difference THP = TowerRelativeHP 0 − TowerRelativeHP 1, BHP = BaseRelativeHP 0 − BaseRelativeHP 1. These two statistics measures the health difference of the towers and bases of the two teams. Incorporating these two statistic in the reward encourages our hero to attack towers of the opposing team and defend its own towers.• Tower Destroyed TD = AliveTower 0 − AliveTower 1. This counts the number of destroyed towers, which rewards the hero when it successfully destroy the opponent's towers.• Winning Game W = AliveBase 0 − AliveBase 1. This value indicates the winning or losing of the game.• Moving forward reward: MF = x + y, where (x, y) is the coordinate of Hero 0: This value is used as part of the reward to guide our hero to move forward and compete actively in the battle field. The overall reward is calculated as a weighted sum of the time differentiated statistics defined above. In specific, the exact formula is r t = 0.5 × 10 −5 (MF t − MF t−1) + 0.001(GD t − GD t−1) + 0.5(HPD t − HPD t−1 DISPLAYFORM0 The coefficients are set roughly inversely proportional to the scale of each statistic. We note that our algorithm is not very sensitive to the change of these coefficients in a reasonable range. In the experiments, we use the default parameters of skills provided by the game environment (usually pointing to the opponent hero's location). We found such kind of simplification does not affect to the overall performance of our agent. In addition, to deal with the periodic problem of the direction of movement, we use (cos(α), sin(α)) to represent the direction and learn a normalized two-dimensional vector instead of a degree (in practice, we add a normalize layer at the end to ensure this). In addition, the 6 discrete actions are not always usable, due to skills level up, lack of Magic Point (MP), or skills Cool Down(CD). In order to deal with this problem, we replace the max k∈ [K] with max k∈[K] and k is usable when selecting the action to perform, and calculating multi-step target as in Equation 5.1.For the network structure, recall that we use a feature vector of 179 dimensions as the state. We set both the value-network and the policy network as multi-layer fully-connected deep networks. The networks are in the same size of 256-128-64 nodes in each hidden layer, with the Relu activation function. During the training and testing processes, we set the frame skipping parameter to 2. This means that we take actions every 3 frames or equivalently, 0.2 second, which adapts to the human reaction time, 0.1 second. We set t max = 20 (4 seconds) to alleviate the delayed reward. In order to encourage exploration, we use -greedy sampling in training with = 0.255. In specific, the first 5 type actions We further smooth the original noisy curves (plotted in light colors) to their running average (plotted in dark colors). In the 3 rows, we plot the average of episode lengths, reward sum averaged for each episode in training, and reward sum averaged for each episode in validation, for the two algorithms respectively. Usually a positive reward sum indicates a winning game, and vice versa. We can see that the proposed algorithm P-DQN learns much faster than its precedent work in our setting. (a) Performance of P-DQN. (b) Performance of DDPG are sampled with probability of 0.05 each and the action "Retreat" with probability 0.005. For actions with additional parameters, since the parameters are in bounded sets, we draw these parameters from a uniform distribution. Moreover, if the sampled action is infeasible, we execute the greedy policy from the feasible ones, so the effective exploration rate is less than. We uses 48 parallel workers with constant learning rate 0.001 in training and 1 worker with deterministic sampling in validation. The training and validating performances are plotted in Figure 3.We implemented the DDPG BID8 ) algorithm within our learning environment to have a fair comparison. The exact network structure is plotted in Figure 1. Each algorithm is allowed to run for 15 million steps, which corresponds to roughly 140 minutes of wall clock time when paralleled with 48 workers. From the experiments , we can see that our algorithm P-DQN can learn the value network and the policy network much faster comparing to the other algorithm. In (a1), we see that the average length of games increases at first, reaches its peak when the two player's strength are close, and decreases when our player can easily defeat the opponent. In addition, in (a2) and (a3), we see that the total rewards in an episode increase consistently in training as well as in test settings. The DDPG algorithm may not be suitable for hybrid actions with both a discrete part and a continuous part. The major difference is that maximization over k when we need to select a action is computed explicitly in P-DQN, instead of approximated implicitly with the policy network as in DDPG. Moreover, with a deterministic policy network, we extend the DQN algorithm to hybrid action spaces of discrete and continuous types, which makes the P-DQN algorithm more suitable for realistic scenarios. Previous deep reinforcement learning algorithms mostly can work with either discrete or continuous action space. In this work, we consider the scenario with discrete-continuous hybrid action space. In contrast of existing approaches of approximating the hybrid space by a discrete set or relaxing it into a continuous set, we propose the parameterized deep Q-network (P-DQN), which extends the classical DQN with deterministic policy for the continuous part of actions. Empirical experiments of training AI for King of Glory, one of the most popular games, demonstrate the efficiency and effectiveness of P-DQN.

We develop VAEs where the encoder takes a model parameter vector as input, so we can do rapid inference for many models