Question: represent the scientific article to find a short tldr

Question: Recent advances in Generative Adversarial Networks (GANs) – in architectural design, training strategies, and empirical tricks – have led nearly photorealistic samples on large-scale datasets such as ImageNet. In fact, for one model in particular, BigGAN, metrics such as Inception Score or Frechet Inception Distance nearly match those of the dataset, suggesting that these models are close to match-ing the distribution of the training set. Given the quality of these models, it is worth understanding to what extent these samples can be used for data augmentation, a task expressed as a long-term goal of the GAN research project. To that end, we train ResNet-50 classifiers using either purely BigGAN images or mixtures of ImageNet and BigGAN images, and test on the ImageNet validation set. Our preliminary suggest both a measured view of state-of-the-art GAN quality and highlight limitations of current metrics. Using only BigGAN images, we find that Top-1 and Top-5 error increased by 120% and 384%, respectively, and furthermore, adding more BigGAN data to the ImageNet training set at best only marginally improves classifier performance. Finally, we find that neither Inception Score, nor FID, nor combinations thereof are predictive of classification accuracy. These suggest that as GANs are beginning to be deployed in downstream tasks, we should create metrics that better measure downstream task performance. We propose classification performance as one such metric that, in addition to assessing per-class sample quality, is more suited to such downstream tasks. Recent years have witnessed a marked improvement in sample quality in Deep Generative Models. One model class in particular, Generative Adversarial Networks BID7, has begun to generate nearly photorealistic images. While applications of adversarial training have found their way into image translation BID16 and style transfer BID5, a typically discussed goal for such models, and in particular conditional ones, is data augmentation. Such models have enjoyed limited success in these tasks thus far for large-scale datasets such as ImageNet, likely because existing models did not generate sufficiently high-quality samples. Recently, however, BigGANs BID4 have generated photorealistic images of ImageNet data up to 512×512 resolution, and moreover, achieve Inception Scores and Frechet Inception Distances similar to the dataset on which they were trained. Such suggest, though do not prove, that BigGANs are indeed capturing the data distribution. If this were true, then it seems plausible that these samples can be used in downstream tasks, especially in situations in which limited labelled data are available. In this work, we test the rather simple hypothesis that BigGANs are indeed useful for data augmentation, or more drastically, data replacement of the original data distribution. To that end, we use BigGANs for two simple experiments. First, we train ImageNet classifiers, replacing the original training set with one produced by BigGAN. Second, we augment the original ImageNet training set with samples from BigGAN. Our working hypothesis is that if BigGANs were indeed capturing the data distribution, then we could use those samples, instead of or in addition to the original training set, to improve performance on classification. That it does not -on replacement, Top-5 classification Though a negative , a more positive byproduct of the work is the introduction of a new metric that can better identify issues with GAN and other generative models. In particular, training a classifier allows us to identify, for conditional generative models, which classes are particularly poor, either due to low quality samples or underrepresentation of dataset diversity. 2.1 SETUP Our experiments are rather simple: we use BigGAN-deep (further denoted as BigGAN) models to either replace or augment the ImageNet training set, train an image classifier, and compare performance on the ImageNet validation set. In the data replacement experiments, we replace the ImageNet training set with one from BigGAN-deep, and each example from the original training set is replaced with a model sample from the same class. In the augmentation experiments, we add to the ImageNet training set, 25%, 50%, or 100% more data from BigGAN. Moreover, since the truncation trick -which resamples dimensions that are outside the mean of the distribution -seems to trade off quality for diversity, we perform experiments for a sweep of truncation parameters: 0.2, 0.42, 0.5, 0.8, 1.0, 1.5, and 2.0. 1 In addition, we compare performance on replacement and augmentation to two traditional GAN metrics: Inception Score BID14 and Frechet Inception Distance (FID) BID10, as these metrics are the current gold standard for GAN comparison. Both rely on a feature space from a classifier trained on ImageNet, suggesting that if metrics are useful at predicting performance on a downstream task, it would indeed be this one. We used ResNet-50 BID9 classifier for our models, with single-crop evaluation. The classifier is trained for 90 epochs using TensorFlow's momentum optimizer, a learning rate schedule linearly increasing from 0.0 to 0.4 for the first 5 epochs, and decreased by a factor of 10 at epochs 30, 60, and 80. It mirrors the 8,192 batch setup of BID8 with gradual warmup. TAB0 shows the performance of classifiers trained on BigGAN datasets compared to the real dataset. At every truncation level, ResNet-50 classifiers trained on BigGAN samples generalize substantially worse to real images than the classifier trained on real data. To better understand why this has occurred, we broke down the performance by class for the best-performing truncation level: 1.5. As shown in the left pane of FIG0, nearly every class suffers a drop in performance compared to the original dataset. Performance on six classes -partridge, red fox, jaguar/panther, squirrel monkey, African elephant, and strawberry -did improve over the original dataset, though the improvement for those classes was marginal. The right pane of FIG0 shows the two best and two worst performing categories, as measured by the difference in classification performance. Notably, for the two worst performing categories and two others -balloon, paddlewheel, pencil sharpener, and spatula -classification accuracy was 0% on the validation set. That said, at the best truncation levels, Top-5 Error is roughly 35%, suggesting that BigGANs are learning nontrivial distributions. Given the performance of BigGAN in the replacement experiments, one should not necessarily expect improved classifier accuracy by augmenting ImageNet training set with BigGAN samples. Figure 2 illustrates the performance of the classifiers when we increase the amount of BigGAN training data. Perhaps somewhat surprisingly, BigGAN models that sample from lower truncation values, and have lower sample diversity, are able to perform better on data augmentation compared to those models that performed the best on the data replacement experiment. In fact, for some of the lowest truncation values, one found modest improvement in classification performance: roughly 3% improvement relative on Top-1 Error (but at the cost of 1.5 times the amount of training time).Finally, Inception Score and FID had very little correlation with performance on either the replacement or augmentation experiments, suggesting that alternative metrics will be needed when we turn our attention to downstream tasks. For our replacement experiments, the correlation coefficient between Top-1 error and FID is 0.16, and Inception Score 0.86, the latter incorrectly suggesting that improved Inception Score is highly correlated with increased error. Moreover, the best-performing methods have rather poor Inception Score and FIDs. That models that perform poorly on Inception Score and Frechet Inception Distance also perform poorly on classification is no surprise; that models that perform well on Inception Scores and FID perform poorly on classification suggests that alternative metrics are needed. One can easily diagnose the issue with Inception Score: as BID2 noted, Inception Score does not account for intra-class diversity, and a training set with little intra-class diversity may make the classifier fail to generalize to a more diverse test set. FID should better account for this lack of diversity at least grossly, as the metric, calculated as F ID(P x, P y) = µ x − µ y 2 + tr(Σ x + Σ y − 2(Σ x Σ y) 1/2 ), compares the covariance matrices of the data and model distribution. By comparison, per-class classification error offers a finer measure of model performance, as it provides us a per-class metric to identify which classes have better or worse performance. While in theory one could calculate a per-class FID, FID is known to suffer from high bias BID3 for low number of samples, likely making the per-class estimates unreliable. The on augmentation highlight different desiderata for samples that are added to the dataset rather than replaced. Clearly, the samples added should be sufficiently different from the data to allow the classifier to better generalize, and yet, poorer sample quality may lead to poorer generalization compared to the original dataset. This may be the reason why extending the dataset with samples generated from a lower truncation value noise -which are higher-quality, but less diverse -perform better on augmentation than replacement. Furthermore, this may also explain why Inception Score, Frechet Inception Distance, and data replacement classification error are not predictive of data augmentation classification performance. This work encompasses two lines of work: GANs for data augmentation and improved evaluation metrics for GANs. For data augmentation, BID0 proposed an image-conditioned model for augmentation, and found improved on smaller datasets. BID6 used a GAN to generate synthetic training data of size 64×64×1 of images of liver lesions. For evaluation metrics, BID15 noted there is difficulty in designing evaluation metrics that will illustrate the general performance in the model. Despite this finding, those interested in measuring the quality of implicit generative models have proposed practical metrics to compare sample quality from different models: which have led to introduction of Inception Score and FID. BID12 recommends the use classifier two-sample tests to test GAN samples as a metric. Other measures attempt to determine other properties of generative models. BID13 constructs synthetic datasets for which precision and recall can be computed approximately and compares Inception Score and FID to changes in precision and recall. Geometry Score BID11 constructs approximate manifolds from data and samples, and uses them for GAN samples to determine whether there was mode collapse. BID1 attempt to determine the support size of GANs by using a Birthday Paradox test, though it requires a human to identify two nearly-identical samples. In this work, we investigated to what extent BigGAN, the state-of-the-art GAN on ImageNet, captures the data distribution, and to what extent those samples can be used for data augmentation. Our demonstrate that despite excellent scores on traditional GAN metrics such as Inception Score and Frechet Inception Distance, current state-of-the-art GAN models do not capture the distribution for large-scale datasets such as ImageNet. Moreover, we found only a modest improvement in classifier performance when the training set was augmented with BigGAN samples. Finally, through classifier metrics outlined in the work, we can identify on which classes BigGAN performed well, and on which ones researchers should focus their future efforts. An open question in this work is how to create metrics predictive of performance on downstream tasks. Even for the classifier metric, on data replacement did not necessarily correlate with those on data augmentation. Better evaluation metrics will help us understand to what extent GANs, or any other Deep Generative Models, can be used for downstream tasks.

Represent the short summary for finding the corresponding scientific article (long)

BigGANs do not capture the ImageNet data distributions and are only modestly successful for data augmentation.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Contrary to previous beliefs, the training performance of deep networks, when measured appropriately, is predictive of test performance, consistent with classical machine learning theory.


Question:
represent the text to find a very short summary (1-2 sentences max)

E.g. We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. L2TL considers joint optimization of vastly-shared weights between models for source and target tasks, and employs adaptive weights for scaling of constituent losses. The adaptation of the weights is based on reinforcement learning, guided with a performance metric on the target validation set. We demonstrate state-of-the-art performance of L2TL given fixed models, consistently outperforming fine-tuning baselines on various datasets. In the regimes of small-scale target datasets and significant label mismatch between source and target datasets, L2TL outperforms previous work by an even larger margin. == We propose learning to transfer learn (L2TL) to improve transfer learning on a target dataset by judicious extraction of information from a source dataset. != An empirical study that provides a novel perspective on few-shot learning, in which a fine-tuning method shows comparable accuracy to more complex state-of-the-art methods in several classification tasks.

Question: Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts of data each day. This wealth of data can help to learn models that can improve the user experience on each device. However, the scale and heterogeneity of federated data presents new challenges in research areas such as federated learning, meta-learning, and multi-task learning. As the machine learning community begins to tackle these challenges, we are at a critical time to ensure that developments made in these areas are grounded with realistic benchmarks. To this end, we propose Leaf, a modular benchmarking framework for learning in federated settings. Leaf includes a suite of open-source federated datasets, a rigorous evaluation framework, and a set of reference implementations, all geared towards capturing the obstacles and intricacies of practical federated environments. With data increasingly being generated on federated networks of remote devices, there is growing interest in empowering on-device applications with models that make use of such data BID25. Learning on data generated in federated networks, however, introduces several new obstacles:Statistical: Data is generated on each device in a heterogeneous manner, with each device associated with a different (though perhaps related) underlying data generating distribution. Moreover, the number of data points typically varies significantly across devices. The number of devices in federated scenarios is typically order of magnitudes larger than the number of nodes in a typical distributed setting, such as datacenter computing. In addition, each device may have significant constraints in terms of storage, computational, and communication capacities. Furthermore, these capacities may also differ across devices due to variability in hardware, network connection, and power. Thus, federated settings may suffer from communication bottlenecks that dwarf those encountered in traditional distributed datacenter settings, and may require faster on-device inference. Privacy and Security: Finally, the sensitive nature of personally-generated data requires methods that operate on federated data to balance privacy and security concerns with more traditional considerations such as statistical accuracy, scalability, and efficiency BID3.Recent works have proposed diverse ways of dealing with these challenges, but many of these efforts fall short when it comes to their experimental evaluation. As an example, consider the federated learning paradigm, which focuses on training models directly on federated networks BID25 BID23. Experimental works focused on federated learning broadly utilize three types of datasets: datasets that do not provide a realistic model of a federated scenario and yet are commonly used, e.g., artificial partitions of MNIST, MNIST-fashion or CIFAR-10 (; BID10 BID7 BID2 BID9 BID28 BID30 ; realistic but proprietary federated datasets, e.g., data from an unnamed social network in, crowdsourced voice commands in BID15, and proprietary data by Huawei in BID4; and realistic federated datasets that are derived from publicly available data, but which are not straightforward to reproduce, e.g., FaceScrub in BID20, Shakespeare in and Reddit in BID10 BID18 BID2.Along the same lines of federated learning, meta-learning is another learning paradigm that could use more realistic benchmarks. The paradigm is a natural fit for federated settings, as the different devices can be easily interpreted as meta-learning tasks BID4. However, the artificially generated tasks considered in popular benchmarks such as Omniglot BID12 BID6 BID29 BID26 and miniImageNet BID24 BID6 BID29 BID26 fail to challenge the current approaches in ways that real-world problems would. More recently, BID27 proposed Meta-Dataset as a more realistic meta-learning benchmark, but tasks still have no real-world interpretation. All of these datasets could thus be categorized as the first type mentioned above (unrealistic yet popular).As a final example, LEAF's datasets can allow researchers and practitioners to test multi-task learning (MTL) methods in regimes with large numbers of tasks and samples, contrary to traditional MTL datasets (e.g., the popular Landmine Detection BID33 BID21 BID32 BID25, Computer Survey BID1 BID0 BID11 and Inner London Education Authority School BID21 BID14 BID0 BID1 BID11) datasets have at most 200 tasks each).In this work, we aim to bridge the gap between artificial datasets that are popular and accessible for benchmarking, and those that realistically capture the characteristics of a federated scenario but that, so far, have been either proprietary or difficult to process. Moreover, beyond establishing a suite of federated datasets, we propose a clear methodology for evaluating methods and reproducing . To this end, we present LEAF, a modular benchmarking framework geared towards learning in massively distributed federated networks of remote devices. LEAF is an open-source benchmarking framework for federated settings. It consists of a suite of open-source datasets, an array of statistical and systems metrics, and a set of reference implementations. As shown in Figure 1, LEAF's modular design allows these three components to be easily incorporated into diverse experimental pipelines. We now detail LEAF's core components. Reference Implementations Metrics Figure 1. LEAF modules and their connections. The Datasets module preprocesses the data and transforms it into a standardized JSON format, which can integrate into an arbitrary ML pipeline. LEAF's Reference Implementations module is a growing repository of common methods used in the federated setting, with each implementation producing a log of various different statistical and systems metrics. This log (or any log generated in an appropriate format) can be used to aggregate and analyze these metrics in various ways. LEAF performs this analysis through its Metrics module. We have curated a suite of realistic federated datasets for LEAF. We focus on datasets where the data has a natural keyed generation process (where each key refers to a particular device); the data is generated from networks of thousands to millions of devices; and the number of data points is skewed across devices. Currently, LEAF consists of three datasets:• Federated Extended MNIST (FEMNIST), which serves as a similar (and yet more challenging) benchmark to the popular MNIST dataset. It is built by partitioning the data in Extended MNIST BID5 based on the writer of the digit/character. • Sentiment140 BID8 We provide statistics on these datasets in TAB1. In LEAF, we provide all necessary pre-processing scripts for each dataset, as well as small/full versions for prototyping and final testing. Moving forward, we plan to add datasets from different domains (e.g. audio, video) and to increase the range of machine learning tasks (e.g. text to speech, translation, compression, etc.).Metrics: Rigorous evaluation metrics are required to appropriately assess how a learning solution behaves in federated scenarios. Currently, LEAF establishes an initial set of metrics chosen specifically for this purpose. For example, we introduce metrics that better capture the entire distribution of performance across devices: performance at the 10th and 90th percentiles and performance stratified by natural hierarchies in the data (e.g. play in the case of the Shakespeare dataset). We also introduce metrics that account for the amount of computing resources needed from the edge devices in terms of number of FLOPS and number of bytes downloaded/uploaded. Finally, LEAF also recognizes the importance of specifying how the accuracy is weighted across devices, e.g., whether every device is equally important, or every data point equally important (implying that power users/devices get preferential treatment). Notably, considering stratified systems and accuracy metrics is particularly important in order to evaluate whether a method will systematically exclude groups of users (e.g., because they have lower end devices) and/or will underperform for segments of the population (e.g., because they produce less data).Reference implementations: In order to facilitate repro- ducibility, LEAF also contains a set of reference implementations of algorithms geared towards federated scenarios. Currently, this set is limited to the federated learning paradigm, and in particular includes reference implementations of minibatch SGD, FedAvg and Mocha BID25. Moving forward we aim to equip LEAF with implementations for additional methods and paradigms with the help of the broader research community. We now show a glimpse of LEAF in action. In particular, we highlight three of LEAF's characteristics:LEAF enables reproducible science: To demonstrate the reproducibility enabled via LEAF, we focus on qualitatively reproducing the that obtained on the Shakespeare dataset for a next character prediction task. In particular, it was noted that for this particular dataset, the FedAvg method surprisingly diverges as the number of local epochs increases. This is therefore a critical setting to understand before deploying methods such as FedAvg. To show how LEAF allows for rapid prototyping of this scenario, we use the reference FedAvg implementation and subsample 118 devices (around 5% of the total) in our Shakespeare data (which can be easily done through our framework). Results are shown in Figure 2, where we indeed see similar divergence behavior in terms of the training loss as we increase the number of epochs. LEAF provides granular metrics: As illustrated in FIG0 and FIG1, our proposed systems and statistical metrics are important to consider when serving multiple clients simultaneously. For statistical metrics, in FIG0 we show the effect of varying the minimum number of samples per user in Sentiment140 (which we denote as k). We see that, while median performance degrades only slightly with data-deficient users (i.e., k = 3), the 25th percentile (bottom of box) degrades dramatically. Meanwhile, for systems metrics, we run minibatch SGD and FedAvg for FEMNIST and calculate the systems budget needed to reach an accuracy threshold of 0.75 in FIG1. We characterize the budget in terms of total number of FLOPS across all devices and total number of bytes uploaded to network. Our Figure 2. Convergence behavior of FedAvg on a subsample of the Shakespeare dataset. We use a learning rate of 0.8 and 10 devices per round for all experiments. We are able to achieve test accuracy comparable to the obtained in. We also qualitatively replicate the divergence in training loss that is observed for large numbers of local epochs (E). demonstrate the improved systems profile of FedAvg when it comes to the communication vs. local computation trade-off, though we note that in general methods may vary across these two dimensions, and it is thus important to consider both aspects depending on the problem at hand. To demonstrate LEAF's modularity, we incorporate its Datasets module into two different experimental pipelines besides FedAvg (which has been our focus so far). In particular, we wish to validate the hypothesis that personalization strategies (be it MTL or meta-learning) outperform competing approaches in statistically heteroge- neous scenarios.1. Our first pipeline explores our hypothesis in regimes where each device holds little data. We use three different kinds of models:• A global SVM which is trained in all of the devices' data at once (Global-SVM).• A local SVM per device that is trained solely on the device's data (Local-SVM).• The same SVM model but trained in the multitask setting presented in BID25 ) (MTL-SVM).2. Our second pipeline corroborates the hypothesis in regimes with no restrictions on the amount of data per device. To do this, we run the popular algorithm Reptile BID22 (which can be shown to be a re-weighed, fine-tuned version of FedAvg) over FEM-NIST and compare it against FedAvg when trained under similar conditions. Results for both sets of experiments are presented in TAB2. For the first set of experiments, we re-cast FEMNIST as a binary classification task (digits vs. characters) and discard devices with more than 192 samples. For the second set, we run each algorithm for 1, 000 rounds, use 5 clients per round, a local learning rate of 10 −3, a training mini-batch size of 10 for 5 mini-batches, and evaluate on an unseen set of test devices. Furthermore, for Reptile we use a linearly decaying meta-learning rate that goes from 2 to 0, and evaluate by fine-tuning each test device for 50 mini-batches of size 5. It is clear that the personalized strategies outperform the competing approaches. We present LEAF, a modular benchmarking framework for learning in federated settings, or ecosystems marked by massively distributed networks of devices. Learning paradigms applicable in such settings include federated learning, metalearning, multi-task learning, and on-device learning. LEAF allows researchers and practitioners in these domains to reason about new proposed solutions under more realistic assumptions than previous benchmarks. We intend to keep LEAF up to date with new datasets, metrics and opensource solutions in order to foster informed and grounded progress in this field TAB1

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present Leaf, a modular benchmarking framework for learning in federated data, with applications to learning paradigms such as federated learning, meta-learning, and multi-task learning.

Represent the TLDR for finding the corresponding scientific article:

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time. We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time. Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset. Explicitly predicting the motion of actors in a dynamic scene is a critical component of intelligent systems. Humans can seamlessly track moving objects in their environment by using cues such as appearance, relative distance, and temporal consistency. The world is rarely experienced in a static way: motion (or its absence) provides essential information to understand a scene. Similarly, incorporating past context through a temporal model is essential to segment and track objects consistently over time and through occlusions. From a computer vision perspective, understanding object motion involves segmenting instances, estimating depth, and tracking instances over time. Instance segmentation, which requires segmenting individual objects at the pixel level, has gained traction with challenging datasets such as COCO , Cityscapes and Mapillary Vistas . Such datasets, which only contain single-frame annotations, do not allow the training of video models with temporally consistent instance segmentation, nor does it allow self-supervised monocular depth estimation, that necessitates consecutive frames. Yet, navigating in the real-world involves a three-dimensional understanding of the other agents with consistent instance segmentation and depth over time. More recently, a new dataset containing video instance segmentation annotations was released, the KITTI Multi-Object and Tracking Dataset . This dataset contains pixel-level instance segmentation on more than 8,000 video frames which effectively enables the training of video instance segmentation models. In this work, we propose a new spatio-temporal embedding loss that learns to map video-pixels to a high-dimensional space 1. This space encourages video-pixels of the same instance to be close together and distinct from other instances. We show that this spatio-temporal embedding loss, jointly with a deep temporal convolutional neural network and self-supervised depth loss, produces consistent instance segmentations over time. The embedding accumulates temporal context thanks to the temporal model, as otherwise, the loss would only be based on appearance. The temporal model is a causal 3D convolutional network, which is only conditioned on past frames to predict the current embedding and is capable of real-time operation. Finally, we show that predicting depth improves the quality of the embedding as knowing the distance to an instance constrains its future location given that objects move smoothly in space. To summarise our novel contributions, we: • introduce a new spatio-temporal embedding loss for video instance segmentation, • show that having a temporal model improves embedding consistency over time, • improve how the embedding disambiguates objects with a self-supervised monocular depth loss, • handle occlusions, contrary to previous IoU based instance correspondence. We demonstrate the efficacy of our method by advancing the state-of-the-art on the KITTI MultiObject and Tracking Dataset . An example of our model's output is given by Section 1. Two main approaches exist for single-image instance segmentation: region-proposal based (; ; ;) and embedding based (; ; ;). The former method relies on a region of interest proposal network that first predicts bounding boxes then estimates the mask of the object inside that bounding box. With such strategy, one pixel can belong to the overlap of many bounding boxes, and it is largely unclear how correspondence between pixels can be learned. We instead favour the embedding based method and extend it to space and time. Capturing the inter-relations of objects using multi-modal cues (appearance, motion, interaction) is difficult, as showcased by the Multi-Object Tracking (MOT) challenge . MOT's goal is to infer the trajectories of objects and cover a wide range of applications such as biology (birds , fish , robot navigation ) and autonomous driving ). and learned a representation of objects that follows the "tracking-by-detection" paradigm where the goal is to connect detections across video frames by finding the optimal assignment of a graph-based tracking formulation (i.e. each detection is a node, and an edge is the similarity score between two detections). Collecting large-scale tracking datasets is necessary to train deep networks, but that process is expensive and time-consuming. introduced video colourisation as a self-supervised method to learn visual tracking. They constrained the colourisation problem of a grayscale image by learning to copy colors from a reference frame, with the pointing mechanism of the model acting as a tracker once it is fully trained. The colourisation model is more robust than optical flow based models, especially in complex natural scenes with fast motion, occlusion and dynamic s. extended the task of multi-object tracking to multi-object tracking and segmentation (MOTS), by considering instance segmentations as opposed to 2D bounding boxes. Motivated by the saturation of the bounding box level tracking evaluations , they introduced the KITTI MOTS dataset, which contains pixel-level instance segmentation on more than 8,000 video frames. They also trained a model which extends Mask R-CNN by incorporating 3D convolutions to integrate temporal information, and the addition of an association head that produces an association vector for each detection, inspired from person re-identification . The temporal component of their model, however, is fairly shallow (one or two layers), and is not causal, as future frames are used to segment past frames. More recently, collected a large-scale dataset from short YouTube videos (3-6 seconds) with video instance segmentation labels, and introduced a densely annotated synthetic dataset with complex occlusions to learn how to estimate the spatial extent of objects beyond what is visible. Contrary to methods relying on region proposals , embeddingbased instance segmentation methods map all pixels of a given instance to a high dimensional space with desirable properties. This overcomes several limitations of region-proposal methods. Firstly, two objects may share the same bounding box and in that situation, it is ambiguous which object mask the model should segment. Secondly, pixels can belong to two separate objects as each prediction is done independently. Finally, the number of detected objects is limited by the fixed number of proposals of the network. We propose a spatio-temporal embedding loss that extends's instance embedding loss to video: each pixel belonging to a given instance in space and time is transformed into a unique location in a high dimensional space, using cues such as appearance, context and motion. More concretely, three terms are used in the loss: the attraction loss (Equation) to ensure pixels from the same instance are close to each other, the repulsion loss (Equation) to ensure two separate instances are far from each other and a regularisation term (Equation) so that instance centers should not diverge too much from the origin. Let us denote the number of instances, K, and the subset of indices, S k, corresponding to all the pixels belonging to instance k in the video. ∀i ∈ S k, y i is the embedding for pixel position i and µ k is the mean embedding of instance k: Where ρ a denotes the attraction radius within a cluster: we want the embedding to be within ρ a of the centroid. 2ρ r denotes the repulsion radius: we want the centroids of two different clusters to be at least 2ρ r apart. Therefore, if we set ρ r > 2ρ a, a given pixel embedding of a cluster will be closer to all the pixel embeddings of its cluster than any other pixel embedding. The spatio-temporal embedding loss is the weighted sum of the attraction, repulsion and regularisation losses: During inference, each pixel of the considered frame is assigned to an instance by randomly picking an unassigned point and aggregating close-by pixels with the mean-shift algorithm . In the ideal case, with a test loss of zero, this will in perfect clustering if the repulsion radius, ρ r, is twice as large as the attraction radius, ρ a. The relative distance of objects is a strong cue to segment instances in space and time, as the motion of objects is temporally smooth. Knowing the previous distance of an object relative to the camera assists tracking as the future position will be constrained by the object's current location. Depth estimation with supervised methods requires a vast quantity of high quality annotated data, which is challenging to acquire in a range of environments as laser measurements can be imprecise in natural scenes with motion and reflections. Because we have access to video in our instance segmentation dataset, we can leverage self-supervised depth losses from monocular videos, where the supervision comes from consecutive temporal frames. In addition to predicting the depth map, ego-motion also has to be inferred, but only during training to constrain the depth network. and , we train a depth network with a separate pose estimation network with the hypothesis during training that scenes are mostly rigid, therefore assuming appearance change is mostly due to camera motion. The pixels that violate this assumption are masked from the view synthesis loss, as they otherwise create infinite holes during inference for objects that are typically seen in motion during training -more details in Appendix A.1. The training signal comes from novel view synthesis: generation of new images of the scene from a different camera pose. Let us denote by (I 1, I 2, ..., I T) a sequence of images, with target view I t and source view I s. The view synthesis loss is given by: withÎ s→t the synthesised view of I t from source image I s using the predicted depthD t and the predicted 4 × 4 camera transformationT t→s predicted from the separate pose network. The projection error function, e, is described by as a weighted sum of L 1, Structural Similarity Index (SSIM) and a smoothness regularisation term. Let us denote by p t the coordinate of a pixel in the target image I t in homogeneous coordinates. Given the camera intrinsic matrix, K, and the mapping ϕ from image plane to camera coordinate, the corresponding pixel in the source image is provided by: The projected coordinates p s are continuous values, we use the Spatial Transformer Network sampling mechanism to bilinearly interpolate the four neighbouring pixels to populate the reconstructed image I s→t. Some pixels are visible in the target image, but are not in the source images, leading to a large projection error. As advocated by , instead of summing, taking the minimum projection error greatly reduces artifacts due to occlusion and in sharper predictions. The ing view synthesis loss is: The ing video instance embedding loss is the weighted sum of the attraction, repulsion, regularisation and geometric view synthesis losses: 4 MODEL ARCHITECTURE Our model contains three components: an encoder, temporal model and the decoders. Each frame is first encoded to a more powerful and compact representation, then the temporal model learns the dynamics of the scene, and finally, the decoders output the instance embedding and depth prediction as illustrated by Figure 2. Figure 2: Our video instance segmentation and depth model architecture. The embedding, z t, is trained to explicitly encode appearance, motion and geometry cues in order to predict an instance embedding and monocular depth prediction. Encoder. We use a ResNet-18 with 14.6M parameters as our encoder, which allows the network to run in real-time on sequences of images. Temporal Model. The scene dynamics is learned with a causal 3D convolutional network composed of blocks of 3D residual convolutions (convolving in both space and time). For a given time index, t, the network only convolves over images from indices s ≤ t to compute the temporal representation z t. It therefore does not use future frames and is completely causal. The temporal model does not decimate the spatial dimension of the encoding, but slowly accumulates information over time from the previous encodings x s with s ≤ t. The temporal model is trained efficiently with convolutions as all input images are available during training, enabling parallel computations with GPUs. However, during inference, the model inherently has to be sequential, but can be made significantly faster by caching the convolutional features over time and eliminating redundant operations, as proposed by for WaveNet . Decoders. The decoders then map the temporal encoding z t to its instance embedding y t of dimension p × height × width, with p the embedding dimension, and depth d t of dimension 1 × height × width. The embedding values belonging to the same instance are pushed together in the high-dimensional space R p, and pulled away from the other instances, over the whole video. Therefore, tracking instances simply requires comparing the mean embedding of a newly segmented instance with previously segmented instances. A distance lower than ρ r indicates a match. To segment the instances, we first run a mask network (trained separately) then we cluster the segmented embeddings using mean shift to discover dense regions of embeddings. Over time, the embeddings are accumulated up until the sequence length corresponding to the sequence length used during training to constrain instances spatio-temporally. This creates increasingly dense regions over time ing in a better clustering. To ensure that embeddings of a particular instance can smoothly vary over time, the embeddings have a life span corresponding to the sequence length of the model. Pose and Mask Model. For the pose network we use a ResNet and for the mask network we use an encoder-decoder model, also based on a ResNet. Further details are in Appendix A.1. Next we describe experimental evidence which demonstrates the performance of our method by advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset . The KITTI Multi-Object Tracking and Segmentation (MOTS) dataset contains 8,008 frames with instance segmentation labels ing in a total of 26,899 annotated cars. It is composed of 21 scenes with a resolution of 375 × 1242 with consistent instance ID labels across time, allowing the training of video instance segmentation models. The frames are annotated at 10 frames per second, which is suitable for self-supervised monocular depth prediction. The ApolloScape dataset ) also contains video instance segmentation labels for 49,287 frames, but the annotations are not consistent in time, rendering the training of a temporal model impossible. NuScenes features 1,000 scenes of 20 seconds with annotations at 2Hz in a diverse range of environments (different weather, daytime, city) but only contains bounding box labels, failing to represent the fine-grained details of instance segmentation. Temporal instance segmentation is also available on short snippets of the DAVIS dataset , but each snippet is recorded by a different camera and is too short to effectively learn a depth model. For this reason, we focus on the KITTI MOTS dataset -it is the only dataset that contains consistent video instance segmentation in a sufficient quantity to train deep models. We halve the input images to our encoder to use an input RGB resolution of 3 × 192 × 640. The ing encoding is 128 × 24 × 80. The decoders then map the temporal encoding z t to its instance embedding y t of dimension p × 192 × 640, with p = 8 the embedding dimension, and depth d t of dimension 1 × 192 × 640. Except for the experiments in Table 4, we train with a sequence length of 5 which corresponds to 0.5 seconds of temporal context since the videos are 10Hz. In the loss function, we set the attraction radius ρ a = 0.5 and repulsion radius ρ r = 1.5. We weight the losses with attraction and repulsion loss weight λ 1 = λ 2 = 1.0, regularisation loss λ 3 = 0.001 and depth loss λ 4 = 1.0. Let us define multi-object tracking and segmentation metrics, which measures the quality of the segmentation as well as the consistency of the predictions over time. Contrary to bounding box detection, where a ground truth box may overlap with several predicted boxes, in instance segmentation, since each pixel is assigned to at most one instance, only one predicted mask can have an Intersection over Union (IoU) larger than a given threshold with a given ground truth mask. Let us denote by H the set of predicted ids, M the set of ground truth ids and g the mapping from hypothesis masks to ground truth masks. g: H → M ∪ ∅ is defined as: We thus define: • True positives as: T P = {h ∈ H|g(h) = ∅}, correctly assigned predicted masks. • False positives as: F P = {h ∈ H|g(h) = ∅}, predicted masks not assigned to any ground truth mask. • False negatives as: F N = {m ∈ M|g −1 (m) = ∅}, ground truth masks not covered by any hypothesis mask. • Soft number of true positives:T P = h∈T P IoU(h, g(h)) Let the function pred: M → M ∪ ∅ map a ground truth mask to its latest tracked predecessor (∅ if the ground truth mask is seen for the first time). The set IDS of ID switches is defined as the set of ground truth masks whose predecessor was tracked by a different ID. , we define the following MOTS metrics: multi-object tracking and segmentation precision (MOTSP), multi-object tracking and segmentation accuracy (MOTSA) and finally the soft multi-object tracking and segmentation accuracy (sMOTSA) that measures segmentation as well as detection and tracking quality. We also measure the average precision (AP), i.e. the normalised area under the precision/recall curve. We compare our model to the following baselines for video instance segmentation and report the in Table 2. • Single-frame embedding loss , previous state-of-the-art method where instance segmentations are propagated in time using intersection-over-union association. • Without temporal model, spatio-temporal embedding loss, without the temporal model. • Without depth, temporal model and spatio-temporal embedding loss, without the depth loss. We also report the of Mask R-CNN and Track R-CNN , even though direct comparison with the latter is not possible as their model was pretrained on Cityscapes and Mapillary Vistas, and is not causal as they use future frames to predict the current frame instance segmentations. Table 2: KITTI MOTS validation set comparing our model with baseline approaches. The static detection metrics (average precision, recall, precision) are evaluated image by image without taking into account the temporal consistency of instance segmentations. As the compared models (Without temporal model, Without depth, Ours) are all using the same mask network, they show similar performance in terms of detection. However, when evaluating performance on metrics that measures temporal consistency (MOTSA and sMOTSA), our best model shows significant improvement over the baselines. The variant without the temporal model performs poorly as it does not have any temporal context to learn a spatio-temporal embedding and therefore only relies on spatial appearance. The temporal model on the other hand learns with the temporal context and local motion, which in a better embedding. Our model, which learns to predict both a spatio-temporal embedding and monocular depth, achieves the best performance. In addition to using cues from appearance and temporal context, estimating depth allows the network to use information from the relative distance of objects to disambiguate them. Finally, we observe that our model outperforms Mask R-CNN on the temporal metrics (MOTSA and sMOTSA) even though the latter exhibits a higher detection accuracy, further demonstrating the temporal consistency quality of our spatio-temporal embedding. Our model first relies on the mask segmentation to isolate which pixel locations to consider for instance clustering. We evaluate the impact of using the ground truth mask against our predicted mask in Table 3. The performance gain is significant, hinting that a better instance segmentation would be possible by improving the mask network. Next, we evaluate the effect of clustering. In the best scenario, the validation loss would be zero, and the clustering would be perfect using the MeanShift algorithm. However, this scenario is unlikely and the clustering algorithm is affected by noisy embeddings. We evaluate the effect of this noise by clustering with the ground-truth mean for each instance, by thresholding with ρ r around the ground truth instance embedding mean. This also in a boost in the evaluation metrics, but most interestingly, a model that uses both ground truth instance embedding mean clustering and ground truth mask performs worse than a model segmented with ground truth mask and our clustering algorithm. This is because our clustering algorithm accumulates embeddings from past frames and therefore creates an attraction force for the mean shift algorithm that enables the instances to be matched more consistently. GT Table 3: Comparing the effect of noisy against ground-truth clustering and mask segmentation on the KITTI MOTS dataset. Our model learns a spatio-temporal embedding that clusters video-pixels from a given instance. Correspondence of instances between frames is achieved by matching detected instances to previous instances if the embedding distance is below the repulsion radius, p a. Instance tracking can occur for an arbitrarily long sequence of time, as long as the embedding changes smoothly over time, which is likely the case as temporal context and depth must evolve gradually. However, when the spatio-temporal embedding is trained over sequences which are too long, the embedding learning collapses. This is because the attractive loss term is detrimental between distant frames, it pressures pixels from the same instance to be have corresponding embeddings when their appearance and depth is no longer similar. It also suggests our model is able to reason over lower order motion cues more effectively than longer term dynamics. This is seen experimentally in Table 4. Table 4: Influence of the sequence length on model performance. This indicates that our model can learn short-term motion features effectively, but not long-term cues. We reason that this is because over longer sequences, the loss prevents the embedding smoothly shifting, which naturally occurs to changing pose, appearance, context and lighting in the scene. We find the optimum sequence length on this dataset to be five. The instance segmentation of our model is consistent across frames as instances are clustered in both space and time. This provides more robust clustering compared to a per-frame approaches. We demonstrate this with the following scenarios showing tracking through partial (Figure 3 and full occlusion ( Figure 5), as well as continuous tracking through noisy detections (Figure 4). Additional examples and failure cases of our model are shown in Appendix A.2 and a video demo can be viewed here: https://youtu.be/pqRPXRUlQ2I In each example, we show from left to right: RGB input image, ground truth instance segmentation, predicted instance segmentation, embedding visualised in 2D, embedding visualised in RGB and predicted monocular depth. The embedding is visualised in 2D with the corresponding mean shift clustering. Each color represents a different instance, the inner circle is the attraction radius of the instance mean embedding, and the outer circle is the repulsion radius of each instance. Additionally, we also visualise the embedding spatially in 3D, by projecting its three principal components to an RGB image. We show in Appendix A.2 that incorporating depth context greatly improves the quality of the embedding, especially in complex scenarios such as partial or total occlusion. We also observe that the embedding is much more structured when incorporating 3D geometry information. We proposed a new spatio-temporal embedding loss that generates consistent instance segmentation over time. The temporal network models the past temporal context and the depth network constrains the embedding to aid disambiguation between objects. We demonstrated that our model could effectively track occluded instances or instances with missed detections, by leveraging the temporal context. Our method advanced the state-of-the-art at video instance segmentation on the KITTI Multi-Object and Tracking Dataset. Encoder. The encoder is a ResNet-18 convolutional layer , with 128 output channels. Temporal model. The temporal model contains 12 residual 3D convolutional blocks, with only the first and last block convolving over time. Each residual block is the succession of: projection layer of kernel size 1×1×1 to halve the number of channels, 3D causal convolutional layer t×3×3, projection layer 1 × 1 × 1 to double the number of channels. We set the temporal kernel size to t = 2, and the number of output channels to 128. Decoders. The decoders for instance embedding and depth estimation are identical and consist of 7 convolutional layers with channels and 3 upsampling layers. The final convolutional layer contains p channels for instance embedding and 1 channel for depth. Depth Masking. During training, we remove from the photometric reprojection loss the pixels that violate the rigid scene assumption, i.e. the pixels whose appearance do not change between adjacents frames. We set the mask M to only include pixels where the reprojection error is lower with the warped imageÎ s→t than the unwarped source image I s: M = min s e(I t,Î s→t) < min s e(I t, I s) Pose Network. The pose network is the succession of a ResNet-18 model followed by 4 convolutions with channels. The last feature map is averaged to output a single 6-DoF transformation matrix. Mask Network. The mask network is trained separately to mask the and is the succession of the Encoder and Decoder described above. The following examples show qualitative and failure examples of our video instance segmentation model on the KITTI Multi-Object and Tracking Dataset. From left to right: RGB input image, ground truth instance segmentation, predicted instance segmentation, embedding visualised in 2D, embedding visualised in RGB and predicted monocular depth. We show that our model greatly benefits from depth estimation, with the learned embedding being more structured, and correctly tracking objects in difficult scenarios such as partial or total occlusion. (a) Without depth estimation. (b) With depth estimation Figure 10: Without depth, the car circled in red is wrongly tracked in frame 5 and 9, while our model correctly tracks it as the network has learned a consistent embedding based not only on appearance, but also on 3D geometry. Also, the RGB projection of the embedding from our model is considerably better and much more structured.

We introduce a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation, even with occlusions and missed detections, using appearance, geometry, and temporal context.

Represent the TLDR for finding the corresponding scientific article!

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.


Question: Represent the article for finding a single-sentence TLDR

Question: As reinforcement learning continues to drive machine intelligence beyond its conventional boundary, unsubstantial practices in sparse reward environment severely limit further applications in a broader range of advanced fields. Motivated by the demand for an effective deep reinforcement learning algorithm that accommodates sparse reward environment, this paper presents Hindsight Trust Region Policy Optimization (HTRPO), a method that efficiently utilizes interactions in sparse reward conditions to optimize policies within trust region and, in the meantime, maintains learning stability. Firstly, we theoretically adapt the TRPO objective function, in the form of the expected return of the policy, to the distribution of hindsight data generated from the alternative goals. Then, we apply Monte Carlo with importance sampling to estimate KL-divergence between two policies, taking the hindsight data as input. Under the condition that the distributions are sufficiently close, the KL-divergence is approximated by another f-divergence. Such approximation in the decrease of variance and alleviates the instability during policy update. Experimental on both discrete and continuous benchmark tasks demonstrate that HTRPO converges significantly faster than previous policy gradient methods. It achieves effective performances and high data-efficiency for training policies in sparse reward environments. Reinforcement Learning has been a heuristic approach confronting a great many real-world problems from playing complex strategic games (; ;) to the precise control of robots (; ;), in which policy gradient methods play very important roles . Among them, the ones based on trust region including Trust Region Policy Optimization (a) and Proximal Policy Optimization have achieved stable and effective performances on several benchmark tasks. Later on, they have been verified in a variety of applications including skill learning , multi-agent control , imitation learning , and have been investigated further to be combined with more advanced techniques (; ;). One unresolved core issue in reinforcement learning is efficiently training the agent in sparse reward environments, in which the agent is given a distinctively high feedback only upon reaching the desired final goal state. On one hand, generalizing reinforcement learning methods to sparse reward scenarios obviates designing delicate reward mechanism, which is known as reward shaping ; on the other hand, receiving rewards only when precisely reaching the final goal states also guarantees that the agent can focus on the intended task itself without any deviation. Despite the extensive use of policy gradient methods, they tend to be vulnerable when dealing with sparse reward scenarios. Admittedly, policy gradient may work in simple and sufficiently rewarding environments through massive random exploration. However, since it relies heavily on the expected return, the chances in complex and sparsely rewarding scenarios become rather slim, which often makes it unfeasible to converge to a policy by exploring randomly. Recently, several works have been devoted to solving the problem of sparse reward, mainly applying either hierarchical reinforcement learning (; ; ;) or a hindsight methodology, including Hindsight Experience Replay , Hindsight Policy Gradient and their extensions . The idea of Hindsight Experience Replay(HER) is to regard the ending states obtained through the interaction under current policy as alternative goals, and therefore generate more effective training data comparing to that with only real goals. Such augmentation overcomes the defects of random exploration and allows the agent to progressively move towards intended goals. It is proven to be promising when dealing with sparse reward reinforcement learning problems. For Hindsight Policy Gradient(HPG), it introduces hindsight to policy gradient approach and improves sample efficiency in sparse reward environments. Yet, its learning curve for policy update still oscillates considerably. Because it inherits the intrinsic high variance of policy gradient methods which has been widely studied in Schulman et al. (2015b), and. Furthermore, introducing hindsight to policy gradient methods would lead to greater variance . Consequently, such exacerbation would cause obstructive instability during the optimization process. To design an advanced and efficient on-policy reinforcement learning algorithm with hindsight experience, the main problem is the contradiction between on-policy data needed by the training process and the severely off-policy hindsight experience we can get. Moreover, for TRPO, one of the most significant property is the approximated monotonic converging process. Therefore, how these advantages can be preserved when the agent is trained with hindsight data also remains unsolved. In this paper, we propose a methodology called Hindsight Trust Region Policy Optimization (HTRPO). Starting from TRPO, a hindsight form of policy optimization problem within trust region is theoretically derived, which can be approximately solved with the Monte Carlo estimator using severely off-policy hindsight experience data. HTRPO extends the effective and monotonically iterative policy optimization procedure within trust region to accommodate sparse reward environments. In HTRPO, both the objective function and the expectation of KL divergence between policies are estimated using generated hindsight data instead of on-policy data. To overcome the high variance and instability in KL divergence estimation, another f -divergence is applied to approximate KL divergence, and both theoretically and practically, it is proved to be more efficient and stable. We demonstrate that on several benchmark tasks, HTRPO can significantly improve the performance and sample efficiency in sparse reward scenarios while maintains the learning stability. From the experiments, we illustrate that HTRPO can be neatly applied to not only simple discrete tasks but continuous environments as well. Besides, it is verified that HTRPO can be generalized to different hyperparameter settings with little impact on performance level. Reinforcement Learning Formulation and Notation. Consider the standard infinite-horizon reinforcement learning formulation which can be defined by tuple (S, A, π, ρ 0, r, γ). S represents the set of states and A denotes the set of actions. π: S → P(A) is a policy that represents an agent's behavior by mapping states to a probability distribution over actions. ρ 0 denotes the distribution of the initial state s 0. Reward function r: S → R defines the reward obtained from the environment and γ ∈ is a discount factor. In this paper, the policy is a differentiable function regarding parameter θ. We follow the standard formalism of state-action value function Q(s, a), state value function V (s) and advantage function A(s, a) in. We also adopt the definition of γ-discounted state visitation distribution as ρ θ (s) = (1 − γ) et al., 2016), in which the coefficient 1 − γ is added to keep the integration of ρ θ (s) as 1. Correspondingly, γ-discounted state-action visitation distribution , also known as occupancy measure , is defined as ρ θ (s, a) = ρ θ (s) × π θ (a|s), in which π θ (a|s) stands for the policy under parameter θ. Trust Region Policy Optimization(TRPO). Schulman et al. (2015a) proposes an iterative trust region method that effectively optimizes policy by maximizing the per-iteration policy improvement. The optimization problem proposed in TRPO can be formalized as follows: in which ρθ(s) = ∞ t=0 γ t P (s t = s). θ denotes the parameter of the new policy whileθ is that of the old one. Trajectory is represented by τ = s 1, a 1, s 2, a 2,.... The objective function L T RP O (θ) can be given out in the form of expeted return: Hindsight Policy Gradient(HPG). After generalizing the concept of hindsight, combines the idea with policy gradient methods. Though goal-conditioned reinforcement learning has been explored for a long time and actively investigated in recent works (; ; ; ; ; ;), HPG firstly extends the idea of hindsight to goal-conditioned policy gradient and shows that the policy gradient can be computed in expectation over all goals. The goal-conditioned policy gradient is derived as follows: Then, by applying hindsight formulation, it rewrites goal-conditioned policy gradient with trajectories conditioned on some other goal g using importance sampling to improve sample efficiency in sparse-reward scenarios. In this paper, we propose an approach that introduces the idea of hindsight to TRPO, called Hindsight Trust Region Policy Optimization(HTRPO), aiming to further improve policy performance and sample efficiency for reinforcement learning with sparse rewards. In Section 3 and Section 4, we demonstrate how to redesign the objective function and the constraints starting from TRPO respectively. In order to apply hindsight methodology, this section presents the main steps for the derivation of HTRPO objective function. Starting from the original optimization problem in TRPO, the objective function can be written in the following variant form: The derivation process of this variant form is shown explicitly in Appendix A.1 and in Schulman et al. (2015a). Given the expression above, we consider the goal-conditioned objective function of TRPO as a premise for hindsight formulation. Similar to equation 4, Lθ(θ) can be correspondingly given out in the following form: For the record, though it seems that equation 6 makes it possible for off-policy learning, it can be used as the objective only when policy π θ is close to the old policy πθ, i.e. within the trust region. Using severely off-policy data like hindsight experience will make the learning process diverge. Therefore, importance sampling need to be integrated to correct the difference of the trajectory distribution caused by changing the goal. Based on the goal-conditioned form of the objective function, the following theorem gives out the hindsight objective function conditioned on some goal g with the distribution correction derived from importance sampling. Theorem 3.1 (HTRPO Objective Function). For the original goal g and an alternative goal g, the object function of HTRPO Lθ(θ) is given by: in which, τ = s 1, a 1, s 2, a 2,..., s t, a t. Appendix A.2 presents an explicit proof on how the hindsight-form objective function derives from equation 6. It will be solved under a KL divergence expectation constraint, which will be discussed in detail in Section 4. Intuitively, equation 7 provides a way to compute the expected return in terms of the advantage with new-goal-conditioned hindsight experiences which are generated from interactions directed by old goals. Naturally, Theorem 3.2 gives out the gradient of HTRPO objective function that will be applied to solve the optimization problem. Detailed steps of computing the gradient is presented in Appendix A.3. Theorem 3.2 (Gradient of HTRPO Objective Function). For the original goal g and an alternative goal g, the gradient ∇ θ Lθ(θ) of HTRPO object function with respect to θ is given by the following expression: in which τ = s 1, a 1, s 2, a 2,..., s t, a t. This section firstly demonstrates some techniques, with strict proof, that can be used to estimate the expectation of KL-divergence and further reduce the variance, and then presents how hindsight is applied to the constraint function of TRPO. In TRPO, the KL divergence expectation under ρθ(s) is estimated by averaging all the values of KL divergence. When they are respectively conditioned on all states collected using the old policy, this kind of estimation is exactly Monte Carlo estimation which is unbiased. However, when we only have access to hindsight experience data, the state distribution may inevitably change and the previous method for estimating the expectation of KL divergence is no longer valid. To solve this problem, we firstly transform the KL divergence to an expectation under occupancy measure ρθ(s, a) = ρθ(s) × πθ(a|s). It can be estimated using collected state-action pair (s, a), whose changed distribution can be corrected by importance sampling. Then, by making use of another f -divergence, the variance of estimation is theoretically proved to be reduced so as to facilitating a more stable training. The constraint function in KL-divergence can be naturally converted to a logarithmic form. Appendix B.1 provides a more explicit version of this conversion. Theorem 4.1 (Logarithmic Form of Constraint Function). Given two policies πθ(a|s) and π θ (a|s), the expectation of their KL-divergence over states s ∼ ρθ(s) is written as: However, simply expanding the KL-divergence into logarithmic form still leaves several problems unhandled. Firstly, the variance remains excessively high, which would cause considerable instability during the learning process. Secondly, current estimation of KL-divergence is of possible negativity. If encountering negative expectation of KL-divergence, the learning process would in fatal instability. The following Theorem 4.2 describes a technique to reduce the variance and Theorem 4.3 gives out the strict proof for the decrease of variance. Theorem 4.2 (Approximation of Constraint Function). For policy πθ(a|s) and π θ (a|s), and for η = π θ (a|s) − πθ(a|s), Theorem 4.2 demonstrates that when θ andθ is of limited difference, the expectation of log πθ(a|s)− log π θ (a|s) can be sufficiently estimated by the expectation of its square. The proof is provided in Appendix B.2. In fact, Es,a∼ρθ(s,a) 1 2 (log πθ(a|s) − log π θ (a|s)) 2 is the expectation of an fdivergence, where f (x) = 1 2 x(log x) 2. Noticeably, f (x) is a strictly convex function when x ∈ (1 e, ∞), and f = 0. Moreover, it is noteworthy that there are two corresponding major improvements through this kind of estimation. Firstly, it is guaranteed to reduce the variance which leads to a more stable performance. This merit will be explained in detail in Theorem 4.3. Another significant improvement is manifested in the elimination of negative KL-divergence, since the estimation presents itself in the form of a square which is always non-negative. log πθ(a|s) − log π θ (a|s). Theorem 4.3 illustrates that there is a decrease from the variance of log πθ(a|s) − log π θ (a|s) to the variance of its square, and furthermore indicates that the variance is effectively reduced. The proof is given in detail in Appendix B.3. In fact, the closer it is betweenθ and θ, the more the variance decreases. Based on Theorem 4.1 to Theorem 4.3, in this paper, we adopt the following form of constraint condition: In Theorem 4.4, we demonstrate that hindsight can also be introduced to the constraint function. The proof follows the methodology similar to that in Section 3, and is deducted explicitly in Appendix B.4. Theorem 4.4 (HTRPO Constraint Function). For the original goal g and an alternative goal g, the constraint between policy πθ(a|s) and policy π θ (a|s) is given by: in which = 1−γ. Theorem 4.4 implies the practicality of using hindsight data under condition g to estimate the expectation. From all illustration above, we give out the final form of the optimization problem for HTRPO: The solving process for HTRPO optimization problem is explicitly demonstrated in Appendix C and the complete algorithm procedure is included in Appendix D. This section demonstrates the validation of HTRPO on several sparse reward benchmark tasks 1. The design of our experiments aims to conduct an in-depth investigation in the following aspects: • How is the effectiveness of HTRPO? • How does each component of HTRPO contribute to its effectiveness? • How is the performance of policy gradient methods trained with hindsight data in continuous environments? • How sensitive is HTRPO to network architecture and some key parameters? We implement HTRPO on a variety of reinforcement learning environments, including Bit Flipping, Grid World and Fetch. Among them, Bit Flipping, Grid World, Fetch Reach and Fetch Push are implemented as descrete-action environments while we also conduct continuous version of experiments in Fetch Reach, Fetch Push and Fetch Slide. A glimpse of these environments is demonstrated in Figure 1 while the detailed introductions are included in Appendix F.1. The reward mechanisms are intentionally modified to sparse reward regulations. Besides, for continuous version of Fetch experiments, we apply an additional policy entropy bonus to encourage more exploration. For each trail of interaction, reward for the agent is set as the remaining number of time steps plus one, and all goals during exploration are chosen uniformly at random for both training and evaluation. During the training process, we terminate one episode either when the maximum number of time steps has elapsed or when the goal state is reached. We evaluate agents' performance by documenting 10 learning trails in the form of average return and their corresponding standard deviation. In Bit Flipping and Grid World environments, the network architecture is of two hidden layers, each with 64 hyperbolic tangent units; in Fetch environment, for both discrete and continuous implementations, the network contains two 256-unit hidden layers. For all environments mentioned above, we compare HTRPO with HPG and TRPO (a), which are chosen as the baseline algorithms. Since HPG is never applied to continuous environments in , we implement HPG to be adapted to continuous environments. Note that the way we scale the time axis is significantly different from that in. Instead of regarding a certain number of training batches as interval between evaluation steps, we directly uses the accumulated time steps the agent takes while interacting with the environments throughout episodes and batches. Besides comparing with baselines, we also ablate each component of HTRPO to investigate how significant it is for the final performance. To be specific, we adopt the "vanilla" estimation of KLdivergence which we call "HTRPO with KL1" instead of the proposed one in Section 4; we also observe the performance of our algorithm without weighted importance sampling, which is denoted as "HTRPO without WIS" in this paper. In discrete environments, we test both the official version of HPG released in and our HPG implementation while for continuous environments of Fetch, we only test our HPG due to the lack of surpport for continuous tasks in. We apply input normalization in et al., the officially released version of HPG eventually converges to similar performances with that of HTRPO in discrete environments, but sometimes, unlike our HPG, it is still far from converging under this time-step evaluation setting. This kind of distinction in converging speed between our HPG and the official HPG may be caused by the reduction of noises, since we use TDerror to update policies instead of the return corrected by importance sampling, which is adopted in HPG. Thus, for the fairness of comparison, in the following analysis, we mainly compare the properties between HTRPO and our HPG. From the we can see that in both discrete and continuous environments, HTRPO outperforms HPG significantly. Aside from assuring a good converging property, the sample efficiency of HTRPO also exceed that of HPG, for it reaches a higher average return within less time in most environments. As for TRPO, though it can converge in several simple tasks like Bit Flipping, Grid World and continuous Fetch Reach, it remains incompetent in dealing with complex control tasks including Fetch Push and Fetch Slide, in all of which HTRPO can learn a good policy. The reason is that for TRPO, it is basically impossible to acquire a positive reward at the beginning of the training in such environments, which makes the policy updates meaningless. How does each component of HTRPO contribute to its effectiveness? In both Figure 2 and Figure 3, "HTRPO with KL1" and "HTRPO without WIS" performs much worse than the complete version of HTRPO. When we estimate the KL-divergence using the "vanilla" KL-divergence defined as equation 9, it causes severe instability, which means that the estimated KL-divergence can be negative with an unacceptable probability. Considering the practicality of the experiment, the corresponding iteration will be skipped without any updates of the policy in this senario. Given the phenomenon stated above, the final performance of "HTRPO with KL1" is much worse and more unstable in all environments. As for the study of Weighted Importance Sampling, it is widely known for significantly reducing the variance , which is once again proved by the of "HTRPO without WIS". Admittedly, we can see that the performance of "HTRPO without WIS" matches the full version of HTRPO in several simple environments in Figure 2 (a)-(d) and Figure 3 (a). However, for more complex environments like Fetch Push and Fetch Slide, the variance is detrimentally larger than that in simple environments. In short, the performance of "HTRPO without WIS" has a severe degradation comparing to the full version of HTRPO. How is the performance of policy gradient methods trained with hindsight data in continuous environments? As mentioned in , it still remains unexplored that to what extent the policy gradient methods trained with hindsight data can solve continuous control tasks. In this section, we will provide the answer. We implement HTRPO in continuous control tasks including Fetch Reach, Fetch Push and Fetch Slide. HPG is tested as well for comparison. From the , we can see that with the help of input normalization, HPG can learn a valid policy in continuous control tasks. Still, HTRPO performs much better than HPG in all three environments, benefiting from a faster and more stable convergence. As illustrated in Figure 3, HTRPO eventually achieves an average success rate of 92% for Fetch Push and 82.5% for Fetch Slide. How sensitive is HTRPO to network architecture and some key parameters? To study the sensitivity of HTRPO to different network architectures, we observe the performance of HTRPO with different network settings. From the demonstrated in Appendix F.2.1, HTRPO achieves commendable performances with all three different network architectures while HPG only converges under certain settings. As for the sensitivity of HTRPO to key parameters, we mainly observe the impact of different number of alternative goals. Based on the learning curves in Appendix F.2.2, we can see that Hindishgt TRPO with more alternative goals achieves better converging speed. We have extended the monotonically converging on-policy algorithm TRPO to accommodate sparse reward environments by adopting the hindsight methodology. The optimization problem in TRPO is scrupulously derived into hindsight formulation and, when the KL-divergence in the constraint function is small enough, it can be tactfully approximated by another f -divergence in order to reduce estimation variance and improve learning stability. Experimental on a variety of environments demonstrate effective performances of HTRPO, and validate its sample efficiency and stable policy update quality in both discrete and continuous scenarios. Therefore, this work reveals HTRPO's vast potential in solving sparse reward reinforcement learning problems. We greatly acknowledge all the fundings in support of this work. With no influence to the optimal solution, we can multiply equation 3 by a constant A.2 THEOREM 3.1 Theorem 3.1 (HTRPO Objective Function). For the original goal g and an alternative goal g, the object function of HTRPO Lθ(θ) is given by: in which, τ = s 1, a 1, s 2, a 2,..., s t, a t. Proof. Starting from equation 6, for every time step t in the expectation, denote so that Split every trajectory τ into τ 1 and τ 2 where τ 1 = s 1, a 1, s 2, a 2,..., s t, a t and τ 2 = s t+1, a t+1,..., then For that γ t π θ (at|st,g) πθ(at|st,g) Aθ(s t, a t, g) is independent from τ 2 conditioned on τ 1, t,a1:t∼pθ(s1:t,a1:t|g) Thus, t,a1:t∼pθ(s1:t,a1:t|g) Following the techniques of importance sampling, the objective function can be rewritten in the form of new goal g: after expanding the objective function and cancelling terms, A.3 THEOREM 3.2 Theorem 3.2 (Gradient of HTRPO Objective Function). For the original goal g and an alternative goal g, the gradient ∇ θ Lθ(θ) of HTRPO object function with respect to θ is given by the following expression: in which τ = s 1, a 1, s 2, a 2,..., s t, a t. Proof. Starting from equation 24, since that π θ (a t |s t, g) is the only term relavant to θ, the corresponding gradient of the objective function is computed by: log πθ(a|s) − log π θ (a|s) Proof. Expand the expectation in equation 2 by the definition of KL-divergence, B.2 THEOREM 4.2 Lemma B.1 Given two distibutions p(x) and q(x), q(x) = p(x) + η(x), in which η(x) is the variation of q(x) at p(x). Proof. Consider the second order Taylor expansion of log q(x) at p(x), For the left side of equation 27, For the first term on the right side of equation 27, Theorem 4.2 (Approximation of Constraint Function). For policy πθ(a|s) and π θ (a|s), and for η = π θ (a|s) − πθ(a|s), in which Var(Y) denotes the variance of Y. Proof. Denote There always exists in which µ 1 is a constant. Then the equation can be converted by the following steps: Thus, when Y = Y 0, the two factors in equation 34, (X 1 − µ 1) and (X 2 − E(X2)) equal to 0 simultaneously. Also, it is easy to notice that when Y ∈ [0, 0.5], X 1 and X 2 are strictly increasing with the increase of Y. Thus, (X 1 − µ 1) and (X 2 − E(X2)) are either both positive or both negative, if not zero. Therefore, Lemma B.3 For any random variable Y, in which Var(Y) denotes the variance of Y. Proof. Apparently, Consequently, For that we have log πθ(a|s) − log π θ (a|s). With the transitivity of inequality, combining equation 40 and equation 41, we know that log πθ(a|s) − log π θ (a|s). B.4 THEOREM 4.4 Theorem 4.4 (HTRPO Constraint Function). For the original goal g and an alternative goal g, the constraint between policy πθ(a|s) and policy π θ (a|s) is given by: in which = 1−γ. Proof. Starting from equation 9, the constraint condition is written as: Multiply the constraint function by a constant Denote the constraint function as fθ(θ), To write the constraint function in goal-conditioned form, In a similar way with the proof for Theorem 3.1, denote every time step of fθ(θ) as fθ(θ, t), in other words, for trajectory τ 1 = s 1, a 1, s 2, a 2,..., s t, a t and τ 2 = s t+1, a t+1,...,. For that 1 2 (log πθ(a t |s t, g) − log π θ (a t |s t, g)) 2 is independent from τ 2 conditioned on τ 1, Accordingly, Furthermore, by importance sampling, for a new goal g, the constraint can be converted to the following form. in which τ = s 1, a 1, s 2, a 2,..., s t, a t. Denote = 1−γ. Based on equation 23, by expanding and canceling terms, the constraint condition can be written as C SOLVING PROCESS FOR HTRPO Based on the final form of HTRPO optimization problem, this section completes the feasibility of this algorithm with estimators for the objective function and the KL-divergence constraint. is obtained from interacting with the environment under a goal g (i). In order to generate hindsight experience, we also need to sample a set of alternative goals G = {g . The Monte Carlo estimation of HTRPO optimization problem with dataset D can be derived as follows: in which λ = N ·N g and g is supposed to follow uniform distribution. However, in experiments, we follow the alternative goal sampling method of HPG . As a , the goals of training data actually follow the distribution of alternative goals instead of uniform distribution, and the objective and KL expectation will be estimated w.r.t. the alternative goal distribution. Therefore, during the learning process, our algorithm is encouraging the agent to achieve the alternative goals. Such mechanism serves as a mutual approach for all hindsight methods (; ;), which can be seen as a merit, for the intention is to guide the agent to achieve the alternative goals and then generalize to the original goals. However, as discussed in , this kind of estimation may in excessive variance, which leads to an unstable learning curve. In order to avoid instability, we adopt the technique of weighted importance sampling introduced in and further convert the optimization problem to the following form: We provide an explicit solution method for the optimization problem above in Appendix C.2. While introducing weighted importance sampling may cause a certain level of bias which is identical to that of HPG , the bias is to decrease in inverse ratio with regard to the increase of the data theoretically . Given the limited resources, we need to tradeoff between reducing bias and enlarging batch size. By picking a appropriate batch size, the improvement of weighted importance sampling is well demonstrated in the experiments. For the HTRPO optimization problem, briefly denote the optimization problem in expression 55 and 56 as: max For any policy parameter θ in the neighborhood of the parameterθ, approximate the optimization problem with linear objective function and quadratic constraint: Noticeably, g(θ) = 0 and ∇ θ g(θ) = 0, which further simplifies the optimization problem to the following form: max Given a convex optimzation problem with a linear objective function under a quadratic constraint, many well-practiced approaches can be taken to solve the problem analytically, among which we adopt the Karush-Kuhn-Tucker(KKT) conditions . For a Lagrangian multiplier λ, Expressions in 60 form the KKT conditions of the optimization problem. Solving the KKT conditions, The policies, however, in this paper are in the form of a neural network, which makes it extremely time-comsuming to compute the Hessian matrix. Thus, we compute [∇ 2 θ g(θ)] −1 ∇ θ f (θ) with conjugate gradient algorithm by solving the following equation: in which [∇ 2 θ g(θ)]x can be practically calculated through the following expansion: Sample a trajectory τ = {(s t, a t, r t, s t+1, g, π θ (a t |s t, g))} T t=1 using current policy θ; 4: end while Collecting data 6: Sample alternative goals G = {g from achieved goals in B origin ; 7: for for τ in B origin do 10: for t = 0 to T do Compute π θ (a t |s t, g (i) ); Modify reward r t |g → r t |g (i); 13: end for 14: 15: In this section, we provide a more comprehensive demonstration for the experiments of HTRPO. In detail, section F.1 narrates a full introduction to each environment; section F.2 gives out the sensitivity analysis of HTRPO including the performance under different network architectures and different numbers of alternative goals, in which we strictly adopt the control variable method and only the studied parameter is altered; section F.3 shows the supplementary materials of the experimental data including learning curves and success rates during the training process. We finetune the hyperparameters according to experience without hyperparameter search due to limited computing resources. k-Bit Flipping. In each episode of this experiment, two arrays of length k are generated. The first array is initialized with all 0's while the second one, usually regarded as the target array, is generated randomly. At each time step, the agent is able to flip one bit of the first array from 0 to 1 or from 1 to 0. Once the first array is exactly the same with the target array, the agent reaches the goal state and is then rewarded. The maximum number of time steps is k. In this experiment, we observe the performance of HTRPO under conditions that k = 8 and that k = 16 respectively. The general process of an 8-Bit Flipping task is demonstrated in Figure 1 (a). Grid World. In this experiment, the agent starts at a position of an 11 × 11 grid with intransitable obstacles, and is trying to reach another randomly chosen position in this grid. The agent is allowed to move up, down, left and right at each time step. Moving into obstacles makes the agent remain in its current position. States of this environment is represented by 2-dimensional integral coordinates and the maximum number of time steps is 32. In Empty Maze environment, there is no impassable obstacles other than the outer boundary, and the agent starts at the upper-left corner of the grid. In Four Rooms environment , walls separate the grid into 4 rooms, each with access to its adjacent rooms through single openings. Example cases of Empty Maze and Four Rooms environments adopted in this paper are demonstrated in Figure 1 (b) and (c). Fetch. Fetch environment contains a 7-DoF Fetch robotic arm with a two-fingered parallel gripper in simulation . In Fetch Reach environment, a target position is randomly chosen and the gripper of Fetch robotic arm needs to be moved upon it. In Fetch Push, the task for the robotic arm is to push a randomly placed block towards the goal state, anther randomly picked position, which is represented by a 3-dimensional Cartesian coordinate. In Fetch Slide, the robotic arm needs to exert a force on the block for it to slide towrds a chosen goal at a certain distance. A pictorial demonstration of this environment is shown in Figure 1 In this experiment, we observe the performance of HTRPO with different network architectures. Specially, we implement the proposed algorithm under 3 different network settings, i.e. networks with a 16-unit layer, two 64-unit layers and two 256-unit layers respectively. For the record, all parameters and other settings remain the same aside from the network architecture. As demonstrated in Figure 4, each row shows the performance under different network architecture settings for each environment. A general can be drawn that networks with more hidden layers and more neurons help to speed up the convergence. However, one difference is that for HTRPO, it converges quickly in all the settings, while for HPG, it converges much slower especially when the network architecture is simple. We believe that the iteratively searching of optimal solution in the trust region helps the network converge rapidly and is more robust to different network architecture. In this experiment, how the number of alternative goals, as a key parameter, affects the performance of HTRPO is studied. We conduct all the experiments, both discrete and continuous with different number of alternative goals. For discrete environments, we set the number of alternative goals to be 10, 30, 100 and ∞ in turn. For continuous environments, we compare the performance under 10, 30, 100 alternative goals respectively. The evaluation curves are shown in 5. From the , we can see that in simple discrete environments, ∞ alternative goals produce the fastest convergence. In complex and continuous environments, 30 and 100 alternative goals lead to comparatively good performance. It is not hard to see that Hindishgt TRPO with more alternative goals achieves better converging speed, which may be credited to the corresponding increase on training samples. This is, to some extent, similar to data augmentation. In this section, we demonstrate the success rates of HTRPO during both evaluation and training. For the record, the actions during the training process are sampled by the distribution output by the network while during the evaluation process, we adopt a greedy strategy to choose the action by the mean value of the distribution. Table 3 lists the success rates of Fetch Push and Fetch Slide during evaluation, in which the ultimate values reflect the mean computed with 1000 test in each iteration. They are the only two environments listed for they are the most complex ones. Figure 7 illustrates the success rate curves during the training process. Figure 9 demonstrates the estimation of KL divergence expectation in the training process. From these data we can see that in the experiments, the approximation of equation 13 can significantly reduce the variance of KL expectation estimation. Besides, the comparison of performance between HTRPO and HTRPO with KL1 also shows the efficiency of this approximation, which helps improve the final performance significantly. Both "HTRPO" and "HTRPO without WIS" use the estimation method in equation 13 with one difference being that "HTRPO without WIS" doesn't adopt weighted importance sampling. Thus, from Figure 9, we can see that "HTRPO" demonstrates the least variance. HTRPO HTRPO with KL1 HTRPO without WIS The curves for KL1 are comparatively lower than those of equation 13. Note that in TRPO, the linear search mechanism adjust the updating step size according to the estimation of KL divergence expectation. It sets a threshold to constrain the KL divergence. For those above the threshold, the updating step size will be reduced to ensure that the estimation of KL divergence estimation falls within the threshold. This explains why the curves for KL1 are comparatively lower. However, since the estimation of KL divergence expectation in HTRPO falls near the expected value, such step size adjustment is rarely triggered. This benefits from the much lower variance of equation 13.

This paper proposes an advanced policy optimization method with hindsight experience for sparse reward reinforcement learning.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.