Question: Represent the scientific article to find a short TLDR

Question: Capsule Networks have shown encouraging on \textit{defacto} benchmark computer vision datasets such as MNIST, CIFAR and smallNORB. Although, they are yet to be tested on tasks where the entities detected inherently have more complex internal representations and there are very few instances per class to learn from and where point-wise classification is not suitable. Hence, this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points. In doing so we introduce \textit{Siamese Capsule Networks}, a new variant that can be used for pairwise learning tasks. We find that the model improves over baselines in the few-shot learning setting, suggesting that capsule networks are efficient at learning discriminative representations when given few samples. We find that \textit{Siamese Capsule Networks} perform well against strong baselines on both pairwise learning datasets when trained using a contrastive loss with $\ell_2$-normalized capsule encoded pose features, yielding best in the few-shot learning setting where image pairs in the test set contain unseen subjects. Convolutional Neural networks (CNNs) have been a mainstay model for a wide variety of tasks in computer vision. CNNs are effective at detecting local features in the receptive field, although the spatial relationship between features is lost when crude routing operations are performed to achieve translation invariance, as is the case with max and average pooling. Essentially, pooling in viewpoint invariance so that small perturbations in the input do not effect the output. This leads to a significant loss of information about the internal properties of present entities (e.g location, orientation, shape and pose) in an image and relationships between them. The issue is usually combated by having large amounts of annotated data from a wide variety of viewpoints, albeit redundant and less efficient in many cases. As noted by hinton1985shape, from a psychology perspective of human shape perception, pooling does not account for the coordinate frames imposed on objects when performing mental rotation to identify handedness BID23; BID19 BID12. Hence, the scalar output activities from local kernel regions that summarize sets of local inputs are not sufficient for preserving reference frames that are used in human perception, since viewpoint information is discarded. Spatial Transformer Networks (STN) BID13 have acknowledged the issue by using dynamic spatial transformations on feature mappings to enhance the geometric invariance of the model, although this approach addresses changes in viewpoint by learning to remove rotational and scale variance, as opposed to viewpoint variance being reflected in the model activations. Instead of addressing translation invariance using pooling operations, BID8 have worked on achieving translation equivariance. The recently proposed Capsule Networks BID24; BID7 have shown encouraging to address these challenges. Thus far, Capsule Networks have only been tested on datasets that have a relatively sufficient number of instances per class to learn from and utilized on tasks in the standard classification setup. This paper extends Capsule Networks to the pairwise learning setting to learn relationships between whole entity encodings, while also demonstrating their ability to learn from little data that can perform few-shot learning where instances from new classes arise during testing (i.e zero-shot prediction). The Siamese Capsule Network is trained using a contrastive loss with 2 -normalized encoded features and demonstrated on face verification tasks. BID8 first introduced the idea of using whole vectors to represent internal properties (referred to as instantiation parameters that include pose) of an entity with an associated activation probability where each capsule represents a single instance of an entity within in an image. This differs from the single scalar outputs in conventional neural networks where pooling is used as a crude routing operation over filters. Pooling performs sub-sampling so that neurons are invariant to viewpoint change, instead capsules look to preserve the information to achieve equivariance, akin to perceptual systems. Hence, pooling is replaced with a dynamic routing scheme to send lowerlevel capsule (e.g nose, mouth, ears etc.) outputs as input to parent capsule (e.g face) that represent part-whole relationships to achieve translation equivariance and untangles the coordinate frame of an entity through linear transformations. The idea has its roots in computer graphics where images are rendered given an internal hierarchical representation, for this reason the brain is hypothesized to solve an inverse graphics problem where given an image the cortex deconstructs it to its latent hierarchical properties. The original paper by BID24 describes a dynamic routing scheme that represent these internal representations as vectors given a group of designated neurons called capsules, which consist of a pose vector u ∈ R d and activation α ∈. The architecture consists of two convolutional layers that are used as the initial input representations for the first capsule layer that are then routed to a final class capsule layer. The initial convolutional layers allow learned knowledge from local feature representations to be reused and replicated in other parts of the receptive field. The capsule inputs are determined using a Iterative Dynamic Routing scheme. A transformation W ij is made to output vector u i of capsule C L i. The length of the vector u i represents the probability that this lower-level capsule detected a given object and the direction corresponds to the state of the object (e.g orientation, position or relationship to upper capsule). The output vector u i is transformed into a prediction vectorû j|i, whereû j|i = W ij u i. Then,û j|i is weighted by a coupling coefficient c ij to obtain s j = i c ijûj|i, where coupling coefficients for each capsule j c ij = 1 and c ij is got by log prior probabilities b ij from a sigmoid function, followed by the softmax, c ij = e bij / k e b ik. Ifû L j|i has high scalar magnitude when multiplied by u L+1 j then the coupling coefficient c ij is increased and the remaining potential parent capsules coupling coefficients are decreased. Routing By Agreement is then performed using coincidence filtering to find tight clusters of nearby predictions. The entities output vector length is represented as the probability of an entity being present by using the nonlinear normalization shown in Equation 1 where vote v j is the output from total input s j, which is then used to compute the agreement a ij = v jûj|i that is added to the log prior b ij. The capsule is assigned a high log-likelihood if densely connected clusters of predictions are found from a subset of s. The centroid of the dense cluster is output as the entities generalized pose. This coincidence filtering step can also be achieved by traditional outlier detection methods such as Random sample consensus (RANSAC) BID3 BID3 classical for finding subsets of the feature space with high agreement. Although, the motivation for using the vector normalization of the instantiation parameters is to force the network to preserve orientation. Lastly, a reconstruction loss on the images was used for regularization which constrains th capsules to learn properties that can better encode the entities. In this paper, we do not use such regularization scheme by autoencoding pairs of input images, instead we use a variant of dropout. BID7 recently describe matrix capsules that perform routing by agreement using the expectation maximization (EM) algorithm, motivated by computer graphics where pose matrices are used to define rotations and translations of objects to account for viewpoint changes. Each parent capsule is considered a Gaussian and the pose matrix of each child capsule are considered data samples of the Gaussian. A given layer L contains a set of capsules C L such that ∀C ∃ {M, α} ∈ C L where pose matrix M ∈ R n×n (n = 4) and activation α ∈ are the outputs. A vote is made V ij = M i W ij for the pose matrix of C L+1 j where W ij ∈ R n×n is a learned viewpoint invariant transformation matrix from capsule where the cost h j is the negative log-probability density weighted by the assignment probabilities r ij, −β u is the negative log probability density per pose matrix computed to describe C L+1 j. If C L+1 j is activated −β a is the cost for describing (µ j, σ 2 j) from lower-level pose data samples along with r ij and λ is the inverse temperature so as the assignment probability becomes higher the slope of the sigmoid curve becomes steeper (represents the presence of an entity instead of the nonlinear vector normalization seen in Equation 1). The network uses 1 standard convolutional layer, a primary capsule layer, 2 intermediate capsule convolutional layer, followed by the final class capsule layer. The matrix capsule network significantly outperformed CNNs on the SmallNORB dataset. LaLonde & Bagci FORMULA7 introduce SegCaps which uses a locally connected dynamic routing scheme to reduce the number of parameters while using deconvolutional capsules to compensate for the loss of global information, showing best performance for segmenting pathological lungs from low dose CT scans. The model obtained a 39% and 95% reduction in parameters over baseline architectures while outperforming both. Bahadori FORMULA7 introduced Spectral Capsule Networks demonstrated on medical diagnosis. The method shows faster convergence over the EM algorithm used with pose vectors. Spatial coincidence filters align extracted features on a 1-d linear subspace. The architecture consists of a 1d convolution followed by 3 residual layers with dilation. Residual blocks R are used as nonlinear transformations for the pose and activation of the first primary capsule instead of the linear transformation that accounts for rotations in CV, since deformations made in healthcare imaging are not fully understood. The weighted votes are obtained as s j,i = α i R j (u i) ∀i where S j is a matrix of concatenated votes that are then decomposed using SVD, where the first singular value dimensions 1 is used to capture most of the variance between votes, thus the activation a j activation is computed as σ η(s DISPLAYFORM0 k is the ratio of all variance explained for all right singular vectors in V, b is optimized and η is decreased during training. The model is trained by maximizing the log-likelihood showing better performance than the spread loss used with matrix capsules and mitigates the problem of capsules becoming dormant. BID31 formalize the capsule routing strategy as an optimization of a clustering loss and a KL regularization term between the coupling coefficient distribution and its past states. The proposed objective function follows as min C,S {L(C, S):= − i j c ij o j|i, s j + α i j c ij log c ij } where o j|i = T ij µ i /||T ij || F and ||T ij || F is the Frobenious norm of T ij. This routing scheme shows significant benefit over the original routing scheme by BID24 as the number of routing iterations increase. Evidently, there has been a surge of interest within the research community. In contrast, the novelty presented in this paper is the pairwise learning capsule network scheme that proposes a different loss function, a change in architecture that compares images, aligns entities across images and describes a method for measuring similarity between final layer capsules such that inter-class variations are maximized and intra-class variations are minimized. Before describing these points in detail, we briefly describe the current state of the art work (SoTA) in face verification that have utilized Siamese Networks. Siamese Networks (SNs) are neural networks that learn relationships between encoded representations of instance pairs that lie on low dimensional manifold, where a chosen distance function d ω is used to find the similarity in output space. Below we briefly describe state of the art convolutional SN's that have been used for face verification and face recognition. BID27 presented a joint identification-verification approach for learning face verification with a contrastive loss and face recognition using cross-entropy loss. To balance loss signals for both identification and verification, they investigate the effects of varying weights controlled by λ on the intra-personal and inter-personal variations, where λ = 0 leaves only the face recognition loss and λ → ∞ leaves the face verification loss. Optimal are found when λ = 0.05 intra personal variation is maximized while both class are distinguished. BID32 propose a center loss function to improve discriminative feature learning in face recognition. The center loss function proposed aims to improve the discriminability between feature representations by minimizing the intra-class variation while keeping features from different classes separable. The center loss is given as DISPLAYFORM0 where z = W T j x i + b j. The c yi is the centroid of feature representations pertaining to the i th class. This penalizes the distance between class centers and minimizes the intra-class variation while the softmax keeps the inter-class features separable. The centroids are computed during stochastic gradient descent as full batch updates would not be feasible for large networks. BID18 proposed Sphereface, a hypersphere embedding that uses an angular softmax loss that constrains disrimination on a hypersphere manifold, motivated by the prior that faces lie on a manifold. The model achieves 99.22 % on the LFW dataset, and competitive on Youtube Face (YTF) and MegaFace. BID25 proposed a triplet similarity embedding for face verification using a triple loss arg min W = α,p,n∈T max(0, α + α T W T W (n − p)) where for T triplet sets lies an anchor class α, positive class p and negative class n, a projection matrix W, (performed PCA to obtain W 0) is minimized with the constraint that W BID9 use deep metric learning for face verification with loss arg min DISPLAYFORM1 DISPLAYFORM2 F | where g(z) = log(1 + e βz)/β, β controls the slope steepness of the logistic function, ||A|| F is the frobenius norm of A and λ is a regularization parameter. Hence, the loss function is made up of a logistic loss and regularization on parameters θ = [W, b]. Best are obtained using a combination of SIFT descriptors, dense SIFT and local binary patterns (LBP), obtaining 90.68% (+/-1.41) accuracy on the LFW dataset. BID21 used an 2 -constraint on the softmax loss for face verification so that the encoded face features lie on the ambit of a hypersphere, showing good improvements in performance. This work too uses an 2 -constraint on capsule encoded face embeddings. FaceNet BID26 too uses a triplet network that combines the Inception network BID28 and a 8-layer convolutional model BID33 which learns to align face patches during training to perform face verification, recognition and clustering. The method trains the network on triplets of increasing difficulty using a negative example mining technique. Similarly, we consider a Siamese Inception Network for the tasks as one of a few comparisons to SCNs. The most relevant and notable use of Siamese Networks for face verification is the DeepFace network BID30. The performance obtained was on par with human level performance on the Faces in the Wild (LFW) dataset and significantly outperformed previous methods. However, it is worth noting this model is trained on a large dataset from Facebook (SFC), therefore the model can be considered to be performing transfer learning before evaluation. The model also carries out some manual steps for detecting, aligning and cropping faces from the images. For detecting and aligning the face a 3D model is used. The images are normalized to avoid any differences in illumination values, before creating a 3D model which is created by first identifying 6 fiducial points in the image using a Support Vector Regressor from a LBP histogram image descriptor. Once the faces are cropped based on these points, a further 67 fiducial point are identified for 3D mesh model, followed by a piecewise affine transformation for each section of the image. The cropped image is then passed to 3 CNN layers with an initial max-pooling layer followed two fully-connected layers. Similar to Capsule Networks, the authors refrain from using max pooling at each layer due to information loss. In contrast to this work, the only preprocessing steps for the proposed SCNs consist of pixel normalization and a reszing of the image. The above work all achieve comparable state of the art for face verification using either a single CNN or a combination of various CNNs, some of which are pretrained on large related datasets. In contrast, this work looks to use a smaller Capsule Network that is more efficient, requires little preprocessing steps (i.e only a resizing of the image and normalization of input features, no aligning, cropping etc.) and can learn from relatively less data. The Capsule Network for face verification is intended to identify enocded part-whole relationships of facial features and their pose that in turn leads to an improved similarity measure by aligning capsule features across paired images. The architecture consists of a 5-hidden layer (includes 2 capsule layers) network with tied weights (since both inputs are from the same domain). The 1 st layer is a convolutional filter with a stride of 3 and 256 channels with kernels κ 1 i ∈ R 9×9 ∀i over the image pairs x 1, x 2 ∈ R 100×100, ing in 20, 992 parameters. The 2 nd layer is the primary capsule layer that takes κ and outputs κ ∈ R 31×31 matrix for 32 capsules, leading to 5.309 × 10 6 parameters (663, 552 weights and 32 biases for each of 8 capsules). The 3 rd layer is the face capsule layer, representing the routing of various properties of facial features, consisting of 5.90 × 10 6 parameters. This layer is then passed to a single fully connected layer by concatenating DISPLAYFORM0 as input, while the sigmoid functions control the dropout rate for each capsule during training. The nonlinear vector normalization shown in Equation 1 is replaced with a tanh function tanh which we found in initial testing to produce better . Euclidean distance, Manhattan distance and cosine similarity are considered as measures between the capsule image encodings. The aforementioned SCN architecture describes the setup for the AT&T dataset. For the LFW dataset, 6 routing iterations are used and 4 for AT&T. To encode paired images x 1, x 2 into vector pairs h 1, h 2 the pose vector of each capsule is vectorized and passed as input to a fully connected layer containing 20 activation units. Hence, for each input there is a lower 20-dimensional representation of 32 capsule pose vectors ing in 512 input features. To ensure all capsules stay active the dropout probability rate is learned for each capsule. The logit function learns the dropout rate of the final capsule layer using Concrete Dropout BID4. This builds on prior work Kingma et al. FORMULA7; BID20 by using a continuous relaxation that is referred to as a concrete distribution, the mask becomes differentiable, unlike the discrete Bernoulli distribution used in Binary dropout. Equation 2 shows the objective function for updating the concrete distribution. For a given capsule probability p c in the last capsule layer, the sigmoid computes the relaxationz on the Bernoulli variable z, where u is drawn uniformly between and log(u c) produces log-uniform noise, and t denotes the temperature values (t = 0.1 in our experiments) which forces probabilities at the extremum when small. The pathwise derivative estimator is used to find a continuous estimation of the dropout mask.z DISPLAYFORM0 Equation FORMULA8 show the dropout loss where the LHS numerator represents the weight regularization term for N weights in the final layer L where γ controls the amount of regularization. The remaining H(p d L) is the dropout entropy term used as dropout regularization. This entropy term H(p d L) is essential to learn the dropout probability for capsule representations, also note when regularization plays a significant role reducing the contrastive loss, d p → 0.5. In our experiments, weight regularization λ ω L = 0.1 is chosen based on a grid search during training for λ θ L = {0.001, 0.01, 0.1, 0.2, 0.5}. This is used as an auxiliary regularization term with the contrastive losses discussed in the next section. DISPLAYFORM1 We also use a straight-through pathwise derivative estimator BID5 to then optimize the dropout probability, following the implementation by BID4. BID7 has also been used to maximize the inter-class distance between the target class and the remaining classes for classifying on the smallNORB dataset. This is given as where the margin m is increased linearly during training to ensure lower-level capsule stay active throughout training. This work instead uses a contrastive margin loss BID2 where the aforementioned capsule encoding similarity function d ω outputs a predicted similarity score. The contrastive loss L c ensures similar vectorized pose encodings are drawn together and dissimilar poses repulse. Equation 4 shows a a pair of images that are passed to the SCN model where DISPLAYFORM0 2 computes the Euclidean distance between encodings and m is the margin. When using Manhattan distance DISPLAYFORM1 A double margin loss that has been used in prior work by BID17 is also considered to affect matching pairs such that to account for positive pairs that can also have high variance in the distance measure. It is worth noting this double margin is similar to the aforementioned margin loss used on class capsules, without the use of λ. Equation 5 shows the double-margin contrastive loss where positive margin m p and negative margin m n are used to find better separation between matching and non-matching pairs. This loss is only used for LFW, given the limited number of instances in AT&T we find the amount of overlap between pairs to be less severe in experimentation. DISPLAYFORM2 The original reconstruction loss DISPLAYFORM3 i ) 2 used as regularization is not used in the pairwise learning setting, instead we rely on the dropout for regularization with exception of the SCN model that uses concrete dropout on the final layer. In the case of using concrete dropout the loss is L conc = L c (ω) + Γ c (p d) and Γ controls the amount of dropout regularization Optimization Convergence can often be relatively slow for face verification tasks, where few informative batch updates (e.g a sample with significantly different pose for a given class) get large updates but soon after the effect is diminished through gradient exponential averaging (originally introduced to prevent α → 0). Motivated by recent findings that improve adaptive learning rates we use AMSGrad BID22. AMSGrad improves over ADAM in some cases by replacing the exponential average of squared gradients with a maximum that mitigates the issue by keeping long-term memory of past gradients. Thus, AMSGrad does not increase or decrease the learning rate based on gradient changes, avoiding divergent or vanishing step sizes over time. Equation 6 presents the update rule, where diagonal of gradient g t is given as DISPLAYFORM4 A. AT&T dataset The AT&T face recognition and verification dataset consists of 40 different subjects with only 10 gray-pixel images per subject in a controlled setting. This smaller dataset allows us to test how SCNs perform with little data. For testing, we hold out 5 subjects so that we are testing on unseen subjects, as opposed to training on a given viewpoint of a subject and testing on another viewpoint of the same subject. Hence, zero-shot pairwise prediction is performed during testing. The LFW consists of 13,000 colored photographed faces from the web. This dataset is significantly more complex not only because there 1680 subjects, with some subjects only consisting of two images, but also because of varied amount of aging, pose, gender, lighting and other such natural characteristics. Each image is 250 × 250, in this work the image is resized to 100×100 and normalized. From the original LFW dataset there has been 2 different versions of the dataset that align the images using funneling BID10 and deep funneling BID11. The latter learns to align the images using Restricted Boltzmann Machines with a group sparsity penalty, showing performance improvements for face verification tasks. The penalty leads to an arrangement of the filters that improved the alignment . This overcomes the problems previous CNNs and models alike had in accounting for pose, orientation and problems Capsule Networks look to address. In contrast, we use the original raw image dataset. Both datasets allow us to test different challenges in face verification. The AT&T dataset provides a useful benchmark for the siamese-based models in the few-shot learning setting as it only provides a limited number of instances and unique subjects, while also being limited to grey-pixel images. In contrast, LFW contains coloured images that are in an unconstrained setting, large in comparison to AT&T and have highly imbalanced classes. Baselines SCNs are compared against well-established architectures for image recognition and verification tasks, namely AlexNet BID15, ResNet-34 and InceptionV3 BID29 with 6 inception layers instead of the original network that uses 8 layers which are used many of the aforementioned papers in Section 3. Table 1 shows best test obtained when using contrastive loss with Euclidean distance between encodings (i.e Mahalanobis distance) for both AT &T and LFW over 100 epochs. Shaded cells correspond to best performing models over test contrastive loss and test accuracy for each respective dataset and loss (e.g LFW+Double Margin (M)). The former uses m = 0.4 and the latter uses m = 0.2, while for the double margin contrastive loss m n = 0.2 matching margin and m p = 0.5 negative matching margin is selected. These settings were chosen during 5-fold cross validation, grid searching over possible margin settings. SCN outperforms baselines on the AT &T dataset after training for 100 epochs. We find that because AT&T contains far fewer instances an adapted dropout rate leads to a slight increase in contrastive loss. Additionally, adding a reconstruction loss with λ r = 1e −4 for both paired images led to a decrease in performance when compared to using dropout with a rate p = 0.2 on all layers except the final layer that encodes the pose vectors. We find for the LFW dataset that the SCN and AlexNet have obtained the best while SCN has 25% less parameters. Additionally, the use of a double margin in better for the LeNet SCN but a slight drop in performance when used with concrete dropout on the final layer (i.e SDropCapNet). Figure 1 illustrates the contrastive loss during training 2 -normalized features for each model tested with various distance measures on AT&T and LFW. We find that SCN yields faster convergence on AT&T, particularly when using Manhattan distance. However for Euclidean distance, we observe a loss variance reduction during training and the best overall performance. Through experiments we find that batch normalized convolutional layers improves performance of the SCN. In batch normalization, (k) and scaled with β (k) so that a (k) = γ (k)x(k) + β (k). This allows the network to learn whether the input range should be more or less diffuse. Batch normalization on the initial convolutional layers reduced variance in loss during training on both the AT &T and LF W datasets. LFW test show that the SCN model takes longer to converge particularly in the early stages of training, in comparison to AlexNet. Figure 2 shows the probability density of the positive pair predictions for each model for all distances between encodings with contrastive loss for the LFW dataset. Accuracy is computed using the aforementioned margins to separate positive and negatives instance pairs, which is set to be constant throughout training since the capsule encoded features are − 2 normalized, similar to the approach taken for FaceNet BID26. We find the variance of predictions is lower in comparison to the remaining models, showing a higher precision in the predictions, particularly for Manhattan distance. Additionally, varying distances for these matching images were close in variance to nonmatching images. This motivated the use of the double margin loss considered for the LFW dataset. DISPLAYFORM0 Finally, the SCN model has between 104-116 % less parameters than Alexnet, 24-27 % Resnet-34 and 127-135% less than the best LeNet baseline for both datasets. However, even considering tied weights between models in the SCN, Capsule Networks are primarily limited in speed even with a reduction in parameters due to the routing iterations that are necessary during training. This paper has introduced the Siamese Capsule Network, a novel architecture that extends Capsule Networks to the pairwise learning setting with a feature 2 -normalized contrastive loss that maximizes inter-class variance and minimizes intra-class variance. The indicate Capsule Networks perform better at learning from only few examples and converge faster when a contrastive loss is used that takes face embeddings in the form of encoded capsule pose vectors. We find Siamese Capsule Networks to perform particularly well on the AT&T dataset in the few-shot learning setting, which is tested on unseen classes (i.e subjects) during testing, while competitive against baselines for the larger Labeled Faces In The Wild dataset.

Represent the TLDR for finding the corresponding scientific article!

A pairwise learned capsule network that performs well on face verification tasks given limited labeled data

Represent this text to find a much much longer article about it with say hundreds-thousands of words!

We generate examples to explain a classifier desicion via interpolations in latent space. The variational auto encoder cost is extended with a functional of the classifier over the generated example path in data space.


Question:
represent the article for finding a single-sentence tldr
Given In this work, we present a novel upper bound of target error to address the problem for unsupervised domain adaptation. Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks. Furthermore, provide an upper bound for target error when transferring the knowledge, which can be summarized as minimizing the source error and distance between marginal distributions simultaneously. However, common methods based on the theory usually ignore the joint error such that samples from different classes might be mixed together when matching marginal distribution. And in such case, no matter how we minimize the marginal discrepancy, the target error is not bounded due to an increasing joint error. To address this problem, we propose a general upper bound taking joint error into account, such that the undesirable case can be properly penalized. In addition, we utilize constrained hypothesis space to further formalize a tighter bound as well as a novel cross margin discrepancy to measure the dissimilarity between hypotheses which alleviates instability during adversarial learning. Extensive empirical evidence shows that our proposal outperforms related approaches in image classification error rates on standard domain adaptation benchmarks. The advent of deep convolutional neural networks brings visual learning into a new era. However, the performance heavily relies on the abundance of data annotated with ground-truth labels. Since traditional machine learning assumes a model is trained and verified in a fixed distribution (single domain), where generalization performance is guaranteed by VC theory (N.), thus it cannot always be applied to real-world problem directly. Take image classification task as an example, a number of factors, such as the change of light, noise, angle in which the image is pictured, and different types of sensors, can lead to a domain-shift thus harm the performance when predicting on test data. Therefore, in many practical cases, we wish that a model trained in one or more source domains is also applicable to another domain. As a solution, domain adaptation (DA) aims to transfer the knowledge learned from a source distribution, which is typically fully labeled into a different (but related) target distribution. This work focus on the most challenging case, i.e, unsupervised domain adaptation (UDA), where no target label is available. suggests that target error can be minimized by bounding the error of a model on the source data, the discrepancy between distributions of the two domains, and a small optimal joint error. Owing to the strong representation power of deep neural nets, many researchers focus on learning domain-invariant features such that the discrepancy of two feature spaces can be minimized. For aligning feature distributions across domains, mainly two strategies have been substantially explored. The first one is bridging the distributions by matching all their statistics . The second strategy is using adversarial learning to build a minimax game between domain discriminator and feature extractor, where a domain discriminator is trained to distinguish the source from the target while the feature extractor is learned to confuse it simultaneously (; ;). In spite of the remarkable empirical accomplished by feature distribution matching schemes, they still suffer from a major limitation: the joint distributions of feature spaces and categories are not well aligned across data domains. As is reported in , such methods fail to generalize in certain closely related source/target pairs, e.g., digit classification adaptation from MNIST to SVHN. One potential reason is when matching marginal distributions of source and target domains, samples from different classes can be mixed together, where the joint error becomes nonnegligible since no hypothesis can classify source and target at the same time. This work aims to address the above problem by incorporating joint error to formalize an optimizable upper bound such that the undesired overlap due to a wrong match can be properly penalized. We evaluate our proposal on several different classification tasks. In some experimental settings, our method outperforms other methods by a large margin. The contributions of this work can be summarized as follows: · We propose a novel upper bound taking joint error into account and theoretically prove that our proposal can reduce to several other methods under certain simplifications. · We construct a constrained hypothesis space such that a much tighter bound can be obtained during optimization. · We adopt a novel measurement, namely cross margin discrepancy, for the dissimilarity of two hypotheses on certain domain to alleviate the instability during adversarial learning and provide reliable performance. The upper bound proposed by invokes numerous approaches focusing on reducing the gap between source and target domains by learning domain-invariant features, which can be achieved through statistical moment matching. Long et al. (2015; use maximum mean discrepancy (MMD) to match the hidden representations of certain layers in a deep neural network. Transfer Component Analysis (TCA) tries to learn a subspace across domains in a Reproducing Kernel Hilbert Space (RKHS) using MMD that dramatically minimize the distance between domain distributions. Adaptive batch normalization (AdaBN) modulates the statistics from source to target on batch normalization layers across the network in a parameterfree way. Another way to learn domain-invariant features is by leveraging generative adversarial network to produce target features that exactly match the source. relax divergence measurement in the upper bound by a worst case which is equivalent to the maximum accuracy that a discriminator can possibly achieve when distinguishing source from target. follow this idea but separate the training procedure into classification stage and adversarial learning stage where an independent feature extractor is used for target. Saito et al. (2017b) explore a tighter bound by explicitly utilizing task-specific classifiers as discriminators such that features nearby the support of source samples will be favored by extractor. introduce margin disparity discrepancy, a novel measurement with rigorous generalization bounds, tailored to the distribution comparison with the asymmetric margin loss to bridge the gap between theory and algorithm. Methods perform distribution alignment on pixel-level in raw input, which is known as image-to-image translation, are also proposed (; ; ; ; ;). Distribution matching may not only bring the source and target domains closer, but also mix samples with different class labels together. Therefore, Saito et al. (2017a);; aim to use pseudo-labels to learn target discriminative representations encouraging a lowdensity separation between classes in the target domain . However, this usually requires auxiliary data-dependent hyper-parameter to set a threshold for a reliable prediction. present conditional adversarial domain adaptation, a principled framework that conditions the adversarial adaptation models on discriminative information conveyed in the classifier predictions, where the back-propagation of training objective is highly dependent on pseudo-labels. We consider the unsupervised domain adaptation as a binary classification task (our proposal holds for multi-class case) where the learning algorithm has access to a set of n labeled points {( . from the source domain S and a set of m unlabeled points {(x i t) ∈ X} m i=1 sampled i.i.d. from the target domain T. Let f S: X → {0, 1} and f T: X → {0, 1} be the optimal labeling functions on the source and target domains, respectively. Let (usually 0-1 loss) denotes a distance metric between two functions over a distribution that satisfies symmetry and triangle inequality. As a commonly used notation, the source risk of hypothesis h: X → {0, 1} is the error w.r.t. the true labeling function f S under domain S, i.e., S (h):= S (h, f S). Similarly, we use T (h) to represent the risk of the target domain. With these notations, the following bound holds: For simplicity, we use The above upper bound is minimized when h = f S, and it is equivalent to T (f S, f T) owing to the triangle inequality: Furthermore, we demonstrate in such case, our proposal is equivalent to an upper bound of optimal joint error λ because: Fig. 1b illustrates a case where common methods fail to penalize the undesirable situation when samples from different classes are mixed together during distribution matching, while our proposal is capable to do so (for simplicity we assume f S takes a specific form, then T (f S, f T) measures the overlapping area 2 and 5, which is equivalent to the optimal joint error λ). Since optimal labeling functions f S, f T are not available during training, we shall further relax the upper bound by taking supreme w.r.t f S, f T within a hypothesis space H: Then minimizing target risk T (h) becomes optimizing a minimax game and since the max-player taking two parameters f 1, f 2 is too strong, we introduce a feature extractor g to make the min-player stronger. Applying g to the source and target distributions, the overall optimization problem can be written as: min However, if we leave H unconstrained, the supreme term can be arbitrary large. In order to obtain a tight bound, we need to restrict the size of hypothesis space as well as maintain the upper bound. For f S ∈ H 1 ≤ H and f T ∈ H 2 ≤ H, the following holds: The constrained subspace for H 1 is trivial as according to its definition, f S must belong to the space consisting of all classifiers for source domain, namely H sc. However, the constrained subspace for H 2 is a little problematic since we have no access to the true labels of the target domain, thus it is hard to locate f T. Therefore, the only thing we can do is to construct a hypothesis space for H 2 that most likely contains f T. As is illustrated in Fig. 1c, when matching distributions of source and target domain, if the ideal case is achieved where the conditional distributions of source and target are perfectly aligned, then it is fare to assume f T ∈ H sc. However, if the worst case is reached where samples from different class are mixed together, then we tend to believe f T / ∈ H sc. Considering this, we present two proposals in the following sections based on different constraints. We assume H 2 is a space where the hypothesis can classify the samples from the source domain with an accuracy of γ ∈, namely H γ sc, such that we can avoid the worst case by choosing a small value for the hyper-parameter γ when a huge domain shift exists. In practice, it is difficult to actually build such a space and sample from it due to a huge computational cost. Instead, we use a weighted source risk to constrain the behavior of f 2 as an approximation to the sample from H γ sc, which leads to the final training objective: Firstly, we build a space consisting of all classifiers for approximate target domain {( based on pseudo labels which can be obtained by the prediction of h during training procedure, namely Ht c . Here, we assume H 2 is an intersection between two hypothesis spaces, i.e. Given enough reliable pseudo labels, we can be confident about f T ∈ H 2 . Analogously, the training objective is given by: The reason we make such an assumption for H 2 can be intuitively explained by Fig. 2 . If H 2 = H sc, then f 2 must perfectly classify the source samples, and it is possible that f 2 does not pass through some target samples (shadow are in 2a), especially when two domains differ a lot. In such case, the feature extractor can move those samples into either side of the decision boundary to reduce the training objective (shadow area) which is not a desired behavior. With an appropriate constraint (2b), as for the extractor, the only way to reduce the objective (shadow area) is to move those samples (orange) inside of f 2. Following the above notations, we consider a score function s(x, y) for multi-class classification where the output indicates the confidence of the prediction on class y. Thus an induced labeling function named l s from X → Y is given by: As a well-established theory, the margin between data points and the classification surface plays a significant role in achieving strong generalization performance. In order to quantify into differentiable measurement as a surrogate of 0-1 loss, we introduce the margin theory developed by , where a typical form of margin loss can be interpreted as: We aim to utilize this concept to further improve the reliability of our proposed method by leveraging this margin loss to define a novel measurement of the discrepancy between two hypotheses f 1, f 2 (e.g. softmax) over a distribution D, namely cross margin discrepancy: Before further discussion, we firstly construct two distributions D f1, D f2 induced by f 1, f 2 respectively, where Then we consider the case where two hypotheses f 1 and f 2 disagree, i.e. y 1 = l f1 (x) = l f2 (x) = y 2, and the primitive loss is defined as: Then the cross margin discrepancy can be viewed as: which is a sum of the margin loss for f 1 on D f2 and the margin loss for f 2 on D f1, if we use the logarithm of softmax as the score function. Thanks to the trick introduced by to mitigate the burden of exploding or vanishing gradients when performing adversarial learning, we further define a dual form as: This dual loss resembles the objective of the generative adversarial network, where two hypotheses try to increase the probability of their own prediction and simultaneously decrease the probability of their opponents; whereas the feature extractor is trained to increase the probability of their opponents, such that the discrepancy can be minimized without unnecessary oscillation. However, a big difference here is when training extractor, GANs usually maximize an alternative term log f 1 (x, y 2) + log f 2 (x, y 1) instead of directly minimizing log(1 − f 1 (x, y 2)) + log(1 − f 2 (x, y 1)) since the original term is close to zero if the discriminator achieves optimum. In our case, the hypothesis can hardly beat the extractor thus the original form can be more smoothly optimized. During the training procedure, the two hypotheses will eventually agree on some points (l f1 (x) = l f2 (x) = y) such that we need to define a new form of discrepancy measurement. Analogously, the primitive loss and its dual form are given by: Another reason why we propose such a discrepancy measurement is that it helps alleviate instability for adversarial learning. As is illustrated in Fig. 3b, during optimization of a minimax game, when two hypotheses try to maximize the discrepancy (shadow area), if one moves too fast around the decision boundary such that the discrepancy is actually maximized w.r.t some samples, then these samples can be aligned on either side to decrease the discrepancy by tuning the feature extractor, which is not a desired behavior. From Fig. 3a, we can see that our proposed cross margin discrepancy is flat for the points around original, i.e. the gradient w.r.t those points nearby the decision boundary will be relatively small, which helps to prevent such failure. propose a novel margin-aware generalization bound based on scoring functions and a new divergence MDD. The training objective used in MDD can be alternatively interpreted as (here (h, f) denotes the margin disparity): min Recall Eq.7, if we set f 1 = f 2 = f and free the constraint of f to any f ∈ H, our proposal degrades exactly to MDD. As is discussed above, when matching distribution, if and only if the ideal case is achieved, where the conditional distributions of induced feature spaces for source and target perfectly match (which is not always possible), can we assume two optimal labeling functions f S, f T to be identical. Besides, an unconstrained hypothesis space for f is definitely not helpful to construct a tight bound. Saito et al. (2017b) propose two task-specific classifiers f 1, f 2 that are used to separate the decision boundary on source domain, such that the extractor is encouraged to produce features nearby the support of the source samples. The objective used in MCD can be alternatively interpreted as (here (f 1, f 2) is quantified by L 1 ): Again, recall Eq.7, if we set γ = 1 and h = f 1, MCD is equivalent to our proposal. As is proved in section 3.1, the upper bound is optimized when h = f S. However, it no longer holds since the upper bound is relaxed by taking supreme to form an optimizabel objective, i.e. setting h = f 1 does not necessarily minimize the objective. Besides, as we discuss above, a fixed γ = 1, i.e H 2 = H sc lacks generality since we have no idea about where f T might be, such that it is not likely to be applicable to those cases where a huge domain shift exists. In this experiment, our proposal is assessed in four types of adaptation scenarios by adopting commonly used digits datasets (Fig. 6 in Appendix),i.e. MNIST , Street View House Numbers (SVHN) , and USPS such that the could be easily compared with other popular methods. All experiments are performed in an unsupervised fashion without any kinds of data augmentation. Details are omitted due to the limit of space (see A.1). We report the accuracy of different methods in Tab. 1. Our proposal outperforms the competitors in almost all settings except a single compared with GPDA . However, their solution requires sampling that increases data size and is equivalent to adding Gaussian noise to the last layer of a classifier, which is considered as a type of augmentation. Our success partially owes to combining the upper bound with the joint error, especially when optimal label functions differ from each other (e.g. MNIST →SVHN). Moreover, as most scenarios are relatively easy for adaptation thus we can be more confident about the hypothesis space constraint owing to reliable pseudo-labels, which leads to a tighter bond during optimization. The demonstrate our proposal can improve generalization performance by adopting both of these advantages. Fig. 4a shows that our original proposal is quite sensitive to the hyper-parameter γ. In short, setting γ = 1 here yields the best performance in most situations, since f S, f T can be quite close after aligning distributions, especially in these easily adapted scenarios. However, in MNIST → SVHN, setting γ = 0.1 gives the optimum which means that f S, f T are so far away due to a huge domain shift that no extractor is capable of introducing an identical conditional distribution in feature space. The improvement is not that much, but at least we outperform the directly comparable MCD and show the importance of hypothesis space constraint. Furthermore, Fig. 4d empirically proves simply minimizing the discrepancy between the marginal distribution does not necessarily lead to a reliable adaptation, which demonstrates the importance of joint error. In addition, Fig. 4b,Fig. 4c show the superiority of the cross margin discrepancy which accelerates the convergence and provides a slightly better . We further evaluate our method on object classification. The VisDA dataset is used here, which is designed for 12-class adaptation task from synthetic object to real object images. Source domain contains 152,397 synthetic images (Fig. 7a in Appendix), which are generated by rendering 3D CAD models. Data of the target domain is collected from MSCOCO consisting of 55,388 real images (Fig. 7b in Appendix). Since the 3D models are generated without the and color diversity, the synthetic domain is quite different from the real domain, which makes it a much more difficult problem than digits adaptation. Again, this experiment is performed in unsupervised fashion and no data augmentation technique excluding horizontal flipping is allowed. Details are omitted due to the limit of space (see A.2). We report the accuracy of different methods in Tab. 2, and find that our proposal outperforms the competitors in all settings. The image structure of this dataset is more complex than that of digits, yet our method provides reliable performance even under such a challenging condition. Another key observation is that some competing methods (e.g., DANN, MCD), which can be categorized as distribution matching based on adversarial learning, perform worse than MDD which simply matches statistics, in classes such as plane and horse, while our methods perform better across all classes, which clearly demonstrates the importance of taking the joint error into account. As for the original proposal (Fig. 5c), performance drops when relaxing the constraint which actually confuses us. Because we expect an improvement here since it is unbelievable that f S, f T eventually lie in a similar space judging from the relatively low prediction accuracy. As for the alternative proposal (Fig. 5d), we test the adaptation performance for different η and the prediction accuracy drastically drops when η goes beyond 0.2. One possible cause is that f 2 and h might almost agree on target domain, such that the prediction of h could not provide more accurate information for the target domain without introducing noisy pseudo labels. Fig. 5a, Fig. 5b again demonstrate the superiority of cross margin discrepancy and the importance of joint error. In this work, we propose a general upper bound that takes the joint error into account. Then we further pursuit a tighter bound with reasonable constraint on the hypothesis space. Additionally, we adopt a novel cross domain discrepancy for dissimilarity measurement which alleviates the instability during adversarial learning. Extensive empirical evidence shows that learning an invariant representation is not enough to guarantee a good generalization in the target domain, as the joint error matters especially when the domain shift is huge. We believe our take an important step towards understanding unsupervised domain adaptation, and also stimulate future work on the design of stronger adaptation algorithms that manage to align conditional distributions without using pseudo-labels from the target domain. layer and a 0.5 rate of dropout is conducted. Nesterov accelerated gradient is used for optimization with a mini-batch size of 32 and an initial learning rate of 10 −3 which decays exponentially. As for the hyper-parameter, we test for γ = {0.1, 0.5, 0.9, 1} and η = {0, 0.5, 0.8, 0.9}. For a direct comparison, we report the accuracy after 10 epochs. Office-Home is a complex dataset (Fig. 8) containing 15,500 images from four significantly different domains: Art (paintings, sketches and/or artistic depictions), Clipart (clip art images), Product (images without ), and Real-world (regular images captured with a camera). In this experiment, following the protocol from , we evaluate our method by fine-tuning a ResNet-50 model pretrained on ImageNet . The model except the last layer combined with a single-layer bottleneck is used as feature extractor and a randomly initialized 2-layer fully-connected network with width 1024 is used as a classifier, where batch normalization is applied to each layer and a 0.5 rate of dropout is conducted. For optimization, we use the SGD with the Nesterov momentum term fixed to 0.9, where the batch size is 32 and learning rate is adjusted according to. From Tab. 3, we can see the adaptation accuracy of the source-only method is rather low, which means a huge domain shift is quite likely to exist. In such case, simply minimizing the discrepancy between source and target might not work as the joint error can be increased when aligning distributions, thus the assumption of the basic theory does not hold anymore. On the other hand, our proposal incorporates the joint error into the target error upper bound which can boost the performance especially when there is a large domain shift. Figure 8: Sample images from the Office-Home dataset . Office-31 (Fig. 9) is a popular dataset to verify the effectiveness of a domain adaptation algorithm, which contains three diverse domains, Amazon from Amazon website, Webcam by web camera and DSLR by digital SLR camera with 4,652 images in 31 unbalanced classes. In this experiment, following the protocol from , we evaluate our method by fine-tuning a ResNet-50 model pretrained on ImageNet . The model used here is almost identical to the one in Office-Home experiment except a different width 2048 for classifiers. For optimization, we use the SGD with the Nesterov momentum term fixed to 0.9, where the batch size is 32 and learning rate is adjusted according to. The on Office-31 are reported in Tab. 4. As for the tasks D→A and W→A, judging from the adaptation accuracy of those previous methods that do not consider the joint error, it is quite likely that samples from different classes are mixed together when matching distributions. Our method shows an advantage in such case which demonstrates that the proposal manage to penalize the undesired matching between source and target. As for the tasks A→W and A→D, our proposal shows relatively high variance and poor performance especially in A→W. One possible reason is that our method depends on building reliable classifiers for the source domain to satisfy the constraint. However, the Amazon dataset contains a lot of noise (Fig. 10) such that the decision boundary of the source classifiers varies drastically in each iteration during training procedure, which can definitely harm the convergence., a positive would be joint error matters for unsupervised domain adaptation especially when the domain shift is huge & a negative would be We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.

Question: In this work we present a new agent architecture, called Reactor, which combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN and Categorical DQN , while giving better run-time performance than A3C . Our first contribution is a new policy evaluation algorithm called Distributional Retrace, which brings multi-step off-policy updates to the distributional reinforcement learning setting. The same approach can be used to convert several classes of multi-step policy evaluation algorithms designed for expected value evaluation into distributional ones. Next, we introduce the β-leaveone-out policy gradient algorithm which improves the trade-off between variance and bias by using action values as a baseline. Our final algorithmic contribution is a new prioritized replay algorithm for sequences, which exploits the temporal locality of neighboring observations for more efficient replay prioritization. Using the Atari 2600 benchmarks, we show that each of these innovations contribute to both the sample efficiency and final agent performance. Finally, we demonstrate that Reactor reaches state-of-the-art performance after 200 million frames and less than a day of training. Model-free deep reinforcement learning has achieved several remarkable successes in domains ranging from super-human-level control in video games and the game of Go BID10, to continuous motor control tasks .Much of the recent work can be divided into two categories. First, those of which that, often building on the DQN framework, act -greedily according to an action-value function and train using minibatches of transitions sampled from an experience replay buffer BID10 BID13 BID5 BID0. These value-function agents benefit from improved sample complexity, but tend to suffer from long runtimes (e.g. DQN requires approximately a week to train on Atari). The second category are the actor-critic agents, which includes the asynchronous advantage actor-critic (A3C) algorithm, introduced by. These agents train on transitions collected by multiple actors running, and often training, in parallel (; BID12 . The deep actor-critic agents train on each trajectory only once, and thus tend to have worse sample complexity. However, their distributed nature allows significantly faster training in terms of wall-clock time. Still, not all existing algorithms can be put in the above two categories and various hybrid approaches do exist BID17 ; BID4 BID14 . We consider a Markov decision process (MDP) with state space X and finite action space A. A (stochastic) policy π(·|x) is a mapping from states x ∈ X to a probability distribution over actions. We consider a γ-discounted infinite-horizon criterion, with γ ∈ the discount factor, and define for policy π the action-value of a state-action pair (x, a) as DISPLAYFORM0 where ({x t} t≥0 ) is a trajectory generated by choosing a in x and following π thereafter, i.e., a t ∼ π(·|x t) (for t ≥ 1), and r t is the reward signal. The objective in reinforcement learning is to find an optimal policy π *, which maximises Q π (x, a). The optimal action-values are given by Q * (x, a) = max π Q π (x, a). The Deep Q-Network (DQN) framework, introduced by , popularised the current line of research into deep reinforcement learning by reaching human-level, and beyond, performance across 57 Atari 2600 games in the ALE. While DQN includes many specific components, the essence of the framework, much of which is shared by Neural Fitted Q-Learning , is to use of a deep convolutional neural network to approximate an action-value function, training this approximate action-value function using the Q-Learning algorithm BID15 and mini-batches of one-step transitions (x t, a t, r t, x t+1, γ t) drawn randomly from an experience replay buffer . Additionally, the next-state action-values are taken from a target network, which is updated to match the current network periodically. Thus, the temporal difference (TD) error for transition t used by these algorithms is given by δ t = r t + γ t max a ∈A Q(x t+1, a ;θ) − Q(x t, a t ; θ),where θ denotes the parameters of the network andθ are the parameters of the target network. Since this seminal work, we have seen numerous extensions and improvements that all share the same underlying framework. Double DQN BID10, attempts to correct for the over-estimation bias inherent in Q-Learning by changing the second term of to Q(x t+1, arg max a ∈A Q(x t+1, a ; θ);θ). The dueling architecture BID13, changes the network to estimate action-values using separate network heads V (x; θ) and A(x, a; θ) with Q(x, a; θ) = V (x; θ) + A(x, a; θ) − 1 |A| a A(x, a ; θ).Recently, BID6 introduced Rainbow, a value-based reinforcement learning agent combining many of these improvements into a single agent and demonstrating that they are largely complementary. Rainbow significantly out performs previous methods, but also inherits the poorer time-efficiency of the DQN framework. We include a detailed comparison between Reactor and Rainbow in the Appendix. In the remainder of the section we will describe in more depth other recent improvements to DQN. The experience replay buffer was first introduced by and later used in DQN . Typically, the replay buffer is essentially a first-in-first-out queue with new transitions gradually replacing older transitions. The agent would then sample a mini-batch uniformly at random from the replay buffer. Drawing inspiration from prioritized sweeping , prioritized experience replay replaces the uniform sampling with prioritized sampling proportional to the absolute TD error .Specifically, for a replay buffer of size N, prioritized experience replay samples transition t with probability P (t), and applies weighted importance-sampling with w t to correct for the prioritization bias, where DISPLAYFORM0 Prioritized DQN significantly increases both the sample-efficiency and final performance over DQN on the Atari 2600 benchmarks BID13. Retrace(λ) is a convergent off-policy multi-step algorithm extending the DQN agent . Assume that some trajectory {x 0, a 0, r 0, x 1, a 1, r 1, . . ., x t, a t, r t, . . .,} has been generated according to behaviour policy µ, i.e., a t ∼ µ(·|x t). Now, we aim to evaluate the value of a different target policy π, i.e. we want to estimate Q π. The Retrace algorithm will update our current estimate Q of Q π in the direction of DISPLAYFORM0 DISPLAYFORM1 is the temporal difference at time s under π, and DISPLAYFORM2 The Retrace algorithm comes with the theoretical guarantee that in finite state and action spaces, repeatedly updating our current estimate Q according to produces a sequence of Q functions which converges to Q π for a fixed π or to Q * if we consider a sequence of policies π which become increasingly greedy w.r.t. the Q estimates . Distributional reinforcement learning refers to a class of algorithms that directly estimate the distribution over returns, whose expectation gives the traditional value function BID2. Such approaches can be made tractable with a distributional Bellman equation, and the recently proposed algorithm C51 showed state-of-the-art performance in the Atari 2600 benchmarks. C51 parameterizes the distribution over returns with a mixture over Diracs centered on a uniform grid, DISPLAYFORM0 with hyperparameters v min, v max that bound the distribution support of size N. In this section we review the actor-critic framework for reinforcement learning algorithms and then discuss recent advances in actor-critic algorithms along with their various trade-offs. The asynchronous advantage actor-critic (A3C) algorithm , maintains a parameterized policy π(a|x; θ) and value function V (x; θ v), which are updated with DISPLAYFORM0 where, A(x t, a t ; DISPLAYFORM1 A3C uses M = 16 parallel CPU workers, each acting independently in the environment and applying the above updates asynchronously to a shared set of parameters. In contrast to the previously discussed value-based methods, A3C is an on-policy algorithm, and does not use a GPU nor a replay buffer. Proximal Policy Optimization (PPO) is a closely related actor-critic algorithm , which replaces the advantage with, DISPLAYFORM2 where ρ t is as defined in Section 2.1.2. Although both PPO and A3C run M parallel workers collecting trajectories independently in the environment, PPO collects these experiences to perform a single, synchronous, update in contrast with the asynchronous updates of A3C.Actor-Critic Experience Replay (ACER) extends the A3C framework with an experience replay buffer, Retrace algorithm for off-policy corrections, and the Truncated Importance Sampling Likelihood Ratio (TISLR) algorithm used for off-policy policy optimization BID14. The Reactor is a combination of four novel contributions on top of recent improvements to both deep value-based RL and policy-gradient algorithms. Each contribution moves Reactor towards our goal of achieving both sample and time efficiency. The Reactor architecture represents both a policy π(a|x) and action-value function Q(x, a). We use a policy gradient algorithm to train the actor π which makes use of our current estimate Q(x, a) of Q π (x, a). Let V π (x 0) be the value function at some initial state x 0, the policy gradient theorem says that DISPLAYFORM0, where ∇ refers to the gradient w.r.t. policy parameters BID9. We now consider several possible ways to estimate this gradient. To simplify notation, we drop the dependence on the state x for now and consider the problem of estimating the quantity DISPLAYFORM1 In the off-policy case, we consider estimating G using a single actionâ drawn from a (possibly different from π) behaviour distributionâ ∼ µ. Let us assume that for the chosen actionâ we have access to an unbiased estimate R(â) of Q π (â). Then, we can use likelihood ratio (LR) method combined with an importance sampling (IS) ratio (which we call ISLR) to build an unbiased estimate of G: DISPLAYFORM2 where V is a baseline that depends on the state but not on the chosen action. However this estimate suffers from high variance. A possible way for reducing variance is to estimate G directly from by using the return R(â) for the chosen actionâ and our current estimate Q of Q π for the other actions, which leads to the so-called leave-one-out (LOO) policy-gradient estimate: This estimate has low variance but may be biased if the estimated Q values differ from Q π. A better bias-variance tradeoff may be obtained by the more general β-LOO policy-gradient estimate: DISPLAYFORM3 DISPLAYFORM4 where β = β(µ, π,â) can be a function of both policies, π and µ, and the selected actionâ. Notice that when β = 1, reduces to, and when β = 1/µ(â), then iŝ DISPLAYFORM5 This estimate is unbiased and can be seen as a generalization ofĜ ISLR where instead of using a state-only dependent baseline, we use a state-and-action-dependent baseline (our current estimate Q) and add the correction term a ∇π(a)Q(a) to cancel the bias. Proposition 1 gives our analysis of the bias of G β-LOO, with a proof left to the Appendix. DISPLAYFORM6 Thus the bias is small when β(a) is close to 1/µ(a), or when the Q-estimates are close to the true Q π values, and unbiased regardless of the estimates if β(a) = 1/µ(a). The variance is low when β is small, therefore, in order to improve the bias-variance tradeoff we recommend using the β-LOO estimate with β defined as: β(â) = min c, 1 µ(â), for some constant c ≥ 1. This truncated 1/µ coefficient shares similarities with the truncated IS gradient estimate introduced in BID14 ) (which we call TISLR for truncated-ISLR): DISPLAYFORM7 The differences are: (i) we truncate 1/µ(â) = π(â)/µ(â) × 1/π(â) instead of truncating π(â)/µ(â), which provides an additional variance reduction due to the variance of the LR ∇ log π(â) = DISPLAYFORM8 (since this LR may be large when a low probability action is chosen), and (ii) we use our Q-baseline instead of a V baseline, reducing further the variance of the LR estimate. In off-policy learning it is very difficult to produce an unbiased sample R(â) of Q π (â) when following another policy µ. This would require using full importance sampling correction along the trajectory. Instead, we use the off-policy corrected return computed by the Retrace algorithm, which produces a (biased) estimate of Q π (â) but whose bias vanishes asymptotically .In Reactor, we consider predicting an approximation of the return distribution function from any state-action pair (x, a) in a similar way as in BID2. The original algorithm C51 described in that paper considered single-step Bellman updates only. Here we need to extend this idea to multi-step updates and handle the off-policy correction performed by the Retrace algorithm, as defined in. Next, we describe these two extensions. Multi-step distributional Bellman operator: First, we extend C51 to multi-step Bellman backups. We consider return-distributions from (x, a) of the form i q i (x, a)δ zi (where δ z denotes a Dirac in z)which are supported on a finite uniform grid DISPLAYFORM0 The coefficients q i (x, a) (discrete distribution) corresponds to the probabilities assigned to each atom z i of the grid. From an observed n-step sequence {x t, a t, r t, x t+1, . . ., x t+n}, generated by behavior policy µ (i.e, a s ∼ µ(·|x s) for t ≤ s < t + n), we build the n-step backed-up return-distribution from (x t, a t). The n-step distributional Bellman target, whose expectation is t+n−1 s=t γ s−t r s + γ n Q(x t+n, a), is given by: DISPLAYFORM1 Since this distribution is supported on the set of atoms {z n i}, which is not necessarily aligned with the grid {z i}, we do a projection step and minimize the KL-loss between the projected target and the current estimate, just as with C51 except with a different target distribution BID2.Distributional Retrace: Now, the Retrace algorithm defined in involves an off-policy correction which is not handled by the previous n-step distributional Bellman backup. The key to extending this distributional back-up to off-policy learning is to rewrite the Retrace algorithm as a linear combination of n-step Bellman backups, weighted by some coefficients α n,a. Indeed, notice that rewrites as DISPLAYFORM2 where α n,a = c t+1... c t+n−1 π(a|x t+n) − I{a = a t+n}c t+n. These coefficients depend on the degree of off-policy-ness (between µ and π) along the trajectory. We have that n≥1 a α n,a = n≥1 c t+1... c t+n−1 (1 − c t+n) = 1, but notice some coefficients may be negative. However, in expectation (over the behavior policy) they are non-negative. Indeed, DISPLAYFORM3 by definition of the c s coefficients. Thus in expectation (over the behavior policy), the Retrace update can be seen as a convex combination of n-step Bellman updates. Then, the distributional Retrace algorithm can be defined as backing up a mixture of n-step distributions. More precisely, we define the Retrace target distribution as: DISPLAYFORM4 where h zi (x) is a linear interpolation kernel, projecting onto the support {z i}: DISPLAYFORM5 We update the current probabilities q(x t, a t) by performing a gradient step on the KL-loss DISPLAYFORM6 Again, notice that some target "probabilities" q * i (x t, a t) may be negative for some sample trajectory, but in expectation they will be non-negative. Since the gradient of a KL-loss is linear w.r.t. its first argument, our update rule provides an unbiased estimate of the gradient of the KL between the expected (over the behavior policy) Retrace target distribution and the current predicted distribution. Remark: The same method can be applied to other algorithms (such as TB(λ) and importance sampling ) in order to derive distributional versions of other off-policy multi-step RL algorithms. Prioritized experience replay has been shown to boost both statistical efficiency and final performance of deep RL agents . However, as originally defined prioritized replay does not handle sequences of transitions and weights all unsampled transitions identically. In this section we present an alternative initialization strategy, called lazy initialization, and argue that it better encodes prior information about temporal difference errors. We then briefly describe our computationally efficient prioritized sequence sampling algorithm, with full details left to the appendix. It is widely recognized that TD errors tend to be temporally correlated, indeed the need to break this temporal correlation has been one of the primary justifications for the use of experience replay . Our proposed algorithm begins with this fundamental assumption. Assumption 1. Temporal differences are temporally correlated, with correlation decaying on average with the time-difference between two transitions. Prioritized experience replay adds new transitions to the replay buffer with a constant priority, but given the above assumption we can devise a better method. Specifically, we propose to add experience to the buffer with no priority, inserting a priority only after the transition has been sampled and used for training. Also, instead of sampling transitions, we assign priorities to all (overlapping) sequences of length n. When sampling, sequences with an assigned priority are sampled proportionally to that priority. Sequences with no assigned priority are sampled proportionally to the average priority of assigned priority sequences within some local neighbourhood. Averages are weighted to compensate for sampling biases (i.e. more samples are made in areas of high estimated priorities, and in the absence of weighting this would lead to overestimation of unassigned priorities).The lazy initialization scheme starts with priorities p t corresponding to the sequences {x t, a t, . . ., x t+n} for which a priority was already assigned. Then it extrapolates a priority of all other sequences in the following way. Let us define a partition (I i) i of the states ordered by increasing time such that each cell I i contains exactly one state s i with already assigned priority. We define the estimated priorityp t to all other sequences asp t = si∈J(t) DISPLAYFORM0 is a collection of contiguous cells (I i) containing time t, and w i = |I i | is the length of the cell I i containing s i. For already defined priorities denotep t = p t. Cell sizes work as estimates of inverse local density and are used as importance weights for priority estimation.2 For the algorithm to be unbiased, partition (I i) i must not be a function of the assigned priorities. So far we have defined a class of algorithms all free to choose the partition (I i) and the collection of cells I(t), as long that they satisfy the above constraints. FIG6 in the Appendix illustrates the above description. Now, with probability we sample uniformly at random, and with probability 1 − we sample proportionally top t. We implemented an algorithm satisfying the above constraints and called it Contextual Priority Tree (CPT). It is based on AVL trees BID11 and can execute sampling, insertion, deletion and density evaluation in O(ln(n)) time. We describe CPT in detail in the Appendix in Section 6.3.We treated prioritization as purely a variance reduction technique. Importance-sampling weights were evaluated as in prioritized experience replay, with fixed β = 1 in. We used simple gradient magnitude estimates as priorities, corresponding to a mean absolute TD error along a sequence for Retrace, as defined in for the classical RL case, and total variation in the distributional Retrace case. In order to improve CPU utilization we decoupled acting from learning. This is an important aspect of our architecture: an acting thread receives observations, submits actions to the environment, and stores transitions in memory, while a learning thread re-samples sequences of experiences from memory and trains on them (Figure 2, left). We typically execute 4-6 acting steps per each learning step. We sample sequences of length n = 33 in batches of 4. A moving network is unrolled over frames 1-32 while the target network is unrolled over frames 2-33.We allow the agent to be distributed over multiple machines each containing action-learner pairs. Each worker downloads the newest network parameters before each learning step and sends delta-updates at the end of it. Both the network and target network are stored on a shared parameter server while each machine contains its own local replay memory. Training is done by downloading a shared network, evaluating local gradients and sending them to be applied on the shared network. While the agent can also be trained on a single machine, in this work we present of training obtained with either 10 or 20 actor-learner workers and one parameter server. In Figure 2 (right) we compare resources and runtimes of Reactor with related algorithms. In some domains, such as Atari, it is useful to base decisions on a short history of past observations. The two techniques generally used to achieve this are frame stacking and recurrent network architectures. We chose the latter over the former for reasons of implementation simplicity and computational efficiency. As the Retrace algorithm requires evaluating action-values over contiguous sequences of trajectories, using a recurrent architecture allowed each frame to be processed by the convolutional network only once, as opposed to n times times if n frame concatenations were used. The Reactor architecture uses a recurrent neural network which takes an observation x t as input and produces two outputs: categorical action-value distributions q i (x t, a) (i here is a bin identifier), and policy probabilities π(a|x t). We use an architecture inspired by the duelling network architecture BID13. We split action-value -distribution logits into state-value logits and advantage logits, which in turn are connected to the same LSTM network BID7.Final action-value logits are produced by summing state-and action-specific logits, as in BID13. Finally, a softmax layer on top for each action produces the distributions over discounted future returns. The policy head uses a softmax layer mixed with a fixed uniform distribution over actions, where this mixing ratio is a hyperparameter (, Section 5.1.3). Policy and Q-networks have separate LSTMs. Both LSTMs are connected to a shared linear layer which is connected to a shared convolutional neural network . The precise network specification is given in TAB6 in the Appendix. Gradients coming from the policy LSTM are blocked and only gradients originating from the Qnetwork LSTM are allowed to back-propagate into the convolutional neural network. We block gradients from the policy head for increased stability, as this avoids positive feedback loops between π and q i caused by shared representations. We used the Adam optimiser , with a learning rate of 5 × 10 −5 and zero momentum because asynchronous updates induce implicit momentum . Further discussion of hyperparameters and their optimization can be found in Appendix 6.1. We trained and evaluated Reactor on 57 Atari games BID1. FIG4 compares the performance of Reactor with different versions of Reactor each time leaving one of the algorithmic improvements out. We can see that each of the algorithmic improvements (Distributional retrace, beta-LOO and prioritized replay) contributed to the final . While prioritization was arguably the most important component, Beta-LOO clearly outperformed TISLR algorithm. Although distributional and non-distributional versions performed similarly in terms of median human normalized scores, distributional version of the algorithm generalized better when tested with random human starts (We evaluated Reactor with target update frequency T update = 1000, λ = 1.0 and β-LOO with β = 1 on 57 Atari games trained on 10 machines in parallel. We averaged scores over 200 episodes using 30 random human starts and noop starts TAB8 in the Appendix). We calculated mean and median human normalised scores across all games. We also ranked all algorithms (including random and human scores) for each game and evaluated mean rank of each algorithm across all 57 Atari games. We also evaluated mean Rank and Elo scores for each algorithm for both human and noop start settings. Please refer to Section 6.2 in the Appendix for more details. TAB2 with several other state-of-art algorithms across 57 Atari games for a fixed random seed across all games BID1. We compare Reactor against are: DQN , Double DQN BID10, DQN with prioritised experience replay BID13, dueling architecture and prioritised dueling BID13, ACER BID14, A3C , and Rainbow BID6. Each algorithm was exposed to 200 million frames of experience, or 500 million frames when followed by 500M, and the same pre-processing pipeline including 4 action repeats was used as in the original DQN paper .In TAB2, we see that Reactor exceeds the performance of all algorithms across all metrics, despite requiring under two days of training. With 500 million frames and four days training we see Reactor's performance continue to improve significantly. The difference in time-efficiency is especially apparent when comparing Reactor and Rainbow (see FIG4 . Additionally, unlike Rainbow, Reactor does not use Noisy Networks BID3, which was reported to have contributed to the performance gains. When evaluating under the no-op starts regime TAB4, Reactor out performs all methods except for Rainbow. This suggests that Rainbow is more sample-efficient when training and evaluation regimes match exactly, but may be overfitting to particular trajectories due to the significant drop in performance when evaluated on the random human starts. Regarding ACER, another Retrace-based actor-critic architecture, both classical and distributional versions of Reactor FIG4) exceeded the best reported median human normalized score of 1.9 with noop starts achieved in 500 million steps. In this work we presented a new off-policy agent based on Retrace actor-critic architecture and show that it achieves similar performance as the current state-of-the-art while giving significant real-time performance gains. We demonstrate the benefits of each of the suggested algorithmic improvements, including Distributional Retrace, beta-LOO policy gradient and contextual priority tree. DISPLAYFORM0 Proof. The bias ofĜ β-LOO is DISPLAYFORM1 As we believe that algorithms should be robust with respect to the choice of hyperparameters, we spent little effort on parameter optimization. In total, we explored three distinct values of learning rates and two values of ADAM momentum (the default and zero) and two values of T update on a subset of 7 Atari games without prioritization using non-distributional version of Reactor. We later used those values for all experiments. We did not optimize for batch sizes and sequence length or any prioritization hyperparamters. Commonly used mean and median human normalized scores have several disadvantages. A mean human normalized score implicitly puts more weight on games that computers are good and humans are bad at. Comparing algorithm by a mean human normalized score across 57 Atari games is almost equivalent to comparing algorithms on a small subset of games close to the median and thus dominating the signal. Typically a set of ten most score-generous games, namely Assault, Asterix, Breakout, Demon Attack, Double Dunk, Gopher, Pheonix, Stargunner, Up'n Down and Video Pinball can explain more than half of inter-algorithm variance. A median human normalized score has the opposite disadvantage by effectively discarding very easy and very hard games from the comparison. As typical median human normalized scores are within the range of 1-2.5, an algorithm which scores zero points on Montezuma's Revenge is evaluated equal to the one which scores 2500 points, as both performance levels are still below human performance making incremental improvements on hard games not being reflected in the overall evaluation. In order to address both problem, we also evaluated mean rank and Elo metrics for inter-algorithm comparison. Those metrics implicitly assign the same weight to each game, and as a is more sensitive of relative performance on very hard and easy games: swapping scores of two algorithms on any game would in the change of both mean rank and Elo metrics. We calculated separate mean rank and Elo scores for each algorithm using of test evaluations with 30 random noop-starts and 30 random human starts TAB8 ). All algorithms were ranked across each game separately, and a mean rank was evaluated across 57 Atari games. For Elo score evaluation algorithm, A was considered to win over algorithm B if it obtained more scores on a given Atari. We produced an empirical win-probability matrix by summing wins across all games and used this matrix to evaluate Elo scores. A ranking difference of 400 corresponds to the odds of winning of 10:1 under the Gaussian assumption. Contextual priority tree is one possible implementation of lazy prioritization FIG6 ). All sequence keys are put into a balanced binary search tree which maintains a temporal order. An AVL tree BID11 ) was chosen due to the ease of implementation and because it is on average more evenly balanced than a Red-Black Tree. Each tree node has up to two children (left and right) and contains currently stored key and a priority of the key which is either set or is unknown. Some trees may only have a single child subtree while Sampling is done by going from the root node up the tree by selecting one of the children (or the current key) stochastically proportional to orange proportions. Sampling terminates once the current (square) key is chosen. Figure 6: Example of a balanced priority tree. Dark blue nodes contain keys with known priorities, light blue nodes have at least one child with at least a single known priority, while ping nodes do not have any priority estimates. Nodes 1, 2 and 3 will obtain priority estimates equal to 2/3 of the priority of key 5 and 1/3 of the priority of node 4. This implies that estimated priorities of keys 1, 2 and 3 are implicitly defined by keys 4 and 6. Nodes 8, 9 and 11 are estimated to have the same priority as node 10.some may have none. In addition to this information, we were tracking other summary statistics at each node which was re-evaluated after each tree rotation. The summary statistics was evaluated by consuming previously evaluated summary statistics of both children and a priority of the key stored within the current node. In particular, we were tracking a total number of nodes within each subtree and mean-priority estimates updated according to rules shown in FIG7. The total number of nodes within each subtree was always known (c in FIG7), while mean priority estimates per key (m in FIG7) could either be known or unknown. If a mean priority of either one child subtree or a key stored within the current node is unknown then it can be estimated to by exploiting information coming from another sibling subtree or a priority stored within the parent node. Sampling was done by traversing the tree from the root node up while sampling either one of the children subtrees or the currently held key proportionally to the total estimated priority masses contained within. The rules used to evaluate proportions are shown in orange in FIG7. Similarly, probabilities of arbitrary keys can be queried by traversing the tree from the root node towards the child node of an interest while maintaining a product of probabilities at each branching point. Insertion, deletion, sampling and probability query operations can be done in O(ln(n)) time. The suggested algorithm has the desired property that it becomes a simple proportional sampling algorithm once all the priorities are known. While some key priorities are unknown, they are estimated by using nearby known key priorities (Figure 6).Each time when a new sequence key is added to the tree, it was set to have an unknown priority. Any priority was assigned only after the key got first sampled and the corresponding sequence got passed through the learner. When a priority of a key is set or updated, the key node is deliberately removed from and placed back to the tree in order to become a leaf-node. This helped to set priorities of nodes in the immediate vicinity more accurately by using the freshest information available. The value of = 0.01 is the minimum probability of choosing a random action and it is hard-coded into the policy network. FIG8 shows the overall network topology while TAB6 specifies network layer sizes. In this section we compare Reactor with the recently published Rainbow agent BID6. While ACER is the most closely related algorithmically, Rainbow is most closely related in terms of performance and thus a deeper understanding of the trade-offs between Rainbow and Reactor may benefit interested readers. There are many architectural and algorithmic differences between Rainbow and Reactor. We will therefore begin by highlighting where they agree. Both use a categorical action-value distribution critic BID2, factored into state and state-action logits BID13, DISPLAYFORM0 Both use prioritized replay, and finally, both perform n-step Bellman updates. Despite these similarities, Reactor and Rainbow are fundamentally different algorithms and are based upon different lines of research. While Rainbow uses Q-Learning and is based upon DQN , Reactor is an actor-critic algorithm most closely based upon A3C . Each inherits some design choices from their predecessors, and we have not performed an extensive ablation comparing these various differences. Instead, we will discuss four of the differences we believe are important but less obvious. First, the network structures are substantially different. Rainbow uses noisy linear layers and ReLU activations throughout the network, whereas Reactor uses standard linear layers and concatenated ReLU activations throughout. To overcome partial observability, Rainbow, inheriting this choice from DQN, uses frame stacking. On the other hand, Reactor, inheriting its choice from A3C, uses LSTMs after the convolutional layers of the network. It is also difficult to directly compare the number of parameters in each network because the use of noisy linear layers doubles the number of parameters, although half of these are used to control noise, while the LSTM units in Reactor require more parameters than a corresponding linear layer would. Second, both algorithms perform n-step updates, however, the Rainbow n-step update does not use any form of off-policy correction. Because of this, Rainbow is restricted to using only small values of n (e.g. n = 3) because larger values would make sequences more off-policy and hurt performance. By comparison, Reactor uses our proposed distributional Retrace algorithm for off-policy correction of n-step updates. This allows the use of larger values of n (e.g. n = 33) without loss of performance. Third, while both agents use prioritized replay buffers , they each store different information and prioritize using different algorithms. Rainbow stores a tuple containing the state x t−1, action a t−1, sum of n discounted rewards n−1 k=0 r t+k k−1 m=0 γ t+m, product of n discount factors n−1 k=0 γ t+k, and next-state n steps away x t+n−1. Tuples are prioritized based upon the last observed TD error, and inserted into replay with a maximum priority. Reactor stores length n sequences of tuples (x t−1, a t−1, r t, γ t) and also prioritizes based upon the observed TD error. However, when inserted into the buffer the priority is instead inferred based upon the known priorities of neighboring sequences. This priority inference was made efficient using the previously introduced contextual priority tree, and anecdotally we have seen it improve performance over a simple maximum priority approach. Finally, the two algorithms have different approaches to exploration. Rainbow, unlike DQN, does not use -greedy exploration, but instead replaces all linear layers with noisy linear layers which induce randomness throughout the network. This method, called Noisy Networks BID3, creates an adaptive exploration integrated into the agent's network. Reactor does not use noisy networks, but instead uses the same entropy cost method used by A3C and many others , which penalizes deterministic policies thus encouraging indifference between similarly valued actions. Because Rainbow can essentially learn not to explore, it may learn to become entirely greedy in the early parts of the episode, while still exploring in states not as frequently seen. In some sense, this is precisely what we want from an exploration technique, but it may also lead to highly deterministic trajectories in the early part of the episode and an increase in overfitting to those trajectories. We hypothesize that this may be the explanation for the significant difference in Rainbow's performance between evaluation under no-op and random human starts, and why Reactor does not show such a large difference.6.6 ATARI Table 4: Scores for each game evaluated with 30 random human starts. Reactor was evaluated by averaging scores over 200 episodes. All scores (except for Reactor) were taken from BID13, and BID6

Represent the short summary for finding the corresponding scientific article (long)

Reactor combines multiple algorithmic and architectural contributions to produce an agent with higher sample-efficiency than Prioritized Dueling DQN while giving better run-time performance than A3C.

Ensemble method for reinforcement learning that weights Q-functions based on accumulated TD errors.