Question: Represent the article for finding a single-sentence TLDR

Question: State-of-the-art face super-resolution methods employ deep convolutional neural networks to learn a mapping between low- and high-resolution facial patterns by exploring local appearance knowledge. However, most of these methods do not well exploit facial structures and identity information, and struggle to deal with facial images that exhibit large pose variation and misalignment. In this paper, we propose a novel face super-resolution method that explicitly incorporates 3D facial priors which grasp the sharp facial structures. Firstly, the 3D face rendering branch is set up to obtain 3D priors of salient facial structures and identity knowledge. Secondly, the Spatial Attention Mechanism is used to better exploit this hierarchical information (i.e. intensity similarity, 3D facial structure, identity content) for the super-resolution problem. Extensive experiments demonstrate that the proposed algorithm achieves superior face super-resolution and outperforms the state-of-the-art. Face images provide crucial clues for human observation as well as computer analysis . However, the performance of most face image tasks, such as face recognition and facial emotion detection , degrades dramatically when the resolution of a facial image is relatively low. Consequently, face super-resolution, also known as face hallucination, was coined to restore a low-resolution face image to its high-resolution counterpart. A multitude of deep learning methods (; ; ; ; a; b) have been successfully applied in face Super-Resolution (SR) problems and achieve state-of-the-art . But super-resolving arbitrary facial images, especially at high magnification factors, is still an open and challenging problem due to the ill-posed nature of the SR problem and the difficulty in learning and integrating strong priors into a face hallucination model. Some researches (; a;) on exploiting the face priors to assist neural networks to capture more facial details have been proposed recently. A face hallucination model incorporating identity priors is presented in. But the identity prior is extracted only from the multi-scale up-sampling in the training procedure and therefore cannot provide enough extra priors to guide the network to achieve a better . Yu et al. (2018a) employ facial component heatmaps to encourage the upsampling stream to generate super-resolved faces with higher-quality details, especially for large pose variations. Although heatmaps can provide global component regions, it cannot learn the reconstruction of detailed edges, illumination or expression priors. Besides, all of these aforementioned face SR approaches ignore facial structure and identity recovery. In contrast to previous methods, we propose a novel face super-resolution method that embeds 3D face structures and identity priors. Firstly, a deep 3D face reconstruction branch is set up to explicitly obtain 3D face render priors which facilitate the face super-resolution branch. Specifically, the 3D face render prior is generated by the ResNet-50 network . It contains rich hierarchical information, such as low-level (e.g., sharp edge, illumination) and perception level (e.g., identity). The Spatial Attention Mechanism is proposed here to adaptively integrate the 3D facial prior into the network. Specifically, we employ the Spatial Feature Transform (SFT) to generate affine transformation parameters for spatial feature modulation. Afterwards, it encourages the network to learn the spatial interdepenencies of features between 3D facial priors and input images after adding the attention module into the network. The main contributions of this paper are: 1. A novel face SR model is proposed by explicitly exploiting facial structure in the form of facial-prior estimation. The estimated 3D facial prior provides not only spatial information of facial components but also their visibility information, which are ignored by the pixel-level content. 2. We propose a feature-fusion-based network to better extract and integrate the face rendered priors by employing the Spatial Attention Mechanism (SAM). 3. We qualitatively and quantitatively explore multi-scale face super-resolution, especially at very low input resolutions. The proposed network achieves better SR criteria and superior visual quality compared to state-of-the-art face SR methods. Face hallucination relates closely to the natural image super-resolution problem. Thus, in this section, we discuss recent research on super-resolution and face hallucination to illustrate the necessary context for our work. Recently, neural networks have demonstrated a remarkable capability to improve SR . Since the pioneering network can learn to map the relationship between LR and HR (a), a lot of CNN architectures have been proposed for SR (b; ; ; ;). Most of the existing high-performance SR networks have residual blocks (Jiwon to go deeper in the network architecture, and achieve better performance. EDSR improves the performance by removing unnecessary batch normalization layers in residual blocks. A residual dense network (RDN) (a) was proposed to exploit the hierarchical features from all the convolutional layers. Zhang et al. (2018b) proposed the very deep residual channel attention networks(RCAN) to discard abundant low-frequency information which hinders the representational ability of CNNs. used a spatial feature transform layer to introduce the semantic prior as an additional input of SR network. presented a wavelet-based CNN approach that can ultra-resolve a very low resolution face image in a unified framework. However, these networks require a lot of time to train the large-scale parameters to obtain good . In our work, we largely decrease the training parameters, but still achieve the superior performance in SR criteria (SSIM and PSNR) and visible quality. Facial Prior Knowledge: Exploiting facial priors in face hallucination, such as spatial configuration of facial components, is the key factor that differentiates it from generic super-resolution tasks. There are some face SR methods that use facial prior knowledge to better super-resolve LR faces. learned subspaces from LR and HR face images respectively, and then reconstructed an HR output from the PCA coefficients of the LR input. set up a Markov Random Field (MRF) to reduce ghosting artifacts because of the misalignments in LR images. These methods are prone to generate severe artifacts, especially in large pose variations and misalignments in LR images. Yu & Porikli (2017b) interweaved multiple spatial transformer networks with the deconvolutional layers to handle unaligned LR faces. Dahl et al. (2017b) leveraged the framework of PixelCNN to super-resolve very low-resolution faces. presented a cascade bi-network, dubbed CBN, to localize LR facial components first and then upsample the facial components; however, CBN may produce ghosting faces when localization errors occur. Recently, Yu et al. (2018a) used a multi-task convolutional neural network (CNN) to incorporate structural information of faces. built a face recognition model that acts as identity priors for the super-resolution network during training. In our paper, we used the 3D face reconstruction branch to extract the facial structure, detailed edges, illumination, and identity priors. Furthermore, we recover these priors in an explicit way. The 3D shapes of facial images can be restored from unconstrained 2d images by the 3D face reconstruction. In this paper, we employ the 3D Morphable Model (3DMM) based on the fusion of parametric descriptions of face attributes (e.g., gender, identity, and distinctiveness) to reconstruct the 3D facial priors. The reconstructed face will inherit the facial features and present the clear and sharp facial components. Given a low-resolution facial image, the 3D rendering branch aims to extract the 3D face coefficients based on the 3D Morphable Model (3DMM). The high-resolution face rendered image is generated after obtaining the 3D coefficients and regarded as the high-resolution facial priors which facilitate the face super-resolution. The 3D coefficients contain abundant hierarchical knowledge, such as identity, facial expression, texture, illumination, and face pose. The proposed face super-resolution framework is presented in Figure 2, and it consists of two branches: the 3D rendering network to extract the facial prior and the Spatial Attention Mechanism aiming to exploit the prior for the face super-resolution problem. It is still a challenge for state-of-the-art edge prediction methods to acquire very sharp facial structures from low-resolution images. Therefore, a 3DMM-based model is proposed to localize the precise facial structure by generating the 3D facial images which are constructed by the 3D coefficient vector. Besides, there exist large face pose variations, such as inplane and out-of-plane rotations. A large amount of data is needed to learn the representative features varying with the facial poses. To address this problem, an inspiration came from the idea that the 3DMM coefficients can analytically model the pose variation with a simple math derivation (; and does not require a large training set, we utilize a face rendering network based on ResNet-50 to regress a face coefficient vector. The output of the ResNet-50 is the representative feature vector of x = (α, β, δ, γ, ζ) ∈ R 239, where α ∈ R 80, β ∈ R 64, δ ∈ R 80, γ ∈ R 9, and ζ ∈ R 6 represent the identity, facial expression, texture, illumination, and face pose, respectively. According to the Morphable model , we transform the face coefficients to a 3D shape S and texture T of the face image as and where S and T are the average values of the S and T. Besides, B t, B id and B exp denote the base vector of texture, identity, and expression calculated by the PCA method. A modified L 2 based loss function for the 3D face reconstruction is presented based on a paired training set where j is the paired image index, and L is the total number of training pairs. i and M denote the pixel index and face region, respectively. A, I and B represent the skin color based attention mask, the sharp image, and the up-sampling of low-resolution image, respectively. R(B i j (x)) denotes the reconstructed face image based on the learned face vector by the ResNet-50 network. The proposed face super-resolution architecture. Our model consists of two branches: the top block is a ResNet-50 Network to extract the 3D facial coefficients and restore a sharp face rendered structure. The bottom block is dedicated to face super-resolution guided by the facial coefficients and rendered sharp face structures which are concatenated by the Spatial Feature Transform. Given the LR images, the generated 3D face rendered reconstructions are shown in Figure 1. The rendered face predictions contain the clear spatial knowledge and good visual quality of facial components which are very close to the information of the ground-truths. The 3D priors grasp very well the pose variations and skin colour, and further embed pose variations into the SR networks which improve the accuracy and stability in face images with large pose variations. Therefore, we concatenate the reconstructed face image as an additional feature in the SR network. The face expression, identity, texture, illumination, and face pose are transformed into four feature maps and fed into the spatial feature transform block of the SR network. As shown in Figure 2, our Spatial Attention Mechanism aims to exploit the 3D face rendered priors which grasp the precise locations of face components and the facial identity. In order to explore the interdependence and correlation of priors and input images between channels, the attention block is added into the Spatial Attention Mechanism. The proposed network, also named the Spatial Attention Mechanism (SAM), consists of three simple parts: a spatial transform block, an attention block, and an upscale module. We import the 3D face priors into the Spatial Attention Transform Block after a convolutional layer. The 3D face priors consist of two parts: one directly from the rendered face images (as the RGB input), and the other from the feature transformation of the coefficient parameters. The feature transformation procedure is described as follows: firstly, the coefficients of (identity, expression, texture, and the fusion of illumination and face pose) are reshaped to a matrix by setting extra elements to zeros. Afterwards, it is expanded to the same size as the LR images by zero-padding, and then scaled to the interval. Finally, the coefficient features are concatenated with the priors from the rendered face images. The Spatial Feature Transform (SFT) learns a mapping function Θ that provides a modulation parameter pair (µ, ν) according to the priors ψ, such as segmentation probability. Instead, the 3D face priors are taken as the input. The outputs of the SFT layer are adaptively controlled by the modulation parameter pair by way of applying an affine transformation spatially to each intermediate feature map. Specifically, the intermediate transformation parameters (µ, ν) are derived from the priors ψ by a mapping function as: and then where N denotes the SR network, and θ represents trainable parameters of the network. The intermediate feature maps are modified by scaling and shifting feature maps according to the transformation parameters: where F denotes the feature maps, and ⊗ is referred to element-wise multiplication. At this step, the SFT layer implements the spatial-wise transformation. Attention mechanism can be viewed as a guide to bias the allocation of available processing resources towards the most informative components as input . Consequently, the channel module is presented to explore the most informative components and the interdependency between the channels. The attention module is composed of a series of residual channel attention blocks (RCAB) shown in Figure 2. Inspired by the integration of channel attention and residual blocks, we ensemble a series of residual channel attention blocks. For the b-th block, the output F b of RCAB is obtained by: where C b denotes the channel attention function. F b−1 is the block's input, and X b is calculated by two stacked convolutional layers. In order to evaluate the performance of our priors and algorithms, we compare them with the startof-art methods qualitatively and quantitatively.. We use open-resource implementations from the authors and train all the networks on the same dataset for a fair comparison. We propose two models: first is the VDSR+ which is the basic VDSR model embedded with the 3D facial prior as extra RGB channel information and the other is our SR network incorporating facial priors by the Spatial Attention Mechanism (SAM). The implementation code will be made available to the public. More are shown in the supplementary material. CelebA and Menpo dataset are used to verify the performance of the algorithm. The training phase uses 162,080 images from the CelebA dataset. In the testing phase, 40,519 images from the CelebA test set are used along with the large-pose-variation test set from the Menpo dataset. The every facial pose test set of Menpo (left, right and semi-frontal) contains 1000 images, respectively. The HR ground-truth images are obtained by center-cropping the facial images and then resizing them to the 128×128 pixels. The LR face images are generated by downsampling HR ground-truths to 32×32 pixels (4 scale) and 16×16 pixels (8 scale). In our network, the ADAM optimizer is used with a batch size of 64 for training, and input images are center-cropped as RGB channels. The initial learning rate is 0.0002 and is divided by 2 every 50 epochs. The whole training process takes 2 days with an NVIDIA Titan X GPU. Quantitative evaluation of the network using PSNR and the structural similarity (SSIM) scores for the CelebA test set are listed in Table 1. Furthermore, in order to analyze the proposed methods' performance and stability regarding to large face pose variations, three case corresponding to different face poses (left, right, and semifrontal) of the Menpo test data are listed in Table 2. CelebA Test: Ours (VDSR+) achieves significantly better (1 dB higher than the remaining best method and 2 db higher than the basic VDSR method in x8 SR) even for the large-scale parameter methods, such as RDN and RCAN. But it does perform slightly worse than ours (SAM). It should be noted that ours (VDSR+) is the same as VDSR except for the extra 3D face priors as the Figure 5: Visual comparison with state-of-the-art methods(×8). The from the proposed method have less visual artifacts and more details on key face components (e.g., eyes, mouth, and nose) RGB channel inputs. It indicates that the 3D priors make a great contribution to the performance improvement (average 1.6 db improvement) of face super-resolution. Menpo Test: To verify the effectiveness and stability of face priors and our proposed network towards large pose variations, the PSNR and SSIM of face poses are listed in Table 2. While ours (SAM) is the best method superior than others, VDSR+ achieves 1.8db improvement compared with the basic VDSR method in the magnification factors (×4). Super-resolution: The qualitative of our methods at different magnifications (×4 and ×8) are shown respectively in Figures 3 and 4. It can be observed that our proposed method recovers clearer faces with finer component details (e.g., nose and eyes). Artifacts: The outputs of most methods (e.g., RCAN, RDN, and Wavelet-SRNet) contain some artifacts around facial components, such as the eyes, nose, and mouth shown in Figure 5. After adding the rendered face priors, ours show clear and sharp facial structures without any ghosting artifacts. It illustrates that the proposed 3D priors can help the network understand the spatial location and the entire face structure. Ablation Study: In this section, we conduct an ablation study to demonstrate the effectiveness of each module. We compare the proposed network with and without using the rendered 3D face priors and the Spatial Attention Mechanism (SAM) in terms of PSNR and SSIM on the test data. As shown in Figure 6 (b, f), the baseline method without rendered faces and SAM tends to generate blurry faces that cannot capture sharp edges. Figure 6 (c and g) shows clearer and sharper facial structures after adding the rendered priors. By using SAM, the visual quality is further improved in Figure 6 (d and h). The quantitative comparisons between (VDSR, our VDSR+, and our SAM) in Tables 1 and 2 also illustrate the effectiveness of the rendered priors and the Spatial Attention Mechanism. Model Size Analysis: Figure 7 shows comparisons of model size and performance. Our networks, VDSR+ and SAM, embedded with 3D priors are more lightweight while still achieving the best performance even compared with other state-of-the-art methods (e.g., RCAN and RDN) with a larger scale of parameters. In this paper, we proposed a novel network that incorporates 3D facial priors of rendered faces and identity knowledge. The 3D rendered branch utilizes the face rendering loss to encourage a highquality guided image providing clear spatial locations of facial components and other hierarchical information (i.e., expression, illumination, and face pose). To well exploit 3D priors and consider the channel correlation between priors and inputs, the Spatial Attention Mechanism is presented by employing the Spatial Feature Transform and Attention block. The comprehensive experimental have demonstrated that the proposed method can deliver the better performance and largely decrease artifacts in comparison with the state-of-the-art methods by using significantly fewer parameters. Semi-Frontal Facial Pose Visualization: For the semi-frontal pose, the SR of RCAN, RDN and Wavelet-SRNet have a lot of artifacts around facial components (e.g., eyes, teeth, nose, mouth). Fortunately, after incorporating the rendered face priors, it largely avoids the appearance of ghosting artifacts, seen in Figure.8 Left Facial Pose Visualization: For the left pose, the high-resolution of the proposed method perform much better. Ours (VDSR+) which exploiting the 3D facial priors can grasp the facial structure knowledge and restore the high-resolution facial components (e.g. mouth) much closer to the ground-truth compared with the basic VDSR method without priors. Right Facial Pose Visualization: For the right pose, the high-resolution of the proposed method are still the best. Adding the facial structure priors can help network to learn the location of facial components even for the large pose variation. High Magnification Factor × 8 Visualization: It is still a challenge to generate the sharp superresolution images for a large magnification factor (×8). The 3D rendered facial priors provide extra facial structure knowledge that are crucial for SR problems. As shown in Figure 12 and 13, the proposed method generates a high visible quality of SR images even for the large magnification factor. Learning Curves with Different Ablation Configurations:To verify the effectiveness of 3D facial structure priors, we design the three different configurations (w/o 3D priors, w/o Spatial Attention Mechanism): baseline methods (i.e., VDSR, SRCNN); baseline incorporating 3D facial priors (i.e., VDSR+,SRCNN+); the method using the Spatial Attention Mechanism and 3D priors (our proposed method: +priors and +SAM). The learning curves of each configuration are plotted to show the effectiveness of the each block. The priors are easy to insert into any network without increasing any parameters, but largely improve the accuracy and the convergence of the algorithms shown in Figure 14. Quantitative Results with Different Ablation Configurations: As shown in Table 3, each block boosts the accuracy of baseline algorithms: the average performance improvement stemming from 3D facial priors and from Spatial Attention Mechanism are 1.6db and 0.57db, respectively. Qualitative Evaluation with different ablation configurations: The baseline incorporated with the facial rendered priors tends to avoid some artifacts around the key facial components and generate more sharp edges compared with the basic baseline method without the facial priors. By adding the Spatial Attention Mechanism, it could help the network better exploit the priors and is easier to generate more sharp facial structures, shown in Figure 15.

Represent the TLDR for finding the corresponding scientific article

We propose a novel face super resolution method that explicitly incorporates 3D facial priors which grasp the sharp facial structures.

represent the short summary for finding the corresponding scientific article (long)

We introduced a shallow featrue extraction network with a large receptive field for stereo matching tasks, which uses a simple structure to get better performance.


Question: We present Cross-View Training (CVT), a simple but effective method for deep semi-supervised learning. On labeled examples, the model is trained with standard cross-entropy loss. On an unlabeled example, the model first performs inference (acting as a "teacher") to produce soft targets. The model then learns from these soft targets (acting as a ``"student"). We deviate from prior work by adding multiple auxiliary student prediction layers to the model. The input to each student layer is a sub-network of the full model that has a restricted view of the input (e.g., only seeing one region of an image). The students can learn from the teacher (the full model) because the teacher sees more of each example. Concurrently, the students improve the quality of the representations used by the teacher as they learn to make predictions with limited data. When combined with Virtual Adversarial Training, CVT improves upon the current state-of-the-art on semi-supervised CIFAR-10 and semi-supervised SVHN. We also apply CVT to train models on five natural language processing tasks using hundreds of millions of sentences of unlabeled data. On all tasks CVT substantially outperforms supervised learning alone, ing in models that improve upon or are competitive with the current state-of-the-art. Deep learning classifiers work best when trained on large amounts of labeled data. However, acquiring labels can be costly, motivating the need for effective semi-supervised learning techniques that leverage unlabeled examples during training. Many semi-supervised learning algorithms rely on some form of self-labeling. In these approaches, the model acts as both a "teacher" that makes predictions about unlabeled examples and a "student" that is trained on the predictions. As the teacher and the student have the same parameters, these methods require an additional mechanism for the student to benefit from the teacher's outputs. One approach that has enjoyed recent success is adding noise to the student's input BID0 BID50. The loss between the teacher and the student becomes a consistency cost that penalizes the difference between the model's predictions with and without noise added to the example. This trains the model to give consistent predictions to nearby data points, encouraging smoothness in the model's output distribution with respect to the input. In order for the student to learn effectively from the teacher, there needs to be a sufficient difference between the two. However, simply increasing the amount of noise can in unrealistic data points sent to the student. Furthermore, adding continuous noise to the input makes less sense when the input consists of discrete tokens, such in natural language processing. We address these issues with a new method we call Cross-View Training (CVT). Instead of only training the full model as a student, CVT adds auxiliary softmax layers to the model and also trains them as students. The input to each student layer is a sub-network of the full model that sees a restricted view of the input example (e.g., only seeing part of an image), an idea reminiscent of cotraining BID1. The full model is still used as the teacher. Unlike when using a large amount of input noise, CVT does not unrealistically alter examples during training. However, the student layers can still learn from the teacher because the teacher has a better, unrestricted view of the input. Meanwhile, the student layers improve the model's representations (and therefore the teacher) as they learn to make accurate predictions with a limited view of the input. Our method can be easily combined with adding noise to the students, but works well even when no noise is added. We propose variants of our method for Convolutional Neural Network (CNN) image classifiers, Bidirectional Long Short-Term Memory (BiLSTM) sequence taggers, and graph-based dependency parsers. For CNNs, each auxiliary softmax layer sees a region of the input image. For sequence taggers and dependency parsers, the auxiliary layers see the input sequence with some context removed. For example, one auxiliary layer is trained to make predictions without seeing any tokens to the right of the current one. We first evaluate Cross-View Training on semi-supervised CIFAR-10 and semi-supervised SVHN. When combined with Virtual Adversarial Training BID39, CVT improves upon the current state-of-the-art on both datasets. We also train semi-supervised models on five tasks from natural language processing: English dependency parsing, combinatory categorical grammar supertagging, named entity recognition, text chunking, and part-of-speech tagging. We use the 1 billion word language modeling benchmark BID3 as a source of unlabeled data. CVT works substantially better than purely supervised training, ing in models that improve upon or are competitive with the current state-of-the-art on every task. We consider these particularly important because many recently proposed semi-supervised learning methods work best on continuous inputs and have only been evaluated on vision tasks BID0 BID50 BID26 BID59. In contrast, CVT can handle discrete inputs such as language very effectively. Semi-supervised learning in general has been widely studied BID2. Early approaches to deep semi-supervised learning pre-train neural models on unlabeled data, which has been successful for applications in computer vision BID21 BID28 and natural language processing BID7 BID46. More recent work incorporates generative models based on autoencoders BID22 BID47 or Generative Adversarial Networks BID55 BID51 into the training. Self-Training. One of the earliest approaches to semi-supervised learning is self-training BID52 BID11. Initially, a classifier is trained on labeled data only. In each subsequent round of training, the classifier, acting as a "teacher," labels some of the unlabeled data and adds it to the training set. Then, acting as a "student," it is retrained on the new training set. The new examples added each round act as noisy "pseudo labels" BID29 that the model can learn from. Many recent approaches train the student with soft targets from the teacher's output distribution rather than a hard label, making the procedure more akin to knowledge distillation BID16.Consistency Training and Distributional Smoothing. Recent works add noise to the student's input BID0 BID50. This trains the model to give consistent predictions to nearby data points, encouraging distributional smoothness in the model. Inspired by the success of adversarial training BID12, BID37 extend this idea by adversarially selecting the perturbation to the input. Other approaches focus on improving the targets provided by the teacher by tracking an exponential moving average of its predictions BID26 or its weights BID59. Our method is complimentary to these previous approaches, and can be combined with them effectively. Co-Training. Co-Training BID1 BID41 trains two models with disjoint views of the input. On unlabeled data, each one acts as a "teacher" for the other model. In contrast, our approach trains a single unified model where auxiliary prediction layers see different, but not necessarily independent views of the input. Auxiliary Prediction Layers. Another way of leveraging unlabeled data is through the addition of auxiliary "self-supervised" losses. These approaches train auxiliary prediction layers on tasks where performance can be measured without human-provided labels. Previous work has jointly trained image classifiers with tasks like relative position and colorization BID9, sequence taggers with language modeling BID48, and reinforcement learning agents with predicting changes in the environment BID20. Unlike these approaches, our auxiliary losses are based on self-labeling, not labels deterministically constructed from the input. Data Augmentation. Data augmentation, such as random translations or crops of input images, bears some similarity to our method in that it also exposes the model to different views of input examples. Data augmentation has become a common practice for both supervised and semi-supervised training of image classifiers BID54. We first provide a general description of Cross-View Training. We then present specific constructions for auxiliary prediction layers that work well for image classification, sequence tagging, and dependency parsing. We use D l = {(x 1, y 1), (x 2, y 2),..., (x N, y N)} to represent a labeled dataset and D ul = {x 1, x 2, ..., x M} to represent an unlabeled dataset. We use p θ (y|x i) to denote the output distribution over classes produced by a model with parameters θ on input x i. Our approach uses a standard cross-entropy loss over the labeled data: DISPLAYFORM0 On unlabeled data, a popular approach is to add a consistency cost encouraging distributional smoothness in the model. First, the model produces soft targets for the current example:ŷ i = p θ (y|x i). The model is then trained to minimize the consistency cost DISPLAYFORM1 where D is a distance function (we use KL divergence) and η is a perturbation to the input that can be chosen randomly or adversarially. As is common in prior work, we hold the teacher's prediction y i fixed during training (i.e., we don't back-propagate through it) so the student learns to imitate the teacher, but not vice versa. Our dependency parsing models use auxiliary layers analogous to the "forward" and "backward" sequence tagging ones. Cross-View Training adds k additional prediction layers p )produced by the model. It outputs a distribution over labels, usually with a softmax layer (an affine transformation followed by a softmax activation function) applied to this representation: DISPLAYFORM2 At test time, only the main prediction layer p θ is used. Each h j is chosen such that it only uses a part of each input x i; the particular choice can depend on the task and model architecture. We propose variants for CNN image classifiers, BiLSTM sequence taggers, and graph-based dependency parsers in sections 3.2, 3.3, and 3.4. We add the distances between the output distributions of the teacher and auxiliary students to the consistency loss, ing in a cross-view consistency (CVC) loss: DISPLAYFORM3 We combine the supervised and CVC losses into the total loss, L = L sup + λ 2 L CVC, and minimize it with stochastic gradient descent. At each step, L sup is computed over a minibatch of labeled examples and L CVC is computed over a minibatch of unlabeled examples. λ 1 and λ 2 are hyperparameters controlling the strength of the auxiliary prediction layers and the strength of the unsupervised loss. For all experiments we set λ 1 = k and λ 2 = 1 unless indicated otherwise. See FIG0 for an illustration of the training procedure. Although adding noise or an adversarial perturbation to the input generally improves , L CVC can be trained without this enhancement (i.e., setting η = 0). In this case, the first term inside the expectation disappears (the student will exactly match the teacher, so the distance is zero). In contrast, L consistency requires a nonzero η to make the student and teacher output different distributions. In most neural networks, a few additional softmax layers is computationally cheap compared to the portion of the model building up representations (such as a CNN or RNN). Therefore our method contributes little overhead to training time over consistency training. CVT does not change inference time because the auxiliary layers are only used during training. Our image recognition models are based on Convolutional Neural Networks, which produce a set of features H(x i) ∈ R n×n×d from an image x i. The first two dimensions of H index into the spatial coordinates of feature vectors and d is the size of the feature vectors. For shallower CNNs, a particular feature vector corresponds to a region of the input image. For example, H 0,0 would be a d-dimensional vector of features extracted from the upper left corner. For deeper CNNs, a particular feature vector would be extracted from the whole image, but still only use a "region" of the representations from an earlier layer. The CNNs in our experiment are all in the first category. The primary prediction layer for our CNNs take as input the mean of H over the first two dimensions, which in a d-dimensional vector that is fed into a softmax layer: p θ (y|x i) = SML(global average pool(H)).We add n 2 auxiliary softmax layers to the top of the CNN. The jth layer takes a single feature vector as input, as shown in the left of FIG1: p j θ (y|x i) = SML(H j/n,j mod n). We also experimented with adding auxiliary softmaxes to the outputs of earlier layers in the CNN, but found this did not improve performance. In sequence tagging, each example (DISPLAYFORM0 We assume an L-layer bidirectional RNN sequence tagging model, which has become standard for many sequence tagging tasks BID13 BID14 . Each layer runs an RNN such as an LSTM BID18 in the forward direction (taking x t i as input at each step t) and the backward direction (taking x T −t+1 i as input at each step) and concatenates the . A softmax layer on top of the outputs of the last BiRNN layer, DISPLAYFORM1 The auxiliary softmax layers take DISPLAYFORM2, the outputs of the forward and backward RNNs in the first BiRNN layer, as inputs. We add the following four softmax layers to the model (see the right of FIG1): DISPLAYFORM3 The "forward" and "backward" prediction layers use the RNN's current output to predict the current token. The "future" and "past" layers use the RNN's previous output (or, equivalently, they predict the label for the next token). The forward layer makes each prediction without seeing the right context of the current token. The future layer makes each prediction without the right context or the current token itself. Therefore it works like a neural language model that, instead of predicting which token comes next in the sequence, predicts which class of token comes next in the sequence. In a dependency parse, words in a sentence are treated as nodes in a graph. Typed directed edges connect the words, forming a tree structure describing the syntactic structure of the sentence. In particular, each word x To give a specific example, in the sentence "The small dog barked", the correct label for "small" would be the edge ("dog", "small", adjectival-modifier).We use a neural graph-based dependency parser similar to the one from BID10. It first runs a BiRNN encoder over the sentence as described in section 3.3, producing a sequence of DISPLAYFORM0 is passed through two separate multilayer perceptrons, one producing a representation for x t i as a head word and one producing a representation for it as a dependent. A bilinear classifier applied to these representations produces a score for each candidate edge. Lastly, these scores are passed through a softmax layer to produce probabilities. Mathematically, the probability of an edge is given as p θ ((u, t, r) DISPLAYFORM1 Where s is the scoring function s(z 1, z 2, r) = MLP head (z 1)(W r + W)MLP dep (z 2). The bilinear classifier uses a weight matrix W r specific to the candidate relation as well as a weight matrix W shared across all relations. We add four auxiliary prediction layers to our model for cross-view training: DISPLAYFORM2 Each auxiliary layer has some missing context (not seeing either the preceding or following words) for the candidate head and candidate dependent. All the parameters for the scoring function of each auxiliary prediction layer are layer-specific. To validate our approach, we evaluate Cross-View Training on two semi-supervised learning benchmarks. These discard most of the labels from standard image image recognition datasets to artificially make them semi-supervised. As a sterner test of our approach, we also apply CVT to five tasks from Natural Language Processing (NLP) using hundreds of millions of unlabeled sentences for semi-supervised learning. Data. We experiment on two semi-supervised image recognition benchmarks. These are constructed from the CIFAR-10 (BID23) and Street View House Numbers (SVHN) BID40 datasets. Following previous work, we make the datasets semi-supervised by only using the provided labels for a subset of the examples in the training set; the rest are treated as unlabeled examples. Model. We use the convolutional neural network from Miyato et al. We add 36 auxiliary softmax layers to the 6 × 6 collection of feature vectors produced by the CNN. Each auxiliary layer sees a patch of the image ranging in size from 21 × 21 pixels (the corner) to 29 × 29 pixels (the center) of the 32 × 32 pixel images. We optimize L with λ 1 = 1 and each minibatch consisting of 32 labeled and 128 unlabeled examples. Miyato et al. use Virtual Adversarial Training (VAT), minimizing L consistency with the input perturbation η chosen adversarially. We train our cross-view models (which instead use L CVC) both with and without this adversarial noise. We report with and without using data augmentation (random translations for SVHN and random translations and horizontal flipping for CIFAR-10) in Table 1.Results. CVT works well as semi-supervised learning method without any noise being added to the student. When random noise is added, it performs close to VAT (the standard-deviation-based confidence intervals intersect) while training much faster (requiring only one backwards pass for each training minibatch, while VAT requires an additional one to compute the adversarial perturbation). Our method can easily be combined with VAT, ing in further improvements and state-of-theart . The benefit of CVT is less when data augmentation is applied, perhaps because random translations of the input expose the model to different "views" in a similar manner as with CVT. We believe the gains on SVHN are smaller than CIFAR-10 because the digits in SVHN occur in the center of the image, so the auxiliary softmaxes seeing the sides and corner do not learn as effectively. We also note that incorporating auxiliary softmax layers into the supervised loss L sup does not improve (see Appendix C). This indicates that the benefit of CVT comes from the improved self-training mechanism, not the additional losses regularizing the model. Model Analysis. To understand why CVT produces better , we compare the behavior of the VAT and CVT (with adversarial noise) models trained on CIFAR-10. First, we record the average value of each feature vector produced by the CNNs when they run over the test set. As shown in the left of FIG5, the CVT model has higher activation strengths for the feature vectors corresponding to the edges of the image. We hypothesize that the VAT model fits to the data while primarily using BID8 f Miyato et al. (2017b) *We found Miyato et al.'s implementation produces slightly different than the ones they report in their paper. Table 1: Error rates on semi-supervised learning benchmarks. We report means and standard deviations from 5 runs. + after a dataset means data augmentation was applied. In contrast, the model with CVT must learn meaningful representations for the edge regions in order to train the corresponding auxiliary softmax layers. As these feature vectors are more useful, their magnitude become larger so they contribute more to the final representation produced by the global average pool. To compare to discriminatory power of the feature vectors, we freeze the weights of the CNNs and add auxiliary softmax layers that are trained from scratch. We then measure the accuracies of the added layers (see the center and right of FIG5 . Unsurprisingly, the VAT model, which only learns representations that will be useful after the average pool, has much lower accuracies from individual feature vectors. The difference is particularly striking in the sides and corners, where CVT accuracies are around 50% higher (they are about 25% higher in the center). This finding further indicates that CVT is improving the model's representations, particularly for the outside parts of images. Data. Although the widely-used benchmarks in the previous section provide validation of our approach, they are small datasets that are artificially made to be semi-supervised. In this section, we show CVT is successful on well-studied tasks where semi-supervised learning is rarely applied. In particular, we train semi-supervised models on the following NLP tasks:• Combinatory Category Grammar (CCG) Supertagging: Labeling words with CCG supertags: lexical categories that encode information about the predicate-argument structure of the sentence. CCG is widely used in syntactic and semantic parsing. We use data from CCGBank BID19 and report word-level accuracy.• Text Chunking: Dividing a sentence into syntactically correlated parts (e.g., a noun phrase followed by a verb phrase). We use the CoNLLL-2000 shared task data (Tjong Kim BID60 and report the F1 score over predicted chunks.• Named Entity Recognition (NER): Identifying and classifying named entities (organizations, places, etc.) in a sentence. We use the CoNLL-2003 dataset (Tjong Kim BID61 and report entity-level F1 score.• Part-of-Speech (POS) Tagging: Labeling words with their syntactic categories (e.g., determiner, adjective, etc.). We use the Wall Street Journal (WSJ) portion of the Penn Treebank BID36 and report word-level accuracy.• Dependency Parsing: Inferring a tree-structure describing the syntactic structure of a sentence. We use the Penn Treebank converted to Stanford Dependencies (version 3.3.0) and report unlabeled and labeled attachment score (UAS and LAS).We use the 1 Billion Word Language Model Benchmark BID3 as a pool of unlabeled sentences for semi-supervised learning. Models. We use a CNN-BiLSTM sequence tagging model BID5. The model first represents each word as the sum of a word embedding and the output of a character-level CNN. This sequence of word representations is then fed through two BiLSTM layers and a softmax layer to produce predictions. See Appendix A for details about the model. Our dependency parser uses the same CNN-BiLSTM encoder as our sequence tagger. As described in Section 3.4, a MLP-Bilinear classifier on top of the encoder makes the predictions. Although it is common for dependency parsers to take words and part-of-speech tags as inputs, our model only takes words as inputs. See Appendix B for details about the model. BID38 were able to apply Virtual Adversarial Training to document classification, but we found VAT ineffective for our word-level tasks. Although we experimented with constraining the word embeddings to unit length and adding random or adversarial perturbations to them during training, it did not improve performance. This is perhaps because, unlike with RGB values in an image, words are discrete, so adding noise to their representations is less meaningful. Instead, we add dropout to the student but not the teacher. Recent work BID48 has shown that jointly training a neural language model with sequence taggers improves . We report accuracies with and without this enhancement (training the language model on the unlabeled data). See TAB1 for sequence tagging and TAB3 for dependency parsing . Results. CVT significantly improves over the supervised baseline on all tasks, both with and without the auxiliary language modeling objective. We report a new state-of-the-art for CCG-supertagging and pure dependency parsing (i.e., without using constituency parse annotations) and competitive with the current state-of-the-art on the other tasks. Our dependency parsing is particularly important because our model does not include part-of-speech tags as input, which other works have shown to improve performance notably BID10 BID4. Of the prior listed in the BID31 b c BID30 d BID62 e BID15 f g * The full TagLM model has many times more parameters than ours. TagLM-2048 is of more comparable size to our models, although still larger. TAB1: Results for sequence tagging tasks. We report the means and standard deviation of 5 runs. "Baseline" trains with L sup, "Consistency" trains with" L sup + L consistency, and "CVT" trains with L sup + L CVC. +LM means language modeling is added as an auxiliary task on the unlabeled data. Depparse UAS Depparse LAS BID15 94.67 92.90 BID35 94.9 93.0 BID53 95.33 - BID10 95 BID25, and BID32 because these train constituency parsers and convert the system outputs to dependency parses. They produce higher scores, but have access to more information during training and do not apply to datasets without constituency annotations. Although the large TagLM model is competitive with ours for Chunking and NER, reducing the size of TagLM to having 2048 hidden units already causes it to perform worse than our model. Although there has been a large body of work successfully applying consistency-cost-based learning to vision tasks, we find it does not provide the same gains for NLP. Training a model with the consistency loss L consistency did not improve over the baseline for sequence tagging and only slightly improved over the baseline for dependency parsing. This is perhaps due to the lack of benefit from adding noise when the input consists of discrete tokens as discussed earlier. CVT, on the other hand, works well as a semi-supervised learning method for NLP. mance, but the "future" and "past" layers are more beneficial than the "forward" and "backward" ones, perhaps because theses provide a more distinct and challenging view of the input. Training Larger NLP Models. Most sequence taggers and dependency parsers in prior use work small LSTMs (hidden state sizes of at most 500 units) because larger models yield little to no gains in performance BID49. We found our own supervised approaches and, to a lesser extent, our models when only using language modeling as the auxiliary task to also not benefit from increasing the model size. In contrast, when using CVT accuracy scales much better with model size (see FIG6 . This suggests the appropriate semi-supervised learning methods may enable the development of larger, more sophisticated models for natural language processing tasks with limited amounts of labeled data.found this to slightly improve accuracy and significantly reduce the variance in accuracy between models trained with different random initializations. For Chunking and Named Entity Recognition, we use a BIOES tagging scheme. The model is trained using SGD with momentum BID45 BID57 . Word embeddings are initialized with GloVe vectors BID42 We use the same 2-layer CNN-BiLSTM encoder and the same hyperparameters as listed in Appendix A. The MLPs used to produce representations for candidate head and dependent words have one hidden layer of size 512 with a ReLU activation and an output layer of size 256. We apply dropout to the output of the hidden layer. We omit punctuation from evaluation, which is standard practice for the PTB-SD 3.3.0 dataset. In initial experiments, we explored whether cross-view losses could benefit purely supervised classifiers. To do this, we trained models with the following objective: DISPLAYFORM0 See Section 3.1 for an explanation of the notation. We hoped that adding auxiliary softmax layers with different views of the input would act as a regularizer on the model. However, we found little to no benefit from this approach. For sequence tagging, improved slightly on CCG and POS but degraded on NER and Chunking. For image recognition, we augmented WideResNet BID63 with auxiliary softmax layers and evaluated it on CIFAR-10 and CIFAR-100. On both datasets, the augmented model performed slightly worse (by ∼0.2% on CIFAR-10 and ∼0.9% on CIFAR-100).We also experimented with using L sup-cv instead of of L sup on semi-supervised CIFAR-10 and CIFAR-10+. Surprisingly, it (slightly) decreased performance for all of the methods we experimented with: supervised training, VAT, CVT, and CVT with adversarial noise. We note we only tried these experiments with λ 1 = 1, but this value of λ 1 did work well for the semi-supervised setting. These negative suggest that the gains are from CVT are from the improved self-training mechanism, not the additional prediction layers regularizing the model.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Self-training with different views of the input gives excellent results for semi-supervised image recognition, sequence tagging, and dependency parsing.

Deep neural networks trained with data augmentation do not require any other explicit regularization (such as weight decay and dropout) and exhibit greater adaptaibility to changes in the architecture and the amount of training data.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: I show how it can be beneficial to express Metropolis accept/reject decisions in terms of comparison with a uniform value, and to then update this uniform value non-reversibly, as part of the Markov chain state, rather than sampling it independently each iteration. This provides a small improvement for random walk Metropolis and Langevin updates in high dimensions. It produces a larger improvement when using Langevin updates with persistent momentum, giving performance comparable to that of Hamiltonian Monte Carlo (HMC) with long trajectories. This is of significance when some variables are updated by other methods, since if HMC is used, these updates can be done only between trajectories, whereas they can be done more often with Langevin updates. This is seen for a Bayesian neural network model, in which connection weights are updated by persistent Langevin or HMC, while hyperparameters are updated by Gibbs sampling. A decision to accept or reject a Metropolis proposal to move from x to x * can be done by checking whether u < π(x *)/π(x), where π is the density function for the distribution being sampled, and u is a uniform random variable. Standard practice is to generate a value for u independently for each decision. I show here that it can be beneficial to instead update u each iteration without completely forgetting the previous value, using a non-reversible method. Doing non-reversible updates to u will not change the fraction of proposals that are accepted, but can in acceptances being clustered together (with rejections similarly clustered). This can be beneficial, for example, when rejections cause reversals of direction, as in variant of Langevin updates with persistent momentum. For any MCMC method, we can augment the variable of interest, x, with density π(x), by a variable s, whose conditional distribution given x is uniform over [0, π(x) ]. The ing joint distribution for x and s is uniform over the region where 0 < s < π(x). This is the view underlying "slice sampling" (Neal 2003), in which s is introduced temporarily, by sampling uniformly from [0, π(x)], and then forgotten once a new x has been chosen. Metropolis updates can also be viewed in this way, with the new x found by accepting or rejecting a proposal, x *, according to whether π(x *) > s, with s newly sampled from [0, π(x) ]. However, it is valid to instead retain s in the state between updates, utilizing its current value for accept/reject decisions, and updating this value when desired by any method that leaves invariant the uniform distribution on [0, π(x)] (since this is the conditional distribution for s given x). Equivalently, one can retain in the state a value u whose distribution is uniform over, independent of x, with u corresponding to s/π(x). Accept/reject decisions are then made by checking whether u < π(x *)/π(x). Note, however, that if x * is accepted, u must then be updated to u π(x)/π(x *), which corresponds to s not changing. Here, I will consider non-reversible updates for u, which translate it by some fixed amount, δ, and perhaps add some noise, reflecting off the boundaries at 0 and 1. It is convenient to express such an update with reflection in terms of a variable v that is uniform over [−1, +1], and define u = |v|. An update for v can then be done as follows: For any δ and any distribution for the noise (not depending on the current value of v), this update leaves the uniform distribution over [−1, +1] invariant. The full state consists of x and v, with x having density π(x) and, independently, v being uniform over [−1, +1], which corresponds to the conditional distribution of s = |v|π(x) given x being uniform over [0, π(x) ]. If a proposed move from x to x * is accepted we change v to v π(x)/π(x *), which leaves s unchanged (allowing the reverse move). Because of this change to v on acceptance, when π(x) varies continuously, it may not be necessary to include noise in the update for v, but if π(x) has only a finite number of possible values, adding noise may be necessary to ensure ergodicity. The hope with these non-reversible updates is that u will move slowly (if δ and the noise amount are small) between values near 0, where acceptance is easy, and values near 1, where acceptance is harder. (But note that u may change in either direction when proposals are accepted, though s will not.) Non-reversibly updating u will not change the overall acceptance rate, but it is expected that acceptances and rejections will become more clustered -with an accepted proposal more likely to be followed by another acceptance, and a rejected proposal more likely to be followed by another rejection. We might wish to intermingle Metropolis updates for x that use v to decide on acceptance with other sorts of updates for x -for example, Gibbs sampling updates, or Metropolis updates accepted in the standard fashion. We can do these updates while ignoring v, and then simply resume use of v afterwards, since v is independent of x. We could also include several independent v variables in the state, using different v values for different classes of updates, but this generalization is not explored here. A small benefit from non-reversible updating of u can be seen with simple random-walk Metropolis updates. Such updates operate as follows: Here are when sampling a 40-dimensional Gaussian distribution with identity covariance matrix: The values for σ used were the stepsizes shown above scaled down by 40 1/2. The autocorrelation times (one plus twice sum of autocorrelations) shown are for groups of 40 iterations (hence the single-iteration autocorrelation time is about 40 times larger). When estimating the mean of a single coordinate, little difference is seen between the standard method and using a non-reversible update for u. But for estimating the mean of the log of the probability density, the non-reversible method is about 1.14 times better. A similar small benefit is seen for simple Langevin updates. One possible explanation for the improvement is that, as noted by , the performance of Metropolis methods in high dimensions is limited by their ability to sample different values for the log of the density. In D dimensions, the log density typically varies over a range proportional to D 1/2. A Metropolis update will typically change the log density only by about one -larger decreases in the log density are unlikely to be accepted, and it follows from reversibility that increases in the log density of much more than one must also be rare (once equilibrium has been reached). Since standard Metropolis methods are reversible, these changes of order one will occur in a random walk, and so around D steps will be needed to traverse the range of log densities of about D 1/2, limiting the speed of mixing. The gain seen from using non-reversible updates for u may come from helping with this problem. When u is small few proposals will be rejected, and the chain will tend to drift towards smaller values for the log density, with the opposite behaviour at times when u is near one. This could reduce the random walk nature of changes in the log density. I obtained more interesting when applying the non-reversible acceptance method to the one-step, non-reversible version of Hamiltonian Monte Carlo (Duane, et al 1987) due to. This method is a "persistent" form of "Langevin" update. See the review by for more discussion of these methods. Hamiltonian Monte Carlo works in an extended state space with momentum variables, p, newly sampled each iteration. It proposes a new value for (x, p) by simulating Hamiltonian dynamics with some number of "leapfrog" steps (and then negating p, so the proposal is reversible). A leapfrog step has the form In the limit as the stepsize η goes to zero, the proposed point will always be accepted. If many leapfrog steps are used, the proposed point can be distant from the current point, avoiding the slowdown from doing a random walk with small steps. In Horowitz's method, only one leapfrog step is done, but a trick is used so that these steps nevertheless usually keep going in the same direction, except on a rejection. These updates operate as follows: 2) Find (x *, p *) from (x ′, p ′) with one "leapfrog" step (as in HMC), with stepsize η. For α near 1, Step only slightly changes p. If Step accepts, the negation in the proposal is canceled by the negation in Step. But a rejection will reverse p, leading the chain to almost double back on itself. Unfortunately, even with this non-reversibility trick, Horowitz's method is not as efficient as HMC with long trajectories. To reduce the rejection rate, and hence random-walk inducing reversals of direction, a small, inefficient step size (η) is needed. But a higher rejection rate would be tolerable if rejections cluster in time, producing long runs of no rejections. For example, rejection-free runs of 20, 0, 20, 0,... are better than rejection-free runs of 10, 10, 10, 10,..., since N of the former runs will move via a random walk a distance proportional to 20(N/2) 1/2 ≈ 14 N 1/2, whereas N of the latter runs will move only a distance proportional to 10 N 1/2. I tried sampling with the standard persistent Langevin method and with persistent Langeving updates with non-reversible updating of u, on a multivariate Gaussian distribution consisting of 16 independent pairs of variables having variances of 1 and correlations of 0.99. (Ie, a 32-dimensional Gaussian with block-diagonal covariance matrix). The plots below show the . The values for η used were the stepsizes shown scaled down by 32 1/6. The curves in different colours are for α ∈ {0.1 η, ..., 0.8 η}. The autocorrelation times shown are for groups of 31 iterations. The non-reversible method is 1.62 times better estimating the mean log probability. The right plots show that rejections are indeed clustered when non-reversible updates are used, which reduces random walks, explaining the improvement. I tried using the persistent Langevin method with non-reversible updates for u to sample the posterior distribution of a Bayesian neural network model. Such models (Neal 1995) typically have hyperparameters controlling the variance of groups of weights in the network. It is convenient to use Gibbs sampling updates for these hyperparameters, alternating such updates with HMC updates for the network weights. However, when long trajectories are used for HMC, as is desirable to reduce random-walk behaviour, the Gibbs sampling updates for hyperparameters are done infrequently. Using persistent Langevin updates for weights would allow hyperparameters to be updated more frequently, perhaps speeding overall convergence. We hope to make this work better by non-reversibly updating u. I tested this approach on a binary classification problem, with 5 inputs, and 300 training cases. A network with one hidden layer of 12 tanh hidden units was used. Only two of the inputs for this problem were relevant, with two more being slightly noisy versions of the two relevant inputs, and one input being independent noise. Five separate hyperparameters controlled the variance of weights out of each inputs. For the best-tuned persistent Langevin method non-reversibly updating u, the average autocorrelation time for the four plausibly-relevant input hyperparameters was 1.25 times smaller than for the best-tuned HMC method. This is an encouraging preliminary .

Represent the short summary for finding the corresponding scientific article (long)

A non-reversible way of making accept/reject decisions can be beneficial

Represent the TLDR for finding the corresponding scientific article.

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: represent this

Question: We introduce ES-MAML, a new framework for solving the model agnostic meta learning (MAML) problem based on Evolution Strategies (ES). Existing algorithms for MAML are based on policy gradients, and incur significant difficulties when attempting to estimate second derivatives using backpropagation on stochastic policies. We show how ES can be applied to MAML to obtain an algorithm which avoids the problem of estimating second derivatives, and is also conceptually simple and easy to implement. Moreover, ES-MAML can handle new types of nonsmooth adaptation operators, and other techniques for improving performance and estimation of ES methods become applicable. We show empirically that ES-MAML is competitive with existing methods and often yields better adaptation with fewer queries. Meta-learning is a paradigm in machine learning that aims to develop models and training algorithms which can quickly adapt to new tasks and data. Our focus in this paper is on meta-learning in reinforcement learning (RL), where data efficiency is of paramount importance because gathering new samples often requires costly simulations or interactions with the real world. A popular technique for RL meta-learning is Model Agnostic Meta Learning (MAML) (;, a model for training an agent which can quickly adapt to new and unknown tasks by performing one (or a few) gradient updates in the new environment. We provide a formal description of MAML in Section 2. MAML has proven to be successful for many applications. However, implementing and running MAML continues to be challenging. One major complication is that the standard version of MAML requires estimating second derivatives of the RL reward function, which is difficult when using backpropagation on stochastic policies; indeed, the original implementation of MAML did so incorrectly, which spurred the development of unbiased higher-order estimators (DiCE, ) and further analysis of the credit assignment mechanism in MAML . Another challenge arises from the high variance inherent in policy gradient methods, which can be ameliorated through control variates such as in T-MAML , through careful adaptive hyperparameter tuning and learning rate annealing . To avoid these issues, we propose an alternative approach to MAML based on Evolution Strategies (ES), as opposed to the policy gradient underlying previous MAML algorithms. We provide a detailed discussion of ES in Section 3.1. ES has several advantages: 1. Our zero-order formulation of ES-MAML (Section 3.2, Algorithm 3) does not require estimating any second derivatives. This dodges the many issues caused by estimating second derivatives with backpropagation on stochastic policies (see Section 2 for details). 2. ES is conceptually much simpler than policy gradients, which also translates to ease of implementation. It does not use backpropagation, so it can be run on CPUs only. 3. ES is highly flexible with different adaptation operators (Section 3.3). 4. ES allows us to use deterministic policies, which can be safer when doing adaptation (Section 4.3). ES is also capable of learning linear and other compact policies (Section 4.2). On the point, a feature of ES algorithms is that exploration takes place in the parameter space. Whereas policy gradient methods are primarily motivated by interactions with the environment through randomized actions, ES is driven by optimization in high-dimensional parameter spaces with an expensive querying model. In the context of MAML, the notions of "exploration" and "task identification" have thus been shifted to the parameter space instead of the action space. This distinction plays a key role in the stability of the algorithm. One immediate implication is that we can use deterministic policies, unlike policy gradients which is based on stochastic policies. Another difference is that ES uses only the total reward and not the individual state-action pairs within each episode. While this may appear to be a weakness, since less information is being used, we find in practice that it seems to lead to more stable training profiles. This paper is organized as follows. In Section 2, we give a formal definition of MAML, and discuss related works. In Section 3, we introduce Evolutionary Strategies and show how ES can be applied to create a new framework for MAML. In Section 4, we present numerical experiments, highlighting the topics of exploration (Section 4.1), the utility of compact architectures (Section 4.2), the stability of deterministic policies (Section 4.3), and comparisons against existing MAML algorithms in the few-shot regime (Section 4.4). Additional material can be found in the Appendix. We first discuss the original formulation of MAML . Let T be a set of reinforcement learning tasks with common state and action spaces S, A, and P(T) a distribution over T. In the standard MAML setting, each task T i ∈ T has an associated Markov Decision Process (MDP) with transition distribution q i (s t+1 |s t, a t), an episode length H, and a reward function R Ti which maps a trajectory τ = (s 0, a 1, ..., a H−1, s H) to the total reward R(τ). A stochastic policy is a function π: S → P(A) which maps states to probability distributions over the action space. A deterministic policy is a function π: S → A. Policies are typically encoded by a neural network with parameters θ, and we often refer to the policy π θ simply by θ. The MAML problem is to find the so-called MAML point (called also a meta-policy), which is a policy θ * that can be'adapted' quickly to solve an unknown task T ∈ T by taking a (few) 1 policy gradient steps with respect to T. The optimization problem to be solved in training (in its one-shot version) is thus of the form: where: is called the adapted policy for a step size α > 0 and P T (·|η) is a distribution over trajectories given task T ∈ T and conditioned on the policy parameterized by η. Standard MAML approaches are based on the following expression for the gradient of the MAML objective function to conduct training: We collectively refer to algorithms based on computing using policy gradients as PG-MAML. Since the adaptation operator U (θ, T) contains the policy gradient is second-order in θ: Correctly computing the gradient with the term using automatic differentiation is known to be tricky. Multiple authors (; ;) have pointed out that the original implementation of MAML incorrectly estimates the term, which inadvertently causes the training to lose'pre-adaptation credit assignment'. Moreover, even when correctly implemented, the variance when estimating can be extremely high, which impedes training. To improve on this, extensions to the original MAML include ProMP , which introduces a new low-variance curvature (LVC) estimator for the Hessian, and T-MAML , which adds control variates to reduce the variance of the unbiased DiCE estimator . However, these are not without their drawbacks: the proposed solutions are complicated, the variance of the Hessian estimate remains problematic, and LVC introduces unknown estimator bias. Another issue that arises in PG-MAML is that policies are necessarily stochastic. However, randomized actions can lead to risky exploration behavior when computing the adaptation, especially for robotics applications where the collection of tasks may involve differing system dynamics as opposed to only differing rewards . We explore this further in Section 4.3. These issues: the difficulty of estimating the Hessian term, the typically high variance of ∇ θ J(θ) for policy gradient algorithms in general, and the unsuitability of stochastic policies in some domains, lead us to the proposed method ES-MAML in Section 3. Aside from policy gradients, there have also been biologically-inspired algorithms for MAML, based on concepts such as the Baldwin effect . However, we note that despite the similar naming, methods such as'Evolvability ES' bear little resemblance to our proposed ES-MAML. The problem solved by our algorithm is the standard MAML, whereas aims to maximize loosely related notions of the diversity of behavioral characteristics. Moreover, ES-MAML and its extensions we consider are all derived notions such as smoothings and approximations, with rigorous mathematical definitions as stated below. Formulating MAML with ES allows us to employ numerous techniques originally developed for enhancing ES, to MAML. We aim to improve both phases of MAML algorithm: the meta-learning training algorithm, and the efficiency of the adaptation operator. Evolution Strategies (ES) (; 2014), which recently became popular for RL , rely on optimizing the smoothing of the blackbox function f: R d → R, which takes as input parameters θ ∈ R d of the policy and outputs total discounted (expected) reward obtained by an agent applying that policy in the given environment. Instead of optimizing the function f directly, we optimize a smoothed objective. We define the Gaussian smoothing of The gradient of this smoothed objective, sometimes called an ES-gradient, is given as (see: ): Note that the gradient can be approximated via Monte Carlo (MC) samples: In ES literature the above algorithm is often modified by adding control variates to equation 4 to obtain other unbiased estimators with reduced variance. The forward finite difference (Forward-FD) estimator is given by subtracting the current policy value f (θ), yielding The antithetic estimator is given by the symmetric difference Notice that the variance of the Forward-FD and antithetic estimators is translation-invariant with respect to f. In practice, the Forward-FD or antithetic estimator is usually preferred over the basic version expressed in equation 4. In the next sections we will refer to Algorithm 1 for computing the gradient though we emphasize that there are several other recently developed variants of computing ES-gradients as well as applying them for optimization. We describe some of these variants in Section 3.3 and appendix A.3. A key feature of ES-MAML is that we can directly make use of new enhancements of ES. To formulate MAML in the ES framework, we take a more abstract viewpoint. For each task T ∈ T, let f T (θ) be the (expected) cumulative reward of the policy θ. We treat f T as a blackbox, and make no assumptions on its structure (so the task need not even be MDP, and f T may be nonsmooth). The MAML problem is then max As argued in (see also Section 2), a major challenge for policy gradient MAML is estimating the Hessian, which is both conceptually subtle and difficult to correctly implement using automatic differentiation. The algorithm we propose obviates the need to calculate any second derivatives, and thus avoids this issue. Suppose that we can evaluate (or approximate) f T (θ) and U (θ, T), but f T and U (·, T) may be nonsmooth or their gradients may be intractable. We consider the Gaussian smoothing J σ of the MAML reward, and optimize J σ using ES methods. The gradient ∇ J σ (θ) is given by and can be estimated by jointly sampling over (T, g) and evaluating f T (U (θ + σg, T)). This algorithm is specified in Algorithm 2 box, and we refer to it as (zero-order) ES-MAML. Data: initial policy θ 0, meta step size β 1 for t = 0, 1,... do 2 Sample n tasks T 1,..., T n and iid vectors g 1,..., g n ∼ N (0, I); Algorithm 2: Zero-Order ES-MAML (general adaptation operator U (·, T)) Data: initial policy θ 0, adaptation step size α, meta step size β, number of queries Sample n tasks T 1,..., T n and iid vectors g 1,..., g n ∼ N (0, I); Algorithm 3: Zero-Order ES-MAML with ESGradient Adaptation The standard adaptation operator U (·, T) is the one-step task gradient. Since f T is permitted to be nonsmooth in our setting, we use the adaptation operator U (θ, T) = θ + α∇ f T σ (θ) acting on its smoothing. Expanding the definition of J σ, the gradient of the smoothed MAML is then given by This leads to the algorithm that we specify in Algorithm 3, where the adaptation operator U (·, T) is itself estimated using the ES gradient in the inner loop. We can also derive an algorithm analogous to PG-MAML by applying a first-order method to the MAML reward E T ∼P(T) f T (θ + α∇ f T (θ)) directly, without smoothing. The gradient is given by which corresponds to equation in when expressed in terms of policy gradients. Every term in this expression has a simple Monte Carlo estimator (see Algorithm 4 in the appendix for the MC Hessian estimator). We discuss this algorithm in greater detail in Appendix A.1. This formulation can be viewed as the "MAML of the smoothing", compared to the "smoothing of the MAML" which is the basis for Algorithm 3. It is the additional smoothing present in equation 6 which eliminates the gradient of U (·, T) (and hence, the Hessian of f T). Just as with the Hessian estimation in the original PG-MAML, we find empirically that the MC estimator of the Hessian (Algorithm 4) has high variance, making it often harmful in training. We present some comparisons between Algorithm 3 and Algorithm 5, with and without the Hessian term, in Appendix A.1.2. Note that when U (·, T) is estimated, such as in Algorithm 3, the ing estimator for ∇ J σ will in general be biased. This is similar to the estimator bias which occurs in PG-MAML because we do not have access to the true adapted trajectory distribution. We discuss this further in Appendix A.2. Algorithm 2 allows for great flexibility in choosing new adaptation operators. The simplest extension is to modify the ES gradient step: we can draw on general techniques for improving the ES gradient estimator, some of which are described in Appendix A.3. Some other methods are explored below. Instead of using i.i.d Gaussian vectors to estimate the ES gradient in U (·, T), we consider samples constructed according to Determinantal Point Processes (DPP). DPP sampling is a method of selecting a subset of samples so as to maximize the'diversity' of the subset. It has been applied to ES to select perturbations g i so that the gradient estimator has lower variance (a). The sampling matrix determining DPP sampling can also be data-dependent and use information from the meta-training stage to construct a learned kernel with better properties for the adaptation phase. In the experimental section we show that DPP-ES can help in improving adaptation in MAML. Nondifferentiable operators U (·, T) can be also used in Algorithm 2. One particularly interesting example is the local search operator given by U (θ, T) = argmax{f T (θ): θ − θ ≤ R}, where R > 0 is the search radius. That is, U (θ, T) selects the best policy for task T which is in a'neighborhood' of θ. For simplicity, we took the search neighborhood to be the ball B(θ, R) here, but we may also use more general neighborhoods of θ. In general, exactly solving for the maximizer of f T over B(θ, R) is intractable, but local search can often be well approximated by a hill climbing algorithm. Hill climbing creates a population of candidate policies by perturbing the best observed policy (which is initialized to θ), evaluates the reward f T for each candidate, and then updates the best observed policy. This is repeated for several iterations. A key property of this search method is that the progress is monotonic, so the reward of the returned policy U (θ, T) will always improve over θ. This does not hold for the stochastic gradient operator, and appears to be beneficial on some difficult problems (see Section 4.1). It has been claimed that hill climbing and other genetic algorithms are competitive with gradient-based methods for solving difficult RL tasks ). Another stochastic algorithm approximating local search is CMA-ES (; ;), which performs more sophisticated search by adapting the covariance matrix of the perturbations. The performance of MAML algorithms can be evaluated in several ways. One important measure is the performance of the final meta-policy: whether the algorithm can consistently produce metapolicies with better adaptation. In the RL setting, the adaptation of the meta-policy is also a function of the number K of queries used: that is, the number of rollouts used by the adaptation operator U (·, T). The meta-learning goal of data efficiency corresponds to adapting with low K. The speed of the meta-training is also important, and can be measured in several ways: the number of metapolicy updates, wall-clock time, and the number of rollouts used for meta-training. In this section, we present experiments which evaluate various aspects of ES-MAML and PG-MAML in terms of data efficiency (K) and meta-training time. Further details of the environments and hyperparameters are given in Appendix A.7. In the RL setting, the amount of information used drastically decreases if ES methods are applied in comparison to the PG setting. To be precise, ES uses only the cumulative reward over an episode, whereas policy gradients use every state-action pair. Intuitively, we may thus expect that ES should have worse sampling complexity because it uses less information for the same number of rollouts. However, it seems that in practice ES often matches or even exceeds policy gradients approaches . Several explanations have been proposed: In the PG case, especially with algorithms such as PPO, the network must optimize multiple additional surrogate objectives such as entropy bonuses and value functions as well as hyperparameters such as the TDstep number. Furthermore, it has been argued that ES is more robust against delayed rewards, action infrequency, and long time horizons . These advantages of ES in traditional RL also transfer to MAML, as we show empirically in this section. ES may lead to additional advantages (even if the numbers of rollouts needed in training is comparable with PG ones) in terms of wall-clock time, because it does not require backpropagation, and can be parallelized over CPUs. In this section, we present two experiments on environments with very sparse rewards where the meta-policy must exhibit exploratory behavior to determine the correct adaptation. The four corners benchmark was introduced in to demonstrate the weaknesses of exploration in PG-MAML. An agent on a 2D square receives reward for moving towards a selected corner of the square, but only observes rewards once it is sufficiently close to the target corner, making the reward sparse. An effective exploration strategy for this set of tasks is for the meta-policy θ * to travel in circular trajectories to observe which corner produces rewards; however, for a single policy to produce this exploration behavior is difficult. In Figure 1, we demonstrate the behavior of ES-MAML on the four corners problem. When K = 20, the same number of rollouts for adaptation as used in , the basic version of Algorithm 3 is able to correctly explore and adapt to the task by finding the target corner. Moreover, it does not require any modifications to encourage exploration, unlike PG-MAML. We further used K = 10, 5, which caused the performance to drop. For better performance in this low-information environment, we experimented with two different adaptation operators U (·, T) in Algorithm 2, which are HC (hill climbing) and DPP-ES. The standard ES gradient is denoted MC. Furthermore, ES-MAML is not limited to "single goal" exploration. We created a more difficult task, six circles, where the agent continuously accrues negative rewards until it reaches six target points to "deactivate" them. Solving this task requires the agent to explore in circular trajectories, similar to the trajectory used by PG-MAML on the four corners task. We visualize the behavior in Figure 2. Observe that ES-MAML with the HC operator is able to develop a strategy to explore the target locations. From Figure 1, we observed that both operators DPP-ES and HC were able to improve exploration performance. We also created a modified task by heavily penalizing incorrect goals, which caused performance to dramatically drop for MC and DPP-ES. This is due to the variance from the MC-gradient, which may in a adapted policy that accidentally produces large negative rewards or become stuck in local-optima (i.e. refuse to explore due to negative rewards). This is also fixed by the HC adaptation, which enforces non-decreasing rewards during adaptation, allowing the ES-MAML to progress. Additional examples on the classic Navigation-2D task are presented in Appendix A.4, highlighting the differences in exploration behavior between PG-MAML and ES-MAML. One of the main benefits of ES is due to its ability to train compact linear policies, which can outperform hidden-layer policies. We demonstrate this on several benchmark MAML problems in the HalfCheetah and Ant environments in Figure 3. In contrast, observed that PG-MAML empirically and theoretically suggested that training with more deeper layers under SGD increases performance. We demonstrate that on the Forward-Backward and Goal-Velocity MAML benchmarks, ES-MAML is consistently able to train successful linear policies faster than deep networks. We also show that, for the Forward-Backward Ant problem, ES-MAML with the new HC operator is the most performant. Using more compact policies also directly speeds up ES-MAML, since fewer perturbations are needed for gradient estimation. We find that deterministic policies often produce more stable behaviors than the stochastic ones that are required for PG, where randomized actions in unstable environments can lead to catastrophic outcomes. In PG, this is often mitigated by reducing the entropy bonus, but this has an undesirable side effect of reducing exploration. In contrast, ES-MAML explores in parameter space, which mitigates this issue. To demonstrate this, we use the "Biased-Sensor CartPole" environment from . This environment has unstable dynamics and sparse rewards, so it requires exploration but is also risky. We see in Figure 4 that ES-MAML is able to stably maintain the maximum reward. We also include in Figure 4 from two other environments, Swimmer and Walker2d, for which it is known that PG is surprisingly unstable, and ES yields better training . Notice that we again find linear policies (L) outperforming policies with one (H) or two (HH) hidden layers. For real-world applications, we may be constrained to use fewer queries K than has typically been demonstrated in previous MAML works. Hence, it is of interest to compare how ES-MAML compares to PG-MAML for adapting with very low K. One possible concern is that low K might harm ES in particular because it uses only the cumulative rewards; if for example K = 5, then the ES adaptation gradient can make use of only 5 values. In comparison, PG-MAML uses K · H state-action pairs, so for K = 5, H = 200, PG-MAML still has 1000 pieces of information available. However, we find experimentally that the standard ES-MAML (Algorithm 3) remains competitive with PG-MAML even in the low-K setting. In Figure 5, we compare ES-MAML and PG-MAML on the Forward-Backward and Goal-Velocity tasks across four environments (HalfCheetah, Swimmer, Walker2d, Ant) and two model architectures. While PG-MAML can generally outperform ES-MAML on the Goal-Velocity task, ES-MAML is similar or better on the Forward-Backward task. Moreover, we observed that for low K, PG-MAML can be highly unstable (note the wide error bars), with some trajectories failing catastrophically, whereas ES-MAML is relatively stable. This is an important consideration in real applications, where the risk of catastrophic failure is undesirable. We have presented a new framework for MAML based on ES algorithms. The ES-MAML approach avoids the problems of Hessian estimation which necessitated complicated alterations in PG-MAML and is straightforward to implement. ES-MAML is flexible in the choice of adaptation operators, and can be augmented with general improvements to ES, along with more exotic adaptation operators. In particular, ES-MAML can be paired with nonsmooth adaptation operators such as hill climbing, which we found empirically to yield better exploratory behavior and better performance on sparse-reward environments. ES-MAML performs well with linear or compact deterministic policies, which is an advantage when adapting if the state dynamics are possibly unstable. but slightly worse than the full PG-MAML, and does not report comparisons with and without the Hessian on RL MAML. argue for the importance of the second-order terms in proper credit assignment, but use heavily modified estimators (LVC, control variates; see Section 2) in their experiments, so the performance is not directly comparable to the'naive' estimator in Algorithm 4. Our interpretation is that Algorithm 4 has high variance, making the Hessian estimates inaccurate, which can slow training on relatively'easier' tasks like ForwardBackward walking but possibly increase the exploration on four corners. We also compare FO-NoHessian against Algorithm 3 on Forward-Backward HalfCheetah and Ant in Figure A2. In this experiment, the two methods ran on servers with different number of workers available, so we measure the score by the total number of rollouts. We found that FO-NoHessian was slightly faster than Algorithm 3 when measured by rollouts on Ant, but FO-NoHessian had notably poor performance when the number of queries was low (K = 5) on HalfCheetah, and failed to reach similar scores as the others even after running for many more rollouts. Since the adapted policy U (θ, T) generally cannot be evaluated exactly, we cannot easily obtain unbiased estimates of f T (U (θ, T)). This problem arises for both PG-MAML and ES-MAML. We consider PG-MAML first as an example. In PG-MAML, the adaptation operator is U (θ, In general, we can only obtain an estimate of ∇ θ E τ ∼P T (τ |θ) [R(τ)] and not its exact value. However, the MAML gradient is given by which requires exact sampling from the adapted trajectories τ ∼ P T (τ |U (θ, T)). Since this is a nonlinear function of U (θ, T), we cannot obtain unbiased estimates of ∇J(θ) by sampling τ generated by an estimate of U (θ, T). In the case of ES-MAML, the adaptation operator is U (θ, We may question whether using an unbiased estimator of f T (U (θ, T)) is likely to improve performance. One natural strategy is to reformulate the objective function so as to make the desired estimator unbiased. This happens to be the case for the algorithm E-MAML, which treats the adaptation operator as an explicit function of K sampled trajectories and "moves the expectation outside". That is, we now have an adaptation operator U (θ, T ; τ 1, . . ., τ K), and the objective function becomes θ + α∇ f T σ (θ), we replace the estimation of U by ESGRAD on line 4 of Algorithm 3 with an improved estimator of ∇ f T σ (θ), which even may depend on data collected during the meta-training stage. Many techniques exist for reducing the variance of the estimator such as Quasi Monte Carlo sampling . Aside from variance reduction, there are also methods with special properties. A.3.1 ACTIVE SUBSPACES Active Subspaces is a method for finding a low-dimensional subspace where the contribution of the gradient is maximized. Conceptually, the goal is to find and update on-the-fly a low-rank subspace L so that the projection ∇f. This should be done in such a way that ∇f T (θ) does not need to be computed explicitly. Optimizing in lower-dimensional subspaces might be computationally more efficient and can be thought of as an example of guided ES methods, where the algorithm is guided how to explore space in the anisotropic way, leveraging its knowledge about function optimization landscape that it gained in the previous steps of optimization. In the context of RL, the active subspace method ASEBO (b) was successfully applied to speed up policy training algorithms. This strategy can be made data-dependent also in the MAML context, by learning an optimal subspace using data from the meta-training stage, and sampling from that subspace in the adaptation step. Regression-Based Optimization (RBO) is an alternative method of gradient estimation. From Taylor series expansion we have. By taking multiple finite difference expressions f (θ + d) − f (θ) for different d, we can recover the gradient by solving a regularized regression problem. The regularization has an additional advantage -it was shown that the gradient can be recovered even if a substantial fraction of the rewards f (θ + d) are corrupted (c). Strictly speaking, this is not based on the Gaussian smoothing as in ES, but is another method for estimating gradients using only zero-th order evaluations. We present a preliminary experiment with RBO and ASEBO gradient adaptation in Figure A3. To be precise, the algorithms used are identical to Algorithm 3 except that in line 4, d On the left plot, we test for noise robustness on the Forward-Backward Swimmer MAML task, comparing standard ES-MAML (Algorithm 3) to RBO-MAML. To simulate noisy data, we randomly corrupt 25% of the queries f T (θ + σg) used to estimate the adaptation operator U (θ, T) with an enormous additive noise. This is the same type of corruption used in (c). Interestingly, RBO does not appear to be more robust against noise than the standard MC estimator, which suggests that the original ES-MAML has some inherent robustness to noise. On the right plot, we compare ASEBO-MAML to ES-MAML on the Goal-Velocity HalfCheetah task in the low-K setting. We found that when measured in iterations, ASEBO-MAML outperforms ES-MAML. However, ASEBO requires additional linear algebra operations and thus uses significantly more wall-clock time (not shown in plot) per iteration, so if measured by real time, then ES-MAML was more effective. Navigation-2D is a classic environment where the agent must explore to adapt to the task. The agent is represented by a point on a 2D square, and at each time step, receives reward equal to its distance from a given target point on the square. Note that unlike the four corners and six circles tasks, the reward for Navigation-2D is dense. We visualize the differing exploration strategies learned by PG-MAML and ES-MAML in Figure A4. Notice that PG-MAML makes many tiny movements in multiple directions to'triangulate' the target location using the differences in reward for different state-action pairs. On the other hand, ES-MAML learns a meta-policy such that each perturbation of the meta-policy causes the agent to move in a different direction (represented by red paths), so it can determine the target location from the total rewards of each path. In Figure A5, we compare ES-MAML and PG-MAML on the Forward-Backward and Goal-Velocity tasks for HalfCheetah, Swimmer, Walker2d, and Ant, using the same values of K that were used in the original experiments of . Figure A5: Comparisons between ES-MAML and PG-MAML using the queries K from . A.6 REGRESSION AND SUPERVISED LEARNING MAML has also been applied to supervised learning. We demonstrate ES-MAML on sine regression , where the task is to fit a sine curve f with unknown amplitude and phase given a set of K pairs (x i, f (x i)). The meta-policy must be able to learn that all of tasks have a common periodic nature, so that it can correctly adapt to an unknown sine curve outside of the points x i. For regression, the loss is the mean-squared error (MSE) between the adapted policy π θ (x) and the true curve f (x). Given data samples {( 2 . Note that unlike in reinforcement learning, we can exactly compute ∇L(θ); for deep networks, this is by automatic differentiation. Thus, we opt to use Tensorflow to compute the adaptation operator U (θ, T) in Algorithm 3. This is in accordance with the general principle that when gradients are available, it is more efficient to use the gradient than to approximate it by a zero-order method . We show several in Figure A6. The adaptation step size is α = 0.01, which is the same as in . For comparison, reports that PG-MAML can obtain a loss of ≈ 0.5 after one adaptation step with K = 5, though it is not specified how many iterations the meta-policy was trained for. ES-MAML approaches the same level of performance, though the number of training iterations required is higher than for the RL tasks, and surprisingly high for what appears to be a simpler problem. This is likely again a reflection of the fact that for problems such as regression where the gradients are available, it is more efficient to use gradients. As an aside, this leads to a related question of the correct interpretation of the query number K in the supervised setting. There is a distinction between obtaining a data sample (x i, f (x i)), and doing a computation (such as a gradient) using that sample. If the main bottleneck is collecting the data {(x i, f (x i)}, then we may be satisfied with any algorithm that performs any number of operations on the data, as long as it uses only K samples. On the other hand, in the (on-policy) RL setting, samples cannot typically be're-used' to the same extent, because rollouts τ sampled with a given Figure A6: The MSE of the adapted policy, for varying number of gradient steps and query number K. Runs are averaged across 3 seeds. policy π θ follow an unknown distribution P(τ |θ) which reduces their usefulness away from θ. Thus, the corresponding notion to rollouts in the SL setting would be the number of backpropagations (for PG-MAML) or perturbations (for ES-MAML), but clearly these have different relative costs than doing simulations in RL. A.7 HYPERPARAMETERS AND SETUPS A.7.1 ENVIRONMENTS Unless otherwise explicitly stated, we default to K = 20 and horizon = 200 for all RL experiments. We also use the standard reward normalization in , and use a global state normalization (i.e. the same mean, standard deviation normalization values for MDP states are shared across workers). For the Ant environments (Goal-Position Ant, Forward-Backward Ant), there are significant differences in weighting on the auxiliary rewards such as control costs, contact costs, and survival rewards across different previous work (e.g. those costs are downweighted in whereas the coefficients are vanilla Gym weightings in ). These auxiliary rewards can lead to local minima, such as the agent staying stationary to collect the survival bonus which may be confused with movement progress when presenting a training curve. To make sure the agent is explicitly performing the required task, we opted to remove such costs in our work and only present the main goal-distance cost and forward-movement reward respectively. For the other environments, we used default weightings and rewards, since they do not change across previous works. Let N be the number of possible distinct tasks possible. We sample tasks without replacement, which is important if N 5, as each worker performs adaptations on all possible tasks. For standard ES-MAML (Algorithm 3), we used the following settings. For ES-MAML and PG-MAML, we took 3 seeded runs, using the default TRPO hyperparameters found in .

We provide a new framework for MAML in the ES/blackbox setting, and show that it allows deterministic and linear policies, better exploration, and non-differentiable adaptation operators.

Represent the short summary for finding the corresponding scientific article (long)