Question: Represent the article for finding a single-sentence TLDR

Question: The available resolution in our visual world is extremely high, if not infinite. Existing CNNs can be applied in a fully convolutional way to images of arbitrary resolution, but as the size of the input increases, they can not capture contextual information. In addition, computational requirements scale linearly to the number of input pixels, and resources are allocated uniformly across the input, no matter how informative different image regions are. We attempt to address these problems by proposing a novel architecture that traverses an image pyramid in a top-down fashion, while it uses a hard attention mechanism to selectively process only the most informative image parts. We conduct experiments on MNIST and ImageNet datasets, and we show that our models can significantly outperform fully convolutional counterparts, when the resolution of the input is that big that the receptive field of the baselines can not adequately cover the objects of interest. Gains in performance come for less FLOPs, because of the selective processing that we follow. Furthermore, our attention mechanism makes our predictions more interpretable, and creates a trade-off between accuracy and complexity that can be tuned both during training and testing time. Our visual world is very rich, and there is information of interest in an almost infinite number of different scales. As a , we would like our models to be able to process images of arbitrary resolution, in order to capture visual information with arbitrary level of detail. This is possible with existing CNN architectures, since we can use fully convolutional processing , coupled with global pooling. However, global pooling ignores the spatial configuration of feature maps, and the output essentially becomes a bag of features 1. To demonstrate why this an important problem, in Figure 1 (a) and (b) we provide an example of a simple CNN that is processing an image in two different resolutions. In (a) we see that the receptive field of neurons from the second layer suffices to cover half of the kid's body, while in (b) the receptive field of the same neurons cover area that corresponds to the size of a foot. This shows that as the input size increases, the final representation becomes a bag of increasingly more local features, leading to the absence of coarselevel information, and potentially harming performance. We call this phenomenon the receptive field problem of fully convolutional processing. An additional problem is that computational resources are allocated uniformly to all image regions, no matter how important they are for the task at hand. For example, in Figure 1 (b), the same amount of computation is dedicated to process both the left half of the image that contains the kid, and the right half that is merely . We also have to consider that computational complexity scales linearly with the number of input pixels, and as a , the bigger the size of the input, the more resources are wasted on processing uninformative regions. We attempt to resolve the aforementioned problems by proposing a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way. The receptive field problem of fully convolutional processing. A simple CNN consisted of 2 convolutional layers (colored green), followed by a global pooling layer (colored red), processes an image in two different resolutions. The shaded regions indicate the receptive fields of neurons from different layers. As the resolution of the input increases, the final latent representation becomes a bag of increasingly more local features, lacking coarse information. (c) A sketch of our proposed architecture. The arrows on the left side of the image demonstrate how we focus on image sub-regions in our top-down traversal, while the arrows on the right show how we combine the extracted features in a bottom-up fashion. In Figure 1 (c) we provide a simplified sketch of our approach. We start at level 1, where we process the input image in low resolution, to get a coarse description of its content. The extracted features (red cube) are used to select out of a predefined grid, the image regions that are worth processing in higher resolution. This process constitutes a hard attention mechanism, and the arrows on the left side of the image show how we extend processing to 2 additional levels. All extracted features are combined together as denoted by the arrows on the right, to create the final image representation that is used for classification (blue cube). We evaluate our model on synthetic variations of MNIST and on ImageNet , while we compare it against fully convolutional baselines. We show that when the resolution of the input is that big, that the receptive field of the baseline 2 covers a relatively small portion of the object of interest, our network performs significantly better. We attribute this behavior to the ability of our model to capture both contextual and local information by extracting features from different pyramid levels, while the baselines suffer from the receptive field problem. Gains in accuracy are achieved for less floating point operations (FLOPs) compared to the baselines, due to the attention mechanism that we use. If we increase the number of attended image locations, computational requirements increase, but the probability of making a correct prediction is expected to increase as well. This is a trade-off between accuracy and computational complexity, that can be tuned during training through regularization, and during testing by stopping processing on early levels. Finally, by inspecting attended regions, we are able to get insights about the image parts that our networks value the most, and to interpret the causes of missclassifications. Attention. Attention has been used very successfully in various problems (; ; ; ;). Most similar to our work, are models that use recurrent neural networks to adaptively attend to a sequence of image regions, called glimpses (; ; ; ;). There are notable technical differences between such models and our approach. However, the difference that we would like to emphasize, is that we model the image content as a hierarchical structure, and we implicitly create a parsing tree , where each node corresponds to an attended location, and edges connect image regions with sub-regions (an example is provided in Appendix A.1). If we decide to store and reuse information, building such a tree structure offers a number of potential benefits, e.g. efficient indexing. We consider this an important direction to explore, but it is beyond the scope of the current paper. Multi-scale representations. We identify four broad categories of multi-scale processing methods. Image pyramid methods extract multi-scale features by processing multi-scale inputs (; ;). Our approach belongs to this category. Encoding schemes take advantage of the inherently hierarchical nature of deep neural nets, and reuse features from different layers, since they contain information of different scale (; ;). Encoding-Decoding schemes follow up the feed-forward processing (encoding) with a decoder, that gradually recovers the spatial resolution of early feature maps, by combining coarse with fine features . Spatial modules are incorporated into the feed forward processing, to alter the feature extraction between layers (; ;). Computational efficiency. We separate existing methods on adjusting the computational cost of deep neural networks, into four categories. ). This is the strategy that we follow in our architecture. We present our architecture by walking through the example in Figure 2, where we process an image with original resolution of 128 × 128 px (1 in the top left corner). In the fist level, we downscale the image to 32 × 32 px and pass it through the feature extraction module, in order to produce a feature vector V 1 that contains a coarse description of the original image. The feature extraction module is a CNN that accepts inputs in a fixed resolution, that we call base resolution. We can provide V 1 as direct input to the classification module in order to get a rapid prediction. If we end processing here, our model is equivalent to a standard CNN operating on 32 × 32 px inputs. However, since the original resolution of the image is 128 × 128 px, we can take advantage of the additional information by moving to the second processing level. In the second level, we feed the last feature map, F 1, of the feature extraction module, to the location module. The location module considers a number of candidate locations within the image described by F 1, and predicts how important it is to process in higher detail each one of them. In this particular example, the candidate locations form a 2 × 2 regular grid, and the location module yields 4 probabilities, that we use as parameters of 4 Bernoulli distributions in order to sample a hard attention mask. Based on this mask, we crop the corresponding regions, and we pass them through the feature extraction module, creating V 21 and V 23. If we want to stop at this processing level, we can directly pass V 21 and V 23 through the aggregation module (skipping the merging module 4). The aggregation module combines the features from individual image regions into a single feature vector V, that describes the original image solely based on fine information. This means that V agg 1 is complementary to V 1, and both vectors are combined by the merging module, which integrates fine information (V agg 1) with its context (V 1), creating a single comprehensive representation V 1. Then, V 1 can be used for the final prediction. We can extend our processing to a third level, where F 21 and F 23 are fed to the location module to create two binary masks. No locations are selected from the image patch described by V 23, and the aggregation module only creates V agg 21. Then, we start moving upwards for the final prediction. In Appendix A.2 we provide additional details about the modules of our architecture. In Appendix A.3 we express the feature extraction process with a single recursive equation. Our model is not end-to-end differentiable, because of the Bernoulli sampling involved in the location selection process. To overcome this problem, we use a variant of REINFORCE : where N · M is the number of images we use for each update, x i is the ith image, y i is its label, and w are the parameters of our model. p(l i |x i, w) is the probability that the sequence of locations l i is attended for image x i, and p(y i |l i, x i, w) is the probability of predicting the correct label after attending l i. The size of our original batch B is N, but in the derivation of we approximate with a Monte Carlo estimator of M samples, the expectation for each image x i in B. Based on this, for simplicity we just consider that our batch has size N · M. b is a baseline that we use to reduce the variance of our estimators, and λ f is a weighting hyperparameter. The first term of L F allows us to update the parameters in order to maximize the probability of each correct label. The second term allows us to update the location selection process, according to the utility of attended locations for the prediction of the correct labels. In Appendix A.4.1 we provide the exact derivation of learning rule. We experimentally identified two problems related to. First, our model tends to attend all the available locations, maximizing the computational cost. To regulate this, we add the following term: where p l i t approximates the expected number of attended locations for image x i base on l i, and t calculates the average expected number of attended locations per image in our augmented batch of N M images. The purpose of R t is to make the average number of attended locations per image equal to c t, which is a hyperparameter selected according to our computational cost requirements. λ t is simply a weighting hyperparameter. The second problem we identified while using learning rule, is that the learned attention policy may not be diverse enough. In order to encourages exploration, we add the following term: where p k is the average probability of attending the kth out of the g candidate locations, that the location module considers every time it is applied during the processing of our N M images. R r encourages the location module to attend with the same average probability c r, locations that are placed at different regions inside the sampling grid. λ r is a weighting hyperparameter. In Appendix A.4.2 we provide additional details about terms and. Our final learning rule is the following: Gradual Learning. The quality of the updates we make by using, depends on the quality of the Monte Carlo estimators. When we increase the number of processing levels that our model can go through, the number of location sequences that can be attended increases exponentially. Based on this, if we allow our model to go through multiple processing levels, and we use a small number of samples in order to have a moderate cost per update, we expect our estimators to have high variance. To avoid this, we separate training into stages, where in the first stage we allow only 2 processing levels, and at every subsequent stage an additional processing level is allowed. This way, we expect the location module to gradually learn which image parts are the most informative, narrowing down the location sequences that have a considerable probability to be attended at each training stage, and allowing a small number of samples to provide acceptable estimators. We call this training strategy gradual learning, and the learning rule that we use at each stage s, is the following: where L s r is equivalent to, with s superscripts indicating that the hyperparameters of each term can be adjusted at each training stage. The maximum number of possible training stages depends on the resolution of the training images. Multi-Level Learning. By following the gradual learning paradigm, a typical behavior that we observe is the following. After the first stage of training, our model is able to classify images with satisfactory accuracy by going through 2 processing levels. After the second stage of training, our model can go through 3 processing levels, and as we expect, accuracy increases since finer information can be incorporated. However, if we force our model to stop processing after 2 levels, the obtained accuracy is significantly lower than the one we were able to achieve after the first stage of training. This is a behavior that we observe whenever we finish a new training stage, and it is an important problem, because we would like to have a flexible model that can make accurate predictions after each processing level. To achieve this, we introduce the following learning rule: where L z r is learning rule with z + 1 processing levels allowed for our model. Each λ s z is a hyperparameter that specifies the relative importance of making accurate predictions after processing level z + 1, while we are at training stage s. We experiment with two datasets, MNIST and ImageNet . MNIST is a small dataset that we can easily modify in order to test different aspects of our models' Figure 3: Example images from our MNIST-based datasets, along with the attended locations of M 28 3 models. We blur the area outside the attended locations, because it is processed only in lower resolution during the first processing level. This way we aim to get a better understanding of what our models "see". behavior, e.g. the localization capabilities of our attention mechanism. ImageNet has over a million training images, and allows us to evaluate our model on a large scale. Data. MNIST is a dataset with images of handwritten digits that range from 0 to 9, leading to 10 different classes. All images are grayscale, and their size is 28 × 28 pixels. The dataset is split in a training set of 55, 000 images, a validation set of 5, 000 images, and a test set of 10, 000 images. We modify the MNIST images by placing each digit at a randomly selected location inside a black canvas of size 56 × 56. We refer to this MNIST variation as plain MNIST, and we further modify it to create noisy MNIST and textured MNIST. In the case of noisy MNIST, we add salt and pepper noise by corrupting 10% of the pixels in every image, with a 0.5 ratio between black and white noise. In the case of textured MNIST, we add textured that is randomly selected from a large image depicting grass. Example images are provided in the first column of Figure 3., each composed of a version M i of our architecture, and a corresponding fully convolutional baseline BL i for comparison. Ideally, we would like the feature extraction processes of the models that we compare to be consisted of the same layers, in order to eliminate important factors of performance variation like the type or the number of layers. To this end, in every pair of models, the baseline BL i is equivalent to the feature extraction module of M i followed by the classification module, with 2 fully connected layers in between. The 2 fully connected layers account for the aggregation and merging modules, which could be considered part of the feature extraction process in M i. We create 3 pairs of models because we want to study how our architecture performs relatively to fully convolutional models with different receptive fields. To achieve this, BL 1 has 1 convolutional layer and receptive field 3 × 3 px, BL 2 has 2 convolutional layers and receptive field 8 × 8 px, and BL 3 has 3 convolutional layers and receptive field 20 × 20 px. We would like to note that all models {M i} 3 i=1 have base resolution 14 × 14 px, and their location modules consider 9 candidate locations which belong to a 3 × 3 regular grid with 50% overlap. In Appendix A.5.1 we provide the exact architectures of all models. Training. We describe how we train one pair of models (M i, BL i) with one of the 3 MNIST-based datasets that we created. The same procedure is repeated for all datasets, and for every pair of models. The original resolution of our dataset is 56 × 56 px, and we rescale it to 2 additional resolutions that Figure 4: Experimental on plain, textured and noisy MNIST. The differences in accuracy between many models were very small, and as a , in the provided graphs we report the average of 20 different evaluations on the validation set, where each time we randomly change the positions of the digits inside the images. For textured and noisy MNIST, we randomly change the and the noise pattern of each image as well. are equal to 28 × 28 px, and 14 × 14 px (example images are provided in the first 3 columns of Figure 3). We split our training procedure in 3 sessions, where at each session we train our models with images of different resolution. In the first session we use images of size 14 × 14 px, and we refer to the ing models as BL 14 i and M 14 i. We note that since the resolution of the input is equal to the base resolution of M i, our model can go through just 1 processing level and the location module is not employed. In our second training session we use images of resolution 28 × 28 px, and the increased resolution of the input allows M i to use the location module and extend processing to 2 processing levels. Based on this, we are able to train M i multiple times by assigning different values to hyperparameter c t, ing to models that attend a different average number of locations per image, and as a , they have different average computational cost and accuracy. We refer to the models from this training session as BL 28 i and M, where n c is the average number of locations that M i attended on the validation set, while trained with c t = c. We also define M 28 i as the set of all M 28,nc i models we trained in this session. In the third training session we use images of resolution 56 × 56 px, and M i is able to go through 3 processing levels. Following our previous notation, we refer to the models of this session as BL. In Appendix A.5.2 we provide additional details about the training sessions, along with the exact hyperparameters that we used to obtain the that follow. Results. In the first row of Figure 4, we present performance graphs for all models on plain MNIST. We note that the annotations under all models indicate the average number of locations n c that we described in the training section. We start by examining Figure 4 (a), where, that demonstrate the interpretability of our models' decisions because of the employed attention mechanism. we depict the performance of models M 3 and BL 3 on images of different resolution. The overlapping blue markers indicate models M 14 3 and BL 14 3, which achieve the same accuracy level. The green markers denote models BL 28 3 and M 28,2.2 3 3, and as we expect, they achieve higher accuracy compared to BL 3, and by observing the performance of all M 3 models, we see that as the resolution of the input and the number of attended locations increases, accuracy increases as well, which is the expected trade-off between computation and accuracy. This trade-off follows a form of logarithmic curve, that saturates in models M operates. We observe that the location module is capable of focusing on the digit, and the generally good performance of the location module is reflected on the accuracy of the model, which is over 99%. However, BL, but are correctly classified by BL 28 3. In both examples, the attended locations partially cover the digits, leading 5 to be interpreted as 1, and 3 to be interpreted as 7. Of course, we are not always able to interpret the cause of a missclassification by inspecting the attended locations, but as the provided examples show, we may be able to get important insights. Besides the performance of our attention mechanism, we don't expect M 28,2.2 3 to achieve higher accuracy compared to BL 28 3. We remind that the images provided to the models are of size 28 × 28 px, the digit within each image covers an area of maximum size 14 × 14 px, and the receptive field of the baseline is 20 × 20 px. As a , the receptive field can perfectly cover the area of the digit, and the extracted features contain both coarse and fine information. Consequently, our model doesn't offer any particular advantage in terms of feature extraction that could be translated in better accuracy. This is something that we expect to change if the resolution of the input increases, or if the receptive field of the baseline gets smaller, as in the cases of BL 2 and BL 1. In Fig. 4 (b) we present the performance of models M 2 and BL 2, and the main difference we observe with (a), is that BL 56 2 demonstrates lower accuracy compared to BL 28 2. We attribute this behavior to the fact that the receptive field of BL 56 2 covers less than 10% of the area occupied by each digit, and as a , BL 56 2 is unable to extract coarse level features which are valuable for classification. Based on this hypothesis, we are able to interpret the behavior of the models in (c) as well. We observe that M In the second and third row of Figure 4, we present our on textured and noisy MNIST respectively. Our previous analysis applies to these as well. In addition, we would like to note that our attention mechanism is robust to textured and salt and pepper noise, as we can see in the respresentative examples provided in the last column of Fig. 3. In Appendix A.5.3 we provide some additional remarks on the reported in Fig. 4. Data. We use the ILSVRC 2012 version of ImageNet, which contains 1, 000 classes of natural images. The training and validation sets contain 1, 281, 167 and 50, 000 images, respectively. The average resolution of the original images is over 256 px per dimension. All images are color images, 3 We don't depict other M 28 3 models that were trained with different ct values, because they don't demonstrate any significant changes in accuracy, and would reduce the clarity of our graphs. In the y-axis we provide the top-1 accuracy on the validation set, while in the x-axis we provide the required number of FLOPs (×10 6) per image. but for simplicity, when we refer to resolution, we will drop the last dimension that corresponds to the color channels. Models. We create two pairs of models by following the same design principles we presented in Section 5.1. Model BL 1 has 3 convolutional layers and receptive field 18 × 18 px, while BL 2 has 4 convolutional layers and receptive field 38 × 38 px. have base resolution 32 × 32 px, and their location modules consider 9 candidate locations which belong to a 3 × 3 regular grid with 50% overlap. In Appendix A.6.1 we provide the architectures of all models. Training. We follow the training procedure we described in Section 5.1. The main difference is that we rescale our images to 4 different resolutions {r × r|r ∈ {32, 64, 128, 256}}, ing to 4 training sessions. We follow the notation we introduced in Section 5.1l, and we denote the models that from our first training session as M For the second training session we have r = 64, for the third r = 128, and for the fourth r = 256, while i ∈ {1, 2}. Finally, we use multi-level learning rule to train models that are able to demonstrate high accuracy after each processing level. The ing models are denoted by {M . In Appendix A.6.2 we provide additional training details. Results. In Figure 6 we provide our experimental . As in Figure 4, markers of different color denote models which are trained with images of different resolution, while the annotations next to markers that correspond to models indicate the average number of locations n c . In the first row of Fig. 6, we depict the performance of models M 2 and BL 2 . First, we would like to note that we observe the trade-off between accuracy and computatinal complexity that we have identified in Fig. 4 . When the number of required FLOPs increases by processing inputs of higher resolution, or by attending a bigger number of locations, accuracy increases as well. By inspecting the behavior of individual models, we observe that BL lack coarse information, and we expect this phenomenon to become even more intense for models BL 1, since they have smaller receptive field. Indeed, as we can see in the second row of Fig. 6, the performance gap between M 1 and BL 1 models is bigger. M We would also like to comment on the behavior of models M ml 2 and M ml 1 that we provide in the graphs of Fig. 6 . We observe that both M ml 2 and M ml 1 are able to maintain comparable performance to models M r 2 and M r 1 respectively, in all processing levels. This shows that we are able to adjust the computational requirements of our models during testing time, by controlling the number of processing levels that we allow them to go through. This is beneficial when we face constraints in the available computational resources, but also when we are processing images which vary in difficultly. Easier images can be classified after a few processing levels, while for harder ones we can extend processing to more levels. Finally, in Figure 7 we provide examples of attended image locations. We proposed a novel architecture that is able to process images of arbitrary resolution without sacrificing spatial information, as it typically happens with fully convolutional processing. This is achieved by approaching feature extraction as a top-down image pyramid traversal, that combines information from multiple different scales. The employed attention mechanism allows us to adjust the computational requirements of our models, by changing the number of locations they attend. This way we can exploit the existing trade-off between computational complexity and accuracy. Furthermore, by inspecting the image regions that our models attend, we are able to get important insights about the causes of their decisions. Finally, there are multiple future research directions that we would like to explore. These include the improvement of the localization capabilities of our attention mechanism, and the application of our model to the problem of budgeted batch classification. In addition, we would like our feature extraction process to become more adaptive, by allowing already extracted features to affect the processing of image regions that are attended later on. Figure 8 we provide the parsing tree that our model implicitly creates. Feature extraction module. It is a CNN that receives as input images of fixed resolution h × w × c, and outputs feature vectors of fixed size 1 × f . In the example of Fig. 2, h = w = 32. Location module. It receives as input feature maps of fixed size f h × f w × f c, and returns g probabilities, where g corresponds to the number of cells in the grid of candidate locations. In the example of Fig. 2, g = 4 since we are using a 2 × 2 grid. Aggregation module. It receives g vectors of fixed size 1 × f, and outputs a vector of fixed size 1 × f . The g input vectors describe the image regions inside a k × k grid. Image regions that were not selected by the location module, are described by zero vectors. The g input vectors are reorganized into a 1 × k × k × f tensor, according to the spatial arrangement of the image regions they describe. The tensor of the reorganized input vectors is used to produce the final output. Merging module. It receives two vectors of fixed size 1 × f, concatenates them into a single vector of size 1 × 2f, and outputs a vector of fixed size 1 × f . Classification module. It receives as input a vector of fixed size 1 × f, and outputs logits of fixed size 1 × c, where c is the number of classes. The logits are fed into a softmax layer to yield class probabilities. The feature extraction process of our model can be described with the following recursive equation: where ⊕ denotes the merging module, agg(·) denotes the aggregation module, loc(·) denotes the outcome of the Bernoulli sampling that is based on the output of the location module, and l Vn is the set of indexes that denote the candidate locations related to the image region described by V n. When recursion ends, V m = V m ∀m. The REINFORCE rule naturally emerges if we optimize the log likelihood of the labels, while considering the attended locations as latent variables . For a batch of N images, the log likelihood is given by the following relation: where x i is the ith image in the batch, y i is its label, and w are the parameters of our model. p(l i |x i, w) is the probability that the sequence of locations l i is attended for image x i, and p(y i |l i, x i, w) is the probability of predicting the correct label after attending l i. Equation 8 describes the log likelihood of the labels in terms of all location sequences that could be attended. Equation 8a shows that is the number of times the location module is applied while sequence l i is attended, and g is the number of candidate locations considered by the location module. In the example of We use Jensen's inequality in equation 8 to derive the following lower bound on the log likelihood: By maximizing the lower bound F, we expect to maximize the log likelihood. The update rule that we use is the partial derivative of F with respect to w, normalized by the number of images in the batch. We get: To derive equation 10 we used the log derivative trick. As we can see, for each image x i we need to calculate an expectation according to p(l i |x i, w). We approximate each expectation with a Monte Carlo estimator of M samples: We get samples from p(l i |x i, w) by repeating the processing of image x i. l i,m is the sequence of locations that is attended during the mth time we process image x i. In order to reduce the variance of the estimators, we use the baseline technique from. In particular, the baseline we use is the exponential moving average of the log likelihood, and is updated after the processing of each batch during training. Our baseline after the nth batch is the following: where x n i is the ith image in the nth batch, y n i is its label, and l i,n is the corresponding attended sequence of locations. Since we use M samples for the Monte Carlo estimator of each image, we simply consider that our batch has size NM to simplify the notation. Our updated learning rule is the following: For simplicity, we drop the indexes that indicate the batch we are processing. is the learning rule we presented in, and this concludes our derivation. We provide the exact equations that we use to calculate quantities p l i t and p k in regularization terms R t and R r respectively. We have: Based on the notation we introduced in Appendix A.4.1, p l i t approximates the expected number of attended locations for image x i, by summing the probabilities from all Bernoulli distributions considered during a single processing of x i under l i. p k computes the average probability of attending the kth out of the g candidate locations, that the location module considers every time it is applied. The average is calculated by considering all the times the location module is applied during the processing of the N M images in our augmented batch. Finally, we would like to note that the values of c t and c r are interdependent. A.5.1 ARCHITECTURES In Tables 1, 2 and 3, we provide the exact architectures of models (M 3, BL 3), (M 2, BL 2) and (M 1, BL 1). We provide details about training one pair of models (M i, BL i) with one of the 3 MNIST-based datasets, and the exact same procedure applies to all pairs and to all datasets. In our first training session we optimize the cross entropy loss to train BL i, and we use learning rule for M i. We remind that because of the input resolution, M i goes through only 1 processing level, and behaves as a regular CNN which is consisted of the feature extraction module followed by the classification module. As a , the only term of learning rule that we actually use, is the first term of L F, and we end up optimizing the cross entropy of the labels. For both models we use the following hyperparameters. We use learning rate 0.01 that drops by a factor of 0.2 after the completion of 80% and 90% of the total number of training steps. We train for 70 epochs, and we use batches of size 128. We use the Adam optimizer with the default values of β 1 = 0.9, The architectures of models M 3 and BL 3 that we used in our experiments with MNIST. "GAP" denotes a global average pooling layer. The variable output sizes of the baseline CNN are approximately calculated to preserve the clarity of our tables. Based on these approximate output sizes, the computation of the number of FLOPs in the corresponding layers is approximate as well. β 2 = 0.999 and = 10 −8. We use xavier initialization for the weights, and zero initialization for the biases. For regularization purposes, we use the following form of data augmentation. Every time we load an image, we randomly change the position of the digit inside the black canvas, as well as the noise pattern and the , in case we are using noisy MNIST or textured MNIST. This data augmentation strategy and the aforementioned hyperparameter's values, are used in the other two training sessions as well. In the second training session we again optimize the cross entropy loss for BL i, and we use learning rule for M i. The resolution of the input allows M i to go through 2 processing levels, and all terms of contribute to the updates of our models' parameters. We would like to note that we stop the gradients' flow from the location module to the other modules, and as a , the second term of, as well as the regularization terms and, affect only the parameters of the location module. We do this to have better control on how the location module learns, since we experienced the problems we described in Section 4. In the same context, we set λ f = 10 −6, λ r = 10 −7 and λ t = 10 −7, which lead to very small updates at every training step. For our Monte Carlo estimators we use 2 samples. These hyperparameter values are used in the third training session as well. The models M i which are reported in Figure 4 and from this training session (green circles), are trained with c t = 2. In the third training session M i can go through 3 processing levels, and we can train it by using either learning rule, or gradual learning. Gradual learning evolves in 2 stages, where in the first stage M i can go through 2 processing levels, while in the second stage it can go through 3. The first training stage is equivalent to the previous training session, since there is an equivalence between going through a different number of processing levels and processing inputs of different resolution. Based on that, we can directly move to the second training stage of gradual learning, by initializing the variables of M i with the learned parameters from one of the M 28 i models that we already trained. Gradual learning creates an imbalance in terms of how M i and BL i are trained, since it evolves in multiple stages. However, if we see gradual learning as a method of training models with images of gradually increasing resolution, it can be applied to the baselines as well. Based on this, we can i, and then apply our standard optimization to the cross entropy loss. In practice, we observe that baselines are almost always benefited by gradual learning, while in some cases our models achieve higher accuracy when they are trained from scratch with learning rule. In general, gradual learning and most modifications to the initial learning rule, ed from our experimentation with ImageNet, which is a much bigger and more diverse dataset of natural images. Consequently, our on the MNIST-based datasets remain almost unchanged even if we simplify our training procedure, e.g. by excluding term from our training rule. However, we kept our training procedure consistent both on the MNIST-based datasets and on ImageNet, we experimented both with gradual learning and with training from scratch at every session, and in the of Figure 4 we report the best performing models. Finally, the models M i which are reported in Figure 4 and from this training session (red circles), are trained with c t = 6 and c t = 12. In Figure 4 (e), BL 56 2 achieves higher accuracy compared to BL 28 2, which wasn't the case in (b). We hypothesize that the fine information provided by the increased resolution, is valuable to disentangle the digits from the distracting , and outweighs the lack of coarse level features that stems from the receptive field size of BL 56 2. In addition, the differences in accuracy between M 1 and BL 1 models in Fig. 4 (f), are considerably bigger compared to the ones recorded in (c). This shows that our models are more robust to distracting textured compared to the baselines. In Fig. 4 (g . This is surprising, because M 56,6.1 3 is processing images of higher resolution, and it should be able to reach at least the same level of accuracy as M 28,2.2 3 . Our explanation for this observation is based on the nature of the data. As we can see in the example images that we provide in the last row of Figure 4, when the resolution of an image is reduced, the noise is blurred out. As a , when M 56,6.1 3 is processing images of higher resolution, it is processing more intense high frequency noise, and since it is using a limited number of locations, its accuracy drops compared to M . This phenomenon is observed in Fig. 4 respectively. The explanation we provided adds a new dimension to the understanding of our models, because so far we were treating fine information as something that is by default beneficial for classification, while high frequency noise of any form may require special consideration in our design and training choices. Tables 4 and 5, we provide the exact architectures of models (M 2, BL 2) and (M 1, BL 1). We provide details about training one pair of models (M i, BL i). We make a distinction between models M 1 and M 2 only when the process we follow, or the values of the hyperparameters that we use, differ. In our first training session we optimize the cross entropy loss to train BL i, and we use learning rule for M i. The base resolution of M i matches the size of the input images (32 × 32 px), and our model goes through only 1 processing level, without using the location module. As a , the only term of learning rule that we actually use, is the first term of L F, and we end up optimizing the cross entropy of the labels. For both models we use the following hyperparameters. We use learning rate 0.001 that drops by a factor of 0.2 after the completion of 80% and 90% of the total number of training steps. We train for 200 epochs, and we use batches of size 128. We use the Adam optimizer with the default values of β 1 = 0.9, β 2 = 0.999 and = 10 −8. We use xavier initialization for the weights, and zero initialization for the biases. For regularization purposes, we use data augmentation that is very similar to the one used by. In particular, given a training image, we get a random The architectures of models M 2 and BL 2 that we used in our experiments on ImageNet. "GAP" denotes a global average pooling layer. The variable output sizes of the baseline CNN are approximately calculated to preserve the clarity of our tables. Based on these approximate output sizes, the computation of the number of FLOPs in the corresponding layers is approximate as well. crop that covers at least 85% of the image area, while it has an aspect ration between 0.5 and 2.0. Since we provide inputs of fixed size to our networks, we resize the image crops appropriately. The resizing is performed by randomly selecting between bilinear, nearest neighbor, bicubic, and area interpolation. Also, we randomly flip the resized image crops horizontally, and we apply photometric distortions according to. The final image values are scaled between −1 and 1. This data augmentation strategy and the aforementioned hyperparameter's values, are used in the other training sessions as well, and in the stages of multi-level learning that we describe later. In the second training session we again optimize the cross entropy loss for BL i, and we use learning rule for M i. The resolution of the input allows M i to go through 2 processing levels, and all terms of contribute to the updates of our models' parameters. As we described in Appendix A.5.2, we stop the gradients' flow from the location module to the other modules. We set λ f = 10 −8, λ r = 10 −9 and λ t = 10 −9, and for our Monte Carlo estimators we use 2 samples. These hyperparameter values are used in the two remaining training sessions as well. The models M 64 i which are reported in Figure 6 are trained with c t ∈ {1.5, 4.5}. In the third training session we use gradual learning for M i, by initializing its variables with the learned parameters of a M 64 i model. The models M 128 2 which are reported in Figure 6 are trained with c t ∈ {3.75, 7, 13.5, 24.75}, and models M 128 1 with c t ∈ {3.75, 8, 11, 13.5, 30}. We use gradual learning for BL i as well, by initializing its parameters with those of BL 64 i before we apply the standard optimization to the cross entropy loss. In the fourth training session we again use gradual learning for M i, by initializing its variables with the learned parameters of a M with c t ∈ {30, 35, 40}. As in the previous session, we optimize the cross entropy loss to train BL i, and we initialize its parameters with those of BL 128 i.

Represent the TLDR for finding the corresponding scientific article

We propose a novel architecture that traverses an image pyramid in a top-down fashion, while it visits only the most informative regions along the way.

represent this text to find a much much longer article about it with say hundreds-thousands of words

For instance, <<Use deep reinforcement learning to design the physical attributes of a robot jointly with a control policy.>>  to <<The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs. An agent's ability to navigate through and interact with its environment depends not just on its skill at planning and controlling its motion, but also on its physical design. Different physical designs are inherently better suited to different tasks and environments. By making appropriate choices during fabrication, mechanical elements can be designed to improve robustness to non-idealities such as errors in perception, delays in actuation, etc., and indeed, make control problem an easier one to solve. At the same time, robots that take different forms may find completely different control strategies to be optimal to complete the same task. Therefore, the physical and computational design of an agent are inherently coupled, and must ideally be jointly optimized if the robot is to successfully complete a task in a particular environment. Consider the development of a legged robot for locomotion. Variations in physical design will require changes to the joint torques in order to preserve a particular locomotion behavior (e.g., a heavier torso requires greater torque at the ankle), and will likely in completely different walking gaits, even when the morphology is preserved. In fact, some changes to design may render locomotion impossible for the target operating environment (e.g., a robot with long feet may be unable to locomote up an incline). Meanwhile, careful choice of bipedal design enables passive walking BID20 BID9 BID4. It is therefore beneficial to not simply consider the robot's design or gait to be fixed, but to optimize both jointly for the target environment and task. Similar co-design can be beneficial in other settings-for example for the control policy and physical characteristics of digits in robotic grippers for grasping. While a robot's physical design and the corresponding control policy are inherently coupled, most existing methods ignore this coupling, instead choosing to alternate between separate design and control phases. Existing approaches that jointly reason over design and control BID7 BID12 BID46 assume knowledge of an accurate model of the robot dynamics and require expert supervision (e.g., to provide a suitable initial design and guide the optimization process). However, these restrictive assumptions limits their applicability to a handful of specific settings, and often yield solutions heavily influenced by expert intuition. In this work, we seek a general approach-one that can optimize a robot's physical characteristics jointly with controllers of a desired complexity (Fig. 1), that can be applied to general tasks in some DISPLAYFORM0 Figure 1: Our algorithm learns a robot's physical design jointly with the control policy. Here we show the learned designs evolving over time for the Hopper (top left), the Walker2d (top right) and the Ant (bottom), each with the default Roboschool design for comparison. Scale is fixed for each robot. Note that these designs correspond to modes of the distribution over robot designs that our algorithm maintains during training.given environment, and that can explore the joint search space of physical design and computational control in a purely data-driven way, without a model of the robot dynamics and independent of the biases of expert intuition. We develop this approach in the context of determining the physical parameters of an articulated agent-the lengths and thicknesses of each limbs in a given morphologythrough joint training with a neural network for control, with the objective of achieving locomotion. Our method maintains a distribution over these physical parameters, and simultaneously trains the parameters of this distribution with those of a neural network controller, using deep reinforcement learning. In this way, we pursue a design distribution and control policy that are jointly optimal for the given task and environment. Experimental show that starting from random initializations, our approach is able to find novel designs and walking gaits that match or exceed the performance of manually designed agents. To the best of our knowledge, our method is the first to successfully carry out such a joint optimization of design and control in a completely model-free manner. Attention has been paid recently to the problem of jointly optimizing the parameters that specify a robot's design and motion (e.g., gaits) or control. Early work in this area takes an evolutionary approach to optimizing a robot's design and controller, typically parameterized as a neural network, for virtual agents BID45 BID30 BID0 and physical robots BID19 BID1. Particularly relevant to our approach is the work of BID12, who relate design and motion parameters via a set of implicit functions that express robot dynamics, desired trajectories, and actuation limits. These functions encode a manifold that is then linearized to model the relationship between design and motion via the implicit function theorem. The method then solves for the desired parameters in a local fashion via constraint-based optimization. Similarly, BID46 describe an approach that jointly reasons over physical design and motion parameters for robots with articulated degrees of freedom (e.g., legged robots). They formulate the problem in the framework of trajectory optimization by incorporating parameters involved in the design of articulated robots, including dimensions, masses, mass distribution, and moments of inertia, together with the contact force and torque variables typically associated with trajectory optimization. They use weighted actuation cost as the objective, subject to a regularization penalty on parameters. Their method is guaranteed to find a feasible design so long as the problem is initialized within a small neighborhood of a feasible solution. Unlike our approach, their method requires that the user provide an estimate of the robot configuration at each time step, as well as an accurate analytic model of constraint dynamics (e.g., foot contact), which is computationally expensive to enforce. BID8 propose a derivative-free strategy that optimizes over muscle routing and muscle-based control for simulated bipeds to realize a design that incorporates biomechanical constraints. describe an evolutionary method that jointly reasons over design and motion parameters for legged robots, while also tuning the parameters of a robust controller that tracks these motions. Their method is limited to biologically inspired quadrupedal foothold patterns, and does not account for contact dynamics. In contrast, our approach applies to arbitrary morphologies and does not require that we model contact dynamics. While the focus on simultaneous optimization over robot design and motion parameters is relatively new, there is a long history of research focused on the related problem of co-design of physical structure and control BID35 BID34. BID29 jointly optimize the link geometry of a high-speed robot arm along with the parameters of a PD joint controller. These methods often rely upon access to an accurate model of the robot dynamics in order to design the controller. In order to reduce the dependence upon a detailed analytical model, BID32 use on-robot experiments to refine their model as part of the iterative design process. More recently, BID49 allow for some degree of uncertainty in the model by including the sensitivity of the design and control parameters to model uncertainty along with the task-specific optimization objectives. However, existing methods still rely upon access to an analytical model of the robot dynamics and are typically limited to simple (e.g., linear) control designs. Our method assumes only that the controller can be modeled via a convolutional neural network, and can thereby learn complex, highly nonlinear control policies with no a priori knowledge of the robot dynamics. Far more attention has been paid to the individual problem of task-driven optimization of robot motion and control. Given an arbitrary robot design chosen by novice users, BID21 describe an interactive design framework that solves for a realizable walking gait that in stable locomotion. Similarly, BID27 synthesize emergent behaviors (motion) for arbitrary morphologies and tasks from high-level specifications, by jointly optimizing over contact and motion. BID28 build upon this work by training recurrent neural networks to serve as feedback controllers capable of producing complex, stable behaviors for a variety of dynamical systems. Their method interleaves supervised learning with trajectory optimization, and incorporates noise to improve generalizability. Meanwhile, there is a large body of literature that formulates motion synthesis as a trajectory optimization problem. This approach has proven effective at respecting contact constraints (e.g., between the foot and ground), which make controlling dynamic motion particularly challenging BID6 BID33 BID10. These approaches have been shown to generate sophisticated behaviors for complex robot designs (e.g., humanoids) BID48, and for robots of arbitrary morphologies using only a high-level specification of the robot's shape, gait, and task BID50. Related, a number of methods interleave trajectory optimization and supervised learning with neural network regression BID14 BID15 BID26 BID28. Unlike our method, the use of trajectory optimization makes these approaches reliant upon knowledge of the model. A great deal of attention of-late has focused on the problem of learning complex control policies directly from low-level sensory input, without any knowledge of the system dynamics. Methods that have proven particularly effective combine neural networks that learn representations of the highdimensional raw sensor input with deep reinforcement learning BID36 BID24 BID41. While much of the work in this area focuses on low-dimensional, discrete action spaces, several methods have been recently proposed that learn continuous control policies through deep reinforcement learning. These techniques have been applied to control simple, simulated robots BID52 BID53 BID51 BID18, perform robot manipulation BID16 BID19 BID11, and control legged robots BID42 BID31.Black-box optimization BID40 BID37 is an alternative to using reinforcement learning to training the control policy. These approaches have the advantage that they do not involve backpropagating gradients, are insensitive to reward sparsity, and can handle long time horizons. While black-box optimization strategies have traditionally been thought of as illsuited to difficult reinforcement learning problems, BID38 recently showed that they perform similarly to state-of-the-art RL methods on difficult continuous control problems, including those that involve locomotion. Closely related is the policy-gradient method of BID44, who define the policy as a distribution over the parameters of a controller, which they then sample over. This in gradient estimates that are far less noisy than is typical of policy gradient algorithms. These approaches are similar to the way in which we learn the robot design, which we formulate as a Gaussian mixture model over design parameters. Indeed, the two referenced methods yield the same gradient estimate for Gaussian parameter distributions BID38.Meanwhile, much work has focused on the problem of determining robot designs that meet the requirements of a particular task. Given a user demonstration of the desired behaviors, BID5 learn optimum kinematic linkages that are capable of reproducing these motions. BID22 synthesize electromechanical robot designs in a compositional fashion based upon a complete user-specified structural specification of the robot. BID23 build upon this work, allowing the user to specify functional objectives via structured English, which is parsed to a formal specification using linear temporal logic. describes a theory for co-design that includes the ability to select discrete robot parts according to functional constraints, but do not reason over geometry or motion. Related to our approach is recent work that jointly optimizes sensor design and inference algorithms for perception systems. BID3 considers the problem of jointly learning a camera sensor's multiplexing pattern along with reconstruction methods for imaging tasks. They model inference as a neural network together and use stochastic gradient descent to backpropagate the loss to a neural layer representation of the multiplexing pattern. Related, BID39 jointly learn design and inference for beacon-based localization. They encode beacon allocation (spatially and across transmission channels) as a differential neural layer that interfaces with a neural network for inference. Joint optimization then follows from standard techniques for training neural networks. In this section, we begin by describing the standard reinforcement learning framework for training agent policies, and then describe how we extend this to also learn the physical design of the agent. In the standard reinforcement learning setting, an agent interacts with its environment, usually a Markov Decision Process, over a number of discrete timesteps. At each time step t, the agent receives a state s t ∈ S and takes action a t ∈ A according to a policy π: S → A. Then, the agent receives a scalar reward r t and the next state s t+1 from the environment. This process continues until a terminal state is reached. The goal of reinforcement learning is to then find a policy π * that maximizes the expected return E[R t], where R t = ∞ i=0 γ i r t+i and γ ∈ is a discount factor. Policy gradient methods are a class of algorithms often used to optimize reinforcement learning problems, due to their ability to optimize cumulative reward and the ease with which the can be used with neural networks and other nonlinear function approximators. Consequently, they are commonly used for reinforcement learning problems that involve complex, continuous action spaces. Policy gradient methods directly parameterize a stochastic policy π θ (a t |s t) and perform stochastic gradient ascent on the expected return. "Vanilla" policy gradient methods compute an estimate of the gradient ∇ θ E[R t] using a sample-based mean computed over ∇ θ log π θ (a t |s t)R t BID54, which yields an unbiased gradient estimate BID47. While the variance in the ing estimate decreases with the number of samples, sampling is computationally expensive. Another way to reduce variance while maintaining an unbiased estimate is to estimate the gradient by comparing the reward to a "baseline" reward b(s t).These methods are effective but can be unstable, especially when used for deep reinforcement learning. Small changes to the policy parameters may cause large changes in the distribution of visited states. Several methods have been proposed to mitigate these effects. Among them, BID41 introduce Trust Region Policy Optimization (TRPO), which imposes a constraint on the KLdivergence between policies before and after an update. Recently, proposed proximal policy optimization (PPO), a first-order class of methods similar to TRPO. PPO alternates between sampling data through interaction with the environment and optimizing the objectivê DISPLAYFORM0 where r t (θ) = π θ (at|st) π θ old (at|st) andÊ t represents an empirical average over a finite sample set. This objective seeks to maximize the expected return while encouraging a small step in policy space. The clipping within the objective removes any incentive for moving r t (θ) outside the interval [1−, 1+]. The net effect of this objective is a simple, robust policy gradient algorithm that attains or matches state-of-the-art on a wide array of tasks We extend the standard reinforcement learning formulation by considering the space of possible robot designs Ω. Specifically, we assume that for every design E ∈ Ω, we can define common state and action spaces S and A, reward function r(s, a), and initial state distribution p 0 (s), that are meaningful to all designs. The designs differ only in the transition dynamics they induce p E (s |s, a) and share a common action space A-to achieve this, we assume a common morphology for all possible designs. Our goal then is to find the optimal design E * and policy π * E * pair that maximizes the expected value of a given reward function. However, this is a non-linear, non-convex optimization problem over the spaces of all possible designs and all possible policies. Solving it exactly would require enumerating all possible designs (possibly through discretization of the design space), learning policies for each, and comparing the ing expected rewards. This is computationally infeasible for all but the simplest of cases. Instead, we develop a gradient-based approach to solving this optimization problem with the following key components: we maintain a multi-modal distribution over the space of physical designs and update this distribution using policy gradient methods in parameter space, similar to BID44 and BID38; and we train a single controller to act on all sampled designs during training, providing this controller with the design parameters of the specific sample it is controlling. We find that the multi-modal stochastic parameterization of the design space allows our method to explore the space more thoroughly (where a uni-modal Gaussian distribution would frequently get trapped in local minima). Moreover, a common controller makes optimization tractable, allowing efficient evaluation of unseen designs. We additionally benefit from learning common strategies across diverse designs, while still adapting different policies to different parts of the design space given the sample parameters as input. Together, these components enable efficient joint exploration of the design and policy spaces. Formally, let p(r; φ) denote the distribution over designs, and π(a t |s t, r; θ) a stochastic control policy parameterized by φ and θ respectively. In our experiments, we use a neural network to model the control policy π, and a Gaussian mixture model as the parametric form of the distribution p(r; φ). Our goal is then to solve the following optimization problem: DISPLAYFORM1 where E r,t [·] is the expectation over robots and trajectories those robots induce. We use stochastic gradient-based updates to optimize Eqn. 2. Our method (Algorithm 3.2) alternates between updating the parameters of the policy and design distributions, θ and φ, respectively. We empirically find this to yield convergence to better solutions in a reasonable number of iterations compared to performing simultaneous updates. We optimize the policy parameters θ using Proximal Policy Optimization. However, instead of collecting data from a single design, we sample a design r after a fixed number T of timesteps according to the distribution p(r; φ). After n iterations of PPO, we freeze the policy parameters and proceed to optimize the parameters of the design distribution. Without knowledge of the model, we optimize the design parameters φ via policy gradient over parameter space, similar to BID44. This is equivalent to black-box optimization methods that have proven effective for complex learning problems BID38. We sample m different designs and compute their returns for a single episode acting under policy π(·; θ). We then use this data to shift the design distribution p(·; φ) in a direction maximizing expected return. These iterations are repeated until convergence. We validate our approach with three commonly used robots provided in OpenAI's Roboschool. Environments in Roboschool are built on top of Bullet Physics, a popular open-source physics engine. In a series of locomotion experiments, we show that our approach not only discovers novel robot designs and gaits, but also outperforms two out of three of the hand-designed robots trained on the same task and yields performance comparable to the third. We evaluate our approach on three legged locomotion tasks within Roboschool: RoboschoolHopper, RoboschoolWalker2d, and RoboschoolAnt (note that we subsequently drop the Roboschool prefix for brevity). FIG0 depicts the default Roboschool design for each robot. These environments describe a locomotion task that has the robots moving along the positive x-axis (to the right in the figures) towards a distant goal. The reward function defined in these environments is a weighted sum of rewards for forward progress and staying upright, and penalties for applying torques and for joints that reach their rotational limits. The episode ends when the robot falls over or reaches a maximum number of timesteps. We learn robot designs using the morphologies specified by the standard Roboschool designs. For each morphology, we parameterize the robot in terms of the length and radius (e.g., mass) of each link (or just the radius in the case of the sphere for the ant body). We impose symmetry, and share parameters across each leg for the Walker2d and Ant designs. This parameterization permits a wide variety of robots of different shapes and sizes, some of which are better suited to the locomotion objective than others, and some designs that do not permit a control policy that in locomotion. We model the control policy π(a t |s t, E; θ p) as a feed forward neural network consisting of three fully-connected layers with 64 units and tanh activation functions. A final layer maps the output to the robot action space. With the exception of the last robot-specific layer, the architecture is the same for all experiments. Meanwhile, we represent the distribution over the parameterized robot design p(·; θ r) using a Gaussian mixture model with four mixture components (each with a diagonal covariance matrix over the different parameters). We initialize the means of each component randomly within a wide range, and initialize the variances in order to span the range of each parameter. We find that our approach maintains high variance distributions during early iterations-thereby continuing exploration of the design space-before committing to a chosen design. The appendix provides further details regardng the evolution of these distributions. Timesteps Ant Figure 3: Training curves that show the evolution of reward across training iterations for the Hopper, Walker2d, and Ant environments over eight different random seeds. The black dashed line corresponds to the highest achievable performance of the corresponding baseline robot. In other figures, we display the best-performing checkpoint of each run. Therefore, we make each training curve transparent after it stops improving. We train our method with eight threads of Proximal Policy Optimization for a total of 300M environment timesteps across all threads. We use the publicly available implementation of PPO provided in OpenAI Baselines BID13. We alternate between 50 policy PPO iterations and 2 design iterations, finding this ratio to yield convergence to good solutions. We set this and the other hyper-parameters on the Hopper and used the same settings for all experiments. We evaluate the performance of our approach to joint optimization over robot design and control on the Hopper, Walker2d, and Ant robots, and compare against a policy trained for the standard Roboschool design. We evaluate the consistency and robustness of our approach relative to the initial robot parameter distribution using several different random seeds. For each robot morphology, we find that our method learns robot designs and the corresponding control policy that are either comparable (Walker2d) or exceed the performance (Hopper and Ant) achievable using the default designs (Fig. 3). We also see that our method achieves these levels of performance by discovering unique robot designs (Fig. 1) together with novel walking gaits (Fig. 4) that can not be achieved with the default designs. Note that our method was able to learn these designs and control policies from a random initialization, without access to a dynamics model or any expert supervision. Our method learns a joint design and controller for the Hopper that outperforms the baseline by as much as 50%. Our learned robot exploits the small size of the foot to achieve faster, more precise walking gaits. Meanwhile, the longer torso of the learned robot improves stability, allowing it to maintain balance while locomoting at a faster pace. We found this behavior to be consistent across several different Hopper experiments, with the method converging to designs with a small foot and long, thin torso. In the appendix, we explore the stability of this design with respect to variations in the environment using the coefficient of friction as an example, and find the improvement to be consistent. For the ant, our optimization yields a physical design that is significantly different from the default design FIG0. Consequently, the learned Ant drastically outperforms the baseline, improving reward by up to 116%. Our method learns a design with a small, lightweight body and extremely long legs. The long legs enable the ant to apply large torque at contact, allowing it to move at a fast pace. Our framework learned different design-control pairs for the Walker2d that perform similarly to the default design. Across several different experiments, we see two distinct, successful designs and walking gaits. Interestingly, neither agent walks primarily on its feet. The first design has small, thick legs and long feet. The controller is then able to exploit the thick middle leg link, which protrudes slightly past the foot, to push off the ground. The long foot then provides additional balance. The second design is similar in geometry to the baseline Walker2d, but moves in a very different way. By lowering the knee joint and lengthening the foot, the Walker2d is able to efficiently balance on its knees and toes. This low stance allows the Walker2d to fully extend its leg backwards, creating a long, powerful stride, similar to that of a sprinter using starting blocks. Figure 4: Here, we compare locomotion of default and learned robots, visualizing both their physical design and corresponding learned gaits. We pick the best for Hopper and Ant, and two of the best for Walker2d (due to diversity in gaits). Note that for each robot type, we show a blend of a fixed number of frames in the same time interval, allowing direct comparison between the speed with which different designs are able to locomote. We proposed what is, to the best of our knowledge, the first model-free algorithm that jointly optimizes over the physical design of a robot and the corresponding control policy, without any need for expert supervision. Given an arbitrary morphology, our robot maintains a distribution over the robot design parameters and learns these parameters together with a neural network controller using policy gradient-based reinforcement learning. This in an assignment to the policy over robot parameters and the control policy that are jointly optimal. We evaluated our approach on a series of different legged robot morphologies, demonstrating that it in novel robot designs and walking gaits, achieving performance that either matches or exceeds that of manually defined designs. Our findings suggest several avenues for future work. The most direct is extending the current approach to find optimized designs for uneven terrain, the presence of obstacles, changes in slope, variations in friction, etc. We are also interested in extending our framework to relax the assumption that the morphology is pre-defined. Finally, we are investigating applications to different types of agents and design spaces beyond legged robots (e.g., end-effectors), and exploring appropriate stochastic parameterization for such designs. The following provides further evaluation of our framework, including the behavior of the design distribution during training and the robustness of the learned designs to environment variations. To provide insight into the training process of our algorithm, we evaluate the evolution of the Gaussian mixture model distribution throughout training of the best performing Hopper experiment. We initialize each component of the mixture model with random means and a diagonal covariance matrix chosen to cover the parameter space. The initial mixture weights are uniform. As shown in FIG1 (left), roughly one third of the way through training, our algorithm converges to the most successful component. Additionally, we find that modes generally do not collapse (FIG1). We find this behavior to be consistent across random seeds and different robots. It is often desirable for a robot to be able to operate in a variety of environments. In this section, we consider the robustness of a learned design to changes in friction. We conducted experiments on the Hopper in which we first learned the design and controller for one friction setting (0.8). We finetuned the controller in environments with different friction settings, while leaving the design fixed. We find the learned design to be reasonably robust with variability comparable to controllers finetuned for the default hand-crafted design (Fig. 6). Additionally, the learned design outperforms the hand-crafted one across the full range of friction values (although, for very low friction values, both designs essentially were unable to learn a successful gait).Note that our framework can incorporate this goal of generalization by simply sampling from a diverse set of environments during training. But at the same time, it may be useful in some applications to seek out solutions that are specifically adapted to a relatively narrower set of environments, gaining better performance within this set at the cost of more general performance. Figure 6: A visualization of accumulated reward for different friction values. The controllers and learned design were trained in an environment with a friction value of 0.8. We finetuned the controllers (but not the designs) for both the learned and hand-crafted designs for 10M timesteps for each friction value. Rewards are reported as an average over 100 episodes.>>

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.