Question: Represent the article for finding a single-sentence TLDR

Question: Deep Convolutional Neural Networks (CNNs) have been repeatedly shown to perform well on image classification tasks, successfully recognizing a broad array of objects when given sufficient training data. Methods for object localization, however, are still in need of substantial improvement. Common approaches to this problem involve the use of a sliding window, sometimes at multiple scales, providing input to a deep CNN trained to classify the contents of the window. In general, these approaches are time consuming, requiring many classification calculations. In this paper, we offer a fundamentally different approach to the localization of recognized objects in images. Our method is predicated on the idea that a deep CNN capable of recognizing an object must implicitly contain knowledge about object location in its connection weights. We provide a simple method to interpret classifier weights in the context of individual classified images. This method involves the calculation of the derivative of network generated activation patterns, such as the activation of output class label units, with regard to each in- put pixel, performing a sensitivity analysis that identifies the pixels that, in a local sense, have the greatest influence on internal representations and object recognition. These derivatives can be efficiently computed using a single backward pass through the deep CNN classifier, producing a sensitivity map of the image. We demonstrate that a simple linear mapping can be learned from sensitivity maps to bounding box coordinates, localizing the recognized object. Our experimental , using real-world data sets for which ground truth localization information is known, reveal competitive accuracy from our fast technique. Deep Convolutional Neural Networks (CNNs) have been shown to be effective at image classification, accurately performing object recognition even with thousands of object classes when trained on a sufficiently rich data set of labeled images BID14. One advantage of CNNs is their ability to learn complete functional mappings from image pixels to object categories, without any need for the extraction of hand-engineered image features BID21. To facilitate learning through stochastic gradient descent, CNNs are (at least approximately) differentiable with regard to connection weight parameters. Image classification, however, is only one of the problems of computer vision. In the task of image classification, each image has a single label, associated with the class identity of the main object in the image, and the goal is to assign correct labels in a manner that generalizes to novel images. This can be accomplished by training a machine learning classifier, such as a CNN, on a large data set of labeled images BID5. In the object localization task, in comparison, the output for a given image is not a class label but the locations of a specified number of objects in the image, usually encoded as bounding boxes. Evaluation of an object localization system generally requires ground truth bounding boxes to compare to the system's output. The detection task is more difficult than the localization task, as the number of objects are not predetermined BID21.In this paper, we focus on object localization, identifying the position in the image of a recognized object. As is common in the localization literature, position information is output in the form of a bounding box. Previously developed techniques for accomplishing this task generally involve searching the image for the object, considering many candidate bounding boxes with different sizes and locations, sometimes guided by an auxilliary algorithm for heuristically identifying regions of interest BID21; BID10; BID13. For each candidate location, the sub-image captured by the bounding box is classified for object category, with the final output Figure 1: Examples of sensitivity maps, displaying the sensitivity of network internal representations to individual pixels, providing information about the locations of the main objects in the source images.bounding box either being the specific candidate region classified as the target object with the highest level of certainty or some heuristic combination of neighboring or overlapping candidate regions with high classification certainty. These approaches tend to be time consuming, often requiring deep CNN classification calculations of many candidate regions at multiple scales. Efforts to speed these methods mostly focus on reducing the number of regions considered, typically by using some adjunct heuristic region proposal algorithm BID10; BID17; BID13. Still, the number of considered regions is often reported to be roughly 2,000 per image. While these approaches can be fairly accurate, their slowness limits their usefulness, particularly for online applications. A noteworthy alternative approach is to directly train a deep CNN to produce outputs that match ground truth localization bounding boxes, using a large image data set that provides both category and localization information for each image. It appears as if some form of this method was used with AlexNet BID14, though details concerning localization, rather than image classification, are difficult to discern from the published literature. A natural approach would be to cast the learning of bounding boxes as a simple regression problem, with targets being the four coordinates that specify a bounding box (e.g., coordinates of upper-left and lower-right corners, or region center coordinates along with region width and height). It is reasonable to consider sharing early layers of a deep CNN, such as those performing convolution and max pooling, between both an image classification network and an object localization network. Indeed, taking such a multitask learning approach BID2 can allow for both object category and object location training data to shape connection weights throughout the network. Thus, the deep CNN would have "two heads", one for image classification, using a classification cross-entropy loss function, and one for object localization, reducing the 2 norm between ground truth and predicted bounding box coordinates BID14. While this approach can produce a network that quickly outputs location information, extensive training on large data sets containing ground truth bounding box information is necessary to produce good generalization. In this paper, we introduce an approach to object localization that is both very fast and robust in the face of limited ground truth bounding box training data. This approach is rooted in the assertion that any deep CNN for image classification must contain, implicit in its connection weights, knowledge about the location of recognized objects BID20. The goal, then, is to interpret the flow of activation in an object recognition network when it is performing image classification so as to extract information about object location. Furthermore, the goal is to do this quickly. Thus, this approach aims to leverage location knowledge that is already latent in extensively trained and tuned image classification networks, without requiring a separate learning process for localization. Our method makes use of the notion of a sensitivity analysis BID26. We propose estimating the sensitivity of the category outputs, or activation patterns at internal network layers, of an image classification CNN to variance in each input pixel, given a specific input image. The is a numeric value for each pixel in the input image that captures the degree to which small changes in that pixel (locally, around its current value) give rise to large changes in the output category. Together, these numeric values form a sensitivity map of the image, encoding image regions that are important for the current classification. Our proposed measure of sensitivity is the partial derivative of activity with regard to each pixel value, evaluated for the current image. For a deep CNN that formally embodies a differentiable mapping (at least approximately) from image pixels to output categories, this partial derivative can be quickly calculated. While many tools currently exist for efficiently calculating such derivatives, we provide a simple algorithm that computes these values through a single backward pass through the image classification network, similar to that used to calculate unit error (delta) values in the backpropagation of error learning algorithm BID18. Thus, we can generate a sensitivity map for an image in about the same amount of time as it takes the employed image classification network to produce an output. Some example sensitivity maps are shown in Figure 1.The idea of using sensitivity information, like that in our sensitivity maps, for a variety of tasks, including localization, has previously appeared in the literature BID24; BID28 BID20. Indeed, some of these past efforts have used more sophisticated measures of sensitivity. In this paper, we show that even our very simple sensitivity measure can produce strong localization performance, and it can do so quickly, without any modifications to the classification network, and even for object categories on which the classification network was not trained. The relationship of the reported here to previously reported work is discussed further in Section 4.As previously mentioned, object localization methods typically encode object location as a bounding box. Since our sensitivity maps encode location differently, in terms of pixels, we propose learning a simple linear mapping from sensitivity maps to bounding box coordinates, allowing our method to output a bounding box for each classified image. We suggest that this linear mapping can be robustly learned from a relatively small training set of images with ground truth bounding boxes, since the sensitivity maps form a much more simple input than the original images. The primary contributions of this paper may be summarized as follows:• We propose a new general approach to performing object localization, interpreting a previously trained image classification network by performing a sensitivity analysis, identifying pixels to which the category output, or a more general internal representation, is particularly sensitive.• We demonstrate how a linear function from the ing sensitivity maps to object location bounding box coordinates may be learned from training images containing ground truth location information.• We provide a preliminary assessment of our approach, measuring object localization performance on the ImageNet and PASCAL VOC data sets using the VGG16 image classification CNN, showing strong accuracy while maintaining short computation times. Calculating derivatives of a function of network output with regard to network parameters, such as connection weights, is a standard part of CNN training. It is common for learning in a deep CNN to involve stochastic gradient decent, which involves such derivatives. In that case, the derivatives are of an objective function with regard to connection weight values. In image classification networks, the objective function is designed to have optima where training images are correctly classified. In the case of object localization, a similar objective function could be designed to minimize differences between output bounding box coordinates and provided ground truth bounding box coordinates, for all images in an appropriately labeled training set. For example, given N training images, stored in the matrix X, with the ground truth 4-dimensional bounding box vector for image x i being y i, and G(x i ; w) being the CNN output vector for image x i given connection weights w, an appropriate loss function would be: DISPLAYFORM0The CNN will produce good estimates of the training image bounding boxes when this loss function is minimized with regard to w. Network weight parameters that minimize this loss, w *, may be sought through stochastic gradient decent, incrementally updating w according to the gradient of (X, w) with regard to w. A primary drawback of this approach is that it requires a large and representative sample of images with ground truth bounding box information. Consider that, once weights are found, the gradient of (X, w *) with regard to X would provide information about the sensitivity of the bounding box loss function with regard to the pixels in the images. This gradient can be calculated as efficiently as the gradient of the loss with regard to the weights, with both depending on the gradient of G(x i ; w) with regard to a subset of its arguments. This means that the gradient of G(x i ; w *) with regard to x i can be efficiently computed, and that gradient would capture the sensitivity of bounding box coordinates with regard to the specific pixels in image x i. Note that this gradient can be calculated for images beyond those in the training set. Knowing which pixels in a novel image play an important role in determining the bounding box provides useful information for object localization. Using this calculation to address the object localization task makes little sense, however, as G(x i ; w *) provides an estimate of object location without a need to consider pixel sensitivity. Rather than training a deep CNN to output bounding boxes, requiring extensive labeled data, we propose calculating the same gradient for a different network -one successfully trained to perform image classification. If we now see G(x i ; w *) as the output of such an image classification network, its gradient with regard to x i would provide information about the sensitivity of the assigned category to individual pixels. Pixels with the largest absolute values of this derivative will, around the input x i, produce the largest changes in the classification decision of the CNN. This can be seen as one measure of how important pixels are for classifying the object in the image. Consider that the object class output is not immediately affected by changes to pixels with a derivative of zero. The calculation of this gradient can be performed as efficiently as a single "backward pass" through the classification network. This is well illustrated by considering the case of a simple layered backpropagation network BID18 in which the "net input" of unit i, η i, is a weighted sum of the activations of units in the previous layer, and the activation of unit i is g(η i), where g(·) is the unit activation function. In this case, we can define a sensitivity value for each unit, s i, as the derivative of the network output with regard to η i. Using the chain rule of calculus, it is easy to show that the sensitivity of an output unit is g (ηi), and, for units in earlier layers the gradients are computed as follows: DISPLAYFORM1 where k iterates over all units in the immediately downstream layer from unit i and w ki is the connection weight from unit i to unit k. This calculation may be performed, layer by layer, from outputs to inputs, until s i values for each pixel input unit are available. This demonstrates how efficiently pixel sensitivity values can be calculated for a given classified image. Of course, there are currently a variety of software packages that include tools for calculating gradients. In the evaluation of our approach in Section 3, we report using the tools provided by TensorFlow BID0. We have proposed using a previously trained image classification network as a source of information about object location, focusing on the gradient of the network output with regard to image pixels. It is interesting to note that it might not be necessary to perform the sensitivity calculation using the full classification network. There is a growing body of research that suggests that, in a well trained image classification CNN, the features that are extracted at the "attention map" layer (i.e., the output of the last convolutional layer) tend to be generally useful for learning a variety of image analysis tasks Razavian et al. FORMULA1;. Inspired by these , we have investigated the possibility of substituting the gradient of the classifier output with regard to pixels with the gradient of the attention map with regard to pixels. This avoids calculations involving final fully connected layers and any classification softmax layer. Generating image sensitivity maps from the attention map layer is slightly faster than our original proposal, but, more importantly, it is possible that general knowledge about object location might be found in the attention map, and using the attention map as the basis of the sensitivity map might actually generalize beyond the categories on which the image classiciation CNN was trained. We have not yet done a formal comparison of these two approaches to constructing the sensitivity map, but example using both approaches are reported in Section 3. Note that computing the gradients of the aggregated values of the last convolution layer with respect to the input pixels are considered as the Gestalt total which can be computed as follows. DISPLAYFORM0 which A n is the activation map of the last convolution layer, H, W and C are the height, width and channels of the last convolution layer; moreover, GT indicates as Gestalt Total. The sensitivity map calculations that have been described, so far, provide a scalar sensitivity value for each input to the image classification deep CNN. Color images, however, are regularly provided to such networks using multiple inputs per image pixel, often encoding each pixel over three color channels. Thus, the gradient calculation will actually produce three sensitivity values for each pixel. Since we hope to produce a sensitivity map that focuses in a general way on location information, it seems reasonable to aggregate the three sensitivity values into one. Since the direction of the sensitivity relationship with the class output is irrelevant, a good first step is to take the absolute value of each derivative. Given that dependence on even a single color channel suggests that a pixel is important for identifying the object, an argument can be made that a pixel should be labeled with the maximum of the three absolute derivatives. Alternatively, it could be argued that all color channels should be taken into account when producing the sensitivity map, in which case it might be better to average the three absolute derivatives. We have explored both of these aggregation methods, with appearing in Section 3. Object localization algorithms typically output the four coordinates of a bounding box to communicate the location of the target object. Such a bounding box is not intrinsic to a sensitivity map, however. Heuristic techniques could be used to identify a rectangular region that captures the majority of the high sensitivity pixels, while avoiding low sensitivity pixels, but we have taken a different approach. We have opted to learn a linear mapping from sensitivity maps to bounding box coordinates, using training images with ground truth location information. It is important to note that learning this mapping is not the same as learning to map from the original images to bounding box coordinates, as has been done in some other object localization systems. Sensitivity maps contain much less information than the original images, so using the sensitivity maps as inputs both reduces the dimensionality of the input to this mapping and makes for a more simple functional relationship between pixels and bounding box coordinates. We expect that this simplification will allow the mapping to bounding box coordinates to be successfully learned using a far smaller set of training images labeled with ground truth object locations. Indeed, we expect that a simple linear mapping could perform well. Formally, we define the parameters of the linear mapping to the four bounding box coordinates as a 4 × M matrix,Ŵ, (where M is the number of pixels in an image) and a 4-dimensional vector of"bias weights",ŵ. Given a sensitivity map, s, the output is (Ŵ s +ŵ). Given a training set of N images, the mapping is found by minimizing the following objective function with regard toŴ and w: DISPLAYFORM0 where s i is the sensitivity map for the i th image, and B i,j is the j th coordinate of the bounding box for the i th image. This learning process amounts to four independent linear regression problems, which can be solved efficiently. Once learned, mapping from sensitivity maps to bounding box coordinates can be done very quickly. With sensitivity map formation requiring only a single backward pass through the image classification network, the whole process -from image, to classification, to sensitivity map, to bounding box -can be performed in little more than twice the time it takes for the network to do object recognition. The code and the sensitivity map of on imageNet as well as PASCAL VOC dataset will be publicly available. We evaluated our proposed method for object localization on two challenging data sets: the PASCAL VOC 2007 BID8 data set and the ImageNet 2012 BID5 data set. The PASCAL VOC 2007 data set was selected due to its use in the existing object localization literature. The ImageNet data set is one of the largest publicly available data sets. It also contains many images annotated with ground truth bounding boxes. We followed the literature with regard to the evaluation criterion applied to our method, using CorLoc, which has been used for weakly supervised localization. The CorLoc metric is defined as the percentage of images in a data set that are correctly localized based on the PASCAL criterion, in which a given localization is considered correct if and only if the intersection over union (IOU) area of the predicted and ground truth bounding boxes is greater than one half: DISPLAYFORM0... where β p is the predicted bounding box and β gt is the ground truth bounding box BID27. To demonstrate that our approach works with an image classification deep CNN that was in no way specialized for our localization method, we opted to use a publicly available network. We used the VGG16 network, shown in FIG0, fully trained BID23. This network provides ImageNet object classes as output, allowing us to calculate sensitivity maps based on the network classification when examining ImageNet data. For the PASCAL VOC 2007 data set, we used the previously described method of calculating derivatives based on the attention map of VGG16, since there is not consistent class correspondnce between the PASCAL VOC 2007 classes and the classes on which VGG16 was trained. To produce sensitivity maps for the PASCAL VOC 2007 data set, we aggregated across color channels by using the maximum absolute derivative across the three inputs for each pixel. For the ImageNet data set, we averaged the absolute derivatives across the three inputs in order to produce pixel sensitivity values. For generating sensitivity maps, we have used a pretrained vgg 16 network, we have used the whole network architecture while we were experimenting on ImageNet dataset, otherwise we have removed the last 3 fully connected layers and computed the Gestalt Total by the last convolution layer. The derivatives in either case computed using just one backward pass to the original pixels. For learning bounding boxes we have used the aggregated sensitivity maps as an input. To learn the mapping from sensitivity maps to bounding box coordinates, we performed linear regression using stochastic gradient decent. Updates were performed in batches of 2,048. The learning rate was initialized to 0.1 and decayed by a factor of 10 every 10,000 iterations. The experiment run on 1 GPU for 4 days. The full PASCAL VOC 2007 data set includes 12,608 training set images and an equal number of testing set images BID8. Each image contains an object of 1 of 20 different categories. We applied our object localization method to this full data set. However, we were unable to find published localization performance data for other methods applied to the full data set, which we might use for comparison to our approach. Work reported in Tang et al. BID27 provides performance data on 6 of the classes: aeroplane, bicycle, boat, bus, horse, and motorbike. Performance on these same classes have also been reported by others Russell et al. FORMULA1; BID4; BID6. TAB1 compares the localization performance of our method with that of other approaches. Note that our method, while being very fast, outperforms the comparison algorithms. Examples of the bounding boxes selected by our method, compared to ground truth, for all 20 classes in the PASCAL VOC 2007 data set are shown in FIG1. Qualitatively, it appears as if our approach is most accurate when there is a single target object with little crowding. However, if the target object is small and in a crowded region of the image, performance is less reliable. While speed is an important property of our method, as is the reuse of classification training for localization, we compared our approach to data from some slower state-of-the-art deep learning techniques for localization that do not necessarily have these properties. We compared our method to R-CNN BID11, , and Poselets BID1. These were chosen due to the ready availability of published localization for these alternative methods on the PASCAL VOC 2007 data set, with the measure of performance being Average CorLoc (or mean Average Precision, mAP). The comparison are given in Table??. Several of the comparison methods display better localization performance than our approach, but it is important to keep in mind that the comparison cases had some important advantages, including taking the time to use a sliding window and access to the class labels on which the network was trained. Recall that our sensitivity maps were produced, in this case, by calculating the sensitivity of the network attention map activity to pixel values. Thus, this comparison illustrates trade-offs between speed, performance, and generalization. Note that as one of the reviewers mentioned it would be worth looking at the if we just use the sensitivity maps and heuristics to draw bounding boxes out of objects. For this experiment, we used a Gaussian smoothing filter to smooth out the sensitivity maps and then we picked top %20 pixels and draw the bounding box out of those pixels as other researchers obtained this experiment before BID28 BID20. Based on our observation we noticed that it could damage the mean CorLoc by %3 in our best observations. However, this process is highly depends on the smoothing σ parameter. The obtained from different σ values are reported in TAB3. ImageNet is a large image data set that has been systematically organized by object category BID5. We executed a large scale evaluation of our approach by using all images in ImageNet that are annotated with ground truth localization information. This subset contains 300,916 images involving 478 object classes. We divided this data set into a training set, a test set, and a validation set by sampling without replacement (i.e., the intersection between each pair of the three sets was empty). There were 225,687 images (75%) in the training set, and there were 45,137 images in each of the other two sets. We compared the performance of our approach with two methods discussed in Tang et al. BID27 for which ImageNet are explicitly reported: Top Objectiveness Box & CoLocalization. Also, we noted that many images in this data set presented the target object in the middle of the image, providing a bias that could be leveraged by learned localization systems. Thus, as a baseline of performance, we calculated the CorLoc performance for a system that blindly offered the same bounding box in the middle of the image, with average size, for every input. The are shown in TAB4. Once again, note the relatively high accuracy performance of our efficient method. Also note that the baseline was comfortingly low. As might be expected, performance varies with class. Our algorithm appears to do well on some objects, such as balls and dogs. One might suspect that failures arise in the linear mapping from sensitivity maps to bounding box coordinates, but a perusal of the sensitivity maps, themselves, suggests that the pixel sensitivity values vary in utility across different object categories. Still, our method performs fairly well across the classes. Note that the IOU does not fall below 0.62 for any class. This suggests that, while some individual images may be problematic, the overall performance for each class is quite good. This universally strong class-specific performance is also displayed in TAB4. Figure 4: Results of the proposed method on different object categories from the ImageNet data set. Each row shows 9 examples in one class. The green boxes are the ground truth, and the red ones are the predicted bounding boxes. The sensitivity analysis approach gives us the sensitivity of every single pixels in all channels in the RGB images and since we are in need of locations we need to aggregate among channels. We proposed two methods, an average function, a maximum function. The first approach is to taking average among channels and the second method is to pick up the maximum numbers among channels. We didn't notice significant difference between these two methods in the localization performance, the only information that comes to light is generating sensitivity maps based on average function is a bit smoother in visual sense than maximum function. The CorLoc between average and maximum aggregation function on ImageNet dataset are 68.7 and 67.9 respectively and the of these two aggregation operators on PASCAL VOC dataset is 39.2 and 40.1,respectively. To analyze our object localization approach since it highly depends to the hardware parameters and network architecture, we decided to analyze the speed in term of forward and backward passes. Our approach only needs two passes, one forward pass for the classification and one backward pass for localization. If we consider each forward/backward pass as n operations we can say our approach is O(N 2) + which means it needs one forward pass, one backward pass and one inference from the linear model. We have presented an approach to object localization based on performing a sensitivity analysis of a previously trained image classification deep CNN. Our method is fast enough to be used in online applications, and it demonstrates accuracy that is superior to some methods that are much slower. It is likely that even better accuracy could be had by incorporating sensitivity analysis information into a more sophisticated bounding box estimator. As previously noted, the idea of using sensitivity information has appeared in previously published work. There are ways in which the reported in this paper are distinct, however. We have moved beyond visualization of network function using sensitivity (or saliency) BID24 to performing direct comparisons between different methods on the localization task. We have shown that using a fast and simple measure of sensitivity can produce comparable performance to that of much slower methods. Our approach produces good generalization without modifying the classification network, as is done in Class Activation Mapping (CAM) BID28. With our PASCAL VOC 2007 , we have shown that our approach can successfully be applied to attention maps, even when the image contains objects belonging to a class on which the classification network was not trained, distinguishing it from. In short, we have demonstrated the power of a simple sensitivity measure for performing localization. Note that our approach may be used with image classifiers other than CNNs. The proposed sensitivity analysis can be conducted on any differentiable classifier, though performance will likely depend on classifer specifics. Indeed, at a substantial time cost, even a black box classifier could be approximately analyzed by making small changes to pixels and observing the effects on activation patterns. The proposed approach is quite general. Indeed, we are currently working on applying sensitivity analysis to deep networks trained on other tasks, with the goal of interpreting network performance on the current input in a useful way. Thus, we see a potentially large range of uses for sensitivity analysis in neural network applications.

Proposing a novel object localization(detection) approach based on interpreting the deep CNN using internal representation and network's thoughts

represent this text to find a much much longer article about it with say hundreds-thousands of words

For instance, <<We propose a differentiable architecture search algorithm for both convolutional and recurrent networks, achieving competitive performance with the state of the art using orders of magnitude less computation resources.>>  to "This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques. Discovering state-of-the-art neural network architectures requires substantial effort of human experts. Recently, there has been a growing interest in developing algorithmic solutions to automate the manual process of architecture design. The automatically searched architectures have achieved highly competitive performance in tasks such as image classification BID35 BID36 BID13 a; BID26 and object detection BID36.The best existing architecture search algorithms are computationally demanding despite their remarkable performance. For example, obtaining a state-of-the-art architecture for CIFAR-10 and ImageNet required 2000 GPU days of reinforcement learning (RL) BID36 or 3150 GPU days of evolution BID26. Several approaches for speeding up have been proposed, such as imposing a particular structure of the search space BID13 a), weights or performance prediction for each individual architecture and weight sharing/inheritance across multiple architectures BID0 BID24; ), but the fundamental challenge of scalability remains. An inherent cause of inefficiency for the dominant approaches, e.g. based on RL, evolution, MCTS BID20, SMBO BID12 or Bayesian optimization BID9, is the fact that architecture search is treated as a black-box optimization problem over a discrete domain, which leads to a large number of architecture evaluations required. In this work, we approach the problem from a different angle, and propose a method for efficient architecture search called DARTS (Differentiable ARchiTecture Search). Instead of searching over a discrete set of candidate architectures, we relax the search space to be continuous, so that the architecture can be optimized with respect to its validation set performance by gradient descent. The data efficiency of gradient-based optimization, as opposed to inefficient black-box search, allows DARTS to achieve competitive performance with the state of the art using orders of magnitude less computation resources. It also outperforms another recent efficient architecture search method, ENAS BID24. Notably, DARTS is simpler than many existing approaches as it does not involve controllers BID35; BID36 BID24 BID33, hypernetworks or performance predictors BID12 ), yet it is generic enough handle both convolutional and recurrent architectures. The idea of searching architectures within a continuous domain is not new BID27; BID30 BID28, but there are several major distinctions. While prior works seek to fine-tune a specific aspect of an architecture, such as filter shapes or branching patterns in a convolutional network, DARTS is able to learn high-performance architecture building blocks with complex graph topologies within a rich search space. Moreover, DARTS is not restricted to any specific architecture family, and is applicable to both convolutional and recurrent networks. In our experiments (Sect. 3) we show that DARTS is able to design a convolutional cell that achieves 2.76 ± 0.09% test error on CIFAR-10 for image classification using 3.3M parameters, which is competitive with the state-of-the-art by regularized evolution BID26 obtained using three orders of magnitude more computation resources. The same convolutional cell also achieves 26.7% top-1 error when transferred to ImageNet (mobile setting), which is comparable to the best RL method BID36. On the language modeling task, DARTS efficiently discovers a recurrent cell that achieves 55.7 test perplexity on Penn Treebank (PTB), outperforming both extensively tuned LSTM BID17 and all the existing automatically searched cells based on NAS BID35 and ENAS BID24.Our contributions can be summarized as follows:• We introduce a novel algorithm for differentiable network architecture search based on bilevel optimization, which is applicable to both convolutional and recurrent architectures.• Through extensive experiments on image classification and language modeling tasks we show that gradient-based architecture search achieves highly competitive on CIFAR-10 and outperforms the state of the art on PTB. This is a very interesting , considering that so far the best architecture search methods used non-differentiable search techniques, e.g. based on RL BID36 or evolution BID26 BID13 ).• We achieve remarkable efficiency improvement (reducing the cost of architecture discovery to a few GPU days), which we attribute to the use of gradient-based optimization as opposed to non-differentiable search techniques.• We show that the architectures learned by DARTS on CIFAR-10 and PTB are transferable to ImageNet and WikiText-2, respectively. The implementation of DARTS is available at https://github.com/quark0/darts 2 DIFFERENTIABLE ARCHITECTURE SEARCH We describe our search space in general form in Sect. 2.1, where the computation procedure for an architecture (or a cell in it) is represented as a directed acyclic graph. We then introduce a simple continuous relaxation scheme for our search space which leads to a differentiable learning objective for the joint optimization of the architecture and its weights (Sect. 2.2). Finally, we propose an approximation technique to make the algorithm computationally feasible and efficient (Sect. 2.3). Following BID36; BID26; BID12 BID25, we search for a computation cell as the building block of the final architecture. The learned cell could either be stacked to form a convolutional network or recursively connected to form a recurrent network. A cell is a directed acyclic graph consisting of an ordered sequence of N nodes. Each node x (i) is a latent representation (e.g. a feature map in convolutional networks) and each directed edge (i, j) is associated with some operation o (i,j) that transforms x (i). We assume the cell to have two input nodes and a single output node. For convolutional cells, the input nodes are defined as the cell outputs in the previous two layers BID36. For recurrent cells, these are defined as the input at the current step and the state carried from the previous step. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. Each intermediate node is computed based on all of its predecessors: A special zero operation is also included to indicate a lack of connection between two nodes. The task of learning the cell therefore reduces to learning the operations on its edges. DISPLAYFORM0 Let O be a set of candidate operations (e.g., convolution, max pooling, zero) where each operation represents some function o(·) to be applied to x (i). To make the search space continuous, we relax the categorical choice of a particular operation to a softmax over all possible operations: DISPLAYFORM0 where the operation mixing weights for a pair of nodes (i, j) are parameterized by a vector α (i,j) of dimension |O|. The task of architecture search then reduces to learning a set of continuous variables α = α (i,j), as illustrated in FIG0. At the end of search, a discrete architecture can be obtained by replacing each mixed operationō (i,j) with the most likely operation, i.e., o (i,j) = argmax o∈O α (i,j) o. In the following, we refer to α as the (encoding of the) architecture. After relaxation, our goal is to jointly learn the architecture α and the weights w within all the mixed operations (e.g. weights of the convolution filters). Analogous to architecture search using RL BID35 BID36 BID24 or evolution BID13 BID26 where the validation set performance is treated as the reward or fitness, DARTS aims to optimize the validation loss, but using gradient descent. Denote by L train and L val the training and the validation loss, respectively. Both losses are determined not only by the architecture α, but also the weights w in the network. The goal for architecture search is to find α * that minimizes the validation loss L val (w *, α *), where the weights w * associated with the architecture are obtained by minimizing the training loss w * = argmin w L train (w, α *).This implies a bilevel optimization problem with α as the upper-level variable and w as the lower-level variable: DISPLAYFORM1 DISPLAYFORM2 The nested formulation also arises in gradient-based hyperparameter optimization BID16 BID22 BID2, which is related in a sense that the architecture α could be viewed as a special type of hyperparameter, although its dimension is substantially higher than scalar-valued hyperparameters such as the learning rate, and it is harder to optimize. Create a mixed operationō DISPLAYFORM0 (ξ = 0 if using first-order approximation) 2. Update weights w by descending ∇ w L train (w, α) Derive the final architecture based on the learned α. Evaluating the architecture gradient exactly can be prohibitive due to the expensive inner optimization. We therefore propose a simple approximation scheme as follows: DISPLAYFORM0 where w denotes the current weights maintained by the algorithm, and ξ is the learning rate for a step of inner optimization. The idea is to approximate w * (α) by adapting w using only a single training step, without solving the inner optimization (equation 4) completely by training until convergence. Related techniques have been used in meta-learning for model transfer BID1, gradientbased hyperparameter tuning BID15 and unrolled generative adversarial networks BID19. Note equation 6 will reduce to ∇ α L val (w, α) if w is already a local optimum for the inner optimization and thus ∇ w L train (w, α) = 0.The iterative procedure is outlined in Alg. 1. While we are not currently aware of the convergence guarantees for our optimization algorithm, in practice it is able to reach a fixed point with a suitable choice of ξ 1. We also note that when momentum is enabled for weight optimisation, the one-step unrolled learning objective in equation 6 is modified accordingly and all of our analysis still applies. Applying chain rule to the approximate architecture gradient (equation 6) yields DISPLAYFORM1 where w = w − ξ∇ w L train (w, α) denotes the weights for a one-step forward model. The expression above contains an expensive matrix-vector product in its second term. Fortunately, the complexity can be substantially reduced using the finite difference approximation. Let be a small scalar 2 and DISPLAYFORM2 Evaluating the finite difference requires only two forward passes for the weights and two backward passes for α, and the complexity is reduced from O(|α||w|) to O(|α| + |w|).First-order Approximation When ξ = 0, the second-order derivative in equation 7 will disappear. In this case, the architecture gradient is given by ∇ α L val (w, α), corresponding to the simple heuristic of optimizing the validation loss by assuming the current w is the same as w * (α). This leads to some speed-up but empirically worse performance, according to our experimental in TAB0. In the following, we refer to the case of ξ = 0 as the first-order approximation, and refer to the gradient formulation with ξ > 0 as the second-order approximation. To form each node in the discrete architecture, we retain the top-k strongest operations (from distinct nodes) among all non-zero candidate operations collected from all the previous nodes. The strength of an operation is defined as DISPLAYFORM0. To make our derived architecture comparable with 1 A simple working strategy is to set ξ equal to the learning rate for w's optimizer. 2 We found = 0.01/ ∇ w L val (w, α) 2 to be sufficiently accurate in all of our experiments. DISPLAYFORM1. The analytical solution for the corresponding bilevel optimization problem is (α *, w *) =, which is highlighted in the red circle. The dashed red line indicates the feasible set where constraint equation 4 is satisfied exactly (namely, weights in w are optimal for the given architecture α). The example shows that a suitable choice of ξ helps to converge to a better local optimum. those in the existing works, we use k = 2 for convolutional cells BID36 BID12 BID26 and k = 1 for recurrent cells BID24.The zero operations are excluded in the above for two reasons. First, we need exactly k non-zero incoming edges per node for fair comparison with the existing models. Second, the strength of the zero operations is underdetermined, as increasing the logits of zero operations only affects the scale of the ing node representations, and does not affect the final classification outcome due to the presence of batch normalization BID8. Our experiments on CIFAR-10 and PTB consist of two stages, architecture search (Sect. 3.1) and architecture evaluation (Sect. 3.2). In the first stage, we search for the cell architectures using DARTS, and determine the best cells based on their validation performance. In the second stage, we use these cells to construct larger architectures, which we train from scratch and report their performance on the test set. We also investigate the transferability of the best cells learned on CIFAR-10 and PTB by evaluating them on ImageNet and WikiText-2 (WT2) respectively. We include the following operations in O: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero. All operations are of stride one (if applicable) and the convolved feature maps are padded to preserve their spatial resolution. We use the ReLU-Conv-BN order for convolutional operations, and each separable convolution is always applied twice BID36 BID26 BID12.Our convolutional cell consists of N = 7 nodes, among which the output node is defined as the depthwise concatenation of all the intermediate nodes (input nodes excluded). The rest of the setup follows BID36; BID12 BID26, where a network is then formed by stacking multiple cells together. The first and second nodes of cell k are set equal to the outputs of cell k − 2 and cell k − 1, respectively, and 1 × 1 convolutions are inserted as necessary. Cells located at the 1/3 and 2/3 of the total depth of the network are reduction cells, in which all the operations adjacent to the input nodes are of stride two. The architecture encoding therefore is (α normal, α reduce), where α normal is shared by all the normal cells and α reduce is shared by all the reduction cells. Detailed experimental setup for this section can be found in Sect. A.1.1. Our set of available operations includes linear transformations followed by one of tanh, relu, sigmoid activations, as well as the identity mapping and the zero operation. The choice of these candidate operations follows BID35; BID24. Each architecture snapshot is re-trained from scratch using the training set (for 100 epochs on CIFAR-10 and for 300 epochs on PTB) and then evaluated on the validation set. For each task, we repeat the experiments for 4 times with different random seeds, and report the median and the best (per run) validation performance of the architectures over time. As references, we also report the (under the same evaluation setup; with comparable number of parameters) of the best existing cells discovered using RL or evolution, including NASNet-A (2000 GPU days), AmoebaNet-A (3150 GPU days) BID26 and ENAS (0.5 GPU day) BID24.function, as done in the ENAS cell BID24. The rest of the cell is learned. Other settings are similar to ENAS, where each operation is enhanced with a highway bypass BID34 and the cell output is defined as the average of all the intermediate nodes. As in ENAS, we enable batch normalization in each node to prevent gradient explosion during architecture search, and disable it during architecture evaluation. Our recurrent network consists of only a single cell, i.e. we do not assume any repetitive patterns within the recurrent architecture. Detailed experimental setup for this section can be found in Sect. A.1.2. To determine the architecture for final evaluation, we run DARTS four times with different random seeds and pick the best cell based on its validation performance obtained by training from scratch for a short period (100 epochs on CIFAR-10 and 300 epochs on PTB). This is particularly important for recurrent cells, as the optimization outcomes can be initialization-sensitive FIG2.To evaluate the selected architecture, we randomly initialize its weights (weights learned during the search process are discarded), train it from scratch, and report its performance on the test set. We note the test set is never used for architecture search or architecture selection. Detailed experimental setup for architecture evaluation on CIFAR-10 and PTB can be found in Sect. A.2.1 and Sect. A.2.2, respectively. Besides CIFAR-10 and PTB, we further investigated the transferability of our best convolutional cell (searched on CIFAR-10) and recurrent cell (searched on PTB) by evaluating them on ImageNet (mobile setting) and WikiText-2, respectively. More details of the transfer learning experiments can be found in Sect. A.2.3 and Sect. A.2.4. BID33 3.54 39.8 96 8 RL AmoebaNet-A BID26 3.34 ± 0.06 3.2 3150 19 evolution AmoebaNet-A + cutout BID26 ) † 3.12 3.1 3150 19 evolution AmoebaNet-B + cutout BID26 2.55 ± 0.05 2.8 3150 19 evolution Hierarchical evolution BID13 3.75 ± 0.12 15.7 300 6 evolution PNAS BID12 3.41 ± 0.09 3.2 225 8 SMBO ENAS + cutout BID24 2.89 4.6 0.5 6 RL ENAS + cutout BID24 3.00 ± 0.14 3.3 1.5 7 gradient-based DARTS (second order) + cutout 2.76 ± 0.09 3.3 4 7 gradient-based * Obtained by repeating ENAS for 8 times using the code publicly released by the authors. The cell for final evaluation is chosen according to the same selection protocol as for DARTS. † Obtained by training the corresponding architectures using our setup. ‡ Best architecture among 24 samples according to the validation error after 100 training epochs. Table 2: Comparison with state-of-the-art language models on PTB (lower perplexity is better). Note the search cost for DARTS does not include the selection cost (1 GPU day) or the final evaluation cost by training the selected architecture from scratch (3 GPU days). BID18 60.7 58.8 24 --manual LSTM + skip connections BID17 60.9 58.3 24 --manual LSTM + 15 softmax experts BID31 BID23 publicly released by the authors. † Obtained by training the corresponding architecture using our setup. ‡ Best architecture among 8 samples according to the validation perplexity after 300 training epochs. BID36 27.2 8.7 5.3 488 2000 RL NASNet-C BID36 27.5 9.0 4.9 558 2000 RL AmoebaNet-A BID26 25.5 8.0 5.1 555 3150 evolution AmoebaNet-B BID26 26.0 8.5 5.3 555 3150 evolution AmoebaNet-C BID26 24.3 7.6 6.4 570 3150 evolution PNAS BID12 25. The CIFAR-10 for convolutional architectures are presented in TAB0. Notably, DARTS achieved comparable with the state of the art BID36 BID26 while using three orders of magnitude less computation resources (i.e. 1.5 or 4 GPU days vs 2000 GPU days for NASNet and 3150 GPU days for AmoebaNet). Moreover, with slightly longer search time, DARTS outperformed ENAS BID24 ) by discovering cells with comparable error rates but less parameters. The longer search time is due to the fact that we have repeated the search process four times for cell selection. This practice is less important for convolutional cells however, because the performance of discovered architectures does not strongly depend on initialization FIG2.Alternative Optimization Strategies To better understand the necessity of bilevel optimization, we investigated a simplistic search strategy, where α and w are jointly optimized over the union of the training and validation sets using coordinate descent. The ing best convolutional cell (out of 4 runs) yielded 4.16 ± 0.16% test error using 3.1M parameters, which is worse than random search. In the second experiment, we optimized α simultaneously with w (without alteration) using SGD, again over all the data available (training + validation). The ing best cell yielded 3.56 ± 0.10% test error using 3.0M parameters. We hypothesize that these heuristics would cause α (analogous to hyperparameters) to overfit the training data, leading to poor generalization. Note that α is not directly optimized on the training set in DARTS. Table 2 presents the for recurrent architectures on PTB, where a cell discovered by DARTS achieved the test perplexity of 55.7. This is on par with the state-of-the-art model enhanced by a mixture of softmaxes BID31, and better than all the rest of the architectures that are either manually or automatically discovered. Note that our automatically searched cell outperforms the extensively tuned LSTM BID17, demonstrating the importance of architecture search in addition to hyperparameter search. In terms of efficiency, the overall cost (4 runs in total) is within 1 GPU day, which is comparable to ENAS and significantly faster than NAS BID35.It is also interesting to note that random search is competitive for both convolutional and recurrent models, which reflects the importance of the search space design. Nevertheless, with comparable or less search cost, DARTS is able to significantly improve upon random search in both cases (2.76 ± 0.09 vs 3.29 ± 0.15 on CIFAR-10; 55.7 vs 59.4 on PTB). TAB4 show that the cell learned on CIFAR-10 is indeed transferable to ImageNet. It is worth noticing that DARTS achieves competitive performance with the state-of-the-art RL method BID36 while using three orders of magnitude less computation resources. TAB6 shows that the cell identified by DARTS transfers to WT2 better than ENAS, although the overall are less strong than those presented in Table 2 for PTB. The weaker transferability between PTB and WT2 (as compared to that between CIFAR-10 and ImageNet) could be explained by the relatively small size of the source dataset (PTB) for architecture search. The issue of transferability could potentially be circumvented by directly optimizing the architecture on the task of interest. BID4 -68.9 --manual LSTM BID18 69.1 66.0 33 -manual LSTM + skip connections BID17 69.1 65.9 24 -manual LSTM + 15 softmax experts BID31 66 We presented DARTS, a simple yet efficient architecture search algorithm for both convolutional and recurrent networks. By searching in a continuous space, DARTS is able to match or outperform the state-of-the-art non-differentiable architecture search methods on image classification and language modeling tasks with remarkable efficiency improvement by several orders of magnitude. There are many interesting directions to improve DARTS further. For example, the current method may suffer from discrepancies between the continuous architecture encoding and the derived discrete architecture. This could be alleviated, e.g., by annealing the softmax temperature (with a suitable schedule) to enforce one-hot selection. It would also be interesting to explore performance-aware architecture derivation schemes based on the one-shot model learned during the search process. A EXPERIMENTAL DETAILS A.1 ARCHITECTURE SEARCH A.1.1 CIFAR-10Since the architecture will be varying throughout the search process, we always use batch-specific statistics for batch normalization rather than the global moving average. Learnable affine parameters in all batch normalizations are disabled during the search process to avoid rescaling the outputs of the candidate operations. To carry out architecture search, we hold out half of the CIFAR-10 training data as the validation set. A small network of 8 cells is trained using DARTS for 50 epochs, with batch size 64 (for both the training and validation sets) and the initial number of channels 16. The numbers were chosen to ensure the network can fit into a single GPU. We use momentum SGD to optimize the weights w, with initial learning rate η w = 0.025 (annealed down to zero following a cosine schedule without restart BID14), momentum 0.9, and weight decay 3 × 10 −4. We use zero initialization for architecture variables (the α's in both the normal and reduction cells), which implies equal amount of attention (after taking the softmax) over all possible ops. At the early stage this ensures weights in every candidate op to receive sufficient learning signal (more exploration). We use Adam BID10 as the optimizer for α, with initial learning rate η α = 3 × 10 −4, momentum β = (0.5, 0.999) and weight decay 10 −3. The search takes one day on a single GPU 3.A.1.2 PTB For architecture search, both the embedding and the hidden sizes are set to 300. The linear transformation parameters across all incoming operations connected to the same node are shared (their shapes are all 300 × 300), as the algorithm always has the option to focus on one of the predecessors and mask away the others. Tying the weights leads to memory savings and faster computation, allowing us to train the continuous architecture using a single GPU. Learnable affine parameters in batch normalizations are disabled, as we did for convolutional cells. The network is then trained for 50 epochs using SGD without momentum, with learning rate η w = 20, batch size 256, BPTT length 35, and weight decay 5 × 10 −7. We apply variational dropout BID3 of 0.2 to word embeddings, 0.75 to the cell input, and 0.25 to all the hidden nodes. A dropout of 0.75 is also applied to the output layer. Other training settings are identical to those in BID18; BID31. Similarly to the convolutional architectures, we use Adam for the optimization of α (initialized as zeros), with initial learning rate η α = 3 × 10 −3, momentum β = (0.9, 0.999) and weight decay 10 −3. The search takes 6 hours on a single GPU. A large network of 20 cells is trained for 600 epochs with batch size 96. The initial number of channels is increased from 16 to 36 to ensure our model size is comparable with other baselines in the literature (around 3M). Other hyperparameters remain the same as the ones used for architecture search. Following existing works BID24 BID36 BID12 BID26, additional enhancements include cutout , path dropout of probability 0.2 and auxiliary towers with weight 0.4. The training takes 1.5 days on a single GPU with our implementation in PyTorch BID21. Since the CIFAR are subject to high variance even with exactly the same setup BID13, we report the mean and standard deviation of 10 independent runs for our full model. To avoid any discrepancy between different implementations or training settings (e.g. the batch sizes), we incorporated the NASNet-A cell BID36 and the AmoebaNet-A cell BID26 into our training framework and reported their under the same settings as our cells."

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent networks are equivalent to trellis networks with special sparsity structure in their weight matrices. Thus trellis networks with general weight matrices generalize truncated recurrent networks. We leverage these connections to design high-performing trellis networks that absorb structural and algorithmic elements from both recurrent and convolutional models. Experiments demonstrate that trellis networks outperform the current state of the art methods on a variety of challenging benchmarks, including word-level language modeling and character-level language modeling tasks, and stress tests designed to evaluate long-term memory retention. The code is available at https://github.com/locuslab/trellisnet.

Represent the short summary for finding the corresponding scientific article (long)

Trellis networks are a new sequence modeling architecture that bridges recurrent and convolutional models and sets a new state of the art on word- and character-level language modeling.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Are HMMs a special case of RNNs? We investigate a series of architectural transformations between HMMs and RNNs, both through theoretical derivations and empirical hybridization and provide new insights.


Question: Represent the scientific article to find a short TLDR!

Question: We propose an end-to-end framework for training domain specific models (DSMs) to obtain both high accuracy and computational efficiency for object detection tasks. DSMs are trained with distillation and focus on achieving high accuracy at a limited domain (e.g. fixed view of an intersection). We argue that DSMs can capture essential features well even with a small model size, enabling higher accuracy and efficiency than traditional techniques. In addition, we improve the training efficiency by reducing the dataset size by culling easy to classify images from the training set. For the limited domain, we observed that compact DSMs significantly surpass the accuracy of COCO trained models of the same size. By training on a compact dataset, we show that with an accuracy drop of only 3.6%, the training time can be reduced by 93%. The framework is based on knowledge distillation BID0 but targets to reduce the accuracy gap 23 between student and teacher models by training the student using a restricted class of domain specific 24 images. Since such training may be conducted on edge-devices, we improve the training efficiency 25 by culling easy-to-classify images with small accuracy penalty. This paper's contribution is summarized below. • We propose an end-to-end framework for training domain specific models (DSMs) to mit-28 igate the tradeoff between object-detection accuracy and computational efficiency. To the 29 best of our knowledge, this is the first successful demonstration of training DSMs for object 30 detection tasks. • By training resnet18-based Faster-RCNN DSMs, we observed a 19.7% accuracy (relative DISPLAYFORM0 compute L train (i) from label(i) and pred(i). Collect Detection ← DSM.predict(image) Figure 1: Object detection of the test image, before and after domain specific training.• Since edge devices will have limited resources, we propose culling the training dataset to 35 significantly reduce the computation resource required for the training. Only training data 36 that has high utility in training is added. This filtering allows us to reduce training time by 93% with an accuracy loss of only 3.6%. DSM framework to train compact models with dataset constructed by domain-specific data. As illustrated in Algorithm 1, our DSM framework consists of preparation of the data and training of the DSM. A large challenge when deploying models in surveillance is preparing the training data 48 since manually labelling frames in videos is cumbersome. To overcome this, we label the dataset 49 used to train the DSM by using the predictions of a much larger teacher model with higher accuracy 50 and treating these predictions as ground truth labels. Furthermore, we compare the prediction on 51 image x i made by the teacher to that of the DSM; we determine whether to store the x i and label Teacher.predict(x i) in our compiled dataset Ω. After the training set is compiled, it is used to train 53 the DSM. Training a object detection model can take hours even with a GPU and can be challenging for 55 applications requiring frequent retraining. We exploit the fact that when the DSM is pretrained on 56 large-scale general dataset, it can already provide good predictions for a large chunk of the domain-57 specific data. This procedure develops a compact dataset Ω that is only composed of data that the 58 DSM finds inconsistent with the prediction made by the teacher model. Keeping data x j that both 59 the DSM and teacher detections are consistent is computationally redundant because it does not 60 contribute to gradient signals. We define L train to quantify the consistency between teacher and 61 DSM: images are for training and the later 3600 images for testing. DISPLAYFORM0 Results. As shown in table 1, we first train our res18 DSM using the full N train = 3600 training 78 images for 10 epochs using stochastic-gradient descent with a learn rate of 10 we aim to train the models with only the domain specific data. We show on

Represent this text to find a much much longer article about it with say hundreds-thousands of words

High object-detection accuracy can be obtained by training domain specific compact models and the training can be very short.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.