Deep-n-Cheap: An Automated Search Framework for Low Complexity Deep Learning
Sourya Dey Saikrishna C. Kanala Keith M. Chugg Peter A. Beerel
cs.LG cs.CV stat.ML




Deep-n-Cheap




S. Dey et al.



University of Southern California, Los Angeles CA 90089, USA
{souryade,kanala,chugg,pabeerel}@usc.edu


Deep-n-Cheap: An Automated Search Framework for Low Complexity Deep LearningSupported by Defense Threat Reduction Agency (DTRA), USA.

    Sourya Dey Saikrishna C. Kanala Keith M. Chugg Peter A. Beerel

    December 30, 2023
======================================================================================================================================




We present Deep-n-Cheap – an open-source AutoML framework to search for deep learning models. This search includes both architecture and training hyperparameters, and supports convolutional neural networks and multi-layer perceptrons. Our framework is targeted for deployment on both benchmark and custom datasets, and as a result, offers a greater degree of search space customizability as compared to a more limited search over only pre-existing models from literature. We also introduce the technique of `search transfer', which demonstrates the generalization capabilities of the models found by our framework to multiple datasets.

Deep-n-Cheap includes a user-customizable complexity penalty which trades off performance with training time or number of parameters. Specifically, our framework results in models offering performance comparable to state-of-the-art while taking 1-2 orders of magnitude less time to train than models from other AutoML and model search frameworks. Additionally, this work investigates and develops various insights regarding the search process. In particular, we show the superiority of a greedy strategy and justify our choice of Bayesian optimization as the primary search methodology over random / grid search.







§ INTRODUCTION

Artificial NN in deep learning systems are critical drivers of emerging technologies such as computer vision, text classification, and autonomous applications. In particular, CNN are used for image related tasks while MLP can be used for general purpose classification tasks. Manually designing these NN is challenging since they typically have a large number of interconnected layers <cit.> and require a large number of decisions to be made regarding hyperparameters. These hyperparameters, as opposed to trainable parameters like weights and biases, are not learned by the network. They need to be specified and adjusted by an external entity, the designer. They can be broadly grouped into two categories – a) architectural hyperparameters, such as the type of each layer and the number of nodes in it, and b) training hyperparameters, such as the learning rate and batch size. The difficulty of manually designing hyperparameters to find a good NN is exacerbated by the fact that several hyperparameters interact with each other to have a combined effect on the final performance. 



  §.§.§ Motivation and Related Work:

The problem of searching for good NN has resulted in several efforts towards automating this process. These efforts include AutoML frameworks such as Auto-Keras <cit.>, AutoGluon <cit.> and Auto-PyTorch <cit.>, which are open source software packages applicable to a variety of tasks and types of NN. The major focus of these efforts is on providing user-friendly toolkits to search for good hyperparameter values. 

Several other efforts place more emphasis on novel techniques for the search process. These can be broadly grouped into NAS efforts such as <cit.>, and efforts that place a larger emphasis on training hyperparameters over architecture <cit.>. 
An alternate grouping is on the basis of search methodology – a) reinforcement learning 
<cit.>, b) evolution / genetic operations 
<cit.>, and c) BO <cit.>. Although the efforts described in this paragraph often come with publicly available software, they are typically not intended for general purpose use, the code release for <cit.> only allows reproducing NN on two datasets. This differentiates them from AutoML frameworks.

Deep NN often suffer from complexity bottlenecks – either in storage, quantified by the total number of trainable parameters N_p, or computational, such as the number of FLOPs or the time taken to perform training and/or inference. Prior efforts on NN search penalize inference complexity in specific ways – latency in <cit.>, FLOPs in <cit.>, and both in <cit.>. However, inference complexity is significantly different from training since the latter includes backpropagation and parameter updates every batch
. For example, the resulting network for CIFAR-10 in <cit.> takes a minute to perform inference, but hours to train. 
Moreover, while there is considerable interest in popular benchmark datasets
, in most real-world applications deep learning models need to be trained on custom datasets for which readymade, pre-trained models do not exist <cit.>. This leads to an increasing number of resource-constrained devices needing to perform training on the fly, self-driving cars. 

The computing platform is also important, 
changing batch size has a greater effect on training time per epoch on GPU than CPU. Therefore, calculating the FLOP count is not always an accurate measure of the time and resources expended in training a NN. Some previous works have proposed pre-defined sparsity <cit.> and stochastic depth <cit.> to reduce training time, while <cit.> focuses on finding the quickest training time to get to a certain level of performance. Note that these are all manual methods, not search frameworks.






  §.§.§ Overview and Contributions:

This paper introduces DnC – an open-source[The code and documentation are available at <https://github.com/souryadey/deep-n-cheap>] AutoML framework to search for deep learning models. We specifically target the training complexity bottleneck by including a penalty for training time per epoch  in our search objective. The penalty coefficient can be varied by the user to obtain a family of networks trading off performance and complexity. Additionally, we also support storage complexity penalties for N_p.

DnC searches for both architecture and training hyperparameters. While the architecture search derives some ideas from literature, we have striven to offer the user a considerable amount of customizability in specifying the search space. This is important for training on custom datasets which can have significantly different requirements than those associated with benchmark datasets.

DnC primarily uses BO and currently supports classification tasks using CNN and MLP. A notable aspect is search transfer, where we found that the best NN obtained from searching over one dataset give good performance on a different dataset. This helps to improve generalization in NN – such as on custom datasets – instead of purely optimizing for specific problems.

The following are the key contributions of this paper:

    
  * Complexity: To the best of our knowledge, DnC is the only AutoML framework targeting training complexity reduction. We show results on several datasets on both GPU and CPU. Our models achieve performance comparable to state-of-the-art, with training times that are 1-2 orders of magnitude less than those for models obtained from other AutoML and search efforts.
    
  * Usability: DnC offers a highly customizable three-stage search interface for both architecture and training hyperparameters. As opposed to Auto-Keras and AutoGluon, our search includes a) batch size that affects training times, and b) architectures beyond pre-existing ones found in literature. As a result, our target users include those who want to train quickly on custom datasets. As an example, our framework achieves the highest performance and lowest training times on the custom Reuters RCV1 dataset <cit.>. We also introduce search transfer to explore generalization capabilities of architectures to multiple datasets under different training hyperparameter settings.
    
  * Insights: We conduct investigations into the search process and draw several insights that will help guide a deeper understanding of NN and search methodologies. We introduce a new similarity measure for BO and a new distance function for NN. We empirically justify the value of our greedy three-stage search approach over less greedy approaches, and the superiority of BO over random and grid search.





The paper is structured as follows –– Sec. <ref> outlines our search methodology, Sec. <ref> our experimental results, Sec. <ref> includes additional investigations and insights, Sec. <ref> compares with related work, and Sec. <ref> concludes the paper.





§ OUR APPROACH

Given a dataset, our framework searches for NN config through sequential stages in multiple search spaces. Each config is trained for the same number of epochs, 100. There have been works on extrapolating NN performance from limited training <cit.>, however we train for a large number of epochs to predict with significant confidence the final performance of a NN after convergence. Configs are mapped to objective values using:
c
f(Config) = f_p(Config) + w_cf_c(Config)

where w_c controls the importance given to the complexity term. The goal of the search is to minimize f. Its components are:
rCl
f_p    =    1 - Best Validation Accuracy  

f_c    =    c/c_0 

where c is the complexity metric for the current config (either  or N_p), and c_0 is a reference value for the same metric (typically obtained for a high complexity config in the space). Lower values of w_c focus more on performance, improving accuracy. One key contribution of this work is characterizing higher values of w_c that lead to reduced complexity NN that train fast – these also reduce the search cost by speeding up the overall search process.

Fig. <ref> summarizes these effects.






 §.§ Three-stage search process







  §.§.§ Stage 1 – Core architecture search:

For CNN, the combined search space consists of the number of conv layers and number of channels in each, 
while for MLP, it is the number of hidden layers and number of nodes in each. Other architectural hyperparameters such as BN and dropout layers and all training hyperparameters are fixed to presets that we found to work well across a variety of datasets and network depths. BO is used to minimize f and the corresponding best config is the Stage 1 result.



  §.§.§ Stage 2 – Advanced architecture search:

This stage starts from the resulting architecture from Stage 1 and uses grid search to search for the following CNN hyperparameters through a sequence of sub-stages – 1) whether to use strides or max pooling layers for downsampling, 2) amount of BN layers, 3) amount of dropout layers and drop probabilities, and 4) amount of shortcut connections. This is not a combined space, instead grid search first picks the downsampling choice leading to the minimum f value, then freezes that and searches over BN, and so on. This ordering yielded good empirical results, however, reordering is supported by the framework. For MLP, there is a single grid search for dropout probabilities. As in the previous stage, training hyperparameters are fixed to presets. The result from Stage 2 is the result from the final sub-stage. 
    


  §.§.§ Stage 3 – Training hyperparameter search:

The architecture is finalized after Stage 2. In Stage 3 – identical for CNN and MLP – we search over the combined space of initial learning rate η, weight decay λ and batch size, using BO to minimize f. 
The final config after Stage 3 comprises both architecture and training hyperparameters. The complete process is summarized in Fig. <ref>.




  §.§.§ Other hyperparameters:

We recognize that there are other hyperparameters in NN such as kernel sizes, decay, etc. These have not been explored in the present work, and are excellent topics for future extensions to the framework. 





 §.§ Bayesian Optimization

BO is useful for optimizing functions that are black-box and/or expensive to evaluate such as f, which requires NN training. 
The initial step when performing BO is to sample n_1 config from the search space, {x_1,⋯,x_n_1}, calculate their corresponding objective values, {f(x_1),⋯, f(x_n_1)}, and form a Gaussian prior. The mean vector μ is filled with the mean of the f values, and covariance matrix Σ is such that Σ_ij = σ(x_i,x_j), where σ(·,·) is a kernel function that takes a high value ∈[0,1] if config x_i and x_j are similar.


c
μ = [ μ(x_1);      ⋮; μ(x_n) ]             Σ = [ σ(x_1,x_1)          ⋯ σ(x_1,x_n);          ⋮          ⋱          ⋮; σ(x_n,x_1)          ⋯ σ(x_n,x_n) ]



Then the algorithm continues for n_2 steps, each step consisting of sampling n_3 config, picking the config with the maximum expected improvement
, computing its f value, and updating μ and Σ accordingly. The reader is referred to <cit.> for a complete tutorial on BO – where eq. (4) in particular has details of expected improvement. Note that BO explores a total of n_1+n_2n_3 states in the search space, but the expensive f computation only occurs for n_1+n_2 states.




  §.§.§ Similarity between NN configurations:

We begin by defining the distance between values of a particular hyperparameter k for two config x_i and x_j. Larger distances denote dissimilarity. We initially considered the distance functions defined in Sections 2 and 3 of <cit.>, 

c
d(x_ik,x_jk) = ω/√(2) √(1-cos(πx_ik-x_jk/u_k-l_k))


but then adopted an alternate one that resulted in similar performance with less tuning. We call it the ramp distance:
c
d(x_ik,x_jk) = ω_k (|x_ik-x_jk|/u_k-l_k)^r_k 

where u_k and l_k are respectively the upper and lower bounds for k, ω_k is a scaling coefficient, and r_k is a fractional power used for stretching small differences. Note that d is 0 when x_ik=x_jk, and reaches a maximum of ω_k when they are the furthest apart. x_ik and x_jk are computed in different ways depending on k:

    
  * If k is batch size or number of layers, x_ik and x_jk are the actual values.
    
  * If k is η or λ, x_ik and x_jk are the logarithms of the actual values.
    
  * When k is the hidden node configuration of a MLP, we sum the nodes together across all hidden layers. This is because we found that the sum has a greater impact on f than considering layers individually, a config with three 300-node hidden layers has a closer f value to a config with one 1000-node hidden layer than a config with three 100-node hidden layers.
    
  * When k is the conv channel configuration of a CNN, we calculate individual distances for each layer. If the number of layers is different, the distance is maximum for each of the extra layers, ω. This idea is inspired from <cit.>, as compared to alternative similarity measures in <cit.>. We follow this layer-by-layer comparison because our prior experiments showed that the representations learned by a certain conv layer in a CNN are similar to those learned by layers at the same depth in different CNN. Additionally, this approach performed better than the summing across layers as in MLP.


Each individual distance d(x_ik,x_jk) is converted to its kernel value σ(x_ik,x_jk) using the squared exponential function, then we take their convex combination for all K hyperparameters using coefficients {s_k} to finally get σ(x_i,x_j). An example is given in Fig. <ref>.
rCl
σ(x_ik,x_jk)    =    exp(-d^2(x_ik, x_jk)/2 )  

σ(x_i,x_j)    =    ∑_k=1^Ks_kσ(x_ik,x_jk) 






The acquisition function we choose is expected improvement, given as:
c
EI(x) = (f^*-μ_post-ξ)P(f^*-μ_post-ξ/σ_post) + σ_post p(f^*-μ_post-ξ/σ_post)

where f^* is the current optimum value of f, μ_post and σ_post are the posterior mean and covariance values obtained by updating the distribution with the current state, P(·) and p(·) are respectively the cumulative and probability density functions of the standard normal distribution, and ξ trades off exploration vs exploitation.

Note that calculating EI(x) is significantly cheaper than computing f(x) due to the former not requiring training a neural network to compute performance. 





§ EXPERIMENTAL RESULTS

This section presents results of our search framework on different datasets for both CNN and MLP classification problems, along with the search settings used. Note that most of these settings can be customized by the user – this leads to one of our key contributions of using limited knowledge from literature to enable wider exploration of NN for various custom problems. We used the Pytorch library on two platforms: a) GPU – an Amazon Web Services p3.2xlarge instance that uses a single NVIDIA V100 GPU with 16 GB memory and 8 vCPUs, and b) CPU – a mid-2014 Macbook Pro CPU with 2.2 GHz Intel Core i7 processor and 16GB 1.6 GHz DDR3 RAM. For BO, we used n_1 = n_2 = 15 and n_3 = 1000.



 §.§ CNN

All CNN experiments are on GPU. The datasets used are CIFAR-10 and -100 with train-validation-test splits of 40k-10k-10k, and FMNIST with 50k-10k-10k. Standard augmentation is always used – channel-wise normalization, random crops from 4 pixel padding on each side, and random horizontal flips. Augmentation requires Pytorch data loaders that incur timing overheads, so we also show results on unaugmented CIFAR-10 where the whole dataset is loaded into memory at the beginning and  reduces as a result.

For Stage 1, we use BO to search over CNN with 4–16 conv layers, the first of which has c_1 ∈{16, 17,⋯, 64} channels and each subsequent layer has c_i+1∈{c_i, c_i+1,⋯,min(2c_i,512)} channels. We allow the number of channels in a layer to have arbitrary integer values, not just fixed to multiples of 8. Kernel sizes are fixed to 3x3. Downsampling precedes layers where c_i crosses 64, 128 and 256 (this is due to GPU memory limitations). During Stage 1, all conv layers are followed by BN and dropout with 30% drop probability. Configs with more than 8 conv layers have shortcut connections. Global average pooling and a softmax classifier follows the conv portion. There are no hidden classifier layers since we empirically obtained no performance benefit. For both Stages 1 and 2, we used the default Adam optimizer with η = 10^-3, decayed by 80% at the half and three-quarter points of training, batch size of 256, and λ = 𝕀(N_p≥10^6)× N_p/10^11, 𝕀 being the indicator function. We empirically found this rule to work well.

For Stage 2, the first grid search is over all possible combinations of using either strides or max pooling for the downsampling layers. Second, we vary the fraction of BN layers through [0,14,12,34]. For example, if there are 7 conv layers, a setting of 12 will place BN layers after conv layers 2, 4, 6 and 7. Third, we vary the fraction of dropout layers in a manner similar to BN, and drop probabilities over [0.1,0.2] for the input layer and [0.15,0.3,0.45] for all other layers. Finally, we search over shortcut connections – none, every 4th layer, or every other layer. Note that any shortcut connection skips over 2 layers.

For Stage 3, we used BO to search over a) η∈{10^x} for x ∈ [1,5], b) λ∈{10^x} for x ∈ [-6,-3], with λ converted to 0 when x<-5, and c) batch sizes in [32, 33,⋯, 512]. We found that batch sizes that are not powers of 2 did not lead to any slowdown on the platforms used.

The penalty function f_c uses normalized , since this is the major bottleneck in developing CNN. Each config was trained for 100 epochs on the train set and evaluated on the validation set to obtain f_p. We ran experiments for 5 different values of w_c: [0,0.01,0.1,1,10]. The best network from each search was then trained on the combined training and validation set, and evaluated on the test set for 300 epochs to get final test accuracies and  values.



As shown in Fig. <ref>, we obtain a family of networks by varying w_c. Performance in the form of test accuracy trades off with complexity in the form of . The latter is correlated with search cost and N_p. The last row of figures directly plot the performance-complexity tradeoff. These curves rise sharply towards the left and flatten out towards the right, indicating diminishing performance returns as complexity is increased. This highlights one of our key contributions – allowing the user to choose fast training NN that perform well.

Taking augmented CIFAR-10 as an example, DnC found the following best config for w_c=0: 14 conv layers with {c} = (50, 52, 53, 59, 95, 96, 97, 120, 193, 239, 351, 385, 488, 496), the 4th layer has a stride of 2 while max pooling follows layers 8 and 10, BN follows all conv layers, dropout with drop probability 0.3 follows every other conv block, and skip connections are present for every other conv block. The best found η remains 10^-3, batch size is 120 and λ = 3.35×10^-5. We note that we achieve good performance with a NN that has irregular {c} values and is also not very deep – the latter is consistent with the findings in <cit.>. Also note that the best config found for w_c=10 only has 4 conv layers.




 §.§ MLP

We ran CPU experiments on the MNIST and FMNIST datasets in permutation-invariant format (images are flattened to a single layer of 784 input pixels) without any augmentation, and GPU experiments on the Reuters RCV1 dataset constructed as given in <cit.>. Each dataset is loaded into memory in its entirety, eliminating data loader overheads.

For Stage 1, we search over 0–2 hidden layers for MNIST and FMNIST, number of nodes in each being 20–400. These numbers change for RCV1 to 0–3 and 50–1000 since it is a larger dataset. Every layer is followed by a dropout layer with 20% drop probability. Training hyperparameters are fixed as in the case of CNN, with the difference that λ = 𝕀(N_p≥10^4)× N_p/10^9 for MNIST and FMNIST and λ = 𝕀(N_p≥10^5)× N_p/10^10 for RCV1. For Stage 2, we do a grid search over drop probabilities in [0,0.1,0.3,0.4,0.5], and for Stage 3, the training hyperparameter search is identical to CNN.

We ran separate searches for individual penalty functions – normalized  and normalized N_p. The latter is owing to the fact that MLP often massively increase the number of parameters and thereby storage complexity of NN <cit.>. The train-validation-test splits for MNIST and FMNIST are 50k-10k-10k, and 178k-50k-100k for RCV1. Candidate networks were trained for 60 epochs and the final networks tested after 180 epochs. As before, w_c ∈ [0,0.01,0.1,1,10] for MNIST and FMNIST. For RCV1, the results for w_c = 10 were mostly similar to w_c = 1, so we replace 10 with 0.03. The plots against w_c are shown in Fig. <ref>, where pink dots are for  penalty and black crosses are for N_p penalty.



The trends in Fig. <ref> are qualitatively similar to those in Fig. <ref>. When penalizing N_p, the two lowest complexity networks in each case have no hidden layers, so they both have exactly the same N_p (results differ due to different training hyperparameters). Of interest is the subfigure on the bottom right, indicating much longer search times when penalizing N_p as compared to . This is because time is not a factor when penalizing N_p, so the search picks smaller batch sizes that increase  with a view to improving performance. Interestingly enough, this does not actually lead to performance benefit as shown in the subfigure on the top-right, where the black crosses occupy similar locations as the pink dots.




§ INVESTIGATIONS AND INSIGHTS




 §.§ Search transfer

One goal of our search framework is to find models that are applicable to a wide variety of problems and datasets suited to different user requirements. To evaluate this aspect, we experimented on whether a NN architecture found from searching through Stages 1 and 2 on dataset A can be applied to dataset B after searching for Stage 3 on it. In other words, how does transferring an architecture compare to `native' config, those searched for through all three stages on dataset B. This process is shown on the left in Fig. <ref>. Note that we repeat Stage 3 of the search since it optimizes training hyperparameters such as weight decay, which are related to the capacity of the network to learn a new dataset. This is contrary to simply transferring the architecture as in <cit.>.







We took the best CNN architectures found from searches on CIFAR-10, CIFAR-100 and FMNIST (as depicted in Fig. <ref>) and transferred them to each other for Stage 3 searching. The results for test accuracy and  are shown on the right in Fig. <ref>. We note that the architectures generally transfer well. In particular, transferring from FMNIST (green crosses in subfigures (a) and (b)) results in slight performance degradation since those architectures have N_p around 1M-2M, while some architectures found from native searches (pink dots) on CIFAR have N_p>20M. However, architectures transferred between CIFAR-10 and -100 often exceed native performance. Moreover, almost all the architectures transferred from CIFAR-100 (green crosses in subfigure (c)) exceed native performance on FMNIST, which again is likely due to bigger N_p. We also note that  values remain very similar on transferring, except for the w_c=0 case where there is absolutely no time penalty.



 §.§ Greedy strategy

Our search methodology is greedy in the sense that it preserves only the best config resulting in the minimum f value from each stage and sub-stage. We also experimented with a non-greedy strategy. Instead of one, we picked the three best config from Stage 1 – {x_1,x_2,x_3}, then ran separate grid searches on each of them to get three corresponding config at the end of Stage 2, and finally picked the three best config for each of their Stage 3 runs for a total of nine different config – {x_11,x_12,x_13,x_21,⋯,x_33}. Following a purely greedy approach would have resulted in only x_11, while following a greedy approach for Stages 1 and 2 but not Stage 3 would have resulted in {x_11,x_12,x_13}. We plotted the losses for each config for five different values of w_c on CIFAR-10 unaugmented (Fig. <ref> shows three of these). In each case we found that following a purely greedy approach yielded best results, which justifies our choice for DnC.






 §.§ Bayesian optimization vs random and grid search



We use Sobol sequencing – a space-filling method that selects points similar to grid search – to select initial points from the search space and construct the BO prior. We experimented on the usefulness of BO by comparing the final search loss f achieved by performing the Stage 1 and 3 searches in four different ways:

    
  * Random search: pick 30 prior points randomly, no optimization steps
    
  * Grid search: pick 30 prior points via Sobol sequencing, no optimization steps
    
  * Balanced BO (DnC default): pick 15 prior points via Sobol sequencing, 15 optimization steps
    
  * Extreme BO: pick 1 initial point, 29 optimization steps


The results in Fig. <ref> are for different w_c on CIFAR-10. BO outperforms random and grid search on each occasion. In particular, more optimization steps are beneficial for low complexity models, while the advantages of BO are not significant for high performing models. We believe that this is due to the fact that many deep nets <cit.> are fairly robust to training hyperparameter settings.




§ COMPARISON TO RELATED WORK



Table <ref> compares features of different AutoML frameworks. To the best of our knowledge, only DnC allows the user to specifically penalize complexity of the resulting models
. This allows our framework to find models with performance comparable to other state-of-the-art methods, while significantly reducing the computational burden of training. This is shown in Table <ref>, which compares the search process and metrics of the final model found for CNN on CIFAR-10, and Table <ref>, which does the same for MLP on FMNIST and RCV1 for DnC and Auto-PyTorch only, since Auto-Keras and AutoGluon do not have explicit support for MLP at the time of writing
.

Note that Auto-Keras and AutoGluon do not support explicitly obtaining the final model from the search, which is needed to perform separate inference on the test set after the search. As a result, in order to have a fair comparison, Tables <ref> and <ref> use metrics from the search process –  is for the train set and the performance metric is best validation accuracy. These are reported for the best model found from each search. Auto-Keras and AutoGluon use fixed batch sizes across all models, however, Auto-PyTorch and DnC also do a search over batch sizes. We have included batch size since it affects . Each config for each search is run for the same number of epochs, as described in Sec. <ref>. The exception is Auto-PyTorch, where a key feature is variable number of epochs. 








We note that for CNN, DnC results in both the fastest  and highest performance. The performance of Proxyless NAS is comparable, while taking 43X more time to train. This highlights one of our key features – the ability to find models with performance comparable to state-of-the-art while massively reducing training complexity. 
The search cost is lowest for the default AutoGluon run, which only runs 3 config. We also did an extended run for ∼100 models on AutoGluon to make it match with DnC and Auto-Keras – this results in the longest search time without significant performance gain.

For MLP, DnC has the fastest search times and lowest  and N_p values – this is a result of it searching over simpler models with few hidden layers. While Auto-PyTorch performs slightly better for the benchmark FMNIST, our framework gives better performance for the more customized RCV1 dataset.




§ CONCLUSION AND FUTURE WORK

In this paper we introduced Deep-n-Cheap – the first AutoML framework that specifically considers training complexity of the resulting models during searching. While our framework can be customized to search over any number of layers, it is interesting that we obtained competitive performance on various datasets using models significantly less deep than those obtained from other AutoML and search frameworks in literature. We also found that it is possible to transfer a family of architectures found using different w_c values between different datasets without performance degradation. The framework uses Bayesian optimization and a three-stage greedy search process – these were empirically demonstrated to be superior to other search methods and less greedy approaches.

DnC currently supports classification using CNN and MLP. Our future plans are to extend to other types of networks such as recurrent and other applications of deep learning such as segmentation, which would also require expanding the set of hyperparameters searched over. The framework is open source and offers considerable customizability to the user. We hope that DnC becomes widely used and provides efficient NN design solutions to many users. The framework can be found at <https://github.com/souryadey/deep-n-cheap>.




splncs04








§ APPENDIX: VALIDITY OF OUR COVARIANCE KERNEL

The validity of our covariance kernel can be proved as follows. We note that since x_ik and x_jk are scalars, d in eq. (<ref>) is the Euclidean distance. It follows from the properties of the squared exponential kernel that σ(x_ik,x_jk) in eq. (<ref>) is a valid kernel function. So if a kernel matrix Σ_k were to be formed such that Σ_k_ij = σ(x_ik,x_jk), then Σ_k would be positive semi-definite. Writing eq. (<ref>) in matrix form gives Σ = ∑_k=1^Ks_kΣ_k. Since a convex combination of positive semi-definite matrices is also positive semi-definite, it follows that Σ is a valid covariance matrix.




§ APPENDIX: ENSEMBLING

One way to increase performance such as test accuracy is by having an ensemble of multiple networks vote on the test set. This comes at a complexity cost since multiple NN need to be trained. We experimented on ensembling by taking the n best networks from BO in Stage 3 of our search. Note that this does not increase the search cost as long as n≤ n_1+n_2. However, it does increase the effective number of parameters by a factor of exactly n (since each of the n best config have the same architecture), and  by some indeterminate factor (since each of the n best config might have a different batch size).



We experimented on CIFAR-10 unaugmented using n=3 and augmented using n=5. The impact on the performance-complexity tradeoff is shown in Fig. <ref>. Note how the plus markers – ensemble results – have slightly better performance at the cost of significantly increased complexity as compared to the circles – single results. However, we did not use ensembling in other experiments since the slight increases in accuracy do not usually justify the significant increases in .




§ APPENDIX: CHANGING HYPERPARAMETERS OF BAYESIAN OPTIMIZATION

The BO process itself has several hyperparameters that can be customized by the user, or optimized using marginal likelihood or Markov chain Monte Carlo methods <cit.>. This section describes the default values we used. Expected improvement involves an exploration-exploitation tradeoff variable ξ. The recommended default is ξ=0.01 <cit.>, however, we tried different values and empirically found ξ=10^-4 to work well. Secondly, f is a noisy function since the computed values of network performance are noisy due to random initialization of weights and biases for each new state. Accordingly, and also considering numerical stability for the matrix inversions involved in BO, our algorithm incorporates a noise term σ_n^2. We calculated its value from the variance in f values as σ_n^2 = 10^-4, which worked well compared to other values we tried.




§ APPENDIX: ADAPTATION TO VARIOUS PLATFORMS

While most deep NN are run on GPUs, situations may arise where GPUs are not readily or freely available and it is desirable to run simpler experiments such as MLP training on CPUs. DnC can adapt its penalty metrics to any platform. For example, the FMNIST results shown in Fig. <ref> were on CPU, while Table <ref> shows results on GPU (to do a fair comparison with other frameworks). As a result, the  values are an order of magnitude faster, while the performance is the same as expected.





§ APPENDIX: CONSTANTS

Finally we discuss potential keys that we considered, but eventually decided not to include in our search framework due to their values remaining relatively constant across the literature. These constants are:

    
  * Activation function: Fixed as ReLU. Activation function is always applied after every conv layer (or after the corresponding BN layer if it exists).
    
  * Padding for convolution: We always pad such that given a stride of 1, the output image dimensions of a layer will be the same as the input to that layer. This means that we apply 1 pixel padding for 3x3 kernels, and 2 pixel padding for 5x5 kernels.
    
  * Parameter initialization: We use He normal initialization <cit.> for weights and a constant value of 0.1 to initialize biases. The latter is according to the recommendations in <cit.>, which suggests a small positive starting value for biases for better convergence when using ReLU activation.
    
  * Dilation: We do not apply dilated convolutions since their use case is primarily semantic segmentation, which is not the primary use case in this work.
    
  * Global average pooling: This is always applied after the conv portion, prior to the MLP.
    
  * Pool size: When max-pooling layers are present, they always use 2x2 non-overlapping pooling windows.
    
  * Style of pre-defined sparsity: Our previous work <cit.> discussed different classes of pre-defined sparsity – random, structured and clash-free. Since the number of channels is a search key, we do not know what the size of the input to the MLP will be after applying global average pooling. Accordingly, using structured or clash-free pre-defined sparsity will detract from the richness of the search space possible to be explored. So we use only random pre-defined sparsity in this work. While random pre-defined sparsity was shown to have slightly inferior performance compared to other classes, this is only true for very low density values, not in the range [0.2,1] that we search over.



