Efficient NAS with FaDE on Hierarchical Spaces
Simon Neumeyer Julian Stier Michael Granitzer


















    
    April 29, 2024
==================




Neural architecture search (NAS) is a challenging problem.
Hierarchical search spaces allow for cheap evaluations of neural network sub modules to serve as surrogate for architecture evaluations.
Yet, sometimes the hierarchy is too restrictive or the surrogate fails to generalize.  
We present FaDE which uses differentiable architecture search to obtain relative performance predictions on finite regions of a hierarchical NAS space.
The relative nature of these ranks calls for a memory-less, batch-wise outer search algorithm for which we use an evolutionary algorithm with pseudo-gradient descent.
FaDE is especially suited on deep hierarchical, respectively multi-cell search spaces, which it can explore by linear instead of exponential cost and therefore eliminates the need for a proxy search space.


Our experiments show that firstly, FaDE-ranks on finite regions of the search space correlate with corresponding architecture performances and secondly, the ranks can empower a pseudo-gradient evolutionary search on the complete neural architecture search space.

    









§ INTRODUCTION


Automatically finding structures of deep neural architectures is an active research field.
The exponentially growing space of directed acyclic graphs (DAGs), their complex geometric structure and the expensive architecture performance evaluation make neural architecture searches (NAS) a challenging problem.
Methods such as evolutionary and genetic algorithms, bayesian searches and differentiable architecture searches compete for the most promising automatic approaches to conduct neural architecture searches <cit.>.

Differentiable architecture search (DARTS) is a successful and popular method to relax the search space into a differentiable hyper-architecture.
This relaxation allows to use differentiable search methods to learn both model weights and architectural parameters to evaluate sub-paths of a hyper-architecture <cit.>.
While the combined and weight-shared hyper-architecture allows for a very fast training of few GPU days, the search space is limited to subspaces of the defined hyper-architecture.
Evolutionary searches, in contrast, are way more dynamic in the way they restrict the search space.
Without any weight-sharing tweaks, this usually comes with a significant higher computational time.





We present a FAst Darts Estimator on hierarchical search spaces () that aims at optimizing chained like hierarchical architectures while not resorting to a proxy domain.
By iteratively fixing a finite set of sub-module architectures per neural sub-module and using DARTS for training, architecture ranks α are estimated based on the relative performance within an iteration.
As the estimations are of relative nature, we require a state-less, batch-wise optimization algorithm to determine from those estimations a new finite set of cell architectures per cell.
To this end, we apply an evolutionary approach which incorporates a pseudo-gradient descent for candidate generation.
runs one independent optimization algorithm per cell which allows it to optimize additional depth with linear instead of exponential cost.

Our contributions contain the first usage of differentiably obtained ranks for neural architecture search in an open-ended search space.
The usage is justified with a correlation analysis.
We provide code and data of our experiments for reproducibility in a github repository.
might be generalized to further types of hierarchical search spaces and could also be employed with other state-less, batch-wise search strategies in open-ended search spaces.








§ FAST DARTS ESTIMATOR
 

We construct chained hierarchical search spaces for neural architectures and use their (relative) estimated performance as architectural ranks.
The obtained -ranks guide a search on the complete (but open) search space.
The structure of finite regions is not arbitrary, but bounded to the set of architectures contained in a hyper-architecture.
We use DARTS to train such a hyper-architecture, to predict the corresponding region of the search space from a trained hyper-architecture, and a mapping of the search space into Euclidean space together with a pseudo gradient descent to guide the exploration of the search space.



  
On Chained Hierarchical NAS Spaces
Motivated by repeated motifs in hand-crafted architectures, <cit.> introduce hierarchy to NAS spaces by considering an architecture to be constructed from several structurally identical sub-modules, so called cells.
By identical the same architecture, yet each cell with its proper weights, is meant.
A cell typically consists of several convolutional layers, each with a variable operation type and with variable connections between layers.
Optimizing the cell architecture is often equivalent to finding the type of convolutional operation for a fixed number of layers and determining which layers are being connected.
The macro-architecture determines how multiple, often identical, cells are being stacked to one complete architecture.
Quite often, the stacking is done in e.g. a simple chain structure <cit.>.
Many search strategies on hierarchical search spaces fix the macro-architecture and solely optimize the architecture of a single cell which is often done on a shallower proxy domain.
For example, in image classification, a proxy domain might consist in switching from CIFAR-100 to CIFAR-10.
Even though such search strategies can achieve very good results, they are not always successful.
On the one hand, the performance of a single cell might not generalize to the performance of an overall neural network architecture consisting of multiple structural copies of that cell.







 §.§ DARTS


DARTS relaxes discrete architecture decisions within a differentiable hyper-architecture.
Liu et al. <cit.> consider a neural network module consisting of a set of edges E, that densely connects several ordered vertices, and a finite set of convolutional operations O.
The goal is to find the top k∈ℕ incoming edges and respective operation per vertex.
To this end, <cit.> dedicate one architecture parameter α_e,o per edge e∈ E and operation o∈ O, and calculate the output for any edge e∈ E as

    Softmax(α_e)^⊤ o(.), o∈ O

Finding the best architecture parameters α aims at solving the bi-level optimization problem

    _αL_val(α, _ωL_train(α, ω))

with L_val and L_train denoting the neural network loss on a dedicated set of training samples each.
<cit.> propose a second-order and a simplified first-order objective for solving equation <ref> via gradient descent.
The latter can be implemented by alternatingly fixing α and training ω on its respective split of training samples and vice verca.
After training, they derive an optimal discrete model by keeping only those operations with large architecture parameters.
A driving factor of <cit.> becoming the baseline work of differentiable architecture search consists in their bi-level optimization algorithm and in their choice of search space.
Liu et al. work on a cell from <cit.> and further adopt their proxy concept, consisting of a smaller dataset and a smaller chain of cells during training.

Gumbel-Softmax
<cit.> criticize the Softmax in equation <ref> learning a well performing combination of architectures to which no single architecture will generalize once selected after training.
Instead, they propose a sampling mechanism in order to only activate the connections of a single architecture during any forward pass, while still applying back-propagation to the complete hyper-architecture.
To this end, instead of Softmax they apply [gumbel_softmax]
    Gumbel-Softmax_τ(x):=Softmax_τ(x + G^n)
 to x∈ℝ^n where τ>0 is a temperature parameter for Softmax and G^n is an i.i.d. vector of the standard Gumbel distributed random variable G.
Gumbel-Softmax adds stochastic noise to a Softmax transformation while supporting differentiability.
For their forward pass they use an additional one-hot encoding on equation <ref>.
<cit.> show that it holds for any τ > 0, x∈ℝ^n: 
[Onehot ∘ Gumbel-Softmax_τ(x)] = Softmax(x)




 §.§ Training a Chained Hierarchical Architecture using DARTS


For an abstract space of cell architectures 𝒮 and a depth d∈ℕ, we build the chained search space 𝒮^d, see <ref>.
Given a window size w∈ℕ, we consider the corresponding space of hyper-architectures as ℋ:=ℋ_d,w(𝒮):=𝒮^d× w, see <ref> (middle).
We use a matrix notation for hyper-architectures for convenience.
Note, that any hyper-architecture H∈𝒮^d× w can be identified with a subset of the search space by considering ×_i≤ d{H_ij |  j≤ w}⊂𝒮^d, the cross product modelling the combinatorics of chaining cells along the depth of the search space.
Hence, a row of H∈𝒮^d× w represents the pool of cells that H features at the corresponding depth.

Using differentiable architecture search (DARTS) <cit.>, we endow a hyper-architecture H∈ℋ with architecture parameters α∈[0,1]^d× w, such that after training H with a bi-level optimization algorithm, architecture parameter α_ij reflects the performance of cell H_ij in the context of exclusive competition within each row of H.
For convenience, α_i, i≤ d, always denotes the architecture parameters after the Softmax transformation.

In our work, the cells themselves serve as building blocks for DARTS as opposed to <cit.> where DARTS is being applied to optimize the architecture of a single cell.
Following <cit.> and <cit.> we choose Gumbel-Softmax <cit.> to incorporate the architecture parameters into the computation path.
Usually, DARTS-based methods consider the architecture parameters as variables over a convex loss surface and optimize them in the same fashion as neural networks weights.
As we want to use the trained architecture parameters as performance predictors, we found it reasonable to firstly regularize the architecture parameters and secondly use a constant learning rate to not interfere with the effect of the former.
Our regularization should serve the purpose of preventing the hyper-architecture from converging too early in favor of certain architectures while supporting such a convergence in later epochs.
To this end, we add the maximum norm on the Gumbel-Softmax regularized architecture parameters to the loss function, scaled by a factor r∈ℝ linearly decreasing over epochs.
Hence, we update our architecture parameters α := Gumbel-Softmax(α^'_i)_i≤ d, α^'∈ℝ^d× w, according to the following objective gradient:

    ∇_α^'(L(α, ω) + r‖α‖_∞)

where L(α, ω) denotes the neural network loss dependent on architecture parameters α and neural network weights ω.

Cell-dependent Regularization
We elaborate on the regularization factor r in <ref> to regularize shallower cells earlier than deeper cells, compare <ref> 
This approach is motivated by a cell only being able to learn if its input is somewhat stable.

requirement_twocolumn
    
    
    

requirement_jmlr
    
    

    requirement_twocolumn
        
    
        
    


Weight Sharing
In addition to implicit weight sharing within the hyper-architecture, implied by a linear number of cells making up for an exponential number of contained architectures, weights might also be shared per row, requiring an implementation of the hyper-architecture where each row itself consists of a hyper-cell that coalesces several cells.
In our experimental setup, for example, where cell architectures are modelled by graphs, we considered the largest graph G_max contained in the search space, fully connected, and implemented a hyper-architecture of which each row solely consists of the implementation of G_max, the hyper-cell.
Now any hyper-architecture can be derived from this hyper-architecture by considering all graphs as sub-graphs of G_max and renouncing on devoting strictly separate computation paths to each cell, but instead only virtually separating them as different paths within the hyper-cell.





 §.§ Deriving FaDE-ranks on Hyper-Architecture


We use α to rank the subset of architectures contained in H based on

    ψ_α:H→ℝ,(H_1k_1,…,H_dk_d) ↦∏_i≤ dα_ik_i

as shown in <ref> (middle).
Note that ψ_α is just one of many ways to apply the information encoded in α to a ranking of corresponding architectures.
The benefit of this approach arises from the practical - not theoretical - assumption of independence along the depth of an architecture, that allows for predicting an exponential search space in linear time.
Training several architectures H_val⊂ H⊂𝒮^d from scratch yields a validation function ρ on H_val.
A rank correlation coefficient between ρ and ψ_α, the latter restricted to the validation set, documents how well ψ_α predicts relative performances of single architectures contained in H.





 §.§ Joint Batch-wise Pseudo Gradient Descent
 

A proper correlation between ρ and ψ_α indicates the usefulness of the information contained in the α parameters and motivates us to use ψ_α in guiding a memory-less, batch-wise search in 𝒮^d. 
The downside of the hyper-architecture approach persists in the relative nature, implying that -ranks ψ_α can in general not be compared among new hyper-architecture evaluations.
Contrary to Pham et al. <cit.> we argue though that changes in weights of a hyper-architecture eventually invalidate former architecture evaluations within that hyper-architecture and hence any information obtained, whether prediction or real evaluation, inhibits a relative nature anyways, its information content fading during search.
Search methods such as gradient descent do not need memory and hence can use the -ranks to navigate a NAS.
To apply gradient descent, we assume the search space to be Euclidean and use finite differences on a batch of -ranks to approximate a gradient.
Details on the caveat of 𝒮 to be Euclidean are being discussed further below.

The overall search works by iteratively sampling hyper-architectures H^(t), t∈ℕ where the i-th row of H^(t+1), i≤ d is solely dependent on the i-th row of the corresponding architecture parameter α^(t), obtained after training H^(t).
Compare <ref>.
Formally, we consider d independent w-dimensional stochastic processes in 𝒮^w and we therefore refer to the search over 𝒮^d being a joint search over 𝒮.
The goal is to find hyper-architectures containing well-performing single architectures.
For any row i≤ d, an anchor point M^(t)_i∈𝒮 is being maintained, with M^(1)_i being randomly initialized. The cells of row i in H^(t+1) are being derived from the standard unit vectors originating from the anchor:

    H^(t)_i={M^(t)_i}∪{M^(t)_i±γ*e_k  |  k≤w/2}

where the dimension of 𝒮 is assumed to be w/2 and where e_k denote the standard unit vectors, k≤w/2.

r0.35
    
    < g r a p h i c s >

    
    < g r a p h i c s >

    
    < g r a p h i c s >

    
        Graph generation:
        a sample in embedding space determines a corresponding bucket from which a graph is drawn.
    
    
    


The hyper-parameter γ>0 controls the width of the local environment around the anchor.
After having obtained the architecture parameters for H^(t)_i by training H^(t), the anchor M^(t+1)_i is being derived from descending M^(t)_i according to the finite differences along the standard unit vectors:

    M^(t+1)_i=M^(t)_i - λ∑_k=1^w/2e_k(β_i(M^(t)_i+γ*e_k) - β_i(M^(t)_i-γ*e_k))

where λ>0 controls the step size of gradient descent and β_i(·) mapping sub-modules to their corresponding architecture parameter after train iteration i.

Note that weight sharing could be considered among successive hyper-architectures.
We tested pre-initializing the normal neural network weights of H^(k+1) with the trained weights of H^(k).



  
Search Space

We focus on the graph attributes of neural network cell architectures.
Therefore we provide a bijective embodiment:𝒢→𝒮 from a space of directed acyclic graphs to the space of cell architectures.
We let embodiment map a directed acyclic graph to a cell architecture by first prepending an input vertex to vertices with no incoming edges and appending an output vertex to all vertices with no outgoing edges.
The input vertex just serves as interface to distribute the input vector x to all source vertices.
On all edges, except those originating from the input vertex, we place structurally identical convolution layers.
Vertices with more than one input edge combine their inputs by summation and ReLU non-linearity.
While channel count and feature map size within a cell are being fixed, between succeeding cells we approximately double the channel count while proportionally reducing the feature map size.

To enable the pseudo gradient descent from <ref>, we use a low-dimensional feature space ℱ and a (stochastic) generator:ℱ→𝒢 that generates graphs from Euclidean vectors.
The generator function is desired to be surjective and smooth with regard to the performance of cell architectures in 𝒮≅𝒢.
There are several possible choices for 𝒢, ℱ and generator, including ℱ to be the parameter space of a graph generation algorithm.
In our experiments we settle for simple choices, starting with a rather small 𝒢 containing directed acyclic graphs with a low number of vertices.
Hoping on smoothness we consider several scaled graph attributes that have been shown <cit.> to correlate with performance of the corresponding sub-modules in a certain embodiment and training setting, to construct the dimensions of ℱ.


r0.35
    

    
    < g r a p h i c s >

    
        Density of softmaxed architecture parameters:
        predicted ranks based on averaged architecture parameter per graph architecture per cell (dark=deep, light=shallow)
    
    
    



We construct an embedding: 𝒢→ℱ by determining the required graph attributes for each graph of 𝒢.
As this embedding is not dense in ℱ, we propose a mapping from the feature space to the space of cell architectures as follows.
ℱ is being separated into disjunct dim(ℱ)-dimensional intervals and define interval:ℱ→2^ℱ to map a point in the feature space to its containing interval.
We then define generator as U∘ embedding^-1∘ interval where U is the discrete uniform sampling.
<ref> visualizes the sampling process via G on an exemplary two-dimensional Euclidean feature space.
In addition to potentially increasing sampling speed, dependent on the implementation, the sampling via intervals adds some appreciated noise to the generator.










  
Related Work

<cit.> extend DARTS <cit.> by sampling a single embedded architecture for each forward pass instead of taking a weighted sum which, besides reducing memory, might even result in more reliable training as the sparsity of a forward pass is the same for hyper-architecture and target architecture.
Also <cit.> and <cit.> feature a sparse forward pass by using Gumbel-Softmax <cit.> instead of a weighted sum for aggregating parallel architecture choices.
<cit.> progressively drop the weakest connections as training progresses to reduce memory footprint.
They use the saved resources to progressively increase depth of the hyper-architecture w.r.t. stacked sub-modules and thus aim at closing the gap between proxy and target domain.
However, they feature only copies of the same sub-module.
<cit.> propose a progressive approach that successively searches a stack of different sub-modules.
Progressive approaches are also used without DARTS, e.g. by starting from a space containing very small network modules and end up in a space containing modules of desirable size <cit.>.
This restricts search space exploration to a constant complexity in each iteration by step-wise building up from well performing smaller modules.

Weight sharing comes implicit in DARTS but is also used in other work, i.e. <cit.> equip a recurrent NAS pipeline with a hyper-architecture such that instead of training discrete architectures proposed by the pipeline from scratch, they are initialized with the corresponding weights of the hyper-architecture.
That enhancement is applicable out of the box to most NAS approaches and shows a significant decrease in resource usage while achieving comparable results.







§ EXPERIMENTS


We consider a multi-cell search space consisting of n_c=4 chained cells, each cell featuring a DAG with less than n_v=6 nodes as cell architecture.
In a first experiment we obtain -ranks on a single hyper-architecture and show that they correlate well with the actual performances of a small subset of architectures contained in the hyper-architecture. 
Another experiment iteratively trains hyper-architectures according to <ref> in order to search the complete search space.
For the latter we present that search results improve over iterations, though not significantly.

l0.4
    

    
    < g r a p h i c s >

    
        Correlation between predicted and evaluated ranks.
    
    
    



Dataset Preprocessing
All experiments are conducted on the CIFAR-10 image classification dataset.
The employed dataset split is 1-1-4, meaning one part was used for testing, one part for architecture training and four parts for weight training.
Between succeeding cells, max pooling is applied for downsampling.
All convolutional cells have been using the same kernel shape of 5× 5.
Gradient clipping during backpropagation was employed with an absolute value of 10.

We used the following hyperparameters across all experiments and provide a github repository for detailled reproducibility:
activation function is ReLU, loss is cross-entropy, DARTS aggregation is hard Gumbel-Softmax, the aggregation func. temperature is ten.
We used 16 or 32 channels for deepest cells and Max() as pooling operation.
For optimization we used a batch size of 128, Adam with β_1=0.1, β_2=10^-3, α=10^-3, ϵ=10^-8, a weight decay of 10^-4, Kaiming as weight initialization, 𝒩(0; 0.5) for the architecture initialization and a gradient clipping value of ten.




 §.§ Validating FaDE-Ranks 


We construct a hyper-architecture that with n_g=5 manually selected DAGs as parallel computation paths per cell.
Once embedded, each DAG comes with the same number of weights, approximately 0.25· 10^6.
The hyper-architecture H spans a finite search space with n_g^n_c=5^4=625 architectures.
After training the H, we obtain -ranks as described in <ref>.

We consider the architecture parameter distributions per cell, averaged over several experiment repetitions, as independent marginals of a joint discrete distribution on the finite architecture search space.
<ref> illustrates the predicted marginal distributions of the trained hyper-architecture.

Training a subset of 16 manually selected discrete architectures enables calculating a spearman rank correlation between -ranks and evaluation ranks, as shown in <ref>.
The correlation coefficient of 0.8 is significant and shows that our methodology predicts the relative performances within the small architecture subset quite well.
That means, the obtained -ranks can be used to guide an open-ended search with local information.

r0.4
    

    
    < g r a p h i c s >

    
        Search trajectories in ℝ^3 per cell per dimension:
        per cell the anchor point of the pseudo gradient descent is plotted for 100 epochs.
        The anchor point coordinates are color-coded by feature space dimension which share a common y-axis as their values are normed to the same interval.
        We observe a rough convergence of trajectories.
    
    
    



From further experiments we observed that weight sharing decreases the obtained correlation compared to the results in <ref>.
However, weight sharing yields quite significant correlation results already with a few number of training epochs.
This may be owed to the fact, that weight sharing implies weights to be trained more often and thus counteracts the reduction in epochs.
We will resort to a fewer number of training epochs while sharing weights in later experiments.




 §.§ NAS on iterative FaDE-Ranks 




We aim at iteratively improving the cell architectures of the hyper-architecture from <ref> according to the methodology described in <ref>.
A pseudo gradient descent serves as optimization strategy on the feature space ℱ.
We consider architectures with 4 cells, assign one feature space per cell and run independent, parallel pseudo gradient descent algorithms on them.
The joint feature space refers to the product of the 4 feature spaces.
Building on <cit.>, e:𝒢→ℱ maps a DAG to three dimensions according to its normed eccentricity variance, degree variance and number of vertices, such that ℱ=ℝ^3.

A single experiments runs with 100 epochs, each featuring five epochs of hyper-architecture optimization.
For each cell i=1,…,4, we obtain feature space trajectories (M^(t)_i)_t=1,…,100.
In order to validate these trajectories, in regular intervals of t, we repeatedly construct an architecture from the DAGs generated from M^(t)_i, i=1,…,4, and evaluate it using 30 epochs of architecture training.
We thus obtain a validation function from the trajectory index space into ℝ.
An increasing function, coarsely measurable by a linear regression on its index, would indicate our search strategy to yield better architectures over time.
<ref> shows individual model evaluations, including a linear regression.
The pearson coefficient of 0.16 is barely significant.

<ref> shows the trajectories in all three dimensions per cell from shallow to deep and its bottom graphic shows the trajectories aggregated over cells.
We interestingly notice that the trajectories already show convergence within 20 epochs.
Note that convergence in number of operations towards the high end of the scale occurs in every cell.
This convergence is in accordance with what one would naturally expect.
We also observe that the deeper the cell the more divergent its trajectories.
There are artifacts that could be attributed to the sparsity of our search space or poor heuristics of our search space sampling, for example the large step sizes of the trajectories and its occasional chainsaw pattern.



To validate the outer Neural Architecture Search, we compared to random search (RS) and bayesian optimization (BO).
For 50 epochs, the search methods propose a point in the joint ℱ for evaluation.


The median accuracy of five such architecture generations and evaluations is being fed back to the search algorithm.
BO, similar to the pseudo gradient descent on , assumes its stochastic models per cell to be independent from each other.
Even though, this is not the case, we argue that the number of epochs is too small for a more complex bayesian model that does not make this assumption of independence.
For BO, we use the Upper Confidence Bounds method with κ=2.5, ξ=0.
We compare the results with the pseudo gradient descent validation results, this time validating the trajectories for the first 25 of 100 outer epochs.
We do not use 50 epochs for the pseudo gradient descent on as the trajectories already converge earlier.
<ref> provides test accuracies of the top 10 epochs per search method.

Points in <ref> are a median of five accuracy evaluations, generated from the same point in ℱ.
The distribution of accuracy evaluations of a single point represents an important criteria on the suitability of the search space.
A wide-spread distribution indicates that the feature space does not capture well architecture features that correlate with the corresponding evaluation performance. 
The mean standard deviation across all data points in our plot is 0.014 which is quite high compared to the observed magnitudes in <ref>.



















§ CONCLUSION & FUTURE WORK


We presented , a method to leverage differentiable architecture search to aggregate path decisions from a fixed hierarchical hyper-architecture into point estimates for an open-ended search.
The aggregated estimates are called -ranks and show a positive rank correlation with individually trained architectures.
Justified with this correlation, -ranks can be used to guide an outer search in a pseudo-gradient descent manner.
The method is generalizable in a way that alternative strategies for the outer search can be employed as long as the relative nature of the ranks are respected.
We see future work in both 1/ the analysis of the quality rank information for global search as well as 2/ experiments with more complex graph feature spaces, e.g. obtained from generative graph models.




splncs04

10




cai2018
Cai, H., Zhu, L., Han, S.: Proxylessnas: Direct neural architecture search on
  target task and hardware  (2018)

Chen2019
Chen1, X., Xie2, L., Wu1, J., Tian, Q.: Progressive darts bridging the depth
  gap between search  (2019)

dong2019
Dong, X., Yang, Y.: Searching for a robust neural architecture in four gpu
  hours. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
  Pattern Recognition. pp. 1761–1770 (2019)

elsken2019neural
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. The
  Journal of Machine Learning Research  20(1),  1997–2017 (2019)

hao2021layered
Hao, J., Zhu, W.: Layered feature representation for differentiable
  architecture search  (2021)

jang2016categorical
Jang, E., Gu, S., Poole, B.: Categorical reparameterization with
  gumbel-softmax. In: International Conference on Learning Representations
  (2017)

liu2018prog
Liu, C., Zoph, B., Neumann, M., Shlens, J., Hua, W., Li, L.J., Fei-Fei, L.,
  Yuille, A., Huang, J., Murphy, K.: Progressive neural architecture search
  (2018)

liu2018darts
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. In:
  International Conference on Learning Representations (2018)

pham2018efficient
Pham, H., Guan, M., Zoph, B., Le, Q., Dean, J.: Efficient neural architecture
  search via parameters sharing. In: International conference on machine
  learning. pp. 4095–4104. PMLR (2018)

stier2019structural
Stier, J., Granitzer, M.: Structural analysis of sparse neural networks.
  Procedia Computer Science  159,  107–116 (2019)

xie2018
Xie, S., Zheng, H., Liu, C., Lin, L.: Snas: stochastic neural architecture
  search. In: International Conference on Learning Representations (2018)

zoph2018
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable
  architectures for scalable image recognition. In: Proceedings of the IEEE
  conference on computer vision and pattern recognition. pp. 8697–8710 (2018)




