MS-DARTS: Mean-Shift Based Differentiable Architecture Search
  Differentiable Architecture Search (DARTS) is an effective continuous
relaxation-based network architecture search (NAS) method with low search cost.
It has attracted significant attentions in Auto-ML research and becomes one of
the most useful paradigms in NAS. Although DARTS can produce superior
efficiency over traditional NAS approaches with better control of complex
parameters, oftentimes it suffers from stabilization issues in producing
deteriorating architectures when discretizing the continuous architecture. We
observed considerable loss of validity causing dramatic decline in performance
at this final discretization step of DARTS. To address this issue, we propose a
Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and
perturbation. Our approach can improve bot the stability and accuracy of DARTS,
by smoothing the loss landscape and sampling architecture parameters within a
suitable bandwidth. We investigate the convergence of our mean-shift approach,
together with the effects of bandwidth selection that affects stability and
accuracy. Evaluations performed on CIFAR-10, CIFAR-100, and ImageNet show that
MS-DARTS archives higher performance over other state-of-the-art NAS methods
with reduced search cost.
