Are the shortcomings of neural networks diminishing? Having worked with neural networks for about half a year, I have experienced first-hand what are often claimed as their main disadvantages, i.e. overfitting and getting stuck in local minima. However, through hyperparameter optimization and some newly invented approaches, these have been overcome for my scenarios. From my own experiments:

Dropout seems to be a very good regularization method.

Batch normalization eases training and keeps signal strength consistent across many layers.

Adadelta consistently reaches very good optima


I have experimented with scikit-learn's implementation of SVM alongside my experiments with neural networks, but I find the performance to be very poor in comparison, even after having done grid-searches for hyperparameters. I realize that there are countless other methods, and that SVM's can be considered a sub-class of NN's, but still.
So, to my question:
With all the newer methods researched for neural networks, have they slowly - or will they - become &quot;superior&quot; to other methods? Neural networks have their disadvantages, as do others, but with all the new methods, have these disadvantages been mitigated to a state of insignificance?
I realize that oftentimes &quot;less is more&quot; in terms of model complexity, but that too can be architected for neural networks. The idea of &quot;no free lunch&quot; forbids us to assume that one approach always will reign superior. It's just that my own experiments - along with countless papers on awesome performances from various NN's - indicate that there might be, at the least, a very cheap lunch.
Neural Networks have other short comings as well. 


It takes much longer and far more resources to train a neural network than something like a random forest. So if you need speed of training or are resource constrained in anyway, you probably should not look at Neural Networks first. Evaluation of a trained deep NN can be much more expensive than competing techniques too.
The effort involved in learning how to architect and train a NN is still much higher than competing methods, like an SVM. People who are just starting out in Data Science should probably use other techniques to learn about the nuances of fitting data before getting involved in neural networks. And although simple NNs with only one or two hyperparameters are often available in 
many data science libraries, they don't perform any better than other techniques so are just another ML black-box technique really.
While we have made a lot of progress in understanding how neural networks do their magic, they are still less accessible and dissectible than most competing methods. So while NNs might solve the problem, they might not give you as many insights as easily as other techniques do.


Looking forward to what other people have to say here.