How do you think about neural networks and ways to design new models?
You are right to guess that there is no solid theoretical way to design neural models. This is in part consistent with the free lunch theorem which says that there is no universal algorithm that can solve all problems efficiently. 

The performance of neural models not only depends on the idea but also on the data. Batchnorm has a valid reason for normalising the input to each hidden layer in neural networks but it doesn't always improve the performance of models. One can find the limitations of current neural architecture for a given data and can come up with a fix that might improve the performance but it doesn't always mean that it will prove to be better. 

As of now the field is more or less driven by an empirical approach rather than a theoretical one when it comes to designing new neural architecture.