Are there dynamic neural networks?
This article on Dynamically Expandable Neural Networks (DEN) (by Harshvardhan Gupta) is based on this paper Lifelong Learning with Dynamically Expandable Networks (by Jeongtae Lee, Jaehong Yoon, Eunho Yang, Sung Ju Hwang)

This presents 3 solutions to increase the capacity of the network if needed retaining whatever useful information from the old model and train the new model:


Selective retraining
Dynamic Network Expansion 
Network Split/Duplication


To me, it seems that such neural network is dynamic and improving. As such, they answer partially your question. If they don't sorry about that.