Are neural networks dynamical systems? Dynamical systems are those whose evolution can be described by a rule, evolves with time and is deterministic. In this context can I say that Neural networks have a rule of evolution which is the activation function $f(\text{sum of product of weights and features})$ ? 

Are neural networks 


dynamical systems, 
linear or nonlinear dynamical systems? 


Can somebody please shed some light on this?
The type of neural networks you have in mind (where each neuron is a threshold function taking on $0$ or $1$ values), with $n$ neurons is a dynamical system with state space $\{0,1\}^n$. After every tick of the clock, the neural network is in some new state, defined by the current values of its neurons.

Because the state space is discrete, I don't think your second question makes sense here. Linear vs. non-linear are terms applied to continuous-time dynamical systems.