I understand how recurrent neural networks work, however I'm trying to build a deep intuitive understanding of their behavior which is difficult for me because they exhibit such complex behaviors.The most difficult thing for me to understand is how a Reccurrent Neural Network can exhibit an oscillatory behavior along with the notion of exploding weights. For one, I'm guessing that oscillatory behaviors are only possible for certain activation functions and configurations. Here are my following questions:1) Can sigmoid activation functions exhibit oscillatory behavior? I've convinced myself that they don't since they have a positive range between 0 and 1 which doesn't allow for negative derivatives, but maybe I'm wrong. Is there a formal proof out there for whether it can or cannot?2) Which activation functions can exhibit oscillatory behaviors and are there proofs out there for them? I believe tanh might have this behavior but I'm not sure.3) What are exploding weight derivatives and how do they occur? There is polar opposite of exploding weight's which seems to be where the weights do not learn and stay stagnate. What causes these issues? I imagine this is dependent upon the activation function as well.Lastly, I began reading this article:http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&ved=0CCsQFjAB&url=http%3A%2F%2Fwww.researchgate.net%2Fprofile%2FChristian_OReilly%2Fpublication%2F52004955_Permanent_oscillations_in_a_3-node_recurrent_neural_network_model%2Flinks%2F0c96052464be171d68000000.pdf&ei=X_E1Vf7ULNGsogSo9YHIBQ&usg=AFQjCNE0fex0s7hY2w_upMNkmaxIseUKww&bvm=bv.91386359,d.cGUbut got confused around page 10 because the author omit's explaining opaque assumptions such as why we are defining the matrix measure as is and so forth(it's a good paper nonetheless). Is there a more transparent paper or lecture out there that can shed light on this subject, or maybe simple explanation for what the author is trying to get at?
Prove Reccurrent Neural Network can exhibit oscillatory behavior
machine learning;neural network;deep learning