What are the limitations of RNNs?
I finally finished the project. Given really short signals and a really small training set, SNNs (I used Echo State Machines and a neural form of SVM) vastly out-performed Deep Learning recurrent neural networks. However, this may be mostly because I'm really bad at training Deep Learning networks.

Specifically, SNNs performed better at classification of various signals I created. Given the following signals:



The various approaches had the following accuracy, where RC = Echo State Machine, FC-SVM = Frequency Component SVM and vRNN = Vanilla Deep Learning Recurrent Neural Network:



SNNs were also more robust to noise:



For more information, including how I desperately tried to improve the Deep Learning classification approach performance, check out my repository and the report I wrote which is where all the figures came from.

Update:
After spending some time away from this project, I think one of the reasons that RNNs do horribly at this project is that they're bad at dealing with really long signals. Had I chunked the signals together with some sort of smoothing as preprocessing, they probably would have performed better.