Advantages of stacking LSTMs?
From What are the advantages of stacking multiple LSTMs? (I'll only update the answer there):


  From {1}:
  
  
    While it is not theoretically clear what is the additional power gained by the deeper architecture, it was observed empirically that
    deep RNNs work better than shallower ones on some tasks. In
    particular, Sutskever et al (2014) report that a 4-layers deep
    architecture was crucial in achieving good machine-translation
    performance in an encoder-decoder framework. Irsoy and Cardie (2014)
    also report improved results from moving from a one-layer BI-RNN to an
    architecture with several layers. Many other works report result using
    layered RNN architectures, but do not explicitly compare to 1-layer
    RNNs.
  
  
  
  
  References:
  
  
  {1} Goldberg, Yoav. "A Primer on Neural Network Models for Natural Language Processing." J. Artif. Intell. Res.(JAIR) 57 (2016): 345-420.
  https://scholar.google.com/scholar?cluster=3704132192758179278&amp;hl=en&amp;as_sdt=0,5
  ; http://u.cs.biu.ac.il/~yogo/nnlp.pdf