So what's the catch with LSTM?
You are right that LSTMs work very well for some problems, but some of the drawbacks are:


LSTMs take longer to train
LSTMs require more memory to train
LSTMs are easy to overfit
Dropout is much harder to implement in LSTMs
LSTMs are sensitive to different random weight initializations


These are in comparison to a simpler model like a 1D conv net, for example.

The first three items are because LSTMs have more parameters.