Why can't LSTMs keep track of the "important parts" of a sequence?
The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places.

To prove this point, I recommend reading Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons".

Addressing specifically this comment in your question:


  I suspect another reason why the LSTM is not able to infer the format is that the comma can be placed in different indexes of the sequence, so it could be losing its importance in the hidden state the longer the sequence is (not sure if that makes sense).


If commas are relevant to solve the task at hand, LSTMs can learn to remember its position or related information. This information is not necessarily diluted by repeated application of reccurrence with long sequences: networks can learn to propagate and promote crucial information from one recurrent state to the next.



Input sequences have arbitrary length, which means that LSTMs need to compress information about seen sequence elements

Rather, the input sequence has an arbitrary length, while the LSTM state vectors have a fixed size. State vectors are the only way for an LSTM to "keep track of important parts". This means that those fixed-size vectors are a bottleneck and there is an information-theoretic upper bound on the amount of information about "important parts" that can be kept by an LSTM.

LSTMs potentially take multiple decisions. For each decision, something else in the input sequence is most important

For tasks such as summarization that you mention in the question, an LSTM makes a series of predictions (predicting the tokens of the summary one token at a time). For each prediction, different things in the input sequence might be important. Put another way, for each decision, another view of the input may be most helpful.

This is a key motivation for using attention networks. Each time an LSTM is making a prediction, an attention network can provide a dynamic, optimally helpful view of the input sequence.