Where does the name 'LSTM' come from?
In Sepp Hochreiter's original paper on the LSTM   where he introduces the algorithm and method to the scientific community, he explains that the long term memory refers to the learned weights and the short term memory refers to the gated cell state values that change with each step through time t. 

edit: quote from paper "Recurrent networks can in principle use their feedback connections to store representations of recent input events in the form of activations ("short-term memory", as opposed to "long-term memory embodied by slowly changing weights)"