How does the cost function of LSTM works?
It is my understanding that there are no differences for loss function optimization in RNN's as compared to any other neural network models.  The loss is calculated at the very end of your network output.  

Say you have this network architecture for predicting a categorical variable:  

Layer Type, Output Size  

Input: (None, 10, 6) = (batch_size,seq len,num_features)  

LSTM: (none, 10,100) =(batch_size,sequence_len,hidden_unit_size)  

TimeDistributed Dense: (None, 10, 10) = (batch_size,sequence_len,num_categories)  

Softmax Activation: (none, 10, 10) = (batch_size, sequence_len, num_categories)  

Here, you have 10 time-steps for the sequence and the final softmax output is giving a prediction for the next value of the categorical variable at every timestep (Assuming 1-hot representation of the categorical feature).  The Y value for each time step for training is simply the 1-hot representation of the categorical value at t+1.  The LSTM layer output gives (batch_size, 10, 100).  So, you now have a 100 dimensional representation of every time-step.  You pass this to the TimeDistributed Dense and then to softmax.  So, at the very end, at the softmax output layer, you have a Y vector of size (batch_size, seq_len, 1) that contains the true output for all 10 time-steps, for every sample in the batch.  You then would use this the same exact way as any other NN model using mini-batches to calculate loss and backprop errors to network weights.