How to ignore vectors of zeros (i.e. paddings) in Keras?
You can use Masking layer (with mask value of zero) before LSTM layer in order to ignore all timesteps with only zeros (i.e. zeros vector). You can find more information about this layer on its documentation. Here is an example from documentation which uses Masking layer:


  Consider a Numpy data array x of shape (samples, timesteps, features),
  to be fed to an LSTM layer. You want to mask sample #0 at timestep #3,
  and sample #2 at timestep #5, because you lack features for these
  sample timesteps. You can do:
  
  
  set x[0, 3, :] = 0. and x[2, 5, :] = 0.
  insert a Masking layer with mask_value=0. before the LSTM layer:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))