KL divergence loss goes to zero while training VAE
take a look at the paper &quot;Generating Sentences from a Continuous Space&quot; by Bowman. In Section 3.1 it is explained why LSTM_VAE tend to this behaviour:
&quot;This problematic tendency in learning is compounded by the lstm decoder’s sensitivity to subtle variation in the hidden states, such as that introduced by the posterior sampling process. This
causes the model to initially learn to ignore ~z and
go after low hanging fruit, explaining the data with
the more easily optimized decoder. Once this has
happened, the decoder ignores the encoder and little to no gradient signal passes between the two,
yielding an undesirable stable equilibrium with the
kl cost term at zero. We propose two techniques
to mitigate this issue.&quot;
....