Adversarial text generation with gradient clipping, still facing gradient explosion
Have you tried tf.clip_by_value? You could set clip_value_min to be -1 and the max to be 1.
If you're using an embedding matrix for your text, the embedding values shouldn't get larger than 1
Also as an aside, people have tried using GAN's for text generation, and the results aren't very good
I am training an text generation model with Generative adversarial networks.
As a common method, I used tf.clip_by_global_norm() to avoid gradient problems. However, even I used gradient clipping, I am still facing gradient exploding problem with error caused by tf.clip_by_global_norm() function.
The document says: If global_norm == infinity then the entries in t_list are all set to NaN to signal that an error occurred.
I can hardly find the origin of problem with this situation since I believed that tf.clip_by_global_norm definitely avoid the gradient problem.