How to debug out of memory in tensorflow2-gpu
These did not occur in 
2.0 alpha TensorFlow but in 2.0. 
Pip installs tensorflow-gpu==2.0.0: has 
leaked memory!
Pip install tensorflow-gpu==2.0.0-alpha: 
it's all right!
Try it out
I am using tensorflow-2 gpu with tf.data.Dataset.
Training on small models works.
When training a bigger model, everything works at first : gpu is used, the first epoch works with no trouble (but I am using most of my gpu memory).
At validation time, I run into a CUDA_ERROR_OUT_OF_MEMORY with various allocation with a smaller and smaller amount of bytes that could not be allocated (ranging from 922Mb to 337Mb).
I currently have no metrics and no callbacks and am using tf.keras.Model.fit.
If I remove the validation data, the training continues.
What is my issue ? how can I debug this ?
In tf1, I could use RunOptions(report_tensor_allocations_upon_oom=True), does any equivalent exist in tf2 ? 
This occurs with tensorflow==2.1.0 .