Keras OOM for data validation using GPU I'm trying to run a deep model using GPU and seems Keras running the validation against the whole validation data set in one batch instead of validating in many batches and that's causing out of memory problem
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[160000,64,64,1] and type double on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:GatherV2]
I did not have this problem when I was running on CPU, it's just happening when I'm running on GPU, my fit code looks like this
history =model.fit(patches_imgs_train,patches_masks_train, batch_size=8, epochs=10,
  shuffle=True, verbose=1, validation_split=0.2) 

When I delete the validation parameter from the fit method the code works, but I need the validation.
There may be two causes to your problem:

In validation the network needs more memory.
There is another problem not directly related to this.

In this Keras issue you can find a discussion of a very similar problem. Basically, you can try:

Reducing the batch size.
If you are using Tensorboad, try disabling it or setting its batch_size parameter.