Encountering "Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED" on a previously working system
There was indeed a system-wide upgrade.
Updating cuda to cuda 10.2 and nvidia-driver to 440 and making libcudnn7 7.6.5 fixed the problem.
Everything was ok around a week ago.
Even though I am running on a server, I really don't think much has changed. 
Wonder what could have caused it.
Tensorflow has version 2.1.0-dev20191015
Anyway, here is the GPU status:
NVIDIA-SMI 430.50
Driver Version: 430.50
CUDA Version: 10.1 
Epoch 1/5
2019-11-29 22:08:00.334979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-11-29 22:08:00.644569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-11-29 22:08:00.647191: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-11-29 22:08:00.647309: E tensorflow/stream_executor/cuda/cuda_dnn.cc:337] Possibly insufficient driver version: 430.50.0
2019-11-29 22:08:00.647347: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cudnn_rnn_ops.cc:1510 : Unknown: Fail to find the dnn implementation.
2019-11-29 22:08:00.647393: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Fail to find the dnn implementation.
At the end, I get:
UnknownError:  [_Derived_]  Fail to find the dnn implementation.
     [[{{node CudnnRNN}}]]
     [[sequential/bidirectional/forward_lstm/StatefulPartitionedCall]] [Op:__inference_distributed_function_18158]
Function call stack:
distributed_function -> distributed_function -> distributed_function
The code gets traced back to here:
174         history = model.fit(training_input, training_output,  epochs=EPOCHES,
    175                             batch_size=BATCH_SIZE,
--> 176                             validation_split=0.1)
Thank you.