CUDNN_STATUS_INTERNAL_ERROR when using both Pytorch and TensorFlow
I'm running a program to process some data, and I inference both a TensorFlow model and a Pytorch model. 

When inferencing either of the models everything works fine. However, when I add the pytorch input my program crashes with this error: 

2018-05-14 12:55:05.525251: E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018-05-14 12:55:05.525280: F tensorflow/core/kernels/conv_ops.cc:717] Check failed: stream-&gt;parent()-&gt;GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo&lt;T&gt;(), &amp;algorithms) 


Note that this already happens before I do anything with Pytorch. No models are loaded, nothing is put on GPU, no devices are checked. 

Does anyone know what might be going wrong, how to fix it, and if there are some parameters I can change? 

Something I already tried is disabling the PyTorch backend using this code: 

import torch.backends.cudnn as cudnn
cudnn.enabled = False


But unfortunately this does not help... 

You'll find in the NVIDIA Forums some references of cuBLAS not playing well with several Python processes interacting with it at the same time. This is referenced in this 1 year old issue for Tensorflow, but it should be the same for any multiple-PyTorch client applications interfacing with GPU through CUDA - and cuBLAS, to be more specific. cuBLAS handles weren't being properly initialized, somehow due to a mixture of issues related to on-disk caching and RAM utilization being too large. 
The solution was both to delete the on-disk cache for cuBLAS, 

sudo rm -rf ~/.nv


and restrict the amount of memory usage for nets.
