Get deadlock after Predict(cuda10.0, cudnn7.6.5,  Tesla T4 GPU)
@ivankxt The TF v2.2 is an older version which is not actively supported. We would recommend you to kindly upgrade to the latest TF version and let us know if the issue still persists?
Thank you!  This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60841">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60841">No</a>
 @ivankxt As per the [documentation](https://www.tensorflow.org/install/source#gpu) could you please try with the cudnn version of 7.4 which will be compatible with the cuda version that you are using?  Also we recommend you to use the latest stable version as TF v2.2 is not actively supported.  The issues would be lesser in the newer version as compared to the older version. please follow the instructions for [gpu](https://www.tensorflow.org/install/pip) support as well. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60841">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60841">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

source

### Tensorflow Version

tf2.2 + tfserving2.2

### Custom Code

Yes

### OS Platform and Distribution

centos7

### Mobile device

_No response_

### Python version

3.6

### Bazel version

3.7.2

### GCC/Compiler version

7.5

### CUDA/cuDNN version

7.6.5

### GPU model and memory

15G

### Current Behaviour?

In our inference service, when executes the predict interface(predictor_->Predict(...)), it gets deadlock.

`
std::unique_ptr<tensorflow::serving::TensorflowPredictor> predictor_;
predictor_->Predict(opt, core_.get(), predict_req, &predict_resp, run_metadata.get());
`

Here is the pstack：

obviously, it's in async execute, and waiting for something

Thread 167 (Thread 0x7f4b1bfa7700 (LWP 81084)):
#0  0x00007f5059939c09 in syscall () from /usr/lib64/libc.so.6
#1  0x00007f505feb1bbb in nsync::nsync_mu_semaphore_p_with_deadline(nsync::nsync_semaphore_s_*, timespec) () from /home/qspace/upload/libtensorflow_serving.so
#2  0x00007f505feaedf9 in nsync::nsync_sem_wait_with_cancel_(nsync::waiter*, timespec, nsync::nsync_note_s_*) () from /home/qspace/upload/libtensorflow_serving.so
#3  0x00007f505feafeeb in nsync::nsync_cv_wait_with_deadline_generic(nsync::nsync_cv_s_*, void*, void (*)(void*), void (*)(void*), timespec, nsync::nsync_note_s_*) () from /home/qspace/upload/libtensorflow_serving.so
#4  0x00007f505feb03c3 in nsync::nsync_cv_wait_with_deadline(nsync::nsync_cv_s_*, nsync::nsync_mu_s_*, timespec, nsync::nsync_note_s_*) () from /home/qspace/upload/libtensorflow_serving.so
#5  0x00007f506168349c in tensorflow::DirectSession::WaitForNotification(tensorflow::Notification*, long long) () from /home/qspace/upload/libtensorflow_serving.so
#6  0x00007f50616834ed in tensorflow::DirectSession::WaitForNotification(tensorflow::Notification*, tensorflow::DirectSession::RunState*, tensorflow::CancellationManager*, long long) () from /home/qspace/upload/libtensorflow_serving.so
#7  0x00007f5061693bb5 in tensorflow::DirectSession::RunInternal(long long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /home/qspace/upload/libtensorflow_serving.so
#8  0x00007f5061695dd5 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::string, tensorflow::Tensor>, std::allocator<std::pair<std::string, tensorflow::Tensor> > > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&) () from /home/qspace/upload/libtensorflow_serving.so
#9  0x00007f5061681313 in tensorflow::DirectSession::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::string, tensorflow::Tensor>, std::allocator<std::pair<std::string, tensorflow::Tensor> > > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) () from /home/qspace/upload/libtensorflow_serving.so
#10 0x00007f5067131cdc in tensorflow::serving::ServingSessionWrapper::Run(tensorflow::RunOptions const&, std::vector<std::pair<std::string, tensorflow::Tensor>, std::allocator<std::pair<std::string, tensorflow::Tensor> > > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<std::string, std::allocator<std::string> > const&, std::vector<tensorflow::Tensor, std::allocator<tensorflow::Tensor> >*, tensorflow::RunMetadata*) () from /home/qspace/upload/libtensorflow_serving.so
#11 0x00007f506714092b in tensorflow::serving::internal::RunPredict(tensorflow::RunOptions const&, tensorflow::MetaGraphDef const&, tensorflow::serving::optional<long long> const&, tensorflow::serving::internal::PredictResponseTensorSerializationOption, tensorflow::Session*, tensorflow::serving::PredictRequest const&, tensorflow::serving::PredictResponse*, tensorflow::RunMetadata*) () from /home/qspace/upload/libtensorflow_serving.so
#12 0x00007f5067131aa0 in tensorflow::serving::TensorflowPredictor::PredictWithModelSpec(tensorflow::RunOptions const&, tensorflow::serving::ServerCore*, tensorflow::serving::ModelSpec const&, tensorflow::serving::PredictRequest const&, tensorflow::serving::PredictResponse*, tensorflow::RunMetadata*) () from /home/qspace/upload/libtensorflow_serving.so
#13 0x00007f5067131c81 in tensorflow::serving::TensorflowPredictor::Predict(tensorflow::RunOptions const&, tensorflow::serving::ServerCore*, tensorflow::serving::PredictRequest const&, tensorflow::serving::PredictResponse*, tensorflow::RunMetadata*) () from /home/qspace/upload/libtensorflow_serving.so
#14 0x0000000002bd91b1 in mmfinderbd::RankModel::Predict (this=0x327bf080, req=..., resp=0x7f47042735d0) at bdegateway/mmfinder/mmfinderbdetfsvr/models/tf/rank_tf_model.cpp:525
#15 0x0000000002b838cc in mmfinderbd::ServerCoreSingleModel::Predict (this=0x2c176aa0 <mmfinderbd::ServerCore::Instance()::instance>, req=..., resp=...) at bdegateway/mmfinder/mmfinderbdetfsvr/core/server_core_single_model.cpp:365
#16 0x0000000002b5df8b in MMFinderBdeTfSvrServiceImpl_PB::InferImpl (this=0x7f4ad84cbec0, head_uin=<optimized out>, req=..., resp=0x7f47042735d0) at bdegateway/mmfinder/mmfinderbdetfsvr/mmfinderbdetfsvrserviceimpl_pb.cpp:93
#17 0x0000000002b731d8 in MMFinderBdeTfSvrDispatcher_PB::Infer (this=this@entry=0x7f4ad84cbe60, uin=<optimized out>, req_buffer=req_buffer@entry=0x7f4ad84cb868, resp_buffer=resp_buffer@entry=0x7f4ad84cb870) at bazel-out/cd7t-opt/genfiles/bdegateway/mmfinder/mmfinderbdetfsvr/skgenerated/sk_mmfinderbdetfsvrdispatcher.pb.cpp:1366
#18 0x0000000002b78569 in MMFinderBdeTfSvrDispatcher_PB::Dispatch (this=this@entry=0x7f4ad84cbe60) at bazel-out/cd7t-opt/genfiles/bdegateway/mmfinder/mmfinderbdetfsvr/skgenerated/sk_mmfinderbdetfsvrdispatcher.pb.cpp:398
#19 0x0000000002b5a91e in MMFinderBdeTfSvrServer::SKServerProc (this=<optimized out>, ctrl_info=0x7f470421b820, ctx=0x7f470421b7a0, in_pkg=0x7f42681054a0, out_pkg=0x7f42681054e0, args=<optimized out>) at ./bdegateway/mmfinder/mmfinderbdetfsvr/mmfinderbdetfsvrserver.h:44
#20 0x000000000671a6c8 in SMCoWorkerMt::CoWorkerIORun (this=0x330b4670, self=0x7f470421b6d0) at comm2/summer/smcoworker.cpp:1138
#21 0x0000000007cc679e in operator() (this=0x7f470421b908) at /home/mmdev/gcc7/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
#22 CoRoutineFunc (co=0x7f470421b8f0) at basic/colib/co_routine.cpp:601
#23 0x0000000000000000 in ?? ()

and what does it exactly waiting for...

Thread 301 (Thread 0x7f4b09dcf700 (LWP 80869)):
#0  0x00007f5059939c09 in syscall () from /usr/lib64/libc.so.6
#1  0x0000000005bf5111 in WaitUntil (t=..., val=0, v=0x7f45f046b750) at mm3rd/abseil-cpp/absl/synchronization/internal/waiter.cc:107
#2  absl::lts_2020_02_25::synchronization_internal::Waiter::Wait (this=this@entry=0x7f45f046b750, t=t@entry=...) at mm3rd/abseil-cpp/absl/synchronization/internal/waiter.cc:151
#3  0x0000000005bf5052 in AbslInternalPerThreadSemWait (t=...) at mm3rd/abseil-cpp/absl/synchronization/internal/per_thread_sem.cc:93
#4  0x00007f5066f87b6d in absl::Mutex::Block(absl::base_internal::PerThreadSynch*) () from /home/qspace/upload/libtensorflow_serving.so
#5  0x00007f5066f8889e in absl::Mutex::LockSlowLoop(absl::SynchWaitParams*, int) () from /home/qspace/upload/libtensorflow_serving.so
#6  0x00007f5066f88dc2 in absl::Mutex::LockSlowWithDeadline(absl::MuHowS const*, absl::Condition const*, absl::synchronization_internal::KernelTimeout, int) () from /home/qspace/upload/libtensorflow_serving.so
#7  0x00007f505f0897ec in absl::Mutex::LockSlow(absl::MuHowS const*, absl::Condition const*, int) () from /home/qspace/upload/libtensorflow_serving.so
#8  0x00007f50601c56fe in stream_executor::gpu::CUDABlas::DoBlasGemm(stream_executor::Stream*, stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, float, stream_executor::DeviceMemory<float> const&, int, stream_executor::DeviceMemory<float> const&, int, float, stream_executor::DeviceMemory<float>*, int) () from /home/qspace/upload/libtensorflow_serving.so
#9  0x00007f5060290c13 in stream_executor::Stream::ThenBlasGemm(stream_executor::blas::Transpose, stream_executor::blas::Transpose, unsigned long long, unsigned long long, unsigned long long, float, stream_executor::DeviceMemory<float> const&, int, stream_executor::DeviceMemory<float> const&, int, float, stream_executor::DeviceMemory<float>*, int) () from /home/qspace/upload/libtensorflow_serving.so
#10 0x00007f5063aebc4f in tensorflow::LaunchMatMul<Eigen::GpuDevice, float, true>::launch(tensorflow::OpKernelContext*, tensorflow::Tensor const&, tensorflow::Tensor const&, Eigen::array<Eigen::IndexPair<long>, 1ul> const&, std::vector<long long, std::allocator<long long> >*, bool, tensorflow::Tensor*) () from /home/qspace/upload/libtensorflow_serving.so
#11 0x00007f5063aec42d in tensorflow::MatMulOp<Eigen::GpuDevice, float, true>::Compute(tensorflow::OpKernelContext*) () from /home/qspace/upload/libtensorflow_serving.so
#12 0x00007f5061878296 in tensorflow::BaseGPUDevice::Compute(tensorflow::OpKernel*, tensorflow::OpKernelContext*) () from /home/qspace/upload/libtensorflow_serving.so
#13 0x00007f50614a80bf in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long) () from /home/qspace/upload/libtensorflow_serving.so
#14 0x00007f50614a8c7f in std::_Function_handler<void (), tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(absl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8ul, std::allocator<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode> >*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*)::{lambda()#2}>::_M_invoke(std::_Any_data const&) () from /home/qspace/upload/libtensorflow_serving.so
#15 0x00007f506189a71f in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::ScheduleWithHint(std::function<void ()>, int, int) () from /home/qspace/upload/libtensorflow_serving.so
#16 0x00007f506189dd1b in tensorflow::thread::ThreadPool::Schedule(std::function<void ()>) () from /home/qspace/upload/libtensorflow_serving.so
#17 0x00007f5061681bb3 in std::_Function_handler<void (std::function<void ()>), tensorflow::DirectSession::RunInternal(long long, tensorflow::RunOptions const&, tensorflow::CallFrameInterface*, tensorflow::DirectSession::ExecutorsAndKeys*, tensorflow::RunMetadata*, tensorflow::thread::ThreadPoolOptions const&)::{lambda(tensorflow::DirectSession::PerPartitionExecutorsAndLib const&, tensorflow::Executor::Args*)#7}::operator()(tensorflow::DirectSession::PerPartitionExecutorsAndLib const&, tensorflow::Executor::Args*) const::{lambda(std::function<void ()>)#1}>::_M_invoke(std::_Any_data const&, std::function<void ()>&&) () from /home/qspace/upload/libtensorflow_serving.so
#18 0x00007f506149ac84 in tensorflow::(anonymous namespace)::ExecutorState::ScheduleReady(absl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8ul, std::allocator<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode> >*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) [clone .part.508] () from /home/qspace/upload/libtensorflow_serving.so
#19 0x00007f50614a3a84 in tensorflow::(anonymous namespace)::ExecutorState::NodeDone(tensorflow::Status const&, absl::InlinedVector<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, 8ul, std::allocator<tensorflow::(anonymous namespace)::ExecutorState::TaggedNode> >*, tensorflow::NodeExecStatsInterface*, tensorflow::(anonymous namespace)::ExecutorState::TaggedNodeReadyQueue*) () from /home/qspace/upload/libtensorflow_serving.so
#20 0x00007f50614a8e4f in tensorflow::(anonymous namespace)::ExecutorState::Process(tensorflow::(anonymous namespace)::ExecutorState::TaggedNode, long long)::{lambda()#6}::operator()() const () from /home/qspace/upload/libtensorflow_serving.so
#21 0x00007f50614fcbd0 in std::_Function_handler<void (tensorflow::Status const&), tensorflow::(anonymous namespace)::IntraProcessRecvAsyncImpl(tensorflow::DeviceMgr const*, tensorflow::LocalRendezvous*, tensorflow::RendezvousInterface::ParsedKey const&, tensorflow::RendezvousInterface::Args const&, std::function<void (tensorflow::Status const&, tensorflow::RendezvousInterface::Args const&, tensorflow::RendezvousInterface::Args const&, tensorflow::Tensor const&, bool)>)::{lambda(tensorflow::Status const&, tensorflow::RendezvousInterface::Args const&, tensorflow::RendezvousInterface::Args const&, tensorflow::Tensor const&, bool)#2}::operator()(tensorflow::Status const&, tensorflow::RendezvousInterface::Args const&, tensorflow::RendezvousInterface::Args const&, tensorflow::Tensor const&, bool)::{lambda(tensorflow::Status const&)#1}>::_M_invoke(std::_Any_data const&, tensorflow::Status const&) () from /home/qspace/upload/libtensorflow_serving.so
#22 0x00007f506186fa29 in tensorflow::GPUUtil::CopyCPUTensorToGPU(tensorflow::Tensor const*, tensorflow::DeviceContext const*, tensorflow::Device*, tensorflow::Tensor*, std::function<void (tensorflow::Status const&)>, bool)::{lambda()#2}::operator()() const () from /home/qspace/upload/libtensorflow_serving.so
#23 0x00007f506189c3e1 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int) () from /home/qspace/upload/libtensorflow_serving.so
#24 0x00007f50618990f3 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /home/qspace/upload/libtensorflow_serving.so
#25 0x0000000007f0a9df in std::execute_native_thread_routine (__p=0x7f4a83d29490) at ../../../../../gcc-7.5.0/libstdc++-v3/src/c++11/thread.cc:83
#26 0x00007f505a533dc5 in start_thread () from /usr/lib64/libpthread.so.0
#27 0x00007f505993f74d in clone () from /usr/lib64/libc.so.6

GPU Info:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.116.03   Driver Version: 525.116.03   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+====
|   0  Tesla T4            On   | 00000000:00:0B.0 Off |                    0 |
| N/A   52C    P0    36W /  70W |   4695MiB / 15360MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+


Please give me some advice! 

### Standalone code to reproduce the issue

```shell
std::unique_ptr<tensorflow::serving::TensorflowPredictor> predictor_;
predictor_->Predict(opt, core_.get(), predict_req, &predict_resp, run_metadata.get());
```


### Relevant log output

_No response_</details>