TF Distributed looks for libcublasLt.so.10, when installed CUDA version is 11.6
Hi, @SuperSecureHuman 

Aplogize for the delay and It seems like there is some problem with `cudatoolkit` and the `cudnn` version could be incompatible with your tensorflow version. If possible could you please help me with the output of these two commands `nvcc --version` and `nvidia-smi` and as per our [official documentation](https://www.tensorflow.org/install/source#gpu) with Tested build configurations for `tensorflow-2.11.0`,`CUDA==11.2` and `cuDNN==8.1` should work. Thank you! Hey!

Here you go

Nvidia-smi

![image](https://user-images.githubusercontent.com/88489071/219923465-f80a2f77-fcac-4fe1-9c02-e2e0c5189672.png)


nvcc --version

```
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0
```


Cuda 11.2 still has the library version at 11, while TF looks for libcublas10.

![image](https://user-images.githubusercontent.com/88489071/219923593-fbd7bc1f-9a7c-46b5-a290-090c4498a070.png)


Anyway, loading 11.2/8.1 and running the same training script on two nodes.

 I have no idea what happened, but the training went fine. Even with cuda 11.6

Because of the queue in the HPC, I am unable to have an interactive session for longer time, through which I can look into this once again.

I'll update once again here, after I try it again I am unable to reproduce the bug now. You may close the issue if needed. I'll reopen if I get it again. Hi, @SuperSecureHuman 

Thank you for the confirmation and it seems like the issue got resolved if you're able to reproduce the bug again, so I would request you to please post new issue and we will look into that issue. Could you please confirm if this issue is resolved for you now ? Please feel free to close the issue if it is resolved ? Thank you! This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.
 I was not able to reproduce the issue again, I'll create a new issue if I get it again. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59710">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59710">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

2.8, 2.11

### Custom Code

Yes

### OS Platform and Distribution

RHEL 7.9

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

11.6/8.4

### GPU model and memory

_No response_

### Current Behaviour?

```shell
When using tf multi worker, the chief node looks for libcublasLt from CUDA 10.2, while the loaded CUDA version is 11.6.

The expected behaviour is that TensorFlow should be using libcublasLt from the currently available CUDA version rather than 10.2

The worker node exits with a communication error - Connection reset by peer
```


### Standalone code to reproduce the issue

```shell
https://www.tensorflow.org/tutorials/distribute/multi_worker_with_keras#train_the_model
```


### Relevant log output

```shell
2023-02-16 09:26:37.626920: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-16 09:26:38.332105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 11641 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:3b:00.0, compute capability: 7.0
2023-02-16 09:26:38.334955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:worker/replica:0/task:0/device:GPU:0 with 11641 MB memory:  -> device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:3b:00.0, compute capability: 7.0
2023-02-16 09:26:38.341938: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> 10.149.255.254:12345, 1 -> 10.149.0.4:23456}
2023-02-16 09:26:38.342153: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:437] Started server with target: grpc://10.149.255.254:12345
2023-02-16 09:26:44.265969: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:776] AUTO sharding policy will apply DATA sharding policy as it failed to apply FILE sharding policy because of the following reason: Found an unshardable source dataset: name: "TensorSliceDataset/_2"
op: "TensorSliceDataset"
input: "Placeholder/_0"
input: "Placeholder/_1"
attr {
  key: "Toutput_types"
  value {
    list {
      type: DT_FLOAT
      type: DT_INT64
    }
  }
}
attr {
  key: "_cardinality"
  value {
    i: 60000
  }
}
attr {
  key: "is_files"
  value {
    b: false
  }
}
attr {
  key: "metadata"
  value {
    s: "\n\024TensorSliceDataset:0"
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: 28
        }
        dim {
          size: 28
        }
      }
      shape {
      }
    }
  }
}
experimental_type {
  type_id: TFT_PRODUCT
  args {
    type_id: TFT_DATASET
    args {
      type_id: TFT_PRODUCT
      args {
        type_id: TFT_TENSOR
        args {
          type_id: TFT_FLOAT
        }
      }
      args {
        type_id: TFT_TENSOR
        args {
          type_id: TFT_INT64
        }
      }
    }
  }
  args {
    type_id: TFT_DATASET
    args {
      type_id: TFT_PRODUCT
      args {
        type_id: TFT_TENSOR
        args {
          type_id: TFT_FLOAT
        }
      }
      args {
        type_id: TFT_TENSOR
        args {
          type_id: TFT_INT64
        }
      }
    }
  }
}

Epoch 1/3
Could not load library libcublasLt.so.10. Error: libcublasLt.so.10: cannot open shared object file: No such file or directory
```
</details>