INVALID_ARGUMENT: Unable to find the relevant tensor remote_handle
@iCosmos76 Kindly check that all IP addresses and ports specified in your cluster configuration are accurate and accessible across machines. Please ensure you're using the appropriate cluster resolver (e.g., tf.distribute.cluster_resolver.TFConfigClusterResolver) and that it's correctly configured with worker and parameter server addresses. Thank you! @sushreebarsa Thank you very much for your help. Indeed, the machines I was trying to connect to were in the cloud and there is a firewall there that does not allow me to interact correctly with my local cluster members. @iCosmos76 Could you please confirm if the issue has been resolved then?

Thank you! Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62880">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62880">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

tensorflow 2.15.0.post1 

### Custom code

Yes

### OS platform and distribution

Manjaro Linux 23.1.3, Linux Ubuntu 22.04

### Mobile device

_No response_

### Python version

3.11.6

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Hello. I am using ParameterServerStrategy to train multiple workers who have different ip addresses. When I train the model on the host machine, the training is successful. But when I try to train a distributed model, this error appears: INVALID_ARGUMENT: Unable to find the relevant tensor remote_handle. Please help me.

### Standalone code to reproduce the issue

```shell
import json
import os
import sys
import pathlib
from keras.preprocessing.image import ImageDataGenerator
from tensorflow import keras
from tensorflow.keras import layers
import tensorflow as tf
import splitfolders 
from tensorflow.keras.applications import ResNet50
from keras.layers import  Dropout, Flatten, Dense

tf.debugging.experimental.enable_dump_debug_info(
    "./tfdbg2_logdir",
    tensor_debug_mode="FULL_HEALTH",
    circular_buffer_size=-1)

os.environ["GRPC_FAIL_FAST"] = "use_caller"
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

cluster_resolver = tf.distribute.cluster_resolver.TFConfigClusterResolver()

def dataset_fn(input_context):

    data_dir = "/home/Kompot/Distributed_calculations/archive/raw-img/"
    data_dir = pathlib.Path(data_dir)

    train_path='./output/train/'
    val_path='./output/val'
    test_path='./output/test'

    class_names=os.listdir(train_path)

    NUM_CLASSES = len(class_names)
    BATCH_SIZE = 32
    IMG_DIM = 224

    train_datagen = ImageDataGenerator(zoom_range=0.15,width_shift_range=0.2,height_shift_range=0.2,shear_range=0.15)
    train_generator = train_datagen.flow_from_directory(train_path,target_size=(IMG_DIM, IMG_DIM),batch_size=BATCH_SIZE)

    dataset = tf.data.Dataset.from_generator(
        lambda: train_generator,
        output_signature=(
            tf.TensorSpec(shape=[None, IMG_DIM, IMG_DIM, 3], dtype=tf.float32),
            tf.TensorSpec(shape=[None, NUM_CLASSES], dtype=tf.float32)
        )
    )

    dataset = dataset.shard(num_shards=input_context.num_input_pipelines, index=input_context.input_pipeline_id)

    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
    # dataset = dataset.shuffle(10)

    return dataset


input_dataset = tf.keras.utils.experimental.DatasetCreator(dataset_fn=dataset_fn)

print("Waiting for workers and pss to connect...")
strategy = tf.distribute.experimental.ParameterServerStrategy(cluster_resolver=cluster_resolver)

with strategy.scope():
    modelT = ResNet50(
        input_shape=(224, 224, 3),
        include_top=False,
        weights='imagenet'
    )

    for layers in modelT.layers:
        layers.trainable = False

    y = Flatten()(modelT.output)
    y = Dropout(0.5)(y)
    y = Dense(10, activation="softmax")(y)

    modelT = keras.Model(modelT.input, y)
    modelT.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"], steps_per_execution='auto')

modelT.fit(input_dataset, epochs=10, steps_per_epoch=605)
```


### Relevant log output

```shell
Epoch 1/10
2024-02-01 09:17:23.866528: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:59] Ignoring an error encountered when deleting remote tensors handles: INVALID_ARGUMENT: Unable to find the relevant tensor remote_handle: Op ID: 2344, Output num: 1
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768243.863097783","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unable to find the relevant tensor remote_handle: Op ID: 2344, Output num: 1","grpc_status":3} [type.googleapis.com/tensorflow.core.platform.ErrorSourceProto='\x08\x05']

ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:1 failed with UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768248.511244031","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768248.511239749","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768243.823390542","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768248.511244031","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768248.511239749","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
It is treated as a transient connectivity failure for now.
ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:1 failed with UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768248.511244031","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768248.511239749","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768243.823390542","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768248.511244031","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768248.511239749","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
It is treated as a transient connectivity failure for now.
ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:1 failed with UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768268.511226712","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768268.511222584","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768263.943495340","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768268.511226712","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768268.511222584","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
It is treated as a transient connectivity failure for now.
ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:1 failed with UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768268.511226712","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768268.511222584","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768263.943495340","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768268.511226712","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768268.511222584","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
It is treated as a transient connectivity failure for now.
ERROR:tensorflow: /job:worker/task:1 encountered the following error when processing closure: UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768283.943821199","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
ERROR:tensorflow: /job:worker/task:1 encountered the following error when processing closure: UnavailableError():failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768283.943821199","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
ERROR:tensorflow:Start cancelling closures due to error UnavailableError(): failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768283.943821199","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
ERROR:tensorflow:Start cancelling closures due to error UnavailableError(): failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768283.943821199","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]

ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:0 failed with CancelledError():Cancelled
Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768284.014644991","description":"Error received from peer ipv4:192.168.83.219:2224","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Cancelled","grpc_status":1} [Op:__inference_train_function_16244]
This derived error is ignored and not reported to users.
ERROR:tensorflow:Remote function on worker /job:worker/replica:0/task:0 failed with CancelledError():Cancelled
Additional GRPC error information from remote target /job:worker/replica:0/task:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768284.014644991","description":"Error received from peer ipv4:192.168.83.219:2224","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Cancelled","grpc_status":1} [Op:__inference_train_function_16244]
This derived error is ignored and not reported to users.
Traceback (most recent call last):
  File "/home/Kompot/Distributed_calculations/./coordinator.py", line 123, in <module>
    modelT.fit(input_dataset, epochs=10, steps_per_epoch=605)
  File "/home/Kompot/Distributed_calculations/venv/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnavailableError: failed to connect to all addresses
Additional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}
Additional GRPC error information from remote target /job:worker/replica:0/task:1 while calling /tensorflow.eager.EagerService/Enqueue:
:{"created":"@1706768283.943821199","description":"Error received from peer ipv4:51.250.11.138:8000","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"failed to connect to all addresses\nAdditional GRPC error information from remote target /job:ps/replica:0/task:0/device:CPU:0 while calling /tensorflow.eager.EagerService/Enqueue:\n:{"created":"@1706768288.511248869","description":"Failed to pick subchannel","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1706768288.511243624","description":"failed to connect to all addresses","file":"external/com_github_grpc_grpc/src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}","grpc_status":14} [Op:__inference_train_function_16244]
2024-02-01 09:18:06.505567: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 14720224220024848622
```
