The available server factories are: [ GRPC_SERVER ] error, while passing grpc+verbs protocol!
Hi **@Maziyar-Na** ,
I was able to reproduce the issue on Colab using TF v2.15. Please find the [gist](https://colab.sandbox.google.com/gist/Venkat6871/bb14bec5ff894b60f6b701406d06659f/62759_2-15-v.ipynb) for your reference.

Thank you! Thanks @Venkat6871 !
Is there any solution? I want to run TF 2.15 using verbs api. Hi @Maziyar-Na ,

IMO, verbs support not available in 2.x versions. There are few issues attached [here](https://github.com/tensorflow/tensorflow/issues?q=is%3Aissue+is%3Aopen+verb) for more details. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62759">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62759">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.15

### Custom code

Yes

### OS platform and distribution

Linux Ubntu 22.04

### Mobile device

N/A

### Python version

3.10

### Bazel version

6.1.0

### GCC/compiler version

clang-16

### CUDA/cuDNN version

N/A

### GPU model and memory

N/A

### Current behavior?

I am trying to run a distributed training code using multi-worker mirrored strategy and I want the code to use grpc+verbs on a RoCE-capable network card. I have built the tensorflow from the source and since the "verbs" option is not there anymore, I've added the following lines in the .bazelrc file in tensorflow root direcory:
build:verbs --define=with_verbs_support=true

And passed --config=verbs and It's compiled successfully. Now when I want to use the grpc+verbs api in the code like the following (based on [https://github.com/tensorflow/tensorflow/issues/37622]) : 

worker_index = 0  # For instance
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': {
        'worker': ["192.168.6.6:12345", "192.168.6.7:12345", "192.168.6.8:123>
    },
    'task': {'type': 'worker', 'index': worker_index},
    'rpc_layer':'grpc+verbs'
})

it gives me error:
tensorflow.python.framework.errors_impl.NotFoundError: No server factory registered for the given ServerDef
(The available server factories are: [ GRPC_SERVER ]) 

Could you please help me figure out how to run distributed training using multi-worker mirrored strategy on tensorflow using the verbs api ?



### Standalone code to reproduce the issue

```shell
import tensorflow as tf
import keras
import os
import json

def get_compiled_model():
    # Make a simple 2-layer densely-connected neural network.
    inputs = keras.Input(shape=(784,))
    x = keras.layers.Dense(256, activation="relu")(inputs)
    x = keras.layers.Dense(256, activation="relu")(x)
    outputs = keras.layers.Dense(10)(x)
    model = keras.Model(inputs, outputs)
    model.compile(
        optimizer=keras.optimizers.Adam(),
        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
        metrics=[keras.metrics.SparseCategoricalAccuracy()],
    )
    return model


def get_dataset():
    batch_size = 32
    num_val_samples = 10000

    # Return the MNIST dataset in the form of a `tf.data.Dataset`.
    (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

    # Preprocess the data (these are Numpy arrays)
    x_train = x_train.reshape(-1, 784).astype("float32") / 255
    x_test = x_test.reshape(-1, 784).astype("float32") / 255
    y_train = y_train.astype("float32")
    y_test = y_test.astype("float32")

    # Reserve num_val_samples samples for validation
    x_val = x_train[-num_val_samples:]
    y_val = y_train[-num_val_samples:]
    x_train = x_train[:-num_val_samples]
    y_train = y_train[:-num_val_samples]
    return (
        tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size),
        tf.data.Dataset.from_tensor_slices((x_val, y_val)).batch(batch_size),
        tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(batch_size),
    )


# Set TF_CONFIG
worker_index = 0  # For instance
os.environ['TF_CONFIG'] = json.dumps({
    'cluster': {
        'worker': ["192.168.6.6:12345", "192.168.6.7:12345", "192.168.6.8:12345"]
    },
    'task': {'type': 'worker', 'index': worker_index},
    'rpc_layer':'grpc+verbs'
})

# Create a MirroredStrategy.
strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
print("Number of devices: {}".format(strategy.num_replicas_in_sync))

# Open a strategy scope.
with strategy.scope():
    # Everything that creates variables should be under the strategy scope.
    # In general this is only model construction & `compile()`.
    model = get_compiled_model()

# Train the model on all available devices.
train_dataset, val_dataset, test_dataset = get_dataset()
model.fit(train_dataset, epochs=2, validation_data=val_dataset)

# Test the model on all available devices.
model.evaluate(test_dataset)
```


### Relevant log output

```shell
2024-01-08 15:35:36.165997: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From /users/maziyar/dnn.py:57: _CollectiveAllReduceStrategyExperimental.__init__ (from tensorflow.python.distribute.collective_all_reduce_strategy) is deprecated and will be removed in a future version.
Instructions for updating:
use distribute.MultiWorkerMirroredStrategy instead
2024-01-08 15:35:37.225240: E tensorflow/core/common_runtime/eager/context_distributed_manager.cc:783] No server factory registered for the given ServerDef: cluster {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "192.168.6.6:12345"
    }
    tasks {
      key: 1
      value: "192.168.6.7:12345"
    }
    tasks {
      key: 2
      value: "192.168.6.8:12345"
    }
  }
}
job_name: "worker"
default_session_config {
  device_count {
    key: "CPU"
    value: 1
  }
  device_count {
    key: "GPU"
    value: 0
  }
  device_filters: "/job:worker/task:0"
  gpu_options {
    experimental {
    }
  }
  allow_soft_placement: true
  graph_options {
    rewrite_options {
      scoped_allocator_optimization: ON
      scoped_allocator_opts {
        enable_op: "CollectiveReduce"
      }
    }
  }
  experimental {
    collective_group_leader: "/job:worker/replica:0/task:0"
    coordination_config {
      service_type: "standalone"
      service_leader: "/job:worker/replica:0/task:0"
      enable_health_check: true
      coordinated_job_list {
        name: "worker"
        num_tasks: 3
      }
    }
  }
}
protocol: "grpc+verbs"

The available server factories are: [ GRPC_SERVER ]
Traceback (most recent call last):
  File "/users/maziyar/dnn.py", line 57, in <module>
    strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py", line 383, in new_func
    return func(*args, **kwargs)
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 255, in __init__
    self).__init__(cluster_resolver, communication_options)
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 186, in __init__
    CollectiveAllReduceExtended(
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 339, in __init__
    self._initialize_strategy(self._cluster_resolver, devices=devices)
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 358, in _initialize_strategy
    self._initialize_multi_worker(cluster_resolver)
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/distribute/collective_all_reduce_strategy.py", line 530, in _initialize_multi_worker
    context.context().ensure_initialized()
  File "/users/maziyar/.local/lib/python3.10/site-packages/tensorflow/python/eager/context.py", line 619, in ensure_initialized
    pywrap_tfe.TFE_EnableCollectiveOps(context_handle, server_def_str)
tensorflow.python.framework.errors_impl.NotFoundError: No server factory registered for the given ServerDef: cluster {
  job {
    name: "worker"
    tasks {
      key: 0
      value: "192.168.6.6:12345"
    }
    tasks {
      key: 1
      value: "192.168.6.7:12345"
    }
    tasks {
      key: 2
      value: "192.168.6.8:12345"
    }
  }
}
job_name: "worker"
default_session_config {
  device_count {
    key: "CPU"
    value: 1
  }
  device_count {
    key: "GPU"
    value: 0
  }
  device_filters: "/job:worker/task:0"
  gpu_options {
    experimental {
    }
  }
  allow_soft_placement: true
  graph_options {
    rewrite_options {
      scoped_allocator_optimization: ON
      scoped_allocator_opts {
        enable_op: "CollectiveReduce"
      }
    }
  }
  experimental {
    collective_group_leader: "/job:worker/replica:0/task:0"
    coordination_config {
      service_type: "standalone"
      service_leader: "/job:worker/replica:0/task:0"
      enable_health_check: true
      coordinated_job_list {
        name: "worker"
        num_tasks: 3
      }
    }
  }
}
protocol: "grpc+verbs"

The available server factories are: [ GRPC_SERVER ]
```
