 TensorFlow device (GPU:0) is being mapped to multiple devices when using tf.estimator api and set visible_device_list with hvd.local_rank()
Hi @mmseerrttt ,

It seems you are using TF1.x related code which is not supported now. I have gone through the [horovod](https://horovod.readthedocs.io/en/stable/tensorflow.html) documentation and found below notes.

<img width="821" alt="Screenshot 2023-06-08 at 9 50 23 AM" src="https://github.com/tensorflow/tensorflow/assets/116063290/e0dc1706-ee31-4638-85f6-faa55ff57373">

Also [Estimator](https://www.tensorflow.org/guide/estimator) API is deprecated now.Please refer below note for same.

Warning: Estimators are not recommended for new code. Estimators run [v1.Session](https://www.tensorflow.org/api_docs/python/tf/compat/v1/Session)-style code which is more difficult to write correctly, and can behave unexpectedly, especially when combined with TF 2 code. Estimators do fall under our [compatibility guarantees](https://tensorflow.org/guide/versions), but will receive no fixes other than security vulnerabilities. See the [migration guide](https://tensorflow.org/guide/migrate) for details.


 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60810">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60810">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tf 2.11

### Custom Code

Yes

### OS Platform and Distribution

Linux Ubuntu 20.04.1

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

12.0

### GPU model and memory

_No response_

### Current Behaviour?

I know there is a same issue, https://github.com/tensorflow/tensorflow/issues/58952, but it has been closed.

The model implemented with tf.estimator api. horovod multi-cards job (horovodrun -np 2 python xxx.py).

it works well using 2.10, but throws Device mapping error when using TF2.11.

FROM [issue58952]( https://github.com/tensorflow/tensorflow/issues/58952), It seems that this error is relative to this [commit]( https://github.com/tensorflow/tensorflow/commit/16ea0f8995a9b087ff7e50bbf5a03e58e3c5002e#:~:text=if-,context.is_custom_device,-(device_string)%3A). When calling context.is_cutom_device(), this API would create TFE_Context that would create TF devices. However, program would also create TF devices when create Session with the aboved mentioned config, and these two TF device creation would conflicts.

It seems that the [session_options used in the ensure_initialized](https://github.com/tensorflow/tensorflow/blob/301f145ca3979b5155a166cbe6cb8c2a3af1bab1/tensorflow/python/eager/context.py#L581) function to [create NewContext](https://github.com/tensorflow/tensorflow/blob/301f145ca3979b5155a166cbe6cb8c2a3af1bab1/tensorflow/python/eager/context.py#L598) are inconsistent with the options used by the NewSession. The options used by NewSession are explicitly set through tf.ConfigProto.gpu_options.visible_device_list=hvd.localrank(). session_config should be at least consistent, or loaded once at most

To reproduce this error, you need to install horovod and using multi card machine to run below code with:

horovodrun -np N python xxx.py

where N denotes the number of processes to be started.


### Standalone code to reproduce the issue

```shell
mport pandas as pd
import tensorflow as tf
import tensorflow.compat.v1 as tf1
global is_mpi
try:
    import horovod.tensorflow as hvd
    hvd.init()
    is_mpi = hvd.size()
except ImportError:
    is_mpi = 0
    print("No MPI horovod support, this is running in no-MPI mode!")

session_config = tf1.ConfigProto()
if is_mpi:
  session_config.gpu_options.visible_device_list = str(hvd.local_rank())

run_config = tf.estimator.RunConfig(session_config=session_config)

x_train = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/train.csv')
x_train['sex'].replace(('male', 'female'), (0, 1), inplace=True)
x_train['alone'].replace(('n', 'y'), (0, 1), inplace=True)
x_train['class'].replace(('First', 'Second', 'Third'), (1, 2, 3), inplace=True)
x_train.drop(['embark_town', 'deck'], axis=1, inplace=True)
y_train = x_train.pop('survived')

# Data setup for TensorFlow 1 with `tf.estimator`
def _input_fn():
  return tf1.data.Dataset.from_tensor_slices((dict(x_train), y_train)).batch(32)

FEATURE_NAMES = [
    'age', 'fare', 'sex', 'n_siblings_spouses', 'parch', 'class', 'alone'
]
feature_columns = []
for fn in FEATURE_NAMES:
  feat_col = tf1.feature_column.numeric_column(fn, dtype=tf.float32)
  feature_columns.append(feat_col)

linear_estimator = tf.estimator.LinearEstimator(
    head=tf.estimator.BinaryClassHead(),
    feature_columns=feature_columns,
    model_dir="./model",
    config=run_config)
linear_estimator.train(input_fn=_input_fn, steps=1000)
```


### Relevant log output

```shell
[1,0]<stderr>:2023-06-08 02:16:09.394853: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
[1,1]<stderr>:2023-06-08 02:16:09.397900: E tensorflow/core/common_runtime/session.cc:91] Failed to create session: ALREADY_EXISTS: TensorFlow device (GPU:0) is being mapped to multiple devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083
[1,1]<stderr>:2023-06-08 02:16:09.397922: E tensorflow/c/c_api.cc:2209] ALREADY_EXISTS: TensorFlow device (GPU:0) is being mapped to multiple devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083
[1,1]<stderr>:Traceback (most recent call last):
[1,1]<stderr>:  File "test.py", line 43, in <module>
[1,1]<stderr>:    linear_estimator.train(input_fn=_input_fn, steps=1000)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 360, in train
[1,1]<stderr>:    loss = self._train_model(input_fn, hooks, saving_listeners)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1186, in _train_model
[1,1]<stderr>:    return self._train_model_default(input_fn, hooks, saving_listeners)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1217, in _train_model_default
[1,1]<stderr>:    return self._train_with_estimator_spec(estimator_spec, worker_hooks,
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1512, in _train_with_estimator_spec
[1,1]<stderr>:    with training.MonitoredTrainingSession(
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 606, in MonitoredTrainingSession
[1,1]<stderr>:    return MonitoredSession(
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 1050, in __init__
[1,1]<stderr>:    super(MonitoredSession, self).__init__(
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 753, in __init__
[1,1]<stderr>:    self._sess = _RecoverableSession(self._coordinated_creator)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 1259, in __init__
[1,1]<stderr>:    _WrappedSession.__init__(self, self._create_session())
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 1264, in _create_session
[1,1]<stderr>:    return self._sess_creator.create_session()
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 906, in create_session
[1,1]<stderr>:    self.tf_sess = self._session_creator.create_session()
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/monitored_session.py", line 665, in create_session
[1,1]<stderr>:    return self._get_session_manager().prepare_session(
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/session_manager.py", line 309, in prepare_session
[1,1]<stderr>:    sess, is_loaded_from_checkpoint = self._restore_checkpoint(
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/training/session_manager.py", line 218, in _restore_checkpoint
[1,1]<stderr>:    sess = session.Session(self._target, graph=self._graph, config=config)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1604, in __init__
[1,1]<stderr>:    super(Session, self).__init__(target, graph, config=config)
[1,1]<stderr>:  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 712, in __init__
[1,1]<stderr>:    self._session = tf_session.TF_NewSessionRef(c_graph, opts)
[1,1]<stderr>:tensorflow.python.framework.errors_impl.AlreadyExistsError: TensorFlow device (GPU:0) is being mapped to multiple devices (1 now, and 0 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083
```
</details>