How to load a model locally, while the model is saved in distributed training remote machines (number of PS(parameter server)>1)?
Hi @cnglen ,

There is a list of known limitations documented [here](https://www.tensorflow.org/tutorials/distribute/parameter_server_training#parameterserverstrategy_general) regarding `ParameterServerStrategy` .

It is not supported to load a saved_model via `tf.saved_model.load` containing sharded variables. 

You may try loading such a saved_model using TensorFlow Serving is expected to work (refer to the [serving tutorial](https://www.tensorflow.org/tfx/tutorials/serving/rest_simple) for details).

Hope this helps. Thanks!
 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60560">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60560">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Support

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

tf2.8

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

Can't load the saved model(saved in distributed training, while number of PS(parameter server)>1).


``` python
# run in  local machine
# model is trained in remote machines
model = tf.keras.models.load_model(d_model)
```

Error Message:

``` bash
Loading a saved_model containing ShardedVariable via `tf.saved_model.load` is not supported. If the model is built using Keras, please use `tf.keras.models.load_model` instead
```

Tried the following saving options, not work. 
1. save_options = tf.saved_model.SaveOptions(experimental_io_device="/job:chief/replica:0/task:0/device:CPU:0")
2. save_options = tf.saved_model.SaveOptions(experimental_io_device="/job:localhost/replica:0/task:0/device:CPU:0")



### Standalone code to reproduce the issue

```shell
self.model = tf.keras.models.load_model(d_model, options=load_locally)
  File "lib/python3.7/site-packages/keras/utils/traceback_utils.py", line 71, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "lib/python3.7/site-packages/tensorflow/python/distribute/sharded_variable.py", line 864, in _raise_when_load
    'Loading a saved_model containing ShardedVariable via '
ValueError: Loading a saved_model containing ShardedVariable via `tf.saved_model.load` is not supported. If the model is built using Keras, please use `tf.keras.models.load_model` instead.
```


### Relevant log output

_No response_</details>