We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
[Horovod on Ray]Failed on example of tensorflow2_mnist_ray.py
**Environment:**
1. Framework: (TensorFlow1.15.0)
3. Horovod version:0.20.0

**Your question:**
Please ask your question here.
Hi @tgaddair @richardliaw I tried the example of Horovod on ray on CPU nodes, but failed.
The ray cluster is ok.

Using TensorFlow backend.
[2m[36m(pid=262081)[0m Using TensorFlow backend.
[2m[36m(pid=262071)[0m Using TensorFlow backend.
Traceback (most recent call last):
  File "tensorflow2_mnist_ray.py", line 94, in <module>
    executor.run(train, kwargs=dict(num_epochs=1))
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/ray/runner.py", line 426, in run
    lambda w: fn(*args, **kwargs)) for worker in self.workers])
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/worker.py", line 1516, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RecursionError): [36mray::BaseHorovodWorker.execute()[39m (pid=262081, ip=10.0.0.131)
  File "python/ray/_raylet.pyx", line 414, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 431, in ray._raylet.execute_task
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 317, in deserialize_objects
    self._deserialize_object(data, metadata, object_id))
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 257, in _deserialize_object
    return self._deserialize_msgpack_data(data, metadata)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 238, in _deserialize_msgpack_data
    python_objects = self._deserialize_pickle5_data(pickle5_data)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py", line 226, in _deserialize_pickle5_data
    obj = pickle.loads(in_band)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)

   .......
  

  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
RecursionError: maximum recursion depth exceeded while calling a Python object



--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `examples/tensorflow2_mnist_ray.py`
Content:
```
1 import tensorflow as tf
2 import horovod.tensorflow.keras as hvd
3 import horovod.keras as hvd
4 
5 import ray
6 from horovod.ray import RayExecutor
7 
8 
9 def train(num_epochs):
10     # Horovod: initialize Horovod.
11     hvd.init()
12 
13     # Horovod: pin GPU to be used to process local rank (one GPU per process)
14     gpus = tf.config.experimental.list_physical_devices('GPU')
15     for gpu in gpus:
16         tf.config.experimental.set_memory_growth(gpu, True)
17     if gpus:
18         tf.config.experimental.set_visible_devices(
19             gpus[hvd.local_rank()], 'GPU')
20 
21     (mnist_images, mnist_labels), _ = \
22         tf.keras.datasets.mnist.load_data(path='mnist-%d.npz' % hvd.rank())
23 
24     dataset = tf.data.Dataset.from_tensor_slices(
25         (tf.cast(mnist_images[..., tf.newaxis] / 255.0, tf.float32),
26          tf.cast(mnist_labels, tf.int64))
27     )
28     dataset = dataset.repeat().shuffle(10000).batch(128)
29 
30     mnist_model = tf.keras.Sequential([
31         tf.keras.layers.Conv2D(32, [3, 3], activation='relu'),
32         tf.keras.layers.Conv2D(64, [3, 3], activation='relu'),
33         tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
34         tf.keras.layers.Dropout(0.25),
35         tf.keras.layers.Flatten(),
36         tf.keras.layers.Dense(128, activation='relu'),
37         tf.keras.layers.Dropout(0.5),
38         tf.keras.layers.Dense(10, activation='softmax')
39     ])
40 
41     # Horovod: adjust learning rate based on number of GPUs.
42     scaled_lr = 0.001 * hvd.size()
43     opt = tf.optimizers.Adam(scaled_lr)
44 
45     # Horovod: add Horovod DistributedOptimizer.
46     opt = hvd.DistributedOptimizer(opt)
47 
48     # Horovod: Specify `experimental_run_tf_function=False` to ensure TensorFlow
49     # uses hvd.DistributedOptimizer() to compute gradients.
50     mnist_model.compile(loss=tf.losses.SparseCategoricalCrossentropy(),
51                         optimizer=opt,
52                         metrics=['accuracy'],
53                         experimental_run_tf_function=False)
54 
55     callbacks = [
56         # Horovod: broadcast initial variable states from rank 0 to all other processes.
57         # This is necessary to ensure consistent initialization of all workers when
58         # training is started with random weights or restored from a checkpoint.
59         hvd.callbacks.BroadcastGlobalVariablesCallback(0),
60 
61         # Horovod: average metrics among workers at the end of every epoch.
62         #
63         # Note: This callback must be in the list before the ReduceLROnPlateau,
64         # TensorBoard or other metrics-based callbacks.
65         hvd.callbacks.MetricAverageCallback(),
66 
67         # Horovod: using `lr = 1.0 * hvd.size()` from the very beginning leads to worse final
68         # accuracy. Scale the learning rate `lr = 1.0` ---> `lr = 1.0 * hvd.size()` during
69         # the first three epochs. See https://arxiv.org/abs/1706.02677 for details.
70         hvd.callbacks.LearningRateWarmupCallback(
71             warmup_epochs=3, initial_lr=scaled_lr, verbose=1),
72     ]
73 
74     # Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.
75     if hvd.rank() == 0:
76         callbacks.append(tf.keras.callbacks.ModelCheckpoint(
77             './checkpoint-{epoch}.h5'))
78 
79     # Horovod: write logs on worker 0.
80     verbose = 1 if hvd.rank() == 0 else 0
81 
82     # Train the model.
83     # Horovod: adjust number of steps based on number of GPUs.
84     mnist_model.fit(dataset, steps_per_epoch=500 // hvd.size(),
85                     callbacks=callbacks, epochs=num_epochs, verbose=verbose)
86 
87 
88 ray.init()
89 settings = RayExecutor.create_settings(timeout_s=30)
90 executor = RayExecutor(settings, num_hosts=1, num_slots=2, use_gpu=False)
91 executor.start()
92 executor.run(train, kwargs=dict(num_epochs=1))
93 executor.shutdown()
94
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/examples/tensorflow2_mnist_ray.py b/examples/tensorflow2_mnist_ray.py
--- a/examples/tensorflow2_mnist_ray.py
+++ b/examples/tensorflow2_mnist_ray.py
@@ -1,6 +1,5 @@
 import tensorflow as tf
 import horovod.tensorflow.keras as hvd
-import horovod.keras as hvd
 
 import ray
 from horovod.ray import RayExecutor

{"golden_diff": "diff --git a/examples/tensorflow2_mnist_ray.py b/examples/tensorflow2_mnist_ray.py\n--- a/examples/tensorflow2_mnist_ray.py\n+++ b/examples/tensorflow2_mnist_ray.py\n@@ -1,6 +1,5 @@\n import tensorflow as tf\n import horovod.tensorflow.keras as hvd\n-import horovod.keras as hvd\n \n import ray\n from horovod.ray import RayExecutor\n", "issue": "[Horovod on Ray]Failed on example of tensorflow2_mnist_ray.py\n**Environment:**\r\n1. Framework: (TensorFlow1.15.0)\r\n3. Horovod version:0.20.0\r\n\r\n**Your question:**\r\nPlease ask your question here.\r\nHi @tgaddair @richardliaw I tried the example of Horovod on ray on CPU nodes, but failed.\r\nThe ray cluster is ok.\r\n\r\nUsing TensorFlow backend.\r\n\u001b[2m\u001b[36m(pid=262081)\u001b[0m Using TensorFlow backend.\r\n\u001b[2m\u001b[36m(pid=262071)\u001b[0m Using TensorFlow backend.\r\nTraceback (most recent call last):\r\n  File \"tensorflow2_mnist_ray.py\", line 94, in <module>\r\n    executor.run(train, kwargs=dict(num_epochs=1))\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/horovod/ray/runner.py\", line 426, in run\r\n    lambda w: fn(*args, **kwargs)) for worker in self.workers])\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/worker.py\", line 1516, in get\r\n    raise value.as_instanceof_cause()\r\nray.exceptions.RayTaskError(RecursionError): \u001b[36mray::BaseHorovodWorker.execute()\u001b[39m (pid=262081, ip=10.0.0.131)\r\n  File \"python/ray/_raylet.pyx\", line 414, in ray._raylet.execute_task\r\n  File \"python/ray/_raylet.pyx\", line 431, in ray._raylet.execute_task\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py\", line 317, in deserialize_objects\r\n    self._deserialize_object(data, metadata, object_id))\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py\", line 257, in _deserialize_object\r\n    return self._deserialize_msgpack_data(data, metadata)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py\", line 238, in _deserialize_msgpack_data\r\n    python_objects = self._deserialize_pickle5_data(pickle5_data)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/ray/serialization.py\", line 226, in _deserialize_pickle5_data\r\n    obj = pickle.loads(in_band)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n\r\n   .......\r\n  \r\n\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 50, in __getattr__\r\n    module = self._load()\r\n  File \"/home/xianyang/sw/miniconda3/envs/test/lib/python3.7/site-packages/tensorflow/__init__.py\", line 44, in _load\r\n    module = _importlib.import_module(self.__name__)\r\nRecursionError: maximum recursion depth exceeded while calling a Python object\r\n\r\n\n", "before_files": [{"content": "import tensorflow as tf\nimport horovod.tensorflow.keras as hvd\nimport horovod.keras as hvd\n\nimport ray\nfrom horovod.ray import RayExecutor\n\n\ndef train(num_epochs):\n    # Horovod: initialize Horovod.\n    hvd.init()\n\n    # Horovod: pin GPU to be used to process local rank (one GPU per process)\n    gpus = tf.config.experimental.list_physical_devices('GPU')\n    for gpu in gpus:\n        tf.config.experimental.set_memory_growth(gpu, True)\n    if gpus:\n        tf.config.experimental.set_visible_devices(\n            gpus[hvd.local_rank()], 'GPU')\n\n    (mnist_images, mnist_labels), _ = \\\n        tf.keras.datasets.mnist.load_data(path='mnist-%d.npz' % hvd.rank())\n\n    dataset = tf.data.Dataset.from_tensor_slices(\n        (tf.cast(mnist_images[..., tf.newaxis] / 255.0, tf.float32),\n         tf.cast(mnist_labels, tf.int64))\n    )\n    dataset = dataset.repeat().shuffle(10000).batch(128)\n\n    mnist_model = tf.keras.Sequential([\n        tf.keras.layers.Conv2D(32, [3, 3], activation='relu'),\n        tf.keras.layers.Conv2D(64, [3, 3], activation='relu'),\n        tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n        tf.keras.layers.Dropout(0.25),\n        tf.keras.layers.Flatten(),\n        tf.keras.layers.Dense(128, activation='relu'),\n        tf.keras.layers.Dropout(0.5),\n        tf.keras.layers.Dense(10, activation='softmax')\n    ])\n\n    # Horovod: adjust learning rate based on number of GPUs.\n    scaled_lr = 0.001 * hvd.size()\n    opt = tf.optimizers.Adam(scaled_lr)\n\n    # Horovod: add Horovod DistributedOptimizer.\n    opt = hvd.DistributedOptimizer(opt)\n\n    # Horovod: Specify `experimental_run_tf_function=False` to ensure TensorFlow\n    # uses hvd.DistributedOptimizer() to compute gradients.\n    mnist_model.compile(loss=tf.losses.SparseCategoricalCrossentropy(),\n                        optimizer=opt,\n                        metrics=['accuracy'],\n                        experimental_run_tf_function=False)\n\n    callbacks = [\n        # Horovod: broadcast initial variable states from rank 0 to all other processes.\n        # This is necessary to ensure consistent initialization of all workers when\n        # training is started with random weights or restored from a checkpoint.\n        hvd.callbacks.BroadcastGlobalVariablesCallback(0),\n\n        # Horovod: average metrics among workers at the end of every epoch.\n        #\n        # Note: This callback must be in the list before the ReduceLROnPlateau,\n        # TensorBoard or other metrics-based callbacks.\n        hvd.callbacks.MetricAverageCallback(),\n\n        # Horovod: using `lr = 1.0 * hvd.size()` from the very beginning leads to worse final\n        # accuracy. Scale the learning rate `lr = 1.0` ---> `lr = 1.0 * hvd.size()` during\n        # the first three epochs. See https://arxiv.org/abs/1706.02677 for details.\n        hvd.callbacks.LearningRateWarmupCallback(\n            warmup_epochs=3, initial_lr=scaled_lr, verbose=1),\n    ]\n\n    # Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.\n    if hvd.rank() == 0:\n        callbacks.append(tf.keras.callbacks.ModelCheckpoint(\n            './checkpoint-{epoch}.h5'))\n\n    # Horovod: write logs on worker 0.\n    verbose = 1 if hvd.rank() == 0 else 0\n\n    # Train the model.\n    # Horovod: adjust number of steps based on number of GPUs.\n    mnist_model.fit(dataset, steps_per_epoch=500 // hvd.size(),\n                    callbacks=callbacks, epochs=num_epochs, verbose=verbose)\n\n\nray.init()\nsettings = RayExecutor.create_settings(timeout_s=30)\nexecutor = RayExecutor(settings, num_hosts=1, num_slots=2, use_gpu=False)\nexecutor.start()\nexecutor.run(train, kwargs=dict(num_epochs=1))\nexecutor.shutdown()\n", "path": "examples/tensorflow2_mnist_ray.py"}], "after_files": [{"content": "import tensorflow as tf\nimport horovod.tensorflow.keras as hvd\n\nimport ray\nfrom horovod.ray import RayExecutor\n\n\ndef train(num_epochs):\n    # Horovod: initialize Horovod.\n    hvd.init()\n\n    # Horovod: pin GPU to be used to process local rank (one GPU per process)\n    gpus = tf.config.experimental.list_physical_devices('GPU')\n    for gpu in gpus:\n        tf.config.experimental.set_memory_growth(gpu, True)\n    if gpus:\n        tf.config.experimental.set_visible_devices(\n            gpus[hvd.local_rank()], 'GPU')\n\n    (mnist_images, mnist_labels), _ = \\\n        tf.keras.datasets.mnist.load_data(path='mnist-%d.npz' % hvd.rank())\n\n    dataset = tf.data.Dataset.from_tensor_slices(\n        (tf.cast(mnist_images[..., tf.newaxis] / 255.0, tf.float32),\n         tf.cast(mnist_labels, tf.int64))\n    )\n    dataset = dataset.repeat().shuffle(10000).batch(128)\n\n    mnist_model = tf.keras.Sequential([\n        tf.keras.layers.Conv2D(32, [3, 3], activation='relu'),\n        tf.keras.layers.Conv2D(64, [3, 3], activation='relu'),\n        tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),\n        tf.keras.layers.Dropout(0.25),\n        tf.keras.layers.Flatten(),\n        tf.keras.layers.Dense(128, activation='relu'),\n        tf.keras.layers.Dropout(0.5),\n        tf.keras.layers.Dense(10, activation='softmax')\n    ])\n\n    # Horovod: adjust learning rate based on number of GPUs.\n    scaled_lr = 0.001 * hvd.size()\n    opt = tf.optimizers.Adam(scaled_lr)\n\n    # Horovod: add Horovod DistributedOptimizer.\n    opt = hvd.DistributedOptimizer(opt)\n\n    # Horovod: Specify `experimental_run_tf_function=False` to ensure TensorFlow\n    # uses hvd.DistributedOptimizer() to compute gradients.\n    mnist_model.compile(loss=tf.losses.SparseCategoricalCrossentropy(),\n                        optimizer=opt,\n                        metrics=['accuracy'],\n                        experimental_run_tf_function=False)\n\n    callbacks = [\n        # Horovod: broadcast initial variable states from rank 0 to all other processes.\n        # This is necessary to ensure consistent initialization of all workers when\n        # training is started with random weights or restored from a checkpoint.\n        hvd.callbacks.BroadcastGlobalVariablesCallback(0),\n\n        # Horovod: average metrics among workers at the end of every epoch.\n        #\n        # Note: This callback must be in the list before the ReduceLROnPlateau,\n        # TensorBoard or other metrics-based callbacks.\n        hvd.callbacks.MetricAverageCallback(),\n\n        # Horovod: using `lr = 1.0 * hvd.size()` from the very beginning leads to worse final\n        # accuracy. Scale the learning rate `lr = 1.0` ---> `lr = 1.0 * hvd.size()` during\n        # the first three epochs. See https://arxiv.org/abs/1706.02677 for details.\n        hvd.callbacks.LearningRateWarmupCallback(\n            warmup_epochs=3, initial_lr=scaled_lr, verbose=1),\n    ]\n\n    # Horovod: save checkpoints only on worker 0 to prevent other workers from corrupting them.\n    if hvd.rank() == 0:\n        callbacks.append(tf.keras.callbacks.ModelCheckpoint(\n            './checkpoint-{epoch}.h5'))\n\n    # Horovod: write logs on worker 0.\n    verbose = 1 if hvd.rank() == 0 else 0\n\n    # Train the model.\n    # Horovod: adjust number of steps based on number of GPUs.\n    mnist_model.fit(dataset, steps_per_epoch=500 // hvd.size(),\n                    callbacks=callbacks, epochs=num_epochs, verbose=verbose)\n\n\nray.init()\nsettings = RayExecutor.create_settings(timeout_s=30)\nexecutor = RayExecutor(settings, num_hosts=1, num_slots=2, use_gpu=False)\nexecutor.start()\nexecutor.run(train, kwargs=dict(num_epochs=1))\nexecutor.shutdown()\n", "path": "examples/tensorflow2_mnist_ray.py"}]}