We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
horovodrun convenience script does not account for 'OpenRTE' in the output of mpirun --version
**Environment:**
1. Framework: (TensorFlow, PyTorch)
2. Framework version: 1.14.0
3. Horovod version: 0.16.4
4. MPI version: 3.1.4/4.0.1
5. CUDA version: 10.1
6. NCCL version: 2.4.8
7. Python version: 3.6
8. OS and version: Ubuntu, Docker
9. GCC version:5.4.0

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
Yes, hasn't been specifically asked
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**
1. horovodrun outputs the following, when using with Open MPI 4.0.1.
```
horovodrun -np 1 -H localhost:1 python pytorch_mnist.py
Open MPI not found in output of mpirun --version.
Traceback (most recent call last):
  File "/opt/conda/bin/horovodrun", line 21, in <module>
    run.run()
  File "/opt/conda/lib/python3.6/site-packages/horovod/run/run.py", line 448, in run
    'horovodrun convenience script currently only supports '
Exception: horovodrun convenience script currently only supports Open MPI.

Choose one of:
1. Install Open MPI 4.0.0+ and re-install Horovod (use --no-cache-dir pip option).
2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).
root@3da487b92c3d:/horovod/examples# mpirun --version
mpirun.real (OpenRTE) 4.0.1

Report bugs to http://www.open-mpi.org/community/help/
```
2. When Open MPI is installed as follows:
```
RUN wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-$OPEN_MPI_VERSION.tar.gz \
 && gunzip -c openmpi-$OPEN_MPI_VERSION.tar.gz | tar xf - \
 && cd openmpi-$OPEN_MPI_VERSION \
 && ./configure --prefix=/home/.openmpi \
 && make all install \
 && cd .. \
 && rm openmpi-$OPEN_MPI_VERSION.tar.gz \
 && rm -rf openmpi-$OPEN_MPI_VERSION
```
3. The horovodrun check expects 'OpenMPI' to be present in the output of `mpirun --version`. [[link](https://github.com/horovod/horovod/blob/master/horovod/run/mpi_run.py)]. However, when installed as above, OpenMPI has the following in output:
```
root@3b5149353790:/horovod/examples# mpirun --version
mpirun.real (OpenRTE) 4.0.1

Report bugs to http://www.open-mpi.org/community/help/
```
4. Either openmpi was installed incorrectly (in which case, can horovod documentation clarify how to install it correctly?), or the horovodrun convenience script does not account for presence of 'OpenRTE' in the `mpirun --version`.

I'm unable to understand when is 'OpenRTE' visible in mpirun --version, and it isn't? I saw the option --enable-orterun-prefix-by-default, but I'm not using it to build open-mpi.

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `horovod/run/mpi_run.py`
Content:
```
1 # Copyright 2019 Uber Technologies, Inc. All Rights Reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 # ==============================================================================
15 
16 from __future__ import print_function
17 import six
18 import traceback
19 import sys
20 import os
21 from horovod.run.common.util import env as env_util, safe_shell_exec, secret, codec
22 
23 # Open MPI Flags
24 _OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']
25 # Spectrum MPI Flags
26 _SMPI_FLAGS = ['-gpu', '-disable_gdr']
27 # MPICH Flags
28 _MPICH_FLAGS = []
29 # Threshold for large cluster MPI issues:
30 _LARGE_CLUSTER_THRESHOLD = 64
31 
32 try:
33     from shlex import quote
34 except ImportError:
35     from pipes import quote
36 
37 
38 def _get_mpi_implementation_flags():
39     output = six.StringIO()
40     command = 'mpirun --version'
41     try:
42         exit_code = safe_shell_exec.execute(command, stdout=output,
43                                             stderr=output)
44         output_msg = output.getvalue()
45     except Exception:
46         print(traceback.format_exc(), file=sys.stderr)
47         return None
48     finally:
49         output.close()
50 
51     if exit_code == 0:
52         if 'Open MPI' in output_msg:
53             return list(_OMPI_FLAGS)
54         elif 'IBM Spectrum MPI' in output_msg:
55             return list(_SMPI_FLAGS)
56         elif 'MPICH' in output_msg:
57             return list(_MPICH_FLAGS)
58         print('Open MPI/Spectrum MPI/MPICH not found in output of mpirun --version.',
59               file=sys.stderr)
60         return None
61     else:
62         print("Was not able to run %s:\n%s" % (command, output_msg),
63               file=sys.stderr)
64         return None
65 
66 
67 def mpi_run(settings, common_intfs, env, command, stdout=None, stderr=None, run_func=safe_shell_exec.execute):
68     """
69     Runs mpi_run.
70 
71     Args:
72         settings: Settings for running MPI.
73                   Note: settings.num_proc and settings.hosts must not be None.
74         common_intfs: Interfaces to include by MPI.
75         env: Environment dictionary to use for running MPI.
76         command: Command and arguments to run as a list of string.
77         stdout: Stdout of the mpi process.
78                 Only used when settings.run_func_mode is True.
79         stderr: Stderr of the mpi process.
80                 Only used when settings.run_func_mode is True.
81         run_func: Run function to use. Must have arguments 'command' and 'env'.
82                   Only used when settings.run_func_mode is True.
83                   Defaults to safe_shell_exec.execute.
84     """
85     mpi_impl_flags = _get_mpi_implementation_flags()
86     if mpi_impl_flags is None:
87         raise Exception(
88             'horovodrun convenience script does not find an installed MPI.\n\n'
89             'Choose one of:\n'
90             '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '
91             '(use --no-cache-dir pip option).\n'
92             '2. Run distributed '
93             'training script using the standard way provided by your'
94             ' MPI distribution (usually mpirun, srun, or jsrun).\n'
95             '3. Use built-in gloo option (horovodrun --gloo ...).')
96 
97     ssh_port_arg = '-mca plm_rsh_args \"-p {ssh_port}\"'.format(
98             ssh_port=settings.ssh_port) if settings.ssh_port else ''
99 
100     # if user does not specify any hosts, mpirun by default uses local host.
101     # There is no need to specify localhost.
102     hosts_arg = '-H {hosts}'.format(hosts=settings.hosts)
103 
104     tcp_intf_arg = '-mca btl_tcp_if_include {common_intfs}'.format(
105         common_intfs=','.join(common_intfs)) if common_intfs else ''
106     nccl_socket_intf_arg = '-x NCCL_SOCKET_IFNAME={common_intfs}'.format(
107         common_intfs=','.join(common_intfs)) if common_intfs else ''
108 
109     # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues
110     if settings.num_hosts and settings.num_hosts >= _LARGE_CLUSTER_THRESHOLD:
111         mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')
112         mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(settings.num_proc))
113 
114     # Pass all the env variables to the mpirun command.
115     mpirun_command = (
116         'mpirun --allow-run-as-root --tag-output '
117         '-np {num_proc} {hosts_arg} '
118         '-bind-to none -map-by slot '
119         '{mpi_args} '
120         '{ssh_port_arg} '
121         '{tcp_intf_arg} '
122         '{nccl_socket_intf_arg} '
123         '{output_filename_arg} '
124         '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables
125         .format(num_proc=settings.num_proc,
126                 hosts_arg=hosts_arg,
127                 mpi_args=' '.join(mpi_impl_flags),
128                 tcp_intf_arg=tcp_intf_arg,
129                 nccl_socket_intf_arg=nccl_socket_intf_arg,
130                 ssh_port_arg=ssh_port_arg,
131                 output_filename_arg='--output-filename ' + settings.output_filename
132                                     if settings.output_filename else '',
133                 env=' '.join('-x %s' % key for key in sorted(env.keys())
134                              if env_util.is_exportable(key)),
135 
136                 extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',
137                 command=' '.join(quote(par) for par in command))
138     )
139 
140     if settings.verbose >= 2:
141         print(mpirun_command)
142 
143     # Execute the mpirun command.
144     if settings.run_func_mode:
145         exit_code = run_func(command=mpirun_command, env=env, stdout=stdout, stderr=stderr)
146         if exit_code != 0:
147             raise RuntimeError("mpirun failed with exit code {exit_code}".format(exit_code=exit_code))
148     else:
149         os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)
150 
151
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/horovod/run/mpi_run.py b/horovod/run/mpi_run.py
--- a/horovod/run/mpi_run.py
+++ b/horovod/run/mpi_run.py
@@ -49,7 +49,7 @@
         output.close()
 
     if exit_code == 0:
-        if 'Open MPI' in output_msg:
+        if 'Open MPI' in output_msg or 'OpenRTE' in output_msg:
             return list(_OMPI_FLAGS)
         elif 'IBM Spectrum MPI' in output_msg:
             return list(_SMPI_FLAGS)

{"golden_diff": "diff --git a/horovod/run/mpi_run.py b/horovod/run/mpi_run.py\n--- a/horovod/run/mpi_run.py\n+++ b/horovod/run/mpi_run.py\n@@ -49,7 +49,7 @@\n         output.close()\n \n     if exit_code == 0:\n-        if 'Open MPI' in output_msg:\n+        if 'Open MPI' in output_msg or 'OpenRTE' in output_msg:\n             return list(_OMPI_FLAGS)\n         elif 'IBM Spectrum MPI' in output_msg:\n             return list(_SMPI_FLAGS)\n", "issue": "horovodrun convenience script does not account for 'OpenRTE' in the output of mpirun --version\n**Environment:**\r\n1. Framework: (TensorFlow, PyTorch)\r\n2. Framework version: 1.14.0\r\n3. Horovod version: 0.16.4\r\n4. MPI version: 3.1.4/4.0.1\r\n5. CUDA version: 10.1\r\n6. NCCL version: 2.4.8\r\n7. Python version: 3.6\r\n8. OS and version: Ubuntu, Docker\r\n9. GCC version:5.4.0\r\n\r\n**Checklist:**\r\n1. Did you search issues to find if somebody asked this question before?\r\nYes, hasn't been specifically asked\r\n4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?\r\n\r\n**Bug report:**\r\n1. horovodrun outputs the following, when using with Open MPI 4.0.1.\r\n```\r\nhorovodrun -np 1 -H localhost:1 python pytorch_mnist.py\r\nOpen MPI not found in output of mpirun --version.\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/bin/horovodrun\", line 21, in <module>\r\n    run.run()\r\n  File \"/opt/conda/lib/python3.6/site-packages/horovod/run/run.py\", line 448, in run\r\n    'horovodrun convenience script currently only supports '\r\nException: horovodrun convenience script currently only supports Open MPI.\r\n\r\nChoose one of:\r\n1. Install Open MPI 4.0.0+ and re-install Horovod (use --no-cache-dir pip option).\r\n2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).\r\nroot@3da487b92c3d:/horovod/examples# mpirun --version\r\nmpirun.real (OpenRTE) 4.0.1\r\n\r\nReport bugs to http://www.open-mpi.org/community/help/\r\n```\r\n2. When Open MPI is installed as follows:\r\n```\r\nRUN wget https://www.open-mpi.org/software/ompi/v4.0/downloads/openmpi-$OPEN_MPI_VERSION.tar.gz \\\r\n && gunzip -c openmpi-$OPEN_MPI_VERSION.tar.gz | tar xf - \\\r\n && cd openmpi-$OPEN_MPI_VERSION \\\r\n && ./configure --prefix=/home/.openmpi \\\r\n && make all install \\\r\n && cd .. \\\r\n && rm openmpi-$OPEN_MPI_VERSION.tar.gz \\\r\n && rm -rf openmpi-$OPEN_MPI_VERSION\r\n```\r\n3. The horovodrun check expects 'OpenMPI' to be present in the output of `mpirun --version`. [[link](https://github.com/horovod/horovod/blob/master/horovod/run/mpi_run.py)]. However, when installed as above, OpenMPI has the following in output:\r\n```\r\nroot@3b5149353790:/horovod/examples# mpirun --version\r\nmpirun.real (OpenRTE) 4.0.1\r\n\r\nReport bugs to http://www.open-mpi.org/community/help/\r\n```\r\n4. Either openmpi was installed incorrectly (in which case, can horovod documentation clarify how to install it correctly?), or the horovodrun convenience script does not account for presence of 'OpenRTE' in the `mpirun --version`.\r\n\r\nI'm unable to understand when is 'OpenRTE' visible in mpirun --version, and it isn't? I saw the option --enable-orterun-prefix-by-default, but I'm not using it to build open-mpi.\n", "before_files": [{"content": "# Copyright 2019 Uber Technologies, Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import print_function\nimport six\nimport traceback\nimport sys\nimport os\nfrom horovod.run.common.util import env as env_util, safe_shell_exec, secret, codec\n\n# Open MPI Flags\n_OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']\n# Spectrum MPI Flags\n_SMPI_FLAGS = ['-gpu', '-disable_gdr']\n# MPICH Flags\n_MPICH_FLAGS = []\n# Threshold for large cluster MPI issues:\n_LARGE_CLUSTER_THRESHOLD = 64\n\ntry:\n    from shlex import quote\nexcept ImportError:\n    from pipes import quote\n\n\ndef _get_mpi_implementation_flags():\n    output = six.StringIO()\n    command = 'mpirun --version'\n    try:\n        exit_code = safe_shell_exec.execute(command, stdout=output,\n                                            stderr=output)\n        output_msg = output.getvalue()\n    except Exception:\n        print(traceback.format_exc(), file=sys.stderr)\n        return None\n    finally:\n        output.close()\n\n    if exit_code == 0:\n        if 'Open MPI' in output_msg:\n            return list(_OMPI_FLAGS)\n        elif 'IBM Spectrum MPI' in output_msg:\n            return list(_SMPI_FLAGS)\n        elif 'MPICH' in output_msg:\n            return list(_MPICH_FLAGS)\n        print('Open MPI/Spectrum MPI/MPICH not found in output of mpirun --version.',\n              file=sys.stderr)\n        return None\n    else:\n        print(\"Was not able to run %s:\\n%s\" % (command, output_msg),\n              file=sys.stderr)\n        return None\n\n\ndef mpi_run(settings, common_intfs, env, command, stdout=None, stderr=None, run_func=safe_shell_exec.execute):\n    \"\"\"\n    Runs mpi_run.\n\n    Args:\n        settings: Settings for running MPI.\n                  Note: settings.num_proc and settings.hosts must not be None.\n        common_intfs: Interfaces to include by MPI.\n        env: Environment dictionary to use for running MPI.\n        command: Command and arguments to run as a list of string.\n        stdout: Stdout of the mpi process.\n                Only used when settings.run_func_mode is True.\n        stderr: Stderr of the mpi process.\n                Only used when settings.run_func_mode is True.\n        run_func: Run function to use. Must have arguments 'command' and 'env'.\n                  Only used when settings.run_func_mode is True.\n                  Defaults to safe_shell_exec.execute.\n    \"\"\"\n    mpi_impl_flags = _get_mpi_implementation_flags()\n    if mpi_impl_flags is None:\n        raise Exception(\n            'horovodrun convenience script does not find an installed MPI.\\n\\n'\n            'Choose one of:\\n'\n            '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '\n            '(use --no-cache-dir pip option).\\n'\n            '2. Run distributed '\n            'training script using the standard way provided by your'\n            ' MPI distribution (usually mpirun, srun, or jsrun).\\n'\n            '3. Use built-in gloo option (horovodrun --gloo ...).')\n\n    ssh_port_arg = '-mca plm_rsh_args \\\"-p {ssh_port}\\\"'.format(\n            ssh_port=settings.ssh_port) if settings.ssh_port else ''\n\n    # if user does not specify any hosts, mpirun by default uses local host.\n    # There is no need to specify localhost.\n    hosts_arg = '-H {hosts}'.format(hosts=settings.hosts)\n\n    tcp_intf_arg = '-mca btl_tcp_if_include {common_intfs}'.format(\n        common_intfs=','.join(common_intfs)) if common_intfs else ''\n    nccl_socket_intf_arg = '-x NCCL_SOCKET_IFNAME={common_intfs}'.format(\n        common_intfs=','.join(common_intfs)) if common_intfs else ''\n\n    # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues\n    if settings.num_hosts and settings.num_hosts >= _LARGE_CLUSTER_THRESHOLD:\n        mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')\n        mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(settings.num_proc))\n\n    # Pass all the env variables to the mpirun command.\n    mpirun_command = (\n        'mpirun --allow-run-as-root --tag-output '\n        '-np {num_proc} {hosts_arg} '\n        '-bind-to none -map-by slot '\n        '{mpi_args} '\n        '{ssh_port_arg} '\n        '{tcp_intf_arg} '\n        '{nccl_socket_intf_arg} '\n        '{output_filename_arg} '\n        '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables\n        .format(num_proc=settings.num_proc,\n                hosts_arg=hosts_arg,\n                mpi_args=' '.join(mpi_impl_flags),\n                tcp_intf_arg=tcp_intf_arg,\n                nccl_socket_intf_arg=nccl_socket_intf_arg,\n                ssh_port_arg=ssh_port_arg,\n                output_filename_arg='--output-filename ' + settings.output_filename\n                                    if settings.output_filename else '',\n                env=' '.join('-x %s' % key for key in sorted(env.keys())\n                             if env_util.is_exportable(key)),\n\n                extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',\n                command=' '.join(quote(par) for par in command))\n    )\n\n    if settings.verbose >= 2:\n        print(mpirun_command)\n\n    # Execute the mpirun command.\n    if settings.run_func_mode:\n        exit_code = run_func(command=mpirun_command, env=env, stdout=stdout, stderr=stderr)\n        if exit_code != 0:\n            raise RuntimeError(\"mpirun failed with exit code {exit_code}\".format(exit_code=exit_code))\n    else:\n        os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)\n\n", "path": "horovod/run/mpi_run.py"}], "after_files": [{"content": "# Copyright 2019 Uber Technologies, Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import print_function\nimport six\nimport traceback\nimport sys\nimport os\nfrom horovod.run.common.util import env as env_util, safe_shell_exec, secret, codec\n\n# Open MPI Flags\n_OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']\n# Spectrum MPI Flags\n_SMPI_FLAGS = ['-gpu', '-disable_gdr']\n# MPICH Flags\n_MPICH_FLAGS = []\n# Threshold for large cluster MPI issues:\n_LARGE_CLUSTER_THRESHOLD = 64\n\ntry:\n    from shlex import quote\nexcept ImportError:\n    from pipes import quote\n\n\ndef _get_mpi_implementation_flags():\n    output = six.StringIO()\n    command = 'mpirun --version'\n    try:\n        exit_code = safe_shell_exec.execute(command, stdout=output,\n                                            stderr=output)\n        output_msg = output.getvalue()\n    except Exception:\n        print(traceback.format_exc(), file=sys.stderr)\n        return None\n    finally:\n        output.close()\n\n    if exit_code == 0:\n        if 'Open MPI' in output_msg or 'OpenRTE' in output_msg:\n            return list(_OMPI_FLAGS)\n        elif 'IBM Spectrum MPI' in output_msg:\n            return list(_SMPI_FLAGS)\n        elif 'MPICH' in output_msg:\n            return list(_MPICH_FLAGS)\n        print('Open MPI/Spectrum MPI/MPICH not found in output of mpirun --version.',\n              file=sys.stderr)\n        return None\n    else:\n        print(\"Was not able to run %s:\\n%s\" % (command, output_msg),\n              file=sys.stderr)\n        return None\n\n\ndef mpi_run(settings, common_intfs, env, command, stdout=None, stderr=None, run_func=safe_shell_exec.execute):\n    \"\"\"\n    Runs mpi_run.\n\n    Args:\n        settings: Settings for running MPI.\n                  Note: settings.num_proc and settings.hosts must not be None.\n        common_intfs: Interfaces to include by MPI.\n        env: Environment dictionary to use for running MPI.\n        command: Command and arguments to run as a list of string.\n        stdout: Stdout of the mpi process.\n                Only used when settings.run_func_mode is True.\n        stderr: Stderr of the mpi process.\n                Only used when settings.run_func_mode is True.\n        run_func: Run function to use. Must have arguments 'command' and 'env'.\n                  Only used when settings.run_func_mode is True.\n                  Defaults to safe_shell_exec.execute.\n    \"\"\"\n    mpi_impl_flags = _get_mpi_implementation_flags()\n    if mpi_impl_flags is None:\n        raise Exception(\n            'horovodrun convenience script does not find an installed MPI.\\n\\n'\n            'Choose one of:\\n'\n            '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '\n            '(use --no-cache-dir pip option).\\n'\n            '2. Run distributed '\n            'training script using the standard way provided by your'\n            ' MPI distribution (usually mpirun, srun, or jsrun).\\n'\n            '3. Use built-in gloo option (horovodrun --gloo ...).')\n\n    ssh_port_arg = '-mca plm_rsh_args \\\"-p {ssh_port}\\\"'.format(\n            ssh_port=settings.ssh_port) if settings.ssh_port else ''\n\n    # if user does not specify any hosts, mpirun by default uses local host.\n    # There is no need to specify localhost.\n    hosts_arg = '-H {hosts}'.format(hosts=settings.hosts)\n\n    tcp_intf_arg = '-mca btl_tcp_if_include {common_intfs}'.format(\n        common_intfs=','.join(common_intfs)) if common_intfs else ''\n    nccl_socket_intf_arg = '-x NCCL_SOCKET_IFNAME={common_intfs}'.format(\n        common_intfs=','.join(common_intfs)) if common_intfs else ''\n\n    # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues\n    if settings.num_hosts and settings.num_hosts >= _LARGE_CLUSTER_THRESHOLD:\n        mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')\n        mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(settings.num_proc))\n\n    # Pass all the env variables to the mpirun command.\n    mpirun_command = (\n        'mpirun --allow-run-as-root --tag-output '\n        '-np {num_proc} {hosts_arg} '\n        '-bind-to none -map-by slot '\n        '{mpi_args} '\n        '{ssh_port_arg} '\n        '{tcp_intf_arg} '\n        '{nccl_socket_intf_arg} '\n        '{output_filename_arg} '\n        '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables\n        .format(num_proc=settings.num_proc,\n                hosts_arg=hosts_arg,\n                mpi_args=' '.join(mpi_impl_flags),\n                tcp_intf_arg=tcp_intf_arg,\n                nccl_socket_intf_arg=nccl_socket_intf_arg,\n                ssh_port_arg=ssh_port_arg,\n                output_filename_arg='--output-filename ' + settings.output_filename\n                                    if settings.output_filename else '',\n                env=' '.join('-x %s' % key for key in sorted(env.keys())\n                             if env_util.is_exportable(key)),\n\n                extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',\n                command=' '.join(quote(par) for par in command))\n    )\n\n    if settings.verbose >= 2:\n        print(mpirun_command)\n\n    # Execute the mpirun command.\n    if settings.run_func_mode:\n        exit_code = run_func(command=mpirun_command, env=env, stdout=stdout, stderr=stderr)\n        if exit_code != 0:\n            raise RuntimeError(\"mpirun failed with exit code {exit_code}\".format(exit_code=exit_code))\n    else:\n        os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)\n\n", "path": "horovod/run/mpi_run.py"}]}