We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Support MPICH when using horovodrun
I'm using MPICH instead of OpenMPI. 
I'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error

```python
[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>
[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root
[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters
```

Horovod version: 0.18.2
MPICH version: 3.2

From looking at it, it looks like certain args are not available in MPICH.
Is horovodrun compatible with MPICH?
Support MPICH when using horovodrun
I'm using MPICH instead of OpenMPI. 
I'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error

```python
[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>
[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root
[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error
[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array
[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments
[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters
```

Horovod version: 0.18.2
MPICH version: 3.2

From looking at it, it looks like certain args are not available in MPICH.
Is horovodrun compatible with MPICH?

--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `horovod/runner/mpi_run.py`
Content:
```
1 # Copyright 2019 Uber Technologies, Inc. All Rights Reserved.
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
4 # you may not use this file except in compliance with the License.
5 # You may obtain a copy of the License at
6 #
7 #     http://www.apache.org/licenses/LICENSE-2.0
8 #
9 # Unless required by applicable law or agreed to in writing, software
10 # distributed under the License is distributed on an "AS IS" BASIS,
11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
14 # ==============================================================================
15 
16 import copy
17 import os
18 import sys
19 
20 from shlex import quote
21 
22 from horovod.runner.common.util import env as env_util, hosts, safe_shell_exec, tiny_shell_exec
23 
24 # MPI implementations
25 _OMPI_IMPL = 'OpenMPI'
26 _SMPI_IMPL = 'SpectrumMPI'
27 _MPICH_IMPL = 'MPICH'
28 _IMPI_IMPL = "IntelMPI"
29 _UNKNOWN_IMPL = 'Unknown'
30 _MISSING_IMPL = 'Missing'
31 
32 # Open MPI Flags
33 _OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']
34 # Spectrum MPI Flags
35 _SMPI_FLAGS = []
36 _SMPI_FLAGS_TCP = ['-tcp']
37 # MPICH Flags
38 _MPICH_FLAGS = []
39 # Intel MPI Flags
40 _IMPI_FLAGS = []
41 
42 # Threshold for large cluster MPI issues:
43 _LARGE_CLUSTER_THRESHOLD = 64
44 # No process binding args
45 _NO_BINDING_ARGS = ['-bind-to none', '-map-by slot']
46 # Process socket binding args
47 _SOCKET_BINDING_ARGS = ['-bind-to socket', '-map-by socket', '-rank-by core']
48 
49 # MPI not found error message
50 _MPI_NOT_FOUND_ERROR_MSG= ('horovod does not find an installed MPI.\n\n'
51                            'Choose one of:\n'
52                            '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '
53                            '(use --no-cache-dir pip option).\n'
54                            '2. Run distributed '
55                            'training script using the standard way provided by your'
56                            ' MPI distribution (usually mpirun, srun, or jsrun).\n'
57                            '3. Use built-in gloo option (horovodrun --gloo ...).')
58 
59 
60 def mpi_available(env=None):
61     return _get_mpi_implementation(env) not in {_UNKNOWN_IMPL, _MISSING_IMPL}
62 
63 
64 def is_open_mpi(env=None):
65     return _get_mpi_implementation(env) == _OMPI_IMPL
66 
67 
68 def is_spectrum_mpi(env=None):
69     return _get_mpi_implementation(env) == _SMPI_IMPL
70 
71 
72 def is_mpich(env=None):
73     return _get_mpi_implementation(env) == _MPICH_IMPL
74 
75 
76 def is_intel_mpi(env=None):
77     return _get_mpi_implementation(env) == _IMPI_IMPL
78 
79 
80 def _get_mpi_implementation(env=None):
81     """
82     Detects the available MPI implementation by invoking `mpirun --version`.
83     This command is executed by the given execute function, which takes the
84     command as the only argument and returns (output, exit code). Output
85     represents the stdout and stderr as a string.
86 
87     Returns one of:
88     - _OMPI_IMPL, _SMPI_IMPL, _MPICH_IMPL or _IMPI_IMPL for known implementations
89     - _UNKNOWN_IMPL for any unknown implementation
90     - _MISSING_IMPL if `mpirun --version` could not be executed.
91 
92     :param env: environment variable to use to run mpirun
93     :return: string representing identified implementation
94     """
95     command = 'mpirun --version'
96     res = tiny_shell_exec.execute(command, env)
97     if res is None:
98         return _MISSING_IMPL
99     (output, exit_code) = res
100 
101     if exit_code == 0:
102         if 'Open MPI' in output or 'OpenRTE' in output:
103             return _OMPI_IMPL
104         elif 'IBM Spectrum MPI' in output:
105             return _SMPI_IMPL
106         elif 'MPICH' in output:
107             return _MPICH_IMPL
108         elif 'Intel(R) MPI' in output:
109             return _IMPI_IMPL
110 
111         print('Unknown MPI implementation given in output of mpirun --version:', file=sys.stderr)
112         print(output, file=sys.stderr)
113         return _UNKNOWN_IMPL
114     else:
115         print('Was unable to run {command}:'.format(command=command), file=sys.stderr)
116         print(output, file=sys.stderr)
117         return _MISSING_IMPL
118 
119 
120 def _get_mpi_implementation_flags(tcp_flag, env=None):
121     if is_open_mpi(env):
122         return list(_OMPI_FLAGS), list(_NO_BINDING_ARGS), _OMPI_IMPL
123     elif is_spectrum_mpi(env):
124         return (list(_SMPI_FLAGS_TCP) if tcp_flag else list(_SMPI_FLAGS)), list(_SOCKET_BINDING_ARGS), _SMPI_IMPL
125     elif is_mpich(env):
126         return list(_MPICH_FLAGS), list(_NO_BINDING_ARGS), _MPICH_IMPL
127     elif is_intel_mpi(env):
128         return list(_IMPI_FLAGS), [], _IMPI_IMPL
129     else:
130         return None, None, None
131 
132 
133 def mpi_run(settings, nics, env, command, stdout=None, stderr=None):
134     """
135     Runs mpi_run.
136 
137     Args:
138         settings: Settings for running MPI.
139                   Note: settings.num_proc and settings.hosts must not be None.
140         nics: Interfaces to include by MPI.
141         env: Environment dictionary to use for running command.
142         command: Command and arguments to run as a list of string.
143         stdout: Stdout of the mpi process.
144                 Only used when settings.run_func_mode is True.
145         stderr: Stderr of the mpi process.
146                 Only used when settings.run_func_mode is True.
147     """
148     if env is not None and not isinstance(env, dict):
149         raise Exception('env argument must be a dict, not {type}: {env}'
150                         .format(type=type(env), env=env))
151 
152     mpi_impl_flags, impl_binding_args, mpi = _get_mpi_implementation_flags(settings.tcp_flag, env=env)
153     if mpi_impl_flags is None:
154         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
155 
156     impi = _IMPI_IMPL == mpi
157 
158     ssh_args = []
159     if settings.ssh_port:
160         ssh_args += [f'-p {settings.ssh_port}']
161     if settings.ssh_identity_file:
162         ssh_args += [f'-i {settings.ssh_identity_file}']
163 
164     mpi_ssh_args = ''
165     if ssh_args:
166         joined_ssh_args = ' '.join(ssh_args)
167         mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \"{joined_ssh_args}\"' if impi else f'-mca plm_rsh_args \"{joined_ssh_args}\"'
168 
169     tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(
170         nics=','.join(nics)) if nics and not impi else ''
171     nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(
172         opt='genv' if impi else 'x',
173         nics=','.join(nics)) if nics else ''
174 
175     # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues
176     host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)
177     if not impi and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:
178         mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')
179         mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))
180 
181     # if user does not specify any hosts, mpirun by default uses local host.
182     # There is no need to specify localhost.
183     hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi else 'H',
184                 hosts=','.join(host_names) if host_names and impi else settings.hosts)
185 
186     ppn_arg = ' '
187     if host_to_slots and impi:
188         ppn = host_to_slots[host_names[0]]
189         for h_name in host_names[1:]:
190             if ppn != host_to_slots[h_name]:
191                 raise Exception('''Different slots in -hosts parameter are not supported in Intel(R) MPI.
192                                  Use -machinefile <machine_file> for this purpose.''')
193         ppn_arg = ' -ppn {} '.format(ppn)
194 
195     if settings.prefix_output_with_timestamp and not impi:
196         mpi_impl_flags.append('--timestamp-output')
197 
198     binding_args = settings.binding_args if settings.binding_args and not impi else ' '.join(impl_binding_args)
199 
200     basic_args = '-l' if impi else '--allow-run-as-root --tag-output'
201 
202     output = []
203     if settings.output_filename:
204         output.append('-outfile-pattern' if impi else '--output-filename')
205         output.append(settings.output_filename)
206 
207     env_list = '' if impi else ' '.join(
208                     '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))
209 
210     # Pass all the env variables to the mpirun command.
211     mpirun_command = (
212         'mpirun {basic_args} '
213         '-np {num_proc}{ppn_arg}{hosts_arg} '
214         '{binding_args} '
215         '{mpi_args} '
216         '{mpi_ssh_args} '
217         '{tcp_intf_arg} '
218         '{nccl_socket_intf_arg} '
219         '{output_filename_arg} '
220         '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables
221         .format(basic_args=basic_args,
222                 num_proc=settings.num_proc,
223                 ppn_arg=ppn_arg,
224                 hosts_arg=hosts_arg,
225                 binding_args=binding_args,
226                 mpi_args=' '.join(mpi_impl_flags),
227                 tcp_intf_arg=tcp_intf_arg,
228                 nccl_socket_intf_arg=nccl_socket_intf_arg,
229                 mpi_ssh_args=mpi_ssh_args,
230                 output_filename_arg=' '.join(output),
231                 env=env_list,
232                 extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',
233                 command=' '.join(quote(par) for par in command))
234     )
235 
236     if settings.verbose >= 2:
237         print(mpirun_command)
238 
239     # we need the driver's PATH and PYTHONPATH in env to run mpirun,
240     # env for mpirun is different to env encoded in mpirun_command
241     for var in ['PATH', 'PYTHONPATH']:
242         if var not in env and var in os.environ:
243             # copy env so we do not leak env modifications
244             env = copy.copy(env)
245             # copy var over from os.environ
246             env[var] = os.environ[var]
247 
248     # Execute the mpirun command.
249     if settings.run_func_mode:
250         exit_code = safe_shell_exec.execute(mpirun_command, env=env, stdout=stdout, stderr=stderr)
251         if exit_code != 0:
252             raise RuntimeError("mpirun failed with exit code {exit_code}".format(exit_code=exit_code))
253     else:
254         os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)
255
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/horovod/runner/mpi_run.py b/horovod/runner/mpi_run.py
--- a/horovod/runner/mpi_run.py
+++ b/horovod/runner/mpi_run.py
@@ -153,7 +153,7 @@
     if mpi_impl_flags is None:
         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
 
-    impi = _IMPI_IMPL == mpi
+    impi_or_mpich = mpi in (_IMPI_IMPL, _MPICH_IMPL)
 
     ssh_args = []
     if settings.ssh_port:
@@ -164,27 +164,27 @@
     mpi_ssh_args = ''
     if ssh_args:
         joined_ssh_args = ' '.join(ssh_args)
-        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \"{joined_ssh_args}\"' if impi else f'-mca plm_rsh_args \"{joined_ssh_args}\"'
+        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \"{joined_ssh_args}\"' if impi_or_mpich else f'-mca plm_rsh_args \"{joined_ssh_args}\"'
 
     tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(
-        nics=','.join(nics)) if nics and not impi else ''
+        nics=','.join(nics)) if nics and not impi_or_mpich else ''
     nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(
-        opt='genv' if impi else 'x',
+        opt='genv' if impi_or_mpich else 'x',
         nics=','.join(nics)) if nics else ''
 
     # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues
     host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)
-    if not impi and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:
+    if not impi_or_mpich and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:
         mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')
         mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))
 
     # if user does not specify any hosts, mpirun by default uses local host.
     # There is no need to specify localhost.
-    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi else 'H',
-                hosts=','.join(host_names) if host_names and impi else settings.hosts)
+    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi_or_mpich else 'H',
+                hosts=','.join(host_names) if host_names and impi_or_mpich else settings.hosts)
 
     ppn_arg = ' '
-    if host_to_slots and impi:
+    if host_to_slots and impi_or_mpich:
         ppn = host_to_slots[host_names[0]]
         for h_name in host_names[1:]:
             if ppn != host_to_slots[h_name]:
@@ -192,19 +192,19 @@
                                  Use -machinefile <machine_file> for this purpose.''')
         ppn_arg = ' -ppn {} '.format(ppn)
 
-    if settings.prefix_output_with_timestamp and not impi:
+    if settings.prefix_output_with_timestamp and not impi_or_mpich:
         mpi_impl_flags.append('--timestamp-output')
 
-    binding_args = settings.binding_args if settings.binding_args and not impi else ' '.join(impl_binding_args)
+    binding_args = settings.binding_args if settings.binding_args and not impi_or_mpich else ' '.join(impl_binding_args)
 
-    basic_args = '-l' if impi else '--allow-run-as-root --tag-output'
+    basic_args = '-l' if impi_or_mpich else '--allow-run-as-root --tag-output'
 
     output = []
     if settings.output_filename:
-        output.append('-outfile-pattern' if impi else '--output-filename')
+        output.append('-outfile-pattern' if impi_or_mpich else '--output-filename')
         output.append(settings.output_filename)
 
-    env_list = '' if impi else ' '.join(
+    env_list = '' if impi_or_mpich else ' '.join(
                     '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))
 
     # Pass all the env variables to the mpirun command.

{"golden_diff": "diff --git a/horovod/runner/mpi_run.py b/horovod/runner/mpi_run.py\n--- a/horovod/runner/mpi_run.py\n+++ b/horovod/runner/mpi_run.py\n@@ -153,7 +153,7 @@\n     if mpi_impl_flags is None:\n         raise Exception(_MPI_NOT_FOUND_ERROR_MSG)\n \n-    impi = _IMPI_IMPL == mpi\n+    impi_or_mpich = mpi in (_IMPI_IMPL, _MPICH_IMPL)\n \n     ssh_args = []\n     if settings.ssh_port:\n@@ -164,27 +164,27 @@\n     mpi_ssh_args = ''\n     if ssh_args:\n         joined_ssh_args = ' '.join(ssh_args)\n-        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \\\"{joined_ssh_args}\\\"' if impi else f'-mca plm_rsh_args \\\"{joined_ssh_args}\\\"'\n+        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \\\"{joined_ssh_args}\\\"' if impi_or_mpich else f'-mca plm_rsh_args \\\"{joined_ssh_args}\\\"'\n \n     tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(\n-        nics=','.join(nics)) if nics and not impi else ''\n+        nics=','.join(nics)) if nics and not impi_or_mpich else ''\n     nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(\n-        opt='genv' if impi else 'x',\n+        opt='genv' if impi_or_mpich else 'x',\n         nics=','.join(nics)) if nics else ''\n \n     # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues\n     host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)\n-    if not impi and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:\n+    if not impi_or_mpich and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:\n         mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')\n         mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))\n \n     # if user does not specify any hosts, mpirun by default uses local host.\n     # There is no need to specify localhost.\n-    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi else 'H',\n-                hosts=','.join(host_names) if host_names and impi else settings.hosts)\n+    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi_or_mpich else 'H',\n+                hosts=','.join(host_names) if host_names and impi_or_mpich else settings.hosts)\n \n     ppn_arg = ' '\n-    if host_to_slots and impi:\n+    if host_to_slots and impi_or_mpich:\n         ppn = host_to_slots[host_names[0]]\n         for h_name in host_names[1:]:\n             if ppn != host_to_slots[h_name]:\n@@ -192,19 +192,19 @@\n                                  Use -machinefile <machine_file> for this purpose.''')\n         ppn_arg = ' -ppn {} '.format(ppn)\n \n-    if settings.prefix_output_with_timestamp and not impi:\n+    if settings.prefix_output_with_timestamp and not impi_or_mpich:\n         mpi_impl_flags.append('--timestamp-output')\n \n-    binding_args = settings.binding_args if settings.binding_args and not impi else ' '.join(impl_binding_args)\n+    binding_args = settings.binding_args if settings.binding_args and not impi_or_mpich else ' '.join(impl_binding_args)\n \n-    basic_args = '-l' if impi else '--allow-run-as-root --tag-output'\n+    basic_args = '-l' if impi_or_mpich else '--allow-run-as-root --tag-output'\n \n     output = []\n     if settings.output_filename:\n-        output.append('-outfile-pattern' if impi else '--output-filename')\n+        output.append('-outfile-pattern' if impi_or_mpich else '--output-filename')\n         output.append(settings.output_filename)\n \n-    env_list = '' if impi else ' '.join(\n+    env_list = '' if impi_or_mpich else ' '.join(\n                     '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))\n \n     # Pass all the env variables to the mpirun command.\n", "issue": "Support MPICH when using horovodrun\nI'm using MPICH instead of OpenMPI. \r\nI'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error\r\n\r\n```python\r\n[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>\r\n[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root\r\n[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error\r\n[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array\r\n[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments\r\n[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters\r\n```\r\n\r\nHorovod version: 0.18.2\r\nMPICH version: 3.2\r\n\r\nFrom looking at it, it looks like certain args are not available in MPICH.\r\nIs horovodrun compatible with MPICH?\nSupport MPICH when using horovodrun\nI'm using MPICH instead of OpenMPI. \r\nI'm able to train models using mpiexec and mpirun on my cluster but when I try to use horovodrun, it fails with the following error\r\n\r\n```python\r\n[ec2-user@master ~]$python3 ./horovodrun -np 8 python3 <train command>\r\n[mpiexec@master.amazon.com] match_arg (utils/args/args.c:159): unrecognized argument allow-run-as-root\r\n[mpiexec@master.amazon.com] HYDU_parse_array (utils/args/args.c:174): argument matching returned error\r\n[mpiexec@master.amazon.com] parse_args (ui/mpich/utils.c:1596): error parsing input array\r\n[mpiexec@master.amazon.com] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to parse user arguments\r\n[mpiexec@master.amazon.com] main (ui/mpich/mpiexec.c:153): error parsing parameters\r\n```\r\n\r\nHorovod version: 0.18.2\r\nMPICH version: 3.2\r\n\r\nFrom looking at it, it looks like certain args are not available in MPICH.\r\nIs horovodrun compatible with MPICH?\n", "before_files": [{"content": "# Copyright 2019 Uber Technologies, Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport copy\nimport os\nimport sys\n\nfrom shlex import quote\n\nfrom horovod.runner.common.util import env as env_util, hosts, safe_shell_exec, tiny_shell_exec\n\n# MPI implementations\n_OMPI_IMPL = 'OpenMPI'\n_SMPI_IMPL = 'SpectrumMPI'\n_MPICH_IMPL = 'MPICH'\n_IMPI_IMPL = \"IntelMPI\"\n_UNKNOWN_IMPL = 'Unknown'\n_MISSING_IMPL = 'Missing'\n\n# Open MPI Flags\n_OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']\n# Spectrum MPI Flags\n_SMPI_FLAGS = []\n_SMPI_FLAGS_TCP = ['-tcp']\n# MPICH Flags\n_MPICH_FLAGS = []\n# Intel MPI Flags\n_IMPI_FLAGS = []\n\n# Threshold for large cluster MPI issues:\n_LARGE_CLUSTER_THRESHOLD = 64\n# No process binding args\n_NO_BINDING_ARGS = ['-bind-to none', '-map-by slot']\n# Process socket binding args\n_SOCKET_BINDING_ARGS = ['-bind-to socket', '-map-by socket', '-rank-by core']\n\n# MPI not found error message\n_MPI_NOT_FOUND_ERROR_MSG= ('horovod does not find an installed MPI.\\n\\n'\n                           'Choose one of:\\n'\n                           '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '\n                           '(use --no-cache-dir pip option).\\n'\n                           '2. Run distributed '\n                           'training script using the standard way provided by your'\n                           ' MPI distribution (usually mpirun, srun, or jsrun).\\n'\n                           '3. Use built-in gloo option (horovodrun --gloo ...).')\n\n\ndef mpi_available(env=None):\n    return _get_mpi_implementation(env) not in {_UNKNOWN_IMPL, _MISSING_IMPL}\n\n\ndef is_open_mpi(env=None):\n    return _get_mpi_implementation(env) == _OMPI_IMPL\n\n\ndef is_spectrum_mpi(env=None):\n    return _get_mpi_implementation(env) == _SMPI_IMPL\n\n\ndef is_mpich(env=None):\n    return _get_mpi_implementation(env) == _MPICH_IMPL\n\n\ndef is_intel_mpi(env=None):\n    return _get_mpi_implementation(env) == _IMPI_IMPL\n\n\ndef _get_mpi_implementation(env=None):\n    \"\"\"\n    Detects the available MPI implementation by invoking `mpirun --version`.\n    This command is executed by the given execute function, which takes the\n    command as the only argument and returns (output, exit code). Output\n    represents the stdout and stderr as a string.\n\n    Returns one of:\n    - _OMPI_IMPL, _SMPI_IMPL, _MPICH_IMPL or _IMPI_IMPL for known implementations\n    - _UNKNOWN_IMPL for any unknown implementation\n    - _MISSING_IMPL if `mpirun --version` could not be executed.\n\n    :param env: environment variable to use to run mpirun\n    :return: string representing identified implementation\n    \"\"\"\n    command = 'mpirun --version'\n    res = tiny_shell_exec.execute(command, env)\n    if res is None:\n        return _MISSING_IMPL\n    (output, exit_code) = res\n\n    if exit_code == 0:\n        if 'Open MPI' in output or 'OpenRTE' in output:\n            return _OMPI_IMPL\n        elif 'IBM Spectrum MPI' in output:\n            return _SMPI_IMPL\n        elif 'MPICH' in output:\n            return _MPICH_IMPL\n        elif 'Intel(R) MPI' in output:\n            return _IMPI_IMPL\n\n        print('Unknown MPI implementation given in output of mpirun --version:', file=sys.stderr)\n        print(output, file=sys.stderr)\n        return _UNKNOWN_IMPL\n    else:\n        print('Was unable to run {command}:'.format(command=command), file=sys.stderr)\n        print(output, file=sys.stderr)\n        return _MISSING_IMPL\n\n\ndef _get_mpi_implementation_flags(tcp_flag, env=None):\n    if is_open_mpi(env):\n        return list(_OMPI_FLAGS), list(_NO_BINDING_ARGS), _OMPI_IMPL\n    elif is_spectrum_mpi(env):\n        return (list(_SMPI_FLAGS_TCP) if tcp_flag else list(_SMPI_FLAGS)), list(_SOCKET_BINDING_ARGS), _SMPI_IMPL\n    elif is_mpich(env):\n        return list(_MPICH_FLAGS), list(_NO_BINDING_ARGS), _MPICH_IMPL\n    elif is_intel_mpi(env):\n        return list(_IMPI_FLAGS), [], _IMPI_IMPL\n    else:\n        return None, None, None\n\n\ndef mpi_run(settings, nics, env, command, stdout=None, stderr=None):\n    \"\"\"\n    Runs mpi_run.\n\n    Args:\n        settings: Settings for running MPI.\n                  Note: settings.num_proc and settings.hosts must not be None.\n        nics: Interfaces to include by MPI.\n        env: Environment dictionary to use for running command.\n        command: Command and arguments to run as a list of string.\n        stdout: Stdout of the mpi process.\n                Only used when settings.run_func_mode is True.\n        stderr: Stderr of the mpi process.\n                Only used when settings.run_func_mode is True.\n    \"\"\"\n    if env is not None and not isinstance(env, dict):\n        raise Exception('env argument must be a dict, not {type}: {env}'\n                        .format(type=type(env), env=env))\n\n    mpi_impl_flags, impl_binding_args, mpi = _get_mpi_implementation_flags(settings.tcp_flag, env=env)\n    if mpi_impl_flags is None:\n        raise Exception(_MPI_NOT_FOUND_ERROR_MSG)\n\n    impi = _IMPI_IMPL == mpi\n\n    ssh_args = []\n    if settings.ssh_port:\n        ssh_args += [f'-p {settings.ssh_port}']\n    if settings.ssh_identity_file:\n        ssh_args += [f'-i {settings.ssh_identity_file}']\n\n    mpi_ssh_args = ''\n    if ssh_args:\n        joined_ssh_args = ' '.join(ssh_args)\n        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \\\"{joined_ssh_args}\\\"' if impi else f'-mca plm_rsh_args \\\"{joined_ssh_args}\\\"'\n\n    tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(\n        nics=','.join(nics)) if nics and not impi else ''\n    nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(\n        opt='genv' if impi else 'x',\n        nics=','.join(nics)) if nics else ''\n\n    # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues\n    host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)\n    if not impi and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:\n        mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')\n        mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))\n\n    # if user does not specify any hosts, mpirun by default uses local host.\n    # There is no need to specify localhost.\n    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi else 'H',\n                hosts=','.join(host_names) if host_names and impi else settings.hosts)\n\n    ppn_arg = ' '\n    if host_to_slots and impi:\n        ppn = host_to_slots[host_names[0]]\n        for h_name in host_names[1:]:\n            if ppn != host_to_slots[h_name]:\n                raise Exception('''Different slots in -hosts parameter are not supported in Intel(R) MPI.\n                                 Use -machinefile <machine_file> for this purpose.''')\n        ppn_arg = ' -ppn {} '.format(ppn)\n\n    if settings.prefix_output_with_timestamp and not impi:\n        mpi_impl_flags.append('--timestamp-output')\n\n    binding_args = settings.binding_args if settings.binding_args and not impi else ' '.join(impl_binding_args)\n\n    basic_args = '-l' if impi else '--allow-run-as-root --tag-output'\n\n    output = []\n    if settings.output_filename:\n        output.append('-outfile-pattern' if impi else '--output-filename')\n        output.append(settings.output_filename)\n\n    env_list = '' if impi else ' '.join(\n                    '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))\n\n    # Pass all the env variables to the mpirun command.\n    mpirun_command = (\n        'mpirun {basic_args} '\n        '-np {num_proc}{ppn_arg}{hosts_arg} '\n        '{binding_args} '\n        '{mpi_args} '\n        '{mpi_ssh_args} '\n        '{tcp_intf_arg} '\n        '{nccl_socket_intf_arg} '\n        '{output_filename_arg} '\n        '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables\n        .format(basic_args=basic_args,\n                num_proc=settings.num_proc,\n                ppn_arg=ppn_arg,\n                hosts_arg=hosts_arg,\n                binding_args=binding_args,\n                mpi_args=' '.join(mpi_impl_flags),\n                tcp_intf_arg=tcp_intf_arg,\n                nccl_socket_intf_arg=nccl_socket_intf_arg,\n                mpi_ssh_args=mpi_ssh_args,\n                output_filename_arg=' '.join(output),\n                env=env_list,\n                extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',\n                command=' '.join(quote(par) for par in command))\n    )\n\n    if settings.verbose >= 2:\n        print(mpirun_command)\n\n    # we need the driver's PATH and PYTHONPATH in env to run mpirun,\n    # env for mpirun is different to env encoded in mpirun_command\n    for var in ['PATH', 'PYTHONPATH']:\n        if var not in env and var in os.environ:\n            # copy env so we do not leak env modifications\n            env = copy.copy(env)\n            # copy var over from os.environ\n            env[var] = os.environ[var]\n\n    # Execute the mpirun command.\n    if settings.run_func_mode:\n        exit_code = safe_shell_exec.execute(mpirun_command, env=env, stdout=stdout, stderr=stderr)\n        if exit_code != 0:\n            raise RuntimeError(\"mpirun failed with exit code {exit_code}\".format(exit_code=exit_code))\n    else:\n        os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)\n", "path": "horovod/runner/mpi_run.py"}], "after_files": [{"content": "# Copyright 2019 Uber Technologies, Inc. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport copy\nimport os\nimport sys\n\nfrom shlex import quote\n\nfrom horovod.runner.common.util import env as env_util, hosts, safe_shell_exec, tiny_shell_exec\n\n# MPI implementations\n_OMPI_IMPL = 'OpenMPI'\n_SMPI_IMPL = 'SpectrumMPI'\n_MPICH_IMPL = 'MPICH'\n_IMPI_IMPL = \"IntelMPI\"\n_UNKNOWN_IMPL = 'Unknown'\n_MISSING_IMPL = 'Missing'\n\n# Open MPI Flags\n_OMPI_FLAGS = ['-mca pml ob1', '-mca btl ^openib']\n# Spectrum MPI Flags\n_SMPI_FLAGS = []\n_SMPI_FLAGS_TCP = ['-tcp']\n# MPICH Flags\n_MPICH_FLAGS = []\n# Intel MPI Flags\n_IMPI_FLAGS = []\n\n# Threshold for large cluster MPI issues:\n_LARGE_CLUSTER_THRESHOLD = 64\n# No process binding args\n_NO_BINDING_ARGS = ['-bind-to none', '-map-by slot']\n# Process socket binding args\n_SOCKET_BINDING_ARGS = ['-bind-to socket', '-map-by socket', '-rank-by core']\n\n# MPI not found error message\n_MPI_NOT_FOUND_ERROR_MSG= ('horovod does not find an installed MPI.\\n\\n'\n                           'Choose one of:\\n'\n                           '1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod '\n                           '(use --no-cache-dir pip option).\\n'\n                           '2. Run distributed '\n                           'training script using the standard way provided by your'\n                           ' MPI distribution (usually mpirun, srun, or jsrun).\\n'\n                           '3. Use built-in gloo option (horovodrun --gloo ...).')\n\n\ndef mpi_available(env=None):\n    return _get_mpi_implementation(env) not in {_UNKNOWN_IMPL, _MISSING_IMPL}\n\n\ndef is_open_mpi(env=None):\n    return _get_mpi_implementation(env) == _OMPI_IMPL\n\n\ndef is_spectrum_mpi(env=None):\n    return _get_mpi_implementation(env) == _SMPI_IMPL\n\n\ndef is_mpich(env=None):\n    return _get_mpi_implementation(env) == _MPICH_IMPL\n\n\ndef is_intel_mpi(env=None):\n    return _get_mpi_implementation(env) == _IMPI_IMPL\n\n\ndef _get_mpi_implementation(env=None):\n    \"\"\"\n    Detects the available MPI implementation by invoking `mpirun --version`.\n    This command is executed by the given execute function, which takes the\n    command as the only argument and returns (output, exit code). Output\n    represents the stdout and stderr as a string.\n\n    Returns one of:\n    - _OMPI_IMPL, _SMPI_IMPL, _MPICH_IMPL or _IMPI_IMPL for known implementations\n    - _UNKNOWN_IMPL for any unknown implementation\n    - _MISSING_IMPL if `mpirun --version` could not be executed.\n\n    :param env: environment variable to use to run mpirun\n    :return: string representing identified implementation\n    \"\"\"\n    command = 'mpirun --version'\n    res = tiny_shell_exec.execute(command, env)\n    if res is None:\n        return _MISSING_IMPL\n    (output, exit_code) = res\n\n    if exit_code == 0:\n        if 'Open MPI' in output or 'OpenRTE' in output:\n            return _OMPI_IMPL\n        elif 'IBM Spectrum MPI' in output:\n            return _SMPI_IMPL\n        elif 'MPICH' in output:\n            return _MPICH_IMPL\n        elif 'Intel(R) MPI' in output:\n            return _IMPI_IMPL\n\n        print('Unknown MPI implementation given in output of mpirun --version:', file=sys.stderr)\n        print(output, file=sys.stderr)\n        return _UNKNOWN_IMPL\n    else:\n        print('Was unable to run {command}:'.format(command=command), file=sys.stderr)\n        print(output, file=sys.stderr)\n        return _MISSING_IMPL\n\n\ndef _get_mpi_implementation_flags(tcp_flag, env=None):\n    if is_open_mpi(env):\n        return list(_OMPI_FLAGS), list(_NO_BINDING_ARGS), _OMPI_IMPL\n    elif is_spectrum_mpi(env):\n        return (list(_SMPI_FLAGS_TCP) if tcp_flag else list(_SMPI_FLAGS)), list(_SOCKET_BINDING_ARGS), _SMPI_IMPL\n    elif is_mpich(env):\n        return list(_MPICH_FLAGS), list(_NO_BINDING_ARGS), _MPICH_IMPL\n    elif is_intel_mpi(env):\n        return list(_IMPI_FLAGS), [], _IMPI_IMPL\n    else:\n        return None, None, None\n\n\ndef mpi_run(settings, nics, env, command, stdout=None, stderr=None):\n    \"\"\"\n    Runs mpi_run.\n\n    Args:\n        settings: Settings for running MPI.\n                  Note: settings.num_proc and settings.hosts must not be None.\n        nics: Interfaces to include by MPI.\n        env: Environment dictionary to use for running command.\n        command: Command and arguments to run as a list of string.\n        stdout: Stdout of the mpi process.\n                Only used when settings.run_func_mode is True.\n        stderr: Stderr of the mpi process.\n                Only used when settings.run_func_mode is True.\n    \"\"\"\n    if env is not None and not isinstance(env, dict):\n        raise Exception('env argument must be a dict, not {type}: {env}'\n                        .format(type=type(env), env=env))\n\n    mpi_impl_flags, impl_binding_args, mpi = _get_mpi_implementation_flags(settings.tcp_flag, env=env)\n    if mpi_impl_flags is None:\n        raise Exception(_MPI_NOT_FOUND_ERROR_MSG)\n\n    impi_or_mpich = mpi in (_IMPI_IMPL, _MPICH_IMPL)\n\n    ssh_args = []\n    if settings.ssh_port:\n        ssh_args += [f'-p {settings.ssh_port}']\n    if settings.ssh_identity_file:\n        ssh_args += [f'-i {settings.ssh_identity_file}']\n\n    mpi_ssh_args = ''\n    if ssh_args:\n        joined_ssh_args = ' '.join(ssh_args)\n        mpi_ssh_args = f'-bootstrap=ssh -bootstrap-exec-args \\\"{joined_ssh_args}\\\"' if impi_or_mpich else f'-mca plm_rsh_args \\\"{joined_ssh_args}\\\"'\n\n    tcp_intf_arg = '-mca btl_tcp_if_include {nics}'.format(\n        nics=','.join(nics)) if nics and not impi_or_mpich else ''\n    nccl_socket_intf_arg = '-{opt} NCCL_SOCKET_IFNAME={nics}'.format(\n        opt='genv' if impi_or_mpich else 'x',\n        nics=','.join(nics)) if nics else ''\n\n    # On large cluster runs (e.g. Summit), we need extra settings to work around OpenMPI issues\n    host_names, host_to_slots = hosts.parse_hosts_and_slots(settings.hosts)\n    if not impi_or_mpich and host_names and len(host_names) >= _LARGE_CLUSTER_THRESHOLD:\n        mpi_impl_flags.append('-mca plm_rsh_no_tree_spawn true')\n        mpi_impl_flags.append('-mca plm_rsh_num_concurrent {}'.format(len(host_names)))\n\n    # if user does not specify any hosts, mpirun by default uses local host.\n    # There is no need to specify localhost.\n    hosts_arg = '-{opt} {hosts}'.format(opt='hosts' if impi_or_mpich else 'H',\n                hosts=','.join(host_names) if host_names and impi_or_mpich else settings.hosts)\n\n    ppn_arg = ' '\n    if host_to_slots and impi_or_mpich:\n        ppn = host_to_slots[host_names[0]]\n        for h_name in host_names[1:]:\n            if ppn != host_to_slots[h_name]:\n                raise Exception('''Different slots in -hosts parameter are not supported in Intel(R) MPI.\n                                 Use -machinefile <machine_file> for this purpose.''')\n        ppn_arg = ' -ppn {} '.format(ppn)\n\n    if settings.prefix_output_with_timestamp and not impi_or_mpich:\n        mpi_impl_flags.append('--timestamp-output')\n\n    binding_args = settings.binding_args if settings.binding_args and not impi_or_mpich else ' '.join(impl_binding_args)\n\n    basic_args = '-l' if impi_or_mpich else '--allow-run-as-root --tag-output'\n\n    output = []\n    if settings.output_filename:\n        output.append('-outfile-pattern' if impi_or_mpich else '--output-filename')\n        output.append(settings.output_filename)\n\n    env_list = '' if impi_or_mpich else ' '.join(\n                    '-x %s' % key for key in sorted(env.keys()) if env_util.is_exportable(key))\n\n    # Pass all the env variables to the mpirun command.\n    mpirun_command = (\n        'mpirun {basic_args} '\n        '-np {num_proc}{ppn_arg}{hosts_arg} '\n        '{binding_args} '\n        '{mpi_args} '\n        '{mpi_ssh_args} '\n        '{tcp_intf_arg} '\n        '{nccl_socket_intf_arg} '\n        '{output_filename_arg} '\n        '{env} {extra_mpi_args} {command}'  # expect a lot of environment variables\n        .format(basic_args=basic_args,\n                num_proc=settings.num_proc,\n                ppn_arg=ppn_arg,\n                hosts_arg=hosts_arg,\n                binding_args=binding_args,\n                mpi_args=' '.join(mpi_impl_flags),\n                tcp_intf_arg=tcp_intf_arg,\n                nccl_socket_intf_arg=nccl_socket_intf_arg,\n                mpi_ssh_args=mpi_ssh_args,\n                output_filename_arg=' '.join(output),\n                env=env_list,\n                extra_mpi_args=settings.extra_mpi_args if settings.extra_mpi_args else '',\n                command=' '.join(quote(par) for par in command))\n    )\n\n    if settings.verbose >= 2:\n        print(mpirun_command)\n\n    # we need the driver's PATH and PYTHONPATH in env to run mpirun,\n    # env for mpirun is different to env encoded in mpirun_command\n    for var in ['PATH', 'PYTHONPATH']:\n        if var not in env and var in os.environ:\n            # copy env so we do not leak env modifications\n            env = copy.copy(env)\n            # copy var over from os.environ\n            env[var] = os.environ[var]\n\n    # Execute the mpirun command.\n    if settings.run_func_mode:\n        exit_code = safe_shell_exec.execute(mpirun_command, env=env, stdout=stdout, stderr=stderr)\n        if exit_code != 0:\n            raise RuntimeError(\"mpirun failed with exit code {exit_code}\".format(exit_code=exit_code))\n    else:\n        os.execve('/bin/sh', ['/bin/sh', '-c', mpirun_command], env)\n", "path": "horovod/runner/mpi_run.py"}]}