title: subprocess.CalledProcessError: Command '['mpirun', '-np', '2', '/usr/bin/python3.7', '-u', 'train.py']' returned non-zero exit status 134
body: <p><strong>Hello, I am using python 3.7 and running the following code :</strong></p>
  <pre><code>if &quot;parent&quot; == mpi_fork(args.num_worker+1): os.exit()
  </code></pre>
  <p><strong>where mpi_fork is</strong></p>
  <pre><code>def mpi_fork(n):
  &quot;&quot;&quot;Re-launches the current script with workers
    Returns &quot;parent&quot; for original parent, &quot;child&quot; for MPI children
    (from https://github.com/garymcintire/mpi_util/)
  &quot;&quot;&quot;
   if n&lt;=1:
      return &quot;child&quot;
   if os.getenv(&quot;IN_MPI&quot;) is None:
      env = os.environ.copy()
      env.update(
        MKL_NUM_THREADS=&quot;1&quot;,
        OMP_NUM_THREADS=&quot;1&quot;,
        IN_MPI=&quot;1&quot;
      )
      print(&quot;\n\nHello\n\n&quot;)
      print( [&quot;mpirun&quot;, &quot;-np&quot;, str(n), sys.executable] + sys.argv)
      subprocess.check_call([&quot;mpirun&quot;, &quot;-np&quot;, str(n),sys.executable] +['-u']+ sys.argv,    env=env)
      return &quot;parent&quot;
    else:
      global nworkers, rank
      nworkers = comm.Get_size()
      rank = comm.Get_rank()
      print('assigning the rank and nworkers', nworkers, rank)
      return &quot;child&quot;
  </code></pre>
  <p><strong>and I've got the following error:</strong></p>
  <pre><code>2020-06-24 10:00:55.817305: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created     TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6271 MB memory) -&gt; physical   GPU (device: 0, name: Quadro P4200, pci bus id: 0000:01:00.0, compute capability: 6.1)
  2020-06-24 10:00:55.858934: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 6.12G (6576128000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.859365: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 5.51G (5918515200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.859810: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.96G (5326663680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.860166: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.46G (4793997312 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.860540: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 4.02G (4314597376 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.861072: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 3.62G (3883137536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.861451: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 3.25G (3494823680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.861801: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 2.93G (3145341184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.862153: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 2.64G (2830807040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.862554: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 2.37G (2547726336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.862936: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 2.13G (2292953600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.863306: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.92G (2063658240 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.863665: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.73G (1857292544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.864025: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.56G (1671563264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.864391: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.40G (1504407040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.864748: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.26G (1353966336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.865104: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.13G (1218569728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.865569: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 1.02G (1096712704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.866050: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 941.32M (987041536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.866422: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 847.18M (888337408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.866779: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 762.47M (799503616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.867130: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 686.22M (719553280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.867488: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 617.60M (647598080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.867860: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 555.84M (582838272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.868222: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 500.25M (524554496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.868577: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 450.23M (472099072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.868930: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 405.21M (424889344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.869288: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 364.69M (382400512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.869652: I tensorflow/stream_executor/cuda/cuda_driver.cc:801] failed to allocate 328.22M (344160512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
  2020-06-24 10:00:55.960490: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch&lt;Distribution&gt;, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: out of memory
  
  [antonio-HP-ZBook-17-G5:07714] *** Process received signal ***
  
  [antonio-HP-ZBook-17-G5:07714] Signal: Aborted (6)
  
  [antonio-HP-ZBook-17-G5:07714] Signal code:  (-6)
  
  2020-06-24 10:00:55.960557: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch&lt;Distribution&gt;, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: out of memory
  
  [antonio-HP-ZBook-17-G5:07715] *** Process received signal ***
  
  [antonio-HP-ZBook-17-G5:07715] Signal: Aborted (6)
  
  [antonio-HP-ZBook-17-G5:07715] Signal code:  (-6)
  
  
  Primary job  terminated normally, but 1 process returned
  a non-zero exit code. Per user-direction, the job has been aborted.
  
  
  mpirun noticed that process rank 1 with PID 0 on node antonio-HP-ZBook-17-G5 exited on signal 6 (Aborted).
  Traceback (most recent call last):
    File &quot;train.py&quot;, line 460, in &lt;module&gt;
      if &quot;parent&quot; == mpi_fork(args.num_worker+1): os.exit()
    File &quot;train.py&quot;, line 433, in mpi_fork
      subprocess.check_call([&quot;mpirun&quot;, &quot;-np&quot;, str(n),sys.executable] +['-u']+ sys.argv, env=env)
    File &quot;/usr/lib/python3.7/subprocess.py&quot;, line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['mpirun', '-np', '2', '/usr/bin/python3.7', '-u', 'train.py']' returned non-zero exit status 134.
  </code></pre>
  <p><strong>Can anyone help me to solve this probmlem?</strong></p>