<p dir="auto"><a href="https://app.circleci.com/pipelines/github/pytorch/pytorch/217203/workflows/9228f0bf-ed03-47fe-9a3d-dc736b8e14c3/jobs/7653966/steps" rel="nofollow">https://app.circleci.com/pipelines/github/pytorch/pytorch/217203/workflows/9228f0bf-ed03-47fe-9a3d-dc736b8e14c3/jobs/7653966/steps</a></p>
<p dir="auto">Same as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="643908803" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/40434" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/40434/hovercard" href="https://github.com/pytorch/pytorch/issues/40434">#40434</a>, but for TensorPipe backend</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sep 21 23:53:07 ERROR:root:Caught exception: 
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py&quot;, line 246, in wrapper
Sep 21 23:53:07     fn()
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/dist_utils.py&quot;, line 70, in new_test_method
Sep 21 23:53:07     return_value = old_test_method(self, *arg, **kwargs)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 481, in test_backward_ddp_inside
Sep 21 23:53:07     self._do_test(DdpMode.INSIDE)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 455, in _do_test
Sep 21 23:53:07     self._master_process(ddp_mode, simulate_uneven_inputs)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 377, in _master_process
Sep 21 23:53:07     ddp_mode, simulate_uneven_inputs, remote_em_rref, remote_net_rref
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 423, in do_test_on_master
Sep 21 23:53:07     ddp_grads, non_ddp_grads = future.wait()
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 177, in _handle_exception
Sep 21 23:53:07     raise result.exception_type(result.msg)
Sep 21 23:53:07 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 21 23:53:07 AssertionError('On WorkerInfo(id=0, name=worker0):\nAssertionError(\'Default process group is not initialized\')\nTraceback (most recent call last):\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 186, in __init__\n    if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1979, in new_group\n    _check_default_pg()\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 211, in _check_default_pg\n    &quot;Default process group is not initialized&quot;\nAssertionError: Default process group is not initialized\n')
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function
Sep 21 23:53:07     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 78, in _call_method
Sep 21 23:53:07     return method(rref.local_value(), *args, **kwargs)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 177, in _handle_exception
Sep 21 23:53:07     raise result.exception_type(result.msg)
Sep 21 23:53:07 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 21 23:53:07 AssertionError('Default process group is not initialized')
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function
Sep 21 23:53:07     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 186, in __init__
Sep 21 23:53:07     if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1979, in new_group
Sep 21 23:53:07     _check_default_pg()
Sep 21 23:53:07   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 211, in _check_default_pg
Sep 21 23:53:07     &quot;Default process group is not initialized&quot;
Sep 21 23:53:07 AssertionError: Default process group is not initialized"><pre class="notranslate"><code class="notranslate">Sep 21 23:53:07 ERROR:root:Caught exception: 
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 246, in wrapper
Sep 21 23:53:07     fn()
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/dist_utils.py", line 70, in new_test_method
Sep 21 23:53:07     return_value = old_test_method(self, *arg, **kwargs)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 481, in test_backward_ddp_inside
Sep 21 23:53:07     self._do_test(DdpMode.INSIDE)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 455, in _do_test
Sep 21 23:53:07     self._master_process(ddp_mode, simulate_uneven_inputs)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 377, in _master_process
Sep 21 23:53:07     ddp_mode, simulate_uneven_inputs, remote_em_rref, remote_net_rref
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 423, in do_test_on_master
Sep 21 23:53:07     ddp_grads, non_ddp_grads = future.wait()
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 177, in _handle_exception
Sep 21 23:53:07     raise result.exception_type(result.msg)
Sep 21 23:53:07 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 21 23:53:07 AssertionError('On WorkerInfo(id=0, name=worker0):\nAssertionError(\'Default process group is not initialized\')\nTraceback (most recent call last):\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 186, in __init__\n    if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1979, in new_group\n    _check_default_pg()\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 211, in _check_default_pg\n    "Default process group is not initialized"\nAssertionError: Default process group is not initialized\n')
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function
Sep 21 23:53:07     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 78, in _call_method
Sep 21 23:53:07     return method(rref.local_value(), *args, **kwargs)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 177, in _handle_exception
Sep 21 23:53:07     raise result.exception_type(result.msg)
Sep 21 23:53:07 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 21 23:53:07 AssertionError('Default process group is not initialized')
Sep 21 23:53:07 Traceback (most recent call last):
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function
Sep 21 23:53:07     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 186, in __init__
Sep 21 23:53:07     if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1979, in new_group
Sep 21 23:53:07     _check_default_pg()
Sep 21 23:53:07   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 211, in _check_default_pg
Sep 21 23:53:07     "Default process group is not initialized"
Sep 21 23:53:07 AssertionError: Default process group is not initialized
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jjlilley/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jjlilley">@jjlilley</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jiayisuse/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jiayisuse">@jiayisuse</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lw">@lw</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/beauby/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/beauby">@beauby</a></p>
<p dir="auto"><a href="https://app.circleci.com/pipelines/github/pytorch/pytorch/217196/workflows/15141629-1343-446c-a9cf-230f9c6b4527/jobs/7655526/steps" rel="nofollow">https://app.circleci.com/pipelines/github/pytorch/pytorch/217196/workflows/15141629-1343-446c-a9cf-230f9c6b4527/jobs/7655526/steps</a></p>
<p dir="auto">Probably same issue as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="643908803" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/40434" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/40434/hovercard" href="https://github.com/pytorch/pytorch/issues/40434">#40434</a>, which affects ProcessGroup.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Sep 22 01:48:29   test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTestWithSpawn) ... 2020-09-22 01:48:29,011 ddp_under_dist_autograd_test.py:342 INFO p:process 0 t:MainThread: Running the trainer #0...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 1 t:MainThread: Running the trainer #1...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 2 t:MainThread: Running the trainer #2...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 3 t:MainThread: Running the trainer #3...
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:344 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:328 INFO p:process 4 t:MainThread: The remote worker is running.
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:362 INFO p:process 5 t:MainThread: Running the master process...
Sep 22 01:48:29 2020-09-22 01:48:29,016 ddp_under_dist_autograd_test.py:353 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
Sep 22 01:48:29 2020-09-22 01:48:29,017 ddp_under_dist_autograd_test.py:353 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
Sep 22 01:48:29 2020-09-22 01:48:29,017 ddp_under_dist_autograd_test.py:353 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
Sep 22 01:48:29 2020-09-22 01:48:29,018 ddp_under_dist_autograd_test.py:375 INFO p:process 5 t:MainThread: Created remote rrefs on master
Sep 22 01:48:29 2020-09-22 01:48:29,019 ddp_under_dist_autograd_test.py:93 INFO p:process 4 t:Dummy-1: Initing RemoteEM with 2 3
Sep 22 01:48:29 2020-09-22 01:48:29,020 ddp_under_dist_autograd_test.py:119 INFO p:process 4 t:Dummy-1: Initing RemoteNet with 5 3
Sep 22 01:48:29 2020-09-22 01:48:29,032 ddp_under_dist_autograd_test.py:353 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
Sep 22 01:48:29 ERROR:root:Caught exception: 
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py&quot;, line 246, in wrapper
Sep 22 01:48:29     fn()
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/dist_utils.py&quot;, line 70, in new_test_method
Sep 22 01:48:29     return_value = old_test_method(self, *arg, **kwargs)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 481, in test_backward_ddp_inside
Sep 22 01:48:29     self._do_test(DdpMode.INSIDE)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 455, in _do_test
Sep 22 01:48:29     self._master_process(ddp_mode, simulate_uneven_inputs)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 377, in _master_process
Sep 22 01:48:29     ddp_mode, simulate_uneven_inputs, remote_em_rref, remote_net_rref
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 423, in do_test_on_master
Sep 22 01:48:29     ddp_grads, non_ddp_grads = future.wait()
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 177, in _handle_exception
Sep 22 01:48:29     raise result.exception_type(result.msg)
Sep 22 01:48:29 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 22 01:48:29 AssertionError('On WorkerInfo(id=0, name=worker0):\nAssertionError(\'Default process group is not initialized\')\nTraceback (most recent call last):\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 186, in __init__\n    if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1979, in new_group\n    _check_default_pg()\n  File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 211, in _check_default_pg\n    &quot;Default process group is not initialized&quot;\nAssertionError: Default process group is not initialized\n')
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function
Sep 22 01:48:29     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 78, in _call_method
Sep 22 01:48:29     return method(rref.local_value(), *args, **kwargs)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 177, in _handle_exception
Sep 22 01:48:29     raise result.exception_type(result.msg)
Sep 22 01:48:29 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 22 01:48:29 AssertionError('Default process group is not initialized')
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py&quot;, line 164, in _run_function
Sep 22 01:48:29     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py&quot;, line 186, in __init__
Sep 22 01:48:29     if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 1979, in new_group
Sep 22 01:48:29     _check_default_pg()
Sep 22 01:48:29   File &quot;/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py&quot;, line 211, in _check_default_pg
Sep 22 01:48:29     &quot;Default process group is not initialized&quot;
Sep 22 01:48:29 AssertionError: Default process group is not initialized
Sep 22 01:48:29 
Sep 22 01:48:29 
Sep 22 01:48:29 exiting process with exit code: 10
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker3 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker2 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker4 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker1 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker0 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 Process 5 terminated with exit code 10, terminating remaining processes.
Sep 22 01:48:29 ERROR (2.361s)"><pre class="notranslate"><code class="notranslate">Sep 22 01:48:29   test_backward_ddp_inside (__main__.TensorPipeDdpUnderDistAutogradTestWithSpawn) ... 2020-09-22 01:48:29,011 ddp_under_dist_autograd_test.py:342 INFO p:process 0 t:MainThread: Running the trainer #0...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 0 t:MainThread: Initing trainer process group by trainer #0 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 1 t:MainThread: Running the trainer #1...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 2 t:MainThread: Running the trainer #2...
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 1 t:MainThread: Initing trainer process group by trainer #1 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:344 INFO p:process 2 t:MainThread: Initing trainer process group by trainer #2 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,012 ddp_under_dist_autograd_test.py:342 INFO p:process 3 t:MainThread: Running the trainer #3...
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:344 INFO p:process 3 t:MainThread: Initing trainer process group by trainer #3 with ranks [0, 1, 2, 3]
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:328 INFO p:process 4 t:MainThread: The remote worker is running.
Sep 22 01:48:29 2020-09-22 01:48:29,013 ddp_under_dist_autograd_test.py:362 INFO p:process 5 t:MainThread: Running the master process...
Sep 22 01:48:29 2020-09-22 01:48:29,016 ddp_under_dist_autograd_test.py:353 INFO p:process 1 t:MainThread: Waiting for shutdown signal on trainer #1...
Sep 22 01:48:29 2020-09-22 01:48:29,017 ddp_under_dist_autograd_test.py:353 INFO p:process 2 t:MainThread: Waiting for shutdown signal on trainer #2...
Sep 22 01:48:29 2020-09-22 01:48:29,017 ddp_under_dist_autograd_test.py:353 INFO p:process 3 t:MainThread: Waiting for shutdown signal on trainer #3...
Sep 22 01:48:29 2020-09-22 01:48:29,018 ddp_under_dist_autograd_test.py:375 INFO p:process 5 t:MainThread: Created remote rrefs on master
Sep 22 01:48:29 2020-09-22 01:48:29,019 ddp_under_dist_autograd_test.py:93 INFO p:process 4 t:Dummy-1: Initing RemoteEM with 2 3
Sep 22 01:48:29 2020-09-22 01:48:29,020 ddp_under_dist_autograd_test.py:119 INFO p:process 4 t:Dummy-1: Initing RemoteNet with 5 3
Sep 22 01:48:29 2020-09-22 01:48:29,032 ddp_under_dist_autograd_test.py:353 INFO p:process 0 t:MainThread: Waiting for shutdown signal on trainer #0...
Sep 22 01:48:29 ERROR:root:Caught exception: 
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 246, in wrapper
Sep 22 01:48:29     fn()
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/dist_utils.py", line 70, in new_test_method
Sep 22 01:48:29     return_value = old_test_method(self, *arg, **kwargs)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 481, in test_backward_ddp_inside
Sep 22 01:48:29     self._do_test(DdpMode.INSIDE)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 455, in _do_test
Sep 22 01:48:29     self._master_process(ddp_mode, simulate_uneven_inputs)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 377, in _master_process
Sep 22 01:48:29     ddp_mode, simulate_uneven_inputs, remote_em_rref, remote_net_rref
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 423, in do_test_on_master
Sep 22 01:48:29     ddp_grads, non_ddp_grads = future.wait()
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 177, in _handle_exception
Sep 22 01:48:29     raise result.exception_type(result.msg)
Sep 22 01:48:29 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 22 01:48:29 AssertionError('On WorkerInfo(id=0, name=worker0):\nAssertionError(\'Default process group is not initialized\')\nTraceback (most recent call last):\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 186, in __init__\n    if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1979, in new_group\n    _check_default_pg()\n  File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 211, in _check_default_pg\n    "Default process group is not initialized"\nAssertionError: Default process group is not initialized\n')
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function
Sep 22 01:48:29     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 78, in _call_method
Sep 22 01:48:29     return method(rref.local_value(), *args, **kwargs)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 177, in _handle_exception
Sep 22 01:48:29     raise result.exception_type(result.msg)
Sep 22 01:48:29 AssertionError: On WorkerInfo(id=0, name=worker0):
Sep 22 01:48:29 AssertionError('Default process group is not initialized')
Sep 22 01:48:29 Traceback (most recent call last):
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py", line 164, in _run_function
Sep 22 01:48:29     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/distributed/ddp_under_dist_autograd_test.py", line 186, in __init__
Sep 22 01:48:29     if ddp_mode in (DdpMode.INSIDE, DdpMode.OUTSIDE)
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1979, in new_group
Sep 22 01:48:29     _check_default_pg()
Sep 22 01:48:29   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 211, in _check_default_pg
Sep 22 01:48:29     "Default process group is not initialized"
Sep 22 01:48:29 AssertionError: Default process group is not initialized
Sep 22 01:48:29 
Sep 22 01:48:29 
Sep 22 01:48:29 exiting process with exit code: 10
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker3 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker2 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker4 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker1 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 [W tensorpipe_agent.cpp:577] RPC agent for worker0 encountered error when reading incoming request from worker5: EOF: end of file (this is expected to happen during shutdown)
Sep 22 01:48:29 Process 5 terminated with exit code 10, terminating remaining processes.
Sep 22 01:48:29 ERROR (2.361s)
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/osalpekar/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/osalpekar">@osalpekar</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jiayisuse/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jiayisuse">@jiayisuse</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/agolynski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/agolynski">@agolynski</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jjlilley/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jjlilley">@jjlilley</a></p>