<p dir="auto">Flaky occurrence:</p>
<p dir="auto"><a href="https://circle.pytorch.org/build-details.html?build_id=2161720" rel="nofollow">https://circle.pytorch.org/build-details.html?build_id=2161720</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Jul 09 19:35:19 test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend) ... Process process 0:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 258, in _bootstrap
Jul 09 19:35:19     self.run()
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 93, in run
Jul 09 19:35:19     self._target(*self._args, **self._kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1661, in _run
Jul 09 19:35:19     getattr(self, self.id().split(&quot;.&quot;)[2])()
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1597, in wrapper
Jul 09 19:35:19     fn(self)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 120, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 136, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19     model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 298, in __init__
Jul 09 19:35:19     self.broadcast_bucket_size)
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19     dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: NCCL error in: /var/lib/jenkins/workspace/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system error
Jul 09 19:35:19 Process process 1:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 258, in _bootstrap
Jul 09 19:35:19     self.run()
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/multiprocessing/process.py&quot;, line 93, in run
Jul 09 19:35:19     self._target(*self._args, **self._kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1661, in _run
Jul 09 19:35:19     getattr(self, self.id().split(&quot;.&quot;)[2])()
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1597, in wrapper
Jul 09 19:35:19     fn(self)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 120, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 136, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File &quot;test_distributed.py&quot;, line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19     model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 298, in __init__
Jul 09 19:35:19     self.broadcast_bucket_size)
Jul 09 19:35:19   File &quot;/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py&quot;, line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19     dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: Connection reset by peer
Jul 09 19:40:19 FAIL

Jul 09 19:40:48 ======================================================================
Jul 09 19:40:48 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
Jul 09 19:40:48 ----------------------------------------------------------------------
Jul 09 19:40:48 Traceback (most recent call last):
Jul 09 19:40:48   File &quot;test_distributed.py&quot;, line 1595, in wrapper
Jul 09 19:40:48     self._join_and_reduce(fn)
Jul 09 19:40:48   File &quot;test_distributed.py&quot;, line 1677, in _join_and_reduce
Jul 09 19:40:48     &quot;Timeout waiting for rank %d to terminate&quot; % rank)
Jul 09 19:40:48 AssertionError: True is not false : Timeout waiting for rank 2 to terminate
Jul 09 19:40:48 
Jul 09 19:40:48 ----------------------------------------------------------------------"><pre class="notranslate"><code class="notranslate">Jul 09 19:35:19 test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend) ... Process process 0:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Jul 09 19:35:19     self.run()
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
Jul 09 19:35:19     self._target(*self._args, **self._kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 1661, in _run
Jul 09 19:35:19     getattr(self, self.id().split(".")[2])()
Jul 09 19:35:19   File "test_distributed.py", line 1597, in wrapper
Jul 09 19:35:19     fn(self)
Jul 09 19:35:19   File "test_distributed.py", line 120, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 136, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19     model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 298, in __init__
Jul 09 19:35:19     self.broadcast_bucket_size)
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19     dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: NCCL error in: /var/lib/jenkins/workspace/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system error
Jul 09 19:35:19 Process process 1:
Jul 09 19:35:19 Traceback (most recent call last):
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
Jul 09 19:35:19     self.run()
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/multiprocessing/process.py", line 93, in run
Jul 09 19:35:19     self._target(*self._args, **self._kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 1661, in _run
Jul 09 19:35:19     getattr(self, self.id().split(".")[2])()
Jul 09 19:35:19   File "test_distributed.py", line 1597, in wrapper
Jul 09 19:35:19     fn(self)
Jul 09 19:35:19   File "test_distributed.py", line 120, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 136, in wrapper
Jul 09 19:35:19     return func(*args, **kwargs)
Jul 09 19:35:19   File "test_distributed.py", line 1548, in test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
Jul 09 19:35:19     model = nn.parallel.DistributedDataParallel(ONLY_SBN_NET.cuda(rank), device_ids=[rank])
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 298, in __init__
Jul 09 19:35:19     self.broadcast_bucket_size)
Jul 09 19:35:19   File "/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 480, in _distributed_broadcast_coalesced
Jul 09 19:35:19     dist._broadcast_coalesced(self.process_group, tensors, buffer_size)
Jul 09 19:35:19 RuntimeError: Connection reset by peer
Jul 09 19:40:19 FAIL

Jul 09 19:40:48 ======================================================================
Jul 09 19:40:48 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
Jul 09 19:40:48 ----------------------------------------------------------------------
Jul 09 19:40:48 Traceback (most recent call last):
Jul 09 19:40:48   File "test_distributed.py", line 1595, in wrapper
Jul 09 19:40:48     self._join_and_reduce(fn)
Jul 09 19:40:48   File "test_distributed.py", line 1677, in _join_and_reduce
Jul 09 19:40:48     "Timeout waiting for rank %d to terminate" % rank)
Jul 09 19:40:48 AssertionError: True is not false : Timeout waiting for rank 2 to terminate
Jul 09 19:40:48 
Jul 09 19:40:48 ----------------------------------------------------------------------
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/unlimblue/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/unlimblue">@unlimblue</a></p>
<h2 dir="auto"><g-emoji class="g-emoji" alias="bug" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f41b.png">🐛</g-emoji> Bug</h2>
<p dir="auto"><code class="notranslate">TestDistBackend.test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value</code> seems to be flaky</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="23:43:08 ======================================================================
23:43:08 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
23:43:08 ----------------------------------------------------------------------
23:43:08 Traceback (most recent call last):
23:43:08   File &quot;test_distributed.py&quot;, line 1986, in wrapper
23:43:08     self._join_and_reduce(fn)
23:43:08   File &quot;test_distributed.py&quot;, line 2079, in _join_and_reduce
23:43:08     &quot;Timeout waiting for rank %d to terminate&quot; % rank)
23:43:08 AssertionError: True is not false : Timeout waiting for rank 1 to terminate"><pre class="notranslate"><code class="notranslate">23:43:08 ======================================================================
23:43:08 FAIL: test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackend)
23:43:08 ----------------------------------------------------------------------
23:43:08 Traceback (most recent call last):
23:43:08   File "test_distributed.py", line 1986, in wrapper
23:43:08     self._join_and_reduce(fn)
23:43:08   File "test_distributed.py", line 2079, in _join_and_reduce
23:43:08     "Timeout waiting for rank %d to terminate" % rank)
23:43:08 AssertionError: True is not false : Timeout waiting for rank 1 to terminate
</code></pre></div>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ezyang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ezyang">@ezyang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gchanan/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gchanan">@gchanan</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zou3519/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zou3519">@zou3519</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jerryzh168/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jerryzh168">@jerryzh168</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pietern/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pietern">@pietern</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mrshenli/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mrshenli">@mrshenli</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/pritamdamania87/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/pritamdamania87">@pritamdamania87</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/zhaojuanmao/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/zhaojuanmao">@zhaojuanmao</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/satgera/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/satgera">@satgera</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/rohan-varma/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/rohan-varma">@rohan-varma</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gqchen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gqchen">@gqchen</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/aazzolini/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/aazzolini">@aazzolini</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/xush6528/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/xush6528">@xush6528</a></p>