<p dir="auto">It seems that masked_fill_ is more dangerous for zero-strided tensors, because the mask may be different for the elements that refer to the single location in memor</p>
<p dir="auto">Probably related issue: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="365607801" data-permission-text="Title is private" data-url="https://github.com/pytorch/pytorch/issues/12230" data-hovercard-type="issue" data-hovercard-url="/pytorch/pytorch/issues/12230/hovercard" href="https://github.com/pytorch/pytorch/issues/12230">#12230</a></p>
<p dir="auto">My Torch version is <code class="notranslate">1.1.0</code>.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import torch
a = torch.zeros(2)
b = a.expand(2, 2)

b *= 2 # breaks
# RuntimeError: mul: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before calling mul

mask = torch.ByteTensor([[0, 1], [1, 0]])
b.masked_fill_(mask, 0) # works, but semantically unsafe"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">torch</span>
<span class="pl-s1">a</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">zeros</span>(<span class="pl-c1">2</span>)
<span class="pl-s1">b</span> <span class="pl-c1">=</span> <span class="pl-s1">a</span>.<span class="pl-en">expand</span>(<span class="pl-c1">2</span>, <span class="pl-c1">2</span>)

<span class="pl-s1">b</span> <span class="pl-c1">*=</span> <span class="pl-c1">2</span> <span class="pl-c"># breaks</span>
<span class="pl-c"># RuntimeError: mul: unsupported operation: more than one element of the written-to tensor refers to a single memory location. Please clone() the tensor before calling mul</span>

<span class="pl-s1">mask</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">ByteTensor</span>([[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>], [<span class="pl-c1">1</span>, <span class="pl-c1">0</span>]])
<span class="pl-s1">b</span>.<span class="pl-en">masked_fill_</span>(<span class="pl-s1">mask</span>, <span class="pl-c1">0</span>) <span class="pl-c"># works, but semantically unsafe</span></pre></div>
<p dir="auto">I discovered this when writing some masking code, it seems that unexpected behavior can arise when using pointwise operations to write to expanded tensors. The code below reproduces and explains the issue:</p>
<p dir="auto">We start off by constructing an example array and constructing a mask from it:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="example_batch = torch.Tensor([
    [[1, 2, 0, 0],
     [2, 1, 3, 0],
     [2, 1, 3, 0]]
])

mask = (example_batch == 0)
&gt;&gt;&gt; mask
tensor([[[0, 0, 1, 1],
         [0, 0, 0, 1],
         [0, 0, 0, 1]]], dtype=torch.uint8)"><pre class="notranslate"><span class="pl-s1">example_batch</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-v">Tensor</span>([
    [[<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>],
     [<span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>, <span class="pl-c1">0</span>],
     [<span class="pl-c1">2</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>, <span class="pl-c1">0</span>]]
])

<span class="pl-s1">mask</span> <span class="pl-c1">=</span> (<span class="pl-s1">example_batch</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">mask</span>
<span class="pl-en">tensor</span>([[[<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>],
         [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>],
         [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>]]], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">torch</span>.<span class="pl-s1">uint8</span>)</pre></div>
<p dir="auto">We then expand a range into our example batch:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="index_filled = torch.arange(1, example_batch.size(-1) + 1).expand_as(example_batch)
&gt;&gt;&gt; index_filled
tensor([[[1, 2, 3, 4],
         [1, 2, 3, 4],
         [1, 2, 3, 4]]])"><pre class="notranslate"><span class="pl-s1">index_filled</span> <span class="pl-c1">=</span> <span class="pl-s1">torch</span>.<span class="pl-en">arange</span>(<span class="pl-c1">1</span>, <span class="pl-s1">example_batch</span>.<span class="pl-en">size</span>(<span class="pl-c1">-</span><span class="pl-c1">1</span>) <span class="pl-c1">+</span> <span class="pl-c1">1</span>).<span class="pl-en">expand_as</span>(<span class="pl-s1">example_batch</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">index_filled</span>
<span class="pl-en">tensor</span>([[[<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>]]])</pre></div>
<p dir="auto">So far, so good, yet the problem appears when we try to do a <code class="notranslate">masked_fill</code> on <code class="notranslate">index_filled</code> using our masked array:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="masked = index_filled.masked_fill_(mask, 0)
&gt;&gt;&gt; masked
tensor([[[1, 2, 0, 0],
         [1, 2, 0, 0],
         [1, 2, 0, 0]]])"><pre class="notranslate"><span class="pl-s1">masked</span> <span class="pl-c1">=</span> <span class="pl-s1">index_filled</span>.<span class="pl-en">masked_fill_</span>(<span class="pl-s1">mask</span>, <span class="pl-c1">0</span>)
<span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">masked</span>
<span class="pl-en">tensor</span>([[[<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>]]])</pre></div>
<p dir="auto">This doesn't seem to be what we would expect from our mask operation, what we where expecting was:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="tensor([[[1., 2., 0., 0.],
         [1., 2., 3., 0.],
         [1., 2., 3., 0.]]])"><pre class="notranslate"><span class="pl-en">tensor</span>([[[<span class="pl-c1">1.</span>, <span class="pl-c1">2.</span>, <span class="pl-c1">0.</span>, <span class="pl-c1">0.</span>],
         [<span class="pl-c1">1.</span>, <span class="pl-c1">2.</span>, <span class="pl-c1">3.</span>, <span class="pl-c1">0.</span>],
         [<span class="pl-c1">1.</span>, <span class="pl-c1">2.</span>, <span class="pl-c1">3.</span>, <span class="pl-c1">0.</span>]]])</pre></div>
<p dir="auto">This behavior also happens with <code class="notranslate">.add_</code> (an ATen operation):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; index_filled.add_(mask.long())
tensor([[[1, 2, 4, 7],
         [1, 2, 4, 7],
         [1, 2, 4, 7]]])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">index_filled</span>.<span class="pl-en">add_</span>(<span class="pl-s1">mask</span>.<span class="pl-en">long</span>())
<span class="pl-en">tensor</span>([[[<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">4</span>, <span class="pl-c1">7</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">4</span>, <span class="pl-c1">7</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">4</span>, <span class="pl-c1">7</span>]]])</pre></div>
<p dir="auto">Verse if we try with <code class="notranslate">.contiguous()</code>...</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; index_filled.contiguous().add_(mask.long())
tensor([[[1, 2, 4, 5],
         [1, 2, 3, 5],
         [1, 2, 3, 5]]])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">index_filled</span>.<span class="pl-en">contiguous</span>().<span class="pl-en">add_</span>(<span class="pl-s1">mask</span>.<span class="pl-en">long</span>())
<span class="pl-en">tensor</span>([[[<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">5</span>],
         [<span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">5</span>]]])</pre></div>
<p dir="auto">The "quick fix" to this is to call <code class="notranslate">.contiguous()</code> before <code class="notranslate">add_</code> or <code class="notranslate">masked_fill</code>, however I feel that this solution is non-obvious if the user is not aware of strides (and non-obvious even if the user is). Maybe a check for this type of case should exist? Thanks to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jekbradbury/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jekbradbury">@jekbradbury</a> for helping me figure out this issue.</p>
<h2 dir="auto">System Info</h2>
<ul dir="auto">
<li>Python version: 3.6</li>
<li>CUDA available: No</li>
<li>PyTorch or Caffe2: PyTorch</li>
<li>How you installed PyTorch (conda, pip, source): Conda</li>
<li>OS: Mac OSX 10.13.6</li>
<li>PyTorch version: 0.4.1</li>
<li>Python version: 3.6</li>
</ul>