In-place operations with PyTorch
I was wondering how to deal with in-place operations in PyTorch. As I remember using in-place operation with autograd has always been problematic.

And actually I’m surprised that this code below works, even though I haven’t tested it I believe this code would have raised an error in version 0.3.1.

Basically I want do is set a certain position of a tensor vector to a certain value in a like:

my_tensor[i] = 42


Working example code:

# test parameter a
a = torch.rand((2), requires_grad=True)
print('a ', a)
b = torch.rand(2)

# calculation
c = a + b

# performing in-place operation
c[0] = 0
print('c ', c)
s = torch.sum(c)
print('s ', s)

# calling backward()
s.backward()

# optimizer step
optim = torch.optim.Adam(params=[a], lr=0.5)
optim.step()

# changed parameter a
print('changed a', a)


Output:

a  tensor([0.2441, 0.2589], requires_grad=True)
c  tensor([0.0000, 1.1511], grad_fn=&lt;CopySlices&gt;)
s  tensor(1.1511, grad_fn=&lt;SumBackward0&gt;)
changed a tensor([ 0.2441, -0.2411], requires_grad=True)


So obviously in version 0.4.1. this works just fine without warnings or errors.

Referring to this article in the documentation: autograd-mechanics


  Supporting in-place operations in autograd is a hard matter, and we
  discourage their use in most cases. Autograd’s aggressive buffer
  freeing and reuse makes it very efficient and there are very few
  occasions when in-place operations actually lower memory usage by any
  significant amount. Unless you’re operating under heavy memory
  pressure, you might never need to use them.


But even though it works, the use of in-place operations is discouraged in most cases.



So my questions are:


How much does the usage of in-place operations affect performance?
How do I get around using in-place operations in such cases where I want to set one element of a tensor to a certain value?


Thanks in advance!

I am not sure about how much in-place operation affect performance but I can address the second query. You can use a mask instead of in-place ops. 

a = torch.rand((2), requires_grad=True)
print('a ', a)
b = torch.rand(2)

# calculation
c = a + b

# performing in-place operation
mask = np.zeros(2)
mask[1] =1
mask = torch.tensor(mask)
c = c*mask
...

