Expanding a non-singleton dimension in PyTorch, but without copying data in memory?
Say that we have a tensor s of size [a,b,c] that is not necessarily contiguous, and b&gt;&gt;1.
I want to expand (but not copy) it in the second dimension for n times to get a tensor of size [a,nb,c].
The issue is that I cannot find a way to do this without explicitly copying data in memory.
The ways I know to do the operation, including
s.repeat_interleave(n,dim=1)
s.unsqueeze(-2).expand(-1,-1,n,-1).contiguous().view([a,-1,c])
s.unsqueeze(-2).expand(-1,-1,n,-1).reshape([a,-1,c])

will perform the copying step, and significantly slow things down.
Anyone know a solution?
Thanks in advance!

I don't think this is possible, and here is a minimal example to illustrate my point.
Consider a torch.Tensor [1, 2, 3], which has size (3,). If we want to expand it without performing a copy, we would create a new view of the tensor. Imagine for example that we want to create a view that contains twice the values of the original tensor, i.e. [1, 2, 3, 1, 2, 3] which would have size (2*3,). But it is not possible to define such a view only playing with the strides, here is why: to step from 1 to 2, or from 2 to 3, we would need the stride value to be 1. But to step from 3 to 1, we would need the stride value to be -2, but the system of strides as implemented cannot have different values for a given axis.
I am not 100% sure that it is not possible. Maybe there exists a very clever trick by using storage_offset parameter of torch.as_strided() function, or something else. Also, Perhaps this feature will be implemented in a future version, for example if you try to set a negative stride value, you have the error
&gt;&gt;&gt; torch.as_strided(torch.tensor([1, 2, 3]), size=(1,), stride=(-1,))
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
RuntimeError: as_strided: Negative strides are not supported at the moment, got strides: [-1]

indicating that this functionality could change in the future (here I used pytorch version 1.13.0).
One could argue that you could first expand without copying in a new dimension using torch.Tensor.expand(), and then flatten() the result, but this does not work, let me explain why. In the documentation of expand(), it is explained that this function returns a new view of the tensor (so this does not do any copy), and in the documentation of flatten(), it is explained that this function will try to return a view of the flattened tensor, and if not possible it will return a copy. Then let's try this method, and check the memory size of the tensor at each step using tensor.storage().nbytes():
&gt;&gt;&gt; a = torch.tensor([1, 2, 3])
&gt;&gt;&gt; print(a)
tensor([1, 2, 3])
&gt;&gt;&gt; print(a.storage().nbytes())
24
&gt;&gt;&gt; b = a.expand([2, 3])
&gt;&gt;&gt; print(b)
tensor([[1, 2, 3],
        [1, 2, 3]])
&gt;&gt;&gt; print(b.storage().nbytes())
24
&gt;&gt;&gt; c = b.flatten()
&gt;&gt;&gt; print(c)
tensor([1, 2, 3, 1, 2, 3])
&gt;&gt;&gt; print(c.storage().nbytes())
48

As we can see, flatten() does not seem capable to return a view of the flattened tensor since c takes twice memory compared to a. And if the pyTorch implementation of flatten() is not capable of doing that, this probably means that indeed it is not possible to do it.
