Extend a tensor of sequences so that every sequence is prepended
I am having a tensor of 3 sequences where every sequence has length 2 and consists of vectors of size 2:
import torch
t = torch.Tensor([[[11,12],[21,22]], [[31,32],[41,42]], [[51,52],[61,62]]])
&gt;&gt;&gt; t
tensor([[[11., 12.],
         [21., 22.]],

        [[31., 32.],
         [41., 42.]],

        [[51., 52.],
         [61., 62.]]])

So t has the structure t[batch, sequencePos, dataPos]. How can I extend every sequence so that it is prepended by a new element [01, 02] (sequence have then length 3) so that I get:
tensor([[[01., 02.],
         [11., 12.],             
         [21., 22.]],

        [[01., 02.],
         [31., 32.]
         [41., 42.]],

        [[01., 02.],
         [51., 52.],
         [61., 62.]]])


You are looking to concatenate two tensors on axis=1, the first one being t:
 &gt;&gt;&gt; t
tensor([[[11., 12.],
         [21., 22.]],

        [[31., 32.],
         [41., 42.]],

        [[51., 52.],
         [61., 62.]]])

The second one is an arrangement:
&gt;&gt;&gt; arr = torch.arange(t.size(-1))
tensor([0, 1])

However, we first need to broadcast it to the correct shape using torch.reshape and torch.repeat:
&gt;&gt;&gt; arr = arr.reshape(1, 1, -1).repeat(len(t), 1, 1)
tensor([[[0, 1]],

        [[0, 1]],

        [[0, 1]]])

At this point, arr.shape is torch.Size([3, 1, 4]).
We are set for concatenating arr and t together, either with torch.cat:
&gt;&gt;&gt; torch.cat((arr, t), dim=1)

or more elegantly with torch.hstack:
&gt;&gt;&gt; torch.hstack((arr, t))
tensor([[[ 0.,  1.],
         [11., 12.],
         [21., 22.]],

        [[ 0.,  1.],
         [31., 32.],
         [41., 42.]],

        [[ 0.,  1.],
         [51., 52.],
         [61., 62.]]])

Notice how this implementation will work with any 3-dimensional input. In the following example t has three columns instead of two:
&gt;&gt;&gt; t
tensor([[[11., 12., 13.],
         [21., 22., 23.]],

        [[31., 32., 33.],
         [41., 42., 43.]],

        [[51., 52., 53.],
         [61., 62., 63.]]])

&gt;&gt;&gt; arr = torch.arange(t.size(-1)).reshape(1, 1, -1).repeat(len(t), 1, 1)
&gt;&gt;&gt; torch.cat((arr, t), dim=1)
tensor([[[ 0.,  1.,  2.],
         [11., 12., 13.],
         [21., 22., 23.]],

        [[ 0.,  1.,  2.],
         [31., 32., 33.],
         [41., 42., 43.]],

        [[ 0.,  1.,  2.],
         [51., 52., 53.],
         [61., 62., 63.]]])


This can even be generalized to n-dimensional tensors. You just have to take care of the reshape and repeat calls where the number of arguments depends on the number of dimensions.
&gt;&gt;&gt; ones = (1,)*(t.ndim-1)
&gt;&gt;&gt; arr = torch.arange(t.size(-1)).reshape(*ones, -1).repeat(len(t), *ones)
&gt;&gt;&gt; torch.cat((arr, t), dim=-2)

