Instruction: Channel first and channel last in convolution

Input:
I saw there are two types of data: channel first and last in the world of convolutional networks.
According to many websites, "channel-first" refers to NCHW format, while "channel-last" is equivalent to NHWC format. This is clear because in channel first format, C is positioned before H and W.
However, ARM seems to have defined "channel-first" as NHWC, as you can see in this paper.

P6: The two most common image data formats are Channel-Width-Height (CHW), i.e. channel last, and Height-Width-Channel (HWC), i.e. channel first. The dimension ordering is the same as that of the data stride. In
an HWC format, the data along the channel is stored with a stride of 1, data along the width is stored
with a stride of the channel count, and data along the height is stored with a stride of (channel count × image width).

This is also reasonable since "Channel first" sounds like MAC operation goes channel-wise like below:
for (N){
  for (H){
    for (W){
      for (C){
      }
    }
  }
}

So there is no fixed definition of channel-first or channel-last, isn't there?
Also, I'm not sure when you say NHWC or NCHW, what do you specifically mean? I guess the important thing is the combination of algorithms and the data arrangement in memory. If the data comes in in NHWC format, you need to design the algorithm like so.
And, since there are no fixed definitions of NHWC and NCHW, I don't think it makes any sense if you just say PyTorch is NCHW, channel-first or something without mentioning how the data arranges in memory.
Or when you hear NCHW, you can realize that the data arrangement in memory is like ch0[0,0], ch1[0, 0], ch2[0, 0], ch0[1, 0], ch1[1, 0], ch2[1, 0], ch0[2, 0], ...?
Can anyone help clarify my understanding of the data format?

Output:
I had originally overlooked the paper you linked where they clearly define the two opponents to how the terms are usually employed in the documentation and elsewhere. There are indeed two different ways to look at CHW and HWC...
TLDR; For end-users CHW is channel-first while HWC is channel last. In this case, we refer to the position of the channel dimension with regards to the other dimensions (H and W). Wether it is before CHW or after HWC is a matter of convention defined by the library used (eg. PyTorch vs. Tensorflow). In terms of memory allocation, it makes sense to call CHW channel last, it means that the channel axis' stride will be last: it will be unfolded last with regards to the other axes of the tensor.

I don't think it makes any sense if you just say PyTorch is NCHW, channel first or something without mentioning how the data arranges in memory.

For the end-user (as in end-developer), it does not matter how memory is allocated or arranged. The important part is to know how to use the API provided by PyTorch to manipulate torch.Tensors. When we say NCHW, we mean 'channel-first', i.e. tensors of shape (batch_size, channel, height, width). In all PyTorch documentation pages, you will find the exact shapes, inputs, and outputs tensors are required to have. It just happens they have chosen to stick with the NCHW convention for 2-dimensional channel tensors.
It makes sense to stick with one format let it be for the underlying implementation - where the memory arrangement does matter - or the end-user itself - who is used to working with a single format.
In TensorFlow for instance, channel is last, so the format used is NHWC.

To come back to how HWC (resp. CHW) was named channel-first (resp. channel-last) in the paper you linked. This has to do with the tensor stride: i.e. the layout of data in memory. Intuitively you can think that format HWC is channel-first because the channel dimension is the first axis to get unfolded.
If you look at this example:
>>> x = torch.rand(2,3,4) # last dimension is the channel axis
tensor([[[0.5567, 0.0276, 0.6491, 0.7933],
         [0.2876, 0.0361, 0.3883, 0.3201],
         [0.6742, 0.0305, 0.5719, 0.4683]],

        [[0.3385, 0.2082, 0.1675, 0.3429],
         [0.6146, 0.0533, 0.6147, 0.2216],
         [0.1855, 0.6107, 0.1716, 0.0071]]])

The underlying memory arangement is actually revealed when flattening the data (assuming the initial tensor's data is contiguous in memory):
>>> x.flatten()
tensor([0.5567, 0.0276, 0.6491, 0.7933, 0.2876, 0.0361, 0.3883, 0.3201, 0.6742,
        0.0305, 0.5719, 0.4683, 0.3385, 0.2082, 0.1675, 0.3429, 0.6146, 0.0533,
        0.6147, 0.2216, 0.1855, 0.6107, 0.1716, 0.0071])

Notice above how the data is laid out: going channel by channel 0.5567, 0.0276, 0.6491, 0.7933, then 0.2876, 0.0361, 0.3883, 0.3201, etc...
In the other format (i.e. CHW) it would have been laid out as 0.5567, 0.2876, 0.6742, then 0.3385, 0.6146, 0.1855, etc...
So it does make sense to call CHW channel-last (HWC as channel-first) when referring to how data is allocated in memory.


Instruction: Python code to convert 1D tensor to 2D tensor

Input:
I am trying to use Binary Cross Entropy Loss (BCE loss) for Simese network.
I have two inputs for BCE loss function:
output (input_dy) → tensor of size [4] , output of neural network
true_labels (y_true) → tensor of size [4], target (true value)

For BCE loss, the input parameters must be of the dimension:
output (input_dy) → [Batch_size, no. of classes]
true_labels (y_true) → [Batch_size]

The following diagram explains the query:enter image description here
I need a function in python using pytorch to convert the dy matrix to a 2D matrix with the output probabilities that sum to 1. [To note: dy should be iterated through length of it, as it is the output of the network for every input ]
Further a 2D array must be represented into one hot encoding, which will be true_labels (that will represent Binary classes with 0 & 1)
I need both output matrix and true_labels matrix for BCE Loss with following dimensions:
output dimension → [4, 2]
true_labels → [4]

Any help is most appreciated!
Thank you in advance.

Output:
It looks to me that your output is binary so you don't really need a 2D matrix for that task.
Also, I'm not quite sure that the BCE Loss (nor BCEWithLogits) requires tensors of different dimensions, they should both have shape (N, *) as far as I know.
Apart from that, for the sake of the question: if you have p(x), you can obtain the other column by simply computing 1 - p(x).
There are many ways to obtain that, a method could be:
# suppose we have a tensor/batch of probabilities
a = torch.tensor([0.4691, 0.9589, 0.7529, 0.9564])
# this gives a 2D matrix with two columns, (1 - p), p
b = torch.stack((1 - p, p), dim=-1)

And that's it!


Instruction: pytorch nn.Module inference

Input:
I am planning on learning Pytorch. However at this stage I would like to ask a question so that I can understand some code I am reading
When you have a class whose base class is nn.Module say
class My_model(nn.Module)

how are inferences supposed to be run there?
In the code I am reading it says
tasks_output, other = my_model(data)

Wouldn't that just be creating an object? (like calling the class constructor)
How, in pytorch, are inference supposed to be made?
(for reference I am talking when my_model is set to my_model.eval())
EDIT: My apologies. I made the mistake of declaring the class and object as one.. I corrected the code

Output:
You are confusion __init__ and __call__.
In your example my_model is a class, therefore calling
my_model_instance = my_model(arguments)

Invoke's my_model.__init__ with arguments. The result of this call is a new instance of my_model in the variable my_model_instance.
Once you instantiated the class my_model as the variable my_model_instance, you can evaluate the model on the training data:
tasks_output, other = my_model_instance(data)

"Calling" (i.e., putting parenthesis after the variable name) the instance of the model causes python to invoke the method __call__ of the class.
In the case of classes derived from nn.Modules this will invoke __call__ of nn.Module that does some pytorch stuff and eventually calls your implementation of forward method of my_class.
Please see this detailed thread on the difference between __init__ and __call__ in python in general.
It is often a convenient follow PEP8 Style Guide for Python Code:

Class names should normally use the CapWords convention.

Function names should be lowercase, with words separated by underscores as necessary to improve readability.
Variable names follow the same convention as function names.


Instruction: Pytorch transformations on GPU, is it worth on big input data?

Input:
I am running a UNet with PyTorch on medical imaging data with a bunch of transformations and augmentations in my preprocessing. However, after digging into the different preprocessing packages like Torchio and MONAI, I noticed that most of the functions, even when they take Tensors as IO, are running things on CPU.
The functions either straight up take numpy arrays as input or call .numpy() on the tensors.
The problem is that my data is composed of 3D images of dimension 91x109x91 that I resize in 96x128x96 so they are pretty big. Hence, running transformations and augmentations on CPU is pretty inefficient I think.
First, it makes my program CPU bound because it takes more time to transform my images than running them through the model (I timed it many times ). Secondly, I checked the GPU usage and it's oscillating between pretty much 0% and 100% at each batch so, it's clearly limited by the CPU. I would like to speed it up if it's possible.
My question is: Why are these packages not using the GPUs? They could at least have hybrid functions taking either a numpy array or a Tensor as input as a lot of numpy functions are available in Torch as well. Is there a good reason to stick to the CPU rather than speeding up the preprocessing by loading the images on GPU at the beginning of the preprocessing?
I translated a simple normalization function to work on GPU and compare the running time between the GPU and CPU version and even on a laptop (NVidia M2000M) the function was 3 to 4 times faster on GPU.
On an ML discord, someone mentioned that GPU-based functions might not give deterministic results and that's why it might not be a good idea but I don't know if that's actually the case.
My preprocessing includes resizing, intensity clamping, z-scoring, intensity rescaling, and then I have some augmentations like random histogram shift/elastic transform/affine transform/bias field.

Output:
A transformation will typically only be faster on the GPU than on the CPU if the implementation can make use of the parallelism offered by the GPU. Typically anything that operates element-wise, or row/column-wise can be made faster on GPU. This therefore concerns most image transformations.
The reason why some libraries don't implement things on GPU is that it requires additional work for each Tensor manipulation library you want to support (Pytorch, Tensorflow, MXNet, ...), and you still have to maintain another CPU implementation anyway. Since you're using PyTorch, checkout the torchvision package that implements many transformations for both GPU and CPU tensors.
For more complex transformations, like elastic deformation, I'm not sure if you can find a GPU version. If not, you might have to write one yourself, or drop this transformation, or pay the cost of copying back-and-forth between CPU and GPU during your data augmentation.
Another solution that some people prefer is to precompute a large set  of transformation on CPU as a separate step and to save the result in a file. The HDF file format is commonly used to save large datasets that can then be read very fast from disk. Since you will be saving a finite set of augmentation, be careful to generate several augmentations for each sample of your dataset to conserve a somewhat random behavior. This is not perfect, but it's a very pragmatic that will likely speed things up quite a bit if your CPU is holding your GPU back.
Regarding the determinism of the GPU, it's true that floating point operations are not by default guaranteed to be deterministic when run on GPU. This is because reordering some floating point operations can make them faster, but the reordering cannot guarantee that the result will be exactly the same (it will be close of course!). This can matter for reproducibility, if you use a seed in your code and get slightly different results. See the Pytorch Documentation to understand other sources of non-determinism.


Instruction: What is the "data.max" of a torch.Tensor?

Input:
I have been browsing the documentation of torch.Tensor, but I have not been able to find this (just similar things).
If a_tensor is a torch.Tensor, what is a_tensor.data.max? What type, etc.?
In particular, I am reading a_tensor.data.max(1)[1] and a_tensor.data.max(1)[1][i].cpu().numpy().

Output:
When accessing .data you are accessing the underlying data of the tensor. The returned object is a Torch.*Tensor as well, however, it won't be linked to any computational graph.
Take this example:
>>> x = torch.rand(4, requires_grad=True)
>>> y = x**2

>>> y
tensor([0.5272, 0.3162, 0.1374, 0.3004], grad_fn=<PowBackward0>)

While y.data is somewhat detached from graph (no grad_fn function), yet it is not a copy of y as y.detach() would return:
>>> y.data
tensor([0.5272, 0.3162, 0.1374, 0.3004]

Therefore, if you modify y.data's components you end modifying y itself:
>>> y.data[0] = 1

>>> y
tensor([1.0000, 0.3162, 0.1374, 0.3004], grad_fn=<PowBackward0>)

Notice how the grad_fn didn't change there. If you had done y[0] = 1, grad_fn would have been updated to <CopySlices>. This shows that modify your tensor's data through .data is not accounted for in terms of gradient, i.e., you won't be able to backpropagate these operations. It is required to work with y - not y.data - when planning to use Autograd.

So, to give an answer to your question: a_tensor.data is a torch.*Tensor, same type as a_tensor, and a_tensor.data.max is a function bound to that tensor.


Instruction: how to save in pytorch an ONNX model with training (autograd) operations?

Input:
In pytorch, is it possible to save an ONNX model to file including the backward operations?
If not, is there any other way in pytorch to save the forward and backward graph as text (json, pbtxt ...)?
Any help will be appreciated.

Output:
it's possible if you wrap the model with ORTModule -
https://github.com/microsoft/onnxruntime-training-examples
There's flag to enable onnx model saving, for example:
model._save_onnx = True
model._save_onnx_prefix = 'MNIST'
However, the onnx graph from fw will be further optimized before generating bw graph. Thus it's specific to ORT, but the training results should be mathematically the same. If you are looking for just fw+bw graph, the output onnx is still a good reference. The onnx could be opened using Netron util - https://github.com/lutzroeder/Netron


Instruction: torch.masked_scatter result did not meet expectations

Input:
my pytorch code:
import torch
x = torch.tensor([[0.3992, 0.2908, 0.9004, 0.4850, 0.6004],
            [0.5735, 0.9006, 0.6797, 0.4152, 0.1732]])
print(x.shape)
mask = torch.tensor([[False, False,  True, False,  True],
            [ True,  True,  True, False, False]])
print(mask.shape)
y = torch.tensor([[0., 0., 0., 0., 0.],
            [0., 0., 0., 0., 0.]])
print(y.shape)
y.masked_scatter_(mask, x)
print(y)

result is:
torch.Size([2, 5])
torch.Size([2, 5])
torch.Size([2, 5])
tensor([[0.0000, 0.0000, 0.3992, 0.0000, 0.2908],
        [0.9004, 0.4850, 0.6004, 0.0000, 0.0000]])

i think the result answer is:
tensor([[0.0000, 0.0000, 0.9004, 0.0000, 0.6004],
        [0.5375, 0.9006, 0.6797, 0.0000, 0.0000]])

my pytorch version is pytorch1.4

Output:
You are right, this is confusing and there is virtually no documentation.
However, the way scatter works (as you have discovered) is that the ith True in a row is given the ith value from the source. So not the value corresponding to the position of the True.
Luckily what you are trying to do can easily be achieved using the normal indexing notation:
>>> y[mask] = x[mask]
>>> y
tensor([[0.0000, 0.0000, 0.9004, 0.0000, 0.6004],
        [0.5735, 0.9006, 0.6797, 0.0000, 0.0000]])


Instruction: AzureML experiment pipeline not using CUDA with PyTorch

Input:
I am running an experiment pipeline to train my model with PyTorch and CUDA.
I created the environment as follow:
    env = Environment.from_conda_specification(model, join(model, 'conda_dependencies.yml'))
    env.docker.enabled = True
    env.environment_variables = {'MODEL_NAME': model, 'BRANCH': branch, 'COMMIT': commit}
    env.docker.base_image = DEFAULT_GPU_IMAGE

    run_config = RunConfiguration()
    run_config.environment = env
    run_config.docker = DockerConfiguration(use_docker=True)

And here is the training step:
train_step = PythonScriptStep(
      name='Model Train',
      source_directory=training_dir,
      compute_target=cluster,
      runconfig=run_config,
      script_name='train_aml.py',
      arguments=[
        '--model', model,
        '--model_output_dir', model_output_dir,
      ],
      inputs=[train_dataset.as_mount()],
      outputs=[model, model_output_dir]
    )

Even though I am using a Standard_NC12_Promo machine when I run my training script, the GPU is not picked up by PyTorch device = torch.device("cuda" if torch.cuda.is_available() else "cpu").
If I try running my script on the same machine but not in an experiment then the GPU is used.
Do you know any potential solutions to this?

Output:
Depending on pytorch version you might need a specific version of cuda. Try cuda 11.0.3 or cuda 11.1 from here
https://github.com/Azure/AzureML-Containers/tree/master/base/gpu
Regarding your code snippet, please move environment variables out of environment object to runconfiguration


Instruction: AttributeError: Can't pickle local object 'pre_datasets..' when implementing Pytorch framework

Input:
I was trying to implement a pytorch framework on CNN.
I'm sure the code is right because it's from a tutorial and it works when I ran it on Jupyter Notebook on GoogleDrive.
But when I tried to localize it as a .py file, it suggest an error:
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'
I know it's about inferencing objects outside a function, but what was the exact matter about this error?
And how should I fix it?

Here's the major part of the code.
def pre_datasets():
    TRAIN_TFM = transforms.Compose(
        [
            transforms.Resize(size=(128, 128)),
            # TODO
            transforms.ToTensor(),
        ]
    )
    train_set = DatasetFolder(
        root=CONFIG["train_set_path"],
        loader=lambda x: Image.open(x),
        extensions="jpg",
        transform=TRAIN_TFM,
    )
    train_loader = DataLoader(
        dataset=train_set,
        batch_size=CONFIG["batch_size"],
        shuffle=True,
        num_workers=CONFIG["num_workers"],
        pin_memory=True,
    )
    return train_loader

def train(train_loader):
    ...
    for epoch in range(CONFIG["num_epochs"]):
    ...
        for batch in train_loader: # error happened here
    ...

if __name__ == "__main__":
    train_loader = pre_datasets()
    train(train_loader)

Here's the error message:
Traceback (most recent call last):
  File "HW03_byCRZ.py", line 197, in <module>
    train(train_loader, valid_loader)
  File "HW03_byCRZ.py", line 157, in train
    for batch in train_loader:
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 355, in __iter__
    return self._get_iterator()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 301, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 914, in __init__
    w.start()
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ceezous/opt/anaconda3/envs/pytorch_env/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'pre_datasets.<locals>.<lambda>'

Output:
I had a similar issue and I used dill like this:
import dill as pickle

and it worked out of the box!


Instruction: Convert Flatten layer from PyTorch to Tensorflow - Equivalent for start_dim and end_dim

Input:
What is the equivalent of the options start_dim and end_dim of the flatten PyTorch layers, in Tensorflow?
With Tensorlfow we only have data_format and it is not customizable.

Output: I don't think there is an identical implementation in tf. However, you can always use tf.reshape and add the shape yourself with a simple function which takes as arguments input, start_dim and end_dim and outputs the corresponding output shape that torch.flatten would give you.


Instruction: Split neural network, load only needed part onto the GPU

Input:
I have a very, very big neural network and an Google Colab Pro Subscripting receiving my 16GB of GPU RAM. Unfortunately, this is not enough. My idea now is, to split the model (Unet) into the encoder and decoder part separately, and proceed like the following:

Load encoder to the GPU
Process the data through the encoder
Load encoder to the cpu, decoder to the GPU
Process the encoder output through the decoder
Load the decoder to the cpu aaaand repeat.

Is this in general possible? I coded an example but it wont work:
def train(epoch, loader, loss_fn, optimizer, scaler, model1, model2):
    model1.train()
    model2.train()

    loop = prog(loader)

    running_loss = []

    for batch_index, (data, target) in enumerate(loop):
        optimizer.zero_grad(set_to_none=True)

        model1 = model1.to(DEVICE)

        data, skip_connections = model1(data.to(DEVICE))
        model1 = model1.cpu()

        model2 = model2.to(DEVICE)
        data = model2(data, skip_connections)

        model2 = model2.cpu()

        target = target.to(DEVICE)

        with torch.cuda.amp.autocast():
            loss = loss_fn(data, target)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        model1 = model1.to(DEVICE)

        loss_value = loss.item()

        loop.set_postfix(info="Epoch {}, train, loss={:.5f}".format(epoch, loss_value))
        running_loss.append(loss_value)

    return s.mean(running_loss)

For the setup / initialization I got the following:
DEVICE = "cuda"

model1 = UNET_FIRST_HALVE(in_channels=4).to(DEVICE)
model2 = UNET_SECOND_HALVE(out_channels=NUM_CLASSES).cpu()

for epoch in range(epochs_done + 1, num_epochs + 1):
    training_loss = train(..., model1, model2)
    .
    .
    .

I get the following error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Surely I understand the error, but I am sure that I push and pull everything at the right time onto and from the GPU... Or maybe there is a better way of splitting a model?

Output:
There are a couple of wrong things with this:

Your autocast block should include the forward on your model
You don't need to go back and forth from CPU to GPU and back, it's already a bottleneck with tensors, imagine with models.
Optional: what the heck is your UNet made of if you can't make it fit on a 16GB device?

I'd first try with standard solutions: reduce the batch size, include AMP (that I see you already included) or even DeepSpeed, which already does some CPU allocation, depending on the memory optimization level. Have a look at it, just in case, this may already solve your problems.
Answering to your question, the only feasible approach I see is to keep a model on cuda, the other on CPU and eventually move inputs/outputs.
def train(epoch, loader, loss_fn, optimizer, scaler, model1, model2):
    model1.to(device1)
    model2.to(device2)
    model1.train()
    model2.train()

    loop = prog(loader)

    running_loss = []

    for batch_index, (data, target) in enumerate(loop):
        optimizer.zero_grad(set_to_none=True)
        # now data nd skip are on device1
        data, skip_connections = model1(data.to(device1))
        # we need to move to device2
        data = data.to(device2)
        skip = skio_connections.to(device2)
        data = model2(data, skip)
        # everything should be on 2
        # move the target to the same device
        target = target.to(device2)
        loss = loss_fn(data, target)
        # backprop
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        loss_value = loss.item()
        loop.set_postfix(info="Epoch {}, train, loss={:.5f}".format(epoch, loss_value))
        running_loss.append(loss_value)

    return s.mean(running_loss)

The main issue I see with this, is that your first model (residing on CPU) should not require the backpropagation part, otherwise it will probably break again. I doubt that PyTorch can build the backward graph between devices.


Instruction: How can I reshape (A,) and (B, C, D) shapes to the single (A, B, C, D)?

Input:
This is my code;
for img_loc in list(self.train_data)[idx]:
   images_set.append(self.load_ucf_image(img_loc))
print(images_set)

And, this is its output
[tensor([[[ 1.7865,  1.8893,  1.9578,  ..., -1.3815, -0.4054,  0.2967],
             [ 1.7694,  1.8722,  1.9578,  ..., -0.6452, -0.4054,  0.1254],
             [ 1.7523,  1.8722,  1.9749,  ..., -0.5082, -0.6623, -0.3541],
             ...,
             [-1.9809, -1.6384, -1.2617,  ..., -1.7754, -1.0562, -0.9020],
             [-2.0494, -1.8268, -1.5014,  ..., -1.1075, -1.4672, -1.7069],
             [-1.9980, -1.8953, -1.4672,  ..., -1.7412, -1.7069, -1.2445]],

            [[ 1.0805,  1.2556,  1.3606,  ..., -1.1954, -0.1975,  0.4678],
             [ 1.1155,  1.2731,  1.3957,  ..., -0.4426, -0.1975,  0.3102],
             [ 1.1155,  1.2731,  1.3957,  ..., -0.2850, -0.4426, -0.1800],
             ...,
             [-1.9657, -1.7556, -1.4580,  ..., -1.8957, -1.2304, -1.1253],
             [-2.0007, -1.9132, -1.6856,  ..., -1.1954, -1.6331, -1.9482],
             [-1.9482, -1.9482, -1.5980,  ..., -1.8431, -1.8782, -1.4230]],

            [[ 1.4548,  1.6640,  1.8034,  ..., -0.4624,  0.5311,  1.3502],
             [ 1.5071,  1.7163,  1.8557,  ...,  0.2871,  0.5311,  1.1411],
             [ 1.5245,  1.7511,  1.9080,  ...,  0.3916,  0.2348,  0.6182],
             ...,
             [-1.4907, -1.3164, -1.0550,  ..., -1.5953, -0.9504, -0.8284],
             [-1.5430, -1.4907, -1.2816,  ..., -0.8633, -1.3164, -1.6476],
             [-1.4907, -1.5256, -1.2119,  ..., -1.5081, -1.5256, -1.1421]]]), tensor([[[-0.4054, -0.3883, -0.4739,  ...,  2.1119,  2.0948,  2.0777],
             [-0.3712, -0.2856, -0.4397,  ...,  2.0948,  2.0777,  2.0434],
             [-0.3541, -0.2513,  0.1083,  ...,  2.0777,  2.0434,  2.0263],
             ...,
             [-1.0219, -1.2617, -1.2103,  ..., -0.0629, -0.1999, -0.2171],
             [-1.0048, -1.3302, -1.3302,  ..., -0.0801, -0.1828, -0.4226],
             [-1.3302, -0.9705, -1.0562,  ..., -0.0458, -0.1486, -0.7993]],

            [[-0.2325, -0.2150, -0.2675,  ...,  1.5707,  1.5007,  1.4132],
             [-0.1975, -0.1099, -0.2500,  ...,  1.6057,  1.5357,  1.4482],
             [-0.2150, -0.1099,  0.2752,  ...,  1.6057,  1.5532,  1.4482],
             ...,
             [-1.1779, -1.4230, -1.3704,  ..., -0.3901, -0.4776, -0.4776],
             [-1.1779, -1.5630, -1.5280,  ..., -0.4076, -0.4601, -0.6527],
             [-1.5630, -1.2304, -1.2829,  ..., -0.3200, -0.3901, -1.0028]],

            [[ 0.7751,  0.7925,  0.6531,  ...,  2.0300,  1.9428,  1.8731],
             [ 0.8099,  0.8971,  0.6356,  ...,  2.0823,  1.9951,  1.9254],
             [ 0.8099,  0.8797,  1.1759,  ...,  2.1171,  2.0300,  1.9603],
             ...,
             [-0.8981, -1.1073, -1.0201,  ..., -0.0092, -0.0790, -0.0441],
             [-0.8981, -1.2293, -1.1944,  ..., -0.0267, -0.0615, -0.2184],
             [-1.2990, -0.9156, -0.9853,  ...,  0.0431, -0.0092, -0.5670]]]), tensor([[[ 1.9749,  1.8208,  1.9749,  ..., -0.8678, -0.8335, -0.8678],
             [ 2.0777,  1.8893,  2.0777,  ..., -0.8507, -0.8164, -0.8507],
             [ 2.1290,  2.0777,  2.1290,  ..., -0.7993, -0.7822, -0.7993],
             ...,
             [-1.2617, -1.3644, -1.4672,  ..., -1.9980, -2.0665, -2.1008],
             [-1.1760, -1.2617, -1.3815,  ..., -2.0665, -2.0837, -2.0837],
             [-1.0904, -1.1760, -1.3302,  ..., -2.1008, -2.0323, -2.0837]],

            [[ 1.3256,  1.1856,  1.3431,  ..., -0.7402, -0.7227, -0.7577],
             [ 1.4482,  1.3081,  1.5007,  ..., -0.7227, -0.7227, -0.7402],
             [ 1.5532,  1.5357,  1.6057,  ..., -0.6702, -0.6877, -0.6877],
             ...,
             [-1.4230, -1.5630, -1.7031,  ..., -2.0007, -2.0357, -2.0357],
             [-1.3354, -1.4405, -1.6155,  ..., -2.0357, -2.0357, -2.0357],
             [-1.2479, -1.3529, -1.5630,  ..., -2.0357, -1.9832, -2.0182]],

            [[ 1.7163,  1.5768,  1.7511,  ...,  0.0082, -0.0092, -0.0441],
             [ 1.8383,  1.6814,  1.8905,  ...,  0.0431,  0.0431, -0.0267],
             [ 1.9254,  1.8905,  1.9777,  ...,  0.0953,  0.0779, -0.0092],
             ...,
             [-1.1073, -1.2293, -1.4036,  ..., -1.6650, -1.6824, -1.6824],
             [-1.0550, -1.1596, -1.3164,  ..., -1.7696, -1.7173, -1.6999],
             [-0.9678, -1.0724, -1.2641,  ..., -1.7696, -1.6650, -1.6999]]]), tensor([[[-0.6794, -0.6452, -0.6452,  ...,  2.1804,  2.1290,  2.0777],
             [-0.6794, -0.6452, -0.5596,  ...,  2.1633,  2.1804,  2.0948],
             [-0.6452, -0.5767, -0.4739,  ...,  2.1462,  2.0948,  2.0434],
             ...,
             [-1.9980, -2.0494, -2.0494,  ..., -1.4158, -1.3130, -1.3130],
             [-2.0152, -2.0665, -1.9980,  ..., -1.3644, -1.2445, -1.1760],
             [-2.0152, -2.0494, -2.0323,  ..., -1.3473, -1.2445, -1.0904]],

            [[-0.6176, -0.5826, -0.5651,  ...,  1.6933,  1.6057,  1.5182],
             [-0.6176, -0.5476, -0.4776,  ...,  1.7633,  1.7108,  1.5707],
             [-0.5826, -0.4776, -0.3901,  ...,  1.7983,  1.6933,  1.6057],
             ...,
             [-2.0007, -2.0357, -2.0357,  ..., -1.7556, -1.5805, -1.4930],
             [-2.0182, -2.0357, -2.0182,  ..., -1.7031, -1.5455, -1.4055],
             [-1.9657, -2.0357, -2.0182,  ..., -1.6856, -1.5455, -1.3179]],

            [[ 0.0779,  0.1825,  0.3045,  ...,  2.0997,  1.9777,  1.9080],
             [ 0.1128,  0.2348,  0.4265,  ...,  2.1346,  2.0648,  1.9428],
             [ 0.1476,  0.3045,  0.5136,  ...,  2.1520,  2.0648,  1.9428],
             ...,
             [-1.6650, -1.7522, -1.7696,  ..., -1.4559, -1.2641, -1.2119],
             [-1.7173, -1.8044, -1.7522,  ..., -1.4036, -1.2119, -1.1073],
             [-1.7173, -1.7870, -1.7696,  ..., -1.3861, -1.2119, -1.0201]]]), tensor([[[-0.3712, -0.4054, -0.3369,  ...,  2.0092,  1.9407,  1.8379],
             [-0.3027, -0.2684, -0.3712,  ...,  2.0263,  1.9749,  1.8550],
             [-0.1486, -0.3198, -0.3712,  ...,  2.0092,  1.9578,  1.9064],
             ...,
             [-1.9638, -2.0494, -2.0494,  ..., -1.9638, -2.0665, -2.0152],
             [-1.9467, -2.0323, -1.9467,  ..., -1.9467, -2.0837, -2.0152],
             [-1.1418, -1.9638, -2.0323,  ..., -2.0323, -2.0323, -1.9980]],

            [[-0.1975, -0.2325, -0.1975,  ...,  1.2906,  1.1856,  1.0280],
             [-0.1275, -0.0924, -0.2325,  ...,  1.3256,  1.2556,  1.1331],
             [-0.0049, -0.1800, -0.2325,  ...,  1.3782,  1.3081,  1.2031],
             ...,
             [-2.0357, -2.0357, -2.0357,  ..., -1.8957, -1.9482, -1.8782],
             [-2.0357, -2.0357, -1.9832,  ..., -1.8782, -1.9482, -1.8256],
             [-1.2129, -2.0007, -2.0357,  ..., -1.9132, -1.8431, -1.7906]],

            [[ 0.7751,  0.7402,  0.7925,  ...,  1.6291,  1.5420,  1.4025],
             [ 0.8448,  0.9145,  0.7576,  ...,  1.6988,  1.6291,  1.4722],
             [ 0.9842,  0.8448,  0.7576,  ...,  1.7685,  1.6988,  1.5768],
             ...,
             [-1.7173, -1.7696, -1.7696,  ..., -1.4210, -1.4559, -1.3861],
             [-1.6476, -1.6999, -1.6127,  ..., -1.4036, -1.4559, -1.3164],
             [-0.8284, -1.6302, -1.6476,  ..., -1.4210, -1.3339, -1.2816]]]), tensor([[[-0.3198, -0.3883, -0.4226,  ...,  1.8893,  1.7352,  1.7352],
             [-0.2684, -0.3369, -0.3541,  ...,  1.8893,  1.7523,  1.7523],
             [-0.1999, -0.2342, -0.3712,  ...,  1.9235,  1.8037,  1.8037],
             ...,
             [-2.0323, -2.0494, -2.1008,  ..., -1.8268, -1.8268, -1.7754],
             [-1.9809, -2.0323, -1.9980,  ..., -1.8097, -1.7583, -1.6727],
             [-1.9638, -2.0494, -2.0665,  ..., -1.7925, -1.7583, -1.6898]],

            [[-0.1625, -0.2325, -0.2675,  ...,  1.0980,  0.8704,  0.8704],
             [-0.1099, -0.1975, -0.2150,  ...,  1.1506,  0.9405,  0.9405],
             [-0.0574, -0.0924, -0.2325,  ...,  1.2206,  1.0630,  1.0105],
             ...,
             [-2.0357, -2.0007, -2.0357,  ..., -1.7906, -1.7206, -1.6331],
             [-2.0007, -2.0182, -1.9832,  ..., -1.7381, -1.6155, -1.4930],
             [-2.0357, -2.0357, -2.0357,  ..., -1.6681, -1.5805, -1.5105]],

            [[ 0.7751,  0.7054,  0.6705,  ...,  1.3677,  1.1237,  1.1237],
             [ 0.8274,  0.7925,  0.7751,  ...,  1.4025,  1.2108,  1.1759],
             [ 0.9319,  0.8971,  0.7576,  ...,  1.4897,  1.3154,  1.2805],
             ...,
             [-1.6999, -1.7173, -1.7696,  ..., -1.2641, -1.1770, -1.1073],
             [-1.6999, -1.7696, -1.7347,  ..., -1.2293, -1.0898, -0.9330],
             [-1.7173, -1.7696, -1.7696,  ..., -1.1770, -1.0201, -0.9504]]]), tensor([[[ 2.0605,  2.0434,  2.0263,  ..., -0.3198, -0.3027, -0.2856],
             [ 2.0434,  2.0263,  2.0263,  ..., -0.2171, -0.2171, -0.2171],
             [ 2.0777,  2.0263,  2.0092,  ..., -0.1999, -0.1828, -0.1999],
             ...,
             [ 0.3652,  0.4337,  0.3823,  ..., -2.0323, -2.0152, -1.9809],
             [ 0.3823,  0.4679,  0.4337,  ..., -2.0152, -2.0323, -2.0494],
             [ 0.3823,  0.3652,  0.4337,  ..., -1.9638, -2.0323, -2.0665]],

            [[ 1.6232,  1.6758,  1.7283,  ..., -0.1450, -0.1800, -0.1625],
             [ 1.6758,  1.7283,  1.8333,  ..., -0.0574, -0.0924, -0.0924],
             [ 1.7283,  1.7983,  1.8859,  ..., -0.0399, -0.0574, -0.0749],
             ...,
             [ 0.1352,  0.2052,  0.1352,  ..., -2.0357, -2.0357, -2.0007],
             [ 0.1877,  0.2402,  0.1702,  ..., -2.0357, -2.0357, -2.0357],
             [ 0.2227,  0.1352,  0.1702,  ..., -1.9657, -2.0357, -2.0357]],

            [[ 1.9603,  2.0300,  2.0648,  ...,  0.6879,  0.6356,  0.6531],
             [ 2.0300,  2.0997,  2.1694,  ...,  0.8448,  0.7576,  0.7576],
             [ 2.1171,  2.1520,  2.2043,  ...,  0.8971,  0.8274,  0.7751],
             ...,
             [ 0.4788,  0.5136,  0.4091,  ..., -1.7870, -1.8044, -1.7347],
             [ 0.5136,  0.5834,  0.4962,  ..., -1.8044, -1.8044, -1.8044],
             [ 0.5485,  0.4788,  0.4962,  ..., -1.7870, -1.8044, -1.8044]]]), tensor([[[-0.3369, -0.3198, -0.3541,  ...,  2.0605,  2.0777,  2.0948],
             [-0.2684, -0.2342, -0.2513,  ...,  2.0605,  2.0948,  2.0777],
             [-0.2342, -0.1657, -0.1828,  ...,  2.0434,  2.0777,  2.0777],
             ...,
             [-2.0494, -1.9980, -1.9124,  ...,  0.5193,  0.5536,  0.4851],
             [-1.9638, -1.9980, -1.9980,  ...,  0.5193,  0.5364,  0.4508],
             [-2.0494, -2.0494, -1.9638,  ...,  0.5193,  0.5022,  0.4166]],

            [[-0.1975, -0.1800, -0.1800,  ...,  1.6933,  1.6758,  1.6232],
             [-0.1450, -0.1099, -0.0924,  ...,  1.7633,  1.7458,  1.6758],
             [-0.1099, -0.0399, -0.0224,  ...,  1.8158,  1.8158,  1.7458],
             ...,
             [-2.0357, -2.0357, -1.9832,  ...,  0.2227,  0.2402,  0.2227],
             [-1.9832, -2.0182, -2.0182,  ...,  0.2577,  0.2752,  0.2227],
             [-2.0357, -2.0357, -1.9832,  ...,  0.2577,  0.2752,  0.2052]],

            [[ 0.5659,  0.5834,  0.6531,  ...,  2.0125,  2.0125,  1.9603],
             [ 0.6705,  0.7054,  0.7925,  ...,  2.0997,  2.0997,  2.0125],
             [ 0.7402,  0.8099,  0.8797,  ...,  2.1346,  2.1346,  2.0474],
             ...,
             [-1.7696, -1.7870, -1.7347,  ...,  0.5485,  0.6182,  0.6182],
             [-1.7173, -1.7522, -1.7522,  ...,  0.5834,  0.6356,  0.6008],
             [-1.8044, -1.8044, -1.7173,  ...,  0.5834,  0.6182,  0.5834]]]), tensor([[[ 2.0777,  2.0605,  2.0777,  ...,  0.1426,  0.1083,  0.0569],
             [ 2.1119,  2.0777,  2.0948,  ...,  0.1939,  0.1597,  0.0569],
             [ 2.0948,  2.0777,  2.0948,  ...,  0.1768,  0.1426,  0.0056],
             ...,
             [ 0.4679,  0.6906,  0.7248,  ..., -0.5253, -0.5082, -0.5938],
             [ 0.0912,  0.6049,  0.7248,  ..., -0.6281, -0.9877, -0.5082],
             [-0.5253,  0.5707,  0.6734,  ..., -0.8678, -0.6452, -0.2342]],

            [[ 1.9559,  2.0084,  2.0784,  ...,  0.2227,  0.1877,  0.1352],
             [ 2.0259,  2.0609,  2.0959,  ...,  0.2227,  0.1877,  0.0826],
             [ 2.0084,  2.0609,  2.0959,  ...,  0.2227,  0.1702,  0.0301],
             ...,
             [ 0.2052,  0.3627,  0.3452,  ..., -0.7927, -0.8277, -0.9153],
             [-0.1450,  0.2752,  0.3452,  ..., -0.9678, -1.3880, -0.9153],
             [-0.7752,  0.2402,  0.3102,  ..., -1.2829, -1.0903, -0.7052]],

            [[ 2.3088,  2.3437,  2.4308,  ...,  1.1934,  1.1585,  1.1062],
             [ 2.3611,  2.3786,  2.4483,  ...,  1.2108,  1.1759,  1.0714],
             [ 2.3437,  2.3786,  2.4483,  ...,  1.1585,  1.1585,  1.0191],
             ...,
             [ 0.6008,  0.7402,  0.7054,  ..., -0.5147, -0.5495, -0.6367],
             [ 0.2348,  0.6182,  0.7054,  ..., -0.6890, -1.1247, -0.6541],
             [-0.3927,  0.5834,  0.6182,  ..., -1.0201, -0.8458, -0.4624]]]), tensor([[[ 2.1119,  2.1633,  2.0948,  ..., -0.1143, -0.1486, -0.1657],
             [ 2.0777,  2.0605,  2.1119,  ..., -0.0458, -0.0629, -0.0972],
             [ 2.1462,  2.1633,  2.0777,  ...,  0.0569,  0.0569,  0.0056],
             ...,
             [ 0.3652,  0.4679,  0.6563,  ..., -2.0323, -1.9638, -1.9980],
             [ 0.1083,  0.4679,  0.6734,  ..., -1.8610, -1.8953, -1.9638],
             [-0.4911,  0.4337,  0.6392,  ..., -0.6109, -1.8097, -1.8953]],

            [[ 1.5007,  1.6408,  1.6933,  ..., -0.1275, -0.1450, -0.1625],
             [ 1.5357,  1.6232,  1.7633,  ..., -0.0399, -0.0224, -0.0574],
             [ 1.6758,  1.7983,  1.8508,  ...,  0.0651,  0.0826,  0.0301],
             ...,
             [ 0.0826,  0.1176,  0.2577,  ..., -2.0357, -1.9832, -1.9832],
             [-0.1450,  0.1527,  0.2927,  ..., -1.9307, -1.9132, -1.9832],
             [-0.7227,  0.1702,  0.3102,  ..., -0.7402, -1.9307, -2.0182]],

            [[ 1.8383,  2.0125,  2.0648,  ...,  0.8099,  0.7925,  0.7751],
             [ 1.8905,  1.9603,  2.1171,  ...,  0.9145,  0.9145,  0.8797],
             [ 2.0125,  2.1520,  2.2043,  ...,  1.0191,  1.0714,  1.0191],
             ...,
             [ 0.5311,  0.5485,  0.6182,  ..., -1.8044, -1.7522, -1.7696],
             [ 0.2871,  0.5311,  0.6531,  ..., -1.6824, -1.6824, -1.7522],
             [-0.3055,  0.5311,  0.6531,  ..., -0.5147, -1.6999, -1.7870]]]), tensor([[[ 1.3755,  1.6153,  1.7694,  ...,  0.0912,  0.2282,  0.1083],
             [ 1.4269,  1.6495,  1.7694,  ...,  0.1768, -0.0801, -0.7650],
             [ 1.3755,  1.6324,  1.8037,  ...,  0.0741, -0.5253, -0.0116],
             ...,
             [-1.5528, -1.5699, -1.8097,  ..., -0.8849, -0.6794, -0.6965],
             [-1.6042, -1.5870, -1.8097,  ..., -0.7479, -0.9534, -1.3302],
             [-1.6898, -1.6727, -1.7412,  ..., -1.6384, -1.3130, -0.7822]],

            [[ 0.6429,  0.9230,  1.0980,  ...,  0.2052,  0.3277,  0.2052],
             [ 0.6954,  0.9405,  1.1331,  ...,  0.2927,  0.0126, -0.6877],
             [ 0.6604,  0.9580,  1.1681,  ...,  0.1702, -0.4951,  0.0476],
             ...,
             [-1.4230, -1.3880, -1.6155,  ..., -0.7927, -0.6527, -0.7752],
             [-1.4230, -1.3704, -1.5805,  ..., -0.6352, -0.9678, -1.4230],
             [-1.5105, -1.4405, -1.5105,  ..., -1.6681, -1.4230, -0.9503]],

            [[ 0.7925,  1.0539,  1.2805,  ...,  1.1585,  1.2457,  1.1062],
             [ 0.8971,  1.1411,  1.3502,  ...,  1.2457,  0.9319,  0.2173],
             [ 0.8971,  1.1759,  1.4200,  ...,  1.0888,  0.4265,  0.9145],
             ...,
             [-0.8284, -0.8284, -1.1421,  ..., -0.4798, -0.3230, -0.4101],
             [-0.8633, -0.8633, -1.1247,  ..., -0.3753, -0.6193, -1.0898],
             [-0.9504, -0.9853, -1.0898,  ..., -1.3164, -1.0201, -0.5495]]]), tensor([[[ 0.0056,  0.0912,  0.1254,  ...,  2.1633,  2.1119,  2.1119],
             [ 0.0569,  0.1597,  0.1254,  ...,  2.1633,  2.1119,  2.1462],
             [ 0.0227,  0.1426,  0.1597,  ...,  2.1804,  2.1462,  2.0948],
             ...,
             [-0.5082, -0.9705, -0.6109,  ...,  0.6049, -0.1999, -1.0219],
             [-0.2171, -0.6452, -0.7650,  ...,  0.6049, -0.6281, -0.9877],
             [-1.1075,  0.0227, -0.6281,  ...,  0.4337, -0.8507, -1.1247]],

            [[ 0.1001,  0.2052,  0.2402,  ...,  1.8859,  1.7458,  1.5882],
             [ 0.1527,  0.2402,  0.2402,  ...,  1.8859,  1.7458,  1.6232],
             [ 0.1176,  0.2227,  0.2577,  ...,  1.9034,  1.7633,  1.6232],
             ...,
             [-0.8978, -1.3704, -0.9503,  ...,  0.3627, -0.3725, -1.0903],
             [-0.6176, -1.0553, -1.1604,  ...,  0.3627, -0.7752, -1.0028],
             [-1.5630, -0.3725, -1.0203,  ...,  0.2227, -1.0028, -1.1429]],

            [[ 1.0191,  1.1585,  1.1934,  ...,  2.2566,  2.0648,  1.9428],
             [ 1.0714,  1.2108,  1.1585,  ...,  2.2914,  2.0997,  1.9777],
             [ 1.0365,  1.1934,  1.1759,  ...,  2.3088,  2.1694,  1.9777],
             ...,
             [-0.6367, -1.1073, -0.6715,  ...,  0.6356, -0.0441, -0.7238],
             [-0.3927, -0.8284, -0.8981,  ...,  0.6356, -0.4973, -0.6541],
             [-1.3164, -0.1487, -0.7587,  ...,  0.4614, -0.7238, -0.7936]]]), tensor([[[ 2.0777,  2.1119,  2.1119,  ...,  0.0569, -0.0629, -0.2171],
             [ 2.0777,  2.1290,  2.1119,  ...,  0.0056, -0.1143, -0.2513],
             [ 2.0777,  2.0948,  2.1290,  ...,  0.0056, -0.1143, -0.1999],
             ...,
             [ 0.4851,  0.5707,  0.5707,  ..., -0.2171, -0.8849, -0.9192],
             [ 0.3481,  0.5707,  0.5536,  ..., -1.1075, -0.7650, -0.3883],
             [-0.1657,  0.5193,  0.5193,  ..., -0.8678, -0.3712, -1.1760]],

            [[ 1.6408,  1.8158,  1.9384,  ...,  0.1527, -0.0224, -0.1625],
             [ 1.6758,  1.8508,  1.9909,  ...,  0.1001, -0.0574, -0.1975],
             [ 1.6758,  1.8683,  2.0084,  ...,  0.0476, -0.0574, -0.1800],
             ...,
             [ 0.2577,  0.2752,  0.2577,  ..., -0.6176, -1.3004, -1.3354],
             [ 0.1352,  0.2752,  0.2402,  ..., -1.5455, -1.1779, -0.7927],
             [-0.3725,  0.2227,  0.2052,  ..., -1.2829, -0.7752, -1.5280]],

            [[ 1.9603,  2.1520,  2.3088,  ...,  1.0714,  0.8971,  0.6705],
             [ 2.0125,  2.2217,  2.3437,  ...,  1.0191,  0.8099,  0.6008],
             [ 2.0125,  2.2217,  2.3960,  ...,  0.9668,  0.8099,  0.6356],
             ...,
             [ 0.5659,  0.5659,  0.5311,  ..., -0.3927, -1.0724, -1.0724],
             [ 0.3916,  0.5485,  0.5136,  ..., -1.3513, -0.9504, -0.5321],
             [-0.1138,  0.4962,  0.4788,  ..., -1.0550, -0.5147, -1.2467]]]), tensor([[[ 0.0398, -0.8678, -0.1486,  ...,  1.6153,  1.0502,  0.7077],
             [ 0.0741, -0.1486, -0.7822,  ...,  1.6495,  1.1358,  0.7933],
             [-0.2513, -0.0972, -0.6965,  ...,  1.6495,  1.1872,  0.7933],
             ...,
             [-0.6452, -1.3815, -0.9192,  ..., -1.9124, -1.6727, -1.7583],
             [-1.1932, -0.7822, -1.3644,  ..., -1.8782, -1.6727, -1.8097],
             [-1.5699, -1.4158, -0.6452,  ..., -1.7925, -1.7069, -1.8268]],

            [[ 0.0826, -0.7927, -0.0574,  ...,  0.8179,  0.2752, -0.0399],
             [ 0.1176, -0.1099, -0.7052,  ...,  0.8529,  0.3277,  0.0476],
             [-0.2325, -0.0399, -0.6176,  ...,  0.8529,  0.3803,  0.0476],
             ...,
             [-0.8627, -1.5105, -0.9678,  ..., -1.7206, -1.5280, -1.6155],
             [-1.4405, -0.9503, -1.4755,  ..., -1.6506, -1.4755, -1.6155],
             [-1.8256, -1.5980, -0.7402,  ..., -1.5280, -1.5105, -1.6331]],

            [[ 1.0191,  0.1128,  0.8622,  ...,  0.9842,  0.3916,  0.1302],
             [ 1.0365,  0.8099,  0.2173,  ...,  1.0191,  0.4614,  0.1999],
             [ 0.6531,  0.8274,  0.2871,  ...,  1.0191,  0.5136,  0.1999],
             ...,
             [-0.5321, -1.1596, -0.6541,  ..., -1.2467, -1.0376, -1.1247],
             [-1.0550, -0.5495, -1.0724,  ..., -1.1944, -0.9678, -1.1421],
             [-1.4384, -1.1944, -0.3055,  ..., -1.0898, -1.0027, -1.1247]]]), tensor([[[-0.3369, -0.2684, -0.1486,  ...,  2.0434,  2.1290,  2.0605],
             [-0.3883, -0.3027, -0.1657,  ...,  2.0605,  2.1462,  2.0605],
             [-0.3883, -0.2684, -0.1486,  ...,  2.0605,  2.1462,  2.0263],
             ...,
             [-1.1075, -1.0219, -0.2684,  ...,  0.4166,  0.3652,  0.2111],
             [-0.5596, -0.9020, -1.1760,  ...,  0.3481,  0.3823,  0.0227],
             [-1.3815, -0.5938, -1.0562,  ...,  0.3138,  0.3652, -0.3712]],

            [[-0.3550, -0.2850, -0.1625,  ...,  1.7458,  1.7458,  1.6057],
             [-0.4076, -0.3200, -0.1800,  ...,  1.7983,  1.7808,  1.6232],
             [-0.4076, -0.2850, -0.1625,  ...,  1.8333,  1.7983,  1.6232],
             ...,
             [-1.5805, -1.5280, -0.7577,  ...,  0.0476,  0.0651, -0.0049],
             [-1.0203, -1.4055, -1.6856,  ...,  0.0126,  0.1352, -0.1450],
             [-1.7556, -0.9853, -1.4755,  ..., -0.0224,  0.1176, -0.5476]],

            [[ 0.5136,  0.6182,  0.7751,  ...,  2.0474,  2.0474,  1.8905],
             [ 0.4614,  0.5834,  0.7576,  ...,  2.1171,  2.0997,  1.9428],
             [ 0.4614,  0.6182,  0.7751,  ...,  2.1868,  2.1520,  1.9603],
             ...,
             [-1.2293, -1.1944, -0.4624,  ...,  0.3568,  0.3916,  0.3742],
             [-0.6715, -1.0724, -1.3861,  ...,  0.3219,  0.4091,  0.1825],
             [-1.4384, -0.6890, -1.2119,  ...,  0.2871,  0.3916, -0.2184]]]), tensor([[[-1.4329, -0.4397,  0.8276,  ..., -0.6281, -0.6281, -0.6623],
             [-0.8335,  0.5022,  1.4783,  ..., -0.5938, -0.5767, -0.5938],
             [ 0.0569,  1.1700,  1.5468,  ..., -0.5767, -0.5424, -0.5253],
             ...,
             [-2.1008, -2.1008, -2.0665,  ..., -1.9809, -1.9295, -1.9467],
             [-2.1008, -2.1179, -2.0665,  ..., -1.9980, -1.9980, -2.0494],
             [-2.0837, -2.1179, -2.0665,  ..., -2.0665, -2.0665, -1.9638]],

            [[-1.8431, -1.0028,  0.1702,  ..., -0.6176, -0.5826, -0.6176],
             [-1.3179, -0.0924,  0.7654,  ..., -0.5826, -0.5301, -0.5476],
             [-0.4951,  0.5203,  0.8179,  ..., -0.5301, -0.4601, -0.4426],
             ...,
             [-2.0182, -2.0182, -2.0007,  ..., -2.0357, -2.0182, -2.0357],
             [-2.0182, -2.0357, -2.0182,  ..., -2.0007, -2.0182, -2.0357],
             [-2.0007, -2.0357, -2.0182,  ..., -2.0182, -2.0357, -2.0007]],

            [[-1.5779, -0.7587,  0.3916,  ...,  0.1999,  0.2522,  0.2522],
             [-1.0724,  0.1651,  1.0017,  ...,  0.2696,  0.3393,  0.3219],
             [-0.2532,  0.7751,  1.0191,  ...,  0.3045,  0.4091,  0.4614],
             ...,
             [-1.8044, -1.8044, -1.8044,  ..., -1.8044, -1.7696, -1.7870],
             [-1.7870, -1.8044, -1.7696,  ..., -1.6999, -1.7173, -1.7696],
             [-1.7696, -1.7696, -1.7347,  ..., -1.6999, -1.7347, -1.6476]]])]

I have append 16 images of 3x112x112 shape, when I check shape of the images_set using this code;
print(np.array(images_set, dtype='object').shape)

I get;
(16,)

and then I check the shape of the first index of the image_set as it is 16 image then using this code;
print(np.array(images_set[0]).shape)

I get, each 16 image have this shape
(3, 112, 112)

how can I make it to this shape `(16, 3, 112, 112)?

Output:
A more general format:
import tensorflow as tf
import numpy as np

#Let's make a prototype of one image using ones (just to reproduce the problem without original data...)
one_liketensor=np.ones((3,112,112))

#Then let's notice the same can be seen in tensor-format as follows:
one_liketensor_as_tensor=tf.constant(one_liketensor)

#Let's define length of the list...where the tensors are...
length_of_list=16

#And then let's make an array of the target shape...
multi_array=np.ones((length_of_list,one_liketensor.shape[0],one_liketensor.shape[1],one_liketensor.shape[2]))
for i in range(length_of_list):
    #For clarificatlin let's multiply each distict "image" with the number i to easily undestand the structure of the result...
    multi_array[i,:]=i*one_liketensor
    #...but naturally the "one_liketensor" is something special data ... thus there is need to take this information directly from this source

#And next let's print the result
print(multi_array)

#And let's transform that to tensor-format
multi_array_as_tensor=tf.constant(multi_array)

#And ... tadaa ... you have the material in the preferred format:
print("Shape of the result is: ",multi_array_as_tensor.shape)

...where the "input information" is the length of the list and the shape (and source) of the tensors;


Instruction: HuggingFace text summarization input data format issue

Input:
I’m trying to fine-tune a model to perform text summarization. I’m using AutoModelForSeq2SeqLM.from_pretrained(), so the following applies to several models (e.g. T5, ProphetNet, BART).
I’ve created a class called CustomDataset, which is a subclass of torch.utils.Dataset. That class contains one field: samples - a list of dictionaries that have encodings and labels keys. Each of the values in each of those dictionaries is a torch.Tensor. Here’s what an entry in samples looks like:
{'encoding': tensor([[21603, 10, 188, 563, 1]]), 'label': tensor([[ 1919, 22003, 22, 7, 1]])}
Here’s how I’m attempting to fine-tune the model using Trainer:
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

training_args = TrainingArguments("test_trainer")
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=data,
)
trainer.train()

The error I’m getting gets thrown on line 63 in transformers\data\data_collator.py. Here’s that line of code:
label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
Here’s the error message:
ValueError: only one element tensors can be converted to Python scalars
I understand why the error message specifically is being thrown - the first["label"] tensor isn’t a one-element tensor, and hence item() can’t be called on it. That’s not why I’m asking this question, though.
I’m assuming that I’m not passing the data correctly, but it seems to me that Trainer should take care of input_ids and decoder_input_ids on its own. I’ve tried to set those manually (passing the encodings as input_ids and the labels as decoder_input_ids) and the model can successfully perform inference, but I haven’t managed to fine-tune it. Where am I making a mistake and how do I fix it?

Output:
Using the name label_ids instead of label fixes the specific problem. label should be used if the label is either an int, a float or a one-element torch.Tensor. For tensors with multiple elements, use label_ids. See data_collator.py, lines 62-71 for details:
if "label" in first and first["label"] is not None:
    label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
    dtype = torch.long if isinstance(label, int) else torch.float
    batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
elif "label_ids" in first and first["label_ids"] is not None:
    if isinstance(first["label_ids"], torch.Tensor):
        batch["labels"] = torch.stack([f["label_ids"] for f in features])
    else:
        dtype = torch.long if type(first["label_ids"][0]) is int else torch.float
        batch["labels"] = torch.tensor([f["label_ids"] for f in features], dtype=dtype)

Also, the name input_ids should be used instead of encoding. Otherwise, an unknown kwarg error gets thrown.


Instruction: Pytorch Lightning Tensorboard Logger Across Multiple Models

Input:
I'm relatively new to Lightning and Loggers vs manually tracking metrics.  I am trying to train two distinct models and have their accuracy and loss plotted on the same charts in tensorboard (or any other logger) within Colab.
What I have right now is basically:
trainer1 = pl.Trainer(gpus=n_gpus, max_epochs=n_epochs, progress_bar_refresh_rate=20, num_sanity_val_steps=0)

trainer2 = pl.Trainer(gpus=n_gpus, max_epochs=n_epochs, progress_bar_refresh_rate=20, num_sanity_val_steps=0)

trainer1.fit(Model1, train_loader, val_loader)
trainer2.fit(Model2, train_loader, val_loader)

#Then later:

%load_ext tensorboard

%tensorboard --logdir lightning_logs/

What I'd like to see at this point are those logged metrics charted together on the same chart, any help would be appreciated.  I've spent some time trying to toy with this but I'm a bit out of my depth on this, thank you!

Output:
The exact chart used for logging a specific metric depends on the key name you provide in the .log() call (its a feature that Lightning inherits from TensorBoard itself)
def validation_step(self, batch, _):
  #  This string decides which chart to use in the TB web interface
  #         vvvvvvvvv
  self.log('valid_acc', acc)

Just use the same string for both .log() calls and have both runs saved in same directory.
logger = TensorBoardLogger(save_dir='lightning_logs/', name='model1')
logger = TensorBoardLogger(save_dir='lightning_logs/', name='model2')

If you run tesnsorboard --logdir ./lightning_logs pointing at the parent directory, you should be able to see both metric in the same chart with the key named valid_acc.


Instruction: GNN with Stable baselines

Input:
I am looking to use DGL or pytorch geometric for building my policy and value networks in stable baselines, however I am struggling to figure out how to send over observations.  The observations must be one of the gym spaces class but I am not sure how to send a graph object that can be used by DGL or Pytorch geometric in this way.
The fundamental question I have is how to send graph observations and where to do the prepossessing necessary to use DGL or pytorch geometric for a custom stable baselines network?  Can I pack the graph into a stable baselines observation space that somehow DGL or pytorch geometric could intake it?
Note: If anyone has a github link with any code that has done this please let me know, I have looked everywhere

Output:
You can serialize your DGL graph object using pickle and convert the resultant byte string into a vector of integers (with each char in the string corresponding to one int).
import dgl
import numpy as np
import pickle

def serialize_graph(graph: dgl.DGLGraph):
    as_byte_string = pickle.dumps(graph)
    as_int_list = [_ for _ in as_byte_string]  # we get ints for free without explicitly casting
    as_float_array = np.array(as_int_list, dtype=np.float32)
    return as_float_array

You can then apply the same operations in reverse to deserialize vector representation of the graph within your custom feature extractor.
import dgl
import pickle
import torch as th

def deserialize_graph(observation: th.Tensor):
    as_int_tensor = observation.to(dtype=th.int32)
    as_char_list = [chr(_) for _ in observation]
    as_byte_string = bytearray(''.join(as_char_list), encoding='latin')
    as_dgl_graph = pickle.loads(as_byte_string)
    return as_dgl_graph


Instruction: Evaluate the model during training affects its performance PyTorch

Input:
In PyTorch, I want to evaluate my model on the validation set every eval_step during training, and I wrote code like this:
def tune(model, loader_train, loader_dev, optimizer, epochs, eval_step):
    for epoch in range(epochs):
        for step,x in enumerate(loader_train):
            optimizer.zero_grad()
            loss = model(x)
            loss.backward()
            optimizer.step()

            if step % eval_step == 0:
                model.eval()
                test(model, loader_dev)
                model.train()

When eval_step = int(len(loader_train)/2) and eval_step = int(len(loader_train)/8), they lead to quite different metric result after training through one whole epoch (which means the second output for the former differs the eighth output for the latter).
Could anyone explain why?

The length of loader_train is 20000 (it depends on batch size), and here is my testing script:
def test(model, loader_dev):
    preds = []
    labels = []
    for step,x in enumerate(loader_dev):
        preds.append(model(x).view(-1))
        labels.apend(x['label'].view(-1))

    metric = cal_metric(preds, labels)
    logger.info(metric)

Output: I think you probably set 'shffule=True' in your dataloader. Even though you fix 'random seed', dataloader in torch will generate different results if you use another dataloader while using current dataloader. In the scenario you describe, it may cause your model get data input in different order and then result in different metric result.


Instruction: Detect if image is blurry using pytorch android API

Input:
So I am working in this app in which if an officer take a picture of car accident and if this picture is blur, it will no accept it. It will only allow images based on quality and resolution. So after research, I found that I can implement this feature using pytorch in which I will save it in my asset and then write kotlin code based on my logic. I found this link: https://pytorch.org/mobile/android/ However, this link shows image classifier example in which user can detect if the image is a cat or a dog, and this is not what I want.
As an example,
If a police officer take a picture via the app of car accident, the app needs to verify if the image resolution and quality is readable and understandable so when user sees the image taken he/she can tell if it is car accident compared to that of when it is blur image and not clear the app will ask the user to take a picture again.
Any tips please ?
Thank you

Output:
For a start you can try to build this using conventional computer vision algorithms.
You will find a few examples trying to implement a similar idea using OpenCV (e.g. here).
If you want to solve such a task through machine learning, you either can try to find a model that is trained to solve such a task online (maybe you can find one if you are lucky), or you have to train such a model yourself.
When using machine learning you need to have a dataset, which in your case you can easily create from a database of images and blurring the examples yourself (see self-supervised learning). You need to decide which machine learning method to use. For a simple detection if an image is blurry, using deep learning and CNNs seems unnecessary. Instead look into extracting some features relating to an images bluriness and use a simple machine learning algorithms such as k-NN.
However I think using machine learning to solve this kind of task is not really necessary. Every digital camera is basically implementing this feature for their autofocus. So I think looking for conventional computer vision solutions with OpenCV, (e.g. edge contrast estimation, image frequency transforms, ...) will get you to your goal quicker.
OpenCV is also available for Android SDK.


Instruction: AWS Sagemaker InvokeEndpoint: operation: Endpoint of account not found

Input:
I've been following this guide here: https://aws.amazon.com/blogs/machine-learning/building-an-nlu-powered-search-application-with-amazon-sagemaker-and-the-amazon-es-knn-feature/
I have successfully deployed the model from my notebook instance. I am also able to generate predictions by calling predict() method from sagemaker.predictor.
This is how I created and deployed the model
class StringPredictor(Predictor):
    def __init__(self, endpoint_name, sagemaker_session):
        super(StringPredictor, self).__init__(endpoint_name, sagemaker_session, content_type='text/plain')

pytorch_model = PyTorchModel(model_data = inputs,
                             role=role,
                             entry_point ='inference.py',
                             source_dir = './code',
                             framework_version = '1.3.1',
                             py_version='py3',
                             predictor_cls=StringPredictor)

predictor = pytorch_model.deploy(instance_type='ml.m5.large', initial_instance_count=4)

From the SageMaker dashboard, I can even see that my endpoint and the status is "in-service"
If I run aws sagemaker list-endpoints I can see my desired endpoint showing up correctly as well.
My issue is when I run this code (outside of sagemaker), I'm getting an error:
import boto3
sm_runtime_client = boto3.client('sagemaker-runtime')
payload = "somestring that is used here"
response = sm_runtime_client.invoke_endpoint(EndpointName='pytorch-inference-xxxx',ContentType='text/plain',Body=payload)

This is the error thrown
botocore.errorfactory.ValidationError: An error occurred (ValidationError) when calling the InvokeEndpoint operation: Endpoint pytorch-inference-xxxx of account xxxxxx not found.

This is quite strange as I'm able to see and run the endpoint just fine from sagemaker notebook and I am able to run the predict() method too.
I have verified the region, endpoint name and the account number.

Output:
I was having the exact same error, I've just fixed mine by setting the correct region.

I have verified the region, endpoint name and the account number.

I know that you have indicated that you have verified the region, but in my case, the remote computer had another region configured. So I just ran the following command on my remote computer
aws configure

And once I set the key ID and secret key again, I set the correct region and the error was gone.


Instruction: Keyerror:None ,I don't understand this problem

Input:
class KITTIRAWDataset(KITTIDataset):

def __init__(self, *args, **kwargs):
    super(KITTIRAWDataset, self).__init__(*args, **kwargs)

def get_image_path(self, folder, frame_index, side):
    self.img_ext='.png'
    f_str = "{:010d}{}".format(frame_index, self.img_ext)
    image_path = os.path.join(
        self.data_path, folder, "image_0{}/data".format(self.side_map[side]), f_str)
    return image_path

def get_depth(self, folder, frame_index, side, do_flip):
    calib_path = os.path.join(self.data_path, folder.split("/")[0])
    # calib_path: D:/SomeExperiments/KITTRawData/2011_09_26
    velo_filename = os.path.join(
        self.data_path,
        folder,
        # Only change is using colon (:) instead of %. For example, instead of %s use {:s} and instead of %d use (:d}
        # 010d: the integer length 10
        "velodyne_points/data/{:010d}.bin".format(int(frame_index)))
    depth_gt = generate_depth_map(calib_path, velo_filename, self.side_map[side])
    depth_gt = skimage.transform.resize(
        depth_gt, self.full_res_shape[::-1], order=0, preserve_range=True, mode='constant')

    if do_flip:
        depth_gt = np.fliplr(depth_gt)
    # print(type(depth_gt)) 'numpy.ndarray'
    return

when I was running my code, an error occured, like below:

I have changed my torch version which refers to the author's markdown, but it can't solve this problem

Output:
KeyError means that you are trying to get a value from a dict with a key that does not exist. In the line displayed you have self.side_map[side]) and KeyError: None means that the key is None, so your side variable have a value None.
That is what we an know looking at the code, the error and without more context


Instruction: Single GPU Pytorch training with SLURM - how to set "ntasks-per-node"?

Input:
I would like to do some simple fine-tuning on a transformers model using a single GPU on a server via SLURM. I haven't used SLURM before and I am not a computer scientist so my understanding of the field is a bit limited. I have done some research and created the script below.
Could you please confirm if it is fit for purpose?
As far as I have understood, a node corresponds to a single computer and "--gres=gpu:1" will use a single gpu. The only thing I haven't understood clearly is "ntasks-per-node". The way I have understood it, because I will run a single python script, this can be equal to 1. Is that correct?
#! /bin/bash

#SBATCH --job-name 'SQuAD'
#SBATCH --output squad_job%J.out
#SBATCH --error squad_error%J.err
#SBATCH --nodes=1
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH --partition=normal
#SBATCH --time=72:00:00

python3 fine_tune_squad.py

Output:
Yes, it will request 1 GPU for running the task. As described in the documentation:

The default is one task per node [...]

Therefore, the default value for --ntasks-per-node is already 1, which means you don't even need to define it. In fact, even --nodes has a default value of 1. Nonetheless, some consider a good practice to explicitly define them to avoid problems, so I'd leave them as you did.


Instruction: min-max normalization of a tensor in PyTorch

Input:
I want to perform min-max normalization on a tensor in PyTorch.
The formula to obtain min-max normalization is

I want to perform min-max normalization on a tensor using some new_min and new_max without iterating through all elements of the tensor.
>>>import torch
>>>x = torch.randn(5, 4)
>>>print(x)
tensor([[-0.8785, -1.6898,  2.2129, -0.8375],
        [ 1.2927, -1.3187, -0.7087, -2.1143],
        [-0.6162,  0.6836, -1.3342, -0.7889],
        [-0.2934, -1.2526, -0.3265,  1.1933],
        [ 1.2494, -1.2130,  1.5959,  1.4232]])

Is there any way to min-max normalize the given tensor between two values new_min, new_max?
Suppose I want to scale the tensor from new_min = -0.25 to new_max = 0.25

Output:
Having defined v_min, v_max, new_min, and new_max as:
>>> v_min, v_max = v.min(), v.max()
>>> new_min, new_max = -.25, .25

You can apply your formula element-wise:
>>> v_p = (v - v_min)/(v_max - v_min)*(new_max - new_min) + new_min
tensor([[-0.1072, -0.2009,  0.2500, -0.1025],
        [ 0.1437, -0.1581, -0.0876, -0.2500],
        [-0.0769,  0.0733, -0.1599, -0.0969],
        [-0.0396, -0.1504, -0.0434,  0.1322],
        [ 0.1387, -0.1459,  0.1787,  0.1588]])

Then check v_p statistics:
>>> v_p.min(), v_p.max()
(tensor(-0.2500), tensor(0.2500))


Instruction: How to get values return by Tuple Object in Maskcrnn libtorch

Input:
I’m new in C++ and libtorch, I try load model by torchscript and execute inference, the codes like below:
   torch::jit::script::Module module;
   try {
       module = torch::jit::load("../../weights/card_extraction/pytorch/2104131340/best_model_27_mAP=0.9981_torchscript.pt");
   }
   catch (const c10::Error& e) {
      std::cerr << "Error to load model\n";
      return -1;
   }
   std::cout << "Load model successful!\n";
   torch::DeviceType device_type;
   device_type = torch::kCPU;
   torch::Device device(device_type, 0);
   module.to(device);

   torch::Tensor sample = torch::zeros({3, 800, 800});
   std::vector<torch::jit::IValue> inputs;
   std::vector<torch::Tensor> images;
   images.push_back(sample);
   /* images.push_back(torch::ones({3, 224, 224})); */

   inputs.push_back(images);

   auto t1 = std::chrono::high_resolution_clock::now();
   auto output = module.forward(inputs);
   auto t2 = std::chrono::high_resolution_clock::now();
   int duration = std::chrono::duration_cast<std::chrono::milliseconds> (t2 - t1).count();
   std::cout << "Inference time: " << duration << " ms" << std::endl;
   std::cout << output << std::endl;

And the result like this:
Load model successful!
[W mask_rcnn.py:86] Warning: RCNN always returns a (Losses, Detections) tuple in scripting (function )
Inference time: 2321 ms
({}, [{boxes: [ CPUFloatType{0,4} ], labels: [ CPULongType{0} ], scores: [ CPUFloatType{0} ], masks: [ CPUFloatType{0,1,800,800} ]}])

How do I get value boxes, labels, scores and masks from return output object using c++ ?
I tried many ways but compile always error with “c10::IValue” error thrown.
And more question, why is the time inference when I convert the model to torchscript, executed by C++ is slower than python?
Many thanks

Output:
You can get access to the element like here: I can parse Tuple arguments and get access to it like in Tensor format. It can help u.
auto output1_t = output.toTuple()->elements()[0].toTensor();
auto output2_t = output.toTuple()->elements()[1].toTensor();

https://discuss.pytorch.org/t/how-can-i-get-access-to-first-and-second-tensor-from-tuple-returned-from-forward-method-in-libtorch-c-c/139741


Instruction: HuggingFace Trainer logging train data

Input:
I'm following this tutorial to train some models:
https://huggingface.co/transformers/training.html
I'd like to track not only the evaluation loss and accuracy but also the train loss and accuracy, to monitor overfitting. While running the code in Jupyter, I do see all of htis:
Epoch   Training Loss   Validation Loss Accuracy    Glue
1   0.096500    0.928782    {'accuracy': 0.625} {'accuracy': 0.625, 'f1': 0.0}
2   0.096500    1.203832    {'accuracy': 0.625} {'accuracy': 0.625, 'f1': 0.0}
3   0.096500    1.643788    {'accuracy': 0.625} {'accuracy': 0.625, 'f1': 0.0}

but when I go into trainer.state.log_history, that stuff is not there. This really doesn't make sense to me.
for obj in trainer.state.log_history:
    print(obj)

{'loss': 0.0965, 'learning_rate': 4.5833333333333334e-05, 'epoch': 0.25, 'step': 1}
{'eval_loss': 0.9287818074226379, 'eval_accuracy': {'accuracy': 0.625}, 'eval_glue': {'accuracy': 0.625, 'f1': 0.0}, 'eval_runtime': 1.3266, 'eval_samples_per_second': 6.03, 'eval_steps_per_second': 0.754, 'epoch': 1.0, 'step': 4}
{'eval_loss': 1.2038320302963257, 'eval_accuracy': {'accuracy': 0.625}, 'eval_glue': {'accuracy': 0.625, 'f1': 0.0}, 'eval_runtime': 1.3187, 'eval_samples_per_second': 6.067, 'eval_steps_per_second': 0.758, 'epoch': 2.0, 'step': 8}
{'eval_loss': 1.6437877416610718, 'eval_accuracy': {'accuracy': 0.625}, 'eval_glue': {'accuracy': 0.625, 'f1': 0.0}, 'eval_runtime': 1.3931, 'eval_samples_per_second': 5.742, 'eval_steps_per_second': 0.718, 'epoch': 3.0, 'step': 12}
{'train_runtime': 20.9407, 'train_samples_per_second': 1.146, 'train_steps_per_second': 0.573, 'total_flos': 6314665328640.0, 'train_loss': 0.07855576276779175, 'epoch': 3.0, 'step': 12}

How do I get these back in an object, and not a printout?
Thanks
Edit: Reproducable code below:
import numpy as np
from datasets import load_metric, load_dataset
from transformers import TrainingArguments, AutoModelForSequenceClassification, Trainer, AutoTokenizer
from datasets import list_metrics

raw_datasets = load_dataset("imdb")

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(8))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(8))
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

training_args = TrainingArguments("IntroToBERT", evaluation_strategy="epoch")
training_args.logging_strategy = 'step'
training_args.logging_first_step = True
training_args.logging_steps = 1
training_args.num_train_epochs = 3
training_args.per_device_train_batch_size = 2
training_args.eval_steps = 1

metrics = {}
for metric in ['accuracy','glue']:
    metrics[metric] = load_metric(metric,'mrpc')

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    out = {}
    for metric in metrics.keys():
        out[metric] = metrics[metric].compute(predictions=predictions, references=labels)
    return out

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)

trainer.train()

# here the printout is as shown

for obj in trainer.state.log_history:
    print(obj)

# here the logging data is displayed

Output:
You can use the methods log_metrics to format your logs and save_metrics to save them. Here is the code:
# rest of the training args
# ...
training_args.logging_dir = 'logs' # or any dir you want to save logs

# training
train_result = trainer.train()

# compute train results
metrics = train_result.metrics
max_train_samples = len(small_train_dataset)
metrics["train_samples"] = min(max_train_samples, len(small_train_dataset))

# save train results
trainer.log_metrics("train", metrics)
trainer.save_metrics("train", metrics)

# compute evaluation results
metrics = trainer.evaluate()
max_val_samples = len(small_eval_dataset)
metrics["eval_samples"] = min(max_val_samples, len(small_eval_dataset))

# save evaluation results
trainer.log_metrics("eval", metrics)
trainer.save_metrics("eval", metrics)

You can also save all logs at once by setting the split parameter in log_metrics and save_metrics to "all" i.e. trainer.save_metrics("all", metrics); but I prefer this way as you can customize the results based on your need.
Here is the complete source provided by transformers  from which you can read more.


Instruction: Negative loss when trying to implement aleatoric uncertainty estimation according to Kendall et al

Input:
I'm trying to implement a neural network with aleatoric uncertainty estimation for regression with pytorch according to
Kendall et al.: "What Uncertainties Do We Need in Bayesian Deep
Learning for Computer Vision?" (Link).
However, while the predicted regression values fit the desired ground truth values quite well, the predicted variance looks weird and the loss gets negative during training.
The paper suggests to have two outputs mean and variance instead of only predicting the regression value. To be more precise, it is suggested to predict mean and log(variance) due to stability reasons. Therefore, my network looks as follows:
class ReferenceResNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fcl1 = nn.Linear(1, 32)
        self.fcl2 = nn.Linear(32, 64)
        self.fcl3 = nn.Linear(64, 128)
        self.fcl_mean = nn.Linear(128,1)
        self.fcl_var = nn.Linear(128,1)

    def forward(self, x):
        x = torch.tanh(self.fcl1(x))
        x = torch.tanh(self.fcl2(x))
        x = torch.tanh(self.fcl3(x))
        mean = self.fcl_mean(x)
        log_var = self.fcl_var(x)
        return mean, log_var

According to the paper, given these outputs, the corresponding loss function consists of a residual regression-part and a regularization term:

where si is the log(variance) predicted by the network.
I implemented this loss-function accordingly:
def loss_function(pred_mean, pred_log_var, y):
    return 1/len(pred_mean)*(0.5 * torch.exp(-pred_log_var)*torch.sqrt(torch.pow(y-pred_mean, 2))+0.5*pred_log_var).sum()

I tried this code on a self-generated toy dataset (see image with results), however, the loss gets negative during training and when I plot the variance over the dataset after training, for me it does not really make sense while the corresponding mean values fit the ground truth quite well:

I already figured out that the negative loss comes from the regularization term as logarithms are negative for values between 0 and 1, however, I don't believe that the absolute value of the regularization term is supposed to grow bigger than the regression part. Does anyone know what is the reason for this and how I can prevent this from happening? And why does my variance look so weird?
For reproduction, my full code looks as follows:
import torch.nn as nn
import torch
import torch.nn.functional as F
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.utils.data.dataset import TensorDataset
from torchvision import datasets, transforms
import math
import numpy as np
import torch.nn.functional as F
import matplotlib.pyplot as plt
from tqdm import tqdm

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class ReferenceRegNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fcl1 = nn.Linear(1, 32)
        self.fcl2 = nn.Linear(32, 64)
        self.fcl3 = nn.Linear(64, 128)
        self.fcl_mean = nn.Linear(128,1)
        self.fcl_var = nn.Linear(128,1)

    def forward(self, x):
        x = torch.tanh(self.fcl1(x))
        x = torch.tanh(self.fcl2(x))
        x = torch.tanh(self.fcl3(x))
        mean = self.fcl_mean(x)
        log_var = self.fcl_var(x)
        return mean, log_var

def toy_function(x):
    return math.sin(x/15-4)+2 + math.sin(x/10-5)

def loss_function(x_mean, x_log_var, y):
    return 1/len(x_mean)*(0.5 * torch.exp(-x_log_var)*torch.sqrt(torch.pow(y-x_mean, 2))+0.5*x_log_var).sum()

BATCH_SIZE = 10
EVAL_BATCH_SIZE = 10
CLASSES = 1
TRAIN_EPOCHS = 50

# generate toy dataset: A train-set in form of a complex sin-curve
x_train_data = np.array([])
y_train_data = np.array([])
for repeat in range(2):
    for i in range(50, 150):
        for j in range(100):
            sampled_x = i+np.random.randint(101)/100
            sampled_y = toy_function(sampled_x)+np.random.normal(0,0.2)
            x_train_data = np.append(x_train_data, sampled_x)
            y_train_data = np.append(y_train_data, sampled_y)
x_eval_data = list(np.arange(50.0, 150.0, 0.1))
y_eval_data = [toy_function(x) for x in x_eval_data]

LOADER_KWARGS = {'num_workers': 0, 'pin_memory': False} if torch.cuda.is_available() else {}
train_set = TensorDataset(torch.Tensor(x_train_data),torch.Tensor(y_train_data))
eval_set = TensorDataset(torch.Tensor(x_eval_data), torch.Tensor(y_eval_data))
train_loader = torch.utils.data.DataLoader(train_set, batch_size=BATCH_SIZE, shuffle=True, **LOADER_KWARGS)
eval_loader = torch.utils.data.DataLoader(eval_set, batch_size=EVAL_BATCH_SIZE, shuffle=False, **LOADER_KWARGS)

TRAIN_SIZE = len(train_loader.dataset)
EVAL_SIZE = len(eval_loader.dataset)

assert (TRAIN_SIZE % BATCH_SIZE) == 0
assert (EVAL_SIZE % EVAL_BATCH_SIZE) == 0

net = ReferenceRegNet().to(DEVICE)
optimizer = optim.Adam(net.parameters(), lr=1e-3)

losses = {}
# train network
for epoch in range(1,TRAIN_EPOCHS+1):

    net.train()
    mean_epoch_loss = 0
    mean_epoch_mse = 0

    # train batches
    for batch_idx, (data, target) in enumerate(tqdm(train_loader), start=1):
        data, target = (data.to(DEVICE)).unsqueeze(dim=1), (target.to(DEVICE)).unsqueeze(dim=1)
        optimizer.zero_grad()

        output_means, output_log_var = net(data)

        target_np = target.detach().cpu().numpy()
        output_means_np = output_means.detach().cpu().numpy()

        loss = loss_function(output_means, output_log_var, target)
        loss_value = loss.item()    # get raw float-value out of loss-tensor
        mean_epoch_loss += loss_value

        # optimize network
        loss.backward()
        optimizer.step()

    mean_epoch_loss = mean_epoch_loss / len(train_loader)
    losses.update({epoch:mean_epoch_loss})

    print("Epoch " + str(epoch) + ": Train-Loss = " + str(mean_epoch_loss))

    net.eval()
    with torch.no_grad():
        mean_loss = 0
        mean_mse = 0

        for data, target in eval_loader:
            data, target = (data.to(DEVICE)).unsqueeze(dim=1), (target.to(DEVICE)).unsqueeze(dim=1)
            output_means, output_log_var = net(data)         # perform prediction
            target_np = target.detach().cpu().numpy()
            output_means_np = output_means.detach().cpu().numpy()
            mean_loss += loss_function(output_means, output_log_var, target).item()

        mean_loss = mean_loss/len(eval_loader)
        #print("Epoch " + str(epoch) + ": Eval-loss = " + str(mean_loss))

fig = plt.figure(figsize=(40,12))   # create a 30x30 inch figure
ax = fig.add_subplot(1,3,1)
ax.set_title("regression value")
ax.set_xlabel("x")
ax.set_ylabel("regression mean")
ax.plot(x_train_data, y_train_data, 'x', color='black')
ax.plot(x_eval_data, y_eval_data, color='red')
pred_means_list = []
output_vars_list_train = []
output_vars_list_test = []
for x_test in sorted(x_train_data):
    x_test = (torch.Tensor([x_test]).to(DEVICE))
    pred_means, output_log_vars = net.forward(x_test)
    pred_means_list.append(pred_means.detach().cpu())
    output_vars_list_train.append(torch.exp(output_log_vars).detach().cpu())
ax.plot(sorted(x_train_data), pred_means_list, color='blue', label = 'training_perform')
pred_means_list = []
for x_test in x_eval_data:
    x_test = (torch.Tensor([x_test]).to(DEVICE))
    pred_means, output_log_vars = net.forward(x_test)
    pred_means_list.append(pred_means.detach().cpu())
    output_vars_list_test.append(torch.exp(output_log_vars).detach().cpu())
ax.plot(sorted(x_eval_data), pred_means_list, color='green', label = 'eval_perform')
plt.tight_layout()
plt.legend()

ax = fig.add_subplot(1,3,2)
ax.set_title("variance")
ax.set_xlabel("x")
ax.set_ylabel("regression var")
ax.plot(sorted(x_train_data), output_vars_list_train, label = 'training data')
ax.plot(x_eval_data, output_vars_list_test, label = 'test data')

plt.tight_layout()
plt.legend()

ax = fig.add_subplot(1,3,3)
ax.set_title("training loss")
ax.set_xlabel("Epoch")
ax.set_ylabel("Loss")
lists = sorted(losses.items())
epoch, loss = zip(*lists)
ax.plot(epoch, loss, label = 'loss')

plt.tight_layout()
plt.legend()
plt.savefig('ref_test.png')

Output:
TLDR: The optimization drives the loss to a minimum where the gradient
becomes zero, regardless of what the nominal loss value is.
A comprehensive explanation by K.Frank:

A smaller loss – algebraically less positive or algebraically more
negative – means (or should mean) better predictions. The
optimization step uses some version of gradient descent to make
your loss smaller. The overall level of the loss doesn’t matter as
far as the optimization goes. The gradient tells the optimizer how
to change the model parameters to reduce the loss, and it doesn’t
care about the overall level of the loss.

An example from the same source:

Consider, for example, optimizing with lossA = MSELoss. Now
imagine optimizing with lossB = lossA - 17.2. The 17.2 doesn’t
really change anything at all. It is true that “perfect” predictions
will yield lossB = -17.2 rather than zero. (lossA will, of course,
be zero for “perfect” predictions.) But who cares?

In your example: you are right, the negative loss value comes from the logarithmic term. This is completely OK and it means that your training is dominated by contributions of high-confidence loss terms. Regarding the high values of variance - can't comment much on it but it should be fine since the loss curve drops as expected.


Instruction: Splitting tensor into sub-tensors in overlapping fashion

Input:
I'm in pytorch and I have a tensor x of size batch_size x d x S. It has to be intended as a batch of sequences of length S, where every sequence element is d dimensional. Every sequence is actually the overlap of multiple sub-sequences, in the following sense:

every sub-sequence is of size past_size + present_size, i.e we have past_size d-dimensional elements followed by other present_size elements
the overlap works as follows: the beginnings of the present_size sections are equispaced by present_size elements, and they are placed in the right-most positions

To make an example, with batch_size=1, d=1, consider x = [1,2,3,4,5,6,7,8,9], where present_size = 2, past_size = 3. The resulting subsequences would be:

[1,2,3,4,5]
[3,4,5,6,7]
[5,6,7,8,9]

The end goal is to produce the splitting of every sequence into the, say, N sub-sequences, to get a tensor of shape batch_size*N x d x past_size+present_size.
My second try is the following:
def seq(x, present_size, total_size=present_size+past_size, N):
   z = x.unfold(-1, total_size, present_size)
   v = torch.flatten(z, start_dim=2)
   s = torch.cat(torch.chunk(v, N, -1), 0)
   return s

Is there a more efficient way? Is it possible to backpropagate through such a function?
Edit
In the above example, N = 3.
Moreover, we have the following relation: N*present_size + past_size = S

Input-output
Here is an example with N=4, present_size = 1, past_size = 2.
x = torch.rand(4,8,6) # d=8, batch_size = 4, 6 = N*present_size + past_size
>>> tensor([[[0.5667, 0.5300, 0.2460, 0.4327, 0.4727, 0.5649],
     [0.0360, 0.6687, 0.0167, 0.5359, 0.9804, 0.8778],
     [0.3703, 0.4884, 0.1505, 0.5463, 0.8114, 0.3270],
     [0.2932, 0.4928, 0.3933, 0.2433, 0.7053, 0.5222],
     [0.6667, 0.2014, 0.7107, 0.7535, 0.2816, 0.6515],
     [0.5285, 0.4150, 0.2557, 0.2144, 0.8317, 0.5448],
     [0.7971, 0.6609, 0.1811, 0.7788, 0.6649, 0.1848],
     [0.6902, 0.3999, 0.8719, 0.7624, 0.5216, 0.3494]],

    [[0.0196, 0.7850, 0.2796, 0.4173, 0.8076, 0.5709],
     [0.4566, 0.4814, 0.0568, 0.8568, 0.9119, 0.4030],
     [0.4031, 0.8887, 0.3782, 0.8015, 0.9835, 0.6043],
     [0.3557, 0.5960, 0.2102, 0.8165, 0.1938, 0.4948],
     [0.8163, 0.7907, 0.3711, 0.6835, 0.8021, 0.1897],
     [0.7790, 0.2621, 0.3769, 0.3830, 0.7140, 0.2309],
     [0.5831, 0.0246, 0.6548, 0.8694, 0.1988, 0.5470],
     [0.1192, 0.2928, 0.4240, 0.2624, 0.7959, 0.4091]],

    [[0.7959, 0.7144, 0.4523, 0.5090, 0.6053, 0.4071],
     [0.4742, 0.0224, 0.9939, 0.9757, 0.0732, 0.6213],
     [0.5211, 0.1149, 0.8218, 0.7061, 0.1807, 0.2822],
     [0.1456, 0.7331, 0.9107, 0.9533, 0.2438, 0.4031],
     [0.0958, 0.2623, 0.0828, 0.2861, 0.0474, 0.8349],
     [0.1740, 0.3658, 0.2416, 0.6735, 0.4013, 0.8896],
     [0.6934, 0.8709, 0.4017, 0.6121, 0.5824, 0.5803],
     [0.4811, 0.1036, 0.4356, 0.6441, 0.5859, 0.4683]],

    [[0.2479, 0.9247, 0.3216, 0.6844, 0.1701, 0.4609],
     [0.3320, 0.4908, 0.0458, 0.9887, 0.4725, 0.7511],
     [0.0594, 0.1978, 0.8830, 0.9126, 0.4821, 0.7731],
     [0.3729, 0.4921, 0.9266, 0.7827, 0.8101, 0.6258],
     [0.4998, 0.7596, 0.1160, 0.3928, 0.4773, 0.7892],
     [0.0215, 0.1325, 0.5940, 0.2094, 0.3109, 0.9281],
     [0.7960, 0.1707, 0.1793, 0.7335, 0.2065, 0.6204],
     [0.6350, 0.9696, 0.5099, 0.7375, 0.7601, 0.1405]]])

r = seq(x, 1, 2+1, 4)
>>> tensor([[[0.5667, 0.5300, 0.2460],
     [0.0360, 0.6687, 0.0167],
     [0.3703, 0.4884, 0.1505],
     [0.2932, 0.4928, 0.3933],
     [0.6667, 0.2014, 0.7107],
     [0.5285, 0.4150, 0.2557],
     [0.7971, 0.6609, 0.1811],
     [0.6902, 0.3999, 0.8719]],

    [[0.0196, 0.7850, 0.2796],
     [0.4566, 0.4814, 0.0568],
     [0.4031, 0.8887, 0.3782],
     [0.3557, 0.5960, 0.2102],
     [0.8163, 0.7907, 0.3711],
     [0.7790, 0.2621, 0.3769],
     [0.5831, 0.0246, 0.6548],
     [0.1192, 0.2928, 0.4240]],

    [[0.7959, 0.7144, 0.4523],
     [0.4742, 0.0224, 0.9939],
     [0.5211, 0.1149, 0.8218],
     [0.1456, 0.7331, 0.9107],
     [0.0958, 0.2623, 0.0828],
     [0.1740, 0.3658, 0.2416],
     [0.6934, 0.8709, 0.4017],
     [0.4811, 0.1036, 0.4356]],

    [[0.2479, 0.9247, 0.3216],
     [0.3320, 0.4908, 0.0458],
     [0.0594, 0.1978, 0.8830],
     [0.3729, 0.4921, 0.9266],
     [0.4998, 0.7596, 0.1160],
     [0.0215, 0.1325, 0.5940],
     [0.7960, 0.1707, 0.1793],
     [0.6350, 0.9696, 0.5099]],

    [[0.5300, 0.2460, 0.4327],
     [0.6687, 0.0167, 0.5359],
     [0.4884, 0.1505, 0.5463],
     [0.4928, 0.3933, 0.2433],
     [0.2014, 0.7107, 0.7535],
     [0.4150, 0.2557, 0.2144],
     [0.6609, 0.1811, 0.7788],
     [0.3999, 0.8719, 0.7624]],

    [[0.7850, 0.2796, 0.4173],
     [0.4814, 0.0568, 0.8568],
     [0.8887, 0.3782, 0.8015],
     [0.5960, 0.2102, 0.8165],
     [0.7907, 0.3711, 0.6835],
     [0.2621, 0.3769, 0.3830],
     [0.0246, 0.6548, 0.8694],
     [0.2928, 0.4240, 0.2624]],

    [[0.7144, 0.4523, 0.5090],
     [0.0224, 0.9939, 0.9757],
     [0.1149, 0.8218, 0.7061],
     [0.7331, 0.9107, 0.9533],
     [0.2623, 0.0828, 0.2861],
     [0.3658, 0.2416, 0.6735],
     [0.8709, 0.4017, 0.6121],
     [0.1036, 0.4356, 0.6441]],

    [[0.9247, 0.3216, 0.6844],
     [0.4908, 0.0458, 0.9887],
     [0.1978, 0.8830, 0.9126],
     [0.4921, 0.9266, 0.7827],
     [0.7596, 0.1160, 0.3928],
     [0.1325, 0.5940, 0.2094],
     [0.1707, 0.1793, 0.7335],
     [0.9696, 0.5099, 0.7375]],

    [[0.2460, 0.4327, 0.4727],
     [0.0167, 0.5359, 0.9804],
     [0.1505, 0.5463, 0.8114],
     [0.3933, 0.2433, 0.7053],
     [0.7107, 0.7535, 0.2816],
     [0.2557, 0.2144, 0.8317],
     [0.1811, 0.7788, 0.6649],
     [0.8719, 0.7624, 0.5216]],

    [[0.2796, 0.4173, 0.8076],
     [0.0568, 0.8568, 0.9119],
     [0.3782, 0.8015, 0.9835],
     [0.2102, 0.8165, 0.1938],
     [0.3711, 0.6835, 0.8021],
     [0.3769, 0.3830, 0.7140],
     [0.6548, 0.8694, 0.1988],
     [0.4240, 0.2624, 0.7959]],

    [[0.4523, 0.5090, 0.6053],
     [0.9939, 0.9757, 0.0732],
     [0.8218, 0.7061, 0.1807],
     [0.9107, 0.9533, 0.2438],
     [0.0828, 0.2861, 0.0474],
     [0.2416, 0.6735, 0.4013],
     [0.4017, 0.6121, 0.5824],
     [0.4356, 0.6441, 0.5859]],

    [[0.3216, 0.6844, 0.1701],
     [0.0458, 0.9887, 0.4725],
     [0.8830, 0.9126, 0.4821],
     [0.9266, 0.7827, 0.8101],
     [0.1160, 0.3928, 0.4773],
     [0.5940, 0.2094, 0.3109],
     [0.1793, 0.7335, 0.2065],
     [0.5099, 0.7375, 0.7601]],

    [[0.4327, 0.4727, 0.5649],
     [0.5359, 0.9804, 0.8778],
     [0.5463, 0.8114, 0.3270],
     [0.2433, 0.7053, 0.5222],
     [0.7535, 0.2816, 0.6515],
     [0.2144, 0.8317, 0.5448],
     [0.7788, 0.6649, 0.1848],
     [0.7624, 0.5216, 0.3494]],

    [[0.4173, 0.8076, 0.5709],
     [0.8568, 0.9119, 0.4030],
     [0.8015, 0.9835, 0.6043],
     [0.8165, 0.1938, 0.4948],
     [0.6835, 0.8021, 0.1897],
     [0.3830, 0.7140, 0.2309],
     [0.8694, 0.1988, 0.5470],
     [0.2624, 0.7959, 0.4091]],

    [[0.5090, 0.6053, 0.4071],
     [0.9757, 0.0732, 0.6213],
     [0.7061, 0.1807, 0.2822],
     [0.9533, 0.2438, 0.4031],
     [0.2861, 0.0474, 0.8349],
     [0.6735, 0.4013, 0.8896],
     [0.6121, 0.5824, 0.5803],
     [0.6441, 0.5859, 0.4683]],

    [[0.6844, 0.1701, 0.4609],
     [0.9887, 0.4725, 0.7511],
     [0.9126, 0.4821, 0.7731],
     [0.7827, 0.8101, 0.6258],
     [0.3928, 0.4773, 0.7892],
     [0.2094, 0.3109, 0.9281],
     [0.7335, 0.2065, 0.6204],
     [0.7375, 0.7601, 0.1405]]])

Output:
Possible method using torch.gather
You can see this problem as reassigning each element to a new position. This has to be done using a tensor containing the indices of the permutation you which to see happening.
If you look at the indices of the last dimension for input x (we will take your example with x.shape = (4, 8, 6)), you have them ordered this way:
tensor([[[0, 1, 2, 3, 4, 5],
         [0, 1, 2, 3, 4, 5],
          ... 4 more
         [0, 1, 2, 3, 4, 5],
         [0, 1, 2, 3, 4, 5]],

        ... 2 more

        [[0, 1, 2, 3, 4, 5],
         [0, 1, 2, 3, 4, 5],
          ... 4 more
         [0, 1, 2, 3, 4, 5],
         [0, 1, 2, 3, 4, 5]]])

Now the permutation of indices should be looking like (considering N=4, present_size=1, and past_size=2). Keep in mind I'm only representing two dimensions among the four x in total:
tensor([[0, 1, 2],
        [1, 2, 3],
        [2, 3, 4],
        [3, 4, 5]])

From there it will be easy to construct the new tensor using torch.gather. The operation will effectively create a tensor out defined in the following way:
out[i][j][k][l] = x[i][j][k][indices[i, j, k, l]]

1. Constructing the tensor of indices
In order to construct such tensor of indices, we will use arrangements. The following are the base indices:
>>> arr = torch.arange(total_size)[None].repeat(N, 1)
tensor([[0, 1, 2],
        [0, 1, 2],
        [0, 1, 2],
        [0, 1, 2]])

to which we add a displacement of present_size accumulated over the rows:
>>> disp = torch.arange(0, total_size + 1, step=present_size)[None].T
tensor([[0],
        [1],
        [2],
        [3]])

The resulting minimal tensor of indices is:
>>> indices = arr + disp
tensor([[0, 1, 2],
        [1, 2, 3],
        [2, 3, 4],
        [3, 4, 5]])

2. Applying torch.gather
First, we need to expand the rows of x to N: the number of rows in the resulting tensor.
>>> x_r = x[None].expand(N, *(-1,)*x.ndim)
>>> x.shape, x_r.shape
(torch.Size([4, 8, 6]), torch.Size([4, 4, 8, 6]))

In order to use torch.gather, we need the input and tensor of indices to have the same shape. To do so we can make views of our tensors using Tensor.expand.
So here we will insert two additional dimensions on indices and expand them to match the sizes of x's first and second axis.
>>> i_r = indices[:, None, None, :].expand(-1, x.size(0), x.size(1), -1)
indices.shape, i_r.shape
(torch.Size([4, 3]), torch.Size([4, 4, 8, 3]))

Then apply the gather function on the last axis of indices:
>>> torch.gather(x_r, dim=-1, index=i_r)
tensor([[[[0.5667, 0.5300, 0.2460],
          [0.0360, 0.6687, 0.0167],
          [0.3703, 0.4884, 0.1505],
          [0.2932, 0.4928, 0.3933],
          [0.6667, 0.2014, 0.7107],
          [0.5285, 0.4150, 0.2557],
          [0.7971, 0.6609, 0.1811],
          [0.6902, 0.3999, 0.8719]],

         ...

        [[0.6844, 0.1701, 0.4609],
         [0.9887, 0.4725, 0.7511],
         [0.9126, 0.4821, 0.7731],
         [0.7827, 0.8101, 0.6258],
         [0.3928, 0.4773, 0.7892],
         [0.2094, 0.3109, 0.9281],
         [0.7335, 0.2065, 0.6204],
         [0.7375, 0.7601, 0.1405]]]])

If you have any questions, please don't hesitate to ask!


Instruction: libtorch and pytorch cannot be installed simultaneously?

Input:
I am learning to develop with PyTorch as well as LibTorch. I have the following line in my ~/.bashrc for dynamic linking of libtorch libraries:
# libtorch linking path
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/.dev_libraries/libtorch/lib/

However, when this path is in LD_LIBRARY_PATH, importing torch in Python reports segmentation fault:
user@host:~$ $LD_LIBRARY_PATH
bash: /home/user/packages/embree-2.16.0.x86_64.linux/lib:/home/user/packages/embree-2.16.0.x86_64.linux/lib::/usr/local/lib/:/usr/local/cuda-11.1/lib64:/usr/local/lib/:/usr/local/cuda-11.1/lib64:/home/user/.dev_libraries/libtorch-cpu/libtorch/lib/: No such file or directory
user@host:~$ python
Python 3.8.10 (default, Jun  2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Segmentation fault (core dumped)
user@host:~$

As soon as I remove that path from the environment variable LD_LIBRARY_PATH, torch can be correctly imported in Python.
I am guessing the cause is that some shared libraries of PyTorch having the same names as the ones in LibTorch. Does this mean PyTorch and LibTorch cannot be installed simultaneously, or is my environment setting incorrect? I'd prefer not to reset LD_LIBRARY_PATH every time I switch between the two.

System specs:

Ubuntu 20.04 + CUDA 11.1 + python 3.8.10 + GCC 9.3.0
pytorch 1.9.0+cu111
libtorch is downloaded from here: https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip

Output:
I faced the same problem too.
You can type
import torch
print(torch.__version__)

to see the version of torch, and use the same version of libtorch, that would solve the problem probably.