Instruction: Split features, preprocess some of them, then join them back together. (hangs forever)

Input:
I'm trying to feed the all features (except the first one) to some layers (nn.Linear + nn.LeakyReLU), get the output, then reassemble the initial data structure and feed it to the last layers. But the training process just hangs forever and I don't get any output.
To be clear, the code works fine without this, but I'm trying to improve the results by preprocessing some of the features before feeding them (with the unprocessed first feature) to the last layer.
Any help would be much appreciated.
Here is my code:
def forward(self, x):
        # save the residual for the skip connection
        res = x[:, :, 0:self.skip]

        xSignal = np.zeros((len(x),len(x[0]),1))
        xParams = np.zeros((len(x),len(x[0]),len(x[0][0])-1))

        # separate data
        for b in range(len(x)):
            for c in range(len(x[b])):
                for d in range(len(x[b][c])):
                    if d == 0:
                        xSignal[b][c][d] = x[b][c][d]
                    else:
                        xParams[b][c][d-1] = x[b][c][d]

        # pass parameters through first network
        xParams = torch.from_numpy(xParams).cuda().float()
        xParams = self.paramsLinear(xParams)
        xParams = self.paramsLeakyRelu(xParams)

        # make new array with output and the signal
        xConcat = np.zeros((len(x),len(x[0]),len(x[0][0])))
        for b in range(len(x)):
            for c in range(len(x[b])):
                for d in range(len(x[b][c])):
                    if d == 0:
                        xConcat[b][c][d] = xSignal[b][c][d]
                    else:
                        xConcat[b][c][d] = xParams[b][c][d-1]

        # convert to tensor
        xConcat = torch.from_numpy(xConcat).cuda().float()

        # pass it through the recurrent part
        xConcat, self.hidden = self.rec(xConcat, self.hidden)
        # then the linear part and return
        return self.lin(xConcat) + res```

Output:
Well, as it turns out, slicing is WAY faster and easier than iterating. And I  also used torch.cat function to put everything back in one tensor.
    def forward(self, x):
        # save the residual for the skip connection
        res = x[:, :, 0:self.skip]

        # split features
        xSignal = x[:, :, 0:1]
        xParams = x[:, :, 1:]

        # pass only some features through first layers
        xParams = self.paramsLinear(xParams)
        xParams = self.paramsLeakyRelu(xParams)

        # put everything back together
        x = torch.cat((xSignal, xParams), 2)

        # pass it through the last layers
        x, self.hidden = self.rec(x, self.hidden)
        # then the linear part and return
        return self.lin(x) + res

It's now training as expected :)


Instruction: RuntimeError: Input type and weight type should be the same

Input:
I’m trying to swap resNet blocks with resNext blocks in my current model. All worked and I even trained the model for 1000+ epochs with the resNet blocks but when I added the following class to the model, it returned this error. (ran without errors in my local CPU but got the error when running in colab)
Added Class :
class GroupConv1D(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size, padding, stride, groups):
    super(GroupConv1D, self).__init__()

    if not in_channels % groups == 0:
        raise ValueError("The input channels must be divisible by the no. of groups")
    if not out_channels % groups == 0:
        raise ValueError("The output channels must be divisible by the no. of groups")

    self.kernel_size = kernel_size
    self.stride = stride
    self.padding = padding
    self.groups = groups

    self.group_in_num = in_channels // groups
    self.group_out_num = out_channels // groups
    self.conv_list = []

    for i in range(self.groups):
        self.conv_list.append(
            nn.Conv1d(
                in_channels=self.group_out_num,
                out_channels=self.group_out_num,
                kernel_size=kernel_size,
                stride=stride,
                padding=padding)
        )

def forward(self, inputs):
    feature_map_list = []
    for i in range(self.groups):
        x_i = self.conv_list[i](
            inputs[:, i * self.group_in_num: (i + 1) * self.group_in_num]
        )
        feature_map_list.append(x_i)

    out = torch.concat(feature_map_list, dim=1)
    return out

The Error :
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
  "__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
  exec(code, run_globals)
File "/content/drive/MyDrive/FYPprototypeTest2/train.py", line 268, in <module>
  cycleGAN.trainModel()
File "/content/drive/MyDrive/FYPprototypeTest2/train.py", line 140, in trainModel
  B_fake = self.A_generator_B(A_real, A_mask)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in
_call_impl
  return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/FYPprototypeTest2/model.py", line 235, in forward
  resnet_block_1 = self.resnet_block_1(conv2d_conv1d)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in
_call_impl
  return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/FYPprototypeTest2/model.py", line 88, in forward
  group_layer = self.groupConv_1(layer_one_GLU)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in
_call_impl
  return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 141, in
forward
  input = module(input)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in
_call_impl
  return forward_call(*input, **kwargs)
File "/content/drive/MyDrive/FYPprototypeTest2/model.py", line 46, in forward
  inputs[:, i * self.group_in_num: (i + 1) * self.group_in_num]
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in
_call_impl
  return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 301, in forward
  return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 298, in
_conv_forward
  self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should
be the same

Help would be hugely appreciated.

Output:
Your problem in your new class GroupConv1D is that you store all your convolution modules in a regular python list self.conv_list instead of using nn Containers.
All methods that affect nn.Modules (e.g., .to(device), .eval(), etc.) are applied recursively to all relevant members of the "root" nn.Module.
However, how can pytorch tell which are the relevant members?
For this you have containers: they group together sub-modules, registers and parameters such that pytorch can recursively apply all relevant nn.Module's methods to them.
See, e.g., this answer.


Instruction: Can't connect to GPU when building PyTorch projects

Input:
Before this, I was able to connect to the GPU through CUDA runtime version 10.2. But then I ran into an error when setting up one of my projects.
Using torch 1.10.1+cu102 (NVIDIA GeForce RTX 3080)
UserWarning:
NVIDIA GeForce RTX 3080 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.

After some readings, it seems that sm_86 is only available for CUDA version 11.0 and above. That's the reason why I upgraded to the latest CUDA version and can't connect to the GPU after this.
I have tried many ways, reinstalling cuda toolkit, PyTorch, torchvision and stuff but nothing works.
CUDA Toolkit I've used:
$ wget https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run
$ sudo sh cuda_11.6.0_510.39.01_linux.run

PyTorch I've installed (tried both conda and pip):
$ conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

$ pip3 install torch==1.10.1+cu113 torchvision==0.11.2+cu113 torchaudio==0.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

These are some basic info:
(base) ubuntu@DESKTOP:~$ python
Python 3.9.5 (default, Jun  4 2021, 12:28:51)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.10.1+cu113'
>>> x = torch.rand(6,6)
>>> print(x)
tensor([[0.0228, 0.3868, 0.9742, 0.2234, 0.5682, 0.7747],
        [0.2643, 0.3911, 0.3464, 0.5072, 0.4041, 0.4268],
        [0.2247, 0.0936, 0.4250, 0.1128, 0.0261, 0.5199],
        [0.0224, 0.7463, 0.1391, 0.8092, 0.3742, 0.2054],
        [0.3951, 0.4205, 0.6270, 0.4561, 0.4784, 0.5958],
        [0.8430, 0.5078, 0.7759, 0.5266, 0.4925, 0.7557]])
>>> torch.cuda.get_arch_list()
[]
>>> torch.cuda.is_available()
False
>>> torch.version.cuda
'11.3'
>>> torch.cuda.device_count()
0

Below are my configurations.
(base) ubuntu@DESKTOP:~$ ls -l /usr/local/ | grep cuda
lrwxrwxrwx  1 root root   21 Jan 24 13:47 cuda -> /usr/local/cuda-11.3/
lrwxrwxrwx  1 root root   25 Jan 17 10:52 cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x 17 root root 4096 Jan 24 13:48 cuda-11.3
drwxr-xr-x 18 root root 4096 Jan 24 10:17 cuda-11.6

ubuntu version:
(base) ubuntu@DESKTOP:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

nvidia-smi:
(base) ubuntu@DESKTOP:~$ nvidia-smi
Mon Jan 24 17:22:42 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.39.01    Driver Version: 511.23       CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:02:00.0 Off |                  N/A |
|  0%   26C    P8     5W / 320W |    106MiB / 10240MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      4009      G   /Xorg                           N/A      |
|    0   N/A  N/A      4025      G   /xfce4-session                  N/A      |
|    0   N/A  N/A      4092      G   /xfwm4                          N/A      |
|    0   N/A  N/A     25903      G   /msedge                         N/A      |
+-----------------------------------------------------------------------------+

nvcc --version:
(base) ubuntu@DESKTOP:~$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Mar_21_19:15:46_PDT_2021
Cuda compilation tools, release 11.3, V11.3.58
Build cuda_11.3.r11.3/compiler.29745058_0

Output:
I'm answering my own question.
PyTorch pip wheels and Conda binaries ship with the CUDA runtime.
But CUDA does not normally come with NVCC, and requires to install separately from conda-forge/cudatoolkit-dev, which is very troublesome during the installation.
So, what I did was that I install NVCC from Nvidia CUDA toolkit.
$ wget https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run

And Conda Pytorch-GPU version
$ conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

As it turns out both installations are not compatible with each other.
Therefore, the step I did to solve this issue:

remove any Conda environments in Ubuntu.
clean the pip list and Conda list until none of any PyTorch, torchvision,  Cuda etc works.
Install Nvidia CUDA toolkit cuda_11.3 first from Nvidia's official website.
Reinstall PyTorch only. $ pip3 install torch==1.10.1+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html


Instruction: Applying non-torch function on loss before calling backward()?

Input: I want to apply a custom non-torch function on the final calculated loss before computing the gradients (calling backward()). An example would be to replace the torch.mean() on the loss vector with a custom pythonic, non-torch mean function. But doing so will break the computation graph. I can not rewrite the custom mean function using torch operators and I am at a loss as how to do this. Any suggestions?

Output:
In pytorch you can easily do this by inheriting from torch.autograd.Function: All you need to do is implement your custom forward() and the corresponding backward() methods. Because I don't know the function you intend to write, I'll demonstrate it by implementing the sine function in a way that works with the automatic differentiation. Note that you need to have a method to compute the derivative of your function with respect to its input to implement the backward pass.
import torch

class MySin(torch.autograd.Function):
    @staticmethod
    def forward(ctx, inp):
        """ compute forward pass of custom function """
        ctx.save_for_backward(inp)  # save activation for backward pass
        return inp.sin()  # compute forward pass, can also be computed by any other library

    @staticmethod
    def backward(ctx, grad_out):
        """ compute product of output gradient with the
        jacobian of your function evaluated at input """
        inp, = ctx.saved_tensors
        grad_inp = grad_out * torch.cos(inp)  # propagate gradient, can also be computed by any other library
        return grad_inp

To use it you can use the function sin = MySin.apply on your input.
There is also another example worked out in the documentation.


Instruction: Adding image size as second input to existing PyTorch model

Input:
I am using pretrained torchvision models in PyTorch and transfer learning to classify my own data set. That is working fine, but I think I could further improve my classification performance. Our images come in different dimensions, all of them are resized to fit the input of my model (e.g. to 224x224 pixels).
However, the original image size often says a lot of the class this image belongs to. So I thought it might help the model to add the original image dimension as second input to the model.
Currently I build my model in PyTorch like this:
model = resnet50(pretrained=True)  # Could be another base model as well
for module, param in zip(model.modules(), model.parameters()):
    if isinstance(module, nn.BatchNorm2d):
        param.requires_grad = False
model.fc = nn.Sequential(
                nn.Linear(2048, 512),
                nn.ReLU(),
                nn.Dropout(0.25),
                nn.Linear(512, 256),
                nn.ReLU(),
                nn.Dropout(0.25),
                nn.Linear(256, num_classes),
            )

Now how would I add another (two-dimensional?) input to that model so that I can feed x and y dimensions of the original image to the model? Also, where does that make most sense - directly into the "beginning" of the model, or better somewhere "in between"?

Output:
One way to inject the data into the model can be directly to the linear layers.
This will have the drawback of not affecting the conv layers.
Note that I injected to the final layer, but this can go in any layer.
model.start = nn.Sequential(
                nn.Linear(2048, 512),
                nn.ReLU(),
                nn.Dropout(0.25),
                nn.Linear(512, 256),
                nn.ReLU(),
                nn.Dropout(0.25),
            )

model.end = nn.Sequential(
                nn.Linear(256 + 2, num_classes),
            )

and your forward should be (pseudocode) something like
def forward(x):
    x1 = model.start(x)
    mid = torch.concatenate([x, extra_2d_data])
    x2 = model.end(mid)
    return x2

See also this


Instruction: Kernel Size for 3D Convolution

Input:
The kernel size of 3D convolution is defined using depth, height and width in Pytorch or TensorFlow. For example, if we consider a CT/MRI image data with 300 slices, the input tensor can be (1,1,300,128,128), corresponding to (N,C,D,H,W). Then, the kernel size can be (3,3,3) for depth, height and width. When doing 3D convolution, the kernel is passed in 3 directions.
However, I was confused if we change the situation from CT/MRI to a colourful video. Let the video has 300 frames, then the input tensor will be (1,3,300,128,128) because of 3 channels for RGB images. I know that for a single RGB image, the kernel size can be 3X3X3 for channels, height and width. But when it comes to a video, it seems both Pytorch and Tensorflow still use depth, height and width to set the kernel size. My question is, if we still use a kernel of (3,3,3), is there a potential fourth dimension for the colour channels?

Output:
Yes.
Actually the convolution operation occurring in a CNN is one dimension higher than its namesake. The channel dimension is always spanned by the entire kernel though, so there's no sliding along the channel dimension. For example, a 2D convolution layer with kernel size set to 5x5 applied to a 3 channel input is actually using a kernel of shape 3x5x5 (assuming channel first notation). Each output channel is the result of convolving the input with a different 3x5x5 kernel, so there is one of these 3x5x5 kernels for each output channel.
This is the same for videos. A 3D convolution layer is actually performing a 4D convolution in the same way. So an input of shape 1x3x300x128x128 with kernel size set to 3x3x3 will actually be performing 4D convolutions with kernels of shape 3x3x3x3.


Instruction: PyTorch - How to process model input in parallel?

Input:
Consider the following simple neural net:
class CustomNN(torch.nn.Module):
    def __init__(self):
        super(CustomNN, self).__init__()

    def forward(self, x):
        sleep(1)
        return x

I am wondering if we can call forward() in parallel. Following the official tutorial, I thought that the following code would work:
x = torch.rand(10, 5).cuda()
futures = [torch.jit.fork(model, x[i,:]) for i in range(10)]
results = [torch.jit.wait(fut) for fut in futures]

I expected this to run in about 1 second, but it still sleeps for the full 10 seconds. Is there any way to call the model in parallel?

Output:
You cannot do this in python, unfortunately, as per torch.jit.fork
To serve TorchScript modules you need to make a C++ application with a proper thread pool – see here
Besides, your code does not include (or you do not show it) the conversion.
You have to wrap the Module with torch.jit.script for it to be scripted (or traced) into ScriptModule
traced_NN = torch.jit.script(CustomNN())

Even still, it won't work, as only pytorch functions (and not even fully), python builtins, and math module is supported in TorchScript (see here)


Instruction: Compute means for different subsets of 1d Tensor

Input:
Imagine the following scenario:
data = torch.Tensor([0.5,0.4,1.2,1.1,0.4,0.4])
indices = torch.Tensor([0,1,1,2,2,2])

What I would like to achieve is the following:
Compute the mean over the subset of samples within data as indexed by indices
subset_means == torch.Tensor([0.5, 0.8, 0.8, 0.63, 0.63, 0.63])

I have not been able to come up with a satisfactory solution so far.

Output:
You can use Tensor.index_put to accumulate values of an array according to some index array. This way you can sum up all values belonging to the same index. In the following snippet I use a separate call with an array of just ones to count the number of occurences of each index, to be able to compute the means from the sums:
import torch
data = torch.tensor([0.5,0.4,1.2,1.1,0.4,0.4])
indices = torch.tensor([0,1,1,2,2,2]).to(torch.long)
# sum groups according to indices
accum = torch.zeros((indices.max()+1, )).index_put((indices,), data, accumulate=True)
# count groups according to indices
cnt = torch.zeros((indices.max()+1,)).index_put((indices,), torch.ones((1,)), accumulate=True)
# compute means and expand according to indices
subset_means = (accum / cnt)[indices]
print(subset_means)
#subset_means == torch.Tensor([0.5, 0.8, 0.8, 0.63, 0.63, 0.63])


Instruction: Batch-wise norm of a tensor

Input:
I have a tensor t of dim n x 3. When I apply torch.linalg.norm it returns one single value. What I need is a batch-wise norm function which will return a tensor with n norms, one for each vector in t.
Thanks for your help.

Output:
It seems the most relevant documentation place is:
https://pytorch.org/docs/stable/generated/torch.linalg.norm.html
In the terminal you could try: python3 and then the following python commands:
>>> from torch import linalg as LA
>>> c = torch.tensor([[1., 2., 3.],
...                   [-1, 1, 4]])
>>> LA.norm(c, dim=0)
tensor([1.4142, 2.2361, 5.0000])
>>> LA.norm(c, dim=1)
tensor([3.7417, 4.2426])

Conclusion:
In your specific case you will need to do:
torch.linalg.norm(t,dim=1)


Instruction: Elegant way to quickly load only a small subset of data in detectron2

Input:
I'm looking for an elegant way to load only a small subset of data in detectron2 in order to speed up the training startup for debugging purposes.
I'm building my own instance segmentation model with detectron2 and running it the usual way:
train_net.py --config-file our_training_config.yaml

But it takes several minutes to load everything...
...
[01/25 13:11:48 d2.data.datasets.coco]: Loading datasets/coco/annotations/instances_train2017.json takes 20.74 seconds.
[01/25 13:11:50 d2.data.datasets.coco]: Loaded 118287 images in COCO format from datasets/coco/annotations/instances_train2017.json
...

I was wondering if there is a parameter/trick/flag which allows one to load only a small subset of examples (say, 100) only to quickly see if all the forward and backward  calls works.
Now it is a bit annoying during the debugging process, since each bug and fix requires another slow run to test if everything works.
Technically one can just cut instances_train2017.json in size, but I believe that there are some less nasty solutions to this problems.

Output:
I had the same problem, then I found RandomSubsetTrainingSampler which comes builtin with detectron2. It allows loading a small fraction of the dataset for training. You can change the config file like that:
DATALOADER:
    SAMPLER_TRAIN: "RandomSubsetTrainingSampler"
    RANDOM_SUBSET_RATIO: 0.1

or you can simply pass a sampler to train loader:
subset_sampler = RandomSubsetTrainingSampler(len(dataset)), 0.1)
build_detection_train_loader(cfg, sampler=subset_sampler)

RANDOM_SUBSET_RATIO is between 0 and 1 so 0.1 means 10% of the training dataset. You can see how it is enabled by default in _train_loader_from_config when building a train loader.
However, it seems that currently there is no such nice way of loading a small part of the validation data using the config file. You can similarly pass a subset sampler to the build_detection_test_loader.


Instruction: How does PyTorch know to which neural network the training loss shall be propagated back if you have multiple neural networks?

Input:
I want to train a neural network with the help of two other neural networks, which are already trained and tested. The input of the network that I want to train is simultaniously inputted to the first static network. The output of the of the network that I want to train is inputted to the second static network. The loss shall be computed on the outputs of the static networks and propagated back to the train network.
# Initialization
var_model_statemapper = NeuralNetwork(9, [('linear', 9), ('relu', None), ('dropout', 0.2), ('linear', 8)])

var_model_panda = NeuralNetwork(9, [('linear', 9), ('relu', None), ('dropout', 0.2), ('linear', 27)])
var_model_panda.load_state_dict(torch.load("panda.pth"))

var_model_ur5 = NeuralNetwork(8, [('linear', 8), ('relu', None), ('dropout', 0.2), ('linear', 24)])
var_model_ur5.load_state_dict(torch.load("ur5.pth"))

var_loss_function = torch.nn.MSELoss()
var_optimizer = torch.optim.Adam(var_model_statemapper.parameters(), lr=0.001)

# Forward Propagation
var_panda_output = var_model_panda(var_statemapper_input)
var_ur5_output = var_model_ur5(var_statemapper_output)
var_train_loss = var_loss_function(var_panda_output, var_ur5_output)

# Backward Propagation
var_optimizer.zero_grad()
var_train_loss.backward()
var_optimizer.step()

You can see that the "var_model_statemapper" is the network that shall be trained. The networks "var_model_panda" and "var_model_ur5" are initialized and their state_dicts are being read from the according ".pth" files, so these networks need to be static. My main question is, which of the networks is updated in the backward propagation? Just the "var_model_statemapper" or all networks? And if the "var_model_statemapper" isn't updated, how do I achive this? And does PyTorch know which network to update just from the initialization of the optimizer?

Output:
Formalizing your pipeline to get a good idea of the setup:
x --- | state_mapper | --> y --- | ur5 | --> ur5_out
 \                                              |
  \                                             ↓
   \--- | panda | --> panda_out ----------- | loss_fn | --> loss

Here is what is happening with lines you provided:
var_optimizer.zero_grad()  # 0.
var_train_loss.backward()  # 1.
var_optimizer.step()       # 2.

Calling zero_grad on an optimizer will clear the cache of all parameter gradients contained in that optimizer. In your case, you have var_optimizer registered with the parameters from var_model_statemapper (the model that you want to optimize).

When you infer loss and backpropagate on it via the backward call, the gradients will propagate through the parameters of all three models.

Then calling step on the optimizer will update the parameters registered in the optimizer you're called it upon. In your case, this means var_optimizer.step() will update all parameters of the model var_model_statemapper alone using the gradients computed in step 1. (namely using the backward call on var_train_loss).

All in all, your current approach will only update the parameters of var_model_statemapper. Ideally, you can freeze models var_model_panda and var_model_ur5 by setting their parameters' requires_grad flag to False. This will save speed on inference and training since their gradients won't be computed and stored during backpropagation.


Instruction: Unable to reload class inherited from nn.Module with importlib

Input:
I am trying to reload a class using importlib, however I am facing an error stating it is not a module, this is an jupyter notebook.
Class code
import torch.nn as nn
import torch.nn.functional as F
import torch

class FeedForwardNeuralNetwork(nn.Module):
    def __init__(self, input_size, layers_data, random_seed=42):
        super(FeedForwardNeuralNetwork, self).__init__()
        torch.manual_seed(random_seed)
        # So that number of dense layers are configurable.
        self.layers = nn.ModuleList()
        for size, activation in layers_data:
            self.layers.append(nn.Linear(input_size, size, bias=False))
            torch.nn.init.xavier_uniform(self.layers[-1].weight)
            self.layers.append(activation)
            input_size = size
    def forward(self, x):
        x = torch.flatten(x, start_dim=1)
        for layer in self.layers:
            x = layer(x)
        return x

Reload code
import importlib
from feed_forward_neural_network import FeedForwardNeuralNetwork

print((type(nn.Module)))
print(type(FeedForwardNeuralNetwork))
importlib.reload(FeedForwardNeuralNetwork)

Error
<class 'type'>
<class 'type'>
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [53], in <module>
      4 print((type(nn.Module)))
      5 print(type(FeedForwardNeuralNetwork))
----> 6 importlib.reload(FeedForwardNeuralNetwork)

File /usr/lib/python3.9/importlib/__init__.py:140, in reload(module)
    134 """Reload the module and return it.
    135
    136 The module must have been successfully imported before.
    137
    138 """
    139 if not module or not isinstance(module, types.ModuleType):
--> 140     raise TypeError("reload() argument must be a module")
    141 try:
    142     name = module.__spec__.name

TypeError: reload() argument must be a module

Environment Details

Python version: 3.9.5
IPython          : 8.0.0
jupyter_client   : 7.1.1
jupyter_core     : 4.9.1

Output:
reload() argument should be a module. You need to write it as
importlib.reload(sys.modules.get(FeedForwardNeuralNetwork.__module__))

<module 'feed_forward_neural_network' from '/home/harshit/Downloads/feed_forward_neural_network/feed_forward_neural_network.py'>


Instruction: Combine features in tensor in pytorch

Input:
I have a tensor with dim 2n x m. I want to compute an output tensor with dim n x m, where the i-th and the i+1-th entry are added together and divided by 2, i.e. (f_0, f_1, f_2, f_3, ...) -> ((f_0+f_1)/2, (f_2+f_3)/2, ...). How can I achieve this without looping over the tensor?
Thanks for your help.

Output:
I would reshape the tensor to (n,2,m) and take the mean of dim 1.
In [7]: x = torch.arange(12).view(4,3).float()

In [8]: x
Out[8]:
tensor([[ 0.,  1.,  2.],
        [ 3.,  4.,  5.],
        [ 6.,  7.,  8.],
        [ 9., 10., 11.]])

In [9]: x.view(2,2,3).mean(dim=1)
Out[9]:
tensor([[1.5000, 2.5000, 3.5000],
        [7.5000, 8.5000, 9.5000]])


Instruction: What makes a pre-trained model in pytorch misclassify an image

Input:
I successfully trained Data Efficient Image Transformer (deit) on cifar-10 dataset with an accuracy of about 95%. However and saved it for later use. I created a separate class to load the model and make inference on just one image. I keep getting different value for prediction every time I run it.
import torch
from models.deit import deit_small_patch16_224

import matplotlib.pyplot as plt
import torch.nn.functional as F
import numpy as np
from PIL import Image
from torchvision.transforms import transforms as transforms
class_names = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

model = deit_small_patch16_224(pretrained=True, use_top_n_heads=8, use_patch_outputs=False)

checkpoint = torch.load("./checkpoint/deit224.t7")
model.load_state_dict(checkpoint, strict=False)
model.head = torch.nn.Linear(in_features=model.head.in_features, out_features=10)
model.eval()

img = Image.open("cats.jpeg")
img_tensor = torch.tensor(np.array(img))/255.0
img_tensor = img_tensor.unsqueeze(0).permute(0, 3, 1, 2)
# print(img_tensor.shape)
with torch.no_grad():
    output = model(img_tensor)
    predicted_class = np.argmax(output)
    print(predicted_class)

Output:
Yes,figured out the error. updated code below
import torch
from models.deit import deit_small_patch16_224
from torch.utils.data import dataset
import torchvision.datasets
import matplotlib.pyplot as plt
import torch.nn.functional as F
import numpy as np
from PIL import Image
from torchvision.transforms import transforms as transforms
class_names = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

model = deit_small_patch16_224(pretrained=True, use_top_n_heads=8, use_patch_outputs=False)

checkpoint = torch.load("./checkpoint/deit224.t7")
state_dict = checkpoint["model"]
new_state_dict = {}
for key in state_dict:
    new_key = '.'.join(key.split('.')[1:])
    new_state_dict[new_key] = state_dict[key]

model.head = torch.nn.Linear(in_features=model.head.in_features, out_features=10)
model.load_state_dict(new_state_dict)
model.eval()

img = Image.open("cats.jpeg")
trans = transforms.ToTensor()
# img_tensor = torch.tensor(np.array(img, dtype=np.float64))/255.0
img_tensor = torch.tensor(np.array(img))/255.0
# img_tensor = torch.tensor(np.array(img))

img_tensor = img_tensor.unsqueeze(0).permute(0, 3, 1, 2)
# print(img_tensor.shape)
with torch.no_grad():
    output = model(img_tensor)
    predicted_class = np.argmax(output)
    print(predicted_class)


Instruction: How does a decaying learning rate schedule with AdamW influence the weight decay parameter?

Input:
According to the Pytorch documentation
https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html
the AdamW optimiser computes at each step the product of the learning rate gamma and the weight decay coefficient lambda. The product
gamma*lambda =: p
is then used as the actual weight for the weight decay step. To see this, consider the second line within the for-loop in the AdamW algorithm:

But what if the learning rate lambda shrinks after each epoch because we use (say) an exponential learning rate decay schedule? Is p consistently computed using the initial learning rate lambda and thus p stays constant during the whole training process? Or does p shrink dynamically as lambda shrinks due to an implicit interaction with the the learning rate decay schedule?
Thanks!

Output:
The function torch.optim._functional.adamw is called each time you step the optimizer using the current parameters of the optimizer (that call occurs at torch/optim/adamw.py:145). This is the function that actually updates the model parameter values. So after a learning-rate scheduler changes the optimizer parameters, the steps afterwards will use those parameters, not the initial ones.
To verify this, the product is recomputed at each step in the code at torch/optim/_functional.py:137.


Instruction: Pytorch: Most computationally and memory efficient way to make a series of concatenations from extracting tensor rows?

Input:
Say that this is my sample tensor
sample = torch.tensor(
    [[2, 7, 3, 1, 1],
        [9, 5, 8, 2, 5],
        [0, 4, 0, 1, 4],
        [5, 4, 9, 0, 0]]
)

I want to have a new tensor, which will consist of concatenations of 2 rows from the sample tensor.
So I have a tensor which contains pairs of the row numbers that I want concatenated into a single row for the new tensor
cat_indices = torch.tensor([[0, 1], [1, 2], [0, 2], [2, 3]])

The current method I am using is this
torch.cat((sample[cat_indices[:,0]], sample[cat_indices[:,1]]), dim=1)

Which gives the desired result
tensor([[2, 7, 3, 1, 1, 9, 5, 8, 2, 5],
        [9, 5, 8, 2, 5, 0, 4, 0, 1, 4],
        [2, 7, 3, 1, 1, 0, 4, 0, 1, 4],
        [0, 4, 0, 1, 4, 5, 4, 9, 0, 0]])

Is this the most memory and computationally efficient method of doing this? I am not sure because I am making two calls to cat_indices, and then I am doing a concatenation operation.
I feel that there should be a way to do this via some sort of view. Perhaps advanced indexing. I've tried things like sample[cat_indices[:,0], cat_indices[:,1]] or sample[cat_indices[0], cat_indices[1]] but I can't make the view come out right.

Output:
What you have should be pretty fast. An alternative is
sample[cat_indices].reshape(cat_indices.shape[0],-1)

You would have to benchmark the performance on your machine though to see which is better.


Instruction: Initialise Pytorch layer with local random number generator

Input:
When writing larger programs that require determinism for random processes, it is generally considered good practice to create function-specific random number generators (rngs) and pass those to the randomness-dependent functions (rather than setting a global seed and have the functions depend on this). See also here.
For example, when I have a function that generates some sample using numpy, I use a rng that I create at the beginning of the script:
# at the beginning of the script
import numpy as np
seed = 5465
rng = np.random.default_rng(seed)
# much later in the script
def generate_sample(rng, size):
  return rng.random(size)
generate_sample(rng, size=5)

I am trying to achieve the same when initialising a torch.nn.Linear layer, i.e. use a pre-defined rng to reproducibly initialise the layer.
Is this (reasonably) possible or am I forced to set the Pytorch global seed via torch.manual_seed() instead?

Output:
Generators are available in Pytorch, and in-place prngs can rely on this input, e.g. the normal distribution generator.
When it comes to torch.nn.Linear, it has no such parameter, as you observed. You can still manually manage the internal state of the global prng by using a decorator that saves the internal state, loads the desired state, execute your nn.Linear, saves its state, and reset the global prng to its original state. By using this, you rely on roch.manual_seed() but also on the state getter and setter functions to reach states further from the seeded states.


Instruction: Image Feature Extraction in PyTorch

Input:
I am going through difficulties to understand this code snippet.
import torch
import torch.nn as nn
import torchvision.models as models

def ResNet152(out_features = 10):
      return getattr(models, "resnet152")(pretrained=False, num_classes = out_features)

def VGG(out_features = 10):
      return getattr(models, "vgg19")(pretrained=False, num_classes = out_features)

In this code segment, features for an input image is extracted by ResNet152 and Vgg19 model. But I have the question, from whether which part of these models the features are being extracted whether the part is last pooling layer or the layer before the classification layer or something else.

Output:
Note that getattr(models, 'resnet152') is equivalent to models.resent152.
Hence, the code below is returning the model itself.
getattr(models, "resnet152")(pretrained=False, num_classes = out_features)
# is same as
models.resnet152(pretrained=False, num_classes = out_features)

Now, if you look at the structure of the model by simply printing it, the last layer is a fully-connected layer, so that is what you're getting as features here.
print(ResNet152())

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
...
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=2048, out_features=10, bias=True)
)

The same is the case for VGG().


Instruction: there is no chage to the dataset even after random affine (pytorch)

Input:
I downloaded two version of mnist dataset: one without any transformation, one with random affinement and normalization.
w/o transformation code:
mnist_train = torchvision.datasets.MNIST(root='MNIST_data/',
                          train=True,
                          transform=transforms.ToTensor(),
                          download=True)

with transformation code:
train_transforms = transforms.Compose(
    [transforms.RandomAffine(degrees=30),
     transforms.ToTensor(),
     transforms.Normalize(mean=(0.5,), std=(0.5,))])

augmented_data = torchvision.datasets.MNIST(root='aug_data/',
                                        train=True,
                                        transform=train_transforms,
                                        download=True)

I printed several samples from those two, but the results were all identical.
can anybody figure out the cause??
thank you

Output:
There is no augmented MNIST dataset and download=True just downloads the same plain MNIST dataset in the roots you have specified.
The transform is applied online when you access an element of the dataset, e.g. augmented_data[0]. In this way, the transformation you have specified is applied to the first image of MNIST.
If you want to check the difference you, therefore, have to display the images accessed in this way with some visualization tool, e.g. by using matplotlib.imshow:
Code
import torch
import torchvision
from torchvision import transforms
import matplotlib.pyplot as plt

mnist_train = torchvision.datasets.MNIST(root='MNIST_data/',
                          train=True,
                          transform=transforms.ToTensor(),
                          download=True)

train_transforms = transforms.Compose(
    [transforms.RandomAffine(degrees=30),
     transforms.ToTensor(),
     transforms.Normalize(mean=(0.5,), std=(0.5,))])

# you can use the same root, since the data is the same
augmented_data = torchvision.datasets.MNIST(root='MNIST_data/',
                                        train=True,
                                        transform=train_transforms,
                                        download=True)

im = mnist_train[0][0][0]
aug_im = augmented_data[0][0][0]

# Normal image
plt.imshow(im, cmap="gray")
plt.colorbar()
plt.show()

# Agumented image
plt.imshow(aug_im, cmap="gray")
plt.colorbar()
plt.show()

Result
Normal Image

Augmented Image

Note the difference of value ranges in the right-sided color bar.


Instruction: How to create batches using PyTorch DataLoader such that each example in a given batch has the same value for an attribute?

Input:
Suppose I have a list, datalist which contains several examples (which are of type torch_geometric.data.Data for my use case). Each example has an attribute num_nodes
For demo purpose, such datalist can be created using the following snippet of code
import torch
from torch_geometric.data import Data # each example is of this type
import networkx as nx # for creating random data
import numpy as np
# the python list containing the examples
datalist = []
for num_node in [9, 11]:
    for _ in range(1024):
        edge_index = torch.from_numpy(
            np.array(nx.fast_gnp_random_graph(num_node, 0.5).edges())
        ).t().contiguous()
        datalist.append(
            Data(
                x=torch.rand(num_node, 5),
                edge_index=edge_index,
                edge_attr=torch.rand(edge_index.size(1))
            )
        )

From the above datalist object, I can create a torch_geometric.loader.DataLoader (which subclasses torch.utils.data.DataLoader) naively (without any constraints) by using the DataLoader constructor as:
from torch_geometric.loader import DataLoader
dataloader = DataLoader(
    datalist, batch_size=128, shuffle=True
)

My question is, how can I use the DataLoader class to ensure that each example in a given batch has the same value for num_nodes attribute?
PS:
I tried to solve it and came up with a hacky solution by combining multiple DataLoader objects using the combine_iterators function snippet from here as follows:
def get_combined_iterator(*iterables):
    nexts = [iter(iterable).__next__ for iterable in iterables]
    while nexts:
        next = random.choice(nexts)
        try:
            yield next()
        except StopIteration:
            nexts.remove(next)

datalists = defaultdict(list)
for data in datalist:
    datalists[data.num_nodes].append(data)
dataloaders = (
    DataLoader(data, batch_size=128, shuffle=True) for data in datalists.values()
)
batches = get_combined_iterator(*dataloaders)

But, I think that there must be some elegant/better method of doing it, hence this question.

Output: If your underlying dataset is map-style, you can use define a torch.utils.data.Sampler which  returns the indices of the examples you want to batch together. An instance of this will be passed as a batch_sampler kwarg to your DataLoader and you can remove the batch_size kwarg as the sampler will form batches for you depending on how you implement it.


Instruction: What is different between DataLoader and DataLoader2 in PyTorch?

Input:
I developed a custom dataset by using the PyTorch dataset class. The code is like that:
class CustomDataset(torch.utils.data.Dataset):

    def __init__(self, root_path, transform=None):
        self.path = root_path
        self.mean = mean
        self.std = std
        self.transform = transform
        self.images = []
        self.masks = []

        for add in os.listdir(self.path):
            # Some script to load file from directory and appending address to relative array
            ...

        self.masks.sort()
        self.images.sort()

    def __len__(self):
        return len(self.images)

    def __getitem__(self, item):
        image_address = self.images[item]
        mask_address = self.masks[item]

        if self.transform is not None:
            augment = self.transform(image=np.asarray(Image.open(image_address, 'r', None)),
                                     mask=np.asarray(Image.open(mask_address, 'r', None)))
            image = Image.fromarray(augment['image'])
            mask = augment['mask']

        if self.transform is None:
            image = np.asarray(Image.open(image_address, 'r', None))
            mask = np.asarray(Image.open(mask_address, 'r', None))

        # Handle Augmentation here

        return image, mask

Then I created an object from this class and passed it to torch.utils.data.DataLoader. Although this works well with DataLoader but with torch.utils.data.DataLoader2 I got a problem. The error is this:

dataloader = torch.utils.data.DataLoader2(dataset=dataset, batch_size=2, pin_memory=True, num_workers=4)

Exception: thread parallelism mode is not supported for old DataSets

My question is why DataLoader2 module was added to PyTorch what is different with DataLoader and what are its benefits?
PyTorch Version: 1.10.1

Output:
You should definitely not use it DataLoader2.
torch.utils.data.DataLoader2 (actually torch.utils.data.dataloader_experimental.DataLoader2)
was added as an experimental "feature" as a future replacement for DataLoader. It is defined here. Currently, it is only accessible on the master branch (unstable) and is of course not documented on the official pages.


Instruction: Is there a way to send patches of image into a transformer model for inference or combine the patches together to make one image?

Input:
I am making inference with a single image of size 224x224  on a vision transformer model (deit). However, I divided the image into 196 patches and manipulated the pixels of one patch to check its behaviour. Each patch is of size 16x16.
On feeding these patches to the model, I got the error: Input image size (16*16) doesn't match (224*224). Of course the model was trained on 224x224 image and needs same size. An idea is to combine these patches together into one complete image but I am having problem in getting through.
The single image shape :([1, 3, 224, 224])
The divided-into-patches image shape: [196, 16, 16, 3]
import torch
from models.deit import deit_small_patch16_224
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import os
from torchvision.transforms import transforms as transforms
from torchvision.utils import make_grid

def into_patches(im, xPieces, yPieces):
    imgwidth, imgheight = im.size
    height = imgheight // yPieces
    width = imgwidth // xPieces
    #fig, axs = plt.subplots(yPieces, xPieces)
    img_list = []
    for i in range(0, yPieces):
        for j in range(0, xPieces):
            box = (j * width, i * height, (j + 1) * width, (i + 1) * height)
            a = im.crop(box)
            np_img = np.asarray(a)
            if i ==6 and j ==5:
                np_img.setflags(write=1)
                np_img[:] =0
            img_list.append(np_img)
    return img_list

class_names = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

model = deit_small_patch16_224(pretrained=True, use_top_n_heads=8, use_patch_outputs=False)

checkpoint = torch.load("./checkpoint/deit224.t7")
state_dict = checkpoint["model"]
new_state_dict = {}
for key in state_dict:
    new_key = '.'.join(key.split('.')[1:])
    new_state_dict[new_key] = state_dict[key]

model.head = torch.nn.Linear(in_features=model.head.in_features, out_features=10)
model.load_state_dict(new_state_dict)
model.eval()

img = Image.open("bird.jpeg")
img = img.resize((224, 224), resample=0)
a = np.array(into_patches(img, 14, 14))

img_tensor = torch.tensor(a)

# print(img_tensor.shape)
 with torch.no_grad():
     output = model(img_tensor)
     predicted_class = np.argmax(output)
     print(predicted_class.item())

I get the following error:-
AssertionError: Input image size (16*3) doesn't match model (224*224).
Is there anyway to combine these 196 patches back into 224x224 image?

Output:
You can use fold and unfold to extract the patches, manipulate them and then re-arrange them back into an image:
# divide the batch of images into non-overlapping patches
u = nnf.unfold(x, kernel_size=16, stride=16, padding=0)
# manipulate patch number 17
u[..., 17] = my_manipulation(u[..., 17])
# fold the patches back together
f = nnf.fold(u, x.shape[-2:], kernel_size=16, stride=16, padding=0)


Instruction: What values does torch.Conv2D return?

Input:
x = torch.randn(3,64,161,161)\
model = nn.Conv2d(64, 1, kernel_size=1) result = model(x)\
print(result.shape)

output : 3, 1, 161, 161

Output has the first two values as 3 and 1. I understood it as :
3 means the number of input channels that the model has initially received.
1 means the number of output channels after the model dealt with the tensor.
Did I understand correctly?

Output:
No, the number of input channels is 64 (the number of output channels is 1). 3 is the batch size in your code snippet.
The convention PyTorch uses for all tensors with 2D multi-channel data is to have shape [N,C,H,W] where

N is batch dimension
C is channel dimension
H is height dimension
W is width dimension

For Conv2d
N will be the same for both the input and output tensors since each batch element in the input tensor produces one corresponding element in the output tensor.
The input tensor's C must be equal to in_channels. This is the first argument you provided to Conv2d. In your case it is 64 so you need to have an input tensor with 64 channels.
The output tensor's C will be equal to out_channels. This is the second argument you provide to Conv2d. In your case it is 1 so the output tensor will have 1 channel.
W and H are the width and height of the data that is being convolved. The output shape will vary depending on the kernel_size, padding, stride, and dilation arguments of Conv2d. The equation relating input W and H to output W and H is

For more information I recommend you check the official documentation.


Instruction: Retraining a Model from 3 Channels (RGB) to 4 Channels (RGBA), can I use the 3 channel weights?

Input:
I need to expand a model from RGB to RGBA. I can handle the code rewrite on the model, but instead of retraining the entire model from scratch, I would love to start it off with it's 3 channel weights + zeros.
Is there an easy way to change torch's save of 3 channel weights into 4?

Output:
Yes, you can do a little bit of "model surgery". Assuming the input to the model is only processed directly by a convolutional layer then you can just replace that conv layer with another that has in_channels set to 4. Then you can set weights to zero and copy over the old weights (and biases if applicable) from the original conv layer.
For example, say we had a simple model that looked like this
import torch
import torch.nn as nn
import torch.nn.functional as F

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=3, padding=1, bias=True)
        self.conv2 = nn.Conv2d(10, 5, kernel_size=3, padding=1, bias=True)
        self.linear = nn.Linear(125, 1)
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        return self.linear(x.flatten(start_dim=1))

model = SimpleModel()

Supposing that the model is trained at this point, we could perform the surgery as follows
y_rgb = torch.randn(1, 3, 5, 5)

# get performance on initial z_rgb
z_rgb = model(y_rgb)

# perform model surgery
with torch.no_grad():
    new_conv1 = nn.Conv2d(4, 10, kernel_size=3, padding=1, bias=True)
    new_conv1.weight.zero_()
    new_conv1.weight[:,:3,...]=model.conv1.weight
    new_conv1.bias.copy_(model.conv1.bias)
    model.conv1 = new_conv1

# add a random alpha channel to y_rgba
y_alpha = torch.randn(1,1,5,5)
y_rgba = torch.cat([y_rgb, y_alpha], dim=1)

# get results on rgba model
z_rgba = model(y_rgba)

# compare z_rgb and z_rgba, print mean-square difference
z_err = ((z_rgba-z_rgb)**2).mean().item()
print('Err:', z_err)

# save results to a new file
torch.save(model.state_dict(), 'checkpoint_rgba.pt')

which should give you an error of zero or very close to zero.
Of course if you don't have a bias term in your first conv layer then you don't need to copy that over.
Assuming you've saved the new state dictionary, then you will probably want to update the model class definition so that your input convolution layer takes 4 channel input instead of 3. Then next time you can directly load the new state dictionary without additional steps.

Now it's not strictly necessary to do the surgery on the model directly. Though I tend to prefer it as I find it easier to verify correctness.
Assuming you saved off the state dictionary for the RGB model, you could also just directly modify the state dictionary.
# assuming you saved RGB model using torch.save(model.state_dict(), 'checkpoint_rgb.pt')
state_dict = torch.load('checkpoint_rgb.pt')
old_weight = state_dict['conv1.weight']
state_dict['conv1.weight'] = torch.zeros(
    old_weight.shape[0],
    old_weight.shape[1]+1,
    old_weight.shape[2],
    old_weight.shape[3]
).type_as(old_weight)
state_dict['conv1.weight'][:,:3,...] = old_weight
torch.save(state_dict, 'checkpoint_rgba.pt')


Instruction: correct shape (BS,H,W,C) not working in torchvision.utils.save_image

Input:
Let's BS be the batch size, H the height, w the weight, and c the number of channels which is 3 in my case.
When I save my image in this shape (BS,C,H,W) with
torchvision.utils.save_image(image, path)

it works very well but the image is unreadable since the format is wrong.
But when I am reshaping my image into the right format which is (BS,H,W,C), The code below does not work
image = image.reshape(BS,H,W,C)
torchvision.utils.save_image(image, path)

Here is the error that I am stuck in:

TypeError: Cannot handle this data type: (1, 1, 256), |u1

Output:
You wanted your image to have size (BS, C, H, W), but you are incorrectly reshaping it.
Assuming the image.shape is (BS, H, W, C), perhaps you meant to perform image = image.permute(0, 3, 1, 2) which moves the channel dimension in the second position, to obtain the required shape.


Instruction: Cannot figure out dense layers dimensions to run the neural network

Input:
I am trying to build a multi layer neural network. I have train data with shape:
train[0][0].shape
(4096,)

Below is my dense layer
from collections import OrderedDict
n_out = 8
net = nn.Sequential(OrderedDict([
                            ('hidden_linear', nn.Linear(4096, 1366)),
                            ('hidden_activation', nn.Tanh()),
                            ('hidden_linear', nn.Linear(1366, 456)),
                            ('hidden_activation', nn.Tanh()),
                            ('hidden_linear', nn.Linear(456, 100)),
                            ('hidden_activation', nn.Tanh()),
                            ('output_linear', nn.Linear(100, n_out))
                            ]))

I am using crossentropy as the loss function. The problem I have is when I train the model with the below code:
 learning_rate = 0.001
 optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)
 n_epochs = 40

for epoch in range(n_epochs):
    for snds, labels in final_train_loader:
         outputs = net(snds.view(snds.shape[0], -1))
         loss = loss_fn(outputs, labels)

         optimizer.zero_grad()
         loss.backward()
         optimizer.step()

     print("Epoch: %d, Loss: %f" % (epoch, float(loss)))

The error I receive is the matrix multiplication error.
 RuntimeError: mat1 and mat2 shapes cannot be multiplied (100x4096 and 456x100)

I have the dimensions wrong but cannot figure out how to get it right.

Output:
The OrderedDict contains three Linear layers associated with the same key, hidden_layer (the same happens with nn.Tanh). In order to make it work you need to provide such layers with a different name:
inp = torch.rand(100, 4096)
net = nn.Sequential(OrderedDict([
                            ('hidden_linear0', nn.Linear(4096, 1366)),
                            ('hidden_activation0', nn.Tanh()),
                            ('hidden_linear1', nn.Linear(1366, 456)),
                            ('hidden_activation1', nn.Tanh()),
                            ('hidden_linear2', nn.Linear(456, 100)),
                            ('hidden_activation2', nn.Tanh()),
                            ('output_linear', nn.Linear(100, n_out))
                            ]))
net(inp)  # now it works!


Instruction: How to input the image with the size 4x4 and replace the Fully Connected Layers using the Convolution Layers?

Input:
I'm a beginner at the PyTorch library, and I got stuck in an exercise.
The code below works for the input image with size 2x2. I'm trying to do the same thing as below but the input image with size 4x4.
The code:
import torch

Assume that we have a 2x2 input image
inputs = torch.tensor([[[[1., 2.],
                         [3., 4.]]]])
inputs.shape

Output: torch.Size([1,1,2,2]

A fully connected layer, which maps the 4 input features two 2 outputs, would be computed as follows:
fc = torch.nn.Linear(4, 2)

weights = torch.tensor([[1.1, 1.2, 1.3, 1.4],
                        [1.5, 1.6, 1.7, 1.8]])
bias = torch.tensor([1.9, 2.0])
fc.weight.data = weights
fc.bias.data = bias
torch.relu(fc(inputs.view(-1, 4)))

Output: torch.Size([2, 1, 2, 2])
Output: torch.Size([2])

Obtain the same outputs if we use convolutional layers where the kernel size is the same size as the input feature array:
conv = torch.nn.Conv2d(in_channels=1,
                       out_channels=2,
                       kernel_size=inputs.squeeze(dim=(0)).squeeze(dim=(0)).size())
print(conv.weight.size())
print(conv.bias.size())

Output: torch.Size([2, 1, 2, 2])
Output: torch.Size([2])
conv.weight.data = weights.view(2, 1, 2, 2)
conv.bias.data = bias
torch.relu(conv(inputs))

Output: tensor([[[[14.9000]],

         [[19.0000]]]], grad_fn=<ReluBackward0>)

Replace the fully connected layer using a convolutional layer when we reshape the input image into a num_inputs x 1 x 1 image:
conv = torch.nn.Conv2d(in_channels=4,
                       out_channels=2,
                       kernel_size=(1, 1))

conv.weight.data = weights.view(2, 4, 1, 1)
conv.bias.data = bias
torch.relu(conv(inputs.view(1, 4, 1, 1)))

Output: tensor([[[[14.9000]],

         [[19.0000]]]], grad_fn=<ReluBackward0>)

So based on this code how to input an image that has a size 4x4 and replace the Fully Connected Layers using Convolution Layers?

Output:
You simply need to change the shape of input and reshape weights as per 4x4.
inputs = torch.randn(1, 1, 4, 4)
fc = torch.nn.Linear(16, 2)
torch.relu(fc(inputs.view(-1, 16)))

# output
tensor([[0.0000, 0.2525]], grad_fn=<ReluBackward0>)

Now, for conv layer
conv = torch.nn.Conv2d(in_channels=1,
                       out_channels=2,
                       kernel_size=inputs.squeeze(dim=(0)).squeeze(dim=(0)).size())
conv.weight.data = fc.weight.data.view(2, 1, 4, 4)
conv.bias.data = fc.bias.data
torch.relu(conv(inputs))

# output
tensor([[[[0.0000]],
         [[0.2525]]]], grad_fn=<ReluBackward0>)

You can read Converting FC layers to CONV layers if not sure how conv layers params are taken.


Instruction: Create a torch tensor with desired values

Input:
I want to create a torch tensor of size 100 with values 10 and 100.
For example: The following gives a tensor of values between 5 and 6.
torch.randint(5,7,(100,))

tensor([6, 6, 6, 5, 5, 6, 6, 6, 6, 5, 6, 6, 6, 6, 6, 5, 6, 5, 5, 6, 5, 5, 5, 5,
        6, 5, 5, 5, 5, 5, 6, 6, 6, 5, 6, 6, 5, 5, 5, 5, 6, 5, 5, 5, 5, 5, 6, 5,
        5, 6, 5, 6, 5, 6, 5, 6, 6, 6, 6, 5, 6, 6, 6, 5, 5, 5, 6, 6, 6, 6, 5, 6,
        5, 5, 5, 5, 6, 6, 5, 6, 6, 6, 5, 5, 6, 6, 5, 6, 6, 6, 5, 5, 5, 5, 5, 6,
        6, 6, 5, 6])

Instead of this, I want a tensor with values 10 and 100 and I do not want the values between the integers 10 and 100. Tensor should just contain 10 and 100. How do I do that?
Thanks in advance.

Output:
You can achieve that by using the python function random.choice() to create a list of random numbers then convert it to a tensor:
import random
import torch
list_numbers = random.choices([100,10], k=100)

random_numbers = torch.Tensor(list_numbers)

print(random_numbers)


Instruction: How to fix error with pytorch conv2d function?

Input:
I am trying to use conv2d function on these two tensors:
Z = np.random.choice([0,1],size=(100,100))
Z = torch.from_numpy(Z).type(torch.FloatTensor)

print(Z)

tensor([[0., 0., 1.,  ..., 1., 0., 0.],
        [1., 0., 1.,  ..., 1., 1., 1.],
        [0., 0., 0.,  ..., 0., 1., 1.],
        ...,
        [1., 0., 1.,  ..., 1., 1., 1.],
        [1., 0., 1.,  ..., 0., 0., 0.],
        [0., 1., 1.,  ..., 1., 0., 0.]

and
filters = torch.tensor(np.array([[1,1,1],
                        [1,0,1],
                        [1,1,1]]), dtype=torch.float32)

print(filters)

tensor([[1., 1., 1.],
        [1., 0., 1.],
        [1., 1., 1.]])

But when I try to do torch.nn.functional.conv2d(Z,filters) this error returns:
RuntimeError: weight should have at least three dimensions
I really don't understand what is the problem here. How to fix it?

Output:
The input to torch.nn.functional.conv2d(input, weight) should be

You can use unsqueeze() to add fake batch and channel dimensions thus having sizes: input: (1, 1, 100, 100) and weight: (1, 1, 3, 3).
torch.nn.functional.conv2d(Z.unsqueeze(0).unsqueeze(0), filters.unsqueeze(0).unsqueeze(0))


Instruction: Run out of memory trying to create a tensor of size [2191, 512] with pytorch to save data from movie frames using CLIP

Input:
I'm using pytorch for the first time and I'm facing a problem I don't think I should. I currently have selected 2919 frames of a movie in jpg. I'm trying to transform all those images into a single tensor. I'm using CLIP to transform each image into a tensor of size [1, 512]. In the end, I expected to have a tensor of size [2919, 512], which should not use that much memory. But my code never finishes running and I can only assume I'm doing something terribly wrong.
First I'm doing my import and loading the model:
import torch
import clip
from glob import glob
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

Secondly, I'm reading the path for all the images and initializing the "film" tensor with random value to overwrite them. I tried generating an empty one and concatenate but that also consumed too much memory:
path_names = glob(r"Films/**/*.jpg")
film = torch.rand((len(files), 512), dtype=torch.float32, device = device)
film_frame_count = 0
    for file in files:
        print("Frame " + str(film_frame_count) + " out of " + str(len(files)))
        film[film_frame_count] = model.encode_image(preprocess(Image.open(file)).unsqueeze(0).to(device))[0]
        film_frame_count += 1
    torch.save(film, 'output_tensor/'+ film_code[1])

If anyone could point it out what I'm doing wrong I would appreciate.

Output:
The problem ended up being caused because pytorch was saving the gradiants for the graph, so I needed to indicate that I didn't want them to be stored with this indicator on top:
with torch.no_grad():
    /*my code*/


Instruction: Pytorch `bachward()` updates multiple models

Input:
Can anyone tell me why the gradients of the discriminator change as well and if there is a way to avoid it?
for i in range(2):

    X_fake = gen_model(z)

    pred_real = disc_model(X)
    pred_fake = disc_model(X_fake.detach())
    disc_loss = (loss_fn(pred_real, y) + loss_fn(pred_fake, y)) / 2

    disc_optimizer.zero_grad()
    disc_loss.backward()
    disc_optimizer.step()

    pred_fake = disc_model(X_fake)
    gen_loss = loss_fn(pred_fake, y)

    gen_optimizer.zero_grad()
    i == 1 and print_grads(disc_model) # Checkpoint 1
    gen_loss.backward()
    i == 1 and print_grads(disc_model) # Checkpoint 2
    gen_optimizer.step()

This is the rest of the code.
import torch
import torch.nn as nn

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self._linear = nn.Sequential( nn.Linear(1, 5) )

    def forward(self, X):
        return self._linear(X)

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self._linear = nn.Sequential( nn.Linear(5, 1) )

    def forward(self, X):
        return self._linear(X)

def print_grads(model):
    for params in model.parameters():
        print(params.grad)

# Build the model and data.
gen_model = Generator()
gen_optimizer = torch.optim.Adam(gen_model.parameters(), 1)

disc_model = Discriminator()
disc_optimizer = torch.optim.Adam(disc_model.parameters(), 1)

loss_fn = torch.nn.BCEWithLogitsLoss()

z = torch.rand((1, 1))
X = torch.rand((1, 5))
y = torch.rand((1, 1))

Output:
The gradients of the discriminator have been updated because you have backpropagated the loss gen_loss through the discriminator up to the generator itself. In your current formulation of the training loop, this is not a problem since only the generator's parameters will get updated on the next line with gen_optimizer.step(). In other words: yes the gradients of the discriminator will be updated (they change from #1 to #2) but these won't be used to update the parameters of the discriminator. As long as you properly clear the gradient cache with zero_grad before backpropagating, you will be fine.
A common practice is to freeze the discriminator when training the generator, this avoids unnecessary gradient computation:
# freeze discriminator
disc_model.requires_grad_(False)

# train the generator
gen_optimizer.zero_grad()
gen_loss.backward()
gen_optimizer.step()

# unfreeze discriminator for next step
disc_model.requires_grad_(True)


Instruction: Torchscript/C++ jit::trace model - Accessing layers parameters

Input:
I have a model I trained in python, traced using torch.jit.trace, and load into C++ using torch::jit::load.
Is there a way to access the last layer to pull the value for the models required output depth (for example, if it is a Conv2D layer going from 16 -> 2, I want to predefine a tensor for a depth [b,d->2,x,y] of 2)?

Output: Not the most elegant way of solving this, but the most straightforward was just passing a dummy tensor through and accessing the shape. Another way I did try was accessing the parameter list and looking for "softmax", unfortunately I couldn't guarantee everyones model will spell it the same way when searching for this. If someone else has a good answer for this feel free to share, but this will have to do for now.


Instruction: Normalize MNIST in PyTorch

Input:
I am trying to normalize MNIST dataset in PyTorch 1.9 and Python 3.8 to be between the range [0, 1] with the code (batch_size = 32).
# Specify path to MNIST dataset-
path_to_data = "path_to_dataset"

# Define transformation(s) to be applied to dataset-
transforms_MNIST = transforms.Compose(
    [
        transforms.ToTensor(),
        transforms.Normalize(mean = (0.1307,), std = (0.3081,))
    ]
)

# Load MNIST dataset-
train_dataset = torchvision.datasets.MNIST(
        # root = './data', train = True,
        root = path_to_data + "data", train = True,
        transform = transforms_MNIST, download = True
        )

test_dataset = torchvision.datasets.MNIST(
        # root = './data', train = False,
        root = path_to_data + "data", train = False,
        transform = transforms_MNIST
        )

# Create training and testing dataloaders-
train_loader = torch.utils.data.DataLoader(
        dataset = train_dataset, batch_size = batch_size,
        shuffle = True
        )

test_loader = torch.utils.data.DataLoader(
        dataset = test_dataset, batch_size = batch_size,
        shuffle = False
        )

print(f"Sizes of train_dataset: {len(train_dataset)} and test_dataet: {len(test_dataset)}")
print(f"Sizes of train_loader: {len(train_loader)} and test_loader: {len(test_loader)}")
# Sizes of train_dataset: 60000 and test_dataet: 10000
# Sizes of train_loader: 1875 and test_loader: 313

# Sanity check-
print(f"train_dataset: min pixel value = {train_dataset.data.min().numpy():.3f} &"
      f" max pixel value = {train_dataset.data.max().numpy():.3f}")
# train_dataset: min pixel value = 0.000 & max pixel value = 255.000

print(f"test_dataset: min pixel value = {test_dataset.data.min().numpy():.3f} &"
      f" max pixel value = {test_dataset.data.max().numpy():.3f}")
# test_dataset: min pixel value = 0.000 & max pixel value = 255.000

print(f"len(train_loader) = {len(train_loader)} & len(test_loader) = {len(test_loader)}")
# len(train_loader) = 1875 & len(test_loader) = 313

# Sanity check-
len(train_dataset) / batch_size, len(test_dataset) / batch_size
# (1875.0, 312.5)

# Get some random batch of training images & labels-
images, labels = next(iter(train_loader))

# You get x images due to the specified batch size-
print(f"images.shape: {images.shape} & labels.shape: {labels.shape}")
# images.shape: torch.Size([32, 1, 28, 28]) & labels.shape: torch.Size([32])

# Get min and max values for normalized pixels in mini-batch-
images.min(), images.max()
# (tensor(-0.4242), tensor(2.8215))

The min and max for 'images' should be between 0 and 1, instead, it is 0.4242 and 2.8215. What is going wrong?

Output:
This happens because Normalize applies what is actually known (also) as a standardization: output = (input - mean) / std.
The normalization you want to achieve is automatically performed when loading the image so you can comment Normalize.


Instruction: Transform not getting applied on CustomDataset Pytorch

Input:
I images with folder structure as following :
root_dir
│
└───folder1
│   │   file011.png
│   │   file012.png
│
└───folder2
|   │   file021.png
|   │   file022.png
|
└───folder2
    │   file031.png
    │   file032.png
...

Now I wanted to create a CustomeDataset without labels in PyTorch (since I am using it for GANs)
So I did the following :
class CurrencyDataset(Dataset):
    '''
    Currency Dataset with no labels
    '''
    def __init__(self, type, transform):
        '''
        Parameters
            type : "Train" or "Test"
            transform : Transformations to be applied
        '''
        root_dir = "indian-currency-notes-classifier/"
        # Storing images in a list
        self.data = []
        self.transform = transform
        dir = os.path.join(root_dir, type)
        for note in os.listdir(dir) :
            note_dir = os.path.join(dir, note)
            for img_name in os.listdir(note_dir):
                img = io.imread(os.path.join(note_dir, img_name))
                self.data.append(img)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        x = self.data[idx]
        if self.transform :
            x = self.transform(x)
        return x

and used the following transformations :
transform = transforms.Compose([
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
    transforms.RandomRotation(15),
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

train_ds = CurrencyDataset("Train", transform)

but on checking the shape and values of the Tensors, I found out that transformations were not getting applied
train_ds.data[0].shape
>> (1072, 1154, 3)

I am a bit new to PyTorch so please let me know If I am doing something wrong here or What needs to be done to make it correct ?

Output:
You forgot to assign the transform object as an attribute of the instance. This, in turn, means self.transform evaluates to None in the __getitem__ function. Simply add the following in the __init__:
self.transform = transform

Additonally, you are not calling the proper function (__getitem__) with train_ds.data[0].shape, instead it should be train_ds[0] (in other words: train_ds.__getitem__(0)).

Finally, your pipeline doesn't have the correct order of transforms as T.Resize expects a torch.Tensor, not a PIL image:
transform = T.Compose([
    T.RandomRotation(15),
    T.Resize((224, 224)),
    T.ToTensor(),
    T.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])])


Instruction: Skip bad data points when loading data using DataLoader

Input:
I am trying to perform an image classification task using mini-imagenet dataset. The data that I want to use, contains a few bad data points(I am not sure why). I would like to load this data and train my model on it. In the process, I want to skip the bad data points completely. How do I do this?
The data loader I am using is as follows:
class MiniImageNet(Dataset):

    def __init__(self, root, train=True,
                transform=None,
                index_path=None, index=None, base_sess=None):
        if train:
            setname = 'train'
        else:
            setname = 'test'
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.train = train  # training set or test set
        self.IMAGE_PATH = os.path.join(root, 'miniimagenet/images')
        self.SPLIT_PATH = os.path.join(root, 'miniimagenet/split')

        csv_path = osp.join(self.SPLIT_PATH, setname + '.csv')
        lines = [x.strip() for x in open(csv_path, 'r').readlines()][1:]

        self.data = []
        self.targets = []
        self.data2label = {}
        lb = -1

        self.wnids = []

        for l in lines:
            name, wnid = l.split(',')
            path = osp.join(self.IMAGE_PATH, name)
            if wnid not in self.wnids:
                self.wnids.append(wnid)
                lb += 1
            self.data.append(path)
            self.targets.append(lb)
            self.data2label[path] = lb

        self.y = self.targets

        if train:
            image_size = 84
            self.transform = transforms.Compose([
                transforms.RandomResizedCrop(image_size),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                    std=[0.229, 0.224, 0.225])])

        else:
            image_size = 84
            self.transform = transforms.Compose([
                transforms.Resize([image_size, image_size]),
                transforms.CenterCrop(image_size),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                    std=[0.229, 0.224, 0.225])])

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):
        path, targets = self.data[i], self.targets[i]
        image = self.transform(Image.open(path).convert('RGB'))
        return image, targets

I tried to use a try-except sequence, but in that case, instead of skipping, the dataloader is returning None, causing an error. How do I completely skip a datapoint in a dataloader?

Output:
Try removing the bad data at the end of the __init__ function.
for i in range(len(self.data) - 1, -1, -1):
    if is_bad_data(self.data[i], self.targets[i]):
        del self.data[i]
        del self.targets[i]


Instruction: Pytorch features and classes from .npy files

Input:
I am very rookie in moving from TensorFlow to Pytorch. In tensorflow, I can simply load features and labels from separate .npy files and train a CNN using them. It is simple as below:
def finetune_resnet(file_train_classes, file_train_features, name_model_to_save):

    #Lets load features and classes first
    print("Loading, organizing and pre-processing features")
    num_classes = 12
    x_train=np.load(file_train_features)
    y_train=np.load(file_train_classes)

    #Defining train as 70% and validation 30% of the data
    #The partition is stratified with a fixed random state
    #Therefore, for all networks, the partition will be the same
    x_train, x_validation, y_train, y_validation = train_test_split(x_train, y_train, test_size=0.30, stratify=y_train, random_state=42)

    print("transforming to categorical")
    y_train = to_categorical(y_train, num_classes)
    y_validation = to_categorical(y_validation, num_classes)

    y_train= tf.constant(y_train, shape=[y_train.shape[0], num_classes])
    y_validation= tf.constant(y_validation, shape=[y_validation.shape[0], num_classes])

    print("preprocessing data")
    #Preprocessing data
    x_train = x_train.astype('float32')
    x_validation=x_validation.astype('float32')
    x_train /= 255.
    x_validation /= 255.

    print("Setting up the network")
    #Parameters for network training
    batch_size = 32
    epochs=300
    sgd = SGD(lr=0.01)
    trainAug = ImageDataGenerator(rotation_range=30,zoom_range=0.15,width_shift_range=0.2,height_shift_range=0.2,shear_range=0.15,horizontal_flip=True,fill_mode="nearest")
    print("Compiling the network")
    #Load model and prepare it for fine tuning
    baseModel = ResNet50(weights="imagenet", include_top=False,
    input_tensor=Input(shape=(224, 224, 3)))
    # construct the head of the model that will be placed on top of the
    # the base model
    headModel = baseModel.output
    headModel = Flatten(name="flatten")(headModel)
    headModel = Dense(512, activation="relu")(headModel)
    headModel = Dropout(0.5)(headModel)
    headModel = Dense(num_classes, activation="softmax")(headModel)

    # place the head FC model on top of the base model (this will become
    # the actual model we will train)
    model = Model(inputs=baseModel.input, outputs=headModel)
    model.compile(loss="categorical_crossentropy", optimizer=sgd, metrics=["accuracy"])

    trainAug.fit(x_train)
    # Fit the model on the batches generated by datagen.flow().
    print("[INFO] training head...")
    H=model.fit(trainAug.flow(x_train, y_train, batch_size=batch_size), steps_per_epoch=x_train.shape[0] // batch_size, epochs=epochs, validation_data=(x_validation, y_validation), callbacks=callbacks)

However, I have no idea how to load, train and evaluate training and testing data if loading these data from .npy files. I checked a tutorial that loads training data from folders, which is not what I want.
How can I train and test a RESNET-50 model starting with imagenet weights loading train and test data from .npy files with Pytorch?
P.s: most of Pytorch training loops require <class 'torch.utils.data.dataloader.DataLoader'> inputs to train. Is that possible to transform my training data in numpy arrays to such a format?
P.s= you can try with my data here

Output:
It seems like you need to create a custom Dataset.
class MyDataSet(torch.utils.data.Dataset):
  def __init__(self, x, y):
    super(MyDataSet, self).__init__()
    # store the raw tensors
    self._x = np.load(file_train_features)
    self._y = np.load(file_train_classes)

  def __len__(self):
    # a DataSet must know it size
    return self._x.shape[0]

  def __getitem__(self, index):
    x = self._x[index, :]
    y = self._y[index, :]
    return x, y

You can further use Dataset methods to split MyDataSet into train and validation (e.g., using torch.utils.data.random_split).
You might also find TensorDataset useful.


Instruction: How to build a tensor from one tensor of contents and another of indices?

Input:
I'm trying to assemble a tensor based on the contents of two other tensors, like so:
I have a 2D tensor called A, with shape I * J, and another 2D tensor called B, with shape M * N, whose elements are indices into the 1st dimension of A.
I want to obtain a 3D tensor C with shape M * N * J such that C[m,n,:] == A[B[m,n],:] for all m in [0, M) and n in [0, N).
I could do this using nested for-loops to iterate over all indices in M and N, assigning the right values to C at each one, but M and N are large so this is quite slow. I suspect there's some nicer, faster way of doing this using clever slicing or a built-in pytorch function, but I don't know what it would be. It looks a bit like somewhere one would use torch.gather(), but that requires all tensors to have the same number of dimensions. Does anyone know how this ought to be done?
EDIT: torch.index_select(input, dim, index) is almost what I want, but it won't work here because it requires that index be a 1D tensor, while my tensor of indices is 2D.

Output:
You could achieve this by flattening the first dimensions which let's you index A. A broadcast will be required to recover the final shape
>>> A[B.flatten(),:].reshape(*B.shape, A.size(-1))

Indexing with A[B.flatten(),:] is equivalent to torch.index_select(A, 0, B.flatten()).


Instruction: Pytorch - Problem with fine tune training from custom features and classes

Input:
The core of my problem is the fact that my features come from NumPy files (.npy).
Therefore I need the following class in my code
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
from torch.utils.data import Dataset, DataLoader
from torchvision.models import resnet50
import time
import copy

class MyDataSet(torch.utils.data.Dataset):
    def __init__(self, x, y, transform=None):
        super(MyDataSet, self).__init__()
        # store the raw tensors
        self._x = np.load(x)
        self._y = np.load(y)
        self.transform = transform

    def __len__(self):
        # a DataSet must know it size
        return self._x.shape[0]

    def __getitem__(self, index):
        x = self._x[index, :]
        y = self._y[index, :]
        return x, y

To convert my NumPy files to DataLoaders I do the following. The code below seems to work (at least, no errors are returned)
#Transform dataset
transform = transforms.Compose([transforms.ToTensor()])
dataset = MyDataSet("train1-features.npy","train1-classes.npy",transform=transform)
dataloader = DataLoader(dataset, batch_size=32)

I am trying to fine-tune a RESNET-50 network in these data with 12 classes. Here is what I do
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

feature_extract = True
batch_size = 8
num_epochs = 15
num_classes=12

model_ft = resnet50(pretrained=True)
set_parameter_requires_grad(model_ft, feature_extract)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, num_classes)
input_size = 224

if torch.cuda.is_available():
    model_ft.cuda()

params_to_update = model_ft.parameters()

print("Params to learn:")
if feature_extract:
    params_to_update = []
    for name,param in model_ft.named_parameters():
        if param.requires_grad == True:
            params_to_update.append(param)
            print("\t",name)
else:
    for name,param in model_ft.named_parameters():
        if param.requires_grad == True:
            print("\t",name)

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9)

# Setup the loss fxn
criterion = nn.CrossEntropyLoss()

Finally, here is the problematic training function
for epoch in range(num_epochs):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(dataloader, 0):

        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        #transfer labels and inputs to cuda()
        inputs,labels=inputs.cuda(), labels.cuda()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model_ft(inputs)
        loss = loss_func(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

This returns me the following error once I execute the code:
Traceback (most recent call last):
  File "train_my_data_example.py", line 89, in <module>
    for i, data in enumerate(dataloader, 0):
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "train_my_data_example.py", line 29, in __getitem__
    y = self._y[index, :]
IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

The error is clearly the dataloader variable, so is this creation ok? I mean, I am loading NumPy data and transforming it to a data loader as below:
transform = transforms.Compose([transforms.ToTensor()])
dataset = MyDataSet("train1-features.npy","train1-classes.npy",transform=transform)
dataloader = DataLoader(dataset, batch_size=32)

Is there any error in my data loader or is the problem the training loop of Pytorch?
P.s: you can reproduce my code by downloading the classes and features here

Output:
You are trying to index the second axis of an array which only has a single dimension. Simply replace y = self._y[index, :] with y = self._y[index].
Actually when positioned last, : is not required as all dimensions are selected by default.


Instruction: Change the layer format to a different image resolution

Input:
There is a class where everything is set to 32x32 image format Taken from here
class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(1, 6, 5) # here I changed the image channel from 3 to 1
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 64, 5)
            self.fc1 = nn.Linear(64 * 5 * 5, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 22) # here I changed the number of output neurons from 10 to 22

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = torch.flatten(x, 1) # flatten all dimensions except batch
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

How to change all this under resolution 96 to 96? Channel 1 (grayscale)?

Output:
At resolution 32x32 the output of conv2 is shaped (1, 64, 5, 5). On the other hand, if the input is at resolution 96x96, it will be (1, 64, 21, 21). This means fc1 needs to have 28_224 input neurons.
>>> self.fc1 = nn.Linear(64 * 21 * 21, 120)

Alternatively, you can use nn.LazyLinear which will infer this number for you, based on the first inference.
>>> self.fc1 = nn.Linear(120)


Instruction: I am getting this error of Float.Tensor and cuda.FloatTenson mismatch

Input:
I am getting this error while running the training code of a model.
Traceback (most recent call last):
  File "train.py", line 273, in <module>
    train_loss[epoch - 1] = process_epoch(
  File "train.py", line 240, in process_epoch
    loss = loss_fn(model, batch)
  File "train.py", line 221, in <lambda>
    loss_fn = lambda model, batch: weak_loss(model, batch, normalization="softmax")
  File "train.py", line 171, in weak_loss
    corr4d = model(batch).to("cuda")
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/srtf/ncnet/lib/model.py", line 263, in forward
    feature_A = self.FeatureExtraction(tnf_batch['source_image'])
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/srtf/ncnet/lib/model.py", line 84, in forward
    features = self.model(image_batch)
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 353, in forward
    return self._conv_forward(input, self.weight)
  File "/home/srtf/anaconda3/envs/ncnet/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 349, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Cuda is there on the system. Where do I need to make changes in the code?

Output:
Your input needs to be sent to the correct device:
>>> corr4d = model(batch.cuda())

Which will copy the batch to the GPU device ('cuda:0' by default).


Instruction: RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x73034 and 200x120)

Input:
Building a Neural Network layers for Skin detection dataset, and got a error here. I know i have done some mistake but cannot figure it out. Error is am getting is after taking image size 224*224 and channels 3: RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x73034 and 200x120)
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 16, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(16, 26, 5)
    self.fc1 = nn.Linear(8 * 5 * 5, 120)
    self.fc2 = nn.Linear(120, 86)
    self.fc3 = nn.Linear(86, 2)

  def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = torch.flatten(x,1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

net = Net().to(device)
print(net)

These are the layers and Net module
<ipython-input-41-8c9bafb31c44> in forward(self, x)
     16     x = self.pool(F.relu(self.conv2(x)))
     17     x = torch.flatten(x,1)
---> 18     x = F.relu(self.fc1(x))
     19     x = F.relu(self.fc2(x))
     20     x = self.fc3(x)

Can anyone help me solve this.

Output:
As Anant said, you need to match the flattened conv2 dimension (73034) to be the input dimension for the fc1 layer.
self.fc1 = nn.Linear(73034, 120)

The formula to calculate the output of each conv layer:
[(height or width) - kernel size + 2*padding] / stride + 1

For the following I will use the dimensions (Channels, Height, Width)
Input (3,224,224) -> conv1 -> (16,220,220) -> pool -> (16,110,110) -> conv2 -> (26,106,106) -> pool -> (26,53,53) -> flatten -> (73034)
It seems your batch size is 4, which refers to the "4" in (4x73034). If you print the dimensions of the output of conv1 or conv2 layers, the format will be (Batch, Channels, Height, Width).


Instruction: Select elements of Tensor based on index tensor along same dimensions

Input:
I have the following two tensors
input shape: 16 32 32 3
index shape: 16 32 32 2
output shape: 16 32 32 3

The formula for the output would be:
output[b, h, w] = input[b, index[b, h, w, 0], index[b, h, w, 1]]

I tried to use torch.gather but I was not able to formulate the previous assignment.
Does anyone know how to do this in an efficient manner? Thanks!
For context: input contains a batch of 16 elemens where each one is a tensor of 32x32 that containts 3D points. index is a mapping from position to 3D point.

Output:
You can achieve this by unraveling the indices from index (on dimensions 1 and 2) in order to index input on a single dimension using torch.gather.
This requires to expand the shape of the indexer to fit the shape of input:
Here is an example with some dummy data:
>>> x = torch.rand(16, 32, 32, 3)
>>> index = torch.randint(0, 10, (16,32,32,2))

Some manipulation on index is required to unravel the values:
>>> unraveled = x.size(1)*index[..., 0] + index[..., 1]
>>> u = unraveled.flatten(1).unsqueeze(-1).expand(-1, -1, x.size(-1))

Now u, reshaped expanded version of index has a shape of (16, 1024, 3).
The indexed tensor also needs to be flattened:
>>> x.flatten(1, 2)
torch.Size([16, 1024, 3])

Finally, you can gather on dim=1 (keep in mind the result needs to be reshaped to the desired shape i.e. the input's shape):
>>> out = input.flatten(1,2).gather(1, u).reshape_as(x)


Instruction: How to perform matrix equality?

Input:
I have two matrices. A with size 160 x 250 and B with size 3200 x 250.
I want to get the set intersection of each row of A with each row of B to get a 160 x 3200 vector. (The set size is 250 elements)
Any ideas how to implement this?
I'm thinking it should require torch.eq, but not sure how to change the dimensions. For example:
result = torch.sum(torch.eq(A[0], B), dim=1) would give me a 3200 element vector comparison with just the 0th row of A. I want for all rows of A (160)

Output:
Assuming you want to check equality between the 160x3200 possible pairs of 250-feature vectors. You can do so with an indexing trick:
>>> (A[None] == B[:, None]).all(-1)


Instruction: PyTorch doubly stochastic normalisation of 3D tensor

Input:
I'm trying to implement double stochastic normalisation of an N x N x P tensor as described in Section 3.2 in Gong, CVPR 2019. This can be done easily in the N x N case using matrix operations but I am stuck with the 3D tensor case. What I have so far is
def doubly_stochastic_normalise(E):
    """E: n x n x f"""
    E = E / torch.sum(E, dim=1, keepdim=True)  # normalised across rows
    F = E / torch.sum(E, dim=0, keepdim=True)  # normalised across cols
    E = torch.einsum('ijp,kjp->ikp', E, F)
return E

but I'm wondering if there is a method without einsum.

Output:
In this setting, you can always fall back to using torch.matmul (batched matrix multiplication to be more precise). However, this requires you to transpose the axis. Recall the matrix multiplication for two 3D inputs, in einsum notation, it gives us:
bik,bkj->bij

Notice how the k dimension gets reduces. To get to this setting, we need to transpose the inputs of the operator. In your case we have:
ijp ? kjp -> ikp
 ↓     ↓      ↑
pij @ pjk -> pik

This translates to:
>>> (E.permute(2,0,1) @ F.permute(2,1,0)).permute(1,2,0)
#       ijp ➝ pij          kjp ➝ pjk      pik ➝ ikp

You can argue your method is not only shorter but also a lot more readable. I would therefore stick with torch.einsum. The reason why the einsum operator is so useful here is because you can perform axes transpositions on the fly.


Instruction: Miniforge Conda "PackagesNotFoundError" on ARM processor for PyTorch

Input:
I am unable to install any packages with miniforge 3 (conda 4.11.0).
I am attempting this on a Jetson Nano Developer Kit running Jetpack. Initially it had conda installed but it seems to have gone missing, so I decided to reinstall conda. It looks like the base version of anaconda/miniconda is having issues running on ARM processors, and so I downloaded miniforge which apparently is working.
I have set up an environment successfully, but attempting to download pytorch gives the following error:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch

Current channels:

  - https://conda.anaconda.org/pytorch/linux-aarch64
  - https://conda.anaconda.org/pytorch/noarch
  - https://conda.anaconda.org/abinit/linux-aarch64
  - https://conda.anaconda.org/abinit/noarch
  - https://conda.anaconda.org/matsci/linux-aarch64
  - https://conda.anaconda.org/matsci/noarch
  - https://conda.anaconda.org/conda-forge/linux-aarch64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

This is for Python 3.7.12. It seems this issue persists no matter what version of pytorch I try to install.
I am however able to install some other packages, as I was able to install beautifulsoup4.

Output:
There is no linux-aarch64 version of pytorch on the default conda channel, see here
This is of course package specific. E.g. there is a linux-aarch64 version of beautifulsoup4 which is why you wre able to install it without an issue.
You can try to install from a different channel that claims to provide a pytorch for aarch64, e.g.
conda install -c kumatea pytorch