Instruction: Understanding Pytorch filter function

Input:
I was going through the documentation of PyTorch framework and found lots of instances where a variable is assigned a function but when it calls the function the parameters change. Not sure on how this works, any pointers would be helpful.
What I do understand -
def func1(word):
    print("hello", word)
var1 = func1

Now in this scenario, var1("world") would print the string hello world.
But what I dont understand is some lines from PyTorch like:
def __init__(self, input_size, num_classes):
    super(NN,self).__init__()
    self.fc1 = nn.Linear(input_size, 50)
    self.fc2 = nn.Linear(50, num_classes)
def forward(self,x):
    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

How do we know that only 1 param should be passed to self.fc2. It seems to be independent of the number of params defined in nn.Linear
Does nn.Linear return a function like func1 that we store in var1 from the earlier example? If so is there any documentation on what is being returned?
I do find the usage for each function in the nn module but is there something that gives more details on how exactly this works?

Output:
nn.Linear is not a function (and neither are all the other layers, like the convolution layers, batchnorms...), but a functor, which means it is a class which implements the __call__ method/operator which is called when you write something like self.fc2(x).
The __call__ operator is implemented in the nn.Module base class, and it's a call to another method _call_impl which itself calls (basically) the forward method. Therefore, thanks to inheritance magic, when you make a class derive from nn.Module, you only need to implement the forward method.
The signature of this method is kinda up to you, but in most cases it will take a tensor as input and return another tensor.
In summary  :
# calls the constructor of nn.Linear. self.fc1 is now a functor
self.fc1 = nn.Linear(20, 10)
# calls the fc1 functor on an input
y = self.fc1(torch.randn(2, 10))
# which is basically doing
y = self.fc1.forward(torch.randn(2, 10))


Instruction: Gradient with respect to the parameters of a specific layer in Pytorch

Input:
I am building a model in pytorch with multiple networks. For example let's consider netA and netB. In the loss function I need to work with the composition netA(netB). In different parts of the optimization I need to calculate the gradient of loss_func(netA(netB)) with respect to only the parameters of netA and in another situation I need to calculate the gradients wrt the parameters of netB. How one should approach the problem?
My approach: In the case of calculating the gradient wrt the parameters of netA I use loss_func(netA(netB.detach())).
If I write loss_func(netA(netB).detach()) it seems that the both parameters of netA and netB are detached.
I tried to use loss_func(netA.detach(netB)) in order to only detach the parameters of netA but it doesn't work. (I get the error that netA doesn't have attribute detach.)

Output:
The gradients are properties of tensors not networks.
Therefore, you can only .detach a tensor.
You can have different optimizers for each network. This way you can compute gradients for all networks all the time, but only update weights (calling step of the relevant optimizer) for the relevant network.


Instruction: Calculate variance with a kernel size in a tensor

Input: Like what nn.Conv2d or nn.AvgPool2d do with a tensor and a kernel size, I would like to calculate the variances of a tensor with a kernel size. How can I achieve this? I guess maybe source code of pytorch should be touched?

Output:
If it's only the variance you are after, you can use the fact that
var(x) = E[x^2] - E[x]^2

Using avg_pool2d you can estimate the local average of x and of x squared:
import torch.nn.functional as nnf

running_var = nnf.avg_pool2d(x**2, kernel_size=2, stride=1) - nnf.avg_pool2d(x, kernel_size=2,stride=1)**2

However, if you want a more general method of performing "sliding window" operations, you should become familiarized with unfold and fold:
u = nnf.unfold(x, kernel_size=2, stride=1)  # get all kernel_size patches as vectors
running_var2 = torch.var(u, unbiased=False, dim=1)
# reshape back to original shape ("folding")
running_var2 = running_var2.reshape(x.shape[0], 1, x.shape[2]-1, x.shape[3]-1)


Instruction: Pytorch load data in mini batches

Input:
I have a folder of images as such
Images
|
|__img1
|   |__img1_b01.tiff
|   |__img1_b02.tiff
|   |__img1_b03.tiff
|   |__img1_b04.tiff
|   |__img1_b05.tiff
|
|__img2
|   |__img2_b02.tiff
|   |__img2_b02.tiff
|   |__img2_b03.tiff
|   |__img2_b04.tiff
|   |__img2_b05.tiff
|
|.. img1000

Each folder represents an image.
Each file in the folders represents a band channel of the image.
Hence each image would have a
I am stuck writting the pytorch custom dataloader to load in batches of 64
So I could have Feature batch shape: torch.Size([64,5, 256, 256])
I have tried the following code
from torchvision import datasets, transforms
from torch.utils import data

dataset = datasets.ImageFolder(root = Images/,
            transform = transforms.ToTensor())

loader = data.DataLoader(dataset, batch_size = 64, shuffle = True)

But it is not giving the results I want which is Feature batch shape: torch.Size([64, 5, 256, 256])

Output:
Using datasets.ImageFolder will make PyTorch treat each "band" image independently and treat the folder names (e.g., img1, img2...) as "class labels".
In order to load 5 image files as different bands/channels of the same image, you'll need to write your own custom Dataset.
This custom Dataset may look something like this:
import torch
import os
from PIL import Image
import numpy as np

class MultiBandDataset(torch.utils.data.Dataset):
  def __init__(self, root, num_bands):
    self.root = root
    self.num_bands = num_bands
    self.imgs = os.listdir(root)  # all `imgNN` folders

  def __len__(self):
    return len(self.imgs)  # number of images = number of subfolders

  def __getitem__(self, index):
    multi_band = []
    # get the subfolder
    subf = os.path.join(self.root, self.imgs[index])
    for band in range(self.num_bands):
      b = Image.open(os.path.join(subf, f'{self.imgs[index]}_b{band+1:02d}.tiff')).convert("F")  # make sure you are reading a single channel from each image. you need to verify this part.
      multi_band.append(np.array(b).astype(np.float32)[None,...])  # add singleton channel dimension
    return np.concatenate(numti_band, axis=0)

Note that you would probably need to re-implement augmentations as well.


Instruction: regarding a design of Train() function

Input:
I once saw the following implementation of neural network. I am confused about the model.train() in the function of Train() . In class CNN_ForecastNet, I do not find the method of train,
class CNN_ForecastNet(nn.Module):
    def __init__(self):
        super(CNN_ForecastNet,self).__init__()
        self.conv1d = nn.Conv1d(3,64,kernel_size=1)
        self.relu = nn.ReLU(inplace=True)
        self.fc1 = nn.Linear(64*2,50)
        self.fc2 = nn.Linear(50,1)

    def forward(self,x):
        x = self.conv1d(x)
        x = self.relu(x)
        x = x.view(-1)
        #print('x size',x.size())
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)

        return x

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = CNN_ForecastNet().to(device)

def Train():

    running_loss = .0

    model.train()

    for idx, (inputs,labels) in enumerate(train_loader):
        inputs = inputs.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        preds = model(inputs.float())
        loss = criterion(preds,labels.float())
        loss.backward()
        optimizer.step()
        running_loss += loss

    train_loss = running_loss/len(train_loader)
    train_losses.append(train_loss.detach().numpy())

    print(f'train_loss {train_loss}')

Output:
As you can find in the documentation, the module train function just set a flag in the model to True (you can use model.eval() to set the flag to False).
This flag is used by some layers for which the behavior changes in eval mode, most notably the dropout and batchnorm layers.


Instruction: How can I load my dataset including the labels in jupyter notebook using anaconda?

Input: can someone help me with load both datasets, the main dataset with the labeled dataset to be load on my Jupiter notebook.

Output:
To load a dataset, depending on what file type the dataset is in, you can use functions such as:
df1 = pd.read_csv("#FileNameHere")

Then concatenate the two dataframes into one with all the attributes and labels.
df = pd.concat([df1, df2], axis=1)


Instruction: what if the size of training set is not the integer multiple of batch size

Input:
I am running the following code against the dataset of PV_Elec_Gas3.csv,  the network architecture is designed as follows
class CNN_ForecastNet(nn.Module):
    def __init__(self):
        super(CNN_ForecastNet,self).__init__()
        self.conv1d = nn.Conv1d(3,64,kernel_size=1)
        self.relu = nn.ReLU(inplace=True)
        self.fc1 = nn.Linear(64*2,50)
        self.fc2 = nn.Linear(50,1)

    def forward(self,x):
        x = self.conv1d(x)
        x = self.relu(x)
        x = x.view(-1)
        #print('x size',x.size())
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)

        return x

The train function is defined as follows,
def Train():

    running_loss = .0

    model.train()

    for idx, (inputs,labels) in enumerate(train_loader):
        inputs = inputs.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        #print('inputs ',inputs)
        preds = model(inputs.float())
        loss = criterion(preds,labels.float())
        loss.backward()
        optimizer.step()
        running_loss += loss

    train_loss = running_loss/len(train_loader)
    train_losses.append(train_loss.detach().numpy())

    print(f'train_loss {train_loss}')

the train_loader is defined as train_loader = torch.utils.data.DataLoader(train,batch_size=2,shuffle=False)  here the batch_size is set as 2.  When running the train function, I got error message as follows. The reason is becaause when the code iterate through the train_loader, the last iteration only have one training point instead of  two as batch_size requires. For this kind of scenario, besides changing the batch size, are there any other options?
This is the error message. I also include the full code to reproduce the error
RuntimeError                              Traceback (most recent call last)
<ipython-input-82-78a49fb8c068> in <module>
     99 for epoch in range(epochs):
    100     print('epochs {}/{}'.format(epoch+1,epochs))
--> 101     Train()
    102     gc.collect()

<ipython-input-82-78a49fb8c068> in Train()
     81         optimizer.zero_grad()
     82         #print('inputs ',inputs)
---> 83         preds = model(inputs.float())
     84         loss = criterion(preds,labels.float())
     85         loss.backward()

~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-82-78a49fb8c068> in forward(self, x)
     57         x = x.view(-1)
     58         #print('x size',x.size())
---> 59         x = self.fc1(x)
     60         x = self.relu(x)
     61         x = self.fc2(x)

~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

~\Anaconda3\envs\pytorchenv\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
-> 1692         output = input.matmul(weight.t())
   1693         if bias is not None:
   1694             output += bias

RuntimeError: mat1 dim 1 must match mat2 dim 0

the following is the code for reproduction of error
import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
from numpy import array
import torch
import gc
import torch.nn as nn
from tqdm import tqdm_notebook as tqdm
from torch.utils.data import Dataset,DataLoader

solar_power = pd.read_csv('PV_Elec_Gas3.csv').rename(columns={'date':'timestamp'}).set_index('timestamp')

train_set = solar_power[:'8/10/2016']

def split_sequence(sequence, n_steps):
    x, y = list(), list()
    for i in range(len(sequence)):

        end_ix = i + n_steps

        if end_ix > len(sequence)-1:
            break
        seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
        x.append(seq_x)
        y.append(seq_y)
    return array(x), array(y)

n_steps = 3
train_x,train_y = split_sequence(train_set.loc[:,"kWh electricity/day"].values,n_steps)

class ElecDataset(Dataset):
    def __init__(self,feature,target):
        self.feature = feature
        self.target = target

    def __len__(self):
        return len(self.feature)

    def __getitem__(self,idx):
        item = self.feature[idx]
        label = self.target[idx]

        return item,label

class CNN_ForecastNet(nn.Module):
    def __init__(self):
        super(CNN_ForecastNet,self).__init__()
        self.conv1d = nn.Conv1d(3,64,kernel_size=1)
        self.relu = nn.ReLU(inplace=True)
        self.fc1 = nn.Linear(64*2,50)
        self.fc2 = nn.Linear(50,1)

    def forward(self,x):
        x = self.conv1d(x)
        x = self.relu(x)
        x = x.view(-1)
        #print('x size',x.size())
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)

        return x

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = CNN_ForecastNet().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)
criterion = nn.MSELoss()

train_losses = []

def Train():

    running_loss = .0

    model.train()

    for idx, (inputs,labels) in enumerate(train_loader):
        inputs = inputs.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        #print('inputs ',inputs)
        preds = model(inputs.float())
        loss = criterion(preds,labels.float())
        loss.backward()
        optimizer.step()
        running_loss += loss

    train_loss = running_loss/len(train_loader)
    train_losses.append(train_loss.detach().numpy())

    print(f'train_loss {train_loss}')

train = ElecDataset(train_x.reshape(train_x.shape[0],train_x.shape[1],1),train_y)
train_loader = torch.utils.data.DataLoader(train,batch_size=2,shuffle=False)

epochs = 1
for epoch in range(epochs):
    print('epochs {}/{}'.format(epoch+1,epochs))
    Train()
    gc.collect()

Output:
NO!!!!

In your forward method you x.view(-1) before passing it to a nn.Linear layer. This "flattens" not only the spatial dimensions on x, but also the batch dimension! You basically mix together all samples in the batch, making your model dependant on the batch size and in general making the predictions depend on the batch as a whole rather than on the individual data points.
Instead, you should:
  ...
  def forward(self, x):
    x = self.conv1d(x)
    x = self.relu(x)
    x = x.flatten(start_dim=1)  # flatten all BUT batch dimension
    x = self.fc1(x)  # you'll probably have to modify in_features of fc1 now
    x = self.relu(x)
    x = self.fc2(x)
    return x

Please see flatten() for more details.

If, for some reason, you must process only "full batches", you can tell DataLoader to drop the last batch by changing the argument drop_last from the default False to True:
train_loader = torch.utils.data.DataLoader(train, batch_size=2, shuffle=False, drop_last=True)


Instruction: How do I use the exported 'best.pt" file from yolov5 colab file to run the trained weights locally?

Input: I have trained my model using yoloV5 on google colab, following the provided tutorial and walkthrough provided for training any custom model: Colab file for training your own custom model. I now have an exported best.pt file after running the last cell in the link provided. Now, I want to make use of this trained weight to run a detection locally on any python script. Is this possible? If so, how do I go about doing this?

Output:
You should follow this step:

Create an empty folder in desktop called ObjectDetection
Open command prompt and change directory to that new folder using
cd ObjectDetection.
Clone yolov5 repo using command - git clone https://github.com/ultralytics/yolov5.git. It will create a new folder called yolov5 inside ObjectDetection folder.
yolov5 folder contains important python file called detect.py which is responsible to detect the objects.
After cloning the repo, enter into yolov5 folder using cd yolov5
Install all the necessary requirements using - pip install -r requirements.txt
Download best.pt from colab and manually paste it inside yolov5 folder.
Also copy the image that you want to test inside yolov5 folder.
Before running inference, make sure that image.png, best.pt and detect.py should be in inside yolov5 folder.
You can then run inference inside yolov5 folder by using this command:
python detect.py --weights best.pt --source image.png
After the process is completed, you can check the result inside path ObjectDetection/yolov5/runs/detect/exp


Instruction: Taking sample from Categorical distribution pytorch

Input:
I'm currently working on a Deep reinforcement learning problem, and I'm using the categorical distribution to help the agent get random action. This is the code.
  def choose_action(self,enc_current_node,goal_node):
      #print('nn')
      #vector=self.convert_vector(observation,end)
      state=T.tensor([[enc_current_node,goal_node]],dtype=T.float)
      pi,v=self.forward(state)
      probs=T.softmax(pi,dim=1)
      print(probs)
      dist=Categorical(probs)
      action=dist.sample().numpy()[0]#take a sample from the categorical dist from 1-22
      return action

the output of the Categorical(props) like this:
probs=T.tensor([[1.5857e-03, 8.9753e-01, 2.8500e-03, 9.0585e-03, 3.6661e-04, 6.8342e-08,
         7.2956e-04, 3.3966e-05, 3.7150e-04, 1.8317e-05, 4.1543e-04, 4.7550e-05,
         5.2323e-05, 1.1337e-03, 1.6356e-05, 6.9848e-03, 2.2993e-03, 1.0874e-06,
         2.0343e-04, 2.3616e-03, 1.3477e-02, 6.1464e-02]])
c=Categorical(probs)
c
>>> output:
>>> Categorical(probs: torch.Size([1, 22]))

now in the function, I sued dist.sample to take a sample of the 22 elements in it
but I notice something that a lot of time the sample method that used in PyTorch result in the same number 90% of the time as you can see here:
list=[]
for i in range(100):
  list.append(c.sample()[0].item())
>>> output:
>>> [1, 1, 1, 21, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 21, 1, 3, 1, 1, 1, 1, 1, 1, 21, 1, 1, 21, 1, 1, 1, 1, 1, 1, 1, 21, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 21, 1, 3, 1, 1, 1, 1, 1, 18, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 21, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3]

as you can see above, the sample method output a lot of 1
my question is there a way to choose a random sample from the Categorical distribution rather than this?

Output:
If you look at your probabilities for sampling probs, you see that the 1th class has the largest probability, and almost all others are < 1%. If you are not familiar with scientific notation, here it is formatted as rounded percentages:
for label, p in enumerate(probs[0]):
    print(f'{label:2}: {100*p:5.2f}%')

 0:  0.16%
 1: 89.75%  <---
 2:  0.28%
 3:  0.91%
 4:  0.04%
 5:  0.00%
 6:  0.07%
 7:  0.00%
 8:  0.04%
 9:  0.00%
10:  0.04%
11:  0.00%
12:  0.01%
13:  0.11%
14:  0.00%
15:  0.70%
16:  0.23%
17:  0.00%
18:  0.02%
19:  0.24%
20:  1.35%
21:  6.15%

Hence ~90% of samples drawn from this will be 1.


Instruction: Does it make sense to backpropagate a loss calculated from an earlier layer through the entire network?

Input:
Suppose you have a neural network with 2 layers A and B. A gets the network input. A and B are consecutive (A's output is fed into B as input). Both A and B output predictions (prediction1 and prediction2) Picture of the described architecture
You calculate a loss (loss1) directly after the first layer (A) with a target (target1). You also calculate a loss after the second layer (loss2) with its own target (target2).
Does it make sense to use the sum of loss1 and loss2 as the error function and back propagate this loss through the entire network? If so, why is it "allowed" to back propagate loss1 through B even though it has nothing to do with it?
This question is related to this question
https://datascience.stackexchange.com/questions/37022/intuition-importance-of-intermediate-supervision-in-deep-learning
but it does not answer my question sufficiently.
In my case, A and B are unrelated modules. In the aforementioned question, A and B would be identical. The targets would be the same, too.
(Additional information)
The reason why I'm asking is that I'm trying to understand LCNN (https://github.com/zhou13/lcnn) from this paper.
LCNN is made up of an Hourglass backbone, which then gets fed into MultiTask Learner (creates loss1), which in turn gets fed into a LineVectorizer Module (loss2). Both loss1 and loss2 are then summed up here and then back propagated through the entire network here.
Even though I've visited several deep learning lectures, I didn't know this was "allowed" or makes sense to do. I would have expected to use two loss.backward(), one for each loss. Or is the pytorch computational graph doing something magical here? LCNN converges and outperforms other neural networks which try to solve the same task.

Output:
Yes, It is "allowed" and also makes sense.
From the question, I believe you have understood most of it so I'm not going to details about why this multi-loss architecture can be useful. I think the main part that has made you confused is why does "loss1" back-propagate through "B"? and the answer is: It doesn't. The fact is that loss1 is calculated using this formula:
loss1 = SOME_FUNCTION(label, y_hat)

and y_hat(prediction1) is only dependent on layers before it. Hence, the gradient of this loss only flows through layers before this section (A) and not the ones after it (B). To better understand this, you could again check the mathematics of artificial neural networks.  The loss2, on the other hand, back-propagates through all of the network (including part A). When you use a cumulative loss (Loss = loss1 + loss2), a framework like Pytorch will automatically follow the gradient of every predicted label to the first layer.


Instruction: pytorch different versions with "pip3 show torch" VS "torch.__version__"

Input:
When checking the installed pytorch version you can do that in two ways:

pip3 show torch (or similar python3 -m pip freeze for all packages)
import torch; torch.__version__

Interestingly the first option (using pip3) returns 1.8.1+cu111, while the second option (torch.__version__) returns 1.7.1 (without the cuda version support string, but cuda is available).
Why do these two methods show different results and which one is the "valid" one?
Important
When I installed pytorch with cuda support I installed the latest version, but downgraded a few weeks ago for different reasons.
Some info:

OS: ubuntu
python installed natively (not in conda, also not using jupyter)

Output: This typically happens when you have multiple versions of the same library installed for some reason (normally mixing conda install and pip install in my experience). I recommend uninstalling oldest versions using the appropriate package manager until you see the expected behavior.


Instruction: How to generate a tensor representing, for each pair of rows in a matrix, whether elements at the same position both = 1?

Input:
I'm trying to compare every row in a matrix to every other row in the same matrix, yielding, for each pair, a row showing whether each element of those two rows both equal 1.
Example: if I have this matrix as input:
[[1,0,1],
 [0,1,1],
 [0,0,0]]

I'd want to get this tensor:
[[[1,0,1],
  [0,0,1],
  [0,0,0]],
 [[0,0,1],
  [0,1,1]
  [0,0,0]],
 [[0,0,0],
  [0,0,0],
  [0,0,0]]]

Alternatively, if it's easier, my eventual goal is to reduce this tensor down to a matrix where each row represents whether ANY elements in a pair of rows from the original matrix both =1. So, the tensor above would get reduced to:
[[1,1,0],
 [1,1,0],
 [0,0,0]]

If it's easier to go straight to that without going through the intermediate tensor, let me know.

Output:
This is somewhat a partial answer, but it solves the first step which seems to be the question at hand:
import numpy as np

x = np.array([[1,0,1],
              [0,1,1],
              [0,0,0]])

output = x * np.repeat(x[:, np.newaxis, :], 3, axis=1)

output
# array([[[1, 0, 1],
#         [0, 0, 1],
#         [0, 0, 0]],

#        [[0, 0, 1],
#         [0, 1, 1],
#         [0, 0, 0]],

#        [[0, 0, 0],
#         [0, 0, 0],
#         [0, 0, 0]]])

In PyTorch, it would be:
import torch

x = torch.tensor([[1,0,1],
                  [0,1,1],
                  [0,0,0]])

output = x * x.unsqueeze(1).repeat_interleave(3, dim=1)


Instruction: How can I avoid a loop if I need to do a matrix multiplication?

Input:
I have the following code:
import numpy as np
import torch

y = torch.ones((1000,10)) #This is the output of a neural network which does not matter here
theta_min = 0; theta_max = np.pi; K = 10; sigma = 10;
z = torch.zeros(y.shape)
for i in range(0,y.shape[0]):
    theta = np.random.uniform(theta_min, theta_max)
    vector = np.reshape(np.exp(-1j * np.arange(0,K) * np.pi * np.sin(theta)),(-1,1))
    vector = torch.tensor(vector)
    alpha = sigma * np.random.randn()
    z[i,:] = alpha * vector @ vector.T @ y[i,:].T

How can I avoid the loop to make the code faster?

Output:
Following the solution of Megan Hardy, I have tried instead of make use of 3d arrays. We will make some useless operations, but it is better than the for loop. The code would look like this:
y = torch.ones((1000,10)).type(torch.complex64)
theta_min = 0; theta_max = np.pi; K = 10; sigma = 10;

theta = torch.rand(y.shape[0],1,1) * np.pi  #Each slice in the 0 dimension will be treated separately
temp = torch.reshape(torch.arange(0,K),(-1,1))    #Necessary so that it doesn't have dimensions (10,), but (10,1)
vector = torch.exp(-1j * temp * np.pi * torch.sin(theta))
matrix = vector @ torch.transpose(vector,1,2)   #Maintain the 0 dimension but exchange dimension 1 and 2

alpha = sigma * torch.rand(1)

z = alpha * matrix @ y.T    #This makes more operations that necessary
temp = range(0,y.shape[0])
z = alpha * z[temp,:,temp]  #Take only the desired columns


Instruction: Dimension out of range (expected to be in range of [-1, 0], but got 1) (pytorch)

Input:
I have a very simple feed forward neural network (pytorch)
import torch
import torch.nn.functional as F
import numpy as np
class Net_1(nn.Module):
    def __init__(self):
        super(Net_1, self).__init__()
        self.fc1 = nn.Linear(5*5, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 3)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x, dim=1)
net = Net_1()

and the input is this 5x5 numpy array
state = [[0, 0, 3, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 2, 1, 0, 0],
        [0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0]]
state = torch.Tensor(state).view(-1)

net(state) throws the following error
Dimension out of range (expected to be in range of [-1, 0], but got 1)

the problem is when F.log_softmax() is applied

Output:
at the point when you call return F.log_softmax(x, dim=1), x is a 1-dimensional tensor with shape torch.Size([3]).
dimension indexing in pytorch starts at 0, so you cannot use dim=1 for a 1-dimensional tensor, you will need to use dim=0.
replace return F.log_softmax(x, dim=1) with return F.log_softmax(x, dim=0) and you'll be good to go.
in the future you can check tensor sizes by adding print(x.shape) in forward.


Instruction: Updating a register_buffer in PyTorch?

Input:
Correct way to update a register_buffer in PyTorch
I'm trying to determine the recommended way to update a register buffer which preserves the buffer's attributes. One "hacky" way to do this is shown below:
import torch
import torch.nn as nn

class SomeModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_buffer("A", torch.tensor(2.0))

    def forward(self, x):
        return x * self.A

    def update_buffer(self, v):
        A = getattr(self, "A")
        setattr(self, "A", torch.tensor(v, dtype=A.dtype, device=A.device))

if __name__ == "__main__":
    model = SomeModule()
    model.to(torch.float64)
    print(model(1.0).dtype)
    model.update_buffer(3.0)
    print(model(1.0).dtype)

When updating many buffers this gets a bit messy. Is there a recommended / better method to accomplish this?

Output:
Generally:
self.A = new_tensor

To copy the dtype and device from the previous tensor:
self.A = new_tensor.type_as(self.A)

If the new tensor has the same shape as the old, or you want to overwrite the current tensor's data with a new value, you can do:
self.A[:] = new_value

Examples:
>>> mod.A = torch.tensor([3, 2, 1], dtype=int, device="cuda")
>>> mod.A
tensor([3, 2, 1], device='cuda:0')
>>> mod.A = torch.Tensor([1.5, 2, 3.5]).type_as(mod.A)
>>> mod.A
tensor([1, 2, 3], device='cuda:0')
>>> mod.A[:] = torch.Tensor([3, 4, 5])
>>> mod.A
tensor([3, 4, 5], device='cuda:0')
>>> mod.A[:] = 7
>>> mod.A
tensor([7, 7, 7], device='cuda:0')
>>> mod.state_dict()
OrderedDict([('A', tensor([7, 7, 7], device='cuda:0'))])


Instruction: No module named 'torch'

Input:
I'm trying to solve this Error:
ModuleNotFoundError: No module named 'torch'
I did the installation of Pytorch using this command:
conda install pytorch -c pytorch
but when I import torch I got the message above.

Output:
You might be on another conda environment.
try:
conda env list

and find the environment that you have installed pytorch on.
then use:
conda activate NAME_OF_YOUR_ENV

to swtich to that environment and run your program there.


Instruction: Why would a Torchscript trace return different looking encoded_inputs compared to the original Transformer model?

Input:
Background
I'm working with a finetuned Mbart50 model that I need sped up for inferencing because using the HuggingFace model as-is is fairly slow with my current hardware. I wanted to use TorchScript because I couldn't get onnx to export this particular model as it seems it will be supported at a later time (I would be glad to be wrong otherwise).
Convert Transformer to a Pytorch trace:
import torch
""" Model data  """
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-one-to-many-mmt", torchscript= True)

tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-one-to-many-mmt")
tokenizer.src_lang = 'en_XX'

dummy = "To celebrate World Oceans Day, we're swimming through a shoal of jack fish just off the coast of Baja, California, in Cabo Pulmo National Park. This Mexican marine park in the Sea of Cortez is home to the northernmost and oldest coral reef on the west coast of North America, estimated to be about 20,000 years old. Jacks are clearly plentiful here, but divers and snorkelers in Cabo Pulmo can also come across many other species of fish and marine mammals, including several varieties of sharks, whales, dolphins, tortoises, and manta rays."

model.config.forced_bos_token_id=250006
myTokenBatch = tokenizer(dummy, max_length=192, padding='max_length', truncation = True, return_tensors="pt")

torch.jit.save(torch.jit.trace(model, [myTokenBatch.input_ids,myTokenBatch.attention_mask]), "././traced-model/mbart-many.pt")

Inference Step:

import torch
 """ Model data  """
from transformers import MBart50TokenizerFast

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-one-to-many-mmt")

model = torch.jit.load('././traced-model/mbart-many.pt')
MAX_LENGTH =  192

tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-one-to-many-mmt")

model.to(device)
model.eval()

tokenizer.src_lang = 'en_XX'

dummy = "To celebrate World Oceans Day, we're swimming through a shoal of jack fish just off the coast of Baja, California, in Cabo Pulmo National Park. This Mexican marine park in the Sea of Cortez is home to the northernmost and oldest coral reef on the west coast of North America, estimated to be about 20,000 years old. Jacks are clearly plentiful here, but divers and snorkelers in Cabo Pulmo can also come across many other species of fish and marine mammals, including several varieties of sharks, whales, dolphins, tortoises, and manta rays."

myTokenBatch = tokenizer(dummy, max_length=192, padding='max_length', truncation = True, return_tensors="pt")

encode, pool , norm  = model(myTokenBatch.input_ids,myTokenBatch.attention_mask)

Expected Encoding Output:
These are tokens that can be decoded to words with MBart50TokenizerFast.

tensor([[250004,    717, 176016,   6661,  55609,      7,  10013,      4,    642,
             25,    107, 192298,   8305,     10,  15756,    289,    111, 121477,
          67155,   1660,   5773,     70, 184085,    111, 118191,      4,  39897,
              4,     23, 143740,  21694,    432,   9907,   5227,      5,   3293,
         181815, 122084,   9201,     23,     70,  27414,    111,  48892,    169,
             83,   5368,     47,     70, 144477,   9022,    840,     18,    136,
          10332,    525, 184518,    456,   4240,     98,     70,  65272, 184085,
            111,  23924,  21629,      4,  25902,   3674,     47,    186,   1672,
              6,  91578,   5369,  10332,      5,  21763,      7,    621, 123019,
          32328,    118,   7844,   3688,      4,   1284,  41767,    136, 120379,
           2590,   1314,     23, 143740,  21694,    432,    831,   2843,   1380,
          36880,   5941,   3789, 114149,    111,  67155,    136, 122084,  21968,
           8080,      4,  26719,  40368,    285,  68794,    111,  54524,   1224,
              4,    148,  50742,      7,      4,  13111,  19379,   1779,      4,
          43807, 125216,      7,      4,    136,    332,    102,  62656,      7,
              5,      2,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1,      1,      1,      1,      1,      1,      1,
              1,      1,      1]])

Actual Output:
I don't know what this is... print(encode)

(tensor([[[[-9.3383e-02, -2.0395e-01,  4.8226e-03,  ...,  1.8068e+00,
            1.1528e-01,  7.0406e-02],
          [-4.4630e-02, -2.2453e-01,  9.5264e-02,  ...,  1.6921e+00,
            1.4607e-01,  4.8238e-02],
          [-7.8206e-01,  1.2699e-01,  1.6467e+00,  ..., -1.7057e+00,
            8.7768e-01,  8.2230e-01],
          ...,
 [-1.2145e-02, -2.1855e-03, -6.0966e-03,  ...,  2.9296e-02,
            2.2141e-03,  3.2074e-02],
          [-1.4671e-02, -2.8995e-03, -5.8610e-03,  ...,  2.8525e-02,
            2.4620e-03,  3.1593e-02],
          [-1.5877e-02, -3.5165e-03, -4.8743e-03,  ...,  2.8930e-02,
            2.9877e-03,  3.3892e-02]]]], grad_fn=<CopyBackwards>))

Output:
Found the answer here: https://stackoverflow.com/a/66117248/13568346

You can't directly convert a seq2seq model (encoder-decoder model) using this method. To convert a seq2seq model (encoder-decoder) you have to split them and convert them separately, an encoder to onnx and a decoder to onnx. you can follow this guide (it was done for T5 which is also a seq2seq model). you need to provide a dummy variable to both encoder and to the decoder separately. by default when converting using this method it provides the encoder the dummy variable.


Instruction: PyTorch's CrossEntropyLoss - how to deal with the sequence length dimension with transformers?

Input:
I'm training a transformer model for text generation.
let's assume:
vocab size = 100
embbeding size = 50
max sequence length = 30
batch size = 32
loss = cross entropy loss

the last layer in the model is a fully connected layer,
mapping from shape [30, 32, 50] to [30, 32, 100].
the idea is that each of the last 30 sequences in the first dimension, I have a target vector I want to calculate loss with.
the issue is that based on the docs, this loss only excepts 2 dims on the prediction and one on the target - so how can I fit my 3D prediction into it?
(and 2D target?)

Output:
Use torch.BCELoss() instead (Binary cross entropy). This expects input and target to be the same size but they can be any size, and should fall within the range [0,1]. It performs cross-entropy loss element-wise.
EDIT: if you expect only one element from the vocab to be output, then you should use CrossEntropyLoss and instead encode your labels as a 1D vector rather than a 2D vector (i.e. do 1-hot decoding). BCE treats each element in the output for a single example as independent from the others, which is not a valid assumption for a multi-class style problem. I originally misread and thought the final output was an embedding, rather than an element from the vocabulary, hence my original suggestion.


Instruction: Why does unet have classes?

Input:
import torch
import torch.nn as nn
import torch.nn.functional as F

class double_conv(nn.Module):
    '''(conv => BN => ReLU) * 2'''
    def __init__(self, in_ch, out_ch):
        super(double_conv, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True)
        )

    def forward(self, x):
        x = self.conv(x)
        return x

class inconv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(inconv, self).__init__()
        self.conv = double_conv(in_ch, out_ch)

    def forward(self, x):
        x = self.conv(x)
        return x

class down(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(down, self).__init__()
        self.mpconv = nn.Sequential(
            nn.MaxPool2d(2),
            double_conv(in_ch, out_ch)
        )

    def forward(self, x):
        x = self.mpconv(x)
        return x

class up(nn.Module):
    def __init__(self, in_ch, out_ch, bilinear=True):
        super(up, self).__init__()

        #  would be a nice idea if the upsampling could be learned too,
        #  but my machine do not have enough memory to handle all those weights
        if bilinear:
            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        else:
            self.up = nn.ConvTranspose2d(in_ch//2, in_ch//2, 2, stride=2)

        self.conv = double_conv(in_ch, out_ch)

    def forward(self, x1, x2):
        x1 = self.up(x1)
        diffX = x1.size()[2] - x2.size()[2]
        diffY = x1.size()[3] - x2.size()[3]
        x2 = F.pad(x2, (diffX // 2, int(diffX / 2),
                        diffY // 2, int(diffY / 2)))
        x = torch.cat([x2, x1], dim=1)
        x = self.conv(x)
        return x

class outconv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(outconv, self).__init__()
        self.conv = nn.Conv2d(in_ch, out_ch, 1)

    def forward(self, x):
        x = self.conv(x)
        return x

class UNet(nn.Module):
    def __init__(self, n_channels, n_classes):
        super(UNet, self).__init__()
        self.inc = inconv(n_channels, 64)
        self.down1 = down(64, 128)
        self.down2 = down(128, 256)
        self.down3 = down(256, 512)
        self.down4 = down(512, 512)
        self.up1 = up(1024, 256)
        self.up2 = up(512, 128)
        self.up3 = up(256, 64)
        self.up4 = up(128, 64)
        self.outc = outconv(64, n_classes)

    def forward(self, x):
        self.x1 = self.inc(x)
        self.x2 = self.down1(self.x1)
        self.x3 = self.down2(self.x2)
        self.x4 = self.down3(self.x3)
        self.x5 = self.down4(self.x4)
        self.x6 = self.up1(self.x5, self.x4)
        self.x7 = self.up2(self.x6, self.x3)
        self.x8 = self.up3(self.x7, self.x2)
        self.x9 = self.up4(self.x8, self.x1)
        self.y = self.outc(self.x9)
        return self.y

When I was reading UNet architecture I have found that it has n_classes as output.
class UNet(nn.Module):
    def __init__(self, n_channels, n_classes):

but why does it have n_classes as it is used for image segmentation?
I am trying to use this code for image denoising and I couldn't figure out what will should the n_classes parameter be, because I don't have any classes.
Does n_classes signify multiclass segmentation? If so, what is the output of binary UNet segmentation?

Output:
Answer

Does n_classes signify multiclass segmentation?

Yes, if you specify n_classes=4 it will output a (batch, 4, width, height) shaped tensor, where each pixel can be segmented as one of 4 classes. Also one should use torch.nn.CrossEntropyLoss for training.

If so, what is the output of binary UNet segmentation?

If you want to use binary segmentation you'd specify n_classes=1 (either 0 for black or 1 for white) and use torch.nn.BCEWithLogitsLoss

I am trying to use this code for image denoising and I couldn't figure out what will should the n_classes parameter be

It should be equal to n_channels, usually 3 for RGB or 1 for grayscale. If you want to teach this model to denoise an image you should:

Add some noise to the image (e.g. using torchvision.transforms)
Use sigmoid activation at the end as the pixels will have value between 0 and 1 (unless normalized)
Use torch.nn.MSELoss for training

Why sigmoid?
Because [0,255] pixel range is represented as [0, 1] pixel value (without normalization at least). sigmoid does exactly that - squashes value into [0, 1] range, hence linear outputs (logits) can have a range from -inf to +inf.

Why not a linear output and a clamp?

In order for the Linear layer to be in [0, 1] range after clamp possible output values from Linear would have to be greater than 0 (logits range to fit the target: [0, +inf])

Why not a linear output without a clamp?

Logits outputted would have to be within [0, 1] range

Why not some other method?

You could do that, but the idea of sigmoid is:

help neural network (any logit value can be outputted)
first derivative of sigmoid is gaussian standard normal, hence it models the probability of many real-life occurring phenomena (see also here for more)


Instruction: Pytorch CPU CUDA device load without gpu

Input:
I found this nice code Pytorch mobilenet which I cant get running on a CPU.
https://github.com/rdroste/unisal
I am new to Pytorch so I am not shure what to do.
In line 174 of the module train.py the device is set:
device = 'cuda:0' if torch.cuda.is_available() else 'cpu'

which is correct as far as I know about Pytorch.
Do I have to change the torch.load too? I tried with no success.
class BaseModel(nn.Module):
    """Abstract model class with functionality to save and load weights"""

    def forward(self, *input):
        raise NotImplementedError

    def save_weights(self, directory, name):
        torch.save(self.state_dict(), directory / f'weights_{name}.pth')

    def load_weights(self, directory, name):
        self.load_state_dict(torch.load(directory / f'weights_{name}.pth'))

    def load_best_weights(self, directory):
        self.load_state_dict(torch.load(directory / f'weights_best.pth'))

    def load_epoch_checkpoint(self, directory, epoch):
        """Load state_dict from a Trainer checkpoint at a specific epoch"""
        chkpnt = torch.load(directory / f"chkpnt_epoch{epoch:04d}.pth")
        self.load_state_dict(chkpnt['model_state_dict'])

    def load_checkpoint(self, file):
        """Load state_dict from a specific Trainer checkpoint"""
        """Load """
        chkpnt = torch.load(file)
        self.load_state_dict(chkpnt['model_state_dict'])

    def load_last_chkpnt(self, directory):
        """Load state_dict from the last Trainer checkpoint"""
        last_chkpnt = sorted(list(directory.glob('chkpnt_epoch*.pth')))[-1]
        self.load_checkpoint(last_chkpnt)

I don’t get it. Where do I have to tell pytorch there is no gpu ?
complete error:
Traceback (most recent call last):
  File "run.py", line 99, in <module>
    fire.Fire()

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)

  File "run.py", line 95, in predict_examples
    example_folder, is_video, train_id=train_id, source=source)

  File "run.py", line 72, in predictions_from_folder
    folder_path, is_video, source=source, model_domain=model_domain)

  File "/home/b256/Data/saliency_models/unisal-master/unisal/train.py", line 871, in generate_predictions_from_path
    self.model.load_best_weights(self.train_dir)

  File "/home/b256/Data/saliency_models/unisal-master/unisal/train.py", line 1057, in model
    self._model = model_cls(**self.model_cfg)

  File "/home/b256/Data/saliency_models/unisal-master/unisal/model.py", line 190, in __init__
    self.cnn = MobileNetV2(**self.cnn_cfg)

  File "/home/b256/Data/saliency_models/unisal-master/unisal/models/MobileNetV2.py", line 156, in __init__
    Path(__file__).resolve().parent / 'weights/mobilenet_v2.pth.tar')

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 367, in load
    return _load(f, map_location, pickle_module)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 538, in _load
    result = unpickler.load()

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 504, in persistent_load
    data_type(size), location)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 113, in default_restore_location
    result = fn(storage, location)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 94, in _cuda_deserialize
    device = validate_cuda_device(location)

  File "/home/b256/anaconda3/envs/unisal36/lib/python3.6/site-packages/torch/serialization.py", line 78, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

Output:
In https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-on-gpu-load-on-cpu you'll see there's a map_location keyword argument to send weights to the proper device:
model.load_state_dict(torch.load(PATH, map_location=device))

From the docs https://pytorch.org/docs/stable/generated/torch.load.html#torch.load

torch.load() uses Python’s unpickling facilities but treats storages,
which underlie tensors, specially. They are first deserialized on the
CPU and are then moved to the device they were saved from. If this
fails (e.g. because the run time system doesn’t have certain devices),
an exception is raised. However, storages can be dynamically remapped
to an alternative set of devices using the map_location argument.
If map_location is a callable, it will be called once for each
serialized storage with two arguments: storage and location. The
storage argument will be the initial deserialization of the storage,
residing on the CPU. Each serialized storage has a location tag
associated with it which identifies the device it was saved from, and
this tag is the second argument passed to map_location. The builtin
location tags are 'cpu' for CPU tensors and 'cuda:device_id' (e.g.
'cuda:2') for CUDA tensors. map_location should return either None or
a storage. If map_location returns a storage, it will be used as the
final deserialized object, already moved to the right device.
Otherwise, torch.load() will fall back to the default behavior, as if
map_location wasn’t specified.
If map_location is a torch.device object or a string containing a
device tag, it indicates the location where all tensors should be
loaded.
Otherwise, if map_location is a dict, it will be used to remap
location tags appearing in the file (keys), to ones that specify where
to put the storages (values).


Instruction: Pytorch Inferencing form the model is giving me different results every time

Input:
I have created and trained one very simple network in pytorch as shown below:
self.task_layers[task][task_layer_key]; TaskLayerManager(
  (taskLayers): ModuleList(
    (0): lc_hidden(
      (dropout_layer): Dropout(p=0.0, inplace=False)
      (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    )
    (1): cnn(
      (cnn_layer): CNN_Text(
        (dropout): Dropout(p=0.1, inplace=False)
        (fc1): Linear(in_features=300, out_features=2, bias=True)
        (convs1): ModuleList(
          (0): Conv2d(1, 300, kernel_size=(5, 768), stride=(1, 1), padding=(4, 0))
        )
      )
    )
  )
)

Layer descriptions:
taskLayers.0.linear_weights      torch.Size([13])
taskLayers.0.layer_norm.weight      torch.Size([768])
taskLayers.0.layer_norm.bias      torch.Size([768])
taskLayers.1.cnn_layer.fc1.weight      torch.Size([2, 300])
taskLayers.1.cnn_layer.fc1.bias      torch.Size([2])
taskLayers.1.cnn_layer.convs1.0.weight      torch.Size([300, 1, 5, 768])
taskLayers.1.cnn_layer.convs1.0.bias      torch.Size([300])

It is a binary classification network that take a 3d tensor as input [N,K,768] and gives output [N,2] tensor
I am not able to figure out "Why at every run it is giving me different results"?
Please help me with this - I am new to pytorch.
And let me know if any other information is needed.

Output:
I suspect this is due to you not having set the model to inference mode with
model.eval()

If you don't do this, your dropout layer(s) will remain activated and randomly dropout p proportion of neurons on each call.


Instruction: How to read this modified unet?

Input:
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from PIL import Image
import matplotlib.pyplot as plt

class Model_Down(nn.Module):
    """
    Convolutional (Downsampling) Blocks.

    nd = Number of Filters
    kd = Kernel size

    """
    def __init__(self,in_channels, nd = 128, kd = 3, padding = 1, stride = 2):
        super(Model_Down,self).__init__()
        self.padder = nn.ReflectionPad2d(padding)
        self.conv1 = nn.Conv2d(in_channels = in_channels, out_channels = nd, kernel_size = kd, stride = stride)
        self.bn1 = nn.BatchNorm2d(nd)

        self.conv2 = nn.Conv2d(in_channels = nd, out_channels = nd, kernel_size = kd, stride = 1)
        self.bn2 = nn.BatchNorm2d(nd)

        self.relu = nn.LeakyReLU()

    def forward(self, x):
        x = self.padder(x)
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.padder(x)
        x = self.conv2(x)
        x = self.bn2(x)
        x = self.relu(x)
        return x

class Model_Skip(nn.Module):
    """

    Skip Connections

    ns = Number of filters
    ks = Kernel size

    """
    def __init__(self,in_channels = 128, ns = 4, ks = 1, padding = 0, stride = 1):
        super(Model_Skip, self).__init__()
        self.conv = nn.Conv2d(in_channels = in_channels, out_channels = ns, kernel_size = ks, stride = stride, padding = padding)
        self.bn = nn.BatchNorm2d(ns)
        self.relu = nn.LeakyReLU()

    def forward(self,x):
        x = self.conv(x)
        x = self.bn(x)
        x = self.relu(x)
        return x

class Model_Up(nn.Module):
    """
    Convolutional (Downsampling) Blocks.

    nd = Number of Filters
    kd = Kernel size

    """
    def __init__(self, in_channels = 132, nu = 128, ku = 3, padding = 1):
        super(Model_Up, self).__init__()
        self.bn1 = nn.BatchNorm2d(in_channels)
        self.padder = nn.ReflectionPad2d(padding)
        self.conv1 = nn.Conv2d(in_channels = in_channels, out_channels = nu, kernel_size = ku, stride = 1, padding = 0)
        self.bn2 = nn.BatchNorm2d(nu)

        self.conv2 =  nn.Conv2d(in_channels = nu, out_channels = nu, kernel_size = 1, stride = 1, padding = 0) #According to supmat.pdf ku = 1 for second layer
        self.bn3 = nn.BatchNorm2d(nu)

        self.relu = nn.LeakyReLU()

    def forward(self,x):
        x = self.bn1(x)
        x = self.padder(x)
        x = self.conv1(x)
        x = self.bn2(x)
        x = self.relu(x)
        x = self.conv2(x)
        x = self.bn3(x)
        x = self.relu(x)
        x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')
        return x

class Model(nn.Module):
    def __init__(self, length = 5, in_channels = 32, out_channels = 3, nu = [128,128,128,128,128] , nd =
                    [128,128,128,128,128], ns = [4,4,4,4,4], ku = [3,3,3,3,3], kd = [3,3,3,3,3], ks = [1,1,1,1,1]):
        super(Model,self).__init__()
        assert length == len(nu), 'Hyperparameters do not match network depth.'

        self.length = length

        self.downs = nn.ModuleList([Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i]) if i != 0 else
                                        Model_Down(in_channels = in_channels, nd = nd[i], kd = kd[i]) for i in range(self.length)])

        self.skips = nn.ModuleList([Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i]) for i in range(self.length)])

        self.ups = nn.ModuleList([Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i]) if i != self.length-1 else
                                        Model_Up(in_channels = ns[i], nu = nu[i], ku = ku[i]) for i in range(self.length-1,-1,-1)]) #Elements ordered backwards

        self.conv_out = nn.Conv2d(nu[0],out_channels,1,padding = 0)
        self.sigm = nn.Sigmoid()

    def forward(self,x):
        s = [] #Skip Activations

        #Downpass
        for i in range(self.length):
            x = self.downs[i].forward(x)
            s.append(self.skips[i].forward(x))

        #Uppass
        for i in range(self.length):
            if (i == 0):
                x = self.ups[i].forward(s[-1])
            else:
                x = self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))

        x = self.sigm(self.conv_out(x)) #Squash to RGB ([0,1]) format
        return x

This code is a modified UNet I am working on. I am facing hard to read and understand code and how the skip connections are connected to upsampling. can anyone please explain it or can it be written in simple more understandable way without nn.ModuleList.
can some one show how this network looks using a diagram?
This is the github link repo link from where i took this code and trying to understand it.

Output:
Here is a functional equivalent of the main Model forward(x) method. It is much more verbose, but it is "unravelling" the flow of operations, making it more easily understandable.
I assumed that the length of the list-arguments are always 5 (i is in the [0, 4] range, inclusive) so I could unpack properly (and it follows the default set of parameters).
def unet_function(x, in_channels = 32, out_channels = 3, nu = [128,128,128,128,128],
                  nd = [128,128,128,128,128], ns = [4,4,4,4,4], ku = [3,3,3,3,3],
                  kd = [3,3,3,3,3], ks = [1,1,1,1,1]):

    ################################
    # DOWN PASS ####################
    ################################

    #########
    # i = 0 #
    #########

    # First Down
    # Model_Down(in_channels = in_channels, nd = nd[i], kd = kd[i])
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2D(in_channels=in_channels, out_channels=nd[0], kernel_size=kd[0], stride=2)(x)
    x = nn.BatchNorm2d(nd[0])(x)
    x = nn.LeakyRelu()(x)
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2d(in_channels = nd[0], out_channels=nd[0], kernel_size = kd[0], stride=1)(x)
    x = nn.BatchNorm2d(nd[0])(x)
    x = nn.LeakyRelu()(x)

    # First skip
    # Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i])
    s0 = nn.Conv2D(in_channels=nd[0], out_channels=ns[0])(x)
    s0 = nn.BatchNorm2d(ns[0])(s0)
    s0 = nn.LeakyreLU()(s0)

    #########
    # i = 1 #
    #########

    # Second Down
    # Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i])
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2D(in_channels=nd[0], out_channels=nd[0], kernel_size=kd[1], stride=2)(x)
    x = nn.BatchNorm2d(nd[0])(x)
    x = nn.LeakyRelu()(x)
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2d(in_channels = nd[0], out_channels=nd[0], kernel_size = kd[1], stride=1)(x)
    x = nn.BatchNorm2d(nd[0])(x)
    x = nn.LeakyRelu()(x)

    # Second skip
    # Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i])
    s1 = nn.Conv2D(in_channels=nd[1], out_channels=ns[1])(x)
    s1 = nn.BatchNorm2d(ns[1])(s1)
    s1 = nn.LeakyreLU()(s1)

    #########
    # i = 2 #
    #########

    # Third Down
    # Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i])
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2D(in_channels=nd[1], out_channels=nd[1], kernel_size=kd[2], stride=2)(x)
    x = nn.BatchNorm2d(nd[1])(x)
    x = nn.LeakyRelu()(x)
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2d(in_channels = nd[1], out_channels=nd[0], kernel_size = kd[2], stride=1)(x)
    x = nn.BatchNorm2d(nd[1])(x)
    x = nn.LeakyRelu()(x)

    # Third skip
    # Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i])
    s2 = nn.Conv2D(in_channels=nd[2], out_channels=ns[2])(x)
    s2 = nn.BatchNorm2d(ns[2])(s2)
    s2 = nn.LeakyreLU()(s2)

    #########
    # i = 3 #
    #########

    # Fourth Down
    # Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i])
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2D(in_channels=nd[2], out_channels=nd[2], kernel_size=kd[3], stride=2)(x)
    x = nn.BatchNorm2d(nd[2])(x)
    x = nn.LeakyRelu()(x)
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2d(in_channels = nd[2], out_channels=nd[2], kernel_size = kd[3], stride=1)(x)
    x = nn.BatchNorm2d(nd[2])(x)
    x = nn.LeakyRelu()(x)

    # Fourth skip
    # Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i])
    s3 = nn.Conv2D(in_channels=nd[3], out_channels=ns[3])(x)
    s3 = nn.BatchNorm2d(ns[3])(s3)
    s3 = nn.LeakyreLU()(s3)

    #########
    # i = 4 #
    #########

    # Fifth Down
    # Model_Down(in_channels = nd[i-1], nd = nd[i], kd = kd[i])
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2D(in_channels=nd[3], out_channels=nd[3], kernel_size=kd[4], stride=2)(x)
    x = nn.BatchNorm2d(nd[3])(x)
    x = nn.LeakyRelu()(x)
    x = nn.ReflectionPad2d(padding=1)(x)
    x = nn.Conv2d(in_channels = nd[3], out_channels=nd[3], kernel_size = kd[4], stride=1)(x)
    x = nn.BatchNorm2d(nd[2])(x)
    x = nn.LeakyRelu()(x)

    # Fifth skip
    # Model_Skip(in_channels = nd[i], ns = ns[i], ks = ks[i])
    x = nn.Conv2D(in_channels=nd[4], out_channels=ns[4])(x)
    x = nn.BatchNorm2d(ns[4])(x)
    x = nn.LeakyreLU()(x)

    ################################
    # UP PASS ######################
    ################################

    #########
    # i = 4 #
    #########

    # First Up
    # Model_Up(in_channels = ns[i], nu = nu[i], ku = ku[i])
    x = nn.BatchNorm2d(in_channel=ns[4])(x)
    x = nn.ReflectionPad2d(padding)(x)
    x = nn.Conv2d(in_channels=ns[4], out_channels=nu[4], kernel_size=ku[4], stride=1, padding=0)(x)
    x = nn.BatchNorm2d(nu[4])(x)
    x = nn.LeakyReLU()(x)
    x = nn.Conv2d(in_channels = nu[4], out_channels=nu[4], kernel_size = 1, stride = 1, padding = 0)(x)
    x = nn.BatchNorm2d(nu[4])(x)
    x = nn.LeakyReLU()(x)
    x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')

    #########
    # i = 3 #
    #########

    # Second Up
    # self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))
    x = torch.cat([x,s3], axis=1) # IMPORTANT HERE
    # Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i])
    x = nn.BatchNorm2d(in_channel=ns[3]+nu[4])(x)
    x = nn.ReflectionPad2d(padding)(x)
    x = nn.Conv2d(in_channels=ns[3]+nu[4], out_channels=nu[3], kernel_size=ku[3], stride=1, padding=0)(x)
    x = nn.BatchNorm2d(nu[3])(x)
    x = nn.LeakyReLU()(x)
    x = nn.Conv2d(in_channels = ns[3]+nu[4], out_channels=nu[3], kernel_size = 1, stride = 1, padding = 0)(x)
    x = nn.BatchNorm2d(nu[3])(x)
    x = nn.LeakyReLU()(x)
    x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')

    #########
    # i = 2 #
    #########

    # Third Up
    # self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))
    x = torch.cat([x,s2], axis=1) # IMPORTANT HERE
    # Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i])
    x = nn.BatchNorm2d(in_channel=ns[2]+nu[3])(x)
    x = nn.ReflectionPad2d(padding)(x)
    x = nn.Conv2d(in_channels=ns[2]+nu[3], out_channels=nu[2], kernel_size=ku[2], stride=1, padding=0)(x)
    x = nn.BatchNorm2d(nu[2])(x)
    x = nn.LeakyReLU()(x)
    x = nn.Conv2d(in_channels = ns[2]+nu[3], out_channels=nu[2], kernel_size = 1, stride = 1, padding = 0)(x)
    x = nn.BatchNorm2d(nu[2])(x)
    x = nn.LeakyReLU()(x)
    x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')

    #########
    # i = 1 #
    #########

    # Fourth Up
    # self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))
    x = torch.cat([x,s1], axis=1) # IMPORTANT HERE
    # Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i])
    x = nn.BatchNorm2d(in_channel=ns[1]+nu[2])(x)
    x = nn.ReflectionPad2d(padding)(x)
    x = nn.Conv2d(in_channels=ns[1]+nu[2], out_channels=nu[1], kernel_size=ku[1], stride=1, padding=0)(x)
    x = nn.BatchNorm2d(nu[1])(x)
    x = nn.LeakyReLU()(x)
    x = nn.Conv2d(in_channels = ns[1]+nu[2], out_channels=nu[1], kernel_size = 1, stride = 1, padding = 0)(x)
    x = nn.BatchNorm2d(nu[1])(x)
    x = nn.LeakyReLU()(x)
    x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')

    #########
    # i = 0 #
    #########

    # Fifth Up
    # self.ups[i].forward(torch.cat([x,s[self.length-1-i]],axis = 1))
    x = torch.cat([x,s0], axis=1) # IMPORTANT HERE
    # Model_Up(in_channels = ns[i]+nu[i+1], nu = nu[i], ku = ku[i])
    x = nn.BatchNorm2d(in_channel=ns[0]+nu[1])(x)
    x = nn.ReflectionPad2d(padding)(x)
    x = nn.Conv2d(in_channels=ns[0]+nu[1], out_channels=nu[0], kernel_size=ku[0], stride=1, padding=0)(x)
    x = nn.BatchNorm2d(nu[0])(x)
    x = nn.LeakyReLU()(x)
    x = nn.Conv2d(in_channels = nu[0], out_channels=nu[0], kernel_size = 1, stride = 1, padding = 0)(x)
    x = nn.BatchNorm2d(nu[0])(x)
    x = nn.LeakyReLU()(x)
    x = F.interpolate(x, scale_factor = 2, mode = 'bilinear')

    ################################
    # OUT ##########################
    ################################

    x = nn.Conv2d(in_channels=nu[0], out_channels=out_channels, kernel_size=1, padding = 0)
    return nn.Sigmoid()(x) #Squash to RGB ([0,1]) format

The two most important parts are:

The skips where the tensor x is processed in a parallel part of the code, not to disturb the main x "pathway".

The produced tensors from the skip parts are then fed back to the "main pathway" starting from the last one. I kept those tensor as individual variables s0 to s3, so that it is more obvious.

From this picture, you can clearly see the down part feeding latter parts. s0 is the longest grey arrow, it is concatenated to the "main pathway" just before the last convolution layers group.

(Not the same U-Net)
You can also understand from it why we don't need to store a s4 : it is directly fed to the next layer, therefore there is no need to store it as a separate variable.
The Module version does store it, but only because it is conveniently stored in a list read in reverse order at the end. Another obvious reason to store them in a list is so we can have any number of Up and Down parts, by changing the parameters accordingly.


Instruction: How and where can i freeze classifier layer?

Input: If I need to freeze the output layer of this model which is doing the classification as I don't need it.

Output:
You are confusing a few things here (I think)
Freezing layers
You freeze the layer if you don't want them to be trained (and don't want them to be part of the graph also).
Usually we freeze part of the network creating features, in your case it would be everything up to self.head.
After that, we usually only train bottleneck (self.head in this case) to fine-tune it for the task at hand.
In case of your model it would be:
def gradient(model, freeze: bool):
    for parameter in transformer.parameters():
        parameter.requires_grad_(not freeze)

transformer = VisionTransformer()
gradient(model, freeze=True)
gradient(model.head, freeze=False)

I only want the features
In this case you have the following line:
self.head = nn.Linear(embed_dim, num_classes) if num_classes > 0 else nn.Identity()

If you specify num_classes as 0 the model will only return the  features, e.g.:
transformer = VisionTransformer(num_classes=0)

I want specific head for my task
Simply override the self.head attribute, for example:
transformer.head = nn.Sequential(
    nn.Linear(embed_dim, 100), nn.ReLU(), nn.Linear(100, num_classes)
)

Or, if you want different number of classes you can specify num_classes to the number of classes you have in your task.
Question in the comment
No, you should freeze everything except head and specify that you want features out, this would do the trick:
def gradient(model, freeze: bool):
    for parameter in transformer.parameters():
        parameter.requires_grad_(not freeze)

transformer = VisionTransformer(num_classes=0)
gradient(model, freeze=True)

Due to that, learned features by VisionTransformer will be preserved (probably what you are after), you don't need self.head at all in this case!


Instruction: Serving service from kfserving github examples created with kubectl, but can not infere

Input:
I have installed minikube cluster and kfserving on a linux desktop.
Then I have followed two tutorials
https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1beta1/custom/torchserve
https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1alpha2/custom/kfserving-custom-model
In the second tutorial I have needed to move "name: custom" from "custom:" section to "container:" section in the yaml file.
I expected that serving service was working and responding to serving requests and pods of the service where in kubernetes.
I use the newest stable versions from May 2021.
But I have same bug in both tutorials. Bellow commands are from the first tutorial. When I prepare docker images with models and run
$ kubectl apply -f torchserve-custom.yaml

command.
I see output like
inferenceservice.serving.kubeflow.org/torchserve-custom created

But I can not run a prediction, because service I get 404 not found.
When I run
$ kubectl get pods --all-namespaces

I do not see any new pods.
But I can deleted service with
 $ kubectl delete -f torchserve-custom

When I have followed other tutorials like: https://github.com/kubeflow/kfserving/tree/master/docs/samples/v1beta1/torchserve everything worked fine and I could run predictions.
Why can`t I run a prediction?
Why any new pods are created?
How to setup serving in kfserving using docker image, if the above tutorials did not worked?

Output: It turned out that my local docker registry wasn't visible from kubernetes. kubectl get events shows InternalError "Unable to fetch image ... "


Instruction: What is the fastest way to load data from multiple csv files

Input:
I am working with multiple csv files, each containing multiple 1D data. I have about 9000 such files and total combined data is about 40 GB.
I have written a dataloader like this:
class data_gen(torch.utils.data.Dataset):
    def __init__(self, files):

        self.files = files
        my_data = np.genfromtxt('/data/'+files, delimiter=',')
        self.dim = my_data.shape[1]
        self.data = []

    def __getitem__(self, i):

        file1 = self.files
        my_data = np.genfromtxt('/data/'+file1, delimiter=',')
        self.dim = my_data.shape[1]

        for j in range(my_data.shape[1]):
            tmp = np.reshape(my_data[:,j],(1,my_data.shape[0]))
            tmp = torch.from_numpy(tmp).float()
            self.data.append(tmp)

        return self.data[i]

    def __len__(self):

        return self.dim

The way I am loading the whole dataset into the dataloader is like through a for loop:
for x_train in tqdm(train_files):
    train_dl_spec = data_gen(x_train)
        train_loader = torch.utils.data.DataLoader(
        train_dl_spec, batch_size=128, shuffle=True, num_workers=8, pin_memory=True)
        for data in train_loader:

But this is working terribly slow. I was wondering if I could store all of that data in one file but I don’t have enough RAM. So is there a way around it?
Let me know if there’s a way.

Output:
I've never used pytorch before, and I confess I don't really know what's going on. Nonetheless I'm almost certain you're using Dataset wrong.
As I understand it, the Dataset is an abstraction of all the data where each index returns a sample. Say each of your 9000 files has 10 rows (samples), 21 would refer to the 3rd file and the 2nd row (using 0-indexing).
Because you have so much data you don't want to load everything into memory. So the Dataset should manage just getting one value, and the DataLoader creates batches of the values.
There's almost certainly some optimisation that can be applied to what I've done, but maybe this can start you off. I created the directory csvs with these files:
❯ cat csvs/1.csv
1,2,3
2,3,4
3,4,5

❯ cat csvs/2.csv
21,21,21
34,34,34
66,77,88

Then I created this Dataset class. It takes a directory as input (where all the CSVs are stored). Then the only thing is stores in memory is the name of every file and the number of lines it has. When an item is requested we find out which file contains that index, and then return the Tensor for that line.
By only ever iterating through files, we never store file contents in memory. An improvement here though would not to iterate over the list of files to find out which one is relevant, and to make use of generators and state when accessing consecutive indexes.
(Because accessing when accessing index 8, in an 10 line file we iterate through the first 7 uselessly, which we can't help. But then when accessing index 9, it would be better to work out that we could just return the next one, rather than iterating through the first 8 lines again.)
import numpy as np
from functools import lru_cache
from pathlib import Path
from pprint import pprint
from torch.utils.data import Dataset, DataLoader

@lru_cache()
def get_sample_count_by_file(path: Path) -> int:
    c = 0
    with path.open() as f:
        for line in f:
            c += 1
    return c

class CSVDataset:
    def __init__(self, csv_directory: str, extension: str = ".csv"):
        self.directory = Path(csv_directory)
        self.files = sorted((f, get_sample_count_by_file(f)) for f in self.directory.iterdir() if f.suffix == extension)
        self._sample_count = sum(f[-1] for f in self.files)

    def __len__(self):
        return self._sample_count

    def __getitem__(self, idx):
        current_count = 0
        for file_, sample_count in self.files:
            if current_count <= idx < current_count + sample_count:
                # stop when the index we want is in the range of the sample in this file
                break  # now file_ will be the file we want
            current_count += sample_count

        # now file_ has sample_count samples
        file_idx = idx - current_count  # the index we want to access in file_
        with file_.open() as f:
            for i, line in enumerate(f):
                if i == file_idx:
                    data = np.array([float(v) for v in line.split(",")])
                    return torch.from_numpy(data)

Now we can use the DataLoader as I believe is intended:
dataset = CSVDataset("csvs")
loader = DataLoader(dataset, batch_size=4)

pprint(list(enumerate(loader)))

"""
[(0,
  tensor([[ 1.,  2.,  3.],
        [ 2.,  3.,  4.],
        [ 3.,  4.,  5.],
        [21., 21., 21.]], dtype=torch.float64)),
 (1, tensor([[34., 34., 34.],
        [66., 77., 88.]], dtype=torch.float64))]
"""

You can see this correctly returns batches of data. Rather than printing this out you can process each batch and only store that batch in memory.
See the docs for further information: https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html#part-3-the-dataloader


Instruction: Adding a feature to a neuron

Input:
Say I have the following model:
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 5)
        self.fc2 = nn.Linear(5, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = self.fc1(x)
        print(x)
        x = torch.relu(x)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Model()

opt = optim.Adam(net.parameters())
features = torch.rand((3,1)) #3 inputs, each of 1D

I can print the value of my neurons (only the first layer here) with print(x):
net(features)
>>>tensor([[ 0.6703,  0.4484, -0.8529,  1.3119,  0.6741],
        [ 0.9112,  0.6496, -1.2960,  1.8264,  0.4547],
        [ 0.7483,  0.5135, -0.9963,  1.4785,  0.6031]],
       grad_fn=<AddmmBackward>)
tensor([[0.0144],
        [0.0575],
        [0.0284]], grad_fn=<AddmmBackward>)

How can I add a "feature" to each neuron that is a string with a name?
e.g.
print(x)
>>> tensor([[ [0.6703, 'neuron_1'],  [0.4484, 'neuron_2'], [-0.8529, 'neuron_3'],  1.3119,  0.6741],... etc.

I'm not sure if I'll need to change the neuron class. I believe in the forward method I will then need to only take the first element of each neurons tensor: neuron_tensor = [neuron_value, neuron_name]
Update 1:
from @Aditya Singh Rathore comment it sounds like it might not be possible to have a string and a value in the same tensor. Is it possible then to have a value instead of a string to represent the neurons?
From before neuron_tensor = [neuron_value, neuron_name] where neuron_name is a string.
Is this possible instead? : neuron_tensor = [neuron_value, neuron_name] where neuron_name is just a value (e.g 1 for neuron 1, 2 for neuron 2)

Output:
What you want is possible, although I'm not sure what you exactly want to do with this. Basically you seem to want to include the "index" in the intermediate tensor for some purpose, and again discard it when passing to the next layer.
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 5)
        self.fc2 = nn.Linear(5, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = self.fc1(x)
        idx_tensor = (torch.arange(x.shape[1]) + 1).unsqueeze(0).repeat_interleave(repeats=x.shape[0], dim=0)
        x = torch.cat([x.unsqueeze(2), idx_tensor.unsqueeze(2)], dim=2)
        print(x)

        x = torch.relu(x[:, :, 0])
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Running this gives:
net = Model()

opt = optim.Adam(net.parameters())
features = torch.rand((3,1))
net(features)

>>>tensor([[[ 0.0817,  1.0000],
         [ 0.8084,  2.0000],
         [ 1.6118,  3.0000],
         [ 0.8658,  4.0000],
         [-0.1583,  5.0000]],

        [[ 0.2881,  1.0000],
         [ 0.6946,  2.0000],
         [ 1.3760,  3.0000],
         [ 0.6098,  4.0000],
         [-0.1240,  5.0000]],

        [[ 0.1919,  1.0000],
         [ 0.7476,  2.0000],
         [ 1.4859,  3.0000],
         [ 0.7291,  4.0000],
         [-0.1400,  5.0000]]], grad_fn=<CatBackward>)

tensor([[-0.2841],
        [-0.2191],
        [-0.2495]], grad_fn=<AddmmBackward>)

Note that a typical tensor cannot be both integer and float at the same time, so the 1, 2, 3 will be stored as float 1.000.., 2.000... etc.
I suggest if your purpose is something like a fancy printing, then maybe look into torch's hook functions? For example, you can do something like:
import pandas as pd

def fc_hook_fn(module, input, output):
    print("\n" + "#" * 60)
    print(f"In layer {module}")
    print("#" * 60 + "\n")
    cols = [f"Neuron-{i + 1}" for i in range(output.shape[1])]
    idx = [f"Input-{i + 1}" for i in range(output.shape[0])]
    neuron_activations = pd.DataFrame(output.detach().numpy(), columns=cols, index=idx)
    print(neuron_activations)

net.fc.register_forward_hook(fc_hook_fn)

Now each time something passes through fc1, the function above will be triggered. You don't need to put your print(x) in the forward method.

############################################################
In layer Linear(in_features=1, out_features=5, bias=True)
############################################################

         Neuron-1  Neuron-2  Neuron-3  Neuron-4  Neuron-5
Input-1 -0.948735 -0.901034 -0.290353 -0.082616 -0.405337
Input-2 -0.725904 -0.801648 -0.302922 -0.045514 -0.580485
Input-3 -0.829738 -0.847960 -0.297065 -0.062802 -0.498870


Instruction: PyTorch - Tensors multiplication along new dimension

Input:
Sorry if already asked, but I can't find the words to look for on Google.
Let's say I have a tensor t1 of size [a,b] and a tensor t2 of size [c].
How can I output a tensor t3 of size [a,b,c], so that:
t3[0, :, 0] = t1[0, :] * t2[0]
t3[0, :, 1] = t1[0, :] * t2[1]
t3[0, :, 2] = t1[0, :] * t2[2]

...

t3[1, :, 0] = t1[1, :] * t2[0]
t3[1, :, 1] = t1[1, :] * t2[1]
t3[1, :, 2] = t1[1, :] * t2[2]

and so on, without a for loop ?
Thanks

Output:
Using torch.kron will give a tensor a x b*c, then using torch.Tensor.reshape could map to a tensor a x b x c :
t3=torch.kron(t1,t2).reshape(a,b,c)


Instruction: Pytorch: Is it able to make a convolution module without bias have bias again?

Input:
After instantiating a 2D convolution with conv = nn.Conv2d(8, 8, 3, bias=False), whose member bias should be None, is it able to give conv a legal bias again (whether with random initialization or determined values)?
I observed that bias in other default convolution modules is of the type Parameter, so I suspect there are extra procedures beyond simply conv.bias = torch.tensor(...) to make the new bias legal for conv.

Output:
Yes, it is possible to set the bias of the conv layer after instantiating. You can use the nn.Parameter class to create bias parameter and assign to conv object's bias attribute.
To show this I have created a simple Conv2d layer and assigned zero to the weights and ones to bias.

conv = nn.Conv2d(1, 1, 1, bias=False)

conv.weight.data = torch.zeros_like(conv.weight)
conv.bias = nn.Parameter(torch.ones((1,)))

inputs = torch.randn(1, 1, 1, 1)

print(conv(inputs))
# tensor([[[[1.]]]], grad_fn=<ThnnConv2DBackward>)


Instruction: Loading or building cuda engine crashes occassionaly after upgrading to TensorRT 7

Input:
I'm trying to run TensorRT inference in C++. Sometimes the code crashes when trying to build a new engine or load the engine from the file. It happens occasionally (sometimes it runs without any problem). I follow the below steps to prepare network:
initLibNvInferPlugins(&gLogger.getTRTLogger(), "");

if (mParams.loadEngine.size() > 0)
{
    std::vector<char> trtModelStream;
    size_t size{0};
    std::ifstream file(mParams.loadEngine, std::ios::binary);

    if (file.good())
    {
        file.seekg(0, file.end);
        size = file.tellg();
        file.seekg(0, file.beg);
        trtModelStream.resize(size);
        file.read(trtModelStream.data(), size);
        file.close();
    }
    IRuntime* infer_Runtime = nvinfer1::createInferRuntime(gLogger);
    if (mParams.dlaCore >= 0)
    {
        infer_Runtime->setDLACore(mParams.dlaCore);
    }

    mEngine = std::shared_ptr<nvinfer1::ICudaEngine>(
                infer_Runtime->deserializeCudaEngine(trtModelStream.data(), size, nullptr), samplesCommon::InferDeleter());

    gLogInfo << "TRT Engine loaded from: " << mParams.loadEngine << endl;

    infer_Runtime->destroy();
    if (!mEngine)
    {
        return false;
    }
    else
    {
        return true;
    }
}

auto builder = SampleUniquePtr<nvinfer1::IBuilder>(nvinfer1::createInferBuilder(gLogger.getTRTLogger()));

const auto explicitBatch = 1U << static_cast<uint32_t>(NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
auto network = SampleUniquePtr<nvinfer1::INetworkDefinition>(builder->createNetworkV2(explicitBatch));
auto config = SampleUniquePtr<nvinfer1::IBuilderConfig>(builder->createBuilderConfig());
auto parser = SampleUniquePtr<nvonnxparser::IParser>(nvonnxparser::createParser(*network, gLogger.getTRTLogger()));

mEngine = nullptr;

parser->parseFromFile(
        locateFile(mParams.onnxFileName, mParams.dataDirs).c_str(), static_cast<int>(gLogger.getReportableSeverity()));

// Calibrator life time needs to last until after the engine is built.
std::unique_ptr<IInt8Calibrator> calibrator;

config->setAvgTimingIterations(1);
config->setMinTimingIterations(1);
config->setMaxWorkspaceSize(4_GiB);
builder->setMaxBatchSize(mParams.batchSize);

mEngine = std::shared_ptr<nvinfer1::ICudaEngine>(
            builder->buildEngineWithConfig(*network, *config), samplesCommon::InferDeleter());

The error occurs here:
[05/12/2021-16:46:42] [I] [TRT] Detected 1 inputs and 1 output network tensors.

16:46:42: The program has unexpectedly finished.

This line crashes when loading existing engine:
mEngine = std::shared_ptr<nvinfer1::ICudaEngine(
    infer_Runtime->deserializeCudaEngine(trtModelStream.data(), size, nullptr), samplesCommon::InferDeleter());

Or when building the engine:
mEngine = std::shared_ptr<nvinfer1::ICudaEngine>(
            builder->buildEngineWithConfig(*network, *config), samplesCommon::InferDeleter());

More info:

TensorRT 7.2.3
Ubuntu 18.04
cuDNN 8.1.1
CUDA 11.1 update1
ONNX 1.6.0
Pytorch 1.5.0

Output:
Finally got it!
I rewrote the CMake.txt and add all required libs and paths and removed duplicate ones. That might be a lib conflict in cuBLAS.


Instruction: Force BERT transformer to use CUDA

Input:
I want to force the Huggingface transformer (BERT) to make use of CUDA.
nvidia-smi showed that all my CPU cores were maxed out during the code execution, but my GPU was at 0% utilization. Unfortunately, I'm new to the Hugginface library as well as PyTorch and don't know where to place the CUDA attributes device = cuda:0 or .to(cuda:0).
The code below is basically a customized part from german sentiment BERT working example
class SentimentModel_t(pt.nn.Module):
      def __init__(self, model_name: str = "oliverguhr/german-sentiment-bert"):
           DEVICE = "cuda:0" if pt.cuda.is_available() else "cpu"
           print(DEVICE)
           super(SentimentModel_t,self).__init__()

           self.model = AutoModelForSequenceClassification.from_pretrained(model_name).to(DEVICE)
           self.tokenizer = BertTokenizerFast.from_pretrained(model_name)

        def predict_sentiment(self, texts: List[str])-> List[str]:
            texts = [self.clean_text(text) for text in texts]
            # Add special tokens takes care of adding [CLS], [SEP], <s>... tokens in the right way for each model.
            input_ids = self.tokenizer.batch_encode_plus(texts,padding=True, add_special_tokens=True, truncation=True, max_length=self.tokenizer.max_len_single_sentence)
            input_ids = pt.tensor(input_ids["input_ids"])

            with pt.no_grad():
                logits = self.model(input_ids)

            label_ids = pt.argmax(logits[0], axis=1)

            labels = [self.model.config.id2label[label_id] for label_id in label_ids.tolist()]
            return labels

EDIT: After applying the suggestions of @KonstantinosKokos (see edited code above) I got a
RuntimeError: Input, output and indices must be on the current device

pointing to
        with pt.no_grad():
           logits = self.model(input_ids)

The full error code can be obtained down below:
<ipython-input-15-b843edd87a1a> in predict_sentiment(self, texts)
     23
     24         with pt.no_grad():
---> 25             logits = self.model(input_ids)
     26
     27         label_ids = pt.argmax(logits[0], axis=1)

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
   1364         return_dict = return_dict if return_dict is not None else self.config.use_return_dict
   1365
-> 1366         outputs = self.bert(
   1367             input_ids,
   1368             attention_mask=attention_mask,

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states, return_dict)
    859         head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)
    860
--> 861         embedding_output = self.embeddings(
    862             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    863         )

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    196
    197         if inputs_embeds is None:
--> 198             inputs_embeds = self.word_embeddings(input_ids)
    199         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    200

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    122
    123     def forward(self, input: Tensor) -> Tensor:
--> 124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
    126             self.norm_type, self.scale_grad_by_freq, self.sparse)

~/PycharmProjects/Test_project/venv/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

Output:
You can make the entire class inherit torch.nn.Module like so:
class SentimentModel_t(torch.nn.Module):
    def __init___(...)
    super(SentimentModel_t, self).__init__()
    ...

Upon initializing your model you can then call .to(device) to cast it to the device of your choice, like so:
sentiment_model = SentimentModel_t(...)
sentiment_model.to('cuda')

The .to() recursively applies to all submodules of the class, model being one of them (hugging face model inherit torch.nn.Module, thus providing an implementation for to()).
Note that this makes choosing device in the __init__() redundant: its now an external context that you can switch to/from easily.

Alternatively, you can hardcode the device by casting the contained BERT model directly into cuda (less elegant):
class SentimentModel_t():
        def __init__(self, ...):
            DEVICE = "cuda:0" if pt.cuda.is_available() else "cpu"
            print(DEVICE)

            self.model = AutoModelForSequenceClassification.from_pretrained(model_name).to(DEVICE)


Instruction: Pytorch Data Generator for extracting 2D images from many 3D cube

Input:
I'm struggling in creating a data generator in PyTorch to extract 2D images from many 3D cubes saved in .dat format
There is a total of 200 3D cubes each having a 128*128*128 shape. Now I want to extract 2D images from all of these cubes along length and breadth.
For example, a is a cube having size 128*128*128
So I want to extract all 2D images along length i.e., [:, i, :] which will get me 128 2D images along the length, and similarly i want to extract along width i.e., [:, :, i], which will give me 128 2D images along the width. So therefore i get a total of 256 2D images from 1 3D cube, and i want to repeat this whole process for all 200 cubes, there by giving me 51200 2D images.
So far I've tried a very basic implementation which is working fine but is taking approximately 10 minutes to run. I want you guys to help me create a more optimal implementation keeping in mind time and space complexity. Right now my current approach has a time complexity of O(n2), can we dec it further to reduce the time complexity
I'm providing below the current implementation
from os.path import join as pjoin
import torch
import numpy as np
import os
from tqdm import tqdm
from torch.utils import data

class DataGenerator(data.Dataset):

    def __init__(self, is_transform=True, augmentations=None):

        self.is_transform = is_transform
        self.augmentations = augmentations
        self.dim = (128, 128, 128)

        seismicSections = [] #Input
        faultSections = [] #Ground Truth
        for fileName in tqdm(os.listdir(pjoin('train', 'seis')), total = len(os.listdir(pjoin('train', 'seis')))):
            unrolledVolSeismic = np.fromfile(pjoin('train', 'seis', fileName), dtype = np.single) #dat file contains unrolled cube, we need to reshape it
            reshapedVolSeismic = np.transpose(unrolledVolSeismic.reshape(self.dim)) #need to transpose the axis to get height axis at axis = 0, while length (axis = 1), and width(axis = 2)

            unrolledVolFault = np.fromfile(pjoin('train', 'fault', fileName),dtype=np.single)
            reshapedVolFault = np.transpose(unrolledVolFault.reshape(self.dim))

            for idx in range(reshapedVolSeismic.shape[2]):
                seismicSections.append(reshapedVolSeismic[:, :, idx])
                faultSections.append(reshapedVolFault[:, :, idx])

            for idx in range(reshapedVolSeismic.shape[1]):
                seismicSections.append(reshapedVolSeismic[:, idx, :])
                faultSections.append(reshapedVolFault[:, idx, :])

        self.seismicSections = seismicSections
        self.faultSections = faultSections

    def __len__(self):
        return len(self.seismicSections)

    def __getitem__(self, index):

        X = self.seismicSections[index]
        Y = self.faultSections[index]

        return X, Y

Please Help!!!

Output:
why not storing only the 3D data in mem, and let the __getitem__ method "slice" it on the fly?
class CachedVolumeDataset(Dataset):
  def __init__(self, ...):
    super(...)
    self._volumes_x = # a list of 200 128x128x128 volumes
    self._volumes_y = # a list of 200 128x128x128 volumes

  def __len__(self):
    return len(self._volumes_x) * (128 + 128)

  def __getitem__(self, index):
    # extract volume index from general index:
    vidx = index // (128 + 128)
    # extract slice index
    sidx = index % (128 + 128)
    if sidx < 128:
      # first dim
      x = self._volumes_x[vidx][:, :, sidx]
      y = self._volumes_y[vidx][:, :, sidx]
    else:
      sidx -= 128
      # second dim
      x = self._volumes_x[vidx][:, sidx, :]
      y = self._volumes_y[vidx][:, sidx, :]
    return torch.squeeze(x), torch.squeeze(y)


Instruction: How to perform bicubic upsampling of image using pytorch?

Input:
I have png image. I want to upsample it using bicubic interpolation. I found this function in pytorch:
nn.functional.upsample(mode = "bicubic)

https://pytorch.org/docs/stable/generated/torch.nn.Upsample.html
But how to apply to my png image? Should I turn my image into some torch tesnor? I just haven't found any example of using this function exactly on png image

Output:
You can do this
import torch
import torchvision.transforms as transforms
from PIL import Image

t = transforms.ToTensor()
img = Image.open("Table.png")

b = torch.nn.functional.upsample(t(img).unsqueeze(0),(500,400),mode = "bicubic")

you can also apply Bicubic using Image
img = Image.open("Table.png")
re = img.resize((400, 400),Image.BICUBIC)


Instruction: Save Pytorch model state_dict with datetime in filename

Input:
I've been trying to save the state_dict of a Pytorch model with
torch.save(agent.qnetwork_local.state_dict(), filename) where  filename = datetime.now().strftime('%d-%m-%y-%H:%M_dqnweights.pth')
type(filename) returns str which shouldn't be a problem with torch.save() and it should output a non-empty file. Instead I get an empty file with just the date and time and nothing after that. Putting the date and in the middle of the filename results in an empty file with everything after the date and time cut off.
torch.save(agent.qnetwork_local.state_dict(), 'checkpoint1.pth') and any time I hardcode the string works and gives me the expected non-empty file.
What is going on and how do I fix this?
I am running this code in a Python v3.6.8 virtualenv with Pytorch v1.8.1+cpu on Windows 10.

Output:
The colon was the problem in filename = datetime.now().strftime('%d-%m-%y-%H:%m_dqnweights.pth') since it was running on windows.
Changing it to filename = datetime.now().strftime('%d-%m-%y-%H_%M_dqnweights.pth') works as expected.


Instruction: How do I check if a tokenizer/model is already saved

Input:
I am using HuggingFace Transformers with PyTorch. My modus operandi is to download a pre-trained model and save it in a local project folder.
While doing so, I can see that .bin file is saved locally, which stands for the model. However, I am also downloading and saving a tokenizer, for which I cannot see any associated file.
So, how do I check if a tokenizer is saved locally before downloading? Secondly, apart from the usual os.path.isfile(...) check, is there any other better way to prioritize local copy usage from a given location before downloading?

Output:
I've used this code in the past for this purpose. You can adapt it to your setting.
from tokenizers import BertWordPieceTokenizer
import urllib
from transformers import AutoTokenizer

def download_vocab_files_for_tokenizer(tokenizer, model_type, output_path, vocab_exist_bool=False):
    vocab_files_map = tokenizer.pretrained_vocab_files_map
    vocab_files = {}
    for resource in vocab_files_map.keys():
        download_location = vocab_files_map[resource][model_type]
        f_path = os.path.join(output_path, os.path.basename(download_location))
        if vocab_exist_bool != True:
            urllib.request.urlretrieve(download_location, f_path)
        vocab_files[resource] = f_path
    return vocab_files

model_type = 'bert-base-uncased'
#initialized tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_type)
#will do this part later

#retrieve vocab file if it's not there
output_path = os.getcwd()+'/vocab_files/'
vocab_file_name = 'bert-base-uncased-vocab.txt'
vocab_exist_bool = os.path.exists(output_path + vocab_file_name)

#get vocab files
vocab_files = download_vocab_files_for_tokenizer(tokenizer, model_type, output_path, vocab_exist_bool=vocab_exist_bool)