Instruction: Argument convention in PyTorch

Input:
I am new to PyTorch and while going through the examples, I noticed that sometimes functions have a different convention when accepting arguments. For example transforms.Compose receives a list as its argument:
transform=transforms.Compose([  # Here we pass a list of elements
   transforms.ToTensor(),
   transforms.Normalize(
      (0.4915, 0.4823, 0.4468),
      (0.2470, 0.2435, 0.2616)
   )
]))

At the same time, other functions receive the arguments individually (i.e. not in a list). For example torch.nn.Sequential:
torch.nn.Sequential(  # Here we pass individual elements
    torch.nn.Linear(1, 4),
    torch.nn.Tanh(),
    torch.nn.Linear(4, 1)
)

This has been a common typing mistake for me while learning.
I wonder if we are implying something when:

the arguments are passed as a list
the arguments are passed as individual items

Or is it simply the preference of the contributing author and should be memorized as is?
Update 1: Note that I do not claim that either format is better. I am merely complaining about lack of consistency. Of course (as Ivan stated in his answer) it makes perfect sense to follow one format if there is a good reason for it (e.g. transforms.Normalize). But if there is not, then I would vote for consistency.

Output:
This is not a convention, it is a design decision.
Yes, torch.nn.Sequential (source) receives individual items, whereas torchvision.transforms.Compose (source) receives a single list of items. Those are arbitrary design choices. I believe PyTorch and Torchvision are maintained by different groups of people, which might explain the difference. One could argue it is more coherent to have the inputs passed as a list since it is as a varied length, this is the approach used in more conventional programming languages such as C++ and Java. On the other hand you could argue it is more readable to pass them as a sequence of separate arguments instead, which what languages such as Python.
In this particular case we would have
>>> fn1([element_a, element_b, element_c]) # single list

vs
>>> fn2(element_a, element_b, element_c) # separate args

Which would have an implementation that resembles:
def fn1(elements):
    pass

vs using the star argument:
def fn2(*elements):
    pass

However it is not always up to design decision, sometimes the implementation is clear to take. For instance, it would be much preferred to go the list approach when the function has other arguments (whether they are positional or keyword arguments). In this case it makes more sense to implement it as fn1 instead of fn2. Here I'm giving second example with keyword arguments. Look a the difference in interface for the first set of arguments in both scenarios:
>>> fn1([elemen_a, element_b], option_1=True, option_2=True) # list

vs
>>> fn2(element_a, element_b, option_1=True, option_2=True) # separate

Which would have a function header which looks something like:
def fn1(elements, option_1=False, option_2=False)
   pass

While the other would be using a star argument under the hood:
def fn2(*elements, option_1=False, option_2=False)
   pass

If an argument is positioned after the star argument it essentially forces the user to use it as a keyword argument...
Mentioning this you can check out the source code for both Compose and Sequential and you will notice how both only expect a list of elements and no additional arguments afterwards. So in this scenario, it might have been preferred to go with Sequential's approach using the star argument... but this is just personal preference!


Instruction: openAI DALL-E ModuleNotFoundError

Input:
I installed DALL-E following the instructions on https://github.com/openai/DALL-E
and got :
---> 10 from dall_e          import map_pixels, unmap_pixels, load_model
11 from IPython.display import display, display_markdown
12
ModuleNotFoundError: No module named 'dall_e'

Output:
I found that it helped when I changed which Python version I was using.
It fixed my issue when I changed mine from 3.7.- to 3.10.7.


Instruction: Changing Learning Rate According to Layer Width in Pytroch

Input:
I am trying to train a network where the learning rate for each layer scales with 1/(layer width). Is there a way to do this in pytorch? I tried changing the learning rate in the optimizer and including it in my training loop but that didn't work. I've seen some people talk about this with Adam, but I am using SGD to train. Here are the chunks where I defined my model and training, if thats any help.
class ConvNet2(nn.Module):
    def __init__(self):
        super(ConvNet2, self).__init__()
        self.network = nn.Sequential(
        nn.Conv2d(3, 8, 3),
        nn.ReLU(),
        nn.Conv2d(8,32, 3),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),

        nn.Conv2d(32, 32, 3),
        nn.ReLU(),
        nn.Conv2d(32,32, 3),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),

        nn.Flatten(),

        nn.Linear(800, 10)
        )

    def forward(self, x):
        return self.network(x)

net2 = ConvNet2().to(device)

def train(network, number_of_epochs):
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(network.parameters(), lr=learning_rate)
    for epoch in range(number_of_epochs):  # loop over the dataset multiple times
        running_loss = 0.0
        for i, (inputs, labels) in enumerate(trainloader):
            # get the inputs
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = network(inputs)
            loss = criterion(outputs, labels)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = network(inputs)
            loss.backward()
            optimizer.step()

Output:
In the documentation you can see that you can specify "per-parameter options". Assuming you only want to specify the learning rate for the Conv2d layers (this is easily customizable in the code below) you could do something like this:
import torch
from torch import nn
from torch import optim
from pprint import pprint

class ConvNet2(nn.Module):
    def __init__(self):
        super(ConvNet2, self).__init__()
        self.network = nn.Sequential(
        nn.Conv2d(3, 8, 3),
        nn.ReLU(),
        nn.Conv2d(8,32, 3),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),

        nn.Conv2d(32, 32, 3),
        nn.ReLU(),
        nn.Conv2d(32,32, 3),
        nn.ReLU(),
        nn.MaxPool2d(2, 2),

        nn.Flatten(),

        nn.Linear(800, 10)
        )

    def forward(self, x):
        return self.network(x)

net2 = ConvNet2()

def getParameters(model):
    getWidthConv2D = lambda layer: layer.out_channels
    parameters = []
    for layer in model.children():
        paramdict = {'params': layer.parameters()}
        if (isinstance(layer, nn.Conv2d)):
            paramdict['lr'] = getWidthConv2D(layer) * 0.1 # Specify learning rate for Conv2D here
        parameters.append(paramdict)
    return parameters

optimizer = optim.SGD(getParameters(net2.network), lr=0.05)
print(optimizer)


Instruction: Custom layer caused the batch dimension mismatch in pytorch. How to fix this problem?

Input:
I have tried to train a GCN model.I defined the custom layer I needed. However, It cause some dimension mismatch when I do some batch training.
the codes are as following :
import math
import numpy as np
import torch

import torch.nn as nn
from torch.nn.parameter import Parameter
from torch.nn.modules.module import Module
import torch.nn.functional as F
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
from tqdm import tqdm

# =============================================================================
# model define
# =============================================================================
class GraphConvolution(Module):
    """
    Simple GCN layer, similar to https://arxiv.org/abs/1609.02907
    """

    def __init__(self, in_features, out_features, bias=True):
        super(GraphConvolution, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.FloatTensor(in_features, out_features))
        if bias:
            self.bias = Parameter(torch.FloatTensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        stdv = 1. / math.sqrt(self.weight.size(1))
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            self.bias.data.uniform_(-stdv, stdv)

    def forward(self, input, adj):
        support = torch.mm(input, self.weight)
        output = torch.spmm(adj, support)
        if self.bias is not None:
            return output + self.bias
        else:
            return output

    def __repr__(self):
        return self.__class__.__name__ + ' (' \
               + str(self.in_features) + ' -> ' \
               + str(self.out_features) + ')'

class GCN(nn.Module):
    def __init__(self, nfeat, nhid, nclass):
        super(GCN, self).__init__()

        self.gc1 = GraphConvolution(nfeat, nhid)
        self.gc2 = GraphConvolution(nhid, nclass)
        self.linear = nn.Linear(nclass, 1)
        # self.dropout = dropout

    def forward(self, x, adj):
        x = F.relu(self.gc1(x, adj))
        # x = F.dropout(x, self.dropout, training=self.training)
        x =  F.relu(self.gc2(x, adj))
        x = self.linear(x)
        return x

def train(dataloader, model, loss_fn, optimizer,adj):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(device), y.to(device)

        # Compute prediction error
        pred = model(X,adj)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def test(dataloader, model, loss_fn,adj):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            pred = model(X,adj)
            test_loss += loss_fn(pred, y).item()
            # correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= num_batches
    # correct /= size
    # Accuracy: {(100*correct):>0.1f}%,
    print(f"Test Error: \n  Avg loss: {test_loss:>8f} \n")

when I run the code :
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using {device} device")

model = GCN(1,1,1).to(device)
print(model)
# model(X).shape

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

epochs = 10
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataloader, model, loss_fn, optimizer,Adjacency_matrix)
    test(test_dataloader, model, loss_fn,Adjacency_matrix)
print("Done!")

I got the error :

when I looking inside this ,I find the model is working well when I drop the dimension of batch-size. How I need to do to tell the model that this dimension is the batch-size which don't need to compute?

Output:
the error you're seeing is due to you trying to matrix multiple a 3d tensor (your input) by your 2D weights.
To get around this you can simply reshape your data, as we only really care about the last dim when doing matmuls:
def forward(self, input, adj):
    b_size = input.size(0)
    input = input.view(-1, input.shape[-1])
    support = torch.mm(input, self.weight)
    output = torch.spmm(adj, support)
    output = output.view(b_size,-1,output.shape[-1])
    if self.bias is not None:
        return output + self.bias
    else:
        return output


Instruction: PyTorch nn.module won't unbatch operations

Input:
I have a nn.Module whose forward function takes in two inputs. Inside the function, I multiply one of the inputs x1 by a set of trainable parameters, and then concatenate them with the other input x2.
class ConcatMe(nn.Module):
    def __init__(self, pad_len, emb_size):
        super(ConcatMe, self).__init__()
        self.W = nn.Parameter(torch.randn(pad_len, emb_size).to(DEVICE), requires_grad=True)
        self.emb_size = emb_size

    def forward(self, x1: Tensor, x2: Tensor):
        cat = self.W * torch.reshape(x2, (1, -1, 1))
        return torch.cat((x1, cat), dim=-1)

From my understanding, one is supposed to be able to write operations in PyTorch's nn.Modules like we would for inputs with a batch size of 1. For some reason, this is not the case. I'm getting an error that indicates that PyTorch is still accounting for batch_size.
x1 =  torch.randn(100,2,512)
x2 = torch.randint(10, (2,1))
concat = ConcatMe(100, 512)
concat(x1, x2)

-----------------------------------------------------------------------------------
File "/home/my/file/path.py, line 0, in forward
    cat = self.W * torch.reshape(x2, (1, -1, 1))
RuntimeError: The size of tensor a (100) must match the size of tensor b (2) at non-singleton dimension 1

I made a for loop to patch the issue as shown below:
class ConcatMe(nn.Module):
    def __init__(self, pad_len, emb_size):
        super(ConcatMe, self).__init__()
        self.W = nn.Parameter(torch.randn(pad_len, emb_size).to(DEVICE), requires_grad=True)
        self.emb_size = emb_size

    def forward(self, x1: Tensor, x2: Tensor):
        batch_size = x2.shape[0]
        cat = torch.ones(x1.shape).to(DEVICE)

        for i in range(batch_size):
            cat[:, i, :] = self.W * x2[i]

        return torch.cat((x1, cat), dim=-1)

but I feel like there's a more elegant solution. Does it have something to do with the fact that I'm creating parameters inside nn.Module? If so, what solution can I implement that doesn't require a for loop.

Output:
From my understanding, one is supposed to be able to write operations in PyTorch's nn.Modules like we would for inputs with a batch size of 1.

I'm not sure where you got this assumption, it is definitely not true - on the contrary: you always need to write them in a way that they can handle the general case of an arbitrary batch dimension.
Judging from your second implementation it seems like you're trying to multiply two tensors with incompatible dimensions. So in order to fix that you'd have to define
        self.W = torch.nn.Parameter(torch.randn(pad_len, 1, emb_size), requires_grad=True)

To understand things like that better it would help to learn about broadcasting.


Instruction: How to compute Cross Entropy Loss for sequences

Input:
I have a sequence continuation/prediction task (input: a sequence of class indices, output: a sequence of class indices) and I use Pytorch.
My neural network returns a tensor of shape (batch_size, sequence_length, numb_classes) where the entries are a number proportional to the propability that the class with this index is the next class in the sequence.
My targets in the training data are of shape (batch_size, sequence_length) (just the sequences of the real predictions).
I want to use the CrossEntropyLoss
My question: How do I use the Cross Entropy Loss function? Which input shapes are required?
Thank you!

Output:
The documentation page of nn.CrossEntropyLoss clearly states:

Input: shape (C), (N, C) or (N, C, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss.
Target: If containing class indices, shape (), (N) or (N, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss where each value should be between [0, C). If containing class probabilities, the input and each value should be between [0, 1].

Just to be crystal clear, "input" refers to the output prediction of your model
while the "target" is the label tensor. In a nutshell, the target must have one less dimension than that of the input. This missing dimension in the target would contain each class logit value. Usually, we say the target is in the dense format, it only contains the class indices corresponding to the true labels.
The example you give corresponds to the use case of:
#input = (batch_size, sequence_length, numb_classes)
#target = (batch_size, sequence_length)

Which is the case of #input = (N, C, d_1) and #target = (N, d_1), i.e;, you need to permute the axes, or tranpose two axes from your input tensor such that it gets a shape of (batch_size, numb_classes, sequence_length) which is (N, C, d_1). You can do so with either torch.Tensor.transpose or torch.Tensor.permute:
>>> input.permute(0,2,1)

or
>>> input.transpose(1,2)


Instruction: How to pass to Linear data with variable length?

Input:
I am trying to do sequence classification by first passing data to RNN and then to Linear, normally I would just reshape the output from [batch_size, sequence_size, hidden_size] to [batch_size, sequence_size*hidden_size] to pass it to Linear, but in this case I have sequence of varying lengths, so the output of RNN might be for example [batch_size, 32, hidden_size] or [batch_size, 29, hidden_size], so I don’t know with what shape to initialize the Linear layer (in place of question marks in the code below). Is it at all possible?
    class RNN(nn.Module):
        def __init__(self, input_size, hidden_size, num_classes=4):
            super().__init__()
            self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
            self.fc = nn.Linear(hidden_size*????, num_classes)

        def forward(self, x):
            #x=[batch_size, sequence_length]
            out, h_n = self.rnn(x)  #out=[batch_size, sequence_length, hidden_size]
            out = torch.reshape(out, (BATCH_SIZE, -1)) #out=[batch_size, sequence_length*hidden_size]
            out = self.fc(out)  #out=[batch_size, num_classes]
            return x

Currently each batch is padded to the longest sequence in the batch, is it better to just pad all the sequences to the same length to get rid of this problem? Is changing shape of input to Linear causing some bad side effects?

Output:
Linear layers are meant to take fixed number of features as inputs. If you really want to pass a variable sized input, you could try some kind of imputation (e.g., create a linear layer suited for maximum length and whenever input size is lesser, pad with mean value of feature). But I don't think that's desirable in this scenario (or most of scenarios for that matter).
Since you want to do sequence classification, why don't you just pass a summary of inputs created by the RNN (output at last time step) instead of the whole output i.e., pass [batch_size, hidden_size] (last output of RNN) as input to linear layer. Then your linear layer will be hidden_size x num_classes
This is how the code would look like:
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes=4):
        super().__init__()
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        #x=[batch_size, sequence_length]
        out, h_n = self.rnn(x)  #out=[batch_size, sequence_length, hidden_size]
        out = out[:, -1, :]  #out=[batch_size, hidden_size]
        out = self.fc(out)  #out=[batch_size, num_classes]
        return x

This would work because RNN can learn to summarise the whole input into the last step. Also, you can try LSTM instead of RNN as it might be able to achieve this summarisation even better (since it can handle long-term dependencies more easily).


Instruction: mat1 and mat2 shapes cannot be multiplied (19x1 and 19x1)

Input:
I have a handmade dataset and all want to do is set a linear regression model with Pytorch.
These are the codes I wrote:
from torch.autograd import Variable

train_x = np.asarray([1,2,3,4,5,6,7,8,9,10,5,4,6,8,5,2,1,1,6])
train_y = train_x * 2

X = Variable(torch.from_numpy(train_x).type(torch.FloatTensor), requires_grad = False).view(19, 1)
y = Variable(torch.from_numpy(train_y).type(torch.FloatTensor), requires_grad = False)
from torch import nn

lr = nn.Linear(19, 1)

loss = nn.MSELoss()
optimizer = torch.optim.SGD(lr.parameters(), lr = 0.01)
output = lr(X) #error occurs here

I guess this is the simplest Pytorch neural network code in the world but it's still giving this error message:
mat1 and mat2 shapes cannot be multiplied (19x1 and 19x1)

I just did all the things on the book but it's still giving this error. Can you help me?

Output:
If you are using a torch.nn.Linear(a,b) as part of a network, then the input must be of shape (n, a), and the output will be of shape (n, b). Therefore you need to make sure that X has shape (n, 19) in your case, so modifying it with
...).view(1, 19)

would do the trick.


Instruction: How to Split a folder with multiple dataset into train and test using PyTorch

Input:
I have a folder with 48 ECG signal files. The files include .dat and .atr ECG signal records and annotation. I want to split them to train and test to train the AI model. I will be using PyTorch and I want to know a simple way to do this in Python.I prefer a custom split with certain number of files to be in train and the rest in test.
Eg: Train : ['101', '104','107']
Test : ['102', '105','106']
Thanks

Output:
Here first you need to store the Input and attribute location using
a dictionary in python with Input file name as key and Attribute file name as Value.
Then you can split the key of the dictionary and use that as input.
from glob import glob

MainFolder="<Your Folder Name>"

Data={}
for file in glob(MainFolder+"/*.dat"):
   At_file=file[:-3]+"atr"
   Data[file]=At_file

# Here Data would have Input and attribute file name as key and value pair

# To split the date:

Key_data=list(Data)
import random
random.shuffle(Key_data)

#Here you specify the split ratio of Training and Testing
split=int(len(Key_data)*(0.8))

Train_in=Key_data[:split]
Test_in=Key_data[split:]
Train_at=[Data[i] for i in Train_in]
Test_at=[Data[i] for i in Test_in]

print(Train_in,Train_at,Test_in,Test_at)

Here Train_in is the Input files and Train_at is its corresponding attribute files
This should solve your problem. Comment if you get any error in implementing the above code.


Instruction: classify labels to -1 or 1

Input:
I have a tensor with values between -1 and 1 . How can I get a new tensor such that where were negative values now there will be one and where were positive numbers now there will be 1? (efficiently)
Namely,
tensor1 = [-0.1, 0.5, 0.08]
new_tensor = [-1, 1, 1]

and zero will be -1 or 1

Output:
With numpy it is trivial:
import numpy as np
tensor1 = [-0.1, 0.5, 0.08]
new_tensor = np.sign(tensor1)
new_tensor[new_tensor==0] = 1


Instruction: PyTorch mat1 and mat2 shapes cannot be multiplied (4x460800 and 80000x16)

Input:
I'm trying to find road lanes using PyTorch. I created dataset and my model. But when I try to train my model, I get mat1 and mat2 shapes cannot be multiplied (4x460800 and 80000x16) error. I've tried other topic's solutions but those solutions didn't help me very much.
My dataset is bunch of road images with their validation images. I have .csv file that contains names of images (such as 'image1.jpg, image2.jpg'). Original size of images and validation images is 1280x720. I convert them 200x200 in my dataset code.
Road image:

Validation image:

Here's my dataset:
import os
import pandas as pd
import random

import torch
import torchvision.transforms.functional as TF
from torch.utils.data import Dataset
from torchvision import transforms
from PIL import Image

class Dataset(Dataset):
    def __init__(self, csv_file, root_dir, val_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.val_dir = val_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index, 0])
        image = Image.open(img_path).convert('RGB')
        mask_path = os.path.join(self.val_dir, self.annotations.iloc[index, 0])
        mask = Image.open(mask_path).convert('RGB')

        transform = transforms.Compose([
            transforms.Resize((200, 200)),
            transforms.ToTensor()
        ])

        if self.transform:
            image = self.transform(image)
            mask = self.transform(mask)

        return image, mask

My model:
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()

        self.cnn_layers = nn.Sequential(
            # Conv2d, 3 inputs, 128 outputs
            # 200x200 image size
            nn.Conv2d(3, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            # Conv2d, 128 inputs, 64 outputs
            # 100x100 image size
            nn.Conv2d(128, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            # Conv2d, 64 inputs, 32 outputs
            # 50x50 image size
            nn.Conv2d(64, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

        self.linear_layers = nn.Sequential(
            # Linear, 32*50*50 inputs, 16 outputs
            nn.Linear(32 * 50 * 50, 16),
            # Linear, 16 inputs, 3 outputs
            nn.Linear(16, 3)
        )

    def forward(self, x):
        x = self.cnn_layers(x)
        x = x.view(x.size(0), -1)
        x = self.linear_layers(x)
        return x

How to avoid this error and train my images on these validation images?

Output:
The answer: In your case, NN input has a shape (3, 1280, 720), not (3, 200, 200) as you want. Probably you have forgotten to modify transform argument in RNetDataset. It stays None, so transforms are not applied and the image is not resized. Another possibility is that it happens due to these lines:
        transform = transforms.Compose([
            transforms.Resize((200, 200)),
            transforms.ToTensor()
        ])

        if self.transform:
            image = self.transform(image)
            mask = self.transform(mask)

You have two variables named transform, but one with self. - maybe you messed them up. Verify it and the problem should go away.
How I came up with it: 460800 is clearly a tensor size after reshaping before linear layers. According to the architecture, tensor processed with self.cnn_layers should have 32 layers, so its height multiplied by width should give 460800 / 32 = 14400. Suppose that its height = H, width = W, so H x W = 14400. Let's understand, what was the original input size in this case? nn.MaxPool2d(kernel_size=2, stride=2) layer divides height and width by 2, and it happens three times. So, the original input size has been 8H x 8W = 64 x 14400 = 936000. Finally, notice that 936000 = 1280 * 720. This can't be a magical coincidence. Case closed!
Another suggestion: even if you apply transforms correctly, your code might not work. Suppose that you have an input of size (4, 3, 200, 200), where 4 is a batch size. Layers in your architecture will process this input as follows:
nn.Conv2d(3, 128, kernel_size=3, stride=1, padding=1) # -> (4, 128, 200, 200)
nn.MaxPool2d(kernel_size=2, stride=2) # -> (4, 128, 100, 100)
nn.Conv2d(128, 64, kernel_size=3, stride=1, padding=1) # -> (4, 64, 100, 100)
nn.MaxPool2d(kernel_size=2, stride=2) # -> (4, 64, 50, 50)
nn.Conv2d(64, 32, kernel_size=3, stride=1, padding=1) # -> (4, 32, 50, 50)
nn.MaxPool2d(kernel_size=2, stride=2) # -> (4, 32, 25, 25)

So, your first layer in self.linear_layers should be not nn.Linear(32 * 50 * 50, 16), but nn.Linear(32 * 25 * 25, 16). With this change, everything should be fine.


Instruction: Utility of wrapping tensor in Variable with requires_grad=False in legacy PyTorch

Input:
I'm using a codebase that was written in 2017/18 and I found the following code:
audio_norm = audio_norm.unsqueeze(0)
audio_norm = torch.autograd.Variable(audio_norm, requires_grad=False)

I am aware that wrapping tensors in Variable formerly allowed their gradients to be incorporated into the computation graph Torch builds for previous versions of Torch (now no longer needed) but I'm confused what the utility of wrapping a tensor in torch.autograd.Variable(my_tensor, requires_grad=False) would be.
Could someone explain if this was an idiom and what the analogous modern Torch code would be? My guess would be calling detach on the tensor to stop its gradients being tracked.
For reference, the relevant line from the codebase is line 45 from the data_utils.py script of NVIDIA's Tacotron 2 implementation. Thanks.

Output:
In PyTorch 0.3.1 and earlier, any tensor involved in a computation that needed to be tracked by autograd had to be wrapped in a Variable. Semantically Variable.requires_grad in PyTorch 0.3.1 and earlier is equivalent to Tensor.requires_grad now. Basically, requires_grad=False simply tells autograd that you will never need the gradient w.r.t. that variable/tensor. Mathematical operations are only ever recorded (i.e. a computation graph is constructed) if at least one input variable/tensor has requires_grad=True.
Note that any code using PyTorch newer than 0.3.1 does not actually require the use of Variable, this includes the code in the repository you provided (which explicitly requires PyTorch >= 1.0). In 0.4 the functionality of Variable was merged into the Tensor class. In modern PyTorch, you simply have to set the requires_grad attribute of the tensor to achieve the same behavior. By default, a new user-defined tensor is already constructed with requires_grad=False, so the modern equivalent of the code you posted is usually to just delete the Variable line. If you aren't sure if the tensor already has requires_grad == False then you could explicitly set it.
audio_norm = audio_norm.unsqueeze(0)
audio_norm.requires_grad_(False)

You can read the legacy documentation here for more information.


Instruction: Why am I wrong in the input of the tensor?

Input:
Here is my class for cnn.
class SimpleCnn(nn.Module):
def __init__(self, n_classes):
    super().__init__()
    self.layer1 = nn.Sequential(   # 224*224
        nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, padding=1),
        nn.ReLU(),
        nn.BatchNorm2d(64),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )
    self.layer2 = nn.Sequential(   # 112*112
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, padding=1),
        nn.ReLU(),
        nn.BatchNorm2d(128),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )
    self.layer3 = nn.Sequential(   # 56*56
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, padding=1),
        nn.ReLU(),
        nn.BatchNorm2d(256),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )
    self.layer3 = nn.Sequential(   # 28*28
        nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.ReLU(),
        nn.BatchNorm2d(512),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )
    self.layer4 = nn.Sequential(   # 14*14
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, padding=1),
        nn.ReLU(),
        nn.BatchNorm2d(512),
        nn.MaxPool2d(kernel_size=2, stride=2)
    )
    self.out1 = nn.Linear(512*7*7, 4096)   # 7*7
    self.out2 = nn.Linear(4096, n_classes)
def forward(self, x):
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)

    x = x.view(1, -1)
    x = self.out1(x)
    logits = self.out2(x)
    return logits

And it returns such a mistake.
RuntimeError: Given groups=1, weight of size [512, 256, 3, 3], expected input[64, 128, 56, 56] to have 256 channels, but got 128 channels instead.
I've seen other mistakes of such a type but can't find where I'm wrong here.
Thank you for your answer.

Output: In your code self.layer3 is first defined but then overwritten (a copy-pasta error I assume?). The error is thrown because in the redefinition of layer3 you assume the input has 256 channels, but the output from self.layer2 only has 128 channels.


Instruction: Saving Image in tensor array as jpg or png

Input:
I am trying to detect the face using mtcnn. The main aim is to detect face, crop and save the cropped image as jpg or png file type.  The code implemented is below.
from facenet_pytorch import MTCNN
from PIL import Image
import numpy as np
from matplotlib import pyplot as plt
img = Image.open("example.jpg")
mtcnn = MTCNN(margin=20, keep_all=True, post_process=False)
faces = mtcnn(img)
print(faces.shape)

This gives the shape
torch.Size([1, 3, 160, 160])

How to save this cropped portion as jpg file.

Output:
torch.save(faces, "faces.torch")

that wont be saved as an image, if you want to save it as an image:
img = Image.fromarray(faces.cpu().detach().numpy()[0])
img.save("faces.png")


Instruction: How to return extra loss from module forward function in PyTorch?

Input:
I made a module, that needs an extra loss term, e.g.
class MyModule:
  def forward(self, x):
    out = f(x)
    extra_loss = loss_f(self.parameters(), x)
    return out, extra_loss

I can't figure out how to make this module embeddable, for example, into a Sequential model: any regular module like Linear put after this one will fail because extra_loss causes the input to Linear to be a tuple, which Linear does not support.
So what I am looking for is extracting that extra loss after running the model forward
my_module = MyModule()

model = Sequential(
  my_module,
  Linear(my_module_outputs, 1)
)

output = model(x)
my_module_loss = ????
loss = mse(label, output) + my_module_loss

Does module composability support this scenario?

Output:
IMHO, hooks here is overreaction. Provided extra_loss is additive, we can use global variable like this:
class MyModule:
    extra_loss =0
    def forward(self, x):
        out = f(x)
        MyModule.extra_loss += loss_f(self.parameters(), x)
        return out

output = model(x)
loss = mse(label, output) + MyModule.extra_loss
MyModule.extra_loss =0


Instruction: TypeError: forward() got an unexpected keyword argument 'baseline value'. How do I correctly load a saved model in Skorch?

Input:
I saved my Skorch neural net model using the below code:
net_b = NeuralNetClassifier(
    Classifier_b,
    max_epochs=50,
    optimizer__momentum= 0.9,
    lr=0.1,
    device=device,
)

#Fit the model on the full data
net_b.fit(merged_X_train, merged_Y_train);

#Test saving
import pickle
with open('MLP.pkl', 'wb') as f:
    pickle.dump(net_b, f)

When I try to load this model again and run it against test data, I receive the following error:
TypeError: forward() got an unexpected keyword argument 'baseline value'

This is my code:
#Split the data
X_train, y_train, X_valid, y_valid,X_test, y_test = train_valid_test_split(rescaled_data, target = 'fetal_health',
                                        train_size=0.8, valid_size=0.1, test_size=0.1)

input_dim = f_df_toscale.shape[1]
output_dim = len(np.unique(f_target))
hidden_dim_a = 20
hidden_dim_b = 12
device = 'cpu'

class Classifier_b(nn.Module):
    def __init__(self,
                 input_dim = input_dim,
                 hidden_dim_a = hidden_dim_b,
                 output_dim = output_dim):

        super(Classifier_b, self).__init__()

        #Take the inputs and pass these to a hidden layer
        self.hidden = nn.Linear(input_dim,hidden_dim_b)

        #Take the hidden layer and pass it through an additional hidden layer
        self.hidden_b = nn.Linear(hidden_dim_a,hidden_dim_b)

        #Take the hidden layer and pass to a multi nerouon output
        self.output = nn.Linear(hidden_dim_b,output_dim)

    def forward(self, x):
        hidden = F.relu(self.hidden(x))
        hidden = F.relu(self.hidden_b(hidden))
        output = F.softmax(self.output(hidden))
        return output

#load the model
with open('MLP.pkl', 'rb') as f:
    model_MLP = pickle.load(f)

#Test the model
y_pred = model_MLP.predict(X_test)
ML = accuracy_score(y_test, y_pred)
print('The accuracy score for the MLP is ', ML)

When I run this model normally in the original notebook everything run fines. But when I try to load my model from a saved state I get the error. Any idea why? I have nothing called 'baseline value'.
Thanks

Output:
The save and load model can be problematic if the code changes. So it is better to use
save_params() and  load_params()
In your case
net_b.save_params(f_params='some-file.pkl')
To load the model first initialize (initializing is very important) and then load parameters
new_net.initialize()
new_net.load_params(f_params='some-file.pkl')


Instruction: Pytorch nn.CrossEntropyLoss() always returns 0

Input:
I am building a multi-class Vision Transformer Network. When passing my values through my loss function, it always returns zero. My output layer consisits of 37 Dense Layers with a softmax-unit on each on of them. criterion is created with nn.CrossEntropyLoss().The output of criterion is 0.0 for every iteration. I am using the colab notebook. I printed out the output and label for one iteration:
for output, label in zip(iter(ouputs_t), iter(labels_t)):
                      loss += criterion(
                          output,
                          # reshape label from (Batch_Size) to (Batch_Size, 1)
                          torch.reshape(label, (label.shape[0] , 1 ))
                          )

output: tensor([[0.1534],
        [0.5797],
        [0.6554],
        [0.4066],
        [0.2683],
        [0.1773],
        [0.7410],
        [0.5136],
        [0.5695],
        [0.3970],
        [0.4317],
        [0.7216],
        [0.8336],
        [0.4517],
        [0.4004],
        [0.5963],
        [0.3079],
        [0.5956],
        [0.3876],
        [0.2327],
        [0.7919],
        [0.2722],
        [0.3064],
        [0.9779],
        [0.8358],
        [0.1851],
        [0.2869],
        [0.3128],
        [0.4301],
        [0.4740],
        [0.6689],
        [0.7588]], device='cuda:0', grad_fn=<UnbindBackward0>)

label: tensor([[0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [1.],
        [1.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [0.],
        [1.],
        [0.]], device='cuda:0')

My Model:
class vit_large_patch16_224_multiTaskNet(nn.Module):
    def __init__(self, output_classes, frozen_feature_layers=False):
        super().__init__()

        vit_base_patch16_224 = timm.create_model('vit_large_patch16_224',pretrained=True)
        self.is_frozen = frozen_feature_layers
        # here we get all the modules(layers) before the fc layer at the end
        self.features = nn.ModuleList(vit_base_patch16_224.children())[:-1]
        self.features = nn.Sequential(*self.features)
        if frozen_feature_layers:
            self.freeze_feature_layers()

        # now lets add our new layers
        in_features = vit_base_patch16_224.head.in_features
        # it helps with performance. you can play with it
        # create more layers, play/experiment with them.
        self.fc0 = nn.Linear(in_features, 512)
        self.bn_pu = nn.BatchNorm1d(512, eps = 1e-5)
        self.output_modules = nn.ModuleList()
        for i in range(output_classes):
            self.output_modules.append(nn.Linear(512, 1))
        # initialize all fc layers to xavier
        for m in self.modules():
            if isinstance(m, nn.Linear):
                torch.nn.init.xavier_normal_(m.weight, gain = 1)

    def forward(self, input_imgs):
        output = self.features(input_imgs)
        final_cs_token = output[:, 0]
        output = self.bn_pu(F.relu(self.fc0(final_cs_token)))
        output_list= list()
        for output_modul in self.output_modules:
          output_list.append(torch.sigmoid(output_modul(output)))
        # Convert List to Tensor
        output_tensor = torch.stack(output_list)
        #
        output_tensor = torch.swapaxes(output_tensor, 0 , 1)
        return output_tensor

    def _set_freeze_(self, status):
        for n,p in self.features.named_parameters():
            p.requires_grad = status
        # for m in self.features.children():
        #     for p in m.parameters():
        #         p.requires_grad=status

    def freeze_feature_layers(self):
        self._set_freeze_(False)

    def unfreeze_feature_layers(self):
        self._set_freeze_(True)

Output:
You are in a multi-class classification scenario, which means you can consider your problem as c-binary class classification done in parallel (where c is the total number of class). Having output_t the logit tensor containing the values outputted by your model's last linear layer and target the ground-truth tensor containing the true classes states for each instance in the batch. You can apply nn.BCEWithLogitsLoss since it works with multi-dimensional tensors out of the box:
With dummy inputs:
>>> output_t = torch.rand(47, 32, 1)
>>> target = torch.randint(0, 2, (47, 32, 1)).float()

Then initializing and calling the loss function:
>>> loss = nn.BCEWithLogitsLoss()
>>> loss(output_t, target)
tensor(0.7246)


Instruction: Building 2 layer neural network with Pytorch

Input:
I am trying to build a simple 2 layer network, it has 2 inputs and 1 output, and the code is as follows:
num_input = 2
num_output = 1

# Input
x1 = torch.rand(1, 2)

# Weights
W1 = torch.rand(2,3)
W2 = torch.rand(3,1)

# biases
b1 = torch.rand(1,3)
b2 = torch.rand(1,1)

# Activation function
def f(inp):
    inp[inp >= 0] = 1
    inp[inp < 0] = 0
    return inp

# Predict output
out = f(torch.mm(x1, W1) + b1)
y=W2*out +b2
print(y)

# Check solution
assert list(y.size()) == [1, num_output], f"Incorrect output size ({y.size()})"
print("nice!")

from this code I always get incorrect output size, can anyone give me a hint, how can I get the correct output size?

Output:
y=out@W2 +b2

You were doing elementwise multiplication. That was not changing the size of the output as you desired.
Just to be clear python 3.5 and above can use this "@" syntax - which is doing the same thing as torch.mm() - that is matrix multiplication.
Dimensions: (Now)
Now you have (1,2) input multiplied with (2,3) weights and adding (1,3) bias. Shape is (1,3) and then you matrix mutiply with (1,3) and (3,1) output is (1,1) and bias to it making the final output size (1,1).

Dimensions (before)

Side Note:
Also you can use the nn.Linear to do all these operations easily without specifying the weights and biases like that.


Instruction: RuntimeError: shape '[128, 3, 5, 4, 5, 4]' is invalid for input of size 185856

Input:
I'm using the following code to extract descriptors from images using a Visual Transformer (vit_b_16) but I get the error: RuntimeError: shape '[128, 3, 5, 4, 5, 4]' is invalid for input of size 185856. Does anyone know what I'm doing wrong and how I can fix it?
def img_to_patch(x, patch_size, flatten_channels=True):

        B, C, H, W = x.shape
        x = x.reshape(B, C, H//patch_size, patch_size, W//patch_size, patch_size)
        x = x.permute(0, 2, 4, 1, 3, 5) # [B, H', W', C, p_H, p_W]
        x = x.flatten(1,2)              # [B, H'*W', C, p_H, p_W]
        if flatten_channels:
            x = x.flatten(2,4)          # [B, H'*W', C*p_H*p_W]
        return x

class AttentionBlock(nn.Module):

    def __init__(self, embed_dim, hidden_dim, num_heads, dropout=0.0):

        super().__init__()

        self.layer_norm_1 = nn.LayerNorm(embed_dim)
        self.attn = nn.MultiheadAttention(embed_dim, num_heads)
        self.layer_norm_2 = nn.LayerNorm(embed_dim)
        self.linear = nn.Sequential(
            nn.Linear(embed_dim, hidden_dim),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, embed_dim),
            nn.Dropout(dropout)
        )

    def forward(self, x):
        inp_x = self.layer_norm_1(x)
        x = x + self.attn(inp_x, inp_x, inp_x)[0]
        x = x + self.linear(self.layer_norm_2(x))
        return x

class VisionTransformer(nn.Module):

    def __init__(self, embed_dim, hidden_dim, num_channels, num_heads, num_layers, num_classes, patch_size, num_patches, dropout=0.0):

        super().__init__()

        self.patch_size = patch_size

        # Layers/Networks
        self.input_layer = nn.Linear(num_channels*(patch_size**2), embed_dim)
        self.transformer = nn.Sequential(*[AttentionBlock(embed_dim, hidden_dim, num_heads, dropout=dropout) for _ in range(num_layers)])
        self.mlp_head = nn.Sequential(
            nn.LayerNorm(embed_dim),
            nn.Linear(embed_dim, num_classes)
        )
        self.dropout = nn.Dropout(dropout)

        # Parameters/Embeddings
        self.cls_token = nn.Parameter(torch.randn(1,1,embed_dim))
        self.pos_embedding = nn.Parameter(torch.randn(1,1+num_patches,embed_dim))

    def forward(self, x):
        # Preprocess input
        print(x.shape)
        x = img_to_patch(x, self.patch_size)
        print(x.shape)
        B, T, _ = x.shape
        x = self.input_layer(x)

        # Add CLS token and positional encoding
        cls_token = self.cls_token.repeat(B, 1, 1)
        x = torch.cat([cls_token, x], dim=1)
        x = x + self.pos_embedding[:,:T+1]

        # Apply Transforrmer
        x = self.dropout(x)
        x = x.transpose(0, 1)
        x = self.transformer(x)

        # Perform classification prediction
        cls = x[0]
        out = self.mlp_head(cls)
        return out

class ViT(pl.LightningModule):

    def __init__(self, model_kwargs, lr):
        super().__init__()
        self.save_hyperparameters()
        self.model = VisionTransformer(**model_kwargs)
       # self.example_input_array = next(iter(train_loader))[0]

    def forward(self, x):
        return self.model(x)

and I'm initialising the Transformer like this:
if network_variant == 'vb16':
    net = ViT(model_kwargs={
               'embed_dim': 256,
               'hidden_dim': 512,
               'num_heads': 8,
               'num_layers': 6,
               'patch_size': 4,
               'num_channels': 3,
               'num_patches': 64,
               'num_classes': num_classes,
               'dropout': 0.2
                },
                lr=3e-4)

This is my first time using PyTorch and Vision Transformers so I'm really not sure what I'm doing wrong.

Output:
The error is saying that numpy is trying to fill a matrix with dimensions 128 * 35454, which requires 153,600 elements. However, the data before the reshape has 185,856 elements. Most likely you're miscounting some indices. The difference is 32256=12843*21, which provides some hints about which indices you're probably miscounting... The only place you're calling reshape() is the second line of img_to_patch(), so I assume that's where the error comes from. (Including the traceback in your question would help confirm this.)
Check the sizes of the variables leading up to that line and confirm they're what you expect.
Looking at your code, the most likely problem is that you're rounding off when dividing H and W by patch_size. If H and W aren't multiples of patch_size, you'll need something to decide which pixels to drop: np.reshape() won't make that decision on its own.


Instruction: Efficiently add numbers to the end of each dimension of a tensor

Input:
I have a tensor x of shape (n, 200). I want to make it shape (n, 218), by appending a tensor of 18 numbers to the end of every "row" of the current tensor. n varies based on the batch size, so I want a way to do this for any n.
As of right now, I have a working solution, but I was wondering if there is a built-in way to do this, I did not see anything in the documentation particularly.
My method is:
import torch.nn.functional as F
x = F.pad(input = x, (0, 18, 0, 0)) # pad each tensor in dim 2 with 18 zeroes
for index in range(x.shape[0]):
    x[index][-18] = nums_to_add # nums_to_add is a tensor with size (1,18)

This works just fine, but I was wondering if there is any easier way to do this, without first padding the zeroes.

Output:
torch.cat() is what you are looking for. Here is a snippet:
import torch
a = torch.randint(1,10,(3,4))
b =  torch.randint(1,10,(3,2))
print(a)
print(b)
a = torch.cat((a,b),axis=1) # axis should be one here
print(a)

Output
tensor([[2, 5, 3, 8],
        [3, 9, 5, 3],
        [9, 4, 9, 9]])
tensor([[6, 4],
        [1, 1],
        [8, 3]])
tensor([[2, 5, 3, 8, 6, 4],
        [3, 9, 5, 3, 1, 1],
        [9, 4, 9, 9, 8, 3]])

Now here is a similar example just used repeat to make it same shape in dim=0 so that we can concatenate it easily. (Trying to follow OP's suggestion exactly)
import torch
a = torch.randint(1,10,(5,200)) # shape(5,200)
b =  torch.randint(1,10,(1,18)).repeat((5,1)) # shape(5,18)
a = torch.cat((a,b),axis=1) # axis should be one here
print(a.shape) # (5,218)

The only tricky part of the above solution is the repeat() part (If you can say it complicated ...), which basically repeats this tensor along the specified dimensions. Check here.


Instruction: Does pytorch support multiple tensor CAddTable?

Input:
I found in official doc that CAddTable should be done as
x = x1 + x2 # instead of CAddTable(x1, x2) in older version

and PyTorch would do the rest of things like autograd
But how about if I have multiple tensors, aka. changing the input above from two tensors to a list of tensors. Could PyTorch still do the similar things?

Output:
Just for a clean display of the code snip in the comment:
x = torch.stack((x1, x2, x3, x4), dim=0)
y = torch.sum(x, dim=0, keepdim=False) # same shape as x1, x2...


Instruction: Pytorch with CUDA local installation fails on Ubuntu

Input:
I am trying to install PyTorch with CUDA.
I followed the instructions (installation using conda) mentioned in
https://pytorch.org/get-started/locally/

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c
pytorch

The conda install command runs without giving any error:
conda list displays the following:
# Name                    Version                   Build  Channel

cudatoolkit               11.3.1               h2bc3f7f_2
pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
torch                     1.10.2                   pypi_0    pypi
torchaudio                0.11.0               py39_cu113    pytorch
torchvision               0.11.3                   pypi_0    pypi

But when I check whether GPU driver and CUDA is enabled and accessible by PyTorch
torch.cuda.is_available()

returns false.
Prior to Pytorch installation, I checked and confirmed the pre-requisites mentioned in
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#system-requirements
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions
Here are my ubuntu server details:
Environment:

OS/kernel:

Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-154-generic x86_64)

Footnote under the table: Table 1. Native Linux Distribution Support in CUDA 11.6
mentions

For Ubuntu LTS on x86-64, the Server LTS kernel (e.g. 4.15.x for
18.04) is supported in CUDA 11.6.

GCC

gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0

GLIBC

ldd (Ubuntu GLIBC 2.27-3ubuntu1.5) 2.27

GPU

GeForce GTX 1080 Ti

Kernel headers and development packages
$ uname -r
4.15.0-176-generic

As per my understanding, conda pytorch installation with CUDA will install the CUDA driver too.
I am not sure where did I went wrong.
Thanks in advance.
EDIT:
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Nov__3_21:07:56_CDT_2017
Cuda compilation tools, release 9.1, V9.1.85

nvcc shows CUDA version 9.1
whereas
$ nvidia-smi
Wed May 11 06:44:31 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0 Off |                  N/A |
| 25%   40C    P8    11W / 250W |     18MiB / 11177MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
| 25%   40C    P8    11W / 250W |      2MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
| 25%   35C    P8    11W / 250W |      2MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      4119      G   /usr/lib/xorg/Xorg                             9MiB |
|    0      4238      G   /usr/bin/gnome-shell                           6MiB |
+-----------------------------------------------------------------------------+

nvidia-smi shows CUDA version 10.0
https://varhowto.com/check-cuda-version/
This article mentions that nvcc refers to CUDA-toolkit whereas nvidia-smi refers to NVIDIA driver.
Q1: Does it shows that there are two different CUDA installation at the system wide level?
Nvidia Cudatoolkit vs Conda Cudatoolkit
The CUDA toolkit (version 11.3.1) I am installing in my conda environment is different from the one installed as system wide level (which is shown by the output of nvcc and nvidia-smi).
Q2: As per the above stackoverflow thread answer, they can be separate. Or is it the reason for my failure to install cuda locally?

Output:
I have solved the issue.
Disclaimer: I am a newbie in CUDA.
Following answer is based on a) what I have read in other threads  b) my experience based on those discussions.
Core Logic:
CUDA driver's version >= CUDA runtime version
Reference: Different CUDA versions shown by nvcc and NVIDIA-smi

In most cases, if nvidia-smi reports a CUDA version that is
numerically equal to or higher than the one reported by nvcc -V, this
is not a cause for concern. That is a defined compatibility path in
CUDA (newer drivers/driver API support "older" CUDA toolkits/runtime
API).

As I am using conda's cudatoolkit:

Driver API: nvidia-smi
Runtime API: conda's cudatoolkit

For cudatoolkit 11.3.1, I was using nvidia-smi CUDA Version: 10.0
Solution:
Upgrade NVIDIA drivers.
Upgraded the NVIDIA drivers following the instruction at https://linuxconfig.org/how-to-install-the-nvidia-drivers-on-ubuntu-18-04-bionic-beaver-linux
Post upgradation, here's the output of nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 27%   46C    P8    12W / 250W |     19MiB / 11177MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:06:00.0 Off |                  N/A |
| 25%   44C    P8    11W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 25%   39C    P8    11W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3636      G   /usr/lib/xorg/Xorg                  9MiB |
|    0   N/A  N/A      4263      G   /usr/bin/gnome-shell                6MiB |
+-----------------------------------------------------------------------------+

Now driver version(11.4) >= runtime version (11.3.1)
PyTorch is now able to use CUDA with GPU:
In [1]: import torch

In [2]: torch.cuda.is_available()
Out[2]: True


Instruction: Define flatten layer in neural network using pytorch

Input:
I'm trying to define a flatten layer before initiating fully connected layer. As my input is a tensor with shape (512, 2, 2), so I want to flatten this tensor before FC layers.
I used to get this error:
empty(): argument 'size' must be tuple of ints, but found element of type Flatten at pos 2

import torch.nn as nn
class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        self.flatten=nn.Flatten()
        self.fc1=nn.Linear(self.flatten,512)
        self.fc2=nn.Linear(512,256)
        self.fc3=nn.Linear(256,3)

    def forward(self,x):
        x=self.flatten(x) # Flatten layer
        x=torch.ReLU(self.fc1(x))
        x=torch.ReLU(self.fc2(x))
        x=torch.softmax(self.fc3(x))
        return x

Output:
This line is not correct:
        self.fc1 = nn.Linear(self.flatten, 512)

the first argument in_features for nn.Linear should be int not the nn.Module
in your case you defined flatten attribute as a nn.Flatten module:
        self.flatten = nn.Flatten()

to fix this issue, you have to pass in_features equals to the number of feature after flattening:
        self.fc1 = nn.Linear(n_features_after_flatten, 512)


Instruction: Issues with Pytorch and Torchvision on MAC M1 - python / conda

Input:
Im trying to run Pytorch and Torchvision on Mac M1. I follow these instructions successfully install pytorch and run it natively on apple - https://betterprogramming.pub/how-to-install-pytorch-on-apple-m1-series-512b3ad9bc6
Issues comes when trying to install Torchvision - I conda install torchvision (version 0.22). But when I try to import it I get the error :
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/__init__.py", line 2, in <module>
    from torchvision import datasets
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/datasets/__init__.py", line 9, in <module>
    from .fakedata import FakeData
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/datasets/fakedata.py", line 3, in <module>
    from .. import transforms
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/transforms/__init__.py", line 1, in <module>
    from .transforms import *
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/transforms/transforms.py", line 17, in <module>
    from . import functional as F
  File "/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/torchvision/transforms/functional.py", line 5, in <module>
    from PIL import Image, ImageOps, ImageEnhance, PILLOW_VERSION
ImportError: cannot import name 'PILLOW_VERSION' from 'PIL' (/Users/gracecolverd/miniconda3/envs/pytorch_m2/lib/python3.8/site-packages/PIL/__init__.py)

It seems to be trying to pull a deprecated function PILLOW_VERSION, but when I check the  PIL init file referenced, there is no mention of the PILLOW_VERSION
"""Pillow (Fork of the Python Imaging Library)

Pillow is the friendly PIL fork by Alex Clark and Contributors.
    https://github.com/python-pillow/Pillow/

Pillow is forked from PIL 1.1.7.

PIL is the Python Imaging Library by Fredrik Lundh and Contributors.
Copyright (c) 1999 by Secret Labs AB.

Use PIL.__version__ for this Pillow version.

;-)
"""

from . import _version

# VERSION was removed in Pillow 6.0.0.
# PILLOW_VERSION was removed in Pillow 9.0.0.
# Use __version__ instead.
__version__ = _version.__version__
del _version

_plugins = [
    "BlpImagePlugin",
    "BmpImagePlugin",
    "BufrStubImagePlugin",
    "CurImagePlugin",
    "DcxImagePlugin",
    "DdsImagePlugin",
    "EpsImagePlugin",
    "FitsStubImagePlugin",
    "FliImagePlugin",
    "FpxImagePlugin",
    "FtexImagePlugin",
    "GbrImagePlugin",
    "GifImagePlugin",
    "GribStubImagePlugin",
    "Hdf5StubImagePlugin",
    "IcnsImagePlugin",
    "IcoImagePlugin",
    "ImImagePlugin",
    "ImtImagePlugin",
    "IptcImagePlugin",
    "JpegImagePlugin",
    "Jpeg2KImagePlugin",
    "McIdasImagePlugin",
    "MicImagePlugin",
    "MpegImagePlugin",
    "MpoImagePlugin",
    "MspImagePlugin",
    "PalmImagePlugin",
    "PcdImagePlugin",
    "PcxImagePlugin",
    "PdfImagePlugin",
    "PixarImagePlugin",
    "PngImagePlugin",
    "PpmImagePlugin",
    "PsdImagePlugin",
    "SgiImagePlugin",
    "SpiderImagePlugin",
    "SunImagePlugin",
    "TgaImagePlugin",
    "TiffImagePlugin",
    "WebPImagePlugin",
    "WmfImagePlugin",
    "XbmImagePlugin",
    "XpmImagePlugin",
    "XVThumbImagePlugin",
]

class UnidentifiedImageError(OSError):
    """
    Raised in :py:meth:`PIL.Image.open` if an image cannot be opened and identified.
    """

    pass

Ive checked the folder above to see if there is a PILLOW init but havnt found anything. Would love to fix this so I dont have to use colab forever! Thanks

Output:
Solved this - the issues was in the Torchvision functional.py script.

In the error it mentioned which files were trying to pull this function
. I searched in /Users/gracecol/miniconda3/envs/tv/lib/python3.10/site-packages/torchvision/transforms

for PILLOW_VERSION and in functional.py I had to replace PILLOW_VERSION with "__version__"


Instruction: Pytorch model object has no attribute 'predict' BERT

Input:
I had train a BertClassifier model using pytorch. After creating my best.pt I would like to make in production my model and using it to predict and classifier starting from a sample, so I resume them from the checkpoint. Otherwise after put it in evaluation and freeze model, I use .predict to make in work on my sample but I'm encountering this Attribute Error. I had also inizialize it before calling the checkpoint. When I am wrong? Thank you for your help!
def save_ckp(state, is_best, checkpoint_path, best_model_path):
    """
    function created to save checkpoint, the latest one and the best one.
    This creates flexibility: either you are interested in the state of the latest checkpoint or the best checkpoint.
    state: checkpoint we want to save
    is_best: is this the best checkpoint; min validation loss
    checkpoint_path: path to save checkpoint
    best_model_path: path to save best model
    """
    f_path = checkpoint_path
    # save checkpoint data to the path given, checkpoint_path
    torch.save(state, f_path)
    # if it is a best model, min validation loss
    if is_best:
        best_fpath = best_model_path
        # copy that checkpoint file to best path given, best_model_path
        shutil.copyfile(f_path, best_fpath)

def load_ckp(checkpoint_fpath, model, optimizer):
    """
    checkpoint_path: path to save checkpoint
    model: model that we want to load checkpoint parameters into
    optimizer: optimizer we defined in previous training
    """
    # load check point
    checkpoint = torch.load(checkpoint_fpath)
    # initialize state_dict from checkpoint to model
    model.load_state_dict(checkpoint['state_dict'])
    # initialize optimizer from checkpoint to optimizer
    optimizer.load_state_dict(checkpoint['optimizer'])
    # initialize valid_loss_min from checkpoint to valid_loss_min
    valid_loss_min = checkpoint['valid_loss_min']
    # return model, optimizer, epoch value, min validation loss
    return model, optimizer, checkpoint['epoch'], valid_loss_min.item()

#Create the BertClassfier class
class BertClassifier(nn.Module):
    """Bert Model for Classification Tasks."""
    def __init__(self, freeze_bert=True):
        """
         @param    bert: a BertModel object
         @param    classifier: a torch.nn.Module classifier
         @param    freeze_bert (bool): Set `False` to fine-tune the BERT model
        """
        super(BertClassifier, self).__init__()

        .......

    def forward(self, input_ids, attention_mask):
        ''' Feed input to BERT and the classifier to compute logits.
         @param    input_ids (torch.Tensor): an input tensor with shape (batch_size,
                       max_length)
         @param    attention_mask (torch.Tensor): a tensor that hold attention mask
                       information with shape (batch_size, max_length)
         @return   logits (torch.Tensor): an output tensor with shape (batch_size,
                       num_labels) '''
         # Feed input to BERT
        outputs = self.bert(input_ids=input_ids,
                             attention_mask=attention_mask)

         # Extract the last hidden state of the token `[CLS]` for classification task
        last_hidden_state_cls = outputs[0][:, 0, :]

         # Feed input to classifier to compute logits
        logits = self.classifier(last_hidden_state_cls)

        return logits

def initialize_model(epochs):
    """ Initialize the Bert Classifier, the optimizer and the learning rate scheduler."""
    # Instantiate Bert Classifier
    bert_classifier = BertClassifier(freeze_bert=False)

    # Tell PyTorch to run the model on GPU
    bert_classifier = bert_classifier.to(device)

    # Create the optimizer
    optimizer = AdamW(bert_classifier.parameters(),
                      lr=lr,    # Default learning rate
                      eps=1e-8    # Default epsilon value
                      )

    # Total number of training steps
    total_steps = len(train_dataloader) * epochs

    # Set up the learning rate scheduler
    scheduler = get_linear_schedule_with_warmup(optimizer,
                                                num_warmup_steps=0, # Default value
                                                num_training_steps=total_steps)
    return bert_classifier, optimizer, scheduler

def train(model, train_dataloader, val_dataloader, valid_loss_min_input, checkpoint_path, best_model_path, start_epochs, epochs, evaluation=True):

    """Train the BertClassifier model."""
    # Start training loop
    logging.info("--Start training...\n")

    # Initialize tracker for minimum validation loss
    valid_loss_min = valid_loss_min_input

    for epoch_i in range(start_epochs, epochs):
        # =======================================
        #               Training
        # =======================================
        # Print the header of the result table
        logging.info((f"{'Epoch':^7} | {'Batch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}"))

        # Measure the elapsed time of each epoch
        t0_epoch, t0_batch = time.time(), time.time()

        # Reset tracking variables at the beginning of each epoch
        total_loss, batch_loss, batch_counts = 0, 0, 0

        # Put the model into the training mode
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):
            batch_counts +=1
            # Load batch to GPU
            b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

            # Zero out any previously calculated gradients
            model.zero_grad()

            # Perform a forward pass. This will return logits.
            logits = model(b_input_ids, b_attn_mask)

            # Compute loss and accumulate the loss values
            loss = loss_fn(logits, b_labels)
            batch_loss += loss.item()
            total_loss += loss.item()

            # Perform a backward pass to calculate gradients
            loss.backward()

            # Clip the norm of the gradients to 1.0 to prevent "exploding gradients"
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and the learning rate
            optimizer.step()
            scheduler.step()

            # Print the loss values and time elapsed for every 20 batches
            if (step % 500 == 0 and step != 0) or (step == len(train_dataloader) - 1):
                # Calculate time elapsed for 20 batches
                time_elapsed = time.time() - t0_batch

                # Print training results
                logging.info(f"{epoch_i + 1:^7} | {step:^7} | {batch_loss / batch_counts:^12.6f} | {'-':^10} | {'-':^9} | {time_elapsed:^9.2f}")

                # Reset batch tracking variables
                batch_loss, batch_counts = 0, 0
                t0_batch = time.time()

        # Calculate the average loss over the entire training data
        avg_train_loss = total_loss / len(train_dataloader)

        logging.info("-"*70)
        # =======================================
        #               Evaluation
        # =======================================
        if evaluation == True:
            # After the completion of each training epoch, measure the model's performance
            # on our validation set.
            val_loss, val_accuracy = evaluate(model, val_dataloader)

            # Print performance over the entire training data
            time_elapsed = time.time() - t0_epoch

            logging.info(f"{epoch_i + 1:^7} | {'-':^7} | {avg_train_loss:^12.6f} | {val_loss:^10.6f} | {val_accuracy:^10.6f} | {time_elapsed:^9.2f}")

            logging.info("-"*70)
        logging.info("\n")

         # create checkpoint variable and add important data
        checkpoint = {
            'epoch': epoch_i + 1,
            'valid_loss_min': val_loss,
            'state_dict': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }

        # save checkpoint
        save_ckp(checkpoint, False, checkpoint_path, best_model_path)

        ## TODO: save the model if validation loss has decreased
        if val_loss <= valid_loss_min:
            print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(valid_loss_min,val_loss))
            # save checkpoint as best model
            save_ckp(checkpoint, True, checkpoint_path, best_model_path)
            valid_loss_min = val_loss

    logging.info("-----------------Training complete--------------------------")

def evaluate(model, val_dataloader):
    """After the completion of each training epoch, measure the model's performance on our validation set."""
    # Put the model into the evaluation mode. The dropout layers are disabled during the test time.
    model.eval()

    # Tracking variables
    val_accuracy = []
    val_loss = []

    # For each batch in our validation set...
    for batch in val_dataloader:
        # Load batch to GPU
        b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)

        # Compute logits
        with torch.no_grad():
            logits = model(b_input_ids, b_attn_mask)

        # Compute loss
        loss = loss_fn(logits, b_labels)
        val_loss.append(loss.item())

        # Get the predictions
        preds = torch.argmax(logits, dim=1).flatten()

        # Calculate the accuracy rate
        accuracy = (preds == b_labels).cpu().numpy().mean() * 100
        val_accuracy.append(accuracy)

    # Compute the average accuracy and loss over the validation set.
    val_loss = np.mean(val_loss)
    val_accuracy = np.mean(val_accuracy)

    return val_loss, val_accuracy

bert_classifier, optimizer, scheduler = initialize_model(epochs=n_epochs)
train(model = bert_classifier ......)

bert_classifier, optimizer, scheduler = initialize_model(epochs=n_epochs)
model, optimizer, start_epoch, valid_loss_min = load_ckp(r"./best_model/best_model.pt", bert_classifier, optimizer)

model.eval()
model.freeze()

sample = {
  "seq": "ABCDE",}

predictions = model.predict(sample)

AttributeError: 'BertClassifier' object has no attribute 'predict'

Output:
Generally, people wrote the prediction function for you.
If not, you need to handle the low level stuff.
After this line, you loaded the trained parameters.
model, optimizer, start_epoch, valid_loss_min = load_ckp(r"./best_model/best_model.pt", bert_classifier, optimizer)
After that, you need to do the model.forward(intput_seq,this_attention_mask_maybe_null).
You can see the forward method here is the : def forward(self, input_ids, attention_mask) in the model.


Instruction: TypeError: cross_entropy_loss(): argument ‘input’ (position 1) must be Tensor, not Linear

Input:
I was following a youtube video and learning to make a chat bot, the teacher explained this step to make the training model, the code compiled perfectly for the teacher but im getting an error. What am i doing wrong?
for epoch in range(num_epochs):
    for (words, labels) in train_loader:
        words = words.to(device)
        labels = labels.to(device, dtype=torch.int64)

        outputs= model(words)
        loss = criterion(outputs,labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if(epoch +1) % 100 == 0:
        print(f'epoch {epoch+1}/{epoch}, loss = {loss.item():.4f}')

print(f'epoch {epoch+1}/{epoch}, loss = {loss.item():.4f}')

NeuralNet:
class NeuralNet(nn.Module):
    def __init__(self,input_size, hidden_size,num_classes):
        super(NeuralNet,self).__init__()
        self.l1 = nn.Linear(input_size,hidden_size)
        self.l2 = nn.Linear(hidden_size,hidden_size)
        self.l3 = nn.Linear(hidden_size,num_classes)
        self.relu = nn.ReLU()

    def forward(self,x):
        out = self.l1(x)
        out = self.relu(out)
        out = self.l2(out)
        out = self.relu(out)
        out = self.l3
        return out

Output:
The issue is with the NeuralNet code specifically in the line:
out = self.l3
You are setting out to be the Linear layer instead of calling the linear layer on the data. Change it to
out = self.l3(out)
and it will work


Instruction: how to set the gradient for a network in pytorch

Input:
I have a model in pytorch. The model can take any shape but lets assume this is the model
torch_model =  Sequential(
    Flatten(),
    Linear(28 * 28, 256),
    Dropout(.4),
    ReLU(),
    BatchNorm1d(256),
    ReLU(),
    Linear(256, 128),
    Dropout(.4),
    ReLU(),
    BatchNorm1d(128),
    ReLU(),
    Linear(128, 10),
    Softmax()
)

I am using SGD optimizer, I want to set the gradient for each of the layers so the SGD algorithm will move the parameters in the direction I want.
Lets say I want all the gradients for all the layers to be ones (torch.ones_like(gradient_shape)) how can I do this?
Thanks?

Output:
In PyTorch, with a model defined as yours above, you can iterate over the layers like this:
for layer in list(torch_model.modules())[1:]:
  print(layer)

You have to add the [1:] since the first module returned is the sequential module itself. In any layer, you can access the weights with layer.weight. However, it is important to remember that some layers, like Flatten and Dropout, don't have weights. A way to check, and then add 1 to each weight would be:
for layer in list(torch_model.modules())[1:]:
  if hasattr(layer, 'weight'):
    with torch.no_grad():
      for i in range(layer.weight.shape[0]):
          layer.weight[i] = layer.weight[i] + 1

I tested the above on your model and it does add 1 to every weight. Worth noting that it won't work without torch.no_grad() as you don't want pytorch tracking the changes.


Instruction: How to get weights from layer of neural network that is defined as a class object in Pytorch?

Input:
I am developing something on top of an already existing framework of code, and I am having some trouble extracting weights from a neural network defined as a class. Code below
import numpy as np
import torch
import torch.nn as nn

class Solver:
    class Head(nn.Module):
        def __init__(self, base):
            super().__init__()
            self.base = base
            self.last_layer = nn.Linear(100, 10)

        def forward(self, x):
            x = self.base(x)
            x = self.last_layer(x)
            return x

    def __init__(self, bases, HeadClass=None):
        self.base = bases
        if HeadClass:
            self.head = self.Head(self.base)
        else:
            self.head = self.Head(self.base)
        print('Head Class:',self.head)

class Full_Solver:
    class Base(nn.Module):
        def __init__(self):
            super().__init__()
            self.linear_1 = nn.Linear(1, 100)
            self.linear_2 = nn.Linear(100, 100)
            self.linear_3 = nn.Linear(100, 100)

        def forward(self, x):
            x = self.linear_1(x)
            x = torch.tanh(x)
            x = self.linear_2(x)
            x = torch.tanh(x)
            x = self.linear_3(x)
            x = torch.tanh(x)
            return x

    def __init__(self, BaseClass=Base()):
        self.base = BaseClass
        print('Base model:',self.base)
        print('Base model type:',type(self.base))

        solver_1 = Solver(self.base)
        print('Full model:',solver_1)
        print('Full model type:',type(solver_1))

xx = Full_Solver()

In the Full_Solver class, I am defining a Base Neural Network, which will add a Head based on some conditions in the Solver class (I have left out all the conditions for sake of brevity). When I do print('Full model:',solver_1), the output I am getting is Full model: <__main__.Solver object at 0x7f83a82e9cd0>. How do I extract the output weights from this class object? (Assume that I just want to extract the randomly assigned weights from this)

Output: In this case it would be print('Full model:',solver_1.head.last_layer.weight)


Instruction: How to calculate log_softmax for list of tensors without breaking autograd in Pytorch

Input:
I'm trying to calculate the log_softmax function of a list of tensors, i.e., a list [t_1, t_2, ..., t_n] where each t_i is of type torch.tensor and each t_i can be of a different, arbitrary shape. I do not want to apply the log_softmax function to each t_i separately, but to all of them as if they were part of the same unique tensor. The output of this function should be a list of tensors with the same shape as the input. Lastly, as I will apply this function to the end layer of a neural network, I want to be able to differentiate this function, i.e., the gradients must flow through it.
Pytorch provides the class torch.nn.LogSoftmax, but I cannot use it as it expects a single tensor as input, instead of a list of tensors. Additionally, I want to calculate the log_softmax function efficiently and in a stable way. To achieve that, I want to use the log-sum-exp trick. Lastly, I want to ignore the last value of the first element of the list (see code snippet below), i.e., not apply log_softmax to it.
This is my current implementation:
def log_softmax(pred_tensors):
    minus_inf = -1000 # Constant that represents minus infinity

    # Calculate the max value
    c = max([preds.amax() if preds is not None else minus_inf for preds in pred_tensors])

    # Calculate log(sum(e^(x_i-c)))
    log_sum_exp = 0
    for r in range(len(pred_tensors)):
        if pred_tensors[r] is not None:

            # Arity 0 -> ignore nullary predicate corresponding to termination condition
            curr_sum =  torch.sum(torch.exp(pred_tensors[r][:-1] - c))   if r == 0 else \
                        torch.sum(torch.exp(pred_tensors[r] - c))
            log_sum_exp += curr_sum

    log_sum_exp = torch.log(log_sum_exp)

    # Calculate log_softmax (apply log_softmax to the original tensor) (except to the termination condition)
    for r in range(len(pred_tensors)):
        if pred_tensors[r] is not None:
            # Arity 0 -> ignore nullary predicate corresponding to termination condition
            if r == 0:
                pred_tensors[r][:-1] -= log_sum_exp + c
            else:
                pred_tensors[r] -= log_sum_exp + c

    return pred_tensors

I have tested it and it works. However, I think my implementation may be breaking the autograd of Pytorch, in lines c = max([preds.amax() if preds is not None else minus_inf for preds in pred_tensors]) and log_sum_exp += curr_sum.
So, my questions are: Is my implementation really breaking autograd? If it is, can you provide an alternative implementation that works with autograd?

Output:
I also posted this question on the Pytorch Forum and was solved there. I post the solution below:
def _log_softmax(self, pred_tensors):
    # Remove the nullary predicate associated with the termination condition, so that it does not
    # affect the log_softmax computation
    term_cond_value = pred_tensors[0][-1]
    pred_tensors[0] = pred_tensors[0][:-1]

    # Calculate log_sum_exp of all the values in the tensors of the list
    # 1) flatten each tensor in the list
    # 2) concatenate them as a unique tensor
    # 3) calculate log_sum_exp
    log_sum_exp = torch.logsumexp(torch.cat([preds.flatten() if preds is not None else torch.empty(0, dtype=torch.float32) for preds in pred_tensors]), dim=-1)

    # Use log_sum_exp to calculate the log_softmax of the tensors in the list
    for r in range(len(pred_tensors)):
        if pred_tensors[r] is not None:
            pred_tensors[r] -= log_sum_exp

    # Append the nullary predicate corresponding to the termination condition
    pred_tensors[0] = torch.cat([pred_tensors[0], term_cond_value.reshape(1)]) # We need reshape() to transform from tensor of dimension 0 to dimension 1

    return pred_tensors

Basically, I firstly removed from the list of tensors the element pred_tensors[0][-1], so that it did not affect the calculations, and appended it to the final list of tensors. Then, since I could not concatenate a list of tensors of different sizes, I first flattened them and then used torch.cat to concatenate them, before using torch.logsumexp to calculate the log_sum_exp with all the values in all the tensors of the list. Then, this value was finally used to calculate the log_softmax of each tensor value, obtaining a list of output tensors with the same shape as the input.


Instruction: Keras: Mimic PyTorch's conv2d and linear/dense weight initialization?

Input: I am porting a model from PyTorch to Keras/Tensorflow, and I want to make sure I'm using the same algorithm for weight initialization. How do I mimic PyTorch's weight initialization in Keras?

Output:
If you refactor the PyTorch initialization code, you'll find that the weight initialization algorithm is surprisingly simple. The comment in that code is correct; just read that comment and mimic it.
Here's working Keras / Tensorflow code that mimics it:
import tensorflow as tf
from tensorflow.keras import layers

class PytorchInitialization(tf.keras.initializers.VarianceScaling):
    def __init__(self, seed=None):
        super().__init__(
            scale=1 / 3, mode='fan_in', distribution='uniform', seed=seed)

# Conv layer
conv = layers.Conv2D(32, 3, activation="relu", padding="SAME",
                     input_shape=(28, 28, 1),
                     kernel_initializer=PytorchInitialization(),
                     bias_initializer=PytorchInitialization())

# Dense / linear layer
classifier = layers.Dense(10,
                          kernel_initializer=PytorchInitialization(),
                          bias_initializer=PytorchInitialization(),


Instruction: What it means when your model can't overfit a small batch of data?

Input:
I am trying to train RNN model to classify sentences into 4 classes, but it doesn’t seem to work. I tried to overfit 4 examples (blue line) which worked, but even as little as 8 examples (red line) is not working, let alone the whole dataset.
I tried different learning rates and sizes of hidden_size and embedding_size but it doesn’t seem to help, what am I missing? I know that if the model is not able to overfit small batch it means the capacity should be increased but in this case increasing capacity has no effect.
The architecture is as follows:
class RNN(nn.Module):
    def __init__(self, embedding_size=256, hidden_size=128, num_classes=4):
        super().__init__()
        self.embedding = nn.Embedding(len(vocab), embedding_size, 0)
        self.rnn = nn.RNN(embedding_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        #x=[batch_size, sequence_length]
        x = self.embedding(x) #x=[batch_size, sequence_length, embedding_size]
        _, h_n = self.rnn(x)  #h_n=[1, batch_size, hidden_size]
        h_n = h_n.squeeze(0)
        out = self.fc(h_n)  #out=[batch_size, num_classes]
        return out

Input data is tokenized sentences, padded with 0 to the longest sentence in the batch, so as an example one sample would be: [2784, 9544, 1321, 120, 0, 0]. The data is from AG_NEWS dataset from torchtext datasets.
The training code:
model = RNN().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LR)
model.train()

for epoch in range(NUM_EPOCHS):
    epoch_losses = []
    correct_predictions = []
    for batch_idx, (labels, texts) in enumerate(train_loader):
        scores = model(texts)
        loss = criterion(scores, labels)

        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

        epoch_losses.append(loss.item())
        correct = (scores.max(1).indices==labels).sum()
        correct_predictions.append(correct)

    epoch_avg_loss = sum(epoch_losses)/len(epoch_losses)
    epoch_avg_accuracy = float(sum(correct_predictions))/float(len(labels))

Output: The issue was due to the vanishing gradient, it got resolved by using batch normalization.


Instruction: How to find score probability of classification model result in pytorch?

Input:
I'm new to pytorch using this I've trained a image classification model, when I test the model with the image I only get label , if I want to get probability of prediction of that class how can I get that ?
 test_image = test_image_tensor.view(1,3,300,300)
 model.eval()
 out = model(test_image)
 ps = torch.exp(out)
 topk,topclass = ps.topk(1,dim=1)
 class_name = idx_to_class[topclass.cpu().numpy()[0][0]]

I'm using above code for prediction which gives only class name  , if I want label score of prediction how can I get it?
Any help or suggestion on this will be appreciated

Output:
The probabilities are the softmax of the predictions:
class_prob = torch.softmax(out, dim=1)
# get most probable class and its probability:
class_prob, topclass = torch.max(class_prob, dim=1)
# get class names
class_name = idx_to_class[topclass.cpu().numpy()[0][0]]


Instruction: nn.CorssEntropyLoss documentation example is not correct?

Input:
Tested the documentation example of nn.CrossEntropyLoss. It doesn't seem to work.
loss = nn.CrossEntropyLoss()

input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5).softmax(dim=1)
output = loss(input, target)
print(output)

error:
RuntimeError: 1D target tensor expected, multi-target not supported

Output:
You might need to upgrade your torch version.
pip install torch --upgrade


Instruction: How to get intermediate output grad in Pytorch model

Input:
we can get loss of last layer by loss = loss_fn(y_pred, y_true), and results in a loss: Tensor
then we call loss.backward() to do back propagation.
after optimizer.step() we could see updated model.parameters()
taking below example
y = Model1(x) # with optimizer1
z = Model2(y) # with optimizer2
loss = loss_fn(z, z_true)
loss.backward()
optimizer2.optimize() # update Model2 parameters

# in order to update Model1 parameters I think we should do
y.backward(grad_tensor=the_output_gradient_from_Model2)
optimizer1.optimize()

How to get the intermediate back propagation result? e.g. the gradient of output grad, which will be taken by y_pred.backward(grad_tensor=grad).
Update: The solution is setting required_grad=True and take Tensor x.grad. Thanks for the answers.
PS: The scenario is I am doing a federated learning, the model is split into 2 parts. The first part takes input and forward to second part. And it need the second part to calculate the loss and back propagate the loss to first part, so that the first part takes the loss and do its own back propagation.

Output:
I will assume you're referring to intermediate gradients when you say "loss of a specific layer".
You can access the gradient of the layer with respect to the output loss by accessing the grad attribute on the parameters of your model which require gradient computation.
Here is a simplistic setup:
>>> f = nn.Sequential(
       nn.Linear(10,5),
       nn.Linear(5,2),
       nn.Linear(2, 2, bias=False),
       nn.Sigmoid())

>>> x = torch.rand(3, 10).requires_grad_(True)
>>> f(x).mean().backward()

Navigate through all the parameters per layer:
>>> for n, c in f.named_children():
...    for p in c.parameters():
...       print(f'<{n}>:{p.grad}')

<0>:tensor([[-0.0054, -0.0034, -0.0028, -0.0058, -0.0073, -0.0066, -0.0037, -0.0044,
             -0.0035, -0.0051],
            [ 0.0037,  0.0023,  0.0019,  0.0040,  0.0050,  0.0045,  0.0025,  0.0030,
              0.0024,  0.0035],
            [-0.0016, -0.0010, -0.0008, -0.0017, -0.0022, -0.0020, -0.0011, -0.0013,
             -0.0010, -0.0015],
            [ 0.0095,  0.0060,  0.0049,  0.0102,  0.0129,  0.0116,  0.0066,  0.0077,
              0.0063,  0.0091],
            [ 0.0005,  0.0003,  0.0002,  0.0005,  0.0006,  0.0006,  0.0003,  0.0004,
              0.0003,  0.0004]])
<0>:tensor([-0.0090,  0.0062, -0.0027,  0.0160,  0.0008])
<1>:tensor([[-0.0035,  0.0035, -0.0026, -0.0106, -0.0002],
            [-0.0020,  0.0020, -0.0015, -0.0061, -0.0001]])
<1>:tensor([-0.0289, -0.0166])
<2>:tensor([[0.0355, 0.0420],
            [0.0354, 0.0418]])


Instruction: Why is the actual runtime still cpu after I set cuda for tensor

Input:
I want to gpu-accelerate a custom function, and below is the original function:
import numpy as np

def py_cpu_nms(dets, thresh):
    """Pure Python NMS baseline."""
    x1 = dets[:, 0]
    y1 = dets[:, 1]
    x2 = dets[:, 2]
    y2 = dets[:, 3]
    scores = dets[:, 4]

    areas = (x2 - x1 + 1) * (y2 - y1 + 1)
    order = scores.argsort()[::-1]

    keep = []
    while order.size > 0:
        i = order[0]
        keep.append(i)
        xx1 = np.maximum(x1[i], x1[order[1:]])
        yy1 = np.maximum(y1[i], y1[order[1:]])
        xx2 = np.minimum(x2[i], x2[order[1:]])
        yy2 = np.minimum(y2[i], y2[order[1:]])

        w = np.maximum(0.0, xx2 - xx1 + 1)
        h = np.maximum(0.0, yy2 - yy1 + 1)
        inter = w * h
        ovr = inter / (areas[i] + areas[order[1:]] - inter)

        inds = np.where(ovr <= thresh)[0]
        order = order[inds + 1]

    return keep

It would use the CPU to compute, but that wasn't fast enough so I wanted to speed it up directly with PyTorch, which I converted into a Torch implementation:
import numpy as np
import torch

def py_cpu_nms(dets, thresh):
    """Pure Python NMS baseline."""
    dets = torch.from_numpy(dets)
    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    dets.cuda()
    x1 = dets[:, 0]
    y1 = dets[:, 1]
    x2 = dets[:, 2]
    y2 = dets[:, 3]
    scores = dets[:, 4]

    areas = ((x2 - x1 + 1) * (y2 - y1 + 1)).cuda()
    order = torch.argsort(scores)
    order = torch.flip(order, dims=[0])
    keep = []
    while order.size()[0] > 0:
        i = order[0]
        keep.append(i)
        xx1 = torch.maximum(x1[i], x1[order[1:]]).cuda()
        yy1 = torch.maximum(y1[i], y1[order[1:]]).cuda()
        xx2 = torch.minimum(x2[i], x2[order[1:]]).cuda()
        yy2 = torch.minimum(y2[i], y2[order[1:]]).cuda()

        w = torch.maximum(torch.tensor(0.0), xx2 - xx1 + 1).cuda()
        h = torch.maximum(torch.tensor(0.0), yy2 - yy1 + 1).cuda()
        inter = (w * h).cuda()
        ovr = inter / (areas[i] + areas[order[1:]] - inter)

        inds = torch.where(ovr <= thresh)[0].cuda()
        order = order[inds + 1].cuda()

    return keep

But in fact, all the calculations still use the CPU, does anyone know why?

Output: cuda() is no in-place. Change your line 3 of the function to dets = dets.cuda().


Instruction: How to apply max_length to truncate the token sequence from the left in a HuggingFace tokenizer?

Input:
In the HuggingFace tokenizer, applying the max_length argument specifies the length of the tokenized text. I believe it truncates the sequence to max_length-2 (if truncation=True) by cutting the excess tokens from the right. For the purposes of utterance classification, I need to cut the excess tokens from the left, i.e. the start of the sequence in order to preserve the last tokens. How can I do that?
from transformers import AutoTokenizer

train_texts = ['text 1', ...]
tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base')
encodings = tokenizer(train_texts, max_length=128, truncation=True)

Output:
Tokenizers have a truncation_side parameter that should set exactly this.
See the docs.


Instruction: Can autograd handle repeated use of the same layer in the same depth of the computation graph?

Input: I have a network which works as follows: The input is split in half; the first half is put through some convolutional layers l1, then the second half is put through the same layers l1 (after the output for the first half of the input has been computed), then the two output representations are concatenated and put through additional layers l2 at once. Now my question (similar to Can autograd in pytorch handle a repeated use of a layer within the same module? but not quite the same setting as in the other question, the same layer was reused in different depths of the computation graph, whereas here, the same layer is used twice within the same depth) is: does autograd handle this properly? I.e. is the backpropagation error for l1 computed with respect to both of its forward passes and the weights are adapted w.r.t. both of these at once?

Output: Autograd does not care how many times you "use" something. This is not how it works. it just builds a graph behind the scenes of the dependencies, using something twice just makes a graph that is not a line, but it will not affect its execution.


Instruction: The input of the forward part of my model is a tuple, cannot be converted to onnx format?

Input:
Test Code:
    #!/usr/bin/env python
    # -*- coding:utf-8 -*-
    import torch
    import torch.nn as nn

    class Model(nn.Module):
        def __init__(self):
            super(Model, self).__init__()
            self.linear = nn.Linear(32, 16)
            self.relu1 = nn.ReLU(inplace=True)
            self.relu2 = nn.ReLU(inplace=True)
            self.fc = nn.Linear(32, 2)

        def forward(self, x):
            x1, x2 = x
            x1 = self.linear(x1)
            x1 = self.relu1(x1)
            x2 = self.linear(x2)
            x2 = self.relu2(x2)
            out = torch.cat((x1, x2), dim=-1)
            out = self.fc(out)
            return out

    model = Model()
    model.eval()

    x1 = torch.randn((2, 10, 32))
    x2 = torch.randn((2, 10, 32))
    x = (x1, x2)

    torch.onnx.export(model,
                  x,
                  'model.onnx',
                  input_names=["input"],
                  output_names=["output"],
                  dynamic_axes={'input': {0: 'batch'}, 'output': {0: 'batch'}}
                  )
    print("Done")

How to convert the above code to onnx?
The input of the forward part of my model is a tuple, cannot be converted to onnx format?
thanks!
The input of the forward part of my model is a tuple, which cannot be converted to onnx format according to the existing methods. Can you tell me how to solve it

Output:
Looking at this issue and this other issue, the parameters are unpacked by default so you need to provide a tuple as argument to torch.onnx.export:
torch.onnx.export(model,
   args=(x,),
   f='model.onnx',
   input_names=["input"],
   output_names=["output"],
   dynamic_axes={'input': {0: 'batch'}, 'output': {0: 'batch'}})


Instruction: Saving the weights of a Pytorch .pth model into a .txt or .json

Input:
I am trying to save the the weights of a pytorch model into a .txt or .json. When writing it to a .txt,
#import torch
model = torch.load("model_path")
string = str(model)
with open('some_file.txt', 'w') as fp:
     fp.write(string)

I get a file where not all the weights are saved, i.e there are ellipsis throughout the textfile. I cannot write it to a JSON since the model has tensors, which are not JSON serializable [unless there is a way that I do not know?] How can I save the weights in the .pth file to some format such that no information is lost, and can be easily seen?
Thanks

Output:
When you are doing str(model.state_dict()), it recursively uses str method of elements it contains. So the problem is how individual element string representations are build. You should increase the limit of lines printed in individual string representation:
torch.set_printoptions(profile="full")

See the difference with this:
import torch
import torchvision.models as models
mobilenet_v2 = models.mobilenet_v2()
torch.set_printoptions(profile="default")
print(mobilenet_v2.state_dict()['features.2.conv.0.0.weight'])
torch.set_printoptions(profile="full")
print(mobilenet_v2.state_dict()['features.2.conv.0.0.weight'])

Tensors are currently not JSON serializable.