Instruction: How can I use my own data to test this Convolutional Neural Network on PyTorch?

Input:
so recently I have been following this tutorial from sentdex on convolutional neural networks and I have been trying to implement his code to test the trained neural network with my own images (in this case, I just pick random pictures from the dataset used in his program). So my intention is to train the neural network, test it and finally save it so I can later load it on a separate python file to use the already trained NN on one single image.
The dataset he uses is "dogs vs cats from microsoft". This is the code where I wrote the neural network program ("main.py").
import cv2
import numpy as np
from tqdm import tqdm
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

REBUILD_DATA = False # set to true to one once, then back to false unless you want to change something in your training data.

class DogsVSCats():
    IMG_SIZE = 100
    CATS = "PetImages/Cat"
    DOGS = "PetImages/Dog"
    TESTING = "PetImages/Testing"
    LABELS = {CATS: 0, DOGS: 1}
    training_data = []

    catcount = 0
    dogcount = 0

    def make_training_data(self):
        for label in self.LABELS:
            print(label)
            for f in tqdm(os.listdir(label)):
                if "jpg" in f:
                    try:
                        path = os.path.join(label, f)
                        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                        img = cv2.resize(img, (self.IMG_SIZE, self.IMG_SIZE))
                        self.training_data.append([np.array(img), np.eye(2)[self.LABELS[label]]])  # do something like print(np.eye(2)[1]), just makes one_hot
                        #print(np.eye(2)[self.LABELS[label]])

                        if label == self.CATS:
                            self.catcount += 1
                        elif label == self.DOGS:
                            self.dogcount += 1

                    except Exception as e:
                        pass
                        #print(label, f, str(e))

        np.random.shuffle(self.training_data)
        np.save("training_data.npy", self.training_data)
        print('Cats:',dogsvcats.catcount)
        print('Dogs:',dogsvcats.dogcount)

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50, 50).view(-1, 1, 50, 50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net()

print(net)

if REBUILD_DATA:
    dogsvcats = DogsVSCats()
    dogsvcats.make_training_data()

training_data = np.load("training_data.npy", allow_pickle=True)
print(len(training_data))

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1  # lets reserve 10% of our data for validation
val_size = int(len(X)*VAL_PCT)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

BATCH_SIZE = 100
EPOCHS = 1

def train(net):
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
            #print(f"{i}:{i+BATCH_SIZE}")
            batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)
            batch_y = train_y[i:i+BATCH_SIZE]

            net.zero_grad()

            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()    # Does the update

        print(f"Epoch: {epoch}. Loss: {loss}")

def test(net):
    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_X))):
            real_class = torch.argmax(test_y[i])
            net_out = net(test_X[i].view(-1, 1, 50, 50))[0]  # returns a list,
            predicted_class = torch.argmax(net_out)

            if predicted_class == real_class:
                correct += 1
            total += 1

    print("Accuracy: ", round(correct/total, 3))

train(net)
test(net)

PATH = './object_detection.pth'
torch.save(net.state_dict(), PATH)

After training the neural network, I want to load it in this next program and simply test the images on the NN. However, every time I run this program, the neural network is trained and tested again, which makes this process much longer and annoying. And also, I think when I run this program and then input the image into the NN, the whole "main.py" is being run.
Please, can someone help me with this? It would be amazing, as I am using this as a base to my Bachelor's thesis. Potentially I would also like to adapt this code to run my own entire dataset through it, it would be incredible if someone would help me do this, as I am a newbie on pytorch.
import cv2
from main import Net, train, test
import numpy as np

classes = ('cat', 'dog')

imsize = 50

net = Net()
net.load_state_dict(torch.load('./object_detection.pth'))

def image_loader(image_name):
    image = cv2.imread(image_name, cv2.IMREAD_GRAYSCALE)
    image = cv2.resize(image, (imsize, imsize))
    image = np.array(image)
    image = torch.Tensor(image)/255
    image = image.view(-1, 1, 50, 50)
    return image

test_image = image_loader("./PetImages/Cat/1021.jpg")

result = net(test_image)
_, predicted = torch.max(result, 1)

print(result)
print(classes[predicted[0]])

Output:
The problem you are facing is nothing related to NN, but the importing part.
In the second code snippet, you import classes and functions your first code snippet. At the same time, the statements will also be executed all the code inside and it is not what we want.
The simplest way to solve it is to collect your code inside a if case to avoid execution during being imported.
The result may look like this:
import cv2
import numpy as np
from tqdm import tqdm
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class DogsVSCats():
    IMG_SIZE = 100
    CATS = "PetImages/Cat"
    DOGS = "PetImages/Dog"
    TESTING = "PetImages/Testing"
    LABELS = {CATS: 0, DOGS: 1}
    training_data = []

    catcount = 0
    dogcount = 0

    def make_training_data(self):
        for label in self.LABELS:
            print(label)
            for f in tqdm(os.listdir(label)):
                if "jpg" in f:
                    try:
                        path = os.path.join(label, f)
                        img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                        img = cv2.resize(img, (self.IMG_SIZE, self.IMG_SIZE))
                        self.training_data.append([np.array(img), np.eye(2)[self.LABELS[label]]])  # do something like print(np.eye(2)[1]), just makes one_hot
                        #print(np.eye(2)[self.LABELS[label]])

                        if label == self.CATS:
                            self.catcount += 1
                        elif label == self.DOGS:
                            self.dogcount += 1

                    except Exception as e:
                        pass
                        #print(label, f, str(e))

        np.random.shuffle(self.training_data)
        np.save("training_data.npy", self.training_data)
        print('Cats:',dogsvcats.catcount)
        print('Dogs:',dogsvcats.dogcount)

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50, 50).view(-1, 1, 50, 50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

def train(net):
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
            #print(f"{i}:{i+BATCH_SIZE}")
            batch_X = train_X[i:i+BATCH_SIZE].view(-1, 1, 50, 50)
            batch_y = train_y[i:i+BATCH_SIZE]

            net.zero_grad()

            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()    # Does the update

        print(f"Epoch: {epoch}. Loss: {loss}")

def test(net):
    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_X))):
            real_class = torch.argmax(test_y[i])
            net_out = net(test_X[i].view(-1, 1, 50, 50))[0]  # returns a list,
            predicted_class = torch.argmax(net_out)

            if predicted_class == real_class:
                correct += 1
            total += 1

    print("Accuracy: ", round(correct/total, 3))

if __name__ == "__main__":
    REBUILD_DATA = False # set to true to one once, then back to false unless you want to change something in your training data.
    net = Net()

    print(net)

    if REBUILD_DATA:
        dogsvcats = DogsVSCats()
        dogsvcats.make_training_data()

    training_data = np.load("training_data.npy", allow_pickle=True)
    print(len(training_data))

    optimizer = optim.Adam(net.parameters(), lr=0.001)
    loss_function = nn.MSELoss()

    X = torch.Tensor([i[0] for i in training_data]).view(-1,50,50)
    X = X/255.0
    y = torch.Tensor([i[1] for i in training_data])

    VAL_PCT = 0.1  # lets reserve 10% of our data for validation
    val_size = int(len(X)*VAL_PCT)

    train_X = X[:-val_size]
    train_y = y[:-val_size]

    test_X = X[-val_size:]
    test_y = y[-val_size:]

    BATCH_SIZE = 100
    EPOCHS = 1
    train(net)
    test(net)

    PATH = './object_detection.pth'
    torch.save(net.state_dict(), PATH)

You can check more information on the official docs: import and main.


Instruction: network values goes to 0 by linear layers

Input:
I designed the Graph Attention Network.
However, during the operations inside the layer, the values of features becoming equal.
class GraphAttentionLayer(nn.Module):
    ## in_features = out_features = 1024
    def __init__(self, in_features, out_features, dropout):
        super(GraphAttentionLayer, self).__init__()
        self.dropout = dropout
        self.in_features = in_features
        self.out_features = out_features

        self.W = nn.Parameter(torch.zeros(size=(in_features, out_features)))
        self.a1 = nn.Parameter(torch.zeros(size=(out_features, 1)))
        self.a2 = nn.Parameter(torch.zeros(size=(out_features, 1)))
        nn.init.xavier_normal_(self.W.data, gain=1.414)
        nn.init.xavier_normal_(self.a1.data, gain=1.414)
        nn.init.xavier_normal_(self.a2.data, gain=1.414)
        self.leakyrelu = nn.LeakyReLU()

    def forward(self, input, adj):
        h = torch.mm(input, self.W)
        a_input1 = torch.mm(h, self.a1)
        a_input2 = torch.mm(h, self.a2)
        a_input = torch.mm(a_input1, a_input2.transpose(1, 0))
        e = self.leakyrelu(a_input)

        zero_vec = torch.zeros_like(e)
        attention = torch.where(adj > 0, e, zero_vec) # most of values is close to 0
        attention = F.softmax(attention, dim=1) # all values are 0.0014 which is 1/707 (707^2 is the dimension of attention)
        attention = F.dropout(attention, self.dropout)
        return attention

The dimension of 'attention' is (707 x 707) and I observed the value of attention is near 0 before the softmax.
After the softmax, all values are 0.0014 which is 1/707.
I wonder how to keep the values normalized and prevent this situation.
Thanks

Output:
Since you say this happens during training I would assume it is at the start. With random initialization you often get near identical values at the end of the network during the start of the training process.
When all values are more or less equal the output of the softmax will be 1/num_elements for every element, so they sum up to 1 over the dimension you chose. So in your case you get 1/707 as all the values, which just sounds to me your weights are freshly initialized and the outputs are mostly random at this stage.
I would let it train for a while and observe if this changes.


Instruction: RuntimeError: Cost Function returns nan values in its 1th output

Input:
I am working on a temporal model to predict future events. Here is the link to my colab notebook. I am facing an issue trying to train the model. I am getting NaN values for Train and valid loss. The loss function is a joint loss consisting of the cross entropy loss and the squared loss.  Link to the blog here.
Tried following solution which didn`t worked -
Smaller learning rate - 0.01, 0.001, 0.0001
class cost_function():
    def __init__(self, yhat, y, L_2=0.001, logEps=1e-8):
        # logEps : log epsilon, very small positive value greater that 0.0
        # CE = - [ ln(y*)(y) + ln(1-y*)(1-y) ],

        self.yhat = yhat
        self.y = y

        self.logEps = logEps
        self.L_2 = L_2

        self.W_out = nn.Parameter(torch.randn(hiddenDimSize, numClass)*0.01)

    def cross_entropy(self):
        ce = -(self.y * torch.log(self.yhat + self.logEps) + (1. - self.y) * torch.log(1. - self.yhat + self.logEps))
        print("Inside CrossEntrophy Loss fn : ", ce)
        return ce

    def prediction_loss(self):
        # return  (torch.sum(torch.sum(self.cross_entropy(), dim=0),dim=1)).float()/  lengths.float()

        tmp_tensor = torch.sum(self.cross_entropy(), dim=0)
        print("Inside PredictionLoss fn : Sum Dim 0", tmp_tensor)
        print("Inside PredictionLoss fn : Sum Dim 1", torch.sum(tmp_tensor,dim=1))
        print("Inside PredictionLoss fn : Final Result ", (torch.sum(tmp_tensor,dim=1)).float()/  lengths.float())
        return (torch.sum(tmp_tensor,dim=1)).float()/  lengths.float()

    def cost(self):
        print("Inside Cost fn :", torch.mean(self.prediction_loss()) + self.L_2 * (self.W_out ** 2).sum())
        return torch.mean(self.prediction_loss()) + self.L_2 * (self.W_out ** 2).sum() # regularize

build_EHRNN class -  I have made modification in the forward method params to resolve the 'h' is not defined error.
torch.manual_seed(1)

class build_EHRNN(nn.Module):
    def __init__(self, inputDimSize=4894, hiddenDimSize=[200,200], batchSize=100, embSize=200, numClass=4894, dropout=0.5, logEps=1e-8):
        super(build_EHRNN, self).__init__()

        self.inputDimSize = inputDimSize
        self.hiddenDimSize = hiddenDimSize
        self.numClass = numClass
        self.embSize = embSize
        self.batchSize = batchSize
        self.dropout = nn.Dropout(p=0.5)
        self.logEps = logEps

        # Embedding inputs
        self.W_emb = nn.Parameter(torch.randn(self.inputDimSize, self.embSize).cuda())
        self.b_emb = nn.Parameter(torch.zeros(self.embSize).cuda())

        self.W_out = nn.Parameter(torch.randn(self.hiddenDimSize, self.numClass).cuda())
        self.b_out = nn.Parameter(torch.zeros(self.numClass).cuda())

        self.params = [self.W_emb, self.W_out,
                       self.b_emb, self.b_out]

    # def forward(self,x, y, h, lengths, mask):
    def forward(self,x, y, lengths, mask):
        self.emb = torch.tanh(torch.matmul(x, self.W_emb) + self.b_emb)
        input_values = self.emb
        self.outputs = [input_values]
        for i, hiddenSize in enumerate([self.hiddenDimSize, self.hiddenDimSize]):  # iterate over layers
            rnn = EHRNN(self.inputDimSize,hiddenSize,self.embSize,self.batchSize,self.numClass) # calculate hidden states
            hidden_state = []
            h = self.init_hidden().cuda()
            for i,seq in enumerate(input_values): # loop over sequences in each batch
                h = rnn(seq, h)
                hidden_state.append(h)
            hidden_state = self.dropout(torch.stack(hidden_state))    # apply dropout between layers
            input_values = hidden_state

        y_linear = torch.matmul(hidden_state, self.W_out)  + self.b_out # fully connected layer
        yhat = F.softmax(y_linear, dim=1)  # yhat
        yhat = yhat*mask[:,:,None]   # apply mask

        # Loss calculation
        cross_entropy = -(y * torch.log(yhat + self.logEps) + (1. - y) * torch.log(1. - yhat + self.logEps))
        last_step = -torch.mean(y[-1] * torch.log(yhat[-1] + self.logEps) + (1. - y[-1]) * torch.log(1. - yhat[-1] + self.logEps))
        prediction_loss = torch.sum(torch.sum(cross_entropy, dim=0),dim=1)/ torch.cuda.FloatTensor(lengths)
        cost = torch.mean(prediction_loss) + 0.000001 * (self.W_out ** 2).sum() # regularize
        return (yhat, hidden_state, cost)

    def init_hidden(self):
        return torch.zeros(self.batchSize, self.hiddenDimSize)  # initial state

model training
artificalData_seqs = np.array(pickle.load(open(os.path.join(GOOGLE_DRV_PATH,BASE_DIR,'data.encodedDxs'),'rb')))
train, test, valid = load_data(artificalData_seqs, artificalData_seqs)

batchSize = 50     # decreased from 100 to 50
n_batches = int(np.ceil(float(len(train[0])) / float(batchSize)))-1
n_batches_valid = int(np.ceil(float(len(valid[0])) / float(batchSize)))-1
model = build_EHRNN(inputDimSize=4894, hiddenDimSize=200, batchSize=50, embSize=200, numClass=4894, dropout=0.5, logEps=1e-8)
model = model.to(device)

 import torch.nn.functional as F
import pdb

optimizer = torch.optim.Adadelta(model.parameters(), lr = 0.001, rho=0.95)
epochs = 10

counter = 0
# with torch.autograd.detect_anomaly():
for e in range(epochs):
    for x, y in train_dl:
        x, y , mask, lengths = padding(x, y, inputDimSize, numClass)
        output, h = model(x, mask)

        loss = cost_function(output, y).cost()
        # pdb.set_trace()
        loss.backward()
        print("loss ",loss)
        nn.utils.clip_grad_norm_(model.parameters(), 5) # Constraining the weight matrix directly == regularization.
        optimizer.step()
        optimizer.zero_grad()

    with torch.no_grad():
            model.eval()
            val_loss = []
            for x_valid, y_valid in valid_dl:
                    x_val, y_val, mask, lengths = padding(x_valid, y_valid, inputDimSize, numClass)
                    outputs_val, hidden_val = model(x_val,  mask)
                    loss = cost_function(outputs_val, y_val).cost()
                    val_loss.append(loss.item())
            model.train()

            print("Epoch: {}/{}...".format(e+1, epochs),
                                "Step: {}...".format(counter),
                                "Training Loss: {:.4f}...".format(loss.item()),
                                "Val Loss: {:.4f}".format(torch.mean(torch.tensor(val_loss))))

Error (Start getting NaNs in the loss)
Inside PredictionLoss fn : Sum Dim 0 tensor([[0.1008, 0.1539, 0.1211,  ..., 0.1533, 0.1218, 0.1418],
        [0.0253, 0.0449, 0.0249,  ..., 0.0439, 0.0134, 0.0332],
        [0.0306, 0.0799, 0.0570,  ..., 0.0790, 0.0484, 0.0678],
        ...,
        [0.0253, 0.0450, 0.0249,  ..., 0.0439, 0.0134, 0.0332],
        [0.0253, 0.0450, 0.0249,  ..., 0.0439, 0.0134, 0.0332],
        [0.0038, 0.0106, 0.0106,  ..., 0.0098, 0.0004, 0.0106]],
       grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([  372.4754,   133.2620,   219.1195,    37.5425,   141.3354,    37.5070,
          229.2947,     0.0000,   379.1829,   217.3962,    80.1226,    37.5074,
          138.4665,    82.1034,    89.7893,    81.8173,    92.8159,   141.8856,
           95.9898,   216.0511,   133.2535,   385.0391,   369.4958,   244.9362,
           37.5088,    37.5087,   141.6083,     0.0000,    95.3367,    37.5074,
          735.0569,   378.0407,    37.5135,    40.7778,    82.0872,   225.9998,
          216.6189,   379.0732,    81.4742,   144.4226,    93.3905,   214.0228,
           37.5078,   224.0793,    88.3753,    41.2919,   140.4855,    37.5086,
          226.6366,   148.7171,   137.9226, 13887.5811,    81.1428,    84.6804,
          226.6779,    37.5065,   223.8841,   220.5979,    83.2484,    37.5080,
           84.5247,   384.2115,    80.1173,     0.0000,   146.9714,    37.6982,
          134.6618,    84.1838,    37.5421,   730.5516,    37.5085,     0.0000,
          215.1523,   136.5673,    81.2887,    37.5080,    94.4181,   140.6268,
          133.9295,   136.2485,    80.1173,   386.2103,    39.0282,     0.0000,
           37.5055,    42.1506,    80.1662,   228.5819,    39.3403,   138.7672,
         1768.6033,   143.5350,    40.2060,   147.7809,    37.5085,   380.9214,
          750.6883,   141.0447,   136.9028,    37.5049],
       grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([5., 3., 4., 1., 3., 1., 4., 0., 5., 4., 2., 1., 3., 2., 2., 2., 2., 3.,
        2., 4., 3., 5., 5., 4., 1., 1., 3., 0., 2., 1., 6., 5., 1., 1., 2., 4.,
        4., 5., 2., 3., 2., 4., 1., 4., 2., 1., 3., 1., 4., 3., 3., 9., 2., 2.,
        4., 1., 4., 4., 2., 1., 2., 5., 2., 0., 3., 1., 3., 2., 1., 6., 1., 0.,
        4., 3., 2., 1., 2., 3., 3., 3., 2., 5., 1., 0., 1., 1., 2., 4., 1., 3.,
        7., 3., 1., 3., 1., 5., 6., 3., 3., 1.])
Inside PredictionLoss fn : Final Result  tensor([  74.4951,   44.4207,   54.7799,   37.5425,   47.1118,   37.5070,
          57.3237,       nan,   75.8366,   54.3491,   40.0613,   37.5074,
          46.1555,   41.0517,   44.8946,   40.9086,   46.4080,   47.2952,
          47.9949,   54.0128,   44.4178,   77.0078,   73.8992,   61.2340,
          37.5088,   37.5087,   47.2028,       nan,   47.6683,   37.5074,
         122.5095,   75.6081,   37.5135,   40.7778,   41.0436,   56.5000,
          54.1547,   75.8146,   40.7371,   48.1409,   46.6952,   53.5057,
          37.5078,   56.0198,   44.1876,   41.2919,   46.8285,   37.5086,
          56.6591,   49.5724,   45.9742, 1543.0646,   40.5714,   42.3402,
          56.6695,   37.5065,   55.9710,   55.1495,   41.6242,   37.5080,
          42.2623,   76.8423,   40.0586,       nan,   48.9905,   37.6982,
          44.8873,   42.0919,   37.5421,  121.7586,   37.5085,       nan,
          53.7881,   45.5224,   40.6443,   37.5080,   47.2090,   46.8756,
          44.6432,   45.4162,   40.0587,   77.2421,   39.0282,       nan,
          37.5055,   42.1506,   40.0831,   57.1455,   39.3403,   46.2557,
         252.6576,   47.8450,   40.2060,   49.2603,   37.5085,   76.1843,
         125.1147,   47.0149,   45.6343,   37.5049], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([2., 1., 1., 2., 3., 5., 3., 3., 1., 4., 2., 2., 2., 3., 2., 2., 5., 2.,
        4., 2., 1., 4., 2., 5., 3., 0., 4., 7., 6., 4., 4., 1., 7., 1., 3., 3.,
        5., 3., 5., 5., 4., 2., 2., 4., 1., 5., 6., 2., 5., 5., 2., 1., 3., 1.,
        4., 4., 3., 3., 0., 2., 2., 4., 2., 2., 1., 0., 3., 2., 3., 0., 1., 2.,
        4., 4., 5., 1., 1., 3., 3., 2., 5., 0., 2., 6., 5., 5., 5., 1., 2., 4.,
        0., 2., 6., 1., 2., 0., 1., 1., 2., 3.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([3., 2., 4., 4., 6., 1., 1., 4., 2., 2., 4., 2., 1., 1., 1., 1., 3., 1.,
        2., 3., 4., 2., 2., 3., 2., 0., 1., 3., 4., 1., 2., 3., 4., 3., 3., 1.,
        2., 3., 3., 2., 1., 4., 6., 3., 4., 2., 3., 0., 1., 1., 3., 7., 2., 3.,
        4., 2., 3., 0., 3., 3., 2., 1., 1., 3., 6., 2., 2., 3., 2., 3., 3., 2.,
        1., 2., 5., 3., 4., 2., 5., 3., 5., 5., 5., 3., 2., 1., 4., 3., 3., 3.,
        1., 4., 3., 2., 5., 3., 6., 4., 3., 2.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([4., 6., 2., 2., 4., 5., 0., 3., 2., 3., 4., 3., 3., 3., 4., 3., 0., 1.,
        2., 2., 4., 2., 4., 4., 1., 0., 3., 0., 5., 5., 1., 2., 3., 2., 2., 3.,
        1., 2., 1., 2., 3., 1., 3., 1., 3., 0., 4., 2., 1., 6., 2., 2., 0., 1.,
        4., 4., 2., 2., 0., 3., 5., 5., 5., 5., 3., 4., 4., 4., 5., 4., 2., 2.,
        3., 1., 5., 6., 3., 4., 1., 3., 5., 1., 3., 2., 3., 6., 2., 4., 3., 2.,
        3., 2., 4., 1., 2., 2., 1., 2., 3., 4.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([1., 5., 1., 5., 3., 0., 1., 4., 3., 3., 5., 5., 2., 0., 1., 2., 3., 2.,
        2., 2., 2., 2., 2., 2., 2., 3., 4., 1., 4., 2., 1., 3., 4., 0., 4., 2.,
        0., 4., 1., 1., 2., 3., 3., 5., 1., 3., 2., 3., 1., 1., 3., 6., 3., 0.,
        2., 3., 2., 3., 3., 4., 1., 5., 2., 6., 2., 1., 5., 3., 2., 1., 3., 5.,
        2., 0., 3., 0., 1., 3., 3., 2., 3., 1., 2., 4., 1., 2., 7., 2., 7., 2.,
        2., 2., 1., 1., 1., 2., 4., 2., 2., 3.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([2., 2., 1., 4., 3., 2., 2., 4., 5., 2., 3., 1., 1., 2., 3., 1., 3., 1.,
        3., 1., 6., 4., 2., 2., 0., 4., 1., 0., 2., 0., 1., 1., 1., 1., 2., 2.,
        3., 1., 3., 2., 0., 1., 3., 2., 4., 1., 2., 1., 6., 2., 1., 2., 1., 2.,
        2., 1., 1., 1., 4., 1., 0., 2., 2., 2., 3., 6., 5., 1., 1., 3., 3., 3.,
        1., 4., 4., 2., 4., 2., 3., 7., 1., 1., 2., 3., 0., 0., 1., 4., 2., 3.,
        1., 2., 3., 2., 7., 0., 2., 3., 4., 4.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([1., 2., 6., 7., 2., 1., 1., 3., 1., 2., 2., 4., 3., 3., 4., 3., 2., 2.,
        5., 2., 1., 3., 2., 2., 2., 1., 4., 3., 2., 2., 1., 4., 2., 0., 2., 2.,
        1., 3., 4., 4., 5., 1., 5., 6., 1., 3., 1., 3., 0., 1., 6., 1., 2., 0.,
        3., 2., 2., 3., 4., 3., 3., 2., 3., 2., 2., 3., 4., 3., 2., 0., 3., 1.,
        1., 3., 1., 4., 7., 2., 2., 1., 1., 1., 2., 3., 3., 2., 2., 3., 1., 3.,
        3., 1., 1., 3., 1., 2., 7., 3., 1., 6.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([2., 2., 4., 2., 5., 1., 2., 4., 2., 2., 2., 5., 1., 2., 1., 4., 1., 2.,
        4., 3., 4., 0., 2., 5., 2., 6., 2., 6., 3., 4., 3., 5., 7., 4., 3., 5.,
        1., 3., 2., 3., 2., 2., 2., 3., 3., 3., 2., 3., 1., 4., 3., 3., 2., 1.,
        6., 9., 2., 1., 6., 3., 1., 5., 1., 6., 2., 2., 6., 2., 4., 2., 4., 3.,
        2., 2., 4., 1., 2., 2., 2., 1., 3., 1., 3., 2., 3., 3., 2., 2., 4., 3.,
        3., 3., 5., 4., 3., 2., 4., 2., 1., 8.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : Sum Dim 1 tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<SumBackward1>)
Inside PredictionLoss fn : lengths tensor([1., 3., 2., 3., 3., 2., 7., 4., 3., 1., 1., 5., 1., 3., 4., 4., 1., 5.,
        1., 3., 3., 1., 4., 3., 0., 4., 4., 2., 4., 1., 2., 4., 3., 2., 3., 2.,
        2., 2., 2., 3., 3., 2., 5., 1., 2., 1., 2., 2., 3., 2., 2., 2., 3., 3.,
        3., 5., 1., 4., 8., 4., 0., 2., 4., 2., 0., 1., 4., 4., 1., 5., 0., 1.,
        3., 1., 2., 3., 3., 4., 3., 4., 2., 4., 2., 3., 4., 1., 1., 2., 4., 1.,
        1., 0., 0., 0., 4., 3., 1., 3., 4., 3.])
Inside PredictionLoss fn : Final Result  tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan], grad_fn=<DivBackward0>)
Inside PredictionLoss fn : Sum Dim 0 tensor([[nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        ...,
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan],
        [nan, nan, nan,  ..., nan, nan, nan]], grad_fn=<SumBackward1>)

Output:
The problem is the values in lengths variable.
In your cost_function.prediction_loss you divide the cross entropy loss by the length of each sequece: (torch.sum(tmp_tensor,dim=1)).float()/  lengths.float().
However, if you look at the values of your lengths tensor:

Inside PredictionLoss fn : lengths tensor([5., 3., 4., 1., 3., 1., 4., 0., 5., 4., 2., 1., 3., 2., 2., 2., 2., 3.,
   2., 4., 3., 5., 5., 4., 1., 1., 3., 0., 2., 1., 6., 5., 1., 1., 2., 4.,
   4., 5., 2., 3., 2., 4., 1., 4., 2., 1., 3., 1., 4., 3., 3., 9., 2., 2.,
   4., 1., 4., 4., 2., 1., 2., 5., 2., 0., 3., 1., 3., 2., 1., 6., 1., 0.,
   4., 3., 2., 1., 2., 3., 3., 3., 2., 5., 1., 0., 1., 1., 2., 4., 1., 3.,
   7., 3., 1., 3., 1., 5., 6., 3., 3., 1.])

You will notice that some of the entries are 0(!). The corresponding values in the loss function are also zero (no loss for zero-length sequence). When you divide zero by zero you get nan.

Some good practices for coding:

If possible, use library functions instead of re-implementing things. These functions are usually tested and optimized and are more numerically stable.
For instance, you can use torch.nn.CrossEntropyLoss that combines both the cross entropy loss and the softmax in a numerically robust manner.

the variable lengths used for loss computation is not explicitly an argument of the loss function or a class member. You should make it an explicit argument.


Instruction: In PyTorch calc Euclidean distance instead of matrix multiplication

Input:
Let say we have 2 matrices:
mat = torch.randn([20, 7]) * 100
mat2 = torch.randn([7, 20]) * 100

n, m = mat.shape

The simplest usual matrix multiplication looks like this:
def mat_vec_dot_product(mat, vect):
    n, m = mat.shape

    res = torch.zeros([n])
    for i in range(n):
        for j in range(m):
            res[i] += mat[i][j] * vect[j]

    return res

res = torch.zeros([n, n])
for k in range(n):
    res[:, k] = mat_vec_dot_product(mat, mat2[:, k])

But what if I need to apply L2 norm instead of dot product? The code is next:
def mat_vec_l2_mult(mat, vect):
    n, m = mat.shape

    res = torch.zeros([n])
    for i in range(n):
        for j in range(m):
            res[i] += (mat[i][j] - vect[j]) ** 2

    res = res.sqrt()

    return res

for k in range(n):
    res[:, k] = mat_vec_l2_mult(mat, mat2[:, k])

Can we do this somehow in an optimal way using Torch or any other libraries? Cause naive O(n^3) Python code works really slow.

Output:
Use torch.cdist for L2 norm - euclidean distance
res = torch.cdist(mat, mat2.permute(1,0), p=2)

Here, I have used permute to swap dim of mat2 from 7,20 to 20,7


Instruction: Tried installing FastAi but I got "ERROR: No matching distribution found for torchvision>=0.7"

Input:
I tried installing Fast Ai using the command
pip install fastai

But then I got this error

ERROR: Could not find a version that satisfies the requirement torchvision>=0.7 (from fastai) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0, 0.4.1, 0.5.0)
ERROR: No matching distribution found for torchvision>=0.7 (from fastai)

What happened and how do I solve this? I tried installing some specific torch versions but none works.

Output:
The error just means that you are missing a required library of the given version PyTorch>=0.7.
As it states in the Instructions if you installing it by using pip make sure to install pytorch manually to the required version. You can see here on instructions for installing PyTorch.


Instruction: How do I compute bootstrapped cross entropy loss in PyTorch?

Input:
I have read some papers that use something called "Bootstrapped Cross Entropy Loss" to train their segmentation network. The idea is to focus only on the hardest k% (say 15%) of the pixels into account to improve learning performance, especially when easy pixels dominate.
Currently, I am using the standard cross entropy:
loss = F.binary_cross_entropy(mask, gt)

How do I convert this to the bootstrapped version efficiently in PyTorch?

Output:
Often we would also add a "warm-up" period to the loss such that the network can learn to adapt to the easy regions first and transit to the harder regions.
This implementation starts from k=100 and continues for 20000 iterations, then linearly decay it to k=15 for another 50000 iterations.
class BootstrappedCE(nn.Module):
    def __init__(self, start_warm=20000, end_warm=70000, top_p=0.15):
        super().__init__()

        self.start_warm = start_warm
        self.end_warm = end_warm
        self.top_p = top_p

    def forward(self, input, target, it):
        if it < self.start_warm:
            return F.cross_entropy(input, target), 1.0

        raw_loss = F.cross_entropy(input, target, reduction='none').view(-1)
        num_pixels = raw_loss.numel()

        if it > self.end_warm:
            this_p = self.top_p
        else:
            this_p = self.top_p + (1-self.top_p)*((self.end_warm-it)/(self.end_warm-self.start_warm))
        loss, _ = torch.topk(raw_loss, int(num_pixels * this_p), sorted=False)
        return loss.mean(), this_p


Instruction: pytorch : How to stack 2 tensors

Input:
I’m trying to stack 2 tensors A.shape=(64,16,16) and B.shape=(64,16,16) in a tensor of shape C.shape=(1,128,16,16)
and non of functions I’ve tried work where
torch.stack => C.shape=(2,64,16,16) and
torch.cat => C.shape=(128,16,16)
can enyones help me

Output:
Concat first and then use unsqueeze to add singleton dimension at 0th position
torch.cat([A, B]).unsqueeze(0)


Instruction: Getting variance values for random samples generated from a standard normal distribution using numpy

Input:
I have a function that gives me probability distributions for each class, in terms of a matrix corresponding to mean values and another matrix corresponding to variance values. For example, if I had four classes then I would have the following outputs:
y_means = [1,2,3,4]
y_variance = [0.01,0.02,0.03,0.04]

I need to do the following calculation to the mean values to continue with the rest of my program:
y_means = np.array(y_means)
y_means = np.reshape(y_means,(y_means.size,1))
A = np.random.randn(10,y_means.size)
y_means = np.matmul(A,y_means)

Here, I have used the numpy.random.randn function to generate random samples from a standard normal distribution, and then multiply this with the matrix with the mean value to obtain a new output matrix. The dimension of the output matrix would then be of the size (10 x 1).
I need to do a similar calculation such that my output_variances will also be a (10 x 1) matrix. But it is not meaningful to multiply the variances in the same way with random samples from a standard normal distribution, because this would result in negative values as well. This is undesirable because my ultimate aim would be to create a normal distribution with these mean values and their corresponding variance values using:
torch.distributions.normal.Normal(loc=y_means, scale=y_variance)

So my question is if there is any method by which I get a variance value for each random sample generated by numpy.random.randn? Because then the multplication of such a matrix would make more sense with output_variance.
Or if there is any other strategy for this that I might be unaware of, please let me know.

Output:
The problem mentioned in the question required another matrix of the same dimension as A that corresponded to a variance measure for the random samples present in A.
Taking a row-wise or column-wise variance of the matrix denoted by A using numpy.var() didn't give a similar 10 x 4 matrix to multiply with y_variance.
I had solved the above problem by using the following approach:
First create a matrix with the same dimensions as A with zero entries, using the following line of code:
A_var = np.zeros_like(A)

then, using torch.distributions, create normal distributions with the values in A as the mean and zeroes as variance:
dist_A = torch.distributions.normal.Normal(loc=torch.Tensor(A), scale=torch.Tensor(A_var))

https://pytorch.org/docs/stable/distributions.html lists all the operations possible on Normal distributions in PyTorch. The sample() method can generate samples from a given distribution for any size. This property was exploited to first generate a sample matrix of size 10 X 10 x 4 and then calculating the variance along axis 0.
np.var(np.array(dist2.sample((10,))),axis=0)

This would result in a variance matrix of size 10 x 4, which can be used for calculations with y_variance.


Instruction: How to Efficiently Find the Indices of Max Values in a Multidimensional Array of Matrices using Pytorch and/or Numpy

Input:
Background
It is common in machine learning to deal with data of a high dimensionality. For example, in a Convolutional Neural Network (CNN) the dimensions of each input image may be 256x256, and each image may have 3 color channels (Red, Green, and Blue). If we assume that the model takes in a batch of 16 images at a time, the dimensionality of the input going into our CNN is [16,3,256,256]. Each individual convolutional layer expects data in the form [batch_size, in_channels, in_y, in_x], and all of these quantities often change layer-to-layer (except batch_size). The term we use for the matrix made up of the [in_y, in_x] values is feature map, and this question is concerned with finding the maximum value, and its index, in every feature map at a given layer.
Why do I want to do this? I want to apply a mask to every feature map, and I want to apply that mask centered at the max value in each feature map, and to do that I need to know where each max value is located. This mask application is done during both training and testing of the model, so efficiency is vitally important to keep computational times down. There are many Pytorch and Numpy solutions for finding singleton max values and indices, and for finding the maximum values or indices along a single dimension, but no (that I could find) dedicated and efficient built-in functions for finding the indices of maximum values along 2 or more dimensions at a time. Yes, we can nest functions that operate on a single dimension, but these are some of the least efficient approaches.
What I've Tried

I've looked at this Stackoverflow question, but the author is dealing with a special-case 4D array which is trivially squeezed to a 3D array. The accepted answer is specialized for this case, and the answer pointing to TopK is misguided because it not only operates on a single dimension, but would necessitate that k=1 given the question asked, thus devlolving to a regular torch.max call.
I've looked at this Stackoverflow question, but this question, and its answer, focus on looking through a single dimension.
I have looked at this Stackoverflow question, but I already know of the answer's approach as I independently formulated it in my own answer here (where I amended that the approach is very inefficient).
I have looked at this Stackoverflow question, but it does not satisfy the key part of this question, which is concerned with efficiency.
I have read many other Stackoverflow questions and answers, as well as the Numpy documentation, Pytorch documentation, and posts on the Pytorch forums.
I've tried implementing a LOT of varying approaches to this problem, enough that I have created this question so that I can answer it and give back to the community, and anyone who goes looking for a solution to this problem in the future.

Standard of Performance
If I am asking a question about efficiency I need to detail expectations clearly. I am trying to find a time-efficient solution (space is secondary) for the problem above without writing C code/extensions, and which is reasonably flexible (hyper specialized approaches aren't what I'm after). The approach must accept an [a,b,c,d] Torch tensor of datatype float32 or float64 as input, and output an array or tensor of the form [a,b,2] of datatype int32 or int64 (because we are using the output as indices).
Solutions should be benchmarked against the following typical solution:
max_indices = torch.stack([torch.stack([(x[k][j]==torch.max(x[k][j])).nonzero()[0] for j in range(x.size()[1])]) for k in range(x.size()[0])])

Output:
The Approach
We are going to take advantage of the Numpy community and libraries, as well as the fact that Pytorch tensors and Numpy arrays can be converted to/from one another without copying or moving the underlying arrays in memory (so conversions are low cost). From the Pytorch documentation:

Converting a torch Tensor to a Numpy array and vice versa is a breeze. The torch Tensor and Numpy array will share their underlying memory locations, and changing one will change the other.

Solution One
We are first going to use the Numba library to write a function that will be just-in-time (JIT) compiled upon its first usage, meaning we can get C speeds without having to write C code ourselves. Of course, there are caveats to what can get JIT-ed, and one of those caveats is that we work with Numpy functions. But this isn't too bad because, remember, converting from our torch tensor to Numpy is low cost. The function we create is:
@njit(cache=True)
def indexFunc(array, item):
    for idx, val in np.ndenumerate(array):
        if val == item:
            return idx

This function if from another Stackoverflow answer located here (This was the answer which introduced me to Numba). The function takes an N-Dimensional Numpy array and looks for the first occurrence of a given item. It immediately returns the index of the found item on a successful match. The @njit decorator is short for @jit(nopython=True), and tells the compiler that we want it to compile the function using no Python objects, and to throw an error if it is not able to do so (Numba is the fastest when no Python objects are used, and speed is what we are after).
With this speedy function backing us, we can get the indices of the max values in a tensor as follows:
import numpy as np

x =  x.numpy()
maxVals = np.amax(x, axis=(2,3))
max_indices = np.zeros((n,p,2),dtype=np.int64)
for index in np.ndindex(x.shape[0],x.shape[1]):
    max_indices[index] = np.asarray(indexFunc(x[index], maxVals[index]),dtype=np.int64)
max_indices = torch.from_numpy(max_indices)

We use np.amax because it can accept a tuple for its axis argument, allowing it to return the max values of each 2D feature map in the 4D input. We initialize max_indices with np.zeros ahead of time because appending to numpy arrays is expensive, so we allocate the space we need ahead of time. This approach is much faster than the Typical Solution in the question (by an order of magnitude), but it also uses a for loop outside the JIT-ed function, so we can improve...
Solution Two
We will use the following solution:
@njit(cache=True)
def indexFunc(array, item):
    for idx, val in np.ndenumerate(array):
        if val == item:
            return idx
    raise RuntimeError

@njit(cache=True, parallel=True)
def indexFunc2(x,maxVals):
    max_indices = np.zeros((x.shape[0],x.shape[1],2),dtype=np.int64)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            max_indices[i,j] = np.asarray(indexFunc(x[i,j], maxVals[i,j]),dtype=np.int64)
    return max_indices

x = x.numpy()
maxVals = np.amax(x, axis=(2,3))
max_indices = torch.from_numpy(indexFunc2(x,maxVals))

Instead of iterating through our feature maps one-at-a-time with a for loop, we can take advantage of parallelization using Numba's prange function (which behaves exactly like range but tells the compiler we want the loop to be parallelized) and the parallel=True decorator argument. Numba also parallelizes the np.zeros function. Because our function is compiled Just-In-Time and uses no Python objects, Numba can take advantage of all the threads available in our system! It is worth noting that there is now a raise RuntimeError in the indexFunc. We need to include this, otherwise the Numba compiler will try to infer the return type of the function and infer that it will either be an array or None. This doesn't jive with our usage in indexFunc2, so the compiler would throw an error. Of course, from our setup we know that indexFunc will always return an array, so we can simply raise and error in the other logical branch.
This approach is functionally identical to Solution One, but changes the iteration using nd.index into two for loops using prange. This approach is about 4x faster than Solution One.
Solution Three
Solution Two is fast, but it is still finding the max values using regular Python. Can we speed this up using a more comprehensive JIT-ed function?
@njit(cache=True)
def indexFunc(array, item):
    for idx, val in np.ndenumerate(array):
        if val == item:
            return idx
    raise RuntimeError

@njit(cache=True, parallel=True)
def indexFunc3(x):
    maxVals = np.zeros((x.shape[0],x.shape[1]),dtype=np.float32)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            maxVals[i][j] = np.max(x[i][j])
    max_indices = np.zeros((x.shape[0],x.shape[1],2),dtype=np.int64)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            x[i][j] == np.max(x[i][j])
            max_indices[i,j] = np.asarray(indexFunc(x[i,j], maxVals[i,j]),dtype=np.int64)
    return max_indices

max_indices = torch.from_numpy(indexFunc3(x))

It might look like there is a lot more going on in this solution, but the only change is that instead of calculating the maximum values of each feature map using np.amax, we have now parallelized the operation. This approach is marginally faster than Solution Two.
Solution Four
This solution is the best I've been able to come up with:
@njit(cache=True, parallel=True)
def indexFunc4(x):
    max_indices = np.zeros((x.shape[0],x.shape[1],2),dtype=np.int64)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            maxTemp = np.argmax(x[i][j])
            max_indices[i][j] = [maxTemp // x.shape[2], maxTemp % x.shape[2]]
    return max_indices

max_indices = torch.from_numpy(indexFunc4(x))

This approach is more condensed and also the fastest at 33% faster than Solution Three and 50x faster than the Typical Solution. We use np.argmax to get the index of the max value of each feature map, but np.argmax only returns the index as if each feature map were flattened. That is, we get a single integer telling us which number the element is in our feature map, not the indices we need to be able to access that element. The math [maxTemp // x.shape[2], maxTemp % x.shape[2]] is to turn that singular int into the [row,column] that we need.
Benchmarking
All approaches were benchmarked together against a random input of shape [32,d,64,64], where d was incremented from 5 to 245. For each d, 15 samples were gathered and the times were averaged. An equality test ensured that all solutions provided identical values. An example of the benchmark output is:

A plot of the benchmarking times as d increased is (leaving out the Typical Solution so the graph isn't squashed):

Woah! What is going on at the start with those spikes?
Solution Five
Numba allows us to produce Just-In-Time compiled functions, but it doesn't compile them until the first time we use them; It then caches the result for when we call the function again. This means the very first time we call our JIT-ed functions we get a spike in compute time as the function is compiled. Luckily, there is a way around this- if we specify ahead of time what our function's return type and argument types will be, the function will be eagerly compiled instead of compiled just-in-time. Applying this knowledge to Solution Four we get:
@njit('i8[:,:,:](f4[:,:,:,:])',cache=True, parallel=True)
def indexFunc4(x):
    max_indices = np.zeros((x.shape[0],x.shape[1],2),dtype=np.int64)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            maxTemp = np.argmax(x[i][j])
            max_indices[i][j] = [maxTemp // x.shape[2], maxTemp % x.shape[2]]
    return max_indices

max_indices6 = torch.from_numpy(indexFunc4(x))

And if we restart our kernel and rerun our benchmark, we can look at the first result where d==5 and the second result where d==10 and note that all of the JIT-ed solutions were slower when d==5 because they had to be compiled, except for Solution Four, because we explicitly provided the function signature ahead of time:

There we go! That's the best solution I have so far for this problem.

EDIT #1
Solution Six
An improved solution has been developed which is 33% faster than the previously posted best solution. This solution only works if the input array is C-contiguous, but this isn't a big restriction since numpy arrays or torch tensors will be contiguous unless they are reshaped, and both have functions to make the array/tensor contiguous if needed.
This solution is the same as the previous best, but the function decorator which specifies the input and return types are changed from
@njit('i8[:,:,:](f4[:,:,:,:])',cache=True, parallel=True)
to
@njit('i8[:,:,::1](f4[:,:,:,::1])',cache=True, parallel=True)
The only difference is that the last : in each array typing becomes ::1, which signals to the numba njit compiler that the input arrays are C-contiguous, allowing it to better optimize.
The full solution six is then:
@njit('i8[:,:,::1](f4[:,:,:,::1])',cache=True, parallel=True)
def indexFunc5(x):
    max_indices = np.zeros((x.shape[0],x.shape[1],2),dtype=np.int64)
    for i in prange(x.shape[0]):
        for j in prange(x.shape[1]):
            maxTemp = np.argmax(x[i][j])
            max_indices[i][j] = [maxTemp // x.shape[2], maxTemp % x.shape[2]]
    return max_indices

max_indices7 = torch.from_numpy(indexFunc5(x))

The benchmark including this new solution confirms the speedup:


Instruction: Convolution - Deconvolution for even and odd size

Input:
I have two different size tensors to put in the network.
C = nn.Conv1d(1, 1, kernel_size=1, stride=2)
TC = nn.ConvTranspose1d(1, 1, kernel_size=1, stride=2)

a = torch.rand(1, 1, 100)
b = torch.rand(1, 1, 101)

a_out, b_out = TC(C(a)), TC(C(b))

The results are
a_out = torch.size([1, 1, 99]) # What I want is [1, 1, 100]
b_out = torch.size([1, 1, 101])

Is there any method to handle this problem?
I need your help.
Thanks

Output:
It is expected behaviour as per documentation. May be padding can be used when even input length is detected to get same length as input.
Something like this
class PadEven(nn.Module):
    def __init__(self, conv, deconv, pad_value=0, padding=(0, 1)):
        super().__init__()
        self.conv = conv
        self.deconv = deconv
        self.pad = nn.ConstantPad1d(padding=padding, value=pad_value)

    def forward(self, x):
        nd = x.size(-1)
        x = self.deconv(self.conv(x))
        if nd % 2 == 0:
            x = self.pad(x)
        return x

C = nn.Conv1d(1, 1, kernel_size=1, stride=2)
TC = nn.ConvTranspose1d(1, 1, kernel_size=1, stride=2)
P = PadEven(C, TC)

a = torch.rand(1, 1, 100)
b = torch.rand(1, 1, 101)

a_out, b_out = P(a), P(b)


Instruction: Using GPU inside docker container - CUDA Version: N/A and torch.cuda.is_available returns False

Input:
I'm trying to use GPU from inside my docker container. I'm using docker with version 19.03 on Ubuntu 18.04.
Outside the docker container if I run nvidia-smi I get the below output.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05    Driver Version: 450.51.05    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   30C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

If I run the samething inside the container created from nvidia/cuda docker image, I get the same output as above and everything is running smoothly. torch.cuda.is_available() returns True.
But If I run the same nvidia-smi command inside any other docker container, it gives the following output where you can see that the CUDA Version is coming as N/A. Inside the containers torch.cuda.is_available() also returns False.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.05    Driver Version: 450.51.05    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            On   | 00000000:00:1E.0 Off |                    0 |
| N/A   30C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I have installed nvidia-container-toolkit using the following commands.
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install nvidia-container-toolkit
sudo systemctl restart docker

I started my containers using the following commands
sudo docker run --rm --gpus all nvidia/cuda nvidia-smi
sudo docker run -it --rm --gpus all ubuntu nvidia-smi

Output:
docker run --rm --gpus all nvidia/cuda nvidia-smi should NOT return CUDA Version: N/A if everything (aka nvidia driver, CUDA toolkit, and nvidia-container-toolkit) is installed correctly on the host machine.
Given that docker run --rm --gpus all nvidia/cuda nvidia-smi returns correctly. I also had problem with CUDA Version: N/A inside of the container, which I had luck in solving:
Please see my answer https://stackoverflow.com/a/64422438/2202107 (obviously you need to adjust and install the matching/correct versions of everything)


Instruction: How can I apply a transformation to a torch tensor

Input:
I have a Torch Tensor z and I would like to apply a transformation matrix mat to z and have the output be exactly the same size as z. Here is the code I am running:
def trans(z):
    print(z)
    mat = transforms.Compose([transforms.ToPILImage(),transforms.RandomRotation(90),transforms.ToTensor()])
    z = Variable(mat(z.cpu()).cuda())
    z = nnf.interpolate(z, size=(28, 28), mode='linear', align_corners=False)
    return z
z = trans(z)

However, I get this error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-e2fc36889ba5> in <module>()
      3 inputs,targs=next(iter(tst_loader))
      4 recon, mean, var = vae.predict(model, inputs[img_idx])
----> 5 out = vae.generate(model, mean, var)

4 frames
/content/vae.py in generate(model, mean, var)
     90     z = trans(z)
     91     z = Variable(z.cpu().cuda())
---> 92     out = model.decode(z)
     93     return out.data.cpu()
     94

/content/vae.py in decode(self, z)
     56
     57     def decode(self, z):
---> 58         out = self.z_develop(z)
     59         out = out.view(z.size(0), 64, self.z_dim, self.z_dim)
     60         out = self.decoder(out)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     89
     90     def forward(self, input: Tensor) -> Tensor:
---> 91         return F.linear(input, self.weight, self.bias)
     92
     93     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1674         ret = torch.addmm(bias, input, weight.t())
   1675     else:
-> 1676         output = input.matmul(weight.t())
   1677         if bias is not None:
   1678             output += bias

RuntimeError: mat1 dim 1 must match mat2 dim 0

How can I successfully apply this rotation transform mat and not get any errors doing so?
Thanks,
Vinny

Output:
The problem is that interpolate expects a batch dimension, and looks like your data does not have one, based on the error message and the successful application of transforms. Since your input is spatial (based on the size=(28, 28)), you can fix that by adding the batch dimension and changing the mode, since linear is not implemented for spatial input:
z = nnf.interpolate(z.unsqueeze(0), size=(28, 28), mode='bilinear', align_corners=False)

If you want z to still have a shape like (C, H, W), then:
z = nnf.interpolate(z.unsqueeze(0), size=(28, 28), mode='bilinear', align_corners=False).squeeze(0)


Instruction: Alternatively train multi task learning model in pytorch - weight updating

Input:
I want to build a multi task learning model on two related datasets with different inputs and targets. The two tasks are sharing lower-level layers but with different header layers, a minimal example:
class MultiMLP(nn.Module):
    """
    A simple dense network for MTL on hard parameter sharing.
    """
    def __init__(self):
        super().__init__()
        self.hidden = nn.Linear(100, 200)
        self.out_task0= nn.Linear(200, 1)
        self.out_task0= nn.Linear(200, 1)

    def forward(self, x):
        x = self.hidden(x)
        x = F.relu(x)
        y_task0 = self.out_task0(x)
        y_task1 = self.out_task1(x)
        return [y_task0, y_task1]

The dataloader is constructed so that the batches are alternatively generated from two datasets, i.e. batch 0, 2, 4, ... from task 0, batch 1, 3, 5, ... from task 1.  I wanted to train the network in this way: only update weights for hidden layer and out_task0 for batches from task 0, and update only hidden and out_task1 for task 1.
I then alternatively switch requires_grad for the corresponding tasks during training as following. But I observed that all weights are updated for every iteration.
...
criterion = MSELoss()
for i, data in enumerate(combined_loader):
    x, y = data[0], data[1]
    optimizer.zero_grad()
    # controller is 0 for task0, 1 for task1
    # altenate the header layer
    controller = i % 2
    task0_mode = True if controller == 0 else False
    for name, param in model.named_parameters():
        if name in ['out_task0.weight', 'out_task0.bias']:
            param.requires_grad = task0_mode
        elif name in ['out_task1.weight', 'out_task1.bias']:
            param.requires_grad = not task0_mode

    outputs = model(x)[controller]
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()

    # Monitor the parameter updates
    for name, p in model.named_parameters():
        if name in ['out_task0.weight', 'out_task1.weight']:
            print(f"Controller: {controller}")
            print(name, p)

Did I miss anything in the training procedure? Or the overall setup will not work?

Output:
Disclaimer: the question has been answered from PyTorch Forum, I put things together here in case someone runs into the same problem, the credit goes to ptrblk
The problem could arise from any variants of stochastic gradient descent(sgd) which utilizes gradients from previous steps, for instance, stochastic gradient descent with momentum(sgd-m), Nesterov accelerated gradient (NAG), Adagrad, RMSprop, Adam and so on. Zero-ing gradient at step t would not affect the terms relying on historical gradients. Thus the weights are still updated with the setting in the posted question.
One can see that from the following code example.
model = nn.Linear(1, 1, bias=False)

#optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.) # same results for w1 and w2
optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.5) # w2 gets updated
#optimizer = torch.optim.Adam(model.parameters(), lr=1.) # w2 gets updated

w0 = model.weight.clone()

out = model(torch.randn(1, 1))
out.mean().backward()
optimizer.step()
w1 = model.weight.clone()

optimizer.zero_grad()
print(model.weight.grad)
optimizer.step()
w2 = model.weight.clone()

print(w1 - w0)
print(w2 - w1)

With the native SGD optimizer, w2 and w1 are the same. But it is not the case for SGD-M and Adam.


Instruction: Parameters of my network on PyTorch are not updated

Input:
I want to make an auto calibration system using PyTorch.
I try to deal with a homogeneous transform matrix as weights of neural networks.
I write a code referring to PyTorch tutorials, but my custom parameters are not updated after backward method is called.
When I print a 'grad' attribute of each parameter, it is a None.
My code is below. Is there anything wrong?
Please give any advise to me. Thank you.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

import numpy as np

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.params = nn.Parameter(torch.rand(6))
        self.rx, self.ry, self.rz = self.params[0], self.params[1], self.params[2]
        self.tx, self.ty, self.tz = self.params[3], self.params[4], self.params[5]

    def forward(self, x):
        tr_mat = torch.tensor([[1, 0, 0, self.params[3]],
                                [0, 1, 0, self.params[4]],
                                [0, 0, 1, self.params[5]],
                                [0, 0, 0, 1]], requires_grad=True)

        rz_mat = torch.tensor([[torch.cos(self.params[2]), -torch.sin(self.params[2]), 0, 0],
                                [torch.sin(self.params[2]), torch.cos(self.params[2]), 0, 0],
                                [0, 0, 1, 0],
                                [0, 0, 0, 1]], requires_grad=True)

        ry_mat = torch.tensor([[torch.cos(self.params[1]), 0, torch.sin(self.params[1]), 0],
                                [0, 1, 0, 0],
                                [-torch.sin(self.params[1]), 0, torch.cos(self.params[1]), 0],
                                [0, 0, 0, 1]], requires_grad=True)

        rx_mat = torch.tensor([[1, 0, 0, 0],
                                [0, torch.cos(self.params[0]), -torch.sin(self.params[0]), 0],
                                [0, torch.sin(self.params[0]), torch.cos(self.params[0]), 0],
                                [0, 0, 0, 1]], requires_grad=True)

        tf1 = torch.matmul(tr_mat, rz_mat)
        tf2 = torch.matmul(tf1, ry_mat)
        tf3 = torch.matmul(tf2, rx_mat)

        tr_local = torch.tensor([[1, 0, 0, x[0]],
                                [0, 1, 0, x[1]],
                                [0, 0, 1, x[2]],
                                [0, 0, 0, 1]])
        tf_output = torch.matmul(tf3, tr_local)
        output = tf_output[:3, 3]
        return output

    def get_loss(self, output):
        pass

model = Net()

input_ex = np.array([[-0.01, 0.05, 0.92],
                    [-0.06, 0.03, 0.94]])

output_ex = np.array([[-0.3, 0.4, 0.09],
                        [-0.5, 0.2, 0.07]])
print(list(model.parameters()))

optimizer = optim.Adam(model.parameters(), 0.001)
criterion = nn.MSELoss()

for input_np, label_np in zip(input_ex, output_ex):
    input_tensor = torch.from_numpy(input_np).float()
    label_tensor = torch.from_numpy(label_np).float()
    output = model(input_tensor)

    optimizer.zero_grad()
    loss = criterion(output, label_tensor)
    loss.backward()
    optimizer.step()

print(list(model.parameters()))

Output:
What happens
Your problem is related to PyTorch's implicit conversion of torch.tensor to float. Let's say you have this:
tr_mat = torch.tensor(
    [
        [1, 0, 0, self.params[3]],
        [0, 1, 0, self.params[4]],
        [0, 0, 1, self.params[5]],
        [0, 0, 0, 1],
    ],
    requires_grad=True,
)

torch.tensor can only be constructed from list which has Python like values, it cannot have torch.tensor inside it. What happens under the hood (let's say) is each element of self.params which can be converted to float is (in this case all of them can, e.g. self.params[3], self.params[4], self.params[5]).
When tensor's value is casted to float it's value is copied into Python counterpart hence it is not part of computational graph anymore, it's a new pure Python variable (which cannot be backpropagated obviously).
Solution
What you can do is choose elements of your self.params and insert them into eye matrices so the gradient flows. You can see a rewrite of your forward method taking this into account:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.params = nn.Parameter(torch.randn(6))

    def forward(self, x):
        sinus = torch.cos(self.params)
        cosinus = torch.cos(self.params)

        tr_mat = torch.eye(4)
        tr_mat[:-1, -1] = self.params[3:]

        rz_mat = torch.eye(4)
        rz_mat[0, 0] = cosinus[2]
        rz_mat[0, 1] = -sinus[2]
        rz_mat[1, 0] = sinus[2]
        rz_mat[1, 1] = cosinus[2]

        ry_mat = torch.eye(4)
        ry_mat[0, 0] = cosinus[1]
        ry_mat[0, 2] = sinus[1]
        ry_mat[2, 0] = -sinus[1]
        ry_mat[2, 2] = cosinus[1]

        rx_mat = torch.eye(4)
        rx_mat[1, 1] = cosinus[0]
        rx_mat[1, 2] = -sinus[0]
        rx_mat[2, 1] = sinus[0]
        rx_mat[2, 2] = cosinus[0]

        tf1 = torch.matmul(tr_mat, rz_mat)
        tf2 = torch.matmul(tf1, ry_mat)
        tf3 = torch.matmul(tf2, rx_mat)

        tr_local = torch.tensor(
            [[1, 0, 0, x[0]], [0, 1, 0, x[1]], [0, 0, 1, x[2]], [0, 0, 0, 1]],
        )
        tf_output = torch.matmul(tf3, tr_local)
        output = tf_output[:3, 3]
        return output

(you may want to double check this rewrite but the idea holds).
Also notice tr_local can be done "your way" as we don't need any values to keep gradient.
requires_grad
You can see requires_grad wasn't used anywhere in the code. It's because what requires gradient is not the whole eye matrix (we will not optimize 0 and 1), but parameters which are inserted into it. Usually you don't need requires_grad at all in your neural network code because:

input tensors are not optimized (usually, those could be when you are doing adversarial attacks or such)
nn.Parameter requires gradient by default (unless frozen)
layers and other neural network specific stuff requires gradient by default (unless frozen)
values which don't need gradient (input tensors) going through layers which do require it (or parameters or w/e) can be backpropagated


Instruction: Compare the efficiency of the data loading methods in deep learning

Input:
I need to load a time-series dataset to train a network. The dataset was split into many chunks train_x_0.npy, train_x_1.npy, ..., train_x_40.npy (41 chunks) because of memory issue when I extract these .npy files from the raw data. However, their sizes are too large (around 1000 GB) that I couldn't load everything into the RAM. I have been considering two ways to solve this problem.

Loading the data chunks using np.load() with argument mmap_mode='r+'. The memory-mapped chunks are stored in a Python list self.data. In the __getitem__(self, idx) method of Pytorch Dataset class, I convert idx to chunk_idx and sample_idx, then get the sample by self.data[chunk_idx][sample_idx].
Extract .npy files again from raw data, and save the data sample-by-sample, i.e. one .npy file is now one sample, not a data chunk. In the __getitem__(self, idx) method, I will get one sample by loading it using np.load(sample_path).

Assuming the Pytorch DataLoader will be used to iterate through all samples, then which method will be faster?
If you have another suggestion to extract the raw data or to load the .npy files, please share your opinion.

Output:
Both suggested approaches will be limited by your filesystem's IO, since each sample will be loaded from disk on-demand (memory mapping does not speed up the actual loading, once a given patch is requested).
Especially when you are planning to train for many epochs, you can achieve a strong speedup by loading your original chunks train_x_0.npy, train_x_1.npy, etc. one (or as many as you can hold in RAM) at a time and training multiple epochs on this chunk before switching to the next.
For this, you would need control over the sample indices requested by the dataloader. For that you could define a sampler which is passed the sample indices available in the respective cached data chunk. In pseudocode, your training loop could look something like this when caching one chunk at a time:
from yourproject import Dataset
from torch.utils.data import DataLoader
from torch.utils.data.sampler import SubsetRandomSampler

dataset = Dataset(train_data_path, ...)
for chunk_idx in range(num_chunks):
  dataset.cache_chunk(chunk_idx)
  chunk_sample_inds = dataset.get_chunk_sample_inds(chunk_idx)
  chunk_sampler = SubsetRandomSampler(chunk_sample_inds)
  chunk_loader = DataLoader(dataset=dataset, sampler=chunk_sampler)
  for chunk_epoch in range(num_chunk_epoch):
    for sample_idx, sample in enumerate(chunk_loader):
       output = model(sample)

Hereby, your Dataset class needs to take care of

caching (loading to RAM) a specified chunk, given a chunk idx (indicated by the cache_chunk method)
returning a list of valid sample indices for a given chunk idx (indicated by the get_chunk_sample_inds method)

If you use a fast GPU (which is often limited by shuffling data back and forth between RAM and VRAM, even for RAM-cached data), you can expect several orders of magnitude speed up using this approach (as opposed to attempting to fill the VRAM from HDD for each sample).


Instruction: Confusion about model.train()

Input: I am a beginner in pytorch. I saw on github that some deep learning models have model.train(), and some don’t, but they can run normally. I want to know if model.train() is necessary? what's the effect?

Output:
train mode or eval mode only matters when you have modules that behave asymmetrically (e.g. BatchNorm, Dropout) in training/testing. I would like to emphasize that it does not affect gradient accumulation at all. Even with asymmetrical modules, one can perfectly train a model in eval mode. Some do this in order to save memory in training using a pretrained ImageNet model.
If you don't have any asymmetrical modules, it does not matter at all.
By default, all modules start with training=True.


Instruction: How exactly does torch / np einsum work internally

Input:
This is a query regarding the internal working of torch.einsum in the GPU. I know how to use einsum. Does it perform all possible matrix multiplications, and just pick out the relevant ones, or does it perform only the required computation?
For example, consider two tensors a and b, of shape (N,P), and I wish to find the dot product of each corresponding tensor ni, of shape (1,P).
Using einsum, the code is:
torch.einsum('ij,ij->i',a,b)

Without using einsum, another way to obtain the output is :
torch.diag(a @ b.t())

Now, the second code is supposed to perform significantly more computations than the first one (eg if N = 2000, it performs 2000 times more computation). However, when I try to time the two operations, they take roughly the same amount of time to complete, which begs the question. Does einsum perform all combinations (like the second code), and picks out the relevant values?
Sample Code to test:
import time
import torch
for i in range(100):
  a = torch.rand(50000, 256).cuda()
  b = torch.rand(50000, 256).cuda()

  t1 = time.time()
  val = torch.diag(a @ b.t())
  t2 = time.time()
  val2 = torch.einsum('ij,ij->i',a,b)
  t3 = time.time()
  print(t2-t1,t3-t2, torch.allclose(val,val2))

Output:
It probably has to do with the fact that the GPU can parallelize the computation of a @ b.t(). This means that the GPU doesn't actually have to wait for each row-column multiplication computation to finish to compute then next multiplication.
If you check on CPU then you see that torch.diag(a @ b.t()) is significantly slower than torch.einsum('ij,ij->i',a,b)  for large a and b.


Instruction: Adding xavier initiliazation in pytorch

Input:
I want to add Xavier initialization to the first layer of my Neural Network, but I am getting an error in this class:
class DemoNN(nn.Module):
    def __init__(self):
        super().__init__()
        torch.manual_seed(0)
        self.net = nn.Sequential(
            nn.Linear(2,2),
            torch.nn.init.xavier_uniform((nn.Linear(2,2)).weights),
            nn.Sigmoid(),

            nn.Linear(2,2),
            nn.Sigmoid(),

            nn.Linear(2,4),
            nn.Softmax()
            )

    def forward(self, X):
        self.net(X)

Output:
You seem to try and initialize the second linear layer within the constructor of an nn.Sequential object.
What you need to do is to first construct self.net and only then initialize the second linear layer as you wish.
Here is how you should do it:
import torch
import torch.nn as nn
class DemoNN(nn.Module):
    def __init__(self):
        super().__init__()
        torch.manual_seed(0)
        self.net = nn.Sequential(
            nn.Linear(2,2),
            nn.Linear(2,2),
            nn.Sigmoid(),

            nn.Linear(2,2),
            nn.Sigmoid(),

            nn.Linear(2,4),
            nn.Softmax()
        )
        torch.nn.init.xavier_uniform_(self.net[1].weight)

    def forward(self, X):
        self.net(X)


Instruction: AssertionError: Torch not compiled with CUDA enabled (problem in torch vision)

Input:
so I am trying to run my object detection program and I keep getting the following error message:
AssertionError: Torch not compiled with CUDA enabled.
I don't understand why this happens, as I have a 2017 MacBook Pro with an AMD GPU, so I have no CUDA enabled GPU.
I added this statement in my code to make sure the device is set to 'cpu', however, it looks as if the program keeps trying to run it through a GPU even though it does not exist.
if torch.cuda.is_available():
    device = torch.device('cuda')

else:
    device = torch.device('cpu')

This is the place where the error happens (4th line):
    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        print("Hey")
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        print("Hey")
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

It would be really great, if anyone could help me with this issue!
Thanks everyone in advance!
PS: I already tried updating the Pytorch version, but still same problem.
Error output:

import os
import pandas as pd
import torch
import torch.utils.data
import torchvision
from PIL import Image
import utils
from engine import train_one_epoch, evaluate
import transforms as T
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

def parse_one_annot(path_to_data_file, filename):
    data = pd.read_csv(path_to_data_file)
    boxes_array = data[data["filename"] == filename][["xmin", "ymin", "xmax", "ymax"]].values

    return boxes_array

class RaccoonDataset(torch.utils.data.Dataset):

    def __init__(self, root, data_file, transforms=None):
        self.root = root
        self.transforms = transforms
        self.imgs = sorted(os.listdir(os.path.join(root, "images")))
        self.path_to_data_file = data_file

    def __getitem__(self, idx):
        # load images and bounding boxes
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        img = Image.open(img_path).convert("RGB")
        box_list = parse_one_annot(self.path_to_data_file,
                                   self.imgs[idx])
        boxes = torch.as_tensor(box_list, dtype=torch.float32)

        num_objs = len(box_list)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)
            return img, target

    def __len__(self):
        return len(self.imgs)

dataset = RaccoonDataset(root="./raccoon_dataset", data_file="./raccoon_dataset/data/raccoon_labels.csv")

dataset.__getitem__(0)

def get_model(num_classes):
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new on
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    return model

def get_transform(train):
    transforms = []
    # converts the image, a PIL image, into a PyTorch Tensor
    transforms.append(T.ToTensor())

    if train:
        # during training, randomly flip the training images
        # and ground-truth for data augmentation
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def main():

    dataset = RaccoonDataset(root="./raccoon_dataset",
                             data_file="raccoon_dataset/data/raccoon_labels.csv",
                             transforms=get_transform(train=True))

    dataset_test = RaccoonDataset(root="./raccoon_dataset",
                                  data_file="raccoon_dataset/data/raccoon_labels.csv",
                                  transforms=get_transform(train=False))

    torch.manual_seed(1)
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-40])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-40:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4,
                                              collate_fn=utils.collate_fn)
    data_loader_test = torch.utils.data.DataLoader(dataset_test, batch_size=1, shuffle=False, num_workers=4,
                                                   collate_fn=utils.collate_fn)

    print("We have: {} examples, {} are training and {} testing".format(len(indices), len(dataset), len(dataset_test)))

    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    num_classes = 2
    model = get_model(num_classes)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)

    # and a learning rate scheduler which decreases the learning rate by
    # 10x every 3 epochs

    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

    # let's train it for 10 epochs

    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    os.mkdir("pytorch object detection/raccoon/")
    torch.save(model.state_dict(), "pytorch object detection/raccoon/model")

if __name__ == '__main__':
    main()

Output: Turns out I had to reinstall torch and torch vision to make everything work


Instruction: Can I speed up inference in PyTorch using autocast (automatic mixed precision)?

Input: The docs (see also this) for autocast in PyTorch only discuss training.  Does it speed things up if I also use autocast for inference?

Output:
Yes it could (may not in some cases though).
You are processing data with lower precision (e.g. float16 vs float32).
Your program has to read and process less data in this case.
This might help with cache locality and hardware specific software (e.g. tensor cores if using CUDA)


Instruction: PyTorch torch.no_grad() versus requires_grad=False

Input:
I'm following a PyTorch tutorial which uses the BERT NLP model (feature extractor) from the Huggingface Transformers library. There are two pieces of interrelated code for gradient updates that I don't understand.
(1) torch.no_grad()
The tutorial has a class where the forward() function creates a torch.no_grad() block around a call to the BERT feature extractor, like this:
bert = BertModel.from_pretrained('bert-base-uncased')

class BERTGRUSentiment(nn.Module):

    def __init__(self, bert):
        super().__init__()
        self.bert = bert

    def forward(self, text):
        with torch.no_grad():
            embedded = self.bert(text)[0]

(2) param.requires_grad = False
There is another portion in the same tutorial where the BERT parameters are frozen.
for name, param in model.named_parameters():
    if name.startswith('bert'):
        param.requires_grad = False

When would I need (1) and/or (2)?

If I want to train with a frozen BERT, would I need to enable both?
If I want to train to let BERT be updated, would I need to disable both?

Additionaly, I ran all four combinations and found:
   with torch.no_grad   requires_grad = False  Parameters  Ran
   ------------------   ---------------------  ----------  ---
a. Yes                  Yes                      3M        Successfully
b. Yes                  No                     112M        Successfully
c. No                   Yes                      3M        Successfully
d. No                   No                     112M        CUDA out of memory

Can someone please explain what's going on? Why am I getting CUDA out of memory for (d) but not (b)? Both have 112M learnable parameters.

Output:
This is an older discussion, which has changed slightly over the years (mainly due to the purpose of with torch.no_grad() as a pattern. An excellent answer that kind of answers your question as well can be found on Stackoverflow already.
However, since the original question is vastly different, I'll refrain from marking as duplicate, especially due to the second part about the memory.
An initial explanation of no_grad is given here:

with torch.no_grad() is a context manager and is used to prevent calculating gradients [...].

requires_grad on the other hand is used

to freeze part of your model and train the rest [...].

Source again the SO post.
Essentially, with requires_grad you are just disabling parts of a network, whereas no_grad will not store any gradients at all, since you're likely using it for inference and not training.
To analyze the behavior of your combinations of parameters, let us investigate what is happening:

a) and b) do not store any gradients at all, which means that you have vastly more memory available to you, no matter the number of parameters, since you're not retaining them for a potential backward pass.
c) has to store the forward pass for later backpropagation, however, only a limited number of parameter (3 million) are stored, which makes this still manageable.
d), however, needs to store the forward pass for all 112 million parameters, which causes you to run out of memory.


Instruction: PyTorch C++ extension: How to index tensor and update it?

Input:
I'm creating a PyTorch C++ extension and after much research I can't figure out how to index a tensor and update its values. I found out how to iterate over a tensor's entries using the data_ptr() method, but that's not applicable to my use case.
Given is a matrix M, a list of lists (blocks) of index pairs P and a function f: dtype(M)^2 -> dtype(M)^2 that takes two values and spits out two new values.
I'm trying to implement the following pseudo code:
for each block B in P:
    for each row R in M:
        for each index-pair (i,j) in B:
            M[R,i], M[R,j] = f(M[R,i], M[R,j])

After all, this code is going to run on the GPU using CUDA, but since I don't have any experience with that, I wanted to first write a pure C++ program and then convert it.
Can anyone suggest how to do this or how to convert the algorithm to do something equivalent?

Output:
What I wanted to do can be done using the
tensor.accessor<scalar_dtype, num_dimensions>()
method. If executing on the GPU instead use scalars.packed_accessor64<scalar_dtype, num_dimensions, torch::RestrictPtrTraits>()
or
scalars.packed_accessor32<scalar_dtype, num_dimensions, torch::RestrictPtrTraits>() (depending on the size of your tensor).
auto num_rows = scalars.size(0);
matrix = torch::rand({10, 8});
auto a = matrix.accessor<float, 2>();
for (auto i = 0; i < num_rows; ++i) {
    auto x = a[i][some_index];
    auto new_x = some_function(x);
    a[i][some_index] = new_x;
}