Instruction: PyTorch - AssertionError: Size mismatch between tensors

Input:
I am trying to adapt a Pytorch script that was created for linear regression. It was originally written to take in a set of random values(created with np.random) as features and targets.
I have now created a dataframe of actual data for analysis:
df = pd.read_csv('file_name.csv')

The df looks like this:
      X1     X2     X3     X4    X5   X6   X7  X8    Y1     Y2
0    0.98  514.5  294.0  110.25  7.0   2  0.0   0  15.55  21.33
1    0.98  514.5  294.0  110.25  7.0   3  0.0   0  15.55  21.33
2    0.98  514.5  294.0  110.25  7.0   4  0.0   0  15.55  21.33
3    0.98  514.5  294.0  110.25  7.0   5  0.0   0  15.55  21.33
4    0.90  563.5  318.5  122.50  7.0   2  0.0   0  20.84  28.28

...and I am currently extracting just two columns(X1 and X2) as my features, and one column(Y1) as my targets, like this:
x = df[['X1', 'X2']]
y = df['Y1']

So features look like this:
      X1     X2
0    0.98  514.5
1    0.98  514.5
2    0.98  514.5
3    0.98  514.5
4    0.90  563.5

and targets look like this:
        Y1
0      15.55
1      15.55
2      15.55
3      15.55
4      20.84

However, when I attempt to convert the features (X1 and X1) and targets(Y1) to tensors, in order to feed them to the NN, the code fails at the line:
dataset = TensorDataset(x_tensor_flat, y_tensor_flat)
I get the error:
line 45, in <module> dataset = TensorDataset(x_tensor, y_tensor)
AssertionError: Size mismatch between tensors

There's clearly some shaping issue at play, but I can't work out what.
I have tried to flatten as well as transposing the tensors, but I get the same error.
Any help would be hugely appreciated.
Here's the full section of code that is causing the issue:
import pandas as pd
import torch
import torch.optim as optim
import torch.nn as nn
from torch.utils.data import Dataset, TensorDataset, DataLoader
from torch.utils.data.dataset import random_split

device = 'cuda' if torch.cuda.is_available() else 'cpu'

df = pd.read_csv('file_name.csv')
x = df[['X1', 'X2']]
y = df['Y1']

x_tensor = torch.from_numpy(np.array(x)).float()
y_tensor = torch.from_numpy(np.array(y)).float()

train_loader = DataLoader(dataset=train_dataset, batch_size=10)
val_loader = DataLoader(dataset=val_dataset, batch_size=10)

class ManualLinearRegression(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 1)

    def forward(self, x):
        return self.linear(x)

def make_train_step(model, loss_fn, optimizer):
    def train_step(x, y):
        model.train()
        yhat = model(x)
        loss = loss_fn(y, yhat)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        return loss.item()
    return train_step

torch.manual_seed(42)

model = ManualLinearRegression().to(device)
loss_fn = nn.MSELoss(reduction='mean')
optimizer = optim.SGD(model.parameters(), lr=1e-1)
train_step = make_train_step(model, loss_fn, optimizer)

n_epochs = 50
training_losses = []
validation_losses = []
print(model.state_dict())

for epoch in range(n_epochs):
    batch_losses = []
    for x_batch, y_batch in train_loader:
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        loss = train_step(x_batch, y_batch)
        batch_losses.append(loss)
    training_loss = np.mean(batch_losses)
    training_losses.append(training_loss)

    with torch.no_grad():
        val_losses = []
        for x_val, y_val in val_loader:
            x_val = x_val.to(device)
            y_val = y_val.to(device)
            model.eval()
            yhat = model(x_val)
            val_loss = loss_fn(y_val, yhat).item()
            val_losses.append(val_loss)
        validation_loss = np.mean(val_losses)
        validation_losses.append(validation_loss)

    print(f"[{epoch+1}] Training loss: {training_loss:.3f}\t Validation loss: {validation_loss:.3f}")

print(model.state_dict())

Output:
The problem is with how you have called the random_split function. Note that it takes lengths as input, not the percentage or ratio of the split. The error is about the same, i.e., the sum of lengths (80+20) that you have specified is not the same as the length of data (5).
The below code snippet should fix your problem. Also, you do not need to flatten tensors... I think.
dataset = TensorDataset(x_tensor, y_tensor)
val_size = int(len(dataset)*0.2)
train_size = len(dataset)- int(len(dataset)*0.2)
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])


Instruction: Load the ,trained model save, in pytorch for chatbots

Input:
I ran the code of this tutorial (link) and after a while learning the model was completed and I chatted with the trained model. After exit the program, in the save directory next to the program file, 8 files with a tar extension were created.
I guess they are trained model saved files, what should I do to load those files and reuse them?

Output:
In "Run the Model", they describe what you should do. More specifically, there is this code in their snippet:
# Set checkpoint to load from; set to None if starting from scratch
loadFilename = None
checkpoint_iter = 4000
#loadFilename = os.path.join(save_dir, model_name, corpus_name,
#                            '{}-{}_{}'.format(encoder_n_layers, decoder_n_layers, hidden_size),
#                            '{}_checkpoint.tar'.format(checkpoint_iter))

And you basically need to uncomment the #loadFilename and set the checkpoint_iter to the iteration you want. After that, you can skip the training part, because you already ran it, and run the evaluation code again.


Instruction: Best way to debug or step over a sequential pytorch model

Input:
I used to write the PyTorch model with nn.Module which included __init__ and forward so that I can step over my model to check how the variable dimension changes along the network.
However I have since realized that you can also do it with nn.Sequential which only requires an __init__, you don't need to write a forward function as below:

However, the problem is when I try to step over this network, it is not easy to check the variable any more. It just jumps to another place and back.
Does anyone know how to do step over in this situation?
P.S: I am using PyCharm.

Output:
You can iterate over the children of model like below and print sizes for debugging. This is similar to writing forward but you write a separate function instead of creating an nn.Module class.
import torch
from torch import nn

model = nn.Sequential(
    nn.Conv2d(1,20,5),
    nn.ReLU(),
    nn.Conv2d(20,64,5),
    nn.ReLU()
)

def print_sizes(model, input_tensor):
    output = input_tensor
    for m in model.children():
        output = m(output)
        print(m, output.shape)
    return output

input_tensor = torch.rand(100, 1, 28, 28)
print_sizes(model, input_tensor)

# output:
# Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) torch.Size([100, 20, 24, 24])
# ReLU() torch.Size([100, 20, 24, 24])
# Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1)) torch.Size([100, 64, 20, 20])
# ReLU() torch.Size([100, 64, 20, 20])

# you can also nest the Sequential models like this. In this case inner Sequential will be considered as module itself.
model1 = nn.Sequential(
    nn.Conv2d(1,20,5),
    nn.ReLU(),
    nn.Sequential(
        nn.Conv2d(20,64,5),
        nn.ReLU()
    )
)

print_sizes(model1, input_tensor)

# output:
# Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1)) torch.Size([100, 20, 24, 24])
# ReLU() torch.Size([100, 20, 24, 24])
# Sequential(
#     (0): Conv2d(20, 64, kernel_size=(5, 5), stride=(1, 1))
#     (1): ReLU()
# ) torch.Size([100, 64, 20, 20])


Instruction: loss defined on repeated evaluation of the same model

Input:
I have a model denoted by f(),
Suppose the target is t, f(x1) = y1 and f(x2) = y2 and my loss is defined as
loss = mse(y1,y2) + mse(y2,t)
Since both y1 and y2 reguires grad, I have received error such as

one of the variables needed for gradient computation has been modified by an inplace operation

My understanding is that, suppose I evaluate y1 first, the graph has been changed upon my evaluation of y2. Should I fix some tensor such as ,e.g., y1_no_grad = y1.detach().numpy() and then use
loss = mse(y1_no_grad,y2) + mse(y2,t)?
However, I still receive error Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy(), which I am not sure if it is because y1_no_grad is a numpy array while y2 is a tensor.
Update:
I realized my problem afterwards. It was due to that I created multiple loss tensors and that I backwarded one loss tensor first which changed the parameters in-place. This caused error when I wanted to backward another loss tensor.
E.g.
f(x1) = y1
f(x2) = y2
f(x3) = y3
...
f(xn) = yn

f(x) = y

for i in range(n):
    optimizer.zero_grad()
    loss = mse(y,yi) + mse(yi,t)
    loss.backward()
    optimizer.step()

To me the solutions are either:
1.accumulate the loss tensors before doing backward , i.e.
for i in range(n):
    loss = mse(y,yi) + mse(yi,t)
    loss.backward()
optimizer.step()

2.Evaluate again before each backward, i.e.:
for i in range(n):
    optimizer.zero_grad()
    y = f(x)
    yi = f(xi)
    loss = mse(y,yi) + mse(yi,t)
    loss.backward()
    optimizer.step()

Output:
Suppose the target is t, f(x1) = y1 and f(x2) = y2 and my loss is defined as
loss = mse(y1,y2) + mse(y2,t)
Since both y1 and y2 reguires grad, I have received error such as

This statement is incorrect. What you've described does not necessiate in-place assignment errors. For example
import torch
from torch.nn.functional import mse_loss

def f(x):
    return x**2

t = torch.ones(1)

x1 = torch.randn(1, requires_grad=True)
x2 = torch.randn(1, requires_grad=True)

y1 = f(x1)
y2 = f(x2)

loss = mse_loss(y1, y2) + mse_loss(y2, t)
loss.backward()

does not produce any errors. Likely your issue is somewhere else.
For the general case you described you should get a computation graph that could be visualized as

The only issue here could be that your function f is not differentiable or is somehow invalid (perhaps in in-place assignment is taking place in f).


Instruction: Creating a tensor where each entry is a function of its index

Input:
I want to create a matrix D that is defined by D[i,j]=d(i-j) where d is some arbitrary function that I can choose.
It can easily be done with loops, but it is very slow. Is there an efficiant way of creating this matrix with torch or numpy?

Output:
You could apply the function (if vectorised) to numpy.indices:
import numpy as np

i, j = np.indices((n, m))
D = d(i - j)


Instruction: RuntimeError: mat1 dim 1 must match mat2 dim 0

Input:
I am still grappling with PyTorch, having played with Keras for a while (which feels a lot more intuitive).
Anyway - I have the nn.linear model code below, which works fine for just one input feature, where:
inputDim = 1

I am now trying to expand the same code to include 2 features, and so I have included another column in my feature dataframe and also set:
inputDim = 2

However, when I run the code, I get the dreaded error:
RuntimeError: mat1 dim 1 must match mat2 dim 0

This error references line 63, which is:
    outputs = model(inputs)

I have gone through several other posts here relating to this dimensionality error, but I still can't see what is wrong with my code. Any help would be appreciated.
The full code looks like this:
import numpy as np
import pandas as pd
import torch
from torch.autograd import Variable
import matplotlib.pyplot as plt

device = 'cuda' if torch.cuda.is_available() else 'cpu'

df = pd.read_csv('Adjusted Close - BAC-UBS-WFC.csv')
x = df[['BAC', 'UBS']]
y = df['WFC']

# number_of_features = x.shape[1]
# print(number_of_features)

x_train = np.array(x, dtype=np.float32)
x_train = x_train.reshape(-1, 1)

y_train = np.array(y, dtype=np.float32)
y_train = y_train.reshape(-1, 1)

class linearRegression(torch.nn.Module):
    def __init__(self, inputSize, outputSize):
        super(linearRegression, self).__init__()
        self.linear = torch.nn.Linear(inputSize, outputSize)

    def forward(self, x):
        out = self.linear(x)
        return out

inputDim = 2
outputDim = 1
learningRate = 0.01
epochs = 500

# Model instantiation
torch.manual_seed(42)
model = linearRegression(inputDim, outputDim)
if torch.cuda.is_available(): model.cuda()

criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learningRate)

# Model training
loss_series = []
for epoch in range(epochs):
    # Converting inputs and labels to Variable
    inputs = Variable(torch.from_numpy(x_train).cuda())
    labels = Variable(torch.from_numpy(y_train).cuda())

    # Clear gradient buffers because we don't want any gradient from previous epoch to carry forward, dont want to cummulate gradients
    optimizer.zero_grad()

    # get output from the model, given the inputs
    outputs = model(inputs)

    # get loss for the predicted output
    loss = criterion(outputs, labels)
    loss_series.append(loss.item())
    print(loss)
    # get gradients w.r.t to parameters
    loss.backward()

    # update parameters
    optimizer.step()

    print('epoch {}, loss {}'.format(epoch, loss.item()))

# Calculate predictions on training data
with torch.no_grad():  # we don't need gradients in the testing phase
    predicted = model(Variable(torch.from_numpy(x_train).cuda())).cpu().data.numpy()

Output:
General advice: For errors with dimension, it usually helps to print out dimensions at each step of the computation.
Most likely in this specific case, you have made mistake in reshaping the input with this x_train = x_train.reshape(-1, 1)
Your input is (N,1) but NN expects (N,2).


Instruction: pytorch initialize two sub-modules with same weights?

Input:
I was wondering if there's a typical way to initialize two parts of one network to same some initial values? Say I got two separate auto-encoders for both query and document, and I would like to initialize the weights of these two auto-encoders to same weights (not sharing the weights).
Thanks!

Output: I think the most easy way would be to init one of the sub-modules at random, save the state_dict and then load_state_dict from the other module.


Instruction: PyTorch: RuntimeError: The size of tensor a (224) must match the size of tensor b (244) at non-singleton dimension 3

Input:
I want to create and train AutoEncoder to extract features and use that features for the clustering algorithms. Right now I am getting errors while calculating the loss.
RuntimeError: The size of tensor a (224) must match the size of tensor b (244) at non-singleton dimension 3
and a warning
UserWarning: Using a target size (torch.Size([1, 3, 224, 244])) that is different to the input size (torch.Size([1, 3, 224, 224])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

return F.mse_loss(input, target, reduction=self.reduction)*

I am using Pytorch.
Can anyone tell me what is wrong with this? In warning and error size of input and output is the same but it is saying it is different.
The summary sizes of input and output images are as follow
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 16, 112, 112]             448
              ReLU-2         [-1, 16, 112, 112]               0
            Conv2d-3           [-1, 32, 56, 56]           4,640
              ReLU-4           [-1, 32, 56, 56]               0
            Conv2d-5           [-1, 64, 18, 18]         100,416
              ReLU-6           [-1, 64, 18, 18]               0
            Conv2d-7            [-1, 128, 3, 3]         401,536
              ReLU-8            [-1, 128, 3, 3]               0
            Conv2d-9            [-1, 256, 1, 1]         295,168
  ConvTranspose2d-10            [-1, 128, 3, 3]         295,040
             ReLU-11            [-1, 128, 3, 3]               0
  ConvTranspose2d-12           [-1, 64, 12, 12]         401,472
             ReLU-13           [-1, 64, 12, 12]               0
  ConvTranspose2d-14           [-1, 24, 28, 28]          75,288
             ReLU-15           [-1, 24, 28, 28]               0
  ConvTranspose2d-16           [-1, 16, 56, 56]           3,472
             ReLU-17           [-1, 16, 56, 56]               0
  ConvTranspose2d-18          [-1, 8, 111, 111]           1,160
             ReLU-19          [-1, 8, 111, 111]               0
  ConvTranspose2d-20          [-1, 3, 224, 224]             603
          Sigmoid-21          [-1, 3, 224, 224]               0
================================================================
Total params: 1,579,243
Trainable params: 1,579,243
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 9.94
Params size (MB): 6.02
Estimated Total Size (MB): 16.54
----------------------------------------------------------------
Min Value of input Image =  tensor(0.0627)
Max Value of input Image =  tensor(0.5098)
Input Image shape =  torch.Size([1, 3, 224, 244])
Output Image shape =  torch.Size([1, 3, 224, 224])

My Autoencoder class is
class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(

            nn.Conv2d(3, 16, 3, stride=2, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.Conv2d(16, 32, 3, stride=2, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.Conv2d(32, 64, 7, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.Conv2d(64, 128, 7, stride=5, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.Conv2d(128, 256, 3, stride=5, padding=1)  # b, 16, 10, 10

        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(256, 128, 3),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(128, 64, 7,stride=3, padding=1,output_padding=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(64, 24, 7,stride=2, padding=1,output_padding=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(24, 16, 3, stride=2, padding=1,output_padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 3, stride=2, padding=1),  # b, 1, 28, 28
            nn.ReLU(True),
            nn.ConvTranspose2d(8,3, 5, stride=2, padding=1,output_padding=1),  # b, 1, 28, 28
            nn.Sigmoid()

        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

and training function is as follow
dataset = DatasetLoader('E:/DAL/Dataset/Images', get_transform(train=True))

torch.manual_seed(1)
indices = torch.randperm(len(dataset)).tolist()
dataset = torch.utils.data.Subset(dataset, indices[:-50])
dataset_test = torch.utils.data.Subset(dataset, indices[-50:])

data_loader = torch.utils.data.DataLoader(
    dataset, batch_size=1, shuffle=True, num_workers=0)

data_loader_test = torch.utils.data.DataLoader(
    dataset_test, batch_size=1, shuffle=False, num_workers=0)

model = autoencoder().cuda()
summary(model, (3, 224, 224))

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,weight_decay=1e-5)

total_loss = 0
for epoch in range(num_epochs):
    for data in data_loader:
        # print(data)
        img = data
        print("Min Value of input Image = ",torch.min(img))
        print("Max Value of input Image = ",torch.max(img))
        img = Variable(img).cuda()
        # ===================forward=====================
        output = model(img)
        print("Input Image shape = ",img.shape)
        print("Output Image shape = ",output.shape)
        loss = criterion(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log========================
    total_loss += loss.data
    print('epoch [{}/{}], loss:{:.4f}'
          .format(epoch+1, num_epochs, total_loss))
    if epoch % 10 == 0:
        pic = to_img(output.cpu().data)
        save_image(pic, './dc_img/image_{}.png'.format(epoch))

torch.save(model.state_dict(), './conv_autoencoder.pth')

Dataloader Class and transform function is as follow
def get_transform(train):
    transforms = []

    transforms.append(T.Resize((224,244)))
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
        transforms.append(T.RandomVerticalFlip(0.5))
    transforms.append(T.ToTensor())
    return T.Compose(transforms)

class DatasetLoader(torch.utils.data.Dataset):
    def __init__(self, root, transforms=None):
        self.root = root
        self.transforms = transforms
        self.imgs = list(sorted(os.listdir(root)))

    def __getitem__(self, idx):
        img_path = os.path.join(self.root, self.imgs[idx])

        img = Image.open(img_path).convert("RGB")

        if self.transforms is not None:
            img = self.transforms(img)

        return img

    def __len__(self):
        return len(self.imgs)

Output:
Im pretty sure you got a typo in your get_transform function:
transforms.append(T.Resize((224,244)))

You probably wanted to resize it to (224, 224) instead of (224, 244).


Instruction: Pytorch GAN model doesn't train: matrix multiplication error

Input:
I'm trying to build a basic GAN to familiarise myself with Pytorch. I have some (limited) experience with Keras, but since I'm bound to do a larger project in Pytorch, I wanted to explore first using 'basic' networks.
I'm using Pytorch Lightning. I think I've added all necessary components. I tried passing some noise through the generator and the discriminator separately, and I think the output has the expected shape. Nonetheless, I get a runtime error when I try to train the GAN (full traceback below):
RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x9 and 25x1)
I noticed that 7 is the size of the batch (by printing out the batch dimensions), even though I specified batch_size to be 64. Other than that, quite honestly, I don't know where to begin: the error traceback doesn't help me.
Chances are, I made multiple mistakes. However, I'm hoping some of you will be able to spot the current error from the code, since the multiplication error seems to point towards a dimensionality problem somewhere. Here's the code.
import os

import pytorch_lightning as pl
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as transforms
from skimage import io
from torch.utils.data import Dataset, DataLoader, random_split
from torchvision.utils import make_grid
from torchvision.transforms import Resize, ToTensor, ToPILImage, Normalize

class DoppelDataset(Dataset):
    """
    Dataset class for face data
    """

    def __init__(self, face_dir: str, transform=None):

        self.face_dir = face_dir
        self.face_paths = os.listdir(face_dir)
        self.transform = transform

    def __len__(self):

        return len(self.face_paths)

    def __getitem__(self, idx):

        if torch.is_tensor(idx):
            idx = idx.tolist()

        face_path = os.path.join(self.face_dir, self.face_paths[idx])
        face = io.imread(face_path)

        sample = {'image': face}

        if self.transform:
            sample = self.transform(sample['image'])

        return sample

class DoppelDataModule(pl.LightningDataModule):

    def __init__(self, data_dir='../data/faces', batch_size: int = 64, num_workers: int = 0):
        super().__init__()
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.num_workers = num_workers

        self.transforms = transforms.Compose([
            ToTensor(),
            Resize(100),
            Normalize(mean=(123.26290927634774, 95.90498110733365, 86.03763122875182),
                      std=(63.20679012922922, 54.86211954409834, 52.31266645797249))
        ])

    def setup(self, stage=None):
        # Initialize dataset
        doppel_data = DoppelDataset(face_dir=self.data_dir, transform=self.transforms)

        # Train/val/test split
        n = len(doppel_data)
        train_size = int(.8 * n)
        val_size = int(.1 * n)
        test_size = n - (train_size + val_size)

        self.train_data, self.val_data, self.test_data = random_split(dataset=doppel_data,
                                                                      lengths=[train_size, val_size, test_size])

    def train_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.test_data, batch_size=self.batch_size, num_workers=self.num_workers)

    def val_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.val_data, batch_size=self.batch_size, num_workers=self.num_workers)

    def test_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.test_data, batch_size=self.batch_size, num_workers=self.num_workers)

class DoppelGenerator(nn.Sequential):
    """
    Generator network that produces images based on latent vector
    """

    def __init__(self, latent_dim: int):
        super().__init__()

        def block(in_channels: int, out_channels: int, padding: int = 1, stride: int = 2, bias=False):
            return nn.Sequential(
                nn.ConvTranspose2d(in_channels=in_channels, out_channels=out_channels, kernel_size=4, stride=stride,
                                   padding=padding, bias=bias),
                nn.BatchNorm2d(num_features=out_channels),
                nn.ReLU(True)
            )

        self.model = nn.Sequential(
            block(latent_dim, 512, padding=0, stride=1),
            block(512, 256),
            block(256, 128),
            block(128, 64),
            block(64, 32),
            nn.ConvTranspose2d(32, 3, kernel_size=4, stride=2, padding=1, bias=False),
            nn.Tanh()
        )

    def forward(self, input):
        return self.model(input)

class DoppelDiscriminator(nn.Sequential):
    """
    Discriminator network that classifies images in two categories
    """

    def __init__(self):
        super().__init__()

        def block(in_channels: int, out_channels: int):
            return nn.Sequential(
                nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=4, stride=2, padding=1,
                          bias=False),
                nn.BatchNorm2d(num_features=out_channels),
                nn.LeakyReLU(0.2, inplace=True),
            )

        self.model = nn.Sequential(
            block(3, 64),
            block(64, 128),
            block(128, 256),
            block(256, 512),
            nn.Conv2d(512, 1, kernel_size=4, stride=1, padding=0, bias=False),
            nn.Flatten(),
            nn.Linear(25, 1),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.model(input)

class DoppelGAN(pl.LightningModule):

    def __init__(self,
                 channels: int,
                 width: int,
                 height: int,
                 lr: float = 0.0002,
                 b1: float = 0.5,
                 b2: float = 0.999,
                 batch_size: int = 64,
                 **kwargs):

        super().__init__()

        # Save all keyword arguments as hyperparameters, accessible through self.hparams.X)
        self.save_hyperparameters()

        # Initialize networks
        # data_shape = (channels, width, height)
        self.generator = DoppelGenerator(latent_dim=self.hparams.latent_dim, )
        self.discriminator = DoppelDiscriminator()

        self.validation_z = torch.randn(8, self.hparams.latent_dim,1,1)

    def forward(self, input):
        return self.generator(input)

    def adversarial_loss(self, y_hat, y):
        return F.binary_cross_entropy(y_hat, y)

    def training_step(self, batch, batch_idx, optimizer_idx):
        images = batch

        # Sample noise (batch_size, latent_dim,1,1)
        z = torch.randn(images.size(0), self.hparams.latent_dim,1,1)

        # Train generator
        if optimizer_idx == 0:

            # Generate images (call generator -- see forward -- on latent vector)
            self.generated_images = self(z)

            # Log sampled images (visualize what the generator comes up with)
            sample_images = self.generated_images[:6]
            grid = make_grid(sample_images)
            self.logger.experiment.add_image('generated_images', grid, 0)

            # Ground truth result (ie: all fake)
            valid = torch.ones(images.size(0), 1)

            # Adversarial loss is binary cross-entropy
            generator_loss = self.adversarial_loss(self.discriminator(self(z)), valid)
            tqdm_dict = {'gen_loss': generator_loss}

            output = {
                'loss': generator_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            }
            return output

        # Train discriminator: classify real from generated samples
        if optimizer_idx == 1:

            # How well can it label as real?
            valid = torch.ones(images.size(0), 1)
            real_loss = self.adversarial_loss(self.discriminator(images), valid)

            # How well can it label as fake?
            fake = torch.zeros(images.size(0), 1)
            fake_loss = self.adversarial_loss(
                self.discriminator(self(z).detach()), fake)

            # Discriminator loss is the average of these
            discriminator_loss = (real_loss + fake_loss) / 2
            tqdm_dict = {'d_loss': discriminator_loss}
            output = {
                'loss': discriminator_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            }
            return output

    def configure_optimizers(self):
        lr = self.hparams.lr
        b1 = self.hparams.b1
        b2 = self.hparams.b2

        # Optimizers
        opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))

        # Return optimizers/schedulers (currently no scheduler)
        return [opt_g, opt_d], []

    def on_epoch_end(self):

        # Log sampled images
        sample_images = self(self.validation_z)
        grid = make_grid(sample_images)
        self.logger.experiment.add_image('generated_images', grid, self.current_epoch)

if __name__ == '__main__':

    # Global parameter
    image_dim = 128
    latent_dim = 100
    batch_size = 64

    # Initialize dataset
    tfs = transforms.Compose([
        ToPILImage(),
        Resize(image_dim),
        ToTensor()
    ])
    doppel_dataset = DoppelDataset(face_dir='../data/faces', transform=tfs)

    # Initialize data module
    doppel_data_module = DoppelDataModule(batch_size=batch_size)

    # Build models
    generator = DoppelGenerator(latent_dim=latent_dim)
    discriminator = DoppelDiscriminator()

    # Test generator
    x = torch.rand(batch_size, latent_dim, 1, 1)
    y = generator(x)
    print(f'Generator: x {x.size()} --> y {y.size()}')

    # Test discriminator
    x = torch.rand(batch_size, 3, 128, 128)
    y = discriminator(x)
    print(f'Discriminator: x {x.size()} --> y {y.size()}')

    # Build GAN
    doppelgan = DoppelGAN(batch_size=batch_size, channels=3, width=image_dim, height=image_dim, latent_dim=latent_dim)

    # Fit GAN
    trainer = pl.Trainer(gpus=0, max_epochs=5, progress_bar_refresh_rate=1)
    trainer.fit(model=doppelgan, datamodule=doppel_data_module)

Full traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-28805d67d74b>", line 1, in <module>
    runfile('/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py', wdir='/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 298, in <module>
    trainer.fit(model=doppelgan, datamodule=doppel_data_module)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 510, in fit
    results = self.accelerator_backend.train()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 57, in train
    return self.train_or_test()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in train_or_test
    results = self.trainer.train()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 561, in train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 550, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 718, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 485, in optimizer_step
    model_ref.optimizer_step(
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1298, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 286, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 144, in __optimizer_step
    optimizer.step(closure=closure, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 708, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 806, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 319, in training_step
    training_step_output = self.trainer.accelerator_backend.training_step(args)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 62, in training_step
    return self._step(self.trainer.model.training_step, args)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 58, in _step
    output = model_step(*args)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 223, in training_step
    real_loss = self.adversarial_loss(self.discriminator(images), valid)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 154, in forward
    return self.model(input)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x9 and 25x1)

Output:
This multiplication problem comes from the DoppelDiscriminator. There is a linear layer
    nn.Linear(25, 1),

that should be
    nn.Linear(9, 1),

based on the error message.


Instruction: Cannot fix ImportError: cannot import name 'ToTensorV2' from 'albumentations.pytorch' on Colab

Input:
I tried all three solutions from this question to solve the problem, but still receive an error:
ImportError: cannot import name 'ToTensorV2' from 'albumentations.pytorch' (/usr/local/lib/python3.7/dist-packages/albumentations/pytorch/__init__.py)

My current code:
!pip install -U albumentations
import albumentations
from albumentations.pytorch import ToTensorV2

Output:
Have you tried factory resetting the runtime? If you haven't, maybe python was still loading the earlier version. I have just tested the accepted solution from the linked question on a fresh colab instance and it worked fine:
!pip install albumentations==0.4.6
import albumentations
from albumentations.pytorch import ToTensorV2


Instruction: Why use Variable() in inference?

Input:
I am learning PyTorch for an image classification task, and I ran into code where someone used a PyTorch Variable() in their function for prediction:
def predict_image(image):
    image_tensor = test_transforms(image).float()
    image_tensor = image_tensor.unsqueeze_(0)
    input = Variable(image_tensor)
    input = input.to(device)
    output = model(input)
    index = output.data.cpu().numpy().argmax()
    return index

Why do they use Variable() here? (even though it works fine without it.)

Output:
You can safely omit it. Variables are a legacy component of PyTorch, now deprecated, that used to be required for autograd:

Variable (deprecated)
WARNING
The Variable API has been deprecated: Variables are no longer necessary to use autograd with tensors. Autograd automatically supports Tensors with requires_grad set to True. Below please find a quick guide on what has changed:

Variable(tensor) and Variable(tensor, requires_grad) still work as expected, but they return Tensors instead of Variables.


Instruction: the Training is stopped for my PyTorch model

Input:
I am creating a model to identify names of items in RNN (LITMS)
i get data , then transform data then create Baches then create Model then create train function Correctly but the training is stop here(not working):

this this my code
        for e in range(epochs):

            # initialize hidden state
            h = net.init_hidden(batch_size)

            for x, y in get_batches(data, batch_size, seq_length):
                print ("the login the loob get_batches is succressfuly")
                counter += 1

                # One-hot encode our data and make them Torch tensors
                x = one_hot_encode(x, n_chars)
                inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

                if(train_on_gpu):
                    inputs, targets = inputs.cuda(), targets.cuda()

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                h = tuple([each.data for each in h])

                # zero accumulated gradients
                net.zero_grad()

                # get the output from the model
                output, h = net(inputs, h)

                # calculate the loss and perform backprop
                loss = criterion(output, targets.view(batch_size*seq_length))
                loss.backward()
                # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
                nn.utils.clip_grad_norm_(net.parameters(), clip)
                opt.step()

                # loss stats
                if counter % print_every == 0:
                    # Get validation loss
                    val_h = net.init_hidden(batch_size)
                    val_losses = []
                    net.eval()
                    for x, y in get_batches(val_data, batch_size, seq_length):
                        # One-hot encode our data and make them Torch tensors
                        x = one_hot_encode(x, n_chars)
                        x, y = torch.from_numpy(x), torch.from_numpy(y)

                        # Creating new variables for the hidden state, otherwise
                        # we'd backprop through the entire training history
                        val_h = tuple([each.data for each in val_h])

                        inputs, targets = x, y
                        if(train_on_gpu):
                            inputs, targets = inputs.cuda(), targets.cuda()

                        output, val_h = net(inputs, val_h)
                        val_loss = criterion(output, targets.view(batch_size*seq_length))

                        val_losses.append(val_loss.item())

                    net.train() # reset to train mode after iterationg through validation data

                    print("Epoch: {}/{}...".format(e+1, epochs),
                        "Step: {}...".format(counter),
                        "Loss: {:.4f}...".format(loss.item()),
                        "Val Loss: {:.4f}".format(np.mean(val_losses)))

I don't know why.
some times i got this error after many trials

please help me if you can
You can find the notebook file here.

Output:
you duplicate this step

remove it then try again


Instruction: Fine-tuning model's classifier layer with new label

Input:
I would like to fine-tune already fine-tuned BertForSequenceClassification model with new dataset containing just 1 additional label which hasn't been seen by model before.
By that, I would like to add 1 new label to the set of labels that model is currently able of classifying properly.
Moreover, I don't want classifier weights to be randomly initialized, I'd like to keep them intact and just update them accordingly to the dataset examples while increasing the size of classifier layer by 1.
The dataset used for further fine-tuning could look like this:
sentece,label
intent example 1,new_label
intent example 2,new_label
...
intent example 10,new_label

My model's current classifier layer looks like this:
Linear(in_features=768, out_features=135, bias=True)

How could I achieve it?
Is it even a good approach?

Output:
You can just extend the weights and bias of your model with new values. Please have a look at the commented example below:
#This is the section that loads your model
#I will just use an pretrained model for this example
import torch
from torch import nn
from transformers import AutoModelForSequenceClassification, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("jpcorb20/toxic-detector-distilroberta")
model = AutoModelForSequenceClassification.from_pretrained("jpcorb20/toxic-detector-distilroberta")
#we check the output of one sample to compare it later with the extended layer
#to verify that we kept the previous learnt "knowledge"
f = tokenizer.encode_plus("This is an example", return_tensors='pt')
print(model(**f).logits)

#Now we need to find out the name of the linear layer you want to extend
#The layers on top of distilroberta are wrapped inside a classifier section
#This name can differ for you because it can be chosen randomly
#use model.parameters instead find the classification layer
print(model.classifier)

#The output shows us that the classification layer is called `out_proj`
#We can now extend the weights by creating a new tensor that consists of the
#old weights and a randomly initialized tensor for the new label
model.classifier.out_proj.weight = nn.Parameter(torch.cat((model.classifier.out_proj.weight, torch.randn(1,768)),0))

#We do the same for the bias:
model.classifier.out_proj.bias = nn.Parameter(torch.cat((model.classifier.out_proj.bias, torch.randn(1)),0))

#and be happy when we compare the output with our expectation
print(model(**f).logits)

Output:
tensor([[-7.3604, -9.4899, -8.4170, -9.7688, -8.4067, -9.3895]],
       grad_fn=<AddmmBackward>)
RobertaClassificationHead(
  (dense): Linear(in_features=768, out_features=768, bias=True)
  (dropout): Dropout(p=0.1, inplace=False)
  (out_proj): Linear(in_features=768, out_features=6, bias=True)
)
tensor([[-7.3604, -9.4899, -8.4170, -9.7688, -8.4067, -9.3895,  2.2124]],
       grad_fn=<AddmmBackward>)


Instruction: RuntimeError: tensors must be 2-D

Input:
I was running a simple MLP network with customized learning algorithms. It worked fine on the training set, but I got this error when I entered additional code to check the test accuracy. How can I fix it?
Test Accuracy code
epochs = 1
for epcoh in range(epochs):
    model_bp.eval()
    model_fa.eval()
    test_loss_bp = 0
    correct_bp = 0
    test_loss_fa = 0
    correct_fa = 0
    with torch.no_grad():
        for idx_batch, (inputs, targets) in enumerate(test_loader):

            output_bp = model_bp(inputs)
            output_fa = model_fa(inputs)
            # sum up batch loss
            test_loss_bp += loss_crossentropy(output_bp, targets).item()
            test_loss_bp += loss_crossentropy(output_fa, targets).item()
            # get the index of the max log-probability
            ## predict_bp = outputs_bp.argmax(dim=1, keepdim=True)
            predict_bp = torch.max(output_bp.data,1)[1]
            correct_bp += predict_bp.eq(targets.view_as(predict_bp)).sum().item()

            predict_fa = torch.max(output_fa.data,1)[1]
            correct_fa += predict_fa.eq(targets.view_as(predict_fa)).sum().item()

    print('Test set: BP Average loss: {:.4f}, Accuracy: {}/{} ({:.4f}%)\n'.format(test_loss_bp, correct_bp, len(test_loader.dataset),
            100. * correct_bp / len(test_loader.dataset)))
    print('Test set: FA Average loss: {:.4f}, Accuracy: {}/{} ({:.4f}%)\n'.format(test_loss_fa, correct_fa, len(test_loader.dataset),
            100. * correct_fa / len(test_loader.dataset)))

Error
I'm curious about the meaning of 'RuntimeError: tensors must be 2-D'. We would appreciate it if you could tell us why it happened and where you made the mistake.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-9b8b6f683e59> in <module>
     16             #targets = targets.to(device)
     17
---> 18             output_bp = model_bp(inputs)
     19             output_fa = model_fa(inputs)
     20             # sum up batch loss

~\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(self, inputs)
    102         """
    103         # first layer
--> 104         linear1 = F.relu(self.linear[0](inputs))
    105
    106         linear2 = self.linear[1](linear1)

~\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(self, input)
     69     def forward(self, input):
     70         # See the autograd section for explanation of what happens here.
---> 71         return LinearFunction.apply(input, self.weight, self.bias)
     72
     73

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(ctx, input, weight, bias)
     11     def forward(ctx, input, weight, bias=None):
     12         ctx.save_for_backward(input, weight, bias)
---> 13         output = input.mm(weight.t())
     14         if bias is not None:
     15             output += bias.unsqueeze(0).expand_as(output)

RuntimeError: tensors must be 2-D

This is my model. And fa_linear, linear : customize network
# load feedforward dfa model
model_fa = fa_linear.LinearFANetwork(in_features=784, num_layers=2, num_hidden_list=[1000, 10]).to(device)

# load reference linear model
model_bp = linear.LinearNetwork(in_features=784, num_layers=2, num_hidden_list=[1000, 10]).to(device)

# optimizers
optimizer_fa = torch.optim.SGD(model_fa.parameters(),
                            lr=1e-4, momentum=0.9, weight_decay=0.001, nesterov=True)
optimizer_bp = torch.optim.SGD(model_bp.parameters(),
                            lr=1e-4, momentum=0.9, weight_decay=0.001, nesterov=True)

loss_crossentropy = torch.nn.CrossEntropyLoss()

# make log file
results_path = 'bp_vs_fa_'
logger_train = open(results_path + 'train_log2.txt', 'w')

linear
from torch.autograd import Function
from torch import nn
import torch
import torch.nn.functional as F

# Inherit from Function
class LinearFunction(Function):
    # Note that both forward and backward are @staticmethods
    @staticmethod
    # bias is an optional argument
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    # This function has only a single output, so it gets only one gradient
    @staticmethod
    def backward(ctx, grad_output):
        # This is a pattern that is very convenient - at the top of backward
        # unpack saved_tensors and initialize all gradients w.r.t. inputs to
        # None. Thanks to the fact               that additional trailing Nones are
        # ignored, the return statement is simple even when the function has
        # optional inputs.
        input, weight, bias = ctx.saved_variables
        grad_input = grad_weight = grad_bias = None
        # These needs_input_grad checks are optional and there only to
        # improve efficiency. If you want to make your code simpler, you can
        # skip them. Returning gradients for inputs that don't require it is
        # not an error.

        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias

class Linear(nn.Module):
    def __init__(self, input_features, output_features, bias=True):
        super(Linear, self).__init__()
        self.input_features = input_features
        self.output_features = output_features

        # nn.Parameter is a special kind of Variable, that will get
        # automatically registered as Module's parameter once it's assigned
        # as an attribute. Parameters and buffers need to be registered, or
        # they won't appear in .parameters() (doesn't apply to buffers), and
        # won't be converted when e.g. .cuda() is called. You can use
        # .register_buffer() to register buffers.
        # nn.Parameters can never be volatile and, different than Variables,
        # they require gradients by default.
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(output_features))
        else:
            # You should always register all possible parameters, but the
            # optional ones can be None if you want.
            self.register_parameter('bias', None)

        # weight initialization
        torch.nn.init.kaiming_uniform(self.weight)
        torch.nn.init.constant(self.bias, 1)

    def forward(self, input):
        # See the autograd section for explanation of what happens here.
        return LinearFunction.apply(input, self.weight, self.bias)

class LinearNetwork(nn.Module):
    def __init__(self, in_features, num_layers, num_hidden_list):
        """
        :param in_features: dimension of input features (784 for MNIST)
        :param num_layers: number of layers for feed-forward net
        :param num_hidden_list: list of integers indicating hidden nodes of each layer
        """
        super(LinearNetwork, self).__init__()
        self.in_features = in_features
        self.num_layers = num_layers
        self.num_hidden_list = num_hidden_list

        # create list of linear layers
        # first hidden layer
        self.linear = [Linear(self.in_features, self.num_hidden_list[0])]
        # append additional hidden layers to list
        for idx in range(self.num_layers - 1):
            self.linear.append(Linear(self.num_hidden_list[idx], self.num_hidden_list[idx+1]))

        # create ModuleList to make list of layers work
        self.linear = nn.ModuleList(self.linear)

    def forward(self, inputs):
        """
        forward pass, which is same for conventional feed-forward net
        :param inputs: inputs with shape [batch_size, in_features]
        :return: logit outputs from the network
        """
        # first layer
        linear1 = F.relu(self.linear[0](inputs))

        linear2 = self.linear[1](linear1)

        return linear2

fa_linear
import torch
import torch.nn.functional as F
import torch.nn as nn
from torch import autograd
from torch.autograd import Variable

class LinearFANetwork(nn.Module):
    """
    Linear feed-forward networks with feedback alignment learning
    Does NOT perform non-linear activation after each layer
    """
    def __init__(self, in_features, num_layers, num_hidden_list):
        """
        :param in_features: dimension of input features (784 for MNIST)
        :param num_layers: number of layers for feed-forward net
        :param num_hidden_list: list of integers indicating hidden nodes of each layer
        """
        super(LinearFANetwork, self).__init__()
        self.in_features = in_features
        self.num_layers = num_layers
        self.num_hidden_list = num_hidden_list

        # create list of linear layers
        # first hidden layer
        self.linear = [LinearFAModule(self.in_features, self.num_hidden_list[0])]
        # append additional hidden layers to list
        for idx in range(self.num_layers - 1):
            self.linear.append(LinearFAModule(self.num_hidden_list[idx], self.num_hidden_list[idx+1]))

        # create ModuleList to make list of layers work
        self.linear = nn.ModuleList(self.linear)

    def forward(self, inputs):
        """
        forward pass, which is same for conventional feed-forward net
        :param inputs: inputs with shape [batch_size, in_features]
        :return: logit outputs from the network
        """

        # first layer
        linear1 = self.linear[0](inputs)

        # second layer
        linear2 = self.linear[1](linear1)

        return linear2

class LinearFAFunction(autograd.Function):

    @staticmethod
    # same as reference linear function, but with additional fa tensor for backward
    def forward(context, input, weight, weight_fa, bias=None):
        context.save_for_backward(input, weight, weight_fa, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    @staticmethod
    def backward(context, grad_output):
        input, weight, weight_fa, bias = context.saved_variables
        grad_input = grad_weight = grad_weight_fa = grad_bias = None

        if context.needs_input_grad[0]:
            # all of the logic of FA resides in this one line
            # calculate the gradient of input with fixed fa tensor, rather than the "correct" model weight
            grad_input = grad_output.mm(weight_fa)
        if context.needs_input_grad[1]:
            # grad for weight with FA'ed grad_output from downstream layer
            # it is same with original linear function
            grad_weight = grad_output.t().mm(input)
        if bias is not None and context.needs_input_grad[3]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_weight_fa, grad_bias

class LinearFAModule(nn.Module):

    def __init__(self, input_features, output_features, bias=True):
        super(LinearFAModule, self).__init__()
        self.input_features = input_features
        self.output_features = output_features

        # weight and bias for forward pass
        # weight has transposed form; more efficient (so i heard) (transposed at forward pass)
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(output_features))
        else:
            self.register_parameter('bias', None)

        # fixed random weight and bias for FA backward pass
        # does not need gradient
        self.weight_fa = nn.Parameter(Variable(torch.FloatTensor(output_features, input_features), requires_grad=False))

        # weight initialization
        torch.nn.init.kaiming_uniform(self.weight)
        torch.nn.init.kaiming_uniform(self.weight_fa)
        torch.nn.init.constant(self.bias, 1)

    def forward(self, input):
        return LinearFAFunction.apply(input, self.weight, self.weight_fa, self.bias)

Output:
You just need to flatten your input before passing it to your model. Something like this:
# ...

# from [batch_size, 1, 28, 28] <- 4-D
# to   [batch_size, 1x28x28]   <- 2-D, as expected
flat_inputs = torch.flatten(inputs)

output_bp = model_bp(flat_inputs)
output_fa = model_fa(flat_inputs)

# ...


Instruction: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [256, 1, 3, 3], but got 3-dimensional input of size [64, 1, 786] instead

Input:
I'm trying to combine the CausalConv1d with Conv2d as the encoder of my VAE. But I got this error which is produced on Encoder part. The CausalConv1d is implemented by a nn.Conv1d network, So it should only have 3-dimensional weight, but why the error says expected 4-dimensional?  And I have another question, why I can't use a single int but only tuple in Pycharm when I set the "kernel_size", "stride" etc. parameters in a Convs layer? Although the official document said both int and tuple are valid. Here is the traceback:
Traceback (most recent call last):
  File "training.py", line 94, in <module>
    training(args)
  File "training.py", line 21, in training
    summary(model, torch.zeros(64, 1, 784), show_input=True, show_hierarchical=False)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\pytorch_model_summary\model_summary.py", line 118, in summary
    model(*inputs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "F:\VAE_reservoir_computing\VAE.py", line 34, in forward
    mu, log_var = self.encoder.forward(image)
  File "F:\VAE_reservoir_computing\CausalCnn_Reservoir.py", line 30, in forward
    x = self.conv1d(x.view(x.shape[0], 1, 784))
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\container.py", line 119, in forward
    input = module(input)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "F:\VAE_reservoir_computing\CausalConv1d.py", line 21, in forward
    conv1d_out = self.conv1d(x)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\conv.py", line 263, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Anaconda\envs\vae_reservior_computing\lib\site-packages\torch\nn\modules\conv.py", line 260, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [256, 1, 3, 3], but got 3-dimensional input of size [64, 1, 786] instead

Here is my code of the Encoder-Decoder part:
import torch
import torch.nn as nn
from CausalConv1d import CausalConv1d

class CausalReservoirEncoder(nn.Module):
    def __init__(self, in_channels, out_channels, num_filters, z_dim,  *args):
        super(CausalReservoirEncoder, self).__init__()
        self.num_filters = num_filters
        self.z_dim = z_dim
        hidden_filters = num_filters

        self.Conv1d = nn.Sequential(
            CausalConv1d(in_channels,out_channels,kernel_size=(3,3),dilation=1,A=False),
            nn.LeakyReLU()
        )
        for p in self.parameters():
            p.requires_grad = False

        self.encoder = nn.Sequential(
            nn.Conv2d(out_channels, self.num_filters, kernel_size=(4, 4), padding=(1, 1), stride=(2, 2)), # 28x28 -> 14x14
            nn.LeakyReLU(),
            nn.Conv2d(hidden_filters, 2 * hidden_filters, kernel_size=(4, 4), padding=(1, 1), stride=(2, 2)), # 14x14 -> 7x7
            nn.LeakyReLU(),
            nn.Flatten(),
            nn.Linear(2*hidden_filters*7*7, self.z_dim)

        )

    def forward(self, x):
        x = self.Conv1d(x.view(x.shape[0], 1, 784))
        h_e = self.encoder(x.view(x.shape[0], -1, 28, 28))
        mu, log_var = torch.chunk(h_e, 2, dim=1)

        return mu, log_var

class CausalReservoirDecoder(nn.Module):
    def __init__(self, z_dim, out_channels, num_filters, **kwargs):
        super(CausalReservoirDecoder, self).__init__()
        self.z_dim = z_dim
        self.num_filters = num_filters
        hidden_filters = num_filters

        self.linear = nn.Sequential(
            nn.Linear(self.z_dim, 2*hidden_filters*7*7),
            nn.LeakyReLU()
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(2*hidden_filters, hidden_filters, kernel_size=(4,4), padding=(1,1), stride=(2,2)), #7x7 -> 14x14
            nn.LeakyReLU(),
            nn.ConvTranspose2d(hidden_filters, out_channels, kernel_size=(4,4), padding=(1,1), stride=(2,2)), # 14x14 -> 28x28
            nn.Sigmoid()
        )

    def forward(self, z):
        x = self.linear(z)
        x = x.reshape(x.shape[0], -1, 7, 7)
        x_recon = self.decoder(x)

        return x_recon

Here is the implementation of the CausalConv1d:
import torch.nn as nn
import torch.nn.functional as F

class CausalConv1d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, dilation, A=False, *args, **kwargs):
        super(CausalConv1d, self).__init__()
        self.kernel_size = kernel_size
        self.dilation = dilation
        self.A = A

        self.padding = (kernel_size[0] - 1) * dilation + A * 1

        self.conv1d = nn.Conv1d(in_channels,out_channels,self.kernel_size,stride=(1,1),padding=(0,0),dilation=dilation,**kwargs)

    def forward(self, x):
        x = F.pad(x, (self.padding, 0))
        conv1d_out = self.conv1d(x)
        if self.A:
            return conv1d_out[:,:,: -1]
        else:
            return conv1d_out

So anyone can give me some suggestions?

Output:
I know this may not be intuitive, but when you use a kernel_size with 2-dim (e.g., (3,3)), then your Conv1d has 4-dim weights. Therefore, to solve your issue, you must change from:
CausalConv1d(in_channels,out_channels,kernel_size=(3,3),dilation=1,A=False),

to:
CausalConv1d(in_channels, out_channels, kernel_size=3, dilation=1, A=False),


Instruction: How to reshape a tensor where one dimension is reduced to 1?

Input:
I'm trying to reshape a tensor using addition and squeeze/normalization from [30,50,32,64] to [30,50,32,1]. And when I use Tensorflow reshape, I'm getting the following error.
o = tf.reshape(o, shape=[30, 50, 32, 1])

ValueError: Cannot reshape a tensor with 3072000 elements to shape [30,50,32,1] (48000 elements) for 'discriminator/embedding/Reshape_4' (op: 'Reshape') with input shapes: [30,50,32,64], [4] and with input tensors computed as partial shapes: input[1] = [30,50,32,1].
Can someone suggest a way around it?

Output:
I think you are confusing reshape operation with reduction. First of all, reshape has nothing to do with normalisation. By definition, any reshape operation should preserve number and value of tensor's elements.
i.e. it's impossible to reshape 30x50x32x64 (3072000 elements) tensor to 30x50x32x1 tensor (48000 elements).
Check out the docs: https://www.tensorflow.org/api_docs/python/tf/reshape
So if you want to apply reduction operation on last tensor's dimension you can do it like this:
In [1]: import tensorflow as tf
In [2]: data = tf.random.normal(shape=[30, 50, 32, 64])
In [3]: tf.math.reduce_mean(data, axis=-1, keepdims=True).shape
Out[3]: TensorShape([30, 50, 32, 1])

Squeezing/unsqueezing can be viewed as a special case of reshape operation. Where you are removing or adding single tone dimensions.
# squeeze example
In [4]: tf.squeeze(tf.math.reduce_mean(data, axis=-1, keepdims=True)).shape
Out[4]: TensorShape([30, 50, 32])

# unsqueeze example
In [5]: tf.expand_dims(tf.math.reduce_mean(data, axis=-1, keepdims=True), axis=0).shape
Out[5]: TensorShape([1, 30, 50, 32, 1])


Instruction: how to store model.state_dict() in a temp var for later use?

Input:
I tried to store the state dict of my model in a variable temporarily and wanted to restore it to my model later, but the content of this variable changed automatically as the model updated.
There is a minimal example:
import torch as t
import torch.nn as nn
from torch.optim import Adam

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(3, 2)

    def forward(self, x):
        return self.fc(x)

net = Net()
loss_fc = nn.MSELoss()
optimizer = Adam(net.parameters())

weights = net.state_dict()
print(weights)

x = t.rand((5, 3))
y = t.rand((5, 2))
loss = loss_fc(net(x), y)

optimizer.zero_grad()
loss.backward()
optimizer.step()

print(weights)

I thought the two outputs would be the same, but I got (outputs may change due to random initialization)
OrderedDict([('fc.weight', tensor([[-0.5557,  0.0544, -0.2277],
        [-0.0793,  0.4334, -0.1548]])), ('fc.bias', tensor([-0.2204,  0.2846]))])
OrderedDict([('fc.weight', tensor([[-0.5547,  0.0554, -0.2267],
        [-0.0783,  0.4344, -0.1538]])), ('fc.bias', tensor([-0.2194,  0.2856]))])

The content of weights changed, which is so weird.
I also tried .copy() and t.no_grad() as following, but they did not help.
with t.no_grad():
    weights = net.state_dict().copy()

Yes, I know that I can save state dict using t.save(), but I just want to figure out what happened in the previous example.
I'm using Python 3.8.5 and Pytorch 1.8.1
Thanks for any help.

Output:
That's how OrderedDict works. Here's a simpler example:
from collections import OrderedDict

# a mutable variable
l = [1,2,3]

# an OrderedDict with an entry pointing to that mutable variable
x = OrderedDict([("a", l)])

# if you change the list
l[1] = 20

# the change is reflected in the OrderedDict
print(x)
# >> OrderedDict([('a', [1, 20, 3])])

If you want to avoid that, you'll have to do a deepcopy rather than a shallow copy:
from copy import deepcopy
x2 = deepcopy(x)

print(x2)
# >> OrderedDict([('a', [1, 20, 3])])

# now, if you change the list
l[2] = 30

# you do not change your copy
print(x2)
# >> OrderedDict([('a', [1, 20, 3])])

# but you keep changing the original dict
print(x)
# >> OrderedDict([('a', [1, 20, 30])])

As Tensor is also mutable, the same behaviour is expected in your case. Therefore, you can use:
from copy import deepcopy

weights = deepcopy(net.state_dict())


Instruction: How to use PyTorch to softmax only the upper triangular elements of a matrix?

Input:
Given input like:
tensor([[[1.9392, -1.9266,  0.9664],
         [0.0000, -1.9266,  0.9664],
         [0.0000, -0.0000,  0.9664]]])

My desired output is:
tensor([[[0.4596,  0.0096, 0.1737],
         [0.0000,  0.0096, 0.1737],
         [0.0000, -0.0000, 0.1737]]])

I.e. just calculating the function over the upper triangular elements.

Output:
You can access the upper triangular elements with torch.triu_indices:
t = tensor([[1.9392, -1.9266, 0.9664],
            [0.0000, -1.9266, 0.9664],
            [0.0000, -0.0000, 0.9664]])

idx = torch.triu_indices(*t.shape)
soft = F.softmax(t[idx[0], idx[1]], dim=0)

If you want to reassign the values as in your desired output:
>>> t[idx[0], idx[1]] = soft
>>> t

tensor([[0.4596,  0.0096, 0.1737],
        [0.0000,  0.0096, 0.1737],
        [0.0000, -0.0000, 0.1737]])


Instruction: The .data attribute for a dataset in torchvision.datasets doesn't work for ImageFolder?

Input:
I can access the training data set of an MNIST object like so:
from torchvision.datasets import MNIST
trainset = MNIST(root='.', train=True, download=True)
print(trainset.data)

However, I'm not able to do the same using ImageFolder. Anybody know the equivalent for ImageFolder?
AttributeError: 'ImageFolder' object has no attribute 'data'

Output:
There isn't any.
Nowadays, everyone assumes that MNIST fits in memory, thus it is preloaded to the data attribute. However, this is usually not possible for ImageDatasets. Therefore, the images are loaded on-the-fly which means, no data attribute for them. You can access the image paths and labels using the self.imgs or self.samples.


Instruction: UnboundLocalError: local variable 'labels' referenced before assignment in PyTorch while doing X-Ray classification

Input:
I am trying to replicate this Kaggle notebook, using my data. I am not posting the whole code here, because it is huge.
Received an error, which cannot fix myself
----> 2 model, history = run_fold(model, criterion, optimizer, scheduler, device=CFG.device, fold=0, num_epochs=CFG.num_epochs)

1 frames
<ipython-input-63-1ac8b53f265b> in train_model(model, criterion, optimizer, scheduler, num_epochs,
dataloaders, dataset_sizes, device, fold)
 23             for inputs, classes in dataloaders[phase]:
 24                 inputs = inputs.to(CFG.device)
---> 25                 labels = labels.to(CFG.device)
 26
 27                 # forward

UnboundLocalError: local variable 'labels' referenced before assignment

My data structure looks the same as the notebook

I have checked similar questions, there are several of them, but this is more tech question how to fix this thing. Maybe something wrong with index? Appreciate any tips.

Output:
There is a typo in the for loop. It should be labels instead of classes:
# ...
for inputs, labels in dataloaders[phase]:
    inputs = inputs.to(CFG.device)
    labels = labels.to(CFG.device)
    # ...


Instruction: model.train() and model.eval() causing nan values

Input:
Hey so I am trying my hand at image classification/transfer learning using the monkey species dataset and the resnet50 with a modified final fc layer to predict just the 10 classes. Eveything is working until I use model.train() and model.eval() then after the first epoch it starts to return nans and the accuracy drops off as you'll see below. I'm curious why is this only when switching to train/eval....?
First I import the model and attach the classifier and freeze the parameters
%%capture
resnet = models.resnet50(pretrained=True)

for param in resnet.parameters():
  param.required_grad = False

in_features = resnet.fc.in_features

# Build custom classifier
classifier = nn.Sequential(OrderedDict([('fc1', nn.Linear(in_features, 512)),
                                        ('relu', nn.ReLU()),
                                        ('drop', nn.Dropout(0.05)),
                                        ('fc2', nn.Linear(512, 10)),
                                        ]))

# ('output', nn.LogSoftmax(dim=1))
resnet.classifier = classifier

resnet.to(device)

Then setting my loss func, optimizer, and shceduler
# Step : Define criterion and optimizer
criterion = nn.CrossEntropyLoss()
# pass the optimizer to the appended classifier layer
optimizer = torch.optim.SGD(resnet.parameters(), lr=0.01)
# Scheduler
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[10], gamma=0.05)

Then setting the training and validation loops
epochs = 20

tr_losses = []
avg_epoch_tr_loss = []
tr_accuracy = []

val_losses = []
avg_epoch_val_loss = []
val_accuracy = []
val_loss_min = np.Inf

resnet.train()
for epoch in range(epochs):
  for i, batch in enumerate(train_loader):
    # Pull the data and labels from the batch
    data, label = batch
    # If available push data and label to GPU
    if train_on_gpu:
      data, label = data.to(device), label.to(device)
    # Compute the logit
    logit = resnet(data)
    # Compte loss
    loss = criterion(logit, label)
    # Clearing the gradient
    resnet.zero_grad()
    # Backpropagate the gradients (accumulte the partial derivatives of loss)
    loss.backward()
    # Apply the updates to the optimizer step in the opposite direction to the gradient
    optimizer.step()
    # Store the losses of each batch
    # loss.item() seperates the loss from comp graph
    tr_losses.append(loss.item())
    # Detach and store the average accuracy of each batch
    tr_accuracy.append(label.eq(logit.argmax(dim=1)).float().mean())
    # Print the rolling batch training loss every 20 batches
    if i % 40 == 0 and not i == 1:
      print(f'Batch No: {i} \tAverage Training Batch Loss: {torch.tensor(tr_losses).mean():.2f}')
  # Print the average loss for each epoch
  print(f'\nEpoch No: {epoch + 1},Training Loss: {torch.tensor(tr_losses).mean():.2f}')
  # Print the average accuracy for each epoch
  print(f'Epoch No: {epoch + 1}, Training Accuracy: {torch.tensor(tr_accuracy).mean():.2f}\n')
  # Store the avg epoch loss for plotting
  avg_epoch_tr_loss.append(torch.tensor(tr_losses).mean())

  resnet.eval()
  for i, batch in enumerate(val_loader):
    # Pull the data and labels from the batch
    data, label = batch
    # If available push data and label to GPU
    if train_on_gpu:
      data, label = data.to(device), label.to(device)
    # Compute the logits without computing the gradients
    with torch.no_grad():
      logit = resnet(data)
    # Compte loss
    loss = criterion(logit, label)
    # Store test loss
    val_losses.append(loss.item())
    # Store the accuracy for each batch
    val_accuracy.append(label.eq(logit.argmax(dim=1)).float().mean())
    if i % 20 == 0 and not i == 1:
      print(f'Batch No: {i+1} \tAverage Val Batch Loss: {torch.tensor(val_losses).mean():.2f}')
  # Print the average loss for each epoch
  print(f'\nEpoch No: {epoch + 1}, Epoch Val Loss: {torch.tensor(val_losses).mean():.2f}')
  # Print the average accuracy for each epoch
  print(f'Epoch No: {epoch + 1}, Epoch Val Accuracy: {torch.tensor(val_accuracy).mean():.2f}\n')
  # Store the avg epoch loss for plotting
  avg_epoch_val_loss.append(torch.tensor(val_losses).mean())

  # Checpoininting the model using val loss threshold
  if torch.tensor(val_losses).float().mean() <= val_loss_min:
    print("Epoch Val Loss Decreased... Saving model")
    # save current model
    torch.save(resnet.state_dict(), '/content/drive/MyDrive/1. Full Projects/Intel Image Classification/model_state.pt')
    val_loss_min = torch.tensor(val_losses).mean()
  # Step the scheduler for the next epoch
  scheduler.step()
  # Print the updated learning rate
  print('Learning Rate Set To: {:.5f}'.format(optimizer.state_dict()['param_groups'][0]['lr']),'\n')

The model starts to train but then slowly becomes nan values
Batch No: 0     Average Training Batch Loss: 9.51
Batch No: 40    Average Training Batch Loss: 1.71
Batch No: 80    Average Training Batch Loss: 1.15
Batch No: 120   Average Training Batch Loss: 0.94

Epoch No: 1,Training Loss: 0.83
Epoch No: 1, Training Accuracy: 0.78

Batch No: 1     Average Val Batch Loss: 0.39
Batch No: 21    Average Val Batch Loss: 0.56
Batch No: 41    Average Val Batch Loss: 0.54
Batch No: 61    Average Val Batch Loss: 0.54

Epoch No: 1, Epoch Val Loss: 0.55
Epoch No: 1, Epoch Val Accuracy: 0.81

Epoch Val Loss Decreased... Saving model
Learning Rate Set To: 0.01000

Batch No: 0     Average Training Batch Loss: 0.83
Batch No: 40    Average Training Batch Loss: nan
Batch No: 80    Average Training Batch Loss: nan

Output:
I see that resnet.zero_grad() is after logit = resnet(data), which causes the gradient to explode in your case.
Please do it as below:
# Clearing the gradient
optimizer.zero_grad()
logit = resnet(data)

# Compute loss
loss = criterion(logit, label)


Instruction: What is the default settings (e.g. hyperparameters) for MatLab's feedforwardnet?

Input:
My code is very simple for one layer of 20 neurons:
FFNN = feedforwardnet(20);
FFNN_trained = train(FFNN,x,y);

This can result in a very good performance in a few hundreds of epochs. I want to reproduce it in Pytorch, so I need to know the details, e.g. learning rate, activation function, optimizer, batch size, when to stop, etc. Also, the data splitting for training/validation/testing seems to be random in feedforwardnet.
Where can I find these details for feedforwardnet? How to specify the training/validation/testing in feedforwardnet?
Thank you for the help. I realise that the levenberg-marquardt method is not available in Pytorch.

Output:
According to the documentations for feedforwardnet, the default setting for this function is to train with the Levenberg-Marquardt backpropagation, aka. damped least-squares -- feedforwardnet(20, 'trainlm') option.
As for the data split, the default seems to be a random 0.7-0.15-0.15 train-validation-test split, using the dividerand function.

From the trainlm page:
trainlm is a network training function that updates weight and bias values according to Levenberg-Marquardt optimization.
trainlm is often the fastest backpropagation algorithm in the toolbox, and is highly recommended as a first-choice supervised algorithm, although it does require more memory than other algorithms.
Training occurs according to trainlm training parameters, shown here with their default values:

net.trainParam.epochs — Maximum number of epochs to train. The default value is 1000.
net.trainParam.goal — Performance goal. The default value is 0.
net.trainParam.max_fail — Maximum validation failures. The default value is 6.
net.trainParam.min_grad — Minimum performance gradient. The default value is 1e-7.
net.trainParam.mu — Initial mu. The default value is 0.001.
net.trainParam.mu_dec — Decrease factor for mu. The default value is 0.1.
net.trainParam.mu_inc — Increase factor for mu. The default value is 10.
net.trainParam.mu_max — Maximum value for mu. The default value is 1e10.
net.trainParam.show — Epochs between displays (NaN for no displays). The default value is 25.
net.trainParam.showCommandLine — Generate command-line output. The default value is false.
net.trainParam.showWindow — Show training GUI. The default value is true.
net.trainParam.time — Maximum time to train in seconds. The default value is inf.

Validation vectors are used to stop training early if the network performance on the validation vectors fails to improve or remains the same for max_fail epochs in a row. Test vectors are used as a further check that the network is generalizing well, but do not have any effect on training.

From Divide Data for Optimal Neural Network Training:
MATLAB provides 4 built-in functions for splitting data:

Divide the data randomly (default) - dividerand
Divide the data into contiguous blocks - divideblock
Divide the data using an interleaved selection - divideint
Divide the data by index - divideind

You can access or change the division function for your network with this property:
net.divideFcn
Each of the division functions takes parameters that customize its behavior. These values are stored and can be changed with the following network property:
net.divideParam


Instruction: Is batch normalisation a nonlinear operation?

Input:
Can batch normalisation be considered a nonlinear operation like relu or sigmoid?
I have come across a resnet block like:
: IBasicBlock(
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (prelu): PReLU(num_parameters=512)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )

so I wonder if batchnormalisation can act as non linear operation

Output: Batch Normalization is non-linear in the data points xi on a per batch training basis, but is linear during inference in the scaling parameters (as these are fixed).


Instruction: PyTorch DataLoader uses same random seed for batches run in parallel

Input:
There is a bug in PyTorch/Numpy where when loading batches in parallel with a DataLoader (i.e. setting num_workers > 1), the same NumPy random seed is used for each worker, resulting in any random functions applied being identical across parallelized batches.
Minimal example:
import numpy as np
from torch.utils.data import Dataset, DataLoader

class RandomDataset(Dataset):
    def __getitem__(self, index):
        return np.random.randint(0, 1000, 2)

    def __len__(self):
        return 9

dataset = RandomDataset()
dataloader = DataLoader(dataset, batch_size=1, num_workers=3)

for batch in dataloader:
    print(batch)

As you can see, for each parallelized set of batches (3), the results are the same:
# First 3 batches
tensor([[891, 674]])
tensor([[891, 674]])
tensor([[891, 674]])
# Second 3 batches
tensor([[545, 977]])
tensor([[545, 977]])
tensor([[545, 977]])
# Third 3 batches
tensor([[880, 688]])
tensor([[880, 688]])
tensor([[880, 688]])

What is the recommended/most elegant way to fix this? i.e. have each batch produce a different randomization, irrespective of the number of workers.

Output:
It seems this works, at least in Colab:
dataloader = DataLoader(dataset, batch_size=1, num_workers=3,
    worker_init_fn = lambda id: np.random.seed(id) )

EDIT:

it produces identical output (i.e. the same problem) when iterated over epochs. – iacob

Best fix I have found so far:
...
dataloader = DataLoader(ds, num_workers= num_w,
           worker_init_fn = lambda id: np.random.seed(id + epoch * num_w ))

for epoch in range ( 2 ):
    for batch in dataloader:
        print(batch)
    print()

Still can't suggest closed form, thing depends on a var (epoch) then called. Ideally It must be something like worker_init_fn = lambda id: np.random.seed(id + EAGER_EVAL(np.random.randint(10000) ) where EAGER_EVAL evaluate seed on loader construction, before lambda is passed as parameter. Is it possible in python, I wonder.


Instruction: PyTorch gradient doesn't flow through a clone of a tensor

Input:
I'm trying to have my model learn a certain function. I have parameters self.a, self.b, self.c that are trainable. I'm trying to force self.b to be in a certain range by using tanh. However, when I run the code it appears as the gradient is flowing through the original parameter (self.b), but not through the clone (self.b_opt)
import torch
import torch.nn as nn
import torch.optim as optim

class model(nn.Module):
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))

            self.b_opt = torch.tanh(self.b.clone())

model_net = model()
#function to learn = 5 * (r > 2) * (3)
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

for epoch in range(10):
  for r in range(5):
    optimizer.zero_grad()
    loss = 5 * (r > 2) * (3) - model_net.a * torch.sigmoid((r - model_net.b_opt)) * (model_net.c)

    loss.backward(retain_graph=True)
    optimizer.step()
  #print(model_net.a)
  print(model_net.b)
  print(model_net.b_opt)
  #print(model_net.c)
  print()

>>> Parameter containing:
tensor([0.4298], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)

Parameter containing:
tensor([-0.0277], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)

Parameter containing:
tensor([-0.5007], requires_grad=True)
tensor([0.7229], grad_fn=<TanhBackward>)

Output:
gradient does actually flows through b_opt since it's the tensor that is involved in your loss function. However, it is not a leaf tensor (it is the result of operations on tensors, specifically a clone and a tanh, you can check with model_net.b_opt.is_leaf), which means it allows gradients to be propagated but does not accumulate them (b_opt.grad does not exist). You cannot update the value of b_opt through backprop.
However the fact that the value of b is updated proves that the gradient does flow ! What you actually need is to recompute b_opt each time b is updated, because here your b_opt is computed once and for all in the init function. That would be something like
...
loss.backward(retain_graph=True)
optimizer.step()
self.b_opt = torch.tanh(self.b.clone())
...

A much simpler way to do all this though would be to get rid of b_opt altogether, and replace your loss with
loss = 5 * (r > 2) * (3) - model_net.a * torch.sigmoid((r - tanh(model_net.b))) * (model_net.c)


Instruction: What is the difference between `on_validation_epoch_end` and `validation_epoch_end` in Pytorch-lightning?

Input:
Inside a LightningModule,Pycharm allows 2 auto complete methods:
class MyModel(LightningModule):

    def on_validation_epoch_end(self):

    def validation_epoch_end(self, outs):

with on_validation_epoch_end referenced in hooks.py
def on_validation_epoch_end(self) -> None:
    """
    Called in the validation loop at the very end of the epoch.
    """
    # do something when the epoch ends

and
validation_epoch_end called in evaluation_loop.py as eval_results = model.validation_epoch_end(eval_results) leading to __run_eval_epoch_end.

What is the purpose of each of those?
I can only assume one is deprecated. Could not find any relevant docs.

Output: Here is a pseudocode that shows when the hooks are called, and I think it makes it quite explicit that you are right : these two functions are redundant (literally called at the same place with the same arguments) and I would say that validation_epoch_end is the one to be considered deprecated here, since it's not mentioned in the doc whereas the hooks (of the form on_event_start/end) are extensively explained


Instruction: importing likelihoods fails

Input:
I am trying to get in touch with Gaussian Process Classification and try to reproduce the example from https://docs.gpytorch.ai/en/stable/examples/01_Exact_GPs/GP_Regression_on_Classification_Labels.html
Following their code, I want to import the Dirichlet Classification Likelihood by
from gpytorch.likelihoods import DirichletClassificationLikelihood

However, this seems not to work, and I get the Error:
ImportError: cannot import name 'DirichletClassificationLikelihood'
from 'gpytorch.likelihoods' (/Library/Frameworks/Python.framework/Versions/3.8/
lib/python3.8/site-packages/gpytorch/likelihoods/__init__.py)

also other likelihoods as FixedNoiseGaussianLikelihood does not work.
I tried to reinstall GPyTorch in Conda, which did not help.
but importing i.e GaussianLikelihood works without problems. Did somebody had similar problems, or knows how to solve this Error?
EDIT: It only does not work for Jupyter Notebook.
Best

Output:
Solved it.
The paths of Jupyter Notebook did not include the new updated library. After updating paths to the wanted folder by
import sys
sys.paths.append("....")

it worked nice.
Best


Instruction: How to create a PyTorch mutable tensor?

Input:
I'm trying to create a copy of a tensor that will change if the original changes.
r = torch.tensor(1.0, requires_grad=True)
p = r.clone()
print('before')
print(r)
print(p)
r = r*5
print('after')
print(r)
print(p)

>>>
before
tensor(1., requires_grad=True)
tensor(1.)
after
tensor(5., grad_fn=<MulBackward0>)
tensor(1.)

I tried with clone(), detach(), and even simply p=r, nothing worked.
Update.
Tried view:
r = torch.tensor(1.0, requires_grad=True)
p = r.view(1)
print('before')
print(r)
print(p)
r = r*5
print('after')
print(r)
print(p)

>>>before
tensor(1., requires_grad=True)
tensor([1.], grad_fn=<ViewBackward>)
after
tensor(5., grad_fn=<MulBackward0>)
tensor([1.], grad_fn=<ViewBackward>)

Output:
What your looking for a is a view(which is a shallow copy of the tensor), numpy also follows this as well, the below contains what you want
test = torch.tensor([100])
test_copy = test.view(1)

print(test, test_copy) # tensor([100]) tensor([100])
test[0] = 200
test, test_copy # (tensor([200]), tensor([200]))

Edit: Made some changes and found out the problem
r = torch.tensor(1.0, requires_grad=True) # Remove requires_grad if you really dont need it
p = r.view(1)
print('before')
print(r)
print(p)
with torch.no_grad(): # Had to use this if not wont be able to do math ops cause gradients are being tracked
  # Problem here was that you redeclared a new variable of r so you wiped out the previous reference to the tensor
  r.mul_(5) # Simple calculations
print('after')
print(r)
print(p)


Instruction: Differentiable convolution between two 1d signals in pytorch

Input:
i need to implement a convolution between a signal and a window in pytorch and i want it to be differentiable. Since i couldn't find an already existing function for tensors (i could only find the ones with learnable parameters) i wrote one myself but, i'm unable to make it work without breaking the computation graph. How could i do it? The function i made is:
def Convolve(a, b):
  conv=torch.zeros(a.shape[0], a.shape[1], requires_grad=True).clone()
  l=b.shape[0]
  r=int((l-1)/2)
  l2=a.shape[1]
  for x in range(a.shape[0]):#for evry signal
    for x2 in range(a.shape[1]):#for every time instant
      for x4 in range(l):#compute the convolution (for every window value)
        if (x2-r+x4<0 or x2-r+x4>=l2):#if the index is out of bonds the result is 0 (to avoid zero padding)
           conv[x][x2]+=0
        else:
          conv[x][x2-r+x4]+=a[x][x2-r+x4]*b[x4]#otherwise is window*signal
  return conv

Where 'a' is a two dimensional tensor (signal index, time) and 'b' is an Hann window.
The lenght of the window is odd.

Output:
it is (fortunately!) possible to achieve this with pytorch primitives. You are probably looking for functional conv1d. Below is how it works. I was not sure whether you wanted the derivative with respect to the input or the weights, so you have both, just keep the requires_grad that fits you needs :
import torch.nn.functional as F
# batch of 3 signals, each of length 11 with 1 channel
signal = torch.randn(3, 1, 11, requires_grad=True)
# convolution kernel of size 3, expecting 1 input channel and 1 output channel
kernel = torch.randn(1, 1, 3, requires_grad=True)
# convoluting signal with kernel and applying padding
output = F.conv1d(signal, kernel, stride=1, padding=1, bias=None)
# output.shape is (3, 1, 11)
# backpropagating some kind of loss through the computational graph
output.sum().backward()
print(kernel.grad)
>>> torch.tensor([[[...]]])


Instruction: Image Net Preprocessing using torch transforms

Input:
I am trying to recreate the data preprocessing on the ImageNet data set done in the original publication "Deep Residual Learning for Image Recognition". As said in their paper in section 3.4:
"Our implementation for ImageNet follows the practice in [21, 41].  The image is resized with its shorter side randomly sampled in[256,480]for scale augmentation [41].A 224×224 crop is randomly sampled from an image or its horizontal flip, with the per-pixel mean subtracted [21]. The standard color augmentation in [21] is used."
I have figured out the parts of randomly cropping the original image or the horizontal flip with a crop size of 224x224. The other two parts I have not. The other two parts are The image is resized with its shorter side randomly sampled in[256,480] for scale augmentation and the The standard color augmentation in [21] is used.
For the first one, I can't find a "random resize" function in torch transforms. The second, where its referencing [21], is (according to [21]) a "perform  PCA  on  the  set  of  RGB  pixel  values  throughout  theImageNet training set". Please refer to ImageNet Classification with Deep Convolutional Neural Networks in section "Data Augmentation" for the full explanation.
How would I recreate this type of preprocessing?

Output:
The first one needs 3 combined transforms, RandomChoice, Resize and RandomCrop.
transforms.Compose([transforms.RandomChoice([transforms.Resize(256),
                                             transforms.Resize(480)]),
                    transforms.RandomCrop(224)
                    ])

For the second one this is what you're looking for but officially Pytorch (and literally everybody else) simply uses this.
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

If you think that's too simple, the standard Tensorflow pre-processing is just
x /= 127.5
x -= 1.


Instruction: index 1 is out of bounds for dimension 0 with size 1

Input:
I am starting to learn about DQN, and I am trying to solve the FrozenLake-v0 problem from scratch by my self and using Pytorch so I will put the whole code since it's connected.
class LinearDeepQNetwork(nn.Module):
  def __init__(self,lr,n_action,input_dim):
    super(LinearDeepQNetwork,self).__init__()
    self.f1=nn.Linear(input_dim,128)
    self.f2=nn.Linear(128,n_action)
    self.optimizer=optim.Adam(self.parameters(),lr=lr)
    self.loss=nn.MSELoss()
    self.device=T.device('cuda' if T.cuda.is_available() else 'cpu')
    self.to(self.device)

  def forward(self,state):
    layer1=F.relu(self.f1(state))
    actions=self.f2(layer1)

    return actions

the second class is the agent, and the problem is in the learning function
nd class is the agent, and the problem is in the learning function
class Agent():
  def __init__(self,input_dim,n_action,lr,gamma=0.99,
               epslion=1.0,eps_dec=1e-5,eps_min=0.01):
    self.input_dim=input_dim
    self.n_action=n_action
    self.lr=lr
    self.gamma=gamma
    self.epslion=epslion
    self.eps_dec=eps_dec
    self.eps_min=eps_min
    self.action_space=[i for i in range(self.n_action)]

    self.Q=LinearDeepQNetwork(self.lr,self.n_action,self.input_dim)

  def choose_action(self,observation):
    if np.random.random()>self.epslion:
      #conveate the state into tensor
      state=T.tensor(observation).to(self.Q.device)
      actions=self.Q.forward(state)
      action=T.argmax(actions).item()
    else:
      action=np.random.choice(self.action_space)

    return action

  def decrement_epsilon(self):
    self.epslion=self.epslion-self.eps_dec \
                  if self.epslion > self.eps_min else self.eps_min

  def OH(self,x,l):
    x = T.LongTensor([[x]])
    one_hot = T.FloatTensor(1,l)
    return one_hot.zero_().scatter_(1,x,1)

  def learn(self,state,action,reward,state_):
    self.Q.optimizer.zero_grad()
    states=Variable(self.OH(state,16)).to(self.Q.device)
    actions=T.tensor(action).to(self.Q.device)
    rewards=T.tensor(reward).to(self.Q.device)
    state_s=Variable(self.OH(state_,16)).to(self.Q.device)

    q_pred=self.Q.forward(states)[actions]

    q_next=self.Q.forward(state_s).max()

    q_target=reward+self.gamma*q_next
    loss=self.Q.loss(q_target,q_pred).to(self.Q.device)
    loss.backward()
    self.Q.optimizer.step()
    self.decrement_epsilon()

now the problem when I run the following code it gives me an error in the learning phase, and it gives me this error index 1 is out of bounds for dimension 0 with size 1.
env=gym.make('FrozenLake-v0')
n_games=5000
scores=[]
eps_history=[]

agent=Agent(env.observation_space.n,env.action_space.n,0.0001)

for i in tqdm(range(n_games)):
  score=0
  done=False
  obs=env.reset()

  while not done:
    action=agent.choose_action(obs)
    obs_,reward,done,_=env.step(action)
    score+=reward

    agent.learn(obs,action,reward,obs_)
    obs=obs_
  scores.append(score)
  eps_history.append(agent.epslion)
  if i % 100 ==0:
    avg_score=np.mean(scores[-100:])
    print(f'score={score}  avg_score={avg_score} epsilon={agent.epslion} i={i}')

I think the problem is in the shape of the values between the NN and the agent class, but I can't figure out the problem.
Error traceback:
IndexError                                Traceback (most recent call last)
<ipython-input-10-2e279f658721> in <module>()
     17     score+=reward
     18
---> 19     agent.learn(obs,action,reward,obs_)
     20     obs=obs_
     21   scores.append(score)

<ipython-input-8-5359b19ec4fa> in learn(self, state, action, reward, state_)
     39     state_s=Variable(self.OH(state_,16)).to(self.Q.device)
     40
---> 41     q_pred=self.Q.forward(states)[actions]
     42
     43     q_next=self.Q.forward(state_s).max()

IndexError: index 1 is out of bounds for dimension 0 with size 1

Output:
since you are calling a tensor that contains a matrix, you need to specify which indices your calling in your case just adding [0] to the forward statement will solve the problem and in the [actions], replace it with [actions.item()]
self.Q.forward(states)[0][actions.item()]