Instruction: why is my loss function only returning NaN values?

Input:
below is my code
import numpy as np
import torch
from torch.utils import data
import torch.nn as nn
import pandas as pd

# PREPPING DATA FROM CSV FILE
csvFile = pd.read_csv('/Users/ericbeep999/Desktop/Web Development/Projects/Python/pytorch/3. Linear Regression/weather.csv')
labels, features = csvFile.iloc[:, 4], csvFile.iloc[:, 5]
#labels - min temp
#features - max temp
labels = torch.tensor(labels, dtype=torch.float32).reshape(-1, 1)
features = torch.tensor(features, dtype=torch.float32).reshape(-1,1)

# READING DATASET
def load_array(data_arrays, batch_size, is_train = True):
    dataset = data.TensorDataset(*data_arrays)
    return data.DataLoader(dataset, batch_size, shuffle= is_train)
batch_size = 20
data_set = load_array((features, labels), batch_size)

#DEFININING MODEL AND PARAMETERS
model = nn.Sequential(nn.Linear(1, 1))
model[0].weight.data.normal_(0, 0.1)
model[0].bias.data.fill_(0)

#DEFINING LOSS FUNCTION AND OPTIMIZATION ALGORITHMN

lossFunc = nn.MSELoss()
learning_rate = 0.01
gradient = torch.optim.SGD(model.parameters(), learning_rate)

#TRAINING MODEL

num_epochs = 100
for epoch in range(num_epochs):
    for X, Y in data_set:

        loss = lossFunc(model(X), Y)
        gradient.zero_grad()
        loss.backward()
        gradient.step()
    loss = lossFunc(model(features), labels)
    print(f'epoch: {epoch + 1}, loss: {loss}')

print(f"{model[0].weight.data}, {model[0].bias.data}")

the csv file I am importing the data from can be found at https://www.kaggle.com/datasets/smid80/weatherww2?datasetId=3759&searchQuery=pytorch
My labels are the min temperature and my features are the max temperature
whenever I run the code, the only thing that prints is
epoch: 1, loss: nan
epoch: 2, loss: nan
epoch: 3, loss: nan
epoch: 4, loss: nan
epoch: 5, loss: nan
epoch: 6, loss: nan
epoch: 7, loss: nan
epoch: 8, loss: nan
epoch: 9, loss: nan
epoch: 10, loss: nan

i don't really understand why it is only printing NaN

Output: I changed your learning rate to 0.001 and it runs without giving NaNs (albeit not learning anything since predicting min temperature from max temperature in that data may not be easily learned). My guess is the issue is with the scale of your input/output data, i.e. they're in the range of something like 0-40 which isn't great for neural networks - it can cause the learning to go unstable more easily. I would suggest you first scale your inputs/outputs to be in a range of [0, 1] or [-1, 1]. I'll direct you to this blog for details on achieving that, they also discuss in more detail why scaling the data is important for learning with neural networks.


Instruction: How to set a 3-d tensor to 0 according to the value of a 2-d tensor

Input:
Suppose I have a 3-d tensor P shaped (B, N, d) and a 2-d tensor Q shaped (B,N), where values in Q are smaller than d. I want to set some values in P to 0 using the indices from Q:
For instance,
P = torch.randn(2,3,6)
Q = torch.tensor([[0,3,4], [2,1,3]])

How can I set P[0,0,0]=0; P[0,1,3]=0; P[0,2,4]=0; P[1,0,2]=0; P[1,1,1]=0; P[1,2,3]=0 using Q and keep other values in P unchanged?

Output:
What you are looking to do is:
P[i, j, Q[i, j]] = 0

This is the perfect use case for torch.Tensor.scatter, which has the effect of placing an arbitrary value at designated positions.
We first need the input and indexer to have matching shapes:
>>> Q_ = Q[...,None].expand_as(P)

And apply the scatter function on dim=2, with the hidden value argument...
P.scatter(dim=2, index=Q_, value=0)
tensor([[[0, 2, 5, 4, 3, 7],
         [4, 1, 5, 0, 3, 8],
         [7, 8, 4, 3, 0, 1]],

        [[7, 1, 0, 1, 1, 3],
         [5, 0, 5, 3, 0, 5],
         [6, 3, 5, 0, 7, 2]]])


Instruction: PyTorch - How to specify an input layer? Is it included by default?

Input:
I am working on a Reinforcement Learning problem in StableBaselines3, but I don't think that really matters for this question. SB3 is based on PyTorch.
I have 101 input features, and even though I designed a neural architecture with the first layer having only 64 nodes, the network still works. Below is a screenshot of my model architecture:

I am concerned because I thought that the first layer of the neural network needed to have a number of nodes equal to the number of input features.
Does PyTorch include an input layer by default, and doesn't display it? If so, how can I know and control what the activation functions etc. are for the input layer?
EDIT:
Here are my imports and basic code, in response to Michael's comment.
import gym
from gym import Env
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from gym import spaces
from gym.utils import seeding
from stable_baselines3.common.vec_env import DummyVecEnv, SubprocVecEnv
from stable_baselines3.common.utils import set_random_seed
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import PPO
import math
import random
import torch as th

from sb3_contrib.common.maskable.policies import MaskableActorCriticPolicy
from sb3_contrib.common.wrappers import ActionMasker
from sb3_contrib.ppo_mask import MaskablePPO
from sb3_contrib.common.envs import InvalidActionEnvDiscrete
from sb3_contrib.common.maskable.evaluation import evaluate_policy
from sb3_contrib.common.maskable.utils import get_action_masks

env = MyCustomEnv(....)
env = ActionMasker(env, mask_fn)  # Wrap to enable masking

# Defining custom neural network architecture
mynetwork = dict(activation_fn=th.nn.LeakyReLU,
                     net_arch=[dict(pi=[64, 64], vf=[64, 64])])

# Maskable PPO behaves just like regular PPO
model = MaskablePPO(MaskableActorCriticPolicy, env, verbose=1, learning_rate=0.0005, gamma=0.975, seed=10, batch_size=256, clip_range=0.2,
                    tensorboard_log="./log1/", policy_kwargs=mynetwork)
# To get the screenshot I gave
print(model.policy)

Output:
We can do a little bit of digging inside Stable Baselines' source code.
Looking inside the MaskableActorCriticPolicy, we can see it builds a MLP extractor by initializing an instance of MlpExtractor and creating the policy_net sub-network here whose layers are defined in this loop. Ultimately the layers feature sizes are dictated by the "pi" and "vf" lists in side of net_arch (see here).
If you trace back this you will notice those parameters can be modified as the net_arch argument on MaskableActorCriticPolicy header.


Instruction: Creating an image grid of tensors out of a batch using matplotlib/pytorch

Input:
I am trying to create a grid of images (e.g. 3 by 3) from a batch of tensors that will be fed into a GAN through a data loader in the next step. With the below code I was able to transform the tensors into images that are displayed in a grid in the right position. The problem is, that they are all displayed in a separate grid as shown here: Figure 1 Figure 2    Figure 5. How can put them in one grid and just get one figure returned with all 9 images?? Maybe I am making it too complicated. :D In the end tensors out of the real_samples have to be transformed and put into a grid.
real_samples = next(iter(train_loader))
for i in range(9):
    plt.figure(figsize=(9, 9))
    plt.subplot(330 + i + 1)
    plt.imshow(np.transpose(vutils.make_grid(real_samples[i].to(device)
                      [:40], padding=1, normalize=True).cpu(),(1,2,0)))
    plt.show()

Output:
And here is how to display a variable number of wonderful CryptoPunks using matplotlib XD:
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import ImageGrid
import matplotlib.image as mpimg

row_count = 3
col_count = 3

cryptopunks = [
    mpimg.imread(f"cryptopunks/{i}.png") for i in range(row_count * col_count)
]

fig = plt.figure(figsize=(8., 8.))
grid = ImageGrid(fig, 111, nrows_ncols=(row_count, col_count), axes_pad=0.1)

for ax, im in zip(grid, cryptopunks):
    ax.imshow(im)

plt.show()

Please note that the code allows you to generate all the images you want, not only 3 times 3. I have a folder called cryptopunks with a lot of images called #.png (e.g., 1.png, ..., 34.png, ...). Just change the row_count and col_count variable values. For instance, for row_count=6 and col_count=8 you get:

If your image files do not have that naming pattern above (i.e., just random names), just replace the first lines with the following ones:
import os
from pathlib import Path

import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import ImageGrid
import matplotlib.image as mpimg

for root, _, filenames in os.walk("cryptopunks/"):
    cryptopunks = [
        mpimg.imread(Path(root, filename)) for filename in filenames
    ]

row_count = 3
col_count = 3

# Here same code as above.

(I have downloaded the CryptoPunks dataset from Kaggle.)


Instruction: my nn.sigmoid() gradient different with my manual calculation

Input:
so im doing a manual calculation of lstm backpropagation in excel and want to compare it to my code, but im having trouble with the gradient of sigmoid at the pytorch. :
the output here:
tensor([[0.8762]], grad_fn=<SigmoidBackward>)
tensor([-0.1238])
epoch:   0 loss: 0.13214068

so the first line is the sigmoid value and the second line is the gradient of sigmoid value. why the value of sigmoid gradient is -0.1238 while the formula of sigmoid gradient are σ(x)⋅(1−σ(x). if i calculate the sigmoid gradient manually the value is 0.10845, but in the code the sigmoid gradient is -0.1238 .is the formula for the sigmoid gradient in pytorch wrong?

Output:
I can't reproduce your error. For one thing, your value of 0.10845 is not correct: remember that it might be computed that way (i.e. z * (1 - z)) because you expect z to be logistic(z) in your implementation. But, in any case, the value I compute agrees with what PyTorch produces.
Here's the logistic function:
import numpy as np

def logistic(z, derivative=False):
    if not derivative:
        return 1 / (1 + np.exp(-z))
    else:
        return logistic(z) * (1 - logistic(z))

logistic(0.8762, derivative=True)

This produces 0.20754992931590668.
Now with PyTorch:
import torch

t = torch.Tensor([0.8762])
t.requires_grad = True
torch.sigmoid(t).backward()
t.grad

This produces tensor([0.2075]).


Instruction: GAN LOSS of Generator and Discriminator Lowest at First Epoch - Is that normal?

Input:
I am trying to train a simple GAN and I noticed that the loss for the generator and discriminator is the lowest in the first epoch? How can that be? Did I miss something?
Below you find the plot of the Loss over the iterations:
Here is the code I was using:
I adapted the code according to your suggest @emrejik. It doesn't seem to have changed much though. I couldn't work with torch.ones() at the suggested lines as I was receiving this error message: "argument size (position 1) must be tuple of ints, not Tensor". Any idea how come?
I ran through 10 epochs and this came out now:
from glob import glob
import sys
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
import torchvision.utils as vutils
import torchvision
from torch.utils.tensorboard import SummaryWriter
from torchinfo import summary
from mpl_toolkits.axes_grid1 import ImageGrid
from skimage import io
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import numpy as np
from tqdm import trange

manual_seed = 999

path = 'Punks'
image_paths = glob(path + '/*.png')
img_size = 28
batch_size = 32

device = 'cuda' if torch.cuda.is_available() else 'cpu'

transform = transforms.Compose(
    [
        transforms.ToPILImage(),
        transforms.Resize(img_size),
        transforms.CenterCrop(img_size),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
    ]
)

class ImageDataset(Dataset):
    def __init__(self, paths, transform):
        self.paths = paths
        self.transform = transform

    def __len__(self):
        return len(self.paths)

    def __getitem__(self, index):
        image_path = self.paths[index]
        image = io.imread(image_path)

        if self.transform:
            image_tensor = self.transform(image)

        return image_tensor

if __name__ == '__main__':

    dataset = ImageDataset(image_paths, transform)

    train_loader = DataLoader(
        dataset, batch_size=batch_size, num_workers=2, shuffle=True)

    class Discriminator(nn.Module):
        def __init__(self):
            super().__init__()
            self.model = nn.Sequential(
                nn.Linear(784*3, 2048),
                nn.ReLU(),
                nn.Dropout(0.3),
                nn.Linear(2048, 1024),
                nn.ReLU(),
                nn.Dropout(0.3),
                nn.Linear(1024, 512),
                nn.ReLU(),
                nn.Dropout(0.3),
                nn.Linear(512, 256),
                nn.ReLU(),
                nn.Dropout(0.3),
                nn.Linear(256, 1),
                nn.Sigmoid(),
            )

        def forward(self, x):
            x = x.view(x.size(0), 784*3)
            output = self.model(x)
            return output

    discriminator = Discriminator().to(device=device)

    class Generator(nn.Module):
        def __init__(self):
            super().__init__()
            self.model = nn.Sequential(
                nn.Linear(100, 256),
                nn.ReLU(),
                nn.Linear(256, 512),
                nn.ReLU(),
                nn.Linear(512, 1024),
                nn.ReLU(),
                nn.Linear(1024, 2048),
                nn.ReLU(),
                nn.Linear(2048, 784*3),
                nn.Tanh(),
            )

        def forward(self, x):
            output = self.model(x)
            output = output.view(x.size(0), 3, 28, 28)
            return output

    generator = Generator().to(device=device)

    lr = 0.0001
    num_epochs = 10
    loss_function = nn.BCELoss()
    #noise = torch.randn(batch_size, 100, device=device)

    optimizer_discriminator = torch.optim.Adam(
        discriminator.parameters(), lr=lr)
    optimizer_generator = torch.optim.Adam(generator.parameters(), lr=lr)

    model = Discriminator().to(device=device)
    summary(model, input_size=(batch_size, 3, 28, 28))

    model = Generator().to(device=device)
    summary(model, input_size=(batch_size, 100))

    image_list = []
    Dis_losses = []
    Gen_losses = []
    iters = 0
    epochs = 0

    for epoch in trange((num_epochs), bar_format='{desc:<5.5}{percentage:3.0f}%|{bar:120}{r_bar}\n'):
        for n, real_samples in enumerate(train_loader):

            batch_size = len(real_samples)
            real_samples = real_samples.to(device=device)
            real_samples_labels = torch.ones((batch_size, 1)).to(device=device)

            latent_space_samples = torch.randn(
                (batch_size, 100)).to(device=device)

            fake_samples = generator(latent_space_samples)
            fake_samples_labels = torch.zeros(
                (batch_size, 1)).to(device=device)

            discriminator.zero_grad()

            output_discriminator_real = discriminator(real_samples)
            loss_discriminator_real = loss_function(
                output_discriminator_real, real_samples_labels)

            output_discriminator_fake = discriminator(fake_samples)
            loss_discriminator_fake = loss_function(
                output_discriminator_fake, fake_samples_labels)

            loss_discriminator = (
                loss_discriminator_real + loss_discriminator_fake)/2

            loss_discriminator.backward()
            optimizer_discriminator.step()

            latent_space_samples = torch.randn(
                (batch_size, 100)).to(device=device)

            generator.zero_grad()

            fake_samples = generator(latent_space_samples)
            output_discriminator_fake = discriminator(fake_samples)
            loss_generator = loss_function(
                output_discriminator_fake, real_samples_labels)

            loss_generator.backward()
            optimizer_generator.step()

            image_list.append(vutils.make_grid(
                fake_samples_labels, padding=2, normalize=True))
            Dis_losses.append(loss_discriminator.item())
            Gen_losses.append(loss_generator.item())
            iters += 1
            epochs += 1

            if n == batch_size - 1:
                print(f"Epoch: {epoch} Loss D.: {loss_discriminator}")
                print(f"Epoch: {epoch} Loss G.: {loss_generator}")

    latent_space_samples = torch.randn((batch_size, 100)).to(device=device)

    plt.figure(figsize=(10, 5))
    plt.title("Generator and Discriminator Loss During Training")
    plt.plot(Dis_losses, label="D")
    plt.plot(Gen_losses, label="G")
    plt.xlabel("iterations")
    plt.ylabel("Loss")
    plt.legend()
    plt.show()

Output:
I didn't see the proper use of loss function for the discriminator. You should give real samples and generated samples separately to the discriminator. I think  you should change your code to a form like this:
fake = generator(noise)
disc_real = disc(real)
loss_disc_real = loss_func(disc_real, torch.ones_like(disc_real))
disc_fake = disc(fake)
loss_disc_fake = loss_func(disc_fake,torch.zeros_like(disc_fake))
loss_disc = (loss_disc_real+loss_disc_fake)/2
....

loss_generator = loss_func(disc_fake,torch.ones_like(disc_fake))
...

plot loss_disc and loss_generator, this should be work


Instruction: How to reduce the size of Bert model(checkpoint/model_state.bin) using pytorch

Input:
I used torch.quantization.quantize_dynamic to reduce the model size but it is reducing my prediction Accuracy score.
I'm using that model file inside the Flask and doing some real time predictions, Due to the large size i'm facing issues while predicting. So could anyone please help me on reducing the bert model size using pytorch and guide me on who to do the real time predictions.

Output:
Here is the doc of torch.quantization.quantize_dynamic where dtype is set to torch.qint8.
so if you dont want your accuracy to drastically decrease use the below syntax
torch.quantization.quantize_dynamic(model, qconfig_spec=None, dtype=torch.float16)


Instruction: Overfitting LSTM pytorch

Input:
I was following the tutorial on CoderzColumn to implement a LSTM for text classification using pytorch. I tried to apply the implementation on the bbc-news Dataset from Kaggle, however, it heavily overfits, achieving a max accuracy of about 60%.
See the train/loss curve for example:

Is there any advice (I am quite new to RNN/LSTM), to adapt the model to prevent that high overfiting?
The model is taken from the above tutorial and looks kind of like this:
class LSTMClassifier(nn.Module):
    def __init__(self, vocab, target_classes, embed_len = 50, hidden_dim=75, n_layers=1):
        super(LSTMClassifier, self).__init__()
        self.n_layers = n_layers
        self.embed_len = embed_len
        self.hidden_dim = hidden_dim
        self.embedding_layer = nn.Embedding(num_embeddings=len(vocab), embedding_dim=embed_len)
      #  self.lstm = nn.LSTM(input_size=embed_len, hidden_size=hidden_dim,dropout=0.2, num_layers=n_layers, batch_first=True)
        self.lstm = nn.LSTM(input_size=embed_len, hidden_size=hidden_dim, num_layers=n_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, len(target_classes))

    def forward(self, X_batch):
        embeddings = self.embedding_layer(X_batch)
        hidden, carry = torch.randn(self.n_layers, len(X_batch), self.hidden_dim), torch.randn(self.n_layers, len(X_batch), self.hidden_dim)
        output, (hidden, carry) = self.lstm(embeddings, (hidden, carry))

        return self.fc(output[:,-1])

I would be really thankful for any adive how to adapt the version in the tutorial to use it more effectively on other datasets

Output:
Have you tried adding nn.Dropout layer before the self.fc?
Check what p = 0.1 / 0.2 / 0.3 will do.
Another thing you can do is to add regularisation to your training via weight_decay parameter:
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4, weight_decay=1e-5)

Use small values first, and increase by 10 times, see which will get you the best result.
Also, goes without saying, make sure that there is no test data points in  train set. Make sure you did not forget to shuffle your train set:
train_loader = DataLoader(train_dataset, batch_size=1024, collate_fn=vectorize_batch, shuffle=True)


Instruction: PyTorch: Handling multiple functions and arguments

Input:
I have a large number of neural networks (such that I have to use list comprehension to produce them), say [f_1, f_2, f_3 ...], collectively denoted by F.
I have the argument X = [x_1, x_2, x_3 ...], such that input tensor x_i is intended for the network f_i. The tensors I have will be big, owing to the data and then the gradients.
Is there an elegant (and efficient) way of obtaining the sequence [f_1(x_1), f_2(x_2), f_3(x_3) ...]?

Output:
You could again use a list comprehension:
out = [f(x) for f,x in zip(F,X)]


Instruction: Difficulties in using jacobian of torch.autograd.functional

Input:
I am solving PDE, so I need the jacobian matrix of the residual with respect to variables.
Here is my code
import torch
from torch.autograd.functional import jacobian
def get_residual (pgnew, swnew):
    residual_w  = 5*(swnew-swold)+T_w*((pgnew[2:,:,:]-pgnew[1:-1,:,:])-(pc[2:,:,:]-pc[1:-1,:,:])) - T_w*((pgnew[1:-1,:,:]-pgnew[0:-2,:,:])-(pc[1:-1,:,:]-pc[0:-2,:,:]))
    residual_g  = 5*((1-swnew)-(1-swold))+T_g*(pgnew[2:,:,:]-pgnew[1:-1,:,:]) - T_g*(pgnew[1:-1,:,:]-pgnew[0:-2,:,:])
    residual    = torch.ravel(torch.column_stack((residual_w,residual_g)))
    return residual

if __name__ == '__main__':
    dt       = 0.01
    T_w      = 10
    T_g      = 12
    swnew     = torch.zeros(3, 1, 1, requires_grad=True, dtype=torch.float64)
    swold     = torch.ones(3, 1, 1, requires_grad=True, dtype=torch.float64)
    pgnew     = 2*torch.ones(5, 1, 1, requires_grad=True, dtype=torch.float64)
    pc        = 3*torch.ones(5, 1, 1, requires_grad=True, dtype=torch.float64)

    unknown  = torch.ravel(torch.column_stack((pgnew[1:-1,:,:],swnew)))

    residual = get_residual(pgnew, swnew)
    print('Check Jacobian \n', jacobian(get_residual, unknown))

I am following this tutorial;
however, it shows an error, namely,

get_residual() missing 1 required positional argument: 'swnew',

so I change it to
print('Check Jacobian \n', jacobian(get_residual(pgnew, swnew),unknown)

Then it shows 'Tensor' object is not callable'
Thank you

Output:
You have to give the jacobian a tuple as second argument, containing the arguments of get_residual:
print('Check Jacobian \n', jacobian(get_residual, (pgnew, swnew)))

(You may need to make sure that they have the shape you want etc.)


Instruction: PyTorch lightening resume training from last epoch and weights

Input: I am using PyTorch Lightening trainer for pre-training a large model. I know I can resume training from old weights but that does not contain old hyper-parameters (lr, last_epoch, etc.). Is there any automatic way to resume training? OR Do I need to overload Checkpoint Callback or CSVLogger to search for old cvs logs and get last epoch number?

Output:
Instead of load from checkpoint, use resume_from_checkpoint in trainer.
# resume from a specific checkpoint
trainer = Trainer(resume_from_checkpoint="some/path/to/my_checkpoint.ckpt")

See more details here https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#resume-from-checkpoint .


Instruction: Good way to down-dimension (extract) of a 3D tensor (or same as numpy)

Input:
I have some data stored in a certain 3D tensor
data1 = torch.ones(3, 3, 3, requires_grad=True, dtype=torch.float64)
data2 = torch.zeros(3, 3, 3, requires_grad=True, dtype=torch.float64)

When I perform the calculation
temp= data1[:,0,0]+data2[:,0,0]

I would like to see the result in form of size ([3])tensor instead of ([3,1,1])
So considering the performance,

I should extract from the data1, data2 or temp?
How to do this?

Output:
As @paime said, you can use single element slices instead of single indexes:
>>> data1[:,:1,:1] + data2[:,:1,:1]
tensor([[[1.]],

        [[1.]],

        [[1.]]], dtype=torch.float64, grad_fn=<AddBackward0>)

Alternatively, there are different ways of unsqueezing multiple dimensions at the same time:

Using two torch.Tensor.unsqueeze
>>> temp.unsqueeze(-1).unsqueeze(-1)

Reshaping the tensor with a view:
>>> temp.view(*temp.shape, 1, 1)

Or with fancy indexing:
>>> temp[..., None, None]


Instruction: Neural Network for Regression using PyTorch

Input:
I am trying to implement a Neural Network for predicting the h1_hemoglobin in PyTorch. After creating a model, I kept 1 in the output layer as this is Regression. But I got the error as below. I'm not able to understand the mistake. Keeping a large value like 100 in the output layer removes the error but renders the model useless as I am trying to implement regression.
Data:

from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)
##### Creating Tensors
X_train=torch.tensor(X_train)
X_test=torch.tensor(X_test)
y_train=torch.LongTensor(y_train)
y_test=torch.LongTensor(y_test)

class ANN_Model(nn.Module):
    def __init__(self,input_features=4,hidden1=20,hidden2=20,out_features=1):
        super().__init__()
        self.f_connected1=nn.Linear(input_features,hidden1)
        self.f_connected2=nn.Linear(hidden1,hidden2)
        self.out=nn.Linear(hidden2,out_features)
    def forward(self,x):
        x=F.relu(self.f_connected1(x))
        x=F.relu(self.f_connected2(x))
        x=self.out(x)
        return x

loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)

epochs = 500
final_losses = []
for i in range(epochs):
    i = i + 1
    y_pred = model.forward(X_train.float())
    loss=loss_function(y_pred, y_train)
    final_losses.append(loss.item())
    if i%10==1:
        print("Epoch number: {} and the loss: {}".format(i, loss.item()))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Error:

Output:
Since you are performing regression, the CrossEntropyLoss() internally implements the NLLLoss() function. The CrossEntropyLoss() expects C classes for C predictions but you have specified only one class. The NLLLoss() tries to index into the prediction logits based on the ground-truth value. E.g., in your case, the ground-truth is a single value 14. The loss step tries to index into the 14th logit of your predictions to get its corresponding value so that it can compute the negative log likelihood on it, which is essentially - -log(probability_k) where k is the index that the ground-truth outputs. Since you have only logit in your predictions, it throws an error - index out of bounds.
For regression problems, you should consider using distance based losses such as MSELoss().
Try replacing your loss function - loss_function = CrossEntropyLoss() with loss_function = MSELoss()


Instruction: How to use fairseq interactive with multiple gpu?

Input:
I am trying to generate new prediction for the model, but I found it is not that intuitive to use fairseq. I found fairseq-interactive could help to generate with a good settings of batch_size, however, it seems that it will use 1 GPU at a time, I wonder if it is possible to use multiple GPU? Hope someone can kindly help!
Many thanks :)

Output: You cannot do this natively within fairseq. The best way to do this is to shard your data and run fairseq-interactive on each shard in the background. Be sure to set CUDA_VISIBLE_DEVICES for each shard so you put each shard's generation on a different GPU. This advice also applies to fairseq-generate (which will be significantly faster for large inference jobs).


Instruction: Using reshape or view in a certain fashion

Input:
import torch
import numpy as np
a = torch.tensor([[1, 4], [2, 5],[3, 6]])
bb=a.detach().numpy()
b = a.view(6).detach().numpy()

Element b is like:
[1 4 2 5 3 6]

How do I reshape back to the following:
[1 2 3 4 5 6]

This is just an example, want some generic answers, even 3D.

Output:
If you want to remain in PyTorch, you can view b in a's shape, then apply a transpose and flatten:
>>> b.view(-1,2).T.flatten()
tensor([1, 2, 3, 4, 5, 6])

In the 3D case, you can perform similar manipulations using torch.transpose which enables you to swap two axes. You get the desired result by combining it with torch.view:

First case (extra dimension last):
>>> b = a.view(-1, 1).expand(-1,3).flatten()
tensor([1, 1, 1, 4, 4, 4, 2, 2, 2, 5, 5, 5, 3, 3, 3, 6, 6, 6])

>>> b.view(-1,2,3).transpose(0,1).flatten()
tensor([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6])

Second case (extra dimension first):
>>> b = a.view(1,-1).expand(3,-1).flatten()
tensor([1, 4, 2, 5, 3, 6, 1, 4, 2, 5, 3, 6, 1, 4, 2, 5, 3, 6])

>>> b.view(3,-1).T.view(-1,2,3).transpose(0,1).flatten()
tensor([1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6])


Instruction: How to understand the code "cc[bb] += aa" in pytorch?

Input:
import torch
aa=torch.tensor([[1,2,3],[4,5,6]]).T
bb=torch.tensor([0,1,1]).T
cc = torch.zeros(2, 2)
cc[bb] += aa

Then the result is cc=tensor([[1., 4.],[3., 6.]]), why?

Output:
Let us reason with pseudo code:
>>> aa = [[1,2,3],
          [4,5,6]].T
>>> aa
[[1,4],
 [2,5],
 [3,6]]

>>> bb = [0,1,1].T
>>> b
[0,
 1,
 1]

>>> cc = zeros(2, 2)
>>> cc
[[0,0],
 [0,0]]

The next instruction is an assignment which consists in first indexing cc with bb values. Here we are picking entire rows from cc using indices in bb. Since there are three rows in bb, the resulting tensor will consist of cc[bb[0]], cc[bb[1]], and cc[bb[2]] but bb[1] and bb[2] are equal which means it comes down to cc[0] and cc[1].
The right-hand side operand is aa and consists of three rows: [1,4], [2,5], and [3,6]. This means that the final operation performed will be equivalent to (row-wise):
cc[0] += [1,4]
cc[1] += [3,6]

Since cc is initialized with zero values, we can sum this up to:
>>> cc[0] = [1,4]
>>> cc[1] = [3,6]

This means that:
>>> cc
[[1,4],
 [3,6]]


Instruction: Count all different pixel values given a set of images

Input:
In order to sanity check masks for the semantic segmentation task, I would like to know how I can find all different pixel values, given a set of images.
I tried:
l = []
for img in glob.glob('/content/Maschere/*png'):
  im = Image.open(img)
  data = torch.from_numpy(np.asarray(im))
  v = torch.unique(data)
  l.append(v)

print(set(l))

The aforementioned code displays the unique pixel values per image, instead, I want get the unique for the whole set of images
NOTE:
I get this output format:
{tensor([  2, 255], dtype=torch.uint8), tensor([  2, 255], dtype=torch.uint8), tensor([  2, 255], dtype=torch.uint8), tensor([  2, 255], dtype=torch.uint8), tensor([  3, 255], dtype=torch.uint8), tensor([  9, 255], dtype=torch.uint8)

I would get this kind of result instead :
tensor([  2, 3, 9 255], dtype=torch.uint8)

Output:
I didnt' test it, but something along the lines of:
l = set()
for img in glob.glob('/content/Maschere/*png'):
  im = Image.open(img)
  data = torch.from_numpy(np.asarray(im))
  v = set(torch.unique(data))
  l.update(v)

print(l)

It maintains a single set which you update with any new values you encounter.


Instruction: Model training converges to a fixed value of loss with low accuracy

Input:
I have been trying to train a simple model on Chinese MNIST Kaggle dataset. However it keeps converging to a CrossEntropyLoss of 2.708050 even with model training setting picked up from pytorch tutorials. The losses change but converge to a high value with low accuracy (everytime < 10%).

There is no error in how the dataset is created.
I had initially tried custom made models, training loops as well as test loops, but those didn't work. Finally, I switched to tested functions to figure out the problem. Here is how I have defined the dataset function, initialized using a list of filepaths and target values converted to indices using a value_idx dictionary.
class Custom_dataset(Dataset):
    def __init__(self,filepath,value):
        data = [cv2.imread(fp,-1) for fp in filepath]
        data = [i/np.max(i) for i in data]
        data = torch.tensor(data, dtype=torch.double)
        data = data.view(-1,1,64,64)

        self.data = data

        self.size = len(value)
        assert len(filepath)==len(value), print('length mismatch')

        self.index = torch.tensor([value_idx[i] for i in value]).reshape(-1,1)
        self.target = torch.zeros((self.size,15)).scatter_(dim=1
                                                           ,index=self.index#.unsqueeze_(dim=1)
                                                           ,value=1)

        print('data shape ',self.data.shape)

    def __getitem__(self,idx):
        return (self.data[idx,:],self.target[idx])

    def __len__(self):
        return self.size

Here is my training loop and test loop (also picked up from the pytorch tutorials page)
def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X,y = X.to(device), y.to(device)
        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

and model definition
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = Sequential(Linear(64*64,64*32)
                       ,ReLU()
                       ,Linear(64*32,32*32)
                       ,ReLU()
                       ,Linear(32*32,16*32)
                       ,ReLU()
                       ,Linear(32*16,16*16)
                       ,ReLU()
                       ,Linear(16*16,15)
                       ,Sigmoid(),)

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

with the final code being
model_mk_47 = NeuralNetwork().double().to(device)
NUM_EPOCHS=10
for i in tqdm(range(NUM_EPOCHS)):
    train_loop(train_dl,model_mk_47, CrossEntropyLoss(), Adam(model_mk_47.parameters()) )

Output:
Calculating accuracy with sigmoid is not an issue as you are using argmax, and softmax and sigmoid will return different values but they will be in the same order. However, one issue i'm seeing with your code is that you are including your activation within your forward pass code. What I might encourage you to do is to remove the Sigmoid from your model. This is because CrossEntropyLoss takes in:

The input is expected to contain raw, unnormalized scores for each class. input has to be a Tensor of size (C)(C) for unbatched input, (minibatch, C)(minibatch,C) or (minibatch, C, d_1, d_2, ..., d_K)(minibatch,C,d1,d2,...,d K) with K \geq 1K≥1 for the K-dimensional case. The last being useful for higher dimension inputs, such as computing cross entropy loss per-pixel for 2D images.

This means it is expecting logits (which are the output of the last layer without any activation). If you want to use a sigmoid (would recommend softmax), use functional softmax after the loss has been calculated (or include a flag which only activates the model when self.training == True in the model code.
I assume you are getting the performance of your model on your train data. If this is not improving it either means your data pipeline is faulty or your approach isn't working. Are you sure that your labels align properly with your images?


Instruction: How to solve RuntimeError: CUDA out of memory?

Input:
I try to run an inference using a cli to get the predictions from a detection and recognition model. With cuda10.2 it takes 15 mins for the inference to complete but I have cuda11.3 which takes 3 hours, I want to reduce this time.
Note : My hardware does not support cuda10.2.
hence I have following packages installed,

cudatoolkit  11.3.1  h2bc3f7f_2
pytorch  1.10.0  py3.7_cuda11.3_cudnn8.2.0_0  pytorch
torchvision  0.11.0  py37_cu113 pytorch

I get this error while I run the inference cli,
RuntimeError: CUDA out of memory. Tried to allocate 2.05 GiB (GPU 0; 5.81 GiB total capacity; 2.36 GiB already allocated; 1.61 GiB free; 2.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Tried :

To change the batch_size both for detection and recognition

Kindly help!
Thank you.

Output:
I use the following solutions whenever I encounter "CUDA out of memory" error. Here are the solutions, from simple to hard:
1- Try to reduce the batch size. First, train the model on each datum (batch_size=1) to save time. If it works without error, you can try a higher batch size but if it does not work, you should look to find another solution.
2- Try to use a different optimizer since some optimizers require less memory than others. For instance, SGD requires less memory than Adam.
3- Try to use a simpler model with fewer parameters.
4- Try to divide the model into two (or more than two) separate parts. Then update each part's parameters separately in each epoch. Note that whenever you want to compute gradients and update parameters of one part, the parameters of the other part of the model should be frozen. This leads to a lower amount of RAM required in each step which may solve the problem.
5- Lastly, if none of the above solutions work, GPU computation cannot be used.
Note that you can use a combination of these solutions which may lead to preventing the error. For instance, perhaps using a smaller batch size and simpler optimizer may work in some situations.


Instruction: BCEWithLogitsLoss Multi-label Classification

Input:
I'm a bit confused about how to accumulate the batch losses to obtain the epoch loss.
Two questions:

Is #1 (see comments below) correct way to calculate loss with masks)
Is #2 correct way to report epoch loss)

optimizer = torch.optim.Adam(model.parameters, lr=1e-3, weight_decay=1e-5)
criterion = torch.nn.BCEWithLogitsLoss(pos_weight=pos_weight)

for epoch in range(10):
    EPOCH_LOSS = 0.

    for inputs, gt_labels, masks in training_dataloader:
        optimizer.zero_grad()
        outputs = model(inputs)

        #1: Is this the correct way to calculate batch loss? Do I multiply batch_loss with outputs.shape[0[ before adding it to epoch_loss?
        batch_loss = (masks * criterion(outputs, gt_labels.float())).mean()
        EPOCH_LOSS += batch_loss
        loss.backward()
        optimizer.step()

    #2: then what do I do here? Do I divide the EPOCH_LOSS with len(training_dataloader)?
    print(f'EPOCH LOSS: {EPOCH_LOSS/len(training_dataloader)}:.3f')

Output:
In your criterion, you have got the default reduction field set (see the docs), so your masking approach won't work. You should use your masking one step earlier (prior to the loss calculation) like so:
batch_loss = (criterion(outputs*masks, gt_labels.float()*masks)).mean()

OR
batch_loss = (criterion(outputs[masks], gt_labels.float()[masks])).mean()

But, without seeing your data it might be a different format. You might want to check that this is working as expected.
In regards to your actual question, it depends on how you want to represent your data. What I would do is just to sum all of the batches' losses and represent that, but you can choose to divide by the number of batches if you want to represent the AVERAGE loss of each batch in the epoch.
Because this is purely an illustrative property of your model, it actually doesn't matter which one you pick, as long as it's consistent between epochs to represent the fact that your model is learning.


Instruction: unable to load torchaudio even after installing

Input:
I'm trying to use torchaudio but I'm unable to import it. I have installed it and it is also visible through the pip list.
<ipython-input-6-4cf0a64f61c0> in <module>
----> 1 import torchaudio

ModuleNotFoundError: No module named 'torchaudio'

pytorch-lightning        1.2.0
torch                    1.10.1
torchaudio               0.10.1
torchvision              0.11.2
WARNING: You are using pip version 21.1.2; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Output:
Since you are using linux and CPU, you should consider uninstalling every pytorch related packages and re-install them via:
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu

as shown here.


Instruction: Dataloades TypeError: __getitem___() takes 1 positional argument but 2 were given

Input:
it's my first time approaching pytorch. I built a dataset class to load tensors by Dataloader, like this:
train_loader = DataLoader(dataset_train, batch_size=6, drop_last=True)

But at the following line:
for i,train_batch in enumerate(train_loader):

I receive this error:
TypeError: __ getitem__() takes 1 positional argument but 2 were given
Any help would be great. I'm stuck on it.
My concern is that it could depend on the libraries versions I'm using:
matplotlib              3.5.2
numpy                   1.23.0
opencv-python           4.6.0.66
torch                   1.12.0
torch-tb-profiler       0.4.0
torchaudio              0.12.0
torchvision             0.13.0
Thank you.

Output:
I believe you expected to enumerate your dataloader:
for i, train_batch in enumerate(dataloader):
    # train loop


Instruction: How can I delete the background outside the drawn contours?

Input:
How can I delete the background outside the drawn contours?
My main goal is measure size of ONLY cardboard boxes.
I have 2 differend code.
First code measuring EVERYTHING with aruco marker.
Second code is detecting boxes with yolo.(I need this because measure code detects everything)
Both of them drawn contours.
My measure code is measuring everything thats why i want to remove background except contoured objects.
how can i manage this?
Please help.

Output:
Create black mask image. For each detection draw it contour on mask:
cv2.drawContours(img, contours, -1, color=(255, 255, 255), thickness=cv2.FILLED)

And after make bitwise_and with this mask.


Instruction: MultiplicativeLR scheduler not working properly when call scheduler.step()

Input:
PytorchLightning Framework, I am configuring the optimizers like this:
    def configure_optimizers(self):
    opt = torch.optim.Adam(self.model.parameters(), lr=cfg.learning_rate)
    #modified to fit lightning
    sch = torch.optim.lr_scheduler.MultiplicativeLR(opt, lr_lambda = 0.95) #decrease of 5% every epoch

    return [opt], [sch]

Then in the training_step, I can either call manually the lr_scheduler or let lightning do it automatically.
Fact is that in any case I got this kind of error:
 lr_scheduler["scheduler"].step()
  File "/home/lsa/anaconda3/envs/randla_36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 152, in step
    values = self.get_lr()
  File "/home/lsa/anaconda3/envs/randla_36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 329, in get_lr
    for lmbda, group in zip(self.lr_lambdas, self.optimizer.param_groups)]
  File "/home/lsa/anaconda3/envs/randla_36/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 329, in <listcomp>
    for lmbda, group in zip(self.lr_lambdas, self.optimizer.param_groups)]
TypeError: 'float' object is not callable

But ifI use any other scheduler, not only VSCode recognize it as belonging to pytorch, I also do not get this error.
Pytorch version 1.10
Lightning Version 1.5

Output:
I think that you need to change the value of `lr_lambda'.
Here is the link to the documentation: https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.MultiplicativeLR.html
lr_lambda (function or list) – A function which computes a multiplicative factor given an integer parameter epoch, or a list of such functions, one for each group in optimizer.param_groups.

So, if you want a decrease of 5% every epoch, then you could do the following:

def configure_optimizers(self):
    opt = torch.optim.Adam(self.model.parameters(), lr=cfg.learning_rate)
    #modified to fit lightning
    lmbda = lambda epoch: 0.95
    sch = torch.optim.lr_scheduler.MultiplicativeLR(opt, lr_lambda = lmbda) #decrease of 5% every epoch

    return [opt], [sch]


Instruction: Upsampling Only the Last Two Dimensions of a 5D Tensor

Input:
I have a 5D tensor x (frames of a video) and I want to upsample the spatial size (the last two dimensions) of this tensor but when I use upsampling, the last three dimensions of the tensor are upsampled. For upsampling I use the following class:
class Upsample(nn.Module):
    def __init__(self, scale_factor, mode, align_corners=False):
        self.interp = interpolate
        self.scale_factor = scale_factor
        self.mode = mode
        self.align_corners=align_corners

    def forward(self, x):
        x = self.interp(x, scale_factor=self.scale_factor, mode=self.mode)
        return x

And for example, the main class that I want to upsample a 5D tensor is as follows (I condensed my code):
class Main(nn.Module):
   def __init__(self):
       super(Main, self).__init__()
       self.upsample = Upsample(scale_factor=2, mode='trilinear')

   def forward(self, x):
        x = self.upsample(x)
        return x

To be clearer, for example by applying upsampling on a tensor of x=(2,4,3,10,20), the outcome based on the aforementioned class is x=(2,4,6,20,40) but I need to have x=(2,4,3,20,40).
What is the problem and how can I solve this?

Output:
The trilinear mode of pytorch's interpolate function only supports interpolation of 5D tensor including your third dimension. If you don't mind in resizing your input tensor, you may reduce the dimension and apply bicubic mode for interpolation.
class Upsample(nn.Module):
    def __init__(self, scale_factor, mode = 'bicubic', align_corners=False):
        self.interp = interpolate
        self.scale_factor = scale_factor
        self.mode = mode
        self.align_corners=align_corners

    def forward(self, x):
        B, C, T, W, H = x.size()
        x = x.reshape(B, C*T, W, H)
        x = self.interp(x, scale_factor=self.scale_factor, mode=self.mode)
        x = x.reshape(B, C, T, W, H)
        return x


Instruction: Cannot import name 'ResNet50_Weights' from 'torchvision.models.resnet'

Input:
I was previously loading a ResNet model with the ResNet50_Weights parameter successfully, but then suddenly I started getting the following error:
Traceback (most recent call last):
  File "splitting_models.py", line 3, in <module>
    from torchvision.models.resnet import ResNet50_Weights
ImportError: cannot import name 'ResNet50_Weights' from 'torchvision.models.resnet' (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torchvision/models/resnet.py)

Here is the import:
from torchvision.models import ResNet50_Weights

How could I go about fixing this?
PyTorch version: 1.2.0
TorchVision version: 0.4.0
EDIT
Upgrading using
pip install --upgrade torch torchvision

to the following versions fixed the issue:
PyTorch version: 1.12.0
TorchVision version: 0.13.0

Output:
The problem took me some days to solve it.
Before running a code containing the Pytorch models, make sure you are connected to a stable network. This because for the first time when you are running a Pytorch model such resnet50, alexnet, resnet18 it downloads its' functionalities, so incase of installation error it caches its downloads and draws such error, if you try to re-run. To solve the problem locate where the cache file is, do delete it and try to re-run the code having a stable network.
for my case:
C:\Users\user/.cache\torch\hub\checkpoints\resnet18-f37072fd.pth

this where the is cached file was, delete it and try to re-run having a stable network.
I hope it will help.
Thanks regards,


Instruction: Pytorch fine tuned CNN model giving always the same prediction in training and validation data

Input:
I decided to move from TensorFlow to Pytorch and I am with some issues in understanding how it works. I tried to follow This Tutorial which has a very simple example of Feature Extraction from ImegeNet CNNs for a binary classification problem.
In summary, in my code, the network is defined and called as follows
#Function for feature extraction
def set_parameter_requires_grad(model, feature_extracting):
    if feature_extracting:
        for param in model.parameters():
            param.requires_grad = False

# Initialize these variables which will be set in this if statement. Each of these
# variables is model specific.
def initialize_model(model_name, num_classes, feature_extract, use_pretrained):

    model_ft = None
    input_size = 0

    if model_name == "resnet":
    """ Resnet18
    """
        if use_pretrained:
            model_ft = models.resnet18(weights="ResNet18_Weights.IMAGENET1K_V1")
        else:
            model_ft = models.resnet18(weights=None)

        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs, num_classes)
        input_size = 224

    return model_ft, input_size

model_ft, input_size = initialize_model("resnet", 2, feature_extract = True, use_pretrained = True)

The transforms I do in the data are defined below:
data_transforms = {
'train': transforms.Compose([
    transforms.RandomResizedCrop(input_size),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),
'val': transforms.Compose([
    transforms.Resize(input_size),
    transforms.CenterCrop(input_size),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
]),

}
I call the training this way:
if feature_extract:
    params_to_update = []

    for name, param in model_ft.named_parameters():
        if param.requires_grad:
            params_to_update.append(param)
            print("\t", name)
else:
    for name, param in model_ft.named_parameters():
        if param.requires_grad:
            print("\t", name)

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9)
criterion = nn.CrossEntropyLoss()
model_ft, hist = train_model(model_ft, dataloaders_dict, criterion, optimizer_ft,
num_epochs=num_epochs, is_inception=(model_name == "inception"))

The model has 94% validation accuracy. After training, I save and load the model again.
# Save the model (SIMPLEST WAY)
print("save model the easiest way")
torch.save(model_ft, "MY_FIRST_TORCH_MODEL")
print("Load model the easiest way")
model = torch.load("MY_FIRST_TORCH_MODEL")
print("Evaluate model")
model.eval()

So here it goes my problem: I don't know how to process a testing data sample in order to make it be tested by the Pytorch model. I tried to load and test a training/validation sample just to be sure that the same accuracies in the training and test will also remain when I do a manual test. Here is what I did:
for name in os.listdir("./hymenoptera_data/train/bees/"):
    # testing one example image
    print("Predicting "+ "./hymenoptera_data/train/bees/"+name)
    img = cv2.imread("./hymenoptera_data/train/bees/"+name)
    img_to_test= cv2.resize(img, (224,224), interpolation = cv2.INTER_AREA)
    print(img_to_test.shape)
    img_to_test = img_to_test.astype('float32')

    test_x = img_to_test.reshape(1, 3, 224, 224)
    test_x = torch.from_numpy(test_x)

    output = model(test_x.cuda())
    pred = torch.argmax(output, dim=1)

    print(output)
    print(pred)

So, no matter which sample I choose to test, the prediction is always the same
What am I doing wrong here?

Should the model be loaded in a different way?
Is my function to test correct, especially considering that here I am doing feature extraction instead of fine tuning?
Should I pre-process the data using the same transforms.compose function used to train and validate?
What else should be added to test correctly an image given the example above?

Output:
Be careful, img_to_test is in the HWC format. You are reshaping the image when you should be transposing its axes from HWC to CHW. You may want to replace the following:
>>> test_x = img_to_test.reshape(1, 3, 224, 224)

With a call to np.transpose or torch.transpose instead:
>>> test_x = image_to_test.transpose(2,0,1).unsqueeze(0)


Instruction: Segfault in pytorch on M1: torch.from_numpy(X).float()

Input:
I'm using an M1.
I'm trying to use pytorch for a conv net.
I have a numpy array that I'm trying to turn into a torch tensor.
When I call
torch.from_numpy(X)
pytorch throws an error that it got a double when it expected a float.
When I call
torch.from_numpy(X).float() on a friends computer, everything is fine.
But when I call this command on my computer, I get a segfault.
Has anyone seen this / know what might be happening / know how to fix?

Output:
What's your pytorch vision? I've encountered the same problem on my Macbook Pro M1, and my pytorch version is 1.12.0 at first. The I downgraded it to version 1.10.0 and the problem is solved. I suspect this has something to do with the compatibility with M1 in newer torch versions.
Actually I first uninstalled torch using pip3 uninstall torch and then reinstalled with pip3 install torch==1.10.0
But if you are using torchvision or some other affiliated packages, you may also need to downgrade them too.


Instruction: How do I save custom functions and parameters in PyTorch?

Input:
Firstly, the network function is defined：
def softmax(X):
    X_exp=torch.exp(X)
    partition=X_exp.sum(1,keepdim=True)
    return X_exp/partition

def net(X):
    return softmax(torch.matmul(X.reshape(-1,W.shape[0]),W)+b)

Then update the function parameters by training
train(net,train_iter,test_iter,cross_entropy,num_epoches,updater)

Finally, the function is saved and loaded for prediction
PATH='./net.pth'
torch.save(net,PATH)
saved_net=torch.load(PATH)
predict(saved_net,test_iter,6)

The prediction results show that the updated parameters W and b are not saved and loaded.
What is the correct way to save custom functions and updated parameters ?

Output:
The correct way is to implement your own nn.Module and then use the provided utilities to save and load the model's state (their weights) on demand.
You must define two functions:

__init__: the class initializer logic where you define your model's parameters.

forward: the function which implements the model's forward pass.

A minimal example would be of the form:
class LinearSoftmax(nn.Module):
    def __init__(self, in_feat, out_feat):
        super().__init__()
        self.W = torch.rand(in_feat, out_feat)
        self.b = torch.rand(out_feat)

    def softmax(X):
        X_exp = torch.exp(X)
        partition = X_exp.sum(1, keepdim=True)
        return X_exp / partition

    def forward(X):
        return softmax(torch.matmul(X.reshape(-1,W.shape[0]),W)+b)

You can initialize a new model by doing:
>>> model = LinearSoftmax(10, 3)

You can then save and load weights W and b of a given instance:

save the dictionary returned by nn.Module.state_dict with torch.save:
>>> torch.save(model.state_dict(), PATH)

load the weight into memory with torch.load and mount on model with nn.Module.load_state_dict
>>> model.load_state_dict(torch.load(PATH))


Instruction: Can I use pytorch .backward function without having created the input forward tensors first?

Input:
I have been trying to understand RNNs better and am creating an RNN from scratch myself using numpy.  I am at the point where I have calculated a Loss but it was suggested to me that rather than do the gradient descent and weight matrix updates myself, I use pytorch .backward function.  I started to read some of the documentation and posts here about how it works and it seems like it will calculate the gradients where a torch tensor has requires_grad=True in the function call.
So it seems that unless create a torch tensor, I am not able to use the .backward.  When I try to do this on the loss scalar, I get a 'numpy.float64' object has no attribute 'backward' error.  I just wanted to confirm.  Thank you!

Output:
Yes, this will only work on PyTorch Tensors.
If the tensors are on CPU, they are basically numpy arrays wrapped into PyTorch Tensors API (i.e., running .numpy() on such a tensor returns exactly the data, it can modified etc.)


Instruction: Split PyTorch tensor into overlapping chunks

Input:
Given a batch of images of shape (batch, c, h, w), I want to reshape it into (-1, depth, c, h, w) such that the i-th "chunk" of size d contains frames i -> i+d. Basically, using .view(-1, d, c, h, w) would reshape the tensor into d-size chunks where the index of the first image would be a multiple of d, which isnt what I want.
Scalar example:
if the original tensor is something like:
[1,2,3,4,5,6,7,8,9,10,11,12] and d is 2;

view() would return : [[1,2],[3,4],[5,6],[7,8],[9,10],[11,12]];
however, I want to get:
[[1,2],[2,3],[3,4],[4,5],[5,6],[6,7],[7,8],[8,9],[9,10],[10,11],[11,12]]

I wrote this function to do so:
def chunk_slicing(data, depth):
    output = []
    for i in range(data.shape[0] - depth+1):
        temp = data[i:i+depth]
        output.append(temp)
    return torch.Tensor(np.array([t.numpy() for t in output]))

However I need a function that is useable as part of a PyTorch model as this function causes this error :
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

Output:
IIUC, You need torch.Tensor.unfold.
import torch
x = torch.arange(1, 13)
x.unfold(dimension = 0,size = 2, step = 1)

tensor([[ 1,  2],
        [ 2,  3],
        [ 3,  4],
        [ 4,  5],
        [ 5,  6],
        [ 6,  7],
        [ 7,  8],
        [ 8,  9],
        [ 9, 10],
        [10, 11],
        [11, 12]])

Another example with size = 3 and step = 2.
>>> torch.arange(1, 10).unfold(dimension = 0,size = 3, step = 2)

tensor([[1, 2, 3],  # window with size = 3
# step : ---1--2---
        [3, 4, 5],  # 'step = 2' so start from 3
        [5, 6, 7],
        [7, 8, 9]])


Instruction: cudnn error when runnung pytorch on Google Colab

Input:
Since yesterday when I try to run Pytorch using GPU on Google Colab I recieve the error provided below. Previously it worked fine. I have tried to install different versions of Pytorch and I have got different errors.
# Use PyTorch to check versions, CUDA version and cuDNN

import torch

print("PyTorch version: ")
print(torch.__version__)
print("CUDA Version: ")
print(torch.version.cuda)
print("cuDNN version is: ")
print(torch.backends.cudnn.version())

PyTorch version:
1.12.0+cu113
CUDA Version:
11.3
cuDNN version is:
---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-23-93b5c974c4be> in <module>()
      8 print(torch.version.cuda)
      9 print("cuDNN version is: ")
---> 10 print(torch.backends.cudnn.version())

1 frames

/usr/local/lib/python3.7/dist-packages/torch/backends/cudnn/__init__.py in version()
     48 def version():
     49     """Returns the version of cuDNN"""
---> 50     if not _init():
     51         return None
     52     return __cudnn_version
/usr/local/lib/python3.7/dist-packages/torch/backends/cudnn/__init__.py in _init()
     39                 raise RuntimeError(
     40                     'cuDNN version incompatibility: PyTorch was compiled against {} '
---> 41                     'but linked against {}'.format(compile_version, runtime_version))
     42         return True
     43 else:

RuntimeError: cuDNN version incompatibility: PyTorch was compiled against (8, 3, 2) but linked against (8, 0, 5)

Output:
Use this code to upgrade Python to newer version(3.9), this solved a problem for me.
!wget -O mini.sh https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
!chmod +x mini.sh
!bash ./mini.sh -b -f -p /usr/local
!conda install -q -y jupyter
!conda install -q -y google-colab -c conda-forge
!python -m ipykernel install --name "py39" --user


Instruction: In simple terms, what is the relationship between the GPU, Nvidia driver, CUDA and cuDNN in the context for using a deep learning framework?

Input:
I have always been doing deep learning on Google Colab or on school clusters that have everything set up nicely. Recently I needed to set up a workstation to do deep learning from scratch and I realized I have very limited understanding of the things that I need to install to run a framework like tensorflow or pytorch on GPU.
So can anyone explain in simple term possible, what is purpose of Nvidia driver, CUDA and cuDNN? How do they work together or on top of one another, and why do I need to install all of them for tensorflow/pytorch?

Output: Python code runs on the CPU, not the GPU. This would be rather slow for complex Neural Network layers like LSTM's or CNN's. Hence, TensorFlow and PyTorch know how to let cuDNN compute those layers. cuDNN requires CUDA, and CUDA requires the NVidia driver. All of the last 3 components are provided by NVidia; they've decided to organize their code that way.


Instruction: nn.Parameter not getting updated not sure about the usage

Input:
I have declared two nn.Parameter() variables with requires_grad=True and I am using those in a different function that's being called inside the init method of the class where variables are declared. lparam and rparam are not getting updated
My question is am I doing it the right way?
if not how it should be done?
here is the code example:
class LG(BaseNetwork):
    def __init__(self, opt):
        super().__init__()
        self.opt = opt
        self.lparam = nn.Parameter(torch.zeros(1), requires_grad=True).cuda(device=opt.gpu_ids[0])
        self.rparam = nn.Parameter(torch.zeros(1), requires_grad=True).cuda(device=opt.gpu_ids[0])

    def foo(self, a, b, k=1.0, lparam=0, rparam=0):
      t = bar(a, b, k=k, lparam=lparam, rparam=rparam)
      return t

    def forward(self, a, b):
       x = self.foo(a, b, k=self.opt.k, lparam=self.lparam, rparam=self.rparam)
       return x

BaseNetwork is just initializing functions  and uses nn.Module
def bar(a, b, k=1.0, lparam=0, rparam=0):
    return n(a) * (b.std() * (k * lparam)) + (b.mean() * (k * rparam))

When I print the named params I can not get lparam and rparam

Output:
thanks, I got the solution here https://discuss.pytorch.org/t/nn-parameter-not-getting-updated-not-sure-about-the-usage/157226
I had to remove the cuda(device=opt.gpu_ids[0]) because it is supposed to get the device that the model is put on.


Instruction: How do I convert a 3*1 vector into a 2*2 lower triangular matrix in pytorch

Input:
I have created a network in pytorch, whose output is B*N*H*W. I want that N is equal to 3 and then convert the output into a 2*2 lower trangular matrix with a upper zero in the 2nd demension. There may be two ways to achieve that.
    class Net(nn.Module):
        def __init__(self, in_ch=3, out_ch=N):
        super().__init__():
            self.net = nn.Sequatial([...])
        def forward(self, x):
            return self.net(x)

First, let the network output be B*4*H*W, and then mutiple a mask of [1,0,1,1] in the 2nd dimension to make one channel to be zero. This way can get a 2*2 lower trangular matrix with a upper zero, while I don't know if there will be information lost. And I don't know how to check the loss of information while training.
    class Net(nn.Module):
        def __init__(self, in_ch=3, out_ch=4):
        super().__init__():
            self.net = nn.Sequatial([...])
            self.mask = torch.FloatTensor([1.,0.,1.,1.]).cuda()
        def forward(self, x):
            return torch.einsum('abij,b->abij', self.net(x), self.mask)

Second, I think that the direct solution is to output a B*3*H*W tensor and convert the 2nd dimension into a 2*2 lower trangular matrix with a upper zero. But I don't have an effictive approach to implementing this. The transform operation in 2nd dimension may be like:
    array[2.3, 5.1, 6.3]  -->  array[[2.3, 0.],
                                     [5.1, 6.3]]

There are still two questions. How to convert the 2nd dimension of B*3*H*W to a tensor of shape B*2*2*H*W and if the first method will make the information lost while training.

Output:
Try this:
You can easily calculate N from len(a) as N*(N+1)/2 = len(a) => N
Here is the numpy version:
a = np.array([2.3, 5.1, 6.3])
N = 2

c = np.zeros((N, N))
c[np.tril_indices(N)] = a

Output:
c
>array([[2.3, 0. ],
       [5.1, 6.3]])

Here is the pytorch version:
a = torch.tensor([2.3, 5.1, 6.3])
c = torch.zeros(N, N)
c[torch.tril_indices(N, N, offset=0).tolist()] = a
c

Output:
tensor([[2.3000, 0.0000],
        [5.1000, 6.3000]])


Instruction: Why this operation results in a tensor full of 'nan'?

Input:
err = reduce((mu_students - teacher_pred)**2, 'b h w vec -> b h w', 'sum')
where mu_students and teacher_pred are 2 tensors with size (1,14,14,256).
The two tensors do not contain nan values before the reduce.
Moreover, this exact same operation work fine with the previous layer of the network, when mu_students and teacher_pred have size = (1,28,28,128). This error presents itself only on one kind of images (but it shouldn't be related to size, because images are always resized to 224)

Output:
Could you provide more about the function reduce and the input tensors?
By assuming that;

(1) reduce is from einops and written as
from einops import reduce as reduce

(2) mu_students and teacher_pred is random tensors,

when I try this code on Google Colab, it works well without NaN.
!pip install einops
from einops import reduce as reduce
mu_students = torch.rand(1,14,14,256)
teacher_pred = torch.rand(1,14,14,256)
err = reduce((mu_students - teacher_pred)**2, 'b h w vec -> b h w', 'sum')

The values are, for example,
# mu_students
tensor([[[[0.4079, 0.5835, 0.6807,  ..., 0.6041, 0.7366, 0.9291],
          [0.0338, 0.9161, 0.7018,  ..., 0.6035, 0.1816, 0.4059],
          [0.5949, 0.9535, 0.1460,  ..., 0.4049, 0.5120, 0.0734], ...

# teacher_pred
tensor([[[[0.7493, 0.2193, 0.6465,  ..., 0.6262, 0.0270, 0.5532],
          [0.5343, 0.9384, 0.9916,  ..., 0.2127, 0.0370, 0.6322],
          [0.3568, 0.7474, 0.5562,  ..., 0.0589, 0.1356, 0.6062], ...
# err
tensor([[[42.6367, 36.5668, 42.7598, 37.0643, 40.7744, 43.6076, 45.5338,
          45.3421, 40.0736, 45.2035, 43.3516, 40.8768, 39.1142, 40.8736],
         [50.8344, 40.1921, 40.7194, 44.3139, 37.7520, 39.2365, ...

I wonder if this also works in your environment.
Do you use specific tensors as mu_students or teacher_pred?


Instruction: AssertionError: If capturable=False, state_steps should not be CUDA tensors

Input:
I get this error while loading model weights of a previous epoch on Google colab. I'm using PyTorch version 1.12.0. I can't downgrade to a lower version as there are external libraries that Im using that require Pytorch 1.12.0
Thanks!

Output:
If you are using pytorch 1.12.0 with cuda binaries 11.6/11.7 then on your shell or command prompt, paste the following;
pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu116

The Adam Optimizer regression was removed in the updated torch version


Instruction: Error happend when import torch (pytorch)

Input:
Try to use pytorch, when I do
import torch
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-2-eb42ca6e4af3> in <module>
----> 1 import torch

C:\Big_Data_app\Anaconda3\lib\site-packages\torch\__init__.py in <module>
    124                 err = ctypes.WinError(last_error)
    125                 err.strerror += f' Error loading "{dll}" or one of its dependencies.'
--> 126                 raise err
    127             elif res is not None:
    128                 is_loaded = True

OSError: [WinError 182] <no description> Error loading "C:\Big_Data_app\Anaconda3\lib\site-packages\torch\lib\shm.dll" or one of its dependencies.

Not sure what happend.
The truth is, I installed pytorch around the end of last year.(I think)
I don't remember how, I installed it because I want try to use it.
But I guess I never used it after I installed. don't remember running some script with pytorch.
And now I start to use it, get this error. Have no clue What I should check first.
That means, I don't know what module, framework, driver, app are related with pytorch.
So I don't know where to beginning, check is there any other module might cause this Error.
If anyone knows how to solve this. Or where to start the problem checking. Please let me know.
Thank you.
My pytorch version is 1.11.0
OS: window 10

Output:
I solved the problem.
Just reinstall your Anaconda.
!!Warning!!: you will lose your lib.
Referring solution:
Problem with Torch 1.11


Instruction: Pytorch-Lightning Misconfiguration Exception; The closure hasn't been executed

Input:
I have been trying to train a torch.nn.TransformerEncoderLayer using the standard Pytorch-Lightning Trainer class. Before the first epoch even starts, I face the following error:
MisconfigurationException: The closure hasn't been executed. HINT: did you call optimizer_closure() in your optimizer_step hook? It could also happen because the optimizer.step(optimizer_closure) call did not execute it internally.
I have very properly defined the configure_optimizers() method in the trainer and it works for every other model (say, LSTM, GRU, MultiHeadAttention). If I replace them with the TransformerEncoder, the aforementioned error pops up.

Here is the model code I am using:
class PositionalEncoder(nn.Module):
    def __init__(self, d_model=512, max_seq_len=512):
        super().__init__()
        self.d_model = d_model
        pe = torch.zeros(max_seq_len,
                         d_model)

        for pos in range(max_seq_len):
            for i in range(0, d_model, 2):
                pe[pos, i] = sin(pos / (10000 ** ((2 * i)/d_model)))
                pe[pos, i+1] = cos(pos / (10000 ** ((2 * (i + 1))/d_model)))

        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x *= sqrt(self.d_model)
        x += self.pe[:,:x.size(1)]
        return x

class TRANSFORMER(pl.LightningModule):
    def __init__(self,
                 input_dim,
                 d_model=512,
                 nhead=8,
                 num_layers=6,
                 dropout=0.5,
                 use_scheduler=True,
                 num_tags=len(TAG2IDX),
                 total_steps=1024,
                 train_dataset=None,
                 val_dataset=None,
                 test_dataset=None):

        super().__init__()
        self.crf = CRF(num_tags=num_tags, batch_first=True)
        self.fc = nn.Linear(d_model, num_tags)
        self.use_scheduler = use_scheduler

        self.embedding = nn.Embedding(num_embeddings=input_dim,
                                      embedding_dim=d_model,
                                      padding_idx=0)

        self.pos_encoder = PositionalEncoder(d_model=d_model)

        self.encoder_layer = nn.TransformerEncoderLayer(d_model=d_model,
                                                        nhead=nhead,
                                                        dropout=dropout,
                                                        activation="gelu",
                                                        batch_first=True)

        self.encoder = nn.TransformerEncoder(encoder_layer=self.encoder_layer,
                                             num_layers=num_layers)
        ## Hyperparameters ##
        self.learning_rate = LEARNING_RATE
        self.weight_decay = WEIGHT_DECAY
        self.total_steps = total_steps
        self.batch_size = BATCH_SIZE
        ## Datasets ##
        self.train_dataset = train_dataset
        self.val_dataset = val_dataset
        self.test_dataset = test_dataset
        ## steps ##
        if self.use_scheduler:
            self.total_steps = len(train_dataset) // self.batch_size

    # create the dataloaders
    # add shuffle only for train_dataloader
    # make sure num_workers is set appropriately and drop_last is set to False
    def train_dataloader(self):
        return DataLoader(self.train_dataset,
                          batch_size=self.batch_size,
                          num_workers=N_JOBS,
                          shuffle=True,
                          drop_last=False)

    def val_dataloader(self):
        return DataLoader(self.val_dataset,
                          batch_size=self.batch_size,
                          num_workers=N_JOBS,
                          shuffle=False,
                          drop_last=False)

    def test_dataloader(self):
        return DataLoader(self.test_dataset,
                          batch_size=self.batch_size,
                          num_workers=N_JOBS,
                          shuffle=False,
                          drop_last=False)

    def forward(self, input_ids, masks):
        out = self.embedding(input_ids)
        out = self.pos_encoder(out)
        out = self.encoder(out, src_key_padding_mask=~masks)
        out = self.fc(out)
        return out

    def _shared_evaluation_step(self, batch, batch_idx):
        ids, masks, lbls = batch
        emissions = self(ids, masks)
        loss = -self.crf(emissions, lbls, mask=masks)
        pred = self.crf.decode(emissions, mask=masks)
        r, p, f1 = f1score(lbls, pred)
        return loss, r, p, f1

    def training_step(self, batch, batch_idx):
        loss, r, p, f1 = self._shared_evaluation_step(batch, batch_idx)
        self.log("train_loss", loss, on_step=False, on_epoch=True, prog_bar=True)
        self.log("train_recall", r, on_step=False, on_epoch=True, prog_bar=True)
        self.log("train_precision", p, on_step=False, on_epoch=True, prog_bar=True)
        self.log("train_f1score", f1, on_step=False, on_epoch=True, prog_bar=True)
        return loss

    def validation_step(self, batch, batch_idx):
        loss, r, p, f1 = self._shared_evaluation_step(batch, batch_idx)
        self.log("val_loss", loss, on_step=False, on_epoch=True, prog_bar=True)
        self.log("val_recall", r, on_step=False, on_epoch=True, prog_bar=True)
        self.log("val_precision", p, on_step=False, on_epoch=True, prog_bar=True)
        self.log("val_f1score", f1, on_step=False, on_epoch=True, prog_bar=True)

    def test_step(self, batch, batch_idx):
        loss, r, p, f1 = self._shared_evaluation_step(batch, batch_idx)
        self.log("test_loss", loss, on_step=False, on_epoch=True, prog_bar=True)
        self.log("test_recall", r, on_step=False, on_epoch=True, prog_bar=True)
        self.log("test_precision", p, on_step=False, on_epoch=True, prog_bar=True)
        self.log("test_f1score", f1, on_step=False, on_epoch=True, prog_bar=True)

    def predict_step(self, batch, batch_idx, dataloader_idx=0):
        ids, masks, _ = batch
        return self.crf.decode(self(ids, masks), mask=masks)

    def configure_optimizers(self):
        optimizer = Ranger(self.parameters(),
                           lr=self.learning_rate,
                           weight_decay=self.weight_decay)

        if self.use_scheduler:
            scheduler = get_cosine_schedule_with_warmup(optimizer=optimizer,
                                                        num_warmup_steps=1,
                                                        num_training_steps=self.total_steps)
            lr_scheduler = {
                'scheduler': scheduler,
                'interval': 'epoch',
                'frequency': 1
            }
            return [optimizer], [lr_scheduler]
        else:
            return [optimizer]

and here is how I am using the trainer class:
trainer = pl.Trainer(accelerator="gpu",
                     max_epochs=EPOCHS,
                     precision=32,
                     log_every_n_steps=1,
                     callbacks=[earlystopping_callback,
                                checkpoint_callback])

Output:
You are right. This happens because the special optimizer you have does not call the closure when passing it to the .step() method. But Lightning relies on this because it calls the step method like this:
optimizer.step(training_step_closure)

where training_step_closure consists of essentially executing the LightningModule.training_step.
It looks like Ranger does not follow the standard contract of calling the closure inside of itself.
To overcome this issue, I recommend switching to manual optimization:

Set self.automatic_optimization = False in your LightningModule.
Modify your training step by inserting manual backward, optimizer step and optionally the lr scheduler call:

Like so:
def training_step(self, batch, batch_idx):
    loss, r, p, f1 = self._shared_evaluation_step(batch, batch_idx)

    # Insert these lines:
    self.manual_backward(loss)

    optimizer = self.optimizers()
    scheduler = self.lr_schedulers()

    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()

    ...
    return loss

No other changes should be necessary.


Instruction: module 'torch' has no attribute 'frombuffer' in Google Colab

Input:
data_root = os.path.join(os.getcwd(), "data")
transform = transforms.Compose(
[
 transforms.ToTensor(),
 transforms.Normalize([0.5], [0.5]),
]
)
fashion_mnist_dataset = FashionMNIST(data_root, download = True, train = True, transform = transform)

Error Message

/usr/local/lib/python3.7/dist-packages/torchvision/datasets/mnist.py in read_sn3_pascalvincent_tensor(path, strict)
524     # we need to reverse the bytes before we can read them with torch.frombuffer().
525     needs_byte_reversal = sys.byteorder == "little" and num_bytes_per_value > 1
--> 526     parsed = torch.frombuffer(bytearray(data), dtype=torch_type, offset=(4 * (nd + 1)))
527     if needs_byte_reversal:
528         parsed = parsed.flip(0)
AttributeError: module 'torch' has no attribute 'frombuffer'

what can i do for this err in Colab

Output:
I tried your code in my Google Colab by adding the codes (to import the libraries) below, but it works well without errors.
import os
from torchvision import transform
from torchvision.datasets import FashionMNIST

I used

torchvision   0.13.0+cu113
google-colab   1.0.0
Runtime GPU   (when I set "None," it also works)

Do you get errors when you also use the same codes above? Do you use another versions?


Instruction: how to multiply three arrays with different dimension in PyTorch

Input:
enter image description here
L array dimension is (d,a) ,B is (a,a,N) and R is (a,d). By multiplying these arrays I have to get an array size of (d,d,N). How could I implement this is PyTorch

Output:
A possible and straightforward approach is to apply torch.einsum (read more here):
>>> torch.einsum('ij,jkn,kl->iln', L, B, R)

Where j and k are the reduced dimensions of L and R respectively. And n is the "batch" dimension of B.

The first matrix multiplication will reduce L@B (let this intermediate result be o):
ij,jkn->ikn

The second matrix multiplication will reduce o@R:
ikn,kl->iln

Which overall sums up to the following form:
ij,jkn,kl->iln


Instruction: Force a neural network to have 0-sum outputs

Input: I have a pytorch neural net with n-dimensional output which I want to have 0-sum during training (my training data, i.e. the true outputs, have 0 sum). Of course I could just add a line computing the sum s and then subtract s/n from each element of the output. But this way, the network would be driven even less to actually finding outputs with zero sum, as this would get taken care of anyways (I've been getting worse test results with this approach). Also, as the true outputs in the training data have 0 sum, obviously the network converges to having almost 0 sum outputs, but not quite. Hence, I was wondering whether there is a smart way to force the network to have outputs that sum to 0, without just brute-force subtracting the sum in the end (which would corrupt learning outputs to have sum 0)? I.e. some sort of solution directly incorporated in the network? (Probably there isn't, at least I couldn't think of any...)

Output: Your approach with "explicitly substracting the mean" is the correct way. The same way we use softmax to nicely parametrise distributions, and you could complain that "this makes the network not learn about probability even more!", but in fact it does, it simply does so in its own, unnormalised space. Same in your case - by subtracting the mean you make sure that you match the target variable while allowing your network to focus on hard problems, and not waste its compute on having to learn that the sum is zero. If you do anything else your network will literally have to learn to compute the mean somewhere and subtract it. There are some potential corner cases where there might be some deep representational reason for mean to be zero that could be argues for, but these cases are rare enough that chances that this is actually happening "magically" in the network are zero (and if you knew it was happening there would be better ways of targeting it than by zero ensuring).


Instruction: Logging in Custom Handler for TorchServe

Input:
I have written a custom handler for an DL model using torch-serve and am trying to understand how to add manual log messages to the handler. I know that I can simply print any messages and it will show them within the MODEL_LOG logger at level INFO.
What if I want to add a custom message at DEBUG level or ERROR level? When I try to initialise a logger using
logger = logging.getLogger('model_log') and add a message within the handler using logger.error I see the output on screen but at INFO level.
What is the best way to create log messages within a handler at a chosen log-level?

Output:
I am trying to do something similar. I found example logger usage in base_handler.py, where the logger is initialized on line 23 as:
logger = logging.getLogger(__name__) and used in several places in the rest of the source file, typically as: logger.debug() or logger.warning(), etc. I am assuming that these messages end up in the relevant files under the logs/ folder of TorchServe. I hope it is useful for your particular use case.


Instruction: Implement "same" padding for convolution operations with dilation > 1, in Pytorch

Input:
I am using Pytorch 1.8.1 and although I know the newer version has padding "same" option, for some reasons I do not want to upgrade it. To implement same padding for CNN with stride 1 and dilation >1, I put padding as follows:
 padding=(dilation*(cnn_kernel_size[0]-1)//2, dilation*(cnn_kernel_size[1]-1)//2))

According to the Pytorch document, I expected the input and output size will be the same, but it did not happen!
It is written in Pytorch document that:
Hout=⌊( Hin + 2×padding[0] − dilation[0]×(kernel_size[0]−1) −1) /stride[0] + 1⌋

Wout=⌊( Win + 2×padding[1] − dilation[1]×(kernel_size[1]−1) −1) /stride[1] + 1⌋

The input of torch.nn.Conv2d was with the shape of (1,1,625,513)
which based on the Conv2d pytorch document, indicates batch size = 1, C in = 1, H in = 625 and Win = 513
and after using:

64 filters
kernel size of (15,15)
stride = (1,1)
dilation =5
padding =(35,35)

Putting those values in the formulas above gives us:
Hout=⌊(625 + 2×35 −5×(15−1) −1) /1 +1⌋=⌊(625 + 70 −5×14 -1) + 1⌋=625

Wout=⌊(513 + 2×35 −5×(15−1) −1) /1 +1⌋=⌊(513 + 70 −5×14 -1) + 1⌋=513

However, the given output shape by pytorch was (1,64,681,569)
I can understand the value of 1 and C out = 64.  But I don't know why H out and W out are not the same as H in and W in? Does anyone has any explanation that can help?

Output:
I figured it out! The reason that I ended up with the wrong dimension was that I didn't put a numeric value for padding. I gave it the numeric value of dilation and based on that it calculate itself the value for padding as
padding=(dilation*(cnn_kernel_size[0]-1)//2, dilation*(cnn_kernel_size[1]-1)//2))
I think Pytorch needs to be given the numeric value of padding because when I change my code and gave the network the value of padding and calculate the dilation based on padding (Dilation = (2*padding)/(kernel size -1) I got the right output shape.


Instruction: Can I create a color image with a GAN consisting of only FC layers?

Input: I understand that in order to create a color image, three channel information of input data must be maintained inside the network. However, data must be flattened to pass through the linear layer. If so, can GAN consisting of only FC layer generate only black and white images?

Output:
Your fully connected network can generate whatever you want. Even three channel outputs. However, the question is: does it make sense to do so? Flattened your input will inherently lose all kinds of spatial and feature consistency that is naturally available when represented as an RGB map.
Remember that an RGB image can be thought of as 3-element features describing each spatial location of a 2D image. In other words, each of the three channels gives additional information about a given pixel, considering these channels as separate entities is a loss of information.


Instruction: Deep Neural Network in Python with 3 inputs and 3 outputs

Input:
I would like to implement a deep neural network in Python (preferably PyTorch, but TensorFlow is also possible) which predicts the next location and the time of the arrival at that location. For the raw data I have a csv file with a sequence of three values: latitude, longitude, and time:
39.984702,116.318417,2008-10-23,02:53:04
39.984683,116.31845,2008-10-23,02:53:10
39.984686,116.318417,2008-10-23,02:53:15
...

The number of such rows is around 100 000. So, here is my question. How should I split the data, normalize it and transform it, in order to feed it into the DNN (preferably GRU or LSTM, but as I read CNN are also possible) and receive as an output a predicted location and time of arrival?
Based on my current research, what should be done is to split the data into sequences (of n length), normalize the values, maybe even change the format of the time (for sure not feeding it as a string), and treat the last value in the sequence as a label during the teaching of the DNN.
A simple code would be really helpful, with my problems of understanding the different dimensions of the input and outputs for the NNs.

Output: Just a tip, for the time I would transform it into an Epoch Unix Timestamp.


Instruction: "PackagesNotFoundError: The following packages are not available from current channels:" While Installing PyTorch in Anaconda

Input:
When trying to install PyTorch inside an Anaconda environment with the command conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge, I get the error:
PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch
  - cudatoolkit=11.6
  - torchaudio

Current channels:

  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://conda.anaconda.org/conda-forge/win-32
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.,

I also tried adding conda-forge to my list of channels and got the same error. Can anyone help me?

Output: You have a 32 bit version of anaconda. You need to uninstall and install a 64 bit version to have access to these packages


Instruction: Custom Operations on Multi-dimensional Tensors

Input:
I am trying to compute the tensor R (see image) and the only way I could explain what I am trying to compute is by doing it on a paper:

o = torch.tensor([[[1, 3, 2], [7, 9, 8], [13, 15, 14], [19, 21, 20], [25, 27, 26]], [[31, 33, 32], [37, 39, 38], [43, 45, 44], [49, 51, 50], [55, 57, 56]]])
p = torch.tensor([[[19, 21, 20], [7, 9, 8], [13, 15, 14], [1, 3, 2], [25, 27, 26]], [[55, 57, 56], [31, 33, 32], [37, 39, 38], [43, 45, 44], [49, 51, 50]]])

# this is O' in image
o_prime = torch.tensor([[0.1, 0.2, 0.3, 0.4, 0.5], [0.6, 0.7, 0.8, 0.9, 0.11]])

# this is P' in image
p_prime = torch.tensor([[1.1, 1.2, 1.3, 1.4, 1.5], [1.6, 1.7, 1.8, 1.9, 1.11]])

# this is R (this is what I need)
r = torch.tensor([[[0, 0, 0, 6.1, 0], [0, 24.2, 0, 0, 0], [0, 0, 42.3, 0, 0], [60.4, 0, 0, 0, 0], [0, 0, 0, 0, 78.5]], [[0, 96.6, 0, 0, 0], [0, 0, 114.7, 0, 0], [0, 0, 0, 132.8, 0], [0, 0, 0, 0, 150.9], [168.11, 0, 0, 0, 0]]])

How do I get R without looping over tensors?
correction: In the image, I forgot to add value of p' along with sum(o) + o'

Output:
You can construct a helper tensor containing the resulting values sum(o) + o' + p':
>>> v = o.sum(2, True) + o_prime[...,None] + o_prime[...,None]
tensor([[[  7.2000],
         [ 25.4000],
         [ 43.6000],
         [ 61.8000],
         [ 80.0000]],

        [[ 98.2000],
         [116.4000],
         [134.6000],
         [152.8000],
         [169.2200]]])

Then you can assemble a mask for the final tensor via broadcasting:
>>> eq = o[:,None] == p[:,:,None]

Ensuring all three elements on the last dimension match:
>>> eq.all(dim=-1)
tensor([[[False, False, False,  True, False],
         [False,  True, False, False, False],
         [False, False,  True, False, False],
         [ True, False, False, False, False],
         [False, False, False, False,  True]],

        [[False, False, False, False,  True],
         [ True, False, False, False, False],
         [False,  True, False, False, False],
         [False, False,  True, False, False],
         [False, False, False,  True, False]]])

Finally, you can simply multiply both tensor and auto-broadcasting will handle the rest:
>>> R = eq.all(dim=-1) * v
tensor([[[  0.0000,   0.0000,   0.0000,   7.2000,   0.0000],
         [  0.0000,  25.4000,   0.0000,   0.0000,   0.0000],
         [  0.0000,   0.0000,  43.6000,   0.0000,   0.0000],
         [ 61.8000,   0.0000,   0.0000,   0.0000,   0.0000],
         [  0.0000,   0.0000,   0.0000,   0.0000,  80.0000]],

        [[  0.0000,   0.0000,   0.0000,   0.0000,  98.2000],
         [116.4000,   0.0000,   0.0000,   0.0000,   0.0000],
         [  0.0000, 134.6000,   0.0000,   0.0000,   0.0000],
         [  0.0000,   0.0000, 152.8000,   0.0000,   0.0000],
         [  0.0000,   0.0000,   0.0000, 169.2200,   0.0000]]])

I wanted to know how do you visualize such problems and then come up with a solution? Any pointers would be beneficial.

I was going to say it depends a lot on the problem at hand but this wouldn't get you very far! I believe having a toolbox of functions/tricks and scenarios you've come across (i.e. experience) helps greatly. This is true for problem-solving more generally speaking. I can try to explain how I came up with the solution and my thought process behind it. The initial idea for this problem is to perform an outer equality check between o and p. By that I mean we are trying to construct a structure which evaluates every (o[i], p[j]) relation in batch-wise.
Turns out this is rather a common operation usually seen as an outer summation or outer product. In fact, this type of operation is also applicable to the equality operator: here we are looking to construct a 5x5 matrix of o[i] == p[j]. Keeping in mind throughout the process we have a leading dimension containing three elements, but that doesn't change the process. We just need to account for it by checking that all three checks are indeed True, hence the all(dim=-1) call.
Since the desired result doesn't depend on column position inside the mask, i.e. result = sum(0) + o' + p' whatever the column index, we can just precompute the results for each row beforehand. The final operation is simple multiply the mask (which of course only contains ones at the desired locations) with the vector of dimensions. Intuitively, all columns will get multiplied by the same value but only the 1s will allow for the value to be set.
But most importantly, we have to acknowledge that your figure did all the hard work. This is in my opinion the first step before starting with any reasoning or implementation. So to summarize, I would suggest:

start with a minimal example reducing the number of variables while still making it relevant for the actual problem it is supposed to solve.

think about how you can solve it step by step by trying to get closer and closer to the solution. Iteratively, and trying out with this minimal setup.

Most importantly it comes with practice, with time you will find it easier to reason about your problem and use the right tools to manipulate your data.


Instruction: Constant training and test accuracy in GCNConv

Input:
I am new to pytorch and I'm trying to write a classifier for graph data. I have a dataset of 91 adj matrices (two classes in ratio 50/41, correlation matrices obtained from fMRI data). Currently I am struggling with classification task: my training and test accuracy doesn't change, although loss looks normal (?). Here's some code of my model:
from torch.nn import Linear,Sigmoid
import torch.nn.functional as F
from torch_geometric.nn import GCNConv, BatchNorm, GraphConv
from torch_geometric.nn import global_mean_pool

class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = GCNConv(dataset.num_node_features, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, hidden_channels)
        self.conv3 = GCNConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, 1)

    def forward(self, x, edge_index, edge_weight, batch):
        # 1. Obtain node embeddings
        h = self.conv1(x, edge_index, edge_weight)
        h = h.relu()
        h = self.conv2(h, edge_index, edge_weight)
        h = h.relu()
        h = self.conv3(h, edge_index, edge_weight)

        # 2. Readout layer
        h = global_mean_pool(h, batch)  # [batch_size, hidden_channels]

        # 3. Apply a final classifier
        h = F.dropout(h, p=0.5, training=self.training)
        h = self.lin(h)

        return h

and training loop:
model = GCN(hidden_channels=32)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
# criterion = torch.nn.CrossEntropyLoss()
criterion = torch.nn.BCELoss()
# criterion = torch.nn.MSELoss()

model = model.to(device)
def train():
    model.train()
    total_loss = 0
    batch_count = 0
    for data in train_loader:  # Iterate in batches over the training dataset.
        data = data.to(device)
        out = model(data.x, data.edge_index,  data.edge_weight, data.batch)  # Perform a single forward pass.
        target = data.y
        target = target.unsqueeze(1)
        target = target.float()
        loss = criterion(out, target)  # Compute the loss.
        loss.backward()  # Derive gradients.
        optimizer.step()  # Update parameters based on gradients.
        optimizer.zero_grad()  # Clear gradients.
        total_loss += loss.detach()
        batch_count += 1
    mean_loss = total_loss/batch_count
    return mean_loss

def test(loader):
    model.eval()
    correct = 0
    for data in loader:  # Iterate in batches over the training/test dataset.
    data = data.to(device)
    out = model(data.x, data.edge_index,  data.edge_weight, data.batch)
    pred = out.argmax(dim=1)  # Use the class with highest probability.
    correct += int((pred == data.y).sum())  # Check against ground-truth labels.
    return correct / len(loader.dataset)  # Derive ratio of correct predictions.

test_acc_summ = []
train_acc_summ = []
loss_summ = []
for epoch in range(1, 100):
    loss = train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    test_acc_summ.append(test_acc)
    train_acc_summ.append(train_acc)
    loss_summ.append(loss)
    print(f'Epoch: {epoch:03d}, Train Acc: {train_acc:.4f}, Test Acc: {test_acc:.4f}, Loss: {loss:.4f}')

The output I'm getting:
Epoch: 001, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 1.0981
Epoch: 002, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 1.0983
Epoch: 003, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 1.1312
Epoch: 004, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 1.0880
Epoch: 005, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.8857
Epoch: 006, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.9774
Epoch: 007, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.8917
Epoch: 008, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.8679
Epoch: 009, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.9000
Epoch: 010, Train Acc: 0.4568, Test Acc: 0.3000, Loss: 0.8371

Train and Test acc remains constant regardless of epochs number, but loss decreases. Is there some bug i don't see, or there's more complex problem?

Output: Try to add nn.Sigmoid() on top of self.lin output and remove dropout