Instruction: Floating point exception (core dumped) for UNet implementation

Input:
I am trying to do an implementation of KiuNet ( https://github.com/jeya-maria-jose/KiU-Net-pytorch ). But when I am executing the train command like so:
python train.py --train_dataset "KiuNet/Train Folder/" --val_dataset "KiuNet/Validation Folder/" --direc 'KiuNet/Results/' --batch_size 1 --epoch 200 --save_freq 10 --modelname "kiunet" --learning_rate 0.0001

I am getting the following error:
Traceback (most recent call last):
  File "KiuNet/KiU-Net-pytorch/train.py", line 235, in <module>
    loss.backward()
  File "/miniconda3/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
    torch.autograd.backward(
  File "/miniconda3/lib/python3.9/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [847,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [958,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [703,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [830,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [831,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [575,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [974,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [77,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [78,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [719,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [720,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [592,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [593,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [209,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [465,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [337,0,0] Assertion `t >= 0 && t < n_classes` failed.

When I am running the train command with CUDA_LAUNCH_BLOCKING=1 I get the following error:
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [840,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [580,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [453,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [326,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [71,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [712,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [198,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [199,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [968,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [959,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [830,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [574,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [702,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [191,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [318,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [319,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [446,0,0] Assertion `t >= 0 && t < n_classes` failed.
../aten/src/ATen/native/cuda/NLLLoss2d.cu:104: nll_loss2d_forward_kernel: block: [0,0,0], thread: [63,0,0] Assertion `t >= 0 && t < n_classes` failed.
Floating point exception (core dumped)

My torch and CUDA version are: '1.13.0+cu117'
My Python version: Python 3.9.12
Any help is much appreciated!

Output:
The repository author mentions the following.
"This bug occurs when the ground truth masks have more classes than the number of classes in prediction. Please make sure you ground truth images have only 0 or 1 labels of pixels if you are training for binary segmentation. The datasets usually have the ground truth as 0 or 255 labels of pixels. So, please convert them to 0's and 1's."


Instruction: Pytorch Parameterized Layer not Updated

Input:
I have a problem here, so I want to make a layer where the weight value (and the bias) is based on the other frozen weight. So, let’s say I have a frozen weight (FW) as a base value, then my current model layer will have weight W = FW + D, where D is the trainable parameter. Later, when I train the model, I hope the only parameter that gets updated is D.
I made this simple code for illustration:
frozen = nn.Linear(100,10)
frozen.weight.requires_grad = False
frozen.bias.requires_grad = False

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(100,10)
        self.dw = nn.Parameter(torch.tensor(1.0, requires_grad=True))
        self.db = nn.Parameter(torch.tensor(1.0, requires_grad=True))

    def forward(self, x):
        # the weight (and the bias) of fc layer is from FW and D
        self.fc.weight = nn.Parameter(torch.add(frozen.weight, self.dw))
        self.fc.bias = nn.Parameter(torch.add(frozen.bias, self.db))
        return torch.sigmoid(self.fc(x))

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100)
y = torch.tensor([0]*9+[1], dtype=torch.float32)

for _ in range(10):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

But when I run that code, the model doesn’t train, and the self.dw and self.db doesn’t change. I am not sure whether my concept is wrong, so it’s not possible to train D, or I made a mistake in the implementation.
I also tried to implement using nn.utils.parameterize, but it still doesn’t work (I am new to using this, so I am not sure I implemented it correctly)
frozen = nn.Linear(100,10)
frozen.weight.requires_grad = False
frozen.bias.requires_grad = False

class Adder(nn.Module):
    def __init__(self, delta, frozen):
        super().__init__()
        self.delta = nn.Parameter(torch.tensor(delta, requires_grad=True))
        self.frozen=frozen
    def forward(self, x):
        return torch.add(self.frozen, self.delta)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(100,10)

    def forward(self, x):
        nn.utils.parametrize.register_parametrization(self.fc, "weight",  Adder(1.0, frozen.weight))
        nn.utils.parametrize.register_parametrization(self.fc, "bias", Adder(1.0, frozen.bias))

        return torch.sigmoid(self.fc(x))

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100)
y = torch.tensor([0]*9+[1], dtype=torch.float32)

for _ in range(10):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Thank you for any responses.

Output:
Instead of recreating new weight and bias by
self.fc.weight = nn.Parameter(torch.add(frozen.weight, self.dw))
self.fc.bias = nn.Parameter(torch.add(frozen.bias, self.db))

You can utilize nn.functional.linear and intermediate variables
weight = self.weight + frozen.weight
bias = self.bias + frozen.bias
F.linear(x, weight, bias)

Complete version:
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, frozen):
        super(Net, self).__init__()
        self.weight = nn.Parameter(torch.ones(10, 100, dtype=torch.float32))
        self.bias = nn.Parameter(torch.zeros(10, dtype=torch.float32))
        self.frozen = frozen

    @property
    def weight_bias(self):
        weight = self.weight + self.frozen.weight
        bias = self.bias + self.frozen.bias
        return weight, bias

    def forward(self, x):
        # the weight (and the bias) of fc layer is from FW and D
        weight, bias = self.weight_bias
        return F.linear(x, weight, bias) # this should return raw logits as required by nn.CrossEntropyLoss

frozen = nn.Linear(100, 10)
frozen.weight.requires_grad = False
frozen.bias.requires_grad = False

model = Net(frozen)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100).unsqueeze(0)
y = torch.tensor([0]*9+[1], dtype=torch.float32).unsqueeze(0)

for _ in range(10):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


Instruction: How to translate this small part of TensorFlow code into pyTorch?

Input:
How to translate this small part of TensorFlow code into pyTorch?
def transforms(x):
        # stft returns spectogram for each sample and each eeg
        # input X contains 3 signals, apply stft for each
        # and get array with shape [samples, num_of_eeg, time_stamps, freq]
        # change dims and return [samples, time_stamps, freq, num_of_eeg]
        spectrograms = tf.signal.stft(x, frame_length=32, frame_step=4, fft_length=64)
        spectrograms = tf.abs(spectrograms)
        return tf.einsum("...ijk->...jki", spectrograms)

Output:
You can find the doc for STFT pytorch implementation here. The rest is fast-forward. It should be:
def transforms(x: torch.Tensor) -> torch.Tensor:
    """Return Fourrier spectrogram."""
    spectrograms = torch.stft(x, win_length=32, n_fft=4, hop_length=64)
    spectrograms = torch.abs(spectrograms)
    return torch.einsum("...ijk->...jki", spectrograms)


Instruction: Weighted mean squared error pytorch with sample weights regression

Input:
I am trying to use weighted mean squared error loss function for the regression task with imbalanced dataset. Basically, I have different weight assigned to each example and I am using the weighted MSE loss function. Is there a way to sample the weight tensor using TensorDataset along with input and output batch samples?
def weighted_mse_loss(inputs, targets, weights=None):
    loss = (inputs - targets) ** 2
    if weights is not None:
        loss *= weights.expand_as(loss)
    loss = torch.mean(loss)
    return loss

train_dataset = torch.utils.data.TensorDataset(x_train, y_train)
weights = torch.rand(len(train_dataset))

for x, y in train_loader:
    optimizer.zero_grad()
    out = model(x)
    loss = weighted_mse_loss(y, out, weights)
    loss.backward()

Output:
If you can get the weights before creating the train dataset:
train_dataset = TensorDataset(x_train, y_train, weights)
for x, y, w in train_dataset:
    ...

Otherwise:
train_dataset = TensorDataset(x_train, y_train)
for (x, y), w in zip(train_dataset, weights):
    ...

You can also use a DataLoader but be carefull about shuffling with the second method


Instruction: How do I create a branched AlexNet in PyTorch?

Input:
I am attempting to create a near identical model architecture to AlexNet, except each channel (Red, Green, and Blue) are disconnected by their own branch and are all concatenated at the end for the classifier.
Similar architecture to this
The base network:
class AlexNet(nn.Module):
    def __init__(self, num_classes: int = 1000, dropout: float = 0.5) -> None:
        super().__init__()
        _log_api_usage_once(self)
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.avgpool = nn.AdaptiveAvgPool2d((6, 6))
        self.classifier = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(256 * 6 * 6, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1)
        x = self.classifier(x)
        return x

Training
def train_epoch(self, epoch, total):
    self.model.train()

    for batch_idx, (features, targets) in enumerate(self.train_loader):
        features = features.to(self.device)
        targets = targets.to(self.device)

        logits = self.model(features)

        loss = self.loss_func(logits, targets)

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

I would like to have each channel be belong to its own feature extraction, but combine to classify.
red = features[:,0:1,:,:]
green = features[:,1:2,:,:]
blue = features[:,2:3,:,:]
logits = self.model([r,g,b])

I have seen people use groups but I am not sure how to implement it fully.
Any help is greatly appreciated

Output:
Since each branch/head would take an image with one channel you could start by just replacing the 3 in the first CNN layer with 1:
nn.Conv2d(1, 64, kernel_size=11, stride=4, padding=2),

Now you can send the three single-channeled images through the self.features layers and concat them before passing them to the self.classifier layers:
import torch
import torch.nn as nn

class AlexNet(nn.Module):
    def __init__(self, num_classes: int=1000, dropout: float=0.5) -> None:
        super().__init__()

        self.features = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, stride=4, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(64, 192, kernel_size=5, padding=2),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.Conv2d(192, 384, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(384, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.avgpool = nn.AdaptiveAvgPool2d((3, 3))
        self.classifier = nn.Sequential(
            nn.Dropout(p=dropout),
            nn.Linear(6912, 4096),
            nn.ReLU(inplace=True),
            nn.Dropout(p=dropout),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),
            nn.Linear(4096, num_classes),
        )

    def forward(self, x_r: torch.Tensor, x_g: torch.Tensor, x_b: torch.Tensor) -> torch.Tensor:
        x_r = self.features(x_r)
        x_r = torch.flatten(self.avgpool(x_r), 1)

        x_g = self.features(x_g)
        x_g = torch.flatten(self.avgpool(x_g), 1)

        x_b = self.features(x_b)
        x_b = torch.flatten(self.avgpool(x_b), 1)

        x = torch.concat((x_r, x_g, x_b), -1)

        x = self.classifier(x)
        return x

model = AlexNet()
img = torch.rand(1, 3, 256, 256)
img_r = torch.rand(1, 1, 256, 256)
img_g = torch.rand(1, 1, 256, 256)
img_b = torch.rand(1, 1, 256, 256)

output = model(img_r, img_g, img_b)

Note that I changed self.avgpool = nn.AdaptiveAvgPool2d((6, 6)) to self.avgpool = nn.AdaptiveAvgPool2d((3, 3)) because the output size of the flattened branches was really big (9216). Now it is 2304 and by concatinating them you get a tensor of size 6912. Hope this helps :)


Instruction: How to get Non-contextual Word Embeddings in BERT?

Input:
I am already installed BERT, But I don't know how to get Non-contextual word embeddings.
For example:

input: 'Apple'
output: [1,2,23,2,13,...] #embedding of 'Apple'

How can i get these word embeddings?
Thank you.
I search some method, but no blogs have written the way.

Output:
Sloved.
import torch
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

model = AutoModel.from_pretrained("bert-base-uncased")

# get the word embedding from BERT
def get_word_embedding(word:str):
    input_ids = torch.tensor(tokenizer.encode(word)).unsqueeze(0)  # Batch size 1
    # print(input_ids)
    outputs = model(input_ids)
    last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
    # output[0] is token vector
    # output[1] is the mean pooling of all hidden states
    return last_hidden_states[0][1]


Instruction: Problem loading parallel datasets even after using SubsetRandomSampler

Input:
I have two parallel datasets dataset1 and dataset2 and following is my code to load them in parallel using SubsetRandomSampler where I provide train_indices for dataloading.
P.S. Even after setting num_workers=0 and seeding np as well as torch, the samples do not get loaded in parallel. Any suggestions are heartily welcome including methods other than SubsetRandomSampler.
import torch, numpy as np
from torch.utils.data import Dataset, DataLoader, SubsetRandomSampler

dataset1 = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
dataset2 = torch.tensor([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

train_indices = list(range(len(dataset1)))
torch.manual_seed(12)
np.random.seed(12)
np.random.shuffle(train_indices)
sampler = SubsetRandomSampler(train_indices)

dataloader1 = DataLoader(dataset1, batch_size=2, num_workers=0, sampler=sampler)
dataloader2 = DataLoader(dataset2, batch_size=2, num_workers=0, sampler=sampler)

for i, (data1, data2) in enumerate(zip(dataloader1, dataloader2)):
  x = data1
  y = data2
  print(x, y)

Output:
tensor([5, 1]) tensor([15, 18])
tensor([0, 2]) tensor([14, 12])
tensor([4, 6]) tensor([16, 10])
tensor([8, 9]) tensor([11, 19])
tensor([7, 3]) tensor([17, 13])

Expected Output:
tensor([5, 1]) tensor([15, 11])
tensor([0, 2]) tensor([10, 12])
tensor([4, 6]) tensor([14, 16])
tensor([8, 9]) tensor([18, 19])
tensor([7, 3]) tensor([17, 13])

Output:
Since I was using a random sampler, the random indices are expected.
To yield the same (shuffled) indices from both DataLoaders, it is better to create the indices first, and then use a custom sampler:
class MySampler(torch.utils.data.sampler.Sampler):
    def __init__(self, indices):
        self.indices = indices

    def __iter__(self):
        return iter(self.indices)

    def __len__(self):
        return len(self.indices)

dataset1 = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
dataset2 = torch.tensor([10, 11, 12, 13, 14, 15, 16, 17, 18, 19])

train_indices = list(range(len(dataset1)))
np.random.seed(12)
np.random.shuffle(train_indices)

sampler = MySampler(train_indices)

dataloader1 = DataLoader(dataset1, batch_size=2, num_workers=0, sampler=sampler)
dataloader2 = DataLoader(dataset2, batch_size=2, num_workers=0, sampler=sampler)

for i, (data1, data2) in enumerate(zip(dataloader1, dataloader2)):
  x = data1
  y = data2
  print(x, y)

P.S. got the solution by cross-posting on Pytorch forums but still want to keep it for future readers. Credits to ptrblck.


Instruction: Why does the pytorch crossEntropyLoss use label encoding, instead of one-hot encoding?

Input:
I'm learning on CrossEntropyLoss module in pytorch.
And the tutor says, you should input target value y with 'label encoded', not 'one-hot encoded'.
Like this
loss = nn.CrossEntropyLoss()
Y = torch.tensor([0])
Y_pred_good = torch.tensor([[2.0, 1.0, 0.1]])
Y_pred_bad = torch.tensor([[0.5, 1.0, 0.3]])

l1 = loss(Y_pred_good, Y)
l2 = loss(Y_pred_bad, Y)

print(l1.item())
print(l2.item())

But I learned that CrossEntropy Loss is caculated with one-hot encoded class information.
Does the pytorch module transform label encoded into one-hot encoded? or is there another way to caculate CELoss with label encoded information?

Output:
There's a difference between the multi-label CE loss, nn.CrossEntropyLoss, and the binary version, nn.BCEWithLogitsLoss.
For the binary case, the implemented loss allows for "soft labels" and thus requires the binary targets to be floats in the range [0, 1].
In contrast, nn.CrossEntropyLoss works with "hard" labels, and thus does not need to encode them in a one-hot fashion.
If you do the math for the multi-class cross-entropy loss, you'll see that it is inefficient to have a one-hot representation for the targets. The loss is -log p_i where i is the true label. One only need to index the proper entry in the predicted probabilities vector. This can be done via multiplication be the one-hot encoded targets, but it is much more efficient to do it be indexing the right entry.

Note: It seems like recent versions of nn.CrossEntropyLoss also support one-hot encoded targets ("smooth labels").


Instruction: overlaying the ground truth mask on an image

Input:
In my project, I extracted frames from a video and in another folder I have ground truth for each frame.
I want to map the ground truth image of each frame of a video (in my case, it is saliency prediction ground truth) on its related frame image. As an example I have the following frame:

And the following is ground truth mask:

and the following is the mapping of ground truth on the frame.

How can I do that. Also, I have two folders that inside each of them, there are several folders that inside each of them the there are stored frames. How can I do this operation with these batch data?
This is the hierarchy of my folders:
frame_folder: folder_1, folder_2, ......
├── frames
│   ├── 601   (601 and 602  and etc are folders that in the inside there are image frames that their name is like 0001.png,0002.png, ...)
│   ├── 602
       .
       .
       .
│   └── 700

 ├── ground truth
    │   ├── 601   (601 and 602  and etc are folders that in the inside there are ground truth masks that their name is like 0001.png,0002.png, ...)
    │   ├── 602
           .
           .
           .
    │   └── 700

Update:
Using the answer proposed by @hkchengrex , I faced with an error. When there is only one folder in the paths, it works well but when I put several folders (frames of different videos) based on the question I face with the following error. the details are in below:

 multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/vtn/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
TypeError: process_video() takes 1 positional argument but 6 were given
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/Video_processing/Saliency_mapping.py", line 69, in <module>
    pool.apply(process_video, videos)
  File "/home/user/miniconda3/envs/vtn/lib/python3.10/multiprocessing/pool.py", line 357, in apply
    return self.apply_async(func, args, kwds).get()
  File "/home/user/miniconda3/envs/vtn/lib/python3.10/multiprocessing/pool.py", line 771, in get
    raise self._value
TypeError: process_video() takes 1 positional argument but 6 were given

Output:
I need to do similar things pretty often. In my favorite StackOverflow fashion, here is a script that you can copy and paste. I hope the code itself is self-explanatory. There are a few things that you can tune and try (e.g., color maps, overlay styles). It uses multiprocessing.Pool for faster batch-processing, resizes the mask to match the shape of the image, assumes the mask is in .png format, and depends on the file structure that you posted.
import os
from os import path
import cv2
import numpy as np

from argparse import ArgumentParser
from multiprocessing import Pool

def create_overlay(image, mask):
    """
    image: H*W*3 numpy array
    mask: H*W numpy array
    If dimensions do not match, the mask is upsampled to match that of the image

    Returns a H*W*3 numpy array
    """
    h, w = image.shape[:2]
    mask = cv2.resize(mask, dsize=(w,h), interpolation=cv2.INTER_CUBIC)

    # color options: https://docs.opencv.org/4.x/d3/d50/group__imgproc__colormap.html
    mask_color = cv2.applyColorMap(mask, cv2.COLORMAP_HOT).astype(np.float32)
    mask = mask[:, :, None] # create trailing dimension for broadcasting
    mask = mask.astype(np.float32)/255

    # different other options that you can use to merge image/mask
    overlay = (image*(1-mask)+mask_color*mask).astype(np.uint8)
    # overlay = (image*0.5 + mask_color*0.5).astype(np.uint8)
    # overlay = (image + mask_color).clip(0,255).astype(np.uint8)

    return overlay

def process_video(video_name):
    """
    Processing frames in a single video
    """
    vid_image_path = path.join(image_path, video_name)
    vid_mask_path = path.join(mask_path, video_name)
    vid_output_path = path.join(output_path, video_name)
    os.makedirs(vid_output_path, exist_ok=True)

    frames = sorted(os.listdir(vid_image_path))
    for f in frames:
        image = cv2.imread(path.join(vid_image_path, f))
        mask = cv2.imread(path.join(vid_mask_path, f.replace('.jpg','.png')), cv2.IMREAD_GRAYSCALE)
        overlay = create_overlay(image, mask)
        cv2.imwrite(path.join(vid_output_path, f), overlay)

parser = ArgumentParser()
parser.add_argument('--image_path')
parser.add_argument('--mask_path')
parser.add_argument('--output_path')
args = parser.parse_args()

image_path = args.image_path
mask_path = args.mask_path
output_path = args.output_path

if __name__ == '__main__':
    videos = sorted(
        list(set(os.listdir(image_path)).intersection(
                set(os.listdir(mask_path))))
    )

    print(f'Processing {len(videos)} videos.')

    pool = Pool()
    pool.map(process_video, videos)

    print('Done.')

Output:

EDIT: Made it work on Windows; changed pool.apply to pool.map.


Instruction: getting AttributeError: 'numpy.ndarray' object has no attribute 'dim' when converting tensorflow code to pytorch

Input:
I was translating my TensorFlow code to PyTorch and suddenly faced this error.
What am I doing wrong here?
    AttributeError                            Traceback (most recent call last)
    <ipython-input-36-058644576709> in <module>
          3 batch_size = 1024
          4 Xtrain = torch.concat(
    ----> 5     [transforms(Xtrain[batch_size*batch:batch_size*(batch +1)]) for batch in range(len(Xtrain)//batch_size+1)],
          6     axis=0
          7 )

    <ipython-input-36-058644576709> in <listcomp>(.0)
          3 batch_size = 1024
          4 Xtrain = torch.concat(
    ----> 5     [transforms(Xtrain[batch_size*batch:batch_size*(batch +1)]) for batch in range(len(Xtrain)//batch_size+1)],
          6     axis=0
          7 )

    <ipython-input-22-9fc8aa48e3e2> in transforms(x)
          1 def transforms(x: torch.Tensor) -> torch.Tensor:
          2     """Return Fourrier spectrogram."""
    ----> 3     spectrograms = torch.stft(x, win_length=32, n_fft=4, hop_length=64)
          4     spectrograms = torch.abs(spectrograms)
          5     return torch.einsum("...ijk->...jki", spectrograms)

    ~\anaconda3\lib\site-packages\torch\functional.py in stft(input, n_fft, hop_length, win_length, window, center, pad_mode, normalized, onesided, return_complex)
        565     #       this and F.pad to ATen.
        566     if center:
    --> 567         signal_dim = input.dim()
        568         extended_shape = [1] * (3 - signal_dim) + list(input.size())
        569         pad = int(n_fft // 2)

    AttributeError: 'numpy.ndarray' object has no attribute 'dim'

The following is the approach I have tried already:

    #!/usr/bin/env python
    # coding: utf-8

    # # Import library

    # In[1]:

    get_ipython().run_line_magic('matplotlib', 'inline')
    get_ipython().run_line_magic('load_ext', 'autoreload')
    get_ipython().run_line_magic('autoreload', '2')
    #%matplotlib qt

    # # Load pooled data

    # In[2]:

    from nu_smrutils import loaddat
    import pandas as pd

    # In[26]:

    import pickle
    import mne
    import numpy as np
    import matplotlib.pyplot as plt
    import seaborn as sns # Seaborn is a Python
    #  data visualization library built on top of Matplotlib.
    import datetime # module supplies classes for manipulating dates and times.

    import nu_smrutils # utils for SMR
    import nu_MIdata_loader # MI data loader

    import tensorflow as tf
    from tensorflow import keras
    from tensorflow.keras import layers

    # In[3]:

    dname = dict(BNCI2014004 = 'aBNCI2014004R.pickle',
                 BNCI2014001 = 'aBNCI2014001R.pickle',
                 Weibo2014   = 'aWeibo2014R.pickle',
                 Physionet   = 'aPhysionetRR.pickle')

    # In[4]:

    # itemname is one of : ['BNCI2014004', 'BNCI2014001', 'Weibo2014', 'Physionet']
    itemname = 'BNCI2014004'
    filename = dname[itemname]
    iname = itemname + '__'

    # In[5]:

    data = loaddat(filename)

    # In[6]:

    data[0]['right_hand'].plot();

    # In[7]:

    from nu_smrutils import load_pooled, augment_dataset, crop_data

    # In[8]:

    subjectIndex = list(range(108))
    class_name = ['left_hand', 'right_hand']

    dat = load_pooled(data, subjectIndex, class_name,
                      normalize = True, test_size = 0.15)

    # # Data augmentation

    # In[9]:

    print(dat.keys())
    dat['xtrain'].shape

    # In[10]:

    get_ipython().run_line_magic('pinfo', 'augment_dataset')

    # In[11]:

    augdata = dict(std_dev = 0.01, multiple = 2)

    # In[12]:

    xtrain, ytrain = augment_dataset(dat['xtrain'], dat['ytrain'],
                                     augdata['std_dev'], augdata['multiple'])

    print("Shape after data augmentation :", xtrain.shape)
    dat['xtrain'], dat['ytrain'] = xtrain, ytrain

    # # Data Cropping

    # In[14]:

    fs = 80 # sampling frequency
    crop_len = 1.5 #or None
    crop = dict(fs = fs, crop_len = crop_len)

    #if crop['crop_len']:
    X_train,y_train = crop_data(crop['fs'],crop['crop_len'],
                                dat['xtrain'], dat['ytrain'],
                                xpercent = 50)

    X_valid,y_valid = crop_data(crop['fs'],crop['crop_len'],
                                dat['xvalid'], dat['yvalid'],
                                xpercent = 50)

    X_test, y_test  = crop_data(crop['fs'],crop['crop_len'],
                                dat['xtest'], dat['ytest'],
                                xpercent = 50)

    dat = dict(xtrain = X_train, xvalid = X_valid, xtest = X_test,
               ytrain = y_train, yvalid = y_valid, ytest = y_test)

    # In[16]:

    print('data shape after cropping :',dat['xtrain'].shape)

    # # Pytorch dataloaders

    # In[18]:

    import torch
    from torch.utils.data import TensorDataset, DataLoader

    def get_data_loaders(dat, batch_size, EEGNET = None):
        # convert data dimensions to into to gray scale image format
        if EEGNET: ### EEGNet model requires the last dimension to be 1
            ff = lambda dat: torch.unsqueeze(dat, dim = -1)
        else:
            ff = lambda dat: torch.unsqueeze(dat, dim = 1)

        x_train, x_valid, x_test = map(ff,(dat['xtrain'], dat['xvalid'],dat['xtest']))
        y_train, y_valid, y_test = dat['ytrain'], dat['yvalid'], dat['ytest']
        print('Input data shape', x_train.shape)

        # TensorDataset & Dataloader
        train_dat    = TensorDataset(x_train, y_train)
        val_dat      = TensorDataset(x_valid, y_valid)

        train_loader = DataLoader(train_dat, batch_size = batch_size, shuffle = True)
        val_loader   = DataLoader(val_dat,   batch_size = batch_size, shuffle = False)

        output = dict(dset_loaders = {'train': train_loader, 'val': val_loader},
                      dset_sizes  =  {'train': len(x_train), 'val': len(x_valid)},
                      test_data   =  {'x_test' : x_test, 'y_test' : y_test})
        return output

    # In[19]:

    dat = get_data_loaders(dat, batch_size = 64)
    dat.keys()

    # In[20]:

    # Sanity check begin
    dset_loaders = dat['dset_loaders']
    dset_sizes = dat['dset_sizes']
    dset_sizes

    dtrain = dset_loaders['train']
    dval   = dset_loaders['val']

    dtr = iter(dtrain)
    dv  = iter(dval)

    # In[21]:

    inputs, labels = next(dtr)
    print(inputs.shape, labels.shape)
    # Sanity check end

    # In[29]:

    augmentdata = dict(std_dev = 0.01,  multiple = 1) # to augment data
    fs = 80

    crop_length = 1.5 #seconds
    crop = dict(fs = fs, crop_length = crop_length) # crop length

    class1, class2 = 'left_hand', 'right_hand'
    s = list(range(108))

    # In[31]:

    def convertY(Y):
        return np.concatenate([Y[:, None], np.where(Y == 0, 1, 0)[:, None]], axis=-1)

    # In[33]:

    def convert(d): # converting tran method
        Xtrain = d['xtrain'].numpy()
        Xval   = d['xvalid'].numpy()
        Xtest  = d['xtest'].numpy()

        Ytrain = convertY(d['ytrain'].numpy())
        Yval   = convertY(d['yvalid'].numpy())
        Ytest  = convertY(d['ytest'].numpy())

        return Xtrain, Xval, Xtest, Ytrain, Yval, Ytest

    # In[34]:

    files = ['aBNCI2014004R.pickle', ]

    # In data we storage sample from different files
    Data = []
    for file in files:
        d = nu_MIdata_loader.EEGDataLoader(file, class_name = [class1, class2])
        d1 = d.load_pooled(s, normalize = True, crop = crop, test_size = 0.01, augmentdata = augmentdata)
        Data.append(convert(d1))

    # In[35]:

    # concatenate all data if there more then one file
    Xtrain = np.concatenate([d[0] for d in Data])
    Xval   = np.concatenate([d[1] for d in Data])
    Xtest  = np.concatenate([d[2] for d in Data])
    Xtrain = np.concatenate([Xtrain, Xval], axis=0)

    Ytrain = np.concatenate([d[3] for d in Data])
    Yval   = np.concatenate([d[4] for d in Data])
    Ytest  = np.concatenate([d[5] for d in Data])
    Ytrain = np.concatenate([Ytrain, Yval], axis=0)

    # In[22]:

    def transforms(x: torch.Tensor) -> torch.Tensor:
        """Return Fourrier spectrogram."""
        spectrograms = torch.stft(x, win_length=32, n_fft=4, hop_length=64)
        spectrograms = torch.abs(spectrograms)
        return torch.einsum("...ijk->...jki", spectrograms)

    # In[36]:

    # Convert data in batchs
    # Cause outofmemort or python crash
    batch_size = 1024
    Xtrain = torch.concat(
        [transforms(Xtrain[batch_size*batch:batch_size*(batch +1)]) for batch in range(len(Xtrain)//batch_size+1)],
        axis=0
    )
    Xtest = torch.concat(
        [transforms(Xtest[batch_size*batch:batch_size*(batch +1)]) for batch in range(len(Xtest)//batch_size+1)],
        axis=0
    )

    # Convert to tensorflow tensors
    Ytrain = torch.cast(Ytrain, dtype='float32')
    Ytest = torch.cast(Ytest, dtype='float32')

Output:
torch.stft is a pytorch function and expects a Tensor as the input. You must convert your numpy array into a tensor and then pass that as the input.
you can use torch.from_numpy to do this.


Instruction: Select pytorch tensor elements by list of indices

Input:
I guess I have a pretty simple problem. Let's take the following tensor of length 6
t = torch.tensor([10., 20., 30., 40., 50., 60.])

Now I would like to to access only the elements at specific indices, lets say at [0, 3, 4]. So I would like to return
# exptected output
tensor([10., 40., 50.])

I found torch.index_select which worked great for a tensor of two dimensions, e.g. dimension (2, 4), but not for the given t for example.
How can access a set of elements based on a given list of indices in a 1-d tensor without using a for loop?

Output:
You can in fact use index_select for this:
t = torch.tensor([10., 20., 30., 40., 50., 60.])
output = torch.index_select(t, 0, torch.LongTensor([0, 3, 4]))
# output: tensor([10., 40., 50.])

You just need to specify the dimension (0) as the second parameter. This is the only valid dimension to specify for a 1-d input tensor.


Instruction: Need help on locating which part of CNN needs to get either .cuda() or .to_device()

Input:
Good day!
I've been struggling with the following error: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument target in method wrapper_nll_loss_forward)
I've been unable to locate where I need to add a part of my model to the GPU. From the error message I gather it should be in the loss function but I've tried all places I could think off, related to the loss function and have been unable to solve it.
Would love some help with this.
My full code can be found here:
https://huggingface.co/AFAD85/CNN_apples/blob/main/CNN%20paper%20clone%20pytorch.ipynb
I tried to isolate all possibly relevant code below:

`transformer = transforms.Compose([
    transforms.Resize((350,350)),
    transforms.ToTensor(),
    transforms.Normalize([0.5,0.5,0.5],
                         [0.5,0.5,0.5])
])`

`class ConvNet(nn.Module):
    def __init__(self,num_classes=4):
        super(ConvNet,self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3,out_channels=128,kernel_size=3,stride=1,padding='valid')
        self.bn1 = nn.BatchNorm2d(num_features=128)
        self.relu1 = nn.ReLU()
        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.conv2 = nn.Conv2d(in_channels=128,out_channels=64,kernel_size=3,stride=1,padding='valid')
        self.bn2 = nn.BatchNorm2d(num_features=64)
        self.relu2 = nn.ReLU()
        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.conv3 = nn.Conv2d(in_channels=64,out_channels=64,kernel_size=3,stride=1,padding='valid')
        self.bn3 = nn.BatchNorm2d(num_features=64)
        self.relu3 = nn.ReLU()
        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.conv4 = nn.Conv2d(in_channels=64,out_channels=32,kernel_size=3,stride=1,padding='valid')
        self.bn4 = nn.BatchNorm2d(num_features=32)
        self.relu4 = nn.ReLU()
        self.pool4 = nn.MaxPool2d(kernel_size=2)

        self.conv5 = nn.Conv2d(in_channels=32,out_channels=32,kernel_size=3,stride=1,padding='valid')
        self.bn5 = nn.BatchNorm2d(num_features=32)
        self.relu5 = nn.ReLU()
        self.pool5 = nn.MaxPool2d(kernel_size=2)

        self.flat = nn.Flatten()
        self.fc1 = nn.Linear(in_features=2592, out_features = 256)
        self.fc2 = nn.Linear(in_features=256, out_features = num_classes)

    def forward(self,input):
        output = self.conv1(input)
        output = self.bn1(output)
        output = self.relu1(output)
        output = self.pool1(output)

        output = self.conv2(output)
        output = self.bn2(output)
        output = self.relu2(output)
        output = self.pool2(output)

        output = self.conv3(output)
        output = self.bn3(output)
        output = self.relu3(output)
        output = self.pool3(output)

        output = self.conv4(output)
        output = self.bn4(output)
        output = self.relu4(output)
        output = self.pool4(output)

        output = self.conv5(output)
        output = self.bn5(output)
        output = self.relu5(output)
        output = self.pool5(output)

        # output = output.view(-1,32,9,9)

        output = self.flat(output)
        output = self.fc1(output)
        output = self.fc2(output)

        return output`

    model = ConvNet(num_classes=4).to(device)

    optimizer = Adam(model.parameters(),lr=0.001,weight_decay=0.0001)
    loss_function = nn.CrossEntropyLoss()

`best_accuracy = 0.0

for epoch in range(num_epochs):

    model.train()
    train_accuracy = 0.0
    train_loss = 0.0

    for i, (images,labels) in enumerate(train_loader):
        if torch.cuda.is_available():

            images = Variable(images.cuda())
            lables = Variable(labels.cuda())

        optimizer.zero_grad()

        outputs = model(images)
        loss = loss_function(outputs,labels)
        loss.backward()
        optimizer.step()

        train_loss += loss.cpu().data*images.size(0)
        _.prediction = torch.max(outputs.data,1)

        train_accuracy += int(torch.sum(prediction==labels.data))

    train_accuracy = train_accuracy/train_count
    train_loss = train_loss/train_count

    #test set evalueren
    model.eval()

    test_accuracy = 0.0
    for i, (images,labels) in enumerate(train_loader):
        if torch.cuda.is_available():
            images = Variable(images.cuda())
            lables = Variable(labels.cuda())

        outputs = model(images)
        _.prediction = torch.max(outputs.data,1)

    test_accuracy = test_accuracy/test_count

    print('Epoch: '+str(epoch)+' Train Loss: '+str(int(train_loss)))+' Train Accuracy: '+str(train_accuracy)+' Test Accuracy: '+str(test_accuracy)

    if test_accuracy > best_accuracy:
        torch.save(model.state_dict(), 'best_checkpoint.model')`

I tried to have the model run an epoch, expecting it to run on the GPU.
I tried adding .cuda() and .to_device() in all places where I expected the problem might lie, but was unable to find the correct one.

Output:
It was due to a typo
            lables = Variable(labels.cuda())

where lables should be labels.


Instruction: reshaping tensors for multi head attention in pytorch - view vs transpose

Input:
I'm learning about the attention operator in the deep learning domain. I understand that to compute multi head attention efficiently in parallel, the input tensors (query, key, value) have to be reshaped properly.
Assuming query, key and value are three tensor of identical shape [N, L, D], in which

N is the batch size
L is the sequence length
D is the hidden/embedding size,

they should be turned into [N*N_H, L, D_H] tensors, where N_H is the number of heads for the attention layer and D_H is the embedding size of each head.
The pytorch code seems to do exactly that. Here below I post the code for reshaping the query tensor (key, value are equally deemed)
q = q.contiguous().view(tgt_len, bsz * num_heads, head_dim).transpose(0, 1)

I don't get why they perform both a view and a transpose call, when the result would be the same by just doing
q = q.contiguous().view(bsz * num_heads, tgt_len, head_dim)

Other than avoiding an additional function call, using view alone also guarantees that the resulting tensor is still contiguous in memory, whereas this doesn't hold (to the best of my knowledge) for transpose. I suppose working with contiguous data is beneficial whenever possible to make computations potentially faster (may lead to fewer memory accesses, better exploiting of spatial locality of data, etc.).
What's the use case for having a transpose call after a view?

Output:
The results AREN'T necessarily the same:
a = torch.arange(0, 2 * 3 * 4)
b = a.view(2, 3, 4).transpose(1, 0)
#tensor([[[ 0,  1,  2,  3],
     [12, 13, 14, 15]],

    [[ 4,  5,  6,  7],
     [16, 17, 18, 19]],

    [[ 8,  9, 10, 11],
     [20, 21, 22, 23]]])

c = a.view(3, 2, 4)
#tensor([[[ 0,  1,  2,  3],
     [ 4,  5,  6,  7]],

    [[ 8,  9, 10, 11],
     [12, 13, 14, 15]],

    [[16, 17, 18, 19],
     [20, 21, 22, 23]]])


Instruction: How to solve AttributeError: 'Tensor' object has no attribute 'zero_grad' in pytorch

Input:
Still working through video tutorial https://www.youtube.com/watch?v=weQ5pShEVic&list=PLbMqOoYQ3Mxw1Sl5iAAV4SJmvnAGAhFvK&index=2 about pytorch but hit another error.
lossFunc = torch.nn.MSELoss()
for i in range(epoch):
        output = net(x)
        loss = lossFunc(output, y)
        loss.zero_grad()
        loss.backward()

        for f in net.parameters():
            f.data.sub_(learning_rate = f.grad.data)
print(output, loss)

Created the network, loss function and wanted to iterate before backpropogattion
but get this error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/var/folders/v_/yq26pm194xj5ckqy8p_njwc00000gn/T/ipykernel_9995/2476130544.py in <module>
      3         output = net(x)
      4         loss = lossFunc(output, y)
----> 5         loss.zero_grad()
      6         loss.backward()
      7

AttributeError: 'Tensor' object has no attribute 'zero_grad'

What gives?

Output:
You should use zero grad for your optimizer.
optimizer = torch.optim.Adam(net.parameters(), lr=0.001)
lossFunc = torch.nn.MSELoss()
for i in range(epoch):
    optimizer.zero_grad()
    output = net(x)
    loss = lossFunc(output, y)
    loss.backward()
    optimizer.step()


Instruction: Using GPU with fastai and pytorch

Input:
I am using fastai and pytorch for image classification. I tried to train it on google colab. But it takes much time to train it on colab and I think the problem is GPU is not set properly. This is how I did it.
import os
import torch
import torchvision as tv
import matplotlib.pyplot as plt
import numpy as np
from torchinfo import summary as torchinfo_summary

#cuda configs
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

#loading dataset
data_dir='/content/drive/MyDrive/tech_related/machine_learning_related/pytorch-etic/data/cat_deer_dog_horse'
#data_dir = os.path.join('..','data','cat_deer_dog_horse')
print(os.listdir(data_dir))
data_classes = os.listdir(os.path.join(data_dir,'train'))
print(data_classes)

#making torch defined dataloaders with dataset
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))  #means and stds of each channel in images of cifar10
train_tfms = tv.transforms.Compose([
    tv.transforms.RandomCrop(32,padding=4,padding_mode='reflect'),
    tv.transforms.RandomHorizontalFlip(),
    tv.transforms.ToTensor()
])
valid_tfms = tv.transforms.Compose([
    tv.transforms.ToTensor()
])

train_ds = tv.datasets.ImageFolder(os.path.join(data_dir,'train'),train_tfms)
valid_ds = tv.datasets.ImageFolder(os.path.join(data_dir,'test'),valid_tfms)
batch_size = 64
train_dl = torch.utils.data.DataLoader(train_ds,batch_size,shuffle=True,pin_memory=True)
valid_dl = torch.utils.data.DataLoader(valid_ds,batch_size,shuffle=True,pin_memory=True)

model_ = tv.models.mobilenet_v2(pretrained=False, num_classes=len(data_classes)).to(device)

#here comes the training part, finding optimum learning rate.
from fastai.vision.all import *
data = DataLoaders(train_dl,valid_dl)
learner = Learner(data, model_, loss_func=torch.nn.functional.cross_entropy, opt_func=Adam, metrics=accuracy)

lr_min,lr_steep,lr_slide,lr_valley = learner.lr_find(suggest_funcs=(minimum,steep,slide,valley))

I checked the runtime I used in colab and it is GPU. Can someone say what am I doing wrong?

Output:
It looks to me like you're using the GPU. You can confirm that you are by running next(learner.model.parameters).is_cuda immediately prior to lr_find.
What jumps out is that you are not loading pretrained weights (pretrained = False) in your model. This would naturally slow down the total time of training. Set pretrained to True and see if you get good results quicker.
Or, if you are concerned about the per-epoch time of training, the other issue might be if your datasets are very large. Large datasets are usually  a good problem to have and, even if the per-epoch time of training is high, the total wall time for training without overfitting is probably similar or better with large datasets.
Just to be thorough, lr_find is not used to actually train. For that, you should use fit or fit_one_cycle with the lr_steep value generated by lr_find.


Instruction: What is the error " TypeError: 'int' object is not callable"?

Input:
Substitute three numpy for the audio and combine them to get the max-min average. I am getting an error with this, what should I do?
import torch
import torchaudio
import torchaudio.transforms as T
import os
import requests
import librosa
import matplotlib.pyplot as plt
# 音声の保存
_SAMPLE_DIR = "_sample_data"
SAMPLE_WAV_URL = "https://pytorch-tutorial-assets.s3.amazonaws.com/VOiCES_devkit/source-16k/train/sp0307/Lab41-SRI-VOiCES-src-sp0307-ch127535-sg0042.wav"
SAMPLE_WAV_PATH = os.path.join(_SAMPLE_DIR, "speech.wav")
def plot_spectrogram(spec, title=None, ylabel="freq_bin", aspect="auto", xmax=None):
    fig, axs = plt.subplots(1, 1)
    axs.set_title(title or "Spectrogram (db)")
    axs.set_ylabel(ylabel)
    axs.set_xlabel("frame")
    im = axs.imshow(librosa.power_to_db(spec), origin="lower", aspect=aspect)
    if xmax:
        axs.set_xlim((0, xmax))
    fig.colorbar(im, ax=axs)
    plt.show(block=False)
def synthesis(sigList):
    maxLength = 0
    tmpLength = 0
    tmpArray = []
    #最大長の音声を探索する
    for i, data in enumerate(sigList):
        if len(data) > tmpLength:
            maxLength = len(data)
            tmpLength = len(data)
            index = i
    #最大長の音声の長さの０埋め配列を定義
    sig = np.zeros(maxLength)

    for i in sigList:
        tmp = i.tolist()    #numpy→list
        #全ての音声を最大長の音声に合わせて０埋めする
        for data in range(maxLength - len(i)):
            tmp.append(0)
        tmpArray.append(tmp)
    #配列３つを合成する
    sig = np.array(tmpArray[0]) + np.array(tmpArray[1]) + np.array(tmpArray[2])
    return sig
def min_max(x, axis=None):
    min = x.min(axis=axis, keepdims=True)
    max = x.max(axis=axis, keepdims=True)
    try:
        z = (x - min) / (max - min)
    except ZeroDivisionError:
        z = (x - min) / min
    return z
waveform, sample_rate = torchaudio.load(filepath=SAMPLE_WAV_URL)

n_fft = 1024
win_length = None
hop_length = 512
window_fn = torch.hann_window
waveforms = waveform.numpy()
k = waveforms
for i in range(2):
  waveforms = np.concatenate([waveforms,k],0)
spectrogram = T.Spectrogram(
    n_fft=n_fft,
    win_length=win_length,
    hop_length=hop_length,
    window_fn=window_fn,
    power=2.0,
)

sig  = min_max(synthesis(waveforms))

spec = spectrogram(sig)
plot_spectrogram(spec[0], title='torchaudio')

spec = spectrogram(sig) This is the line where the error occurs.
Detailed error is TypeError
Traceback (most recent call last)
<ipython-input-44-a0a6c4ba7770> in <module>
      70 sig = min_max(synthesis(waveforms))
      71
---> 72 spec = spectrogram(sig)
      73 plot_spectrogram(spec[0], title='torchaudio')

2 frames
/usr/local/lib/python3.7/dist-packages/torchaudio/functional/functional.py in spectrogram(waveform, pad, window, n_fft, hop_length, win_length, power, normalized, center, pad_mode, onesided, return_complex)
     106
     107# pack batch
--> 108 shape = waveform.size()
     109 waveform = waveform.reshape(-1, shape[-1])
     110

TypeError: 'int' object is not callable

Output:
According to the docs for Torchaudio Spectrogram, the parameter that's passed to its return value (spectrogram() in your code) needs to be a PyTorch Tensor. In your code, you're giving it a Numpy array instead, because that's what your function synthesis() returns.
You can convert a Numpy ndarray into a Tensor with torch.from_numpy. For example:
spec = spectrogram(torch.from_numpy(sig))


Instruction: Can PyTorch L-BFGS be used to optimize a complex parameter?

Input:
A brief description of my model:

Consists of a single parameter X of dtype ComplexDouble and shape (20, 20, 20, 3). For reference, this must be complex because I need to perform FFTs etc. on it
X is used to compute a real scalar value, Y as the output
The objective is to minimise the value of Y using autograd to optimize the value of X.

Simple gradient descent-based optimizers like torch.optim.SGD and torch.optim.Adam seem to work fine for this process. I would like to extend this to L-BFGS.
The problem is upon using
optimizer = optim.LBFGS(solver.parameters())

def closure():
    optimizer.zero_grad()
    Y = model.forward()
    Y.backward()
    return Y

for i in range(steps):
    optimizer.step(closure)

I get the error
File "xx\Python\Python38\lib\site-packages\torch\optim\lbfgs.py", line 410, in step
    if gtd > -tolerance_change:
RuntimeError: "gt_cpu" not implemented for 'ComplexDouble'

According to the source file, it's computing the directional derivative to be complex which disrupts the algorithm.
Is there any way to get L-BFGS working for my complex parameter (e.g. using an alternative library) or is this fundamentally impossible? I had some ideas about replacing these "faulty" dot products with something like real(a.conj() * b)) but I wasn't sure whether that would work.

Output: My intuition was correct. I replaced every occurence of a.dot(b) in the file with torch.real(a.conj().dot(b)) and L-BFGS is working great!


Instruction: Pytorch torch.linalg.qr is differentiable?

Input:
I have a neural network that involves the calculation of the QR decomposition of the input matrix X. Such a matrix is rectangular and it has maximal rank.
My question is if this operation still allows to make the gradients propagate backward during the training or if there can be some issue.

Output:
You are right in your suspicions. It is not always differentiable. Depending on the mode parameter, there are three different cases you need to consider.
From the documentation;

The parameter mode chooses between the full and reduced QR
decomposition. If A has shape (*, m, n), denoting k = min(m, n)
mode= ‘reduced’ (default): Returns (Q, R) of shapes (, m, k), (, k,
n) respectively. It is
always differentiable.
mode= ‘complete’: Returns (Q, R) of shapes (, m, m), (, m, n)
respectively. It is
differentiable for m <= n.
mode= ‘r’: Computes only the reduced R. Returns (Q, R) with Q empty
and R of shape (*, k, n).
It is never differentiable.


Instruction: Pytorch-Lightning ModelCheckpoint get paths of saved checkpoints

Input:
I am using PytorchLightning and beside others a ModelCheckpoint which saves models with a formated filename like `filename="model_{epoch}-{val_acc:.2f}"
In a process I want to load these checkpoints again, for simplicity lets say I want only the best via save_top_k=N.
As the filename is dynamic I wonder how can I retrieve the checkpoint easily is there a built in attribute or via the trainer that gives the saved checkpoints?
For example like
checkpoint_callback.get_top_k_paths()

I know I can do it with glob and model_dir but wondering if there is a one line solution built in somehwere.

Output:
you can retrieve the best model path after training from the checkpoint
# retrieve the best checkpoint after training
checkpoint_callback = ModelCheckpoint(dirpath='my/path/')
trainer = Trainer(callbacks=[checkpoint_callback])
model = ...
trainer.fit(model)
checkpoint_callback.best_model_path

To find all the checkpoints you can get the list of files in the dirpath where the checkpoints are saved.


Instruction: Why does PyGad fitness_function not work when inside of a class?

Input:
I am trying to train a genetic algorithm but for some reason it does not work when it's stored inside of a class. I have two equivalent pieces of code but the one stored inside of a class fails. It returns this..
raise ValueError("The fitness function must accept 2 parameters:
1) A solution to calculate its fitness value.
2) The solution's index within the population.

The passed fitness function named '{funcname}' accepts {argcount} parameter(s).".format(funcname=fitness_func.__code__.co_name, argcount=fitness_func.__code__.co_argcount))
ValueError: The fitness function must accept 2 parameters:
1) A solution to calculate its fitness value.
2) The solution's index within the population.

The passed fitness function named 'fitness_func' accepts 3 parameter(s).

Here is the simplified version of the one that doesnt work.
import torch
import torch.nn as nn
import pygad.torchga
import pygad

class NN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.linear1 = nn.Linear(input_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size, hidden_size)
        self.linear3 = nn.Linear(hidden_size, hidden_size)
        self.linear4 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        x = self.linear3(x)
        x = self.linear4(x)
        return x

class Coin:
    def __init__(self):
        self.NeuralNet = NN(1440, 1440, 3)

    def fitness_func(self, solution, solution_idx):
        return 0

    def trainModel(self):

        torch_ga = pygad.torchga.TorchGA(model=self.NeuralNet, num_solutions=10)

        ga_instance = pygad.GA(num_generations=10,
                               num_parents_mating=2,
                               initial_population=torch_ga.population_weights,
                               fitness_func=self.fitness_func)

        ga_instance.run()

if __name__ == "__main__":
    coin = Coin()
    coin.trainModel()

Here is the simplified version of the one that does work.
import torch
import torch.nn as nn
import pygad.torchga
import pygad

class NN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.linear1 = nn.Linear(input_size, hidden_size)
        self.linear2 = nn.Linear(hidden_size, hidden_size)
        self.linear3 = nn.Linear(hidden_size, hidden_size)
        self.linear4 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        x = self.linear3(x)
        x = self.linear4(x)
        return x

def fitness_func(solution, solution_idx):
    return 0

def trainModel():

    NeuralNet = NN(1440, 1440, 3)

    torch_ga = pygad.torchga.TorchGA(model=NeuralNet, num_solutions=10)

    ga_instance = pygad.GA(num_generations=10,
                            num_parents_mating=2,
                            initial_population=torch_ga.population_weights,
                            fitness_func=fitness_func)

    ga_instance.run()

if __name__ == "__main__":
    trainModel()

Both of these should work the same but they don't

Output:
When you look at the pygad code you can see it's explicitly checking that the fitness function has exactly two parameters:
        # Check if the fitness function accepts 2 paramaters.
        if (fitness_func.__code__.co_argcount == 2):
            self.fitness_func = fitness_func
        else:
            self.valid_parameters = False
            raise ValueError("The fitness function must accept 2 parameters:\n1) A solution to calculate its fitness value.\n2) The solution's index within the population.\n\nThe passed fitness function named '{funcname}' accepts {argcount} parameter(s).".format(funcname=fitness_func.__code__.co_name, argcount=fitness_func.__code__.co_argcount))

So if you want to use it in a class you'll need to make it a static method so you aren't required to pass in self:
@staticmethod
def fitness_func(solution, solution_idx):
    return 0


Instruction: Runtime error on WGan-gp algorithm when running on GPU

Input:
I am a newbie in pytorch  and running the WGan-gp algorithm on google colab using GPU runtime. I encountered the error below. The algorithm works fine when at None runtime i.e cpu.
Error generated during training
0%|          | 0/3 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-7e1d4849a60a> in <module>
     19             # Calculate gradient penalty on real and fake images
     20             # generated by generator
---> 21             gp = gradient_penalty(netCritic, real_image, fake, device)
     22             critic_loss = -(torch.mean(critic_real_pred)
     23                             - torch.mean(critic_fake_pred)) + LAMBDA_GP * gp

<ipython-input-15-f84354d74f37> in gradient_penalty(netCritic, real_image, fake_image, device)
      8     # image
      9     # interpolated image ← alpha *real image  + (1 − alpha) fake image
---> 10     interpolated_image = (alpha*real_image) + (1-alpha) * fake_image
     11
     12     # calculate the critic score on the interpolated image

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Snippet of my WGan-gp code
def gradient_penalty(netCritic, real_image, fake_image, device=device):

    batch_size, channel, height, width = real_image.shape

    # alpha is selected randomly between 0 and 1
    alpha = torch.rand(batch_size, 1, 1, 1).repeat(1, channel, height, width)
    # interpolated image=randomly weighted average between a real and fake
    # image
    # interpolated image ← alpha *real image  + (1 − alpha) fake image
    interpolated_image = (alpha*real_image) + (1-alpha) * fake_image

    # calculate the critic score on the interpolated image
    interpolated_score = netCritic(interpolated_image)

    # take the gradient of the score wrt to the interpolated image
    gradient = torch.autograd.grad(inputs=interpolated_image,
                                   outputs=interpolated_score,
                                   retain_graph=True,
                                   create_graph=True,
                                   grad_outputs=torch.ones_like
                                   (interpolated_score)
                                   )[0]
    gradient = gradient.view(gradient.shape[0], -1)
    gradient_norm = gradient.norm(2, dim=1)
    gradient_penalty = torch.mean((gradient_norm - 1)**2)
    return gradient_penalty

n_epochs = 2000
cur_step = 0
LAMBDA_GP = 10
display_step = 50
CRITIC_ITERATIONS = 5
nz = 100

for epoch in range(n_epochs):
    # Dataloader returns the batches
    for real_image, _ in tqdm(dataloader):
        cur_batch_size = real_image.shape[0]
        real_image = real_image.to(device)
        for _ in range(CRITIC_ITERATIONS):
            fake_noise = get_noise(cur_batch_size, nz, device=device)
            fake = netG(fake_noise)
            critic_fake_pred = netCritic(fake).reshape(-1)
            critic_real_pred = netCritic(real_image).reshape(-1)

            # Calculate gradient penalty on real and fake images
            # generated by generator
            gp = gradient_penalty(netCritic, real_image, fake, device)
            critic_loss = -(torch.mean(critic_real_pred)
                            - torch.mean(critic_fake_pred)) + LAMBDA_GP * gp
            netCritic.zero_grad()
            # To make a backward pass and retain the intermediary results
            critic_loss.backward(retain_graph=True)
            optimizerCritic.step()

        # Train Generator: max E[critic(gen_fake)] <-> min -E[critic(gen_fake)]
        gen_fake = netCritic(fake).reshape(-1)
        gen_loss = -torch.mean(gen_fake)
        netG.zero_grad()
        gen_loss.backward()
        # Update optimizer
        optimizerG.step()

        # Visualization code ##
        if cur_step % display_step == 0 and cur_step > 0:
            print(f"Step{cur_step}: GenLoss: {gen_loss}: CLoss: {critic_loss}")
            display_images(fake)
            display_images(real_image)
            gen_loss = 0
            critic_loss = 0
        cur_step += 1

I tried to introduce cuda() at the lines 10 and 21 indicated in the error output.But not working.

Output:
Here is one approach to solve this kind of error:

Read the error message and locate the exact line where it occured:
... in gradient_penalty(netCritic, real_image, fake_image, device)
      8     # image
      9     # interpolated image ← alpha *real image  + (1 − alpha) fake image
---> 10     interpolated_image = (alpha*real_image) + (1-alpha) * fake_image
     11
     12     # calculate the critic score on the interpolated image

RuntimeError: Expected all tensors to be on the same device,
              but found at least two devices, cuda:0 and cpu!

Look for input tensors that have not been properly transferred to the correct device. Then look for intermediate tensors that have not been transferred.
Here alpha is assigned to a random tensor but no transfer is done!
>>> alpha = torch.rand(batch_size, 1, 1, 1) \
                 .repeat(1, channel, height, width)

Fix the issue and test:
>>> alpha = torch.rand(batch_size, 1, 1, 1, device=fake_image.device) \
                 .repeat(1, channel, height, width)


Instruction: The shuffling order of DataLoader in pytorch

Input:
I am really confused about the shuffle order of DataLoader in pytorch.
Supposed I have a dataset:
datasets = [0,1,2,3,4]

In scenario I, the code is:
torch.manual_seed(1)

G = torch.Generator()
G.manual_seed(1)

ran_sampler = RandomSampler(data_source=datasets,generator=G)
dataloader = DataLoader(dataset=datasets,sampler=ran_sampler)

the shuffling result is 0,4,2,3,1.

In scenario II, the code is:
torch.manual_seed(1)

G = torch.Generator()
G.manual_seed(1)

ran_sampler = RandomSampler(data_source=datasets)
dataloader = DataLoader(dataset=datasets, sampler=ran_sampler, generator=G)

the shuffling result is 1,3,4,0,2.

In scenario III, the code is:
torch.manual_seed(1)

G = torch.Generator()
G.manual_seed(1)

ran_sampler = RandomSampler(data_source=datasets, generator=G)
dataloader = DataLoader(dataset=datasets, sampler=ran_sampler, generator=G)

the shuffling result is 4,1,3,0,2.
Can someone explain what is going on here?

Output:
Based on your code, I did a little modification (on scenario II) and inspection:
datasets = [0,1,2,3,4]

torch.manual_seed(1)
G = torch.Generator()
G = G.manual_seed(1)

ran_sampler = RandomSampler(data_source=datasets, generator=G)
dataloader = DataLoader(dataset=datasets, sampler=ran_sampler)
print(id(dataloader.generator)==id(dataloader.sampler.generator))
xs = []
for x in dataloader:
    xs.append(x.item())
print(xs)

torch.manual_seed(1)
G = torch.Generator()
G.manual_seed(1)

# this is different from OP's scenario II because in that case the ran_sampler is not initialized with the right generator.
dataloader = DataLoader(dataset=datasets, shuffle=True, generator=G)
print(id(dataloader.generator)==id(dataloader.sampler.generator))
xs = []
for x in dataloader:
    xs.append(x.item())
print(xs)

torch.manual_seed(1)
G = torch.Generator()
G.manual_seed(1)

ran_sampler = RandomSampler(data_source=datasets, generator=G)
dataloader = DataLoader(dataset=datasets, sampler=ran_sampler, generator=G)
print(id(dataloader.generator)==id(dataloader.sampler.generator))
xs = []
for x in dataloader:
    xs.append(x.item())
print(xs)

The outputs are:
False
[0, 4, 2, 3, 1]
True
[4, 1, 3, 0, 2]
True
[4, 1, 3, 0, 2]

The reason why the above three seemingly equivalent setups lead to different outcomes is that there are two different generators actually being used inside the DataLoader, one of which is None, in the first scenario.
To make it clear, let's analyze the source. It seems that the generator not only decides the random number generation of the _index_sampler inside DataLoader but also affects the initialization of _BaseDataLoaderIter. See the source code
        if sampler is None:  # give default samplers
            if self._dataset_kind == _DatasetKind.Iterable:
                # See NOTE [ Custom Samplers and IterableDataset ]
                sampler = _InfiniteConstantSampler()
            else:  # map-style
                if shuffle:
                    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
                else:
                    sampler = SequentialSampler(dataset)  # type: ignore[arg-type]

and
        self.sampler = sampler
        self.batch_sampler = batch_sampler
        self.generator = generator

and
    def _get_iterator(self) -> '_BaseDataLoaderIter':
        if self.num_workers == 0:
            return _SingleProcessDataLoaderIter(self)
        else:
            self.check_worker_number_rationality()
            return _MultiProcessingDataLoaderIter(self)

and
class _BaseDataLoaderIter(object):
    def __init__(self, loader: DataLoader) -> None:
        ...
        self._index_sampler = loader._index_sampler

Scenario II & Scenario III

Both setups are equivalent. We pass a generator to DataLoader and do not specify the sampler. DataLoader automatically creates a RandomSampler object with the generator and assign the same generator to self.generator.

Scenario I

We pass a sampler to DataLoader with the right generator but do not explicitly specify the keyword argument generator in DataLoader.__init__(...). DataLoader initializes the sampler with the given sampler but uses the default generator None for self.generator and the _BaseDataLoaderIter object returned by self._get_iterator().


Instruction: mini-batch gradient descent, loss doesn't improve and accuracy very low

Input:
I’m trying to implement mini-batch gradient descent on the popular iris dataset, but somehow I don’t manage to get the accuracy of the model above 75-80%. Also the loss does not decrease and is rather stuck at around 0.45, even when I set the number of iterations to 10000.
Something im missing here ?
class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear_stack =  nn.Sequential(
         nn.Linear(4,128),
         nn.ReLU(),
         nn.Linear(128,64),
         nn.ReLU(),
         nn.Linear(64,3),
         )
    def forward(self, x):
        logits = self.linear_stack(x)
        return logits

training loop, batchsize per epoch = 10.
transform_label maps [0,1,2] to the labels.
lr = 0.01
model = NeuralNetwork()
optim = torch.optim.Adam(model.parameters(), lr=lr)
loss = torch.nn.CrossEntropyLoss()

n_iters = 1000
steps = n_iters/10
LOSS = []
for epochs in range(n_iters):
    for i,(inputs, labels) in enumerate(train_loader):
        out = model(inputs)
        train_labels = transform_label(labels)
        l = loss(out, train_labels)
        l.backward()
        #update weights
        optim.step()
        optim.zero_grad()
    LOSS.append(l.item())
    if epochs%steps == 0:
        print(f"\n epoch: {int(epochs+steps)}/{n_iters}, loss: {sum(LOSS)/len(LOSS)}")
        #if i % 1 == 0:
            #print(f" steps: {i+1}, loss : {l.item()}")

output:

epoch: 100/1000, loss: 1.0636296272277832

epoch: 400/1000, loss: 0.5142968013338076

epoch: 500/1000, loss: 0.49906910391073867

epoch: 900/1000, loss: 0.4586030915751588

epoch: 1000/1000, loss: 0.4543738731996598

Is it possible to calculate the loss like that or should I use torch.max()? If I do so I get this Error:

Expected floating point type for target with class probabilities, got Long

Output:
you didn't provide enough data and code to reproduce the problem. I wrote a complete and working code to train your model on the IRIS dataset.
Imports and Classes.
import torch
from torch import nn
import pandas as pd
from torch.utils.data import Dataset, DataLoader
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.metrics import classification_report

class MyDataset(Dataset):
    def __init__(self, X, Y):
        assert len(X) == len(Y)
        self.X = X
        self.Y = Y

    def __len__(self):
        return len(self.X)

    def __getitem__(self, item):
        x = self.X[item]
        y = self.Y[item]
        return x, y

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear_stack =  nn.Sequential(
         nn.Linear(4,128),
         nn.ReLU(),
         nn.Linear(128,64),
         nn.ReLU(),
         nn.Linear(64,3),
         )
    def forward(self, x):
        logits = self.linear_stack(x)
        return logits

Read and Preprocess the data.
# Dataset was downloaded from https://archive.ics.uci.edu/ml/machine-learning-databases/iris/
df = pd.read_csv("iris.data", names=["x1", "x2", "x3", "x4", "label"])
X, Y = df[['x1', "x2", "x3", "x4"]], df['label']

# First, we transform the labels to numbers 0,1,2
encoder = LabelEncoder()
Y = encoder.fit_transform(Y)

# We split the dataset to train and test
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2, random_state=123)

# Due to the nature of Neural Networks, we standardize the inputs to get better results
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

train_dataset = MyDataset(X_train, Y_train)
test_dataset = MyDataset(X_test, Y_test)

train_loader = DataLoader(train_dataset, batch_size=8)
test_loader = DataLoader(test_dataset, batch_size=8)

Train the model.
lr = 0.01
model = NeuralNetwork()
optim = torch.optim.Adam(model.parameters(), lr=lr)
loss = torch.nn.CrossEntropyLoss()

n_iters = 1000
steps = n_iters/10
LOSS = []

for epochs in range(n_iters):
    for i,(inputs, labels) in enumerate(train_loader):
        optim.zero_grad()
        out = model(inputs.float())
        l = loss(out, labels)
        l.backward()
        optim.step()
    LOSS.append(l.item())
    if epochs%steps == 0:
        print(f"\n epoch: {int(epochs+steps)}/{n_iters}, loss: {sum(LOSS)/len(LOSS)}")

output:

Then, we need to run the model on test data to calculate the metrics.
preds = []
with torch.no_grad():
    for i,(inputs, labels) in enumerate(test_loader):
        out = model(inputs.float())
        preds.extend(list(torch.argmax(out, axis=1).cpu().numpy()))

To get the metrics, you can use "classification_report".
print(classification_report(y_true=Y_test, y_pred=preds))

output:

I hope my answer helps you.


Instruction: PyTorch vectorized sum different from looped sum

Input:
I am using torch 1.7.1 and I noticed that vectorized sums are different from sums in a loop if the indices are repeated. For example:
import torch

indices = torch.LongTensor([0,1,2,1])
values = torch.FloatTensor([1,1,2,2])
result = torch.FloatTensor([0,0,0])

looped_result = torch.zeros_like(result)

for i in range(indices.shape[0]):
    looped_result[indices[i]] += values[i]

result[indices] += values

print('result:',result)
print('looped result:', looped_result)

results in:
 result tensor: ([1., 2., 2.])
 looped result tensor: ([1., 3., 2.])

As you can see the looped variable has the correct sums while the vectorized one doesn’t. Is it possible to avoid the loop and still get the correct result?

Output:
The issue here is that you're indexing result multiple times at the same index, which is bound to fail for this inplace operation. Instead what you'd need to use is index_add or index_add_, e.g. (as a continuation of your snippet):
>>> result_ia = torch.zeros_like(result)
>>> result_ia.index_add_(0, indices, values)
tensor([1., 3., 2.]


Instruction: Torch: Input type and weight type (torch.cuda.FloatTensor) should be the same

Input:
Note: I have already seen similar questions: the same error, tell torch not to use GPU, but the answers do not work for me.
I have installed PyTorch version 1.13.0+cu117 (the latest), and the code structure is as follows (an image classification task):
# os.environ["CUDA_VISIBLE_DEVICES"]=""    # required?
device = torch.device("cpu")               # use CPU
...
train_set = DataLoader(
    torchvision.datasets.ImageFolder(path, transform), **kwargs
)
...
model = myCNN().to(device)
optimizer = SGD(args)
loss = CrossEntropyLoss()

train()

I want to train on CPU.
For dataloader, in accordance to this, I've set pin_memory=True and non_blocking=pin_memory. The error persists even on setting pin_memory=False.
The training loop has the following structure:
for epoch in n_epochs:
    model.train()
    inputs, labels = inputs.to(device, non_blocking=non_blocking), labels.to(device, non_blocking=non_blocking)
    Compute loss, back-propagate

The error traceback (on calling train()):
Traceback (most recent call last):
  File "code.py", line 233, in <module>
    train()
  File "code.py", line 122, in train
    outputs = model(inputs)
  File "...\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "code.py", line 87, in forward
    output = self.network(input)
  File "...\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "...\torch\nn\modules\container.py", line 204, in forward
    input = module(input)
  File "...\torch\nn\modules\module.py", line 1190, in _call_impl
    return forward_call(*input, **kwargs)
  File "...\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "...\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

Edit: There was a comment regarding possible issues due to the model. The model is roughly:
class myCNN(nn.Module):
    def __init__(self, ...other args...):
        super().__init__()

        self.network = nn.Sequential(

        nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size),

        ... similar convolutional layers ...

        nn.Flatten(),
        nn.Linear(in_features, out_features)

        )

    def forward(self, input):
        output = self.network(input)
        return output

Since I have transferred both model and data to the same device, what could be the reason of this error? How to correct it?

Output:
The issue was due to incorrect usage of summary from torchinfo. It does a forward pass (if input size is provided), and the device is (by default) selected on basis of torch.cuda.is_available().
If device (as specified in the question) argument is given to summary, the training happens just fine.


Instruction: manual_seed for mps devices in pytorch

Input:
How do I set up a manual seed for mps devices using pytorch?
With cuda devices the code should work like this:
if torch.cuda.is_available():
  torch.cuda.manual_seed(0)

I'm using an apple m1 chip. The following statement returns True:
torch.backends.mps.is_available()

But following statement is not possible:
torch.backends.mps.manual_seed(0)

Output:
According to the docs:

You can use torch.manual_seed() to seed the RNG for all devices (both CPU and CUDA):

import torch
torch.manual_seed(0)


Instruction: ERROR in CNN Pytorch; shape '[-1, 192]' is invalid for input of size 300000

Input:
I want to change kernal size to 3, output channels of convolutional layers to 8 and 16 respectively. But when i try to change it i get an error message The following code is working fine but when I change kernal size and output channels like this:
    self.conv1 = nn.Conv2d(in_channels=1,out_channels=**8**,kernel_size=**3**)
    self.conv2 = nn.Conv2d(in_channels=**8**,out_channels=**16**,kernel_size=**3**)
    self.fc1 = nn.Linear(in_features=**16*2*2**,out_features=128)

It generate an error for invalid input size.
working code

class Network(nn.Module):
  def __init__(self):
    super(Network,self).__init__()
    self.conv1 = nn.Conv2d(in_channels=1,out_channels=6,kernel_size=5)
    self.conv2 = nn.Conv2d(in_channels=6,out_channels=12,kernel_size=5)
    self.fc1 = nn.Linear(in_features=12*4*4,out_features=128)
    self.fc2 = nn.Linear(in_features=128,out_features=64)
    self.out = nn.Linear(in_features=64,out_features=10)
  def forward(self,x):
    #input layer
    x = x
    #first hidden layer
    x = self.conv1(x)
    x = F.relu(x)
    x = F.max_pool2d(x,kernel_size=2,stride=2)
    #second hidden layer
    x = self.conv2(x)
    x = F.relu(x)
    x = F.max_pool2d(x,kernel_size=2,stride=2)
    #third hidden layer
    x = x.reshape(-1,12*4*4)
    x = self.fc1(x)
    x = F.relu(x)
    #fourth hidden layer
    x = self.fc2(x)
    x = F.relu(x)

    #output layer
    x = self.out(x)
    return x

batch_size = 1000
train_dataset = FashionMNIST(
    '../data', train=True, download=True,
    transform=transforms.ToTensor())
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = FashionMNIST(
    '../data', train=False, download=True,
    transform=transforms.ToTensor())
testloader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

model = Network()

losses = []
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())
epochs = 1

for i in range(epochs):
    batch_loss = []
    for j, (data, targets) in enumerate(trainloader):
        optimizer.zero_grad()
        ypred = model(data)
        loss = criterion(ypred, targets.reshape(-1))
        loss.backward()
        optimizer.step()
        batch_loss.append(loss.item())
    if i>10:
        optimizer.lr = 0.0005
    losses .append(sum(batch_loss) / len(batch_loss))
    print('Epoch {}:\tloss {:.4f}'.format(i, losses [-1]))

Output:
By changing your kernel size and output size in intermediate filters, you also change the size of your intermediate activations.
I suppose your input data is of size (1,28,28) (the usual size for FashionMNIST).
In your original code, before the layer self.fc1, after two 2D convolutionnal layers and two maxpools, the shape of your activations will be (12, 4, 4). However, if you change your kernel size to 3 and output channels of convolutional layers to 8 and 16, this shape will change. It will now be (16, 5, 5). Thus, you have to change your network. Try the following:
class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1,out_channels=8,kernel_size=3)
        self.conv2 = nn.Conv2d(in_channels=8,out_channels=16,kernel_size=3)
        self.fc1 = nn.Linear(in_features=16*5*5,out_features=128)
        self.fc2 = nn.Linear(in_features=128,out_features=64)
        self.out = nn.Linear(in_features=64,out_features=10)

    def forward(self,x):
        #input layer
        x = x

        #first hidden layer
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x,kernel_size=2,stride=2)

        #second hidden layer
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x,kernel_size=2,stride=2)

        #third hidden layer
        x = x.reshape(-1,16*5*5)
        x = self.fc1(x)
        x = F.relu(x)

        #fourth hidden layer
        x = self.fc2(x)
        x = F.relu(x)

        #output layer
        x = self.out(x)
        return x

Check Pytorch's documentation for the Conv2D and Maxpool layers.
The output size after a Conv2D layer is:
H_out = ⌊[H_in + 2*padding[0] − dilation[0]*(kernel_size[0]−1)−1]/stride[0] +1⌋

W_out = ⌊[W_in + 2*padding[1] - dilation[1]*(kernel_size[1]-1)-1]/stride[1] +1⌋

As you use the default values, the output size  after the first convolutionnal layer will be :
H_out = W_out = 28+0-2-1+1=26

The maxpool following will divide this size by 2, and after the second convolutionnal layer the size will be:
13+0-2-1+1=11

The second maxpool will divide this by 2 again, taking the floor value, which is 5. Thus, the output shape after the second layer will be (n, 16, 5, 5). Before the first fully connected layer, this has to be flattened. This is why the input features of self.fc1 is 16*5*5.


Instruction: Memory issue in running multiple processes on GPU

Input:
This question can be viewed related to my other question.
I tried running multiple machine learning processes in parallel (with bash). These are written using PyTorch. After a certain number of concurrent programs (10 in my case), I get the following error:
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

As mentioned in this answer,

...it could occur because the VRAM memory limit was hit (which is rather non-intuitive from the error message).
For my case with PyTorch model training, decreasing batch size helped. You could try this or maybe decrease your model size to consume less VRAM.

I tried the solution mentioned here, to enforce a per-process GPU memory usage limit, but this issue persists.
This problem does not occur with a single process, or a fewer number of processes. Since only one context runs at a single time instant, why does this cause memory issue?
This issue occurs with/without MPS. I thought it could occur with MPS, but not otherwise, as MPS may run multiple processes in parallel.

Output:
Since only one context runs at a single time instant, why does this cause memory issue?

Context-switching doesn't dump the contents of GPU "device" memory (i.e. DRAM) to some other location.  If you run out of this device memory, context switching doesn't alleviate that.
If you run multiple processes, the memory used by each process will add up (just like it does in the CPU space) and GPU context switching (or MPS or time-slicing) does not alleviate that in any way.
It's completely expected that if you run enough processes using the GPU, eventually you will run out of resources.  Neither GPU context switching nor MPS nor time-slicing in any way affects the memory utilization per process.


Instruction: how to change the out_features of densenet121 model?

Input:
How to change the out_features of densenet121 model?
I am using the code below to train the model:
from torch.nn.modules.dropout import Dropout

class Densnet121(nn.Module):
    def __init__(self):
        super(Densnet121, self).__init__()
        self.cnn1 = nn.Conv2d(in_channels=3 , out_channels=64 , kernel_size=3 , stride=1 )
        self.Densenet_121 = models.densenet121(pretrained=True)
        self.gap = AvgPool2d(kernel_size=2, stride=1, padding=1)
        self.bn1 = nn.BatchNorm2d(1024)
        self.do1 = nn.Dropout(0.25)
        self.linear = nn.Linear(256,256)
        self.bn2 = nn.BatchNorm2d(256)
        self.do2 = nn.Dropout(0.25)
        self.output = nn.Linear(64 * 64 * 64,2)
        self.act = nn.ReLU()

    def densenet(self):
        for param in self.Densenet_121.parameters():
            param.requires_grad = False
        self.Densenet_121.classifier = nn.Linear(1024, 1024)
        return self.Densenet_121

    def forward(self, x):
        img = self.act(self.cnn1(x))
        img = self.densenet(img)

        img = self.gap(img)
        img = self.bn1(img)
        img = self.do1(img)
        img = self.linear(img)
        img = self.bn2(img)
        img = self.do2(img)
        img = torch.flatten(img, 1)
        img = self.output(img)

        return img

When training this model, I face the following error:

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 64, 62, 62] to have 3 channels, but got 64 channels instead

Output:
Your first conv layer outputs a tensor of shape (b, 64, h, w) while the following layer, the densenet model expects 3 channels. Hence the error that was raised:

"expected input [...] to have 3 channels, but got 64 channels instead"

Unfortunately, this value is hardcoded in the source of the Densenet class, see reference.
One workaround however is to overwrite the first convolutional layer after the densenet has been initialized. Something like this should work:
# First gather the conv layer specs
conv = self.Densenet_121.features.conv0
kwargs = {k: getattr(conv, k) for k in
   ('out_channels', 'stride', 'kernel_size', 'padding', 'bias')}

# overwrite with identical specs with new in_channels
model.features.conv0 = nn.Conv2d(in_channels=64, **kwargs)

Alternatively, you can do:
w = model.features.conv0.weight
w.data = torch.rand(len(w), 64, *w.shape[:2])

Which replaces the underlying convolutional layer weight without affecting its metadata (eg. conv.in_channels remains equal to 3), this could have side effects. So I would recommend following the first approach.


Instruction: Pytorch's share_memory_() vs built-in Python's shared_memory: Why in Pytorch we don't need to access the shared memory-block?

Input:
Trying to learn about the built-in multiprocessing and Pytorch's multiprocessing packages, I have observed a different behavior between both. I find this to be strange since Pytorch's package is fully-compatible with the built-in package.
Concretely, I'm refering to the way variables are shared between processes. In Pytorch, tensor's are moved to shared_memory via the inplace operation share_memory_(). On the other hand, we can get the same result with the built-in package by using the shared_memory module.
The difference between both that I'm struggling to understand is that, with the built-in version, we have to explicitely access the shared memory-block inside the launched process. However, we don't need to do that with the Pytorch version.
Here is a Pytorch's toy example showing this:
import time

import torch
# the same behavior happens when importing:
# import multiprocessing as mp
import torch.multiprocessing as mp

def get_time(s):
    return round(time.time() - s, 1)

def foo(a):
    # wait ~1sec to print the value of the tensor.
    time.sleep(1.0)
    with lock:
        #-------------------------------------------------------------------
        # WITHOUT explicitely accessing the shared memory block, we can observe
        # that the tensor has changed:
        #-------------------------------------------------------------------
        print(f"{__name__}\t{get_time(s)}\t\t{a}")

# global variables.
lock = mp.Lock()
s = time.time()

if __name__ == '__main__':
    print("Module\t\tTime\t\tValue")
    print("-"*50)

    # create tensor and assign it to shared memory.
    a = torch.zeros(2).share_memory_()
    print(f"{__name__}\t{get_time(s)}\t\t{a}")

    # start child process.
    p0 = mp.Process(target=foo, args=(a,))
    p0.start()

    # modify the value of the tensor after ~0.5sec.
    time.sleep(0.5)
    with lock:
        a[0] = 1.0

    print(f"{__name__}\t{get_time(s)}\t\t{a}")
    time.sleep(1.5)

    p0.join()

which outputs (as expected):
Module          Time            Value
--------------------------------------------------
__main__        0.0             tensor([0., 0.])
__main__        0.5             tensor([1., 0.])
__mp_main__     1.0             tensor([1., 0.])

And here is a toy example with the built-in package:
import time
import multiprocessing as mp
from multiprocessing import shared_memory

import numpy as np

def get_time(s):
    return round(time.time() - s, 1)

def foo(shm_name, shape, type_):
    #-------------------------------------------------------------------
    # WE NEED TO explicitely access the shared memory block to observe
    # that the array has changed:
    #-------------------------------------------------------------------
    existing_shm = shared_memory.SharedMemory(name=shm_name)
    a = np.ndarray(shape, type_, buffer=existing_shm.buf)

    # wait ~1sec to print the value.
    time.sleep(1.0)
    with lock:
        print(f"{__name__}\t{get_time(s)}\t\t{a}")

# global variables.
lock = mp.Lock()
s = time.time()

if __name__ == '__main__':
    print("Module\t\tTime\t\tValue")
    print("-"*35)

    # create numpy array and shared memory block.
    a = np.zeros(2,)
    shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
    a_shared = np.ndarray(a.shape, a.dtype, buffer=shm.buf)
    a_shared[:] = a[:]
    print(f"{__name__}\t{get_time(s)}\t\t{a_shared}")

    # start child process.
    p0 = mp.Process(target=foo, args=(shm.name, a.shape, a.dtype))
    p0.start()

    # modify the value of the vaue after ~0.5sec.
    time.sleep(0.5)
    with lock:
        a_shared[0] = 1.0

    print(f"{__name__}\t{get_time(s)}\t\t{a_shared}")
    time.sleep(1.5)

    p0.join()

which equivalently outputs, as expected:
Module          Time            Value
-----------------------------------
__main__        0.0             [0. 0.]
__main__        0.5             [1. 0.]
__mp_main__     1.0             [1. 0.]

So what I'm strugging to understand is why we don't need to follow the same steps in both versions, built-in and Pytorch's, i.e. how Pytorch is able to avoid the need to explicitely access the shared memory-block?
P.S. I'm using a Windows OS and Python 3.9

Output:
pytorch has a simple wrapper around shared memory, python's shared memory module is only a wrapper around the underlying OS dependent functions.
the way it can be done is that you don't serialize the array or the shared memory themselves, and only serialize what's needed to create them by using the __getstate__ and __setstate__ methods from the docs, so that your object acts as both a proxy and a container at the same time.
the following bar class can double for a proxy and a container this way, which is useful if the user shouldn't have to worry about the shared memory part.
import time
import multiprocessing as mp
from multiprocessing import shared_memory
import numpy as np

class bar:
    def __init__(self):
        self._size = 10
        self._type = np.uint8
        self.shm = shared_memory.SharedMemory(create=True, size=self._size)
        self._mem_name = self.shm.name
        self.arr = np.ndarray([self._size], self._type, buffer=self.shm.buf)

    def __getstate__(self):
        """Return state values to be pickled."""
        return (self._mem_name, self._size, self._type)

    def __setstate__(self, state):
        """Restore state from the unpickled state values."""
        self._mem_name, self._size, self._type = state
        self.shm = shared_memory.SharedMemory(self._mem_name)
        self.arr = np.ndarray([self._size], self._type, buffer=self.shm.buf)

def get_time(s):
    return round(time.time() - s, 1)

def foo(shm, lock):
    # -------------------------------------------------------------------
    # without explicitely access the shared memory block we observe
    # that the array has changed:
    # -------------------------------------------------------------------
    a = shm

    # wait ~1sec to print the value.
    time.sleep(1.0)
    with lock:
        print(f"{__name__}\t{get_time(s)}\t\t{a.arr}")

# global variables.
s = time.time()

if __name__ == '__main__':
    lock = mp.Lock()  # to work on windows/mac.

    print("Module\t\tTime\t\tValue")
    print("-" * 35)

    # create numpy array and shared memory block.
    a = bar()
    print(f"{__name__}\t{get_time(s)}\t\t{a.arr}")

    # start child process.
    p0 = mp.Process(target=foo, args=(a, lock))
    p0.start()

    # modify the value of the vaue after ~0.5sec.
    time.sleep(0.5)
    with lock:
        a.arr[0] = 1.0

    print(f"{__name__}\t{get_time(s)}\t\t{a.arr}")
    time.sleep(1.5)

    p0.join()

python just makes it much easier to hide such details inside the class without bothering the user with such details.
Edit: i wish they'd make locks non-inheritable so your code can raise an error on the lock, instead you'll find out one day that it doesn't actually lock ... After it crashes your application in production.


Instruction: Conv1d Pytorch on the columns of a matrix

Input:
I would like to compute, in the most efficient way, the following operation:
cc = nn.Conv1d(1,1,3,1,1,bias=False)
U = torch.randn(1,1,10,10)
V = torch.zeros_like(U)
for i in range(10):
    V[:,:,:,i] = cc(U[:,:,:,i])

In other words, I would like to apply a 1d convolution on the columns of the input U. Of course the for loop is too slow and I am sure there is a more efficient way to solve the problem. However, I can not find any idea to achieve that.

Output:
You can apply a 2D convolution with a rectangular shape window ie. 1x3:
conv = nn.Conv2d(1,1, kernel_size=(1,3), padding=1, bias=False)