Instruction: Scipy minimize.optimize LBFGS vs PyTorch LBFGS

Input:
I have written some code with scipy.optimize.minimize using the LBFGS algorithm. Now I want to implement the same with PyTorch.
SciPy:
res = minimize(calc_cost, x_0, args = const_data, method='L-BFGS-B', jac=calc_grad)
def calc_cost(x, const_data):
   # do some calculations with array "calculation" as result
   return np.sum(np.square(calculation)) #this returns a scalar!
def calc_grad(x, const_data):
   # do some calculations which result in array "calculation"
   return np.ravel(calculation) #in PyTorch this returns without ravel!

Now in PyTorch I am following this example. However, I want to use my own gradient calculation. This results in the error RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([3, 200, 200]) and output[0] has a shape of torch.Size([]). I understand that the shape/size of my gradient should be the same as the objective function (i.e. here a scalar), but this is not what I need (see above). How do I adapt the following code in a way that it does the same calculations as the SciPy version:
optimizer = optim.LBFGS([x_0], history_size=10, max_iter=10, line_search_fn="strong_wolfe")
h_lbfgs = []
for i in range(10):
    optimizer.zero_grad()
    objective = calc_cost(x_0, const_data)
    objective.backward(gradient = calc_gradient(x_0, const_data))
    optimizer.step(lambda: calc_cost(x_0, const_data))
    h_lbfgs.append(objective.item())

I have had a look at the PyTorch docs already but don't quite understand how they apply here:

https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html
https://pytorch.org/docs/stable/optim.html#optimizer-step-closure

Output:
The problem is that I was using the wrong "objective" function. What I am trying to optimize is the x_0 array, therefore I had to alter my code as follows:
for i in range(10):
   optimizer.zero_grad()
   x_0.backward(gradient = calc_gradient(x_0, const_data))
   optimizer.step(lambda: calc_cost(x_0, const_data))
   h_lbfgs.append(objective.item())


Instruction: Why is my pytorch Autoencoder giving me a "mat1 and mat2 shapes cannot be multiplied" error?

Input:
I know this is because the shapes don't match for the multiplication, but why when my code is similar to most example code I found:
import torch.nn as nn
...
#input is a 256x256 image
num_input_channels = 3
self.encoder = nn.Sequential(
            nn.Conv2d(num_input_channels*2**0, num_input_channels*2**1, kernel_size=3, padding=1, stride=2), #1 6 128 128
            nn.Tanh(),
            nn.Conv2d(num_input_channels*2**1, num_input_channels*2**2, kernel_size=3, padding=1, stride=2), #1 12 64 64
            nn.Tanh(),
            nn.Conv2d(num_input_channels*2**2, num_input_channels*2**3, kernel_size=3, padding=1, stride=2), #1 24 32 32
            nn.Tanh(),
            nn.Conv2d(num_input_channels*2**3, num_input_channels*2**4, kernel_size=3, padding=1, stride=2), #1 48 16 16
            nn.Tanh(),
            nn.Conv2d(num_input_channels*2**4, num_input_channels*2**5, kernel_size=3, padding=1, stride=2), #1 96 8 8
            nn.Tanh(),
            nn.Conv2d(num_input_channels*2**5, num_input_channels*2**6, kernel_size=3, padding=1, stride=2), #1 192 4 4
            nn.LeakyReLU(),
            nn.Conv2d(num_input_channels*2**6, num_input_channels*2**7, kernel_size=3, padding=1, stride=2), #1 384 2 2
            nn.LeakyReLU(),
            nn.Conv2d(num_input_channels*2**7, num_input_channels*2**8, kernel_size=2, padding=0, stride=1), #1 768 1 1
            nn.LeakyReLU(),
            nn.Flatten(),
            nn.Linear(768, 1024*32),
            nn.ReLU(),
            nn.Linear(1024*32, 256),
            nn.ReLU(),
        ).cuda()

I get the error "RuntimeError: mat1 and mat2 shapes cannot be multiplied (768x1 and 768x32768)"
To my understanding I should end up with a Tensor of shape [1,768,1,1] after the convolutions
and [1,768] after flattening, so I can use a fully connected Linear layer that goes to 1024*32 in size (by which I tried to add some more ways for the neural net to store data/knowledge).
Using nn.Linear(1,1024*32) runs with a warning later: "UserWarning: Using a target size (torch.Size([3, 256, 256])) that is different to the input size (torch.Size([768, 3, 256, 256]))". I think it comes from my decoder, though
What am I not understanding correctly here?

Output:
All torch.nn Modules require batched inputs, and it seems in your case you have no batch dimension. Without knowing your code I'm assuming you are using
my_input.shape == (3, 256, 256)

But you will need to add a batch dimension, that is, you need to have
my_input.shape == (1, 3, 256, 256)

You can easily do that by introducing a dummy dimension using:
my_input = my_input[None, ...]


Instruction: Command Line stable diffusion runs out of GPU memory but GUI version doesn't

Input:
I installed the GUI version of Stable Diffusion here. With it I was able to make 512 by 512 pixel images using my GeForce RTX 3070 GPU with 8 GB of memory:

However when I try to do the same thing with the command line interface, I run out of memory:
Input:
>> C:\SD\stable-diffusion-main>python scripts/txt2img.py --prompt "a close-up portrait of a cat by pablo picasso, vivid, abstract art, colorful, vibrant" --plms --n_iter 3 --n_samples 1 --H 512 --W 512
Error:
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 8.00 GiB total capacity; 6.13 GiB already allocated; 0 bytes free; 6.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
If I reduce the size of the image to 256 X 256, it gives a result, but obviously much lower quality.
So part 1 of my question is why do I run out of memory at 6.13 GiB when I have 8 GiB on the card, and part 2 is what does the GUI do differently to allow 512 by 512 output? Is there a setting I can change to reduce the load on the GPU?
Thanks a lot,
Alex

Output:
This might not be the only answer, but I solved it by using the optimized version here. If you already have the standard version installed, just copy the "OptimizedSD" folder into your existing folders, and then run the optimized txt2img script instead of the original:
>> python optimizedSD/optimized_txt2img.py --prompt "a close-up portrait of a cat by pablo picasso, vivid, abstract art, colorful, vibrant" --H 512 --W 512 --seed 27 --n_iter 2 --n_samples 10 --ddim_steps 50
It's quite slow on my computer, but produces 512 X 512 images!
Thanks,
Alex


Instruction: What exactly is meant by param_groups in pytorch?

Input:
I would like to update learning rates corresponding to each weight matrix and each bias in pytorch during training. The answers here and here and many other answers I found online talk about doing this using the model's param_groups which to the best of my knowledge applies learning rates in groups, not layer weight/bias specific. I also want to update the learning rates during training, not pre-setting them with torch.optim.
Any help is appreciated.

Output:
Updates to model parameters are handled by an optimizer in PyTorch. When you define the optimizer you have the option of partitioning the model parameters into different groups, called param groups. Each param group can have different optimizer settings. For example one group of parameters could have learning rate of 0.1 and another could have learning rate of 0.01.
To do what you're asking, you can just make every parameter belong to a different param group. You'll need some way to keep track of which param group corresponds to which parameter. Once you've defined the optimizer with different groups you can update the learning rate whenever you want, including at training time.
For example, say we have the following simple linear model
import torch
import torch.nn as nn
import torch.optim as optim

class LinearModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 20)
        self.layer2 = nn.Linear(20, 1)

    def forward(self, x):
        return self.layer2(self.layer1(x))

model = LinearModel()

and suppose we want learning rates for each trainable parameter initialized according to the following:
learning_rates = {
    'layer1.weight': 0.01,
    'layer1.bias': 0.1,
    'layer2.weight': 0.001,
    'layer2.bias': 1.0}

We can use this dictionary to define a different learning rate for each parameter when we initialize the optimizer.
# Build param_group where each group consists of a single parameter.
# `param_group_names` is created so we can keep track of which param_group
# corresponds to which parameter.
param_groups = []
param_group_names = []
for name, parameter in model.named_parameters():
    param_groups.append({'params': [parameter], 'lr': learning_rates[name]})
    param_group_names.append(name)

# optimizer requires default learning rate even if its overridden by all param groups
optimizer = optim.SGD(param_groups, lr=10)

Alternatively, we could omit the 'lr' entry and each param group would be initialized with the default learning rate (lr=10 in this case).
At training time if we wanted to update the learning rates we could do so by iterating over each of the optimizer.param_groups and updating the 'lr' entry for each of them. For example, in the following simplified training loop, we update the learning rates before each step.
for i in range(10):
    output = model(torch.zeros(1, 10))
    loss = output.sum()
    optimizer.zero_grad()
    loss.backward()

    # we can change the learning rate whenever we want for each param group
    print(f'step {i} learning rates')
    for name, param_group in zip(param_group_names, optimizer.param_groups):
        param_group['lr'] = learning_rates[name] / (i + 1)
        print(f'    {name}: {param_group["lr"]}')

    optimizer.step()

which prints
step 0 learning rates
    layer1.weight: 0.01
    layer1.bias: 0.1
    layer2.weight: 0.001
    layer2.bias: 1.0
step 1 learning rates
    layer1.weight: 0.005
    layer1.bias: 0.05
    layer2.weight: 0.0005
    layer2.bias: 0.5
step 2 learning rates
    layer1.weight: 0.0033333333333333335
    layer1.bias: 0.03333333333333333
    layer2.weight: 0.0003333333333333333
    layer2.bias: 0.3333333333333333
step 3 learning rates
    layer1.weight: 0.0025
    layer1.bias: 0.025
    layer2.weight: 0.00025
    layer2.bias: 0.25
step 4 learning rates
    layer1.weight: 0.002
    layer1.bias: 0.02
    layer2.weight: 0.0002
    layer2.bias: 0.2
step 5 learning rates
    layer1.weight: 0.0016666666666666668
    layer1.bias: 0.016666666666666666
    layer2.weight: 0.00016666666666666666
    layer2.bias: 0.16666666666666666
step 6 learning rates
    layer1.weight: 0.0014285714285714286
    layer1.bias: 0.014285714285714287
    layer2.weight: 0.00014285714285714287
    layer2.bias: 0.14285714285714285
step 7 learning rates
    layer1.weight: 0.00125
    layer1.bias: 0.0125
    layer2.weight: 0.000125
    layer2.bias: 0.125
step 8 learning rates
    layer1.weight: 0.0011111111111111111
    layer1.bias: 0.011111111111111112
    layer2.weight: 0.00011111111111111112
    layer2.bias: 0.1111111111111111
step 9 learning rates
    layer1.weight: 0.001
    layer1.bias: 0.01
    layer2.weight: 0.0001
    layer2.bias: 0.1


Instruction: cuda out of memory during training

Input:
I am using Pytorch to do a cat-dog classification. I keep getting a Cuda out of memory problem during training and validation. If I only run the training, I don't have this issue. But when I add a valiation process, I get this oom issue. I cannot see what is happenning.
I have tried: change batchsize to 1; torch.cuda.empty_cache(); and tensor.cpu() for all variables.
RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 8.00 GiB total capacity; 7.21 GiB already allocated; 0 bytes free; 7.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Output:
Could you please update your question to show your code?
Also check that you do use with torch.no_grad() for validation, because otherwise it might compute the gradients and thus consume more memory.


Instruction: Getting the probability density value for a given distribution in PyTorch

Input:
Consider the following code for generating a random sample from a normal distribution with given mean and standard deviation
# import the torch module
import torch

# create the mean with 5 values
mean = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0])

# create the standard deviation with 5 values
std = torch.tensor([1.22, 0.78, 0.56, 1.23, 0.23])

# create normal distribution
print(torch.normal(mean, std))

Output:
tensor([-0.0367,  1.7494,  2.3784,  4.2227,  5.0095])

But I want to calculate the probability density value of the normal distribution for a particular sample given the mean and standard deviation. Is there any function available in PyTorch for doing the same?
Note that I can get the pdf value by coding the analytical expression for the normal distribution with the given mean and standard deviation. But, I want to use an inbuilt from PyTorch.

Output:
There's torch.distributions, providing some helpful methods. How about:
dist = torch.distributions.normal.Normal(mean, std)
print(torch.exp(dist.log_prob(torch.Tensor(my_value))))

The result seems to be the same as scipy.stats.norm.pdf() yields.


Instruction: Writing a custom pytorch dataloader iter with pre-processing on batch

Input:
A typical custom PyTorch Dataset looks like this,
class TorchCustomDataset(torch.utils.data.Dataset):

    def __init__(self, filenames, speech_labels):
        pass

    def __len__(self):
        return 100

    def __getitem__(self, idx):
        return 1, 0

Here, with __getitem__ I can read any file, and apply any pre-processing for that specific file.
What if I want to apply some tensor-level pre-processing to the whole batch of data? Technically, it's possible to just iterate through the data loader to get the batch sample and apply the pre-processing on it.
But how to do it with a custom data loader? In short, what will be the __getitem__ equivalent for data loader to apply some operation on the whole batch of data?

Output: You can override the collate_fn of DataLoader: This function takes the individual items from the underlying Dataset and forms the batch. You can add your custom pre-processing at that point by modifying the collate_fn.


Instruction: How to replace PyTorch model layer's tensor with another layer of same shape in Huggingface model?

Input:
Given a Huggingface model, e.g.
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-large-uncased", num_labels=2)

I can access a layer's tensor as such:
# Shape [1024, 1024]
model.state_dict()["bert.encoder.layer.0.attention.self.query.weight"]

[out]:
tensor([[ 0.0167, -0.0422, -0.0425,  ...,  0.0302, -0.0341,  0.0251],
        [ 0.0323,  0.0347, -0.0041,  ..., -0.0722,  0.0031, -0.0351],
        [ 0.0387, -0.0293, -0.0694,  ...,  0.0492,  0.0201, -0.0727],
        ...,
        [ 0.0035,  0.0081, -0.0337,  ...,  0.0460,  0.0268,  0.0747],
        [ 0.0513,  0.0131,  0.0735,  ..., -0.0127,  0.0144, -0.0400],
        [ 0.0385,  0.0013, -0.0272,  ...,  0.0148,  0.0399,  0.0339]])

Given the another tensor of the same shape that I've pre-defined from somewhere else, in this case, for illustration, I'm creating a random tensor but this can be any tensor that is pre-defined.
import torch
replacement_layer = torch.rand([1024, 1024])

Note: I'm not trying to replace a layer with a random tensor but replace it with a pre-defined one.
When I try to do this to replace the layer tensor through the state_dict(), it didn't seem to work:
import torch
from transformers import AutoModelForSequenceClassification

# The model with a layer that we want to replace.
model = AutoModelForSequenceClassification.from_pretrained("bert-large-uncased", num_labels=2)

# A replacement layer.
replacement_layer = torch.rand([1024, 1024])

# Replacing the layer in the statedict.
model.state_dict()["bert.encoder.layer.0.attention.self.query.weight"] = replacement_layer

# Check that the layer is replaced. No, it is not =(
assert torch.equal(
    model.state_dict()["bert.encoder.layer.0.attention.self.query.weight"],
    replacement_layer)

How to replace PyTorch model layer's tensor with another layer of same shape in Huggingface model?

Output:
A state_dict is something special. It is an on-the-fly copy more than it is the actual contents of a model, if that makes sense.
You can directly access a model's layers by dot notation. Note that 0 often indicates an index rather than a string. You'll also need to transform your tensor into a torch Parameter for it to work within a model.
So this should work:
model.bert.encoder.layer[0].attention.self.query.weight = torch.nn.Parameter(replacement_layer)

or in full:
# Note I used the base model for testing
import torch
from transformers import AutoModelForSequenceClassification

# The model with a layer that we want to replace.
model: torch.nn.Module = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)

# A replacement layer.
replacement_layer = torch.rand([768, 768])

model.bert.encoder.layer[0].attention.self.query.weight = torch.nn.Parameter(replacement_layer)

# Check that the layer is replaced
assert torch.equal(
    model.state_dict()["bert.encoder.layer.0.attention.self.query.weight"],
    replacement_layer)

assert torch.equal(
    model.bert.encoder.layer[0].attention.self.query.weight,
    replacement_layer)

print("Succes!")


Instruction: Precision,recall, F1 score with Sklearn on Pytorch

Input:
I've been looking through samples but am unable to understand how to integrate the precision, recall and f1 metrics for my model. My code is as follows:
for epoch in range(num_epochs):

#Calculate Accuracy (stack tutorial no n_total)
n_correct = 0
n_total = 0

for i, (words, labels) in enumerate(train_loader):
    words = words.to(device)
    labels = labels.to(dtype=torch.long).to(device)

    # Forward pass
    outputs = model(words)
    loss = criterion(outputs, labels)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    #feedforward tutorial solution
    _, predicted = torch.max(outputs, 1)
    n_correct += (predicted == labels).sum().item()
    n_total += labels.shape[0]

accuracy = 100 * n_correct/n_total

#Push to matplotlib
train_losses.append(loss.item())
train_epochs.append(epoch)
train_acc.append(accuracy)

#Loss and Accuracy
if (epoch+1) % 10 == 0:
    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.2f}, Acc: {accuracy:.2f}')

Output:
Since you have the predicted and the labels variables, you can aggregate them during the epoch loop and convert them to numpy arrays to calculate the required metrics.
At the beginning of the epoch, initialize two empty lists; one for true labels and one for ground truth labels.
for epoch in range(num_epochs):
    predicted_labels, ground_truth_labels = [], []
    ...

Then, keep appending the respective entries to each list during the epoch:
   ...
    _, predicted = torch.max(outputs, 1)
    n_correct += (predicted == labels).sum().item()

    # appending
    predicted_labels.append(predicted.cpu().detach().numpy())
    ground_truth_labels.append(labels.cpu().detach().numpy())

...

Then, at the epoch end, you could use precision_recall_fscore_support with predicted_labels and ground_truth_labels as inputs.
Notes:

You'll probably have to refer something like this to flatten the above two lists.
Read about torch.no_grad() to apply it as a good practice during the calculations of metrics.


Instruction: gpu memory is still occupied after validation phase is finished, pytorch

Input:
As far as I know, when training and validating a model with GPU, GPU memory is mainly used for loading data, forward & backward. and to what I know, I think GPU memory usage should be same 1)before training, 2)after training, 3)before validation, 4)after validation. But in my case, GPU memory used in the validation phase is still occupied in the training phase and vice versa. It is not increasing per epoch so I'm sure it is not a common mistake like loss.item().
Here is the summary of my question

Shouldn't the GPU memory used in one phase be cleaned up before another(except for model weights)?
If it should, are there any noob mistakes I'm making here..?

Thank you for your help.
Here is the code for training loop
eval_result = evaluate(model,val_loader,True,True)
print(eval_result)

print('start training')
for epoch in range(num_epoch):
    model.train()
    time_ = datetime.datetime.now()
    for iter_, data in enumerate(tr_loader):
        x, y = data
        x = x.to(device).view(x.shape[0],1,*(x.shape[1:]))
        y = y.to(device).long()

        pred = model.forward(x)
        loss = loss_fn(pred,y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # print
        print_iter = 16
        if (iter_+1) % print_iter == 0:
            elapsed = datetime.datetime.now() - time_
            expected = elapsed * (num_batches / print_iter)
            _epoch = epoch + ((iter_ + 1) / num_batches)
            print('\rTRAIN [{:.3f}/{}] loss({}) '
                'elapsed {} expected per epoch {}'.format(
                    _epoch,num_epoch, loss.item(), elapsed, expected)
                 ,end="\t\t\t")
            time_ = datetime.datetime.now()

    print()
    eval_result = evaluate(model,val_loader,True,True)
    print(eval_result)
    scheduler.step(eval_result[0])

    if (epoch+1) %1 == 0:
        save_model(model, optimizer, scheduler)

I've read about how making a validation phase a function helps since python is function scoping language.
so the evaluate() is
def evaluate(model, val_loader, get_acc = True, get_IOU = True):
    """
    pred: Tensor of shape B C D H W
    label Tensor of shape B D H W
    """
    val_loss = 0
    val_acc = 0
    val_IOU = 0
    with torch.no_grad():
        model.eval()
        for data in tqdm(val_loader):
            x, y = data
            x = x.to(device).view(x.shape[0],1,*(x.shape[1:]))
            y = y.to(device).long()

            pred = model.forward(x)

            loss = loss_fn(pred,y)
            val_loss += loss.item()

            pred = torch.argmax(pred, dim=1)

            if get_acc:
                total = np.prod(y.shape)
                total = total if total != 0 else 1
                val_acc += torch.sum((pred == y)).cpu().item()/total
            if get_IOU:
                iou = 0
                for class_num in range(1,8):
                    iou += torch.sum((pred==class_num)&(y==class_num)).cpu().item()\
                        / torch.sum((pred==class_num)|(y==class_num)).cpu().item()
                val_IOU += iou/7

        val_loss /= len(val_loader)
        val_acc /= len(val_loader)
        val_IOU /= len(val_loader)
    return (val_loss, val_acc, val_IOU)

and here is GPU usage in colab. 1 is the point where the evaluate() is first called, and 2 is when the train started.

Output: Allocating GPU memory is slow. PyTorch retains the GPU memory it allocates, even after no more tensors referencing that memory remain. You can call torch.cuda.empty_cache() to free any GPU memory that isn't accessible.


Instruction: I am kinda new to the pytorch, now struggling with a classification problem

Input:
I built a very simple structure
class classifier (nn.Module):
def __init__(self):
    super().__init__()
    self.classify = nn.Sequential(
        nn.Linear(166,80),
        nn.Tanh(),
        nn.Linear(80,40),
        nn.Tanh(),
        nn.Linear(40,1),
        nn.Softmax()
    )
def forward (self, x):
    pred = self.classify(x)
    return pred
model = classifier()

The loss function and optimizer are defined as
criteria = nn.BCEWithLogitsLoss()
iteration = 1000
learning_rate = 0.1
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

and here is the training and evaluation section
for epoch in range (iteration):
  model.train()
  y_pred = model(x_train)
  loss = criteria(y_pred,y_train)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  model.eval()
  with torch.inference_mode():
      test_pred = model(x_test)
      test_loss = criteria(test_pred, y_test)

  if epoch % 100 == 0:
      print(loss)
      print(test_loss)

I received the same loss values, and by debugging, I found that the weights were not being updated.

Output:
The problem is in the network architecture: you are using a Softmax layer on a single valued output at the end. As per the definition of the softmax function, for a output vector x, we have, for index i:
softmax(x_i) = e^{x_i} / sum_j (e^{x_j})

Here, you only have a single valued output. Due to this, the output of your neural network is always 1, irrespective of the inputs or the weights. To fix this, remove the Softmax layer at the end. An activation function like Sigmoid might be more appropriate, and in fact you are already applying this when using the BCEWithLogitsLoss.


Instruction: Pytorch transformation for just certain batch

Input:
Hi is there any method for apply trasnformation for certain batch?
It means, I want apply trasnformation for just last batch in every epochs.
What I tried is here
import torch

class test(torch.utils.data.Dataset):
    def __init__(self):
        self.source = [i for i in range(10)]

    def __len__(self):
        return len(self.source)

    def __getitem__(self, idx):
        print(idx)
        return self.source[idx]

ds = test()
dl = torch.utils.data.DataLoader(dataset = ds, batch_size = 3,
                                shuffle = False, num_workers = 5)

for i in dl:
    print(i)

because I thought that if I could get idx number, it would be possible to apply for certain batchs.
However If using num_workers outputs are
0
1
2
3
964

57

8
tensor([0, 1, 2])
tensor([3, 4, 5])
tensor([6, 7, 8])
tensor([9])

which are not I thought
without num_worker
0
1
2
tensor([0, 1, 2])
3
4
5
tensor([3, 4, 5])
6
7
8
tensor([6, 7, 8])
9
tensor([9])

So the question is

Why idx works so with num_workers?
How can I apply trasnform for certain batchs (or certain idx)?

Output:
When you have num_workers > 1, you have multiple subprocesses doing data loading in parallel. So what is likely happening is that there is a race condition for the print step, and the order you see in the output depends on which subprocess goes first each time.

For most transforms, you can apply them on a specific batch simply by calling the transform after the batch has been loaded. To do this just for the last batch, you could do something like:
for batch_idx, batch_data in dl:
    # check if batch is the last batch
    if ((batch_idx+1) * batch_size) >= len(ds):
        batch_data = transform(batch_data)


Instruction: Change image labels when using pytorch

Input:
I am loading an image dataset with pytorch as seen below:
dataset = datasets.ImageFolder('...', transform=transform)
loader = DataLoader(dataset, batch_size=args.batchsize)

The dataset is i na folder with structure as seen below:
dataset/
  class_1/
  class_2/
  class_3/

So in result each image in class_1 folder has a label of 0..etc.
However i would like to change these labels and randomly assign a label to each image in the dataset. What i tried is:
 new_labels = [random.randint(0, 3) for i in range(len(dataset.targets))]
 dataset.targets = new_labels

This however does not change the labels as i wanted due to some errors later in model training.
Is this the correct way to do it or is tehre a more appropriate one?

Output:
You can have a transformation for the labels:
import random

class rand_label_transform(object):
  def __init__(self, num_labels):
    self.num_labels = num_labels

  def __call__(self, labels):
    # generate new random label
    new_label = random.randint(0, self.num_labels - 1)
    return new_label

dataset = datasets.ImageFolder('...', transform=transform, target_transform=rand_label_transform(num_labels=3))

See ImageFolder for more details.


Instruction: how to convert ML Project from a GPU project to CPU project?

Input:
I am learning ML and i want to re train a AI model for lane detection.
I want to be familiar with the ML training process. The accuracy/result is not my primary goal and i do not need a best ML model for lane detection.
I found this AI model and want to try it out. But i have been facing a problem:

I do not have a GPU, so i wish i can train this model with my CPU. But sadly some part of this code is written with CUDA. Is there a way, i can convert this GPU code to CPU code only?

Should i find another AI-model only for the CPU training?

Output:
you can use the tensor.to(device) command to move a tensor to a device.
The .to() command is also used to move a whole model to a device, like in the post you linked to.
Another possibility is to set the device of a tensor during creation using the device= keyword argument, like in t = torch.tensor(some_list, device=device)
To set the device dynamically in your code, you can use
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

to set cuda as your device if possible.
Above is the answer for how to add CUDA in the code. SO Use Cntrl + F and remove all the keywords which forces code to run on GPU. Such as "device", "to"


Instruction: How to apply function element wise to 2D tensor

Input:
Very simple question but I have been struggling with this forever now.
import torch
t = torch.tensor([[2,3],[4,6]])
overlap = [2, 6]
f = lambda x: x in overlap

I want:
torch.tensor([[True,False],[False,True]])

Both the tensor and overlap are very big, so efficiency is wished here.

Output:
I found an easy way. Since torch is implemented through numpy array the following works and is performant:
import torch
import numpy as np
t = torch.tensor([[2,3],[4,6]])
overlap = [2, 6]
f = lambda x: x in overlap
mask = np.vectorize(f)(t)

Found here.


Instruction: Docker Image for Image prediction using Pytorch Image Models (timm)

Input:
I want to create a docker image for Image Prediction using Timm which gives outputs in JSON format like {"predicted": "cat", "confidence": "0.99"}.
I am using this code.
from __future__ import print_function
import argparse
import torch
import timm
import urllib
import json
import torchvision.io as io
from PIL import Image as img
from timm.data import resolve_data_config
from timm.data.transforms_factory import create_transform

# Training settings
parser = argparse.ArgumentParser(description='Assignment 1')
parser.add_argument('--model',type=str)
parser.add_argument('--image',type=str)

if __name__ =='__main__':
args =parser.parse_args()
model = timm.create_model(args.model,pretrained=True)
model.eval()
config = resolve_data_config({},model=model)
transform = create_transform(**config)

url, filename = (args.image, "cat.jpg")
urllib.request.urlretrieve(url, filename)
img = img.open(filename).convert('RGB')
tensor = transform(img).unsqueeze(0)  # transform and add batch dimension
import torch
with torch.no_grad():
    out = model(tensor)
probabilities = torch.nn.functional.softmax(out[0], dim=0)
print({"predicted" : "cat", "confidence" : f"{int(probabilities.max()*100)/100}"})

After running a test script, the output is:
Output is not valid json ! got: {'predicted': 'cat', 'confidence': '0.99'}

After using json.dump I am getting this error.
Traceback (most recent call last):
File "/opt/src/main.py", line 25, in <module>
urllib.request.urlretrieve(url, filename)
File "/usr/local/lib/python3.9/urllib/request.py", line 239, in
urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/lib/python3.9/urllib/request.py", line 214, in
urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.9/urllib/request.py", line 523, in
open
response = meth(req, response)
File "/usr/local/lib/python3.9/urllib/request.py", line 632, in
 http_response
response = self.parent.error(
File "/usr/local/lib/python3.9/urllib/request.py", line 561, in
error
return self._call_chain(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 494, in
_call_chain
result = func(*args)
File "/usr/local/lib/python3.9/urllib/request.py", line 641, in
http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Output:
You need to use json.dumps:
import json

...

print(json.dumps({"predicted" : "cat", "confidence" : f"{int(probabilities.max()*100)/100}"}))


Instruction: Normally distributed vector given mean, stddev|variance

Input:
how to do this in torch
 np.random.normal(loc=mean,scale=stdev,size=vsize)

i.e. by providing mean, stddev or variance

Output:
There's a similar torch function torch.normal:
torch.normal(mean=mean, std=std)

If mean and std are scalars, this will produce a scalar value. You can simply repeat them for the target you want it to be, eg:
mean = 2
std = 10
size = (3,3)
r = torch.normal(mean=torch.full(size,mean).float(),std=torch.full(size,mean).float())
print(r)

> tensor([[ 2.2263,  1.1374,  4.5766],
    [ 3.4727,  2.6712,  2.4878],
    [-0.1787,  2.9600,  2.7598]])


Instruction: Pytorch silent data corruption

Input:
I am on a workstation with 4 A6000 GPUs. Moving a Torch tensor from one GPU to another GPU corrupts the data, silently!!!
See the simple example below.
x
>tensor([1], device='cuda:0')

x.to(1)
>tensor([1], device='cuda:1')

x.to(2)
>tensor([0], device='cuda:2')

x.to(3)
>tensor([0], device='cuda:3')

Any ideas what is the cause of this issue?
Other info that might be handy:
(there was two nvlinks which I manually removed trying to solve the problem)
        GPU0    GPU1    GPU2    GPU3    CPU Affinity    NUMA Affinity
GPU0     X      SYS     SYS     SYS     0-63            N/A
GPU1    SYS      X      SYS     SYS     0-63            N/A
GPU2    SYS     SYS      X      SYS     0-63            N/A
GPU3    SYS     SYS     SYS      X      0-63            N/A

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

NVIDIA-SMI 510.47.03    Driver Version: 510.47.03    CUDA Version: 11.6

Edit: adding some screenshots
It seems to be stateful. Changes which GPUs work fine together after starting a new python runtime.

Output:
The solution was to disable IOMMU. On our server, in the BIOS settings
 /Advanced/AMD CBS/NBIO Common Options/IOMMU -> IOMMU - Disabled
See the PyTorch issues thread for more information.


Instruction: Larger batch size cause larger loss

Input:
I am trying to solve a regression problem using pytorch. I have a pre-trained model to start with. When I was tuning hyperparameters, I found my batch size and train/validation loss have a weird correlation. Specifically:
batch size = 16 -\> train/val loss around 0.6 (for epoch 1)
batch size = 64 -\> train/val loss around 0.8 (for epoch 1)
batch size = 128 -\> train/val loss around 1 (for epoch 1)

I want to know if this is normal, or there is something wrong with my code.
optimizer: SGD with learning rate of 1e-3
Loss function:
def rmse(pred, real):
    residuals = pred - real
    square = torch.square(residuals)
    sum_of_square = torch.sum(square)
    mean = sum_of_square / pred.shape[0]
    root = torch.sqrt(mean)
    return root

train loop:
def train_loop(dataloader, model, optimizer, epoch):
    num_of_batches = len(dataloader)
    total_loss = 0
    for batch, (X, y) in enumerate(dataloader):
        optimizer.zero_grad()

        pred = model(X)
        loss = rmse(pred, y)

        loss.backward()
        optimizer.step()

        total_loss += loss.item()

        #lr_scheduler.step(epoch*num_of_batches+batch)
        #last_lr = lr_scheduler.get_last_lr()[0]

    train_loss = total_loss / num_of_batches
    return train_loss

test loop:
def test_loop(dataloader, model):
    size = len(dataloader.dataset)
    num_of_batches = len(dataloader)
    test_loss = 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += rmse(pred, y).item()

    test_loss /= num_of_batches
    return test_loss

Output:
I'll start with an a. analogy, b. dive into the math, and then c. end with a numerical experiment.
a.) What you are witnessing is roughly the same phenomenon as the difference between stochastic and batched gradient descent. In the analog case, the "true" gradient or direction in which the learned parameters should be shifted minimizes the loss over the entire training set of data. In stochastic gradient descent, the gradient shifts the learned parameters in the direction that minimizes the loss for a single example. As the size of the batch is increased from 1 towards the size of the overall dataset, the gradient estimated from the minibatch becomes closer to the gradient for the whole dataset.
Now, is stochastic gradient descent useful at all, given that it is imprecise wrt the whole dataset? Absolutely. In fact, the noise in this estimate can be useful for escaping local minima in the optimization. Analogously, any noise in your estimate of loss wrt the whole dataset is likely nothing to worry about.
b.) But let's next look at why this behavior occurs. RMSE is defined as:

where N is the total number of examples in your dataset. And if RMSE were calculated this way, we would expect the value to be roughly the same (and to approach exactly the same value as N becomes large). However, in your case, you are actually calculating the mean epoch loss as:

where B is the number of minibatches per epoch, and b is the number of examples per minibatch:

Thus, epoch loss is the average RMSE per minibatch. Rearranging, we can see:

when B is large (B = N) and the minibatch size is 1,

which clearly has quite different properties than RMSE defined above. However, as B becomes small B = 1, and minibatch size is N,

which is exactly equal to RMSE above. So as you increase the batch size, the expected value for the quantity you compute moves between these two expressions. This explains the (roughly square root) scaling of your loss with different minibatch sizes. Epoch loss is an estimate of RMSE (which can be thought of as the standard deviation of model prediction error). One training goal could be to drive this error standard deviation to zero, but your expression for epoch loss is also likely a good proxy for this. And both quantities are themselves proxies for whatever model performance you actually hope to obtain.
c. You can try this for yourself with a trivial toy problem. A normal distribution is used as a proxy for model error.
EXAMPLE 1: Compute RMSE for whole dataset ( of size 10000 x b)
import torch
for b in [1,2,3,5,9,10,100,1000,10000,100000]:
  b_errors = []
  for i in range (10000):
    error = torch.normal(0,100,size = (1,b))
    error = error **2
    error = error.mean()
    b_errors.append(error)

RMSE = torch.sqrt(sum(b_errors)/len(b_errors))
print("Average RMSE for b = {}: {}".format(N,RMSE))

Result:
Average RMSE for b = 1: 99.94982147216797
Average RMSE for b = 2: 100.38357543945312
Average RMSE for b = 3: 100.24600982666016
Average RMSE for b = 5: 100.97154998779297
Average RMSE for b = 9: 100.06820678710938
Average RMSE for b = 10: 100.12358856201172
Average RMSE for b = 100: 99.94219970703125
Average RMSE for b = 1000: 99.97941589355469
Average RMSE for b = 10000: 100.00338745117188

EXAMPLE 2: Compute Epoch Loss with B = 10000
import torch
for b in [1,2,3,5,9,10,100,1000,10000,100000]:

b_errors = []
for i in range (10000):
    error = torch.normal(0,100,size = (1,b))
    error = error **2
    error = error.mean()
    error = torch.sqrt(error)
    b_errors.append(error)

avg = (sum(b_errors)/len(b_errors)
print("Average Epoch Loss for b = {}: {}".format(b,avg))

Result:
Average Epoch Loss for b = 1: 80.95650482177734
Average Epoch Loss for b = 2: 88.734375
Average Epoch Loss for b = 3: 92.08515930175781
Average Epoch Loss for b = 5: 95.56260681152344
Average Epoch Loss for b = 9: 97.49445343017578
Average Epoch Loss for b = 10: 97.20250701904297
Average Epoch Loss for b = 100: 99.6297607421875
Average Epoch Loss for b = 1000: 99.96969604492188
Average Epoch Loss for b = 10000: 99.99618530273438
Average Epoch Loss for b = 100000: 100.00079345703125


Instruction: What is the most efficient way to make a method that is able to process single and multi dimensional arrays in python?

Input:
I was using pytorch and realized that for a linear layer you could pass not only 1d tensors but multidmensional tensors as long as the last dimensions matched. Multi dimensional inputs in pytorch Linear method?
I tried looping over each item, but is that what pytorch does?
I'm having trouble thinking how you would program looping with more dimensions, maybe recursion but that seems messy.
What is the most efficient way to implement this?

Output:
I tried looping over each item, but is that what pytorch does?
The short answer is yes, loops are used, but it's more complicated than you probably think. If input is a 2D tensor (a matrix), then the output of a linear operation is computed as input @ weight.T + bias using an external BLAS library's GEMM operation. Otherwise it uses torch.matmul(input, weight.T) + bias which uses broadcast semantics to compute a batched version of the operation. Broadcasting is a semantic, not an implementation, so how the broadcasting is performed is going to be backend-dependent. Ultimately some form of looping combined with parallel processing will be used for most of these implementation.
To go a little deeper, lets take a look at the PyTorch implementation of the linear layer. This quickly leads down some rabbit holes since PyTorch uses different backend libraries for performing linear algebra operations efficiently on the hardware available (libraries like oneAPI, Intel MKL, or MAGMA) but perhaps understanding some of the details can help.
Starting at the C++ entrypoint to nn.functional.linear:
Tensor linear(const Tensor& input, const Tensor& weight, const Tensor& bias) {
  if (input.is_mkldnn()) {
    return at::mkldnn_linear(input, weight, bias);
  }

  if (input.dim() == 2 && bias.defined()) {
    // Fused op is marginally faster.
    return at::addmm(bias, input, weight.t());
  }
  auto output = at::matmul(input, weight.t());
  if (bias.defined()) {
    output.add_(bias);
  }
  return output;
}

There are three cases here.

input.is_mkldnn(). This condition occurs if the input tensor is in the MKL-DNN format (Tensor.to_mkldnn) and will make PyTorch use the at::mkldnn_linear function, which in turn makes calls to ideep, which in turn makes calls to the oneDNN library (previous known as Intel MKL-DNN), which ultimately selects a specific general matrix-matrix multiplication (GEMM) routine dependent on platform and data types. The simplest implementation is the reference implementation, and from that we can see that they use a parallel-for loop (note the anonymous function they use uses a quadruple nested for-loop). In practice the reference implementation probably isn't used, instead, you would probably be calling the x86 optimized version compiled with the Xbyak JIT assembler to produce highly optimized code. I'm not going to pretend to follow all the details of the optimized code, but efficient GEMM is a heavily studied topic that I only have a passing knowledge of.

input.dim() == 2 && bias.defined(). This condition means that input is a 2D tensor (shape [B,M]) and bias is defined. In this case pytorch uses the addmm function. This efficiently computes the output as input @ weight.T + bias where @ is matrix multiplication. There are multiple implementations of addmm registered in PyTorch depending on what types of tensors are being used. The dense-CPU specific version is here which eventually makes calls to an external BLAS library's GEMM subroutine. The backend used is likely Intel MKL but you can check using print(torch.__config__.parallel_info()). Whichever BLAS implementation is being used, its certainly a highly optimized implementation of matrix multiplication similar to the oneDNN implementation, probably using multi-threading and optimized compilation.

If neither of the previous two conditions are met then PyTorch uses the torch.matmul function, which performs a broadcasted version of input @ weight.T where input is shape [..., M]. The result of this operation is a tensor of shape [..., N]. Similar to addmm, there are multiple implementations of this function depending on the tensor types but an external library will ultimately be used that uses parallelization and optimized matrix-multiplication subroutines. After the broadcasted matrix-multiplication a broadcasted add_ operation is used to add the bias term (if bias is defined).


Instruction: What should the input to DeepLabV3 be in training mode?

Input:
I am trying to train a deeplabv3_resnet50 model on a custom dataset, but get the error ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 256, 1, 1]) when trying to do the forward pass. The following minimal example produces this error:
import torch
import torchvision

model = torchvision.models.segmentation.deeplabv3_resnet50(weights="DEFAULT")
model.train()

batch_size = 1
nbr_of_channels = 3
img_height, img_width = (500, 500)
input = torch.rand((batch_size, nbr_of_channels, img_height, img_width))
model(input)

I do not understand this at all. What is meant by got input size torch.Size([1, 256, 1, 1]), and what should I do differently?

Output:
The error you get is from a deep BatchNorm layer: deep in the backbone, the feature map size is reduced to 1x1 pixels. As a result, BatchNorm cannot compute std of the feature map when batch size is only one.
For any batch_size > 1 this will work:
batch_size = 2  # Need bigger batches for training
nbr_of_channels = 3
img_height, img_width = (500, 500)
input = torch.rand((batch_size, nbr_of_channels, img_height, img_width))
model(input)  # working with batch_size > 1


Instruction: Pytorch lightning: see input/ouptut size in model summary when using nn.ModuleList

Input:
When I use nn.ModuleList() to define layers in my pytorch lightning model, their "In sizes" and "Out sizes" in ModelSummary are "?"
Is their a way to have input/output sizes of layer in model summary, eventually using something else than nn.ModuleList() to define layers from a list of arguments.
Here is a dummy model:
(12 is the batch size)
import torch
import torch.nn as nn
from pytorch_lightning import LightningModule
from pytorch_lightning.utilities.model_summary import ModelSummary

class module_list_dummy(LightningModule):
    def __init__(self,
                 layers_size_list,
                ):
        super().__init__()
        self.example_input_array = torch.zeros((12,100), dtype=torch.float32)
        self.fc11 = nn.Linear(100,50)
        self.moduleList = nn.ModuleList()
        input_size = 50
        for layer_size in layers_size_list:
            self.moduleList.append(nn.Linear(input_size, layer_size))
            input_size = layer_size
        self.loss_fn = nn.MSELoss()

    def forward(self, x):
        out = self.fc11(x)
        for layer in self.moduleList:
            out = layer(out)
        return out

    def training_step(self, batch, batch_idx):
        x, y = batch
        out = self.fc11(X)
        for layer in self.moduleList:
            out = layer(out)
        loss = torch.sqrt(self.loss_fn(out, y))
        self.log('train_loss', loss)
        return loss

    def validation_step(self, batch, batch_idx):
        x, y = batch
        out = self.fc11(X)
        for layer in self.moduleList:
            out = layer(out)
        loss = torch.sqrt(self.loss_fn(y_hat, y))
        self.log('val_loss', loss)
        return loss

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)
        return optimizer

the following code print the summary:
net = module_list_dummy(layers_size_list=[30,20,1])
summary = ModelSummary(net)
print(summary)

But the output is:
| Name       | Type       | Params | In sizes  | Out sizes
------------------------------------------------------------------
0 | fc11       | Linear     | 5.0 K  | [12, 100] | [12, 50]
1 | moduleList | ModuleList | 2.2 K  | ?         | ?
2 | loss_fn    | MSELoss    | 0      | ?         | ?
------------------------------------------------------------------
7.2 K     Trainable params
0         Non-trainable params
7.2 K     Total params
0.029     Total estimated model params size (MB)

I would expect to have:
1 | moduleList | ModuleList | 2.2 K  | [12,50]   | [12,1]

or even better something like
1  | Linear | Linear | ... | [12,50] |  [12,30]
2  | Linear | Linear | ... | [12,30] |  [12,20]
3  | Linear | Linear | ... | [12,20] |  [12,1]

I did this to check that those layer are being used in forward:
x = torch.zeros((12,100), dtype=torch.float32)
net(x).shape

and they are ( model output is size [12,1])

Output:
In your case you are using the layers in the list in a sequential manner:
for layer in self.moduleList:
    out = layer(out)

However, nn.ModuleList does not force you to run your layers sequentially. You could be doing this:
out = self.moduleList[3](out)
out = self.moduleList[1](out)
out = self.moduleList[0](out)

Hence, ModuleList cannot itself be interpreted as a "layer" with an input- and output shape. It is completely arbitrary how the user might integrate the layers inside the list in their graph.
In your specific case where you run the layers sequentially, you could define a nn.Sequential instead. This would look like this:
self.layers = nn.Sequential()
for layer_size in layers_size_list:
    self.layers.append(nn.Linear(input_size, layer_size))
    input_size = layer_size

And then in your forward:
out = self.layers(input)

Now, the model summary in Lightning will also show the shape sizes, because nn.Sequential's output shape is now directly the output shape of the last layer inside of it.
I hope this answer helps your understanding.


Instruction: RuntimeError: Could not infer dtype of generator

Input:
I build a training data for a model using pytorch.
def shufflerow(tensor1, tensor2, axis):
    row_perm = torch.rand(tensor1.shape[:axis+1]).argsort(axis)  # get permutation indices
    for _ in range(tensor1.ndim-axis-1): row_perm.unsqueeze_(-1)
    row_perm = row_perm.repeat(*[1 for _ in range(axis+1)], *(tensor1.shape[axis+1:]))  # reformat this for the gather operation
    return tensor1.gather(axis, row_perm),tensor2.gather(axis, row_perm)
class Dataset:
    def __init__(self, observation, next_observation):
        self.data =(observation, next_observation)
        indices = torch.randperm(observation.shape[0])
        self.train_samples = (observation[indices ,:], next_observation[indices ,:])
        self.test_samples = shufflerow(observation, next_observation, 0)

I also have this function which examine whether the data converted to torch.tensor and set the device
def to_tensor(x, device):
    if torch.is_tensor(x):
        return x
    elif isinstance(x, np.ndarray):
        return torch.from_numpy(x).to(device=device, dtype=torch.float32)
    elif isinstance(x, list):
        if all(isinstance(item, np.ndarray) for item in x):
           return [torch.from_numpy(item).to(device=device, dtype=torch.float32) for item in x]
    elif isinstance(x, tuple):
        return (torch.from_numpy(item).to(device=device, dtype=torch.float32) for item in x)
    else:
        print(f"X:{x} and X's type{type(x)}")
        return torch.tensor(x).to(device=device, dtype=torch.float32)

But passing the input data that basically looks like this through the Dataset class
data=Dataset(s1,s2)
print(data.train_samples)
(tensor([[-0.3121, -0.9500,  1.4518],
        [-0.9903, -0.1391, -4.4141],
        [-0.9645, -0.2642,  5.0233],
        [-0.6413,  0.7673, -4.5495],
        [-0.3073,  0.9516, -1.0128],
        [-0.5495,  0.8355,  3.4044],
        [-0.5710, -0.8209, -3.2716],
        [-0.9388,  0.3445,  3.9225],
        [-0.8402, -0.5423, -4.0820]]), tensor([[-0.2723, -0.9622,  0.8342],
        [-0.9958,  0.0912, -4.6186],
        [-0.8747, -0.4847,  4.7741],
        [-0.5495,  0.8355,  3.4044],
        [-0.7146,  0.6996,  4.2841],
        [-0.7128, -0.7014, -3.7148],
        [-0.9915,  0.1303,  4.4200],
        [-0.9358, -0.3526, -4.2585]]))

I am getting this error message
-> 1725         self._target_samples = to_tensor(true_samples)
   1726         self._steps = []

/content/data_gen.py in to_tensor(x)
   1368     else:
   1369         print(f"X:{x} and X's type{type(x)}")
-> 1370         return torch.tensor(x).to(device=device, dtype=torch.float32)

X:<generator object to_tensor.<locals>.<genexpr> at 0x7f380235d6d0> and X's type<class 'generator'>
RuntimeError: Could not infer dtype of generator

Any suggestion, why I am getting this error?

Output:
The expression (torch.from_numpy(item).to(device=device, dtype=torch.float32) for item in x) isn't creating a tuple, it's a generator expression. Since it's in a case where you test for tuples, I suspect you wanted a tuple instead of a generator. Try:
elif isinstance(x, tuple):
    return tuple(torch.from_numpy(item).to(device=device, dtype=torch.float32) for item in x)


Instruction: some parameters appear in more than one parameter group

Input:
When I try the below I get the error ValueError: some parameters appear in more than one parameter group. However, inspecting the model it is not clear to me what is the overlapping module.
The only possiblity as to why it might think so may be because lm_head and transformer.wte have parameters named weight. I'm wondering if this name is what is causing this error.
I am doing this so that I can have the lower layers "moving slowly" compared to the upper layers. Happy to hear if there is an alternative way to do these discriminative learning rates where I don't have overlapping parameters (if any).
import torch
from transformers import AutoModelForCausalLM

language_model = AutoModelForCausalLM.from_pretrained("gpt2")
FREEZE_LAYERS = 2

caption_params = [
        {"params": language_model.lm_head.parameters() , "lr": 1e-4},
        {"params": language_model.transformer.ln_f.parameters() , "lr": 1e-4},
        {"params": language_model.transformer.h[FREEZE_LAYERS:].parameters() , "lr": 5e-5},
        {"params": language_model.transformer.wte.parameters() , "lr": 1e-5},
]
optimizer = torch.optim.Adam(caption_params)

Output:
The error message is diagnosing the problem correctly: there are some parameters that appear in more than one parameter group. You can prove this to yourself by doing the following:
>>> parameter_ids = [[id(p) for p in group["params"]] for group in caption_params]
>>> parameter_ids[0]
[140666221372896]
>>> parameter_ids[3]
[140666221372896]

This reveals that the first and last parameter groups, each of which contains a single large embedding tensor, are actually holding a reference to the same exact tensor. What is this tensor? Let's look at it, using both routes of reference to further show it's the same thing:
>>> a = next(language_model.lm_head.parameters())
>>> a
Parameter containing:
tensor([[-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],
        [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],
        [-0.1275,  0.0479,  0.1841,  ...,  0.0899, -0.1297, -0.0879],
        ...,
        [-0.0445, -0.0548,  0.0123,  ...,  0.1044,  0.0978, -0.0695],
        [ 0.1860,  0.0167,  0.0461,  ..., -0.0963,  0.0785, -0.0225],
        [ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207]],
       requires_grad=True)
>>> b = next(language_model.transformer.wte.parameters())
>>> b
Parameter containing:
tensor([[-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],
        [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],
        [-0.1275,  0.0479,  0.1841,  ...,  0.0899, -0.1297, -0.0879],
        ...,
        [-0.0445, -0.0548,  0.0123,  ...,  0.1044,  0.0978, -0.0695],
        [ 0.1860,  0.0167,  0.0461,  ..., -0.0963,  0.0785, -0.0225],
        [ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207]],
       requires_grad=True)
>>> a is b
True

This makes sense, because many Transformer-based models tie the weights used in mapping between word IDs and word representations at the beginning (the initial Embedding layer) and end (the LM head) of the model.
For your specific problem, you can either accept that the tied weights will be moving at the same LR, or you can untie them by cloning and assigning a new copy of the parameter to one of the two modules.


Instruction: Why my neural network isn`t learning and why it prediction is equal on all test dataset?

Input:
I try to train my first simple neural network. I downloaded dataset from kaggle with already written on tensorflow model. I try to repeat this model on pytorch, but my network obviously not training and prediction does not change from example to example. Please help me. What wrong I do?
My model
class NeuralNetwork(nn.Module):
def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.conv_2d_1 = Conv2d(3, out_channels=16, kernel_size=16, padding=0)
    self.pooling1 = MaxPool2d((2, 2))
    self.conv_2d_2 = Conv2d(16, out_channels=32, kernel_size=32, padding=0)
    self.pooling2 = MaxPool2d((2, 2))
    self.conv_2d_3 = Conv2d(32, out_channels=64, kernel_size=16, padding=0)
    self.pooling3 = MaxPool2d((2, 2))
    self.l1 = Linear(64*25, 128)
    self.l2 = Linear(128, 2)
    self.sigmoid = Sigmoid()
    self.dropout = Dropout(0.2)
    self.relu = ReLU()
    self.softmax = softmax
    self.flatten = tr.nn.Flatten()

def forward(self, x):
    x.requires_grad = True
    x = self.conv_2d_1(x)
    x = self.relu(self.pooling1(x))
    x = self.conv_2d_2(x)
    x = self.relu(self.pooling2(x))
    x = self.conv_2d_3(x)
    x = self.relu(self.pooling3(x))
    x = self.flatten(x)
    x = x.reshape(64*25)
    x = self.l1(x)
    x = self.l2(x)
    return x

Optimizer and loss function
loss_fn = CrossEntropyLoss()
optimizer = Adam(model.parameters(), lr=1e-3)

Training cycle
from matplotlib import pyplot as plt
test1 = tr.Tensor([0, 1]).to('cuda:1')
test2 = tr.Tensor([1, 0]).to('cuda:1')
epochs_number = 10
loss_data = np.zeros((train_clear.shape[0] + train_ill.shape[0]) * epochs_number)
loss_data_cursor = 0
model.train()

for k in range(epochs_number):
    i = 0
    while i < max(train_clear.shape[0], train_ill.shape[0]) + 1:
        try:
            optimizer.zero_grad()
            prediction = model(tr.Tensor(train_clear[i]).to('cuda:1'))
            loss = loss_fn(prediction, test1)
            loss.backward()
            optimizer.step()
            # print(loss, prediction, test1)
            loss_item = loss.item()
            loss_data[loss_data_cursor] = loss_item
            loss_data_cursor += 1
        except:
            pass
        try:
            prediction = model(tr.Tensor(train_ill[i]).to('cuda:1'))
            loss = loss_fn(prediction, test2)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            # print(loss, prediction, test1)
            loss_item = loss.item()
            loss_data[loss_data_cursor] = loss_item
            loss_data_cursor += 1
        except:
            pass
        i += 1
    plt.plot(loss_data)
    plt.show()

Training loss

The test
model.train(mode=False)
prediction1 = model(tr.Tensor(train_clear[0]).to('cuda:1'))
prediction2 = model(tr.Tensor(train_ill[5]).to('cuda:1'))
prediction1, prediction2, model.parameters(), test_clear[0].shape
rand = tr.randn(test_clear[0].shape).to('cuda:1')
prediction3 = model(rand)
prediction1, prediction2, prediction3, model.parameters(), test_clear[0].shape

Test results
 (tensor([ 0.2431, -0.3565], device='cuda:1', grad_fn=<AddBackward0>),
 tensor([ 0.2431, -0.3565], device='cuda:1', grad_fn=<AddBackward0>),
 tensor([ 58.9911, -75.1154], device='cuda:1', grad_fn=<AddBackward0>),
 <generator object Module.parameters at 0x00000183A4845970>,
 (3, 180, 180))

Output: I finded solution. My problem was a using of torch.nn.ReLU instead of torch.nn.functional.relu. I finded that it isn`t equivalent methods and my problem was solved when I replace it.


Instruction: why `local_rank` is zero in DDP even I set visible CUDA as 2?

Input:
There are 3 GPUs in my system.
I want to run on the last one i.e. 2. For this reason, I set gpu_id as 2 in my configuration file as well as CUDA_VISIBLE_DEVICES=2. But in my program, the following line always assigns the 0th GPU.
        local_rank = torch.distributed.get_rank()
        torch.cuda.set_device(local_rank)

How to fix this issue?

Output:
When setting CUDA_VISIBLE_DEVICES=2 you tell the OS to only expose the third GPU to your process. That is, as far as PyTorch is concerned, there is only one GPU. Therefore torch.distributed.get_world_size() returns 1 (and not 3).
The rank of this GPU, in your process, will be 0 - since there are no other GPUs available for the process. But as far as the OS is concerned - all processing are done on the third GPU that was allocated to the job.


Instruction: RuntimeError: The size of tensor a (38) must match the size of tensor b (34) at non-singleton dimension 3

Input:
I studied Resnet 50 using cifar-10
but, I faced RuntimeError.
Here is code
class BasicBlock(nn.Module):
    def __init__(self, in_planes, planes, stride = 1):
        super(BasicBlock, self).__init__()

        self.conv1 = nn.Conv2d(in_planes, planes, kernel_size = 1, stride = stride, padding = 1, bias = False)
        self.bn1 = nn.BatchNorm2d(planes)

        self.conv2 = nn.Conv2d(planes, planes, kernel_size = 3, stride = 1, padding = 1, bias = False)
        self.bn2 = nn.BatchNorm2d(planes)

        self.conv3 = nn.Conv2d(planes, planes * 4, kernel_size = 1, stride = 1, padding = 1, bias = False)
        self.bn3 = nn.BatchNorm2d(planes * 4)

        if stride != 1:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, planes, kernel_size = 1, stride = stride, bias = False),
                nn.BatchNorm2d(planes)
            )
        else:
            self.shortcut = nn.Sequential()

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out = self.bn3(self.conv3(out))
        out += self.shortcut(x) #shortcut connection
        out = F.relu(out)

and Error is
RuntimeError: The size of tensor a (38) must match the size of tensor b (34) at non-singleton dimension 3
How can I fix it?

Output: You don't want padding in your self.conv1 nor self.conv3, because you'll be increasing your image size by 2 (1 each size) each time. Padding should only be used to avoid reducing your image size when using a kernel size of above 1.


Instruction: PyTorch model output different dimension when using DataParallel

Input:
I implemented my PyTorch model with DataParallel for multi-GPU training. However, it seems that the model doesn't consistently output the right dimension. In the training loop, it seems that the model gave the correct output dimension for the first two batches, but it failed to do so for the third batch and caused an error when calculating the loss:

I also tried to use the solution from this post but it didn't help.

Output: It seems like you are left with only one sample for the last batch. Try setting drop_last=True in your Dataloader: This will discard the last "not-full" batch.


Instruction: Simultaneous Batch and Channel slicing in PyTorch

Input:
In PyTorch I have an RGB tensor imgA of batch size 256. I want to retain the green channel for first 128 batches and red channel for remaining 128 batches, something like below:
imgA[:128,2,:,:] = imgA[:128,1,:,:]
imgA[128:,2,:,:] = imgA[128:,0,:,:]
imgA = imgA[:,2,:,:].unsqueeze(1)

or same can be achieved like
imgA = torch.cat((imgA[:128,1,:,:].unsqueeze(1),imgA[128:,0,:,:].unsqueeze(1)),dim=0)

but as I have multiple such images like imgA, imgB, imgC, etc what is the fastest way of achieving the above goal?

Output:
A slicing-based solution can be achieved using torch.gather and repeat_interleave:
select = torch.tensor([1, 0], device=imgA.device)
imgA = = imgA.gather(dim=1, index=select.repeat_interleave(128, dim=0).view(256, 1, 1, 1).expand(-1, -1, *imgA.shape[-2:]))

You can also do that using matrix multiplication and repeat_interleave:
# select c=1 for first half and c=0 for second
select = torch.tensor([[0, 1],[1, 0],[0, 0]], dtype=imgA.dtype, device=imgA.device)
imgA = torch.einsum('cb,bchw->bhw',select.repeat_interleave(128, dim=1), imgA).unsqueeze(dim=1)


Instruction: How to load one model’s output as another model’s parameters and do end-to-end optimization

Input:
Here we have an abstract of this problem:
Assuming that we have two models: ResNet and EfficienNet, respectively.
The first model is as follow (ResNet):
def __init__(self, in_channels, out_channels, num_classes):

    super().__init__()
    self.conv1_0 = _conv3x3(3, 32, stride=2)
    self.bn1_0 = _bn(32)
    self.conv1_1 = _conv3x3(32, 32, stride=1)
    self.bn1_1 = _bn(32)
    self.conv1_2 = _conv3x3(32, 64, stride=1)
    self.relu = nn.ReLU()
    self.pad = torch.nn.ReplicationPad2d(padding=(0, 0, 1, 1))
    self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2)
    self.end_point = _fc(225, num_classes)

def forward(self, x):
    x = self.conv1(x)
    #and so on...
    out = self.end_point(x)
    return out

while the second model has been downloaded as follow (Efficient):
efficientNet = models.efficientnet_b5().to(device)

So, we have two models but the first is developed to scratch and the second one is imported by torchvision.models library.
Now, we want the EfficientNet to take the output of ResNet, then make an end-to-end optimization from EfficientNet’s output to first layer of ResNet.
I know an easier way, which consist of changing the codes and directly using ResNet’s result as an input of Efficient.forward(). However, Efficient model is too complex, making such a change is difficult.
For mathematical reasons, assuming that we have antoher easy model between the ResNet and EfficientNet, that it's called gumbel_model.
So, in conclusion we have 3 models but we got just one target labels about the last one of them, then we can only calculate the loss of the last model (Efficent Net).
When we calculate the loss of the last model, we actually write the three rows as follow:
optimizer.zero_grad()
loss.backforward()
optimizer.step()

Where the optimizer is as follow:
optimizer = optim.SGD([dict(params=efficientNet.parameters(), lr=LR)])

Is it correct that we are backpropagationing only the last model?
To backpropagation end to end models, is it necessary to add parameters of ResNet like the first arg of optim.SGD? If Yes, how could we get params by ResNet? (see above)
I have tried using some codes during one epoch, as follow :
efficientNet = get_efficient_trained(device, out_features=1, in_features=3,path_model)
predictor = ResNet(ResidualBlockBase, layer_config, num_classes=num_cl)

for i, imgs in enumerate(dataloader):
    inputs, labels = imgs
    inputs, labels = inputs.to(device), labels.to(device)
    predictor_output = predictor(inputs)
    predictor_gumbel_output = gumbel(predictor_output)
    optimizer.zero_grad()
    outputs = efficientnet(predictor_gumbel_output).torch.squeeze(outputs, 1)
    loss = loss_fn(outputs, labels)
    loss.backward()
    optimizer.step()
return model

But from the results I have the apprehension that only EfficientNet is just trained.
Are there any ways to let the back-optimization reach to ResNet?
How could I solve this type of problem?*
Waiting for responses.
I really thank you all in advance.

Output:
The params argument of the optimizer specifies all the parameters you want to optimize. Here, as you are only passing the parameters of EfficientNet, only those get optimized, as you suspect.
To optimize for all parameters end-to-end, simply pass them all when initializing the optimizer. This can be done as:
optimizer = optim.SGD(list(efficientNet.parameters()) + list(gumbel.parameters()) + list(predictor.parameters()), lr=LR)


Instruction: How to select indices according to another tensor in pytorch?

Input:
I have two tensors a and b. And I want to retrive the values of b according to the positions of max values in a. That is,
max_values, indices = torch.max(a, dim=0, keepdim=True)
However, I do not know how to use the indices to retrive the values of b. Can anybody helps to solve it? Thanks a lot!!
Edit:
Sorry for not describing my problem concretely. To give a minimal example, the value of tensors a and b are:
    a = torch.tensor([[1,2,4],[2,1,3]])
    b = torch.tensor([[10,24,2],[23,4,5]])

If I use torch.max(a, dim=0, keepdim=True), it will return:
    max:  tensor([[2, 2, 4]])
    indices:   tensor([[1, 0, 0]])

What I want to obtain is the selected value of tensor b according to the indices of max values of a in dim=0, that is,
    tensor([[23, 24, 2]])

I have tried b[indices], whereas the result is not what I want:
    tensor([[[ 2,  3,  5],
             [10, 30, 40],
             [10, 30, 40]]])

Output:
You can use torch.gather:
torch.gather(b, dim=0, index=indices)


Instruction: Pytorch : TypeError when I call my data_transformation function inside where I define my train_dataset object

Input:
When I try to make an object from my get_data function :
   train = get_data(root ="My_train_path",
                     transform = data_transforms[TRAIN] )

it returns an TypeError: 'function' object is not subscriptable.

data_dir = 'my_dataset_dir'

TEST  = 'test'
TRAIN = 'train'
VAL   = 'val'
def data_transforms(phase):
    if phase == TRAIN:
        transform = A.Compose([
            A.CLAHE(clip_limit=4.0, p=0.7),
            A.CoarseDropout(max_height=8, max_width=8, max_holes=8, p=0.5),
            A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
            ToTensorV2(),

        ])

    if phase == VAL:
        transform = A.Compose([
            A.Resize(height=256,width=256),
            A.CenterCrop(height=224,width=224),
            A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
            ToTensorV2(),
        ])

    if phase == TEST:
        transform = A.Compose([
            A.Resize(height=256,width=256),
            A.CenterCrop(height=224,width=224),
            A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
            ToTensorV2(),
        ])

    return transform

def get_data(root,transform):

  image_dataset = CustomImageFolder(root=".",
                                    transform = transform,
                                                         )
  return image_dataset

def make_loader(dataset, batch_size,shuffle,num_workers):
    loader = torch.utils.data.DataLoader(dataset=dataset,
                                         batch_size=batch_size,
                                         shuffle=shuffle,
                                         pin_memory=True, num_workers=num_workers)
    return loader

Output:
The problem is exactly as the error message says it: data_transforms is a function you have defined, and you want to call it with the training phase as the argument. However, you are erroneously subscripting the function, with your use of square brackets ([]). To fix this, replace the square brackets with parentheses (()), as is done for a function call.
That is,
train = get_data(root ="My_train_path",
                 transform = data_transforms(TRAIN) )


Instruction: What is the difference between sagemaker-pytorch-training-toolkit and sagemaker-training-toolkit in SageMaker?

Input:
When porting PyTorch code / models to SageMaker, which one should we use:
PyTorch Training Toolkit (https://github.com/aws/sagemaker-pytorch-training-toolkit/) or
SageMaker Training Toolkit (https://github.com/aws/sagemaker-training-toolkit)? What's the difference when using these toolkits?

Output:
The SageMaker PyTorch Training Toolkit repository used to be the repository for the Sagemaker Pytorch Training Containers, and similarly the SageMaker PyTorch Inference Toolkit was the repository for the SageMaker PyTorch Inference containers.
At some point, AWS has started to directly use the DockerFiles of the Deep Learning containers from the AWS Deep Learning Containers repository so the repositories above were renamed because now AWS has used them to build a library that gets installed into the DL containers to make them SageMaker-compatible for training.
Example: From here https://github.com/aws/sagemaker-pytorch-training-toolkit/blob/master/setup.py Example of building a package that then gets installed in the DL container here: https://github.com/aws/deep-learning-containers/blob/1596489c9002cea08f8a2a7d2f4642c4b3727d52/pytorch/training/docker/1.6.0/py3/Dockerfile.cpu#L112


Instruction: Pytorch crashes with error IndexError: Target 32 is out of bounds

Input:
I am attempting to train a model using CIFAR-100 dataset, on CPU.
But, I get an error:
Traceback (most recent call last):
  File "recog.py", line 68, in <module>
    loss = criterion(outputs, labels)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py", line 1152, in forward
    label_smoothing=self.label_smoothing)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2846, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
IndexError: Target 32 is out of bounds.

I took a snippet from here and modified it a little.
Code:
import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = torchvision.datasets.CIFAR100(root='./dataone', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR100(root='./dataone', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,
                                         shuffle=False, num_workers=2)
classes = ('aquatic mammals','fish','flowers','food containers','fruit and vegetables','household electrical devices','household furniture','insects','large carnivores','large man-made outdoor things','large natural outdoor scenes','large omnivores and herbivores','medium-sized mammals','non-insect invertebrates','people','reptiles','small mammals','trees','vehicles 1','vehicles 2')

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

#criterion = nn.CrossEntropyLoss()
#optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')
            running_loss = 0.0

print('Finished Training')

Also, the target number is always different. I use Python 3.9, with the last pytorch.
When I attempt to do the same thing,but with CIFAR-10, it works perfectly. I'm stuck.
Please help.

Output:
Your model only predicts 10 classes. CIFAR100 has 100 classes.
Change
self.fc3 = nn.Linear(84, 10)

to
self.fc3 = nn.Linear(84, 100)


Instruction: How retain_grad() in pytorch works? I found its position changes the grad result

Input:
in a simple test in pytorch, I want to see grad in a non-leaf tensor, so I use retain_grad():
import torch
a = torch.tensor([1.], requires_grad=True)
y = torch.zeros((10))
gt = torch.zeros((10))

y[0] = a
y[1] = y[0] * 2
y.retain_grad()

loss = torch.sum((y-gt) ** 2)
loss.backward()
print(y.grad)

it gives me a normal output:
tensor([2., 4., 0., 0., 0., 0., 0., 0., 0., 0.])

but when I use retain grad() before y[1] and after y[0] is assigned:
import torch
a = torch.tensor([1.], requires_grad=True)
y = torch.zeros((10))
gt = torch.zeros((10))

y[0] = a
y.retain_grad()
y[1] = y[0] * 2

loss = torch.sum((y-gt) ** 2)
loss.backward()
print(y.grad)

now the output changes to:
tensor([10.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.])

I can't understand the result at all.

Output:
Okay so what's going on is really weird.
What .retain_grad() essentially does is convert any non-leaf tensor into a leaf tensor, such that it contains a .grad attribute (since by default, pytorch computes gradients to leaf tensors only).
Hence, in your first example, after calling y.retain_grad(), it basically converted y into a leaf tensor with an accessible .grad attribute.
However, in your second example, you initially converted the entire y tensor into a leaf tensor; then, you created a non-leaf tensor (y[1]) within your leaf tensor (y), which is what caused the confusion.
y = torch.zeros((10))  # y is a non-leaf tensor

y[0] = a  # y[0] is a non-leaf tensor
y.retain_grad()  # y is a leaf tensor (including y[1])
y[1] = y[0] * 2  # y[1] is a non-leaf tensor, BUT y[0], y[2], y[3], ..., y[9] are all leaf tensors!

The confusing part is:
y[1] after calling y.retain_grad() is now a leaf tensor with a .grad attribute. However, y[1] after the computation (y[1] = y[0] * 2) is now not a leaf tensor with a .grad attribute; it is now treated as a new non-leaf variable/tensor.
Therefore, when calling loss.backward(), the Chain rule of the loss w.r.t y, and particularly looking at the Chain rule of the loss w.r.t leaf y[1] now looks something like this:


Instruction: In pytorch, how to calculate gradient for a element in a tensor when it is used to calculate another element in this tensor?

Input:
In this pytorch code:
import torch
a = torch.tensor([2.], requires_grad=True)
y = torch.zeros((10))
gt = torch.zeros((10))

y[0] = a
y[1] = y[0] * 2
y.retain_grad()

loss = torch.sum((y-gt) ** 2)
loss.backward()
print(y.grad)

I want y[0]'s gradient to consist 2 parts:

loss backward to y[0] itself.
y[0] is used to calculate y[1], so it should have the part of y[1]'s gradient.

but when I run this code, there is only part 1 in y[0]'s gradient.
So how to make y[0]'s gradient to have all 2 parts?
edit: the output is:
tensor([4., 8., 0., 0., 0., 0., 0., 0., 0., 0.])

but I expect:
tensor([20., 8., 0., 0., 0., 0., 0., 0., 0., 0.])

Output:
y[0] and y[1] are two different elements, therefore they have different grad. The only thing that "binds" them is the underlying relation to a. If you inspect the grad of a, you'll see:
print(a.grad)

tensor([20.])

That is, the two parts of the gradients are combined in a.grad.


Instruction: How to use architecture of T5 without pretrained model (Hugging face)

Input: I would like to study the effect of pre-trained model, so I want to test t5 model with and without pre-trained weights. Using pre-trained weights is straight forward, but I cannot figure out how to use the architecture of T5 from hugging face without the weights. I am using Hugging face with pytorch but open for different solution.

Output:
https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Model
"Initializing with a config file does not load the weights associated with the model, only the configuration."
for without weights  create a T5Model with config file
from transformers import AutoConfig
from transformers import T5Tokenizer, T5Model
model_name = "t5-small"
config = AutoConfig.from_pretrained(model_name)
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5Model.from_pretrained(model_name)
model_raw = T5Model(config)


Instruction: How do I use a pt file in Pytorch to predict the label of a new data?

Input:
This is my training model run.py, my data is a one-dimensional matrix with one row and one category.
import numpy as np # linear algebra
import pandas as pd
import os
for dirname, _, filenames in os.walk('./kaggle'):
    for filename in filenames:
        print(os.path.join(dirname, filename))
import torch
from torch.utils.data import DataLoader
from torch import nn,optim
import sys
from tqdm import tqdm
import io
import torch.utils.model_zoo as model_zoo
import torch.onnx

def my_DataLoader(train_root,test_root,batch_size = 100, val_split_factor = 0.2):

    train_df = pd.read_csv(train_root, header=None)
    test_df = pd.read_csv(test_root, header=None)

    train_data = train_df.to_numpy()
    test_data = test_df.to_numpy()

    train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(train_data[:, :-1]).float(),
                                                   torch.from_numpy(train_data[:, -1]).long(),)#
    test_dataset = torch.utils.data.TensorDataset(torch.from_numpy(test_data[:, :-1]).float(),
                                                  torch.from_numpy(test_data[:, -1]).long())

    train_len = train_data.shape[0]
    val_len = int(train_len * val_split_factor)
    train_len -= val_len

    train_dataset, val_dataset = torch.utils.data.random_split(train_dataset, [train_len, val_len])

    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True)
    test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

    return train_loader, val_loader, test_loader
class  conv_net(nn.Module):

    def __init__(self, num_of_class):
        super(conv_net, self).__init__()

        self.model = nn.Sequential(
            #nn.Conv1d(1, 16, kernel_size=5, stride=1, padding=2),
            #nn.Conv1d(1, 16, kernel_size=1, stride=1),
            nn.Conv1d(1, 16, kernel_size=1, stride=1),
            nn.BatchNorm1d(16),
            nn.ReLU(),
            nn.MaxPool1d(2),

            nn.Conv1d(16, 64, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm1d(64),
            nn.ReLU(),
            nn.MaxPool1d(2),
        )

        #self.relu = nn.ReLU()
        self.linear = nn.Sequential(
            #nn.Linear(5120,32),
            nn.Linear(5120,32),
            nn.LeakyReLU(inplace=True),
            nn.Linear(32, num_of_class),
        )

    def forward(self,x):
        #org = x
        x = x.unsqueeze(1)
        x = self.model(x)
        #x = self.relu(x)
        # print(x.shape)
        x = x.view(x.size(0), -1)
        #x [b, 2944]
        # print(x.shape)
        x = self.linear(x)
        return x
batch_size=32
lr = 3e-3
epochs = 150
torch.manual_seed(1234)
#device = torch.device("cpu:0 cuda:0" if torch.cuda.is_available() else "cpu")
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print("using {} device.".format(device))

def evalute(model, loader):
    model.eval()

    correct = 0
    total = len(loader.dataset)
    val_bar = tqdm(loader, file=sys.stdout)
    for x, y in val_bar:
        x, y = x.to(device), y.to(device)
        with torch.no_grad():
            logits = model(x)
            pred = logits.argmax(dim=1)
        correct += torch.eq(pred, y).sum().float().item()

    return correct / total
def main():

    train_loader, val_loader, test_loader = my_DataLoader('./kaggle/train.csv',
                                                          './kaggle/test.csv',
                                                          batch_size=batch_size,
                                                          val_split_factor=0.2)

    model = conv_net(8).to(device)
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criteon = nn.CrossEntropyLoss()
    # Print model's state_dict
    print(model)

    best_acc, best_epoch = 0, 0
    global_step = 0

    for epoch in range(epochs):

        train_bar = tqdm(train_loader, file=sys.stdout)
        for step, (x, y) in enumerate(train_bar):
            # x: [b, 187], y: [b]
            x, y = x.to(device), y.to(device)

            model.train()
            logits = model(x)
            loss = criteon(logits, y)

            optimizer.zero_grad()
            loss.backward()

            # for param in model.parameters():
            #     print(param.grad)

            optimizer.step()

            train_bar.desc = "train epoch[{}/{}] loss:{:.3f}".format(epoch + 1,
                                                                     epochs,
                                                                     loss)

            global_step += 1
        if epoch % 1 == 0:  # You can change the validation frequency as you wish

            val_acc = evalute(model, val_loader)
            print('val_acc = ',val_acc)
            if val_acc > best_acc:
                best_epoch = epoch
                best_acc = val_acc
                # Export the model
                name_pt = 'best3.pt'
                torch.save(model.state_dict(), name_pt)
    print('best acc:', best_acc, 'best epoch:', best_epoch)
    model.load_state_dict(torch.load(name_pt))
    print('loaded from ckpt!')

    test_acc = evalute(model, test_loader)
    print('test acc:', test_acc)

if __name__ == '__main__':
    main()

Then I try to make predictions and modify with reference to other people's code
import torch
from torchvision.transforms import transforms
import pandas as pd
from PIL import Image
from run import conv_net
from pathlib import Path

name_pt = 'best3.pt'
model = conv_net(8)
checkpoint = torch.load(name_pt)
model.load_state_dict(checkpoint)

testdata = './kaggle/onedata.csv'
test_df = pd.read_csv(testdata, header=None)
test_data = test_df.to_numpy()
csv = torch.utils.data.TensorDataset(torch.from_numpy(test_data[:, :]).float())

output = model(csv)

prediction = int(torch.max(output.data, 1)[1].numpy())
print(prediction)

if (prediction == 0):
    print ('other')
if (prediction == 1):
    print ('100%PET')
if (prediction == 2):
    print ('100% Cotton')
if (prediction == 3):
    print ('100% Nylon')
if (prediction == 4):
    print ('>70% PET')
if (prediction == 5):
    print ('<70% PET')
if (prediction == 6):
    print ('Spandex/PET Spandex<5%')
if (prediction == 7):
    print ('Spandex/PET Spandex>5%')

Something went wrong
  File "C:\Users\54-0461100-01\Desktop\for_spec_train\run.py", line 70, in forward
    x = x.unsqueeze(1)
AttributeError: 'TensorDataset' object has no attribute 'unsqueeze'

Most of the questions are for images, not found on CSV files.Any help is appreciated if you have any suggestions.
By the way this is my data format.
LJ column are labels,train and test set are same format
enter image description here
onedata format
enter image description here

Output:
When calling output = model(csv) you are passing the model a 'TensorDataset' object as the input instead of a tensor. You can access the tensors in this object by indexing it. https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#TensorDataset
Additionally, you can avoid the TensorDataset object all together by replacing
csv = torch.utils.data.TensorDataset(torch.from_numpy(test_data[:, :]).float())

with
csv = torch.from_numpy(test_data[:, :]).float()


Instruction: how the code '-input[range(target.shape[0]),target]' works?

Input:
I'm learing pytorch.Reading the official tutorial,I met the preplexing code.
input is a tensor, so is target.
def nll(input,target):
    return -input[range(target.shape[0]),target].mean()

And the pred is:

target is:

the '-input[range(target.shape[0]),target]' is:

Output shows this is not substracting target from input or merging two tensors

Output:
The code input[range(target.shape[0]), target] simply picks, from each row i of input the element at column indicated by the corresponding element of target, that is target[i].
In other words, if out = input[range(target.shape[0]), target] then out[i] = input[i, target[i]].
This is very similar to torch.gather.


Instruction: Solving Sylvester equations in PyTorch

Input:
I'm trying to solve a Sylvester matrix equation of the form

AX + XB = C

From what I've seen, these equations are usually solved with the Bartels-Stewart algorithm taking successive Schur decompositions. I'm aware scipy.linalg already has a solve_sylvester function, but I'm integrating the solution to the Sylvester equation into a neural network, so I need a way to calculate gradients to make A, B, and C learnable. Currently, I'm just solving a linear system with torch.linalg.solve using the Kronecker product and vectorization trick, but this has terrible runtime complexity. I haven't found any PyTorch support for Sylvester equations, let alone Schur decompositions, but before I try to implement Barters-Stewart on the GPU, is there a simpler way to find the gradients?

Output:
Initially I wrote a solution that would give complex X based on Bartels-Stewart algorithm for the m=n case. I had some problems because the eigenvector matrix is not accurate enough. Also the real part gives the real solution, and the imaginary part must be a solution for AX - XB = 0
import torch

def sylvester(A, B, C, X=None):
    m = B.shape[-1];
    n = A.shape[-1];
    R, U = torch.linalg.eig(A)
    S, V = torch.linalg.eig(B)
    F = torch.linalg.solve(U, (C + 0j) @ V)
    W = R[..., :, None] - S[..., None, :]
    Y = F / W
    X = U[...,:n,:n] @ Y[...,:n,:m] @ torch.linalg.inv(V)[...,:m,:m]
    return X.real if all(torch.isreal(x.flatten()[0])
                for x in [A, B, C]) else X

As can be verified on the GPU with
device='cuda'
# Try different dimensions
for batch_size, M, N in [(1, 4, 4), (20, 16, 16), (6, 13, 17), (11, 29, 23)]:
    print(batch_size, (M, N))
    A = torch.randn((batch_size, N, N), dtype=torch.float64,
                    device=device, requires_grad=True)
    B = torch.randn((batch_size, M, M), dtype=torch.float64,
                    device=device, requires_grad=True)
    X = torch.randn((batch_size, N, M), dtype=torch.float64,
                    device=device, requires_grad=True)
    C = A @ X - X @ B
    X_ = sylvester(A, B, C)
    C_ = (A) @ X_ - X_ @ (B)
    print(torch.max(abs(C - C_)))
    X.sum().backward()

A faster algorithm, but inaccurate in the current pytorch version is
def sylvester_of_the_future(A, B, C):
    def h(V):
        return V.transpose(-1,-2).conj()
    m = B.shape[-1];
    n = A.shape[-1];
    R, U = torch.linalg.eig(A)
    S, V = torch.linalg.eig(B)
    F = h(U) @ (C + 0j) @ V
    W = R[..., :, None] - S[..., None, :]
    Y = F / W
    X = U[...,:n,:n] @ Y[...,:n,:m] @ h(V)[...,:m,:m]
    return X.real if all(torch.isreal(x.flatten()[0]) for x in [A, B, C]) else X

I will leave it here maybe in the future it will work properly.


Instruction: How to give TemporalFusionTransformer model a name?

Input:
Im working on creating .py scripts that preprocess some data and then train a TemporalFusionTransformer model. After the training, I have a function that logs the evaluation metrics in a .txt file, whose name should be model_name@datetime.txt. Everywhere that I have looked, searched, in the docs, on forums, articles, I cannot find a way to give my models a custom name? Any idea how to do this?
Edit: Can someone with >1500 reputation please add the tag temporalfusiontransformer in he tags section. Users below 1500 reputation (like me) cannot add new tags to the site.

Output:
You could create a custom class inheriting the original that requires and stores a name property on top of what other functionality the model provides, e.g.
class NamedTFT(TemporalFusionTransformer):
    def __init__(self, name: str, *args, **kwargs):
        super(NamedTFT, self).__init__(*args, **kwargs)
        self.name = name

then you could grab the model's name afterwards.


Instruction: Create a skip connection in a neural network - non Resnet

Input:
I want to add a skip connection to my neural network; I'm not trying to
implement a ResNet, just a regular MLP. I can't find a resource that doesn't point to resnet or densenet. I tried naively adding layers, but it's throwing an error; I'd appreciate the help. thank you
    input_size = 615
    output_size = 40
    model = torch.nn.Sequential()
    layer_0 = model.add_module("linear_0", torch.nn.Linear(input_size, 2048))
    activ_0 = model.add_module("activation_0", ReLU())
    layer_1 = model.add_module("linear_1", torch.nn.Linear(2048, 2048))
    activ_1 = model.add_module("activation_1", ReLU())
    layer_2 = model.add_module("linear_2", torch.nn.Linear(2048, 2048))
    # skip connection
    skip_0 =  model.add_module(layer_1 + layer_2)
    activ_2 = model.add_module("activation_2", ReLU())
    layer_3 = model.add_module("linear_3", torch.nn.Linear(2048, output_size))

Output:
They way skip connections are usually implemented is in the forward function.
For example:
from torch import nn
import torch.nn.functional as nnf

class MLPWithSkip(nn.Module):
  def __init__(self, input_size, output_size):
    self.linear_modules = nn.ModuleList([nn.Linear(input_size, 2048),
                                         nn.linear(2048, 2048),
                                         nn.Linear(2048, 2048),
                                         nn.linear(2048, output_size)])

  def forward(self, x):
    h = []
    for layer in self.linear_modules[:-1]:
      x = layer(x)
      h.append(x)  # store the features
      x = nnf.relu(x)
    # implement the skip
    x = nnf.relu(h[-1] + h[-2])
    y = self.linear_modules[-1](x)
    return y


Instruction: What is the correct implementation for rounding a 2D tensor given the rounding values in a 1D tensor?

Input:
This is what I have done so far:
def round_values(predictions):
    # Rounding values
    rounded = torch.tensor([1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])

    # Find nearest rounding value
    dif = predictions.unsqueeze(2) - rounded.view(1,1,-1)
    indexes = torch.argmin(abs(dif), dim=2)

    # Fill tensor with nearest round value
    # TO IMPROVE
    rounded_predictions = torch.zeros(*predictions.shape)
    for i in range(rounded_predictions.shape[0]):
        for j in range(rounded_predictions.shape[1]):
            index = indexes[i,j]
            rounded_predictions[i,j] = rounded[index]

    return rounded_predictions

a = 4*torch.rand((6,6)) + 1
print(a)
print(round_values(a))

The double loop is terrible. I would like something like rounded[indexes], so it returns a tensor with the same shape as indexes with the values of rounded, but I have not found a way to do it at tensor level instead of element-wise. I would like  Any idea?

Output:
A thousand times faster for a 100 x 100 array (on cpu)
def round_values_bob(predictions):
    return torch.clamp(torch.round(predictions * 2) / 2, 1, 5)