Instruction: Problem with Deep Sarsa algorithm which work with pytorch (Adam optimizer) but not with keras/Tensorflow (Adam optimizer)

Input:
I have a deep sarsa algorithm which work great on Pytorch on lunar-lander-v2 and I would use with Keras/Tensorflow. It use mini-batch of size 64 which are used 128 time to train at each episode.
There are the results I get. As you can see, it work great with Pytorch but not with Keras / Tensorflow... So I think I do not correctly implement the training function is Keras/Tensorflow (code is below).
It seems that loss is oscillating in Keras because epsilon go to early to slow value but it work very great in Pytorch...
Do you see something that could explain why it do not work in Keras/Tensorflow please? Thanks a lot for your help and any idea that could help me...

Network information:
It use Adam optimizer, and a network with two layers : 256 and 128, with relu on each:
class Q_Network(nn.Module):
def __init__(self, state_dim , action_dim):
    super(Q_Network, self).__init__()
    self.x_layer = nn.Linear(state_dim, 256)
    self.h_layer = nn.Linear(256, 128)
    self.y_layer = nn.Linear(128, action_dim)
    print(self.x_layer)

def forward(self, state):
    xh = F.relu(self.x_layer(state))
    hh = F.relu(self.h_layer(xh))
    state_action_values = self.y_layer(hh)
    return state_action_values

For keras/Tensorflwo I use this one:
def CreationModele(dimension):
  entree_etat = keras.layers.Input(shape=(dimension))

  sortie = keras.layers.Dense(units=256, activation='relu')(entree_etat)
  sortie = keras.layers.Dense(units=128, activation='relu')(sortie)
  sortie = keras.layers.Dense(units=4)(sortie)

  modele = keras.Model(inputs=entree_etat,outputs=sortie)
  return modele

Training code
In Pytorch, the training is done by:
def update_Sarsa_Network(self, state, next_state, action, next_action, reward, ends):

    actions_values = torch.gather(self.qnet(state), dim=1, index=action.long())

    next_actions_values = torch.gather(self.qnet(next_state), dim=1, index=next_action.long())

    next_actions_values = reward + (1.0 - ends) * (self.discount_factor * next_actions_values)

    q_network_loss = self.MSELoss_function(actions_values, next_actions_values.detach())
    self.qnet_optim.zero_grad()
    q_network_loss.backward()
    self.qnet_optim.step()
    return q_network_loss

And in Keras/Tensorflow by:
mse = keras.losses.MeanSquaredError(
    reduction=keras.losses.Reduction.SUM)

@tf.function
def train(model, batch_next_states_tensor, batch_next_actions_tensor, batch_reward_tensor, batch_end_tensor, batch_states_tensor, batch_actions_tensor, optimizer, gamma):
  with tf.GradientTape() as tape:
    # EStimation des valeurs des actions courantes
    actions_values = model(batch_states_tensor)                                                          # (mini_batch_size,4)
    actions_values = tf.linalg.diag_part(tf.gather(actions_values,batch_actions_tensor,axis=1))         # (mini_batch_size,)
    actions_values = tf.expand_dims(actions_values,-1)                                                  # (mini_batch_size,1)

    # EStimation des valeurs des actions suivantes
    next_actions_values = model(batch_next_states_tensor)                                                          # (mini_batch_size,4)
    next_actions_values = tf.linalg.diag_part(tf.gather(next_actions_values,batch_next_actions_tensor,axis=1))   # (mini_batch_size,)
    cibles = batch_reward_tensor + (1.0 - batch_end_tensor)*gamma*tf.expand_dims(next_actions_values,-1)         # (mini_batch_size,1)

    error = mse(cibles, actions_values)

  grads = tape.gradient(error, model.trainable_variables)
  optimizer.apply_gradients(zip(grads, model.trainable_variables))

  return error

Error function and Optimizer code
The optimizer is Adam in Pytorch and Tensorflow with lr=0.001. In Pytorch:
def __init__(self, state_dim, action_dim):
    self.qnet = Q_Network(state_dim, action_dim)
    self.qnet_optim = torch.optim.Adam(self.qnet.parameters(), lr=0.001)
    self.discount_factor = 0.99
    self.MSELoss_function = nn.MSELoss(reduction='sum')
    self.replay_buffer = ReplayBuffer()
    pass

In Keras / Tensorflow:
alpha = 1e-3

# Initialise le modèle
modele_Keras = CreationModele(8)

optimiseur_Keras = keras.optimizers.Adam(learning_rate=alpha)

Output:
Ok I finnaly foud a solution by de-correlate target and action value using two model, one being updated periodically for target values calculation.
I use a model for estimating the epsilon-greedy actions and computing the Q(s,a) values and a fixed model (but periodically uptated with the weight of the previous model) for calculate the targer r+gamma*Q(s',a').
Here is my result :


Instruction: How to freeze param when I use transfer learning in python-pytorch

Input:
I want to learn only the first layer by transfer learning and fix(freeze) the parameters of the other layers.
but I was required to **requires_grad = True **.
How can I solve this problem?
The following is a description of what we tried and the errors we encountered.
from efficientnet_pytorch import EfficientNet

model_b0 = EfficientNet.from_pretrained('efficientnet-b0')
num_ftrs = model_b0._fc.in_features
model_b0._fc = nn.Linear(num_ftrs, 10)

for param in model_b0.parameters():
    param.requires_grad = False

last_layer = list(model_b0.children())[-1]

print(f'except last layer: {last_layer}')
for param in last_layer.parameters():
    param.requires_grad = True

criterion = nn.CrossEntropyLoss()
optimizer_ft = optim.SGD(model_b0.parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_b0 = train_model(model_b0, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=3)

if I change requires_grad = True , above code can run.
and error is
      4 optimizer_ft = optim.SGD(model_b7.parameters(), lr=0.001, momentum=0.9)
      5 exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
----> 7 model_b0 = train_model(model_b7, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=15)

Cell In [69], line 43, in train_model(model, criterion, optimizer, scheduler, num_epochs)
     41 loss = criterion(outputs, labels)
---> 43 loss.backward()
     44 optimizer.step()

\site-packages\torch\_tensor.py:396, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    394         create_graph=create_graph,
    395         inputs=inputs)
--> 396 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

\site-packages\torch\autograd\__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    172 # calls in the traceback and some print out the last line
--> 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

thank you for read!

Output:
There are several possible causes of this problem:

The input:
RuntimeError: element 0 of variables does not require grad and does not have a grad_fn

The tensor you passed in does not have the requires_grad=True.
Make sure your new Variable is with requires_grad = True:
var_xs_h = Variable(xs_h.data, requires_grad=True)

The requires_grad:  Freeze last layers of the model

As mentioned by the Pytorch forum moderator ptrblck:

If you are setting requires_grad = False for all parameters, the error
message is expected, as Autograd won’t be able to calculate any
gradients, since no parameter requires them.

Which I think your case is similar to the latter one, you can read the second post.
Another suggestion by ptrblck in debugging.
# standard use case
x = torch.randn(1, 1)
print(x.requires_grad)
# > False

lin = nn.Linear(1, 1)
out = lin(x)
print(out.grad_fn)
# > <AddmmBackward0 object at 0x7fcea08c5610>
out.backward()
print(lin.weight.grad)
# > tensor([[-0.9785]])
print(x.grad)
# > None

# input requires grad
x = torch.randn(1, 1, requires_grad=True)
print(x.requires_grad)
# > True

lin = nn.Linear(1, 1)
out = lin(x)
print(out.grad_fn)
# > <AddmmBackward0 object at 0x7fcea08d4640>
out.backward()
print(lin.weight.grad)
# > tensor([[1.6739]])
print(x.grad)
# >tensor([[0.0300]])


Instruction: Iterating over torch tensor

Input:
What is the best and fastest way to iterate over Tensor. It is confusing why do I get tensor instead of value..
got this :
  [ x for x in t]
  Out[122]: [tensor(-0.12), tensor(-0.11), tensor(0.68), tensor(0.68), tensor(0.17)]

but expected this behavior :
  [ x for x in t.numpy() ]
  Out[123]: [-0.11932722, -0.114598714, 0.67563725, 0.6756373, 0.16548502]

I would prefer not to convert to numpy if possible ?

Output:
With numpy everything is simpler because np.arrays are just a collection of numbers always stored on CPU. Therefore, if you iterate over an np.array you get these float numbers.
However, in PyTorch, tensors store not only numbers but also their gradients. Additionally, PyTorch tensors may be stored on CPU or GPU. Thus, in order to preserve all this "side-information", PyTorch returns single-element tensors when iterating over a tensor.
If you insist on getting simple "numbers" from a tensor, you can use tensor.item():
 [x.item() for x in t]

Or, tensor.tolist():
t.tolist()

For more information on the differences between numpy np.arrays and torch.tensors, see this answer.


Instruction: How to freeze part of selected layer(eg nn.Linear()) of a model in Pytorch?

Input:
question:
fc = nn.Linear(n,3);
I want to freeze the parameters of the third output of the fc when I train this layer.

Output:
According to @PlainRavioli , it's not possible yet and you can set the gradient to zero so the current weights do not change.
But you have to do this after calling loss.backward() and before calling optimizer.step().
So being fc = nn.Linear(n,3), for freezing the parameters of the third output:
loss.backward()
fc.weight.grad[2,:] = torch.zeros_like(fc.weight.grad[2,:])
fc.bias.grad[2] = torch.zeros_like(fc.bias.grad[2])
optimizer.step()

Calling loss.backward() computes dloss/dx and do  x.grad += dloss/dx. So before this operation, gradients are set to None.


Instruction: 'RuntimeError: Expected object of scalar type Long but got scalar' for torch.nn.CrossEntropyLoss()

Input:
I'm using this loss function for xlm-roberta-large-longformer and it gives me this error:
    import torch.nn.functional as f
    from scipy.special import softmax

    loss_func = torch.nn.CrossEntropyLoss()
    output = torch.softmax(logits.view(-1,num_labels), dim=0).float()
    target = b_labels.type_as(logits).view(-1,num_labels)
    loss = loss_func(output, target)
    train_loss_set.append(loss.item())

when I try
b_labels.type_as(logits).view(-1,num_labels).long()

it tells me

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

What should I do?

Output:
Your target tensor should contain integers corresponding to the correct class labels and should not be a one/multi-hot encoding of the class.
You can extract the class labels from a one-hot encoding format using argmax:
>>> b_labels.argmax(1)


Instruction: Pytorch input tensor is correct data type, but throws RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #3

Input:
I am trying to write a simple CNN using PyTorch, but am getting an error in my first layer. The following lines of code will produce the error RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #3 'mat1' in call to _th_addmm_, despite me setting the input to type Long. The print statement in the 3rd line also gives torch.LongTensor, yet the fourth line still throws the error.
import torch
import torch.nn as nn

data = torch.randint(low=0, high=255, size=[2, 1, 1024, 1024], dtype=torch.int64)
model = nn.Conv2d(1, 3, kernel_size=3, padding=1, bias=False)
print(data.type())
out = model(data)

Output:
The type of your data is Long, but the type of the weights of your model is Float. You need to change the type of your data if you are planning to train a model:
import torch
import torch.nn as nn

data = torch.randint(low=0, high=255, size=[2, 1, 1024, 1024], dtype=torch.float32)
model = nn.Conv2d(1, 3, kernel_size=3, padding=1, bias=False)
print(data.type())
out = model(data)


Instruction: How to use the command "with" conditionally without duplication of code

Input:
I am trying to use with before executing a block of code, but only if a condition is met, but (at least the common) usage of with doesn't appear to support that unless I duplicate the block of code.
More concretely, I know I can do the following:
if condition:
    with blah_blah():
        my_code_block
else:
    my_code_block

But that's unsatisfying during development since any change I make to my_code_block must be made twice.  What I want to do (conceptually) is:
if condition:
    with blah_blah():
else:
    my_code_block

That doesn't work, though.  Is there a way to accomplish what I'm trying to do?
For anyone that's interested in my particular use, I'm trying to write code that runs a batch of examples in pytorch, with torch.no_grad() if I'm in evaluation mode and without it if I'm in train mode.  So what I want to do becomes
if mode == 'eval':
    with torch.no_grad():
else:
    run_batch(features, labels)

Output:
Use the with statement, but with a nullcontext context manager if necessary.
from contextlib import nullcontext

with blah_blah() if condition else nullcontext():
    my_code_block

nullcontext takes an optional argument that it will return if your with statement expects something to be bound with as. For example,
with nullcontext("hello") as f:
    print(f)  # outputs "hello"


Instruction: stable-baselines3 PPO model loaded but not working

Input:
I am trying to make an AI agent for playing OpenAI Gym CarRacing environment and I am having trouble loading saved models. I train them, they work, I save them and load them and suddenly the car doesn't even move. I even tried downloading models from other people and when loaded, the car just doesn't move.
I am on Ubuntu 20.04 in VS Code in a Jupyter notebook using
gym==0.21.0, stable-baselines3==1.6.0, python==3.7.0
import gym
from stable_baselines3 import PPO
from stable_baselines3.common.evaluation import evaluate_policy
import os

I make the environment
environment_name = "CarRacing-v0"
env = gym.make(environment_name)

I create the PPO model and make it learn for a couple thousand timesteps. Now when I evaluate the policy, the car renders as moving.
log_path = os.path.join('Training', 'Logs')
model = PPO("CnnPolicy", env, verbose=1, tensorboard_log=log_path)
model.learn(total_timesteps=4000)
evaluate_policy(model, env, n_eval_episodes=1, render=True)

I save the model
ppo_path = os.path.join('Training', 'Saved Models', 'PPO_Car_Testing')
model.save(ppo_path)

now I delete the model and load the saved one and when I evaluate it the car just doesn't move as if it always got action do nothing. I tried models learning for 2k timesteps up to a model which has been learning for 2 million timesteps.
del model
model = PPO("CnnPolicy", env, verbose=1, tensorboard_log=log_path)
ppo_path_load = os.path.join('Training', 'Saved Models', 'PPO_2m_Driving_model')
model.load(ppo_path_load, env)
evaluate_policy(model, env, n_eval_episodes=1, render=True)

Any ideas why the models load incorrectly?

Output:
it seems like you model didn't load Correctly
you gave the code wrong
model = PPO("CnnPolicy", env, verbose=1, tensorboard_log=log_path)
ppo_path_load = os.path.join('Training', 'Saved Models', 'PPO_2m_Driving_model')
model.load(ppo_path_load, env)

Change it to as
model = PPO("CnnPolicy", env, verbose=1, tensorboard_log=log_path)
ppo_path_load = os.path.join('Training', 'Saved Models', 'PPO_2m_Driving_model')
model = ppo.load(ppo_path_load, env)

change RLALGORITHM to your Rl-agorithm such as PPO or A2C, etc
model = RLALGORITHM.load(ppo_path_load, env)


Instruction: Efficient pseudo-inverse for PyTorch 2D convolution

Input:
Background:
Thanks for your attention! I am learning the basic knowledge of 2D convolution, linear algebra and PyTorch. I encounter the implementation problem about the psedo-inverse of the convolution operator. Specifically, I have no idea about how to implement it in an efficient way. Please see the following problem statements for details. Any help/tip/suggestion is welcomed.
(Thanks a lot for your attention!)

The Original Problem:
I have an image feature x with shape [b,c,h,w] and a 3x3 convolutional kernel K with shape [c,c,3,3]. There is y = K * x. How to implement the corresponding pseudo-inverse on y in an efficient way?

There is [y = K * x = Ax], how to implement [x_hat = (A^+)y]?

I guess that there should be some operations using torch.fft. However, I still have no idea about how to implement it. I do not know if there exists an implementation previously.
import torch
import torch.nn.functional as F

c = 32
K = torch.randn(c, c, 3, 3)
x = torch.randn(1, c, 128, 128)
y = F.conv2d(x, K, padding=1)

print(y.shape)

# How to implement pseudo-inverse for y = K * x in an efficient way?

Some of My Efforts:
I may know that the 2D convolution is a linear operator. It is equivalent to a "matrix product" operator. We can actually write out the matrix form of the convolution and calculate its psedo-inverse. However, I think this type of operation will be inefficient. And I have no idea about how to implement it in an efficient way.
According to Wikipedia, the psedo-inverse may satisfy the property of A(A_pinv(x))=x, where A is the convolutional operator, A_pinv is its psedo-inverse, and x may be any image feature.
(Thanks again for reading such a long post!)

Output:
This takes the problem to another level.
The convolution itself is a linear operation, you can determine the matrix of the operation and solve a least square problem directly [1], or compute the pseudo-inverse as you mentioned, and then apply to different outputs and predicting a projection of the input.
I am changing your code to using padding=0
import torch
import torch.nn.functional as F

# your code
c = 32
K = torch.randn(c, c, 1, 1)
x = torch.randn(4, c, 128, 128)
y = F.conv2d(x, K, bias=torch.zeros((c,)))

Also, as you probably already suggested the convolution can be computed as ifft(fft(h)*fft(x)). However, the conv2d function is a cross-correlation, so you have to conjugate the filter leading to ifft(fft(h)*fft(x)), also you have to apply this to two axes, and you have to make sure the FFT is calcuated using the same representation (size), since the data is real, we can apply multi-dimensional real FFT. To be complete, conv2d works on multiple channels, so we have to calculate summations of convolutions. Since the FFT is linear, we can simply compute the summations on the frequency domain
using einsum.
s = y.shape[-2:]
K_f = torch.fft.rfftn(K, s)
x_f = torch.fft.rfftn(x, s)
y_f = torch.einsum('jkxy,ikxy->ijxy', K_f.conj(), x_f)
y_hat = torch.fft.irfftn(y_f, s)

Except for the borders it should be accurate (remember FFT computes a cyclic convolution).
torch.max(abs(y_hat[:,:,:-2,:-2] - y[:,:,:,:]))

Now, notice the pattern jk,ik->ij on the einsum, that means y_f[i,j] = sum(K_f[j,k] * x_f[i,k]) = x_f @ K_f.T, if @ is the matrix product on the first two dimensions. So to invert this operation we have to can interpret the first two dimensions as matrices. The function pinv will compute pseudo-inverses on the last two axes, so in order to use that we have to permute the axes. If we right multiply the output by the pseudo-inverse of transposed K_f we should invert this operation.
s = 128,128
K_f = torch.fft.rfftn(K, s)
K_f_inv = torch.linalg.pinv(K_f.T).T
y_f = torch.fft.rfftn(y_hat, s)
x_f = torch.einsum('jkxy,ikxy->ijxy', K_f_inv.conj(), y_f)
x_hat = torch.fft.irfftn(x_f, s)
print(torch.mean((x - x_hat)**2) / torch.mean((x)**2))

Nottice that I am using the full convolution, but the conv2d actually cropped the images. Let's apply that
y_hat[:,:,128-(k-1):,:] = 0
y_hat[:,:,:,128-(k-1):] = 0

Repeating the calculation you will see that the input is not accurate anymore, so you have to be careful about what you do with your convolution, but in some situations where you can get this to work it will be in fact efficient.
s = 128,128
K_f = torch.fft.rfftn(K, s)
K_f_inv = torch.linalg.pinv(K_f.T).T
y_f = torch.fft.rfftn(y_hat, s)
x_f = torch.einsum('jkxy,ikxy->ijxy', K_f_inv.conj(), y_f)
x_hat = torch.fft.irfftn(x_f, s)
print(torch.mean((x - x_hat)**2) / torch.mean((x)**2))


Instruction: How to pass `--gpus all` option to Docker with Go SDK?

Input:
I have seen how to do some basic commands such as running a container, pulling images, listing images, etc from the SDK examples.
I am working on a project where I need to use the GPU from within the container.
My system has GPU, I have installed the drivers, and I have also installed the nvidia-container-runtime.
If we remove Go SDK from the scene for a moment, I can run the following command to get the nvidia-smi output on my host system:
docker run -it --rm --gpus all nvidia/cuda:10.0-base nvidia-smi

I have to do this via the SDK. Here is the code to start with. This code prints "hello world". But in actual I will be running nvidia-smi command at that place:
package main

import (
    "context"
    "os"

    "github.com/docker/docker/api/types"
    "github.com/docker/docker/api/types/container"
    "github.com/docker/docker/client"
    "github.com/docker/docker/pkg/stdcopy"
)

func main() {
    ctx := context.Background()
    cli, err := client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
    if err != nil {
        panic(err)
    }

    RunContainer(ctx, cli)
}

func RunContainer(ctx context.Context, cli *client.Client) {
    reader, err := cli.ImagePull(ctx, "nvidia/cuda:10.0-base", types.ImagePullOptions{})
    if err != nil {
        panic(err)
    }

    defer reader.Close()
    // io.Copy(os.Stdout, reader)

    resp, err := cli.ContainerCreate(ctx, &container.Config{
        Image: "nvidia/cuda:10.0-base",
        Cmd:   []string{"echo", "hello world"},
        // Tty:   false,
    }, nil, nil, nil, "")

    if err != nil {
        panic(err)
    }

    if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
        panic(err)
    }

    statusCh, errCh := cli.ContainerWait(ctx, resp.ID, container.WaitConditionNotRunning)

    select {
    case err := <-errCh:
        if err != nil {
            panic(err)
        }
    case <-statusCh:
    }

    out, err := cli.ContainerLogs(ctx, resp.ID, types.ContainerLogsOptions{ShowStdout: true})
    if err != nil {
        panic(err)
    }

    stdcopy.StdCopy(os.Stdout, os.Stderr, out)
}

Output:
see: https://github.com/docker/cli/blob/9ac8584acfd501c3f4da0e845e3a40ed15c85041/cli/command/container/opts.go#L594
import "github.com/docker/cli/opts"

// ...

gpuOpts := opts.GpuOpts{}
gpuOpts.Set("all")

resp, err := cli.ContainerCreate(ctx, &container.Config{
    Image: "nvidia/cuda:10.0-base",
    Cmd:   []string{"echo", "hello world"},
    // Tty:   false,
}, &container.HostConfig{Resources: container.Resources{DeviceRequests: gpuOpts.Value()}}, nil, nil, "")


Instruction: Gradients not populating as expected

Input:
Sorry, I know questions of this sort have been asked a lot, but I still don't understand the behavior of autograd.
A simple example is below:
ce_loss=torch.nn.BCELoss()
par=torch.randn((1,n),requires_grad=True)
act=torch.nn.Sigmoid()

y_hat=[]
for obs in data:
    y_hat.append(act(par@obs))

loss=ce_loss(torch.tensor(y_hat,requires_grad=True),y)
loss.backward()

After applying backward, the grad of par remains None (although it is a leaf node with requires_grad=True).
Any tips?

Output:
It is simply because torch.tensor(...) create a new leaf of the computational graph. It means by definition that the operation inside torch.tensor are blocked, in particular the computation using elements of par (and so, the grads are never computed). Note that adding requires_grad=True doesn't change anything because it always creates a leaf (with grads) that forgot the previous operations by definition of a leaf.
I suggest you an other way to make your computation without iterate on data and using native parallelization:
batch_size, n = 8, 10  # or something else

# Random data and labels to reproduce the code
data = torch.randn((batch_size, n))
y = torch.randn((batch_size, ))

y = y.unsqueeze(1)  # size (batch_size, 1)
ce_loss = torch.nn.BCELoss()
par = torch.randn((1, n), requires_grad=True)
act = torch.nn.Sigmoid()

y_hat = act(data @ par.T)  # compute all predictions in parallel

loss = ce_loss(y_hat, y)  # automatically reduced to scalar (mean)
loss.backward()

print(par.grad)  # no longer None!


Instruction: PyTorch: how to use torchvision.transforms.AugMIx with torch.float32?

Input:
PyTorch: how to use torchvision.transforms.AugMIx with torch.float32?
I am trying to apply data augmentation in image dataset by using torchvision.transforms.AugMIx, but I have the following error: TypeError: Only torch.uint8 image tensors are supported, but found torch.float32.
I tried to convert it to int, but I have another error.

My code where I am trying to use the AugMix function:
transform = torchvision.transforms.Compose(
    [
        torchvision.transforms.Resize((224, 224)),  # resize to 224*224
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),  # normalization
        torchvision.transforms.AugMix()
    ]
)
to_tensor = torchvision.transforms.ToTensor()
Image.MAX_IMAGE_PIXELS = None

class BreastDataset(torch.utils.data.Dataset):

    def __init__(self, json_path, data_dir_path='./dataset', clinical_data_path=None, is_preloading=True):
        self.data_dir_path = data_dir_path
        self.is_preloading = is_preloading

        with open(json_path) as f:
            print(f"load data from {json_path}")
            self.json_data = json.load(f)

    def __len__(self):
        return len(self.json_data)

    def __getitem__(self, index):
        label = int(self.json_data[index]["label"])
        patient_id = self.json_data[index]["id"]
        patch_paths = self.json_data[index]["patch_paths"]

        data = {}
        if self.is_preloading:
            data["bag_tensor"] = self.bag_tensor_list[index]
        else:
            data["bag_tensor"] = self.load_bag_tensor([os.path.join(self.data_dir_path, p_path) for p_path in patch_paths])

        data["label"] = label
        data["patient_id"] = patient_id
        data["patch_paths"] = patch_paths

        return data

    def load_bag_tensor(self, patch_paths):
        """Load a bag data as tensor with shape [N, C, H, W]"""

        patch_tensor_list = []
        for p_path in patch_paths:
            patch = Image.open(p_path).convert("RGB")
            patch_tensor = transform(patch)  # [C, H, W]
            patch_tensor = torch.unsqueeze(patch_tensor, dim=0)  # [1, C, H, W]
            patch_tensor_list.append(patch_tensor)

        bag_tensor = torch.cat(patch_tensor_list, dim=0)  # [N, C, H, W]

        return bag_tensor

Any help is appreciated! Thank you in advance!

Output:
For me applying AugMix first and then ToTensor() worked
transformation = transforms.Compose([
                    transforms.AugMix(severity= 6,mixture_width=2),
                    transforms.ToTensor(),
                    transforms.RandomErasing(),
                    transforms.RandomGrayscale(p = 0.35)
                    ])


Instruction: Pytorch tensor and it's transpose have different storage

Input:
I was reading the book Deep Learning with Pytorch and was trying out an example which shows that a tensor and it's transpose share the same storage.
However, when i tried it out on my local machine, I can see that the storage is different for both. Just wanted to understand why this might be the case here ?
The code i tried and the output is as below:
>>> points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
>>> points_t = torch.transpose(points,0,1)
>>> points_t
tensor([[4., 5., 2.],
        [1., 3., 1.]])
>>> id(points.storage())==id(points_t.storage())
False
>>> id(points.storage())
2796700202176
>>> id(points_t.storage())
2796700201888

My python version is 3.9.7 and pytorch version is 1.11.0

Output:
You need to compare the pointer of storages instead of taking the id of it.
>>> points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
>>> points_t = torch.transpose(points,0,1)
>>> points_t
tensor([[4., 5., 2.],
        [1., 3., 1.]])
>>> points.storage().data_ptr() == points_t.storage().data_ptr()
True

The reason you are getting False for id comparison is that Python objects (points and points_t) are different objects but the underlying storages (the memory that you allocate to keep the data) are the same.


Instruction: How to perform multiplication along axes in pytorch?

Input:
I have 2 tensors X and Y - X has shape (20,4,300) and Y has shape(20,300) . How to perform multiplication such that I have an result of shape (20,4). The corresponding techinique in keras is
doc_product = Dot(axes=(2,1))([X,Y])

I would like to know how the same can be done in pytorch?

Output:
Your most versatile function for matrix multiplication is torch.einsum: it allows you specify the dimensions along which to multiply and the order of the dimensions of the output tensor.
In your case it would look like:
dot_product = torch.einsum('bij,bj->bi')


Instruction: Problem with nested network on pytorch ."TypeError: forward() missing 1 required positional argument: 'x'"

Input:
I attempt to create an architecture consisting of one convolutional filter and one layer of three convolutional filters. I first build the inner layer with the name "MysmallNet(nn.module)", and then I build "MybigNet" calling the small network. This is my code.
#In[]
class MysmallNet(nn.Module):
    def __init__(self):
        super(MysmallNet, self).__init__()
        # TODO Task 3: Design Your Network
        self.Convlayer_1 = nn.Conv2d(in_channels = 16, out_channels = 16, kernel_size = 3, stride = 1,padding=1)
        self.Convlayer_2 = nn.Conv2d(in_channels=16,out_channels=16,kernel_size=3,stride=1, padding=1)
        self.Convlayer_3 = nn.Conv2d(in_channels=16,out_channels=16,kernel_size=3,stride=1, padding=1)

    def forward(self, x):
        # TODO Task 3: Design Your Network
        residual1 = x
        x = self.Convlayer_1(x)
        x = self.Convlayer_2(x)
        x = self.Convlayer_3(x)
        return x

MysmallNetV2= MysmallNet()

class MybigNet(nn.Module):
    def __init__(self):
        super(MybigNet, self).__init__()

        self.Convlayer_1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3,stride=1,padding=1)
        self.smallNet= MysmallNetV2()

    def forward(self, x):
        x = self.Convlayer_1(x)
        x = self.smallNet(x)
        return x

modelBig = MybigNet()

I have the issue when I save my model as "modelBig". The displayed error is :
TypeError: forward() missing 1 required positional argument: 'x'

Output:
Your definition of big net is wrong, it should be:
class MybigNet(nn.Module):
    def __init__(self):
        super(MybigNet, self).__init__()

        self.Convlayer_1 = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3,stride=1,padding=1)
        self.smallNet= MysmallNet()

    def forward(self, x):
        x = self.Convlayer_1(x)
        x = self.smallNet(x)
        return x

This should solve the issue.


Instruction: How to extract Integer from Pytorch Tensor

Input:
This is a part of code... VSCODE declares variable  > xyxy as 'list'
                for *xyxy, conf, cls in reversed(det):
                if save_txt:  # Write to file
                    xywh = (xyxy2xywh(torch.tensor(xyxy).view(1, 4)) / gn).view(-1).tolist()  # normalized xywh
                    line = (cls, *xywh, conf) if opt.save_conf else (cls, *xywh)  # label format
                    with open(txt_path + '.txt', 'a') as f:
                        f.write(('%g ' * len(line)).rstrip() % line + '\n')

                if save_img or view_img:  # Add bbox to image
                    label = f'{names[int(cls)]} {conf:.2f}'
                    plot_one_box(xyxy, im0, label=label, color=colors[int(cls)], line_thickness=1)

                else :
                    pass #

        # Print time (inference + NMS)
        print(f'{s}Done. ({(1E3 * (t2 - t1)):.1f}ms) Inference, ({(1E3 * (t3 - t2)):.1f}ms) NMS')
        #print(xyxy_custom = xyxy.numpy()`

Printing variable xyxy gives this output (possibly pytorch tensor)
[tensor(513., device='cuda:0'), tensor(308., device='cuda:0'), tensor(661., device='cuda:0'), tensor(394., device='cuda:0')]

I want to extract integers from this list (for example output should be [513,308,661,394]
i have tried print(xyxy.list()) or print(xyxy.numpy(). this gives an error
AttributeError: 'list' object has no attribute 'list'

Output:
You can convert the elements of the list into integers using .item():
xyxy = [int(e_.item()) for e_ in xyxy]


Instruction: Pytorch gradient calulation of one Tensor

Input:
I'm a beginner in pytorch and I'm probably stuck on a relatively trivial problem, but it's not clearing up for me at the moment.
When calculating the gradient of a tensor I get a constant gradient of 1.
Shouldn't the gradient of a constant however result in 0?
Here is a minimal example:
import torch
x = torch.tensor(50., requires_grad=True)

y = x

y.backward()

print(x.grad)
#Ouput: tensor(1.)

So why is the ouput 1 and not 0?

Output: You are not computing the gradient of a constant, but that of the variable x which has a constant value 50. The derivative of x with respect to x is 1.


Instruction: Pytorch custom dataset is super slow

Input:
During training it takes ages to load one batch of data. What can cause this problem?
I am new to Pytorch, i had been working with tensorflow for a while, this my first attempt to create something like this.
I wrote a custom dataset which gets its images from folders, it gets stored in a dataframe which will be splited it into train val sets.
class CustomDataset(torch.utils.data.Dataset):
def __init__(self, root, split, train_ratio = 0.85, val_ratio = 0.1, transform=None):
    self.root = root

    self.train_ratio = train_ratio
    self.val_ratio = val_ratio
    self.test_ratio = 1 - (self.train_ratio + self.val_ratio)

    df = self.folder2pandas()
    self.split = split
    self.data = self.splitDf(df)
    self.transform = transform

def __len__(self):
    return len(self.data)

def __getitem__(self, idx):
    row = self.data.iloc[[idx]]

    x = row.values[0][0]
    y = row.values[0][1]

    x = cv2.imread(x)

    if self.sourceTransform:
        x = self.sourceTransform(x)

    return x, y

def folder2pandas(self):
    tuples = []
    for folder, subs, files in os.walk(self.root):
        for filename in files:
            path = os.path.abspath(os.path.join(folder, filename))
            tuples.append((path, folder.split('\\')[-1]))
    return pd.DataFrame(tuples, columns=["x", "y"])

def splitDf(self, df):
    df = df.sort_values(by=['x'], ascending=True).reset_index(drop=True)
    train_idxs = df.loc[range(0, int(self.train_ratio * len(df)))]
    val_idxs = df.loc[range(int(self.train_ratio * len(df)),
                            int(self.train_ratio * len(df)) + int(self.val_ratio * len(df)))]
    test_idxs = df.loc[range(
        int(self.train_ratio * len(df)) + int(self.val_ratio * len(df)), len(df))]

    if self.split == 'train':
        return train_idxs
    elif self.split == 'val':
        return val_idxs
    elif self.split == 'test':
        return test_idxs

Augmentations:
train_transforms = transforms.Compose([
transforms.Resize((224,224)),
transforms.RandomChoice([
    transforms.RandomAutocontrast(),
    transforms.ColorJitter(brightness=0.3, contrast=0.5, saturation=0.1, hue=0.1),
    transforms.GaussianBlur(kernel_size=(5,5), sigma=(0.1, 2.0)),
    transforms.Grayscale(num_output_channels=3),
    transforms.RandomVerticalFlip(),]),
transforms.RandomHorizontalFlip(0.5),
transforms.ToTensor(),
transforms.Normalize(res[0].numpy(), res[1].numpy()),
])

val_transforms = transforms.Compose([
    transforms.Resize((224,224)),
    transforms.ToTensor(),
    transforms.Normalize(res[0].numpy(), res[1].numpy()),
])

Initializing datasets:
In 'resources' folder there are two folder which name's represents the labels (BinaryClassification).
train_set=CustomDataset(root="resources/",split='train', transform=train_transforms)
val_set=CustomDataset(root="resources/",split='val', transform=val_transforms)

Giving datasets to dataloader:
trainloader = torch.utils.data.DataLoader(train_set, shuffle = True, batch_size=32, num_workers=4)
testloader = torch.utils.data.DataLoader(val_set, shuffle = True, batch_size=32, num_workers=4)

Output:
Putting the solution of the comments in a cleaner way:
The creation of several workers was taking large amount of time. It seems that on windows the creation of processes can have weird behaviours in terms of time.
As __getitem__() is not called, the problem is not in data loading per se, try removing the num_workers parameters.
testloader = torch.utils.data.DataLoader(val_set, shuffle = True, batch_size=32)

Then if this works, try increasing it and check the behaviour.


Instruction: Pytorch random choose an index with condition

Input:
I have a tensor that stores whether or not an index in available
available = torch.Tensor([1,1,0,0,1,0])

and I want to return an index of either 0, 1, or 4 given that available[0],available[1],available[4] all equal to 1 with the same possibility.
can somebody help me with this? Thanks

Output:
Torch makes this easy. You can use multinomial as per this answer:
num_samples = 1
available.multinomial(num_samples, replacement=False)

Here, num_samples indicates how many samples you'd like to draw.
Because you have 1s and 0s already, your available tensor naturally gives the correct weights for the multinomial function.
If you are going to draw more than 3 samples, this will error unless you change replacement to True.


Instruction: Why are some nn.Linear layers not quantized by Pytorch?

Input:
I'm quantizing the Swin transformer (static PTQ) using the following function:
def static_quantize(m, data_loader):
    backend = 'qnnpack'
    torch.backends.quantized.engine = backend
    m.eval()

    m.qconfig = torch.quantization.get_default_qconfig(backend)
    torch.quantization.prepare(m, inplace=True)

    with torch.no_grad():
        for i, data in enumerate(data_loader):
            if i >= 100:
                break
            result = m(return_loss=False, **data)

    torch.quantization.convert(m, inplace=True)

    return m

Most modules, including linear layers, do get quantized. However some linear layers of a SwinBlock are skipped, as you can see here:
(3): SwinBlockSequence(
  (blocks): ModuleList(
    (0): SwinBlock(
      (quant): Quantize(scale=tensor([0.3938]), zero_point=tensor([122]), dtype=torch.quint8)
      (dequant): DeQuantize()
      (norm1): QuantizedLayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (attn): ShiftWindowMSA(
        (w_msa): WindowMSA(
          (quant): Quantize(scale=tensor([0.0294]), zero_point=tensor([155]), dtype=torch.quint8)
          (dequant): DeQuantize()
          (qkv): QuantizedLinear(in_features=768, out_features=2304, scale=0.039033032953739166, zero_point=133, qscheme=torch.per_tensor_affine)
          (attn_drop): Dropout(p=0, inplace=False)
          (proj): QuantizedLinear(in_features=768, out_features=768, scale=0.0369536317884922, zero_point=110, qscheme=torch.per_tensor_affine)
          (proj_drop): Dropout(p=0, inplace=False)
          (softmax): Softmax(dim=-1)
        )
        (drop): DropPath()
      )
      (norm2): QuantizedLayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (ffn): FFN( // <------- HERE (children not quantized)
        (activate): GELU()
        (layers): Sequential(
          (0): Sequential(
            (0): Linear(in_features=768, out_features=3072, bias=True)
            (1): GELU()
            (2): Dropout(p=0, inplace=False)
          )
          (1): Linear(in_features=3072, out_features=768, bias=True)
          (2): Dropout(p=0, inplace=False)
        )
        (dropout_layer): DropPath()
      )
    )

I am referring to the FFN submodule, where nothing is quantized. However, it contains linear layers, which ought to pose no problems for quantization.
Here's how FFN is added to the module:
        _ffn_cfgs = {
            'embed_dims': embed_dims,
            'feedforward_channels': int(embed_dims * ffn_ratio),
            'num_fcs': 2,
            'ffn_drop': 0,
            'dropout_layer': dict(type='DropPath', drop_prob=drop_path),
            'act_cfg': dict(type='GELU'),
            **ffn_cfgs
        }
        self.norm2 = build_norm_layer(norm_cfg, embed_dims)[1]
        self.ffn = FFN(**_ffn_cfgs)

Here's the source code for FFN:

@FEEDFORWARD_NETWORK.register_module()
class FFN(BaseModule):
    """Implements feed-forward networks (FFNs) with identity connection.

    Args:
        embed_dims (int): The feature dimension. Same as
            `MultiheadAttention`. Defaults: 256.
        feedforward_channels (int): The hidden dimension of FFNs.
            Defaults: 1024.
        num_fcs (int, optional): The number of fully-connected layers in
            FFNs. Default: 2.
        act_cfg (dict, optional): The activation config for FFNs.
            Default: dict(type='ReLU')
        ffn_drop (float, optional): Probability of an element to be
            zeroed in FFN. Default 0.0.
        add_identity (bool, optional): Whether to add the
            identity connection. Default: `True`.
        dropout_layer (obj:`ConfigDict`): The dropout_layer used
            when adding the shortcut.
        init_cfg (obj:`mmcv.ConfigDict`): The Config for initialization.
            Default: None.
    """

    @deprecated_api_warning(
        {
            'dropout': 'ffn_drop',
            'add_residual': 'add_identity'
        },
        cls_name='FFN')
    def __init__(self,
                 embed_dims=256,
                 feedforward_channels=1024,
                 num_fcs=2,
                 act_cfg=dict(type='ReLU', inplace=True),
                 ffn_drop=0.,
                 dropout_layer=None,
                 add_identity=True,
                 init_cfg=None,
                 **kwargs):
        super().__init__(init_cfg)
        assert num_fcs >= 2, 'num_fcs should be no less ' \
            f'than 2. got {num_fcs}.'
        self.embed_dims = embed_dims
        self.feedforward_channels = feedforward_channels
        self.num_fcs = num_fcs
        self.act_cfg = act_cfg
        self.activate = build_activation_layer(act_cfg)

        layers = []
        in_channels = embed_dims
        for _ in range(num_fcs - 1):
            layers.append(
                Sequential(
                    Linear(in_channels, feedforward_channels), self.activate,
                    nn.Dropout(ffn_drop)))
            in_channels = feedforward_channels
        layers.append(Linear(feedforward_channels, embed_dims))
        layers.append(nn.Dropout(ffn_drop))
        self.layers = Sequential(*layers)
        self.dropout_layer = build_dropout(
            dropout_layer) if dropout_layer else torch.nn.Identity()
        self.add_identity = add_identity

    @deprecated_api_warning({'residual': 'identity'}, cls_name='FFN')
    def forward(self, x, identity=None):
        """Forward function for `FFN`.

        The function would add x to the output tensor if residue is None.
        """
        out = self.layers(x)
        if not self.add_identity:
            return self.dropout_layer(out)
        if identity is None:
            identity = x
        return identity + self.dropout_layer(out)

Output:
The problem is very silly: Linear in this case referred to an mmcv wrapper class for nn.Linear. Quantizing the wrapper class is not supported.
class Linear(torch.nn.Linear):

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # empty tensor forward of Linear layer is supported in Pytorch 1.6
        if x.numel() == 0 and obsolete_torch_version(TORCH_VERSION, (1, 5)):
            out_shape = [x.shape[0], self.out_features]
            empty = NewEmptyTensorOp.apply(x, out_shape)
            if self.training:
                # produce dummy gradient to avoid DDP warning.
                dummy = sum(x.view(-1)[0] for x in self.parameters()) * 0.0
                return empty + dummy
            else:
                return empty

        return super().forward(x)

By the looks of it (since I'm using PyTorch 1.8.1) this can be easily remedied by modifying the FFN class to use nn.Linear.


Instruction: How is it possible that a list was given as a parameter to a function that expects a tuple?

Input:
I was watching a tutorial on PyTorch and coding along and got stuck on function torch.randint
According to the documentation:

torch.randint(low=0, high, size, \*, generator=None, out=None,
dtype=None, layout=torch.strided, device=None, requires_grad=False) →
Tensor

here, size is :

size (tuple) – a tuple defining the shape of the output tensor.

The YouTuber wrote
  random_idx = torch.randint(0, len(train_data), size=[1]).item()

But [1] is not a tuple, it is a list. How is this possible?
I also tested it with a tuple and it worked just fine and every usage of randint() i found on the internet provides a tuple for size. E.g. size = (1,2) or size = (1,1).
I searched the source code for torch.randint but could not find it. I searched GitHub, PyTorch docs and even tried to find it in a local PyTorch library.

Output:
The documentation states that this should be a tuple however in practice, the definition of randint() is:
def randint(low: _int, high: _int, size: _size, ... )

Where _size is defined as:
(type alias) _size: Type[Size] | Type[List[int]] | Type[Tuple[int, ...]]

So in practice, the requirement is for the size parameter to be of type Size, List of int or Tuple of int, which will pretty much behave the same in this case
EDIT:
As stated above, indeed, typing is only an indicative in Python, so if you use any type of variable, there won't be any issue if the function itself doesn't raise an error. For the question of why the function acts accordingly and returns what is expected, this is because of the first part of the answer :)


Instruction: What do I do wrong when install OpenKiwi?

Input:
I tried to work with OpenKiwi to Anaconda3
and after installation (pip install openkiwi)
I execute following code
(I do this because I want to create openkiwi vocabulary) :
import warnings
from collections import defaultdict
import torchtext
from kiwi.constants import PAD, START, STOP, UNALIGNED, UNK, UNK_ID

And than I have an error message:
ImportError                               Traceback (most recent call last)
<ipython-input-6-ea850b280bef> in <module>
      4 import torchtext
      5
----> 6 from kiwi.constants import PAD, START, STOP, UNALIGNED, UNK, UNK_ID

ImportError: cannot import name 'UNK_ID' from 'kiwi.constants' (C:\Users\Mike\anaconda3\lib\site-packages\kiwi\constants.py)

Anaconda3 has following versions: >!

pytorch-lightning: 1.7.6
pytorch-nlp: 0.5.0
torch: 1.4.0
torch metrics   0.9.3
torch: text 0.13.1
transformers: 3.5.1

Output: UNK_ID is no longer a constant in the latest version of OpenKiwi. That second link shows some code targeting version 1.4 or so, while OpenKiwi is at version 2.1 now. Just drop the UNK_ID from the import line, and replace the one in the code with 0.


Instruction: AssertionError when running U-Net script

Input:
This is a continuation of this problem. While I ironed out the problems I still get another issue. Would anyone be able to help me in this regards?
Looks like the predicted mask and actual mask have different sizes?
The output code is below:
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
/tmp/ipykernel_18/459131192.py in <module>
     25             with torch.set_grad_enabled(phase == "train"):
     26                 y_pred = unet(x)
---> 27                 loss = dsc_loss(y_pred, y_true)
     28                 running_loss += loss.item()
     29

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/tmp/ipykernel_18/3969884729.py in forward(self, y_pred, y_true)
      6
      7     def forward(self, y_pred, y_true):
----> 8         assert y_pred.size() == y_true.size()
      9         y_pred = y_pred[:, 0].contiguous().view(-1)
     10         y_true = y_true[:, 0].contiguous().view(-1)

AssertionError:

The below is the U-Net model. Please have a look.
unet_network.py:
#Unet
#https://github.com/mateuszbuda/brain-segmentation-pytorch

from collections import OrderedDict

import torch
import torch.nn as nn

class UNet(nn.Module):

    def __init__(self, in_channels=3, out_channels=1, init_features=8):
        super(UNet, self).__init__()

        features = init_features
        self.encoder1 = UNet._block(in_channels, features, name="enc1")
        self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder2 = UNet._block(features, features * 2, name="enc2")
        self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder3 = UNet._block(features * 2, features * 4, name="enc3")
        self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2)
        self.encoder4 = UNet._block(features * 4, features * 8, name="enc4")
        self.pool4 = nn.MaxPool2d(kernel_size=2, stride=2)

        self.bottleneck = UNet._block(features * 8, features * 16, name="bottleneck")

        self.upconv4 = nn.ConvTranspose2d(
            features * 16, features * 8, kernel_size=2, stride=2
        )
        self.decoder4 = UNet._block((features * 8) * 2, features * 8, name="dec4")
        self.upconv3 = nn.ConvTranspose2d(
            features * 8, features * 4, kernel_size=2, stride=2
        )
        self.decoder3 = UNet._block((features * 4) * 2, features * 4, name="dec3")
        self.upconv2 = nn.ConvTranspose2d(
            features * 4, features * 2, kernel_size=2, stride=2
        )
        self.decoder2 = UNet._block((features * 2) * 2, features * 2, name="dec2")
        self.upconv1 = nn.ConvTranspose2d(
            features * 2, features, kernel_size=2, stride=2
        )
        self.decoder1 = UNet._block(features * 2, features, name="dec1")

        self.conv = nn.Conv2d(
            in_channels=features, out_channels=out_channels, kernel_size=1
        )

    def forward(self, x):
        enc1 = self.encoder1(x)
        enc2 = self.encoder2(self.pool1(enc1))
        enc3 = self.encoder3(self.pool2(enc2))
        enc4 = self.encoder4(self.pool3(enc3))

        bottleneck = self.bottleneck(self.pool4(enc4))

        dec4 = self.upconv4(bottleneck)
        dec4 = torch.cat((dec4, enc4), dim=1)
        dec4 = self.decoder4(dec4)
        dec3 = self.upconv3(dec4)
        dec3 = torch.cat((dec3, enc3), dim=1)
        dec3 = self.decoder3(dec3)
        dec2 = self.upconv2(dec3)
        dec2 = torch.cat((dec2, enc2), dim=1)
        dec2 = self.decoder2(dec2)
        dec1 = self.upconv1(dec2)
        dec1 = torch.cat((dec1, enc1), dim=1)
        dec1 = self.decoder1(dec1)
        return torch.sigmoid(self.conv(dec1))

    @staticmethod
    def _block(in_channels, features, name):
        return nn.Sequential(
            OrderedDict(
                [
                    (
                        name + "conv1",
                        nn.Conv2d(
                            in_channels=in_channels,
                            out_channels=features,
                            kernel_size=3,
                            padding=1,
                            bias=False,
                        ),
                    ),
                    (name + "norm1", nn.BatchNorm2d(num_features=features)),
                    (name + "relu1", nn.ReLU(inplace=True)),
                    (
                        name + "conv2",
                        nn.Conv2d(
                            in_channels=features,
                            out_channels=features,
                            kernel_size=3,
                            padding=1,
                            bias=False,
                        ),
                    ),
                    (name + "norm2", nn.BatchNorm2d(num_features=features)),
                    (name + "relu2", nn.ReLU(inplace=True)),
                ]
            )
        )

Thanks & Best Regards
Schroter Michael

Output:
Your error stems from the difference in number of channels between the prediction (pred=torch.Size([5, 1, 512, 512])) and the target (y_true=torch.Size([5, 3, 512, 512])).
For a target with 3 channels, you need your pred to have three channels as well. That is, you need to configure your UNet to have out_channels=3 instead of the default of 1.


Instruction: Fine tuning resnet18 for cifar10

Input:
I just want fine tuning ResNet18 on cifar10 datasets. so I just want to change the last linear layer from 1000 to 10.
I tried use children function to get the previous layers
ResModel = resnet18(weights=ResNet18_Weights)
model = nn.Sequential(
    *list(ResModel.children())[:-1],
    nn.Linear(512,10)
)

so it raised error
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32768x1 and 512x10)
and then I tried this way ResModel.fc=nn.Linear(512,10) it works fine.
so why?

Output:
The difference between stacking all layers into a single nn.Sequential and overriding only the last layer is the forward function:
Your ResModel is of type torchvision.models.ResNet, while your model is a simple nn.Sequential. The forward pass of ResNet has an additional flatten operation before the last linear layer -- you do not have this operation in your nn.Sequential model.


Instruction: How to automate layers addition to neural network models in pytorch

Input:
I am dealing with a model in pytorch and I want to automate the layers and activations addition to the model. This code is my simple model:
import torch
from torch import nn
import torch.nn.functional as F
class NeuralNetwork(nn.Module):
    def __init__(self, n_inputs, n_hidden_unit, n_output):
        super().__init__()
        l1 = nn.Linear(n_inputs, n_hidden_unit)
        a1 = nn.Sigmoid()
        l2 = nn.Linear(n_hidden_unit, n_output)
        l = [l1, a1, l2]
        self.module_list = nn.ModuleList(l)

    def forward(self, x):
        for f in self.module_list:
            x = f(x)
        return x

model = NeuralNetwork(n_inputs=10, n_hidden_unit=30, n_output=2)
model

As you see two layers and one activation is added manually but i want to for example have two lists or numpy arrays of them and then call the lists into my model. Lists will look like the following:
connections = [(10, 30), (30, 2)]
activation = [nn.Sigmoid()]

A similar thing I did using Sequential model:
layers = []
layers.append(nn.Linear(10, 30))
layers.append(nn.Sigmoid())
layers.append(nn.Linear(30, 2))

model = nn.Sequential(*layers)
model

Output:
You can just use a loop:
def __init__(self, connections, activation):
    super().__init__()
    l = []
    for layer_idx, (n_input, n_output) in enumerate(connections):
        l.append(nn.Linear(n_input, n_output))
        if layer_idx < len(activation):
            l.append(activation[layer_idx])
    self.module_list = nn.ModuleList(l)


Instruction: Efficiently check whether each column in a PyTorch tensor has a corresponding reversed column

Input:
I have a collection of tensors of common shape (2,ncol). Example:
torch.tensor([[1, 2, 3, 7, 8], [3, 3, 1, 8, 7]], dtype=torch.long)

For each tensor, I want to determine if, for each column [[a], [b]], the reversed column [[b], [a]] is also in the tensor. For example, in this case, since ncol is odd, I can immediately say that this is not the case. But in this other example
torch.tensor([[1, 2, 3, 7, 8, 4], [3, 3, 1, 8, 7, 2]], dtype=torch.long)

I would actually have to perform the check. A naive solution would be
test = torch.tensor([[1, 2, 3, 7, 8, 4], [3, 3, 1, 8, 7, 2]], dtype=torch.long)

def are_column_paired(matrix: torch_geometric.data.Data) -> bool:
    ncol = matrix.shape[1]
    if ncol % 2 != 0:
        all_paired = False
        return all_paired

    column_has_match = torch.zeros(ncol, dtype=torch.bool)
    for i in range(ncol):
        if column_has_match[i]:
            continue
        column = matrix[:, i]
        j = i + 1
        while not (column_has_match[i]) and (j <= (ncol - 1)):
            if column_has_match[j]:
                j = j + 1
                continue
            current_column = matrix[:, j]
            current_column = current_column.flip(dims=[0])
            if torch.equal(column, current_column):
                column_has_match[i], column_has_match[j] = True, True
            j = j + 1

    all_paired = torch.all(column_has_match).item()

    return all_paired

But of course this is slow and possibly not pythonic. How can I write a more efficient code?
PS note that while test here is very small, in the actual use case I expect ncol to be O(10^5).

Output:
Here is one possible simple approach. It is likely not the most efficient you can get, but is much faster than your current solution. The idea is to simply check if the sorting the columns in the original and row-flipped tensors are identical. I believe the time complexity of this approach is O(n logn), as opposed to O(n^2) in your case.
def are_columns_paired(matrix):
    flipped_matrix = matrix.flip(dims=[0])
    matrix_sorted = matrix[:,matrix[1].argsort()] # sort second row
    matrix_sorted = matrix_sorted[:, matrix_sorted[0].sort(stable=True)[1]] # sort first row, keeping positions in second row fixed when there is a tie
    flipped_matrix = flipped_matrix[:,flipped_matrix[1].argsort()]
    flipped_matrix = flipped_matrix[:, flipped_matrix[0].sort(stable=True)[1]]
    return (matrix_sorted == flipped_matrix).all()

Here, for both the original and flipped matrix, we sort the columns, first based on the first row, and when there is a tie, based on the second row.
I tested both approaches on a randomly generated tensor with ncol=2000000 and values ranging from 0 to 999999. The above code ran in about 1 second, while the approach from the question did not provide a solution even after an hour.


Instruction: UserWarning: Trying to infer the `batch_size` from an ambiguous collection

Input:
I have a pytorch lightning module like this:
class GraphLevelGNN(pl.LightningModule):

    def __init__(self,**model_kwargs):
        super().__init__()
        # Saving hyperparameters
        self.save_hyperparameters()

        self.model = GraphGNNModel(**model_kwargs)
        self.loss_module = nn.BCEWithLogitsLoss()

    def forward(self, data, mode="train"):
        x, edge_index, batch_idx = data.x, data.edge_index, data.batch
        x = self.model(x.cpu(), edge_index.cpu(), batch_idx.cpu())
        x = x.squeeze(dim=-1)

        if self.hparams.c_out == 1:
            preds = (x > 0).float().cpu()
            data.y = data.y.float().cpu()
        else:
            preds = x.argmax(dim=-1).cpu()

        loss = self.loss_module(x.cpu(), data.y.cpu())
        acc = (preds.cpu() == data.y.cpu()).sum().float() / preds.shape[0]
        f1 = f1_score(preds.cpu(),data.y.cpu())  ##change f1/precision and recall was just testing
        precision = precision_score(preds.cpu(),data.y.cpu())
        recall = recall_score(preds.cpu(),data.y.cpu())

        return loss, acc, f1,precision, recall,preds

    def configure_optimizers(self):
        optimizer = optim.SGD(self.parameters(),lr=0.1) # High lr because of small dataset and small model
        return optimizer

    def training_step(self, batch, batch_idx):
        loss, acc, _,_,_,_ = self.forward(batch, mode="train")
        self.log('train_loss', loss,on_epoch=True,logger=True)
        self.log('train_acc', acc,on_epoch=True,logger=True)
        return loss

    def validation_step(self, batch, batch_idx):
        loss, acc, _,_,_,_ = self.forward(batch, mode="val")
        self.log('val_acc', acc,on_epoch=True,logger=True)
        self.log('val_loss', loss,on_epoch=True,logger=True)

    def test_step(self, batch, batch_idx):
        loss,acc, f1,precision, recall,preds = self.forward(batch, mode="test")
        self.log('test_acc', acc,on_epoch=True,logger=True)
        self.log('test_f1', f1,on_epoch=True,logger=True)
        self.log('test_precision', precision,on_epoch=True,logger=True)
        self.log('test_recall', recall,on_epoch=True,logger=True)

When I run the code, I get a warning:
(train_fn pid=404034) /opt/conda/lib/python3.7/site-packages/pytorch_lightning/utilities/data.py:99: UserWarning: Trying to infer the `batch_size` from an ambiguous collection. The batch size we found is 127. To avoid any miscalculations, use `self.log(..., batch_size=batch_size)`.

I'm not clear, which function am I meant to add the extra self.log to?

Output:
This warning means PyTorch Lightning has trouble inferring the batch size of your training perhaps because the batch contains different element types with varying amounts of elements inside them. To make sure it uses the correct batch_size for loss and metric computation. You can specify it yourself as described on the warning message. By setting the batch_size argument on each log call, e.g.
self.log('train_acc', acc, on_epoch=True, logger=True, batch_size=batch_size)


Instruction: PyTorch Datapipes and how does overwriting the datapipe classes work?

Input:
Pytorch Datapipes are a new inplace dataset loaders for large data that can be fed into Pytorch models through streaming, for reference these are

Official Doc: https://pytorch.org/data/main/tutorial.html
A crash-course post explaining the usage https://sebastianraschka.com/blog/2022/datapipes.html

Given a myfile.csv file, initialised as csv_file variable in code, the file looks like this, :
imagefile,label
train/0/16585.png,0
train/0/56789.png,0
...

In the example code, that uses datapipes that reads a csv_file and then create a iterable dataset using torchdata.datapipes and we see something like:
from torchdata import datapipes as dp

def build_data_pipe(csv_file, transform, len=1000, batch_size=32):
    new_dp = dp.iter.FileOpener([csv_file])

    new_dp = new_dp.parse_csv(skip_lines=1)
    # returns tuples like ('train/0/16585.png', '0')

    new_dp = new_dp.shuffle(buffer_size=len)

    ...
    # More code that returns `new_dp` variable that looks like some
    # lazy-loaded unevaluated/materialized Iterable objects.

    return new_dp

If we look at each step and the return to new_dp, we see:
>>> from torchdata import datapipes as dp

# The first initialize a FileOpenerIterDataPipe type
>>> new_dp = dp.iter.FileOpener(["myfile.csv"])
>>> new_dp
FileOpenerIterDataPipe

# Then after that the API to the DataPipes allows some overwriting/subclassing
# by calling a partial function, e.g.
>>> new_dp.parse_csv
functools.partial(<function IterDataPipe.register_datapipe_as_function.<locals>.class_function at 0x213123>, <class 'torchdata.datapipes.iter.util.plain_text_reader.CSVParserIterDataPipe'>, False, FileOpenerIterDataPipe)

>>> new_dp = new_dp.parse_csv(skip_lines=1)
>>> new_dp
CSVParserIterDataPipe

It looks like the new_dp.parse_csv(skip_lines=1) is trying do a a new initialization through a MixIn between CSVParserIterDataPipe and FileOpenerIterDataPipe but I'm not exactly sure what's happening.
To fully get a working datapipe, there's a whole bunch of other new_dp = new_dp.xxx() to call. And my question are,
Q1. Can't the DataPipe be initialize in a non-sequetial way? (P/S: This didn't work as expected)
from torchdata imnport datapipes as dp

class MyDataPipe(dp.iterGenericDataPipe):
    def __init__(self, csv_file, skip_lines=1, shuffle_buffer=1000):
        super().__init__([csv_file])
        self.parse_csv(skip_lines=1)
        self.new_dp.shuffle(buffer_size=shuffle_buffer)

But given that we have to overwrite the new_dp, seems like we might have to do something like:
from torchdata imnport datapipes as dp
class MyDataPipe(dp.iterGenericDataPipe):
    def __init__(self, csv_file, skip_lines=1, shuffle_buffer=1000):
        super().__init__([csv_file])
        self = self.parse_csv(skip_lines=1)
        self = self.new_dp.shuffle(buffer_size=shuffle_buffer)

Q2.  Is self = self.xxx() an anti-pattern in Python?
Q3. How else to initialize a DataPipe if we don't do self = self.xxx() ?

Output:
It looks like you're trying to chain together a series of torch DataPipes, namely:

FileOpener / open_files
CSVParser / parse_csv
Shuffler / shuffle

The official torchdata tutorial at https://pytorch.org/data/0.4/tutorial.html does so using a function (e.g. def custom_data_pipe()), but you seem to prefer a class-based approach (e.g. class CustomDataPipe). Let's call this a DataPipeLine.
An additional complication is that you're trying to apply an inheritance-style torch.utils.data.Dataset to a composition-style torchdata.datapipes.iter.IterDataPipe. Presumably, the reason you're doing this is to create a configurable 'dataset', e.g. one that can skip N lines, has a shuffle buffer of B, etc. Now there's a few things wrong about this, but let's go with it.
Bad example (please don't use)
from torchdata.datapipes import functional_datapipe
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper

@functional_datapipe("csv_processor_and_batcher")
class MyDataPipeLine(IterDataPipe):
    def __init__(
        self,
        source_datapipe: IterDataPipe[str],
        skip_lines: int = 1,
        shuffle_buffer: int = 1000,
    ):
        super().__init__()
        self.source_datapipe: IterDataPipe[str] = source_datapipe

        self.chained_datapipe = (
            self.source_datapipe.open_files()
            .parse_csv(skip_lines=skip_lines)
            .shuffle(buffer_size=shuffle_buffer)
        )

    def __iter__(self):
        for item in self.chained_datapipe:
            yield item

And the way you would use it is:
dp = IterableWrapper(iterable=["file1.csv", "file2.csv"])
dp_custom = dp.csv_processor_and_batcher()

dataloader = torch.utils.data.DataLoader(dataset=dp_custom)
for batch in dataloader:
    print(batch)

Now to be honest, this is really not recommended (and I'm half regretting writing up this answer already) because the reason torchdata exists is to have compositional DataPipes, i.e. each DataPipe should be specialized to do one thing only rather than many things. Also, you won't be streaming data properly, as the iterator will need to run your data through all 3 functions (open_files, parse_csv, shuffle) per file, instead of doing things piecewise (in a parallelizable way), thus defeating the whole purpose of using torchdata!
What you probably want is to 1) Read up more on composition and pipe-ing:

https://realpython.com/inheritance-composition-python
https://pandas.pydata.org/pandas-docs/version/1.5/reference/api/pandas.DataFrame.pipe.html

Then 2) write something like the below. I'm using a LightningDataModule not only because it's cool, but because it's closer to the thing you actually want to subclass:
Better example
from typing import Optional

from torch.utils.data import DataLoader2
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper
import pytorch_lightning as pl

class MyDataPipeModule(pl.LightningDataModule):
    def __init__(
        self,
        csv_files: list[str],
        skip_lines: int = 1,
        shuffle_buffer: int = 1000,
    ):
        super().__init__()
        self.csv_files: list[str] = csv_files
        self.skip_lines: int = skip_lines
        self.shuffle_buffer: int = shuffle_buffer
        # Run the datapipe composition setup()
        self.setup()

    def setup(self, stage: Optional[str] = None) -> IterDataPipe:
        self.dp_chained_datapipe: IterDataPipe = (
            IterableWrapper(iterable=self.csv_files)
            .open_files()
            .parse_csv(skip_lines=self.skip_lines)
            .shuffle(buffer_size=self.shuffle_buffer)
        )
        return self.dp_chained_datapipe

    def train_dataloader(self) -> DataLoader2:
        return DataLoader2(dataset=self.dp_chained_datapipe)

Usage:
datamodule = MyDataPipeModule(csv_files=["file1.csv", "file2.csv"])
model: pl.LightningModule = MyModel()

trainer = pl.Trainer(accelerator="auto", max_epochs=3)
trainer.fit(model=model, datamodule=datamodule)

Maybe not quite the answer you expected, but I'd encourage you to experiment a bit more. The key bit is to switch your mindset from inheritance (subclassing) to composition (chaining/pipe-ing).
P.S. Gonna throw in a shameless plug on some tutorials I wrote at https://zen3geo.readthedocs.io/en/v0.4.0/walkthrough.html. It's a bit geospatial specific, but might be helpful to get a feel of the DataPipe-way of working. Good luck!


Instruction: Why couldn't I feed a 4-tuple to nn.ReplicationPad2d()?

Input:
I'm applying yolov5 on kitti raw image [C, H, W] = [3, 375, 1242]. Therefore I need to pad the image so that the H and W being dividable by 32. I'm using nn.ReplicationPad2d to do the padding: [3, 375, 1242] -> [3, 384, 1248].
In the official tutorial of nn.ReplicationPad2d it was said that we give a 4-tuple to indicate padding sizes for left, right, top and bottom.
The Problem is:
When I give a 4-tuple (0, pad1, 0, pad2), it claims that: 3D tensors expect 2 values for padding
When I give a 2-tuple (pad1, pad2), the pad can be implemented but it seems that only W was padded by pad1+pad2, while H stays unchanged. Because I 'll get a tensor of size [3, 375, 1257].
1257-1242 = 15 = 9+6, where 9 was supposed to pad H and 6 pad W.
I could not figure out what is the problem here...
thanks in advance
Here is my code:
def paddingImage(img, divider=32):
    if img.shape[1]%divider != 0 or img.shape[2]%divider != 0:
        padding1_mult = int(img.shape[1] / divider) + 1
        padding2_mult = int(img.shape[2] / divider) + 1
        pad1 = (divider * padding1_mult) - img.shape[1]
        pad2 = (divider * padding2_mult) - img.shape[2]

        # pad1 = 32 - (img.shape[1]%32)
        # pad2 = 32 - (img.shape[2]%32)
        # pad1 = 384 - 375    # 9
        # pad2 = 1248 - 1242  # 6

        #################### PROBLEM ####################
        padding = nn.ReplicationPad2d((pad1, pad2))
        #################### PROBLEM ####################

        return padding(img)
    else:
        return img

Where img was given as a torch.Tensor in the main function:
# ...
image_tensor = torch.from_numpy(image_np).type(torch.float32)
image_tensor = paddingImage(image_tensor)
image_np = image_tensor.numpy()
# ...

Output:
PyTorch expects the input to ReplicationPad2d to be batched image tensors. Therefore, we can unsqueeze to add a 'batch dimension'.
def paddingImage(img, divider=32):
    if img.shape[1]%divider != 0 or img.shape[2]%divider != 0:
        padding1_mult = int(img.shape[1] / divider) + 1
        padding2_mult = int(img.shape[2] / divider) + 1
        pad1 = (divider * padding1_mult) - img.shape[1]
        pad2 = (divider * padding2_mult) - img.shape[2]

        # pad1 = 32 - (img.shape[1]%32)
        # pad2 = 32 - (img.shape[2]%32)
        # pad1 = 384 - 375    # 9
        # pad2 = 1248 - 1242  # 6

        padding = nn.ReplicationPad2d((0, pad2, 0, pad1))
        # Add a extra batch-dimension, pad, and then remove batch-dimension
        return torch.squeeze(padding(torch.unsqueeze(img,0)),0)
    else:
        return img

Hope this helps!
EDIT
As GoodDeeds mentions, this is resolved with later versions of PyTorch. Either upgrade PyTorch, or, if that's not an option, use the code above.


Instruction: CPU inference in libtorch causes OOM with repeated calls to forward

Input:
I have some libtorch code that is doing inference on the cpu using a model trained in pytorch that is then exported to torchscript. The code below is a simplified version of a method that is being repeatedly called.
void Backend::perform(std::vector<float *> in_buffer,
                      std::vector<float *> out_buffer) {
  c10::InferenceMode guard;

  at::Tensor tensor_out;
  at::Tensor tensor_in = torch::zeros({ 1, 16, 2 });
  std::vector<torch::jit::IValue> inputs = { tensor_in };

  // calling forward on the model "decode," this is where
  // the memory leak happens
  tensor_out = m_model.get_method("decode")(inputs).toTensor();

  auto out_ptr = tensor_out.contiguous().data_ptr<float>();

  for (int i(0); i < out_buffer.size(); i++) {
    memcpy(out_buffer[i], out_ptr + i * n_vec, n_vec * sizeof(float));
  }
}

m_model is the .ts file loaded via:
m_model = torch::jit::load(path);
m_model.eval();

Every call it seems that more of the torch graph is being allocated, and it isn’t being freed causing the program to eventually OOM and crash. Commenting out the forward call causes the the memory usage to stabilize.
My understanding is that InferenceMode guard should turn off autograd memory buildup which seems to be the normal cause of these issues.
I tried mimicking this in pytorch (by repeatedly calling forward from a loop), and there’s no memory issues which seems to point to this being a libtorch issue rather than an issue with the model itself.
My system:
OS: Windows 10/11
pytorch version: 1.11.0
libtorch version: 1.11.0

Output:
This ended up being a bug in the windows implementation of libtorch. Memory leaks can happen when calling forward on a separate thread from the main thread (https://github.com/pytorch/pytorch/issues/24237), and moving the forward call to the main thread fixed the issue.
Even though the issue is marked closed the bug is still present.


Instruction: Pytorch DataLoader changes dict return values

Input:
Given a Pytorch dataset that reads a JSON file as such:
import csv

from torch.utils.data import IterableDataset
from torch.utils.data import DataLoader2, DataLoader

class MyDataset(IterableDataset):
    def __init__(self, jsonfilename):
        self.filename = jsonfilename

    def __iter__(self):
        with open(self.filename) as fin:
            reader = csv.reader(fin)
            headers = next(reader)
            for line in reader:
                yield dict(zip(headers, line))

content = """imagefile,label
train/0/16585.png,0
train/0/56789.png,0"""

with open('myfile.json', 'w') as fout:
    fout.write(content)

ds = MyDataset("myfile.json")

When I loop through the dataset, the return values are dict of each line of the json, e.g.
ds = MyDataset("myfile.json")

for i in ds:
    print(i)

[out]:
{'imagefile': 'train/0/16585.png', 'label': '0'}
{'imagefile': 'train/0/56789.png', 'label': '0'}

But when I read the Dataset into a DataLoader, it returns the values of the dict as lists instead of the values themselves, e.g.
ds = MyDataset("myfile.json")
x = DataLoader(dataset=ds)

for i in x:
    print(i)

[out]:
{'imagefile': ['train/0/16585.png'], 'label': ['0']}
{'imagefile': ['train/0/56789.png'], 'label': ['0']}

Q (part1) : Why does the DataLoader changes the value of the dict to a list?
and also
Q (part2) : How to make the DataLoader return just the values of the dict instead of the list of value when running __iter__ with the DataLoader? Is there some arguments/options to use in DataLoader to do this?

Output:
The reason is the default collate behaviour in torch.utils.data.DataLoader, which determines how data samples in a batch are merged. By default, the torch.utils.data.default_collate collate function is used, which transforms mappings as:

Mapping[K, V_i] -> Mapping[K, default_collate([V_1, V_2, …])]

and strings as:

str -> str (unchanged)

Note that if you set batch_size to 2 in your example, you get:
{'imagefile': ['train/0/16585.png', 'train/0/56789.png'], 'label': ['0', '0']}

as a consequence of these transforms.
Assuming you do not need batching, you can get your desired output by disabling it by setting batch_size=None. More information on this here: Loading Batched and Non-Batched Data.


Instruction: PyTorch: Logging during model.fit()

Input:
I'm using pytorch/fastai for training models. Since I'm working with remote machines, I am running the scripts using nohup python $1 >$2 2>&1 & with redirection to logging file like "log123.txt".
My problem is that during the model.fit() phase with scheduler, I can't see the progress in the file after each epoch like in console and the results are written to my logging file after the model.fit() finishes training. It works fine when I watch the process in console.
Is there any workaround for this?

Output:
This may be due to file buffering. You can disable it for python process like this:

Run your script with -u options. Example:

python -u my_script.py > log.txt

Use PYTHONUNBUFFERED:

PYTHONUNBUFFERED=1 python my_script.py > log.txt
or
export PYTHONUNBUFFERED=1
python my_script.py > log.txt


Instruction: Difference between conv2d and conv2dtranspose with kernel size 1

Input:
I understand that conv2d is used for downsampling and conv2dtranspose is the opposite (upsampling). However, assuming we are not using stride or padding here. Is there a difference between the two?
Downsampling means reducing the size of input dimension. for example If you have an input of (Batch Size = 5, Channel = 3, Height = 8, Width = 8), if you reduce the height and width using maxpooling (stride=2 kernel_size=2) the output becomes (Batch Size = 5, Channel = 3, Height = 4, Width = 4). That's downsampling, the opposite is upsampling (Increasing the Height and Width dimension)
for example:
classifier1 = torch.nn.Conv2d(in_channels=10, out_channels=5, kernel_size=1)
classifier2 = torch.nn.Conv2dTranspose(in_channels=10, out_channels=5, kernel_size=1)

Output:
Operation wise, no difference. ConvTranspose2d() inserts stride - 1 zeros inbetween all rows and columns, adds kernel size - padding - 1 padding zeros, then does exactly the same stuff as Conv2d(). Default arguments result in no changes.
Though, if you actually run them back to back like this on the same input, the results will vary unless you explicitly equalize the inital weights, of course.


Instruction: the result from torch.concat() is stored in cpu(memory)?

Input:
the code
c = torch.rand((2000, 64, 64)).to('cuda')
d = torch.rand((2000, 64, 64)).to('cuda')
t3 = time.time()
s1 = c+d
s2 = torch.concat((a, b), dim=2)
t4 = time.time()

s1's device is gpu, but s2's device is cpu.
So I can't understand it. What is the principle of this?

Output:
Torch will do an operation if all necessary variable for the operation are on the same device.
I suppose that a and b where on CPU thus torch.concat((a, b), dim=2) is too.
When  you did .to('cuda'), you have moved c and d to GPU, thus s1 is on GPU too.


Instruction: Python - Torchmetric SSIM depends on batch size

Input:
sry for the quick question, I just want to know wether I found a bug or if I do not understand something here. I got the following sample, where I print the ssim of the torchmetrics library of two tensors with the batchsize 8 and the single calculated values mean.
Why are they not the same?
ssim = StructuralSimilarityIndexMeasure(kernel_size=(5, 5))

A = torch.zeros([8, 1, 500, 500])
B = torch.randn([8, 1, 500, 500])

print(ssim(A, B))

ssim1 = ssim(A[0].unsqueeze(0), B[0].unsqueeze(0))
ssim2 = ssim(A[1].unsqueeze(0), B[1].unsqueeze(0))
ssim3 = ssim(A[2].unsqueeze(0), B[2].unsqueeze(0))
ssim4 = ssim(A[3].unsqueeze(0), B[3].unsqueeze(0))
ssim5 = ssim(A[4].unsqueeze(0), B[4].unsqueeze(0))
ssim6 = ssim(A[5].unsqueeze(0), B[5].unsqueeze(0))
ssim7 = ssim(A[6].unsqueeze(0), B[6].unsqueeze(0))
ssim8 = ssim(A[7].unsqueeze(0), B[7].unsqueeze(0))

print((ssim1 + ssim2 + ssim3 + ssim4 + ssim5 + ssim6 + ssim7 + ssim8) / 8)

Console output:
tensor(0.0404)
tensor(0.0340)

Python version 3.8.10
TrochMetrics version 0.9.1
PyTroch version 1.10.1+cu113
Or is this a git issue?

Output:
The difference comes from the data_range parameter. Please refer to the documentations:
data_range: Range of the image. If ``None``, it is determined from the image (max - min)

https://torchmetrics.readthedocs.io/en/stable/image/structural_similarity.html#structural-similarity-index-measure-ssim
By default it is None so the batch and individual examples of batch will have different data_range derived from the data:
if data_range is None:
    data_range = max(preds.max() - preds.min(), target.max() - target.min())

https://github.com/Lightning-AI/metrics/blob/19355a9d2c51b3b39b311694d7b8e6856a73eae6/src/torchmetrics/functional/image/ssim.py#L129
If you set specific data_range like this:
ssim = StructuralSimilarityIndexMeasure(kernel_size=(5, 5), data_range=255)

The results will be the same.


Instruction: How to handle Pytorch Dataset with transform function that returns >1 output per row of data?

Input:
Given a myfile.csv file that looks like:
imagefile,label
train/0/16585.png,0
train/0/56789.png,0

The goal is to create a Pytorch DataLoader that when looped return 2x the data points, e.g.
>>> dp = MyDataPipe(csvfile)
>>> for row in dp.train_dataloader:
...     print(row)
...
(tensor([1.23, 4.56, 7.89]), 0)
(tensor([9.87, 6.54, 3.21]), 1)
(tensor([9.99, 8.88, 7.77]), 0)
(tensor([1.11, 2.22, 9.87]), 1)

I've tried writing the dataloader if we are just expect the same no. of dataloader's row as per the input file, this works:
import torch

from torch.utils.data import DataLoader2
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper
import pytorch_lightning as pl

content = """imagefile,label
train/0/16585.png,0
train/0/56789.png,0"""

with open('myfile.csv', 'w') as fout:
    fout.write(content)

def optimus_prime(row):
    """This functions returns two data points with some arbitrary vectors.
    >>> row = {'imagefile': 'train/0/16585.png', label: 0}
    >>> optimus_prime(row)
    (tensor([1.23, 4.56, 7.89]), 0)
    """
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector1 = torch.rand(3)
    return vector1, row['label']

class MyDataPipe(pl.LightningDataModule):
    def __init__(
        self,
        csv_files: list[str],
        skip_lines: int = 0,
        tranform_func: Callable = None
    ):
        super().__init__()
        self.csv_files: list[str] = csv_files
        self.skip_lines: int = skip_lines

        # Initialize a datapipe.
        self.dp_chained_datapipe: IterDataPipe = (
            IterableWrapper(iterable=self.csv_files)
            .open_files()
            .parse_csv_as_dict(skip_lines=self.skip_lines)
        )

        if tranform_func:
            self.dp_chained_datapipe = self.dp_chained_datapipe.map(tranform_func)

    def train_dataloader(self, batch_size=1) -> DataLoader2:
        return DataLoader2(dataset=self.dp_chained_datapipe, batch_size=batch_size)

dp = MyDataPipe('myfile.csv', tranform_func=optimus_prime)

for row in dp.train_dataloader:
    print(row)

If the optimus_prime function returns 2 data points, how do I setup the Dataloader such that it can collate the 2 data points accordingly?
How to formulate the collate function or tell the Dataloader that there's 2 inputs in each .map(tranform_func) output? E.g. if I change the function to:
def optimus_prime(row):
    """This functions returns two data points with some arbitrary vectors.
    >>> row = {'imagefile': 'train/0/16585.png', label: 0}
    >>> optimus_prime(row)
    (tensor([1.23, 4.56, 7.89]), 0), (tensor([3.21, 6.54, 9.87]), 1)
    """
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector1 = torch.rand(3)
    yield vector1, row['label']
    yield vector2, not row['label']

I've also tried the following and it works but I need to run the optimus_prime function twice, but the 2nd .map(tranform_func) throws a TypeError: tuple indices must be integers or slice not str...

def optimus_prime_1(row):
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector1 = torch.rand(3)
    yield vector1, row['label']

def optimus_prime_2(row):
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector2 = torch.rand(3)
    yield vector2, not row['label']

class MyDataPipe(pl.LightningDataModule):
    def __init__(
        self,
        csv_files: list[str],
        skip_lines: int = 0,
        tranform_funcs: list[Callable] = None
    ):
        super().__init__()
        self.csv_files: list[str] = csv_files
        self.skip_lines: int = skip_lines

        # Initialize a datapipe.
        self.dp_chained_datapipe: IterDataPipe = (
            IterableWrapper(iterable=self.csv_files)
            .open_files()
            .parse_csv_as_dict(skip_lines=self.skip_lines)
        )

        if tranform_funcs:
            for tranform_func in tranform_funcs:
                self.dp_chained_datapipe = self.dp_chained_datapipe.map(tranform_func)

    def train_dataloader(self, batch_size=1) -> DataLoader2:
        return DataLoader2(dataset=self.dp_chained_datapipe, batch_size=batch_size)

dp = MyDataPipe('myfile.csv', tranform_funcs=[optimus_prime_1, optimus_prime_2])

for row in dp.train_dataloader:
    print(row)

Output:
From https://discuss.pytorch.org/t/how-to-handle-pytorch-dataset-with-transform-function-that-returns-1-output-per-row-of-data/162160, there's a reference to use .flatmap() instead of .map():

https://pytorch.org/data/main/generated/torchdata.datapipes.iter.FlatMapper.html

https://pytorch.org/data/main/generated/torchdata.datapipes.iter.Mapper.html

By changing the transformation function to return N no. of data points per row of data form the csvfile, e.g.
def optimus_prime(row):
    """This functions returns two data points with some arbitrary vectors.
    >>> row = {'imagefile': 'train/0/16585.png', label: 0}
    >>> optimus_prime(row)
    (tensor([1.23, 4.56, 7.89]), 0)
    """
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector1 = torch.rand(3)
    vector2 = torch.rand(3)
    return [(vector1, row['label']), (vector2, row['label'])]

Changing the code to use the .flatmap() as such works:

class MyDataPipe(pl.LightningDataModule):
    def __init__(
        self,
        csv_files,
        skip_lines=0
    ):
        super().__init__()
        self.csv_files: list[str] = csv_files
        self.skip_lines: int = skip_lines

        # Initialize a datapipe.
        self.dp_chained_datapipe: IterDataPipe = (
            IterableWrapper(iterable=self.csv_files)
            .open_files()
            .parse_csv_as_dict(skip_lines=self.skip_lines)
        )

        self.dp_chained_datapipe = self.dp_chained_datapipe.flatmap(optimus_prime)

    def train_dataloader(self, batch_size=1) -> DataLoader2:
        return DataLoader2(dataset=self.dp_chained_datapipe, batch_size=batch_size)

Full working example:
import torch

from torch.utils.data import DataLoader2
import pytorch_lightning as pl
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper

content = """imagefile,label
train/0/16585.png,0
train/0/56789.png,0"""

with open('myfile.csv', 'w') as fout:
    fout.write(content)

def optimus_prime(row):
    """This functions returns two data points with some arbitrary vectors.
    >>> row = {'imagefile': 'train/0/16585.png', label: 0}
    >>> optimus_prime(row)
    (tensor([1.23, 4.56, 7.89]), 0)
    """
    # We are using torch.rand here but there is an actual function
    # that converts the png file into a vector.
    vector1 = torch.rand(3)
    vector2 = torch.rand(3)
    return [(vector1, row['label']), (vector2, row['label'])]

class MyDataPipe(pl.LightningDataModule):
    def __init__(
        self,
        csv_files,
        skip_lines=0
    ):
        super().__init__()
        self.csv_files: list[str] = csv_files
        self.skip_lines: int = skip_lines

        # Initialize a datapipe.
        self.dp_chained_datapipe: IterDataPipe = (
            IterableWrapper(iterable=self.csv_files)
            .open_files()
            .parse_csv_as_dict(skip_lines=self.skip_lines)
        )

        self.dp_chained_datapipe = self.dp_chained_datapipe.flatmap(optimus_prime)

    def train_dataloader(self, batch_size=1) -> DataLoader2:
        return DataLoader2(dataset=self.dp_chained_datapipe, batch_size=batch_size)

dp = MyDataPipe(['myfile.csv'])

for row in dp.train_dataloader():
    print(row)

[out]:
[tensor([[0.6003, 0.1200, 0.5175]]), ('0',)]
[tensor([[0.0628, 0.7004, 0.3169]]), ('0',)]
[tensor([[0.0623, 0.4608, 0.7456]]), ('0',)]
[tensor([[0.7454, 0.5326, 0.7459]]), ('0',)]


Instruction: Building Pytorch form source fails using the provided Dockerfile

Input:
I'm trying to build a docker image that I can use as a development environment for modifying Pytorch. There is a Dockerfile provided in the repo, and I'm trying the following:

git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
DOCKER_BUILDKIT=1 docker build -t pytorchtest .

But the docker build results in the following error:
...
#20 28.80 Performing C++ SOURCE FILE Test HAS_WERROR_CAST_FUNCTION_TYPE failed with the following output:
#20 28.80 Change Dir: /opt/pytorch/build/CMakeFiles/CMakeTmp
#20 28.80
#20 28.80 Run Build Command(s):/usr/bin/make -f Makefile cmTC_09005/fast && /usr/bin/make  -f CMakeFiles/cmTC_09005.dir/build.make CMakeFiles/cmTC_09005.dir/build
#20 28.80 make[1]: Entering directory '/opt/pytorch/build/CMakeFiles/CMakeTmp'
#20 28.80 Building CXX object CMakeFiles/cmTC_09005.dir/src.cxx.o
#20 28.80 /usr/bin/c++ -DHAS_WERROR_CAST_FUNCTION_TYPE  -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format  -fPIE   -Werror=cast-function-type -o CMakeFiles/cmTC_09005.dir/src.cxx.o -c /opt/pytorch/build/CMakeFiles/CMakeTmp/src.cxx
#20 28.80 cc1plus: error: -Werror=cast-function-type: no option -Wcast-function-type
#20 28.80 CMakeFiles/cmTC_09005.dir/build.make:77: recipe for target 'CMakeFiles/cmTC_09005.dir/src.cxx.o' failed
#20 28.80 make[1]: *** [CMakeFiles/cmTC_09005.dir/src.cxx.o] Error 1
#20 28.80 make[1]: Leaving directory '/opt/pytorch/build/CMakeFiles/CMakeTmp'
#20 28.80 Makefile:127: recipe for target 'cmTC_09005/fast' failed
#20 28.80 make: *** [cmTC_09005/fast] Error 2
#20 28.80
#20 28.80
#20 28.80 Source file was:
#20 28.80 int main() { return 0; }
#20 DONE 29.0s
------
executor failed running [/bin/sh -c TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1 7.0+PTX 8.0" TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"     python setup.py install]: exit code: 1

I cannot get the error logs because they exist in the temporary filesystem for the image building process.
I find it somewhat strange that a building a stable release image is failing. Am I doing something wrong?

The Dockerfile:
# syntax = docker/dockerfile:experimental
#
# NOTE: To build this you will need a docker version > 18.06 with
#       experimental enabled and DOCKER_BUILDKIT=1
#
#       If you do not use buildkit you are not going to have a good time
#
#       For reference:
#           https://docs.docker.com/develop/develop-images/build_enhancements/
ARG BASE_IMAGE=ubuntu:18.04
ARG PYTHON_VERSION=3.8

FROM ${BASE_IMAGE} as dev-base
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        ca-certificates \
        ccache \
        # cmake=3.10.2-1ubuntu2.18.04.2 \
        cmake \
        curl \
        git \
        libjpeg-dev \
        libpng-dev && \
    rm -rf /var/lib/apt/lists/*
RUN /usr/sbin/update-ccache-symlinks
RUN mkdir /opt/ccache && ccache --set-config=cache_dir=/opt/ccache
ENV PATH /opt/conda/bin:$PATH

FROM dev-base as conda
ARG PYTHON_VERSION=3.8
# Automatically set by buildx
ARG TARGETPLATFORM
# translating Docker's TARGETPLATFORM into miniconda arches
RUN case ${TARGETPLATFORM} in \
         "linux/arm64")  MINICONDA_ARCH=aarch64  ;; \
         *)              MINICONDA_ARCH=x86_64   ;; \
    esac && \
    curl -fsSL -v -o ~/miniconda.sh -O  "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh"
COPY requirements.txt .
RUN chmod +x ~/miniconda.sh && \
    ~/miniconda.sh -b -p /opt/conda && \
    rm ~/miniconda.sh && \
    /opt/conda/bin/conda install -y python=${PYTHON_VERSION} cmake conda-build pyyaml numpy ipython && \
    /opt/conda/bin/python -mpip install -r requirements.txt && \
    /opt/conda/bin/conda clean -ya

FROM dev-base as submodule-update
WORKDIR /opt/pytorch
COPY . .
RUN git submodule update --init --recursive --jobs 0

FROM conda as build
WORKDIR /opt/pytorch
COPY --from=conda /opt/conda /opt/conda
COPY --from=submodule-update /opt/pytorch /opt/pytorch
RUN --mount=type=cache,target=/opt/ccache \
    TORCH_CUDA_ARCH_LIST="3.5 5.2 6.0 6.1 7.0+PTX 8.0" TORCH_NVCC_FLAGS="-Xfatbin -compress-all" \
    CMAKE_PREFIX_PATH="$(dirname $(which conda))/../" \
    python setup.py install || cat /opt/pytorch/build/CMakeFiles/CMakeError.log

Output:
The issue was with the COPY --from=submodule-update /opt/pytorch /opt/pytorch instruction. Some .bzl files were not getting copied. More precisely they were not getting added to the Docker build context because of a .dockerignore file. I've added the following line to the end of the .dockerignore and now it works:
!*.bzl

As far as I understand, this is a bug. These files are committed to the repo, so they should get copied.


Instruction: In PyTorch, how do I update a neural network via the average gradient from a list of losses?

Input:
I have a toy reinforcement learning project based on the REINFORCE algorithm (here's PyTorch's implementation) that I would like to add batch updates to. In RL, the "target" can only be created after a "prediction" has been made, so standard batching techniques do not apply. As such, I accrue losses for each episode and append them to a list l_losses where each item is a zero-dimensional tensor. I hold off on calling .backward() or optimizer.step() until a certain number of episodes have passed in order to create a sort of pseudo batch.
Given this list of losses, how do I have PyTorch update the network based on their average gradient? Or would updating based on the average gradient be the same as updating on the average loss (I seem to have read otherwise elsewhere)?
My current method is to create a new tensor t_loss from torch.stack(l_losses), and then run t_loss = t_loss.mean(), t_loss.backward(), optimizer.step(), and zero the gradient, but I'm unsure if this is equivalent to my intents? It's also unclear to me if I should have been running .backward() on each individual loss instead of concatenating them in a list (but holding on the .step() part until the end?

Output:
Gradient is a linear operation so gradient of the average is the same as the average of the gradient.
Take some example data
import torch
a = torch.randn(1, 4, requires_grad=True);
b = torch.randn(5, 4);

You could store all the losses and compute the mean as you are doing,
a.grad = None
x = (a * b).mean(axis=1)
x.mean().backward() # gradient of the mean
print(a.grad)

Or every iteration to compute the back propagation to get the contribution of that loss to the gradient.
a.grad = None
for bi in b:
    (a * bi / len(b)).mean().backward()
print(a.grad)

Performance
I don't know the internal details of the pytorch backward implementation, but I can tell that
(1) the graph is destroyed by default after the backward pass ratain_graph=True or create_graph=True to backward().
(2) The gradient is not kept except for leaf tensors, unless you specify retain_grad;
(3) if you evaluate a model twice using different inputs, you can perform the backward pass to individual variables, this means that they have separate graphs. This can be verified with the following code.
a.grad = None
# compute all the variables in advance
r = [ (a * b / len(b)).mean() for bi in b ]
for ri in r:
    # This depends on the graph of r[i] but the graph or r[i-1]
    # was already destroyed, it means that r[i] graph is independent
    # of r[i-1] graph, hence they require separate memory.
    ri.backward()  # this will remove the graph of ri
print(a.grad)

So if you update the gradient after each episode it will accumulate the gradient of the leaf nodes, that's all the information you need for the next optimization step, so you can discard that loss freeing up resources for further computations. I would expect a memory usage reduction, potentially even a faster execution if the memory allocation can efficiently use the just deallocated pages for the next allocation.