Instruction: I am running into a gradient computation inplace error

Input:
I am running this code (https://github.com/ayu-22/BPPNet-Back-Projected-Pyramid-Network/blob/master/Single_Image_Dehazing.ipynb) on a custom dataset but I am running into this error.
RuntimeError: one of the variables needed for gradient computation has been modified by an in place operation: [torch. cuda.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Please refer to the code link above for clarification of where the error is occurring.
I am running this model on a custom dataset, the data loader part is pasted below.
    import torchvision.transforms as transforms
    train_transform = transforms.Compose([
    transforms.Resize((256,256)),
    #transforms.RandomResizedCrop(256),
    #transforms.RandomHorizontalFlip(),
    #transforms.ColorJitter(),
    transforms.ToTensor(),
    transforms.Normalize([0.5,0.5,0.5], [0.5,0.5,0.5])
 ])

class Flare(Dataset):
  def __init__(self, flare_dir, wf_dir,transform = None):
    self.flare_dir = flare_dir
    self.wf_dir = wf_dir
    self.transform = transform
    self.flare_img = os.listdir(flare_dir)
    self.wf_img = os.listdir(wf_dir)

  def __len__(self):
     return len(self.flare_img)
  def __getitem__(self, idx):
    f_img = Image.open(os.path.join(self.flare_dir, self.flare_img[idx])).convert("RGB")
    for i in self.wf_img:
        if (self.flare_img[idx].split('.')[0][4:] == i.split('.')[0]):
            wf_img = Image.open(os.path.join(self.wf_dir, i)).convert("RGB")
            break
    f_img = self.transform(f_img)
    wf_img = self.transform(wf_img)

   return f_img, wf_img

flare_dir = '../input/flaredataset/Flare/Flare_img'
wf_dir = '../input/flaredataset/Flare/Without_Flare_'
flare_img = os.listdir(flare_dir)
wf_img = os.listdir(wf_dir)
wf_img.sort()
flare_img.sort()
print(wf_img[0])
train_ds = Flare(flare_dir, wf_dir,train_transform)
train_loader = torch.utils.data.DataLoader(dataset=train_ds,
                                       batch_size=BATCH_SIZE,
                                       shuffle=True)

To get a better idea of the dataset class , you can compare my dataset class with the link pasted above

Output:
Your code is stuck in what is called the "Backpropagation" of your GAN Network.
What you have defined your backward graph should follow is the following:
def backward(self, unet_loss, dis_loss):
        dis_loss.backward(retain_graph = True)
        self.dis_optimizer.step()

        unet_loss.backward()
        self.unet_optimizer.step()

So in your backward graph, you are propagating the dis_loss which is the combination of the discriminator and adversarial loss first and then you are propagating the unet_loss which is the combination of UNet, SSIM and ContentLoss but the unet_loss is connected to discriminator's output loss. So the pytorch is confused and gives you this error as you are taking the optimizer step of dis_loss before even storing the backward graph for unet_loss and I would recommend you to change the code as follows:
def backward(self, unet_loss, dis_loss):
        dis_loss.backward(retain_graph = True)
        unet_loss.backward()

        self.dis_optimizer.step()
        self.unet_optimizer.step()

And this will start your training! but you can experiment with your retain_graph=True.
And great work on the BPPNet Work.


Instruction: Dimensionality problem with PyTorch Conv layers

Input:
I'm trying to train a neural network in PyTorch with some input signals. The layers are conv1d. The shape of my input is [100, 10], meaning 100 signals of a length of 10.
But when I execute the training, I have this error:
Given groups=1, weight of size [100, 10, 1], expected input[1, 1, 10] to have 10 channels, but got 1 channels instead
config = [10, 100, 100, 100, 100, 100, 100, 100]
batch_size = 1
epochs = 10
learning_rate = 0.001
kernel_size = 1

class NeuralNet(nn.Module):

        def __init__(self, config, kernel_size=1):
             super().__init__()

             self.config = config

             self.layers = nn.ModuleList([nn.Sequential(
                    nn.Conv1d(self.config[i], self.config[i + 1], kernel_size = kernel_size),
                    nn.ReLU())
                    for i in range(len(self.config)-1)])

             self.last_layer = nn.Linear(self.config[-1], 3)
             self.layers.append(nn.Flatten())
             self.layers.append(self.last_layer)

      def forward(self, x):

            for i, l in enumerate(self.layers):
                x = l(x)
            return x

def loader(train_data, batch_size):

  inps = torch.tensor(train_data[0])
  tgts = torch.tensor(train_data[1])

  inps = torch.unsqueeze(inps, 1)
  dataset = TensorDataset(inps, tgts)
  train_dataloader = DataLoader(dataset, batch_size = batch_size)

  return train_dataloader

At first, my code was without the unsqueez(inps) line and I had the exact same error, but then I added this line thinking that I must have an input of size (num_examples, num_channels, lenght_of_signal) but it didn't resolve the problem at all.
Thank you in advance for your answers

Output:
nn.Conv1d expects input with shape of form (batch_size, num_of_channels, seq_length). It's parameters allow to directly set number of ouput channels (out_channels) and change length of output using, for example, stride. For conv1d layer to work correctly it should know number of input channels (in_channels), which is not the case on first convolution: input.shape == (batch_size, 1, 10), therefore num_of_channels = 1, while convolution in self.layers[0] expects this value to be equal 10 (because in_channels set by self.config[0] and self.config[0] == 10). Hence to fix this append one more value to config:
config = [10, 100, 100, 100, 100, 100, 100, 100]  # as in snippet above
config = [1] + config

At this point convs should be working fine, but there is another obstacle in self.layers -- linear layer at the end. So if kernel_size of 1 was used, then after final convolution batch will have shape (batch_size, 100, 10), and after flatten (batch_size, 100 * 10), while last_layer expects input of shape (batch_size, 100). So, if length of sequence after final conv layer is known (which is certainly the case if you're using kernel_size of 1 with default stride of 1 and default padding of 0 -- length stays same), last_layer should be defined as:
self.last_layer = nn.Linear(final_length * self.config[-1], 3)

and in snippet above final_length can be set to 10 (since conditions in previous brackets satisfied). To catch idea of how shapes in conv1d transformed take look at simple example in gif below (here batch_size is equal to 1):


Instruction: Pad a numpym array to meet a required size

Input: I need to pass an array of a specific shape (4,5) to a function. However when this array is initially generated it may be less than the required shape e.g. (2,5) or (1,5). How would I pad this array to meet my required (4,5) shape?

Output:
For a 2D array,
np.pad(x, ((num_rows_before, num_rows_after), (num_cols_before, num_cols_after)))

Will get you the desired shape.
Example:
In [11]: x
Out[11]: array([[8, 3, 5, 1, 5]])

In [12]: np.pad(x, ((3, 0), (0, 0)))
Out[12]:
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [8, 3, 5, 1, 5]])

In [13]: np.pad(x, ((0, 3), (0, 0)))
Out[13]:
array([[8, 3, 5, 1, 5],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])

In general, you can pass n 2-tuples for an n-dimensional array, where each 2-tuple consists of before-after pairs of integers that dictate how much to pad along each axis and in each direction.


Instruction: ESRGAN Issue — "Torch not compiled with CUDA enabled"

Input:
Need help! Trying to run a python script with a pre-trained model (ESRGAN), but every time I face this error message. What could be the problem?
Before that I've installed the following dependencies:

pip install pytorch cuda
pip install opencv-python glob2

Output:
Fixed the problem by tweaking a code a bit, changing a device type from 'cuda' to 'cpu'.
device = torch.device('cpu')


Instruction: PyTorch computes gradients as None

Input:
class Parent(nn.Module):

  def __init__(self,in_features,z_dim, img_dim):
        super().__init__()
        self.my_child1 = Child1 (z_dim, img_dim)
        self.my_child2 = Child2 (in_features)

  def forward(self,input):
         input=self.my_child1(input)
         input=self.my_child2(input)
         return input

  def forward1(self,input):
         input=self.my_child1(input)
         return input

  def forward2(self,input):
         input=self.my_child2(input)
         return input

class Child2(nn.Module):
    def __init__(self, in_features):
        super().__init__()
        self.child2 = nn.Sequential(
            nn.Linear(in_features, 128),
            nn.LeakyReLU(0.01),
            nn.Linear(128, 1),
            nn.Sigmoid(),
        )

    def forward(self, x):
        return self.child2(x)

class Child1(nn.Module):
    def __init__(self, z_dim, img_dim):
        super().__init__()
        self.child1 = nn.Sequential(
            nn.Linear(z_dim, 256),
            nn.LeakyReLU(0.01),
            nn.Linear(256, img_dim),
            nn.Tanh(),
        )

    def forward(self, x):
        return self.child1(x)

criterion=nn.BCELoss()
noise = torch.randn(batch_size, z_dim).to(device)
model=Parent(in_features,z_dim, img_dim)
output1=model(noise)
loss1=criterion(output1,torch.ones_like(output1))
loss2=criterion(output1,torch.zeroes_like(output1))
loss3=(loss1+loss2)/2
model.zero_grad()
loss3.backward(retain_graph=True)
print(loss3.grad)

I have not used any optimizer here because updating the parameters are done using a seperate formula which I will employ only after I get the gradients. The formula requires the gradients to be stored in a matrix. However, the gradient always prints “None”.

Output:
You can get the computed gradient for every parameter in your model with:
gradient = [el.grad for el in model.parameters()]
print(gradient)


Instruction: Convolutional neural network throws ErrorTypeError: __init__() takes 1 positional argument but 2 were given

Input:
I am passing a batch of 256 images x_s with the following dimensions [256, 3, 560, 448]. However I whenever i try to feed my images to the CNN i get the following error:
ErrorTypeError: __init__() takes 1 positional argument but 2 were given'

Not sure what it means here by '1 positional argument'. I am passing in the images using an iterator which I created. Below is my code for the training loop up until the point where it breaks:
for e in range(num_epochs):
  print(f'Epoch{e+1:04d}/ {num_epochs:04d}', end='\n================\n')
  dl_source_iter = iter(dl_source)
  dl_target_iter = iter(dl_target)

  for batch in range(max_batches):
    optimizer.zero_grad()

    p = float(batch + e * max_batches) / (num_epochs *max_batches)
    grl_lambda = 2. / (1. + np.exp(-10 * p)) - 1

    x_s, y_s = next(dl_source_iter)

    y_s_domain = torch.zeros(256, dtype=torch.long)

    class_pred, domain_pred = Cnn(x_s, grl_lambda) #This is the line which throws an error

Here is my convolutional neural network:
class Cnn(nn.Module):
  def __init__(self):
    super(Cnn, self).__init__()

    self.feature_extract= nn.Sequential(
        nn.Conv2d(3, 64, 5, 1, 1),
        nn.BatchNorm2d(64),
        nn.MaxPool2d(2),
        nn.ReLU(True),
        nn.Conv2d(64, 50, 5, 1, 1),
        nn.BatchNorm2d(50),
        nn.MaxPool2d(2),
        nn.ReLU(True),
        nn.Dropout2d(),
    )

    self.num_cnn_features = 50*5*5
    self.class_classifier = nn.Sequential(
        nn.Linear(self.num_cnn_features, 200),
        nn.BatchNorm1d(200),
        nn.Dropout2d(),
        nn.ReLU(True),
        nn.Linear(200, 200),
        nn.BatchNorm1d(200),
        nn.ReLU(True),
        nn.Linear(200, 182),
        nn.LogSoftmax(dim = 1),

    )

    self.DomainClassifier = nn.Sequential(
        nn.Linear(self.num_cnn_features, 100),
        nn.BatchNorm1d(100),
        nn.ReLU(True),
        nn.Linear(100, 2),
        nn.LogSoftmax(dim=1)

    )

  def forward(self, x, grl_lambda=1.0):

    features = self.feature_extract(x)
    features = features.view(-1, self.num_cnn_features)
    features_grl = GradientReversalFn(features, grl_lambda)
    class_pred = self.class_classifier(features)
    domain_pred = self.DomainClassifier(features_grl)
    return class_pred, domain_pred

Does anyone have any guesses as to why this might be happening? I can't seem to figure out what is going wrong. Any help would be greatly appreciated.

Output:
You need to create a Cnn object before you can pass data to it. You are calling the Cnn class  constructor __init__, which expects no arguments, rather than the forward method for an instance of the Cnn class, which is what you actually want to do.
# outside of loop
model = Cnn()

# inside loop
class_pred, domain_pred = model(x_s, grl_lambda)


Instruction: Pytorch dot product across rows from different arrays

Input:
I am trying to code up something similar to the positional encoding in the transformers paper.
In order to do so I need to do the following:
For the following three matrices, I want to concatenate them at row level (i.e. the first row from each one stacked together, the second rows together, etc.), and then apply dot product between each matrix and its transpose, and finally, flatten them and stack them together. I'll clarify this in the following example:
x = torch.tensor([[1,1,1,1],
                  [2,2,2,2],
                  [3,3,3,3]])
y = torch.tensor([[0,0,0,0],
                  [0,0,0,0],
                  [0,0,0,0]])
z = torch.tensor([[4,4,4,4],
                  [5,5,5,5],
                  [6,6,6,6]])

concat = torch.cat([x, y, z], dim=-1).view(-1, x.shape[-1])
print(concat)

tensor([[1, 1, 1, 1],
        [0, 0, 0, 0],
        [4, 4, 4, 4],
        [2, 2, 2, 2],
        [0, 0, 0, 0],
        [5, 5, 5, 5],
        [3, 3, 3, 3],
        [0, 0, 0, 0],
        [6, 6, 6, 6]])

# Here I get each three rows together, and then apply dot product, flatten, and stack them.
concat = torch.stack([
            torch.flatten(
                torch.matmul(
                    concat[i:i+3, :], # 3 is the number of tensors (x,y,z)
                    torch.transpose(concat[i:i+3, :], 0, 1))
                )
            for i in range(0, concat.shape[0], 3)
            ])

print(concat)

tensor([[  4,   0,  16,   0,   0,   0,  16,   0,  64],
        [ 16,   0,  40,   0,   0,   0,  40,   0, 100],
        [ 36,   0,  72,   0,   0,   0,  72,   0, 144]])

Finally, I was able to get the final matrix that I want. My question is, is there a way to achieve this without using a loop as I did in the final step? I want everything to be in tensors.

Output:
The loop you introduce only needs to be there to get a "list of slices" of the data, which is practically the same as reshaping it. You are basically introducing a additional dimension, in which there are 3 entries. Basically from shape [n, k] to [n, 3, k].
For working directly with tensors, you can just call .reshape to get to the same shape. After that, the rest of the code you use also works almost the exact same. The transpose has to be changed slightly, due to the change in dimensions.
All in all, what you want can be achieved with:
concat2 = concat.reshape((-1, 3, concat.shape[1]))
torch.flatten(
  torch.matmul(
    concat2,
    concat2.transpose(1,2)
  ),
  start_dim=1,
)


Instruction: conda does not install old version of pytorch

Input:
I'm trying to install Pytorch 1.7 on a remote cluster using anaconda.
I have cloned the base environment from the anaconda module installed on remote:
module load anaconda
conda activate
conda create --clone base --name myenv

I have then done:
conda install pytorch==1.7.1 cudatoolkit=11.0 -c pytorch

as recommended in the Pytorch documentation. I  get an error message,
PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch==1.7.1

On my personal laptop I can install pytorch 1.7 without issues using conda. I have conda 4.11.0 on my laptop and 4.9.2 on remote; the version of anaconda is the same. I cannot upgrade conda to 4.11 on remote as there are inconsistencies.
I have also tried to install my own version of anaconda from https://repo.anaconda.com/archive/Anaconda3-2021.05-Linux-ppc64le.sh**, with matched version of anaconda and conda to those of my laptop. Still no success at installing pytorch 1.7 with the command above.
Do you have any suggestions?
Thanks!
**the ppc64le extension is apparently the correct one, any x86 version refuses to install, although it's unclear to me why that is so..

Output:
ppc64le is the issue here. Your laptop has (most likely) an amd64 based CPU while your remote computer has ppc64le as their CPU architecture. This means that pytorch packages compatible with your laptop will not be compatible with your remote computer. You can find out which pytorch versions are available for that CPU architecture by running conda search:
conda search --subdir='linux-ppc64le' -c pytorch pytorch
Loading channels: done
# Name                       Version           Build  Channel
pytorch                       1.10.2 cpu_py310hef0c51e_0  pkgs/main
pytorch                       1.10.2 cpu_py37h6f0ae12_0  pkgs/main
pytorch                       1.10.2 cpu_py38h6f0ae12_0  pkgs/main
pytorch                       1.10.2 cpu_py39h6f0ae12_0  pkgs/main

Note, only 1.10.2 seems to be available.
If you need a specific version of pytorch other than that, you will probably need to download and install from source.


Instruction: The parameters of the model with custom loss function doesn’t upgraded thorough its learning over epochs

Input:
Thank you for reading my post.
I’m currently developing the peak detection algorithm using CNN to determine the ideal convolution kernel which is representable as the ideal mother wavelet function that will maximize the peak detection accuracy.
To begin with, I created my own IoU loss function and the simple model and tried to run the learning. The execution itself worked without any errors, but somehow it failed.
The parameters of the model with custom loss function doesn't upgraded thorough its learning over epochs
My own loss function is described as below.
def IoU(inputs: torch.Tensor, labels: torch.Tensor,
             smooth: float=0.1, threshold: float = 0.5, alpha: float = 1.0):
  '''
  - alpha: a parameter that sharpen the thresholding.
    if alpha = 1 -> thresholded input is the same as raw input.
  '''

  thresholded_inputs = inputs**alpha / (inputs**alpha + (1 - inputs)**alpha)
  inputs = torch.where(thresholded_inputs < threshold, 0, 1)
  batch_size = inputs.shape[0]

  intersect_tensor = (inputs * labels).view(batch_size, -1)
  intersect = intersect_tensor.sum(-1)

  union_tensor = torch.max(inputs, labels).view(batch_size, -1)
  union = union_tensor.sum(-1)

  iou = (intersect + smooth) / (union + smooth)  # We smooth our devision to avoid 0/0
  iou_score = iou.mean()

  return 1- iou_score

and my training model is,
class MLP(nn.Module):
  def __init__(self):
    super().__init__()
    self.net = nn.Sequential(
        nn.Conv1d(1, 1, kernel_size=32, stride=1, padding=16),
        nn.Linear(257, 256),
        nn.LogSoftmax(1)
    )
  def forward(self, x):
    return self.net(x)

model = MLP()
opt = optim.Adadelta(model.parameters())

# initialization of the kernel of Conv1d
def init_kernel(m):
  if type(m) == nn.Conv1d:
    nn.init.kaiming_normal_(m.weight)
    print(m.weight)
    plt.plot(m.weight[0][0].detach().numpy())

model.apply(init_kernel)

def step(x, y, is_train=True):
  opt.zero_grad()

  y_pred = model(x)
  y_pred = y_pred.reshape(-1, 256)

  loss = IoU(y_pred, y)
  loss.requires_grad = True
  loss.retain_grad = True

  if is_train:
    loss.backward()
    opt.step()

  return loss, y_pred

and lastly, the execution code is,
from torch.autograd.grad_mode import F

train_loss_arr, val_loss_arr = [], []
valbose = 10
epochs = 200

for e in range(epochs):
  train_loss, val_loss, acc = 0., 0., 0.,
  for x, y in train_set.as_numpy_iterator():
    x = torch.from_numpy(x)
    y = torch.from_numpy(y)
    model.train()
    loss, y_pred = step(x, y, is_train=True)
    train_loss += loss.item()
  train_loss /= len(train_set)

  for x, y ,in val_set.as_numpy_iterator():
    x = torch.from_numpy(x)
    y = torch.from_numpy(y)
    model.eval()
    with torch.no_grad():
      loss, y_pred = step(x, y, is_train=False)
    val_loss += loss.item()
  val_loss /= len(val_set)

  train_loss_arr.append(train_loss)
  val_loss_arr.append(val_loss)

  # visualize current kernel to check whether the learning is on progress safely.
  if e % valbose == 0:
    print(f"Epoch[{e}]({(e*100/epochs):0.2f}%):  train_loss: {train_loss:0.4f}, val_loss: {val_loss:0.4f}")
    fig, axs = plt.subplots(1, 4, figsize=(12, 4))
    print(y_pred[0], y_pred[0].shape)
    axs[0].plot(x[0][0])
    axs[0].set_title("spectra")
    axs[1].plot(y_pred[0])
    axs[1].set_title("y pred")
    axs[2].plot(y[0])
    axs[2].set_title("y true")
    axs[3].plot(model.state_dict()["net.0.weight"][0][0].numpy())
    axs[3].set_title("kernel1")
    plt.show()

with these programs, I tried to evaluate this simple model, however, model parameters didn't change at all over epochs.
Visualization of the results at epoch 0 and 30.
epoch 0:
prediction and kernel at epoch0
epoch 30:
prediction and kernel at epoch30
As you can see, the kernel has not be modified through its learning over epochs.
I took a survey to figure out what causes this problem for hours but I'm still not sure how to fix my loss function and model into trainable ones.
Thank you.

Output:
Try printing the gradient after loss.backward() with:
y_pred.grad()

I suspect what you'll find is that after a backward pass, the gradient of y_pred is zero. This means that either a.) gradient is not enabled for one or more of the variables at which the computation graph has a node, or b.) (more likely) you are using an operation which is not differentiable.
In your case, at a minimum torch.where is non-differentiable, so you'll need to replace that. Thersholding operations are non-differentiable and are generally replaced with "soft" thresholding operations (see Softmax instead of max function for classification) so that gradient computation still works. Try replacing this with a soft threshold or no threshold at all.


Instruction: .float() in pytorch changes the value of an int

Input:
import torch
torch.set_printoptions(precision=1, sci_mode=False)

numeric_seq_id = 2021080918959999952

t = torch.tensor(numeric_seq_id)
tt = torch.tensor(numeric_seq_id).float() # !!!

print(t, tt)

output is
tensor(2021080918959999952) tensor(2021080905052848128.)

We could see that tt's value is changed after .float() transform.
Why is here such a difference on the value?

ps. pytorch's version = 1.10.1
python's version = 3.8

Output:
This is not pytorch specific, but an artifact of how floats (or doubles) are represented in memory (see this question for more details), which we can also see in numpy:
import numpy as np

np_int = np.int64(2021080918959999952)
np_float = np.float32(2021080918959999952)
np_double = np.float64(2021080918959999952)

print(np_int, int(np_float), int(np_double))

Output:
2021080918959999952 2021080905052848128 2021080918960000000


Instruction: Splitting pytorch dataloader into numpy arrays

Input:
In principle I'd like to do the opposite of what was done here https://datascience.stackexchange.com/questions/45916/loading-own-train-data-and-labels-in-dataloader-using-pytorch.
I have a Pytorch dataloader train_dataloader with shape (2000,3). I want to store the 3 dataloader columns in 3 separate numpy arrays. (The first column of the dataloader contains the data, the second column contains the labels.)
I managed to do it for the last batch of the train_dataloader (see below), but unfortunately couldn't make it work for the whole train_dataloader.
for X, y, ind in train_dataloader:
    pass

train_X = np.asarray(X, dtype=np.float32)
train_y = np.asarray(y, dtype=np.float32)

Any help would be very much appreciated!

Output:
You can collect all the data:
all_X = []
all_y = []
for X, y, ind in train_dataloader:
  all_X.append(X)
  all_y.append(y)
train_X = torch.cat(all_X, dim=0).numpy()
train_y = torch.cat(all_y, dim=0).numpy()


Instruction: Pytorch DataLoader shuffle=False?

Input:
I used Pytorch DataLoader to create My "batch-data" loder,but I got some problem.
As the definition of the pytorch DataLoader Shuffer.
shuffle (bool, optional) – set to True to have the data reshuffled at every epoch (default: False)

the data will be reshuffled after every epoch.
But,though I set shuffle to False,I will probably also get the completely different batch every iteration in the same epoch which I expect .
testData = torchvision.datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

CurrentFoldTestDataLoader = data.DataLoader(testData, batch_size=32, shuffle=False)
for i in range(1000):
    test_features, test_labels = next(iter(CurrentFoldTestDataLoader))
    print(i,test_labels)

Here I got the same batch in every iteration.
0 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
1 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
2 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
3 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
4 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
5 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
6 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
7 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
8 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
9 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])
10 tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
        1, 4, 6, 0, 9, 3, 8, 8])

Why is this? Is my understanding of the definition of shuffle inaccurate?

Output:
The problem with your code is that you are re-instantiating the same iterator for each step in the for cycle. With shuffle=False the iterator generates the same first batch of images. Try to instantiate the loader outside the cycle instead:
loader = data.DataLoader(testData, batch_size=32, shuffle=False)

for i, data in enumerate(loader):
    test_features, test_labels = data
    print(i, test_labels)


Instruction: Partially stacking Tensors

Input: I have some output from a convolutional layer in PyTorch of the shape [b,c,h] where b is my batches, c is my channels, and h is the features. I want to stack these to feed into a fully-connected layer without changing the batches, so that they are in the shape [b, c*h]. How can I do this?

Output:
Seems like a simple reshape or view should work:
input.shape # [b,c,h]

reshaped = input.view(input.shape[0],-1) # [b,c*h]


Instruction: Cross layer in pytorch

Input:
How can I add a cross layer in pytorch (https://arxiv.org/abs/1708.05123). I would like to add such a cross layer into my neural network that comes before the deep layers as a way to better combine my features that are coming from a range of sources.
Thanks!

Output:
There is a PyTorch implementation of the DCN architecture from paper you linked in the DeepCTR-Torch library. You can see the implementation here: https://github.com/shenweichen/DeepCTR-Torch/blob/master/deepctr_torch/models/dcn.py
If you pip install deepctr-torch then you can just use the CrossNet layer as a torch.Module:
from deepctr_torch.layers.interaction import CrossNet

crossnet = CrossNet(in_features=512)


Instruction: Pytorch's autograd issue with joblib

Input:
There seems to be a problem mixing pytorch's autograd with joblib. I need to get gradient in parallel for a lot of samples. Joblib works fine with other aspects of pytorch, however, when mixing with autograd it gives errors. I made a very small example which shows serial version works fine but the parallel version crashes.
from joblib import Parallel, delayed
import numpy as np
torch.autograd.set_detect_anomaly(True)
tt = lambda x, grad=True: torch.tensor(x, requires_grad=grad)

def Grad(X, Out):
    return autograd.grad(Out, X, create_graph=True, allow_unused=False)[0]

xs, ys = [], []
for i in range(10):
    xi = tt(np.random.rand()).float()
    yi = xi * xi
    xs += [xi]
    ys += [yi]

Grads_serial = [Grad(x, y) for x, y in zip(xs, ys)]
print("Grads_serial", Grads_serial)
Grads_parallel = Parallel(n_jobs=2)([delayed(Grad)(x, y) for x, y in zip(xs, ys)])
print("Grads_parallel", Grads_parallel)

The error message is not very helpful as well:
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

Output:
Joblib is not copying the graph associated with the operations to the different process. One way to work around it is to perform the computation inside the job.
import torch
from torch import autograd
from joblib import Parallel, delayed
import numpy as np
torch.autograd.set_detect_anomaly(False)
tt = lambda x, grad=True: torch.tensor(x, requires_grad=grad)

def Grad(X, Out):
    # This will compute yi in the job, and thus will
    # create the graph here
    yi = Out[0](*Out[1])
    # now the differentiation works
    return autograd.grad(yi, X, create_graph=True, allow_unused=False)[0]

torch.set_num_threads(1)
xs, ys = [], []
for i in range(10):
    xi = tt(np.random.rand()).float()
    yi = lambda xi: xi * xi, [xi]
    xs += [xi]
    ys += [yi]

Grads_serial = [Grad(x, y) for x, y in zip(xs, ys)]
print("Grads_serial", Grads_serial)
Grads_parallel = Parallel(n_jobs=2)([delayed(Grad)(x, y) for x, y in zip(xs, ys)])
print("Grads_parallel", Grads_parallel)

Edit
More philosophical questions are
(1) does it make sense to use joblib parallelism, if you can simply vectorize your operations and let torch to use intraoperator parallelism?
(2) mak14 mentioned using threading backend, it is good that it fixes your example. But multiple threads will use only one CPU, it makes sense for IO bounded jobs, like making HTTP requests, but not for CPU bounded operations.
Edit #2
The existence of torch.multiprocessing suggests that gradients require some special treatment, you could attempt to write a backend to joblib using torch.multiprocessing instead of multiprocessing or threading.
Here you find an overview to how graphs are constructed in both frameworks
https://www.tensorflow.org/guide/intro_to_graphs
https://pytorch.org/blog/computational-graphs-constructed-in-pytorch/
But I fear that to give a definite answer as to why one works and not the other will have to look into the implementation.


Instruction: How to change an image which has dimensions (512, 512, 3) to a tensor of size ([1, 3, 224, 224])

Input:
I was trying to implement a paper where the input dimensions are meant to be a tensor of size ([1, 3, 224, 224]). My current image size is (512, 512, 3).
How do I resize and convert in order to input to the model?

Output:
Assuming image already converted to torch.Tensor and has shape (512, 512, 3), one of possible ways:
from torchvision.transforms import Resize

image = image.permute((2, 0, 1))  # convert to (C, H, W) format

image = image.unsqueeze(0)  # add fake batch dimension

resize = Resize((224, 224))

new_image = resize(image)

Now new_image.shape equals to (1, 3, 224, 224)


Instruction: Pytorch tensor - randomly replace values that meet condition

Input:
I have a Pytorch tensor mask of dimensions,
torch.Size([8, 24, 24])

with unique values,
> torch.unique(mask, return_counts=True)
(tensor([0, 1, 2]), tensor([2093, 1054, 1461]))

I wish to randomly replace the number of 2s to 0s, such that the unique values and counts in the tensor become,
> torch.unique(mask, return_counts=True)
(tensor([0, 1, 2]), tensor([2500, 1054, 1054]))

I have tried using torch.where to no success. How can this be achieved?

Output:
One of the possible solutions is through flattening via view and numpy.random.choice:
from numpy.random import choice

idx = torch.where(mask.view(-1) == 2)[0]  # get all indicies of 2 in flat tensor

num_to_change = 2500 - 2093 # as follows from example abow

idx_to_change = choice(idx, size=num_to_change, replace=False)

mask.view(-1)[idx_to_change] = 0


Instruction: Customizing optimizer in pytorch lightning

Input:
Here I have implemented a custom optimizer in normal pytorch. I am trying to do the same thing in pytorch lightning but don't know how to.
def run_epoch(data_iter, model, loss_compute, model_opt):
    "Standard Training and Logging Function"
    start = time.time()
    total_tokens = 0
    total_loss = 0
    tokens = 0
    sofar = 0
    for i, batch in enumerate(data_iter):
        sofar = sofar + len(batch.src)

        output = model.forward(batch.src, batch.trg,
                            batch.src_mask, batch.trg_mask)

        loss = loss_compute(output, batch.trg_y, batch.ntokens)
        loss.backward()
        if model_opt is not None:
            model_opt.step()
            model_opt.optimizer.zero_grad()

        total_loss += loss
        total_tokens += batch.ntokens
        tokens += batch.ntokens
        tokens = 0
    return total_loss / total_tokens

class CustomOptimizer:

    def __init__(self, model_size, factor, warmup, optimizer):
        self.optimizer = optimizer
        self._step = 0
        self.warmup = warmup
        self.factor = factor
        self.model_size = model_size
        self._rate = 0

    def step(self):
        self._step += 1
        rate = self.rate()
        for p in self.optimizer.param_groups:
            p['lr'] = rate
        self._rate = rate
        self.optimizer.step()

    def rate(self, step=None):
        "Implement `lrate` above"
        if step is None:
            step = self._step
        return self.factor * (self.model_size ** (-0.5) * min(step ** (-0.5), step *
               self.warmup ** (-1.5)))

if __name__ == "__main__":
    model = create_model(V, V, N=2)

    customOptimizer = CustomOptimizer(model.src_embed[0].d_model,
    1, 400,
    torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98),
    eps=1e-9))

    for epoch in range(10):
        model.train()
        run_epoch(data, model,
                  LossCompute(model.generator, LabelSmoothing),
                  customOptimizer)

I tried my best to follow the pytorch lightning official documentation and the code below is my attempt. The code does run smoothly without error. But the losses in each epoch goes down very slowly. So I use the debugger in pycharm and find out that the learning rate of customOptimizer at line customOptimizer.step() always stays as the same value "5.52471728019903e-06". Whereas in the implmentation in normal pytorch shown above does successfully change the learning rate as the training goes on.
class Model(pl.LightningModule)
    def __init__(self, ....)
        self.automatic_optimization = False
        :
        :
        :
   :
   :
   :
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9)

    def training_step(self, batch, batch_idx):
        optimizer = self.optimizers()
        customOptimizer =
        CustomOptimizer(self.src_embed[0].d_model, 1, 400,
                          optimizer.optimizer)
        batch = Batch(batch[0], batch[1])
        out = self(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
        out = self.generator(out)
        labelSmoothing = LabelSmoothing(size=tgt_vocab, padding_idx=1, smoothing=0.1)
        loss = labelSmoothing(out.contiguous().view(-1, out.size(-1)),
               batch.trg_y.contiguous().view(-1)) / batch.ntokens
        loss.backward()
        customOptimizer.step()
        customOptimizer.optimizer.zero_grad()
        log = {'train_loss': loss}
        return {'loss': loss, 'log': log}

if __name__ == '__main__':
    if True:
        model = model(......)
        trainer = pl.Trainer(max_epochs=5)
        trainer.fit(model, train_dataloaders=trainLoader)

Output:
Thank you it works.
class Model(pl.LightningModule)
    def __init__(self, ....)
        self.automatic_optimization = False
        self.customOptimizer = None
        :
        :
        :
   :
   :
   :
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9)

    def training_step(self, batch, batch_idx):
        if self.customOptimizer = None:
           optimizer = self.optimizers()
           self.customOptimizer =
           CustomOptimizer(self.src_embed[0].d_model, 1, 400,
                          optimizer.optimizer)
        batch = Batch(batch[0], batch[1])
        out = self(batch.src, batch.trg, batch.src_mask, batch.trg_mask)
        out = self.generator(out)
        labelSmoothing = LabelSmoothing(size=tgt_vocab, padding_idx=1, smoothing=0.1)
        loss = labelSmoothing(out.contiguous().view(-1, out.size(-1)),
               batch.trg_y.contiguous().view(-1)) / batch.ntokens
        loss.backward()
        self.customOptimizer.step()
        self.customOptimizer.optimizer.zero_grad()
        log = {'train_loss': loss}
        return {'loss': loss, 'log': log}

if __name__ == '__main__':
    if True:
        model = model(......)
        trainer = pl.Trainer(max_epochs=5)
        trainer.fit(model, train_dataloaders=trainLoader)```


Instruction: network quantization——Why do we need "zero_point"? Why symmetric quantization doesn't need "zero point"?

Input:
I have Googled all the days, but can't still find the answer I need. There must be some misunderstandings in my brain. Could you please help me out?
1. Why do we need "zero_point"?

quantization：q=round(r/scale)+zero_point

I think that the zero_point (as an offset) shifts the scaled data to a proper position, for examle, in the figure below for unsigned 2 bits quantization, the zero point shifts [-1,2] to {0,1,2,3}

Am I right about this?
If I am wrong, please help correct me;
If I am right, then zero point is neceesary here (symmetric quantization), and why Jacob in IAO, Section 2.1 said zero-point is for zero-padding? It seems to me that this is just an outcome, not the root reason ?

2. Why doesn't  symmetric quantization need "zero point"?
In Goolge White paper and some blogs, it it said that symmetric quantization dose not need zero point (since zero_point=0):

I can understand it in signed quantization, since both the floating range and the quantized range are symmetric, making zero_point=0.
However, how can we ignore zero_point in unsigned quantization where the quantized range [0,2^b-1] is not symmetric?  Under this situation, it seems to me that we must have a positive zero point to shift the scaled data to the range [0, 2^b-1] as the figure below:

Output:
I haven't found or got definitive answe yet.
But I convince myself as follow:
1.Why do we need "zero_point"?
The zero_point is definitely an offset or bias, shifting the scaled data to a proper position. There should be no doubts about it.
But what "the motivation" that Jocab methioned is that "Z is of the same tye as quantized q", instead of "having a zero point". The former makes sure the real "0" is quantized without error, thus when inferring in the quantized manner, it is the zero point (the same type as q) that is padded (instead of the value "0") in zero_padding (wihout error).

Why doesn't symmetric quantization need "zero point"?

I think the "if signed" and "if un-signed" in the formulas (7)-(9) of white paper are talking about the signs of x, i.e., the real, unquantized, floating-point values, instead of the quantized one. This means, signed floating-point values are quantized to signed fixed-point integer, with zero-point=0; and unsigned floating-point to unsigned fixed-point interger, with zero-point=0 as well.


Instruction: Electra sequence classification with pytorch lightning issues with 'pooler_output'

Input:
I'm working on a sentence classification task with multiple binary labels attached to each sentence. I'm using Electra and pytorch lightning to do the job, but I've run into a problem. When I'm running the trainer.fit(model, data) I get the following error:
AttributeError: 'tuple' object has no attribute 'pooler_output'
The error is referring to line 13 in the section where I'm defining pl.LightningModule:
class CrowdCodedTagger(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.electra = ElectraModel.from_pretrained(ELECTRA_MODEL_NAME, return_dict=False) #changed ElectraModel to ElectraForSequenceClassification
    self.classifier = nn.Linear(self.electra.config.hidden_size, n_classes)
    self.n_training_steps = n_training_steps
    self.n_warmup_steps = n_warmup_steps
    self.criterion = nn.BCELoss()

  def forward(self, input_ids, attention_mask, labels=None):
    output = self.electra(input_ids, attention_mask=attention_mask)
    output = self.classifier(output.pooler_output) # <---- this is the line the error is referring to.
    output = torch.sigmoid(output)
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output

  def training_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("train_loss", loss, prog_bar=True, logger=True)
    return {"loss": loss, "predictions": outputs, "labels": labels}

  def validation_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("val_loss", loss, prog_bar=True, logger=True)
    return loss

  def test_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("test_loss", loss, prog_bar=True, logger=True)
    return loss

  def training_epoch_end(self, outputs):

    labels = []
    predictions = []
    for output in outputs:
      for out_labels in output["labels"].detach().cpu():
        labels.append(out_labels)
      for out_predictions in output["predictions"].detach().cpu():
        predictions.append(out_predictions)

    labels = torch.stack(labels).int()
    predictions = torch.stack(predictions)

    for i, name in enumerate(LABEL_COLUMNS):
      class_roc_auc = auroc(predictions[:, i], labels[:, i])
      self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)

  def configure_optimizers(self):

    optimizer = AdamW(self.parameters(), lr=2e-5)

    scheduler = get_linear_schedule_with_warmup(
      optimizer,
      num_warmup_steps=self.n_warmup_steps,
      num_training_steps=self.n_training_steps
    )

    return dict(
      optimizer=optimizer,
      lr_scheduler=dict(
        scheduler=scheduler,
        interval='step'
      )
    )

Can anyone point me in a direction to fix the error?
EXAMPLE OF DATA STRUCTURE (in CSV):
sentence                      label_1          label_2          label_3
Lorem ipsum dolor sit amet    1                0                1
consectetur adipiscing elit   0                0                0
sed do eiusmod tempor         0                1                1
incididunt ut labore et       1                0                0
Lorem ipsum dolor sit amet    1                0                1

Output:
ELECTRA has no pooler layer like BERT (compare the return section for further information).
In case you only want to use the [CLS] token for your sequence classification, you can simply take the first element of the last_hidden_state (initialize electra without return_dict=False):
output = self.classifier(output.last_hidden_state[:, 0])


Instruction: Pytorch RuntimeError: expected scalar type Double but found Float

Input:
I just started learning Pytorch and created my first LSTM. The dataset is time series data. Below is the training code.
Using .double() does not fix the error.
It is running in a Windows 11 environment.
import torch
import torch.nn as nn
from torch.optim import SGD
import math
import numpy as np

class Predictor(nn.Module):
    def __init__(self, inputDim, hiddenDim, outputDim):
        super(Predictor, self).__init__()

        self.rnn = nn.LSTM(input_size = inputDim,
                            hidden_size = hiddenDim,
                            batch_first = True)
        self.output_layer = nn.Linear(hiddenDim, outputDim)

    def forward(self, inputs, hidden0=None):
        output, (hidden, cell) = self.rnn(inputs, hidden0)
        output = self.output_layer(output[:, -1, :])

        return output

def mkDataSet(train_x, train_y=None):

    t_train_x = []
    t_train_y = []

    sequence_length = 50
    data_length = train_x.shape[0]

    for offset in range(data_length-sequence_length):
        t_train_x.append([train_x.iloc[offset + i] for i in range(sequence_length)])
        try:
            t_train_y.append([train_y.iloc[offset + i] for i in range(sequence_length)])
        except:
            pass

    return t_train_x, t_train_y

def mkRandomBatch(train_x, train_t, batch_size=10):
    batch_x = []
    batch_t = []

    for _ in range(batch_size):
        idx = np.random.randint(0, len(train_x) - 1)
        batch_x.append(train_x[idx])
        try:
            batch_t.append(train_t[idx])
        except:
            pass

    return torch.tensor(batch_x), torch.tensor(batch_t)

def main(train_x=train_x, train_y=train_y, test_x=test_x):
    training_size = 10000
    test_size = 1000
    epochs_num = 1000
    hidden_size = 5
    batch_size = 100

    train_x_origin, train_y_origin, test_x_origin = train_x.copy(), train_y.copy(), test_x.copy()

    train_x, train_t = mkDataSet(train_x, train_y)
    test_x = mkDataSet(test_x)

    model = Predictor(train_x_origin.shape[1], hidden_size, 1)
    criterion = nn.MSELoss()
    optimizer = SGD(model.parameters(), lr=0.01)

    for epoch in range(epochs_num):
        # training
        running_loss = 0.0
        training_accuracy = 0.0
        for i in range(int(training_size / batch_size)):
            optimizer.zero_grad()

            data, label = mkRandomBatch(train_x, train_t, batch_size)

            output = model(data)

            loss = criterion(output, label)
            loss.backward()
            optimizer.step()

            running_loss += loss.data[0]
            training_accuracy += np.sum(np.abs((output.data - label.data).numpy()) < 0.1)

        #test
        test_accuracy = 0.0
        for i in range(int(test_size / batch_size)):
            offset = i * batch_size
            data, label = torch.tensor(test_x[offset:offset+batch_size])
            output = model(data, None)

        training_accuracy /= training_size

        print('%d loss: %.3f, training_accuracy: %.5f, test_accuracy: %.5f' % (
            epoch + 1, running_loss, training_accuracy))

if __name__ == '__main__':
    main(train_x, train_y, test_x)

Then I have this error:
RuntimeError                              Traceback (most recent call last)
.ipynb Cell 26' in <cell line: 113>()
    109         print('%d loss: %.3f, training_accuracy: %.5f, test_accuracy: %.5f' % (
    110             epoch + 1, running_loss, training_accuracy))
    113 if __name__ == '__main__':
--> 114     main(train_x, train_y, test_x)

.ipynb Cell 26' in main(train_x, train_y, test_x)
     87 optimizer.zero_grad()
     89 data, label = mkRandomBatch(train_x, train_t, batch_size)
---> 91 output = model(data)
     93 loss = criterion(output, label)
     94 loss.backward()

File ~\Documents\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

.ipynb Cell 26' in Predictor.forward(self, inputs, hidden0)
     16 def forward(self, inputs, hidden0=None):
---> 17     output, (hidden, cell) = self.rnn(inputs, hidden0)
     18     output = self.output_layer(output[:, -1, :])
     20     return output

File ~\Documents\lib\site-packages\torch\nn\modules\module.py:1110, in Module._call_impl(self, *input, **kwargs)
   1106 # If we don't have any hooks, we want to skip the rest of the logic in
   1107 # this function, and just call forward.
   1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110     return forward_call(*input, **kwargs)
   1111 # Do not call functions when jit is used
   1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~\Documents\lib\site-packages\torch\nn\modules\rnn.py:761, in LSTM.forward(self, input, hx)
    759 self.check_forward_args(input, hx, batch_sizes)
    760 if batch_sizes is None:
--> 761     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
    762                       self.dropout, self.training, self.bidirectional, self.batch_first)
    763 else:
    764     result = _VF.lstm(input, batch_sizes, hx, self._flat_weights, self.bias,
    765                       self.num_layers, self.dropout, self.training, self.bidirectional)

RuntimeError: expected scalar type Double but found Float

I think there is something wrong with 'data' and I print its type by print(type(data)):
torch.Tensor

which is float instead of double. Do you know what`s wrong? Thanks for your help!

Output:
Your input data to the model is tensor of type Double, while the model expects a float tensor. Do this in the last line of mkRandomBatch() function:
 return torch.tensor(batch_x).float(), torch.tensor(batch_t)

You may or may not get a similar error for label tensor as well during loss calculation, in that case try converting label tensor to float as well.


Instruction: The DQN model cannot correctly come out the expected scores

Input:
I am working on a DQN training model of the game "CartPole-v1". In this model, the system did not remind any error information in the terminal. However, The result evaluation got worse.This is the output data:
episode: 85 score: 18 avarage score: 20.21 epsilon: 0.66
episode: 86 score: 10 avarage score: 20.09 epsilon: 0.66
episode: 87 score: 9 avarage score: 19.97 epsilon: 0.66
episode: 88 score: 14 avarage score: 19.90 epsilon: 0.65
episode: 89 score: 9 avarage score: 19.78 epsilon: 0.65
episode: 90 score: 10 avarage score: 19.67 epsilon: 0.65
episode: 91 score: 14 avarage score: 19.60 epsilon: 0.64
episode: 92 score: 13 avarage score: 19.53 epsilon: 0.64
episode: 93 score: 17 avarage score: 19.51 epsilon: 0.64
episode: 94 score: 10 avarage score: 19.40 epsilon: 0.63
episode: 95 score: 16 avarage score: 19.37 epsilon: 0.63
episode: 96 score: 16 avarage score: 19.33 epsilon: 0.63
episode: 97 score: 10 avarage score: 19.24 epsilon: 0.62
episode: 98 score: 13 avarage score: 19.17 epsilon: 0.62
episode: 99 score: 12 avarage score: 19.10 epsilon: 0.62
episode: 100 score: 11 avarage score: 19.02 epsilon: 0.61
episode: 101 score: 17 avarage score: 19.00 epsilon: 0.61
episode: 102 score: 11 avarage score: 18.92 epsilon: 0.61
episode: 103 score: 9 avarage score: 18.83 epsilon: 0.61

I'll show my code here. Firstly I constructed a neuron network:
import random
from torch.autograd import Variable
import torch as th
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import matplotlib.pyplot as plt
import gym
from collections import deque

# construct a neuron network (prepare for step1, step3.2 and 3.3)
class DQN(nn.Module):
    def __init__(self, s_space, a_space) -> None:

        # inherit from DQN class in pytorch
        super(DQN, self).__init__()
        self.fc1 = nn.Linear(s_space, 360)
        self.fc2 = nn.Linear(360, 360)
        self.fc3 = nn.Linear(360, a_space)

    # DNN operation architecture
    def forward(self, input):
        out = self.fc1(input)
        out = F.relu(out)
        out = self.fc2(out)
        out = F.relu(out)
        out = self.fc3(out)
        return out

Instead of newing an agent class, I directly created the select function, which is used for select the corresponding action according to epsilon, and the back propagation function by gradient globally:
# define the action selection according to epsilon using neuron network (prepare for step3.2)
def select(net, epsilon, env, state):

    # randomly select an action if not greedy
    if(np.random.rand() <= epsilon):
        action = env.action_space.sample()
        return action
    # select the maximum reward action by NN and the given state if greedy
    else:
        actions = net(Variable(th.Tensor(state))).detach().numpy()
        action = np.argmax(actions[0])
        return action

This is the back propagation function and the decreasing of epsilon:
# using loss function to improve neuron network (prepare for step3.3)
def backprbgt(net, store, batch_size, gamma, learning_rate):

    # step1: create loss function and Adam optimizer
    loss_F = nn.MSELoss()
    opt = th.optim.Adam(net.parameters(),lr=learning_rate)

    # step2: extract the sample in memory
    materials = random.sample(store, batch_size)

    # step3: Calculate arguments of loss function:

    for t in materials:

        Q_value = net(Variable(th.Tensor(t[0])))

        # step3.1 Calculate tgt_Q_value in terms of greedy:
        reward = t[3]
        if(t[4] == True):
            tgt = reward
        else:
            tgt = reward + gamma * np.amax(net(Variable(th.Tensor(t[2]))).detach().numpy()[0])
        # print(tgt)
        # tgt_Q_value = Variable(th.Tensor([[float(tgt)]]), requires_grad=True)

        # print("Q_value:",Q_value)
        Q_value[0][t[1]] = tgt
        tgt_Q_value = Variable(th.Tensor(Q_value))
        # print("tgt:",tgt_Q_value)

        # step3.2 Calculate evlt_Q_value

        # index = th.tensor([[t[1]]])
        # evlt_Q_value = Q_value.gather(1,index)  # gather tgt into the corresponding action
        evlt_Q_value = net(Variable(th.Tensor(t[0])))
        # print("evlt:",evlt_Q_value)

        # step4: backward and optimization
        loss = loss_F(evlt_Q_value, tgt_Q_value)
        # print(loss)
        opt.zero_grad()
        loss.backward()
        opt.step()

# step5: decrease epsilon for exploitation
def decrease(epsilon, min_epsilon, decrease_rate):
    if(epsilon > min_epsilon):
        epsilon *= decrease_rate

After that, the parameters and training progress are like this:
# training process

# step 1: set parameters and NN
episode = 1500
epsilon = 1.0
min_epsilon = 0.01
dr = 0.995
gamma = 0.9
lr = 0.001
batch_size = 40
memory_store = deque(maxlen=1500)

# step 2: define game category and associated states and actions
env = gym.make("CartPole-v1")
s_space = env.observation_space.shape[0]
a_space = env.action_space.n

net = DQN(s_space, a_space)
score = 0

# step 3: trainning
for e in range(0, episode):

    # step3.1: at the start of each episode, the current result should be refreshed

    # set initial state matrix
    s = env.reset().reshape(-1, s_space)

    # step3.2: iterate the state and action
    for run in range(500):

        # select action and get the next state according to current state "s"
        a = select(net, epsilon, env, s)
        obs, reward, done, info = env.step(a)

        next_s = obs.reshape(-1,s_space)
        s = next_s

        score += 1

        if(done == True):
            reward = -10.0
            memory_store.append((s,a,next_s,reward,done))
            avs = score / (e+1)
            print("episode:", e+1, "score:", run+1, "avarage score: {:.2f}".format(avs), "epsilon: {:.2}".format(epsilon))
            break

        # safe sample data
        memory_store.append((s, a, next_s, reward, done))

        if(run == 499):
            print("episode:", e+1, "score:", run+1, "avarage score:", avs)

    # step3.3 whenever the episode reach the integer time of batch size,
    # we should backward to implore the NN
    if(len(memory_store) > batch_size):
        backprbgt(net, memory_store, batch_size, gamma, lr) # here we need a backprbgt function to backward
        if(epsilon > min_epsilon):
            epsilon = epsilon * dr

In the entire progress of training, there was no error or exception reminds. However, instead of the score increasing, the model performed lower score in the later steps. I think the theory of this model is correct but cannot find where the error appears although I tried lots of methods improving my code, including rechecking the input arguments of network, modifing the data structure of two arguments of loss function, etc. I paste my code here and hope to get some help on how to fix it. Thanks!

Output:
Check out the code. For most parts it's the same as in snippet above, but there is some changes:

for step in replay buffer (which is called in code memory_store) namedtuple is used, and in update it's much easier to read t.reward, than looking what every index doing in step t

class DQN has method update, it's better to keep optimizer as attribute of class, than create it every time when calling function backprbgt

usage of torch.autograd.Variable here is unnecessary, so it's also was taken away

update in backprbgt taken per batch

decrease size of hidden layer from 360 to 32, while increase batch size from 40 to 128

updating network once in 10 episodes, but on 10 batches in replay buffer

average score prints out every 50 episodes based on 10 last episodes

add seeds

Also for RL it's take a long time to learn anything, so hoping that after 100  episodes it'll be close to even 100 points is somewhat optimistic. For the code in link averaging on 5 runs results in following dynamics
X axis -- number of episodes (yeah, 70 K, but it's like 20 minutes of real time)
Y axis -- number of steps in episode
As can be seen after 70K episodes algorithm achieves reward comparable to highest possible in this environment (highest -- 500). By tweaking hyperparameters faster rate can be achieved, but also remember it's DQN without any modification.


Instruction: Speeding up pytorch operations for custom message dropout

Input:
I am trying to implement message dropout in my custom MessagePassing convolution in PyTorch Geometric. Message dropout consists of randomly ignoring p% of the edges in the graph. My idea was to randomly remove p% of them from the input edge_index in forward().
The edge_index is a tensor of shape (2, num_edges) where the 1st dimension is the "from" node ID and the 2nd is the "to" node ID". So what I thought I could do is select a random sample of range(N) and then use it to mask out the rest of the indices:
    def forward(self, x, edge_index, edge_attr=None):
        if self.message_dropout is not None:
            # TODO: this is way too slow (4-5 times slower than without it)
            # message dropout -> randomly ignore p % of edges in the graph i.e. keep only (1-p) % of them
            random_keep_inx = random.sample(range(edge_index.shape[1]), int((1.0 - self.message_dropout) * edge_index.shape[1]))
            edge_index_to_use = edge_index[:, random_keep_inx]
            edge_attr_to_use = edge_attr[random_keep_inx] if edge_attr is not None else None
        else:
            edge_index_to_use = edge_index
            edge_attr_to_use = edge_attr

        ...

However, it is way too slow, it makes an epoch go for 5' instead of 1' without (5 times slower). Is there a faster way to do this in PyTorch?
Edit: The bottleneck seems to be the random.sample() call, not the masking. So I guess what I should be asking is for faster alternatives to that.

Output:
I managed to create a boolean mask using PyTorch's Dropout from Functional which is much faster. Now an epoch takes ~1' again. Better than other solutions with permutations that I found elsewhere.
    def forward(self, x, edge_index, edge_attr=None):
        if self.message_dropout is not None:
            # message dropout -> randomly ignore p % of edges in the graph
            mask = F.dropout(torch.ones(edge_index.shape[1]), self.message_dropout, self.training) > 0
            edge_index_to_use = edge_index[:, mask]
            edge_attr_to_use = edge_attr[mask] if edge_attr is not None else None
        else:
            edge_index_to_use = edge_index
            edge_attr_to_use = edge_attr

        ...


Instruction: How to turn 'Detections' object into string

Input:
I have this code which uses YOLOV5 and the weights of my trained model for inferencing images to detect and recognize the Egyptian currency. The object 'results' is of datatype 'Detections'. Is there someway to convert it into a string object as I need the label output for some purpose? I tried the str() function but it didn't work.
import torch
import os
import cv2

# Model
model = torch.hub.load(r'C:\Users\HAYA\PycharmProjects\curency_recognition\yolov5-master\yolov5-master', 'custom', path=r'C:\Users\HAYA\PycharmProjects\curency_recognition\_best.pt', source='local')
# Image
im = [r'E:\_currency.jpg']
# Inference
results = model(im)
# results
results.print()
results.save()  # or .show()
results.show()
results.xyxy[0]  # img1 predictions (tensor)
results.pandas().xyxy[0]

Output:
The following function is kind of "buggy" but it can give you the required results
def results_parser(results):
  s = ""
  if results.pred[0].shape[0]:
    for c in results.pred[0][:, -1].unique():
      n = (results.pred[0][:, -1] == c).sum()  # detections per class
      s += f"{n} {results.names[int(c)]}{'s' * (n > 1)}, "  # add to string
  return s

I run the default yolov5s on Zidane image from the original repository, after getting results object I did the following:
# Model
model = torch.hub.load(r'/content/yolov5//', 'custom', path=r'/content/yolov5/yolov5s.pt', source='local')
# Image
im = [r'runs/detect/exp/zidane.jpg']
# Inference
results = model(im)
print(results_parser(results))

and the output is:
2 persons, 2 ties,

I wish you find this answer helpful, have a good day)


Instruction: PyTorch autoencoder produces only one image

Input:
I'm new to PyTorch, and am having trouble with a basic autoencoder for a MNIST dataset. An autoencoder is a neural network that trains so that the output recovers the input through narrower layers in between, so that it learns a lower-dimensional representation of a high-dimensional input space.
The error is that all of my trained examples are outputting the same output. I'm not sure where the error is; I modified an online tutorial for this, and I believe all the different images aren't supposed to be outputted into one output. Could anyone help me find out if there is any simple bug or a wrong setting that I didn't spot?
Here is a code snippet that reproduces my problem.
# AE one-block
import time
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
import torch
import torchvision
from torch import nn
from torchvision import transforms
from torchvision.transforms import ToTensor
import torch.nn.functional as F
from torch.utils.tensorboard import SummaryWriter
from torch.optim.lr_scheduler import ExponentialLR
import torchvision.datasets as datasets
from torch.utils.data import DataLoader
from torchvision import datasets

class AE(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.flatten = nn.Flatten()

        self.encoder = torch.nn.Sequential(
            torch.nn.Linear(28 * 28, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 64),
            torch.nn.ReLU(),
            torch.nn.Linear(64, 36),
            torch.nn.ReLU(),
            torch.nn.Linear(36, 18),
            torch.nn.ReLU(),
            torch.nn.Linear(18, 9)
        )

        self.decoder = torch.nn.Sequential(
            torch.nn.Linear(9, 18),
            torch.nn.ReLU(),
            torch.nn.Linear(18, 36),
            torch.nn.ReLU(),
            torch.nn.Linear(36, 64),
            torch.nn.ReLU(),
            torch.nn.Linear(64, 128),
            torch.nn.ReLU(),
            torch.nn.Linear(128, 28 * 28),
            torch.nn.Sigmoid()
        )

    def forward(self, x):
        x = self.flatten(x)
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

# Model initialization
model = AE()
loss_function = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 1e-1, weight_decay = 1e-8)
tensor_transform = transforms.ToTensor()
dataset = datasets.MNIST(root = "./data",
                         train = True,
                         download = True,
                         transform = tensor_transform)
loader = torch.utils.data.DataLoader(dataset = dataset,
                                     batch_size = 32,
                                     shuffle = True)

# Train
epochs = 5
outputs = []
losses = []
for epoch in range(epochs):
    tic = time.monotonic()
    print(f'Epoch = {epoch}')
    for (image, _) in loader:
      image = image.reshape(-1, 28*28)
      reconstructed = model(image)
      loss = loss_function(reconstructed, image)
      optimizer.zero_grad()
      loss.backward()
      optimizer.step()
      losses.append(loss)
    outputs.append((epochs, image, reconstructed))
    toc = time.monotonic()
    print(f'Time taken = {round(toc - tic, 2)}')

# Calculate difference between image outputs
im_batches = [image_batch for (image_batch, _) in loader]
only_image = model(im_batches[0]).detach().numpy()[0]
diff_total = 0
for i in range(len(im_batches)):
    im_out = model(im_batches[i]).detach().numpy()
    diff = np.linalg.norm(im_out - only_image)
print(f'Difference between outputs = {diff_total}')

# Show image outputs
im_out1 = model(im_batches[0]).detach().numpy()
im_out2 = model(im_batches[1]).detach().numpy()
for i in range(3):
    plt.imshow(im_out1[i].reshape(28, 28))
    plt.show()
for i in range(3):
    plt.imshow(im_out2[i].reshape(28, 28))
    plt.show()

My Python computation prints that the "difference between outputs" is zero, indicating that all output images from first 10 batches have the same output. A visual examination through directly looking at the first few images also reveals that outputs look like a strange amalgamation of all MNIST digit images.

Output: Decrease learning rate of optimizer from 1e-1 (which is really big) down to 1e-4 (which is somewhat standard) and increase number of epochs from 5 to 10, outputs will no longer be same. When printing "difference between outputs" variable diff_total is used and it does not changed, in cycle variable diff is computed and there is no interaction with diff_total. So even when epochs == 0 and model outputs are random, "difference between outputs" will also be equal to 0. Also it's better for memory consumption to do losses.append(loss.item()).


Instruction: MultiHeadAttention giving very different values between versions (Pytorch/Tensorflow

Input:
I'm trying to recreate a transformer that was written in Pytorch and make it Tensorflow. Everything was going pretty well until each version of MultiHeadAttention started giving extremely different outputs. Both methods are an implementation of multi-headed attention as described in the paper "Attention is all you Need", so they should be able to achieve the same output.
I'm converting
self_attn = nn.MultiheadAttention(dModel, nheads, dropout=dropout)

to
self_attn = MultiHeadAttention(num_heads=nheads, key_dim=dModel, dropout=dropout)

For my tests, dropout is 0.
I'm calling them with:
self_attn(x,x,x)

where x is a tensor with shape=(10, 128, 50)
As expected from the documentation, the Pytorch version returns a tuple, (the target sequence length, embedding dimension), both with dimensions [10, 128, 50].
I'm having trouble getting the TensorFlow version to do the same thing. With Tensorflow I only get one tensor back, (size [10, 128, 50]) and it looks like neither the target sequence length or embedding dimension tensor from pytorch.
Based on the Tensorflow documentation I should be getting something comparable.
How can I get them to operate the same way? I'm guessing I'm doing something wrong with Tensorflow but I can't figure out what.

Output:
nn.MultiheadAttention outputs by default tuple with two tensors:

attn_output -- result of self-attention operation

attn_output_weights -- attention weights averaged(!) over heads

At the same time tf.keras.layers.MultiHeadAttention outputs by default only one tensor attention_output (which corresponds to attn_output of pytorch). Attention weights of all heads also will be returned if parameter return_attention_scores is set to True, like:
output, scores = self_attn(x, x, x, return_attention_scores=True)

Tensor scores also should be averaged to achieve full correspondence with pytorch:
scores = tf.math.reduce_mean(scores, 1)

While rewriting keep in mind that by default (as in snippet in question) nn.MultiheadAttention expects input in form (seq_length, batch_size, embed_dim), but tf.keras.layers.MultiHeadAttention expects it in form (batch_size, seq_length, embed_dim).


Instruction: Why do I get the error: ZeroDivisionError: 0.0 cannot be raised to a negative power error when loading deit model from Timm

Input:
I am trying to make inference on the DeiT small variant from timm.
from timm.models import create_model
model = create_model('deit_small_patch16_224', pretrained=True)

But I get the error:
self.scale = head_dim ** -0.5
ZeroDivisionError: 0.0 cannot be raised to a negative power.

However, creating a different model with model = create_model('deit_tiny_patch16_224', pretrained=True), I can make inference successfully and it works pretty fine. I understand this error is as a result of a zero being divided by a non-zero value or when being raised to the power of non-zero. But I don't quite get why this model is flagging an error from timm even before loading any data.

Output: It was an error with my version of timm. Upgrading timm: pip install --upgrade timm solved the problem.


Instruction: Given groups=1, weight of size [10, 1, 5, 5], expected input[2, 3, 28, 28] to have 1 channels, but got 3 channels instead

Input:
I am trying to run  CNN with train MNIST, but test on my own written digits. To do that I wrote the following code but I getting an error in title of this questions:
I am trying to run  CNN with train MNIST, but test on my own written digits. To do that I wrote the following code but I getting an error in title of this questions:
batch_size = 64
train_dataset = datasets.MNIST(root='./data/',
                               train=True,
                               transform=transforms.ToTensor(),
                               download=True)
test_dataset = ImageFolder('my_digit_images/', transform=transforms.ToTensor())
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        #print(self.conv1.weight.shape)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv3 = nn.Conv2d(20, 20, kernel_size=3)
       #print(self.conv2.weight.shape)
        self.mp = nn.MaxPool2d(2)
        self.fc = nn.Linear(320, 10)

    def forward(self, x):
        in_size = x.size(0)
        x = F.relu(self.conv1(x))
        #print(x.shape)
        x = F.relu(self.mp(self.conv2(x)))
        x = F.relu(self.mp(self.conv3(x)))

        #print("2.", x.shape)
       # x = F.relu(self.mp(self.conv3(x)))
        x = x.view(in_size, -1)  # flatten the tensor
        #print("3.", x.shape)
        x = self.fc(x)
        return F.log_softmax(x)
model = Net()
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)
def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test():
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        data, target = Variable(data, volatile=True), Variable(target)
        output = model(data)
        test_loss += F.nll_loss(output, target, size_average=False).data
        pred = output.data.max(1, keepdim=True)[1]
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

Output:
MNIST dataset contains black and white 1-channel images, while yours are 3-channeled RGB probably. Either recode your images or preprocess it like
img = img[:,0:1,:,:]

You can do it with custom transform, adding it after transforms.ToTensor()


Instruction: LibTorch with OpenCV: version GOMP_5.0 not found

Input:
I'm trying to use OpenCV and LibTorch in the same project. Libtorch is installed in /usr/include/libtorch, downloaded from the PyTorch website. I'm using the cxx11 ABI version for CUDA 11.3.
Here's my CMakeLists.txt file:
cmake_minimum_required(VERSION 3.23 FATAL_ERROR)
project(chess-rl VERSION 1.0)

find_package( OpenCV REQUIRED )

set(CMAKE_PREFIX_PATH /usr/include/libtorch/share/cmake/)

find_package(Torch REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${OpenCV_LIBS})
include_directories(${TORCH_INCLUDE_DIRS})
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

file (GLOB SOURCE_FILES
  ${PROJECT_SOURCE_DIR}/src/*.cc
  ${PROJECT_SOURCE_DIR}/src/chess/*.cc
)

add_executable(${PROJECT_NAME} ${SOURCE_FILES})
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS} "${TORCH_LIBRARIES}" )
set_property(TARGET ${PROJECT_NAME} PROPERTY CXX_STANDARD 20)

The program has a #include <opencv2/opencv.hpp> line at the top.
Compilation works fine, but running the executable gives me this error:
/usr/include/libtorch/lib/libgomp-52f2fd74.so.1: version `GOMP_5.0' not found (required by /usr/lib/libvtkCommonCore.so.1)

I believe libtorch is trying to use a library that is incompatible with OpenCV.
If I run the program with LD_PRELOAD=/usr/lib/libgomp.so ./build/my-program, it runs fine.
How can I fix this error without having to use that environment variable? Is there a way to link that particular library in CMake?

Output:
I faced the same problem, and your thread contributed to one solution.
You need to add a shared library in your CMakelists.txt file.
add_library(libName SHARED IMPORTED)
set_property(TARGET libName PROPERTY IMPORTED_LOCATION "/usr/lib/libgomp.so")
target_link_libraries(PROJECT_NAME
libName)

I hope I could help you
With kind regards


Instruction: How to add multiple layers to an RNN module for sentiment analysis? Pytorch

Input:
I am trying to create a sentiment analysis model with Pytorch (newbie)
import torch.nn as nn

class RNN(nn.Module):
 def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim, dropout):
    super().__init__() #to call the functions in the superclass
    self.embedding = nn.Embedding(input_dim, embedding_dim) #Embedding layer to create dense vector instead of sparse matrix
    self.rnn = nn.RNN(embedding_dim, hidden_dim)
    self.fc = nn.Linear(hidden_dim, output_dim)
    self.dropout = nn.Dropout(dropout)

def forward(self, text):
    embedded = self.embedding(text)
    output, hidden = self.rnn(embedded)
    hidden = self.dropout(hidden[-1,:,:])
    nn.Sigmoid()
    return self.fc(hidden)

However, the accuracy is below 50% and I would like to add an extra layer, maybe another linear before feeding it to the last linear to get the prediction. What kind of layers can I add after the RNN and before the last Linear? and also what should I feed it with?
I have tried simply adding another
output, hidden= self.fc(hidden)

but I get

ValueError: too many values to unpack (expected 2)

Which I believe is because the output of the previous layer with activation and dropout is different. The help is greatly appreciated.
Thanks

Output:
You were very close, just change your forward call to:
import torch.nn.functional as F
class model_RNN(nn.Module):
 def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim, dropout):
    super().__init__() #to call the functions in the superclass
    self.embedding = nn.Embedding(input_dim, embedding_dim) #Embedding layer to create dense vector instead of sparse matrix
    self.rnn = nn.RNN(embedding_dim, hidden_dim)
    self.hidden_fc = nn.Linear(hidden_dim,hidden_dim)
    self.out_fc = nn.Linear(hidden_dim, output_dim)
    self.dropout = nn.Dropout(dropout)

def forward(self, text):
    embedded = self.embedding(text)
    output, hidden = self.rnn(embedded)
    hidden = self.dropout(hidden[-1,:,:])
    hidden = F.relu(torch.self.hidden_fc(hidden))
    return self.out_fc(hidden)

Just a note, calling nn.Sigmoid() won't do anything to your model output because it will just create a sigmoid layer but won't call it on your data. What you want is probably torch.sigmoid(self.fc(hidden)). Although I would say it's not recommended to use an output activation because some common loss functions require the raw logits. Make sure you apply the sigmoid after the model call in eval mode though!


Instruction: Trying to achieve same result with Pytorch and Tensorflow MultiheadAttention

Input:
I'm trying to recreate a transformer written in Pytorch and implement it in Tensorflow. The problem is that despite both the documentation for the Pytorch version and Tensorflow version, they still come out pretty differently.
I wrote a little code snippet to show the issue:
import torch
import tensorflow as tf
import numpy as np

class TransformerLayer(tf.Module):
    def __init__(self, d_model, nhead, dropout=0):
        super(TransformerLayer, self).__init__()
        self.self_attn = torch.nn.MultiheadAttention(d_model, nhead, dropout=dropout)

batch_size = 2
seq_length = 5
d_model = 10

src = np.random.uniform(size=(batch_size, seq_length, d_model))
srcTF = tf.convert_to_tensor(src)
srcPT = torch.Tensor(src.reshape((seq_length, batch_size, d_model)))

self_attnTF = tf.keras.layers.MultiHeadAttention(key_dim=10, num_heads=5, dropout=0)
transformer_encoder = TransformerLayer(d_model=10, nhead=5, dropout=0.0)

output, scores = self_attnTF(srcTF, srcTF, srcTF, return_attention_scores=True)
print("Tensorflow Attendtion outputs:", output)
print("Tensorflow (averaged) weights:", tf.math.reduce_mean(scores, 1))
print("Torch Attendtion outputs:", transformer_encoder.self_attn(srcPT,srcPT,srcPT)[0])
print("Torch attention output weights:", transformer_encoder.self_attn(srcPT,srcPT,srcPT)[1])

and the result is:
Tensorflow Attendtion outputs: tf.Tensor(
[[[ 0.02602757 -0.14134401  0.00855263  0.4735083  -0.01851891
   -0.20382246 -0.18152176 -0.21076852  0.08623976 -0.33548725]
  [ 0.02607442 -0.1403394   0.00814065  0.47415024 -0.01882939
   -0.20353754 -0.18291879 -0.21234266  0.08595885 -0.33613583]
  [ 0.02524654 -0.14096384  0.00870436  0.47411725 -0.01800703
   -0.20486829 -0.18163288 -0.21082559  0.08571021 -0.3362339 ]
  [ 0.02518575 -0.14039244  0.0090138   0.47431853 -0.01775141
   -0.20391947 -0.18138805 -0.2118245   0.08432849 -0.33521986]
  [ 0.02556361 -0.14039293  0.00876258  0.4746476  -0.01891363
   -0.20398234 -0.18229616 -0.21147579  0.08555281 -0.33639923]]

 [[ 0.07844199 -0.1614371   0.01649148  0.5287745   0.05126739
   -0.13851154 -0.09829871 -0.1621251   0.01922669 -0.2428589 ]
  [ 0.07844222 -0.16024739  0.01805423  0.52941847  0.04975721
   -0.13537636 -0.09829231 -0.16129729  0.01979005 -0.24491176]
  [ 0.07800542 -0.160701    0.01677295  0.52902794  0.05082911
   -0.13843337 -0.09805533 -0.16165744  0.01928401 -0.24327613]
  [ 0.07815789 -0.1600025   0.01757433  0.5291927   0.05032986
   -0.1368022  -0.09849522 -0.16172451  0.01929555 -0.24438493]
  [ 0.0781548  -0.16028519  0.01764914  0.52846324  0.04941286
   -0.13746066 -0.09787872 -0.16141161  0.01994199 -0.2440269 ]]], shape=(2, 5, 10), dtype=float32)
Tensorflow (averaged) weights: tf.Tensor(
[[[0.199085   0.20275716 0.20086522 0.19873264 0.19856   ]
  [0.2015336  0.19960018 0.20218948 0.19891861 0.19775811]
  [0.19906266 0.20318432 0.20190334 0.19812575 0.19772394]
  [0.20074987 0.20104568 0.20269363 0.19744729 0.19806348]
  [0.19953248 0.20176074 0.20314851 0.19782843 0.19772986]]

 [[0.2010009  0.20053487 0.20004745 0.20092985 0.19748697]
  [0.20034568 0.20035927 0.19955876 0.20062163 0.19911464]
  [0.19967113 0.2006859  0.20012529 0.20047483 0.19904283]
  [0.20132652 0.19996871 0.20019794 0.20008174 0.19842513]
  [0.2006393  0.20000939 0.19938737 0.20054278 0.19942114]]], shape=(2, 5, 5), dtype=float32)
Torch Attendtion outputs: tensor([[[ 0.1097, -0.4467, -0.0719, -0.1779, -0.0766, -0.1247,  0.1557,
           0.0051, -0.3932, -0.1323],
         [ 0.1264, -0.3822,  0.0759, -0.0335, -0.1084, -0.1539,  0.1475,
          -0.0272, -0.4235, -0.1744]],

        [[ 0.1122, -0.4502, -0.0747, -0.1796, -0.0756, -0.1271,  0.1581,
           0.0049, -0.3964, -0.1340],
         [ 0.1274, -0.3823,  0.0754, -0.0356, -0.1091, -0.1547,  0.1477,
          -0.0272, -0.4252, -0.1752]],

        [[ 0.1089, -0.4427, -0.0728, -0.1746, -0.0756, -0.1202,  0.1501,
           0.0031, -0.3894, -0.1242],
         [ 0.1263, -0.3820,  0.0718, -0.0374, -0.1063, -0.1562,  0.1485,
          -0.0271, -0.4233, -0.1761]],

        [[ 0.1061, -0.4369, -0.0685, -0.1696, -0.0772, -0.1173,  0.1454,
           0.0012, -0.3860, -0.1201],
         [ 0.1265, -0.3820,  0.0762, -0.0325, -0.1082, -0.1560,  0.1501,
          -0.0271, -0.4249, -0.1779]],

        [[ 0.1043, -0.4402, -0.0705, -0.1719, -0.0791, -0.1205,  0.1508,
           0.0018, -0.3895, -0.1262],
         [ 0.1260, -0.3805,  0.0775, -0.0298, -0.1083, -0.1547,  0.1494,
          -0.0276, -0.4242, -0.1768]]], grad_fn=<AddBackward0>)
Torch attention output weights: tensor([[[0.2082, 0.2054, 0.1877, 0.1956, 0.2031],
         [0.2100, 0.2079, 0.1841, 0.1943, 0.2037],
         [0.2007, 0.1995, 0.1929, 0.1999, 0.2070],
         [0.1995, 0.1950, 0.1976, 0.2002, 0.2077],
         [0.1989, 0.1969, 0.1970, 0.2024, 0.2048]],

        [[0.2095, 0.1902, 0.1987, 0.2027, 0.1989],
         [0.2090, 0.1956, 0.1997, 0.2004, 0.1952],
         [0.2047, 0.1869, 0.2006, 0.2121, 0.1957],
         [0.2073, 0.1953, 0.1982, 0.2014, 0.1978],
         [0.2089, 0.2003, 0.1953, 0.1957, 0.1998]]], grad_fn=<DivBackward0>)

The output weights look similar but the base attention outputs are way off. Is there any way to make the Tensorflow model come out more like the Pytorch one? Any help would be greatly appreciated!

Output:
In MultiHeadAttention there is also a projection layer, like
Q = W_q @ input_query + b_q
K = W_k @ input_keys + b_k
V = W_v @ input_values + b_v

Matrices W_q, W_k and W_v and biases b_q, b_k, b_v are initialized randomly, so difference in outputs should be expected (even between outputs of two distinct layers in pytorch on same input). After self-attention operation there is one more projection and it's also initialized randomly. Weights can be set manually in tensorflow by calling method set_weights of self_attnTF.
Correspondence between weights in tf.keras.layers.MultiHeadAttention and nn.MultiheadAttention not so clear, as an example: torch shares weights between heads, while tf keeps them unique. So if you are using weights of pretrained model from pytorch and try to put them in tensorflow model (for whatever reason) it'll certainly take more than five minutes.
Results should be the same if after initializing pytorch model and tensorflow model you step through their parameters and assign them identical values.


Instruction: TypeError: setup() got an unexpected keyword argument 'stage'

Input:
I am trying to train my q&a model through pytorch_lightning. However while running the command trainer.fit(model,data_module) I am getting the following error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-72-b9cdaa88efa7> in <module>()
----> 1 trainer.fit(model,data_module)

4 frames
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _call_setup_hook(self)
   1488
   1489         if self.datamodule is not None:
-> 1490             self.datamodule.setup(stage=fn)
   1491         self._call_callback_hooks("setup", stage=fn)
   1492         self._call_lightning_module_hook("setup", stage=fn)

TypeError: setup() got an unexpected keyword argument 'stage'

I have installed and imported pytorch_lightning.
Also I have defined data_module = BioQADataModule(train_df, val_df, tokenizer, batch_size = BATCH_SIZE) where BATCH_SIZE = 2, N_EPOCHS = 6.
The model I have used is as follows:-
model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)

Also, I have defined the class for the model as follows:-
    class BioQAModel(pl.LightningModule):

      def __init__(self):
        super().__init__()
        self.model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)

      def forward(self, input_ids, attention_mask, labels=None):
        output = self.model(
            input_ids = encoding["input_ids"],
            attention_mask = encoding["attention_mask"],
            labels=labels
        )

        return output.loss, output.logits

      def training_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("train_loss", loss, prog_bar=True, logger=True)
        return loss

      def validation_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("val_loss", loss, prog_bar=True, logger=True)
        return loss

      def test_step(self, batch, batch_idx):
        input_ids = batch["input_ids"]
        attention_mask = batch["attention_mask"]
        labels = batch["labels"]
        loss, outputs = self(input_ids, attention_mask, labels)
        self.log("test_loss", loss, prog_bar=True, logger=True)
        return loss

      def configure_optimizers(self):
        return AdamW(self.parameters(), lr=0.0001)

For any additional information required, please specify.
Edit 1: Adding BioQADataModule:
class BioQADataModule(pl.LightningDataModule):

  def __init__(
      self,
      train_df: pd.DataFrame,
      test_df: pd.DataFrame,
      tokenizer: T5Tokenizer,
      batch_size: int = 8,
      source_max_token_len = 396,
      target_max_token_len = 32
    ):
      super().__init__()
      self.batch_size = batch_size
      self.train_df = train_df
      self.test_df = test_df
      self.tokenizer = tokenizer
      self.source_max_token_len = source_max_token_len
      self.target_max_token_len = target_max_token_len

  def setup(self):
    self.train_dataset = BioQADataset(
        self.train_df,
        self.tokenizer,
        self.source_max_token_len,
        self.target_max_token_len
    )

    self.test_dataset = BioQADataset(
        self.test_df,
        self.tokenizer,
        self.source_max_token_len,
        self.target_max_token_len
    )

  def train_dataloader(self):
    return DataLoader(
        self.train_dataset,
        batch_size = self.batch_size,
        shuffle = True,
        num_workers = 4
    )

  def val_dataloader(self):
    return DataLoader(
        self.train_dataset,
        batch_size = 1,
        shuffle = True,
        num_workers = 4
    )

  def test_dataloader(self):
    return DataLoader(
        self.train_dataset,
        batch_size = 1,
        shuffle = True,
        num_workers = 4
    )

Output:
You need to add an extra argument stage=None to your setup method:
def setup(self, stage=None):
    self.train_dataset = BioQADataset(
        self.train_df,
        self.tokenizer,
        self.source_max_token_len,
        self.target_max_token_len
    )

    self.test_dataset = BioQADataset(
        self.test_df,
        self.tokenizer,
        self.source_max_token_len,
        self.target_max_token_len
    )

I've played with Pytorch Lightning myself for multi-GPU training here. Although some of the code is a bit outdated (metrics are a standalone module now), you might find it useful.


Instruction: Using Dice metric in pytorch "torchmetrics" : dice_score() missing 2 required positional arguments: 'preds' and 'target'

Input:
I'm trying to use Dice metric from pytorch "torchmetrics". I found an example for using accuracy metric. like below :
from torchmetrics.classification import Accuracy

train_accuracy = Accuracy()
valid_accuracy = Accuracy()

for epoch in range(epochs):
    for x, y in train_data:
        y_hat = model(x)

        # training step accuracy
        batch_acc = train_accuracy(y_hat, y)
        print(f"Accuracy of batch{i} is {batch_acc}")

    for x, y in valid_data:
        y_hat = model(x)
        valid_accuracy.update(y_hat, y)

    # total accuracy over all training batches
    total_train_accuracy = train_accuracy.compute()

    # total accuracy over all validation batches
    total_valid_accuracy = valid_accuracy.compute()

    print(f"Training acc for epoch {epoch}: {total_train_accuracy}")
    print(f"Validation acc for epoch {epoch}: {total_valid_accuracy}")

    # Reset metric states after each epoch
    train_accuracy.reset()
    valid_accuracy.reset()

However, when I replaced "Accuracy()"  with "Dice_score()". like below:
from torchmetrics.functional import dice_score

train_accuracy =dice_score()
valid_accuracy =dice_score()

I got below error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-43-726045592283> in <module>
      3 from torchmetrics.functional import dice_score
      4
----> 5 train_accuracy_2 =dice_score()# Accuracy()
      6 valid_accuracy_2 =dice_score()# Accuracy()
      7

TypeError: dice_score() missing 2 required positional arguments: 'preds' and 'target'

Is there an example of using "Dice" metric from "torchmetrics"

Output:
torchmetrics.classification.dice_score is the functional interface to the Dice score. That means it is a stateless function that expects the ground truth and predictions. There doesn't seem to be a module interface to the Dice score, like there is with accuracy.
torchmetrics.classification.Accuracy is a class that maintains state. Under the hood, it uses the functional interface, which is torchmetrics.functional.accuracy.
This is not enforced in any way, but typically classes are named with CamelCase and functions are named with snake_case.


Instruction: Distributed sequential windowed data in pytorch

Input:
At every epoch of my training, I need to split my dataset in n batches of t consecutive samples. For example, if my data is [1,2,3,4,5,6,7,8,9,10], n = 2 and t = 3 then valid batches would be
[1-2-3, 4-5-6] and [7-8-9, 10-1-2]
[2-3-4, 8-9-10] and [5-6-7, 1-2-3]

My old version is the following, but it samples every point in the data, meaning that I would parse the whole dataset t times per epoch.
train_dataset = list(range(n))
train_sampler = None
if distributed:
    train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=bsize, shuffle=(train_sampler is None),
        pin_memory=True, sampler=train_sampler)

for epoch in range(epochs):
    if distributed:
        train_sampler.set_epoch(epoch)

    for starting_i in train_loader:
        batch = np.array([np.mod(np.arange(i, i + t), n) for i in starting_i])

I have now implemented my own sampling function that splits the data into random batches where each sample is far from the two closest exactly t.  In the non-distributed scenario, I can do
for epoch in range(epochs):
    pad = np.random.randint(n)
    train_loader = np.mod(np.arange(pad, n + pad, t), n)
    np.random.shuffle(train_loader)
    train_loader = np.array_split(train_loader,
        np.ceil(len(train_loader) / bsize))

    for starting_i in train_loader:
        batch = np.array([np.mod(np.arange(i, i + t), n) for i in starting_i])

How do I make this version distributed? Do I need to make a custom torch.nn.parallel.DistributedDataParallel or torch.utils.data.DataLoader?
I have checked the DistributedSampler class
and my guess is that I have to override the __iter__ method. Am I right?
How does DistributedSampler split the dataset? Is it sequentially among num_replicas?
Say num_replicas = 2. Would my dataset be split into [1,2,3,4,5] and [6,7,8,9,10] between the 2 workers? Or is it random? Like [1,4,7,3,10] and [2,9,5,8,6]? First case would be ok for me because keeps samples sequential, but second would not.

Output:
I ended up making my own Dataset where the data is [t, t + window, ... t + n * window]. Every time it is called it randomizes the starting indices of the window. Then the sampler does the shuffling as usual. For reproducibility, it has a set_seed method similar to set_epoch of samplers.
class SequentialWindowedDataset(Dataset):
    def __init__(self, size, window):
        self.size = size
        self.window = window
        self.seed = 0
        self.data = np.arange(0, self.size, self.window)

    def __getitem__(self, index):
        rng = np.random.default_rng(self.seed)
        pad = rng.integers(0, self.size)
        data = (self.data + pad) % self.size
        return data[index]

    def __len__(self):
        return len(self.data)

    def set_seed(self, seed):
        self.seed = seed

The following version randomizes the data outside the call and it is much much faster.
class SequentialWindowedDataset(Dataset):
    def __init__(self, size, window):
        self.size = size
        self.window = window
        self.data = np.arange(0, self.size, self.window)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)

    def randomize(self, seed):
        rng = np.random.default_rng(seed)
        pad = rng.integers(0, self.size)
        self.data = (self.data + pad) % self.size


Instruction: Is this a correct way to training U-Net using different GPUs

Input:
U-Net code:
class UNet(nn.Module):
    def __init__(self):
        super(UNet, self).__init__()
        self.c1 = convBlock(1, 64).to('cuda:0')
        self.d1 = downSample(64).to('cuda:0')
        self.c2 = convBlock(64, 128).to('cuda:0')
        self.d2 = downSample(128).to('cuda:0')
        self.c3 = convBlock(128, 256).to('cuda:0')
        self.d3 = downSample(256).to('cuda:1')
        self.c4 = convBlock(256, 512).to('cuda:1')
        self.d4 = downSample(512).to('cuda:1')
        self.c5 = convBlock(512, 1024).to('cuda:1')
        self.u1 = upSample(1024).to('cuda:1')
        self.c6 = convBlock(1024, 512).to('cuda:1')
        self.u2 = upSample(512).to('cuda:1')
        self.c7 = convBlock(512, 256).to('cuda:1')
        self.u3 = upSample(256).to('cuda:1')
        self.c8 = convBlock(256, 128).to('cuda:1')
        self.u4 = upSample(128).to('cuda:0')
        self.c9 = convBlock(128, 64).to('cuda:0')
        self.out = nn.Conv3d(64, 1, 3, 1, 1).to('cuda:0')
        self.th = nn.Sigmoid().to('cuda:0')

    def forward(self, x):
        L1 = self.c1(x.to('cuda:0'))
        L2 = self.c2(self.d1(L1.to('cuda:0')).to('cuda:0'))
        L3 = self.c3(self.d2(L2.to('cuda:0')).to('cuda:0'))
        L4 = self.c4(self.d3(L3.to('cuda:1')).to('cuda:1'))
        L5 = self.c5(self.d4(L4.to('cuda:1')).to('cuda:1'))
        R4 = self.c6(self.u1(L5.to('cuda:1'), L4.to('cuda:1')).to('cuda:1'))
        R3 = self.c7(self.u2(R4.to('cuda:1'), L3.to('cuda:1')).to('cuda:1'))
        R2 = self.c8(self.u3(R3.to('cuda:1'), L2.to('cuda:1')).to('cuda:1'))
        R1 = self.c9(self.u4(R2.to('cuda:0'), L1.to('cuda:0')).to('cuda:0'))

        return self.th(self.out(R1.to('cuda:0')).to('cuda:0'))

convBlock, downSample, upSample is layer in my own code.
I want to train 3DU-Net, but the GPU memory is not enough, so I want to use multiple GPUs to train this model.
I assign different U-net layers to different GPUs.
I want to ask if this is the correct way to use different GPUs to train models? And what's the best way to run multiple GPU training python scripts using the PyTorch module in the Linux server?

Output:
Your code should work, but I'd suggest using some sort of variable to transfer submodel/tensor to different gpus. Something like this is what I've been using:
class MyModel(nn.Module):
    def __init__(self, split_bool: bool = False):
        self.submodule1 = ...
        self.submodule2 = ...

        self.split_bool = split_bool
        if split_bool:
            self.submodule1.cuda(0)
            self.submodule2.cuda(1)

    def forward(self, x):
        x = self.submodule1(x)
        if self.split_bool:
            x = x.cuda(1) # Transfer tensor to second GPU
        return self.submodule2(x)

For multiple training it really depends on your server. Are you using tensorboard/tensorboardX to plot results? You can launch multiple training script with different parameters with tmux, or even write your own bash script.