Instruction: How to enable Intel Extension for Pytorch(IPEX) in my python code?

Input:
I would like to use Intel Extension for Pytorch in my code to increase overall performance. Referred this GitHub(https://github.com/intel/intel-extension-for-pytorch) for installation.
Currently, I am trying out a hugging face summarization PyTorch sample(https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py). Below is the trainer API used for training.

   # Initialize our Trainer
    trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics if training_args.predict_with_generate else None,
    )

I am not aware of enabling Ipex in this code. Can anyone help me with this?
Thanks in Advance!

Output:
The key changes that are required to enable IPEX are:
    #Import the library:
    import intel_extension_for_pytorch as ipex
    #Apply the optimizations to the model for its datatype:
    model = ipex.optimize(model)
    #torch.channels_last should be applied to both of the model object and data to raise CPU resource usage efficiency.
    model = model.to(memory_format=torch.channels_last)
    data = data.to(memory_format=torch.channels_last)

Also, please check out, https://intel.github.io/intel-extension-for-pytorch/latest/tutorials/examples.html for IPEX examples. Please check out IPEX official page https://www.intel.com/content/www/us/en/developer/tools/oneapi/extension-for-pytorch.html.


Instruction: How to enable mixed precision training while using Intel Extension for PyTorch (IPEX)?

Input: I am working on Dog-Cat classifier using Intel extension for Pytorch (Ref - https://github.com/amitrajitbose/cat-v-dog-classifier-pytorch). I want to reduce the training time for my model. How do I enable mixed precision in my code? Referred this github(https://github.com/intel/intel-extension-for-pytorch) for training my model.

Output:
Mixed precision for Intel Extension for PyTorch can be enabled using below commands,
    # For Float32
    model, optimizer = ipex.optimize(model, optimizer=optimizer)
    # For BFloat16
    model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.bfloat16)

Please check out the link, https://intel.github.io/intel-extension-for-pytorch/cpu/latest/index.html and https://www.intel.com/content/www/us/en/developer/tools/oneapi/extension-for-pytorch.html to learn more about Intel Extension for PyTorch.


Instruction: Summing over product of tensor elements and vector

Input:
I'm trying to write a python code for a higher order (d=4) factorization machine that returns the scalar result y of

Where x is  a vector of some length n, v is a vector of length n, w is an upper triangular matrix of size n by n, and t is a rank 4 tensor of size n by n by n by n. The easiest implementation is just for loops over each index:
for i in range(0,len(x)):
    for j in range(0,len(x)):
        for k in range(0,len(x)):
            for l in range(0,len(x)):
                y += t[i,j,k,l] * x[i] * x[j] * x[k] * x[l]

The first two terms are easily calculated:
y = u @ x + x @ v @ x.T

My question- is there a better way of calculating the sum over the tensor than a nested for-loop? (currently looking at possible solutions in pytorch)

Output:
This seems like a perfect fit for torch.einsum:
>>> torch.einsum('ijkl,i,j,k,l->', t, *(x,)*4)

In expanded form, this looks like torch.einsum('ijkl,i,j,k,l->', t, x, x, x, x) and computes the value defined by your four for loops:
for i, j, k, l in cartesian_prod:
    y += t[i,j,k,l] * x[i] * x[j] * x[k] * x[l]

Where cartesian_prod is the cartesian product: range(len(x))^4


Instruction: How to iterate over Dataloader until a number of samples is seen?

Input:
I'm learning pytorch, and I'm trying to implement a paper about the progressive growing of GANs. The authors train the networks on the given number of images, instead of for a given number of epochs.
My question is: is there a way to do this in pytorch, using default DataLoaders? I'd like to do something like:
loader = Dataloader(..., total=800000)
for batch in iter(loader):
   ... #do training

And the loader loops itself automatically until 800000 samples are seen.
I think that I'd be a better way, than to calculate the number of times you have to loop through the dataset by yourself

Output:
You can use torch.utils.data.RandomSampler and sample from your dataset. Here is a minimal setup example:
class DS(Dataset):
    def __len__(self):
        return 5
    def __getitem__(self, index):
        return torch.empty(1).fill_(index)

>>> ds = DS()

Initialize a random sampler providing num_samples and setting replacement to True i.e. the sampler is forced to draw instances multiple times if len(ds) < num_samples:
>>> sampler = RandomSampler(ds, replacement=True, num_samples=10)

Then plug this sampler to a new torch.utils.data.DataLoader:
>>> dl = DataLoader(ds, sampler=sampler, batch_size=2)

>>> for batch in dl:
...     print(batch)
tensor([[6.],
        [4.]])
tensor([[9.],
        [2.]])
tensor([[9.],
        [2.]])
tensor([[6.],
        [2.]])
tensor([[0.],
        [9.]])


Instruction: How to solve this Issue "ModuleNotFoundError: No module named 'torch.tensor'"

Input:
How to solve this issue
Traceback (most recent call last):
File "C:/Users/arulsuju/Desktop/OfflineSignatureVerification-master/OfflineSignatureVerification-master/main.py", line 4, in
from Preprocessing import convert_to_image_tensor, invert_image
File "C:\Users\arulsuju\Desktop\OfflineSignatureVerification-master\OfflineSignatureVerification-master\Preprocessing.py", line 4, in
from torch.tensor import Tensor
ModuleNotFoundError: No module named 'torch.tensor'
in Python I am using from torch.tensor import Tensor

Output:
In line 4, change this
 from torch.tensor import Tensor

TO
 from torch import Tensor


Instruction: torch.cuda.is_available() is False only in Jupyter Lab/Notebook

Input:
I tried to install CUDA on my computer. After doing so I checked in my Anaconda Prompt and is appeared to work out fine.

However, when I started Jupyter Lab from the same environment, torch.cuda.is_available() returns False. I managed to find and follow this solution, but the problem still persisted for me.

Does anybody have any idea why? Thank you so much!

Output:
Seems to be a problem with similar causes:
Not able to import Tensorflow in Jupyter Notebook
You are probably using other environment than the one you are using outside jupyter.
Try open Anaconda Navigator, navigate to Environments and active your env, navigate to Home and install jupyter notebook, then lunch jupyter notebook from the navigator. This should solve the issue.
In linux, you can check using !which python inside jupyter.
In windows you can use:
import sys
os.path.dirname(sys.executable)

To find where is the python that you are using.
See if the path matches.


Instruction: Indexing a multi-dimensional tensor using only one dimension

Input:
I have a PyTorch tensor b with the shape:  torch.Size([10, 10, 51]). I want to select one element between the 10 possible elements in the dimension d=1 (middle one) using a numpy array: a = np.array([0,1,2,3,4,5,6,7,8,9]). this is just a random example.
I wanted to do:
b[:,a,:] but that isn't working

Output:
Your solution is likely torch.index_select (docs)
You'll have to turn a into a tensor first, though.
a_torch = torch.from_numpy(a)
answer = torch.index_select(b, 1, a_torch)


Instruction: Retrieve only the last hidden state from lstm layer in pytorch sequential

Input:
I have a pytorch model:
model = torch.nn.Sequential(
        torch.nn.LSTM(40, 256, 3, batch_first=True),
        torch.nn.Linear(256, 256),
        torch.nn.ReLU()
)

And for the LSTM layer, I want to retrieve only the last hidden state from the batch to pass through the rest of the layers. Ex:
_, (hidden, _) = lstm(data)
hidden = hidden[-1]

Though, that example only works for a subclassed model. I need to somehow do this on a nn.Sequential() model that way when I save it, it can properly be converted to a tensorflow.js model. The reason I can't make and train this model in tensorflow.js is because I'm trying to implement this repo: Resemblyzer in tensorflow.js while still using the same weights as the pretrained Resemblyzer model which was made in pytorch as a subclassed model. I thought of using the torchvisions.transformations.Lambda() transformation but I would assume that would make it incompatible with tensorflow.js. Is there any way to make this possible while still allowing the model to convert properly?

Output:
You could split up your sequential but only doing so in the forward definition of your model on inference. Once defined:
model = nn.Sequential(nn.LSTM(40, 256, 3, batch_first=True),
                      nn.Linear(256, 256),
                      nn.ReLU())

You can split it:
>>> lstm, fc = model[0], model[1:]

Then infer in two steps:
>>> out, (hidden, _) = lstm(data)
>>> hidden = hidden[-1]
>>> out = fc(out)       # <- or fc(out[-1]) depending on what you want


Instruction: How should I replace the sentence torch.Assert from pytorch-1.7.1 to some other sentence in pytorch-1.5.0

Input: The thing is I need to reimplement a GAN model using torch1.5.0 but the previous torch1.7.1 version codes contains a torch.Assert sentence to do symbolic assert. What sentence should I use to do the same thing?

Output:
Just use python's native assert. That's what it does under the hood.
torch._assert(x == y, 'assertion message')

to be replaced with
assert x == y, 'assertion message'


Instruction: the usage of @ operator in an implementation of extending nn.Module

Input:
I saw the following code segment for extending nn.Mudule. What I do not understand is the input_ @ self.weight in forward function. I can understand that it is try to use the weight information of input_. But @ is always used as decorator, why it can be used this way?
class Linear(nn.Module):
    def __init__(self, in_size, out_size):
        super().__init__()
        self.weight = nn.Parameter(torch.randn(in_size, out_size))
        self.bias = nn.Parameter(torch.randn(out_size))

    def forward(self, input_):
        return self.bias + input_ @ self.weight

linear = Linear(5, 2)
assert isinstance(linear, nn.Module)
assert not isinstance(linear, PyroModule)

example_input = torch.randn(100, 5)
example_output = linear(example_input)
assert example_output.shape == (100, 2)

Output: The @ is a shorthand for the __matmul__ function: the matrix multiplication operator.


Instruction: Inverse operation to padding in Jax

Input:
I'm trying to learn how to use Jax and I stumbled upon the problem of converting the torch.nn.functionnal.pad function into Jax. There is a function to perform padding but I would like in the same way as in PyTorch use negative numbers in the padding (e.g F.pad(array, [-1,-1])).
Does anyone have an idea or had the same problem ?

Output:
The jax.lax.pad function accepts negative padding indices, although the API is a bit different than that of torch.nn.functional.pad. For example:
from jax import lax
import jax.numpy as jnp

x = jnp.ones((2, 3))
y = lax.pad(x, padding_config=[(0, 0, 0), (1, 1, 0)], padding_value=0.0)
print(y)
# [[0. 1. 1. 1. 0.]
#  [0. 1. 1. 1. 0.]]

x = lax.pad(y, padding_config=[(0, 0, 0), (-1, -1, 0)], padding_value=0.0)
print(x)
# [[1. 1. 1.]
#  [1. 1. 1.]]

If you wish, you could wrap this with a function that has similar semantics to the torch version. Here's a quick attempt:
def jax_pad(input, pad, mode='constant', value=0):
  """JAX implementation of torch.nn.functional.pad

  Warning: this has not been thoroughly tested!
  """
  if mode != 'constant':
    raise NotImplementedError("Only mode='constant' is implemented")
  assert len(pad) % 2 == 0
  assert len(pad) // 2 <= input.ndim
  pad = list(zip(*[iter(pad)]*2))
  pad += [(0, 0)] * (input.ndim - len(pad))
  return lax.pad(
      input,
      padding_config=[(i, j, 0) for i, j in pad[::-1]],
      padding_value=jnp.array(value, input.dtype))

x = jnp.ones((2, 3))
y = jax_pad(x, (1, 1))
print(y)
# [[0. 1. 1. 1. 0.]
#  [0. 1. 1. 1. 0.]]

x = jax_pad(y, (-1, -1))
print(x)
# [[1. 1. 1.]
#  [1. 1. 1.]]


Instruction: Is there an actual minimum input image size for popular computer vision models? (E.g., vgg, resnet, etc.)

Input:
According to the documentation on pre-trained computer vision models for transfer learning (e.g., here), input images should come in "mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224".
However, when running transfer learning experiments on 3-channel images with height and width smaller than expected (e.g., smaller than 224), the networks generally run smoothly and often get decent performances.
Hence, it seems to me that the "minimum height and width" is somehow a convention and not a critical parameter. Am I missing something here?

Output: There is a limitation on your input size which corresponds to the receptive field of the last convolution layer of your network. Intuitively, you can observe the spatial dimensionality decreasing as you progress through the network. At least this is the case for feature extractor CNNs which aim at extracting feature embeddings from the input image. That is most pre-trained models such as vanilla VGG, and ResNets networks do not retain spatial dimensionality. If the input of a convolutional layer is smaller than the kernel size (even if/when padded), then you simply won't be able to perform the operation.


Instruction: How to multiply a set of masks over an array of n matrices or tensors in python without using loops?

Input:
I have N binary masks, along with N x M matrices. I wish to apply the ith mask onto the M matrices at the ith index of the matrix array. Their data types can be either torch tensors or numpy arrays. To illustrate,
If I have an array arr:
arr = torch.rand((2, 3, 3, 3))
or
arr = tensor([[[[0.2336, 0.4841, 0.4121],
          [0.9342, 0.8496, 0.8332],
          [0.4670, 0.8158, 0.7891]],

         [[0.5791, 0.2391, 0.8501],
          [0.9811, 0.0087, 0.0655],
          [0.6587, 0.3105, 0.0931]],

         [[0.8892, 0.8104, 0.9181],
          [0.1605, 0.5280, 0.0905],
          [0.2149, 0.8851, 0.7125]]],

        [[[0.9969, 0.8589, 0.7479],
          [0.4013, 0.5922, 0.0252],
          [0.9267, 0.8123, 0.0711]],

         [[0.7931, 0.6477, 0.0947],
          [0.5969, 0.7751, 0.5662],
          [0.1785, 0.0310, 0.9135]],

         [[0.1490, 0.3623, 0.3670],
          [0.3710, 0.7887, 0.1310],
          [0.2052, 0.0244, 0.6891]]]])

and I generate a mask using:
mask = arr[:, 0, 0:, :] > 0.5
For example:
mask = tensor([[[False, False, False],
         [ True,  True,  True],
         [False,  True,  True]],

        [[ True,  True,  True],
         [False,  True, False],
         [ True,  True, False]]])

i.e. of shape (2, 3, 3). For each set of 3 x 3 x 3 matrices, I want to multiply the corresponding single mask, i.e.
result = mask[i] * arr[i, :, :, :]

or result should ideally be:
result = tensor([[[[0, 0, 0],
          [0.9342, 0.8496, 0.8332],
          [0, 0.8158, 0.7891]],

         [[0, 0, 0],
          [0.9811, 0.0087, 0.0655],
          [0, 0.3105, 0.0931]],

         [[0, 0, 0],
          [0.1605, 0.5280, 0.0905],
          [0, 0.8851, 0.7125]]],

        [[[0.9969, 0.8589, 0.7479],
          [0, 0.5922, 0],
          [0.9267, 0.8123, 0]],

         [[0.7931, 0.6477, 0.0947],
          [0, 0.7751, 0],
          [0.1785, 0.0310, 0]],

         [[0.1490, 0.3623, 0.3670],
          [0, 0.7887, 0],
          [0.2052, 0.0244, 0]]]])

I am using this in a relatively long running piece of code, hence I want to avoid using loops which might blow up the code. I have tried using np.multiply, np.dot and np.matmul, but I keep facing dimensionality issues. Can anyone help me with this?
The following code shows the operation I want to perform, removing the for loop:
for i, m in enumerate(arr):
  result[i] = mask[i] * arr[i]

Output:
By using an integer indexing, you removed one of the dimension from arr (2nd axis), and you can't broadcast an array with shape (2, 3, 3) with (2, 3, 3, 3). To make them compatible again, you can add the dimension back by reshaping:
mask.reshape((2, 1, 3, 3)) * arr

Or keep the dimension from the first place by using slice:
mask = arr[:, :1, :, :] > 0.5

and then you can just do mask * arr.


Instruction: Problem with Graph Neural Network in PyTorch Geometric

Input:
I'm trying to understand what is wrong with the following GNN model implemented in PyTorch
class Net(torch.nn.Module):
    def __init__(self):
        super().__init__()

        self.conv = SAGEConv(dataset.num_features,
                             dataset.num_classes,
                             aggr="max") # max, mean, add ...)
    def forward():
        x = self.conv(data.x, data.edge_index)
        return F.log_softmax(x, dim=1)

but I get the following error when trying to run a training loop:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-34-f3ee8050af6a> in <module>
      1 best_val_acc = test_acc = 0
      2 for epoch in range(1,100):
----> 3     train()
      4     _, val_acc, tmp_test_acc = test()
      5     if val_acc > best_val_acc:

<ipython-input-14-64df4e2a24f9> in train()
      2     model.train()
      3     optimizer.zero_grad()
----> 4     F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
      5     optimizer.step()
      6

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() takes 0 positional arguments but 1 was given

I'm adding more details as requested on how I call the model :
def train():
    model.train()
    optimizer.zero_grad()
    F.nll_loss(model()[data.train_mask], data.y[data.train_mask]).backward()
    optimizer.step()

def test():
    model.eval()
    logits, accs = model(), []
    for _, mask in data('train_mask', 'val_mask', 'test_mask'):
        pred = logits[mask].max(1)[1]
        acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()
        accs.append(acc)
    return accs

Output:
Function torch.nn.Module.forward should have at minimum one argument: self. In your case, you have two: self and your input data.
    def forward(self, data): # <-
        x = self.conv(data.x, data.edge_index)
        return F.log_softmax(x, dim=1)


Instruction: NLLLoss is just a normal negative function?

Input:
I'm having trouble understanding nn.NLLLoss().
Since the code below always prints True then what's the difference between the nn.NLLLoss() and using the negative sign (-)?
import torch
while 1:
   b = torch.randn(1)
   print(torch.nn.NLLLoss()(b, torch.tensor([0])) == -b[0])

Output:
In your case you only have a single output value per batch element and the target is 0. The nn.NLLLoss loss will pick the value of the predicted tensor corresponding to the index contained in the target tensor. Here is a more general example where you have a total of five batch elements each having three logit values:
>>> logits = torch.randn(5, 3, requires_grad=True)
>>> y = torch.tensor([1, 0, 2, 0, 1])
>>> y_hat = torch.softmax(b, -1)

Tensors y and y_hat correspond to the target tensor and estimated distributions respectively. You can implement nn.NLLLoss with the following:
>>> -y_hat[torch.arange(len(y_hat)), y]
tensor([-0.2195, -0.1015, -0.3699, -0.5203, -0.1171], grad_fn=<NegBackward>)

Compared to the built-in function:
>>> F.nll_loss(y_hat, y, reduction='none')
tensor([-0.2195, -0.1015, -0.3699, -0.5203, -0.1171], grad_fn=<NllLossBackward>)

Which is quite different to -y_hat alone.


Instruction: Is there a way to overide the backward operation on nn.Module

Input:
I am looking for a nice way of overriding the backward operation in nn.Module for example:
class LayerWithCustomGrad(nn.Module):
    def __init__(self):
        super(LayerWithCustomGrad, self).__init__()
        self.weights = nn.Parameter(torch.randn(200))

    def forward(self,x):
        return x * self.weights

    def backward(self,grad_of_c): # This gets called during loss.backward()
        # grad_of_c comes from the gradient of b*23
        grad_of_a = some_operation(grad_of_c)

        # perform extra computation
        # and more computation

        self.weights.grad = another_operation(grad_of_a,grad_of_c)
        return grad_of_a # and the grad of parameter "a" will receive this

layer = LayerWithCustomGrad()

a = nn.Parameter(torch.randn(200),requires_grad=True)
b = layer(a)
c = b*23

Some of the projects I work on contains layers with non differentiable functions, I will love it if there is some how away to connect two broken graphs and/or modify gradients of graphs that already exist.
It will also be great if there is a possible method of doing it in tensor flow

Output:
The way PyTorch is built you should first implement a custom torch.autograd.Function which will contain the forward and backward pass for your layer. Then you can create a nn.Module to wrap this function with the necessary parameters.
In this tutorial page you can see the ReLU being implemented. I will show here are to build a torch.autograd.Function and its nn.Module wrapper.
class F(torch.autograd.Function):
    """Both forward and backward are static methods."""

    @staticmethod
    def forward(ctx, input, weights):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input, weights)
        return input*weights

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the inputs: here input and weights
        """
        input, weights = ctx.saved_tensors
        grad_input = weights.clone()*grad_output
        grad_weights = input.clone()*grad_output
        return grad_input, grad_weights

The nn.Module will initialize the parameters and call F to handle the actual operation computation for forward/backward pass.
class LayerWithCustomGrad(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Parameter(torch.rand(10))
        self.fn = F.apply

    def forward(self, x):
        return self.fn(x, self.weights)

Now we can try to infer and backpropagate:
>>> layer = LayerWithCustomGrad()
>>> x = torch.randn(10, requires_grad=True)
>>> y = layer(x)
tensor([ 0.2023,  0.7176,  0.3577, -1.3573,  1.5185,  0.0632,  0.1210,  0.1566,
         0.0709, -0.4324], grad_fn=<FBackward>)

Notice the <FBackward> as grad_fn: this is the backward function of F bound to the previous inference we made with x.
>>> y.mean().backward()

>>> x.grad # i.e. grad_input in F.backward
tensor([0.0141, 0.0852, 0.0450, 0.0922, 0.0400, 0.0988, 0.0762, 0.0227, 0.0569,
        0.0309])

>>> layer.weights.grad # i.e. grad_weights in F.backward
tensor([-1.4584, -2.1187,  1.5991,  0.9764,  1.8956, -1.0993, -3.7835, -0.4926,
         0.9477, -1.2219])


Instruction: Continue training with torch.save and torch.load - key error messages

Input:
I am new to Torch and using a code template for a masked-cnn model. In order to be prepared if the training is interrupted, I have used torch.save and torch.load in my code, but I think I cannot use this alone for continuing training sessions? I start training by:
model = train_mask_net(64)

This calls the function train_mask_net where I have included torch.save in the epoch loop. I wanted to load one of the saved models and continue training with torch.load in front of the loop, but I got "key error" messages for the optimizer, loss and epoch call. Should I have made a specific checkpoint function as I have seen in some tutorials or is there a possibility that I can continue training with the files saved by the torch.saved command?
def train_mask_net(num_epochs=1):
    data = MaskDataset(list(data_mask.keys()))
    data_loader = torch.utils.data.DataLoader(data, batch_size=8, shuffle=True, num_workers=4)

    model = XceptionHourglass(max_clz+2)
    model.cuda()
    dp = torch.nn.DataParallel(model)
    loss = nn.CrossEntropyLoss()

    params = [p for p in dp.parameters() if p.requires_grad]
    optimizer = torch.optim.RMSprop(params, lr=2.5e-4,  momentum=0.9)
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=6,
                                                   gamma=0.9)

    checkpoint = torch.load('imaterialist2020-pretrain-models/maskmodel_160.model_ep17')
    #print(checkpoint)
    model.load_state_dict(checkpoint)
    #optimizer.load_state_dict(checkpoint)
    #epoch = checkpoint['epoch']
    #loss = checkpoint['loss']

    for epoch in range(num_epochs):
        print(epoch)
        total_loss = []
        prog = tqdm(data_loader, total=len(data_loader))
        for i, (imag, mask) in enumerate(prog):
            X = imag.cuda()
            y = mask.cuda()
            xx = dp(X)
            # to 1D-array
            y = y.reshape((y.size(0),-1))  # batch, flatten-img
            y = y.reshape((y.size(0) * y.size(1),))  # flatten-all
            xx = xx.reshape((xx.size(0), xx.size(1), -1))  # batch, channel, flatten-img
            xx = torch.transpose(xx, 2, 1)  # batch, flatten-img, channel
            xx = xx.reshape((xx.size(0) * xx.size(1),-1))  # flatten-all, channel

            losses = loss(xx, y)

            prog.set_description("loss:%05f"%losses)
            optimizer.zero_grad()
            losses.backward()
            optimizer.step()

            total_loss.append(losses.detach().cpu().numpy())
            torch.save(model.state_dict(), MODEL_FILE_DIR+"maskmodel_%d.model"%attr_image_size[0]+'_ep'+str(epoch)+'_tsave')

        prog, X, xx, y, losses = None, None, None, None, None,
        torch.cuda.empty_cache()
        gc.collect()
    return model

I don't think its necessary, but the xceptionhour class looks like this:
class XceptionHourglass(nn.Module):
    def __init__(self, num_classes):
        super(XceptionHourglass, self).__init__()
        self.num_classes = num_classes

        self.conv1 = nn.Conv2d(3, 128, 3, 2, 1, bias=True)
        self.bn1 = nn.BatchNorm2d(128)
        self.mish = Mish()

        self.conv2 = nn.Conv2d(128, 256, 3, 1, 1, bias=True)
        self.bn2 = nn.BatchNorm2d(256)

        self.block1 = HourglassNet(4, 256)
        self.bn3 = nn.BatchNorm2d(256)
        self.block2 = HourglassNet(4, 256)
...

Output:
torch.save(model.state_dict(), PATH) only saves the model weights.
To also save optimizer, loss, epoch, etc., change it to:
torch.save({'model': model.state_dict(),
            'optimizer': optimizer.state_dict(),
            'loss': loss,
            'epoch': epoch,
            # ...
            }, PATH)

To load them:
checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

More on it here.


Instruction: RuntimeError: expected scalar type Float but found Double

Input:
My code is as follows:
net = nn.Linear(54, 7)
optimizer = optim.SGD(net.parameters(), lr=lr, momentum=0)
logloss = torch.nn.CrossEntropyLoss()
for i in range(niter):
    optimizer.zero_grad()
    y_2 = torch.from_numpy(np.array(y, dtype='float64'))
    X_2 = torch.from_numpy(np.array(X, dtype='float64'))
    outputs = net(X_2)
    print(loss)
    loss.backward()
    optimizer.step()

And I got the following error message
---> 57             outputs = net(X_2)
     58             print(np.shape(outputs))
     59             loss = logloss(outputs, y_2)

~\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     94
     95     def forward(self, input: Tensor) -> Tensor:
---> 96         return F.linear(input, self.weight, self.bias)
     97
     98     def extra_repr(self) -> str:

~\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1845     if has_torch_function_variadic(input, weight):
   1846         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1847     return torch._C._nn.linear(input, weight, bias)
   1848
   1849

RuntimeError: expected scalar type Float but found Double

Can you specify what is my problems, thank you. I except I have transform the the results into float through torch.from_numpy(np.array(y, dtype='float64')), but do not work.

Output: You need to cast your tensors to float32, either with dtype='float32' or calling float() on your input tensors.


Instruction: StopIteration Error occurs during training while running the train.py file

Input:
I am trying to run a code from github. The file is called train.py. It is supposed to run a Neural Network for training on a dataset. However, I get the following error
(QGN) ubuntu@ip-172-31-13-114:~/QGN$ python train.py
Input arguments:
id               ade20k
arch_encoder     resnet50
arch_decoder     QGN_dense_resnet34
weights_encoder
weights_decoder
fc_dim           2048
list_train       ./data/train_ade20k.odgt
list_val         ./data/validation_ade20k.odgt
root_dataset     ./data/
num_gpus         0
batch_size_per_gpu 2
num_epoch        20
start_epoch      1
epoch_iters      5000
optim            SGD
lr_encoder       0.02
lr_decoder       0.02
lr_pow           0.9
beta1            0.9
weight_decay     0.0001
deep_sup_scale   1.0
prop_weight      2.0
enhance_weight   2.0
fix_bn           0
num_val          500
num_class        150
transform_dict   None
workers          40
imgSize          [300, 375, 450, 525, 600]
imgMaxSize       1000
cropSize         0
padding_constant 32
random_flip      True
seed             1337
ckpt             ./ckpt
disp_iter        20
visualize        False
result           ./result
gpu_id           0
Model ID: ade20k-resnet50-QGN_dense_resnet34-batchSize0-LR_encoder0.02-LR_decoder0.02-epoch20-lossScale1.0-classScale2.0
# samples: 20210
1 Epoch = 5000 iters
Starting Training!
Traceback (most recent call last):
  File "train.py", line 355, in <module>
    main(args)
  File "train.py", line 217, in main
    train(segmentation_module, iterator_train, optimizers, history, epoch, args)
  File "train.py", line 33, in train
    batch_data = next(iterator)
  File "/home/ubuntu/QGN/lib/utils/data/dataloader.py", line 274, in __next__
    raise StopIteration
StopIteration
Segmentation fault (core dumped)

The code from train.py (lines 211 to 231) is as follows
'''
Main loop
history = {'train': {'epoch': [], 'loss': [], 'acc': []}}

print('Starting Training!')

for epoch in range(args.start_epoch, args.num_epoch + 1):
    train(segmentation_module, iterator_train, optimizers, history, epoch, args)

    # checkpointing
    checkpoint(nets, history, args, epoch)

    # evaluation
    args.weights_encoder = os.path.join(args.ckpt, 'encoder_epoch_' + str(epoch) + '.pth')
    args.weights_decoder = os.path.join(args.ckpt, 'decoder_epoch_' + str(epoch) + '.pth')
    iou = eval_train(args)

    # adaptive class weighting
    adjust_crit_weights(segmentation_module, iou, args)

print('Training Done!')

'''
I am not sure if I have shared all the required information. I would appreciate if ant help could be provided to resolve this issue. Just to inform, I have tried using the try and except method as  shared on github on the link https://github.com/amdegroot/ssd.pytorch/issues/214. However the error still persists.
The code from line 30 in train.py is as follows
   # main loop
    tic = time.time()
    for i in range(args.epoch_iters):
        batch_data = next(iterator)
        data_time.update(time.time() - tic)

        segmentation_module.zero_grad()

I ammended the above code as follows
   # main loop
     loader_train = torchdata.DataLoader(
        dataset_train,
        batch_size=args.num_gpus,  # we have modified data_parallel
        shuffle=False,  # we do not use this param
        collate_fn=user_scattered_collate,num_workers=int(args.workers),
        drop_last=True,
        pin_memory=True)

    tic = time.time()
    for i in range(args.epoch_iters):
        try:
            batch_data = next(iterator)
        except StopIteration:
            iterator = iter(loader_train)
            batch_data = next(iterator)
        data_time.update(time.time() - tic)

        segmentation_module.zero_grad()

But still no joy. The error still remains.

Output:
TL;DR
Your args.epoch_iters is larger than the number of batches in loader_train. Python raises StopIteration error when you ask for more batches than there actually are.
When you iterate over some pythonic collection of elements (e.g., list, tuple, DataLoader...) python needs to know when it reaches the end of that collection. It is done by raising StopIteration exception. for loop in python explicitly listens to this exception and uses it to know when to stop. Alas, in your code you do not use a for loop over loader_train, but rather over range(args.epoch_iter) and you use next(iterator) to get the batches.


Instruction: When to put pytorch tensor on GPU?

Input:
I'm experimenting with running neural network on GPU using pytorch, and my data have some unusual shape so I use Dataset and DataLoader to generate data batch. My code runs fine on CPU but I'm a little confused on when is the right timing to put the data on GPU:

My data size is small enough to be put all together on GPU, should I put all data on GPU before fitting, so that all DataLoader and Dataset operations only take place on GPU in order to get optimal execution speed?
Another possibility is to leave all data on CPU which could be useful when the data size become larger. In that case, should I call batch.to("cuda") for each batch generated from DataLoader?
Should I also put the model on GPU first before training? It is a small enough model to be put on GPU.
My raw data are numpy array, hence I have the freedom to write Dataset that returns numpy array in __getitem()___ method, or convert the numpy array to pytorch tensor and write Dataset that returns pytorch tensor. Is one method preferred over the other?

Output:
Let me clear up a thing. at the time of passing the data through the model, both your model and the data(that specific batch) have to be on the same device.
To automate your code to work on both GPU and non-GPU environments you might use this line.
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

So if You are looking to using GPU for training you must put the model on GPU before training. personally, I prefer loading the model to GPU whenever I am creating the model object. [Answer 3]
model = MyModel().to(device)

Then you also need to put your data in GPU. One obvious option is putting all data at once. But I would suggest against that. Because however small your dataset is, You will always get better performance by putting one batch at a time in GPU, rather than whole data at once. I know you are thinking, there is a time delay for taking each batch from CPU to GPU. You are right!
But putting one batch at a time would allow you a larger batch size, and a larger batch size will always win in terms of performance, compared to smaller batch sizes and loading all data at a time.[Answer 1 & 2]
for x_data, y_data in train_dataloader:
    x_data, y_data = x_data.to(device), y_data.to(device)

Finally, about writing the __getitem__,  the Numpy array to PyTorch tensor will be handled by the data loader so your getitem can return Numpy arrays. But I feel good when I see the conversion written explicitly in my code. It gives me the sense of a complete and easy-to-understand pipeline.[Answer 4]


Instruction: No CUDA GPUs are available

Input:
i get this error from the method during the model training process. i am using the google colab to run the code. the google colab dont have any GPU. Is their any other way i can make the code run without requiring cuda cpu.
How can i fix this error ?
def train_model(model, train_loader, val_loader, epoch, loss_function, optimizer, path, early_stop):

    # GPU
   #device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
#     device = torch.device("cpu")
    device = torch.device("cuda")
    model = model.to(device)

    patience, eval_loss = 0, 0

    # train
    for i in range(epoch):
        total_loss, count = 0, 0
        y_pred = list()
        y_true = list()
        for idx, (x, y) in tqdm(enumerate(train_loader), total=len(train_loader)):
            x, y = x.to(device), y.to(device)
            u, m = model(x)
            predict = torch.sigmoid(torch.sum(u*m, 1))
            y_pred.extend(predict.cpu().detach().numpy())
            y_true.extend(y.cpu().detach().numpy())
            loss = loss_function(predict, y.float())
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            total_loss += float(loss)
            count += 1

        train_auc = roc_auc_score(np.array(y_true), np.array(y_pred))
        torch.save(model, path.format(i+1))
        print("Epoch %d train loss is %.3f and train auc is %.3f" % (i+1, total_loss / count, train_auc))

        # verify
        total_eval_loss = 0
        model.eval()
        count_eval = 0
        val_y_pred = list()
        val_true = list()
        for idx, (x, y) in tqdm(enumerate(val_loader), total=len(val_loader)):
            x, y = x.to(device), y.to(device)
            u, m = model(x)
            predict = torch.sigmoid(torch.sum(u*m, 1))
            val_y_pred.extend(predict.cpu().detach().numpy())
            val_true.extend(y.cpu().detach().numpy())
            loss = loss_function(predict, y.float())
            total_eval_loss += float(loss)
            count_eval += 1
        val_auc = roc_auc_score(np.array(y_true), np.array(y_pred))
        print("Epoch %d val loss is %.3fand train auc is %.3f" % (i+1, total_eval_loss / count_eval, val_auc))

Output: Just remove the line where you create your torch.device() and remove all the .to(device) functions where you use it. Then you also don't need to write .cpu().detach() also. You can simply write predict.numpy() as such. When you write device = torch.device("cuda") you are creating a GPU device and you are then transferring your model and data into the GPU device when its not available. This is the reason for the error.


Instruction: What does a single "||" mean in pytorch-geometric documents?

Input: For example the "||" (\Vert) in https://pytorch-geometric.readthedocs.io/en/latest/modules/nn.html#torch_geometric.nn.conv.GATConv

Output:
That documentation page includes a link to an arxiv paper that includes the following (bottom of page three)...

where  represents transposition and || is the concatenation operation.

So yes, || is the concatenation operator.


Instruction: Get the value at a specific index in PyTorch

Input:
I have a ground truth label array for size 5.
y=tensor([958,  85, 244, 182, 294])

I have the output for scores array  of shape :  [5,1000]
scores   =  tensor([[ 1.0406,  1.1808,  4.4227,  ...,  4.6864,  8.0145,  5.2128],
        [ 6.9101,  4.6083,  6.9259,  ...,  9.7415,  9.6305,  9.3974],
        [ 7.6097,  4.0396,  4.4560,  ...,  3.4892, 11.6411, 2],
        [ 1.0693,  4.6295,  5.3638,  ..., 10.9041, 10.8380,  9.2077],
        [ 1.7085,  1.4938,  8.6876,  ..., 15.1423,  9.6055,  9.8920]],
       grad_fn=<ViewBackward>)

I want the value from scores array based on the corresponding index of y. So for y[0], which is 958, I want the corresponding value from scores[1], position 958.
Is there some direct Pytorch function I can use?

Output:
Yes, you can do it by using your y array as an index:
scores[torch.arange(5), y]


Instruction: torchaudio: Error opening '_sample_data\\steam.mp3': File contains data in an unknown format

Input:
I'm new to torch audio and i'm following the this tutorial step by step. I'm having a problem loading an mp3 audio using torchaudio.info(path).
Here is my code:
metadata = torchaudio.info(SAMPLE_MP3_PATH)
print(metadata)

Here is the error that i'm getting:
..

RuntimeError: Error opening '_sample_data\\steam.mp3': File contains data in an unknown format.

torch: v1.9.1+cpu
torchaudio: v0.9.1

Output:
torchaudio.info will call its backend to show the information.
If you use it in windows, and the backend is Soundfile, then this problem will occur. Because Soundfile does not support mp3 format.
You can use the below code to see the available formats.
import soundfile as sf
sf.available_formats()


Instruction: Distributing CUDA runtime to customers but it's too big

Input:
At my company, we are building software that we need to push to customers when we update software (It's being pushed to custom hardware).
We have a GPU on that custom hardware that is fixed, but sometimes, we might need to upgrade the CUDA and CUDNN runtime if we upgrade things in our software (such as libtorch).
The problem now is that because of this, we have to ship CUDA and CUDNN together, which bloats the size of the binaries to over 2GB.
While the actual size of our executable is only 100MB. Is there any smart way around this?

Output:
https://pytorch.org doesn't advertize it, but there is a static version of libtorch available (replace 'shared' with 'static' in the URL).
Link against those libraries instead. Your binary will be a bit bigger (depending on how much of the library your code is using), but on the plus side you'll be saving 1.2GB there, because you don't have to ship the libraries.
CUDA and cuDNN should also have static versions available, although they might be missing in some re-distributions (like in Anaconda).


Instruction: What is the difference edge_weight and edge_attr in Pytorch Geometric

Input:
I want to handle weighted undirected graphs in Pytorch Geometric.
The node features are 50 dimensional. I found that this can be handled by the x attribute of the torch_geometric.data.data class.
The weights of the edges are scalar values.
We found out that edge_attr and edge_weight are the attributes to handle edges.
I think I should probably use edge_weight, is this correct?
Also, what is the difference between edge_attr and edge_weight?
I'm not very good at English, so I apologize for that.
I hope I can get a good answer.
Thank you.

Output: The difference between edge_weight and edge_attr is that edge_weight is always one-dimensional (one value per edge) and that edge_attribute can be multi-dimensional. You can check the cheatsheet for the support of the models.


Instruction: Number of neuron in CNN architecture

Input:
I am using a certain CNN architecture, however, I am not sure how to calculate the exact number of neuron I have in it.
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=16,
                               kernel_size=(7, 7), padding=(1, 1),
                               stride=(2, 2))

        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32,
                               kernel_size=(7, 7), padding=(1, 1),
                               stride=(2, 2))

        self.conv3 = nn.Conv2d(in_channels=32, out_channels=64,
                               kernel_size=(3, 3), padding=(2, 2),
                               stride=(2, 2))

        self.fc1 = nn.Linear(64 * 1 * 1, 8)

There is also 2D-maxpooling after each convolution layer with stride of 2.
I can get the number of parameters and Gmacs in my network, but I am not sure how to get the number of neurons?
Is there a certain way to calculate them?
Thanks.

Output:
One quick way to get the total count is to

Fetch all parameters with nn.Module.parameters;
Convert the generator to a flattened tensor with torch.nn.utils.parameters_to_vector;
Find the total number of elements with torch.Tensor.numel.

Which corresponds to:
>>> p2v(model.parameters()).numel()
44936

Having imported parameters_to_vector from torch.nn.utils as p2v

If you want to count the parameters yourself:

Convolutions when counting kernels and biases. Given number of input channels in_c, output channels out_c, and kernel size k:
conv = lambda in_c, out_c, k: k*k*in_c*out_c + out_c

Fully-connected layers: just a two-dimensional matrix with biases:
fc = lambda in_c, out_c: in_c*out_c + out_c

Max-pool layers are non-parametrized layers: 0 parameters.

All in all, this gives you:
>>> conv(1, 16, 7) + conv(16, 32, 7) + conv(32, 64, 3) + fc(64, 8)
44936

The word neurons is just an abstraction. If you consider it to be the output dimension for each given layer then:

For convolution layers, it will depend on the spatial dimension of the input. So given the spatial dimension x, the kernel size k, the padding p, and the stride s:
conv = lambda x, k, p, s: math.floor((x+2*p - k)/ s + 1)

For fully connected layers, it corresponds to the number of output features.


Instruction: How to Convert a Linear Model to Conv1 in Pytorch?

Input:
I am new to Pytroch and I cannot transform Keras models that I have in my mind in to it.
I have a really simple linear model in Pytorch as follows:
class linear_model(nn.Module):
    def __init__(self, output, activation=nn.ReLU):
        super(linear_model, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 26),
            activation(),
            nn.Linear(26, 6)
        )

    def forward(self, x):
        out = self.net(x)
        return out

I want to use Conv1d instead of a linear layer. Something like this:
class resnet(nn.Module):
    def __init__(self, output, activation=nn.ReLU):
        super(resnet, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(26, 64),
            activation(),
            nn.Conv1d(64, 64, kernel_size=5,padding=2),
            activation(),
            nn.MaxPool1d(3,stride=1, padding=1),
            nn.Conv1d(64, 32, 3,padding=1,stride=2),
            activation(),
            nn.MaxPool1d(3,stride=1, padding=1),
            nn.Linear(32, 6)
        )

    def forward(self, x):
        out = self.net(x)
        return out

The first Conv1d needs an [64,64,6] input and I provide it an [10,64] input (the batch size is 10).
To summarize, I have an array [1,26] and I want to feed into the network, increase its size to [1,64], then reduce it to [1,6] at the end.
I'm conflicting with the concepts of Keras and Pytorch. How should I correct the problem?
Edit:
I have changed the model, and now it's working. After all, I have no clue what's going on there. Here is the new one:
class custom_model(nn.Module):
def __init__(self, output, activation=nn.ReLU):
    super(custom_model, self).__init__()
    self.net = nn.Sequential(
        nn.Linear(num_features, 64),
        nn.Dropout(p=0.5),
        nn.ReLU(),
        nn.Conv1d(1, 64, kernel_size=5,padding=2),
        activation(),
        nn.MaxPool1d(3,stride=1, padding=1),
        nn.Conv1d(64, 32, 3,padding=1,stride=2),
        activation(),
        nn.MaxPool1d(3,stride=1, padding=1),
        nn.Conv1d(32, 6, 1),
        activation(),
        #nn.Conv1d(6, 6, 1)
        nn.Linear(32, output),
        nn.Dropout(p=0.5),
        nn.ReLU(),
        nn.Linear(6, 1),

    )

def forward(self, x):
    x = torch.unsqueeze(x,1)
    out = self.net(x)

    return out

And, changed the way I calculate the loss to make it compatible:
outputs = model(inputs)
outputs_squeezed = torch.squeeze(outputs)
loss_value = loss(outputs_squeezed, y_train)

The model I devised is not necessary correct (I mean, the concept behind it). I just cannot figure out what is happening. To make it clear, I cannot understand what are the input and output of the Conv1d and how I can connect them correctly.

Output:
The data you are passing is that of single-point data. The difference between data of shape [B,N,L] for 1D tensors and [B,N] for single-point tensors is critical for the application of N-D convolutions (in this case B is batch size, L is sequence length, and N is feature depth).
To solve this for your case, just add a dimension to your data so that N=1, because at the moment you do not have a third dimension. However, according to your question I am unsure if you're missing the batch (first) dimension or the feature (second) because you say you provide both, but not simultaneously? Nevertheless, I think this can be entirely amended by:
input = input[:,None,:]

Or
input = torch.unsqueeze(input,1)


Instruction: (with cpu)Pytorch: IndexError: index out of range in self. (with cuda)Assertion `srcIndex < srcSelectDimSize` failed. How to solve?

Input:
Today I get the following error when I use BERT with Pytorch and cuda: /pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion srcIndex < srcSelectDimSize failed.
Epoch [1/100]
Iter:      0,  Train Loss:   1.1,  Train Acc: 39.06%,  Val Loss:   1.0,  Val Acc: 51.90%,  Time: 0:00:04 *
Iter:     10,  Train Loss:  0.99,  Train Acc: 57.81%,  Val Loss:   1.0,  Val Acc: 52.01%,  Time: 0:00:11 *
Iter:     20,  Train Loss:   1.0,  Train Acc: 42.19%,  Val Loss:  0.99,  Val Acc: 52.01%,  Time: 0:00:17 *
Iter:     30,  Train Loss:   1.0,  Train Acc: 40.62%,  Val Loss:  0.99,  Val Acc: 52.12%,  Time: 0:00:23 *
Iter:     40,  Train Loss:   1.0,  Train Acc: 50.00%,  Val Loss:  0.98,  Val Acc: 52.12%,  Time: 0:00:29 *
Iter:     50,  Train Loss:   1.1,  Train Acc: 43.75%,  Val Loss:  0.98,  Val Acc: 52.12%,  Time: 0:00:35 *
Traceback (most recent call last):
  File "/content/drive/MyDrive/Prediction/run.py", line 38, in <module>
    train(config, model, train_iter, dev_iter, test_iter)
  File "/content/drive/MyDrive/Prediction/train_eval.py", line 50, in train
    outputs = model(trains)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/drive/MyDrive/Prediction/models/BERT+Covid.py", line 68, in forward
    output  = self.bert(context, attention_mask=mask)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 1005, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 589, in forward
    output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 475, in forward
    past_key_value=self_attn_past_key_value,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 408, in forward
    output_attentions,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 323, in forward
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [234,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

 #......I SKIPPED SEVERAL LINES DUE TO THE CHARACTER LIMITATION

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:702: indexSelectLargeIndex: block: [235,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed

to find where exactly went wrong, I run my code again with CPU, and I got this error: IndexError: index out of range in self.
traceback (most recent call last):
  File "/content/drive/MyDrive/Prediction/run.py", line 37, in <module>
    train(config, model, train_iter, dev_iter, test_iter)
  File "/content/drive/MyDrive/Prediction/train_eval.py", line 49, in train
    outputs = model(trains)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/drive/MyDrive/Prediction/models/BERT+Covid.py", line 66, in forward
    output  = self.bert(context, attention_mask=mask, )
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 993, in forward
    past_key_values_length=past_key_values_length,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py", line 215, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2043, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

According to the guidance I found online, I have ensured the following issue:

the input length didn't exceed the maximum length in the model (the pad size I set is 98, and I had tried to print out the shape of the input before the line went wrong. it did be (batch_size, pad_size)).

len(tokenizer)==model.config.vocab_size, so this is not the problem

I have no idea now what could be the problem, could anybody help me?
My model structure is:
class Model(nn.Module):

    def __init__(self, config):
        super(Model, self).__init__()
        self.modelConfig = BertConfig.from_pretrained('./bert_pretrain/config.json')
        self.bert = BertModel.from_pretrained(config.bert_path,config=self.modelConfig)
        for param in self.bert.parameters():
            param.requires_grad = False
        self.cls_fc_layer = FCLayer(config.hidden_size, config.word_size, config.dropout_rate)
        self.label_classifier = FCLayer(
            config.word_size+config.numerical_size,
            config.num_classes,
            config.dropout_rate,
            use_activation=False,
        )

    def forward(self, x):
        context = x[0]  # input token ids
        mask = x[2]  # mask
        numerical=x[3] #size(batch_size,18)

        output  = self.bert(context, attention_mask=mask)
        pooled_output=output[1]
        ##size(batch_size,768)
        pooled_output = self.cls_fc_layer(pooled_output)
        ##size(batch_size,18)
        concat_h = torch.cat([pooled_output, numerical], dim=-1)
        ##size(batch_size,36)
        logits = self.label_classifier(concat_h)
        return logits

Output:
I have resolved it!!!
By printing out the  maximum input_ids each batch
for i, (trains, labels) in enumerate(train_iter):
            print("train max input:", torch.max(trains[0]))
            print("train min input:", torch.min(trains[0]))
            print("train max label:", torch.max(labels))
            print("train min label:", torch.min(labels))

I got the following output, the max input_id == 21128, while the length of my tokenizer == 21128, which means the maximum input_id should be 21127, this is where the index is out of range!
train max input: tensor(21128, device='cuda:0')
train min input: tensor(0, device='cuda:0')
train max label: tensor(2, device='cuda:0')
train min label: tensor(0, device='cuda:0')

The reason why this error occurs might be because I manually changed the vocab.txt file of the bert model (Sorry I was new to that...), and I resolved this problem by reloading the original BERT model/vocab and config


Instruction: Pytorch required_grad=False does not freeze network parameters when running on GPU

Input:
I'm trying to freeze a layer of a toy model when training using Pytorch. In the following code, when I run the code on CPU, the layer isn't updated. (Please see the code line print("%.8f" % np.max(np.abs(before -after)))). However, when I run the code on GPU, the layer is updated. What is wrong with my implementation?
import torch
from torch import nn
from torch.autograd import Variable
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

def toNP(x):
    return x.detach().to('cpu').numpy()

# toy feed-forward net
class Sub_Net(nn.Module):
    def __init__(self):
        super(Sub_Net, self).__init__()

        self.fc1 = nn.Linear(10, 3)
        self.fc2 = nn.Linear(3, 3)
        self.fc3 = nn.Linear(3, 3)
        self.fc4 = nn.Linear(3, 3)
        self.fc5 = nn.Linear(3, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.fc4(x)
        x = self.fc5(x)
        return x

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.Sub_Net = Sub_Net()

        self.fc1 = nn.Linear(10, 3)
        self.fc2 = nn.Linear(3, 3)
        self.fc3 = nn.Linear(3, 3)
        self.fc4 = nn.Linear(3, 3)
        self.fc5 = nn.Linear(3, 1)

    def forward(self, x):

        y = self.Sub_Net(x)

        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.fc4(x)
        x = self.fc5(x)
        return x+y

def generator_step(net, optimizer, criterion, input, target):

    output = net(input)
    loss = criterion(output, target)
    net.zero_grad()
    loss.backward()
    optimizer.step()

def discrimination_step(net, optimizer, criterion, input, target):

    for param in net.Sub_Net.parameters():
        param.requires_grad = False

    before = toNP(net.Sub_Net.fc2.weight)

    output = net(input)
    loss = criterion(output, target)
    net.zero_grad()
    loss.backward()
    optimizer.step()

    after = toNP(net.Sub_Net.fc2.weight)

    print("%.8f" % np.max(np.abs(before -after)) )

# Run model on GPU
# net = Net().type(torch.cuda.FloatTensor)
# random_input = Variable(torch.randn(10, )).cuda()
# random_target = Variable(torch.randn(1, )).cuda()

# Run model on CPU
net = Net()
random_input = Variable(torch.randn(10, ))
random_target = Variable(torch.randn(1, ))

# loss
criterion = nn.MSELoss()
optimizer = optim.Adam(filter(lambda p: p.requires_grad, net.parameters()), lr=0.1)

for epoch in range(1, 10):

    generator_step(net, optimizer, criterion, random_input, random_target)
    discrimination_step(net, optimizer, criterion, random_input, random_target)

Results when running on CPU
0.00000000
0.00000000
0.00000000
0.00000000
0.00000000
0.00000000
0.00000000
0.00000000
0.00000000

Results when running on GPU
0.06700575
0.04242781
0.03090768
0.02379489
0.01885229
0.01519108
0.01237211
0.01014686
0.00836059

Output:
It seems it's not CPU / GPU acting differently, but it is about .to('cpu') function in your toNP function.
If given tensor is on GPU, it returns copied tensor on cpu, while it returns given original object when the given tensor is already on CPU.
Please refer more on this site.
To clarify, I've added print function to your discriminator_step as follows:
def discrimination_step(net, optimizer, criterion, input, target):

    for param in net.Sub_Net.parameters():
        param.requires_grad = False

    before = toNP(net.Sub_Net.fc2.weight)
    print(f'Before:\n{before}')

    output = net(input)
    loss = criterion(output, target)
    net.zero_grad()
    loss.backward()
    optimizer.step()

    print(f'Before:\n{before}')
    after = toNP(net.Sub_Net.fc2.weight)
    print(f'After:\n{after}')

    print("dff: %.8f" % np.max(np.abs(before -after)) )

Then the code yields results when on CPU (for 1 epoch):
Before:
[[-0.0222426   0.06449176  0.41833472]
 [-0.3276776  -0.22486973  0.38021228]
 [-0.37726757  0.26268137 -0.05000275]]
Before:
[[ 0.04476321  0.13149747  0.48534054]
 [-0.2606718  -0.15786391  0.44721812]
 [-0.31026173  0.32968715  0.01700307]]
After:
[[ 0.04476321  0.13149747  0.48534054]
 [-0.2606718  -0.15786391  0.44721812]
 [-0.31026173  0.32968715  0.01700307]]
dff: 0.00000000

and on GPU:
Before:
[[-0.06808002  0.39740798  0.55723506]
 [-0.17421165 -0.36702433 -0.4208245 ]
 [-0.37865937 -0.52346057 -0.15856335]]
Before:
[[-0.06808002  0.39740798  0.55723506]
 [-0.17421165 -0.36702433 -0.4208245 ]
 [-0.37865937 -0.52346057 -0.15856335]]
After:
[[-0.13508584  0.4644138   0.6242409 ]
 [-0.24121748 -0.30001852 -0.35381868]
 [-0.31165355 -0.5904664  -0.22556916]]
dff: 0.06700583

It shows before values are changed since the returned before tensor shares the same storage with net.Sub_Net.fc2.weight when on CPU.
Layers are updated regardless of CPU and GPU since they are already in parameter groups of the Adam optimizer.


Instruction: Exporting PyTorch Lightning model to ONNX format not working

Input:
am using Jupyter Lab to run. It has pre-installed tf2.3_py3.6 kernel installed in it. It has 2 GPUS in it.
PyTorch Lightning Version (e.g., 1.3.0): '1.4.6'
PyTorch Version (e.g., 1.8): '1.6.0+cu101'
Python version: 3.6
OS (e.g., Linux): system='Linux'
CUDA/cuDNN version: 11.2
How you installed PyTorch (conda, pip, source): pip

I am saving the best model in checkpoint.
I am doing multi-label classification using Hugging face model. After training the model I want to export the model using ONNX format. The input is attention mask, input ids.
Here is the DataModule Class
N_EPOCHS = 30
BATCH_SIZE = 10

class  SRDataModule(pl.LightningDataModule):

    def __init__(self, X_train,y_train, X_test,y_test, tokenizer, batch_size=8, max_token_len=512):
        super().__init__()
        self.batch_size = batch_size
        self.train_df = X_train
        self.test_df = X_test
        self.train_lab = y_train
        self.test_lab = y_test
        self.tokenizer = tokenizer
        self.max_token_len = max_token_len

    def setup(self, stage=None):
        self.train_dataset = SRDataset(
          self.train_df,
          self.train_lab,
          self.tokenizer,
          self.max_token_len
        )

        self.test_dataset = SRDataset(
          self.test_df,
          self.test_lab,
          self.tokenizer,
          self.max_token_len
    )

    def train_dataloader(self):
        return DataLoader(
          self.train_dataset,
          batch_size=self.batch_size,
          shuffle=True,
          num_workers=10
        )

    def val_dataloader(self):
        return DataLoader(
          self.test_dataset,
          batch_size=self.batch_size,
          num_workers=10
        )

    def test_dataloader(self):
        return DataLoader(
          self.test_dataset,
          batch_size=self.batch_size,
          num_workers=10
        )

Here is the model class:
class SRTagger(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.save_hyperparameters()
    self.bert = BertModel.from_pretrained(BERT_MODEL_NAME, return_dict=True)
    self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)
    self.n_training_steps = n_training_steps
    self.n_warmup_steps = n_warmup_steps
    self.criterion = nn.BCELoss()

  def forward(self, input_ids, attention_mask, labels=None):
    output = self.bert(input_ids, attention_mask=attention_mask)
    output = self.classifier(output.pooler_output)
    output = torch.sigmoid(output)
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output

  def training_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)

    self.log("train_loss", loss, prog_bar=True, logger=True)
    return {"loss": loss, "predictions": outputs, "labels": labels}

  def validation_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("val_loss", loss, prog_bar=True, logger=True)
    return loss

  def test_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["labels"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("test_loss", loss, prog_bar=True, logger=True)
    return loss

  def training_epoch_end(self, outputs):

    labels = []
    predictions = []
    for output in outputs:
      for out_labels in output["labels"].detach().cpu():
        labels.append(out_labels)
      for out_predictions in output["predictions"].detach().cpu():
        predictions.append(out_predictions)

    labels = torch.stack(labels).int()
    predictions = torch.stack(predictions)

    for i, name in enumerate(LABEL_COLUMNS):
      class_roc_auc = auroc(predictions[:, i], labels[:, i])
      self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)

  def configure_optimizers(self):

    optimizer = optim.RAdam(self.parameters(), lr=2e-4)

    scheduler = get_linear_schedule_with_warmup(
      optimizer,
      num_warmup_steps=self.n_warmup_steps,
      num_training_steps=self.n_training_steps
    )

    return dict(
      optimizer=optimizer,
      lr_scheduler=dict(
        scheduler=scheduler,
        interval='step'
      )
    )

Sample Data
sample_batch = next(iter(DataLoader(train_dataset, batch_size=10, num_workers=2)))
sample_batch["input_ids"].shape, sample_batch["attention_mask"].shape

(torch.Size([10, 512]), torch.Size([10, 512]))
sample_batch.keys()
dict_keys(['text_data', 'input_ids', 'attention_mask', 'labels'])

Model
model = SRTagger(
  n_classes=100,
  n_warmup_steps=warmup_steps,
  n_training_steps=total_training_steps
)

ONNX code
# # Export the model
torch.onnx.export(model,                     # model being run
                  ##since model is in the cuda mode, input also need to be
                  (sample_batch["input_ids"],sample_batch["attention_mask"]),              # model input (or a tuple for multiple inputs)
                  "model_torch_export.onnx", # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['output'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})

Error
RuntimeError: output 1 (0
[ CPULongType{} ]) of traced region did not have observable data dependence with trace inputs; this probably indicates your program cannot be understood by the tracer.

Output: Cuz at onnx conversion, the model output must be tensor, not list, tuple, or dict. You can check your forward of model.


Instruction: Challenge in replacing SelfAttention with ImageLinearAttention in Vision Transformer

Input:
When I am replacing ImageLinearAttention with SelfAttention in Vision Transformer, with the code as follows, I get a RuntimeError. The code for ImageLinearAttention is from https://github.com/lucidrains/linear-attention-transformer/blob/master/linear_attention_transformer/images.py except I removed number of channels as you see in commented code.
class ImageLinearAttention(nn.Module):
    def __init__(self, chan, chan_out = None, kernel_size = 1, padding = 0, stride = 1, key_dim = 64, value_dim = 64, heads = 8, norm_queries = True):
        super().__init__()
        self.chan = chan
        chan_out = chan if chan_out is None else chan_out

        self.key_dim = key_dim
        self.value_dim = value_dim
        self.heads = heads

        self.norm_queries = norm_queries

        conv_kwargs = {'padding': padding, 'stride': stride}
        self.to_q = nn.Conv2d(chan, key_dim * heads, kernel_size, **conv_kwargs)
        self.to_k = nn.Conv2d(chan, key_dim * heads, kernel_size, **conv_kwargs)
        self.to_v = nn.Conv2d(chan, value_dim * heads, kernel_size, **conv_kwargs)
        print('value dim: ', value_dim)
        print('chan out: ', chan_out)
        print('kernel_size: ', kernel_size)
        out_conv_kwargs = {'padding': padding}
        print('out_conv_kwargs: ', out_conv_kwargs)
        print('in_chan: ', value_dim * heads)
        self.to_out = nn.Conv2d(value_dim * heads, chan_out, kernel_size, **out_conv_kwargs)

    def forward(self, x, context = None):
        print('x.shape: ', x.shape)
        print('*x.shape is: ', *x.shape)
        print('heads: ', self.heads)
        #b, c, h, w, k_dim, heads = *x.shape, self.key_dim, self.heads
        b, h, w, k_dim, heads = *x.shape, self.key_dim, self.heads
        q, k, v = (self.to_q(x), self.to_k(x), self.to_v(x))
        q, k, v = map(lambda t: t.reshape(b, heads, -1, h * w), (q, k, v))
        q, k = map(lambda x: x * (self.key_dim ** -0.25), (q, k))

        if context is not None:
            #context = context.reshape(b, c, 1, -1)
            context = context.reshape(b, 1, -1)
            ck, cv = self.to_k(context), self.to_v(context)
            ck, cv = map(lambda t: t.reshape(b, heads, k_dim, -1), (ck, cv))
            k = torch.cat((k, ck), dim=3)
            v = torch.cat((v, cv), dim=3)

        k = k.softmax(dim=-1)

        if self.norm_queries:
            q = q.softmax(dim=-2)

        context = torch.einsum('bhdn,bhen->bhde', k, v)
        out = torch.einsum('bhdn,bhde->bhen', q, context)
        out = out.reshape(b, -1, h, w)
        out = self.to_out(out)
        return out

Error is:
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [384, 512, 1, 1], but got 3-dimensional input of size [1, 1984, 512] instead
Also, my data fed to transformer is of size torch.Size([1983, 512]) and my batch size is 1.
Full log is:
$ bash scripts/train.sh
train: True test: False cam: False
preparing datasets and dataloaders......
total_train_num:  176
creating models......
n_class:  2
in_dim:  512
value dim:  64
chan out:  512
kernel_size:  1
out_conv_kwargs:  {'padding': 0}
in_chan:  768
in_dim:  512
value dim:  64
chan out:  512
kernel_size:  1
out_conv_kwargs:  {'padding': 0}
in_chan:  768

=>Epoches 1, learning rate = 0.0010000, previous best = 0.0000
torch.Size([1983, 512])
features size:  torch.Size([1983, 512])
/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:129: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:154: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
max_feature_num:  1983
batch feature size:  torch.Size([1, 1983, 512])
x.shape:  torch.Size([1, 1984, 512])
*x.shape is:  1 1984 512
heads:  12
Traceback (most recent call last):
  File "main.py", line 148, in <module>
    preds,labels,loss = trainer.train(sample_batched, model)
  File "/SeaExp/mona/research/code/cc/helper.py", line 71, in train
    pred,labels,loss = model.forward(feats, labels, masks)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/research/code/cc/models/Transformer.py", line 31, in forward
    out = self.transformer(X)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/research/code/cc/models/linear_att_ViT.py", line 262, in forward
    feat = self.transformer(emb)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/research/code/cc/models/linear_att_ViT.py", line 206, in forward
    out = layer(out)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/research/code/cc/models/linear_att_ViT.py", line 174, in forward
    out = self.attn(out)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/research/code/cc/models/linear_att_ViT.py", line 92, in forward
    q, k, v = (self.to_q(x), self.to_k(x), self.to_v(x))
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/SeaExp/mona/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 439, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [384, 512, 1, 1], but got 3-dimensional input of size [1, 1984, 512] instead

The original SelfAttention code is:
class SelfAttention(nn.Module):
    def __init__(self, in_dim, heads=8, dropout_rate=0.1):
        super(SelfAttention, self).__init__()
        self.heads = heads
        self.head_dim = in_dim // heads
        self.scale = self.head_dim ** 0.5

        self.query = LinearGeneral((in_dim,), (self.heads, self.head_dim))
        self.key = LinearGeneral((in_dim,), (self.heads, self.head_dim))
        self.value = LinearGeneral((in_dim,), (self.heads, self.head_dim))
        self.out = LinearGeneral((self.heads, self.head_dim), (in_dim,))

        if dropout_rate > 0:
            self.dropout = nn.Dropout(dropout_rate)
        else:
            self.dropout = None

    def forward(self, x):
        b, n, _ = x.shape

        q = self.query(x, dims=([2], [0]))
        k = self.key(x, dims=([2], [0]))
        v = self.value(x, dims=([2], [0]))

        q = q.permute(0, 2, 1, 3)
        k = k.permute(0, 2, 1, 3)
        v = v.permute(0, 2, 1, 3)

        attn_weights = torch.matmul(q, k.transpose(-2, -1)) / self.scale
        attn_weights = F.softmax(attn_weights, dim=-1)
        out = torch.matmul(attn_weights, v)
        out = out.permute(0, 2, 1, 3)

        out = self.out(out, dims=([2, 3], [0, 1]))

        return out

How can I fix this error? I am calling the ImageSelfAttention as following in the Encoder block of the Vision Transformer:
class EncoderBlock(nn.Module):
    def __init__(self, in_dim, mlp_dim, num_heads, dropout_rate=0.1, attn_dropout_rate=0.1):
        super(EncoderBlock, self).__init__()

        self.norm1 = nn.LayerNorm(in_dim)
        #self.attn = SelfAttention(in_dim, heads=num_heads, dropout_rate=attn_dropout_rate)
        ## note Mona: not sure if I am correctly passing the params
        # what about attn_dropout_rate=0.1
        ## I don't know
        print('in_dim: ', in_dim)
        self.attn = ImageLinearAttention(chan=in_dim, heads=num_heads, key_dim=32)
        if dropout_rate > 0:
            self.dropout = nn.Dropout(dropout_rate)
        else:
            self.dropout = None
        self.norm2 = nn.LayerNorm(in_dim)
        self.mlp = MlpBlock(in_dim, mlp_dim, in_dim, dropout_rate)

    def forward(self, x):
        residual = x
        out = self.norm1(x)
        out = self.attn(out)
        if self.dropout:
            out = self.dropout(out)
        out += residual
        residual = out

        out = self.norm2(out)
        out = self.mlp(out)
        out += residual
        return out

The code for SelfAttention and how to use it in encoder is mostly from https://github.com/asyml/vision-transformer-pytorch/blob/main/src/model.py

Output: Looks like image self attention works on 4 dimensional inputs of shape (batch, dim, height, width) suited for images and self attention works on 3 dimensional inputs of shape (batch, sequence length, dim) suited for NLP tasks. Maybe the input has to be reshaped before feeding to self attention.


Instruction: Batching irregularities with data loader

Input:
I have some data in .txt files and an instance formed by two lines which both have 100 elements in them. First line defines the problem and the second line defines the solution. Even though it is not a great idea I tried to use a supervised setting among the data. However, I am facing problems with batching. I have added the code for both the data loader and the main for loop that does the job.
The problem I get is that if
I set the batch_size to 5 and preds array has the correct form. However, labels array has one more dimension and instead of having 5 integers in it, it has 5 complete problem solutions.
I believe the problem is in the data loader but couldn't solve it. I am kinda new to the concept, I have been trying to find this for over a week but nothing has settled so far.
Data Loader:
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import pdb
import numpy as np
from torch.utils.data import Dataset

class load_dataset(Dataset):
    def __init__(self, data_file='data.txt', transform=None):
        super().__init__()
        data = np.loadtxt(data_file)
        data = torch.Tensor(data)
        self.data = data[::2]
        self.targets = data[1::2]

    def __len__(self):
        return len(self.targets)

    def __getitem__(self, index):
        adj, target = self.data[index], self.targets[index]
        return adj, target

Main Loop:
for inputs, labels in loaders["train"]:
    inputs, labels = inputs.view([batch_size, 100]), labels.data
    scores = mps(inputs)
    _, preds = torch.max(scores, 1)
    print("preds: ")
    print(preds)
    print("labels: ")
    print(labels)

Output:
preds:
tensor([0, 0, 0, 0, 0])
labels:
tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,
         0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
         0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
         0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.,
         0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.,
         0., 0., 0., 0., 1., 0., 0., 0., 0., 0.]])

Output:
You haven't shown how you defined your dataloader, but assuming you are wrapping load_dataset with a torch.utils.data.DataLoader and setting batch_size=5.
If you set your batch size to 5, then you will have 5 "problems" and the corresponding 5 "solutions" in a single batch. Each having 100 components. This means inputs and labels will be two tensors shaped as (batch_size=5, 100).