Instruction: Semantic Segmentation runtime error at loss function

Input:
I am using a costume model for segmentation (SETRModel). The model output shape is (nBatch, 256, 256) and the code below confirms it (note that the channel is squeezed out). The target shape is the same (It’s a PILMask).
When I start training, I get a runtime error (see below) related to the loss function. What am I doing wrong?
```
size = 480
half= (256, 256)
splitter = FuncSplitter(lambda o: Path(o).parent.name == 'validation')

dblock = DataBlock(blocks=(ImageBlock, MaskBlock(codes)),
               get_items=get_relevant_images,
               splitter=splitter,
               get_y=get_mask,
               item_tfms=Resize((size,size)),
               batch_tfms=[*aug_transforms(size=half), Normalize.from_stats(*imagenet_stats)])

dls = dblock.dataloaders(path/'images', bs=4)

model = SETRModel(patch_size=(32, 32),
            in_channels=3,
            out_channels=1,
            hidden_size=1024,
            num_hidden_layers=8,
            num_attention_heads=16,
            decode_features=[512, 256, 128, 64])

# Create a Learner using a custom model
loss = nn.BCEWithLogitsLoss()
learn = Learner(dls, model, loss_func=loss, lr=1.0e-4, cbs=callbacks, metrics=[Dice()])

# Let's test and make sure the loss function is happy with its inputs
learn.eval()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

t1 = torch.rand(4, 3, 256, 256).to(device)
print("input: " + str(t1.shape))

pred = learn.model(t1).to(device)
print("output: " + str(pred.shape))

# prints this:
# input: torch.Size([4, 3, 256, 256])
# output: torch.Size([4, 256, 256])

target = next(iter(learn.dls.train))[1]
target = target.type(torch.float32).to(device)
target.size(), pred.size()

# prints this:
# (torch.Size([4, 256, 256]), torch.Size([4, 256, 256]))

loss(pred, target)

# prints this:
# TensorMask(0.6844, device='cuda:0', grad_fn=<AliasBackward>)

# so, the loss function is happy with its inputs

learn.fine_tune(50)

# prints this:
# ---------------------------------------------------------------------------
# RuntimeError                              Traceback (most recent call last)
# <ipython-input-114-0e514c73651a> in <module>()
# ----> 1 learn.fine_tune(50)

# 19 frames
# /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in         binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction,     pos_weight)
#    2827 pixel_shuffle = _add_docstr(torch.pixel_shuffle, r"""
#    2828 Rearranges elements in a tensor of shape :math:`(*, C \times r^2, H, W)` to a
# -> 2829 tensor of shape :math:`(*, C, H \times r, W \times r)`.
#    2830
#    2831 See :class:`~torch.nn.PixelShuffle` for details.

# RuntimeError: result type Float can't be cast to the desired output type Long

Output:
This is something that happens when you use PyTorch inside fastai (I believe this should be fixed).
Just create custom loss_func. For example:
def loss_func(output, target): return CrossEntropyLossFlat()(out, targ.long())

and pass it when creating the DataBlock:
dblock = DataBlock(... , loss_func=loss_func, ...)


Instruction: RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same - PyTorch

Input:
I'm trying to push both my mode and data, images and labels, to run on the GPU by doing:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Followed by:
count = 0
loss_list = []
iteration_list = []
accuracy_list = []
epochs = 30

for epoch in range(epochs):
    for i, (images, labels) in enumerate(trainloader):
            net = net.to(device)
            images.to(device)
            labels.to(device)

            optimizer.zero_grad()
            outputs = net(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            count += 1

            if count % 50 == 0:
                correct = 0
                total = 0

                for i, (images, labels) in enumerate(testloader):
                    images.to(device)
                    labels.to(device)

                    outputs = net(images)
                    predicted = torch.max(outputs.data, 1)[1]
                    total += len(labels)
                    correct += (predicted == labels).sum()
                accuracy = 100 * correct / float(total)

                loss_list.append(loss.data)
                iteration_list.append(count)
                accuracy_list.append(accuracy)

                if count % 500 == 0:
                    print("Iteration: {} Loss: {} Accuracy: {} %".format(count, loss.data, accuracy))

I'm explicitly pushing my model and data to device however I am met by the error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-341-361b906da73d> in <module>()
     12
     13             optimizer.zero_grad()
---> 14             outputs = net(images)
     15             loss = criterion(outputs, labels)
     16             loss.backward()

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

I feel like I'm doing the right thing by pushing both model and data to GPU but I can't figure out why it's not working. Does somebody know what's going wrong? Thank you in advance.

Output: Your weights are saved on your gpu but your input is on your cpu. You can change that by: images.cuda()


Instruction: Dataloader worker exited unexpectedly while running on Visual Studio. But runs okay on Google Colab

Input:
So I have this dataloader that loads data from hdf5 but exits unexpectedly when I am using num_workers>0 (it works ok when 0). More strangely, it works okay with more workers on google colab, but not on my computer.
On my computer I have the following error:
Traceback (most recent call last):
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 986, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\multiprocessing\queues.py", line 105, in get
raise Empty
_queue.Empty
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 2, in
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 517, in next
data = self._next_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1182, in _next_data
idx, data = self._get_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 1148, in _get_data
success, data = self._try_get_data()
File "C:\Users\Flavio Maia\AppData\Roaming\Python\Python37\site-packages\torch\utils\data\dataloader.py", line 999, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 12332) exited unexpectedly
Also, my getitem function is:
def __getitem__(self,index):
  desired_file = int(index/self.file_size)
  position = index % self.file_size

  h5_file = h5py.File(self.files[desired_file], 'r')

  image = h5_file['Screenshots'][position]
  rect = h5_file['Rectangles'][position]
  numb = h5_file['Numbers'][position]

  h5_file.close()

  image = torch.from_numpy(image).float()
  rect = torch.from_numpy(rect).float()
  numb = torch.from_numpy( np.asarray(numb) ).float()

  return (image, rect, numb)

Does anyone have any idea what can be causing this empty queue?

Output:
Windows can't handle num_workers > 0 . You can just set it to 0 which is fine. What also should work: Put all your train / test script in a train/test() function and call it under if __name__ == "__main__":
For example like this:
class MyDataLoder(torch.utils.data.Dataset):
    train_set = create_dataloader()
    . . .

def train():
    test_set = create_dataloader()
    . . .

def test():
    . . .

if __name__ == "__main__":
    train()
    test()


Instruction: Creating a model which weights are the sum of weights of 2 different neural networks

Input:
I am doing an experiment of transfer learning.
I trained 2 CNNs that have exactly the same structure, one for MNIST and one for SVHN.
I obtained the parameters (weights  and bias) of the 2 models.
Now, I want to combine (sum, or other operations) these weights. A thing like this:
modelMNIST.parameters()
modelSVHN.parameters()

#now the new model
model3 = MyCNN(1)
model3.parameters = modelMNIST.parameters()+modelSVHN.parameters()

If I do in this way, I obtain this error:
SyntaxError: can't assign to function call
And in this way:
model3.block_1[0].weight = modelMNIST.block_1[0].weight + modelSVHN.block_1[0].weight

I get this error:
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

Is there any way to combine weights of different models?

Output:
My solution is this:
class VGG16SUM(nn.Module):

    def __init__(self, model1, model2, num_classes):
        super(VGG16SUM, self).__init__()

        # calculate same padding:
        # (w - k + 2*p)/s + 1 = o
        # => p = (s(o-1) - w + k)/2

        self.block_1 = nn.Sequential(
            nn.Conv2d(in_channels=1,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      # (1(32-1)- 32 + 3)/2 = 1
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,
                      out_channels=64,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_2 = nn.Sequential(
            nn.Conv2d(in_channels=64,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(in_channels=128,
                      out_channels=128,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_3 = nn.Sequential(
            nn.Conv2d(in_channels=128,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.Conv2d(in_channels=256,
                      out_channels=256,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(256),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.block_4 = nn.Sequential(
            nn.Conv2d(in_channels=256,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.Conv2d(in_channels=512,
                      out_channels=512,
                      kernel_size=(3, 3),
                      stride=(1, 1),
                      padding=1),
            nn.BatchNorm2d(512),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=(2, 2),
                         stride=(2, 2))
        )

        self.classifier = nn.Sequential(
            nn.Linear(2048, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(p=0.25),
            nn.Linear(4096, num_classes),
        )

        for p_out, p_in1, p_in2 in zip(self.parameters(), model1.parameters(), model2.parameters()):
            p_out.data = nn.Parameter(p_in1 +p_in2);

    def forward(self, x):

        x = self.block_1(x)
        x = self.block_2(x)
        x = self.block_3(x)
        x = self.block_4(x)
        # x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x
        #logits = self.classifier(x)
        #probas = F.softmax(logits, dim=1)
        # probas = nn.Softmax(logits)
        #return probas
        # return logits

It works!!!


Instruction: torch concat 1D to a 2D tensor

Input:
I have a issue on concat 2 tensor,
say I have x and y:
x = torch.randn(35, 50)
y = torch.randn(35)

How do I concat every y value in to x[0] to make x has a shape 35,51?
I tried:
for i in y:
    for a in range(x.shape[0]):
        x[a] = torch.cat((x[a],i),0)

Still getting shape error. Any smart way of doing it?

Output:
This should work:
z = torch.cat([x,y.reshape(-1,1)], axis=1)
print(z.shape)

Output:
torch.Size([35, 51])


Instruction: Pytorch: create a mask that is larger than the n-th quantile of each 2D tensor in a batch

Input:
I have a torch.Tensor of shape (2, 2, 2) (can be bigger), where the values are normalized within range [0, 1].
Now I am given a positive integer K, which tells me that I need to create a mask where for each 2D tensor inside the batch, values are 1 if it is larger than 1/k of all the values, and 0 elsewhere. The return mask also has shape (2, 2, 2).
For example, if I have a batch like this:
tensor([[[1., 3.],
         [2., 4.]],
        [[5., 7.],
         [9., 8.]]])

and let K=2, it means that I must mask the values where they are greater than 50% of all the values inside each 2D tensor.
In the example, the 0.5 quantile is 2.5 and 7.5, so this is the desired output:
tensor([[[0, 1],
         [0, 1]],
        [[0, 0],
         [1, 1]]])

I tried:
a = torch.tensor([[[0, 1],
                   [0, 1]],
                  [[0, 0],
                   [1, 1]]])
quantile = torch.tensor([torch.quantile(x, 1/K) for x in a])
torch.where(a > val, 1, 0)

But this is the result:
tensor([[[0, 0],
         [0, 0]],
        [[1, 0],
         [1, 1]]])

Output:
t = torch.tensor([[[1., 3.],
         [2., 4.]],
        [[5., 7.],
         [9., 8.]]])

t_flat = torch.reshape(t, (t.shape[0], -1))
quants = torch.quantile(t_flat, 1/K, dim=1)
quants = torch..reshape(quants, (quants.shape[0], 1, 1))
res = torch.where(t > val, 1, 0)

and after this res is:
tensor([[[0, 1],
         [0, 1]],

        [[0, 0],
         [1, 1]]])

which is what you wanted


Instruction: How to get modules of 3d tensors in 4d torch tensor?

Input:
I have a torch tensor with 4 dimensions.
It's shape is [50, 1, 1, 200]. I have to get a list with modules of 200 hundred 3d tensors. Which is the easiest way to do so?

Output:
Did you try torch.unbind (https://pytorch.org/docs/stable/generated/torch.unbind.html)?
a = torch.rand(50, 1, 1, 200)
b = torch.unbind(dim=3)

len(b)      # 200
b[0].shape  # torch.Size([50, 1, 1])


Instruction: PyTorch: apply mask with different shape

Input:
I have a tensor of shape (60, 3, 32, 32) and a boolean mask of shape (60, 32, 32). I want to apply this mask to the tensor. The output tensor should have shape (60, 3, 32, 32), and values are kept if the mask is 1, else 0.
How can I do that fast?

Output:
Let t be the tensor and m be the mask. You can use:
t * m.unsqueeze(1)


Instruction: attn_output_weights in MultiheadAttention

Input:
I wanna know if the matrix of the attn_output_weight can demonstrate the relationship between every word-pair in the input sequence.
In my project, I draw the heat map based on this output and it shows like this:

However, I can hardly see any information from this heat map.
I refer to other people's work, their heat map is like this.  At least the diagonal of the matrix should have the deep color.
Then I wonder if my method to draw the heat map is correct or not (i.e. directly using the output of the attn_output_weight ) If this is not the correct way, could you please tell me how to draw the heat map?

Output:
It seems your range of values is rather limited. In the target example the range of values lies between [0, 1], since each row represents the softmax distribution. This is visible from the definition of attention:

I suggest you normalize each row / column (according to the attention implementation you are using) and finally visualize the attention maps in the range [0, 1]. You can do this using the arguments vmin and vmax respectively in matplotlib plottings.
If this doesn't solve the problem, maybe add a snippet of code containing the model you are using and the visualization script.


Instruction: What is the simplest way to continue training a pre-trained BERT model, on a specific domain?

Input:
I want to use a pre-trained BERT model in order to use it on a text classification task (I'm using Huggingface library). However, the pre-trained model was trained on domains that are different than mine, and I have a large unannotated dataset that can be used for fine-tuning it. If I use only my tagged examples and fine-tune it "on the go" while training on the specific task (BertForSequenceClassification), the dataset is too small for adapting the language model for the specific domain. What it the best way to do so?
Thanks!

Output:
Let's clarify a couple of points first to reduce some ambiguity.

BERT uses two pretraining objectives: Masked Language Modeling (MLM) and Next Sentence Prediction.
You mentioned having a large unannotated dataset, which you plan on using to fine-tune your BERT model. This is not how fine-tuning works. In order to fine-tune your pretrained model, you would need an annotated dataset i.e. document & class pair for sequence classification downstream task.

So what can you do? First, extend your general domain tokenizer with your unannotated dataset consisting of domain-specific vocabulary. Then, using this extended tokenizer you can continue pretraining on MLM and/or NSP objectives to modify your word embeddings. Finally, fine-tune your model using an annotated dataset.


Instruction: Randomly set some elements in a tensor to zero (with low computational time)

Input:
I have a tensor of shape (3072,1000) which represents the weights in my neural network. I want to:

randomly set 60% of its elements to zero.
After updating the weights, keep 60% of the elements equal to zero but again randomly i.e., not the same previous elements.

Note: my network is not the usual artificial neural network which uses backpropagation algorithm but it is a biophysical model of the neurons in the brain so I am using special weight updating rules. Therefore, I think the ready functions in pytorch, if any, might not be helpful.
I tried the follwoing code, it is working but it takes so long because after every time I update my weight tensor, I have to run that code to set the weight tensor again to be 60% zeros
row_indices = np.random.choice(np.size(mytensor.weight, 0),
                       replace=False,size=int(np.size(mytensor.weight, 0)* 0.6))
column_indices = np.random.choice(np. size(mytensor.weight, 1),
                       replace=False, size=int(np. size(mytensor.weight, 1) * 0.6))
for r in row_indices:
    for c in column_indices:
        (mytensor.weight)[r][c] = 0

Output:
You can use the dropout function for this:
import torch.nn.functional as F

my_tensor.weight = F.dropout(my_tensor.weight, p=0.6)


Instruction: PyTorch tensor declared as torch.long becomes torch.int64

Input:
I am new to PyTorch so I haven't worked a lot with PyTorch Tensors. Something I am puzzled about is if I declare the dytpe of a tensor as torch.long, and then check the dtype it is int64. For example:
In [62]: a = torch.tensor([[0, 1, 1, 2],
                       [1, 0, 2, 1]], dtype=torch.long)
         a.dtype

Out[62]: torch.int64

I am probably making some silly mistake.
Why is this happening?
Edit:
     89         if isinstance(edge_index, Tensor):
---> 90             assert edge_index.dtype == torch.long
     91             assert edge_index.dim() == 2
     92             assert edge_index.size(0) == 2

In my case a is edge_index.

Output: From the the documentation we can see that torch.long and torch.int64 are synonymous and both refer to the 64-bit signed integer type.


Instruction: How to check accuracy on BCELoss Pytorch?

Input:
I'm trying to use Pytorch to take a HeartDisease.csv and predict whether the patient has heart disease or not... the .csv provides 13 inputs and 1 target
I'm using BCELoss and I'm having trouble understanding how to write an accuracy check function.
My num_samples is correct but not my num_correct. I think this is a result of not understanding the predictions tensor. Right now my num_correct is usually over 8000 while my num_samples is 303...
Any insight on how to write this check accuracy function is much appreciated
I wrote this on a google co lab
#imports
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
import pandas as pd

#create fully connected network
class NN(nn.Module):
  def __init__(self, input_size, num_classes):
    super(NN, self).__init__()
    self.outputs = nn.Linear(input_size, 1)

  def forward(self, x):
     x = self.outputs(x)
     return torch.sigmoid(x)

#set device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

#hyperparameters
input_size = 13 # 13 inputs
num_classes = 1 # heartdisease or not
learning_rate = 0.001
batch_size = 64
num_epochs = 1

#load data
class MyDataset(Dataset):
    def __init__(self, root, n_inp):
        self.df = pd.read_csv(root)
        self.data = self.df.to_numpy()
        self.x , self.y = (torch.from_numpy(self.data[:,:n_inp]),
                           torch.from_numpy(self.data[:,n_inp:]))
    def __getitem__(self, idx):
        return self.x[idx, :], self.y[idx,:]
    def __len__(self):
        return len(self.data)

train_dataset = MyDataset("heart.csv", input_size)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle =True)
test_dataset = MyDataset("heart.csv", input_size)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle =True)

#initialize network
model = NN(input_size=input_size, num_classes=num_classes).to(device)

#loss and optimizer
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

#train network
for epoch in range(num_epochs):
  for batch_idx, (data, targets) in enumerate(train_loader):
    #get data to cuda if possible
    data = data.to(device=device)
    targets = targets.to(device=device)

    #forward
    scores = model(data.float())
    targets = targets.float()
    loss = criterion(scores, targets)

    #backward
    optimizer.zero_grad()
    loss.backward()

    #grad descent or adam step
    optimizer.step()

#check accuracy of model
def check_accuracy(loader, model):
  num_correct = 0
  num_samples = 0
  model.eval()

  with torch.no_grad():
    for x, y in loader:
      x = x.to(device=device)
      y = y.to(device=device)

      scores = model(x.float())
      _, predictions = scores.max(1)
      num_correct += (predictions == y).sum()
      num_samples += predictions.size(0)

    print("Got {} / {} with accuracy {}".format(num_correct, num_samples, float(num_correct)/float(num_samples)*100))

  model.train()

print("checking accuracy on training data")
check_accuracy(train_loader, model)

print("checking accuracy on test data")
check_accuracy(test_loader, model)

Output:
Note: Don't fool yourself. A single linear layer + a sigmoid + BCE loss = logistic regression. This is a linear model, so just take note of that when referring to it as a "neural network", which is a term usually reserved for similar networks but with at least one hidden layer and nonlinear activations.
The sigmoid layer at the end of your model's forward() function returns an (N,1)-sized tensor, where N is the batch size. In other words, it returns a scalar for every data point. Each scalar is a value between 0 and 1 (this is the range of the sigmoid function).
The idea is to interpret those scalars as probabilities corresponding to the positive class. Suppose 1 corresponds to heart disease, and 0 corresponds to no heart disease; heart disease is the positive class, and no heart disease is the negative class. Now suppose a score is 0.6. This might be interpreted as a 60% chance that the associated label is heart disease, and a 40% chance that the associated label is no heart disease. This interpretation of the sigmoid output is what motivates the BCE loss to begin with (it's ultimately just a negative log likelihood).
So what you might do is check if your scores are greater than 0.5. If so, predict heart disease. If not, predict no heart disease.
Right now, you're computing maximums from the scores across dimension 1, which does nothing because dimension 1 is already of size 1; taking the maximum of a single value simply gives you that value.
Try something like this:
def check_accuracy(loader, model):
  num_correct = 0
  num_samples = 0
  model.eval()

  with torch.no_grad():
    for x, y in loader:
      x = x.to(device=device)
      y = y.to(device=device)

      scores = model(x.float())
      // Create a Boolean tensor (True for scores > 0.5, False for others)
      // and then cast it to a long tensor (Trues -> 1, Falses -> 0)
      predictions = (scores > 0.5).long()
      num_correct += (predictions == y).sum()
      num_samples += predictions.size(0)

    print("Got {} / {} with accuracy {}".format(num_correct, num_samples, float(num_correct)/float(num_samples)*100))

  model.train()

You may also want to squeeze your prediction and target tensors to size (N) instead of (N,1), though I'm not sure it's necessary in your case.


Instruction: How I can convert model.pt to model.h5?

Input:
After using YOLOv5 to train model weights as .pt file,
how can I convert the weights file (model.pt) to hdf5 file (model.h5)?
Running python train.py --batch 16 --epochs 3 --data mydata.yaml --weights yolov5s.pt, the result is given by best.pt file at subfolder of YOLOv5, how can I convert it to h5 file?

Output:
Do this installation and get started here
This could be a lengthy procedure ...    as you are aware that .pt only contains weights and not model architecture hence your model class should also be present in  your conversion code
Edit: New links are added


Instruction: How do you test a custom dataset in Pytorch?

Input:
I've been following tutorials in Pytorch that use datasets from Pytorch that allow you to enable whether you'd like to train using the data or not... But now I'm using a .csv and a custom dataset.
class MyDataset(Dataset):
    def __init__(self, root, n_inp):
        self.df = pd.read_csv(root)
        self.data = self.df.to_numpy()
        self.x , self.y = (torch.from_numpy(self.data[:,:n_inp]),
                           torch.from_numpy(self.data[:,n_inp:]))
    def __getitem__(self, idx):
        return self.x[idx, :], self.y[idx,:]
    def __len__(self):
        return len(self.data)

How can I tell Pytorch not to train my test_dataset so I can use it as a reference of how accurate my model is?
train_dataset = MyDataset("heart.csv", input_size)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle =True)
test_dataset = MyDataset("heart.csv", input_size)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle =True)

Output:
In pytorch, a custom dataset inherits the class Dataset. Mainly it contains two methods __len__() is to specify the length of your dataset object to iterate over and __getitem__() to return a batch of data at a time.
Once the dataloader objects are initialized (train_loader and test_loader as specified in your code), you need to write a train loop and a test loop.
def train(model, optimizer, loss_fn, dataloader):
    model.train()
    for i, (input, gt) in enumerate(dataloader):
        if params.use_gpu: #(If training using GPU)
            input, gt = input.cuda(non_blocking = True), gt.cuda(non_blocking = True)
        predicted = model(input)
        loss = loss_fn(predicted, gt)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

and your test loop should be:
def test(model,loss_fn, dataloader):
    model.eval()
    for i, (input, gt) in enumerate(dataloader):
        if params.use_gpu: #(If training using GPU)
            input, gt = input.cuda(non_blocking = True), gt.cuda(non_blocking = True)
        predicted = model(input)
        loss     = loss_fn(predicted, gt)

In additional you can use metrics dictionary to log your predicted, loss, epochs etc,. The main difference between training and test loop is that we exclude back propagation (zero_grad(), backward(), step()) in inference stage.
Finally,
for epoch in range(1, epochs + 1):
    train(model, optimizer, loss_fn, train_loader)
    test(model, loss_fn, test_loader)


Instruction: How to move data_parallel model to a specific cuda device?

Input:
I currently need to use a pretrained model by setting it on a specific cuda device. The pretrained model is defined as below:
DataParallel(
  (module): MobileFaceNet(
    (conv1): Conv_block(
      (conv): Conv2d(3, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (prelu): PReLU(num_parameters=64)
    )
    (conv2_dw): Conv_block(
      (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=64, bias=False)
      (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (prelu): PReLU(num_parameters=64)
    )
    (conv_23): Depth_Wise(
      (conv): Conv_block(
        (conv): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=128)
      )
      (conv_dw): Conv_block(
        (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=128, bias=False)
        (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=128)
      )
      (project): Linear_block(
        (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (conv_3): Residual(
      (model): Sequential(
        (0): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (project): Linear_block(
            (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (project): Linear_block(
            (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (2): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (project): Linear_block(
            (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (3): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=128, bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=128)
          )
          (project): Linear_block(
            (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
    )
    (conv_34): Depth_Wise(
      (conv): Conv_block(
        (conv): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=256)
      )
      (conv_dw): Conv_block(
        (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=256, bias=False)
        (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=256)
      )
      (project): Linear_block(
        (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (conv_4): Residual(
      (model): Sequential(
        (0): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (2): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (3): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (4): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (5): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
    )
    (conv_45): Depth_Wise(
      (conv): Conv_block(
        (conv): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=512)
      )
      (conv_dw): Conv_block(
        (conv): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=512, bias=False)
        (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (prelu): PReLU(num_parameters=512)
      )
      (project): Linear_block(
        (conv): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (conv_5): Residual(
      (model): Sequential(
        (0): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
        (1): Depth_Wise(
          (conv): Conv_block(
            (conv): Conv2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (conv_dw): Conv_block(
            (conv): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=256, bias=False)
            (bn): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
            (prelu): PReLU(num_parameters=256)
          )
          (project): Linear_block(
            (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
          )
        )
      )
    )
    (conv_6_sep): Conv_block(
      (conv): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (prelu): PReLU(num_parameters=512)
    )
    (conv_6_dw): Linear_block(
      (conv): Conv2d(512, 512, kernel_size=(7, 7), stride=(1, 1), groups=512, bias=False)
      (bn): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (conv_6_flatten): Flatten()
    (linear): Linear(in_features=512, out_features=512, bias=False)
    (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
)

If I conventionally declare
model.to(device)

with device on cuda:1, then it makes error when forwarding:
model(imgs)

RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1

I think this is because the model was previously trained with data parallel utils in pytorch.
How can I properly set the model to the device that I specifically want?

Output:
You should get the neural network out of DataParallel first.
Assuming your DataParallel is named model you could do:
device = torch.device("cuda:1")
module = model.module.to(device)


Instruction: Elegant Numpy Tensor product

Input:
I need to take the product over two tensors in numpy (or pytorch):
I have
A = np.arange(1024).reshape(8,1,128)
B = np.arange(9216).reshape(8, 128, 9)

And want to obtain C, with dot products summing over the last dim of A (axis=2) and the middle dim of B (axis=1). This should have dimensions 8x9. Currently, I am doing:
C = np.zeros([8, 9])
for i in range(8):
    C[i,:] = np.matmul(A[i,:,:], B[i,:,:])

How to do this elegantly?
I tried:
np.tensordot(weights, features, axes=(2,1)).

but it returns 8x1x8x9.

Output:
One way would be to use numpy.einsum.
C = np.einsum('ijk,ikl->il', A, B)

Or you could use broadcasted matrix multiply.
C = (A @ B).squeeze(axis=1)
# equivalent: C = np.matmul(A, B).squeeze(axis=1)


Instruction: How do I load a local model with torch.hub.load?

Input:
I need to avoid downloading the model from the web (due to restrictions on the machine installed).
This works, but it downloads the model from the Internet
model = torch.hub.load('pytorch/vision:v0.9.0', 'deeplabv3_resnet101', pretrained=True)

I have placed the .pth file and the hubconf.py file in the /tmp/ folder and changed my code to
model = torch.hub.load('/tmp/', 'deeplabv3_resnet101', pretrained=True, source='local')

but to my surprise, it still downloads the model from the Internet. What am I doing wrong? How can I load the model locally?
Just to give you a bit more details, I'm doing all this in a Docker container that has a read-only volume at runtime, so that's why the download of new files fails.

Output:
There are two approaches you can take to get a shippable model on a machine without an Internet connection.

Load DeepLab with a pretrained model on a normal machine, use a JIT compiler to export it as a graph, and put it into the machine. The Script is easy to follow:
 # To export
 model = torch.hub.load('pytorch/vision:v0.9.0', 'deeplabv3_resnet101', pretrained=True).eval()
 traced_graph = torch.jit.trace(model, torch.randn(1, 3, H, W))
 traced_graph.save('DeepLab.pth')

 # To load
 model = torch.jit.load('DeepLab.pth').eval().to(device)

In this case, the weights and network structure is saved as computational graph, so you won't need any extra files.

Take a look at torchvision's GitHub repository.
There's a download URL for DeepLabV3 with Resnet101 backbone weights.
You can download those weights once, and then use deeplab from torchvision with pretrained=False flag and load weights manually.
 model = torch.hub.load('pytorch/vision:v0.9.0', 'deeplabv3_resnet101', pretrained=False)
 model.load_state_dict(torch.load('downloaded weights path'))

Take in consideration, there might be a ['state_dict'] or some similar parent key in state dict, where you would use:
 model.load_state_dict(torch.load('downloaded weights path')['state_dict'])


Instruction: How to translate a conv2D in keras or tensorflow which is already implemented in PyTorch?

Input:
I have the following function in pytorch implementation for replacing a conv2D layer with 3 different layers:
first_layer = torch.nn.Conv2d(in_channels=3, \
            out_channels=3, kernel_size=1,
            stride=1, padding=0, dilation = (1,1), bias=False)

core_layer = torch.nn.Conv2d(in_channels=3, \
            out_channels=16, kernel_size=(3,3),
            stride=(1,1), padding=(1,1), dilation=(1,1),
            bias=False)

last_layer = torch.nn.Conv2d(in_channels=16, \
                                 out_channels=64], kernel_size=1, stride=1,
                                 padding=0, dilation=(1,1), bias=True)

last_layer.bias.data = layer.bias.data

first_layer.weight.data = \
        torch.transpose(first, 1, 0).unsqueeze(-1).unsqueeze(-1)
last_layer.weight.data = last.unsqueeze(-1).unsqueeze(-1)
core_layer.weight.data = core

new_layers = [first_layer, core_layer, last_layer]
y = nn.Sequential(*new_layers)

where, 'first' represents a random 3 by 3 matrix.
'core' represents a tensor of shape [16,3,3,3]
'last' represents another random matrix of  size (64,16).
When I tried to translate this into keras, I have the following :
first_layer = tf.keras.layers.SeparableConv2D(3, kernel_size=1, strides = (1,1), padding = 'same',  dilation_rate = (1,1), use_bias = False )

core_layer = tf.keras.layers.Conv2D(16, kernel_size=3, strides = (1,1), padding = (1,1), dilation_rate = (1,1), use_bias = False)

last_layer = tf.keras.layers.SeparableConv2D(64, kernel_size=1, strides = (1,1), \
                           padding = 'same', dilation_rate = (1,1), use_bias =True )

first_layer = tf.expand_dims(tf.expand_dims(tf.transpose(first, perm = [1,0]),0),0)

last_layer = tf.expand_dims(tf.expand_dims(last, 0),0)
core_layer = core
new_layers = [first_layer, core_layer, last_layer]

when I tried to get back the weights of the model in keras, I am getting a list with no weights at all. The convolution is not performed. Any idea on how to proceed further/ any other approached of transforming the above pytorch implementation to keras or tensorflow?

Output:
You are missing exactly the last step in Keras transformation.
There is also a Sequential() class in TensorFlow(Keras) that you can use to instantiate the model.
I haven't checked for the exact match between TF and PyTorch, but this should be your starting point to solve your problem.
model = tf.keras.Sequential([first_layer, core_layer, last_layer])
y = model(x)


Instruction: How to convert probability to angle degree in a head-pose estimation problem?

Input:
I reused code from others to make head-pose prediction in Euler angles. The author trained a classification network that returns bin classification results for the three angles, i.e. yaw, roll, pitch. The number of bins is 66. They somehow convert the probabilities to the corresponding angle, as written from line 150 to 152 here. Could someone help to explain the formula?
These are the relevant lines of code in the above file:
[56]  model = hopenet.Hopenet(torchvision.models.resnet.Bottleneck, [3, 4, 6, 3], 66) # a variant of ResNet50
[80]  idx_tensor = [idx for idx in xrange(66)]
[81]  idx_tensor = torch.FloatTensor(idx_tensor).cuda(gpu)
[144] yaw, pitch, roll = model(img)
[146] yaw_predicted = F.softmax(yaw)
[150] yaw_predicted = torch.sum(yaw_predicted.data[0] * idx_tensor) * 3 - 99

Output:
If we look at the training code, and the authors' paper,* we see that the loss function is a sum of two losses:

the raw model output (vector of probabilities for each bin category):

[144] yaw, pitch, roll = model(img)

a linear combination of the bin predictions (the predicted continuous angle):

[146] yaw_predicted = F.softmax(yaw)
[150] yaw_predicted = torch.sum(yaw_predicted.data[0] * idx_tensor) * 3 - 99

Since 3 * softmax(label_weighted_sum(output)) - 99 is the final layer in training the regression loss (but is not explicitly a part of the model's forward), this must be applied to the raw output to convert it from the vector of bin probabilities to a single angle prediction.

*

3.2. The Multi-Loss Approach
All previous work which predicted head pose using convolutional networks regressed all three Euler angles directly using a mean squared error loss. We notice that this approach does not achieve the best results on our large-scale synthetic training data.
We propose to use three separate losses, one for each angle. Each loss is a combination of two components: a binned pose classification and a regression component. Any
backbone network can be used and augmented with three fully-connected layers which predict the angles. These three fully-connected layers share the previous convolutional layers of the network.
The idea behind this approach is that by performing bin classification we use the very stable softmax layer and cross-entropy, thus the network learns to predict the neighbourhood of the pose in a robust fashion. By having three cross-entropy losses, one for each Euler angle, we have three signals which are backpropagated into the network
which improves learning. In order to obtain a fine-grained predictions we compute the expectation of each output angle for the binned output. The detailed architecture is shown
in Figure 2.
We then add a regression loss to the network, namely a mean-squared error loss, in order to improve fine-grained predictions. We have three final losses, one for each angle,
and each is a linear combination of both the respective classification and the regression losses. We vary the weight of the regression loss in Section 4.4 and we hold the weight of
the classification loss constant at 1. The final loss for each Euler angle is the following:

Where H and MSE respectively designate the crossentropy and mean squared error loss functions.


Instruction: Batch inference of softmax does not sum to 1

Input:
I am working with REINFORCE algorithm with PyTorch. I noticed that the batch inference/predictions of my simple network with Softmax doesn’t sum to 1 (not even close to 1). I am attaching a minimum working code so that you can reproduce it. What am I missing here?
import numpy as np
import torch

obs_size = 9
HIDDEN_SIZE = 9
n_actions = 2

np.random.seed(0)

model = torch.nn.Sequential(
        torch.nn.Linear(obs_size, HIDDEN_SIZE),
        torch.nn.ReLU(),
        torch.nn.Linear(HIDDEN_SIZE, n_actions),
        torch.nn.Softmax(dim=0)
    )

state_transitions = np.random.rand(3, obs_size)

state_batch = torch.Tensor(state_transitions)
pred_batch = model(state_batch)  # WRONG PREDICTIONS!
print('wrong predictions:\n', *pred_batch.detach().numpy())
# [0.34072137 0.34721774] [0.30972624 0.30191955] [0.3495524 0.3508627]
# DOES NOT SUM TO 1 !!!

pred_batch = [model(s).detach().numpy() for s in state_batch]  # CORRECT PREDICTIONS
print('correct predictions:\n', *pred_batch)
# [0.5955179  0.40448207] [0.6574412  0.34255883] [0.624833   0.37516695]
# DOES SUM TO 1 AS EXPECTED

Output:
Although PyTorch lets us get away with it, we don’t actually provide an input with the right dimensionality. We have a model that takes one input and produces one output, but PyTorch nn.Module and its subclasses are designed to do so on multiple samples at the same time. To accommodate multiple samples, modules expect the zeroth dimension of the input to be the number of samples in the batch.

Deep Learning with PyTorch

That your model works on each individual sample is an implementation nicety. You have incorrectly specified the dimension for the softmax (across batches instead of across the variables), and hence when given a batch dimension it is computing the softmax across samples instead of within samples:

nn.Softmax requires us to specify the dimension along which the softmax function is applied:
softmax = nn.Softmax(dim=1)

In this case, we have two input vectors in two rows (just like when we work with
batches), so we initialize nn.Softmax to operate along dimension 1.

Change torch.nn.Softmax(dim=0) to torch.nn.Softmax(dim=1) to get appropriate results.


Instruction: Is it possible to resolve TypeError: argument 'input' (position 1) must be Tensor error without retraining the model?

Input:
I have made a model in PyTorch for use in an openAI Gym environment. I have made it in the following way:
class Policy(nn.Module):
    def __init__(self, s_size=8, h_size=16, a_size=4):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, 32)
        self.fc3 = nn.Linear(32, 64)
        self.fc4 = nn.Linear(64, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.softmax(x, dim=1 )

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

I then save it's state in a dictionary and use it as following :
env = gym.make('LunarLander-v2')

policy = Policy().to(torch.device('cpu'))
policy.load_state_dict(torch.load('best_params_cloud.ckpt', map_location='cpu'))
policy.eval()
ims = []
rewards = []
state = env.reset()
for step in range(STEPS):
    img = env.render(mode='rgb_array')
    action,log_prob = policy(state)
        # print(action)
    state,reward,done,i_ = env.step(action)
    rewards.append(reward)
    # print(reward,done)
    cv2_im_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    pil_im = Image.fromarray(cv2_im_rgb)

    draw = ImageDraw.Draw(pil_im)

    # Choose a font
    font = ImageFont.truetype("Roboto-Regular.ttf", 20)

    # Draw the text
    draw.text((0, 0), f"Step: {step} Action : {action} Reward: {int(reward)} Total Rewards: {int(np.sum(rewards))} done: {done}", font=font,fill="#FDFEFE")

    # Save the image
    img = cv2.cvtColor(np.array(pil_im), cv2.COLOR_RGB2BGR)
    im = plt.imshow(img, animated=True)
    ims.append([im])
    if done:
        env.close()

        break

Writer = animation.writers['pillow']
writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)
im_ani = animation.ArtistAnimation(fig, ims, interval=50, repeat_delay=3000,
                                    blit=True)
im_ani.save('ll_train1.gif', writer=writer)

But this returns the error:
TypeError                                 Traceback (most recent call last)
<ipython-input-3-da32222edde2> in <module>
      9 for step in range(STEPS):
     10     img = env.render(mode='rgb_array')
---> 11     action,log_prob = policy(state)
     12         # print(action)
     13     state,reward,done,i_ = env.step(action)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-2-66d42ebb791e> in forward(self, x)
     33
     34     def forward(self, x):
---> 35         x = F.relu(self.fc1(x))
     36         x = F.relu(self.fc2(x))
     37         x = F.relu(self.fc3(x))

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

~\anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

TypeError: linear(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

I tried to change the forward function by adding the following line of code:
def forward(self, x):
        x = torch.tensor(x,dtype=torch.float32,device=DEVICE).unsqueeze(0) //Added this line
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.softmax(x, dim=1 )

But this also returns an error : ValueError: not enough values to unpack (expected 2, got 1)
The policy took a lot of time to train, and I am trying to avoid retraining it, is there a workaround for it to run without retraining?

Output:
This error is not related to your model.
forward function only returns the probability distribution but what you need is the action and corresponded probability (output of Policy.act).
Change your code from
for step in range(STEPS):
    img = env.render(mode='rgb_array')

    # This line causes the error.
    action,log_prob = policy(state)

to
for step in range(STEPS):
    img = env.render(mode='rgb_array')

    # This line causes the error.
    action,log_prob = policy.act(state)


Instruction: Wav2Vec pytorch element 0 of tensors does not require grad and does not have a grad_fn

Input:
I am retraining a wav2vec model from hugging face for classification problem. I have 5 classes and the input is a list of tensors [1,400].
Here is how I am getting the model
num_labels = 5
model_name = "Zaid/wav2vec2-large-xlsr-53-arabic-egyptian"
model_config = AutoConfig.from_pretrained(model_name, num_labels=num_labels)  ##needed for the visualizations
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(model_name)
model = Wav2Vec2ForCTC.from_pretrained(model_name, config=model_config)

Here is the model updated settings
# Freeze the pre trained parameters
for param in model.parameters():
    param.requires_grad = False
criterion = nn.MSELoss().to(device)
optimizer = AdamW(model.parameters(), lr=2e-5, eps=1e-6)

# Add three new layers at the end of the network
model.classifier = nn.Sequential(
    nn.Linear(768, 256),
    nn.Dropout(0.25),
    nn.ReLU(),
    nn.Linear(256, 64),
    nn.Dropout(0.25),
    nn.ReLU(),
    nn.Linear(64, 2),
    nn.Dropout(0.25),
    nn.Softmax(dim=1)
)

Then the training loop
print_every = 300

total_loss = 0
all_losses = []
model.train()
for epoch in range(2):
    print("Epoch number: ", epoch)
    for row in range(16918):
        Input = torch.tensor(trn_ivectors[row]).double()
        label = torch.tensor(trn_labels[row]).long().to(device)
        label = torch.unsqueeze(label,0).to(device)
        #print("Label", label.shape)
        Input = torch.unsqueeze(Input,1).to(device)
        #print(Input.shape)
        optimizer.zero_grad()

        #Input.requires_grad = True
        Input = F.softmax(Input[0], dim=-1)

        if label == 0:
            label = torch.tensor([1.0, 0.0]).float().to(device)
        elif label == 1:
            label = torch.tensor([0.0, 1.0]).float().to(device)

        # print(overall_output, label)

        loss = criterion(Input, label)
        total_loss += loss.item()

        loss.backward()
        optimizer.step()

        if idx % print_every == 0 and idx > 0:
            average_loss = total_loss / print_every
            print("{}/{}. Average loss: {}".format(idx, len(train_data), average_loss))
            all_losses.append(average_loss)
            total_loss = 0

torch.save(model.state_dict(), "model_after_train.pt")

Unfortunately when I try to train the program it gives me the following error
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Please I would appreciate if you could tell me how to fix this error. I have been searching a lot on a way fixing it but didn't fix it
Thanks

Output:
Please try adding
requires_grad = True


Instruction: PyTorch LSTM for Daily Stock Return Prediction - Train loss is consistently lower than test loss

Input:
I was wondering if someone could share some ideas for why my training loss begins at a higher level than the test loss?
I am trying to run an LSTM on daily stock return data as the only input and using the 10 previous days to predict the price on the next day. Training/test/validation sets do not overlap, so there is no leakage. Not using any regularisation that would impact the training data only.
Really confused at the moment as I cannot seem to find the error.
I will include the code below but its quite long
    # Defining the LSTM class

import torch
import torch.nn as nn
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler, MinMaxScaler

class LSTM(nn.Module):
  def __init__(self, n_inputs, n_hidden, num_layers, n_outputs):
    super(LSTM, self).__init__()
    self.D = n_inputs
    self.M = n_hidden
    self.K = n_outputs
    self.L = num_layers

    self.rnn = nn.LSTM(
        input_size=self.D,
        hidden_size=self.M,
        num_layers=self.L,
        batch_first=True)
    self.fc = nn.Linear(self.M, self.K)

  def forward(self, X):
    # initial hidden states
    h0 = torch.zeros(self.L, X.size(0), self.M).to(device)
    c0 = torch.zeros(self.L, X.size(0), self.M).to(device)

    # get RNN unit output
    out, _ = self.rnn(X, (h0, c0))

    # we only want h(T) at the final time step
    out = self.fc(out[:, -1, :])
    return out

# Defining a function to train the LSTM

def full_gd(model,
            loss_function,
            optimizer,
            X_train,
            y_train,
            X_test,
            y_test,
            no_epochs):

  # Stuff to store
  train_losses = np.zeros(no_epochs)
  test_losses = np.zeros(no_epochs)

  for it in range(no_epochs):
    # zero the parameter gradients
    optimizer.zero_grad()

    # Forward pass
    outputs = model(X_train)
    loss = loss_function(outputs, y_train)

    # Backward and optimize
    loss.backward()
    optimizer.step()

    # Save losses
    train_losses[it] = loss.item()

    # Test loss
    test_outputs = model(X_test)
    test_loss = loss_function(test_outputs, y_test)
    test_losses[it] = test_loss.item()

    if (it + 1) % 10 == 0:
      print(f'Epoch {it+1}/{no_epochs}, Train Loss: {loss.item():.4f}, Test Loss: {test_loss.item():.4f}')

  return train_losses, test_losses

# Import sklearn's StandardScaler to scale the returns data
scaler = StandardScaler()
scaler.fit(data[:3*len(data)//5])
historical_returns = scaler.transform(data)

# Creating the dataset to train the LSTM. D is the number of input features. T is the number of data points used in forecasting

T = 10
D = 1
X = []
Y = []

for t in range(len(historical_returns) - T):
    x = historical_returns[t:t+T]
    X.append(x)
    y = historical_returns[t+T]
    Y.append(y)

X_historical = np.array(X).reshape(-1, T, 1)
Y_historical = np.array(Y).reshape(-1, 1)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Splitting the data into a 60/20/20 train/validation/test split. No random split is used here as this is a time series dataset

x_train1 = torch.from_numpy(X_historical[:3*len(historical_returns)//5].astype(np.float32))
y_train1 = torch.from_numpy(Y_historical[:3*len(historical_returns)//5].astype(np.float32))

x_val1 = torch.from_numpy(X_historical[-2*len(historical_returns)//5: -1*len(historical_returns)//5].astype(np.float32))
y_val1 = torch.from_numpy(Y_historical[-2*len(historical_returns)//5: -1*len(historical_returns)//5].astype(np.float32))

x_test1 = torch.from_numpy(X_historical[-1*len(historical_returns)//5:].astype(np.float32))
y_test1 = torch.from_numpy(Y_historical[-1*len(historical_returns)//5:].astype(np.float32))

# move data to GPU
x_train1, y_train1 = x_train1.to(device), y_train1.to(device)
x_val1, y_val1 = x_val1.to(device), y_val1.to(device)
x_test1, y_test1 = x_test1.to(device), y_test1.to(device)

x_train1 = x_train1.reshape(-1, T, 1)
x_test1 = x_test1.reshape(-1, T, 1)
x_val1 = x_val1.reshape(-1, T, 1)

# Define the model parameters
Hidden = 10
model = LSTM(1, Hidden, 1, 1)
model.to(device)
loss_function = nn.MSELoss()

learning_rate = 0.01
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model

no_epochs = 200
train_losses, validation_losses = full_gd(model,
                                    loss_function,
                                    optimizer,
                                    x_train1,
                                    y_train1,
                                    x_val1,
                                    y_val1,
                                    no_epochs)

# Plot training and validation loss

plt.figure(figsize=(12,8))
plt.plot(train_losses, label='train loss')
plt.plot(validation_losses, label='test loss')
plt.legend()
plt.show()

Output:
Well there might be several reasons.

Your task is difficult, or it is hard with the data you have.
Your validation split contains very easy tasks.

Another natural reason to this issue is rising from the dataset size, since validation split is relatively smaller than the training split. Theoretically by random guesses (this is somewhat the model's initial state), you are more likely to fail on large number of guesses.
Your model seems that it couldn't learn, it perform poorly on training data, that is undesired. Keep in mind that RNNs are hard to train though. You can try some potential aids like, increasing the epoch size, making the model more complex. If you can compare your results with another work, you should do it. That'd guide you how good or badly you made the experiment.


Instruction: Pytorch: smarter way to reduce dimension by reshape

Input:
I want to reshape a Tensor by multiplying the shape of first two dimensions.
For example,
1st_tensor: torch.Size([12, 10]) to torch.Size([120])
2nd_tensor: torch.Size([12, 10, 5, 4]) to torch.Size([120, 5, 4])
I.e. The first two dimensions shall be merged into one, while the other dimensions shall remain the same.
Is there a smarter way than
1st_tensor.reshape(-1,)
2nd_tensor.reshape(-1,5,4),
that can adapt to the shape of different Tensors?

Test cases:
import torch
tests = [
    torch.rand(11, 11),
    torch.rand(12, 15351, 6, 4),
    torch.rand(13, 65000, 8)
    ]

Output:
For tensor t, you can use:
t.reshape((-1,)+t.shape[2:])

This uses -1 for flattening the first two dimensions, and then uses t.shape[2:] to keep the other dimensions identical to the original tensor.
For your examples:
>>> tests = [
...     torch.rand(11, 11),
...     torch.rand(12, 15351, 6, 4),
...     torch.rand(13, 65000, 8)
...     ]
>>> tests[0].reshape((-1,)+tests[0].shape[2:]).shape
torch.Size([121])
>>> tests[1].reshape((-1,)+tests[1].shape[2:]).shape
torch.Size([184212, 6, 4])
>>> tests[2].reshape((-1,)+tests[2].shape[2:]).shape
torch.Size([845000, 8])


Instruction: How to find input that maximizes output of a neural network using pytorch

Input:
I have a pytorch network, that have been trained and weights are updated (complete training).
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(1, H)
        self.fc2 = nn.Linear(1, H)
        self.fc3 = nn.Linear(H, 1)

    def forward(self, x, y):
        h1 = F.relu(self.fc1(x)+self.fc2(y))
        h2 = self.fc3(h1)
        return h2

After training, I want to maximize the output of the network with respect to input. In other words, I want to optimize the input to maximize the neural network output, without changing weights. How can I achieve that.
My trial, but it doesn't make sense:
in     = torch.autograd.Variable(x)
out    = Net(in)
grad   = torch.autograd.grad(out, input)

Output:
Disable gradients for the network.
Set your input tensor as a parameter requiring grad.
Initialize an optimizer wrapping the input tensor.
Backprop with some loss function and a goal tensor
...
Profit!

import torch

f = torch.nn.Linear(10, 5)
f.requires_grad_(False)
x = torch.nn.Parameter(torch.rand(10), requires_grad=True)
optim = torch.optim.SGD([x], lr=1e-1)
mse = torch.nn.MSELoss()
y = torch.ones(5)  # the desired network response

num_steps = 5  # how many optim steps to take
for _ in range(num_steps):
   loss = mse(f(x), y)
   loss.backward()
   optim.step()
   optim.zero_grad()

But make sure that your goal tensor is well defined wrt. the network's monotonicity, otherwise you might end up with nans.


Instruction: ValueError: only one element tensors can be converted to Python scalars when using torch.Tensor on list of tensors

Input:
I have a list of tensors:
object_ids = [tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.])]

Intuitively, it seems like I should be able to create a new tensor from this:
torch.as_tensor(object_ids, dtype=torch.float32)

But this does NOT work. Apparently, torch.as_tensor and torch.Tensor can only turn lists of scalars into new tensors. it cannot turn a list of d-dim tensors into a d+1 dim tensor.

Output:
You can use torch.stack.
In your example:
>>> object_ids = [tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.])]
>>> torch.stack(object_ids)
tensor([[2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.],
        [2., 3.]])


Instruction: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [640]] is at version 4;

Input:
I want to use pytorch DistributedDataParallel for adversarial training. The loss function is trades.The code can run in DataParallel mode. But in DistributedDataParallel mode, I got this error.
When I change the loss to AT, it can run successfully. Why can't run with trades loss? The two loss functions are as follows：
-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/data/zsd/defense/my_adv_training/muti_gpu_2.py", line 170, in main_worker
    train_loss = train(train_loader, model, optimizer, epoch, local_rank, args)
  File "/data/zsd/defense/my_adv_training/muti_gpu_2.py", line 208, in train
    loss = trades(model,x, y,optimizer, args.epsilon,args.step_size,args.num_steps,beta=6.0)
  File "/data/zsd/defense/my_adv_training/loss_functions.py", line 137, in trades
    loss_kl.backward()
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/lthpc/.conda/envs/bba/lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [640]] is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

.
for i, (x, y) in enumerate(train_loader):
    # measure data loading time
    x,y = x.cuda(local_rank, non_blocking=True),  y.cuda(local_rank, non_blocking=True)

    loss = trades(model,x, y,optimizer, args.epsilon,args.step_size,args.num_steps,beta=6.0)
    torch.distributed.barrier()

    optimizer.zero_grad()
    loss.backward(retain_graph=True)
    optimizer.step()

def trades():
    model.eval()
    criterion_kl = nn.KLDivLoss(reduction='sum')
    x_adv = x.detach() + 0.001 * torch.randn_like(x).detach()
    nat_output = model(x)
    for _ in range(num_steps):
        x_adv.requires_grad_()
        with torch.enable_grad():
            loss_kl = criterion_kl(F.log_softmax(model(x_adv), dim=1),
                               F.softmax(nat_output, dim=1))
        loss_kl.backward()
        eta = step_size * x_adv.grad.sign()
        x_adv = x_adv.detach() + eta
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)
    model.train()
    x_adv = Variable(x_adv, requires_grad=False)
    optimizer.zero_grad()
    # calculate robust loss
    logits = model(x)
    loss_natural = nn.CrossEntropyLoss()(logits, y)
    loss_robust = (1.0 / x.size(0)) * criterion_kl(F.log_softmax(model(x_adv), dim=1),
                           F.softmax(logits, dim=1))
    loss = loss_natural + beta * loss_robust
    return loss

def AT():
    model.eval()
    x_adv = x.detach() + torch.from_numpy(np.random.uniform(-epsilon,
epsilon, x.shape)).float().cuda()
    x_adv = torch.clamp(x_adv, 0.0, 1.0)
    for k in range(num_steps):
        x_adv.requires_grad_()
        output = model(x_adv)
        model.zero_grad()
        with torch.enable_grad():
            loss = nn.CrossEntropyLoss()(output, y)
        loss.backward()
        eta = step_size * x_adv.grad.sign()
        x_adv = x_adv.detach() + eta
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)
    x_adv = Variable(x_adv, requires_grad=False)

    model.train()
    logits_adv = model(x_adv)
    loss = nn.CrossEntropyLoss()(logits_adv, y)
    return loss

Output:
I changed the code of trades and solved this error. But I don't know why this works.
def trades():
    model.eval()
    criterion_kl = nn.KLDivLoss(reduction='sum')
    x_adv = x.detach() + 0.001 * torch.randn_like(x).detach()
    nat_output = model(x)
    for _ in range(num_steps):
        x_adv.requires_grad_()
        with torch.enable_grad():
            loss_kl = criterion_kl(F.log_softmax(model(x_adv), dim=1),
                           F.softmax(nat_output, dim=1))
        grad = torch.autograd.grad(loss_kl, [x_adv])[0]
        x_adv = x_adv.detach() + step_size * torch.sign(grad.detach())
        x_adv = torch.min(torch.max(x_adv, x - epsilon), x + epsilon)
        x_adv = torch.clamp(x_adv, 0.0, 1.0)
    model.train()
    x_adv = Variable(x_adv, requires_grad=False)
    optimizer.zero_grad()
    # calculate robust loss
    logits = model(x)
    loss_natural = nn.CrossEntropyLoss()(logits, y)
    loss_robust = (1.0 / x.size(0)) * criterion_kl(F.log_softmax(model(x_adv), dim=1),
                       F.softmax(logits, dim=1))
    loss = loss_natural + beta * loss_robust
    return loss


Instruction: TypeError: img should be PIL Image. Got  even though using latest pytorch version

Input:
I had to create a new pytorch environment in Anaconda, I created my code and it worked fine in my old environment. I then created the new environment with the same version of pytorch and cuda 10.1 (but then updated to cuda 11, same as the old one. WHen i try to run the same code i get the error:
TypeError: img should be PIL Image. Got <class 'torch.Tensor'>
When trying to apply any transformation to my tensors, for example the following code gives me an error:
def randRoll(batch, deg):
rotator = torchvision.transforms.RandomRotation(deg)
batch = rotator(batch)
return batch

Nothing has changed and I can't understand why i would get this.
Any suggestions?

Output:
Solved
Turns out somehow torchvision 0.2.2 had been installed instead of the latest 0.9.1 (which my other environment used).
This was solved by uninstalling torchvision using
conda remove torchvision

then installing torchvision using pip (using conda install gave me version 0.2.2)
pip install torchvision

I also had to reinstall six using pip.


Instruction: Pixelwise regression. How to go from Nx1xHxW to Nx3xHxW?

Input:
I have a Nx1xHxW feature maps. I need to add a second head that generates Nx3xHxW representing pixel wise regression with a triplet for each pixel .
The question is: how would you go from Nx1xHxW to Nx3xHxW? A fully connected layer would be too expensive in terms of introduced parameters.
That's what I am trying a 1x1x3 convolutional filter with stride 1 defined in PyTorch as nn.Conv2d(1, 3, (1, 1), stride=1, bias=True) but results does not seem encouraging. Any suggestion would be welcome.
Best

Output:
You can expand the dimension of the data at any point in the forward function with non-parametric operations to force the output into this shape. For instance:
def forward(input):
       input = input.repeat(1,3,1,1)
       output = self.layers(input)

       return output

or:
def forward(input):
    intermediate = self.layers(input)
    intermediate.repeat(1,3,1,1)
    output = self.more_layers(intermediate)

    return output

Theoretically, there is some nonlinear function that produces the 3d pixelwise output given a 1-dimensional input. You can try and learn this nonlinear function using a series of NN layers, but, as you indicated above, this may not give great results and moreover may be difficult to learn well. Instead, you can simply expand the input at some point so that you are instead learning a 3d to 3d pixelwise nonlinear function with NN layers. torch.repeat and other similar operations are differentiable so shouldn't cause an issue with learning.


Instruction: calculating accuracy (in lstm model)

Input:
I am trying to get my accuracy, and I have this code:
            num_correct = 0.0
            for inputs, labels in dataloader(
            valid_features, valid_labels, batch_size=batch_size, sequence_length=20):

                top_val, top_class = torch.exp(output).topk(1)
                num_correct += torch.sum(top_class.squeeze() == labels)
#...
            print(#...,
                  "Accuracy: {:.3f}".format(num_correct*1.0 / len(valid_labels) *1.0)

It always prints 0.000, so I decided to print the raw values going into num_correct:
print(top_class.squeeze(), labels)
tensor([ 1,  3,  3,  ...,  3,  4,  3], device='cuda:0') tensor([ 1,  1,  3,  ...,  3,  3,  3], device='cuda:0')
tensor([ 4,  3,  1,  ...,  4,  4,  3], device='cuda:0') tensor([ 4,  3,  1,  ...,  4,  4,  3], device='cuda:0')
tensor([ 2,  4,  2,  ...,  4,  4,  4], device='cuda:0') tensor([ 3,  4,  1,  ...,  4,  4,  4], device='cuda:0')
tensor([ 0,  1,  3,  ...,  2,  3,  0], device='cuda:0') tensor([ 0,  1,  3,  ...,  2,  2,  0], device='cuda:0')

these appear pretty highly accurate. So .. I could extract this out to numpy and be done with it, but there is a pytorch way.

Output:
This works:
num_correct += torch.sum(torch.eq(top_class.squeeze(), labels)).item()


Instruction: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach()

Input:
I'm new on PyTorch and I'm trying to code with it
so I have a function called OH which tack a number and return a vector like this
  def OH(x,end=10,l=12):
    x = T.LongTensor([[x]])
    end = T.LongTensor([[end]])
    one_hot_x = T.FloatTensor(1,l)
    one_hot_end = T.FloatTensor(1,l)
    first=one_hot_x.zero_().scatter_(1,x,1)
    second=one_hot_end.zero_().scatter_(1,end,1)
    vector=T.cat((one_hot_x,one_hot_end),dim=1)
    return vector
OH(0)
output:
tensor([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 1., 0.]])

now I have a NN that takes this output and return number but this warning always appear in my compiling
online.act(OH(obs))
output:
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:17: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
4

I tried to to use online.act(OH(obs).clone().detach()) but it give me the same warning
and the code works fine and give good results but I need to understand this warning
Edit
the following is my NN that has the act function
class Network(nn.Module):
  def __init__(self,lr,n_action,input_dim):
    super(Network,self).__init__()
    self.f1=nn.Linear(input_dim,128)
    self.f2=nn.Linear(128,64)
    self.f3=nn.Linear(64,32)
    self.f4=nn.Linear(32,n_action)
    #self.optimizer=optim.Adam(self.parameters(),lr=lr)
    #self.loss=nn.MSELoss()
    self.device=T.device('cuda' if T.cuda.is_available() else 'cpu')
    self.to(self.device)

  def forward(self,x):
    x=F.relu(self.f1(x))
    x=F.relu(self.f2(x))
    x=F.relu(self.f3(x))
    x=self.f4(x)
    return x

  def act(self,obs):
    state=T.tensor(obs).to(device)
    actions=self.forward(state)
    action=T.argmax(actions).item()

    return action

Output:
the problem is that you are receiving a tensor on the act function on the Network and then save it as a tensor
just remove the tensor in the action like this
  def act(self,obs):
    #state=T.tensor(obs).to(device)
    state=obs.to(device)
    actions=self.forward(state)
    action=T.argmax(actions).item()


Instruction: How to convert this nested JSON file in pandas df?

Input:
I have to deal with this database for a project:
In particular, I need to obtain a pandas df to formatting these data as input for a neural network in an NLP task. The Json format is the following:
json file
├── "data"
│   └── [i]
│       ├── "paragraphs"
│       │   └── [j]
│       │       ├── "context": "paragraph text"
│       │       └── "qas"
│       │           └── [k]
│       │               ├── "answers"
│       │               │   └── [l]
│       │               │       ├── "answer_start": N
│       │               │       └── "text": "answer"
│       │               ├── "id": "&lt;uuid&gt;"
│       │               └── "question": "paragraph question?"
│       └── "title": "document id"
└── "version": 1.1

I tried hard using .json_normalize method but I can't get any results. I've noticed that most of my attempts (those which don't end with an error) end with recognizing as indexes just "data" and "version" and as the only object the rest of the text, as here:
f = open("SQuAD_it-test.json", "r",encoding="Latin-1" )

data = json.load(f)

df = pd.json_normalize(data)
df.sample(1)

                                         data         version
0   [{'paragraphs': [{'qas': [{'question': 'Quando...   1.1

And if I try to visualize more samples an error occurs to me which tells the population is only 1.
My desired output is something like this, selecting the indexes to use which can be at different level of the tree:
df.sample(5)

        title    context   question  text  answer_start
        str1     str6      str11     str16    N1
        str2     str7      str12     str17    N2
        str3     str8      str13     str13    N3
        str4     str9      str14     str18    N4
        str5     str10     str15     str19    N5

I've also worked on the argument of .json_normalize.
But I'm not able to fully comprehend the explanation. Could you help me?

Output:
Since the given json has many nested fileds, we can use record_path and meta arguments to get the desired dataframe:
df = pd.json_normalize(data, record_path=['data', 'paragraphs', 'qas', 'answers'],
     meta=[['data','title'], ['data', 'paragraphs','context'],
     ['data', 'paragraphs', 'qas','question']])

Note that the output keys will not be in the exact order as given in the desired output table. Also, keys will have slightly different (fully qualified) names.
print(df.keys())

Ouput:
Index(['text', 'answer_start', 'data.title', 'data.paragraphs.context',
   'data.paragraphs.qas.question'],
  dtype='object')


Instruction: Training a model on GPU is very slow

Input:
I am using A100-SXM4-40GB Gpu but training is terribly slow. I tried two models, a simple classification on cifar and a Unet on Cityscapes. I tried my code on other GPUs and it worked totally fine, but I do not know why training on this high capacity GPU is super slow.
I would appreciate any help.
Here are some other properties of GPUs.
GPU 0: A100-SXM4-40GB
GPU 1: A100-SXM4-40GB
GPU 2: A100-SXM4-40GB
GPU 3: A100-SXM4-40GB

Nvidia driver version: 460.32.03
cuDNN version: Could not collect

Output: Thank you for your answer. Before trying your answer, I decided to uninstall anaconda and reinstall it and this solved the problem.


Instruction: pytorch change input image size

Input:
I am new to pytorch and I am following a tutorial but when i try to modify the code to use 64x64x3 images instead of 32x32x3 images, i get a buch of errors. Here is the code from the tutorial:
import torch
from torch.utils.data import DataLoader
import torchvision
import torchvision.transforms as transforms
from torchvision.datasets import ImageFolder

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Resize(32),
     transforms.RandomCrop(32),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = ImageFolder("Train", transform=transform)
trainloader = DataLoader(trainset, shuffle=True, batch_size=batch_size, num_workers=0)

classes = ('Dog', 'Cat')

import matplotlib.pyplot as plt
import numpy as np

# functions to show an image

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(batch_size)))

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

print("training started")

from tqdm import tqdm

for epoch in range(5):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in tqdm(enumerate(trainloader, 0), desc=f"epoch: {epoch + 1}"):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

PATH = './net.pth'
torch.save(net.state_dict(), PATH)

If I change the ´transforms.Resize(32)´ and ´transforms.RandomCrop(32)´ to 64 (to get 64x64x3 images), I get this error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\Documents\pyth\classifier\train_classifier.py in <module>
     86
     87         # forward + backward + optimize
---> 88         outputs = net(inputs)
     89         loss = criterion(outputs, labels)
     90         loss.backward()

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Documents\pyth\classifier\train_classifier.py in forward(self, x)
     57         x = self.pool(F.relu(self.conv1(x)))
     58         x = self.pool(F.relu(self.conv2(x)))
---> 59         x = x.view(-1, 10816+1)
     60         x = F.relu(self.fc1(x))
     61         x = F.relu(self.fc2(x))

RuntimeError: shape '[-1, 10817]' is invalid for input of size 10816
´´´

and if i try to change the parameters of ´x.view(...)´ i get this error

´´´
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~\Documents\pyth\classifier\train_classifier.py in <module>
     86
     87         # forward + backward + optimize
---> 88         outputs = net(inputs)
     89         loss = criterion(outputs, labels)
     90         loss.backward()

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Documents\pyth\classifier\train_classifier.py in forward(self, x)
     58         x = self.pool(F.relu(self.conv2(x)))
     59         x = x.view(-1, 16 * 2 * 5 * 5)
---> 60         x = F.relu(self.fc1(x))
     61         x = F.relu(self.fc2(x))
     62         x = self.fc3(x)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\Anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

~\Anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: mat1 and mat2 shapes cannot be multiplied (2x800 and 400x120)
´´´

Output:
I think this should work because after performing 2nd Pooling operation the output feature map is coming N x C x 13 x 13

self.fc1 = nn.Linear(16 * 13 * 13, 120)

x = x.view(-1, 16 * 13 * 13)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 13 * 13, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 13 * 13)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


Instruction: RuntimeError: Given groups=1, weight of size [32, 1, 5, 5], expected input[256, 3, 256, 256] to have 1 channels, but got 3 channels instead

Input:
I am trying to run following code but getting an error:
import torch.nn as nn
import torch.nn.functional as F

class EmbeddingNet(nn.Module):
    def __init__(self):
        super(EmbeddingNet, self).__init__()
        self.convnet = nn.Sequential(nn.Conv2d(1, 32, 5), nn.PReLU(),
                                     nn.MaxPool2d(2, stride=2),
                                     nn.Conv2d(32, 64, 5), nn.PReLU(),
                                     nn.MaxPool2d(2, stride=2))

        self.fc = nn.Sequential(nn.Linear(64 * 4 * 4, 256),
                                nn.PReLU(),
                                nn.Linear(256, 256),
                                nn.PReLU(),
                                nn.Linear(256, 2)
                                )

    def forward(self, x):
        output = self.convnet(x)
        output = output.view(output.size()[0], -1)
        output = self.fc(output)
        return output

    def get_embedding(self, x):
        return self.forward(x)

class EmbeddingNetL2(EmbeddingNet):
    def __init__(self):
        super(EmbeddingNetL2, self).__init__()

    def forward(self, x):
        output = super(EmbeddingNetL2, self).forward(x)
        output /= output.pow(2).sum(1, keepdim=True).sqrt()
        return output

    def get_embedding(self, x):
        return self.forward(x)'''enter code here

Output:
Error is very simple .Its saying instead of 1 channel you have given 3 channel images.
one change would be in this block
class EmbeddingNet(nn.Module):
  def __init__(self):
    super(EmbeddingNet, self).__init__()
    self.convnet = nn.Sequential(nn.Conv2d(3, 32, 5),  #instead of 1 i have made it 3
                                 nn.PReLU(),
                                 nn.MaxPool2d(2, stride=2),
                                 nn.Conv2d(32, 64, 5), nn.PReLU(),
                                 nn.MaxPool2d(2, stride=2))

    self.fc = nn.Sequential(nn.Linear(64 * 4 * 4, 256),
                            nn.PReLU(),
                            nn.Linear(256, 256),
                            nn.PReLU(),
                            nn.Linear(256, 2)
                            )

EDIT to next error:
change to this
self.fc = nn.Sequential(nn.Linear(64 * 61 * 61, 256), #here is the change
                    nn.PReLU(),
                    nn.Linear(256, 256),
                    nn.PReLU(),
                    nn.Linear(256, 2)
                    )


Instruction: Decreasing number of nodes each layers in torch.nn.lstm

Input:
Is there an easy way to decrease the number of nodes in each layer by a factor? I don't see this option on the documentation page, perhaps there is a similar function I can use though instead of manually defining each layer?
    self.lstm = nn.LSTM(
        input_size=input_size,
        hidden_size=hidden_size,
        num_layers=num_layers,
        batch_first=True,
        dropout=0.2,
    )  # lstm

Output:
Not that I know of, but writing it from scratch is straightforward:
def _constant_scale(initial: int, factor: int) -> int:
   return initial//factor

class StackedLSTM(Module):
   def __init__(self, input_size: int, hidden_sizes: list[int], *args, **kwargs):
       super(StackedLSTM, self).__init__()
       self.layers = ModuleList([LSTM(input_size=xs, hidden_size=hs, *args, **kwargs) for xs, hs in zip([input_size] + hidden_sizes, hidden_sizes)])

   def forward(self, x: Tensor, hc: Optional[tuple[Tensor, Tensor]] = None) -> Tensor:
       for layer in self.layers:
           x, _ = layer(x, hc)
           hc = None
       return x

hidden_sizes = [_constant_scale(300, 2**i) for i in range(3)]
sltm = StackedLSTM(100, hidden_sizes)
x = torch.rand(10, 32, 100)
h = torch.rand(1, 32, 300)
c = torch.rand(1, 32, 300)
out = sltm(x, (h, c))
print(out.shape)
# torch.Size([10, 32, 75])