Instruction: What's the proper way to update a leaf tensor's values (e.g. during the update step of gradient descent)

Input:
Toy Example
Consider this very simple implementation of gradient descent, whereby I attempt to fit a linear regression (mx + b) to some toy data.
import torch

# Make some data
torch.manual_seed(0)
X = torch.rand(35) * 5
Y = 3 * X + torch.rand(35)

# Initialize m and b
m = torch.rand(size=(1,), requires_grad=True)
b = torch.rand(size=(1,), requires_grad=True)

# Pass 1
yhat = X * m + b    #  Calculate yhat
loss = torch.sqrt(torch.mean((yhat - Y)**2)) # Calculate the loss
loss.backward()     # Reverse mode differentiation
m = m - 0.1*m.grad  # update m
b = b - 0.1*b.grad  # update b
m.grad = None       # zero out m gradient
b.grad = None       # zero out b gradient

# Pass 2
yhat = X * m + b    #  Calculate yhat
loss = torch.sqrt(torch.mean((yhat - Y)**2)) # Calculate the loss
loss.backward()     # Reverse mode differentiation
m = m - 0.1*m.grad  # ERROR

The first pass works fine, but the second pass errors on the last line, m = m - 0.1*m.grad.
Error
/usr/local/lib/python3.7/dist-packages/torch/_tensor.py:1013: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations. (Triggered internally at  aten/src/ATen/core/TensorBody.h:417.)
  return self._grad

My understanding of why this happens is that, during Pass 1, this line
m = m - 0.1*m.grad

copies m into a brand new tensor (i.e. a totally separate block of memory). So, it goes from being a leaf tensor to a non-leaf tensor.
# Pass 1
...
print(f"{m.is_leaf}")  # True
m = m - 0.1*m.grad
print(f"{m.is_leaf}")  # False

So, how does one perform an update?
I've seen it mentioned that one could use something along the lines of m.data = m - 0.1*m.grad, but I haven't seen much discussion about this technique.

Output:
You're observation is correct, in order to perform the update you should:

Apply the modification with in-place operators.

Wrap the calls with torch.no_grad context manager.

For instance:
with torch.no_grad():
    m -= 0.1*m.grad  # update m
    b -= 0.1*b.grad  # update b


Instruction: How to modify a 2d tensor to 3d in libtorch?

Input:
I have a 2d array vector<vector>, I have coverted it to tensor, but how to modify the dimension of the tensor, I want to modify the dimension from 2d to 3d?
std::vector<std::vector<float>> voice(434, std::vector<float>(80))
ifstream fp("data.txt");
if (! fp) {
    cout << "Error, file couldn't be opened" << endl;
    return 1;
}
for(int i=0;i<80;i++)
{
    for(int j=0;j<434;j++)
    {
        if ( ! fp )
        {
            std::cout << "read error" << std::endl;
            break;
        }
        fp >> voice[i][j]
    }
}

auto options = torch::TensorOptions().dtype(at::kDouble);
auto tensor = torch::zeros({80,434}, options);
for (int i = 0; i < 80; i++)
    tensor.slice(0, i,i+1) = torch::from_blob(vect[i].data(), {434}, options);

Now the tensor is 80 * 434, how can I add one dimension in this tensor to 3d, I want 1 * 80 * 434

Output:
auto tensor = torch::zeros({80,434}, options);

followed by this line
auto tensor = tensor.view({1, 80, 434});

I would recommend to create another variable instead of tensor in the second line, like:
auto transformed_tensor = tensor.view({1, 80, 434});


Instruction: Print Bert model summary using Pytorch

Input:
Hi I would like to print the model summary of my BERT model for text classification. I am using command print(summary(model, inputsize=(channels, height, width)). I would like to know what would be the dimensions of input_size in case of text classification?
I have use print(model) as well but the output is confusing and I want to see the output in the layered form.
Below is my model summary.
BertClassifier(
  (bert): BertModel(
    (embeddings): BertEmbeddings(
      (word_embeddings): Embedding(28996, 768, padding_idx=0)
      (position_embeddings): Embedding(512, 768)
      (token_type_embeddings): Embedding(2, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): BertEncoder(
      (layer): ModuleList(
        (0): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (1): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (2): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (3): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (4): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (5): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (6): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (7): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (8): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (9): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (10): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (11): BertLayer(
          (attention): BertAttention(
            (self): BertSelfAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): BertSelfOutput(
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): BertIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
          )
          (output): BertOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
    (pooler): BertPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (dropout): Dropout(p=0.5, inplace=False)
  (linear1): Linear(in_features=768, out_features=256, bias=True)
  (linear2): Linear(in_features=256, out_features=141, bias=True)
  (relu): ReLU()
)

Output:
I used torch-summary module-
pip install torch-summary

summary(model,input_size=(768,),depth=1,batch_dim=1, dtypes=[‘torch.IntTensor’])


Instruction: Mask to bounding box for narrow features (cracks)

Input:
I am trying to adapt the code detailed here https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html for crack detection using the  Kanha dataset (https://github.com/khanhha/crack_segmentation).
My problem is that when converting masks to bounding boxes, some are empty - and thus invalid.

ValueError: All bounding boxes should have positive height and width.
Found invalid box

Should I add a minimum area clause before appending boxes[], boxes.append([xmin, ymin, xmax, ymax])? Or does anyone have another idea how to make progress?
import os
import numpy as np
import torch
from PIL import Image

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

from engine import train_one_epoch, evaluate
import utils
import transforms as T

class CracksDataset(object):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "images"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "masks"))))

    def __getitem__(self, idx):
        # load images and masks
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        mask_path = os.path.join(self.root, "masks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

    return model

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def main():
    # train on the CPU
    device = torch.device('cpu')
    #device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    # our dataset has two classes only - background and person
    num_classes = 2
    # use our dataset and defined transformations
    dataset = CracksDataset('datasets', get_transform(train=True))
    dataset_test = CracksDataset('datasets', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=2, shuffle=True, num_workers=1,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=2, shuffle=False, num_workers=1,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = get_model_instance_segmentation(num_classes)

    # move model to the right device
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    print("That's it!")

if __name__ == "__main__":
    main()

Output:
I have done exactly the same for a project and I believe the issue comes from detection that will have a width or a height of 1 pixel only.
In this case, xmin = xmax or ymin = ymax which creates the issue.
I just added 1 to my xmax and ymax to ensure boxes are never empty.
boxes.append([xmin, ymin, xmax+1, ymax+1])


Instruction: How to use yolov5 api with flask offline?

Input:
I was able to run the Flask app with yolov5 on a PC with an internet connection. I followed the steps mentioned in yolov5 docs and used this file: yolov5/utils/flask_rest_api/restapi.py,
But I need to achieve the same offline(On a particular PC). Now the issue is, when I am using the following:
model = torch.hub.load("ultralytics/yolov5", "yolov5", force_reload=True)

It tries to download model from internet. And throws an error.
Urllib.error.URLError: <urlopen error [Errno - 2] name or service not known>

How to get the same results offline.
Thanks in advance.

Output:
If you want to run detection offline, you need to have the model already downloaded.
So, download the model (for example yolov5s.pt) from https://github.com/ultralytics/yolov5/releases and store it for example to the yolov5/models.
After that, replace
# model = torch.hub.load("ultralytics/yolov5", "yolov5s", force_reload=True)  # force_reload to recache

with
model = torch.hub.load(r'C:\Users\Milan\Projects\yolov5', 'custom', path=r'C:\Users\Milan\Projects\yolov5\models\yolov5s.pt', source='local')

With this line, you can run detection also offline.
Note: When you start the app for the first time with the updated torch.hub.load, it will download the model if not present (so you do not need to download it from https://github.com/ultralytics/yolov5/releases).


Instruction: How can I get the gradients of two losses in pytorch

Input:
I am trying to get the gradients of two losses in the following code snippet but all I get is None (AttributeError: ‘NoneType’ object has no attribute ‘data’)
img = img.to(device)
#img.requires_grad = True
input = model(img)
input_prediction = input.max(1, keepdim=True)[1]

btarget = torch.tensor(2).unsqueeze(0).to(device)
x_prime.requires_grad = True
x_prime_output = model(x_prime)
x_prime_pred =  x_prime_output.max(1, keepdim=True)[1]

l_target_loss = F.nll_loss(x_prime_output, btarget)
# model.zero_grad()
l_target_loss.retain_grad()
l_target_loss.backward(retain_graph = True)
target_grad = l_target_loss.grad.data

l_argmax_loss = F.nll_loss(x_prime_output, input_prediction.squeeze(0))
l_argmax_loss.retain_grad()
l_argmax_loss.backward()
l_argmax_grad = l_argmax_loss.grad.data

when I tried getting target_grad and l_argmax_grad, I get None.
Edit: I am actually trying to get the gradient of the l_target_loss w.r.t to the input x and the gradient of the  l_argmax_loss w.r.t to the input x as well

Output:
You need to retain the gradient on that tensor with retain_grad, by default it is not cached in memory:
>>> l_target_loss.retain_grad()
>>> l_target_loss.backward(retain_graph=True)


Instruction: EarlyStopping callback in PyTorch Lightning problem

Input:
I try to train Neural Network model in PyTorch Lightning and training fails on validation step where it executes EarlyStopping callback.
The relevant part of the model is below. See, in particular, validation_step which must log the metrics necessary for Early stopping.
class DialogActsLightningModel(pl.LightningModule):

    def __init__(self, config):
        super().__init__()

        self.config = config

        self.model = ContextAwareDAC(
            model_name=self.config['model_name'],
            hidden_size=self.config['hidden_size'],
            num_classes=self.config['num_classes'],
            device=self.config['device']
        )
        self.tokenizer = AutoTokenizer.from_pretrained(config['model_name'])

    def forward(self, batch):
        logits = self.model(batch)
        return logits

    def validation_step(self, batch, batch_idx):
        input_ids, attention_mask, targets = batch['input_ids'], batch['attention_mask'], batch['label'].squeeze()
        logits = self(batch)
        loss = F.cross_entropy(logits, targets)
        acc = accuracy_score(targets.cpu(), logits.argmax(dim=1).cpu())
        f1 = f1_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])
        precision = precision_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])
        recall = recall_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])
        return {"val_loss": loss, "val_accuracy": torch.tensor([acc]), "val_f1": torch.tensor([f1]),
                "val_precision": torch.tensor([precision]), "val_recall": torch.tensor([recall])}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        avg_acc = torch.stack([x['val_accuracy'] for x in outputs]).mean()
        avg_f1 = torch.stack([x['val_f1'] for x in outputs]).mean()
        avg_precision = torch.stack([x['val_precision'] for x in outputs]).mean()
        avg_recall = torch.stack([x['val_recall'] for x in outputs]).mean()
        wandb.log({"val_loss": avg_loss, "val_accuracy": avg_acc, "val_f1": avg_f1, "val_precision": avg_precision,
                   "val_recall": avg_recall})
        return {"val_loss": avg_loss, "val_accuracy": avg_acc, "val_f1": avg_f1, "val_precision": avg_precision,
                "val_recall": avg_recall}

When I run training in the following way:
from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint
from pytorch_lightning.loggers import WandbLogger
import pytorch_lightning as pl
import os

from trainer import DialogActsLightningModel
import wandb

wandb.init()

logger = WandbLogger(
    name="model_name",
    entity='myname',
    save_dir=config["save_dir"],
    project=config["project"],
    log_model=True,
)
early_stopping = EarlyStopping(
    monitor="val_accuracy",
    min_delta=0.001,
    patience=5,
)

model = DialogActsLightningModel(config=config)

trainer = pl.Trainer(
    logger=logger,
    gpus=[0],
    checkpoint_callback=True,
    callbacks=[early_stopping],
    default_root_dir=MODELS_DIRECTORY,
    max_epochs=config["epochs"],
    precision=config["precision"],
    limit_train_batches=10, # run for only 10 batches, debug mode
    limit_test_batches=10,
    limit_val_batches=10
)

trainer.fit(model)

I've got an error, but the model should have calculated and logged the metric "val_accuracy" during validation step.
Epoch 0: 100%
20/20 [00:28<00:00, 1.41s/it, loss=1.95]

/opt/conda/lib/python3.9/site-packages/pytorch_lightning/callbacks/early_stopping.py in _validate_condition_metric(self, logs)
    149         if monitor_val is None:
    150             if self.strict:
--> 151                 raise RuntimeError(error_msg)
    152             if self.verbose > 0:
    153                 rank_zero_warn(error_msg, RuntimeWarning)

RuntimeError: Early stopping conditioned on metric `val_accuracy` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: ``

What I am doing wrong? How to fix it?

Output:
if you use pytorch-lightning latest version you should want to log the val_accuracy or val_loss while you calling early stopping or similar functions. for more please check out the code below.i think this will definitely helpful for you...
def validation_step(self, batch, batch_idx):
    input_ids, attention_mask, targets = batch['input_ids'], batch['attention_mask'], batch['label'].squeeze()
    logits = self(batch)
    loss = F.cross_entropy(logits, targets)
    acc = accuracy_score(targets.cpu(), logits.argmax(dim=1).cpu())
    f1 = f1_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])
    precision = precision_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])
    recall = recall_score(targets.cpu(), logits.argmax(dim=1).cpu(), average=self.config['average'])

    ##########################################################################
    ##########################################################################
    self.log("val_accuracy", torch.tensor([acc])     # try this line
    ##########################################################################
    ##########################################################################

    return {"val_loss": loss, "val_accuracy": torch.tensor([acc]), "val_f1": torch.tensor([f1]),
            "val_precision": torch.tensor([precision]), "val_recall": torch.tensor([recall])}

If This Post is Useful Please Up vote


Instruction: Applying a 2D function to a 4D tensor PyTorch

Input:
I got a 2D function that takes a matrix - 2D tensor with shape (28, 28)
and I got a tensor, lets say (64, 10, 28, 28) - it's a tensor that contains a batch of 64 images that passed through a (10 kernels) conv2d layer.
Now, I want to activate on the last two dimentions of the tensor, the (28,28) bit, a 2D function.
Now I did that in a very inefficient way:
def activation_func(input):

    for batch_idx in range(input.shape[0]):
        for channel_inx in range(input.shape[1]):
            input[batch_idx][channel_inx] = 2D_function(input[batch_idx][channel_inx])

    return input

which is highly inefficient as I noticed.
is there any way of doing this efficiently?
I can write the entire code If necessary
EDIT:
def 2D_function(input):
    global indices # yes I know, I will remove this global stuff later
    # indices = [(i, j) for i in range(1, 28, 4) for j in range(1, 28, 4)]

    for x, y in indices:
        relu_decision = relu(input[x, y]) # standard relu - relu(x)=(x>1)*x
        if not relu_decision:
            # zero out the patch
            input[x - 1: x + 3, y - 1: y + 3] = 0
    return input

Output:
In such cases, I use a Kronecker product trick:
import torch

torch.set_printoptions(linewidth=200)  # you can better see how the mask is shaped

# simulating an input
input = torch.rand(1, 1, 28, 28) - 0.5

ids = torch.meshgrid((torch.arange(1, 28, 4), torch.arange(1, 28, 4)))

# note that relu(x) = (x > 0.) * x, so adjust it to your needs
relus = torch.nn.functional.relu(input[(slice(None), slice(None), *ids)]).to(bool)

A = torch.ones(4, 4)
# generate a block matrix with ones in positions where blocks are set to 0 in correspondence of relus = 0
mask = torch.kron(relus, A)
print(mask.shape)
output = input * mask

print(mask[0, 0])
print(output[0, 0])


Instruction: Pytorch error - RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Float'

Input:
I am getting an error very similar to this.
My error as below:
-----------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-85-5a223a19e3f5> in <module>

      8                                   save_run = 'Yes',
      9                                   return_progress_dict = 'Yes',
---> 10                                   hide_text = 'No')

<ipython-input-84-023bc49b2138> in train_CNN(model, optimizer, train_dataloader, epochs, run_number, val_dataloader, save_run, return_progress_dict, hide_text)
     63             print(labels[0].dtype)
     64             print("------------")
---> 65             loss = F.cross_entropy(probs, labels)
     66
     67             total_loss += loss.item()

/usr/local/anaconda/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847
   2848

RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Float'

I am making sure that both my labels and probs have same datatype - float
probs[0].dtype:
torch.float32
probs[0]:
tensor([-0.8244, -0.5771], device='cuda:0', grad_fn=<SelectBackward0>)
probs[0].dtype:
torch.float32
------------
labels:
tensor([0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 1., 0., 0., 1., 1., 1., 1., 1.,
        0., 0., 1., 1., 0., 1., 0., 0., 0., 1., 0., 1., 1., 0., 0., 0., 0., 0.,
        0., 0., 1., 0., 0., 1., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1.,
        1., 1., 0., 0., 1., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 0.,
        0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 0., 1., 0., 1., 0., 0., 1., 0.,
        1., 1., 1., 0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 0., 1., 1., 1., 0.,
        1., 1., 1., 0., 1., 0., 0., 0., 0., 0., 1., 0., 1., 1., 1., 0., 0., 0.,
        0., 0., 0., 1., 0., 0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 0.,
        1., 0., 0., 0., 1., 1.], device='cuda:0')
labels.dtype:
torch.float32
labels[0].dtype:
torch.float32
------------

My function is as below
def train_CNN(model, optimizer, train_dataloader, epochs, run_number,
          val_dataloader=None, save_run=None, return_progress_dict = None, hide_text = None):

    # Tracking lowest validation loss
    lowest_val_loss = float('inf')
    if return_progress_dict == 'Yes':
        progress_dict = {run_number: {'Epoch':[], 'Avg_Training_Loss':[], 'Validation_Loss':[], 'Validation_Accuracy':[]} }

    # Start training loop
    if hide_text != "Yes":
        print("Start training...\n")
        print(f"{'Epoch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}")
        print("-"*60)

    for epoch_i in range(epochs):
        # =======================================
        #               Training
        # =======================================

        # Tracking time and loss
        t0_epoch = time.time()
        total_loss = 0

        # Put the model into the training mode
        model.train()

        for step, batch in enumerate(train_dataloader):
            # Load batch to CPU
            data, labels = tuple(t.to(device) for t in batch)
            #labels = labels.type(torch.LongTensor)   #23Feb2022 casting to long as per https://stackoverflow.com/questions/69742930/runtimeerror-nll-loss-forward-reduce-cuda-kernel-2d-index-not-implemented-for
            #labels.to(device)

            # Zero out any previously calculated gradients
            optimizer.zero_grad()

            # Perform a forward pass. This will return logits.
            probs = model(data)

            # Compute loss and accumulate the loss values
            print("------------")
            print ("probs[0].dtype:")
            print(probs[0].dtype)
            print ("probs[0]:")
            print (probs[0])
            #print (probs)
            probs=probs.type(torch.cuda.FloatTensor)

            #probs=probs.type(torch.cuda.LongTensor)
            print ("probs[0].dtype:")
            print(probs[0].dtype)
            print("------------")
            labels = labels.type(torch.cuda.FloatTensor)
            #labels=labels.type(torch.cuda.LongTensor)

            #labels = labels.type(torch.cuda.DoubleTensor)
            #x_cuda = Variable(x, requires_grad=True).cuda()

            print("labels:")
            print (labels)
            print("labels.dtype:")
            print (labels.dtype)
            print("labels[0].dtype:")
            print(labels[0].dtype)
            print("------------")
            loss = F.cross_entropy(probs, labels)

            total_loss += loss.item()

            # Perform a backward pass to calculate gradients
            loss.backward()

            # Update parameters
            optimizer.step()

any suggestions? I will try to give a reproducible example soon
##########################update 1--------------------
I tried labels=labels.type(torch.cuda.LongTensor). Probs are still float32 and still getting the error RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'

Output:
labels should be of type torch.long and not torch.float32.
I think you just commented out the line that does this casting in your code.


Instruction: Anaconda Xgboost unable to find GPU

Input:
It seems that Anaconda is unable recognise my GPU,
GPU is RTX2070 (Driver version 510.47.03),
system Ubuntu 20.04,
cudatoolkit 11.3.1,
cudnn 8.2.1,
XGboost 1.5.2 via pip install.
When I run XGboost with GPU enable it shows:
XGBoostError: [01:24:12] ../src/gbm/gbtree.cc:531: Check failed: common::AllVisibleGPUs() >= 1 (0 vs. 1) : No visible GPU is found for XGBoost.
I also used GPUtil to check the visible GPU, it is showing 0 GPU.
Below is the output from nvidia-smi

Can anyone help? how come the GPU in the system is no visible to any package in anaconda? (Pytorch, XGBoost etc.)

Output: Appeared to be a driver issue, tried the same code in another computer with two RTX 3090 and got no issue


Instruction: writing a custom dataset to pytorch

Input:
I am having data of numpy arrays with shape (400, 46, 55, 46) here 400 are the samples and 46,55,46 is the image.350 samples for training and remaining 50 for validation
np.max(data[1]), np.min(data[1]), len(data[1])
Output: (2941.0, -43.0, 46)

Now i want to load the data into pytorch model for that i need to write a custom dataloader as i am new to pytorch i am finding hard to wrie can someone help

Output:
You can use a combination of torch.utils.data.TensorData and torch.utils.data.random_split to construct the two datasets and wrap them with torch.utils.data.DataLoader:
>>> data = np.random.rand(400, 46, 55, 46)

# Datasets initialization
>>> ds = TensorDataset(torch.from_numpy(data))
>>> train_ds, valid_ds = random_split(ds, (350, 50))

# Dataloader wrappers
>>> train_dl, valid_dl = DataLoader(train_ds), DataLoader(valid_ds)


Instruction: AttributeError: 'list' object has no attribute 'view' while training network

Input:
I have a pytorch which i am trying to train but i am getting this error AttributeError: 'list' object has no attribute 'view'. Dont know why i am getting this.
sample data
data = np.random.rand(400, 46, 55, 46)
ds = TensorDataset(torch.from_numpy(data))
train_ds, valid_ds = random_split(ds, (350, 50))
train_dl, valid_dl = DataLoader(train_ds), DataLoader(valid_ds)

model
class AutoEncoder(pl.LightningModule):
  def __init__(self):
    super(AutoEncoder, self).__init__()
    self.encoder = nn.Sequential(
        nn.Linear(46*55*46, 400),
        nn.Tanh())
    self.decoder = nn.Sequential(
        nn.Linear(400, 46*55*46),
        nn.Sigmoid())

  def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

  def configure_optimizers(self):
    optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
    return optimizer

  def training_step(self, train_batch, batch_idx):
    x = train_batch
    x = x.view(x.size(0), -1)
    z = self.encoder(x)
    x_hat = self.decoder(z)
    loss = F.mse_loss(x_hat, x)
    self.log('train_loss', loss)
    return loss

  def validation_step(self, val_batch, batch_idx):
    x = val_batch
    x = x.view(x.size(0), -1)
    z = self.encoder(x)
    x_hat = self.decoder(z)
    loss = F.mse_loss(x_hat, x)
    self.log('val_loss', loss)
model = AutoEncoder()

Error
AttributeError                            Traceback (most recent call last)

<ipython-input-18-11e725b78922> in <module>()
      1 trainer = pl.Trainer()
----> 2 trainer.fit(model, train_dl, valid_dl)

16 frames

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloaders, val_dataloaders, datamodule, train_dataloader, ckpt_path)
    739             train_dataloaders = train_dataloader
    740         self._call_and_handle_interrupt(
--> 741             self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    742         )
    743

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    683         """
    684         try:
--> 685             return trainer_fn(*args, **kwargs)
    686         # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7
    687         except KeyboardInterrupt as exception:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    775         # TODO: ckpt_path only in v1.7
    776         ckpt_path = ckpt_path or self.resume_from_checkpoint
--> 777         self._run(model, ckpt_path=ckpt_path)
    778
    779         assert self.state.stopped

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run(self, model, ckpt_path)
   1197
   1198         # dispatch `start_training` or `start_evaluating` or `start_predicting`
-> 1199         self._dispatch()
   1200
   1201         # plugin will finalized fitting (e.g. ddp_spawn will load trained model)

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _dispatch(self)
   1277             self.training_type_plugin.start_predicting(self)
   1278         else:
-> 1279             self.training_type_plugin.start_training(self)
   1280
   1281     def run_stage(self):

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in start_training(self, trainer)
    200     def start_training(self, trainer: "pl.Trainer") -> None:
    201         # double dispatch to initiate the training loop
--> 202         self._results = trainer.run_stage()
    203
    204     def start_evaluating(self, trainer: "pl.Trainer") -> None:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in run_stage(self)
   1287         if self.predicting:
   1288             return self._run_predict()
-> 1289         return self._run_train()
   1290
   1291     def _pre_training_routine(self):

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run_train(self)
   1309             self.progress_bar_callback.disable()
   1310
-> 1311         self._run_sanity_check(self.lightning_module)
   1312
   1313         # enable train mode

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run_sanity_check(self, ref_model)
   1373             # run eval step
   1374             with torch.no_grad():
-> 1375                 self._evaluation_loop.run()
   1376
   1377             self.call_hook("on_sanity_check_end")

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py in run(self, *args, **kwargs)
    143             try:
    144                 self.on_advance_start(*args, **kwargs)
--> 145                 self.advance(*args, **kwargs)
    146                 self.on_advance_end()
    147                 self.restarting = False

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py in advance(self, *args, **kwargs)
    108         dl_max_batches = self._max_batches[dataloader_idx]
    109
--> 110         dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
    111
    112         # store batch level output per dataloader

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py in run(self, *args, **kwargs)
    143             try:
    144                 self.on_advance_start(*args, **kwargs)
--> 145                 self.advance(*args, **kwargs)
    146                 self.on_advance_end()
    147                 self.restarting = False

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py in advance(self, data_fetcher, dataloader_idx, dl_max_batches, num_dataloaders)
    120         # lightning module methods
    121         with self.trainer.profiler.profile("evaluation_step_and_end"):
--> 122             output = self._evaluation_step(batch, batch_idx, dataloader_idx)
    123             output = self._evaluation_step_end(output)
    124

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py in _evaluation_step(self, batch, batch_idx, dataloader_idx)
    215             self.trainer.lightning_module._current_fx_name = "validation_step"
    216             with self.trainer.profiler.profile("validation_step"):
--> 217                 output = self.trainer.accelerator.validation_step(step_kwargs)
    218
    219         return output

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py in validation_step(self, step_kwargs)
    237         """
    238         with self.precision_plugin.val_step_context():
--> 239             return self.training_type_plugin.validation_step(*step_kwargs.values())
    240
    241     def test_step(self, step_kwargs: Dict[str, Union[Any, int]]) -> Optional[STEP_OUTPUT]:

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in validation_step(self, *args, **kwargs)
    217
    218     def validation_step(self, *args, **kwargs):
--> 219         return self.model.validation_step(*args, **kwargs)
    220
    221     def test_step(self, *args, **kwargs):

<ipython-input-12-16d602e3e66b> in validation_step(self, val_batch, batch_idx)
     29   def validation_step(self, val_batch, batch_idx):
     30     x = val_batch
---> 31     x = x.view(x.size(0), -1)
     32     z = self.encoder(x)
     33     x_hat = self.decoder(z)

AttributeError: 'list' object has no attribute 'view'

Output:
As indicated by the error log, it is in this line:
 29   def validation_step(self, val_batch, batch_idx):
 30     x = val_batch
 31     x = x.view(x.size(0), -1)        # here is your problem

x or vali_batch is a list object, and a list does not have an attribute view() since it is not a tensor.  If you want to convert a list to a tensor, you can simply use:
x = torch.tensor(val_batch)

Or you can convert val_batch to a tensor earlier in your code during loading and processing the data.


Instruction: RuntimeError: DataLoader worker exited unexpectedly

Input:
I am new to PyTorch and Machine Learning so I try to follow the tutorial from here:
https://medium.com/@nutanbhogendrasharma/pytorch-convolutional-neural-network-with-mnist-dataset-4e8a4265e118
By copying the code step by step I got the following error for no reason. I tried the program on another computer and it gives syntax error. However, my IDE didn't warn my anything about syntax. I am really confused how I can fix the issue. Any help is appreciated.
RuntimeError: DataLoader worker exited unexpectedly

Here is the code.
import torch
from torchvision import datasets
from torchvision.transforms import ToTensor
import torch.nn as nn
import matplotlib.pyplot as plt
from torch.utils.data import DataLoader
from torch import optim
from torch.autograd import Variable

train_data = datasets.MNIST(
    root='data',
    train=True,
    transform=ToTensor(),
    download=True,
)
test_data = datasets.MNIST(
    root='data',
    train=False,
    transform=ToTensor()
)
print(train_data)
print(test_data)

print(train_data.data.size())
print(train_data.targets.size())

plt.imshow(train_data.data[0], cmap='gray')
plt.title('%i' % train_data.targets[0])
plt.show()

figure = plt.figure(figsize=(10, 8))
cols, rows = 5, 5
for i in range(1, cols * rows + 1):
    sample_idx = torch.randint(len(train_data), size=(1,)).item()
    img, label = train_data[sample_idx]
    figure.add_subplot(rows, cols, i)
    plt.title(label)
    plt.axis("off")
    plt.imshow(img.squeeze(), cmap="gray")
plt.show()

loaders = {
    'train': DataLoader(train_data,
                        batch_size=100,
                        shuffle=True,
                        num_workers=1),

    'test': DataLoader(test_data,
                       batch_size=100,
                       shuffle=True,
                       num_workers=1),
}

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=1,
                padding=2,
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        # fully connected layer, output 10 classes
        self.out = nn.Linear(32 * 7 * 7, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        # flatten the output of conv2 to (batch_size, 32 * 7 * 7)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output, x  # return x for visualization

cnn = CNN()
print(cnn)

loss_func = nn.CrossEntropyLoss()
print(loss_func)
optimizer = optim.Adam(cnn.parameters(), lr=0.01)
print(optimizer)
num_epochs = 10

def train(num_epochs, cnn, loaders):
    cnn.train()

    # Train the model
    total_step = len(loaders['train'])

    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(loaders['train']):
            # gives batch data, normalize x when iterate train_loader
            b_x = Variable(images)  # batch x
            b_y = Variable(labels)  # batch y

            output = cnn(b_x)[0]
            loss = loss_func(output, b_y)

            # clear gradients for this training step
            optimizer.zero_grad()

            # backpropagation, compute gradients
            loss.backward()
            # apply gradients
            optimizer.step()

            if (i + 1) % 100 == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                      .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))
                pass

        pass

    pass

train(num_epochs, cnn, loaders)

def evalFunc():
    # Test the model
    cnn.eval()
    with torch.no_grad():
        correct = 0
        total = 0
        for images, labels in loaders['test']:
            test_output, last_layer = cnn(images)
            pred_y = torch.max(test_output, 1)[1].data.squeeze()
            accuracy = (pred_y == labels).sum().item() / float(labels.size(0))
            pass

        print('Test Accuracy of the model on the 10000 test images: %.2f' % accuracy)

    pass

evalFunc()

sample = next(iter(loaders['test']))
imgs, lbls = sample

actual_number = lbls[:10].numpy()

test_output, last_layer = cnn(imgs[:10])
pred_y = torch.max(test_output, 1)[1].data.numpy().squeeze()
print(f'Prediction number: {pred_y}')
print(f'Actual number: {actual_number}')

Output: If you are working on jupyter notebook. The problem is more likely to be num_worker. You should set num_worker=0. You can find here some solutions to follow. Because unfortunately, jupyter notebook has some issues with running multiprocessing.


Instruction: SSLCertVerificationError when downloading pytorch datasets via torchvision

Input:
I am having trouble downloading the CIFAR-10 dataset from pytorch. Mostly it seems like some SSL error which I don't really know how to interpret. I have also tried changing the root to various other folders but none of them works. I was wondering whether it is a permission type setting on my end but I am inexperienced. Would appreciate some help to fix this!
The code executed is here:
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=1)

The error is reproduced here:
---------------------------------------------------------------------------
SSLCertVerificationError                  Traceback (most recent call last)
File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:1354, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
   1353 try:
-> 1354     h.request(req.get_method(), req.selector, req.data, headers,
   1355               encode_chunked=req.has_header('Transfer-encoding'))
   1356 except OSError as err: # timeout error

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:1256, in HTTPConnection.request(self, method, url, body, headers, encode_chunked)
   1255 """Send a complete request to the server."""
-> 1256 self._send_request(method, url, body, headers, encode_chunked)

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:1302, in HTTPConnection._send_request(self, method, url, body, headers, encode_chunked)
   1301     body = _encode(body, 'body')
-> 1302 self.endheaders(body, encode_chunked=encode_chunked)

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:1251, in HTTPConnection.endheaders(self, message_body, encode_chunked)
   1250     raise CannotSendHeader()
-> 1251 self._send_output(message_body, encode_chunked=encode_chunked)

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:1011, in HTTPConnection._send_output(self, message_body, encode_chunked)
   1010 del self._buffer[:]
-> 1011 self.send(msg)
   1013 if message_body is not None:
   1014
   1015     # create a consistent interface to message_body

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:951, in HTTPConnection.send(self, data)
    950 if self.auto_open:
--> 951     self.connect()
    952 else:

File C:\ProgramData\Miniconda3\envs\pDL\lib\http\client.py:1425, in HTTPSConnection.connect(self)
   1423     server_hostname = self.host
-> 1425 self.sock = self._context.wrap_socket(self.sock,
   1426                                       server_hostname=server_hostname)

File C:\ProgramData\Miniconda3\envs\pDL\lib\ssl.py:500, in SSLContext.wrap_socket(self, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, session)
    494 def wrap_socket(self, sock, server_side=False,
    495                 do_handshake_on_connect=True,
    496                 suppress_ragged_eofs=True,
    497                 server_hostname=None, session=None):
    498     # SSLSocket class handles server_hostname encoding before it calls
    499     # ctx._wrap_socket()
--> 500     return self.sslsocket_class._create(
    501         sock=sock,
    502         server_side=server_side,
    503         do_handshake_on_connect=do_handshake_on_connect,
    504         suppress_ragged_eofs=suppress_ragged_eofs,
    505         server_hostname=server_hostname,
    506         context=self,
    507         session=session
    508     )

File C:\ProgramData\Miniconda3\envs\pDL\lib\ssl.py:1040, in SSLSocket._create(cls, sock, server_side, do_handshake_on_connect, suppress_ragged_eofs, server_hostname, context, session)
   1039             raise ValueError("do_handshake_on_connect should not be specified for non-blocking sockets")
-> 1040         self.do_handshake()
   1041 except (OSError, ValueError):

File C:\ProgramData\Miniconda3\envs\pDL\lib\ssl.py:1309, in SSLSocket.do_handshake(self, block)
   1308         self.settimeout(None)
-> 1309     self._sslobj.do_handshake()
   1310 finally:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)

During handling of the above exception, another exception occurred:

URLError                                  Traceback (most recent call last)
Input In [8], in <module>
----> 1 trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
      2 trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=1)

File C:\ProgramData\Miniconda3\envs\pDL\lib\site-packages\torchvision\datasets\cifar.py:66, in CIFAR10.__init__(self, root, train, transform, target_transform, download)
     63 self.train = train  # training set or test set
     65 if download:
---> 66     self.download()
     68 if not self._check_integrity():
     69     raise RuntimeError('Dataset not found or corrupted.' +
     70                        ' You can use download=True to download it')

File C:\ProgramData\Miniconda3\envs\pDL\lib\site-packages\torchvision\datasets\cifar.py:144, in CIFAR10.download(self)
    142     print('Files already downloaded and verified')
    143     return
--> 144 download_and_extract_archive(self.url, self.root, filename=self.filename, md5=self.tgz_md5)

File C:\ProgramData\Miniconda3\envs\pDL\lib\site-packages\torchvision\datasets\utils.py:427, in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
    424 if not filename:
    425     filename = os.path.basename(url)
--> 427 download_url(url, download_root, filename, md5)
    429 archive = os.path.join(download_root, filename)
    430 print("Extracting {} to {}".format(archive, extract_root))

File C:\ProgramData\Miniconda3\envs\pDL\lib\site-packages\torchvision\datasets\utils.py:130, in download_url(url, root, filename, md5, max_redirect_hops)
    127     _download_file_from_remote_location(fpath, url)
    128 else:
    129     # expand redirect chain if needed
--> 130     url = _get_redirect_url(url, max_hops=max_redirect_hops)
    132     # check if file is located on Google Drive
    133     file_id = _get_google_drive_file_id(url)

File C:\ProgramData\Miniconda3\envs\pDL\lib\site-packages\torchvision\datasets\utils.py:78, in _get_redirect_url(url, max_hops)
     75 headers = {"Method": "HEAD", "User-Agent": USER_AGENT}
     77 for _ in range(max_hops + 1):
---> 78     with urllib.request.urlopen(urllib.request.Request(url, headers=headers)) as response:
     79         if response.url == url or response.url is None:
     80             return url

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:222, in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220 else:
    221     opener = _opener
--> 222 return opener.open(url, data, timeout)

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:525, in OpenerDirector.open(self, fullurl, data, timeout)
    522     req = meth(req)
    524 sys.audit('urllib.Request', req.full_url, req.data, req.headers, req.get_method())
--> 525 response = self._open(req, data)
    527 # post-process response
    528 meth_name = protocol+"_response"

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:542, in OpenerDirector._open(self, req, data)
    539     return result
    541 protocol = req.type
--> 542 result = self._call_chain(self.handle_open, protocol, protocol +
    543                           '_open', req)
    544 if result:
    545     return result

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:502, in OpenerDirector._call_chain(self, chain, kind, meth_name, *args)
    500 for handler in handlers:
    501     func = getattr(handler, meth_name)
--> 502     result = func(*args)
    503     if result is not None:
    504         return result

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:1397, in HTTPSHandler.https_open(self, req)
   1396 def https_open(self, req):
-> 1397     return self.do_open(http.client.HTTPSConnection, req,
   1398         context=self._context, check_hostname=self._check_hostname)

File C:\ProgramData\Miniconda3\envs\pDL\lib\urllib\request.py:1357, in AbstractHTTPHandler.do_open(self, http_class, req, **http_conn_args)
   1354         h.request(req.get_method(), req.selector, req.data, headers,
   1355                   encode_chunked=req.has_header('Transfer-encoding'))
   1356     except OSError as err: # timeout error
-> 1357         raise URLError(err)
   1358     r = h.getresponse()
   1359 except:

URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1131)>

Output:
Turn off the ssl verification.
import ssl
ssl._create_default_https_context = ssl._create_unverified_context


Instruction: Split torch tensor dimension on two

Input:
If I have a torch tensor of shape [2, 12] is it possible to make from it a tensor of shape [3, 2, 4] in such a way that data will be split on chunks along the last dimension? View function does not change the order of data, but here I need change the order, that the new dimension will be the first, and the second dimension would not be changed
Here is example image of how data must be split
[2, 12]
............
............

[3, 2, 4]
....|....|....
....|....|....

torch.split function does exactly what I need in sense of shape, but it outputs not a tensor, but a tuple

Output:
I'm assuming you want to split the 12 sized dimension into the leading 3 and trailing 4 dimensions. You simply need to view and permute dimensions:
X = torch.permute(X.view(2, 3, 4), (1, 0, 2))


Instruction: What is the difference between "modules" and "functionals" in PyTorch?

Input:
In PyTorch, we have torch.nn.funtional and torch.nn, and the classes/functions within the former are typically referred to as "functions" while the latter as "modules".
There seems to be a lot of overlap between the two, so I'm wondering what are each used for and what their differences are?

Output:
Here are the differences:

torch.nn.functional is the base functional interface (in terms of programming paradigm) to apply PyTorch operators on torch.Tensor.

torch.nn contains the wrapper nn.Module that provide a object-oriented interface to those operators.

So indeed there is a complete overlap, modules are a different way of accessing the operators provided by those functions.
Every single tensor operator in PyTorch is available in the form of a function and its wrapper class. For instance F.conv2d, F.relu, F.dropout, F.batch_norm etc... have corresponding modules nn.Conv2d, nn.ReLU, nn.Dropout, nn.BatchNorm2d, 2d, 3d.


Instruction: Getting image path through a torchvision dataloader using local images

Input:
I want to use a dataloader in my script.
normaly the default function call would be like this.
dataset = ImageFolderWithPaths(
    data_dir,
    transforms.Compose([
            transforms.ColorJitter(0.1, 0.1, 0.1, 0.1),
            transforms.Resize((img_size_XY, img_size_XY)),
            transforms.ToTensor(),
            transforms.Normalize(_mean , _std)
        ])
)

dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=False,
    num_workers=2
)

and to iterate through this dataloader i use
for inputs, labels , paths in _dataloader:
    break

now i need to collect the path for each image.
i found in github this code: (https://gist.github.com/andrewjong/6b02ff237533b3b2c554701fb53d5c4d)
class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method. this is the method that dataloader calls
    def __getitem__(self, index):
        # this is what ImageFolder normally returns
        original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
        # the image file path
        path = self.imgs[index][0]
        # make a new tuple that includes original and the path
        tuple_with_path = (original_tuple + (path,))
        return tuple_with_path

# EXAMPLE USAGE:
# instantiate the dataset and dataloader
data_dir = "your/data_dir/here"
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
    print(inputs, labels, paths)

But this code does not take transforms into account, like in my original code.
Can anybody help in how I should go about making it work with that?

Output:
Since ImageFolderWithPaths inherits from datasets.ImageFolder as shown in the code from GitHub and datasets.ImageFolder has the following arguments including transform: (see here for more info)

torchvision.datasets.ImageFolder(root: str, transform:
Optional[Callable] = None, target_transform: Optional[Callable] =
None, loader: Callable[[str], Any] = ,
is_valid_file: Optional[Callable[[str], bool]] = None)

Solution: you can use your transformations directly when you instantiate ImageFolderWithPaths.
import torch
from torchvision import datasets
from torch.utils.data import DataLoader

class ImageFolderWithPaths(datasets.ImageFolder):

    def __getitem__(self, index):

        img, label = super(ImageFolderWithPaths, self).__getitem__(index)

        path = self.imgs[index][0]

        return (img, label ,path)

# put here your root directory not subfolders directory
# subfolders should be names of classes or encodings
root_dir = "training"

transform = transforms.Compose([transforms.Resize((32, 32)),
                                transforms.ToTensor()]) # my transformations.

dataset = ImageFolderWithPaths(root_dir,transform=transform) # add transformation directly

dataloader = DataLoader(dataset)

for inputs, labels, paths in dataloader:
    print(inputs.shape, labels, paths)

# output
torch.Size([1, 3, 32, 32]) tensor([0]) ('training\\0\\1.jpg',)
torch.Size([1, 3, 32, 32]) tensor([0]) ('training\\0\\1000.jpg',)
torch.Size([1, 3, 32, 32]) tensor([0]) ('training\\0\\10005.jpg',)
torch.Size([1, 3, 32, 32]) tensor([0]) ('training\\0\\10010.jpg',)

I also edited the code from github because there is no torch.utils.DataLoader but torch.utils.data.DataLoader.


Instruction: DataLoaders - wrapping an iterable over the Dataset means?

Input:
I am reading the official documentation of Dataloaders:
https://pytorch.org/tutorials/beginner/basics/data_tutorial.html
And there is this sentence "DataLoader wraps an iterable around the Dataset.."
I know that dataloaders are used to iterate over the dataset but what I don't get is that what does wrapping an iterable over the dataset means? I want an insight of the theoretical point of view.

Output: The sentence means that a DataLoader can be used to iterate the contents of a Dataset. For example, if you've got a Dataset of 1000 images, you can iterate certain attributes in the order that they've been stored in the Dataset and nothing else by itself. In the other hand, a DataLoader that wraps that Dataset allows you to iterate the data in batches, shuffle the data, apply functions, sample data, etc. Just checkout the Pytorch docs on torch.utils.data.DataLoader and you'll see all of the options included.


Instruction: Saving tensors to a .pt file in order to create a dataset

Input:
I was tasked with the creation of a dataset to test the functionality of the code we're working on.
The dataset must have a group of tensors that will be used later on in a generative model.
I'm trying to save the tensors to a .pt file, but I'm overwriting the tensors thus creating a file with only one. I've read about torch.utils.data.dataset but I'm not able to figure out by my own how to use it.
Here is my code:
import torch
import numpy as np

from torch.utils.data import Dataset

#variables that will be used to create the size of the tensors:
num_jets, num_particles, num_features = 1, 30, 3

for i in range(100):
    #tensor from a gaussian dist with mean=5,std=1 and shape=size:
    tensor = torch.normal(5,1,size=(num_jets, num_particles, num_features))

    #We will need the tensors to be of the cpu type
    tensor = tensor.cpu()

    #save the tensor to 'tensor_dataset.pt'
    torch.save(tensor,'tensor_dataset.pt')

#open the recently created .pt file inside a list
tensor_list = torch.load('tensor_dataset.pt')

#prints the list. Just one tensor inside .pt file
print(tensor_list)

Output:
Reason: You overwrote tensor x each time in a loop, therefore you did not get your list, and you only had x at the end.
Solution: you have the size of the tensor, you can initialize a tensor first and iterate through lst_tensors:
import torch
import numpy as np
from torch.utils.data import Dataset

num_jets, num_particles, num_features = 1, 30, 3

lst_tensors = torch.empty(size=(100,num_jets, num_particles, num_features))

for i in range(100):

    lst_tensors[i] = torch.normal(5,1,size=(num_jets, num_particles, num_features))

    lst_tensors[i] = lst_tensors[i].cpu()

torch.save(lst_tensors,'tensor_dataset.pt')

tensor_list = torch.load('tensor_dataset.pt')

print(tensor_list.shape)   # [100,1,30,3]


Instruction: Pytorch: How to get the first N item from dataloader

Input:
There are 3000 pictures in my list, but I only want the first N of them, like 1000, for training.
I wonder how can I achieve this by changing the loop code:
for (image, label) in enumerate(train_loader):

Output:
for (image, label) in list(enumerate(train_loader))[:1000]:

This is not a good way to partition training and validation data though.
First, the dataloader class supports lazy loading (examples are not loaded into memory until needed) whereas casting as a list will require all data to be loaded into memory, likely triggering an out-of-memory error. Second, this may not always return the same 1000 elements if the dataloader has shuffling. In general, the dataloader class does not support indexing so is not really suitable for selecting a specific subset of our dataset. Casting as a list works around this but at the expense of the useful attributes of the dataloader class.
Best practice is to use a separate data.dataset object for the training and validation partitions, or at least to partition the data in the dataset rather than relying on stopping the training after the first 1000 examples. Then, create a separate dataloader for the training partition and validation partition.


Instruction: Difference between pooling and global pooling in torch_geometric.nn

Input: I realised that torch_geometric library offers both global pooling layers and pooling layers, but I don't really understand what is the difference between these 2 when applied to Graph Neural Networks?

Output:
The difference is how the pooling is performed.
Global pooling gives you one supernode that contains the aggregated features from the whole graph.
Local pooling operation on the other hand create clusters and aggregates nodes in them.
Among local pooling you can find for instance Top-K pooling algorithm, SAGPool etc. They both have parameter called "ratio" that lets you specify how many nodes should be removed. Local pooling can give you a bit of hierarchical approach.


Instruction: How to sort a one hot tensor according to a tensor of indices

Input:
Given the below tensor:
tensor = torch.Tensor([[1., 0., 0., 0., 0.],
                       [0., 1., 0., 0., 0.],
                       [0., 0., 1., 0., 0.],
                       [0., 0., 0., 0., 1.],
                       [1., 0., 0., 0., 0.],
                       [1., 0., 0., 0., 0.],
                       [0., 0., 0., 1., 0.],
                       [0., 0., 0., 0., 1.]])

and below is the tensor containing the indices:
indices = torch.tensor([2, 6, 7, 5, 4, 0, 3, 1])

How can I sort tensor using the values inside of indices?
Trying with sorted gives the error:

TypeError: 'Tensor' object is not callable`.

While numpy.sort gives:

ValueError: Cannot specify order when the array has no fields.`

Output:
You can use the indices like this:
tensor = torch.Tensor([[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 0., 1.],
[1., 0., 0., 0., 0.],
[1., 0., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 1.]])
indices = torch.tensor([2, 6, 7, 5, 4, 0, 3, 1])
sorted_tensor = tensor[indices]
print(sorted_tensor)
# output
tensor([[0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1.],
        [0., 1., 0., 0., 0.]])


Instruction: Pytorch : Expected all tensors on same device

Input:
I have my model and inputs moved on the same device but I still get the runtime error :
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

Here is my code,
First my model implementation :
import torch
import torch.nn.functional as F

class Net(torch.nn.Module):
    def __init__(self, n_hiddens, n_feature= 2, n_output= 1):
        super().__init__()
        self.hiddens = []
        n_hidden_in = n_feature
        for n_hidden in n_hiddens :
          self.hiddens.append( torch.nn.Linear(n_hidden_in, n_hidden) )   # hidden layer
          n_hidden_in = n_hidden

        self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        for hidden in self.hiddens :
          x = F.relu(hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        return x

Then I define my dataloaders. Here, X and y are numpy arrays
from torch.utils.data import TensorDataset, DataLoader
# Split training/test
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state= 42)

X_train_tensor = torch.from_numpy(X_train)
y_train_tensor = torch.from_numpy(y_train)

X_test_tensor = torch.from_numpy(X_test)
y_test_tensor = torch.from_numpy(y_test)

train_dataset = TensorDataset(X_train_tensor, y_train_tensor) # create your datset
train_dataloader = DataLoader(train_dataset, batch_size= 1000) # create your dataloader

test_dataset = TensorDataset(X_test_tensor, y_test_tensor) # create your datset
test_dataloader = DataLoader(test_dataset, batch_size= 1000) # create your dataloader

Here I train my model. The error occurs during the line "outputs = regressor(inputs)"
NUM_EPOCHS = 2000
BATCH_SIZE =  1000
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(f"Device used : {device}")

# 1 hidden layer
total_num_nodes = 256
regressor = Net(n_hiddens= [total_num_nodes]).to(device)
optimizer = torch.optim.SGD(regressor.parameters(), lr=0.2, momentum= 0.1, nesterov= True)
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss

for epoch in range(NUM_EPOCHS):
  running_loss = 0.0

  for i, data in enumerate(train_dataloader, 0):
    inputs, values = data
    inputs = inputs.float().to(device)
    values = values.float().to(device)
    optimizer.zero_grad()   # clear gradients for next train
    print(f"Input device is : cuda:{inputs.get_device()}")
    print(f"Target value device is : cuda:{values.get_device()}")
    print(f"Is model on cuda ? : {next(regressor.parameters()).is_cuda}")
    outputs = regressor(inputs) # <-- This is where I have the error
    loss = loss_func(outputs, values)
    loss.backward()         # backpropagation, compute gradients
    optimizer.step()        # apply gradients

Here are the outputs of my print statements :
Device used : cuda:0
Input device is : cuda:0
Target value device is : cuda:0
Is model on cuda ? :True

This should mean that my model and my tensors are all on the same device so why do I still have this error ?
The error log is :
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-5234b830bebc> in <module>()
     24       print(f"Target value device is : cuda:{values.get_device()}")
     25       print(f"Is model on cuda ? : {next(regressor.parameters()).is_cuda}")
---> 26       outputs = regressor(inputs)
     27       loss = loss_func(outputs, values)
     28       loss.backward()         # backpropagation, compute gradients

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-4-56c54b30b771> in forward(self, x)
     16     def forward(self, x):
     17         for hidden in self.hiddens :
---> 18           x = F.relu(hidden(x))      # activation function for hidden layer
     19         x = self.predict(x)             # linear output
     20         return x

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
    101
    102     def forward(self, input: Tensor) -> Tensor:
--> 103         return F.linear(input, self.weight, self.bias)
    104
    105     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849
   1850

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

Thank you very much

Output:
TL;DR  use nn.ModuleList instead of a pythonic one to store the hidden layers in Net.
All your hidden layers are stored in a simple pythonic list self.hidden in Net. When you move your model to GPU, using .to(device), pytorch has no way to tell that all the elements of this pythonic list should also be moved to the same device.
however, if you make self.hidden = nn.ModuleLis(), pytorch now knows to treat all elements of this special list as nn.Modules and recursively move them to the same device as Net.
See these answers 1, 2, 3 for more details.


Instruction: How to interpret the output format of a model?

Input:
Noob here, hard to elaborate my question without an example,
so I use a model on the MNIST data that classifies digits based on number images.
# Load data
trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

Why does model end up with 64 (row) x 10 (column) matrix?
I thought nn.Linear(64, 10) means a layer that has 64 input neurons to 10 neurons. Shouldn't it be an array of 10 probabilities?
and Why output activation function has dim=1 not dim=0?
Isn't each row of 10 columns for an epoch? Shouldn't LogSoftmax being used to calculate the possibility of each digit?
I'm ...lost.
I have spent 2 hr on this, still can't find the answer, sorry for the noob question!

Output:
We usually have our data in the form of (BATCH SIZE, INPUT SIZE) which here in your case would be (64, 784).
What this means is that in every batch you have 64 images and each image has 784 features.
Regarding your model this is what it outputs :
model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))
print(model)

# Sequential(
#  (0): Linear(in_features=784, out_features=128, bias=True)
#  (1): ReLU()
#  (2): Linear(in_features=128, out_features=64, bias=True)
#  (3): ReLU()
#  (4): Linear(in_features=64, out_features=10, bias=True)
#  (5): LogSoftmax(dim=1)
# )

Let's go through how the data will flow through this model.

You have input of shape (64, 784)
It passes through first Linear layer, where each image of 784 features is converted to have 128 features so output is of shape (64, 128)
ReLU does not change the shape just the values so shape is again (64, 128)
Next Linear layer converts 128 features to 64 so now output shape is (64, 64).
Again ReLU layer just changes values so shape is still (64, 64)
This last Linear layer maps 64 input features to 10 output ones so shape is now (64, 10).
Lastly we have the LogSoftmax layer. Here we provided dim=1 because we want to calculate output possibility for each of the possible 10 digits for each of the 64 images in out batch. The dim=0 is the batch and dim=1 is the outputs for digits that is we we provide dim=1. After this your output will have shape (64, 10).

Therefore at the end, each image in the batch will have possibility for each of the 10 digits.

I thought nn.Linear(64, 10) means a layer that has 64 input neurons to 10 neurons.

That is correct. Another point to remember is that batch dimension is not specified in the layers of the models. We define layers to operate on each image. Your second last Linear layer output 64 values for an image so last Linear layer converts it to 10 values and then applies LogSoftmax to it.
This operation is simply repeated for all 64 images in a batch efficiently using matrix operations.
You might be confusing your batch_size=64 with the input_features=64 of Linear layer which is entirely unrelated.


Instruction: GCP Vertex AI Training: Auto-packaged Custom Training Job Yields Huge Docker Image

Input:
I am trying to run a Custom Training Job in Google Cloud Platform's Vertex AI Training service.
The job is based on a tutorial from Google that fine-tunes a pre-trained BERT model (from HuggingFace).
When I use the gcloud CLI tool to auto-package my training code into a Docker image and deploy it to the Vertex AI Training service like so:
$BASE_GPU_IMAGE="us-docker.pkg.dev/vertex-ai/training/pytorch-gpu.1-7:latest"
$BUCKET_NAME = "my-bucket"

gcloud ai custom-jobs create `
--region=us-central1 `
--display-name=fine_tune_bert `
--args="--job_dir=$BUCKET_NAME,--num-epochs=2,--model-name=finetuned-bert-classifier" `
--worker-pool-spec="machine-type=n1-standard-4,replica-count=1,accelerator-type=NVIDIA_TESLA_V100,executor-image-uri=$BASE_GPU_IMAGE,local-package-path=.,python-module=trainer.task"

... I end up with a Docker image that is roughly 18GB (!) and takes a very long time to upload to the GCP registry.
Granted the base image is around 6.5GB but where do the additional >10GB come from and is there a way for me to avoid this "image bloat"?
Please note that my job loads the training data using the datasets Python package at run time and AFAIK does not include it in the auto-packaged docker image.

Output:
The image size shown in the UI is the virtual size of the image. It is the compressed total image size that will be downloaded over the network. Once the image is pulled, it will be extracted and the resulting size will be bigger. In this case, the PyTorch image's virtual size is 6.8 GB while the actual size is 17.9 GB.
Also, when a docker push command is executed, the progress bars show the uncompressed size. The actual amount of data that’s pushed will be compressed before sending, so the uploaded size will not be reflected by the progress bar.
To cut down the size of the docker image, custom containers can be used. Here, only the necessary components can be configured which would result in a smaller docker image. More information on custom containers here.


Instruction: How can I determine validation loss for faster RCNN (PyTorch)?

Input:
I followed this tutorial for object detection:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
and their GitHub repository that contains the following train_one_epoch and evaluate functions:
https://github.com/pytorch/vision/blob/main/references/detection/engine.py
However, I want to calculate losses during validation. I  implemented this for the evaluation loss, where essentially to obtain losses, model.train() needs to be on:
@torch.no_grad()
def evaluate_loss(model, data_loader, device):
    val_loss = 0
    model.train()
    for images, targets in data_loader:
        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        loss_dict = model(images, targets)

        losses = sum(loss for loss in loss_dict.values())

        # reduce losses over all GPUs for logging purposes
        loss_dict_reduced = utils.reduce_dict(loss_dict)
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())
        val_loss += losses_reduced

  validation_loss = val_loss/ len(data_loader)
  return validation_loss

I then place it after the learning rate scheduler step in my for loop:
 for epoch in range(args.num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, train_data_loader, device, epoch, print_freq=10)

        # update the learning rate
        lr_scheduler.step()

        validation_loss = evaluate_loss(model, valid_data_loader, device=device)

        # evaluate on the test dataset
        evaluate(model, valid_data_loader, device=device)

Does this look correct or can it interfere with training or produce inaccurate validation losses?
If ok, by using this, is there is a simple way in applying early stopping for validation loss?
I'm considering just adding something like this after the evaluate model function shown above:
torch.save({
            'epoch': epoch,
            'model_state_dict': net.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'validation loss': valid_loss,
            }, PATH)

where I also aim to save the model at every epoch for checkpointing purposes. However I need to determine the validation "loss" for saving the "best" model.

Output:
So it turns out no stages of the pytorch fasterrcnn return losses when model.eval() is set. However, you can just manually use the forward code to generate the losses in evaluation mode:
from typing import Tuple, List, Dict, Optional
import torch
from torch import Tensor
from collections import OrderedDict
from torchvision.models.detection.roi_heads import fastrcnn_loss
from torchvision.models.detection.rpn import concat_box_prediction_layers
def eval_forward(model, images, targets):
    # type: (List[Tensor], Optional[List[Dict[str, Tensor]]]) -> Tuple[Dict[str, Tensor], List[Dict[str, Tensor]]]
    """
    Args:
        images (list[Tensor]): images to be processed
        targets (list[Dict[str, Tensor]]): ground-truth boxes present in the image (optional)
    Returns:
        result (list[BoxList] or dict[Tensor]): the output from the model.
            It returns list[BoxList] contains additional fields
            like `scores`, `labels` and `mask` (for Mask R-CNN models).
    """
    model.eval()

    original_image_sizes: List[Tuple[int, int]] = []
    for img in images:
        val = img.shape[-2:]
        assert len(val) == 2
        original_image_sizes.append((val[0], val[1]))

    images, targets = model.transform(images, targets)

    # Check for degenerate boxes
    # TODO: Move this to a function
    if targets is not None:
        for target_idx, target in enumerate(targets):
            boxes = target["boxes"]
            degenerate_boxes = boxes[:, 2:] <= boxes[:, :2]
            if degenerate_boxes.any():
                # print the first degenerate box
                bb_idx = torch.where(degenerate_boxes.any(dim=1))[0][0]
                degen_bb: List[float] = boxes[bb_idx].tolist()
                raise ValueError(
                    "All bounding boxes should have positive height and width."
                    f" Found invalid box {degen_bb} for target at index {target_idx}."
                )

    features = model.backbone(images.tensors)
    if isinstance(features, torch.Tensor):
        features = OrderedDict([("0", features)])
    model.rpn.training=True
    #model.roi_heads.training=True

    #####proposals, proposal_losses = model.rpn(images, features, targets)
    features_rpn = list(features.values())
    objectness, pred_bbox_deltas = model.rpn.head(features_rpn)
    anchors = model.rpn.anchor_generator(images, features_rpn)

    num_images = len(anchors)
    num_anchors_per_level_shape_tensors = [o[0].shape for o in objectness]
    num_anchors_per_level = [s[0] * s[1] * s[2] for s in num_anchors_per_level_shape_tensors]
    objectness, pred_bbox_deltas = concat_box_prediction_layers(objectness, pred_bbox_deltas)
    # apply pred_bbox_deltas to anchors to obtain the decoded proposals
    # note that we detach the deltas because Faster R-CNN do not backprop through
    # the proposals
    proposals = model.rpn.box_coder.decode(pred_bbox_deltas.detach(), anchors)
    proposals = proposals.view(num_images, -1, 4)
    proposals, scores = model.rpn.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)

    proposal_losses = {}
    assert targets is not None
    labels, matched_gt_boxes = model.rpn.assign_targets_to_anchors(anchors, targets)
    regression_targets = model.rpn.box_coder.encode(matched_gt_boxes, anchors)
    loss_objectness, loss_rpn_box_reg = model.rpn.compute_loss(
        objectness, pred_bbox_deltas, labels, regression_targets
    )
    proposal_losses = {
        "loss_objectness": loss_objectness,
        "loss_rpn_box_reg": loss_rpn_box_reg,
    }

    #####detections, detector_losses = model.roi_heads(features, proposals, images.image_sizes, targets)
    image_shapes = images.image_sizes
    proposals, matched_idxs, labels, regression_targets = model.roi_heads.select_training_samples(proposals, targets)
    box_features = model.roi_heads.box_roi_pool(features, proposals, image_shapes)
    box_features = model.roi_heads.box_head(box_features)
    class_logits, box_regression = model.roi_heads.box_predictor(box_features)

    result: List[Dict[str, torch.Tensor]] = []
    detector_losses = {}
    loss_classifier, loss_box_reg = fastrcnn_loss(class_logits, box_regression, labels, regression_targets)
    detector_losses = {"loss_classifier": loss_classifier, "loss_box_reg": loss_box_reg}
    boxes, scores, labels = model.roi_heads.postprocess_detections(class_logits, box_regression, proposals, image_shapes)
    num_images = len(boxes)
    for i in range(num_images):
        result.append(
            {
                "boxes": boxes[i],
                "labels": labels[i],
                "scores": scores[i],
            }
        )
    detections = result
    detections = model.transform.postprocess(detections, images.image_sizes, original_image_sizes)  # type: ignore[operator]
    model.rpn.training=False
    model.roi_heads.training=False
    losses = {}
    losses.update(detector_losses)
    losses.update(proposal_losses)
    return losses, detections

Testing this code gives me:
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

# load a model pre-trained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

# replace the classifier with a new one, that has
# num_classes which is user-defined
num_classes = 2  # 1 class (person) + background
# get number of input features for the classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
# replace the pre-trained head with a new one
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
losses, detections = eval_forward(model,torch.randn([1,3,300,300]),[{'boxes':torch.tensor([[100,100,200,200]]),'labels':torch.tensor([0])}])

{'loss_classifier': tensor(0.6594, grad_fn=<NllLossBackward0>),
'loss_box_reg': tensor(0., grad_fn=<DivBackward0>),
 'loss_objectness': tensor(0.5108, grad_fn=<BinaryCrossEntropyWithLogitsBackward0>),
 'loss_rpn_box_reg': tensor(0.0160, grad_fn=<DivBackward0>)}


Instruction: Only Tensors of floating point and complex dtype can require gradients

Input:
I am receiving the following error when I run a convolution operation inside a torch.no_grad() context:

RuntimeError: Only Tensors of floating-point and complex dtype can require gradients.

import torch.nn as nn
import torch

with torch.no_grad():
    ker_t = torch.tensor([[1, -1] ,[-1, 1]])
    in_t = torch.tensor([ [14, 7, 6, 2,] , [4 ,8 ,11 ,1], [3, 5, 9 ,10], [12, 15, 16, 13] ])
    print(in_t.shape)
    in_t = torch.unsqueeze(in_t,0)
    in_t = torch.unsqueeze(in_t,0)
    print(in_t.shape)

    conv = nn.Conv2d(1, 1, kernel_size=2,stride=2,dtype=torch.long)
    conv.weight[:] = ker_t
    conv(in_t)

Now I am sure if I turn my input into floats this message will go away, but I want to work in integers.
But I was under the impression that if I am in a with torch.no_grad() context it should turn off the need for gradients.

Output:
The need for gradients comes from nn.Conv2d when it registers the weights of the convolution layer.
However, if you are only after the forward pass, you do not need to use a convolution layer: you can use the underlying convolution function:
import torch.nn.functional as nnf

ker_t = torch.tensor([[1, -1] ,[-1, 1]])[None, None, ...]
in_t = torch.tensor([ [14, 7, 6, 2,] , [4 ,8 ,11 ,1], [3, 5, 9 ,10], [12, 15, 16, 13] ])[None, None, ...]
out = nnf.conv2d(in_t, ker_t, stride=2)

Will give you this output:
tensor([[[[11, -6],
          [ 1, -4]]]])