Instruction: Index out of range error while training dataset

Input:
I am trying to train MaskRCNN to detect and segment apples using the dataset from this paper,

github link to code being used

I am simply following the instructions as provided in the ReadMe file..

Here is the output on console

(venv) PS > python train_rcnn.py --data_path 'D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\datasets\apples-minneapple' --model mrcnn --epochs 50 --output-dir 'D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples'

mrcnn
Namespace(batch_size=2, data_path='D:\\Research Report\\tensorflow\\Mask_RCNN-TRIALS\\Mask_RCNN-master\\datasets\\apples-minneapple', dataset='AppleDataset', device='cuda', epochs=50, lr=0.02, lr_gamma=0.1, lr_step_size=8, lr_steps=[8, 11], model='mrcnn', momentum=0.9, output_dir='D:\\Research Report\\tensorflow\\Mask_RCNN-TRIALS\\Mask_RCNN-master\\samples\\apples', print_freq=20, resume='', weight_decay=0.0001, workers=4)
Loading data
Creating data loaders
Creating model
Start training
Epoch: [0]  [  0/335]  eta: 1:00:28  lr: 0.000080  loss: 2.4100 (2.4100)  loss_classifier: 0.8481 (0.8481)  loss_box_reg: 0.4164 (0.4164)  loss_objectness: 0.9299 (0.9299)  loss_rpn_box_reg: 0.2157 (0.2157)  time: 10.8327  data: 7.9925  max mem: 2733
Epoch: [0]  [ 20/335]  eta: 0:06:18  lr: 0.001276  loss: 1.4465 (1.4728)  loss_classifier: 0.5526 (0.5496)  loss_box_reg: 0.3586 (0.3572)  loss_objectness: 0.2666 (0.3418)  loss_rpn_box_reg: 0.2233 (0.2242)  time: 0.7204  data: 0.0132  max mem: 3247
Epoch: [0]  [ 40/335]  eta: 0:04:48  lr: 0.002473  loss: 0.9622 (1.2287)  loss_classifier: 0.2927 (0.4276)  loss_box_reg: 0.3188 (0.3314)  loss_objectness: 0.1422 (0.2491)  loss_rpn_box_reg: 0.2168 (0.2207)  time: 0.7408  data: 0.0210  max mem: 3282
Epoch: [0]  [ 60/335]  eta: 0:04:05  lr: 0.003669  loss: 0.7924 (1.0887)  loss_classifier: 0.2435 (0.3654)  loss_box_reg: 0.2361 (0.2983)  loss_objectness: 0.1289 (0.2105)  loss_rpn_box_reg: 0.1898 (0.2144)  time: 0.7244  data: 0.0127  max mem: 3432
Epoch: [0]  [ 80/335]  eta: 0:03:37  lr: 0.004865  loss: 0.7438 (1.0117)  loss_classifier: 0.2565 (0.3376)  loss_box_reg: 0.2193 (0.2799)  loss_objectness: 0.0776 (0.1835)  loss_rpn_box_reg: 0.1983 (0.2108)  time: 0.7217  data: 0.0127  max mem: 3432
Epoch: [0]  [100/335]  eta: 0:03:14  lr: 0.006062  loss: 0.7373 (0.9490)  loss_classifier: 0.2274 (0.3156)  loss_box_reg: 0.2193 (0.2654)  loss_objectness: 0.0757 (0.1643)  loss_rpn_box_reg: 0.1867 (0.2037)  time: 0.7291  data: 0.0132  max mem: 3432
Epoch: [0]  [120/335]  eta: 0:02:54  lr: 0.007258  loss: 0.8275 (0.9243)  loss_classifier: 0.2689 (0.3094)  loss_box_reg: 0.2315 (0.2602)  loss_objectness: 0.0867 (0.1539)  loss_rpn_box_reg: 0.1883 (0.2008)  time: 0.7270  data: 0.0134  max mem: 3432
Epoch: [0]  [140/335]  eta: 0:02:35  lr: 0.008455  loss: 0.7886 (0.9057)  loss_classifier: 0.2573 (0.3029)  loss_box_reg: 0.2246 (0.2539)  loss_objectness: 0.0724 (0.1455)  loss_rpn_box_reg: 0.2459 (0.2035)  time: 0.7170  data: 0.0124  max mem: 3432
Epoch: [0]  [160/335]  eta: 0:02:17  lr: 0.009651  loss: 0.7588 (0.8878)  loss_classifier: 0.2341 (0.2948)  loss_box_reg: 0.2226 (0.2486)  loss_objectness: 0.1032 (0.1427)  loss_rpn_box_reg: 0.2020 (0.2016)  time: 0.7139  data: 0.0118  max mem: 3432
Epoch: [0]  [180/335]  eta: 0:02:01  lr: 0.010847  loss: 0.7340 (0.8744)  loss_classifier: 0.2331 (0.2898)  loss_box_reg: 0.2120 (0.2441)  loss_objectness: 0.1086 (0.1392)  loss_rpn_box_reg: 0.1993 (0.2012)  time: 0.7800  data: 0.0584  max mem: 3432
Epoch: [0]  [200/335]  eta: 0:01:45  lr: 0.012044  loss: 0.8106 (0.8694)  loss_classifier: 0.2616 (0.2873)  loss_box_reg: 0.2208 (0.2411)  loss_objectness: 0.1117 (0.1397)  loss_rpn_box_reg: 0.1927 (0.2014)  time: 0.7344  data: 0.0143  max mem: 3432
Epoch: [0]  [220/335]  eta: 0:01:29  lr: 0.013240  loss: 0.8191 (0.8610)  loss_classifier: 0.2581 (0.2848)  loss_box_reg: 0.2140 (0.2382)  loss_objectness: 0.0860 (0.1362)  loss_rpn_box_reg: 0.2177 (0.2018)  time: 0.7213  data: 0.0126  max mem: 3432
Epoch: [0]  [240/335]  eta: 0:01:13  lr: 0.014437  loss: 0.7890 (0.8590)  loss_classifier: 0.2671 (0.2842)  loss_box_reg: 0.2094 (0.2357)  loss_objectness: 0.1175 (0.1360)  loss_rpn_box_reg: 0.2256 (0.2030)  time: 0.7576  data: 0.0564  max mem: 3432
Epoch: [0]  [260/335]  eta: 0:00:57  lr: 0.015633  loss: 0.8631 (0.8587)  loss_classifier: 0.2900 (0.2849)  loss_box_reg: 0.2089 (0.2337)  loss_objectness: 0.0925 (0.1350)  loss_rpn_box_reg: 0.2271 (0.2050)  time: 0.7371  data: 0.0220  max mem: 3432
Epoch: [0]  [280/335]  eta: 0:00:42  lr: 0.016830  loss: 0.8464 (0.8580)  loss_classifier: 0.2679 (0.2840)  loss_box_reg: 0.2156 (0.2321)  loss_objectness: 0.0940 (0.1346)  loss_rpn_box_reg: 0.2345 (0.2073)  time: 0.7379  data: 0.0143  max mem: 3432
Epoch: [0]  [300/335]  eta: 0:00:27  lr: 0.018026  loss: 0.7991 (0.8519)  loss_classifier: 0.2485 (0.2819)  loss_box_reg: 0.2125 (0.2305)  loss_objectness: 0.0819 (0.1315)  loss_rpn_box_reg: 0.2217 (0.2080)  time: 0.8549  data: 0.1419  max mem: 3450
Epoch: [0]  [320/335]  eta: 0:00:11  lr: 0.019222  loss: 0.6906 (0.8432)  loss_classifier: 0.2362 (0.2791)  loss_box_reg: 0.2036 (0.2285)  loss_objectness: 0.0662 (0.1285)  loss_rpn_box_reg: 0.1801 (0.2070)  time: 0.7257  data: 0.0238  max mem: 3450
Epoch: [0]  [334/335]  eta: 0:00:00  lr: 0.020000  loss: 0.7822 (0.8441)  loss_classifier: 0.2501 (0.2785)  loss_box_reg: 0.2224 (0.2285)  loss_objectness: 0.1135 (0.1296)  loss_rpn_box_reg: 0.1948 (0.2075)  time: 0.7249  data: 0.0139  max mem: 3450
Epoch: [0] Total time: 0:04:18 (0.7707 s / it)
Traceback (most recent call last):
  File "train_rcnn.py", line 143, in <module>
    main(args)
  File "train_rcnn.py", line 109, in main
    evaluate(model, data_loader_test, device=device)
  File "C:\Users\___\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\autograd\grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\engine.py", line 78, in evaluate
    coco = get_coco_api_from_dataset(data_loader.dataset)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\coco_utils.py", line 205, in get_coco_api_from_dataset
    return convert_to_coco_api(dataset)
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\utility\coco_utils.py", line 154, in convert_to_coco_api
    img, targets = ds[img_idx]
  File "D:\Research Report\tensorflow\Mask_RCNN-TRIALS\Mask_RCNN-master\samples\apples\data\apple_dataset.py", line 22, in __getitem__
    mask_path = os.path.join(self.root_dir, "masks", self.masks[idx])
IndexError: list index out of range

This is the file that is run in order to train the network

import datetime
import os
import time

import torch
import torch.utils.data
import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

from data.apple_dataset import AppleDataset
from utility.engine import train_one_epoch, evaluate

import utility.utils as utils
import utility.transforms as T

######################################################
# Train either a Faster-RCNN or Mask-RCNN predictor
# using the MinneApple dataset
######################################################

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def get_maskrcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model

def get_frcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
    return model

def main(args):
    print(args)
    device = args.device

    # Data loading code
    print("Loading data")
    num_classes = 2
    dataset = AppleDataset(os.path.join(args.data_path, 'train'), get_transform(train=True))
    dataset_test = AppleDataset(os.path.join(args.data_path, 'test'), get_transform(train=False))

    print("Creating data loaders")
    data_loader = torch.utils.data.DataLoader(dataset, batch_size=args.batch_size, shuffle=True,
                                              num_workers=args.workers, collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(dataset_test, batch_size=1,
                                                   shuffle=False, num_workers=args.workers,
                                                   collate_fn=utils.collate_fn)

    print("Creating model")
    # Create the correct model type
    if args.model == 'maskrcnn':
        model = get_maskrcnn_model_instance(num_classes)
    else:
        model = get_frcnn_model_instance(num_classes)

    # Move model to the right device
    model.to(device)

    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=args.lr, momentum=args.momentum, weight_decay=args.weight_decay)

    #  lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=args.lr_step_size, gamma=args.lr_gamma)
    lr_scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=args.lr_steps, gamma=args.lr_gamma)

    if args.resume:
        checkpoint = torch.load(args.resume, map_location='cpu')
        model.load_state_dict(checkpoint['model'])
        optimizer.load_state_dict(checkpoint['optimizer'])
        lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])

    print("Start training")
    start_time = time.time()
    for epoch in range(args.epochs):
        train_one_epoch(model, optimizer, data_loader, device, epoch, args.print_freq)
        lr_scheduler.step()

        if args.output_dir:
            torch.save(model.state_dict(), os.path.join(args.output_dir, 'model_{}.pth'.format(epoch)))

        # evaluate after every epoch
        evaluate(model, data_loader_test, device=device)

    total_time = time.time() - start_time
    total_time_str = str(datetime.timedelta(seconds=int(total_time)))
    print('Training time {}'.format(total_time_str))

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description='PyTorch Detection Training')
    parser.add_argument('--data_path', default='~~~~', help='dataset')
    parser.add_argument('--dataset', default='AppleDataset', help='dataset')
    parser.add_argument('--model', default='maskrcnn', help='model')
    parser.add_argument('--device', default='cuda', help='device')
    parser.add_argument('-b', '--batch-size', default=2, type=int)
    parser.add_argument('--epochs', default=13, type=int, metavar='N', help='number of total epochs to run')
    parser.add_argument('-j', '--workers', default=4, type=int, metavar='N', help='number of data loading workers (default: 16)')
    parser.add_argument('--lr', default=0.02, type=float, help='initial learning rate')
    parser.add_argument('--momentum', default=0.9, type=float, metavar='M', help='momentum')
    parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float, metavar='W', help='weight decay (default: 1e-4)', dest='weight_decay')
    parser.add_argument('--lr-step-size', default=8, type=int, help='decrease lr every step-size epochs')
    parser.add_argument('--lr-steps', default=[8, 11], nargs='+', type=int, help='decrease lr every step-size epochs')
    parser.add_argument('--lr-gamma', default=0.1, type=float, help='decrease lr by a factor of lr-gamma')
    parser.add_argument('--print-freq', default=20, type=int, help='print frequency')
    parser.add_argument('--output-dir', default='.', help='path where to save')
    parser.add_argument('--resume', default='', help='resume from checkpoint')

    args = parser.parse_args()
    print(args.model)
    assert(args.model in ['mrcnn', 'frcnn'])

    if args.output_dir:
        utils.mkdir(args.output_dir)

    main(args)

Apple_dataset.py is as follows

import os
import numpy as np
import torch
from PIL import Image

#####################################
# Class that takes the input instance masks
# and extracts bounding boxes on the fly
#####################################
class AppleDataset(object):
    def __init__(self, root_dir, transforms):
        self.root_dir = root_dir
        self.transforms = transforms

        # Load all image and mask files, sorting them to ensure they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root_dir, "images"))))
        self.masks = list(sorted(os.listdir(os.path.join(root_dir, "masks"))))

    def __getitem__(self, idx):
        # Load images and masks
        img_path = os.path.join(self.root_dir, "images", self.imgs[idx])
        mask_path = os.path.join(self.root_dir, "masks", self.masks[idx])

        img = Image.open(img_path).convert("RGB")
        mask = Image.open(mask_path)     # Each color of mask corresponds to a different instance with 0 being the background

        # Convert the PIL image to np array
        mask = np.array(mask)
        obj_ids = np.unique(mask)

        # Remove background id
        obj_ids = obj_ids[1:]

        # Split the color-encoded masks into a set of binary masks
        masks = mask == obj_ids[:, None, None]

        # Get bbox coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        h, w = mask.shape
        for ii in range(num_objs):
            pos = np.where(masks[ii])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])

            if xmin == xmax or ymin == ymax:
                continue

            xmin = np.clip(xmin, a_min=0, a_max=w)
            xmax = np.clip(xmax, a_min=0, a_max=w)
            ymin = np.clip(ymin, a_min=0, a_max=h)
            ymax = np.clip(ymax, a_min=0, a_max=h)
            boxes.append([xmin, ymin, xmax, ymax])

        # Convert everything into a torch.Tensor
        boxes = torch.as_tensor(boxes, dtype=torch.float32)

        # There is only one class (apples)
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])

        # All instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

    def get_img_name(self, idx):
        return self.imgs[idx]

How do i fix the index going out of range??? OR what is the underlying issue that needs to be addressed here??

EDIT1: ok.. so whats happening here is i have two folders "train" and "test".. the training folder has the images and masks, while test folder has only images.. the apple_dataset.py is written such that its looking for masks folder in both train and test folders.. i think i need to change the code such that it looks for masks in only the train folder and not the test set

Output:
fixed this by creating a dummy folder called 'masks' in the "test" folder.. just copy paste the one from 'train' with all the masks.. the train and predict scripts wont really use it, so shoudnt be any issues here..

also look at this issue for more changes that need to be made


Instruction: cmake - linking static library pytorch cannot find its internal functions during build

Input:
I'm trying to build a program using cmake. For several reasons, the program must be built using static libraries rather than dynamic libraries, and I need to use PyTorch so this is what I've done:

Downloaded and installed PyTorch static library (I've found libtorch.a in the proper path, in /home/me/pytorch/torch/lib)
Made CMakeLists.txt with the following contents:

cmake_minimum_required(VERSION 3.5.1 FATAL_ERROR)
project(example-app LANGUAGES CXX)
find_package(Torch REQUIRED)
add_executable(example-app example-app.cpp argparse/argparse.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}" -static -fopenmp)
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)

FYI, example-app.cpp is the file with the main function, and argparse/ is a directory with some source code for functions called in example-app.cpp

It works until cmake -DCMAKE_PREFIX_PATH=/home/me/pytorch/torch .., but the following build incurs some errors, saying it could not find the reference to some functions, namely functions starting with fbgemm::. fbgemm is (as long as I know) some sort of GEMM library used in implementing PyTorch.

It seems to me that while linking the static PyTorch library, its internal libraries like fbgemm stuff have not been linked properly, but I'm not an expert on cmake and honestly not entirely sure.

Am I doing something wrong, or is there a workaround for this problem? Any help or push in the right direction would be greatly appreciated.

P.S.

The exact error has not been posted because it is way too long, but it consists of mostly undefined reference to ~ errors. If looking at the error message might be helpful for some people, I'd be happy to edit the question and post it.
building and running the file works fine if I remove the parts that require the library's functions from the code without commenting out #include <torch/torch.h> from example-app.cpp.

Output:
Lately went through similar process with static linking of PyTorch and to be honest it wasn't too pretty.

I will outline the steps I have undertaken (you can find exact source code in torchlambda, here is CMakeLists.txt (it also includes AWS SDK and AWS Lambda static builds), here is a script building pytorch from source ( cloning and building via /scripts/build_mobile.sh with only CPU support)),
though it's only with CPU support (though similar steps should be fine if you need CUDA, it will get you started at least).

Pytorch static library

Pre-built static PyTorch

First of all, you need pre-built static library files (all of them need to be static, hence no .so, only those with .a extension are suitable).

Tbh I've been looking for those provided by PyTorch on installation page, yet there is only shared version.
In one GitHub issue I've found a way to download them as follows:

Instead of downloading (here via wget) shared libraries:

$ wget https://download.pytorch.org/libtorch/cu101/libtorch-shared-with-deps-1.4.0.zip

you rename shared to static (as described in this issue), so it would become:

$ wget https://download.pytorch.org/libtorch/cu101/libtorch-static-with-deps-1.4.0.zip

Yet, when you download it there is no libtorch.a under lib folder (didn't find libcaffe2.a either as indicated by this issue), so what I was left with was building explicitly from source.

If you have those files somehow (if so, please provide where you got them from please), you can skip the next step.

Building from source

For CPU version I have used /pytorch/scripts/build_mobile.sh file, you can base your version off of this if GPU support is needed (maybe you only have to pass -DUSE_CUDA=ON to this script, not sure though).

Most important is cmake's -DBUILD_SHARED_LIBS=OFF in order to build everything as static library. You can also check script from my tool which passes arguments to build_mobile.sh as well.

Running above will give you static files in /pytorch/build_mobile/install by default where there is everything you need.

CMake

Now you can copy above build files to /usr/local (better not to unless you are using Docker as torchlambda) or set path to it from within your CMakeLists.txt like this:

set(LIBTORCH "/path/to/pytorch/build_mobile/install")

# Below will append libtorch to path so CMake can see files
set(CMAKE_PREFIX_PATH "${CMAKE_PREFIX_PATH};${LIBTORCH}")

Now the rest is fine except target_link_libraries, which should be (as indicated by this issue, see related issues listed there for additional reference) used with -Wl,--whole-archive linker flag, which brought me to this:

target_link_libraries(example-app PRIVATE -lm
        -Wl,--whole-archive "${TORCH_LIBRARIES}"
        -Wl,--no-whole-archive
        -lpthread
        ${CMAKE_DL_LIBS})

You may not need either of -lm, -lpthread or ${CMAKE_DL_LIBS}, though I needed it when building on Amazon Linux AMI.

Building

Now you are off to building your application. Standard libtorch way should be fine but here is another command I used:

mkdir build && \
  cd build &&  \
  cmake .. && \
  cmake --build . --config Release

Above will create build folder where example-app binary should be now safely located.

Finally use ld build/example-app to verify everything from PyTorch was statically linked, see aforementioned issue point 5., your output should look similar.


Instruction: Vectorized way to shuffle a given tensor using pytorch

Input:
I have a tensor A of shape (1,12,2,2) as follows:

  ([[[[1., 3.],
      [9., 11.],

     [[ 2.,  4.],
      [10., 12.]],

     [[ 5.,  7.],
      [13., 15.]],

     [[ 6.,  8.],
      [14., 16.]],

     [[17., 19.],
      [25., 27.]],

     [[18., 20.],
      [26., 28.]],

     [[21., 23.],
      [29., 31.]],

     [[22., 24.],
      [30., 32.]],

     [[33., 35.],
      [41., 43.]],

     [[34., 36.],
      [42., 44.]],

     [[37., 39.],
      [45., 47.]],

     [[38., 40.],
      [46., 48.]]]])

I want to shuffle it using pytorch to produce the following tensor B of shape (1,3,4,4):

tensor([[[[ 1.,  6.,  3.,  8.],
          [21., 34., 23., 36.],
          [ 9., 14., 11., 16.],
          [29., 42., 31., 44.]],

         [[ 2., 17.,  4., 19.],
          [22., 37., 24., 39.],
          [10., 25., 12., 27.],
          [30., 45., 32., 47.]],

         [[ 5., 18.,  7., 20.],
          [33., 38., 35., 40.],
          [13., 26., 15., 28.],
          [41., 46., 43., 48.]]]])

I have implemented this using two for loops as follows:

B = torch.zeros(1,3,4,4, dtype=torch.float)
ctr = 0
for i in range(2):
    for j in range(2):
        B[:,:,i:4:2,j:4:2] = A[:,ctr:ctr+3,:,:]
        ctr = ctr+3

I'm looking for any way to implement this in a vectorized manner in pytorch without these for loops. Maybe using functions like .permute(), etc.

Output:
Just generalising the above solution for any upsampling factor 'r' like in pixel shuffle.

B = A.reshape(-1,r,3,s,s).permute(2,3,0,4,1).reshape(1,3,rs,rs)

Here 's' is the spatial resolution of each channel in 'A' and 'r' is the upsampling factor. For the particular case, r=2 and s=2. This solution should work for arbitary values of 'r' with appropriate size of 'A'.

So for the problem in hand s=2, r=2 and so the solution goes as

B = A.reshape(-1,2,3,2,2).permute(2,3,0,4,1).reshape(1,3,4,4)

as posted by @ddoGas

Similarly if 'A' had been of size (1, 192, 356, 532) and wanting to upsample by r=8 do

B = A.reshape(-1,8,3,356,532).permute(2,3,0,4,1).reshape(1,3,2848,4256)


Instruction: What input shapes are required for a classifier that is using a pre-trained network (Pytorch)?

Input:
I'm fairly new to deeplearning, python, and pytorch so please bear with me!

I'm trying to understand Transfer Learning in Pytorch using two different Pretrained Networks: Vgg11 and Densenet121.
I've run data of shape (3 x 224 x 224) through the "features" part of the above networks, and the output shapes are as follows:

Vgg11 features output shape: 512 x 7 x 7

Densenet121 features output shape: 1024 x 7 x7

Now, I'm trying to make my own Classifier to use instead of the Pre-trained one. Upon checking both pre-trained classifiers, I see the Vgg11 classifier has in the first layer:

  (0): Linear(in_features=25088, out_features=4096, bias=True)

While the Densenet121 has in the first layer:

  (classifier): Linear(in_features=1024, out_features=1000, bias=True))

The Vgg one makes sense, since if you flatten the output of the "features" part, you get 512 x 7 x 7 = 25,088.

How does the Densenet one have only 1024 dimensions? If you flatten the output of its "features" part, you get 1024 x 7 x 7 = 50,176

Are there steps that I am missing for either of them? Are there ways to check the input and output shapes of each layer and find out exactly what's happening?

Thank you.

Output: As mentioned in Table 1 in the DenseNet paper, DenseNet-121 uses something called Global Average Pooling, which is an extreme way of pooling where a tensor of dimensions d x h x w is reduced to d x 1 x 1.


Instruction: Resizing PyTorch tensor with grad to smaller size

Input:
I am trying to downsize a tensor from let's say (3,3) to (1, 1), but I want to keep the original tensor:

import torch

a = torch.rand(3, 3)
a_copy = a.clone()
a_copy.resize_(1, 1)

I need requires_grad=True in my initial tensor but PyTorch forbids this me from trying to resize the copy:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()
a_copy.resize_(1, 1)

Throws an error:

Traceback (most recent call last):
  File "pytorch_test.py", line 7, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Clone and Detach

I tried to .clone() and .detach() as well:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone().detach()

with torch.no_grad():
    a_copy.resize_(1, 1)

which gives this error instead:

Traceback (most recent call last):
  File "pytorch_test.py", line 14, in <module>
    a_copy.resize_(1, 1)
RuntimeError: set_sizes_contiguous is not allowed on a Tensor created from .data or .detach().
If your intent is to change the metadata of a Tensor (such as sizes / strides / storage / storage_offset)
without autograd tracking the change, remove the .data / .detach() call and wrap the change in a `with torch.no_grad():` block.
For example, change:
    x.data.set_(y)
to:
    with torch.no_grad():
        x.set_(y)

This behaviour had been stated in the docs and #15070.

With no_grad()

So, following what they said in the error message, I removed .detach() and used no_grad() instead:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.clone()

with torch.no_grad():
    a_copy.resize_(1, 1)

But it still gives me an error about grad:

Traceback (most recent call last):
  File "pytorch_test.py", line 21, in <module>
    a_copy.resize_(1, 1)
RuntimeError: cannot resize variables that require grad

Similar questions

I have looked at Resize PyTorch Tensor but it the tensor in that example retains all original values.
I have also looked at Pytorch preferred way to copy a tensor which is the method I am using to copy the tensor.

I am using PyTorch version 1.4.0

Output:
I think you should first detach and then clone:

a = torch.rand(3, 3, requires_grad=True)
a_copy = a.detach().clone()
a_copy.resize_(1, 1)

Note: a.detach() returns a new tensor detached from the current graph (it doesn't detach a itself from the graph as a.detach_() does). But because it shares storage with a, you also should clone it. In this way, whatever you do with the a_copy won't affect a. However I am not sure why a.detach().clone() works but a.clone().detach() gives error.

Edit:

The following code also works (which is probably a better solution):

a = torch.rand(3, 3, requires_grad=True)

with torch.no_grad():
    a_copy = a.clone()
    a_copy.resize_(1, 1)


Instruction: How to sort the pytorch tensors by specific key value?

Input:
I'm new to Pytorch. Given a tensor set, I need to sort these tensors by the key value.
For example,

A =
[[0.9133, 0.5071, 0.6222, 3.],
 [0.5951, 0.9315, 0.6548, 1.],
 [0.7704, 0.0720, 0.0330, 2.]]

My expected result after sorting is:

A' =
[[0.5951, 0.9315, 0.6548, 1.],
 [0.7704, 0.0720, 0.0330, 2.],
 [0.9133, 0.5071, 0.6222, 3.]]

I tried to use sorted function in python, but it was time-consuming in my training process.
How to achieve it more efficiently?
Thanks!

Output:
%%timeit -r 10 -n 10
A[A[:,-1].argsort()]

38.6 µs ± 23 µs per loop (mean ± std. dev. of 10 runs, 10 loops each)

%%timeit -r 10 -n 10
sorted(A, key = lambda x: x[-1])

69.6 µs ± 34.8 µs per loop (mean ± std. dev. of 10 runs, 10 loops each)

Both output

tensor([[0.5951, 0.9315, 0.6548, 1.0000],
        [0.7704, 0.0720, 0.0330, 2.0000],
        [0.9133, 0.5071, 0.6222, 3.0000]])

Then there is

%%timeit -r 10 -n 10
a, b = torch.sort(A, dim=-2)

The slowest run took 8.45 times longer than the fastest. This could mean that an intermediate result is being cached.
14.3 µs ± 18.1 µs per loop (mean ± std. dev. of 10 runs, 10 loops each)

with a as the sorted tensor and b as the indices


Instruction: How can I have a PyTorch Conv1d work over a vector?

Input:
I understand Conv1d strides in one dimension. But my input is of shape [64, 20, 161], where 64 is the batches, 20 is the sequence length and 161 is the dimension of my vector.

I'm not sure how to set up my Conv1d to stride over the vector.

I'm trying:

self.conv1 = torch.nn.Conv1d(batch_size, 20, 161, stride=1)

but getting:

RuntimeError: Given groups=1, weight of size 20 64 161, expected input[64, 20, 161] to have 64 channels, but got 20 channels instead

Output:
According to the documentation:

torch.nn.Conv1d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')

in_channels is the number of channels in your input, number of channels usually an computer vision term, in your case this number is 20.
out_channels size of your output, it depends on how much output you want.

For 1D convolution, you can think of number of channels as "number of input vectors" and "number of output feature vectors". And size (not number) of output feature vectors are decided from other parameters like kernel_size, strike, padding, dilation.

An example usage:

t = torch.randn((64, 20, 161))
conv = torch.nn.Conv1d(20, 100)
conv(t)

Note: You never specify batch size in torch.nn modules, first dimension is always assumed to be batch size.


Instruction: how to get value 0 or 1 in siamese neural network?

Input:
I'm trying to use siamese neural network.
Here I want to compare 2 types of images and get the results of the score,
This is the code for the test model to produce the score

in this case i use pytorch

model = Siamese()

# Load state_dict
model.load_state_dict(torch.load('/Users/tania/Desktop/TA/model/model-batch-1001.pth'))

# Create the preprocessing transformation
from torchvision import transforms
transforms = transforms.ToTensor()

# load image(s)
from PIL import Image
x1 = Image.open('table.PNG')
x2 = Image.open('table.PNG')
# Transform

x1 = transforms(x1)
x2 = transforms(x2)

x1 = torch.stack([x1])
x2 = torch.stack([x2])

model.eval()

# Get prediction
output = model(x1,x2)
print (output)

so i got the score like this,

the score is -14.1640

basically in siamese if the image is the same then it produces a value of 1 and if different it will produce a value of 0

how do I get the value of 0 or 1 so that I know whether the image is the same or not?

please help me, i am newbie in neural network

Output:
To get an output between 0 and 1, you need to transform your values with an activation function, in order to map them to a probability. This can be done with the Sigmoid function, defined as:

It returns a probability with range (0,1) (exclusive), where values 0 < y < 0.5 can be interpreted as a negative label, whereas values 0.5 <= y < 1 can be interpreted as being positive. In PyTorch, this can be implemented:

output = model(x1,x2)
output = torch.sigmoid(output)

Hope this helps!


Instruction: Inference time varies over different GPUs using Torch

Input:
I get a bug when running the below inference code. In the function recognize(), it takes 0.4s to finish prediction. It takes another 3s to return the result preds_str to the caller function. I found that if I set gpu_id=0 in file config, it returns instantly. How can I fix this bug? Thanks in advance.

def recognize(imgs, model, demo_loader):

          t = time()
          model.eval()
          with torch.no_grad():
              for image_tensors, image_path_list in demo_loader:
                    batch_size = image_tensors.size(0)
                    image = image_tensors.to(config.device)
                    # For max length prediction
                    length_for_pred = torch.IntTensor([config.batch_max_length] * batch_size).to(config.device)
                    text_for_pred = torch.LongTensor(batch_size, config.batch_max_length + 1).fill_(0).to(config.device)

                    preds = model(image, text_for_pred, is_train=False)
                    _, preds_index = preds.max(2)
                    preds_str = converter.decode(preds_index, length_for_pred)

           print('time elapsed before return:'time()-t) #0.4s
           return preds_str
def main():
     model = Model()
     self.model.cuda(config.device)
     model = torch.nn.DataParallel(model, device_ids=[config.device], output_device=[config.device]).to(config.device)
     model.load_state_dict(torch.load(config.saved_model, map_location=config.device))
     AlignCollate_demo = AlignCollate(imgH=config.imgH, imgW=config.imgW, keep_ratio_with_pad=config.PAD)
     imgs_dataset = ImageDataset(imgs)
     demo_loader = torch.utils.data.DataLoader(imgs_dataset, batch_size=config.batch_size,shuffle=False,num_workers=int(config.workers),collate_fn=AlignCollate_demo, pin_memory=True)
     start_time = time()
     # imgs = [img1, img2, ....]
     preds_str = recognize(imgs, model, demo_loader)
     print('time elapsed after return', time()-start_time) #3.4s

Config file:

    class ConfigWordRecognizer:
        gpu_id = 1 #troublesome line here
        device = torch.device('cuda:{}'.format(gpu_id) if torch.cuda.is_available() else 'cpu')
        imgH = 32
        imgW = 100
        batch_size = 80
        workers = 8
        batch_max_length = 25

Output:
I found the solution from this post.
I set CUDA_VISIBLE_DEVICES=1, gpu_id=0. Then, I remove

model = torch.nn.DataParallel(model, device_ids=[config.device], output_device=[config.device]).to(config.device)

and change

model.load_state_dict(torch.load(config.saved_model, map_location=config.device))

to

model.load_state_dict(self.copyStateDict(torch.load(self.config.saved_model, map_location=self.config.device)))

Copy stateDict function:

def copyStateDict(self, state_dict):
        if list(state_dict.keys())[0].startswith("module"):
            start_idx = 1
        else:
            start_idx = 0
        new_state_dict = OrderedDict()
        for k, v in state_dict.items():
            name = ".".join(k.split(".")[start_idx:])
            new_state_dict[name] = v
        return new_state_dict

The model works well on gpu1. But I still don't understand why if I set 'gpu_id=0', it works well on gpu0 without copyStateDict


Instruction: How to apply OpenCV filters on Pytorch dataset?

Input:
I have used below steps to pre-process single image using OpenCV. Now, I want to apply these pre-processing steps to my entire dataset before training the model in Pytorch. How can this be done?

im = cv2.imread(image_path)
im_nonoise = cv2.medianBlur(im, 3)
imgray = cv2.cvtColor(im_nonoise,cv2.COLOR_BGR2GRAY)
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
cl1 = clahe.apply(imgray)
ret,thresh = cv2.threshold(cl1,110,255,0)
image, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
img = cv2.drawContours(image, contours, -1, (250,100,120))

And I load the data using

data = datasets.ImageFolder(train_dir,transform=transform)
train_loader = torch.utils.data.DataLoader(data,batch_size=batch_size,sampler=train_sampler)

Output:
You can build your own dataset class (derived from ImageFolder) and overload only the __getitem__ method:

class MySpecialDataset(datasets.ImageFolder):
  def __init__(self, root, loader=default_loader, is_valid_file=None):
    super(MySpecialDataset, self).__init__(root=root, loader=loader, is_valid_file=is_valid_file)

  def __getitem__(self, index):
    image_path, target = self.samples[index]
    # do your magic here
    im = cv2.imread(image_path)
    im_nonoise = cv2.medianBlur(im, 3)
    imgray = cv2.cvtColor(im_nonoise,cv2.COLOR_BGR2GRAY)
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
    cl1 = clahe.apply(imgray)
    ret,thresh = cv2.threshold(cl1,110,255,0)
    image, contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
    img = cv2.drawContours(image, contours, -1, (250,100,120))
    # you need to convert img from np.array to torch.tensor
    # this has to be done CAREFULLY!
    sample = torchvision.transforms.ToTensor()(img)
    return sample, target

Once you have this dataset, you can use it with the basic pytorch's DataLoader:

data = MySpecialDataset(train_dir)
train_loader = torch.utils.data.DataLoader(data,batch_size=batch_size,sampler=train_sampler)


Instruction: AttributeError: ‘NoneType’ object has no attribute ‘register_forward_hook’

Input:
I’m trying to register a forward hook function to the last conv layer of my network. I first printed out the names of the modules via:

for name, _ in model.named_modules():
    print(name)

Which gave me "0.conv" as the module name. However, when I tried to do the following, the above error was triggered by line 4:

def hook_feature(module, in_, out_):
    features.append(out_.cpu().data.numpy())

model._modules.get("0.conv").register_forward_hook(hook_feature)

Here is my named_modules() output:

...
0.decoder1.dec1conv2
0.decoder1.dec1norm2
0.decoder1.dec1relu2
0.conv
1
1.outc1
1.outc2

What am I doing wrong and how do I fix it? Thanks!

Output:
Your modules are stored in an hierarchical way. To get to '0.conv', you need to

models._modules["0"]._modules.get("conv").register_forward_hook(hook_feature)


Instruction: Why does a torch gradient increase linearly every time the function is backpropagated?

Input:
I am trying to understand how PyTorch backpropagation works, using the following code.

import torch
import numpy
x = torch.tensor(numpy.e, requires_grad=True)
y = torch.log(x)
y.backward()
print(x.grad)

The result is tensor(0.3679), as expected, which is 1 / x, which is the derivative of log(x) w.r.t. x with x = numpy.e. However, if I run the last 3 lines again WITHOUT re-assigning x, i.e. do

y = torch.log(x)
y.backward()
print(x.grad)

then I will get tensor(0.7358), which is twice the previous result.
Why does this happen?

Output:
Gradients are accumulated until cleared. From the docs (emphasis mine):

  This function accumulates gradients in the leaves - you might need to zero them before calling it.

This zeroing can be done by way of x.grad.zero_() or, in the case of a torch.optim.Optimizer, optim.zero_grad().


Instruction: RuntimeError occurs in PyTorch backward function

Input:
I am trying to calculate the grad of a variable in PyTorch. However, there was a RuntimeError which tells me that the shape of output and grad must be the same. However, in my case, the shape of output and grad cannot be the same. Here is my code to reproduce:

import numpy as np
import torch
from torch.autograd import Variable as V

ne = 3
m, n = 79, 164
G = np.random.rand(m, n).astype(np.float64)
w = np.random.rand(n, n).astype(np.float64)
z = -np.random.rand(n).astype(np.float64)

G = V(torch.from_numpy(G))
w = V(torch.from_numpy(w))
z = V(torch.from_numpy(z), requires_grad=True)
e, v = torch.symeig(torch.diag(2 * z - torch.sum(w, dim=1)) + w, eigenvectors=True, upper=False)
ssev = torch.sum(torch.pow(e[-ne:] * v[:, -ne:], 2), dim=1)
out = torch.sum(torch.matmul(G, ssev.reshape((n, 1))))
out.backward(z)
print(z.grad)

The error message is: RuntimeError: Mismatch in shape: grad_output[0] has a shape of torch.Size([164]) and output[0] has a shape of torch.Size([])

Similar calculation is allowed in TensorFlow and I can successfully get the gradient I want:

import numpy as np
import tensorflow as tf

m, n = 79, 164
G = np.random.rand(m, n).astype(np.float64)
w = np.random.rand(n, n).astype(np.float64)
z = -np.random.rand(n).astype(np.float64)

def tf_function(z, G, w, ne=3):
    e, v = tf.linalg.eigh(tf.linalg.diag(2 * z - tf.reduce_sum(w, 1)) + w)
    ssev = tf.reduce_sum(tf.square(e[-ne:] * v[:, -ne:]), 1)
    return tf.reduce_sum(tf.matmul(G, tf.expand_dims(ssev, 1)))

z, G, w = [tf.convert_to_tensor(_, dtype=tf.float64) for _ in (z, G, w)]
z = tf.Variable(z)
with tf.GradientTape() as g:
    g.watch(z)
    out = tf_function(z, G, w)
print(g.gradient(out, z).numpy())

My tensorflow version is 2.0 and my PyTorch version is 1.14.0. I am using Python3.6.9. In my opinion, calculating the gradients when the output and the variables have different shapes is very reasonable and I don't think I made any mistake.Can anyone help me with this problem? I really appreciate it!

Output:
First of all you don't need to use numpy and then convert to Variable (which is deprecated by the way), you can just use G = torch.rand(m, n) etc. Second, when you write out.backward(z), you are passing z as the gradient of out, i.e. out.backward(gradient=z), probably due to the misconception that "out.backward(z) computes the gradient of z, i.e. dout/dz". Instead, this argument is meant to be gradient = d[f(out)]/dout for some function f (e.g. a loss function) and it's the tensor used to compute vector-Jacobian product dout/dz * df/dout. Therefore, the reason why you got the error is because your out (and its gradient df/dout) is a scalar (zero-dimensional tensor) and z is a tensor of size n, leading to a mismatch in shapes.
To fix the problem, as you have already figured out by yourself, just replace out.backward(z) with out.backward(), which is equivalent to out.backward(gradient=torch.tensor(1.)), since in your case out is a scalar and f(out) = out, so d[f(out)]/dout = d(out)/d(out) = tensor(1.). If your out was a non-scalar tensor, then out.backward() would not work and instead you would have to use out.backward(torch.ones(out.shape)) (again assuming that f(out) = out). In any case, if you need to pass gradient to the out.backward(), make sure that it has the same shape as the out.


Instruction: PyTorch AutoEncoder - Decoded output dimension not the same as input

Input:
I am building a Custom Autoencoder to train on a dataset. My model is as follows

class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder,self).__init__()

        self.encoder = nn.Sequential(
        nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=128,out_channels=256,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=256,out_channels=512,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=512,out_channels=1024,kernel_size=5,stride=2),
        nn.ReLU(inplace=True)
        )

        self.decoder = nn.Sequential(
        nn.ConvTranspose2d(in_channels=1024,out_channels=512,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=512,out_channels=256,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=256,out_channels=128,kernel_size=5,stride=2),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=3,stride=1),
        nn.ReLU(inplace=True),
        nn.ConvTranspose2d(in_channels=32,out_channels=3,kernel_size=3,stride=1),
        nn.ReLU(inplace=True)
        )

    def forward(self,x):
        x = self.encoder(x)
        print(x.shape)
        x = self.decoder(x)
        return x

def unit_test():
    num_minibatch = 16
    img = torch.randn(num_minibatch, 3, 512, 640).cuda(0)
    model = AutoEncoder().cuda()
    model = nn.DataParallel(model)
    output = model(img)
    print(output.shape)

if __name__ == '__main__':
    unit_test()

As you can see, my input dimension is (3, 512, 640) but my output after passing it through the decoder is (3, 507, 635). Am I missing something while adding the Conv2D Transpose layers ?

Any help would be appreciated. Thanks

Output:
The mismatch is caused by the different output shapes of ConvTranspose2d layer. You can add output_padding of 1 to first and third transpose convolution layer to solve this problem.

i.e. nn.ConvTranspose2d(in_channels=1024,out_channels=512,kernel_size=5,stride=2, output_padding=1) and nn.ConvTranspose2d(in_channels=256,out_channels=128,kernel_size=5,stride=2, output_padding=1)

As per the documentation:

  When stride > 1, Conv2d maps multiple input shapes to the same output shape. output_padding is provided to resolve this ambiguity by effectively increasing the calculated output shape on one side.

Decoder layers' shapes before adding output_padding:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
   ConvTranspose2d-1        [-1, 512, 123, 155]      13,107,712
              ReLU-2        [-1, 512, 123, 155]               0
   ConvTranspose2d-3        [-1, 256, 249, 313]       3,277,056
              ReLU-4        [-1, 256, 249, 313]               0
   ConvTranspose2d-5        [-1, 128, 501, 629]         819,328
              ReLU-6        [-1, 128, 501, 629]               0
   ConvTranspose2d-7         [-1, 64, 503, 631]          73,792
              ReLU-8         [-1, 64, 503, 631]               0
   ConvTranspose2d-9         [-1, 32, 505, 633]          18,464
             ReLU-10         [-1, 32, 505, 633]               0
  ConvTranspose2d-11          [-1, 3, 507, 635]             867
             ReLU-12          [-1, 3, 507, 635]               0

After adding padding:

================================================================
   ConvTranspose2d-1        [-1, 512, 124, 156]      13,107,712
              ReLU-2        [-1, 512, 124, 156]               0
   ConvTranspose2d-3        [-1, 256, 251, 315]       3,277,056
              ReLU-4        [-1, 256, 251, 315]               0
   ConvTranspose2d-5        [-1, 128, 506, 634]         819,328
              ReLU-6        [-1, 128, 506, 634]               0
   ConvTranspose2d-7         [-1, 64, 508, 636]          73,792
              ReLU-8         [-1, 64, 508, 636]               0
   ConvTranspose2d-9         [-1, 32, 510, 638]          18,464
             ReLU-10         [-1, 32, 510, 638]               0
  ConvTranspose2d-11          [-1, 3, 512, 640]             867
             ReLU-12          [-1, 3, 512, 640]               0


Instruction: How to combine datasets in PyTorch to return image and numpy file simultaneously

Input:
I am trying to build a dataloader that will take images and poses. The images are saved in the form of .jpg files, and the poses in the form of .npy files. The images and poses are in different folders but have the same sub-folder structure and name. The sub-folders are in the form of classes, i.e., each class has a corresponding folder. I want to apply image transformations and then return the images (for which I am using torchvision datasets.ImageFolder). For the poses, I am using torchvision datasets.DatasetFolder. How do I combine these two datasets so that I get both pose and image of the same name simultaneously?

class ReIDFolder_images(datasets.ImageFolder):

    def __init__(self, root, transform):
        super().__init__(root, transform)
        targets = np.asarray([s[1] for s in self.samples])
        self.targets = targets
        self.img_num = len(self.samples)
        print(self.img_num)

    def _get_cam_id(self, path):
        camera_id = []
        filename = os.path.basename(path)
        camera_id = filename.split('c')[1][0]
        return int(camera_id)-1

    def _get_pos_sample(self, target, index, path):
        pos_index = np.argwhere(self.targets == target)
        pos_index = pos_index.flatten()
        pos_index = np.setdiff1d(pos_index, index)
        if len(pos_index)==0:  # in the query set, only one sample
            return path
        else:
            rand = random.randint(0,len(pos_index)-1)
        return self.samples[pos_index[rand]][0]

    def _get_neg_sample(self, target):
        neg_index = np.argwhere(self.targets != target)
        neg_index = neg_index.flatten()
        rand = random.randint(0,len(neg_index)-1)
        return self.samples[neg_index[rand]]

    def __getitem__(self, index):
        path, target = self.samples[index]
        sample = self.loader(path)

        pos_path = self._get_pos_sample(target, index, path)
        pos = self.loader(pos_path)

        if self.transform is not None:
            sample = self.transform(sample)
            pos = self.transform(pos)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return sample, target, pos

class ReIDFolder_poses(datasets.DatasetFolder):

    def __init__(self, root):
        super().__init__(root, loader=self.npy_loader, extensions='.npy')

        targets = np.asarray([s[1] for s in self.samples])
        self.targets = targets
        self.img_num = len(self.samples)
        print(self.img_num)

    def npy_loader(self, path):
        sample = torch.Tensor(np.load(path))
        return sample

    def _get_cam_id(self, path):
        camera_id = []
        filename = os.path.basename(path)
        camera_id = filename.split('c')[1][0]
        return int(camera_id)-1

    def _get_pos_sample(self, target, index, path):
        pos_index = np.argwhere(self.targets == target)
        pos_index = pos_index.flatten()
        pos_index = np.setdiff1d(pos_index, index)
        if len(pos_index)==0:  # in the query set, only one sample
            return path
        else:
            rand = random.randint(0,len(pos_index)-1)
        return self.samples[pos_index[rand]][0]

    def _get_neg_sample(self, target):
        neg_index = np.argwhere(self.targets != target)
        neg_index = neg_index.flatten()
        rand = random.randint(0,len(neg_index)-1)
        return self.samples[neg_index[rand]]

    def __getitem__(self, index):
        path, target = self.samples[index]
        sample = self.loader(path)

        pos_path = self._get_pos_sample(target, index, path)
        pos = self.loader(pos_path)

        return sample, target, pos

Output: I was able to solve this problem! It turns out I didn't have to inherit datasets.DatasetFolder. Since the labels were the same, I just created one class which inherits datasets.ImageFolder, and fed a modified path to the function npy_loader.


Instruction: how to use torchaudio with torch xla on google colab tpu

Input:
I'm trying to run a pytorch script which is using torchaudio on a google TPU. To do this I'm using pytorch xla following this notebook, more specifically I'm using this code cell to load the xla:

!pip install torchaudio
import os
assert os.environ['COLAB_TPU_ADDR'], 'Make sure to select TPU from Edit > Notebook settings > Hardware accelerator'

VERSION = "20200220"  #@param ["20200220","nightly", "xrt==1.15.0"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

import torch

import torchaudio

import torch_xla

however this is incompatible with the version of torchaudio that I need as: ERROR: torchaudio 0.4.0 has requirement torch==1.4.0, but you'll have torch 1.5.0a0+e95282a which is incompatible.

I couldn't find anywhere how to load torch 1.4.0 using pytorch xla.

I tried to use the nightly version of torch audio but that gives the error as follows:

!pip install torchaudio_nightly -f https://download.pytorch.org/whl/nightly/torch_nightly.html

import os
assert os.environ['COLAB_TPU_ADDR'], 'Make sure to select TPU from Edit > Notebook settings > Hardware accelerator'

VERSION = "20200220"  #@param ["20200220","nightly", "xrt==1.15.0"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

import torch
import torchaudio

import torch_xla

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-968e9d93c06f> in <module>()
      9
     10 import torch
---> 11 import torchaudio
     12
     13 import torch_xla

/usr/local/lib/python3.6/dist-packages/torchaudio/__init__.py in <module>()
      3
      4 import torch
----> 5 import _torch_sox
      6
      7 from .version import __version__, git_version

ImportError: /usr/local/lib/python3.6/dist-packages/_torch_sox.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_29E

---------------------------------------------------------------------------

So how would I go to load the stable version or 1.4.0 version of pytorch using xla or is there any other workaround for this situation?

Thanks a lot for your help!

Output:
I tested using the notebook below;
Getting Started with PyTorch on Cloud TPUs

After changing the cell containing;

FROM:

    VERSION = "20200325"  #@param ["1.5" , "20200325", "nightly"]
    !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
    !python pytorch-xla-env-setup.py --version $VERSION

TO:

    VERSION = "20200325"  #@param ["1.5" , "20200325", "nightly"]
    !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
    !pip install torchvision
    !pip install torch==1.4.0
    !pip install torchaudio==0.4.0
    %matplotlib inline
    !python pytorch-xla-env-setup.py --version $VERSION

All cells ran successfully, and the import statements below threw no errors;

    # imports pytorch
      import torch

    # imports the torch_xla package
      import torch_xla
      import torch_xla.core.xla_model as xm


Instruction: Using fastai (pytorch) in c# code, how to normalize Bitmap with mean and std?

Input:
A few days ago i switched from tensorflow to fastai for my c# Project. But now i am facing a problem with my normalisation. For both i use an onnx pipeline to load the model and the data.

var onnxPipeline = mLContext.Transforms.ResizeImages(resizing: ImageResizingEstimator.ResizingKind.Fill, outputColumnName: inputName,
                                                                 imageWidth: ImageSettings.imageWidth, imageHeight: ImageSettings.imageHeight,
                                                                 inputColumnName: nameof(ImageInputData.Image))
                .Append(mLContext.Transforms.ExtractPixels(outputColumnName: inputName, interleavePixelColors: true, scaleImage: 1 / 255f))
                .Append(mLContext.Transforms.ApplyOnnxModel(outputColumnName: outputName, inputColumnName: inputName, modelFile: onnxModelPath));

var emptyData = mLContext.Data.LoadFromEnumerable(new List<ImageInputData>());
var onnxModel = onnxPipeline.Fit(emptyData);

with

    class ImageInputData
    {
        [ImageType(ImageSettings.imageHeight, ImageSettings.imageWidth)]
        public Bitmap Image { get; set; }

        public ImageInputData(byte[] image)
        {
            using (var ms = new MemoryStream(image))
            {
                Image = new Bitmap(ms);
            }
        }
        public ImageInputData(Bitmap image)
        {
            Image = image;
        }
    }

After using fastai i learned, that the models get better accuracy if the data is normalized with a specific mean and standard deviation (because i used the resnet34 model it should be means { 0.485, 0.456, 0.406 } stds = { 0.229, 0.224, 0.225 } respectively).
So the pixelvalues (for each color ofc.) have to be transformed with those values to match the trainings images. But how can i achive this in C#?
What i tried so far is:

int imageSize = 256;
double[] means = new double[] { 0.485, 0.456, 0.406 }; // used in fastai model
double[] stds = new double[] { 0.229, 0.224, 0.225 };
Bitmap bitmapImage = inputBitmap;
Image image = bitmapImage;

Color[] pixels = new Color[imageSize * imageSize];
for (int x = 0; x < bitmapImage.Width; x++)
{
   for (int y = 0; y < bitmapImage.Height; y++)
   {
      Color pixel = bitmapImage.GetPixel(x, y);
      pixels[x + y] = pixel;
      double red = (pixel.R - (means[0] * 255)) / (stds[0] * 255); // *255 to scale the mean and std values to the Bitmap
      double gre = (pixel.G - (means[1] * 255)) / (stds[1] * 255);
      double blu = (pixel.B - (means[2] * 255)) / (stds[2] * 255);
      Color pixel_n = Color.FromArgb(pixel.A, (int)red, (int)gre, (int)blu);
      bitmapImage.SetPixel(x, y, pixel_n);

   }
}

Ofcourse its not working, because the Colorvalues can`t be negative (which i realised only later).
But how can i achive this normalisation between -1 and 1 for my model in C# with the onnx-model?

Is there a different way to feed the model or to handle the normalisation?

Any help would be appreciated!

Output:
One way to solve this problem is to switch from an onnx pipeline to an onnx Inferencesession, which is in my view simpler and better to understand:

public List<double> UseOnnxSession(Bitmap image, string onnxModelPath)
{
      double[] means = new double[] { 0.485, 0.456, 0.406 };
      double[] stds = new double[] { 0.229, 0.224, 0.225 };

      using (var session = new InferenceSession(onnxModelPath))
      {
          List<double> scores = new List<double>();
          Tensor<float> t1 = ConvertImageToFloatData(image, means, stds);
          List<float> fl = new List<float>();

          var inputMeta = session.InputMetadata;
          var inputs = new List<NamedOnnxValue>()
          {
             NamedOnnxValue.CreateFromTensor<float>("input_1", t1)
          };
          using (var results = session.Run(inputs))
          {
              foreach (var r in results)
              {
                  var x = r.AsTensor<float>().First();
                  var y = r.AsTensor<float>().Last();
                  var softmaxScore = Softmax(new double[] { x, y });
                  scores.Add(softmaxScore[0]);
                  scores.Add(softmaxScore[1]);
              }
           }
           return scores;
       }
}

// Create your Tensor and add transformations as you need.
public static Tensor<float> ConvertImageToFloatData(Bitmap image, double[] means, double[] std)
{
      Tensor<float> data = new DenseTensor<float>(new[] { 1, 3, image.Width, image.Height });
      for (int x = 0; x < image.Width; x++)
      {
           for (int y = 0; y < image.Height; y++)
           {
               Color color = image.GetPixel(x, y);
               var red = (color.R - (float)means[0] * 255) / ((float)std[0] * 255);
               var gre = (color.G - (float)means[1] * 255) / ((float)std[1] * 255);
               var blu = (color.B - (float)means[2] * 255) / ((float)std[2] * 255);
               data[0, 0, x, y] = red;
               data[0, 1, x, y] = gre;
               data[0, 2, x, y] = blu;
            }
       }
       return data;
}

Also i have to use my own Softmax method on these scores to get the real probabilities out of my model:

        public double[] Softmax(double[] values)
        {
            double[] ret = new double[values.Length];
            double maxExp = values.Select(Math.Exp).Sum();
            for (int i = 0; i < values.Length; i++)
            {
                ret[i] = Math.Round((Math.Exp(values[i]) / maxExp), 4);
            }
            return ret;
        }

Hope this helps someone who has a similar Problem.


Instruction: Batches of points with the same label on Pytorch

Input:
I want to train a neural network using gradient descent on batches that contain N training points each. I would like these batches to only contain points with the same label, instead of being randomly sampled from the training set.

For example, if I'm training using MNIST, I would like to have batches that look like the following:

batch_1 = {0,0,0,0,0,0,0,0}

batch_2 = {3,3,3,3,3,3,3,3}

batch_3 = {7,7,7,7,7,7,7,7}

.....

and so on.

How can I do it using pytorch?

Output:
One way to do it is to create subsets and dataloaders for each class and then iterate by randomly switching between the dataloaders at each iteration:

import torch
from torch.utils.data import DataLoader, Subset
from torchvision.datasets import MNIST
from torchvision import transforms
import numpy as np

dataset = MNIST('path/to/mnist_root/',
                transform=transforms.ToTensor(),
                download=True)

class_inds = [torch.where(dataset.targets == class_idx)[0]
              for class_idx in dataset.class_to_idx.values()]

dataloaders = [
    DataLoader(
        dataset=Subset(dataset, inds),
        batch_size=8,
        shuffle=True,
        drop_last=False)
    for inds in class_inds]

epochs = 1

for epoch in range(epochs):
    iterators = list(map(iter, dataloaders))
    while iterators:
        iterator = np.random.choice(iterators)
        try:
            images, labels = next(iterator)
            print(labels)
            # do_more_stuff()

        except StopIteration:
            iterators.remove(iterator)

This will work with any dataset (not just the MNIST).
Here's the result of printing the labels at each iteration:

tensor([6, 6, 6, 6, 6, 6, 6, 6])
tensor([3, 3, 3, 3, 3, 3, 3, 3])
tensor([0, 0, 0, 0, 0, 0, 0, 0])
tensor([5, 5, 5, 5, 5, 5, 5, 5])
tensor([8, 8, 8, 8, 8, 8, 8, 8])
tensor([0, 0, 0, 0, 0, 0, 0, 0])
...
tensor([1, 1, 1, 1, 1, 1, 1, 1])
tensor([1, 1, 1, 1, 1, 1])

Note that by setting drop_last=False, there will be batches, here and there, with less than batch_size elements. By setting it to True, the batches will be all of equal size, but some data points will be dropped.


Instruction: Getting ModuleNotFoundError while the module is next to the calling file

Input:
I am in a weird situation where in order for my package to be called just fine, one single file (models.py) needs to be next to the calling script and also inside the package folder where it is used.

To make it a bit more clear, this is how the package organization looks like :

-FV_dir
 ---__init__.py
 ---F_V.py
 ---models.py
 ---utils.py
 ---service_utils.py
 ---subModule1_dir
    ----__init__.py
    ----detector.py
    ----utils.py
    ----subdir1
        --- etc
    ----subdir2
        --- etc
    ----subdir3

This whole package is placed in the site-packages, so it's usable system-wide. And there is a user script that uses this package like this:
service_client.py:

from FV.service_utils import ServiceCore
from FV.utils import a_helper_function

def run():
  service = ServiceCore()
  service.run()

if __name__ == "__main__":
   run()

The ServiceCore itself uses F_V.py which is the main module here. The F_V module itself uses the models.py and utils.py next to it like  this :

F_V.py :

from FV.utils import func1, func2
from FV.models import model1, model2, model3
...

Now the problem is, if the models.py is not next to the client code (service_client.py) it just complains that the module is not found : here is an example error I get when this is the case:

           └─19146 /home/user1/anaconda3/bin/python3 /home/user1/Documents/service_client.py

Mar 17 19:22:54 ubuntu python3[19146]:     self.fv = FaceVerification(**cfg['Face_Verification']['ARGS'])
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 58, in __init__
Mar 17 19:22:54 ubuntu python3[19146]:     self._init_model()
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 80, in _init_model
Mar 17 19:22:54 ubuntu python3[19146]:     checkpoint = torch.load(self.model_checkpoint_path, map_location=torch.device('cpu'))
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 529, in
Mar 17 19:22:54 ubuntu python3[19146]:     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
Mar 17 19:22:54 ubuntu python3[19146]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 702, in
Mar 17 19:22:54 ubuntu python3[19146]:     result = unpickler.load()
Mar 17 19:22:54 ubuntu python3[19146]: ModuleNotFoundError: No module named 'models'

If I remove the models.py next to the F_V.py obviously F_V.py would complain, as it's directly using it :

           └─19216 /home/user1/anaconda3/bin/python3 /home/user1/Documents/fv_service_linux.py

Mar 17 19:27:33 ubuntu systemd[1532]: Started FV Service.
Mar 17 19:27:34 ubuntu python3[19216]: Traceback (most recent call last):
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/Documents/fv_service_linux.py", line 83, in <module>
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.service_utils import ServiceCore
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/service_utils.py", line 21, in <mod
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.F_V import FaceVerification
Mar 17 19:27:34 ubuntu python3[19216]:   File "/home/user1/anaconda3/lib/python3.7/site-packages/FV/F_V.py", line 17, in <module>
Mar 17 19:27:34 ubuntu python3[19216]:     from FV.models import resnet18, resnet50, resnet101
Mar 17 19:27:34 ubuntu python3[19216]: ModuleNotFoundError: No module named 'FV.models'

So the only way to get this to work is to have models.py next to the client code as well. I can't understand why this is happening, as the client code doesn't even directly interacts with models.py. What am I missing here?

Update:

In the F_V.py I use Pytorch and load a pretrained model. I thought this was unrelated and I was doing something wrong with regards to the packaging in Python, however, it turns out this was indeed the culprit. read the answer for more information.

Output:
This actually happened to be a Pytorch related issue, in which this whole mess has been caused by this simple command when loading the model inside F_V.py :

if self.device == 'cpu':
    checkpoint = torch.load(self.model_checkpoint_path, map_location=torch.device('cpu'))
else:
    checkpoint = torch.load(self.model_checkpoint_path)
# the culprit!
self.model = checkpoint['model'].module

This way of storing and loading the model is very bad, the pretrained models were initially wrapper around by a nn.DataParallel and the person who saved them, did it like this :

def save_checkpoint(epoch, epochs_since_improvement, model, metric_fc, optimizer, acc, is_best):
    print('saving checkpoint ...')
    state = {'epoch': epoch,
             'epochs_since_improvement': epochs_since_improvement,
             'acc': acc,
             'model': model,
             'metric_fc': metric_fc,
             'optimizer': optimizer}
    # filename = 'checkpoint_' + str(epoch) + '_' + str(loss) + '.tar'
    filename = 'checkpoint.tar'
    torch.save(state, filename)
    # If this checkpoint is the best so far, store a copy so it doesn't get overwritten by a worse checkpoint
    if is_best:
        torch.save(state, 'BEST_checkpoint.tar')

As you can see, he used the whole model ('model': model) in the state_dict and save that away. He should have used the state_dict().
This is bad since the same files/dir structure hierarchy /everything must be retained all the time, everywhere that this model is going to be used.
and we were affected by this as well as you can see. All the client services
relied on models.py and needed it to be next to them while they didn't even use it.
Initially I thought therefore in order to solve the issue, we have to instantiate the models ourselves and then load the weights manually.

if self.model_name == 'r18':
    self.model = resnet18(pretrained=False, use_se=use_se)
elif self.model_name == 'r50':
    self.model = resnet50(pretrained=False, use_se=use_se)
elif self.model_name == 'r101':
    self.model = resnet101(pretrained=False, use_se=use_se)
else:
    raise Exception(f"Model name: '{self.model_name}' is not recognized.")

# load the model weights
self.model.load_state_dict(checkpoint['model'].module.state_dict())

Note that since the model was initially a nn.DataParallel model, in order to access the model itself, we use the .module property and then use the models state_dict() to initialized the model and hopefully this solves the issue.
However, it seems this is not the case, and since the model is saved that way, it seems there is no way to get rid of such dependencies this way.
instead convert your model into torch script and then save the model.
This way you can get rid of all the nuisance .

Solution 1:

Try converting your model into torch script and then use that instead:

def convert_model(model, input=torch.tensor(torch.rand(size=(1,3,112,112)))):
        model = torch.jit.trace(self.model, input)
        torch.jit.save(model,'/home/Rika/Documents/models/model.tjm')

and then loaded this version instead:

# load the model
self.model = torch.jit.load('/home/Rika/Documents/models/model.tjm')

Solution 2:

simply save the model's state_dict() again and use  that instead :
I myself ended up doing :

self.model = checkpoint['model'].module
# create the new checkpoint based on what you need
torch.save({'state_dict' : self.model.state_dict(), 'use_se':True},
            '/home/Rika/Documents/BEST_checkpoint_r18_2.tar')

and started using the new checkpoint and so far everything has been good


Instruction: L1 regulariser Pytorch acting opposite to what I expect

Input:
I'm trying to add an L1 penalty to a specific layer of a neural network, and I have the code below (in which I attempt to add l1 penalty to the first layer). If I run it for lambda = 0 (i.e. no penalty), the output gets very close to the expected weights those being [10, 12, 2, 11, -0.25]) and if I run for enough epochs or reduce batch size it will get it exactly, as in the output below:

  mlp.0.weight

  Parameter containing:

  tensor([[  9.8657, -11.8305,   2.0242,  10.8913,  -0.1978]],
         requires_grad=True)

Then, when I run it for a large lambda, say 1000, I would expect these weights to shrink towards zero as there is a large penalty being added to the loss that we are trying to minimise. However, the opposite happens and the weights explode, as in the output below (for lam = 1000)

  mlp.0.weight

  Parameter containing:

  tensor([[-13.9368,   9.9072,   2.2447, -11.6870,  26.7293]],
         requires_grad=True)

If anyone could help me, that'd be great. I'm new to pytorch (but not the idea of regularisation), so I'm guessing it's something in my code that is the problem.

Thanks

import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
from sklearn.linear_model import LinearRegression

class TrainDataset(Dataset):
    def __init__(self, data):
        self.data = data
    def __len__(self):
        return self.data.shape[0]
    def __getitem__(self, ind):
        x = self.data[ind][1:]
        y = self.data[ind][0]
        return x, y

class TestDataset(TrainDataset):
    def __getitem__(self, ind):
        x = self.data[ind]
        return x

torch.manual_seed(94)

x_train = np.random.rand(1000, 5)
y_train = x_train[:, 0] * 10 - x_train[:, 1] * 12 + x_train[:, 2] * 2 + x_train[:, 3] * 11 - x_train[:, 4] * 0.25
y_train = y_train.reshape(1000, 1)
x_train.shape
y_train.shape
train_data = np.concatenate((y_train, x_train), axis=1)

train_set = TrainDataset(train_data)

batch_size = 100
train_loader = DataLoader(train_set, batch_size=batch_size, shuffle=True)

class MLP(nn.Module):
    def __init__(self):
        super(MLP, self).__init__()
        self.mlp = nn.Sequential(nn.Linear(5, 1, bias=False))

    def forward(self, x_mlp):
        out = self.mlp(x_mlp)
        return out

device = 'cpu'

model = MLP()

optimizer = torch.optim.SGD(model.parameters(), lr=0.02, momentum=0.82)

criterion = nn.MSELoss()

epochs = 5
lam = 0
model.train()

for epoch in range(epochs):
    losses = []
    for batch_num, input_data in enumerate(train_loader):
        optimizer.zero_grad()

        x, y = input_data
        x = x.to(device).float()
        y = y.reshape(batch_size, 1)
        y = y.to(device)

        output = model(x)
        for name, param in model.named_parameters():
            if name == 'mlp.0.weight':
                l1_norm = torch.norm(param, 1)

        loss = criterion(output, y) + lam * l1_norm
        loss.backward()

        optimizer.step()
        print('\tEpoch %d | Batch %d | Loss %6.2f' % (epoch, batch_num, loss.item()))

for name, param in model.named_parameters():
    if param.requires_grad:
        print(name)
        print(param)

Output: I found that if I use Adagrad as the optimiser instead of SGD, it acts as expected. Will need to look into the difference of those now, but this can be considered answered.


Instruction: Invalid device id when using pytorch dataparallel！

Input:
Environment：

Win10
Pytorch 1.3.0
python3.7

Problem：

I am using dataparallel in Pytorch to use the two 2080Ti GPUs. Code are like below:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = Darknet(opt.model_def)
model.apply(weights_init_normal)

model = nn.DataParallel(model, device_ids=[0, 1]).to(device)

But when run this code, I encounter errors below:

Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/PyTorch-YOLOv3-master/train.py", line 74, in <module>
    model = nn.DataParallel(model, device_ids=[0, 1]).to(device)
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 133, in __init__
    _check_balance(self.device_ids)
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 19, in _check_balance
    dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 19, in <listcomp>
    dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\cuda\__init__.py", line 337, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id

When I debug into it, I find the function device_count() in get_device_properties() returns 1 while I have 2 GPU on my machine. And torch._C._cuda_getDeviceCount() returns 2 in Anaconda Prompt. What is wrong?

Qustion:

How to solve this problem?
How can I manage to use the two GPUs using dataparallel?
Thank you guys!

Output:
Basically as pointed out by @ToughMind, we need specify
os.environ["CUDA_VISIBLE_DEVICES"] = "0, 1"

It depends though on the CUDA devices available in one's unit, so if someone has one GPU it may be appropriate to put, for example,
os.environ["CUDA_VISIBLE_DEVICES"] = "0"


Instruction: What is the multi-label binary classification?

Input:
Reading PyTorch docs  BCEWithLogitsLoss:

  where c is the class number (c > 1 for multi-label binary classification, c = 1 for single-label binary classification), nn is the number of the sample in the batch and p_cp
  c is the weight of the positive answer for the class cc .

What is the multi-label binary classification? Binary assume only two labels AFIK

Output:
Multi-label in terms of binary classification means that both the classes can be true class for a single example.

For example, in case of dog-cat classifier, for an image containing both dog and cat, it'll predict both dog and cat.

  In the multi-label problem there is no constraint on how many of the classes the instance can be assigned to. Wiki


Instruction: SpaCy-transformers regression output

Input:
I would like to have a regression output instead of the classification. For instance: instead of n classes I want a floating point output value from 0 to 1.

Here is the minimalistic example from the package github page:

    import spacy
    from spacy.util import minibatch
    import random
    import torch

    is_using_gpu = spacy.prefer_gpu()
    if is_using_gpu:
        torch.set_default_tensor_type("torch.cuda.FloatTensor")

    nlp = spacy.load("en_trf_bertbaseuncased_lg")
    print(nlp.pipe_names) # ["sentencizer", "trf_wordpiecer", "trf_tok2vec"]
    textcat = nlp.create_pipe("trf_textcat", config={"exclusive_classes": True})
    for label in ("POSITIVE", "NEGATIVE"):
        textcat.add_label(label)
    nlp.add_pipe(textcat)

    optimizer = nlp.resume_training()
    for i in range(10):
        random.shuffle(TRAIN_DATA)
        losses = {}
        for batch in minibatch(TRAIN_DATA, size=8):
            texts, cats = zip(*batch)
            nlp.update(texts, cats, sgd=optimizer, losses=losses)
        print(i, losses)
    nlp.to_disk("/bert-textcat")

Is there an easy way to make trf_textcat work as a regressor? Or would it mean extending the library?

Output:
I have figured out a workaround: extract vector representations from the nlp pipeline as:

vector_repres = nlp('Test text').vector

After doing so for all the text entries, You end up with a fixed-dimensional representation of the texts. Assuming You have the continuous output values, feel free to use any estimator, including Neural Network with a linear output.

Note that the vector representation is an average of the vector embeddings of all the words in the text - it might be a sub-optimal solution for Your case.


Instruction: How to interpret the P numbers that fairseq generate produces?

Input:
Using fairseq-generate.py, with the transformer architecture, each translation produces a section like this:

Why is it rare to discover new marine mammal species?
S-0     Why is it rare to discover new marine mam@@ mal species ?
H-0     -0.0643349438905716     Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins?
P-0     -0.0763 -0.1849 -0.0956 -0.0946 -0.0735 -0.1150 -0.1301 -0.0042 -0.0321 -0.0171 -0.0052 -0.0062 -0.0015

With this explanation:

  H is the hypothesis along with an average log-likelihood; and P is the positional score per token position, including the end-of-sentence marker

I'm wondering if it is reasonable to say a low (absolute) number in the P row means higher confidence in that particular word? E.g. does -0.07 for "Pourquoi" means it was happier about that than it was (-0.1849) for "est-il"? And the low -0.0015 at the end means it was really confident the sentence should end there.

Background: What I'm trying to work out is if I can use either the H number, or somehow to use the individual P numbers, to get a confidence measure in its translation. I've been analyzing a handful of translations against the H number and didn't notice much correspondence between it and my subjective opinion of translation quality. But I've a couple where I thought it was particularly poor - it had missed a bit of key information - and the final P number was a relatively high -0.6099 and -0.3091 (The final P number is -0.11 or so on most of them.)

Output:
Q: I'm wondering if it is reasonable to say a low (absolute) number in the P row means higher confidence in that particular word?

Yes. As the docs says, "P is the positional score per token position". The score is actually the log probability, therefore the higher (i.e., the lower absolute number) the more "confident". The source-code may not be that easy to follow, but the scores are generated by the SequenceScorer, and there you can see that scores are normalized (which includes a log either if when you're using a single model or an ensemble). Moreover, when printing the scores, they convert them from base e to 2:
print('P-{}\t{}'.format(
    sample_id,
    ' '.join(map(
        lambda x: '{:.4f}'.format(x),
        # convert from base e to base 2
        hypo['positional_scores'].div_(math.log(2)).tolist(),
))

Q: What I'm trying to work out is if I can use either the H number, or somehow to use the individual P numbers, to get a confidence measure in its translation.

It turns out that the H value is simply the average of the P values, as you can see here:
score_i = avg_probs_i.sum() / tgt_len

also converted to base 2. You can check that in your example:
import numpy as np
print(np.mean([-0.0763,-0.1849 ,-0.0956 ,-0.0946 ,-0.0735 ,-0.1150 ,-0.1301 ,-0.0042 ,-0.0321 ,-0.0171 ,-0.0052 ,-0.0062 ,-0.0015]))
# >>> -0.06433076923076922

Another measurement that is often used to assess the performance of a language model is Perplexity. And a good thing is that perplexity can be easily computed based on the P values, as shown in the Language Model example of the fairseq repository:
# Compute perplexity for a sequence
en_lm.score('Barack Obama is coming to Sydney and New Zealand')['positional_scores'].mean().neg().exp()
# tensor(15.1474)

I'm not an expert on NLP, so I can't really tell you which one you should use in your case.


Instruction: when I run my network. I got an error,one of the variables needed for gradient computation has been modified by an inplace operation

Input:
This is a part of code in my train.py:

output_c1[output_c1 > 0.5] = 1.
output_c1[output_c1 < 0.5] = 0.

dice = DiceLoss()
loss = dice(output_c1,labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()

The error is:

Traceback (most recent call last):
  File "D:\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3296, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-78553e2886de>", line 1, in <module>
    runfile('F:/experiment_code/U-net/train.py', wdir='F:/experiment_code/U-net')
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "D:\pycharm\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "F:/experiment_code/U-net/train.py", line 99, in <module>
    loss.backward()
  File "D:\Anaconda3\lib\site-packages\torch\tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "D:\Anaconda3\lib\site-packages\torch\autograd\__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 2, 224, 224]], which is output 0 of SigmoidBackward, is at version 2; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

When I comment this two lines of code:

output_c1[output_c1 > 0.5] = 1.
output_c1[output_c1 < 0.5] = 0.

it can run. I think the error comes from here but I don't know how to solve it.

Output:
output_c1[output_c1 > 0.5] = x is an inplace operation, meaning that it modifies directly the tensor instead of returning a modified copy. Hence it makes the computation of the gradient impossible.

To solve your usecase, you could use the function torch.round to binarize your output :

binary_output_c1 = torch.round(output_c1)
# and don't forget to change the computation of your loss function!
dice = DiceLoss()
loss = dice(binary_output_c1,labels)


Instruction: RuntimeError in Pytorch when increasing batch size to more than 1

Input:
This code for my custom data loader runs smoothly with batch_size=1, but when I increase batch size I get the following Error:

RuntimeError: Expected object of scalar type Double but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'

import numpy as np
import matplotlib.pyplot as plt
import matplotlib
matplotlib.use("TkAgg")
import os, h5py
import PIL
#------------------------------
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
#------------------------------
from data_augmentation import *
#------------------------------
dtype = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor

class NiftiDataset(Dataset):

    def __init__(self,transformation_params,data_path, mode='train',transforms=None ):
        """
        Parameters:
            data_path (string): Root directory of the preprocessed dataset.
            mode (string, optional): Select the image_set to use, ``train``, ``valid``
            transforms (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data_path = data_path
        self.mode = mode
        self.images = []
        self.labels = []
        self.W_maps = []
        self.centers = []
        self.radiuss = []
        self.pixel_spacings = []
        self.transformation_params = transformation_params
        self.transforms = transforms
        #-------------------------------------------------------------------------------------
        if self.mode == 'train':
            self.data_path = os.path.join(self.data_path,'train_set')
        elif self.mode == 'valid':
            self.data_path = os.path.join(self.data_path,'validation_set')
        #-------------------------------------------------------------------------------------
        for _, _, f in os.walk(self.data_path):
            for file in f:
                hdf_file = os.path.join(self.data_path,file)
                data = h5py.File(hdf_file,'r') # Dictionary
                # Preprocessing of Input Image and Label
                patch_img, patch_gt, patch_wmap = PreProcessData(file, data, self.mode, self.transformation_params)
                #print(type(data))
                self.images.append(patch_img) # 2D image
                #print('image shape is : ',patch_img.shape)
                self.labels.append(patch_gt) # 2D label
                #print('label shape is : ',patch_img.shape)
                self.W_maps.append(patch_wmap) # Weight_Map
                # self.centers.append(data['roi_center'][:]) # [x,y]
                # self.radiuss.append(data['roi_radii'][:]) # [R_min,R_max]
                # self.pixel_spacings.append(data['pixel_spacing'][:]) # [x , y , z]

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image = self.images[index]
        label = self.labels[index]
        W_map = self.W_maps[index]
        if self.transforms is not None:
            image, label, W_maps = self.transforms(image, label, W_map)

        return image, label, W_map
#=================================================================================================

if __name__ == '__main__':
    # Test Routinue to check your threaded dataloader
    # ACDC dataset has 4 labels
    n_labels = 4
    path = './hdf5_files'
    batch_size = 1
    # Data Augmentation Parameters
    # Set patch extraction parameters
    size1 = (128, 128)
    patch_size = size1
    mm_patch_size = size1
    max_size = size1

    train_transformation_params = {
        'patch_size': patch_size,
        'mm_patch_size': mm_patch_size,
        'add_noise': ['gauss', 'none1', 'none2'],
        'rotation_range': (-5, 5),
        'translation_range_x': (-5, 5),
        'translation_range_y': (-5, 5),
        'zoom_range': (0.8, 1.2),
        'do_flip': (False, False),
        }

    valid_transformation_params = {
        'patch_size': patch_size,
        'mm_patch_size': mm_patch_size}

    transformation_params = { 'train': train_transformation_params,
                              'valid': valid_transformation_params,
                              'n_labels': 4,
                              'data_augmentation': True,
                              'full_image': False,
                              'data_deformation': False,
                              'data_crop_pad': max_size}
#====================================================================
dataset = NiftiDataset(transformation_params=transformation_params,data_path=path,mode='train')
dataloader = DataLoader(dataset=dataset,batch_size=2,shuffle=True,num_workers=0)

dataiter = iter(dataloader)
data = dataiter.next()
images, labels,W_map = data

#===============================================================================
# Data Visualization
#===============================================================================
print('image: ',images.shape,images.type(),'label: ',labels.shape,labels.type(),
    'W_map: ',W_map.shape,W_map.type())

img   = transforms.ToPILImage()(images[0,0,:,:,0].float())
lbl   = transforms.ToPILImage()(labels[0,0,:,:].float())
W_mp  = transforms.ToPILImage()(W_map [0,0,:,:].float())

plt.subplot(1,3,1)
plt.imshow(img,cmap='gray',interpolation=None)
plt.title('image')
plt.subplot(1,3,2)
plt.imshow(lbl,cmap='gray',interpolation=None)
plt.title('label')
plt.subplot(1,3,3)
plt.imshow(W_mp,cmap='gray',interpolation=None)
plt.title('Weight Map')
plt.show()

I have noticed some strange things such as Tensor types are different even though images and labels and weight maps are images with same type and size.
The Error Traceback:

Traceback (most recent call last):
  File "D:\Saudi_CV\Vibot\Smester_2\2_Medical Image analysis\Project_2020\OUR_Project\data_loader.py", line 118, in <module>
    data = dataiter.next()
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 64, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: Expected object of scalar type Double but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'
[Finished in 19.9s with exit code 1]

Output:
The problem was solved through this solution explained on this page link

        image = torch.from_numpy(self.images[index]).type(torch.FloatTensor)
        label = torch.from_numpy(self.labels[index]).type(torch.FloatTensor)
        W_map = torch.from_numpy(self.W_maps[index]).type(torch.FloatTensor)


Instruction: How can I create a PyTorch tensor with all zeroes and a 1 in the middle of the third dimension?

Input:
I have a tensor, torch.Size([161, 161, 11]) and I want to set it all to zeroes, which I can do with: self.conv1.weight.data = torch.zeros(self.conv1.weight.data.size())

Except, I want column 6 (the middle) of the third dimension to be all ones. How do I do that?

Output:
You can assign it afterwards:

self.conv1.weight.data[:, :, 6] = 1.0

Or in case this tensor is trainable:

with torch.no_grad():
    self.conv1.weight.data[:, :, 6] = 1.0


Instruction: Torch/numpy: groupby pandas alternative

Input:
Is there some efficient way to rewrite the following code to avoid installing & importing pandas and using torch/numpy instead? I am used to work with pandas, so I wrote it like this, but I am trying to learn numpy and torch, so I am looking for alternative solutions that do not use pandas.

bins = torch.LongTensor(3072).random_(0, 35)
weights = torch.rand((3072))
df = pd.DataFrame({'weights': weights.numpy(), 'bins': bins.numpy()})
bins_sum = df.groupby('bins').weights.sum().values

So, basically: how, without using pandas, get a sum of weights grouped by bins?

Output:
You can compute unique elements of bins via torch.unique (the values to group by) and then use index masks for accessing the corresponding elements in weights:

unique = torch.unique(bins)
result = torch.zeros(unique.size(), dtype=weights.dtype)
for i, val in enumerate(unique):
    result[i] += weights[bins == val].sum()


Instruction: Considering transforms in torchaudio, is window length etc. in audio frames or milliseconds

Input: I am preprocessing audio recordings using torchaudio. The documentation lists win_length (int) – Window size. (Default: n_fft) Does this value, which defaults to 400, represent the window length in frames or milliseconds?

Output:
The n_fft variable is used in the Spectrogram class. And it's forward function documentation mentions

  where n_fft is the number of Fourier bins


Instruction: Access the output of several layers of pretrained DistilBERT model

Input:
I am trying to access the output embeddings from several different layers of the pretrained "DistilBERT" model. ("distilbert-base-uncased")

bert_output = model(input_ids, attention_mask=attention_mask)

The bert_output seems to return only the embedding values of the last layer for the input tokens.

Output:
If you want to get the output of all the hidden layers, you need to add the output_hidden_states=True kwarg to your config.

Your code will look something like

from transformers import DistilBertModel, DistilBertConfig

config = DistilBertConfig.from_pretrained('distilbert-base-cased', output_hidden_states=True)
model = DistilBertModel.from_pretrained('distilbert-base-cased', config=config)

The hidden layers will be made available as bert_output[2]


Instruction: Does the generator's optimizer also train the discriminator?

Input:
While learning about GANs, I noticed that the code examples exhibit this pattern:

The discriminator is trained like this:

d_optim.zero_grad()

real_pred = d(real_batch)
d_loss = d_loss_fn(real_pred, torch.ones(real_batch_size, 1))
d_loss.backward()

fake_pred = d(g(noise_batch).detach())
d_loss = d_loss_fn(fake_pred, torch.zeros(noise_batch_size, 1))
d_loss.backward()

d_optim.step()

The generator is trained like this:

g_optim.zero_grad()

fake_pred = d(g(noise_batch))
g_loss = g_loss_fn(fake_pred, torch.ones(noise_batch_size, 1))
g_loss.backward()

g_optim.step()

It is mentioned that d(g(noise_batch).detach()) is written for the discriminator instead of d(g(noise_batch)) to prevent d_optim.step() from training g, but nothing is said about d(g(noise_batch)) for the generator; would g_optim.step() also train d?

In fact, why do we d(g(noise_batch).detach()) if, for example, d_optim = torch.optim.SGD(d.parameters(), lr=0.001)? Does this not specify that d.parameters() and not also g.parameters() are to be updated?

Output:
TLDR: optimizer will update only the parameters specified to it, whereas backward() call computes the gradients for all variables in the computation graph. So, it is useful to detach() the variables for which gradient computation is not required at that instant.

I believe the answer lies in the way things are implemented within PyTorch.

tensor.detach() creates a tensor that shares storage with tensor that does not require grad. So, effectively, you cut off the computation graph. That is, doing fake_pred = d(g(noise_batch).detach()) will detach (cut off) the computation graph of the generator.
When you call backward() on the loss, gradients are calculated for the entire computation graph (irrespective of whether optimizer uses it or not). Thus, cutting off the generator part will avoid the gradient computations for the generator weights (since they are not required).
Also, only the parameters passed to particular optimizer are updated when optimizer.step() is called. So, the g_optim will only optimize the parameters passed to it (You don't explicitly mention which parameters are passed to g_optim). Similarly, d_optim will only update d.parameters() since you explicitly specify that.