Instruction: How to delete maxpooling layer from pytorch network

Input:
I have "simple" Unet with resnet encoder on pytorch v3.1, which works pretty fine:

class UNetResNet(nn.Module):
    """PyTorch U-Net model using ResNet(34, 101 or 152) encoder.

    UNet: https://arxiv.org/abs/1505.04597
    ResNet: https://arxiv.org/abs/1512.03385

    Args:
            encoder_depth (int): Depth of a ResNet encoder (34, 101 or 152).
            num_classes (int): Number of output classes.
            num_filters (int, optional): Number of filters in the last layer of decoder. Defaults to 32.
            dropout_2d (float, optional): Probability factor of dropout layer before output layer. Defaults to 0.2.
            pretrained (bool, optional):
                False - no pre-trained weights are being used.
                True  - ResNet encoder is pre-trained on ImageNet.
                Defaults to False.
            is_deconv (bool, optional):
                False: bilinear interpolation is used in decoder.
                True: deconvolution is used in decoder.
                Defaults to False.

    """

    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d

        if encoder_depth == 34:
            self.encoder = torchvision.models.resnet34(pretrained=pretrained)
            bottom_channel_nr = 512
        elif encoder_depth == 101:
            self.encoder = torchvision.models.resnet101(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 152:
            self.encoder = torchvision.models.resnet152(pretrained=pretrained)
            bottom_channel_nr = 2048
        else:
            raise NotImplementedError('only 34, 101, 152 version of Resnet are implemented')

        self.pool = nn.MaxPool2d(2, 2)

        self.relu = nn.ReLU(inplace=True)

        self.conv1 = nn.Sequential(self.encoder.conv1,
                                   self.encoder.bn1,
                                   self.encoder.relu,
                                   self.pool)

        self.conv2 = self.encoder.layer1

        self.conv3 = self.encoder.layer2

        self.conv4 = self.encoder.layer3

        self.conv5 = self.encoder.layer4

        self.center = DecoderBlockV2(bottom_channel_nr, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec5 = DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec4 = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8,
                                   is_deconv)
        self.dec3 = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2,
                                   is_deconv)
        self.dec2 = DecoderBlockV2(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                                   is_deconv)
        self.dec1 = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
        self.dec0 = ConvRelu(num_filters, num_filters)
        self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

    def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        pool = self.pool(conv5) # that pool layer I would like to delete
        center = self.center(pool)

        dec5 = self.dec5(torch.cat([center, conv5], 1))

        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

By deleting that pool layer from forward we come to next forward function:

def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        center = self.center(conv5) #now center connects with conv5

        dec5 = self.dec5(torch.cat([center, conv5], 1))

        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

Applying only such changes and we will face such error:

~/Desktop/ml/salt/unet_models.py in forward(self, x)
    397         center = self.center(conv5)
    398
--> 399         dec5 = self.dec5(torch.cat([center, conv5], 1))
    400
    401         dec4 = self.dec4(torch.cat([dec5, conv4], 1))

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 4 and 8 in dimension 2 at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:111

So we have to somehow change dec4, but how?

More info:

    class DecoderBlockV2(nn.Module):
        def __init__(self, in_channels, middle_channels, out_channels, is_deconv=True):
            super(DecoderBlockV2, self).__init__()
            self.in_channels = in_channels

            if is_deconv:
                """
                    Paramaters for Deconvolution were chosen to avoid artifacts, following
                    link https://distill.pub/2016/deconv-checkerboard/
                """

                self.block = nn.Sequential(
                    ConvRelu(in_channels, middle_channels),
                    nn.ConvTranspose2d(middle_channels, out_channels, kernel_size=4, stride=2,
                                       padding=1),
            nn.BatchNorm2d(out_channels),
                    nn.ReLU(inplace=True)
                )
            else:
                self.block = nn.Sequential(
                    nn.Upsample(scale_factor=2, mode='bilinear'),
                    ConvRelu(in_channels, middle_channels),
                    ConvRelu(middle_channels, out_channels),
                )

        def forward(self, x):
            return self.block(x)

class ConvRelu(nn.Module):
    def __init__(self, in_, out):
        super().__init__()
        self.conv = conv3x3(in_, out)
        self.activation = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.activation(x)
        return x

Output:
Your error stems from the fact that you are trying to concat (torch.cat) center and conv5 - but the dimensions of these tensors do not match.
Originally, you had the following spatial dimensions

conv5:  4x4
pool:   2x2
center: 4x4  # upsampled
This way you can concat center and conv5 since they are both 4x4 tensors.
However, if you remove the pooling layer, you end up with
conv5:  4x4
center: 8x8  # upsampling more than you need
Now you cannot concat center and conv5 since their spatial dimensions mismatch.

What can you do?

One option is to remove center as well as the pooling, leaving you with

dec5 = self.dec5(conv5)

Another option is to remove the nn.Upsample part from the block of center, leaving you with

self.block = nn.Sequential(
                ConvRelu(in_channels, middle_channels),
                ConvRelu(middle_channels, out_channels),
            )


Instruction: How to free up all memory pytorch is taken from gpu memory

Input:
I have some kind of high level code, so model training and etc. are wrapped by pipeline_network class. My main goal is to train new model every new fold.

for train_idx, valid_idx in cv.split(meta_train[DEPTH_COLUMN].values.reshape(-1)):

        meta_train_split, meta_valid_split = meta_train.iloc[train_idx], meta_train.iloc[valid_idx]

        pipeline_network = unet(config=CONFIG, suffix = 'fold' + str(fold), train_mode=True)

But then I move on to 2nd fold everything fails out of gpu memory:

RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58

At the end of epoch I tried to manually delete that pipeline with no luck:

 def clean_object_from_memory(obj): #definition
    del obj
    gc.collect()
    torch.cuda.empty_cache()

clean_object_from_memory( clean_object_from_memory) # calling

Calling this didn't help as well:

def dump_tensors(gpu_only=True):
        torch.cuda.empty_cache()
        total_size = 0
        for obj in gc.get_objects():
            try:
                if torch.is_tensor(obj):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
                elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
            except Exception as e:
                pass

How can reset pytorch then I move on to the next fold?

Output: Try delete the object with del and then apply torch.cuda.empty_cache(). The reusable memory will be freed after this operation.


Instruction: PyTorch Gradient Descent

Input:
I am trying to manually implement gradient descent in PyTorch as a learning exercise. I have the following to create my synthetic dataset:

import torch
torch.manual_seed(0)
N = 100
x = torch.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*torch.randn(N,1)

Then I create my predictive function (y_pred) as shown below.

w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)
y_pred = w*x+b
mse = torch.mean((y_pred-y_obs)**2)

which uses MSE to infer the weights w,b. I use the block below to update the values according to the gradient.

gamma = 1e-2
for i in range(100):
  w = w - gamma *w.grad
  b = b - gamma *b.grad
  mse.backward()

However, the loop only works in the first iteration. The second iteration onwards w.grad is set to None. I am fairly sure the reason this happens is because I am setting w as a function of it self (I might be wrong).

The question is how do I update the weights properly with the gradient information?

Output:
You should call the backward method before you apply the gradient descent.
You need to use the new weight to calculate the loss every iteration.
Create new tensor without gradient tape every iteration.

The following code works fine on my computer and gives w=5.1 & b=2.2 after 500 iterations training.

Code:

import torch
torch.manual_seed(0)
N = 100
x = torch.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 0.2*torch.randn(N,1)

w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)

gamma = 0.01
for i in range(500):
    print(i)
    # use new weight to calculate loss
    y_pred = w * x + b
    mse = torch.mean((y_pred - y_obs) ** 2)

    # backward
    mse.backward()
    print('w:', w)
    print('b:', b)
    print('w.grad:', w.grad)
    print('b.grad:', b.grad)

    # gradient descent, don't track
    with torch.no_grad():
        w = w - gamma * w.grad
        b = b - gamma * b.grad
    w.requires_grad = True
    b.requires_grad = True

Output:

499
w: tensor([5.1095], requires_grad=True)
b: tensor([2.2474], requires_grad=True)
w.grad: tensor([0.0179])
b.grad: tensor([-0.0576])


Instruction: Pytorch saving and reloading model

Input:
I have the following structure for a VGG16 model:

<bound method Module.state_dict of VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Network(
    (hidden_layers): ModuleList(
      (0): Linear(in_features=25088, out_features=4096, bias=True)
    )
    (output): Linear(in_features=4096, out_features=102, bias=True)
    (dropout): Dropout(p=0.5)
  )

When saving the model with the following code:

checkpoint = {'input_size': 25088,
              'output_size': 102,
              'hidden_layers': [each.out_features for each in model.hidden_layers],
              'state_dict': model.state_dict()}

torch.save(checkpoint, 'checkpoint.pth')

I get the following error:

AttributeError                            Traceback (most recent call last)
<ipython-input-13-b4654570e6e8> in <module>()
      2 checkpoint = {'input_size': 25088,
      3               'output_size': 102,
----> 4               'hidden_layers': [each.out_features for each in model.hidden_layers],
      5               'state_dict': model.state_dict()}
      6

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533
    534     def __setattr__(self, name, value):

  AttributeError: 'VGG' object has no attribute 'hidden_layers'

However, VGG has hidden_layers.
How to save A VGG with learning transfer?

Output:
torch.save(model.state_dict(), 'checkpoint.pth')

state_dict = torch.load('checkpoint.pth')
model.load_state_dict(state_dict)


Instruction: Setting custom loss causes RuntimeError: Variable data has to be a tensor, but got Variable in Pytorch

Input:
I have custom class for the net definition:

class PyTorchUNet(Model):
    ....
    def set_loss(self):
            if self.activation_func == 'softmax': #this is working example
                 loss_function = partial(mixed_dice_cross_entropy_loss,
                                dice_loss=multiclass_dice_loss,
                                cross_entropy_loss=nn.CrossEntropyLoss(),
                                dice_activation='softmax',
                                dice_weight=self.architecture_config['model_params']['dice_weight'],
                                cross_entropy_weight=self.architecture_config['model_params']['bce_weight']
                                )

            elif self.activation_func == 'sigmoid':
                    loss_function = designed_loss #setting will cause error on validation

            else:
                raise Exception('Only softmax and sigmoid activations are allowed')
            self.loss_function = [('mask', loss_function, 1.0)]

    def designed_loss(output, target):
       target = target.long() # this should make variable to tensor
       return lovasz_hinge(output, target)

    # this is just as it from github
    def lovasz_hinge(logits, labels, per_image=True, ignore=None):
        """
        Binary Lovasz hinge loss
          logits: [B, H, W] Variable, logits at each pixel (between -\infty and +\infty)
          labels: [B, H, W] Tensor, binary ground truth masks (0 or 1)
          per_image: compute the loss per image instead of per batch
          ignore: void class id
        """
        if per_image:
            loss = mean(lovasz_hinge_flat(*flatten_binary_scores(log.unsqueeze(0), lab.unsqueeze(0), ignore))
                              for log, lab in zip(logits, labels))
        else:
            loss = lovasz_hinge_flat(*flatten_binary_scores(logits, labels, ignore))
        return loss

    def lovasz_hinge_flat(logits, labels):
        """
        Binary Lovasz hinge loss
          logits: [P] Variable, logits at each prediction (between -\infty and +\infty)
          labels: [P] Tensor, binary ground truth labels (0 or 1)
          ignore: label to ignore
        """
        if len(labels) == 0:
            # only void pixels, the gradients should be 0
            return logits.sum() * 0.
        signs = 2. * labels.float() - 1.
        errors = (1. - logits * Variable(signs))
        errors_sorted, perm = torch.sort(errors, dim=0, descending=True)
        perm = perm.data
        gt_sorted = labels[perm]
        grad = lovasz_grad(gt_sorted)
        loss = torch.dot(F.elu(errors_sorted), Variable(grad))
        return loss
    def mean(l, ignore_nan=False, empty=0):
        """
        nanmean compatible with generators.
        """
        l = iter(l)
        if ignore_nan:
            l = ifilterfalse(np.isnan, l)
        try:
            n = 1
            acc = next(l)
        except StopIteration:
            if empty == 'raise':
                raise ValueError('Empty mean')
            return empty
        for n, v in enumerate(l, 2):
            acc += v
        if n == 1:
            return acc
        return acc / n

Working example:

def mixed_dice_cross_entropy_loss(output, target, dice_weight=0.5, dice_loss=None,
                                  cross_entropy_weight=0.5, cross_entropy_loss=None, smooth=0,
                                  dice_activation='softmax'):
    num_classes_without_background = output.size(1) - 1
    dice_output = output[:, 1:, :, :]
    dice_target = target[:, :num_classes_without_background, :, :].long()
    cross_entropy_target = torch.zeros_like(target[:, 0, :, :]).long()
    for class_nr in range(num_classes_without_background):
        cross_entropy_target = where(target[:, class_nr, :, :], class_nr + 1, cross_entropy_target)
    if cross_entropy_loss is None:
        cross_entropy_loss = nn.CrossEntropyLoss()
    if dice_loss is None:
        dice_loss = multiclass_dice_loss
    return dice_weight * dice_loss(dice_output, dice_target, smooth,
                                   dice_activation) + cross_entropy_weight * cross_entropy_loss(output,
                                                                                cross_entropy_target)

def multiclass_dice_loss(output, target, smooth=0, activation='softmax'):
    """Calculate Dice Loss for multiple class output.

    Args:
        output (torch.Tensor): Model output of shape (N x C x H x W).
        target (torch.Tensor): Target of shape (N x H x W).
        smooth (float, optional): Smoothing factor. Defaults to 0.
        activation (string, optional): Name of the activation function, softmax or sigmoid. Defaults to 'softmax'.

    Returns:
        torch.Tensor: Loss value.

    """
    if activation == 'softmax':
        activation_nn = torch.nn.Softmax2d()
    elif activation == 'sigmoid':
        activation_nn = torch.nn.Sigmoid()
    else:
        raise NotImplementedError('only sigmoid and softmax are implemented')

    loss = 0
    dice = DiceLoss(smooth=smooth)
    output = activation_nn(output)
    num_classes = output.size(1)
    target.data = target.data.float()
    for class_nr in range(num_classes):
        loss += dice(output[:, class_nr, :, :], target[:, class_nr, :, :])
    return loss / num_classes

As result I keep getting:

RuntimeError: Variable data has to be a tensor, but got Variable

How to fix the problem?

Output:
Are you still use pytorch 0.3?

if yes, the following snippet may help

tensor = var.data


Instruction: Use pretrained embedding in Spanish with Torchtext

Input:
I am using Torchtext in an NLP project. I have a pretrained embedding in my system, which I'd like to use. Therefore, I tried:
my_field.vocab.load_vectors(my_path)

But, apparently, this only accepts the names of a short list of pre-accepted embeddings, for some reason. In particular, I get this error:
Got string input vector "my_path", but allowed pretrained vectors are ['charngram.100d', 'fasttext.en.300d', ..., 'glove.6B.300d']

I found some people with similar problems, but the solutions I can find so far are "change Torchtext source code", which I would rather avoid if at all possible.
Is there any other way in which I can work with my pretrained embedding? A solution that allows to use another Spanish pretrained embedding is acceptable.
Some people seem to think it is not clear what I am asking. So, if the title and final question are not enough: "I need help using a pre-trained Spanish word-embedding in Torchtext".

Output:
It turns out there is a relatively simple way to do this without changing Torchtext's source code. Inspiration from this Github thread.

1. Create numpy word-vector tensor

You need to load your embedding so you end up with a numpy array with dimensions (number_of_words, word_vector_length):

my_vecs_array[word_index] should return your corresponding word vector.

IMPORTANT. The indices (word_index) for this array array MUST be taken from Torchtext's word-to-index dictionary (field.vocab.stoi). Otherwise Torchtext will point to the wrong vectors!

Don't forget to convert to tensor:

my_vecs_tensor = torch.from_numpy(my_vecs_array)

2. Load array to Torchtext

I don't think this step is really necessary because of the next one, but it allows to have the Torchtext field with both the dictionary and vectors in one place.

my_field.vocab.set_vectors(my_field.vocab.stoi, my_vecs_tensor, word_vectors_length)

3. Pass weights to model

In your model you will declare the embedding like this:

my_embedding = toch.nn.Embedding(vocab_len, word_vect_len)

Then you can load your weights using:

my_embedding.weight = torch.nn.Parameter(my_field.vocab.vectors, requires_grad=False)

Use requires_grad=True if you want to train the embedding, use False if you want to freeze it.

EDIT: It looks like there is another way that looks a bit easier! The improvement is that apparently you can pass the pre-trained word vectors directly during the vocabulary-building step, so that takes care of steps 1-2 here.


Instruction: How to use PNASNet5 as encoder in Unet in pytorch

Input:
I want use PNASNet5Large as encoder for my Unet here is my wrong aproach for the PNASNet5Large but working for resnet:

class UNetResNet(nn.Module):
def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d

        if encoder_depth == 34:
            self.encoder = torchvision.models.resnet34(pretrained=pretrained)
            bottom_channel_nr = 512
        elif encoder_depth == 101:
            self.encoder = torchvision.models.resnet101(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 152: #this works
            self.encoder = torchvision.models.resnet152(pretrained=pretrained)
            bottom_channel_nr = 2048
        elif encoder_depth == 777: #coded version for the pnasnet
            self.encoder = PNASNet5Large()
            bottom_channel_nr = 4320 #this unknown for me as well

        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU(inplace=True)
        self.conv1 = nn.Sequential(self.encoder.conv1,
                                   self.encoder.bn1,
                                   self.encoder.relu,
                                   self.pool)

        self.conv2 = self.encoder.layer1 #PNASNet5Large doesn't have such layers
        self.conv3 = self.encoder.layer2
        self.conv4 = self.encoder.layer3
        self.conv5 = self.encoder.layer4
        self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

        self.dec5 =  DecoderBlock(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
        self.dec4 = DecoderBlock(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec3 = DecoderBlock(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
        self.dec2 = DecoderBlock(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                                   is_deconv)
        self.dec1 = DecoderBlock(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
        self.dec0 = ConvRelu(num_filters, num_filters)
        self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

    def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)
        center = self.center(conv5)
        dec5 = self.dec5(torch.cat([center, conv5], 1))
        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec3(torch.cat([dec4, conv3], 1))
        dec2 = self.dec2(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)
        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

1) How to get how many bottom channels pnasnet has. It ends up following way:

...
 self.cell_11 = Cell(in_channels_left=4320, out_channels_left=864,
                            in_channels_right=4320, out_channels_right=864)
        self.relu = nn.ReLU()
        self.avg_pool = nn.AvgPool2d(11, stride=1, padding=0)
        self.dropout = nn.Dropout(0.5)
        self.last_linear = nn.Linear(4320, num_classes)

Is 4320 the answer or not, in_channels_left and out_channels_left - something new for me

2) Resnet has somekind of 4 big layers which I use and encoders in my Unet arch, how get similar layer from pnasnet

I'm using pytorch 3.1 and this is the link to the Pnasnet directory

3) AttributeError: 'PNASNet5Large' object has no attribute 'conv1' - so doesn't have conv1 as well

UPD: tried smth like this but failed

class UNetPNASNet(nn.Module):
    def init(self, encoder_depth,  num_classes, num_filters=32, dropout_2d=0.2,
                     pretrained=False, is_deconv=False):
            super().init()
            self.num_classes = num_classes
            self.dropout_2d = dropout_2d
            self.encoder = PNASNet5Large()
            bottom_channel_nr = 4320
            self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

        self.dec5  =  DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
        self.dec4  = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
        self.dec3  = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
        self.dec2  = DecoderBlockV2(num_filters * 4 * 4, num_filters * 4 * 4, num_filters, is_deconv)
        self.dec1  = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
        self.dec0  = ConvRelu(num_filters, num_filters)
        self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

def forward(self, x):
        features = self.encoder.features(x)
        relued_features = self.encoder.relu(features)
        avg_pooled_features = self.encoder.avg_pool(relued_features)
        center = self.center(avg_pooled_features)
        dec5 = self.dec5(torch.cat([center, avg_pooled_features], 1))
        dec4 = self.dec4(torch.cat([dec5, relued_features], 1))
        dec3 = self.dec3(torch.cat([dec4, features], 1))
        dec2 = self.dec2(dec3)
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)
        return self.final(F.dropout2d(dec0, p=self.dropout_2d))

RuntimeError: Given input size: (4320x4x4). Calculated output size: (4320x-6x-6). Output size is too small at /opt/conda/conda-bld/pytorch_1525796793591/work/torch/lib/THCUNN/generic/SpatialAveragePooling.cu:63

Output:
So you want to use PNASNetLarge instead o ResNets as encoder in your UNet architecture. Let's see how ResNets are used. In your __init__:

self.pool = nn.MaxPool2d(2, 2)
self.relu = nn.ReLU(inplace=True)
self.conv1 = nn.Sequential(self.encoder.conv1,
                           self.encoder.bn1,
                           self.encoder.relu,
                           self.pool)

self.conv2 = self.encoder.layer1
self.conv3 = self.encoder.layer2
self.conv4 = self.encoder.layer3
self.conv5 = self.encoder.layer4

So you use the ResNets up to layer4, which is the last block before the average pooling, the sizes you're using for resnet are the ones after the average pooling, therefore I assume there is a self.encoder.avgpool missing after self.conv5 = self.encoder.layer4. The forward of a ResNet in torchvision.models looks like this:

def forward(self, x):
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.relu(x)
    x = self.maxpool(x)

    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)

    x = self.avgpool(x)
    x = x.view(x.size(0), -1)
    x = self.fc(x)

    return x

I guess you want to adopt a similar solution for PNASNet5Large (use the architecture up to the average pooling layer).

1) To get how many channels your PNASNet5Large has, you need to look at the output tensor size after the average pooling, for example by feeding a dummy tensor to it. Also notice that while ResNet are commonly used with input size (batch_size, 3, 224, 224), PNASNetLarge uses (batch_size, 3, 331, 331).

m = PNASNet5Large()
x1 = torch.randn(1, 3, 331, 331)
m.avg_pool(m.features(x1)).size()
torch.Size([1, 4320, 1, 1])

Therefore yes, bottom_channel_nr=4320 for your PNASNet.

2) Being the architecture totally different, you need to modify the __init__ and forward of your UNet. If you decide to use PNASNet, I suggest you make a new class:

class UNetPNASNet(nn.Module):
    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                     pretrained=False, is_deconv=False):
            super().__init__()
            self.num_classes = num_classes
            self.dropout_2d = dropout_2d
            self.encoder = PNASNet5Large()
            bottom_channel_nr = 4320
            self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

            self.dec5 =  DecoderBlock(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
            self.dec4 = DecoderBlock(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
            self.dec3 = DecoderBlock(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
            self.dec2 = DecoderBlock(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                                       is_deconv)
            self.dec1 = DecoderBlock(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
            self.dec0 = ConvRelu(num_filters, num_filters)
            self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

        def forward(self, x):
            features = self.encoder.features(x)
            relued_features = self.encoder.relu(features)
            avg_pooled_features = self.encoder.avg_pool(relued_features)
            center = self.center(avg_pooled_features)
            dec5 = self.dec5(torch.cat([center, conv5], 1))
            dec4 = self.dec4(torch.cat([dec5, conv4], 1))
            dec3 = self.dec3(torch.cat([dec4, conv3], 1))
            dec2 = self.dec2(torch.cat([dec3, conv2], 1))
            dec1 = self.dec1(dec2)
            dec0 = self.dec0(dec1)
            return self.final(F.dropout2d(dec0, p=self.dropout_2d))

3) PNASNet5Large doesn't have a conv1 attribute indeed. You can check it by

'conv1' in list(m.modules())
False


Instruction: PyTorch - create padded tensor from sequences of variable length

Input:
I am looking for a good (efficient and preferably simple) way to create padded tensor from sequences of variable length / shape. The best way I can imagine so far is a naive approach like this:

import torch
seq = [1,2,3]      # seq of variable length
max_len = 5        # maximum length of seq
t = torch.zeros(5) # padding value
for i, e in enumerate(seq):
    t[i] = e
print(t)

Output:

tensor([ 1.,  2.,  3.,  0.,  0.])

Is there a better way to do so?

I haven't found something yet, but I guess there must be something better.

I'm thinking of some function to extend the sequence tensor to the desired shape with the desired padding. Or something to create the padded tensor directly from the sequence. But of course other approaches are welcome too.

Output:
Make your variable length sequence a torch.Tensor and use torch.nn.functional.pad

import torch
import torch.nn.functional as F

seq = torch.Tensor([1,2,3])      # seq of variable length
print(F.pad(seq, pad=(0, 2), mode='constant', value=0))
 1
 2
 3
 0
 0
[torch.FloatTensor of size 5]

Signature of F.pad is:

input: input tensor that is your variable length sequence.
pad: m-elem tuple, where (m/2) ≤ input dimensions and m is even. In 1D case first element is how much padding to the left and second element how much padding to the right of your sequence.
mode: fill the padding with a constant or by replicating the border or reflecting the values.
value: the fill value if you choose a constant padding.


Instruction: How to change channel dimension of am image?

Input: Suppose I have torch tensor with shape = [x,y,z,21] where x = batch_size, y = image_width, z= image_height. The above tensor represents batch of images with 21 channels. How should I convert it to size = [ x,y,z,3 ] ?

Output:
[x,y,z,21] -> [x,y,z,1] -> [x,y,z,3]

for segmentation results predicts with size [x,y,z,21]

segmentation class index result with size [x,y,z,1]

# for pytorch, the right format for image is [batch, channels, height, width]
# however your image format [batch, height, width, channels]
result=predicts.argmax(-1)

the index combie the color map will help you! view voc color map for detial


Instruction: Tensorflow vs PyTorch: convolution doesn't work

Input:
I'm trying to test if Tensorflow convolution output matches PyTorch convolution output with the same weights.

Here's my code in which I copy the weights from Tensorflow to Torch, convolve and compare outputs:

import tensorflow as tf
import numpy as np
import math
from math import floor, ceil
import os
import math
import datetime
from scipy import misc
from PIL import Image
import model
import torch
from torch import nn
import common
import torch.nn.functional as F

sess = tf.Session()
np.random.seed(1)
tf.set_random_seed(1)

#parameters
kernel_size = 3
input_feat = 4
output_feat = 4

#inputs
npo = np.random.random((1,5,5, input_feat))
x = tf.convert_to_tensor(npo, tf.float32)
x2 = torch.tensor(np.transpose(npo, [0, 3, 1, 2])).double()

#the same weights
weights = np.random.random((kernel_size,kernel_size,input_feat,output_feat))
weights_torch = np.transpose(weights, [3, 2, 1, 0])

#convolving with tensorflow
w = tf.Variable(weights, name="testconv_W", dtype=tf.float32)
res = tf.nn.conv2d(x, w, strides=[1, 1, 1, 1], padding="VALID")

sess.run(tf.global_variables_initializer())

#convolving with torch
torchres = F.conv2d(x2, torch.tensor(weights_torch), padding=0, bias=torch.zeros((output_feat)).double())

#comparing the results
print(np.mean(np.transpose(sess.run(res), [0, 3, 1, 2])) - torch.mean(torchres).detach().numpy())

It outputs

0.15440369065716908

Why? Why is there such a big difference? Is the Tensorflow conv2d implementation incorrect? Why doesn't it match PyTorch? Am I doing something wrong? On kernel size 1 everything works fine.
Please help.

Output:
You might try x2 = torch.tensor(np.transpose(npo, [0, 3, 2, 1])).double()
instead of x2 = torch.tensor(np.transpose(npo, [0, 3, 1, 2])).double()


Instruction: Unsupported GNU version! gcc versions later than 6 are not supported! - causes importing cpp_extension

Input:
Then I import it says before error this:

/home/dex/anaconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py:118: UserWarning:

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (c++) may be ABI-incompatible with PyTorch!
Please use a compiler that is ABI-compatible with GCC 4.9 and above.
See https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html.

See https://gist.github.com/goldsborough/d466f43e8ffc948ff92de7486c5216d6
for instructions on how to install GCC 4.9 or higher.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(ABI_INCOMPATIBILITY_WARNING.format(compiler))

But following gist  trying to install 4.9 and 5.9 same errors
Cant get installed, also on 18.04 but with other error:

  dex@dexpc:~$ sudo apt-get install software-properties-common
[sudo] password for dex:
Reading package lists... Done
Building dependency tree
Reading state information... Done
software-properties-common is already the newest version (0.96.24.32.4).
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
dex@dexpc:~$ sudo apt-get update
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:4 http://ppa.launchpad.net/graphics-drivers/ppa/ubuntu bionic InRelease
Hit:5 http://dl.google.com/linux/chrome/deb stable Release
Hit:6 http://ppa.launchpad.net/noobslab/apps/ubuntu bionic InRelease
Hit:7 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:8 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:9 http://ppa.launchpad.net/teejee2008/ppa/ubuntu bionic InRelease
Get:10 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease [109 kB]
Hit:11 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic InRelease
Ign:13 http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic InRelease
Err:14 http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic Release
  404  Not Found [IP: 91.189.95.83 80]
Reading package lists... Done
E: The repository 'http://ppa.launchpad.net/upubuntu-com/ppa/ubuntu bionic Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
dex@dexpc:~$ sudo apt-get install gcc-5.9 g++-5.9
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package gcc-5.9
E: Couldn't find any package by glob 'gcc-5.9'
E: Couldn't find any package by regex 'gcc-5.9'
E: Unable to locate package g++-5.9
E: Couldn't find any package by glob 'g++-5.9'
E: Couldn't find any package by regex 'g++-5.9'

dex@dexpc:~$ gcc --version
gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0

Output:
Two steps to fix the problem:

1)
There should be a copy of sources.list at  /usr/share/doc/apt/examples/sources.list   Copy your /etc/apt/sources.list to save the ppa changes you entered and start with the fresh copy.  Then run the

sudo apt-get update
sudo apt-get dist-upgrade

Then add in your ppas from your saved copy of sources list, and repeat the

sudo apt-get update
sudo apt-get dist-upgrade

Just in case, here's the contents of the fresh file:

# See sources.list(5) manpage for more information
# Remember that CD-ROMs, DVDs and such are managed through the apt-cdrom tool.
deb http://us.archive.ubuntu.com/ubuntu xenial main restricted
deb-src http://us.archive.ubuntu.com/ubuntu xenial main restricted

deb http://security.ubuntu.com/ubuntu xenial-security main restricted
deb-src http://security.ubuntu.com/ubuntu xenial-security main restricted

deb http://us.archive.ubuntu.com/ubuntu xenial-updates main restricted
deb-src http://us.archive.ubuntu.com/ubuntu xenial-updates main restricted

2) To get rif off obsolete ppa

sudo apt-add-repository -r ppa:armagetronad-dev/ppa
sudo apt update -q


Instruction: Pytorch saving and loading a VGG16 with knowledge transfer

Input:
I am saving a VGG16 with knowledge transfer by using the following statement:

torch.save(model.state_dict(), 'checkpoint.pth')

and reloading by using the following statement:

state_dict = torch.load('checkpoint.pth')
model.load_state_dict(state_dict)

That works as long as I reload the VGG16 model and give it the same settings as before with the following code:

model = models.vgg16(pretrained=True)
model.cuda()
for param in model.parameters(): param.requires_grad = False

class Network(nn.Module):
    def __init__(self, input_size, output_size, hidden_layers, drop_p=0.5):

#             input_size: integer, size of the input
#             output_size: integer, size of the output layer
#             hidden_layers: list of integers, the sizes of the hidden layers
#             drop_p: float between 0 and 1, dropout probability

        super().__init__()
        # Add the first layer, input to a hidden layer
        self.hidden_layers = nn.ModuleList([nn.Linear(input_size, hidden_layers[0])])

        # Add a variable number of more hidden layers
        layer_sizes = zip(hidden_layers[:-1], hidden_layers[1:])
        self.hidden_layers.extend([nn.Linear(h1, h2) for h1, h2 in layer_sizes])
        self.output = nn.Linear(hidden_layers[-1], output_size)
        self.dropout = nn.Dropout(p=drop_p)

    def forward(self, x):
        ''' Forward pass through the network, returns the output logits '''

        # Forward through each layer in `hidden_layers`, with ReLU activation and dropout
        for linear in self.hidden_layers:
            x = F.relu(linear(x))
            x = self.dropout(x)

        x = self.output(x)
        return F.log_softmax(x, dim=1)

classifier = Network(25088, 102, [4096], drop_p=0.5)
model.classifier = classifier

How to avoid this?
How can I reload the model without having to reload a VGG16 and redefine a classifier?

Output:
Why not redefine a VGG16 like model directly?
view vgg.py for detail

class VGG_New(nn.Module):
    def __init__(self, features, num_classes=1000, init_weights=True):
        super(VGG, self).__init__()
        self.features = features
        # change here with you code
        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )
        if init_weights:
            self._initialize_weights()

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
                if m.bias is not None:
                    nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.BatchNorm2d):
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)
            elif isinstance(m, nn.Linear):
                nn.init.normal_(m.weight, 0, 0.01)
                nn.init.constant_(m.bias, 0)

then load weight for features only

pretrained_dict=torch.load(vgg_weight)
model_dict=vgg_new.state_dict()
# 1. filter out unnecessary keys
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
# or filter with key value
# pretrained_dict = {k: v for k, v in pretrained_dict.items() if k.find('classifier')==-1}
# 2. overwrite entries in the existing state dict
model_dict.update(pretrained_dict)
vgg_new.load_state_dict(model_dict)


Instruction: Assign Torch and Tensorflow models two separate GPUs

Input:
I am comparing two pre-trained models, one is in Tensorflow and one is in Pytorch, on a machine that has multiple GPUs. Each model fits on one GPU. They are both loaded in the same Python script. How can I assign one GPU to the Tensorflow model and another GPU to the Pytorch model?

Setting CUDA_VISIBLE_DEVICES=0,1 only tells both models that these GPUs are available - how can I (within Python I guess), make sure that Tensorflow takes GPU 0 and Pytorch takes GPU 1?

Output:
You can refer to torch.device. https://pytorch.org/docs/stable/tensor_attributes.html?highlight=device#torch.torch.device

In particular do

device=torch.device("gpu:0")
tensor = tensor.to(device)

or to load a pretrained model

device=torch.device("gpu:0")
model = model.to(device)

to put tensor/model on gpu 0.

Similarly tensorflow has tf.device. https://www.tensorflow.org/api_docs/python/tf/device. Its usage is described here https://www.tensorflow.org/guide/using_gpu

for tensorflow to load model on gpu:0 do,

with tf.device("gpu:0"):
     load_model_function(model_path)


Instruction: Pytorch saving model UserWarning: Couldn't retrieve source code for container of type Network

Input:
When saving model in Pytorch by using:

torch.save(model, 'checkpoint.pth')

I get the following warning:

  /opt/conda/lib/python3.6/site-packages/torch/serialization.py:193:
  UserWarning: Couldn't retrieve source code for container of type
  Network. It won't be checked for correctness upon loading.   "type " +
  obj.name + ". It won't be checked "

When I load it I get the following error:

state_dict = torch.load('checkpoint_state_dict.pth')
model = torch.load('checkpoint.pth')
model.load_state_dict(state_dict)

AttributeError                            Traceback (most recent call last)
<ipython-input-2-6a79854aef0f> in <module>()
      2 state_dict = torch.load('checkpoint_state_dict.pth')
      3 model = 0
----> 4 model = torch.load('checkpoint.pth')
      5 model.load_state_dict(state_dict)

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    301         f = open(f, 'rb')
    302     try:
--> 303         return _load(f, map_location, pickle_module)
    304     finally:
    305         if new_fd:

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    467     unpickler = pickle_module.Unpickler(f)
    468     unpickler.persistent_load = persistent_load
--> 469     result = unpickler.load()
    470
    471     deserialized_storage_keys = pickle_module.load(f)

AttributeError: Can't get attribute 'Network' on <module '__main__'>

Why is not possible to save a model and reload it entirely?

Output:
Saving

torch.save({'state_dict': model.state_dict()}, 'checkpoint.pth.tar')

Loading

model = describe_model()
checkpoint = torch.load('checkpoint.pth.tar')
model.load_state_dict(checkpoint['state_dict'])


Instruction: How do deep learning frameworks such as PyTorch handle memory when using multiple GPUs?

Input: I have recently run into a situation where I am running out of memory on a single Nvidia V100. I have limited experience using multiple GPUs to train networks so I'm a little unsure on how the data parallelization process works. Lets say I'm using a model and batch size that requires something like 20-25GB of memory. Is there any way to take advantage of the full 32GB of memory I have between two 16GB V100s? Would PyTorch's DataParallel functionality achieve this? I suppose there is also the possibility of breaking the model up and using model parallelism as well. Please excuse my lack of knowledge on this subject. Thanks in advance for any help or clarification!

Output:
You should keep model parallelism as your last resource and only if your model doesn't fit in the memory of a single GPU (with 16GB/GPU you have plenty of room for a gigantic model).

If you have two GPUs, I would use data parallelism. In data parallelism you have a copy of your model on each GPU and each copy is fed with a batch. The gradients are then gathered and used to update the copies.

Pytorch makes it really easy to achieve data parallelism, as you just need to wrap you model instance in nn.DataParallel:

model = torch.nn.DataParallel(model, device_ids=[0, 1])
output = model(input_var)


Instruction: How do I use torch.stack?

Input: How do I use torch.stack to stack two tensors with shapes a.shape = (2, 3, 4) and b.shape = (2, 3) without an in-place operation?

Output:
Stacking requires same number of dimensions. One way would be to unsqueeze and stack. For example:

a.size()  # 2, 3, 4
b.size()  # 2, 3
b = torch.unsqueeze(b, dim=2)  # 2, 3, 1
# torch.unsqueeze(b, dim=-1) does the same thing

torch.stack([a, b], dim=2)  # 2, 3, 5


Instruction: Pytorch: List of layers returns 'optimizer got an empty parameter list'

Input:
I have a defined model and defined layer, I add n instances of my defined layer to a list in the init function of my model as follow:

    self.layers = []
    for i in range(len(nhid)-1):
        self.layers.append(MyLayer(nhid[i], nhid[i+1]))

but when I create optimizer by

optim.Adam(model.parameters(),lr=args.lr, weight_decay=args.weight_decay)

it says:

  ValueError: optimizer got an empty parameter list

But when I write it for two layers as follow I got no error:

self.layer1 = MyLayer[nhid[0],nhid[1]]
self.layer2 = MyLayer[nhid[1],nhid[2]]

Output:
I solved the problem by using nn.ModuleList() as follow:

temp = []
for i in range(len(nhid)-1):
    temp.append(MyLayer(nhid[i], nhid[i+1]))
self.layers = nn.ModuleList(temp)

I also read about nn.Sequential(), but I didn't find out how to use it in a correct way.


Instruction: PyTorch autograd -- grad can be implicitly created only for scalar outputs

Input:
I am using the autograd tool in PyTorch, and have found myself in a situation where I need to access the values in a 1D tensor by means of an integer index. Something like this:

def basic_fun(x_cloned):
    res = []
    for i in range(len(x)):
        res.append(x_cloned[i] * x_cloned[i])
    print(res)
    return Variable(torch.FloatTensor(res))

def get_grad(inp, grad_var):
    A = basic_fun(inp)
    A.backward()
    return grad_var.grad

x = Variable(torch.FloatTensor([1, 2, 3, 4, 5]), requires_grad=True)
x_cloned = x.clone()
print(get_grad(x_cloned, x))

I am getting the following error message:

[tensor(1., grad_fn=<ThMulBackward>), tensor(4., grad_fn=<ThMulBackward>), tensor(9., grad_fn=<ThMulBackward>), tensor(16., grad_fn=<ThMulBackward>), tensor(25., grad_fn=<ThMulBackward>)]
Traceback (most recent call last):
  File "/home/mhy/projects/pytorch-optim/predict.py", line 74, in <module>
    print(get_grad(x_cloned, x))
  File "/home/mhy/projects/pytorch-optim/predict.py", line 68, in get_grad
    A.backward()
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 90, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I am in general, a bit skeptical about how using the cloned version of a variable is supposed to keep that variable in gradient computation. The variable itself is effectively not used in the computation of A, and so when you call A.backward(), it should not be part of that operation.

I appreciate your help with this approach or if there is a better way to avoid losing the gradient history and still index through a 1D tensor with requires_grad=True!

**Edit (September 15):**

res is a list of zero-dimensional tensors containing squared values of 1 to 5. To concatenate in a single tensor containing [1.0, 4.0, ..., 25.0], I changed return Variable(torch.FloatTensor(res)) to torch.stack(res, dim=0), which produces tensor([ 1.,  4.,  9., 16., 25.], grad_fn=<StackBackward>).

However, I am getting this new error, caused by the A.backward() line.

Traceback (most recent call last):
  File "<project_path>/playground.py", line 22, in <module>
    print(get_grad(x_cloned, x))
  File "<project_path>/playground.py", line 16, in get_grad
    A.backward()
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/tensor.py", line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 84, in backward
    grad_tensors = _make_grads(tensors, grad_tensors)
  File "/home/mhy/.local/lib/python3.5/site-packages/torch/autograd/__init__.py", line 28, in _make_grads
    raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs

Output:
I changed my basic_fun to the following, which resolved my problem:

def basic_fun(x_cloned):
    res = torch.FloatTensor([0])
    for i in range(len(x)):
        res += x_cloned[i] * x_cloned[i]
    return res

This version returns a scalar value.


Instruction: Linear Regression with CNN using Pytorch: input and target shapes do not match: input [400 x 1], target [200 x 1]

Input:
Let me explain the objective first. Let's say I have 1000 images each with an associated quality score [in range of 0-10]. Now, I am trying to perform the image quality assessment using CNN with regression(in PyTorch). I have divided the images into equal size patches. Now, I have created a CNN network in order to perform the linear regression.

Following is the code:

class MultiLabelNN(nn.Module):
    def __init__(self):
        super(MultiLabelNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.fc1 = nn.Linear(3200,1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, 1)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = x.view(-1, 3200)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        return x

While running this code of network I am getting following error

input and target shapes do not match: input [400 x 1], target [200 x 1]

the target shape is [200x1] is because I have taken the batch size of 200. I found the solution that if I change "self.fc1 = nn.Linear(3200,1024)" and "x = x.view(-1, 3200)" here from 3200 to 6400 my code runs without any error.

Similarly, It will throw an error input and target shapes do not match: input [100 x 1], target [200 x 1] if I put 12800 instead of 6400

Now my doubt is that I am not able to understand the reason behind this. If I am giving 200 images as input to my network then why the input shape is getting affected while changing the parameters when I move from convolutional layer to fully connected layer. I hope I have clearly mentioned my doubt. Even though I anybody has any doubt please ask me. It will be a great help. Thanks in advance.

Output:
class MultiLabelNN(nn.Module):
    def __init__(self):
        super(MultiLabelNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.fc1 = nn.Linear(6400,1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, 1)

   def forward(self, x):
       #shape of x is (b_s, 32,32,1)
       x = self.conv1(x) #shape of x is (b_s, 28,28,132)
       x = F.relu(x)
       x = self.pool(x) #shape of x now becomes (b_s X 14 x 14 x 32)
       x = self.conv2(x) # shape(b_s, 10x10x64)
       x = F.relu(x)#size is (b_s x 10 x 10 x 64)
       x = x.view(-1, 3200) # shape of x is now(b_s*2, 3200)
       #this is the problem
       #you can fc1 to be of shape (6400,1024) and that will work
       x = self.fc1(x)
       x = F.relu(x)
       x = self.fc2(x)
       x = F.relu(x)
       x = self.fc3(x)
       return x

I think this should work. Let me know if some errors still remain.


Instruction: How is a 3-d tensor indexed by two 2d tensors?

Input:
Here's a snapshot from line 15-20 in DIM

def random_permute(X):
    X = X.transpose(1, 2)
    b = torch.rand((X.size(0), X.size(1))).cuda()
    idx = b.sort(0)[1]
    adx = torch.range(0, X.size(1) - 1).long()
    X = X[idx, adx[None, :]].transpose(1, 2)

    return X

where X is a tensor of size [64, 64, 128], idx a tensor of size [64, 64], adx  a tensor of size [64].
How does X = X[idx, adx[None, :]] work? How can we use two 2d tensors to index a 3d tensor? What really happens to X after this indexing?

Output:
Things will be more clear if we consider a smaller concrete example. Let

x = np.arange(8).reshape(2, 2, 2)
b = np.random.rand(2, 2)
idx = b.argsort(0) # e.g. idx=[[1, 1], [0, 0]]
adx = np.arange(2)[None, :] # [[0, 1]]
y = x[idx, adx] # implicitly expanding 'adx' to [[0, 1], [0, 1]]

In this example, we'll have y as

y[0, 0] = x[idx[0, 0], adx[0, 0]]=x[1, 0]
y[0, 1] = x[idx[0, 1], adx[0, 1]]=x[1, 1]
y[1, 0] = x[idx[1, 0], adx[1, 0]]=x[0, 0]
...

It may be helpful to see how we do the same in tensorflow:

d0, d1, d2 = x.shape.as_list()
b = np.random.rand(d0, d1)
idx = np.argsort(b, 0)
idx = idx.reshape(-1)
adx = np.arange(0, d1)
adx = np.tile(adx, d0)

y = tf.reshape(tf.gather_nd(x, zip(idx, adx)), (d0, d1, d2))


Instruction: Pytorch: Size Mismatch during running a test image through a trained CNN

Input:
I am going through tutorials to train/test a convolutional neural network(CNN), and I am having an issue with prepping a test image to run it through the trained network. My initial guess is that it has something to do with having a correct format of the tensor input for the net.

Here is the code for the Net.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init as I

class Net(nn.Module):

def __init__(self):
    super(Net, self).__init__()

    ## 1. This network takes in a square (same width and height), grayscale image as input
    ## 2. It ends with a linear layer that represents the keypoints
    ## this last layer output 136 values, 2 for each of the 68 keypoint (x, y) pairs

    # input size 224 x 224
    # after the first conv layer, (W-F)/S + 1 = (224-5)/1 + 1 = 220
    # after one pool layer, this becomes (32, 110, 110)
    self.conv1 = nn.Conv2d(1, 32, 5)

    # maxpool layer
    # pool with kernel_size = 2, stride = 2
    self.pool = nn.MaxPool2d(2,2)

    # second conv layer: 32 inputs, 64 outputs , 3x3 conv
    ## output size = (W-F)/S + 1 = (110-3)/1 + 1 = 108
    ## output dimension: (64, 108, 108)
    ## after another pool layer, this becomes (64, 54, 54)
    self.conv2 = nn.Conv2d(32, 64, 3)

    # third conv layer: 64 inputs, 128 outputs , 3x3 conv
    ## output size = (W-F)/S + 1 = (54-3)/1 + 1 = 52
    ## output dimension: (128, 52, 52)
    ## after another pool layer, this becomes (128, 26, 26)
    self.conv3 = nn.Conv2d(64,128,3)

    self.conv_drop = nn.Dropout(p = 0.2)
    self.fc_drop = nn.Dropout(p = 0.4)

    # 64 outputs * 5x5 filtered/pooled map  = 186624
    self.fc1 = nn.Linear(128*26*26, 1000)
    #
    self.fc2 = nn.Linear(1000, 1000)

    self.fc3 = nn.Linear(1000, 136)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = self.pool(F.relu(self.conv3(x)))
    x = self.conv_drop(x)

    # prep for linear layer
    # flattening
    x = x.view(x.size(0), -1)

    # two linear layers with dropout in between
    x = F.relu(self.fc1(x))
    x = self.fc_drop(x)
    x = self.fc2(x)
    x = self.fc_drop(x)
    x = self.fc3(x)

    return x

Maybe my calculation in layer inputs is wrong?

And here is the test-running code block: (you can think of 'roi' as a standard numpy image.)

# loop over the detected faces from your haar cascade
for i, (x,y,w,h) in enumerate(faces):

    plt.figure(figsize=(10,5))
    ax = plt.subplot(1, len(faces), i+1)
    # Select the region of interest that is the face in the image
    roi = image_copy[y:y+h, x:x+w]

    ## TODO: Convert the face region from RGB to grayscale
    roi = cv2.cvtColor(roi, cv2.COLOR_RGB2GRAY)

    ## TODO: Normalize the grayscale image so that its color range falls in [0,1] instead of [0,255]
    roi = np.multiply(roi, 1/255)

    ## TODO: Rescale the detected face to be the expected square size for your CNN (224x224, suggested)
    roi = cv2.resize(roi, (244,244))

    roi = roi.reshape(roi.shape[0], roi.shape[1], 1)
    roi = roi.transpose((2, 0, 1))

    ## TODO: Change to tensor
    roi = torch.from_numpy(roi)
    roi = roi.type(torch.FloatTensor)
    roi = roi.unsqueeze(0)
    print (roi.shape)

    ## TODO: run it through the net
    output_pts = net(roi)

And I get the error message saying:

RuntimeError: size mismatch, m1: [1 x 100352], m2: [86528 x 1000] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:2033

The caveat is if I run my trained network in the provided test suite (where the tensor inputs are already prepped), it gives no errors there and runs as it is supposed to. I think that means there's nothing wrong with the design of the network architecture itself. I think there's something wrong with the way that I am prepping the image.

The output of the 'roi.shape' is:

torch.Size([1, 1, 244, 244])

Which should be okay because ([batch_size, color_channel, x, y]).

UPDATE: I have printed out the shape of the layers during running through the net. It turns out the matching input dimensions for FC are different for the test image for the task and the given test images from the test suite. Then I'm almost 80% sure that my prepping the input image for the net is wrong. But how can they have different matching dimensions if the input tensor for both has the exact same dimension ([1,1,244,244])?

when using the provided test suite (where it runs fine):

input: torch.Size([1, 1, 224, 224])
layer before 1st CV: torch.Size([1, 1, 224, 224])
layer after 1st CV pool: torch.Size([1, 32, 110, 110])
layer after 2nd CV pool: torch.Size([1, 64, 54, 54])
layer after 3rd CV pool: torch.Size([1, 128, 26, 26])
flattend layer for the 1st FC: torch.Size([1, 86528])

When prepping/running the test image:

input: torch.Size([1, 1, 244, 244])
layer before 1st CV: torch.Size([1, 1, 244, 244])
layer after 1st CV pool: torch.Size([1, 32, 120, 120]) #<- what happened here??
layer after 2nd CV pool: torch.Size([1, 64, 59, 59])
layer after 3rd CV pool: torch.Size([1, 128, 28, 28])
flattend layer for the 1st FC: torch.Size([1, 100352])

Output:
Did you noticed you have this line in the image preparation.

## TODO: Rescale the detected face to be the expected square size for your CNN (224x224, suggested)
roi = cv2.resize(roi, (244,244))

so you just resized it to 244x244 and not to 224x224.


Instruction: Perform a batch matrix - multiple weight matrices multiplications in pytorch

Input:
I have a batch of matrices A with size torch.Size([batch_size, 9, 5]) and weight matrices B with size torch.Size([3, 5, 6]). In Keras, a simple K.dot(A, B) is able to handle the matrix multiplication to give an output with size (batch_size, 9, 3, 6). Here, each row in A is multiplied to the 3 matrices in B to form a (3x6) matrix.

How do you perform a similar operation in torch. From the documentation, torch.bmm requires that A and B must have the same batch size, so I tried this:

B = B.unsqueeze(0).repeat((batch_size, 1, 1, 1))
B.size() # torch.Size([batch_size, 3, 5, 6])
torch.bmm(A,B) # gives an error

  RuntimeError: invalid argument 2: expected 3D tensor, got 4D

Well, the error is expected but how do I perform such an operation?

Output:
You can use einstein notation to describe the operation you want as bxy,iyk->bxik. So, you can use einsum to calculate it.

torch.einsum('bxy,iyk->bxik', (A, B)) will give you the answer you want.


Instruction: Changing the np array does not change the Torch Tensor automatically?

Input:
I was going through the basic tutorials of PyTorch and came across conversion between NumPy arrays and Torch tensors. The documentation says:

  The Torch Tensor and NumPy array will share their underlying memory locations, and changing one will change the other.

But, this does not seem to be the case in the below code:

import numpy as np

a = np.ones((3,3))
b = torch.from_numpy(a)

np.add(a,1,out=a)
print(a)
print(b)

In the above case I see the changes automatically reflected in the output:

[[2. 2. 2.]
 [2. 2. 2.]
 [2. 2. 2.]]
tensor([[2., 2., 2.],
        [2., 2., 2.],
        [2., 2., 2.]], dtype=torch.float64)

But same doesn't happen when I write something like this:

a = np.ones((3,3))
b = torch.from_numpy(a)

a = a + 1
print(a)
print(b)

I get the following output:

[[2. 2. 2.]
 [2. 2. 2.]
 [2. 2. 2.]]
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]], dtype=torch.float64)

What am I missing here?

Output:
Any time you write a = sign in Python you are creating a new object.

So the right-hand side of your expression in the second case uses the original a and then evaluates to a new object i.e. a + 1, which replaces this original a. b still points to the memory location of the original a, but now a points to a new object in memory.

In other words, in a = a + 1, the expression a + 1 creates a new object, and then Python assigns that new object to the name a.

Whereas, with a += 1, Python calls a's in-place addition method (__iadd__) with the argument 1.

The numpy code: np.add(a,1,out=a) , in the first case takes care of adding that value to the existing array in-place.

(Thanks to @Engineero and @Warren Weckesser for pointing out these explanations in the comments)


Instruction: Compile PyTorch with additional linker options

Input:
I have a modified version of the gloo library. I am able to compile and run programs that use this library (similar to what you can find in gloo/gloo/examples).

Now, I want to build pytorch with my library.
I replaced the third_party/gloo folder in PyTorch with my version of gloo and I am trying to compile it.

However, my version of gloo requires some additional libraries and special linker options. Where should these linker options be added in the pytorch build system?

Without these linker options, my compilation stops with the linker error:

/pytorch/build/lib/libcaffe2_gpu.so: undefined reference to <my code>
/pytorch/build/lib/libcaffe2.so: undefined reference to <my code>

Output:
The additional linker options should be added to:

the Caffe2_DEPENDENCY_LIBS variable in pytorch/caffe2/CMakeLists.txt with the command:
list(APPEND Caffe2_DEPENDENCY_LIBS <linker_options>)
the C10D_LIBS variable in pytorch/torch/lib/c10d/CMakeLists.txt with the command:
list(APPEND C10D_LIBS <linker_options>)

The additional libraries should have Position Independent Code (they must be compiled with the -fPIC flag).


Instruction: How does a 2x2 deconv kernel with stride=2 work?

Input:
For example, if the feature map is 8x8, than I use such a deconv and the feature map becomes 16x16, I'm confused that what the difference between:

deconv(kernel_size=2, stride=2, padding='valid')

and

deconv(kernel_size=3, stride=2, padding='same')

Since they will both make feature map 2 times larger, how do they work respectively?

Output:
I think you'll find the explanations and interactive demo on this web page very helpful.

Specifically, setting stride=2 will double your output shape regardless of kernel size.
kernel_size determine how many output pixels are affected by each input pixel.
Setting stride=2 and kernel_size=2 simply "duplicates" your kernel on the output. Consider this 1D example. Suppose your kernel is [a, b] and your input is [A, B, ...], then the output is

[A*a, A*b, B*a, B*b, ...]

For kernel_size=3, the output becomes

[A*a, A*b, A*c+B*a, B*b, B*c+C*a, ...]


Instruction: Expected tensor for argument #1 'input' to have the same dimension

Input:
Using below code I create 10 instances of each training data of which each has 100 dimensions.
Each of the 100 dimension contains 3 dimensions. Therefore it's shape is : (3, 100, 10). This emulates 10 instances of 100 pixels each with 3 channels to emulate an RGB value

I've set this model to just classify between 1 and 0.

When applying the softmax layer I receive the error :

  RuntimeError: Expected tensor for argument #1 'input' to have the same
  dimension as tensor for 'result'; but 4 does not equal 3 (while
  checking arguments for cudnn_convolution)

I'm using 0.4.0 (checked using print(torch.__version__) )
How to correctly set the dimension for the softmax layer? As I think my dimensions are correct?

%reset -f

import os
import torch
from skimage import io, transform
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils
import torch.utils.data as data_utils
import torchvision
import numpy as np
from sklearn.preprocessing import scale
import torch.nn.functional as F
import torch.nn as nn
import torch.nn.functional as F
from random import randint

batch_size_value = 10

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 10

# Create 3000 instance and reshape to (3 , 100, 10) , this emulates 10 instances of 100 pixels
# each with 3 channels to emulate an RGB value

for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 3000).reshape((3 , 100, 10))
    train_dataset.append(image_x)

mu, sigma = 100, 0.80 # mean and standard deviation
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 3000).reshape((3 , 100, 10))
    train_dataset.append(image_x)

labels_1 = [1 for i in range(num_instances)]
labels_0 = [0 for i in range(num_instances)]

labels = labels_1 + labels_0

print(labels)

x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()

my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

# print(x2)

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('device' , device)
# device = 'cpu'

# Hyper parameters
num_epochs = 10
num_classes = 2
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.fc1   = nn.Linear(864, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, num_classes)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)

model = ConvNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
#         print(images)

        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i % 10) == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Update :

Removing these lines :

out = F.relu(self.conv2(out))
out = F.max_pool2d(out, 2)

fixes issue of dimensions being smaller than kernel.

Output:
There is one structured problem and one bug in your code. Here is the solution.

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=1)  # kernel_size 5----> 1
        self.fc1   = nn.Linear(384, 120)              # FC NodeCount864--> 384
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, num_classes)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)
        return out                        # Don't forget to return the output


Instruction: TypeError: unhashable type: 'list' when calling .iloc()

Input:
I'm currently doing some AI research for a project and for that I have to get used to a framework called "Pytorch". That's fine and all but following the official tutorial (found here) the code doesn't run properly.

The idea is that I analyse a set of facial features from a prepared dataset and then do something with it (haven't gotten to that part yet). But when I run this piece of code:

img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc([index, 0]))  # At this point 'index' is 0

The dataset is initialized like this:

face_dataset = fDataset(csv_file='faces/face_landmarks.csv', root_dir='faces/')

And here is where the error pops up:

for i in range(len(face_dataset)):
    sample = face_dataset[i]  # <-- right there

That leads to the getter function:

def __getitem__(self, index):
    img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc([index, 0]))
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[index, 1:].as_matrix()
    landmarks = landmarks.astype('float').reshape(-1, 2)
    sample = {'image': image, 'landmarks': landmarks}

Found in my FaceLandmarksDataset(Dataset): class I simply get the error of the title. This is strange I find, because I can read the dataset just fine as a frame in PyCharm:

Where the first picture is clearly visible. I have checked as well that it is in the folder that I'm looking in.

Can anyone help out? :)

Output:
You don't need the parentheses with iloc:

self.landmarks_frame.iloc[index, 0]


Instruction: NVidia 1080ti eGPU Ubuntu 16.04.5 LTS - PyTorch / Tensorflow without root permissions

Input:
I have some trouble in setting up my system properly. My system consists of:

Intel NUC7i7BNH
ASUS ROG with a NVidia 1080ti

My GPU is correctly detected by lspci:

06:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 85ea
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at 2000 [size=128]
        Expansion ROM at c5000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384

and with the commands below I installed drivers for the GPU, CUDA and cuDNN:

# Show thunderbolt port / authorize eGPU
$ cat /sys/bus/thunderbolt/devices/0-1/device_name
$ echo 1 | sudo tee -a /sys/bus/thunderbolt/devices/0-1/authorized

# eGPU - on Ubuntu 16.04 - nvidia-384
$ sudo ubuntu-drivers devices
$ sudo ubuntu-drivers autoinstall
$ sudo apt-get install nvidia-modprobe

# CUDA - Download CUDA from Nvidia - http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/
$ wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run
$ chmod +x cuda_9.0.176_384.81_linux-run
$ ./cuda_9.0.176_384.81_linux-run --extract=$HOME
$ sudo ./cuda-linux.9.0.176-22781540.run
$ sudo ./cuda-samples.9.0.176-22781540-linux.run
$ sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
$ sudo ldconfig
# add :/usr/local/cuda/bin (including the ":") at the end of the PATH="/blah:/blah/blah" string (inside the quotes).
$ sudo nano /etc/environments

# CUDA samples - Check installation
$ cd /usr/local/cuda-9.0/samples
$ sudo make
$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery

# cuDNN
$ wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb
$ dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb

# Authorization (Security risk!)
$ sudo nano /etc/udev/rules.d/99-local.rules
# Add
ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"

With sudo prime-select nvidia I can switch between the eGPU and the Intel integrated one. After a logout + login the eGPU seems to work (nvidia-smi):

Fri Sep 21 09:25:18 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   47C    P0    84W / 275W |    214MiB / 11172MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      6734      G   /usr/lib/xorg/Xorg                           138MiB |
|    0      7081      G   kwin_x11                                      23MiB |
|    0      7084      G   /usr/bin/krunner                               2MiB |
|    0      7092      G   /usr/bin/plasmashell                          47MiB |
+-----------------------------------------------------------------------------+

The CUDA samples work also:

$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery
/usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1080 Ti"
  CUDA Driver Version / Runtime Version          9.0 / 9.0
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 11172 MBytes (11715084288 bytes)
  (28) Multiprocessors, (128) CUDA Cores/MP:     3584 CUDA Cores
  GPU Max Clock rate:                            1683 MHz (1.68 GHz)
  Memory Clock rate:                             5505 Mhz
  Memory Bus Width:                              352-bit
  L2 Cache Size:                                 2883584 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 6 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS

Let's come to the problem: The following snippet works only, if I execute python with root permissions.

$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.Tensor([0]).cuda()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: CUDA error: unknown error

vs.

$ sudo ~/miniconda3/envs/PyTorch/bin/python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.Tensor([0]).cuda()
tensor([0.], device='cuda:0')

TLDR:
My CUDA installation seems to work and my eGPU is correctly recognized by the system. However in PyTorch (and also Tensorflow) an 'Unknown Error' is thrown, when I try to move same data onto the eGPU, although it is recognized by the frameworks. When I execute the same code with root permissions everything works like a charm. Does somebody know, where I have to adopt some permissions, such that the code can be executed as standard user?

Output:
After trying out different versions of the nvidia driver, I came to the conclusion, that there is a problem with version 384.

I deleted all nvidia driver related stuff via sudo apt-get purge nvidia. and installed version 390:

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo ubuntu-drivers devices
$ sudo apt install nvidia-390

And here the result:

(PyTorch) max@MaxKubuntuNUC:~$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.tensor([1]).cuda()
tensor([1], device='cuda:0')

Now everything works like a charm.


Instruction: How to convert RGB images to grayscale in PyTorch dataloader?

Input:
I've downloaded some sample images from the MNIST dataset in .jpg format. Now I'm loading those images for testing my pre-trained model.

# transforms to apply to the data
trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

# MNIST dataset
test_dataset = dataset.ImageFolder(root=DATA_PATH, transform=trans)

# Data loader
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

Here DATA_PATH contains a subfolder with the sample image.

Here's my network definition

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.network2D = nn.Sequential(
           nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2),
           nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
           nn.ReLU(),
           nn.MaxPool2d(kernel_size=2, stride=2))
        self.network1D = nn.Sequential(
           nn.Dropout(),
           nn.Linear(7 * 7 * 64, 1000),
           nn.Linear(1000, 10))

    def forward(self, x):
        out = self.network2D(x)
        out = out.reshape(out.size(0), -1)
        out = self.network1D(out)
        return out

And this is my inference part

# Test the model
model = torch.load("mnist_weights_5.pth.tar")
model.eval()

for images, labels in test_loader:
   outputs = model(images.cuda())

When I run this code, I get the following error:

RuntimeError: Given groups=1, weight of size [32, 1, 5, 5], expected input[1, 3, 28, 28] to have 1 channels, but got 3 channels instead

I understand that the images are getting loaded as 3 channels (RGB). So how do I convert them to single channel in the dataloader?

Update:
I changed transforms to include Grayscale option

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,)), transforms.Grayscale(num_output_channels=1)])

But now I get this error

TypeError: img should be PIL Image. Got <class 'torch.Tensor'>

Output:
I found an extremely simple solution to this problem. The required dimensions of the tensor are [1,1,28,28] whereas the input tensor is of the form [1,3,28,28]. So I need to read just 1 channel from it

images = images[:,0,:,:]

This gives me a tensor of the form [1,28,28]. Now I need to convert this to a tensor of the form [1,1,28,28]. Which can be done like this

images = images.unsqueeze(0)

So putting the above two lines together, the prediction part of the code can be written like this

for images, labels in test_loader:
   images = images[:,0,:,:].unsqueeze(0) ## Extract single channel and reshape the tensor
   outputs = model(images.cuda())


Instruction: How does the reshape work before the fully connected layer in the following CNN model?

Input:
Consider the convolutional neural network (two convolutional layers):

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

The fully connected layer fc is to have 7*7*32 inputs coming in. The above:

out = out.reshape(out.size(0), -1) leads to a tensor with size of (32, 49).
This doesn't seem right as the dimensions of input for the dense layer is different. What am I missing here?

[Note that in Pytorch the input is in the following format: [N, C, W, H] so no. of channels comes before the width and height of image]

source: https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/convolutional_neural_network/main.py#L35-L56

Output:
If you look at the output of each layer you can easily understand what you are missing.

def forward(self, x):
    print ('input', x.size())
    out = self.layer1(x)
    print ('layer1-output', out.size())
    out = self.layer2(out)
    print ('layer2-output', out.size())
    out = out.reshape(out.size(0), -1)
    print ('reshape-output', out.size())
    out = self.fc(out)
    print ('Model-output', out.size())
    return out

test_input = torch.rand(4,1,28,28)
model(test_input)

OUTPUT:

('input', (4, 1, 28, 28))
('layer1-output', (4, 16, 14, 14))
('layer2-output', (4, 32, 7, 7))
('reshape-output', (4, 1568))
('Model-output', (4, 10))

Conv2d layer doesn't change the height and width of the tensor. only changes the channel of tensor because of stride and padding. MaxPool2d layer halves the height and width of the tensor.

inpt    = 4,1,28,28
conv1_output = 4,16,28,28
max_output   = 4,16,14,14
conv2_output = 4,32,14,14
max2_output  = 4,32,7,7
reshapeutput = 4,1585 (32*7*7)
fcn_output   = 4,10

N --> Input Size, F --> Filter Size, stride-> Stride Size, pdg-> Padding size

ConvTranspose2d;

OutputSize = N*stride + F - stride - pdg*2

Conv2d;

OutputSize = (N - F)/stride + 1 + pdg*2/stride [e.g. 32/3=10 it ignores after the comma]


Instruction: Training on variable length data - EEG data classification

Input:
I'm a student working on a project involving using EEG data to perform lie detection. I will be working with raw EEG data from 2 channels and will record the EEG data during the duration that the subject is replying to the question. Thus, the data will be a 2-by-variable length array stored in a csv file, which holds the sensor readings from each of the two sensors. For example, it would look something like this:

Time (ms) | Sensor 1 | Sensor 2|
--------------------------------
10        | 100.2    | -324.5  |
20        | 123.5    | -125.8  |
30        | 265.6    | -274.9  |
40        | 121.6    | -234.3  |
             ....
2750      | 100.2    | -746.2  |

I want to predict, based on this data, whether the subject is lying or telling the truth (thus, binary classification.) I was planning on simply treating this as structured data and training based on that. However, on second thought, that wouldn't work at all because of a few reasons:

The order in which the data is organized matters as it is continuous time data.
The length of the data is variable as, again, it's time data and the time it takes for the subject to lie/tell the truth is inconsistent.
I don't know how to deal with there being multiple channels of data.

How would I go about setting up a training model for this type of data? I think this is a "time series classification" problem, but I'm not sure. Any kind of help would be greatly appreciated. Thank you in advance!

Output: After doing some more research, I decided to use an LSTM network with the Keras framework running on top of TensorFlow. LSTMs deal with time series data and the Keras layer allows for multiple feature time series data to be fed into the network, so if anyone is having a similar problem as mine, then LSTMs or RNNs are the way to go.