Instruction: why "RuntimeError CUDA out of memory" in testing?

Input:
The same model ran fine for training with batch-size=5. I reduced the batch size from 80 to 5 during training because of the same error. I am using a GPU with 11GB of memory instead of Titan X (12GB memory), the one used by the author in actual experiment.

However, now in testing, which only has batch-size=1, it is not running.

The issue is in I-frame model testing phase, the other two models have successfully produced results for testing.

Following is my testing command:

time python test.py --arch resnet152 --data-name ucf101 --representation iframe --data-root data/ucf101/mpeg4_videos --test-list data/datalists/ucf101_split1_test.txt --weights ucf101_iframe_model_iframe_model_best.pth.tar --save-scores iframe_score_file

I have used nvidia-smi to make sure nothing else is running on the GPU.

Following is the actual error message:

RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 10.92 GiB total capacity; 10.12 GiB already allocated; 245.50 MiB free; 21.69 MiB cached)

What could be the issue and how it can be fixed?

EDIT: By removing the following two lines from test.py, it starts running without an memeory issue, but it is taking ages to process:

net = torch.nn.DataParallel(net.cuda(devices[0]), device_ids=devices)
net.eval()

Yes, the above lines are for GPU based parallel processing.

Still, is there a solution to my problem?

Output:
I have run this model/testing on GPU with memory upto 8GB, by adding the following flag in the testing command given in the question:

--test-crops 1


Instruction: Pytorch tensor indexing

Input:
I am currently working on converting some code from tensorflow to pytorch, I encountered problem with tf.gather func, there is no direct function to convert it in pytorch.

What I am trying to do is basically indexing, I have two tensors, feature tensor shapes of [minibatch, 60, 2] and indexing tensor [minibatch, 8], say like first tensor is tensor A, and the second one is B.

In Tensorflow, it is directly converted with tf.gather(A, B, batch_dims=1)

How do I achieve this in pytorch?

I have tried A[B] indexing. This seems not work

and A[0]B[0] works, but output of shape is [8, 2]

I need the shape of [minibatch, 8, 2]

It will probably work if I stack tensor like [stack, 8, 2] but I have no idea how to do it

tensorflow
out = tf.gather(logits, indices, batch_dims=1)

pytorch
out = A[B] -> something like this will be great

Output shape of [minibatch, 8, 2]

Output:
I think you are looking for torch.gather

out = torch.gather(A, 1, B[..., None].expand(*B.shape, A.shape[-1]))


Instruction: How to fix 'Expected object of scalar type Float but got scalar type Double for argument #4 'mat1''?

Input:
I am trying to build an lstm model. My model code is below.

My input has 4 features, Sequence length of 5 and batch size of 32.

        class RNN(nn.Module):

            def __init__(self, feature_dim, output_size, hidden_dim, n_layers, dropout=0.5):
                """
                Initialize the PyTorch RNN Module
                :param feature_dim: The number of input dimensions of the neural network
                :param output_size: The number of output dimensions of the neural network
                :param hidden_dim: The size of the hidden layer outputs
                :param dropout: dropout to add in between LSTM/GRU layers
                """
                super(RNN, self).__init__()

                # set class variables
                self.output_size = output_size
                self.n_layers = n_layers
                self.hidden_dim = hidden_dim

                # define model layers
                self.lstm = nn.LSTM(feature_dim, hidden_dim, n_layers, batch_first=True)

                self.fc = nn.Linear(hidden_dim, output_size)

                self.dropout = nn.Dropout(dropout)

            def forward(self, nn_input, hidden):
                """
                Forward propagation of the neural network
                :param nn_input: The input to the neural network
                :param hidden: The hidden state
                :return: Two Tensors, the output of the neural network and the latest hidden state
                """

                # Get Batch Size
                batch_size = nn_input.size(0)

                # Pass through LSTM layer
                lstm_out, hidden = self.lstm(nn_input, hidden)
                # Stack up LSTM outputs
                lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
                # Add dropout and pass through fully connected layer
                x = self.dropout(lstm_out)
                x = self.fc(lstm_out)

                # reshape to be batch_size first
                output = x.view(batch_size, -1, self.output_size)
                # get last batch of labels
                out = output[:, -1]

                # return one batch of output word scores and the hidden state
                return out, hidden

            def init_hidden(self, batch_size):
                '''
                Initialize the hidden state of an LSTM/GRU
                :param batch_size: The batch_size of the hidden state
                :return: hidden state of dims (n_layers, batch_size, hidden_dim)
                '''
                # Implement function

                # initialize   state with zero weights, and move to GPU if available
                weight = next(self.parameters()).data

                if is_gpu_available:
                    hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device),
                          weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device))
                else:
                    hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                              weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
                return hidden

When I train, I got the error

RuntimeError                              Traceback (most recent call last)
    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in <module>
          3
          4 # training the model
    ----> 5 trained_rnn = train_rnn(rnn, batch_size, optimizer, num_epochs, show_every_n_batches)
          6
          7 # saving the trained model

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in train_rnn(rnn, batch_size, optimizer, n_epochs, show_every_n_batches)
         18
         19             # forward, back prop
    ---> 20             loss, hidden = forward_back_prop(rnn, optimizer, inputs, labels, hidden)
         21             # record loss
         22             batch_losses.append(loss)

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in forward_back_prop(rnn, optimizer, inp, target, hidden)
         22
         23     # get the output from the model
    ---> 24     output, h = rnn(inp, h)
         25
         26    # calculate the loss and perform backprop

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        491             result = self._slow_forward(*input, **kwargs)
        492         else:
    --> 493             result = self.forward(*input, **kwargs)
        494         for hook in self._forward_hooks.values():
        495             hook_result = hook(self, input, result)

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in forward(self, nn_input, hidden)
         36
         37         # Pass through LSTM layer
    ---> 38         lstm_out, hidden = self.lstm(nn_input, hidden)
         39         # Stack up LSTM outputs
         40         lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        491             result = self._slow_forward(*input, **kwargs)
        492         else:
    --> 493             result = self.forward(*input, **kwargs)
        494         for hook in self._forward_hooks.values():
        495             hook_result = hook(self, input, result)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
        557             return self.forward_packed(input, hx)
        558         else:
    --> 559             return self.forward_tensor(input, hx)
        560
        561

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward_tensor(self, input, hx)
        537         unsorted_indices = None
        538
    --> 539         output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
        540
        541         return output, self.permute_hidden(hidden, unsorted_indices)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward_impl(self, input, hx, batch_sizes, max_batch_size, sorted_indices)
        520         if batch_sizes is None:
        521             result = _VF.lstm(input, hx, self._get_flat_weights(), self.bias, self.num_layers,
    --> 522                               self.dropout, self.training, self.bidirectional, self.batch_first)
        523         else:
        524             result = _VF.lstm(input, batch_sizes, hx, self._get_flat_weights(), self.bias,

    RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

I am not able to figure the cause of this error. How to fix it? Please help.

Also, is it the correct way of implementing the LSTM or is there a better way to achieve the same?

Output:
torch.nn.LSTM does not need any initialization, as it's initialized to zeros by default (see documentation).
Furthermore, torch.nn.Module already has predefined cuda() method, so one can move module to GPU simply, hence you can safely delete init_hidden(self, batch_size).

You have this error because your input is of type torch.Double, while modules by default use torch.Float (as it's accurate enough, faster and smaller than torch.Double).
You can cast your input Tensors by calling .float(), in your case it could look like that:

def forward(self, nn_input, hidden):
    nn_input = nn_input.float()
    ... # rest of your code

Finally, there is no need for hidden argument if it's always zeroes, you can simply use:

lstm_out, hidden = self.lstm(nn_input) # no hidden here

as hidden is zeroes by default as well.


Instruction: Pytorch tensor to numpy gives "()" as shape

Input:
I have a pytorch tensor

span_end = tensor([[[13]]])

I do the following

span_end = span_end.view(1).squeeze().data.numpy()
            print(type(span_end))
            print(span_end.shape)

This gives me the following output

<class 'numpy.ndarray'>
()

Then later when I try to access the 0th element of span_end I get IndexError because the shape is null somehow. What am I doing wrong here?

Output:
tensor.squeeze() will remove all dimensions of size 1, which in this case all of them are therefore it will result in a tensor with no dimensions.

Removing that statement will work.

import torch
span_end = torch.tensor([[[13]]])
span_end = span_end.view(1).numpy()
print(type(span_end))
print(span_end.shape)
print(span_end[0])

Outputs:

<class 'numpy.ndarray'>
(1,)
13


Instruction: img should be PIL Image. Got

Input:
I'm trying to iterate through a loader to check if it's working, however the below error is given:

TypeError: img should be PIL Image. Got <class 'torch.Tensor'>

I've tried adding both transforms.ToTensor() and transforms.ToPILImage() and it gives me an error asking for the opposite. i.e, with ToPILImage(), it will ask for tensor, and vice versa.

# Imports here
%matplotlib inline
import matplotlib.pyplot as plt
from torch import nn, optim
import torch.nn.functional as F
import torch
from torchvision import transforms, datasets, models
import seaborn as sns
import pandas as pd
import numpy as np

data_dir = 'flowers'
train_dir = data_dir + '/train'
valid_dir = data_dir + '/valid'
test_dir = data_dir + '/test'

#Creating transform for training set
train_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.RandomHorizontalFlip(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

#Creating transform for test set
test_transforms = transforms.Compose(
[transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],[0.229, 0.224, 0.225])])

#transforming for all data
train_data = datasets.ImageFolder(train_dir, transform=train_transforms)
test_data = datasets.ImageFolder(test_dir, transform = test_transforms)
valid_data = datasets.ImageFolder(valid_dir, transform = test_transforms)

#Creating data loaders for test and training sets
trainloader = torch.utils.data.DataLoader(train_data, batch_size = 32,
shuffle = True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)
images, labels = next(iter(trainloader))

It should allow me to simply see the image once I run plt.imshow(images[0]), if its working correctly.

Output:
transforms.RandomHorizontalFlip() works on PIL.Images, not torch.Tensor. In your code above, you are applying transforms.ToTensor() prior to transforms.RandomHorizontalFlip(), which results in tensor.

But, as per the official pytorch documentation here,

  transforms.RandomHorizontalFlip() horizontally flip the given PIL
  Image randomly with a given probability.

So, just change the order of your transformation in above code, like below:

train_transforms = transforms.Compose([transforms.Resize(255),
                                       transforms.CenterCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])


Instruction: Does the input of model require gradient?

Input: In pytorch, when do we need to set the require_grad of input tensor to be True?

Output:
Usually, the inputs are fixed - we do not change the images, we only infer the labels/outputs from the fixed input images.
Since they are fixed - there is no need to compute a gradient w.r.t the input, only w.r.t trainable parameters.

Having said that, there are cases where you want to change the inputs. For instance, when you want to visualize features. See, e.g., this nice post.


Instruction: Predicting sequence of grid coordinates with PyTorch

Input:
I have a similar open question here on Cross Validated (though not implementation focused, which I intend this question to be, so I think they are both valid).

I'm working on a project that uses sensors to monitor a persons GPS location. The coordinates will then be converted to a simple-grid representation. What I want to try and do is after recording a users routes, train a neural network to predict the next coordinates, i.e. take the example below where a user repeats only two routes over time, Home->A and Home->B.

I want to train an RNN/LSTM with sequences of varying lengths e.g. (14,3), (13,3), (12,3), (11,3), (10,3), (9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3) and then also predict with sequences of varying lengths e.g. for this example route if I called

route = [(14,3), (13,3), (12,3), (11,3), (10,3)] //pseudocode
pred = model.predict(route)

pred should give me (9,3) (or ideally even a longer prediction e.g. ((9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3))

How do I feed such training sequences to the init and forward operations identified below?

self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
out, hidden = self.rnn(x, hidden)

Also, should the entire route be a tensor or each set of coordinates within the route a tensor?

Output:
I'm not very experienced with RNNs, but I'll give it a try.

A few things to pay attention to before we start:
1. Your data is not normalized.
2. The output prediction you want (even after normalization) is not bounded to [-1, 1] range and therefore you cannot have tanh or ReLU activations acting on the output predictions.

To address your problem, I propose a recurrent net that given a current state (2D coordinate) predicts the next state (2D coordinates). Note that since this is a recurrent net, there is also a hidden state associated with each location. At first, the hidden state is zero, but as the net sees more steps, it updates its hidden state.

I propose a simple net to address your problem. It has a single RNN layer with 8 hidden states, and a fully connected layer on to to output the prediction.

class MyRnn(nn.Module):
  def __init__(self, in_d=2, out_d=2, hidden_d=8, num_hidden=1):
    super(MyRnn, self).__init__()
    self.rnn = nn.RNN(input_size=in_d, hidden_size=hidden_d, num_layers=num_hidden)
    self.fc = nn.Linear(hidden_d, out_d)

  def forward(self, x, h0):
    r, h = self.rnn(x, h0)
    y = self.fc(r)  # no activation on the output
    return y, h

You can use your two sequences as training data, each sequence is a tensor of shape Tx1x2 where T is the sequence length, and each entry is two dimensional (x-y).

To predict (during training):

rnn = MyRnn()
pred, out_h = rnn(seq[:-1, ...], torch.zeros(1, 1, 8))  # given time t predict t+1
err = criterion(pred, seq[1:, ...])  # compare prediction to t+1

Once the model is trained, you can show it first k steps and continue to predict the next steps:

rnn.eval()
with torch.no_grad():
  pred, h = rnn(s[:k,...], torch.zeros(1, 1, 8, dtype=torch.float))
  # pred[-1, ...] is the predicted next step
  prev = pred[-1:, ...]
  for j in  range(k+1, s.shape[0]):
    pred, h = rnn(prev, h)  # note how we keep track of the hidden state of the model. it is no longer init to zero.
    prev = pred

I put everything together in a colab notebook so you can play with it.
For simplicity, I ignored the data normalization here, but you can find it in the colab notebook.

What's next?
These types of predictions are prone to error accumulation. This should be addressed during training, by shifting the inputs from the ground truth "clean" sequences to the actual predicted sequences, so the model will be able to compensate for its errors.


Instruction: Object detection torchvision : IOError: [Errno 2] No such file or directory:

Input:
I'm working on clothes detection in an image. For this, i'm using the package TorchVision on python, and i want to start from a pretrained model and to fintetune it for my classes (different kind of clothes). My images are in a zip file. When i run the function train_one_epoch, i obtain this error :

IOError: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "<ipython-input-16-4557e2f6ca8b>", line 51, in __getitem__
    img = Image.open(img_path).convert("RGB")
  File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2530, in open
    fp = builtins.open(filename, "rb")
IOError: [Errno 20] Not a directory: 'gdrive/My Drive/donnees_kaggle_iMaterialist/train.zip/062fc2042f49d10fa625151cb652c3da.jpg'

Here is a part of my code :

python

num_classes = 46 + 1
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

dataset_train = FashionDataset('gdrive/My Drive/donnees_kaggle_iMaterialist/train.zip/', # "../input/imaterialist-fashion-2019-FGVC6/train/",
                               'gdrive/My Drive/donnees_kaggle_iMaterialist/train.csv.zip', #"../input/imaterialist-fashion-2019-FGVC6/train.csv",
                               256,
                               256,
                               transforms=get_transform(train=True))

model_ft = get_instance_segmentation_model(num_classes)
model_ft.to(device)

data_loader = torch.utils.data.DataLoader(
    dataset_train, batch_size=4, shuffle=True, num_workers=8,
    collate_fn=lambda x: tuple(zip(*x)))

params = [p for p in model_ft.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=0.001,
                            momentum=0.9, weight_decay=0.0005)

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=5,
                                               gamma=0.1)
num_epochs = 10

for epoch in range(num_epochs):
    train_one_epoch(model_ft, optimizer, data_loader, device, epoch, print_freq=10)
    lr_scheduler.step()

torch.save(model_ft.state_dict(), "model.bin")

Output:
You are getting above error because train.zip is a zip file, not a folder. So, first unzip train.zip file using below code:

import zipfile
zr1 = zipfile.ZipFile("/content/gdrive/My Drive/donnees_kaggle_iMaterialist/train.zip", 'r')
zr1.extractall("/content/gdrive/My Drive/donnees_kaggle_iMaterialist/train/")
zr1.close()

And, then change 'gdrive/My Drive/donnees_kaggle_iMaterialist/train.zip/' to 'gdrive/My Drive/donnees_kaggle_iMaterialist/train/'

Also, train.csv.zip is a zip file, extract it first before actually using it. Use below code:

zr2 = zipfile.ZipFile("/content/gdrive/My Drive/donnees_kaggle_iMaterialist/train.csv.zip", 'r')
zr2.extractall("/content/gdrive/My Drive/donnees_kaggle_iMaterialist/")
zr2.close()

And, then change 'gdrive/My Drive/donnees_kaggle_iMaterialist/train.csv.zip' to something like 'gdrive/My Drive/donnees_kaggle_iMaterialist/train.csv.csv'.

Note one thing here that extracted csv file name would be train.csv.csv(with extension) as zipfile name was train.csv.zip.


Instruction: Pytorch equivalent of tf.map_fn with parallel_iterations?

Input: I'm executing a PyTorch function for n_iters times on a GPU. Currently I'm using a for loop for this. However, when n_iters is large, this is very inefficient. I'm wondering whether there is a PyTorch equivalent of tf.map_fn with the parallel_iterations functionality so that I can execute all the iterations in parallel?

Output:
I have searched intensively and I was not able to find any function in pytorch thats equivalent to tf.map_fn that exposes number of parallel_iterations to be set by the user.

While exploring, I have found that there is a function named 'nn.DataParallel' but this function replicates the model or the operation that you want to run on multiple gpus and then returns the results which is not equivalent to number of parallel_iterations in tf.map_fn.

But for now in Pytorch, using the for loop is the only way to go right now.


Instruction: While using GPU for PyTorch models, getting the CUDA error: Unknown error?

Input:
I am trying to use a pre-trained model using PyTorch. While loading the model to GPU, it is giving the following error:

Traceback (most recent call last):
  File "model\vgg_model.py", line 45, in <module>
    vgg_model1 = VGGFeatureExtractor(True).double().to(device)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 386, in to
    return self._apply(convert)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 193, in _apply
    module._apply(fn)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 193, in _apply
    module._apply(fn)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 199, in _apply
    param.data = fn(param.data)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\nn\modules\module.py", line 384, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
  File "C:\Users\myidi\Anaconda3\envs\openpose\lib\site-packages\torch\cuda\__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: CUDA error: unknown error

I have a Windows 10 Laptop, Nvidia 940m GPU, Latest Pytorch and CUDA Toolkit 9.0 (Also tried on 10.0).

I have tried re-installing the GPU drivers, restarted my machine, re-installed PyTorch, Torchvision and CUDA Toolkit.

While using the following to see if PyTorch is detecting a GPU:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

I am getting the following output: device(type='cuda').

What could be the possible issues? I have tried the solution mentioned here: https://github.com/pytorch/pytorch/issues/20990 and the issue still persists.

I simply put the torch.cuda.current_device() after import torch but the issue still persists.

Output:
Strangely, this worked by using CUDA Toolkit 10.1. I don't know why the latest one is not the default one on PyTorch website in the section where they provide the commands to download the libraries.

Used the following command to install the libraries: conda install pytorch torchvision cudatoolkit=10.1 -c pytorch


Instruction: Using nn.Linear() and nn.BatchNorm1d() together

Input:
I don't understand how BatchNorm1d works when the data is 3D, (batch size, H, W).

Example

Input size: (2,50,70)
Layer: nn.Linear(70,20)
Output size: (2,50,20)

If I then include a batch normalisation layer it requires num_features=50:

BN    : nn.BatchNorm1d(50)

and I don't understand why it isn't 20:

BN    : nn.BatchNorm1d(20)

Example 1)

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.bn11 = nn.BatchNorm1d(50)
        self.fc11 = nn.Linear(70,20)

    def forward(self, inputs):
        out = self.fc11(inputs)
        out = torch.relu(self.bn11(out))
        return out

model = Net()
inputs = torch.Tensor(2,50,70)
outputs = model(inputs)

Example 2)

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.bn11 = nn.BatchNorm1d(20)
        self.fc11 = nn.Linear(70,20)

    def forward(self, inputs):
        out = self.fc11(inputs)
        out = torch.relu(self.bn11(out))
        return out

model = Net()
inputs = torch.Tensor(2,50,70)
outputs = model(inputs)

Example 1 works.
Example 2 throws the error:

RuntimeError: running_mean should contain 50 elements not 20

2D example:

Input size: (2,70)
Layer: nn.Linear(70,20)
BN: nn.BatchNorm1d(20)

I thought the 20 in the BN layer was due to there being 20 nodes output by the linear layer and each one requires a running means/std for the incoming values.

Why in the 3D case, if the linear layer has 20 output nodes, the BN layer doesn't have 20 features?

Output:
One can find the answer inside torch.nn.Linear documentation.

It takes input of shape (N, *, I) and returns (N, *, O), where I stands for input dimension and O for output dim and * are any dimensions between.

If you pass torch.Tensor(2,50,70) into nn.Linear(70,20), you get output of shape (2, 50, 20) and when you use BatchNorm1d it calculates running mean for first non-batch dimension, so it would be 50. That's the reason behind your error.


Instruction: How to declare a scalar as a parameter in pytorch?

Input:
I am new to pytorch. I want to know how to declare a scalar as a parameter. I am wondering what's the difference between the following two ways?

x = torch.randn(1,1, requires_grad=True)

and

tensor = torch.randn(1,1)
x = Variable(tensor, requires_grad=True)

Output:
As per the pytorch official documentation here,

The Variable API has been deprecated: Variables are no longer necessary to use autograd with tensors. Autograd automatically supports Tensors with requires_grad set to True.

Variable(tensor) and Variable(tensor, requires_grad) still work as expected, but they return Tensors instead of Variables.
var.data is the same thing as tensor.data.
Methods such as var.backward(), var.detach(), var.register_hook() now work on tensors with the same method names.

In addition, one can now create tensors with requires_grad=True using factory methods such as torch.randn(), torch.zeros(), torch.ones(), and others like the following:

autograd_tensor = torch.randn((2, 3, 4), requires_grad=True)


Instruction: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED while using flair

Input:
I have been using https://github.com/zalandoresearch/flair#example-usage

tried using flair to experiment flair but then I don't know why I am  not able to use the GPU.
and tried the following:

>>> from flair.data import Sentence
>>> from flair.models import SequenceTagger
>>> sentence = Sentence('I love Berlin .')
>>> tagger = SequenceTagger.load('ner')
2019-07-20 17:52:15,062 loading file /home/vz/.flair/models/en-ner-conll03-v0.4.pt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/flair/nn.py", line 103, in load
    model = cls._init_model_with_state_dict(state)
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py", line 205, in _init_model_with_state_dict
    locked_dropout=use_locked_dropout,
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py", line 166, in __init__
    self.to(flair.device)
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 386, in to
    return self._apply(convert)
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 127, in _apply
    self.flatten_parameters()
  File "/home/vz/miniconda3/envs/gp/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 123, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Can anyone please help me as to how to fix this error ?
Thanks in advance.

Output:
The error is with my machine and CUDNN requirement i would suggest every one to install pytorch with conda so the way to install should be something like this

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

to Eradicate any kind of issues with the installation.


Instruction: Cannot Iterate through PyTorch MNIST dataset

Input:
I am trying load the MNIST dataset in Pytorch and use the built-in dataloader to iterate through the training examples. However I get an error when calling next() on the iterator. I don't have this problem with CIFAR10.

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 128

dataset = torchvision.datasets.MNIST(root='./data', train=True, transform=transform, download=True)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=4)
dataiter = iter(dataloader)
dataiter.next() # ERROR
# RuntimeError: output with shape [1, 28, 28] doesn't match the broadcast shape [3, 28, 28]

I am using Python 3.7.3 with PyTorch 1.1.0

Output:
MNIST dataset consists of grayscaled images, i.e., each image has just 1 channel, while  CIFAR10 dataset consists of color images, i.e., each image has 3 channels.

So, incase of MNIST dataset, replace to transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) to transforms.Normalize([0.5], [0.5]).


Instruction: Pytorch Linear Layer now automatically reshape the input?

Input:
I remember in the past, nn.Linear only accepts 2D tensors.

But today, I discover that nn.Linear now accepts 3D, or even tensors with arbitrary dimensions.

X = torch.randn((20,20,20,20,10))
linear_layer = nn.Linear(10,5)
output = linear_layer(X)
print(output.shape)
>>> torch.Size([20, 20, 20, 20, 5])

When I check the documentation for Pytorch, it does say that it now takes

  Input: :math:(N, *, H_{in}) where :math:* means any number of
            additional dimensions and :math:H_{in} = \text{in\_features}

So it seems to me that Pytorch nn.Linear now reshape the input by x.view(-1, input_dim) automatically.

But I cannot find any x.shape or x.view in the source code:

class Linear(Module):
    __constants__ = ['bias']

    def __init__(self, in_features, out_features, bias=True):
        super(Linear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        if bias:
            self.bias = Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)

    @weak_script_method
    def forward(self, input):
        return F.linear(input, self.weight, self.bias)

    def extra_repr(self):
        return 'in_features={}, out_features={}, bias={}'.format(
            self.in_features, self.out_features, self.bias is not None
        )

Can anyone confirms this?

Output:
torch.nn.Linear uses torch.nn.functional.linear function under the hood, that's where the operations are taking places (see documentation).

It looks like this (removed docstrings and decorators for brevity):

def linear(input, weight, bias=None):
    if input.dim() == 2 and bias is not None:
        # fused op is marginally faster
        ret = torch.addmm(bias, input, weight.t())
    else:
        output = input.matmul(weight.t())
        if bias is not None:
            output += bias
        ret = output
    return ret

First case is addmm, which implements beta*mat + alpha*(mat1 @ mat2) and is supposedly faster (see here for example).

Second operation is matmul, and as one can read in their docs it performs various operations based on the shape of tensors provided (five cases, not going to copy them blatantly here).

In summary it preserves dimensions between first batch and last features dimension. No view() is used whatsoever, especially not this x.view(-1, input_dim), check the code below:

import torch

tensor1 = torch.randn(10, 3, 4)
tensor2 = torch.randn(10, 4, 5)

print(torch.matmul(tensor1, tensor2).shape)
print(torch.matmul(tensor1, tensor2).view(-1, tensor1.shape[1]).shape)

which gives:

torch.Size([10, 3, 5]) # preserves input's 3
torch.Size([50, 3]) # destroys the batch even


Instruction: How to fix "RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.LongTensor"

Input:
I encountered this bug when I am running the machine translation code.

  RuntimeError                              Traceback (most recent call
  last)  in
        5 decoder = Decoder(len(out_vocab), embed_size, num_hiddens, num_layers,
        6                   attention_size, drop_prob)
  ----> 7 train(encoder, decoder, dataset, lr, batch_size, num_epochs)

   in train(encoder, decoder, dataset,
  lr, batch_size, num_epochs)
       13             dec_optimizer.zero_grad()
       14             l = batch_loss(encoder, decoder, X, Y, loss)
  ---> 15             l.backward()
       16             enc_optimizer.step()
       17             dec_optimizer.step()

  /usr/lib64/python3.6/site-packages/torch/tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      105                 products. Defaults to False.
      106         """
  --> 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      108
      109     def register_hook(self, hook):

  /usr/lib64/python3.6/site-packages/torch/autograd/init.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       91     Variable._execution_engine.run_backward(
       92         tensors, grad_tensors, retain_graph, create_graph,
  ---> 93         allow_unreachable=True)  # allow_unreachable flag
       94
       95

  RuntimeError: Function AddBackward0 returned an invalid gradient at
  index 1 - expected type torch.FloatTensor but got torch.LongTensor

I think the bug is in the batch_loss function. But I don't know why and can't fix it.

def batch_loss(encoder, decoder, X, Y, loss):
    batch_size = X.shape[0]
    enc_state = None
    enc_outputs, enc_state = encoder(X, enc_state)
    # 初始化解码器的隐藏状态
    dec_state = decoder.begin_state(enc_state)
    # 解码器在最初时间步的输入是BOS
    dec_input = torch.tensor([out_vocab.stoi[BOS]] * batch_size)
    # 我们将使用掩码变量mask来忽略掉标签为填充项PAD的损失
    mask, num_not_pad_tokens = torch.ones(batch_size), 0
    l = torch.tensor([0])
    for y in Y.t():
        dec_output, dec_state = decoder(dec_input, dec_state, enc_outputs)
        l = l + (mask * loss(dec_output, y)).sum()
        dec_input = y  # 使用强制教学
        num_not_pad_tokens += mask.sum().item()
        # 当遇到EOS时，序列后面的词将均为PAD，相应位置的掩码设成0
        mask = mask * (y != out_vocab.stoi[EOS]).float()
    return l / num_not_pad_tokens

def train(encoder, decoder, dataset, lr, batch_size, num_epochs):
    d2lt.params_init(encoder, init=nn.init.xavier_uniform_)
    d2lt.params_init(decoder, init=nn.init.xavier_uniform_)

    enc_optimizer = optim.Adam(encoder.parameters(), lr=lr)
    dec_optimizer = optim.Adam(decoder.parameters(), lr=lr)
    loss = nn.CrossEntropyLoss(reduction='none')
    data_iter = tdata.DataLoader(dataset, batch_size, shuffle=True)
    for epoch in range(num_epochs):
        l_sum = 0.0
        for X, Y in data_iter:
            enc_optimizer.zero_grad()
            dec_optimizer.zero_grad()
            l = batch_loss(encoder, decoder, X, Y, loss)
            l.backward()
            enc_optimizer.step()
            dec_optimizer.step()
            l_sum += l.item()
        if (epoch + 1) % 10 == 0:
            print("epoch %d, loss %.3f" % (epoch + 1, l_sum / len(data_iter)))

Looking forward to a positive reply.

Output: Thank Proyag. Just replace l = torch.tensor([0]) with l = torch.tensor([0], dtype=torch.float) solves my problem.


Instruction: How to compute product between two sets of features in pytorch using a single loop?

Input:
I wan to compute the product between two sets of feature matrices X and Y
of dimensions (H,W,12) each:

Inefficiently I would do:

H = []
for i in range(12):
    for j in range(12):
        h = X[:,:,i]*Y[:,:,j]
        H.append(h)

which will output H of dimension (H,W,144)

How can this be done in pytorch without iterating in two loops?

I have tried used tensordot solutions but cant replicate the behavior.

Output:
I am not sure this is the most efficient, but you can do something like this (warning: ugly code ahead =]):

import torch

# I choose not to use random -- easier to verify, IMO
a = torch.Tensor([[[1,2],[3,4],[5,6]],[[1,2],[3,4],[5,6]]])
b = torch.Tensor([[[1,2],[3,4],[5,6]],[[1,2],[3,4],[5,6]]])

c = torch.bmm(
    a.view(-1, a.size(-1), 1),
    b.view(-1, 1, b.size(-1))
).view(*(a.shape[:2]), -1)

print(c)

print(a.shape)
print(b.shape)
print(c.shape)

Output:

tensor([[[ 1.,  2.,  2.,  4.],
         [ 9., 12., 12., 16.],
         [25., 30., 30., 36.]],

        [[ 1.,  2.,  2.,  4.],
         [ 9., 12., 12., 16.],
         [25., 30., 30., 36.]]])

torch.Size([2, 3, 2])  # a
torch.Size([2, 3, 2])  # b
torch.Size([2, 3, 4])  # c

Basically, the outer product. Let me know if you need me to explain.

Timings

While using the torch.bmm, 16 out of 32 cores were being used. I used a GeForce RTX 2080 Ti to run the CUDA version (GPU usage was ~97% during execution). Note that the dimensions used on GPU timings are x10, otherwise it is just too fast.

Script:

import timeit

setup = '''
import torch
a = torch.randn(({H}, {W}, 12))
b = torch.randn(({H}, {W}, 12))
'''

setup_cuda = setup.replace("))", ")).to(torch.device('cuda'))")

bmm = '''
c = torch.bmm(
    a.view(-1, a.size(-1), 1),
    b.view(-1, 1, b.size(-1))
).view(*(a.shape[:2]), -1)
'''

loop = '''
c = []
for i in range(a.size(-1)):
    for j in range(b.size(-1)):
        c.append(a[:, :, i] * b[:, :, j])
c = torch.stack(c).permute(1, 2, 0)
'''

min_dim = 10
max_dim = 100
num_repeats = 10

print('BMM')
for d in range(min_dim, max_dim+1, 10):
    print(d, min(timeit.Timer(bmm, setup=setup.format(H=d, W=d)).repeat(num_repeats, 1000)))

print('LOOP')
for d in range(min_dim, max_dim+1, 10):
    print(d, min(timeit.Timer(loop, setup=setup.format(H=d, W=d)).repeat(num_repeats, 1000)))

print('BMM - CUDA')
for d in range(min_dim*10, (max_dim*10)+1, 100):
    print(d, min(timeit.Timer(bmm, setup=setup_cuda.format(H=d, W=d)).repeat(num_repeats, 1000)))

Output:

BMM
10 0.022082214010879397
20 0.034024904016405344
30 0.08957623899914324
40 0.1376199919031933
50 0.20248223491944373
60 0.2657837320584804
70 0.3533527449471876
80 0.42361779196653515
90 0.6103016039123759
100 0.7161333339754492

LOOP
10 1.7369094720343128
20 1.8517447559861466
30 1.9145489090587944
40 2.0530637570191175
50 2.2066439649788663
60 2.394576688995585
70 2.6210166650125757
80 2.9242434420157224
90 3.5709626079769805
100 5.413458575960249

BMM - CUDA
100 0.014253990724682808
200 0.015094103291630745
300 0.12792395427823067
400 0.307440347969532
500 0.541196970269084
600 0.8697826713323593
700 1.2538292426615953
800 1.6859236396849155
900 2.2016236428171396
1000 2.764942280948162


Instruction: RuntimeError: Expected hidden size (2, 24, 50), got (2, 30, 50)

Input:
I am trying to build a model for learning assigned scores (real numbers) to some sentences in a data set. I use RNNs (in PyTorch) for this purpose.
I have defined a model:

class RNNModel1(nn.Module):

    def forward(self, input ,hidden_0):

        embedded = self.embedding(input)
        output, hidden = self.rnn(embedded, hidden_0)
        output=self.linear(hidden)
        return output , hidden

Train function is as:

def train(model,optimizer,criterion,BATCH_SIZE,train_loader,clip):
    model.train(True)
    total_loss = 0
    hidden = model._init_hidden(BATCH_SIZE)

    for i, (batch_of_data, batch_of_labels) in enumerate(train_loader, 1):
        hidden=hidden.detach()
        model.zero_grad()
        output,hidden= model(batch_of_data,hidden)
        loss = criterion(output, sorted_batch_target_scores)
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm(model.parameters(), clip)
        optimizer.step()
    return total_loss/len(train_loader.dataset)

when I run the code I receive this error:

  RuntimeError: Expected hidden size (2, 24, 50), got (2, 30, 50)

Batch size=30, Hidden size=50, Number of Layers=1, Bidirectional=True.

I receive that error in the last batch of data.
I checked the description of RNNs in PyTorch to solve this problem. RNNs in PyTorch have two input arguments and two output arguments. The input arguments are input and h_0. h_0 is a tensor includes initial hidden state for each element in batch of size(num_layers*num_directions, batch, hidden size).
The output arguments are output ans h_n. h_n is a tensor includes hidden state for t=seq_len of size (num_layers*num_directions, batch, hidden size).

in all batches (except the last batch) the size of h_0 and h_n is the same. but in the last batch, perhaps number of elements is less than batch size. Thesefore the size of h_n is (num_layersnum_directions, remained_elements_in_last_batch, hidden size) but the size of h_0 is still (num_layersnum_directions, batch_size, hidden size).

So I receive that error in the last batch of data.

How can I solve this problem and handle the situation in which the size of h_0 and h_n is different?

Thanks in advance.

Output:
This error happens when the number of samples in data set is not a multiple of the size of the batch. Ignoring the last batch can solve the problem. For identifying the last batch, check the number of elements in each batch. If that was less than BATCH_SIZE then it is the last batch in data set.

if(len(batch_of_data)==BATCH_SIZE):
    output,hidden= model(batch_of_data,hidden)


Instruction: Trying to understand cross_entropy loss in PyTorch

Input:
This is a very newbie question but I'm trying to wrap my head around cross_entropy loss in Torch so I created the following code:

x = torch.FloatTensor([
                        [1.,0.,0.]
                       ,[0.,1.,0.]
                       ,[0.,0.,1.]
                       ])

print(x.argmax(dim=1))

y = torch.LongTensor([0,1,2])
loss = torch.nn.functional.cross_entropy(x, y)

print(loss)

which outputs the following:

tensor([0, 1, 2])
tensor(0.5514)

What I don't understand is given my input matches the expected output why is the loss not 0?

Output:
That is because the input you give to your cross entropy function is not the probabilities as you did but the logits to be transformed into probabilities with this formula:

probas = np.exp(logits)/np.sum(np.exp(logits), axis=1)

So here the matrix of probabilities pytorch will use in your case is:

[0.5761168847658291,  0.21194155761708547,  0.21194155761708547]
[0.21194155761708547, 0.5761168847658291, 0.21194155761708547]
[0.21194155761708547,  0.21194155761708547, 0.5761168847658291]


Instruction: PyTorch Binary Classification - same network structure, 'simpler' data, but worse performance?

Input:
To get to grips with PyTorch (and deep learning in general) I started by working through some basic classification examples. One such example was classifying a non-linear dataset created using sklearn (full code available as notebook here)

n_pts = 500
X, y = datasets.make_circles(n_samples=n_pts, random_state=123, noise=0.1, factor=0.2)
x_data = torch.FloatTensor(X)
y_data = torch.FloatTensor(y.reshape(500, 1))

This is then accurately classified using a pretty basic neural net

class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.linear = nn.Linear(input_size, H1)
        self.linear2 = nn.Linear(H1, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.linear(x))
        x = torch.sigmoid(self.linear2(x))
        return x

    def predict(self, x):
        pred = self.forward(x)
        if pred >= 0.5:
            return 1
        else:
            return 0

As I have an interest in health data I then decided to try and use the same network structure to classify some a basic real-world dataset. I took heart rate data for one patient from here, and altered it so all values > 91 would be labelled as anomalies (e.g. a 1 and everything <= 91 labelled a 0). This is completely arbitrary, but I just wanted to see how the classification would work. The complete notebook for this example is here.

What is not intuitive to me is why the first example reaches a loss of 0.0016 after 1,000 epochs, whereas the second example only reaches a loss of 0.4296 after 10,000 epochs

Perhaps I am being naive in thinking that the heart rate example would be much easier to classify. Any insights to help me understand why this is not what I am seeing would be great!

Output:
TL;DR
Your input data is not normalized.

use x_data = (x_data - x_data.mean()) / x_data.std()
increase the learning rate optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

You'll get

convergence in only 1000 iterations.
More details
The key difference between the two examples you have is that the data x in the first example is centered around (0, 0) and has very low variance.
On the other hand, the data in the second example is centered around 92 and has relatively large variance.
This initial bias in the data is not taken into account when you randomly initialize the weights which is done based on the assumption that the inputs are  roughly normally distributed around zero.
It is almost impossible for the optimization process to compensate for this gross deviation - thus the model gets stuck in a sub-optimal solution.
Once you normalize the inputs, by subtracting the mean and dividing by the std, the optimization process becomes stable again and rapidly converges to a good solution.
For more details about input normalization and weights initialization, you can read section 2.2 in He et al Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (ICCV 2015).
What if I cannot normalize the data?
If, for some reason, you cannot compute mean and std data in advance, you can still use nn.BatchNorm1d to estimate and normalize the data as part of the training process.  For example
class Model(nn.Module):
    def __init__(self, input_size, H1, output_size):
        super().__init__()
        self.bn = nn.BatchNorm1d(input_size)  # adding batchnorm
        self.linear = nn.Linear(input_size, H1)
        self.linear2 = nn.Linear(H1, output_size)

    def forward(self, x):
        x = torch.sigmoid(self.linear(self.bn(x)))  # batchnorm the input x
        x = torch.sigmoid(self.linear2(x))
        return x

This modification without any change to the input data, yields similar convergance after only 1000 epochs:

A minor comment
For numerical stability, it is better to use nn.BCEWithLogitsLoss instead of nn.BCELoss. For this end, you need to remove the torch.sigmoid from the forward() output, the sigmoid will be computed inside the loss.
See, for example, this thread regarding the related sigmoid + cross entropy loss for binary predictions.


Instruction: How to get rid of Variable API in PyTorch.autograd?

Input:
I am forwarding, and backpropping tensor data X through two simple nn.Module PyTorch models instances, model1 and model2.

I can't get this process to work without usage of the depreciated Variable API.

So this works just fine:

    y1 = model1(X)
    v = Variable(y1.data, requires_grad=training)         # Its all about this line!
    y2 = model2(v)
    criterion = nn.NLLLoss()
    loss = criterion(y2, y)
    loss.backward()
    y1.backward(v.grad)
    self.step()

But this will throw an error:

    y1 = model1(X)
    y2 = model2(y1)
    criterion = nn.NLLLoss()
    loss = criterion(y2, y)
    loss.backward()
    y1.backward(y1.grad) # it breaks here
    self.step()
>>> RuntimeError: grad can be implicitly created only for scalar outputs

I just can't seem to find a relevant difference between v in the first implementation, and y1 in the second. In both cases requires_grad is set to True. The only thing I could find was that y1.grad_fn=<ThnnConv2DBackward> and v.grad_fn=<ThnnConv2DBackward>

What am I missing here? What (tensor attributes?) do I not know about, and if Variable is depreciated, what other implementation would work?

Output:
After some investigation I came to the following two solutions.
The solution provided elsewhere in this thread retained the computation graph manually, without an option the free them, thus running fine initially, but causing OOM errors later on.

The first solution is to tie the models together using the built in torch.nn.Sequential as such:

model = torch.nn.Sequential(Model1(), Model2())

it's as easy as that. It looks clean and behaves exactly like an ordinary model would.

The alternative is to simply tie them together manually:

        model1 = Model1()
        model2 = Model2()
        y1 = model1(X)
        y2 = model2(y1)
        loss = criterion(y2, y)
        loss.backward()

My fear that this would only backpropagate model2 turned out to be unsubstantiated, since model1 is also stored in the computation graph that is back propagated over.
This implementation enabled inceased transparancy of the interface between the two models, compared to the previous implementation.


Instruction: Assigning a parameter to the GPU sets is_leaf as false

Input:
If I create a Parameter in PyTorch, then it is automatically assigned as a leaf variable:

x = torch.nn.Parameter(torch.Tensor([0.1]))
print(x.is_leaf)

This prints out True. From what I understand, if x is a leaf variable, then it will be updated by the optimiser.

But if I then assign x to the GPU:

x = torch.nn.Parameter(torch.Tensor([0.1]))
x = x.cuda()
print(x.is_leaf)

This prints out False. So now I cannot assign x to the GPU and keep it as a leaf node.

Why does this happen?

Output:
Answer is in is_leaf documentation and here is your exact case:

>>> b = torch.rand(10, requires_grad=True).cuda()
>>> b.is_leaf
False
# b was created by the operation that cast a cpu Tensor into a cuda Tensor

Citing documentation further:

  For Tensors that have requires_grad which is True, they will be leaf
  Tensors if they were created by the user. This means that they are not
  the result of an operation and so grad_fn is None.

In your case, Tensor was not created by you, but was created by PyTorch's cuda() operation (leaf is the pre-cuda b).


Instruction: How to do random search for hyperparameters on different GPUs in parallel?

Input:
Assuming my model uses only one GPU but Virtual Machine has 4.

How to leverage all GPUs for this code?

channel_1_range = [8, 16, 32, 64]
channel_2_range = [8, 16, 32, 64]
kernel_size_1_range = [3, 5, 7]
kernel_size_2_range = [3, 5, 7]
max_count = 40
for count in range(max_count):
    reg = 10**np.random.uniform(-3, 0)
    learning_rate = 10**np.random.uniform(-6, -3)
    channel_1 = channel_1_range[np.random.randint(low=0, high=len(channel_1_range))]
    channel_2 = channel_2_range[np.random.randint(low=0, high=len(channel_2_range))]
    kernel_size_1 = kernel_size_1_range[np.random.randint(low=0, high=len(kernel_size_1_range))]
    kernel_size_2 = kernel_size_2_range[np.random.randint(low=0, high=len(kernel_size_2_range))]

    model = ThreeLayerConvNet(in_channel=3, channel_1=channel_1, kernel_size_1=kernel_size_1, \
        channel_2=channel_2, kernel_size_2=kernel_size_2, num_classes=10)
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    engine = Engine(loader_train=loader_train, loader_val=loader_val, device=device, dtype=dtype, print_every=100, \
        verbose=False)
    engine.train(model, optimizer, epochs=1, reg=reg)

    print("Reg: {0:.2E}, LR: {1:.2E}, Ch_1: {2:2} [{4}], Ch_2: {3:2} [{5}], Acc: {6:.2f} [{7:.2f}], {8:.2f} secs". \
         format(reg, learning_rate, channel_1, channel_2, kernel_size_1, kernel_size_2, \
               engine.accuracy, engine.accuracy_train, engine.duration))

One option is to move this to standalone console app, start N instances (N == number of GPUs) and aggregate results (one output file).

Is it possible to do it directly in Python so I could continue to use jupyter notebook?

Output:
In pytorch you can distribute your models on different GPUs. I think in your case it's the device parameter that allows you to specify the actual GPU:

device1 = torch.device('cuda:0')
device2 = torch.device('cuda:1')
             .
             .
             .
devicen = torch.device('cuda:n')

I don't remember the exact details but if my memory serves me well, you might need to make your code non-blocking by using threading or multiprocessing (better go with multiprocessing to be sure, the GIL might cause you some problems otherwise if you fully utilize your process).
In your case that would mean to parallelise your for loop. For instance by having Queue containing all models and then spawning threads/ processes, allowing you to consume them (where the number of spawned processed, working on the queue, correspond to a GPU each).

So to answer your question, yes you can do it in pure Python (I did a while back, so I'm 100% positive). You can even let one GPU process multiple models (but make sure to calculate your VRAM correctly beforehand). Whether it's actually worth it, compared to just starting multiple jobs is up to you though.

As a little sidenote, if you run it as 'standalone' script, it might still use the same GPU if the GPU number isn't automatically adjusted, otherwise PyTorch might try using DataParallel distribution...


Instruction: Use class from file based of name in string form (Python)

Input:
Currently, I am importing every class I have in a python file in my main "Runner" script. The goal is to, based on a string given, instantiate the class that has the same name as the string given.

For example:

If there is a class called 'Test':

class Test():

if the String is "Test", it will call that class.

Output:
It's seems a reflection case, that are explained here.

Example from ref:

module = __import__(module_name)
class_ = getattr(module, class_name)
instance = class_()


Instruction: Multiply a specific column in each row by -1

Input:
I have a pytorch tensor A, that's of size (n,m) and a list of indices for size n, such that each entry of 0 <= indices[i] < m. For each row i of A, I want to multiply A[i, indices[i]] *= -1, in a vectorized way. Is there an easy way to do this?

 A = torch.tensor([[1,2,3],[4,5,6]])
 indices = torch.tensor([1, 2])

 #desired result
 A = [[1,-2,3],[4,5,-6]]

Output:
Sure there is, fancy indexing is the way to go:
import torch

A = torch.tensor([[1, 2, 3], [4, 5, 6]])
indices = torch.tensor([1, 2]).long()

A[range(A.shape[0]), indices] *= -1

Remember indices must be torch.LongTensor type. You could cast it if you have float using .long() member function.


Instruction: How to multiply a tensor with a vector?

Input:
I have 2 tensors

a = torch.tensor([1,2])
b = torch.tensor([[[10,20],
                   [30,40]],
                  [[1,2],
                   [3,4]]])

and I would like to combine them in such a way that

a ? b = tensor([[[10,20],
                 [30,40]],
                [[ 2, 4],
                 [ 6, 8]]])

(and then sum over the 0th dimension, in the end I want to do a weighted sum)

I've tried:

""" no idea how to interpret that """
a @ b
tensor([[ 70, 100],
        [  7,  10]])

b @ a
tensor([[ 50, 110],
        [  5,  11]])

for i in range(b.size()[0]): # works but I don't think this will work with autograd
      b[i] *= a[i]

a * b              # multiplies right side by 2
tensor([[[10, 40],
         [30, 80]],

        [[ 1,  4],
         [ 3,  8]]])

a.unsqueeze(1)     # multiplies bottom side by 2
tensor([[[10, 20],
         [60, 80]],

        [[ 1,  2],
         [ 6,  8]]])

a.unsqueeze(2) * b # dimension out of range

Output: This should workc = a.unsqueeze(1).unsqueeze(1) * b


Instruction: Is there a way to do Pytorch element wise equality treating each dimension as an element?

Input:
I have two tensors and I want to check for equality treating an array in one dimension as the element

I have 2 tensors

lo = torch.Tensor(([1., 1., 0.],
                   [0., 1., 1.],
                   [0., 0., 0.],
                   [1., 1., 1.]))
lo = torch.Tensor(([1., 1., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.]))

I've tried using
torch.eq(lee, lo)
which returns a tensor like

tensor([[1, 1, 1],
        [1, 0, 0],
        [1, 1, 1],
        [0, 0, 0]], dtype=torch.uint8)

Is there a way to have the output become

tensor([1, 0, 1, 0])

as the only complete element that matches is the first?

edit:
I've come up with this solution

lee = lee.tolist()
lo = lo.tolist()
out = []
for i, j in enumerate(lee):
  if j == lo[i]:
    out.append(1)
  else:
    out.append(0)

and out will be [1, 0, 1, 0]
But is there an easier way?

Output:
You can simply use torch.all(tensor, dim).

code:

l1 = torch.Tensor(([1., 1., 0.],
                   [0., 1., 1.],
                   [0., 0., 0.],
                   [1., 1., 1.]))
l2 = torch.Tensor(([1., 1., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.]))
print(torch.eq(l1, l2))
print(torch.all(torch.eq(l1, l2),  dim=0)) # equivalent to dim = -2
print(torch.all(torch.eq(l1, l2),  dim=1)) # equivalent to dim = -1

output:

tensor([[1, 1, 1],
        [1, 0, 0],
        [1, 1, 1],
        [0, 0, 0]], dtype=torch.uint8)
tensor([0, 0, 0], dtype=torch.uint8)
tensor([1, 0, 1, 0], dtype=torch.uint8) # your desired output


Instruction: How to solve strange cuda error in PyTorch?

Input:
I moved to PyTorch from Keras. I'm very new to the whole moving to  CUDA thing. I've spent hours, surfing the web and haven't been able to find anything? The fix is probably something a line or two. I'd appreciate it if someone knows how to solve this issue?

Here' my code, First I define my u-net model as a class of nn.Module like the following code:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils

class unet(nn.Module):
def __init__(self):
    super(unet, self).__init__()
    self.conv1 = nn.Conv3d(1, 32, 3, padding=1)
    self.conv1_1 = nn.Conv3d(32, 32, 3, padding=1)
    self.conv2 = nn.Conv3d(32, 64, 3, padding=1)
    self.conv2_2 = nn.Conv3d(64, 64, 3, padding=1)
    self.conv3 = nn.Conv3d(64, 128, 3, padding=1)
    self.conv3_3 = nn.Conv3d(128, 128, 3, padding=1)
    self.convT1 = nn.ConvTranspose3d(128, 64, 3, stride=(2,2,2), padding=1, output_padding=1)
    self.conv4 = nn.Conv3d(128, 64, 3, padding=1)
    self.conv4_4 = nn.Conv3d(64, 64, 3, padding=1)
    self.convT2 = nn.ConvTranspose3d(64, 32, 3,stride=(2,2,2), padding=1, output_padding=1)
    self.conv5 = nn.Conv3d(64, 32, 3, padding=1)
    self.conv5_5 = nn.Conv3d(32, 32, 3, padding=1)
    self.conv6 = nn.Conv3d(32, 1 ,3, padding=1)

def forward(self, inputs):
    conv1 = F.relu(self.conv1(inputs))
    conv1 = F.relu(self.conv1_1(conv1))
    pool1 = F.max_pool3d(conv1, 2)

    conv2 = F.relu(self.conv2(pool1))
    conv2 = F.relu(self.conv2_2(conv2))
    pool2 = F.max_pool3d(conv2, 2)
    conv3 = F.relu(self.conv3(pool2))
    conv3 = F.relu(self.conv3_3(conv3))
    conv3 = self.convT1(conv3)

    up1 = torch.cat((conv3, conv2), dim=1)
    conv4 = F.relu(self.conv4(up1))
    conv4 = F.relu(self.conv4_4(conv4))

    conv4 = self.convT2(conv4)
    up2 = torch.cat((conv4, conv1), dim=1)
    conv5 = F.relu(self.conv5(up2))
    conv5 = F.relu(self.conv5_5(conv5))

    conv6 = F.relu(self.conv6(conv5))

    return conv6

Then I run my unet like the following code. note that when defining the module I set it to the cuda. I also set the input data and its labels to the cuda.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = unet().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
datasets = torch.utils.data.TensorDataset(data_recon, data_truth)
train_loader = DataLoader(datasets, batch_size=2, shuffle=True)

def training_loop(n_epochs, optimizer, model, loss_fn, train_loader):
    for epoch in range(1, n_epochs + 1):
        loss_train = 0
        for imgs, labels in train_loader:
            imgs.to(device)
            labels.to(device)
            outputs = model(imgs)
            loss = loss_fn(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            loss_train += loss.item()

        print('{} Epoch {}, Training loss    {}'.format(datetime.datetime.now(), epoch, float(loss_train)))

training_loop(50, optimizer, model, loss_fn, train_loader)

But I get this error:

  RuntimeError                              Traceback (most recent call
  last)  in
  ----> 1 training_loop(50, optimizer, model, loss_fn, train_loader)

   in training_loop(n_epochs, optimizer,
  model, loss_fn, train_loader)
        5             imgs.to(device)
        6             labels.to(device)
  ----> 7             outputs = model(imgs)
        8             loss = loss_fn(outputs, labels)
        9

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      491             result = self._slow_forward(*input, **kwargs)
      492         else:
  --> 493             result = self.forward(*input, **kwargs)
      494         for hook in self._forward_hooks.values():
      495             hook_result = hook(self, input, result)

   in forward(self, inputs)
       18
       19     def forward(self, inputs):
  ---> 20         conv1 = F.relu(self.conv1(inputs))
       21         conv1 = F.relu(self.conv1_1(conv1))
       22         pool1 = F.max_pool3d(conv1, 2)

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      491             result = self._slow_forward(*input, **kwargs)
      492         else:
  --> 493             result = self.forward(*input, **kwargs)
      494         for hook in self._forward_hooks.values():
      495             hook_result = hook(self, input, result)

  /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in
  forward(self, input)
      474                             self.dilation, self.groups)
      475         return F.conv3d(input, self.weight, self.bias, self.stride,
  --> 476                         self.padding, self.dilation, self.groups)
      477
      478

  RuntimeError: Expected object of backend CPU but got backend CUDA for
  argument #2 'weight'

I've spent hours, surfing the web and haven't been able to find anything? The fix is probably something a line or two. I'd appreciate it if someone knows how to solve this issue?

Output:
The problem is with this line

imgs.to(device)
labels.to(device)

.to(device) returns a new Tensor and won't change imgs and labels. So the cuda error is valid. You can simply fix it by assigning new tensor as follows:

imgs = imgs.to(device)
labels = labels.to(device)


Instruction: Why is my classifier not giving me binary results?

Input:
I created a linear ReLu network that is supposed to over-fit to my data. I use BCEWithLogisticLoss as the loss function. I was using it to classify 3d points. Since the data were small enough I didn't care to make into batches. And it worked fine. However now that I've implemented batches into it it seems that the predicted values aren't what I expect (i.e 0 or 1) instead it give me numbers like -25.4562 I didn't change anything else from the network only the batches.

I tried the binary loss function BSELoss however it seems to be a bug in the version of pytorch, so I can't use it. You can take a look at my code below:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# We load the training data
Samples, Ocupancy = common.load_samples()
for i in range(0,Ocupancy.shape[0]):
    if Ocupancy[i] > 1 or Ocupancy[i] < 0:
        print("upsie")

max = np.amax(Samples)
min = np.amin(Samples)

x_test = torch.from_numpy(Samples.astype(np.float32)).to(device)
y_test = torch.from_numpy(Ocupancy.astype(np.float32)).to(device)

train_data = CustomDataset(x_test, y_test)

train_loader = DataLoader(dataset=train_data, batch_size= 22500, shuffle=False) # Batches_size equal to the number of points in each slice

phi = common.MLP(3, 1).to(device)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(phi.parameters(), lr = 0.01)
epoch = 10

fit_start_time = time.time()

for epoch in range(epoch):
    for x_batch, y_batch in train_loader:
        #optimizer.zero_grad()

        x_train = x_batch.to(device)

        y_train = y_batch.to(device)

        y_pred = phi(x_batch)
        print(y_pred)

        # Compute Loss
        loss = criterion(y_pred.squeeze(), y_batch.squeeze())

        print('Epoch {}: train loss: {}'.format(epoch, loss.item()))    # Backward pass

        loss.backward()
        optimizer.step()

fit_end_time = time.time()

print("Total time = %f" % (fit_end_time - fit_start_time))
min = -2
max = 2
resolution = 0.05
X,Y,Z  = np.mgrid[min:max:resolution,min:max:resolution,min:max:resolution] # sample way more

xyz = torch.from_numpy(np.vstack([X.ravel(), Y.ravel(),Z.ravel()]).transpose().astype(np.float32)).to(device)

eval = LabelData(xyz)

eval_loader = DataLoader(dataset=eval, batch_size= 22500, shuffle=False) # Make bigger batches

# feed the network bit by bit?
i = 0
for x_batch in eval_loader:
    phi.eval()
    labels =  phi(x_batch).to(device)
print(labels)

visualization_iso(X,Y,Z,labels)

I expect to the predicted values be 0 or 1, or at least be a probability however it gives me big numbers that I don't understand. Like :19.5953
Please take a look at my code and if you spot any big mistakes please tell me so. I'm really confused since it work quite well before I expanded the size of the data I was using.

Regards

Output:
I might be wrong but I am trying to answer based on your code in the question.

You are using BCEwithlogitsloss which means that the model is expected to output logits. logits are the output just before using the sigmoid activation. Recall, sigmoid activation is used to convert outputs to probability (essentially to be between 0 and 1). Logits can be any real number.

Based on this, I think you should pass the output of the model through a sigmoid activation i.e. F.sigmoid(phi(x_batch)). Or you can also just check if your model's output is greater than 0 or less than 0. If greater than 0, the label should be 1.


Instruction: Is there a way to update the weights of a layer/variable directly after a training step?

Input:
There are ways to get and update the weight directly outside of training, but what I am looking to do is after the each training step and after the gradients have updated a particular variable or layer, I would like to save those weights to file and then replace the layer or variable weights with brand new ones. And then continue with the next step (forward pass using the new values in the variable or layer, then backward pass with the calculated loss/gradients) of training.

I have thought about just calling each training step individually, but I am wondering if that is somehow very time/memory inefficient.

Output:
You can try to use a Callback to do that.

Define the function you want:

def afterBatch(batch, logs):
    model.save_weights('weights'+str(batch)) #maybe you want also a method to save the current epoch....

    #option 1
    model.layers[select_a_layer].set_weights(listOfNumpyWeights) #not sure this works

    #option 2
    K.set_value(model.layers[select_a_layer].kernel, newValueInNumpy)
       #depending on the layer you have kernel, bias and maybe other weight names
       #take a look at the source code of the layer you are changing the weights

Use a LambdaCallback:

from keras.callbacks import LambdaCallback

batchCallback = LambdaCallback(on_batch_end = aterBatch)
model.fit(x, y, callbacks = [batchCallback, ....])

There are weight updates after every batch (this might be too much if you are saving weights every time). You can also try on_epoch_end instead of on_batch_end.


Instruction: In PyTorch, what is the input into nll_loss?

Input:
I am looking at the tutorial here: https://pytorch.org/tutorials/beginner/fgsm_tutorial.html

import torch.nn.functional as F
loss = F.nll_loss(output, target)

In the above two lines of code, what exactly is "target"? They load the data set for target but never discuss what it is exactly. The documentation is also hard to understand.

Output:
Check yourself by running below code:

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),
            ])),
        batch_size=1, shuffle=True)
for data, target in test_loader:
    print(data, target)
    break

Here, data is basically a grayscaled MNIST image and target is the label between 0 and 9.

So, in loss = F.nll_loss(output, target), output is the model prediction(what the model predicted on giving an image/data) and target is the actual label of the given image.

Furthermore, in the above example, check below lines:

output = model(data) # shape [1, 10]
init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability

# If the initial prediction is wrong, don't bother attacking, just move on
if init_pred.item() != target.item():
   continue

# Calculate the loss
loss = F.nll_loss(output, target)

In the above code, only those output-target pairs are passed into F.nll_loss loss function, where the model is predicting correctly. In case, it is unable to predict the label correctly, then all the operations(including loss calculation) after that are skipped and it continues with the next example in the test_loader.


Instruction: How can I randomly shuffle the labels of a Pytorch Dataset?

Input:
I am new to Pytorch, and I am having troubles with some technicalities. I have downloaded the MNIST dataset, using the following command:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            download=True)

I now need to run some experiments on this dataset, but using random labels. How can I shuffle/reassign them randomly?

I am trying to do it manually, but it tells me that " 'tuple' object does not support item assignment". How can I do it then?

Second question: How can I remove a training point from the dataset? It gives me the same error, when I try to do it.

Thank you!!

Output:
If you only want to shuffle the targets, you can use target_transform argument. For example:

train_dataset = dsets.MNIST(root='./data',
                            train=True,
                            transform=transforms.ToTensor(),
                            target_transform=lambda y: torch.randint(0, 10, (1,)).item(),
                            download=True)

If you want some more elaborate tweaking of the dataset,
you can wrap mnist completely

class MyTwistedMNIST(torch.utils.data.Dataset):
  def __init__(self, my_args):
    super(MyTwistedMNIST, self).__init__()
    self.orig_mnist = dset.MNIST(...)

  def __getitem__(self, index):
    x, y = self.orig_mnist[index]  # get the original item
    my_x = # change input digit image x ?
    my_y = # change the original label y ?
    return my_x, my_y

  def __len__(self):
    return self.orig_mnist.__len__()

If there are elements of the original mnist you want to completely discard, than by wrapping around the original mnist, your MyTwistedMNIST class can return len smaller than self.orig_mnist.__len__() reflecting the amount of actual mnist examples you want to handle. Moreover, you will need to map the new index of examples to the original mnist index.


Instruction: Why casting input and model to float16 doesn't work?

Input:
I'm trying to change inputs and a deep learning model to flaot16, since I'm using T4 GPU and they work much faster with fp16.
Here's part of the code: I first have my model and then made some dummy data point for the sake of figuring the data casting figured out first (I ran it with the whole batch and got the same error).

model = CRNN().to(device)
model = model.type(torch.cuda.HalfTensor)

data_recon = torch.from_numpy(data_recon)
data_truth = torch.from_numpy(data_truth)

dummy = data_recon[0:1,:,:,:,:] # Gets just one batch
dummy = dummy.to(device)
dummy = dummy.type(torch.cuda.HalfTensor)

model(dummy)

And here's the error I get:

> ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call
> last) <ipython-input-27-1fe8ecc524aa> in <module>
> ----> 1 model(dummy)
>
> /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
> in __call__(self, *input, **kwargs)
>     491             result = self._slow_forward(*input, **kwargs)
>     492         else:
> --> 493             result = self.forward(*input, **kwargs)
>     494         for hook in self._forward_hooks.values():
>     495             hook_result = hook(self, input, result)
>
> <ipython-input-12-06f39f9304a1> in forward(self, inputs, test)
>      57
>      58             net['t%d_x0'%(i-1)] = net['t%d_x0'%(i-1)].view(times, batch, self.filter_size, width,
> height)
> ---> 59             net['t%d_x0'%i] = self.bcrnn(inputs, net['t%d_x0'%(i-1)], test)
>      60             net['t%d_x0'%i] = net['t%d_x0'%i].view(-1, self.filter_size, width, height)
>      61
>
> /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
> in __call__(self, *input, **kwargs)
>     491             result = self._slow_forward(*input, **kwargs)
>     492         else:
> --> 493             result = self.forward(*input, **kwargs)
>     494         for hook in self._forward_hooks.values():
>     495             hook_result = hook(self, input, result)
>
> <ipython-input-11-b687949e9ce5> in forward(self, inputs,
> input_iteration, test)
>      31         hidden = initial_hidden
>      32         for i in range(times):
> ---> 33             hidden = self.CRNN(inputs[i], input_iteration[i], hidden)
>      34             output_forward.append(hidden)
>      35         output_forward = torch.cat(output_forward)
>
> /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
> in __call__(self, *input, **kwargs)
>     491             result = self._slow_forward(*input, **kwargs)
>     492         else:
> --> 493             result = self.forward(*input, **kwargs)
>     494         for hook in self._forward_hooks.values():
>     495             hook_result = hook(self, input, result)
>
> <ipython-input-10-15c0b221226b> in forward(self, inputs,
> hidden_iteration, hidden)
>      23     def forward(self, inputs, hidden_iteration, hidden):
>      24         in_to_hid = self.i2h(inputs)
> ---> 25         hid_to_hid = self.h2h(hidden)
>      26         ih_to_ih = self.ih2ih(hidden_iteration)
>      27
>
> /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py
> in __call__(self, *input, **kwargs)
>     491             result = self._slow_forward(*input, **kwargs)
>     492         else:
> --> 493             result = self.forward(*input, **kwargs)
>     494         for hook in self._forward_hooks.values():
>     495             hook_result = hook(self, input, result)
>
> /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in
> forward(self, input)
>     336                             _pair(0), self.dilation, self.groups)
>     337         return F.conv2d(input, self.weight, self.bias, self.stride,
> --> 338                         self.padding, self.dilation, self.groups)
>     339
>     340
>
> RuntimeError: Input type (torch.cuda.FloatTensor) and weight type
> (torch.cuda.HalfTensor) should be the same

Output:
Check out your implementation of CRNN. My guess is that you have "hidden" state tensor stored in the model, but not as a "buffer" but just as a regular tensor. Therefore, when casting the model to float16 the hidden state remains float32 and causes you this error.

Try to store the hidden state as a register in the module (see register_buffer for more info).
Alternatively, you can explicitly cast to float16 any member tensor in the module by overloading the .to() method of your model.


Instruction: What does "unsqueeze" do in Pytorch?

Input:
The PyTorch documentation says:

Returns a new tensor with a dimension of size one inserted at the specified position. [...]
>>> x = torch.tensor([1, 2, 3, 4])
>>> torch.unsqueeze(x, 0)
tensor([[ 1,  2,  3,  4]])
>>> torch.unsqueeze(x, 1)
tensor([[ 1],
        [ 2],
        [ 3],
        [ 4]])

Output:
If you look at the shape of the array before and after, you see that before it was (4,) and after it is (1, 4) (when second parameter is 0) and (4, 1) (when second parameter is 1). So a 1 was inserted in the shape of the array at axis 0 or 1, depending on the value of the second parameter.

That is opposite of np.squeeze() (nomenclature borrowed from MATLAB) which removes axes of size 1 (singletons).


Instruction: "RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead"?

Input:
I am trying to use a pre-trained model. Here's where the problem occurs

Isn't the model supposed to take in a simple colored image? Why is it expecting a 4-dimensional input?

RuntimeError                              Traceback (most recent call last)
<ipython-input-51-d7abe3ef1355> in <module>()
     33
     34 # Forward pass the data through the model
---> 35 output = model(data)
     36 init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
     37

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in forward(self, input)
    336                             _pair(0), self.dilation, self.groups)
    337         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 338                         self.padding, self.dilation, self.groups)
    339
    340

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 3 3, but got 3-dimensional input of size [3, 224, 224] instead

Where

inception = models.inception_v3()
model = inception.to(device)

Output:
As Usman Ali wrote in his comment, pytorch (and most other DL toolboxes) expects a batch of images as an input. Thus you need to call

output = model(data[None, ...])

Inserting a singleton "batch" dimension to your input data.

Please also note that the model you are using might expect a different input size (3x229x229) and not 3x224x224.


Instruction: I have a GPU and CUDA installed in Windows 10 but Pytorch's torch.cuda.is_available() returns false; how can I correct this?

Input:
I have PyTorch installed on a Windows 10 machine with a Nvidia GTX 1050 GPU. I have installed the CUDA Toolkit and tested it using Nvidia instructions and that has gone smoothly, including execution of the suggested tests.

However, torch.cuda.is_available() returns False. How can I fix this?

Output:
I also had the same issue.
And running this => a=torch.cuda.FloatTensor(), gave the assertion error AssertionError: Torch not compiled with CUDA enabled . ...which kind of cleared that i was running pytorch without cuda.
Steps:

Make sure you have un-installed Pytorch by invoking the following command:
pip uninstall torch

Go to https://pytorch.org/get-started/locally/ and select your system configurations(as shown in the figure).

Copy the exact command from the Run this command dialog and run it on your terminal.


Instruction: Variable length of text with cnn pytorch

Input:
I am a newbie with pytorch and I am wondering what is the best practice for variable length sentences sequences in CNNs.

I want to use CNN for feature selection on top of word emmbeding generated by fasttext and then feed the output into LSTM .

Now I know pytorch has a dynamic graph and I was wondering if there is a way to this except padding

Output:
Although PyTorch has dynamic graph construction, it is not possible to dynamically construct different graphs for different sequences in the same batch.
As such, it is possible to perform stochastic gradient descent, using individual sequences with arbitrary length, but it is not possible to perform batch gradient descent without padding.

To make training more efficient and avoid training over padded values, PyTorch has both the torch.nn.utils.rnn.pack_padded_sequence function and the torch.nn.utils.rnn.pad_packed_sequence to remove and then replace padding for batch operations.

# pack_padded_sequence so that padded items in the sequence won't be shown to the LSTM
X = torch.nn.utils.rnn.pack_padded_sequence(x, X_lengths, batch_first=True)

# now run through LSTM
X, self.hidden = self.lstm(X, self.hidden)

# undo the packing operation
X, _ = torch.nn.utils.rnn.pad_packed_sequence(X, batch_first=True)


Instruction: What are the difference between .bin and .pt pytorch saved model types?

Input:
Sometimes I see .bin files for pretrained pytorch, like the one here

https://github.com/allenai/scibert#pytorch-models

However, the files are usually saved as .pt files.

What's the difference between these two parameter weights file formats? Why are there two?

Output:
There is no difference as it's just an extension. When it comes to UNIX-like OSes one can open the file no matter the extension (see here), Windows on the other hand is built with them in mind (here).
torch can read either .bin or .pt or .anything so it's probably convention employed by the creators of that repository.
Standard approach is to use .pt or .pth, though the second extension collides with Python's text file readable by interpreter, so .pt seems the best idea for now (see this github issue).


Instruction: Can GPU supports multiple jobs without delay?

Input:
So I am running PyTorch deep learning job using GPU
but the job is pretty light.

My GPU has 8 GB but the job only uses 2 GB. Also GPU-Util is close to 0%.

|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   36C    P2    45W / 210W |   1155MiB /  8116MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

based on GPU-Util and memory I might be able to fit in another 3 jobs.

However, I am not sure if that will affect the overall runtime.

If I run multiple jobs on same GPU does that affects the overall runtime?

I think tried once and I think there was delay.

Output:
Yes you can. One option is to use NVIDIA's Multi-Process Service (MPS) to run four copies of your model on the same card.

This is the best description I have found of how to do it: How do I use Nvidia Multi-process Service (MPS) to run multiple non-MPI CUDA applications?

If you are using your card for inference only, then you can host several models (either copies, or different models) on the same card using NVIDIA's TensorRT Inference Service.


Instruction: Can not find

Input:
I create a pytorch extention, followed by this link
but it throws out the error that

  fatal error: torch/extension.h: No such file or directory.

ubuntu 18.04
code::blocks 17.04
gcc 7.4.0

#include <torch/extension.h>

I expect the code to run.
Is there any document to fix this problem?

Output:
 is the canonical header file that create Python bindings for C++/Cuda extensions.
So Try upgrading the version of Pytorch if it's not in newest version.


Instruction: How to split data into train and test sets using torchvision.datasets.Imagefolder?

Input: In my custom dataset, one kind of image is in one folder which torchvision.datasets.Imagefolder can handle, but how to split the dataset into train and test?

Output:
You can use torch.utils.data.Subset to split your ImageFolder dataset into train and test based on indices of the examples.
For example:

orig_set = torchvision.datasets.Imagefolder(...)  # your dataset
n = len(orig_set)  # total number of examples
n_test = int(0.1 * n)  # take ~10% for test
test_set = torch.utils.data.Subset(orig_set, range(n_test))  # take first 10%
train_set = torch.utils.data.Subset(orig_set, range(n_test, n))  # take the rest


Instruction: using huggingface's pytorch- transformers GPT-2 for classifcation tasks

Input:
I want to use GPT-2 to make a text classifier model. I am not really sure what head should I add after I extracted features through the GPT-2. for eample I have a sequence.

import pytorch_transformers as pt
import torch
text=test.iloc[1,1]
text
'If a fire wanted fanning, it could readily be fanned with a newspaper, and as the government grew weaker, I have no doubt that leather and iron acquired durability in proportion, for, in a very short time, there was not a pair of bellows in all Rotterdam that ever stood in need of a stitch or required the assistance of a hammer.'
len(text)

74
tokenizer = pt.GPT2Tokenizer.from_pretrained('gpt2')
model = pt.GPT2Model.from_pretrained('gpt2')
zz = tokenizer.tokenize(text)
z1=torch.tensor([tokenizer.convert_tokens_to_ids(zz)])
z1
tensor([[ 1532,   257,  2046,  2227,  4336,   768,    11,   340,   714, 14704,
           307,   277,  3577,   351,   257,  7533,    11,   290,   355,   262,
          1230,  6348, 17642,    11,   314,   423,   645,  4719,   326, 11620,
           290,  6953,  9477, 26578,   287,  9823,    11,   329,    11,   287,
           257,   845,  1790,   640,    11,   612,   373,   407,   257,  5166,
           286,  8966,  1666,   287,   477, 18481,   353, 11043,   326,  1683,
          6204,   287,   761,   286,   257, 24695,   393,  2672,   262,  6829,
           286,   257, 15554,    13]])
output,hidden=model(z1)
ouput.shape
torch.Size([1, 74, 768])

the output of GPT2 is n x m x 768 for me, which n is the batch size,m is the number of tokens in the seqence(for example I can pad/truncate to 128.), so I can not do what as the paper said for a classification task just add a fully connected layer in the tail.And I searched on google, few GPT-2 classification task is mensioned.
I am not sure what is correct. Should I do flatten/max pooling/average pooling before the  fully connected layer or something else?

Output:
" so I can not do what as the paper said for a classification task just add a fully connected layer in the tail." - This is the answer to your question.

Usually, transformers like BERT and Roberta, have bidirectional self-attention and they have the [CLS] token where we feed in to the classfier. Since GPT-2 is left-right you need to feed the final token of the embeddings sequence.

P.S - Can you put the link to the paper.


Instruction: Backward function in PyTorch

Input:
I have some question about pytorch's backward function I don't think I'm getting the right output :
import numpy as np
import torch
from torch.autograd import Variable
a = Variable(torch.FloatTensor([[1,2,3],[4,5,6]]), requires_grad=True)
out = a * a
out.backward(a)
print(a.grad)

the output is
tensor([[ 2.,  8., 18.],
        [32., 50., 72.]])

maybe it's 2*a*a
but i think the output suppose to be
tensor([[ 2.,  4., 6.],
        [8., 10., 12.]])

2*a. cause d(x^2)/dx=2x

Output:
Please read carefully the documentation on backward() to better understand it.
By default, pytorch expects backward() to be called for the last output of the network - the loss function. The loss function always outputs a scalar and therefore, the gradients of the scalar loss w.r.t all other variables/parameters is well defined (using the chain rule).
Thus, by default, backward() is called on a scalar tensor and expects no arguments.
For example:
a = torch.tensor([[1,2,3],[4,5,6]], dtype=torch.float, requires_grad=True)
for i in range(2):
  for j in range(3):
    out = a[i,j] * a[i,j]
    out.backward()
print(a.grad)

yields

tensor([[ 2.,  4.,  6.],
        [ 8., 10., 12.]])

As expected: d(a^2)/da = 2a.
However, when you call backward on the 2-by-3 out tensor (no longer a scalar function) - what do you expects a.grad to be? You'll actually need a 2-by-3-by-2-by-3 output: d out[i,j] / d a[k,l](!)
Pytorch does not support this non-scalar function derivatives.  Instead, pytorch assumes out is only an intermediate tensor and somewhere "upstream" there is a scalar loss function, that through chain rule provides d loss/ d out[i,j]. This "upstream" gradient is of size 2-by-3 and this is actually the argument you provide backward in this case: out.backward(g) where g_ij = d loss/ d out_ij.
The gradients are then calculated by chain rule d loss / d a[i,j] = (d loss/d out[i,j]) * (d out[i,j] / d a[i,j])
Since you provided a as the "upstream" gradients you got
a.grad[i,j] = 2 * a[i,j] * a[i,j]

If you were to provide the "upstream" gradients to be all ones
out.backward(torch.ones(2,3))
print(a.grad)

yields

tensor([[ 2.,  4.,  6.],
        [ 8., 10., 12.]])

As expected.
It's all in the chain rule.


Instruction: i want to convert the code below(neural network) from keras to pytorch

Input:
i want to use pytorch instead of keras but i failed to do it my self

Keras

def _model(self):
    model = Sequential()
    model.add(Dense(units=64, input_dim=self.state_size,
    activation="relu"))
    model.add(Dense(units=32, activation="relu"))
    model.add(Dense(units=8, activation="relu"))
    model.add(Dense(self.action_size, activation="linear"))
    model.compile(loss="mse", optimizer=Adam(lr=0.001))

    return model

Pytorch

class Model(nn.Module):
    def __init__(self, input_dim):
        super(Model, self).__init__()
        self.fc1 = nn.ReLU(input_dim, 64)
        self.fc2 = nn.ReLU(64,32)
        self.fc3 = nn.Relu(32, 8)
        self.fc4 = nn.Linear(8, 3)
model = Model()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Output:
model = Model()

You need to provide an argument when you call Model() since your __init__(self, input_dims) requires an argument.

It should be this:

model = Model(<integer dimensions for your network>)


Instruction: Using automatic differentiation libraries to compute partial derivatives of an arbitrary tensor

Input:
(Note: this is not a question about back-propagation.)
I am trying so solve on a GPU a non-linear PDE using PyTorch tensors in place of Numpy arrays. I want to calculate the partial derivatives of an arbitrary tensor, akin to the action of the center finite-difference numpy.gradient function. I have other ways around this problem, but since I am already using PyTorch, I'm wondering if it is possible use the autograd module (or, in general, any other autodifferentiation module) to perform this action.

I have created a tensor-compatible version of the numpy.gradient function - which runs a lot faster. But perhaps there is a more elegant way of doing this. I can't find any other sources that address this question, either to show that it's possible or impossible; perhaps this reflects my ignorance with the autodifferentiation algorithms.

Output:
I've had this same question myself: when numerically solving PDEs, we need access to spatial gradients (which the numpy.gradients function can give us) all the time - could it be possible to use automatic differentiation to compute the gradients, instead of using finite-difference or some flavor of it?

"I'm wondering if it is possible use the autograd module (or, in general, any other autodifferentiation module) to perform this action."

The answer is no: as soon as you discretize your problem in space or time, then time and space become discrete variables with a grid-like structure, and are not explicit variables which you feed into some function to compute the solution to the PDE.

For example, if I wanted to compute the velocity field of some fluid flow u(x,t), I would discretize in space and time, and I would have u[:,:] where the indices represent positions in space and time.

Automatic differentiation can compute the derivative of a function u(x,t). So why can't it compute the spatial or time derivative here? Because you've discretized your problem. This means you don't have a function for u for arbitrary x, but rather a function of u at some grid points. You can't differentiate automatically with respect to the spacing of the grid points.

As far as I can tell, the tensor-compatible function you've written is probably your best bet. You can see that a similar question has been asked in the PyTorch forums here and here. Or you could do something like

dx = x[:,:,1:]-x[:,:,:-1]

if you're not worried about the endpoints.


Instruction: Pytorch pretrained model not recognizing my image

Input:
I have an image that is a torch.Tensor format. I want to feed it directly into a pre-trained classifier, like Inception v3.

However, it's being predicted incorrectly (no error message, just output is wrong). I guess this is because I didn't normalize it (in accordance with: https://pytorch.org/docs/stable/torchvision/models.html), so that's what I'm trying to do.

The problem is, the normalization requires numpy input. However, to get numpy, I do this and it gives me an error:

----> 9 image = data.numpy()[0].transpose((1, 2, 0))  # [image_size, image_size, RGB]

RuntimeError: Can't call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

I cannot call detach because I require gradients to flow through the image (which was generated by other functions). Is there a way to bypass converting this to a numpy. If not, how do I keep the gradient flow?

Output:
One option is to scale images into the interval [0, 1], before using the mean and standard deviation vectors as tensors.

import torch

data = (data - data.min()) / (data.max() - data.min())  # rescale to [0, 1]
mean = torch.tensor([[[0.485, 0.456, 0.406]]])
std = torch.tensor([[[0.229, 0.224, 0.225]]])
data = (data - mean) / std  # normalise with tensors


Instruction: ImportError: cannot import name 'warmup_linear'

Input:
While trying to import warmup_linear, I'm getting this error

ImportError: cannot import name 'warmup_linear'

Import -

from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear

Requirements file

boto3==1.9.198
botocore==1.12.198
certifi==2019.6.16
chardet==3.0.4
docutils==0.14
h5py==2.9.0
idna==2.8
jmespath==0.9.4
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
numpy==1.17.0
Pillow==6.1.0
python-dateutil==2.8.0
pytorch-pretrained-bert==0.6.2
PyYAML==5.1.1
regex==2019.6.8
requests==2.22.0
s3transfer==0.2.1
scipy==1.3.0
seqeval==0.0.12
six==1.12.0
torch==1.1.0
torchvision==0.3.0
tqdm==4.32.2
urllib3==1.25.3

What needs to be done to import 'warmup_linear'?

Output:
The version 0.4.0 doesn't give this issue.
pip install pytorch_pretrained_bert==0.4.0
or downgrading to 0.6.1 solved it.


Instruction: Pytorch: turning a [1,x] sized tensor into an [x] sized tensor

Input:
While trying to load in Pytorch 0.4.0 a model that has probably been produced by Pytorch 0.3.1, I keep getting such errors:

While copying the parameter named "conv1_7x7_s2_bn.bias", whose dimensions in the
model are torch.Size([64]) and whose dimensions in the checkpoint are torch.Size([1, 64]).

I thought that if I had applied transpose on each tensor, then it would work, but it is still failing, as the dimension turns into [64, 1], rather than [64], which I need.

I can I remove the redundant dimension and thus turn the 1-row matrix into a vector?

Note: When calling torch.flatten, I get:

AttributeError: module 'torch' has no attribute 'flatten'

Output:
Removing empty dimensions is called "squeezing". NumPy does it, Tensorflow does it and PyTorch does it.

So the correct command is:

torch.squeeze(tensor)