Instruction: Difference between nn.MaxPool2d vs.nn.functional.max_pool2d?

Input:
Whats the difference between: nn.MaxPool2d(kernel_size, stride) and nn.functional.max_pool2d(t, kernel_size, stride)?
The first one I define in the module and the second in the forward function?
Thanks

Output:
They are essentially the same. The difference is that torch.nn.MaxPool2d is an explicit nn.Module that calls through to torch.nn.functional.max_pool2d() it its own forward() method.

You can look at the source for torch.nn.MaxPool2d here and see the call for yourself: https://pytorch.org/docs/stable/_modules/torch/nn/modules/pooling.html#MaxPool2d

Reproduced below:

def forward(self, input):
        return F.max_pool2d(input, self.kernel_size, self.stride,
                            self.padding, self.dilation, self.ceil_mode,
                            self.return_indices)

Why have two approaches for the same task? I suppose it's to suit the coding style of the many people who might use PyTorch. Some prefer a stateful approach while others prefer a more functional approach.

For example having torch.nn.MaxPool2d means that we could very easily drop it into a nn.Sequential block.

model = nn.Sequential(
          nn.Conv2d(1,3,3),
          nn.ReLU(),
          nn.MaxPool2d((2, 2))
        )


Instruction: Taking a derivative through torch.ge, or how to explicitly define a derivative in pytorch

Input:
I am trying to set up a network in which one layer maps from real numbers to {0, 1} (i.e. makes output binary).

What I tried

While I was able to find that torch.ge provides such functionality, whenever I want to train any parameter occurring before that layer in a network PyTorch breaks.

I have been also trying to find if there is any way in PyTorch/autograd, to override the derivative of a module by hand. More specifically in this cause, I would just like to pass derivative through the torch.ge, without changing it.

Minimal Example

Here is a minimal example I produced, which uses a typical neural network training structure in PyTorch.

import torch
import torch.nn as nn
import torch.optim as optim

class LinearGE(nn.Module):
    def __init__(self, features_in, features_out):
        super().__init__()
        self.fc = nn.Linear(features_in, features_out)

    def forward(self, x):
        return torch.ge(self.fc(x), 0)

x = torch.randn(size=(10, 30))
y = torch.randint(2, size=(10, 10))

# Define Model
m1 = LinearGE(30, 10)

opt = optim.SGD(m1.parameters(), lr=0.01)

crit = nn.MSELoss()

# Train Model
for x_batch, y_batch in zip(x, y):
    # zero the parameter gradients
    opt.zero_grad()

    # forward + backward + optimize
    pred = m1(x_batch)
    loss = crit(pred.float(), y_batch.float())
    loss.backward()
    opt.step()

What I encountered

When I run the above code the following error occurs:

File "__minimal.py", line 33, in <module>
    loss.backward()
...
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

This error makes sense since torch.ge function is not differentiable. However, since MaxPool2D is also not differentiable, I believe that there are ways of mitigating non-differentiability in PyTorch.

It would be great if someone could point me to any source which can help me either implement my own backprop for a custom module, or any way of avoiding this error message.

Thanks!

Output:
Two things I noticed

If your input x is 10x30 (10 examples, 30 features)and the number of output node is 10, then the parameter matrix is 30x10. The expected output matrix is 10x10 (10 examples 10 output nodes)
ge = greater than and equal to. As the code indicated, x >= 0 element wise. We can use relu.

class LinearGE(nn.Module):
    def __init__(self, features_in, features_out):
        super().__init__()
        self.fc = nn.Linear(features_in, features_out)
        self.relu = nn.ReLU(inplace=True)

    def forward(self, x):
        return self.relu(self.fc(x))

or torch.max

torch.max(self.fc(x), 0)[0]


Instruction: How can I change the following code from pytorch to tensorflow?

Input:
I want to change the follow pytorch network (v1.2) to tensorflow. I am confusing between tf.nn.conv2d and tf.keras.layers.Conv2D what should I choose?

import torch.nn as nn
nn.Sequential(nn.Conv2d(in_planes, out_planes, kernel_size=kernel_size, stride=stride, padding=padding, dilation=dilation, bias=True),
    nn.BatchNorm2d(out_planes),
    nn.ReLU(inplace=True))

Output:
tf.nn.conv2d is functional api and tf.keras.layers.Conv2D is layer-class api. You should use the latter one. It's quite as similar as the relationship between torch.nn.functional.conv2d and torch.nn.Conv2D.

import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Conv2D, ReLU, BatchNormalization

model = Sequential()
model.add(Conv2D(filters=10, kernel_size=3, strides=1))
model.add(BatchNormalization())
model.add(ReLU())


Instruction: torch.softmax and torch.sigmoid are not equivalent in the binary case

Input:
Given:
x_batch = torch.tensor([[-0.3, -0.7], [0.3, 0.7], [1.1, -0.7], [-1.1, 0.7]])

and then applying torch.sigmoid(x_batch):
tensor([[0.4256, 0.3318],
        [0.5744, 0.6682],
        [0.7503, 0.3318],
        [0.2497, 0.6682]])

gives a completely different result to torch.softmax(x_batch,dim=1):
tensor([[0.5987, 0.4013],
        [0.4013, 0.5987],
        [0.8581, 0.1419],
        [0.1419, 0.8581]])

As per my understanding, isn't the softmax is exactly the same as the sigmoid in the binary case?

Output:
You are misinformed. Sigmoid and softmax are not equal, even for the 2 element case.
Consider x = [x1, x2].
sigmoid(x1) = 1 / (1 + exp(-x1))

but
softmax(x1) = exp(x1) / (exp(x1) + exp(x2))
            = 1 / (1 + exp(-x1)/exp(-x2))
            = 1 / (1 + exp(-(x1 - x2))
            = sigmoid(x1 - x2)

From the algebra we can see an equivalent relationship is
softmax(x, dim=1) = sigmoid(x - fliplr(x))

or in pytorch
x_softmax = torch.sigmoid(x_batch - torch.flip(x_batch, dims=(1,))


Instruction: Error: symeig_cpu: the algorithm failed to converge: 6 off-diagonal elements of an intermediate tridiagonal form did not converge to zero

Input:
I am trying to use https://github.com/Michaelvll/DeepCCA

After 20-40 iterations, It gives the following error:

RuntimeError: symeig_cpu: the algorithm failed to converge; 6 off-diagonal elements of an intermediate tridiagonal form did not converge to zero.

Error is generated from https://github.com/Michaelvll/DeepCCA/blob/master/objectives.py#L46

[D1, V1] = torch.symeig(SigmaHat11, eigenvectors=True)

System Configuration:

Windows 10.
Python 3.7
Pytorch 1.2.0

How can I debug this?

Output: I have come across a similar error. Its root cause was a failure in the Cholesky decomposition at some point down the road, because a tensor was singular.


Instruction: Non LSTM: Trying to backward through the graph a second time, but the buffers have already been freed

Input:
Note that unlike other questions, this is not about any RNN structure. I wish to create a model that has changing gradients, and will look like below. The breakpoints are manually supplied.

The model that I have created is as follows:
class Trend(nn.Module):
    """
    Broken Trend model, with breakpoints as defined by user.
    """
    def __init__(self, breakpoints):
        super().__init__()
        self.bpoints = breakpoints[None, :]
        self.init_layer = nn.Linear(1,1) # first linear bit
        # extract gradient and bias
        w = self.init_layer.weight
        b = self.init_layer.bias
        self.params = [[w,b]] # save it to buffer

        if len(breakpoints>0):
            # create deltas which is how the gradient will change
            deltas = torch.randn(len(breakpoints)) / len(breakpoints) # initialisation
            self.deltas = nn.Parameter(deltas) # make it a parameter

            for d, x1 in zip(self.deltas, breakpoints):
                y1 = w *x1 + b # find the endpoint of line segment (x1, y1)
                w = w + d # add on the delta to gradient
                b = y1 - w * x1 # find new bias of line segment
                self.params.append([w,b]) # add to buffer

        # create buffer
        self.wb = torch.zeros(len(self.params), len(self.params[0]))

    def __copy2array(self):
        """
        Saves parameters into wb
        """
        for i in range(self.wb.shape[0]):
            for j in range(self.wb.shape[1]):
                self.wb[i,j] = self.params[i][j]

    def forward(self, x):
        # get the line segment area (x_sec) for each x
        x_sec = x >= self.bpoints
        x_sec = x_sec.sum(1)
        self.__copy2array() # copy across parameters into matrix

        # get final prediction y = mx +b for relevant section
        return x*self.wb[x_sec][:,:1] + self.wb[x_sec][:,1:]

However, once I attempt to train it I get the error RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
I obtained the above plot by doing:
time = torch.arange(700).float()[:,None]
y_pred = model(time)
plt.plot(time, y_pred.detach().numpy())
plt.show()

So we know the forward pass is working as expected. However the backward pass is not quite working. Was wondering what I need to change to get it working.
If you're wondering why __copy2array is being used, when I tried to use torch.Tensor(self.params) it destroyed the gradients in those parameters. Thanks in advance.

Output: Since your answer does not contain your complete code it is hard to judge, but I recommend trying what the error message says: Replace .backward() with .backward( retain_graph=True). This means that the gradient is not deleted after the update.


Instruction: Pytorch simulation fails to converge on convex loss function when not initialized with 0

Input:
My code works when the weights initialized with 0. When I initialize them according to some seed, they fail to converge. This should be a bug since the loss function is convex.

I filtered two labels from MNIST (0 and 1), and then I trained a logistic regression model using pytorch. Since I use only 200 training samples (and 784 parameters), the model should quickly converge to 100% accuracy on the training set. This is not the case when the weights initialize by some seed.

I had some problem to share my code on stackoverflow, so here is a link to the code: https://drive.google.com/file/d/1ELe8TIWrXMiXgsB63B0Ss43GPr719rGc/view?usp=sharing

Output:
Your data are not rescaled and normalized. If you look at the images variable in your training loop it's between 0 and 255 this is in all likelihood hurting your training process.

There are cleaner ways to subsample the dataset as you want, but without modifying too much of your code, using this data loading definition

import torchvision.transforms as transforms

#Load Dataset
preprocessing = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.1307,), (0.3081,))
                                   ])
train_dataset = dsets.MNIST(root='./data', train=True, transform=preprocessing, download=True)

#Filter samples by label (to get binary classification) and by number of training samples
Binary_filter=torch.add(train_dataset.targets==1, train_dataset.targets==0)
train_dataset.data, train_dataset.targets = train_dataset.data[Binary_filter],train_dataset.targets[Binary_filter]

TrainSet_filter=torch.cat((torch.ones(num_of_training_samples)
                         ,torch.zeros(len(train_dataset.targets)-num_of_training_samples)),0).bool()
train_dataset.data, train_dataset.targets = train_dataset.data[TrainSet_filter], train_dataset.targets[TrainSet_filter]

#Make Dataset Iterable
train_loader = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)

I have ~100% accuracy in about 5-10 epochs.


Instruction: Is there another step to importing Pytorch and using it?

Input:
I'm trying to test Pytorch but the first step is to get it running and usable on my computer. I have it to an extent but I need it to function in VS code which it does, it's just that it reads as an error despite working.

I've set up pytorch locally on my computer. I can run the test scripts fine in VS Code's integrated terminal.

The issue comes up when I try to do some work normally in VS code.

from __future__ import print_function
import torch
x = torch.rand(5, 3) //This line has an error. torch.rand(5,3) is not callable)
print(x)

It says that module 'torch' has no 'rand' member. But it still outputs correctly.
Running this in the terminal as a python file or in debug mode results in.

(base) c:\Users\Sean\Desktop\Test\hello>D:\Anaconda\python.exe c:\Users\Sean\.vscode\extensions\ms-python.python-2019.10.44104\pythonFiles\ptvsd_launcher.py --default --client --host localhost --port 63625 c:\Users\Sean\Desktop\Test\hello\something.py
tensor([[0.5449, 0.1669, 0.4740],
        [0.3079, 0.0447, 0.9543],
        [0.9137, 0.3987, 0.5736],
        [0.1788, 0.4932, 0.5584],
        [0.1632, 0.6285, 0.4483]])

(base) c:\Users\Sean\Desktop\Test\hello>D:/Anaconda/Scripts/activate

(base) c:\Users\Sean\Desktop\Test\hello>conda activate Anaconda
Could not find conda environment: Anaconda
You can list all discoverable environments with `conda info --envs`.

So it's working, what I want is to not have it coming up as an error. It'll be hard to find the actual bugs later on if it continues and I feel it might cause problems if not addressed. Any help with telling me the root issue and how to go about fixing it would be appreciated.

Output:
Found the answer. The issue was pylint not recognizing Pytorch or Numpty methods. The functions still worked but the error messages make it hard to see actual error messages.

Fixed by adding the following to user settings.

"python.linting.pylintArgs": [
"--errors-only",
"--generated-members=numpy.* ,torch.* ,cv2.* , cv.*"
]

Recorded Error fix:
https://github.com/pytorch/pytorch/issues/701


Instruction: Weighted summation of embeddings in pytorch

Input:
I have a sequence of 12 words which I represent using a 12x256 matrix (using word embeddings). Let us refer to these as . I wish to take this as input and output a 1x256 vector. However I don't want to use a (12x256) x 256 dense layer. Instead I want to create the output embedding using a weighted summation of the 12 embeddings

where the wi s are scalars (thus there is weight sharing).

How can I create trainable wi s in pytorch? I am new and only familiar with the standard modules like nn.Linear.

Output:
You can implement this via 1D convolution with kernel_size = 1

import torch

batch_size=2

inputs = torch.randn(batch_size, 12, 256)
aggregation_layer = torch.nn.Conv1d(in_channels=12, out_channels=1, kernel_size=1)
weighted_sum = aggregation_layer(inputs)

Such convolution will have 12 parameters. Each parameter will be a equal to e_i in formula you provided.

In other words this convolution will ran over dimetion with size 256 and sum it with learnable weights.


Instruction: Pytorch linear/affine layer parameters confusing

Input:
I'm on the Pytorch documentation (https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html) and I'm not really understanding why they are making the the affine layer (16 * 6 * 6, 120).  I understand that the last outputs from the convolution layer were 16 and the output here is 120, but even with their annotation, I'm not understanding where the 6 * 6 comes from.  Can someone please explain?

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

Output:
The 6x6 comes from the height and width of x after it has been passed through your convolutions and maxpools.

Here is a simplified version where you can see how the shape changes at each point. It may help to print out the shapes in their example so you can see exactly how everything changes.

import torch
import torch.nn as nn
import torch.nn.functional as F

conv1 = nn.Conv2d(1, 6, 3)
conv2 = nn.Conv2d(6, 16, 3)

# Making a pretend input similar to theirs.
# We define an input with 1 batch, 1 channel, height 32, width 32
x = torch.ones((1,1,32,32))

# Simulating forward()

x = F.max_pool2d(F.relu(conv1(x)), (2, 2))
print(x.shape)  # torch.Size([1, 6, 15, 15])  1 batch, 6 channels, height 15, width 15

x = F.max_pool2d(F.relu(conv2(x)), 2)
print(x.shape)  # torch.Size([1, 16, 6, 6])  1 batch, 16 channels, height 6, width 6

Next they flatten x and pass it through fc1 which accepts 16*6*6 and produces 120 outputs.


Instruction: Recreating char level RNN for generating text

Input:
I tied to follow a book on deep learning, where there is an chapter about generating text in the style of an example. They used an char level RNN with two LSTM layers in it to generate text in the style of shakespare. But the code in the book (also online: https://github.com/DOsinga/deep_learning_cookbook/blob/master/05.1%20Generating%20Text%20in%20the%20Style%20of%20an%20Example%20Text.ipynb) is written in keras and I only use pytorch. So i tied to recreate it exactly in pytorch, with same network structure and hyperparameters.

So after recreating it and making it work without errors it trained it and it only learned to write the most common character, a space. Then i tried to overfit it on one realy simple sentence, so I had to decrease the sequence lenght to 8. This also did not work, but when decreasing the hidden size of the LSTMs to only 32 it learned it nearly perfectly.
So then I continued working on the original text and started to play arround with the hidden size, learning rate, optimizer (also tried adam) and trained it even longer. The best I could achieve were some random letters, still with a lot of spaces and somtimes something like "her", but far from readable, with still an quite high loss. I used RMSprop with lr=0.01 and a hidden size of 128 over 20000 epochs. I also tried to initialize the hidden state and cell state to zero.

The problem is, that my results are far worse than those in the book, but I did exactly the same just in pytorch. Can someone please tell me, what I should try or what I have done wrong. Any help is appreciated!
PS: Sorry for my bad english.

Here is my code with the original hyperparameters:

#hyperparameters
batch_size = 256
seq_len = 160
hidden_size = 640
layers = 2

#network structure
class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(len(chars),hidden_size,layers)
        self.linear = nn.Linear(hidden_size,len(chars))
        self.softmax = nn.Softmax(dim=2)
    def forward(self,x,h,c):
        x,(h,c) = self.lstm(x,(h,c))
        x = self.softmax(self.linear(x))
        return x,h,c

#create network, optimizer and criterion
rnn = RNN().cuda()
optimizer = torch.optim.RMSprop(rnn.parameters(),lr=0.01)
criterion = nn.CrossEntropyLoss()

#training loop
plt.ion()
losses = []
loss_sum = 0
for epoch in range(10000):
    #generate input and target filled with zeros
    input = numpy.zeros((seq_len,batch_size,len(chars)))
    target = numpy.zeros((seq_len,batch_size))
    for batch in range(batch_size):
        #choose random starting index in text
        start = random.randrange(len(text)-seq_len-1)
        #generate sequences for that batch filled with zeros
        input_seq = numpy.zeros((seq_len+1,len(chars)))
        target_seq = numpy.zeros((seq_len+1))
        for i,char in enumerate(text[start:start+seq_len+1]):
            #convert character to index
            idx = char_to_idx[char]
            #set value of index to one (one-hot-encoding)
            input_seq[i,idx] = 1
            #set value to index (only label)
            target_seq[i] = idx
        #insert sequences into input and target
        input[:,batch,:] = input_seq[:-1]
        target[:,batch] = target_seq[1:]
    #convert input and target from numpy array to pytorch tensor on gpu
    input = torch.from_numpy(input).float().cuda()
    target = torch.from_numpy(target).long().cuda()

    #initialize hidden state and cell state to zero
    h0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    c0 = torch.zeros(layers,batch_size,hidden_size).cuda()
    #run the network on the input
    output,h,c = rnn(input,h0,c0)
    #calculate loss and perform gradient descent
    optimizer.zero_grad()
    loss = criterion(output.view(-1,len(chars)),target.view(-1))
    loss.backward()
    optimizer.step()

Plot of the loss with original hyperparameters:

Example of target and output after training:

Target:  can bring this instrument of honour
    again into his native quarter, be magnanimous in the enterprise,
    and go on; I will grace the attempt for a worthy e
Output:

Plot of the loss with hidden size of 128 over 20000 epochs (best results):

Output:
I later finally found a way to achive something close to real sentences, maybe it will help someone. Here is an example result:

  -I have not seen him and the prince was a signt of the streme of the sumpering of the property of th

In my case the important change was to not inizialize the hidden and cell state to zero every batch but only every epoch. For this to work I had to rewrite the batch generator, so that it produces batches following on each other.


Instruction: Creating a Feed Forward NN Model in Pytorch with a dynamic number of hidden layers

Input:
Why are these two code segments not equivalent:
Segment 1: Creating a model with 2 layers.

class FNNModule(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim1, hidden_dim2, non_linear_function):
        super().__init__()
        self.hidden1 = nn.Linear(input_dim, hidden_dim1)
        self.hidden2 = nn.Linear(hidden_dim1, hidden_dim2)
        self.non_linear_function = non_linear_function()
        self.final_linear = nn.Linear(hidden_dim2, output_dim)

    def forward(self, x):
        out = self.hidden1(x)
        out = self.non_linear_function(out)
        out = self.hidden2(x)
        out = self.non_linear_function(out)
        out = self.final_linear(out)
        return out

Segment Two: Creating the same model but changing code where hidden_layers is a variable:

class FNNModuleVar(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim_array = [], non_linear_function_array=[]):
        super().__init__()
        self.linear_functions = []
        self.non_linear_functions = [x() for x in non_linear_function_array]
        self.hidden_layers = len(hidden_dim_array)
        for l in range(self.hidden_layers):
            self.linear_functions.append(nn.Linear(input_dim, hidden_dim_array[l]))
            input_dim = hidden_dim_array[l]
        self.final_linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = x
        for i in range(self.hidden_layers):
            out = self.linear_functions[i](out)
            out = self.non_linear_functions[i](out)
        out = self.final_linear(x)
        return out
modelVar = FNNModuleVar(input_dim, output_dim, [100, 50], [nn.Tanh, nn.Tanh])
model = FNNModule(input_dim, output_dim, 100, 50, nn.Tanh)

When I try to iterate through modelVar.parameters() and model.parameters() I see that I have very different models.

What am I doing wrong in modelVar?

Output:
Those modules are called as you would expect them to be called they are just not visible to the Module. In order to make them visible you can wrap them in a nn.ModuleList like this:

class FNNModuleVar(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim_array = [], non_linear_function_array=[]):
        super().__init__()
        self.linear_functions = []
        self.non_linear_functions = [x() for x in non_linear_function_array]
        self.hidden_layers = len(hidden_dim_array)
        for l in range(self.hidden_layers):
            self.linear_functions.append(nn.Linear(input_dim, hidden_dim_array[l]))
            input_dim = hidden_dim_array[l]
        self.linear_functions = nn.ModuleList(self.linear_functions)
        self.final_linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = x
        for i in range(self.hidden_layers):
            out = self.linear_functions[i](out)
            out = self.non_linear_functions[i](out)
        out = self.final_linear(out)
        return out

printing the models now would yield:

FNNModule(
  (hidden1): Linear(in_features=50, out_features=100, bias=True)
  (hidden2): Linear(in_features=100, out_features=50, bias=True)
  (non_linear_function): Tanh()
  (final_linear): Linear(in_features=50, out_features=100, bias=True)
)
FNNModuleVar(
  (linear_functions): ModuleList(
    (0): Linear(in_features=50, out_features=100, bias=True)
    (1): Linear(in_features=100, out_features=50, bias=True)
  )
  (final_linear): Linear(in_features=50, out_features=100, bias=True)
)

More details: https://pytorch.org/docs/stable/nn.html#torch.nn.ModuleList


Instruction: Runtime error 999 when trying to use cuda with pytorch

Input:
I installed Cuda 10.1 and the latest Nvidia Driver for my Geforce 2080 ti. I try to run a basic script to test if pytorch is working and I get the following error:

RuntimeError: cuda runtime error (999) : unknown error at ..\aten\src\THC\THCGeneral.cpp:50

Below is the code im trying to run:

import torch
torch.cuda.current_device()
torch.cuda.is_available()
torch.cuda.get_device_name(0)

Output:
Restarting my computer fixed this for me.

But for a less invasive fix, you can also try this solution (from a tensorflow issue thread):

sudo rmmod nvidia_uvm
sudo rmmod nvidia
sudo modprobe nvidia
sudo modprobe nvidia_uvm


Instruction: Can pytorch optimize sequential operations (like a tensorflow graph or JAX's jit)?

Input:
Originally, tensorflow and pytorch had a fundamental difference:

tensorflow is based on a computional graph. Building this graph and evaluating it in a session are two separate steps. While it is being used, the graph doesn't change, which allows for optimizations.
torch eagerly evaluates operations on a tensor. This makes the API more convenient (no sessions) but also looses the potential to recognize and optimize operations that always occur in sequence.

Now this difference is becoming less clear. Tensorflow has answered to the popularity of torch with tf eager. There is also the JAX project, which builds on the same underlying framework as tensorflow (XLA). JAX has no concept of a session. But it allows you to compile multiple operations together by simply calling jit.

Since Tensorflow has moved to cover PyTorch functionality, is PyTorch also working on integrating Tensorflow advantages? Is there something like a session or jit functionality in PyTorch (or on its roadmap) ?

The API docs have a jit section, but as far as I can see, that is more about exporting your models.

Output:
As you mentioned, there is a torch.jit and it's purpose is also to introduce optimization in the exported graph (e.g. kernel fusion, optimization of constants etc.). IIRC you can find some source code regarding those in their github repo here, though I'm not sure whether those are explicitly mentioned somewhere in the docs (or explicitly enough to be remembered).

Since 1.3 there is also quantization introduced (see here for some introduction). In tutorials section, namely here you can see explicit fusion of Conv2d, BatchNorm and ReLU in order to improve performance. Ofc there also exists specific stuff like using int instead of float for weights (quantization), mixed arithmetic (using half float precision whenever possible, see NVidia's Apex) and others.

Last but not least, I don't think for a well written model using vectorized operations and exported with torchscript you are gonna see really substantial runtime differences because of some generic graph optimization. Still it differs whether you are going to use GPU, CPU, TPU, what are their versions, whether you are after inference only or training as well etc. It's pretty hard to pinpoint how fast tensorflow is in comparison to pytorch (besided some well-known issues in both frameworks). All in all it depends and measurements vary a lot AFAIK.

BTW. When it comes to advantages of each framework their core indeed starts to cover similar things (PyTorch got mobile support lately, see here). Real difference is still different underlying approach and what each framework has to do to circumvent those limitations.


Instruction: Is there a method in Pytorch to count the number of unique values in a way that can be back propagated?

Input:
Given the following tensor (which is the result of a network [note the grad_fn]):

tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.],
       grad_fn=<MvBackward>)

Which we will define as:

xx = torch.tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.]).requires_grad_(True)

I would like to define an operation which counts the number of occurrences of each value in such a way that the operation will output the following tensor:

tensor([2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 7, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
        0, 1, 1])

i.e. there are 2 zeros, 1 one, 2 thirteens, etc... the total number of possible values is set upstream, but in this example is 243

So far I have tried the following approaches, which successfully produce the desired tensor, but do not do so in a way that allows computing gradients back through the network:

Attempt 1

tt = []
for i in range(243):
    tt.append((xx == i).unsqueeze(0))
torch.cat(tt,dim=0).sum(dim=1)

Attempt 2

tvtensor = torch.tensor([i for i in range(243)]).unsqueeze(1).repeat(1,xx.shape[0]).float().requires_grad_(True)
(xx==tvtensor).sum(dim=1)

EDIT: Added Attempt

Attempt 3

-- Didn't really expect this to back prop, but figured I would give it a try anyway

ll = torch.zeros((1,243))
for x in xx:
    ll[0,x.long()] += 1

Any help is appreciated

EDIT: As requested the end goal of this is the following:

I am using a technique for calculating structural similarity between two time sequences. One is real and the other is generated. The technique is outlined in this paper (https://link.springer.com/chapter/10.1007/978-3-642-02279-1_33) where a time series is converted to a sequence of code words and the distribution of code words (similar to the way that Bag of Words is used in NLP) is used to represent the time series. Two series are considered similar when the two signal distributions are similar. This is what the counting statistics tensor is for.

What is desired is to be able to construct a loss function which consumes this tensor and measures the distance between the real and generated signal (euclidian norm on the time domain data directly does not work well and this approach claimed better results), so that it can update the generator appropriately.

Output:
I would do it with unique method (only to count occurrences):

if you want to count the occurrences, you have to add the parameter return_counts=True

I did it in the version 1.3.1

This is the fast way to count occurrences, however is a non-differentiable operation, therefore, this method is not recommendable (anyway I have described the way to count ocurrences). To perform what you want, I think you should turn the input into a distribution by means of a differentiable function (softmax is the most used) and then, use some way to measure the distance between distributions (output and target) like cross-entropy, KL (kullback-liebler), JS or wasserstein.


Instruction: Updating a BERT model through Huggingface transformers

Input:
I am attempting to update the pre-trained BERT model using an in house corpus. I have looked at the Huggingface transformer docs and I am a little stuck as you will see below.My goal is to compute simple similarities between sentences using the cosine distance but I need to update the pre-trained model for my specific use case.

If you look at the code below, which is precisely from the Huggingface docs. I am attempting to "retrain" or update the model and I assumed that special_token_1 and special_token_2 represent "new sentences" from my "in house" data or corpus. Is this correct? In summary, I like the already pre-trained BERT model but I would like to update it or retrain it using another in house dataset. Any leads will be appreciated.

import tensorflow as tf
import tensorflow_datasets
from transformers import *

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

SPECIAL_TOKEN_1="dogs are very cute"
SPECIAL_TOKEN_2="dogs are cute but i like cats better and my
brother thinks they are more cute"

tokenizer.add_tokens([SPECIAL_TOKEN_1, SPECIAL_TOKEN_2])
model.resize_token_embeddings(len(tokenizer))
#Train our model
model.train()
model.eval()

Output:
BERT is pre-trained on 2 tasks: masked language modeling (MLM) and next sentence prediction (NSP). The most important of those two is MLM (it turns out that the next sentence prediction task is not really that helpful for the model's language understanding capabilities - RoBERTa for example is only pre-trained on MLM).
If you want to further train the model on your own dataset, you can do so by using BERTForMaskedLM in the Transformers repository. This is BERT with a language modeling head on top, which allows you to perform masked language modeling (i.e. predicting masked tokens) on your own dataset. Here's how to use it:
from transformers import BertTokenizer, BertForMaskedLM
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased', return_dict=True)

inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt")
labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]

outputs = model(**inputs, labels=labels)
loss = outputs.loss
logits = outputs.logits

You can update the weights of BertForMaskedLM using loss.backward(), which is the main way of training PyTorch models. If you don't want to do this yourself, the Transformers library also provides a Python script which allows you perform MLM really quickly on your own dataset. See here (section "RoBERTa/BERT/DistilBERT and masked language modeling"). You just need to provide a training and test file.
You don't need to add any special tokens. Examples of special tokens are [CLS] and [SEP], which are used for sequence classification and question answering tasks (among others). These are added by the tokenizer automatically. How do I know this? Because BertTokenizer inherits from PretrainedTokenizer, and if you take a look at the documentation of its __call__ method here, you can see that the add_special_tokens parameter defaults to True.


Instruction: Create a torch::Tensor from C/C++ array without using "from_blob(...)..."

Input:
Using the C++ libtorch frontend for Pytorch
I want to create a  torch::Tensor from a C++ double[] array.  Comming from a legacy C/C++ API.
I could not find a simple documentation about the subject not in docs nor in the forums.
Something like:
double array[5] = {1, 2, 3, 4, 5};   // or double *array;
auto tharray = torch::Tensor(array, 5, torch::Device(torch::kCUDA));

The only thing I found is to use torch::from_blob but then I would have to clone() and use to(device) if I wanted to use it with CUDA.
double array[] = { 1, 2, 3, 4, 5};. // or double *array;
auto options = torch::TensorOptions().dtype(torch::kFloat64);
torch::Tensor tharray = torch::from_blob(array, {5}, options);

Is there any cleaner way of doing so?

Output:
You can read more about tensor creation here: https://pytorch.org/cppdocs/notes/tensor_creation.html

I don't know of any way to create a tensor from an array without using from_blob but you can use TensorOptions to control various things about the tensor including its device.

Based on your example you could create your tensor on the GPU as follows:

double array[] = { 1, 2, 3, 4, 5};
auto options = torch::TensorOptions().dtype(torch::kFloat64).device(torch::kCUDA, 1);
torch::Tensor tharray = torch::from_blob(array, {5}, options);


Instruction: I have a pytorch image classifier training, and I want to pause training and save the weights at time of program pause. Can I do this?

Input: I'm in the middle of training a classifier that's been training for a few days now, but my problem is that I  didn't code in to save .pt checkpoints throughout training, and so I'll only end up with a weights file when the program is done with all of its epochs. Is there a way to pause training (PAUSE BREAK) and save the model's weights right now?

Output:
Unfortunately, PyTorch does not have a native API for this at the moment.
For the current job, you could use an IDE like PyDev or Pycharm to attach a debugger to the running process and set a break point somewhere in your code and extract the weights and biases.

For future jobs, you could always create checkpoints inside the epochs loop and save the learned model there. This link will help.


Instruction: In torch.distributed, how to average gradients on different GPUs correctly?

Input:
In torch.distributed, how to average gradients on different GPUs correctly?

Modified from https://github.com/seba-1511/dist_tuto.pth/blob/gh-pages/train_dist.py, the codes below can successfully make use of both GPUs (can be checked with nvidia-smi).

But one thing difficult to understand is whether the 'average_gradients' below is indeed the correct way of averaging gradients on the two models on the two GPUs. Like the codes below,  the two 'model = Net()' run with two processes are two models on two different GPUs, but and the line 'average_gradients(model)' just 'averages' gradients of the model on one GPU, not two models on the two GPUs.

The question is that is the codes below indeed a correct way for averaging gradients on the two GPU? If true, how to read, how to understand the codes? If not, what is the correct way of averaging gradients on the two models below?

import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from math import ceil
from random import Random
from torch.multiprocessing import Process
from torchvision import datasets, transforms
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"

class Partition(object):
    """ Dataset-like object, but only access a subset of it. """

    def __init__(self, data, index):
        self.data = data
        self.index = index
    def __len__(self):
        return len(self.index)

    def __getitem__(self, index):
        data_idx = self.index[index]
        return self.data[data_idx]

class DataPartitioner(object):
    """ Partitions a dataset into different chuncks. """
    def __init__(self, data, sizes=[0.7, 0.2, 0.1], seed=1234):
        self.data = data
        self.partitions = []
        rng = Random()
        rng.seed(seed)
        data_len = len(data)
        indexes = [x for x in range(0, data_len)]
        rng.shuffle(indexes)
        for frac in sizes:
            part_len = int(frac * data_len)
            self.partitions.append(indexes[0:part_len])
            indexes = indexes[part_len:]

    def use(self, partition):
        return Partition(self.data, self.partitions[partition])

class Net(nn.Module):
    """ Network architecture. """

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x)

def partition_dataset():
    """ Partitioning MNIST """
    dataset = datasets.MNIST(
        './data',
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize((0.1307, ), (0.3081, ))
        ]))
    size = dist.get_world_size()
    bsz = int(256 / float(size))
    partition_sizes = [1.0 / size for _ in range(size)]
    partition = DataPartitioner(dataset, partition_sizes)
    partition = partition.use(dist.get_rank())
    train_set = torch.utils.data.DataLoader(
        partition, batch_size=bsz, shuffle=True)
    return train_set, bsz

def average_gradients(model):
    """ Gradient averaging. """
    size = float(dist.get_world_size())
    for param in model.parameters():
        dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)
        param.grad.data /= size

def run(rank, size):
    """ Distributed Synchronous SGD Example """
    # print("107 size = ", size)
    # print("dist.get_world_size() = ", dist.get_world_size()) ## 2

    torch.manual_seed(1234)
    train_set, bsz = partition_dataset()
    device = torch.device("cuda:{}".format(rank))

    model = Net()
    model = model.to(device)
    optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

    num_batches = ceil(len(train_set.dataset) / float(bsz))
    for epoch in range(10):
        epoch_loss = 0.0
        for data, target in train_set:
            # data, target = Variable(data), Variable(target)
           # data, target = Variable(data.cuda(rank)), Variable(target.cuda(rank))
            data, target = data.to(device), target.to(device)

            optimizer.zero_grad()
            output = model(data)
            loss = F.nll_loss(output, target)
            epoch_loss += loss.item()
            loss.backward()
            average_gradients(model)
            optimizer.step()
        print('Rank ',
              dist.get_rank(), ', epoch ', epoch, ': ',
              epoch_loss / num_batches)
        # if epoch == 4:
            # from utils import module_utils
            # module_utils.save_model()

def init_processes(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_processes, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Output:
My solution is to use DistributedDataParallel instead of DataParallel like below.

The code

for param in self.model.parameters():
    torch.distributed.all_reduce(param.grad.data)

can work successfully.

class DDPOptimizer:
    def __init__(self, model, torch_optim=None, learning_rate=None):
        """
        :param parameters:
        :param torch_optim: like torch.optim.Adam(parameters, lr=learning_rate, eps=1e-9)
            or optim.SGD(model.parameters(), lr=0.01, momentum=0.5)
        :param is_ddp:
        """
        if torch_optim is None:
            torch_optim = torch.optim.Adam(model.parameters(), lr=3e-4, eps=1e-9)

        if learning_rate is not None:
            torch_optim.defaults["lr"] = learning_rate

        self.model = model
        self.optimizer = torch_optim

    def optimize(self, loss):
        self.optimizer.zero_grad()
        loss.backward()
        for param in self.model.parameters():
            torch.distributed.all_reduce(param.grad.data)

        self.optimizer.step()
    pass

def run():
    """ Distributed Synchronous SGD Example """

    module_utils.initialize_torch_distributed()
    start = time.time()

    train_set, bsz = partition_dataset()
    model = Net()

    local_rank = torch.distributed.get_rank()
    device = torch.device("cuda", local_rank)
    model = model.to(device)

    sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)
    optimizer = DDPOptimizer(model, torch_optim=sgd)

    # optimizer = NoamOptimizerDistributed(100, 1, 10, model)

    num_batches = math.ceil(len(train_set.dataset) / float(bsz))

    epoch, end_epoch = 1, 10

    while epoch <= end_epoch:
        epoch_loss = 0.0
        for data, target in train_set:
            data, target = data.to(device), target.to(device)

            output = model(data)
            loss = F.nll_loss(output, target)
            epoch_loss += loss.item()
            optimizer.optimize(loss)

        print('Rank ', dist.get_rank(), ', epoch ', epoch, ': ', epoch_loss / num_batches)
        # if epoch % 6 == 0:
        #     if local_rank == 0:
        #         module_utils.save_model(model, "a.pt")
        epoch += 1

    print("Time take to train: ", time.time() - start)


Instruction: Usage of data.to(device) with cuda GPUs

Input:
I'm trying to implement a neural network to run across 8 GPUs and i just want clarification on what exactly these commands do

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
data.to(device)

Will this automatically spread the training across the 8 GPUs ?
Thanks!

Output:
No. The code snippet will move the model and data to GPU if CUDA is available, otherwise, it will put them in CPU.

torch.device('cuda') refers to the current cuda device
torch.device('cuda:0') refer to the cuda device with index=0

To use all the 8 GPUs, you can do something like:

if torch.cuda.device_count() > 1:
    model = torch.nn.DataParallel(model)

Note:

torch.cuda.device_count() returns the number of GPUs available.
You do not need to call: data = torch.nn.DataParallel(data)

Why? Because torch.nn.DataParallel

  is a container that parallelizes the application of the given module by splitting the input across the specified devices by chunking in the batch dimension (other objects will be copied once per device). In the forward pass, the module is replicated on each device, and each replica handles a portion of the input. During the backward pass, gradients from each replica are summed into the original module.

The batch size should be larger than the number of GPUs used.


Instruction: How to resize a PyTorch tensor?

Input:
I have a PyTorch tensor of size (5, 1, 44, 44) (batch, channel, height, width), and I want to 'resize' it to (5, 1, 224, 224)
How can I do that? What functions should I use?

Output:
It seems like you are looking for interpolate (a function in nn.functional):
import torch.nn.functional as nnf

x = torch.rand(5, 1, 44, 44)
out = nnf.interpolate(x, size=(224, 224), mode='bicubic', align_corners=False)

If you really care about the accuracy of the interpolation, you should have a look at ResizeRight: a pytorch/numpy package that accurately deals with all sorts of "edge cases" when resizing images. This can have an effect when directly merging features of different scales: inaccurate interpolation may result in misalignments.


Instruction: How to optimize a simulation metric with deep learning without target values?

Input:
I am trying to use an RNN model that outputs bus routes and its input is the demand matrix. The bus routes are then used in a simulation which spits out a metric of how the routes performed. The question is, since there is no target value of bus routes, how do I back propagate the simulation result?

To explain the question with simple python code:

"""
The model is an RNN that takes 400,24,24 matrix as input

dimension 0 represents time, dimension 1 represents departure bus stop and dimension 2 represents the arrival bus stop. Each value is a count of the number of passengers who departed at a bus stop with an arrival bus stop in mind in a specific time

output is 64,24 matrix which will be reshaped to 8,8,24
dimension 0 is the sequence index, dimension 1 is the index of bus (there are 8 buses), dimension 2 is the softmaxed classifier dimension of 24 different bus stops. From the output, 8 bus stops are picked per bus with a sequence

These sequences are then used for path generations of buses and they are evaluated from a simulation
"""

model.train()
optimizer.zero_grad()
out = model(demand)#out is 64,24 demand is 400,24,24
demand, performance = simulation(out)#assume performance as float
#here the out has grad_fn but the performance does not
loss = SOME_NUMBER - performance
loss = torch.FloatTensor(loss)
#here I need to back propagate and it is the confusing part

#simply doing loss.backward() does nothing because no grad_fn

#out.backward() requires 64,24 gradients computed somehow from 1 #metric, causes complete divergence within few steps

optimizer.step()

Output:
How does the model output represent the bus routes? Maybe you could try a reinforced learning approach. Take a look at Deep-Q Learning, It basically takes and input vector (the state of the system) and outputs an action (usually represented by an index in your output layer), then it computes the reward of that action and uses it to train the model (without the need of target values).

Here are some resources that might help you get started:

https://towardsdatascience.com/double-deep-q-networks-905dd8325412

https://arxiv.org/pdf/1802.09477.pdf

https://arxiv.org/pdf/1509.06461.pdf

Hope this was useful.

UPDATE

There is a second option, you could define a custom loss function. Generally these functions only take two arguments, the predicted_y and the target_y, in your case, there is no target_y, so you could pass a dummy target_y and not use it inside the function (I assume that you could call your simulation process inside that function, and return the metric as the "loss"). Here are examples in PyTorch and Keras.

Keras: Make a custom loss function in keras

PyTorch:PyTorch custom loss function


Instruction: What is Adaptive average pooling and How does it work?

Input:
I recently came across a method in Pytorch when I try to implement AlexNet.
I don't understand how it works. Please explain the idea behind it with some examples. And how it is different from Maxpooling or Average poling in terms of Neural Network functionality

  nn.AdaptiveAvgPool2d((6, 6))

Output:
In average-pooling or max-pooling, you essentially set the stride and kernel-size by your own, setting them as hyper-parameters. You will have to re-configure them if you happen to change your input size.

In Adaptive Pooling on the other hand, we specify the output size instead. And the stride and kernel-size are automatically selected to adapt to the needs. The following equations are used to calculate the value in the source code.

Stride = (input_size//output_size)
Kernel size = input_size - (output_size-1)*stride
Padding = 0


Instruction: Pinning memory is actually slower in PyTorch?

Input:
I'm wondering why would pinning memory in PyTorch make things even slower. By reading the code of torch.utils.data.dataloader, I found the pin_memory=True option of DataLoader simply calls .pin_memory() on each batch before returning them. The returned tensor is still on CPU, and I have to call .cuda(non_blocking=True) manually after this. Therefore, the whole process would be

for x in some_iter:
    yield x.pin_memory().cuda(non_blocking=True)

I compared the performance of this with

for x in some_iter:
    yield x.cuda()

Here is the actual code

a = torch.rand(1024, 655360)

%%time
for i in a:
    i.pin_memory().cuda(non_blocking=True)
# CPU times: user 1.35 s, sys: 55.8 ms, total: 1.41 s
# Wall time: 396 ms

%%time
for i in a:
    i.pin_memory().cuda()
# CPU times: user 1.6 s, sys: 12.2 ms, total: 1.62 s
# Wall time: 404 ms

%%time
for i in a:
    i.cuda(non_blocking=True)
# CPU times: user 855 ms, sys: 3.87 ms, total: 859 ms
# Wall time: 274 ms

%%time
for i in a:
    i.cuda()
# CPU times: user 314 ms, sys: 12 µs, total: 314 ms
# Wall time: 313 ms

As a result, not pinning memory both uses less CPU time, and is faster in terms of actual time. Shouldn't pinning memory make data transfer asynchronous and therefore be faster? If that's not the case, why would we do pin memory?

PS. I thought about the possibility of pinning a whole TensorDataset in advance (rather than pinning batches each time). But this cannot pin a tensor that is bigger than GPU memory:

a = np.memmap('../dat/R/train.3,31,31B', '3,31,31B', 'r')
a.nbytes // 2**30
## 68

torch.from_numpy(a).pin_memory()
## ---------------------------------------------------------------------------
## RuntimeError                              Traceback (most recent call last)
## <ipython-input-36-d6f2d74da8e7> in <module>
## ----> 1 torch.from_numpy(a).pin_memory()
##
## RuntimeError: cuda runtime error (2) : out of memory at /tmp/pip-req-build-58y_cjjl/aten/src/THC/THCCachingHostAllocator.cpp:296

And if I do want to pin a small tensor, why don't I directly move the whole tensor into GPU memory in advance?

Output:
TL:DR
Your code is slower, because you allocate a new block of pinned memory each time you call the generator. Allocating new memory each time requires synchronization each time making it much slower than non-pinned memory. Likely, you are measuring this overhead.
Your code example in the edit fails in the THCCachingHostAllocator.cpp. It's not the GPU running out of memory, but your host denying you to allocate 68GB of pinned physical memory.

Pinning memory is actually slower in PyTorch?

Creating or releasing pinned memory (cudaHostAlloc()/cudaFreeHost() via the CUDA Runtime) is much slower than malloc/free because it involves synchronization between the devices (GPU and host). Likely, what you are measuring is - to a large extent - this overhead, as you are incrementally allocating pinned memory.

Shouldn't pinning memory make data transfer asynchronous and therefore be faster? If that's not the case, why would we do pin memory?

It can, but not if you halt/join to synchronize before each transfer in order to allocate the memory.
What pinning memory ultimately does is that it prevents the memory block from being swapped out by the OS; it is guaranteed to remain in RAM. This guarantee enables the GPU's DMA to operate on that block without going through the CPU (which has to check, among other things, if the data needs to be swapped back in). Thus, the CPU is free to do other stuff in the meantime.
It is not a perfect analogy, but you could think about pinned memory as shared memory between the GPU and the host. Both parties can operate on it without informing the other party; a bit like multiple threads in a process. This can be much faster if you implement non-blocking code. However, it can also be much slower if parties end up joining all the time.
Contrast this to the non-pinned approach, where the CPU loads the data from RAM (swapped in if necessary) and then sends it to the GPU. Not only is it slower (needs to go through the northbridge twice), but it also keeps the thread (and hence one CPU core) busy. Python also has the infamous GIL, so it could be that your entire application is waiting for that synchronous I/O.
If you want to use pinned memory to shuffle batches of data into the GPU, then one way to do it is to use pinned memory as a (circular) buffer. The CPU can load the data from disk, apply preprocessing, and place the batch into the buffer. The GPU can then fetch batches from the buffer in its own time and do the inference. If the implementation is done well, then the GPU will not idle more than necessary, and there is no more need for synchronization between the host and the GPU.

And if I do want to pin a small tensor, why don't I directly move the whole tensor into GPU memory in advance?

If you don't need to access the tensor from the CPU and it fits onto the GPU, then there is indeed no need to put it into pinned memory.
In your example, you are opening a memory-mapped numpy array memmap, and then ask to transfer it to pinned memory. A memory-mapped file works very similar to paged memory in that data that doesn't fit the RAM anymore is flushed to disk, and loaded back in when it is accessed again.
This "swapping" can not happen for pinned memory, because we need to guarantee that the entire block resides in RAM at all dimes. Hence, we need to first load the entire array into host memory - a contiguous block of 68 GB -, likely creating a copy of the array in the process to not destroy the memmap object, and then we need to pin that memory block, telling the host to forfeit 68GB of managed physical memory to our application. Either of these two steps can be denied by the OS and raise an OutOfMemory error.
This is pretty much what you are seeing, as you fail in the THCCachingHostAllocator.cpp.


Instruction: pytorch math with exponents less than 1 return nan 's

Input:
torch.pow() returns nan when it's given the exponent argument between ranges -1 and 1

a = torch.randn(1,3)
a

>> tensor([[-1.7871, -0.5375, -0.1164]])

torch.pow(a, 2)      #or a**2

>> tensor([[3.1938, 0.2889, 0.0136]])

torch.pow(a,0.5)     #or a**0.5

>> tensor([[nan, nan, nan]])

expect result:

tensor([[-1.3368, -0.7331, -0.3412]])

Edit: turns out that ** works the same way as well. a**2 does the same thing with the tensor as torch.pow(a,2).  a**0.5 returns nan's like torch.pow does.

Output:
The issue is that square root of a negative number is a non-real complex number.

If you want to keep the sign and take the square root of its absolute value, the following code does the trick

torch.sign(a) * torch.pow(torch.abs(a), 0.5)


Instruction: Face alignment in pytorch

Input:
I am trying to do face alignment on 300W dataset. I am using ResNet50 and L1 loss for training. My code looks like this.

batch_size = 10
image_size = 128

net = torchvision.models.resnet50(pretrained=True)
num_ftrs = net.fc.in_features
net.fc = nn.Linear(num_ftrs, 136) # 136 because 68 points with 2 dim. so 136= 68*2

def train():
    device = torch.device("cuda:0" if torch.cuda.is_available() else
           "cpu")

    optimiser = optim.Adam(net.parameters(), lr=0.001,
            weight_decay=0.0005)

    criterion = L1Loss(reduction='sum')

    for epoch in range(int(0), 200000):
        for batch, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimiser.zero_grad()

            outputs = net(inputs).reshape(-1, 68, 2)

            loss = criterion(outputs, labels)
            loss.backward()
            optimiser.step()
            running_loss += loss.item()

            sys.stdout.write(
            '\rTrain Epoch: {} Batch {} avg_Loss_per_batch: {:.2f}
            '.format(epoch, batch, running_loss/(batch+1)))
            sys.stdout.flush()

The trainloader is with images and points. The ground-truths are shaped as (batch, 68, 2). We have 68 points on the face on 2 dimensional space.

The papers suggests that the ResNet50 should get an error of 10 (metric: pixel) for a 256*256 image with L1 loss. I am getting error around 500-800 on validation set even after 5000 epoch.

I am training images with 256*256 resolution with ground truth of 68 points such as ((x1,y1),(x2,y2)....(x68,y68)) and I have trained over 5000 epoch with many learning rates. My validation code looks like this,

def validater(load_weights=False):
    device = torch.device("cuda:0" if torch.cuda.is_available() else
          "cpu")
    net.eval()
    net.to(device)

    with torch.no_grad():
        for batch, data in enumerate(testloader, 0):
            inputs, labels = data
            inputs, labels  = inputs.to(device), labels.to(device)

            outputs = net(inputs).reshape(-1, 68, 2)

            loss = criterion(outputs, labels)

            loss2 = np.linalg.norm(labels.to('cpu') - outputs.to('cpu'))

            sys.stdout.write('\rTest Epoch: {} Batch {} total_L1_Loss:
                {:.2f} avg_L1_Loss_per_img: {:.2f} total_norm_loss:
                 {:.2f}'.format(
                0, batch, avg_loss, avg_loss/batch/batch_size,
                avg_loss2))
            sys.stdout.flush()

    print()

What is wrong with my code ?

PS: I normalise the imgs with the following code

    img = cv2.normalize(img, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)

After 4000 epoch I get outputs like this where yellow dots are ground truth and blue ones are predicted

Output:
From your output image you can tell the error is smaller on the top left landmarks and grows larger towards the lower right part of the face.
The landmarks you are trying to predict are (x, y) coordinates relative to the top left corner of the image. As you can see, your model's prediction error grows proportionally to the norm of each coordinate. This is not an uncommon phenomenon: when you model predicts a landmark close to the origin (e.g. left eye) it makes "small" predictions as the norm of this landmark is also small, the learned weights are small and therefore the errors are also small. On the other side, when predicting landmarks far from the origin (right part of mouth) the model need to make "large" predictions as the norm of these landmarks is large. Consequently, the trained weights are larger resulting with cruder errors.

To mitigate this, you should pre-process your data (train and test) and normalize the coordinates of the landmarks to be 1:
1. relative to the center of the image
2. relative to image size
That is, instead of (x, y) coordinates in the range of [0, width]x[0, height] you should have the landmarks in the range [-1, 1]x[-1, 1].
After prediction, to get the original coordinates you simply need to shift them and scale them by image size.

1 I am assuming here all faces in the training set are roughly the same size and located roughly in the center of the images. If your settings are "in the wild" where faces can be of any size at any place in the image I'm afraid this will not work.


Instruction: Pytorch - Batch Normalizaiton simple question

Input:
I implemented a model with batch normalization:

class FFNet(torch.nn.Module):
    def __init__(self, D_in, H_1, H_2, D_out):

        super(FFNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H_1)
        self.linear2 = torch.nn.Linear(H_1, H_2)
        self.bn2 = torch.nn.BatchNorm1d(H_2)
        self.linear4 = torch.nn.Linear(H_2, D_out)

    def forward(self, x):

        h_relu_1=F.relu(self.linear1(x))
        h_relu_2=F.relu(self.bn2(self.linear2(h_relu_1)))
        y_pred=self.linear4(h_relu_2)
        return y_pred

Also, I wrote the training loop:

for epoch in range(epoches):

    running_loss = 0.0
    cnt = 0
    for i, data in enumerate(train_data, 0):
        local_X, local_y = data
        y_pred = model.forward(local_X)
        loss = criterion(y_pred, local_y)
        optimizer.zero_grad()
        #loss = criterion(y_pred, Y_local_output)
        loss.backward() # back props
        optimizer.step()
        running_loss = running_loss + loss.item()
        cnt+=1

    Validation_loss = 0.0
    cnt2 = 0
    # Validation
    for i, data in enumerate(validation_data, 0):
        Val_X, Val_Y = data

        y_pred = model.forward(Val_X)

        loss=criterion(y_pred, Val_Y)
        Validation_loss = Validation_loss + loss.item()
        cnt2+=1

I have two questions:
1. Is there no need to use model.train() in this code?
2. How to evaluate this model using eval? I have one data sample whose size is (1xD_in), and batch size is more than 1. When I use the below code, there is an error:

    test_single = torch.tensor([aa, ab, ac, ad, ae, af, ag])
    test_single = test_single.unsqueeze(0)

    model.eval()
    [bb,cc] = model.forward(test_single)

The error is 'not enough values to unpack (expected 2, got 1)'

Output:
If you have batch normalization, then you do need to use model.train() and model.eval() while training and evaluating respectively.

The 2nd part (the unsqueezing code) is not wrong. However, there is only one output of your model (see the return statement of your model's forward function), which causes the error i.e. you try to unpack 2 values whereas there is only one. So, you can't do

[bb,cc] = model.forward(test_single)

You have to do

out = model.forward(test_single)

I tried this and it works.


Instruction: How to parallelize model prediction from a pytorch model?

Input: I have a PyTorch instance segmentation model that predicts the masks of images one by one. Is there any way to parallelize this task so that it can be completed faster. I have tried to use multiprocessing pool.apply_async() to call the method that performs prediction passing the required arguments, but it throws me segmentation fault. Any help is much appreciated. Thanks.

Output:
In general, you should be able to use torch.stack to stack multiple images together into a batch and then feed that to your model. I can't say for certain without seeing your model, though. (ie. if your model was built to explicitly handle one image at a time, this won't work)

model = ... # Load your model

input1 = torch.randn(3, 32, 32)  # An RGB input image
input2 = torch.randn(3, 32, 32)  # An RGB input image

model_input = torch.stack((input1, input2))
print(model_input.shape)         # (2, 3, 32, 32)

result = model(model_input)
print(result.shape)              # (2, 1, 32, 32) 2 output masks, one for each image

If you've trained the model yourself this will look familiar as it's essentially how we feed batches of images to the network during training.

You can stack more than two together, it will typically be limited by the amount of GPU memory you have available to you.


Instruction: How can you specify docker flags when a step is ran on Google Cloud Build?

Input:
I need to be able to specify --ipc=host to the docker command that runs a step. How do I do that?

A little more context.... When running the tests in one of my Google Cloud Build steps I am running into this error:

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).

This error is common when using pytorch, like I am, in docker but it is easily addressed with the flag --ipc=host. I don't know where to put this flag in my step's settings in the cloudbuild.yml.

Right now my step looks like this:

- id: 'run-tests'
  name: 'gcr.io/project/app:test-$BRANCH_NAME'
  entrypoint: 'bash'
  args:
    - '-c'
    - './bin/run-tests.sh'

I have tried adding the "ipc=host" to the args but those go to bash and not the docker command running the image.

Output:
I am going to try this option out myself, but I think you can build a container with ipc=host set. There is no option to do so during a docker build, but you can run a container with ipc=host, then create a new image using docker commit.

The basic premise is:

docker build
docker run --ipc=host
docker commit
docker push

More details and discussion here: https://github.com/moby/moby/issues/24743

5/4 Edit

While this works, the host may have the default shm setting, so you won't get a larger size. Instead, you should probably use docker run --shm-size=256m or similar to give your docker container more shared memory. This requires you to use the docker cloud builder image and configure your own docker run command.


Instruction: How does loss.backward() relate to the appropriate parameters of the model?

Input:
I'm new in PyTorch and I'm having trouble understanding how loss knows to compute the gradients through loss.backward()?

Sure, I understand that the parameters need to have requires_grad=True and I understand that it sets x.grad to the appropriate gradient only for the optimizer later to perform the gradient update.

The optimizer is linked to the model parameters when it's instantiated, but the loss is never linked to the model.

I've been going through this thread, but I don't think anyone answered it clearly and the person that started the thread seems to have the same issue as I do.

What happens when I have two different networks with two different loss functions and two different optimizers. I will easily link the optimizers to each of the networks, but how will the loss functions know how to compute the gradients for each of their appropriate network if I never link them together?

Output: Loss is itself a tensor which is derived from the parameters of the network. A graph is implicitly constructed where each new tensor, including loss, points back to the tensors which were involved with it's construction. When you apply loss.backward() pytorch follows the graph backwards and populates the .grad member of each tensor with the partial dervative of loss with respect to that tensor using the chain rule (i.e. backpropagation)


Instruction: Is there a similar function in tensorflow like load_state_dict() in Pytorch?

Input:
Like it has been described, I am wondering is there a similar function in tensorflow for load_state_dict() like the one does in Pytorch. To demonstrate a scenario, please refer to the code following:

# Suppose we have two correctly initialized neural networks: net2 and net1
# Using Pytorch
net2.load_state_dict(net1.state_dict())

Does anyone have any idea?

Output:
Below code may help in achieveing the same in tensorflow:

Save the model

w1 = tf.Variable(tf.truncated_normal(shape=[10]), name='w1')
w2 = tf.Variable(tf.truncated_normal(shape=[20]), name='w2')
tf.add_to_collection('vars', w1)
tf.add_to_collection('vars', w2)
saver = tf.train.Saver()
sess = tf.Session()
sess.run(tf.global_variables_initializer())
saver.save(sess, 'my-model')
# `save` method will call `export_meta_graph` implicitly.
# you will get saved graph files:my-model.meta

To Restore the model

sess = tf.Session()
new_saver = tf.train.import_meta_graph('my-model.meta')
new_saver.restore(sess, tf.train.latest_checkpoint('./'))
all_vars = tf.get_collection('vars')
for v in all_vars:
    v_ = sess.run(v)
    print(v_)


Instruction: Calculate binary entropy loss using a function in pytorch

Input:
I have a problem about calculating binary cross entropy. The way I know that works out in pytorch is:

import torch
import torch.nn as nn
import torch.nn.functional as F
def lossfunc():
    return F.binary_cross_entropy

criterion = lossFunc()
input = torch.randn((3, 2), requires_grad=True)
target = torch.rand((3, 2), requires_grad=False)
loss = criterion(torch.sigmoid(input),target)

But how to complete the lossfunc() in such way, because I don't know how to pass the arguments to the function:

#the function that add sigmoid to input and calculate the binary cross entropy loss
def lossfunc():
   return

criterion = lossFunc()
input = torch.randn((3, 2), requires_grad=True)
target = torch.rand((3, 2), requires_grad=False)
loss = criterion(input,target)

Output:
I think you're confusing the nn api with the functional F api. In functional api, loss function F.binary_cross_entropy can be used as a function directly.

In nn api, you need to create an object of the loss class such as criterion = nn.BCELoss()

Thus, you can simply do:

def lossFunc(input, target):
   return F.binary_cross_entropy(torch.sigmoid(input),target)

input = torch.randn((3, 2), requires_grad=True)
target = torch.rand((3, 2), requires_grad=False)
loss = lossFunc(input,target)

Also, PyTorch provides nn.nn.BCEWithLogitsLoss() and F.binary_cross_entropy_with_logits() that combines both sigmoid and binary cross-entropy.


Instruction: Explanation of build_vocab in torch and it's association with pre-trained embeddings

Input: Can some explain me what is build_vocab in torch, it is not clear from online documentation? Why do we need it and it's relation to pre-trained embeddings?

Output:
I think you confuse pytorch and torchtext here. In torchtext (a package that provides processing utilities and popular datasets for natural language) you can run build_vocab of a Field to iterate over your dataset in order to build up the vocabulary.

Take also a look here:
https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Field.build_vocab


Instruction: How can I fix this pytorch error on Windows? (ModuleNotFoundError: No module named 'torch')

Input:
Edit: You might want to skip to the end of the question first, I've followed some advice in comments / answers and the current error is different from the original (appears to be related to numpy possibly).

This error ModuleNotFoundError: No module named 'torch' shows up in tons of threads, I've been trying solutions all day. I'll go through my troubleshooting steps one by one, using the solutions suggested in threads.

System info:
Windows 10

First thing I did was follow the instructions on Pytorch, installed Anaconda and did this using the correct settings for my machine (Note: I tried Python v3.7 before trying v3.8 in these screenshots, none of the solutions worked with that either):

As you can see, that should be good to go, according to the instructions.

So I go into the python terminal and try to import pytorch, like so:

ModuleNotFoundError: No module named 'torch' Great, so what now? Well I paste the error into Google and begin my 4 hour wild goose chase.

First result, stack overflow answer: No module named "Torch"
 Let's try the selected answer, it requires some version-related syntax so lets check my python version:

Alright so as directed by the answer:

  Try to install PyTorch using pip:

  First create a conda environment using:

conda create -n env_pytorch python=3.6

Ok:

  Activate the environment using:

source activate env_pytorch

That doesnt work, but if we activate using the instructions given by the prompt, we can do so:

  Now install PyTorch using pip:

pip install torchvision --user ( this will install both torch and torchvision)

Hmmm.. well that went up in flames, so the following...

  Now go to python shell and import using the command:

import torch
import torchvision

...doesn't do anything new, same error as before.

Well, to the next thread, on PyTorch GitHub: https://github.com/pytorch/pytorch/issues/4827

They're trying to use Jupyter, so I tried this, is was another long process like the above that went up in flames, and I really dont want to need to use Jupyter anyway, so we'll skip this one.

Another Pytorch GitHub thread: https://github.com/pytorch/pytorch/issues/12004

  @edtky Could you please give me the output of the following commands
  in CMD?

  where conda.exe
  where pip.exe
  where python.exe

Sure I'll give it a shot:

  @edtky Looks like you have two Python environments. Please try
  importing torch in Anaconda Prompt.

Oh well, I already did that. No bueno.

Another thread: https://discuss.pytorch.org/t/modulenotfounderror-no-module-named-torch/7309 suggests:

  In that case you’ve probably forgotten to activate the environment
  where pytorch is installed. It can also be the library missing in your
  PYTHONPATH variable.

Well I did activate the environment as shown above, but I dont know anything about a PYTHONPTH variable, seems like the PyTorch setup guide wouldve mentioned if I needed to manually do that, I have no clue how to do it and you aren't explaining, so lets look for other answers.

Someone made a whole article to give us this little gym of advice: https://medium.com/@valeryyakovlev/anaconda-no-module-named-torch-ead10946de66

  Another beginner error I encountered when started to use pytorch in
  anaconda environment
      import torch ModuleNotFoundError: No module named ‘torch’ the proper way to install pytorch to anaconda is following
      conda install -c pytorch pytorch It’s not enough to simply run “conda install pytorch” — the package won’t be found. So first
  activate your conda profile with “source activate {your_profile}” and
  then run the command conda install -c...

Ok thats new info, let's try that command again now that our env is activated:

Ok that's a lot of green, let's try now...

Well we can't win 'em all, so lets go onto the next thread: https://forums.fast.ai/t/modulenotfounderror-no-module-named-torch-windows-10/12438/2

  I had also faced the similar problem , I just installed torch and torchvision using pip and it worked …

Ok! Let's try:

Oh well, another solution up in flames..

  I ran into a similar issue with Windows 10. In the end I could only get torch installed with Miniconda.

Alrighty, lets try it!

Alright, cool, moment of truth:

Awesome! You just read through 25 minutes of me re-producing all my attempts to solve this problem, and it doesnt even include the hour I spend down a rabbit hole trying to use Jupyter, which failed equally as miserably. I think it's time to post the question to StackOverflow!

Edit 1:

An answer points out that one of my logs was an error python 3.8 isn't compatible with pytorch, good point I'll fix that. After unintalling 3.8 and installing 3.7:

And no luck! Remember I actually mentioned in my first paragraph that while I was trying 3.8 in these screenshots, the first time around I did all of this with 3.7

Edit 2:

I forgot to install after activating the environment in the previous edit. Once I fixed that, there's a new error:

Output:
Pytorch requires 3.5 <= python < 3.8. Setup an environment with:

conda create -n pytorch python=3.7
conda activate pytorch
conda install pytorch

You should also make sure that you launch the installed python interpreter from this environment (YourAnacondaInstallDirectory\envs\pytorch\python.exe) from the activated conda environment! The later is important because conda will export certain environment variables (have a look at this for a related issue caused by missing envionment variables).


Instruction: How to set python environment variables in VS Code?

Input:
I know how to add arguments for the Python script I want to run. For example, if test.py is my script file and it has one argument like '--batch_size' then I can edit launch.json in VS Code and set "args": ["--batch_size", "32"]

But I don't know how to add environmental arguments for Python itself. For example, Python has -m environmental variable, which runs library module as a script. If I want to run python -m torch.distributed.launch test.py --batch_size 32, what should I edit in VS Code to run the debugger?

UPDATE:
Here is my launch.json

Output:
-m is not environmental variable. It's just a regular argument.

To run python -m torch.distributed.launch test.py --batch_size 32 use args  "args": ["-m", "torch.distributes.launch" ,"--batch_size", "32"] Also you need to run python itself instead of running script to pass these args to it ("program": "python3").

To set actual environment variables use "env": {"ENV_VAR_NAME1": "value of ENV_VAR_NAME1", "ENVVAR2": "Value for ENVVAR2"}

Here you can find more information about launch.json configuration


Instruction: Pytorch get "RuntimeError: CUDA error: device-side assert triggered"

Input:
Snippet from my code :

max = torch.tensor([3])
if USE_CUDA: max = max.cuda()
max_embedding = self.max_embedding(max) # dim of max_embedding: 1*5

item_dict = {}
for item in item_list:
    item = torch.tensor(item)
    if USE_CUDA: item = item.cuda()
    item_embedding = self.item_embedding(item) # dim of item_embedding: 1*20

embedded = torch.cat((max_embedding, item_embedding), 1)

But I get error of "RuntimeError: CUDA error: device-side assert triggered".
The output by adding CUDA_LAUNCH_BLOCKING=1:

/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [3,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/THC/THCTensorIndex.cu:308: void indexSelectSmallIndex(TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim =2, SrcDim = 2, IdxDim = -2: block: [0,0,0], thread: [4,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "mytest.py", line 33, in <module>
     if USE_CUDA: item = item.cuda()
RuntimeError: CUDA error: device-side assert triggered

How to fix it?

Output: This is a typical case of index-out-of-bounds error manifested itself in the context of embeddings. Check this link for solution to a similar problem.


Instruction: Saving Numpy Array as Lab Image via PIL

Input:
Hello everyone!

I am writing a function which gets two Pytorch-Tensors as input and merges parts of both tensors to a new array which will be converted to a Lab-image afterwards.

net_input is a tensor with 3 channels (L, a, b) and output is a tensor with 2 channels (a, b).
Now, I want to take the L-channel from net_input and the a-channel and the b-channel from output.

The first tensor, net_input, has values from -1 to 1. The second tensor, output, has values from 0 - 1.

Thus, both of them must be mapped to values from -128 to 127. I do this, using interp1d

Apparently the saved image has some unwanted patterns and is incorrect.

original:  . grayscale:

from PIL import Image as pil
import numpy as np
from PIL import Image
from scipy.interpolate import interp1d
from datetime import datetime

def output_to_image(net_input, output, index):
    # net_input.size()  -->     torch.Size([1, 3, 225, 225]) # l, a, b channel
    # output.size()     -->     torch.Size([1, 2, 225, 225]) # a, b channel

    # A - B - Channel
    map_a_b = interp1d([0.0, 1.0], [-128, 127])

    # L-Channel
    map_l = interp1d([-1.0, 1.0], [-128, 127])

    height = output.size()[2]
    width = output.size()[3]

    l = net_input.detach().numpy()[0][0]    # l channel
    a = output.detach().numpy()[0][0]   # a channel
    b = output.detach().numpy()[0][1]   # b channel

    # pdb.set_trace()

    img_arr = np.empty([height, width, 3])  # lab img array
    l_arr = np.empty([height, width])   # grayscale img array

    for h in range(0, height-1):
        for w in range(0, width-1):
            img_arr[h][w] = [map_l(l[h][w]), map_a_b(a[h][w]), map_a_b(b[h][w])]
            l_arr[h][w] = map_l(l[h][w])

    now = datetime.now()

    img = Image.fromarray(img_arr, "LAB")
    img.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-lab-img{index}.tiff")

    gray = Image.fromarray(l_arr, "L")
    gray.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-gray-img{index}.jpg")

This is how L, a and b look like:

(Pdb) >? print(l)
[[-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 ...
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]]
(Pdb) >? print(a)
[[0.51877767 0.5208904  0.5310791  ... 0.5340722  0.51334995 0.522657  ]
 [0.5142181  0.50250506 0.5197009  ... 0.51169556 0.5332947  0.5155644 ]
 [0.53288984 0.51795006 0.5224927  ... 0.51704454 0.53655064 0.50311136]
 ...
 [0.5270468  0.5071506  0.52318716 ... 0.5217321  0.53424454 0.5011423 ]
 [0.5216123  0.53247094 0.5254119  ... 0.53089285 0.5259453  0.532716  ]
 [0.53135234 0.5184961  0.51334924 ... 0.5131047  0.51930845 0.51474   ]]
(Pdb) >? print(b)
[[0.4812223  0.47910962 0.46892092 ... 0.46592775 0.48665005 0.47734302]
 [0.48578197 0.49749494 0.4802992  ... 0.4883045  0.46670532 0.48443565]
 [0.46711013 0.48204994 0.47750723 ... 0.4829555  0.46344927 0.4968886 ]
 ...
 [0.47295317 0.49284932 0.47681284 ... 0.47826794 0.46575552 0.49885774]
 [0.47838774 0.46752912 0.47458804 ... 0.46910718 0.47405463 0.46728405]
 [0.46864766 0.48150396 0.48665074 ... 0.48689532 0.48069155 0.48526   ]]
(Pdb)

Edit:

After adjusting the interpolation range to 0..255, like Mark suggested, the output looks a little bit different:

import numpy as np
from PIL import Image
from datetime import datetime

def output_to_image(net_input, output, index):
    # net_input.size()  -->     torch.Size([1, 3, 225, 225])
    # output.size()     -->     torch.Size([1, 3, 225, 225])

    height = output.size()[2]
    width = output.size()[3]

    l = net_input.detach().numpy()[0][0]    # l channel
    a = output.detach().numpy()[0][0]   # a channel
    b = output.detach().numpy()[0][1]   # b channel

    #pdb.set_trace()

    img_arr = np.empty([height, width, 3])  # lab img array
    l_arr = np.empty([height, width])   # grayscale img array

    for h in range(0, height-1):
        for w in range(0, width-1):
            img_arr[h][w] = [map_l(l[h][w]), map_a_b(a[h][w]), map_a_b(b[h][w])]
            l_arr[h][w] = map_l(l[h][w])

    now = datetime.now()

    img = Image.fromarray(img_arr, "LAB")
    img.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-lab-img{index}.tiff")

    gray = Image.fromarray(l_arr, "L")
    gray.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-gray-img{index}.jpg")

def map_l(x):
    return (x+1)*127

def map_a_b(x):
    return x*127

Output:
If you do this with PIL:

im = Image.new('LAB',(80,80),color=(255,0,0)).save('a.tif')

you'll get a cyan colour the same as if you use this colour converter and put in

L/a/b = 100/-128/-128

If you do this:

im = Image.new('LAB',(80,80),color=(255,255,0)).save('a.tif')

you'll find that corresponds to

L/a/b=100/128/-128

on the online converter.

And if you do this:

im = Image.new('LAB',(80,80),color=(255,255,255)).save('a.tif')

you'll find that corresponds to

L/a/b=100/128/128

on the online converter.

So, I think you are targeting the wrong limits if using PIL. Your ranges for L, a and b should all be 0..255.

As regards scaling, I wouldn't use interp1d() at all. If I need to scale the range -1..1 to the range 0..255, I would add 1 which brings me to the range 0..2 and then multiply by 255/2.


Instruction: Why does pre-trained ResNet18 have a higher validation accuracy than training?

Input: For PyTorch's tutorial on performing transfer learning for computer vision (https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html), we can see that there is a higher validation accuracy than training accuracy. Applying the same steps to my own dataset, I see similar results. Why is this the case? Does it have something to do with ResNet 18's architecture?

Output:
Assuming there aren't bugs in your code and the train and validation data are in the same domain, then there are a couple reasons why this may occur.

Training loss/acc is computed as the average across an entire training epoch. The network begins the epoch with one set of weights and ends the epoch with a different (hopefully better!) set of weights. During validation you're evaluating everything using only the most recent weights. This means that the comparison between validation and train accuracy is misleading since training accuracy/loss was computed with samples from potentially much worse states of your model. This is usually most noticeable at the start of training or right after the learning rate is adjusted since the network often starts the epoch in a much worse state than it ends. It's also often noticeable when the training data is relatively small (as is the case in your example).
Another difference is the data augmentations used during training that aren't used during validation. During training you randomly crop and flip the training images. While these random augmentations are useful for increasing the ability of your network to generalize they aren't performed during validation because they would diminish performance.

If you were really motivated and didn't mind spending the extra computational power you could get a more meaningful comparison by running the training data back through your network at the end of each epoch using the same data transforms used for validation.


Instruction: Using MIT Indoor scene database in CNN

Input:
I'm an engineering student and kind of a noob to programming. I'm taking an AI course, currently trying to do my final project.

I have to create a CNN net, I have to use de MIT Indoor scene database (it can be found here: http://web.mit.edu/torralba/www/indoor.html). I don't have a problem doing the CNN since I've done a few before in the semester using CIFAR10, but I'm having trouble with this one since I don't know who to use this set of images.
I think I need to create a dataset of my own, I've tried with PyTorch using https://pytorch.org/tutorials/beginner/data_loading_tutorial.html, but I get confused because I don't have a .csv with features, I have a lot of .xml files with several features for each picture. Also, I don't have a file just saying "bedroom, bar, etc" as I've seen in other tutorials.

I would rather use PyTorch since I can use the "train_test_split" function, but if anyone could help me to understand how to make those 15620 my input to the net, I would really appreciate it.

Output:
you can generate your own csv files, although, you might not need it.
There is a good tutorial on pytorch website https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#load-data, which is very similar or easily applicable to your case.

MIT Indoor dataset has the images one folder per class, and the txt files mentioned on the website are the train / test splits.

So, if you create the following folder structure:

train
|- class 1
|- class 2
...
|- class n

and the same for val / test, it should be straight-foward to use (adapt) the datasets.ImageFolder example for your case.


Instruction: How to compose several matrices into a big matrix diagonally in Pytorch

Input:
I have several matrices, let's say [M1,M2,M3,M4]. Each matrix has a different shape. How do I compose these matrices into one big matrix diagonally like:
[[M1, 0, 0, 0]
 [0, M2, 0, 0]
 [0, 0, M2, 0]
 [0, 0, 0, M2]]

Example:
M1 = [[1,2],[2,1]]
M2 = [[1,2]]
M3 = [[3]]
M4 = [[3,4,5],[4,5,6]]

To compose this big matrix:
[[1, 2, 0, 0, 0, 0, 0]
 [2, 1, 0, 0, 0, 0, 0]
 [0, 0, 1, 2, 0, 0, 0]
 [0, 0, 0, 0, 3, 4, 5]
 [0, 0, 0, 0, 4, 5, 6]]

Output:
Use PyTorch's torch.block_diag():
>>> torch.block_diag(M1,M2,M3,M4)

tensor([[1, 2, 0, 0, 0, 0, 0, 0],
        [2, 1, 0, 0, 0, 0, 0, 0],
        [0, 0, 1, 2, 0, 0, 0, 0],
        [0, 0, 0, 0, 3, 0, 0, 0],
        [0, 0, 0, 0, 0, 3, 4, 5],
        [0, 0, 0, 0, 0, 4, 5, 6]])


Instruction: How to free GPU memory for a specific tensor in PyTorch?

Input:
I’m currently running a deep learning program using PyTorch and wanted to free the GPU memory for a specific tensor.

I’ve thought of methods like del and torch.cuda.empty_cache(), but del doesn’t seem to work properly (I’m not even sure if it frees memory at all) and torch.cuda.empty_cache() seems to free all unused memory, but I want to free memory for just a specific tensor.

Is there any functionality in PyTorch that provides this?

Thanks in advance.

Output:
del operator works but you won't see a decrease in the GPU memory used as the memory is not returned to the cuda device. It is an optimization technique and from the user's perspective, the memory has been "freed". That is, the memory is available for making new tensors now.

Source: Pytorch forum


Instruction: how to batch dialog dataset in pytorch?

Input:
I want to do a task oriented dialog chatbot which is used to book restaurant.Because every dialog has different sequences(eg. some has 5 turns of dialogs which is 10 sentences while another may has 6 turns of dialogs which is 12 sentences totally),I don't know how to batch dataset.

Could you give me some tutorial or github example?

Output:
There are some related questions to this on Stackoverflow. I liked the explanation/answer provided here. The tldr version is to use Packed Sequence. The answer I linked to provides the following example (copied from the link):

a = [torch.tensor([1,2,3]), torch.tensor([3,4])]
b = torch.nn.utils.rnn.pad_sequence(a, batch_first=True)
>>>>
tensor([[ 1,  2,  3],
    [ 3,  4,  0]])
torch.nn.utils.rnn.pack_padded_sequence(b, batch_first=True, lengths=[3,2])
>>>>PackedSequence(data=tensor([ 1,  3,  2,  4,  3]), batch_sizes=tensor([ 2,  2,  1]))


Instruction: How to set up loss function in PyTorch for Soft-Actor-Critic

Input:
I'm trying to implement a custom loss function for a soft Q-learning, actor-critic policy gradient algorithm in PyTorch. This comes from the following paper Learning from Imperfect Demonstrations. The structure of the algorithm is similar to deep q-learning, in that we are using a network to estimate Q-values, and we use a target network to stabilize results. Unlike DQN, however, we calculate V(s) from Q(s) by:

This is simple enough to calculate with PyTorch. My main question has to do with how to set up the loss function. Part of the update equation is expressed as:

Note that Q_hat comes from the target network. How can I go about putting something like this into a loss function? I can compute values for V and Q, but how can I handle the gradients in this case? If anyone can point me towards a similar example that would be much appreciated.

Output:
This turns out to be fairly simple assuming you can calculate V, Q, and Q^. After discussing this with some people offline I was able to get pytorch to calculate this loss by setting it up as:

loss = (Q-V)*(Q-Q_hat).detach()
optimizer.zero_grad()
loss.backward()
optimizer.step()


Instruction: Pytorch customize weight

Input:
I have a network

class Net(nn.Module)

and two different weights w0 and w1 (concatenate weights of all layers into a vector). Now I want to optimize the network on the line connecting w0 and w1, which means that the weight will have the form theta  * w0 + (1-theta) * w1. So now the parameter I want to optimize is no longer the weight itself, but the theta.

How can I implement this? In Pytorch, how can I define the parameter to be theta, and set the weight to be form I want. To be specific, if I create a new class

NetOnLine(nn.Module)

how should I write the forward(self, X) function?

Output:
You can define the parameter theta in your net as an nn.Parameter. You'd define the forward function the same way as normal - pass the data through the layers or operations you want and then return it.

Here's a minimal example, where I train a "network" to learn to multiply a Tensor by 2:

import numpy as np
import torch

class SampleNet(torch.nn.Module):
    def __init__(self):
        super(SampleNet, self).__init__()

        self.theta = torch.nn.Parameter(torch.rand(1))

    def forward(self, x):
        x = x * self.theta.expand_as(x)  # expand_as() to match sizes
        return x

train_data = np.random.rand(1000, 10)
train_data[:, 5:] = 2 * train_data[:, :5]
train_data = torch.Tensor(train_data)

sample_net = SampleNet()

optimizer = torch.optim.Adam(params=sample_net.parameters())
mse_loss = torch.nn.MSELoss()

for epoch in range(5):
    for data in train_data:
        x = data[:5]
        y = data[5:]

        optimizer.zero_grad()
        prediction = sample_net(x)
        loss = mse_loss(y, prediction)
        loss.backward()
        optimizer.step()

    print(f"Epoch {epoch}, Loss {loss.data.item()}")
print(f"Learned theta: {sample_net.theta.data.item()}")

which prints out

Epoch 0, Loss 0.03369491919875145
Epoch 1, Loss 0.0018534092232584953
Epoch 2, Loss 1.2343853995844256e-05
Epoch 3, Loss 2.2044337466553543e-09
Epoch 4, Loss 4.0527581290916714e-12
Learned theta: 1.999994158744812