Instruction: TypeError: cannot unpack non-iterable int object

Input:
Im trying to make my first CNN using pyTorch and am following online help and code already people wrote. i am trying to reproduce their results. I'm using the Kaggle Dogs Breed Dataset for this and below is the error I get. The trainloader does not return my images and labels and any attempt to get them leads in an error:

Traceback (most recent call last):
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1664, in <module>
    main()
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/sbzfk/PycharmProjects/my_FCN_attempt/Kaggle_Dogs_Competition.py", line 85, in <module>
    img, label = next(iter(train_loader))
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:/Users/sbzfk/PycharmProjects/my_FCN_attempt/Kaggle_Dogs_Competition.py", line 42, in __getitem__
    img = self.transform(img)
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\transforms.py", line 34, in __call__
    img = t(img)
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\transforms.py", line 187, in __call__
    w, h = img.size
TypeError: cannot unpack non-iterable int object

Below is my code:

class DogsDataset(Dataset):

    def __init__(self, filenames, labels, root_dir, transform=None):
        assert len(filenames) == len(labels)        # if the two are not of equal length throw an error
        self.filenames = filenames
        self.labels = labels
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.filenames)

    def __getitem__(self, idx):
        this_img = join(self.root_dir, 'train', self.filenames[idx]+'.jpg')
        print(this_img)
        img = io.imread(this_img)
        label = self.labels[idx]
        print(label)

        if self.transform:
            img = self.transform(img)

        return [img, label]

batch_size = 64
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

dataset_root = expanduser(join('~', 'Documents', 'kaggle_dogs_dataset'))
# join will intelligently join directories irrespective of OS, and expanduser will
# replace with /home/ in linux or the username in Windows

csv_file = pd.read_csv(join(dataset_root, 'labels.csv'))        # csv file has two columns, id which are filenames and breed which are labels
filenames = csv_file.id.values              # convert that column to an array, id is the column name and values converty to numpy array
# le = LabelEncoder()
# labels = le.fit_transform(csv_file.breed)           # this will just encode the names between 0 to models-1 , basically changing strings to integers
labels = csv_file.breed.values

filenames_train, filenames_eval, labels_train, labels_eval = train_test_split(filenames, labels,
                                                                              test_size=0.1, stratify=labels)       # this is an import from sklearn as the name implies, it randomly splits data into train and eval, 10% of it to test and rest train

data_transform = transforms.Compose([transforms.Scale(224),
                                     transforms.CenterCrop(224),
                                     transforms.ToTensor(),
                                     transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])

dog_train = DogsDataset(filenames_train, labels_train, dataset_root, transform=data_transform)
train_loader = DataLoader(dog_train, batch_size, shuffle=True)

dog_eval = DogsDataset(filenames_eval, labels_eval, dataset_root, transform=data_transform)
eval_loader = DataLoader(dog_eval, batch_size, shuffle=True)

def im_show(axis, inp):
    """Denormalize and show"""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    axis.imshow(inp)

img, label = next(iter(train_loader))
print(img.size(), label.size())
fig = plt.figure(1, figsize=(16, 4))
grid = ImageGrid(fig, 111, nrows_ncols=(1, 4), axes_pad=0.05)
for i in range(img.size()[0]):
    ax = grid[i]
    im_show(ax, img[i])

Ive tried debugging it line by line and with transform=none I seem to read all the images, only with transform=data_transform I seem to get this error.

Output:
It seems like you are using torchvision's image transforms. Some of these transforms are expecting as input a PIL.Image object, rather than a tensor or numpy array.
You are using io.imread to read ths image file, and I suspect this io is not PIL.Image resulting with a numpy array.
Make sure you pass PIL.Image objects to transforms and that your DogsDataset returns a 3D tensor for image (C-H-W shaped).


Instruction: Order of CUDA devices

Input:
I saw this solution, but it doesn't quite answer my question; it's also quite old so I'm not sure how relevant it is.

I keep getting conflicting outputs for the order of GPU units. There are two of them: Tesla K40 and NVS315 (legacy device that is never used). When I run deviceQuery, I get

Device 0: "Tesla K40m"
...
Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0

Device 1: "NVS 315"
...
Device PCI Domain ID / Bus ID / location ID:   0 / 3 / 0

On the other hand, nvidia-smi produces a different order:

 0  NVS 315

 1  Tesla K40m

Which I find very confusing. The solution I found for Tensorflow (and a similar one for Pytorch) is to use

import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"

PCI Bus ID is 4 for Tesla and 3 for NVS, so it should set it to 3 (NVS), is that right?

In pytorch I set

os.environ['CUDA_VISIBLE_DEVICES']='0'
...
device = torch.cuda.device(0)
print torch.cuda.get_device_name(0)

to get Tesla K40m

when I set instead

os.environ['CUDA_VISIBLE_DEVICES']='1'
device = torch.cuda.device(1)
print torch.cuda.get_device_name(0)

to get

UserWarning:
    Found GPU0 NVS 315 which is of cuda capability 2.1.
    PyTorch no longer supports this GPU because it is too old.

  warnings.warn(old_gpu_warn % (d, name, major, capability[1]))
NVS 315

So I'm quite confused: what's the true order of GPU devices that tf and pytorch use?

Output:
By default, CUDA orders the GPUs by computing power. GPU:0 will be the fastest GPU on your host, in your case the K40m.

If you set CUDA_DEVICE_ORDER='PCI_BUS_ID' then CUDA orders your GPU depending on how you set up your machine meaning that GPU:0 will be the GPU on your first PCI-E lane.

Both Tensorflow and PyTorch use the CUDA GPU order. That is consistent with what you showed:

os.environ['CUDA_VISIBLE_DEVICES']='0'
...
device = torch.cuda.device(0)
print torch.cuda.get_device_name(0)

Default order so GPU:0 is the K40m since it is the most powerful card on your host.

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ['CUDA_VISIBLE_DEVICES']='0'
...
device = torch.cuda.device(0)
print torch.cuda.get_device_name(0)

PCI-E lane order, so GPU:0 is the card with the lowest bus-id in your case the NVS.


Instruction: AttributeError: 'Tensor' has no attribute: 'backwards'

Input:
I'm posting this question for those who made the same error I did. I got this error when trying to compute my gradients:

criterion = torch.nn.CrossEntropyLoss()
loss = criterion(y_hat, y_truth)
loss.backwards()

Output: It's loss.backward(), not loss.backwards().


Instruction: Why pytorch DataLoader behaves differently on numpy array and list?

Input:
The only difference is one of the parameter passed to DataLoader is in type "numpy.array" and the other is in type "list", but the DataLoader gives totally different results.

You can use the following code to reproduce it:

from torch.utils.data import DataLoader,Dataset
import numpy as np

class my_dataset(Dataset):
    def __init__(self,data,label):
        self.data=data
        self.label=label
    def __getitem__(self, index):
        return self.data[index],self.label[index]
    def __len__(self):
        return len(self.data)

train_data=[[1,2,3],[5,6,7],[11,12,13],[15,16,17]]
train_label=[-1,-2,-11,-12]

########################### Look at here:

test=DataLoader(dataset=my_dataset(np.array(train_data),train_label),batch_size=2)
for i in test:
    print ("numpy data:")
    print (i)
    break

test=DataLoader(dataset=my_dataset(train_data,train_label),batch_size=2)
for i in test:
    print ("list data:")
    print (i)
    break

The result is:

numpy data:
[tensor([[1, 2, 3],
        [5, 6, 7]]), tensor([-1, -2])]
list data:
[[tensor([1, 5]), tensor([2, 6]), tensor([3, 7])], tensor([-1, -2])]

Output:
This is because how batching is handled in torch.utils.data.DataLoader. collate_fn argument decides how samples from samples are merged into a single batch. Default for this argument is undocumented torch.utils.data.default_collate.

This function handles batching by assuming numbers/tensors/ndarrays are primitive data to batch and lists/tuples/dicts containing these primitives as structure to be (recursively) preserved. This allow you to have a semantic batching like this:

(input_tensor, label_tensor) -> (batched_input_tensor, batched_label_tensor)
([input_tensor_1, input_tensor_2], label_tensor) -> ([batched_input_tensor_1, batched_input_tensor_2], batched_label_tensor)
{'input': input_tensor, 'target': target_tensor} -> {'input': batched_input_tensor, 'target': batched_target_tensor}

(Left side of -> is output of dataset[i], while right side is batched sample from torch.utils.data.DataLoader)

Your example code is similar to example 2 above: list structure is preserved while ints are batched.


Instruction: Multi label classification in pytorch

Input:
I have a multi-label classification problem. I have 11 classes, around 4k examples. Each example can have from 1 to 4-5 label. At the moment, i'm training a classifier separately for each class with log_loss. As you can expect, it is taking quite some time to train 11 classifier, and i would like to try another approach and to train only 1 classifier. The idea is that the last layer of this classifer would have 11 nodes, and would output a real number by classes which would be converted to a proba by a sigmoid. The loss I want to optimize is the mean of the log_loss on all classes.

Unfortunately, i'm some kind of noob with pytorch, and even by reading the source code of the losses, i can't figure out if one of the already existing losses does exactly what i want, or if I should create a new loss, and if that's the case, i don't really know how to do it.

To be very specific, i want to give for each element of the batch one vector of size 11(which contains a real number for each label (the closer to infinity, the closer this class is predicted to be 1), and 1 vector of size 11 (which contains a 1 at every true label), and be able to compute the mean log_loss on all 11 labels, and optimize my classifier based on that loss.

Any help would be greatly appreciated :)

Output:
You are looking for torch.nn.BCELoss. Here's example code:
import torch

batch_size = 2
num_classes = 11

loss_fn = torch.nn.BCELoss()

outputs_before_sigmoid = torch.randn(batch_size, num_classes)
sigmoid_outputs = torch.sigmoid(outputs_before_sigmoid)
target_classes = torch.randint(0, 2, (batch_size, num_classes))  # randints in [0, 2).

loss = loss_fn(sigmoid_outputs, target_classes)

# alternatively, use BCE with logits, on outputs before sigmoid.
loss_fn_2 = torch.nn.BCEWithLogitsLoss()
loss2 = loss_fn_2(outputs_before_sigmoid, target_classes)
assert loss == loss2


Instruction: Module not found in pycharm (Windows)

Input:
I wanted to install Pytorch via anaconda and it worked but PyCharm can't find the module (

  ModuleNotFoundError: No module named 'torch'

I also have CUDA installed but when I looked up to add a the package with pycharm it also gives an error. When I added the anaconda interpreter I can't run the code. I use Python 3.6

Output: Ok I solved this problem: First install anaconda and open the prompt then type conda install pytorch -c pytorch and pip3 install torchvision. Then go to PyCharm and create an Project and set the Project Interpreter to the Anaconda one (there is in the path: \Anaconda.x.x\python.exe ). Then you go to the Run settings and click Run... and then you go to Edit Configurations and then you select the Project Default interpreter and apply and you should be done! Thanks to the ppl who helped me =)


Instruction: Recurrent network (RNN) won't learn a very simple function (plots shown in the question)

Input:
So I am trying to train a simple recurrent network to detect a "burst" in an input signal. The following figure shows the input signal (blue) and the desired (classification) output of the RNN, shown in red.

So the output of the network should switch from 1 to 0 whenever the burst is detected and stay like with that output. The only thing that changes between the input sequences used to train the RNN is at which time step the burst occurs.

Following the Tutorial on https://github.com/MorvanZhou/PyTorch-Tutorial/blob/master/tutorial-contents/403_RNN_regressor.py, I cannot get a RNN to learn. The learned RNN always operates in a "memoryless" way, i.e., does not use memory to make its predictions, as shown in the following example behavior:

The green line shows the predicted output of the network. What do I do wrong in this example so that the network cannot be learned correctly? Isn't the network task quite simple?

I'm using:

torch.nn.CrossEntropyLoss as loss function
The Adam Optimizer for learning
A RNN with 16 internal/hidden nodes and 2 output nodes. They use the default activation function of the torch.RNN class.

The experiment has been repeated a couple of times with different random seeds, but there is little difference in the outcomes. I've used the following code:

import torch
import numpy, math
import matplotlib.pyplot as plt

nofSequences = 5
maxLength = 130

# Generate training data
x_np = numpy.zeros((nofSequences,maxLength,1))
y_np = numpy.zeros((nofSequences,maxLength))
numpy.random.seed(1)
for i in range(0,nofSequences):
    startPos = numpy.random.random()*50
    for j in range(0,maxLength):
        if j>=startPos and j<startPos+10:
            x_np[i,j,0] = math.sin((j-startPos)*math.pi/10)
        else:
            x_np[i,j,0] = 0.0
        if j<startPos+10:
            y_np[i,j] = 1
        else:
            y_np[i,j] = 0

# Define the neural network
INPUT_SIZE = 1
class RNN(torch.nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.rnn = torch.nn.RNN(
            input_size=INPUT_SIZE,
            hidden_size=16,     # rnn hidden unit
            num_layers=1,       # number of rnn layer
            batch_first=True,
        )
        self.out = torch.nn.Linear(16, 2)

    def forward(self, x, h_state):
        r_out, h_state = self.rnn(x, h_state)

        outs = []    # save all predictions
        for time_step in range(r_out.size(1)):    # calculate output for each time step
            outs.append(self.out(r_out[:, time_step, :]))
        return torch.stack(outs, dim=1), h_state

# Learn the network
rnn = RNN()
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.01)
h_state = None      # for initial hidden state

x = torch.Tensor(x_np)    # shape (batch, time_step, input_size)
y = torch.Tensor(y_np).long()

torch.manual_seed(2)
numpy.random.seed(2)

for step in range(100):

    prediction, h_state = rnn(x, h_state)   # rnn output

    # !! next step is important !!
    h_state = h_state.data        # repack the hidden state, break the connection from last iteration

    loss = torch.nn.CrossEntropyLoss()(prediction.reshape((-1,2)),torch.autograd.Variable(y.reshape((-1,))))         # calculate loss
    optimizer.zero_grad()                   # clear gradients for this training step
    loss.backward()                         # backpropagation, compute gradients
    optimizer.step()                        # apply gradients

    errTrain = (prediction.max(2)[1].data != y).float().mean()
    print("Error Training:",errTrain.item())

For those who want to reproduce the experiment, the plot is drawn using the following code (using Jupyter Notebook):

steps = range(0,maxLength)
plotChoice = 3

plt.figure(1, figsize=(12, 5))
plt.ion()           # continuously plot

plt.plot(steps, y_np[plotChoice,:].flatten(), 'r-')
plt.plot(steps, numpy.argmax(prediction.detach().numpy()[plotChoice,:,:],axis=1), 'g-')
plt.plot(steps, x_np[plotChoice,:,0].flatten(), 'b-')

plt.ioff()
plt.show()

Output:
From the documentation of tourch.nn.RNN, the RNN is actually an Elman network, and have the following properties seen here.
The output of an Elman network is only dependent on the hidden state, while the hidden state is dependent on the last input and the previous hidden state.

Since we have set “h_state = h_state.data”, we actually use the hidden state of the last sequence to predict the first state of the new sequence, which will result in an output heavily dependent on the last output of the previous sequence (which was 0). The Elman network can’t separate if we are in the beginning of the sequence or at the end, it only "sees" the state and last input.

To fix this we can insted set “h_state = None”.
Now every new sequence start with an empty state. This result in the following prediction (where green line again shows the prediction).
Now we start off at 1, but quickly dips down to 0 before the puls push it back up again.
The Elman network can account for some time dependency, but it is not good at remembering long term dependencies, and converge towards an "most common output" for that input.

So to fix this problem, I suggest using a network which is well known for handling long term dependencies well, namely the Long short-term memory (LSTM) rnn, for more information see torch.nn.LSTM. Keep "h_state = None" and change torch.nn.RNN to torch.nn.LSTM.

for complete code and plot see below

import torch
import numpy, math
import matplotlib.pyplot as plt

nofSequences = 5
maxLength = 130

# Generate training data
x_np = numpy.zeros((nofSequences,maxLength,1))
y_np = numpy.zeros((nofSequences,maxLength))
numpy.random.seed(1)
for i in range(0,nofSequences):
    startPos = numpy.random.random()*50
    for j in range(0,maxLength):
        if j>=startPos and j<startPos+10:
            x_np[i,j,0] = math.sin((j-startPos)*math.pi/10)
        else:
            x_np[i,j,0] = 0.0
        if j<startPos+10:
            y_np[i,j] = 1
        else:
            y_np[i,j] = 0

# Define the neural network
INPUT_SIZE = 1
class RNN(torch.nn.Module):
    def __init__(self):
        super(RNN, self).__init__()

        self.rnn = torch.nn.LSTM(
            input_size=INPUT_SIZE,
            hidden_size=16,     # rnn hidden unit
            num_layers=1,       # number of rnn layer
            batch_first=True,
        )
        self.out = torch.nn.Linear(16, 2)

    def forward(self, x, h_state):
        r_out, h_state = self.rnn(x, h_state)

        outs = []    # save all predictions
        for time_step in range(r_out.size(1)):    # calculate output for each time step
            outs.append(self.out(r_out[:, time_step, :]))
        return torch.stack(outs, dim=1), h_state

# Learn the network
rnn = RNN()
optimizer = torch.optim.Adam(rnn.parameters(), lr=0.01)
h_state = None      # for initial hidden state

x = torch.Tensor(x_np)    # shape (batch, time_step, input_size)
y = torch.Tensor(y_np).long()

torch.manual_seed(2)
numpy.random.seed(2)

for step in range(100):

    prediction, h_state = rnn(x, h_state)   # rnn output

    # !! next step is important !!
    h_state = None

    loss = torch.nn.CrossEntropyLoss()(prediction.reshape((-1,2)),torch.autograd.Variable(y.reshape((-1,))))         # calculate loss
    optimizer.zero_grad()                   # clear gradients for this training step
    loss.backward()                         # backpropagation, compute gradients
    optimizer.step()                        # apply gradients

    errTrain = (prediction.max(2)[1].data != y).float().mean()
    print("Error Training:",errTrain.item())

###############################################################################
steps = range(0,maxLength)
plotChoice = 3

plt.figure(1, figsize=(12, 5))
plt.ion()           # continuously plot

plt.plot(steps, y_np[plotChoice,:].flatten(), 'r-')
plt.plot(steps, numpy.argmax(prediction.detach().numpy()[plotChoice,:,:],axis=1), 'g-')
plt.plot(steps, x_np[plotChoice,:,0].flatten(), 'b-')

plt.ioff()
plt.show()


Instruction: What does the underscore suffix in PyTorch functions mean?

Input:
In PyTorch, many methods of a tensor exist in two versions - one with a underscore suffix, and one without. If I try them out, they seem to do the same thing:

In [1]: import torch

In [2]: a = torch.tensor([2, 4, 6])

In [3]: a.add(10)
Out[3]: tensor([12, 14, 16])

In [4]: a.add_(10)
Out[4]: tensor([12, 14, 16])

What is the difference between

torch.add and torch.add_
torch.sub and torch.sub_
...and so on?

Output:
You have already answered your own question that the underscore indicates in-place operations in PyTorch. However I want to point out briefly why in-place operations can be problematic:

First of all on the PyTorch sites it is recommended to not use in-place operations in most cases. Unless working under heavy memory pressure it is more efficient in most cases to not use in-place operations.
https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd
Secondly there can be problems calculating the gradients when using in-place operations:

  Every tensor keeps a version counter, that is incremented every time
  it is marked dirty in any operation. When a Function saves any tensors
  for backward, a version counter of their containing Tensor is saved as
  well. Once you access self.saved_tensors it is checked, and if it is
  greater than the saved value an error is raised. This ensures that if
  you’re using in-place functions and not seeing any errors, you can be
  sure that the computed gradients are correct.
  Same source as above.

Here is a shot and slightly modified example taken from the answer you've posted:

First the in-place version:

import torch
a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
adding_tensor = torch.rand(3)
b = a.add_(adding_tensor)
c = torch.sum(b)
c.backward()
print(c.grad_fn)

Which leads to this error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-c38b252ffe5f> in <module>
      2 a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
      3 adding_tensor = torch.rand(3)
----> 4 b = a.add_(adding_tensor)
      5 c = torch.sum(b)
      6 c.backward()

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

Secondly the non in-place version:

import torch
a = torch.tensor([2, 4, 6], requires_grad=True, dtype=torch.float)
adding_tensor = torch.rand(3)
b = a.add(adding_tensor)
c = torch.sum(b)
c.backward()
print(c.grad_fn)

Which works just fine - output:

<SumBackward0 object at 0x7f06b27a1da0>

So as a take-away I just wanted to point out to carefully use in-place operations in PyTorch.


Instruction: pytorch modify array with list of indices

Input:
Suppose I have a list of indices and wish to modify an existing array with this list. Currently the only way I can do this is by using a for loop as follows. Just wondering if there is a faster/ efficient way.

torch.manual_seed(0)
a = torch.randn(5,3)
idx = torch.Tensor([[1,2], [3,2]], dtype=torch.long)
for i,j in idx:
    a[i,j] = 1

I initially assumed that gather or index_select would go some way in answering this question, but looking at documentation this doesn't seem to be the answer.

In my particular case, a is a 5 dimensional vector and idx is a Nx5 vector. So the output (after subscripting with something like a[idx]) I'd expect is a (N,) shaped vector.

Answer

Thanks to @shai below, the answer that I was seeking was:
a[idx.t().chunk(chunks=2,dim=0)]. Taken from this SO answer.

Output:
It's quite simple

a[idx[:,0], idx[:,1]] = 1

You can find a more general solution in this thread.


Instruction: RuntimeError: "exp" not implemented for 'torch.LongTensor'

Input:
I am following this tutorial: http://nlp.seas.harvard.edu/2018/04/03/attention.html
to implement the Transformer model from the "Attention Is All You Need" paper.

However I am getting the following error :
RuntimeError: "exp" not implemented for 'torch.LongTensor'

This is the line, in the PositionalEnconding class, that is causing the error:

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

When it is being constructed here:

pe = PositionalEncoding(20, 0)

Any ideas?? I've already tried converting this to perhaps a Tensor Float type, but this has not worked.

I've even downloaded the whole notebook with accompanying files and the error seems to persist in the original tutorial.

Any ideas what may be causing this error?

Thanks!

Output:
I happened to follow this tutorial too.

For me I just got the torch.arange to generate float type tensor

from

position = torch.arange(0, max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

to

position = torch.arange(0., max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

Just a simple fix. But now it works for me. It is possible that the torch exp and sin previously support LongTensor but not anymore (not very sure about it).


Instruction: pytorch how to compute grad after clone a tensor

Input:
My simple code:

import torch

x = torch.randn(4, requires_grad=True).cuda()
y = torch.randn(4, requires_grad=True).cuda()
z = torch.zeros(4)
z = torch.clone(x)
z.retain_grad()
h = (z + y) * z
l = torch.randn(4).cuda()
loss = (l - h).pow(2).sum()
loss.backward()
print('x.grad=', x.grad)
print('z.grad=', z.grad)

output:

x.grad= None
z.grad= tensor([-15.3401,  -3.2623,  -2.1670,   0.1410], device='cuda:0')

Why x.grad is None but not same as z.grad ?
What should I do if I want they are same?

Output: You need to call x.retain_grad() after declaring x if you want to keep the grad of tensor x.


Instruction: PyTorch: manually setting weight parameters with numpy array for GRU / LSTM

Input:
I'm trying to fill up GRU/LSTM with manually defined parameters in pytorch.

I have numpy arrays for parameters with shapes as defined in their documentation (https://pytorch.org/docs/stable/nn.html#torch.nn.GRU).

It seems to work but I'm not sure whether the returned values are correct.

Is this a right way to fill up GRU/LSTM with numpy parameters?

gru = nn.GRU(input_size, hidden_size, num_layers,
              bias=True, batch_first=False, dropout=dropout, bidirectional=bidirectional)

def set_nn_wih(layer, parameter_name, w, l0=True):
    param = getattr(layer, parameter_name)
    if l0:
        for i in range(3*hidden_size):
            param.data[i] = w[i*input_size:(i+1)*input_size]
    else:
        for i in range(3*hidden_size):
            param.data[i] = w[i*num_directions*hidden_size:(i+1)*num_directions*hidden_size]

def set_nn_whh(layer, parameter_name, w):
    param = getattr(layer, parameter_name)
    for i in range(3*hidden_size):
        param.data[i] = w[i*hidden_size:(i+1)*hidden_size]

l0=True

for i in range(num_directions):
    for j in range(num_layers):
        if j == 0:
            wih = w0[i, :, :3*input_size]
            whh = w0[i, :, 3*input_size:]  # check
            l0=True
        else:
            wih = w[j-1, i, :, :num_directions*3*hidden_size]
            whh = w[j-1, i, :, num_directions*3*hidden_size:]
            l0=False

        if i == 0:
            set_nn_wih(
                gru, "weight_ih_l{}".format(j), torch.from_numpy(wih.flatten()),l0)
            set_nn_whh(
                gru, "weight_hh_l{}".format(j), torch.from_numpy(whh.flatten()))
        else:
            set_nn_wih(
                gru, "weight_ih_l{}_reverse".format(j), torch.from_numpy(wih.flatten()),l0)
            set_nn_whh(
                gru, "weight_hh_l{}_reverse".format(j), torch.from_numpy(whh.flatten()))

y, hn = gru(x_t, h_t)

numpy arrays are defined as following:

rng = np.random.RandomState(313)
w0 = rng.randn(num_directions, hidden_size, 3*(input_size +
               hidden_size)).astype(np.float32)
w = rng.randn(max(1, num_layers-1), num_directions, hidden_size,
              3*(num_directions*hidden_size + hidden_size)).astype(np.float32)

Output:
That is a good question, and you already give a decent answer. However, it reinvents the wheel - there is a very elegant Pytorch internal routine that will allow you to do the same without as much effort - and one that is applicable for any network.

The core concept here is PyTorch's state_dict. The state dictionary effectively contains the parameters organized by the tree-structure given by the relationship of the nn.Modules and their submodules, etc.

The short answer

If you only want the code to load a value into a tensor using the state_dict, then try this line (where the dict contains a valid state_dict):

`model.load_state_dict(dict, strict=False)`

where strict=False is crucial if you want to load only some parameter values.

The long answer - including an introduction to PyTorch's state_dict

Here's an example of how a state dict looks for a GRU (I chose input_size = hidden_size = 2 so that I can print the entire state dict):

rnn = torch.nn.GRU(2, 2, 1)
rnn.state_dict()
# Out[10]:
#     OrderedDict([('weight_ih_l0', tensor([[-0.0023, -0.0460],
#                         [ 0.3373,  0.0070],
#                         [ 0.0745, -0.5345],
#                         [ 0.5347, -0.2373],
#                         [-0.2217, -0.2824],
#                         [-0.2983,  0.4771]])),
#                 ('weight_hh_l0', tensor([[-0.2837, -0.0571],
#                         [-0.1820,  0.6963],
#                         [ 0.4978, -0.6342],
#                         [ 0.0366,  0.2156],
#                         [ 0.5009,  0.4382],
#                         [-0.7012, -0.5157]])),
#                 ('bias_ih_l0',
#                 tensor([-0.2158, -0.6643, -0.3505, -0.0959, -0.5332, -0.6209])),
#                 ('bias_hh_l0',
#                 tensor([-0.1845,  0.4075, -0.1721, -0.4893, -0.2427,  0.3973]))])

So the state_dict all the parameters of the network. If we have "nested" nn.Modules, we get the tree represented by the parameter names:

class MLP(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.lin_a = torch.nn.Linear(2, 2)
        self.lin_b = torch.nn.Linear(2, 2)

mlp = MLP()
mlp.state_dict()
#    Out[23]:
#        OrderedDict([('lin_a.weight', tensor([[-0.2914,  0.0791],
#                            [-0.1167,  0.6591]])),
#                    ('lin_a.bias', tensor([-0.2745, -0.1614])),
#                    ('lin_b.weight', tensor([[-0.4634, -0.2649],
#                            [ 0.4552,  0.3812]])),
#                    ('lin_b.bias', tensor([ 0.0273, -0.1283]))])

class NestedMLP(torch.nn.Module):
    def __init__(self):
        torch.nn.Module.__init__(self)
        self.mlp_a = MLP()
        self.mlp_b = MLP()

n_mlp = NestedMLP()
n_mlp.state_dict()
#   Out[26]:
#        OrderedDict([('mlp_a.lin_a.weight', tensor([[ 0.2543,  0.3412],
#                            [-0.1984, -0.3235]])),
#                    ('mlp_a.lin_a.bias', tensor([ 0.2480, -0.0631])),
#                    ('mlp_a.lin_b.weight', tensor([[-0.4575, -0.6072],
#                            [-0.0100,  0.5887]])),
#                    ('mlp_a.lin_b.bias', tensor([-0.3116,  0.5603])),
#                    ('mlp_b.lin_a.weight', tensor([[ 0.3722,  0.6940],
#                            [-0.5120,  0.5414]])),
#                    ('mlp_b.lin_a.bias', tensor([0.3604, 0.0316])),
#                    ('mlp_b.lin_b.weight', tensor([[-0.5571,  0.0830],
#                            [ 0.5230, -0.1020]])),
#                    ('mlp_b.lin_b.bias', tensor([ 0.2156, -0.2930]))])

So - what if you want to not extract the state dict, but change it - and thereby the network's parameters? Use nn.Module.load_state_dict(state_dict, strict=True) (link to the docs)
This method allows you to load an entire state_dict with arbitrary values into an instantiated model of the same kind as long as the keys (i.e. the parameter names) are correct and the values (i.e. the parameters) are torch.tensors of the right shape.
If the strict kwarg is set to True (the default), the dict you load has to exactly match the original state dict, except for the values of the parameters. That is, there has to be one new value for each parameter.

For the GRU example above, we need a tensor of the correct size (and the correct device, btw) for each of 'weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0'. As we sometimes only want to load some values (as I think you want to do), we can set the strict kwarg to False - and we can then load only partial state dicts, as e.g. one that only contains parameter values for 'weight_ih_l0'.

As a practical advice, I'd simply create the model you want to load values into, and then print the state dict (or at least a list of the keys and the respective tensor sizes)

print([k, v.shape for k, v in model.state_dict().items()])

That tells you what the exact name of the parameter is you want to change. You then simply create a state dict with the respective parameter name and tensor, and load it:

from dollections import OrderedDict
new_state_dict = OrderedDict({'tensor_name_retrieved_from_original_dict': new_tensor_value})
model.load_state_dict(new_state_dict, strict=False)


Instruction: Pytorch (0.4.1) can not find GPU (Nvidia V100)

Input:
I have installed Pytorch(0.4.1) with CUDA 9.0 via anaconda.
I also have installed Nvidia driver (390.30).
Through nvidia-smi, I can see all GPUs(V100) with their situation.
But I always get False when calling torch.cuda.is_avaliable().
I also tried to downgrade the CUDA version from 9.0 to 8.0, but the situation was just the same.
Tried to install Pytorch again but nothing changed. Could anyone give me a hand, please?

Output:
It seems that it's a Nvidia container issue.
After downgrading the Nvidia container from 18.08 to 18.07, all code runs well.


Instruction: Bool value of Tensor with more than one value is ambiguous in Pytorch

Input:
I want to create a model in pytorch, but I can't
compute the loss.
It's always return Bool value of Tensor with more
than one value is ambiguous
Actually, I run example code, it work.

loss = CrossEntropyLoss()
input = torch.randn(8, 5)
input
target = torch.empty(8,dtype=torch.long).random_(5)
target
output = loss(input, target)

Here is my code,

################################################################################
##
##
import torch
from torch.nn import Conv2d, MaxPool2d, Linear, CrossEntropyLoss, MultiLabelSoftMarginLoss
from torch.nn.functional import relu, conv2d, max_pool2d, linear, softmax
from torch.optim import adadelta
##
##
##  Train
Train = {}
Train["Image"]    = torch.rand(2000, 3, 76, 76)
Train["Variable"] = torch.rand(2000, 6)
Train["Label"] = torch.empty(2000, dtype=torch.long).random_(2)
##
##
##  Valid
Valid = {}
Valid["Image"]    = torch.rand(150, 3, 76, 76)
Valid["Variable"] = torch.rand(150, 6)
Valid["Label"]    = torch.empty(150, dtype=torch.long).random_(2)
################################################################################
##
##
##  Model
ImageTerm    = Train["Image"]
VariableTerm = Train["Variable"]
Pip = Conv2d(in_channels=3, out_channels=32, kernel_size=(3,3), stride=1, padding=0)(ImageTerm)
Pip = MaxPool2d(kernel_size=(2,2), stride=None, padding=0)(Pip)
Pip = Conv2d(in_channels=32, out_channels=64, kernel_size=(3,3), stride=1, padding=0)(Pip)
Pip = MaxPool2d(kernel_size=(2,2), stride=None, padding=0)(Pip)
Pip = Pip.view(2000, -1)
Pip = torch.cat([Pip, VariableTerm], 1)
Pip = Linear(in_features=18502, out_features=1000 , bias=True)(Pip)
Pip = Linear(in_features=1000, out_features=2 , bias=True)(Pip)
##
##
##  Loss
Loss = CrossEntropyLoss(Pip, Train["Label"])

The error is on  Loss = CrossEntropyLoss(Pip, Train["Label"]),
thanks.

Output:
In your minimal example, you create an object "loss" of the class "CrossEntropyLoss". This object is able to compute your loss as
loss(input, target)

However, in your actual code, you try to create the object "Loss", while passing Pip and the labels to the "CrossEntropyLoss" class constructor.
Instead, try the following:
loss = CrossEntropyLoss()
loss(Pip, Train["Label"])

Edit (explanation of the error message): The error Message Bool value of Tensor with more than one value is ambiguous appears when you try to cast a tensor into a bool value. This happens most commonly when passing the tensor to an if condition, e.g.
input = torch.randn(8, 5)
if input:
    some_code()

The second argument of the CrossEntropyLoss class constructor expects a boolean. Thus, in the line
Loss = CrossEntropyLoss(Pip, Train["Label"])

the constructor will at some point try to use the passed tensor Train["Label"] as a boolean, which throws the mentioned error message.


Instruction: Pytorch: Getting the correct dimensions for final layer

Input:
Pytorch newbie here! I am trying to fine-tune a VGG16 model to predict 3 different classes. Part of my work involves converting FC layers to CONV layers. However, the values of my predictions don't fall between 0 to 2 (the 3 classes).

Can someone point me to a good resource on how to compute the correct dimensions for the final layer?

Here are the original fC layers of VGG16:

(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )

My code for converting FC layers to CONV:

 def convert_fc_to_conv(self, fc_layers):
        # Replace first FC layer with CONV layer
        fc = fc_layers[0].state_dict()
        in_ch = 512
        out_ch = fc["weight"].size(0)
        first_conv = nn.Conv2d(512, out_ch, kernel_size=(1, 1), stride=(1, 1))

        conv_list = [first_conv]
        for idx, layer in enumerate(fc_layers[1:]):
            if isinstance(layer, nn.Linear):
                fc = layer.state_dict()
                in_ch = fc["weight"].size(1)
                out_ch = fc["weight"].size(0)
                if idx == len(fc_layers)-4:
                    in_ch = 3
                conv = nn.Conv2d(out_ch, in_ch, kernel_size=(1, 1), stride=(1, 1))
                conv_list += [conv]
            else:
                conv_list += [layer]
            gc.collect()

        avg_pool = nn.AvgPool2d(kernel_size=2, stride=1, ceil_mode=False)
        conv_list += [avg_pool, nn.Softmax()]
        top_layers = nn.Sequential(*conv_list)
        return top_layers

Final model architecture:

    Model(
    (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))

    (classifier): Sequential(
    (0): Conv2d(512, 4096, kernel_size=(1, 1), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Conv2d(4096, 3, kernel_size=(1, 1), stride=(1, 1))
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): AvgPool2d(kernel_size=2, stride=1, padding=0)
    (7): Softmax()
  )
)

Summary of the model:

            Conv2d-1         [-1, 64, 224, 224]           1,792
              ReLU-2         [-1, 64, 224, 224]               0
            Conv2d-3         [-1, 64, 224, 224]          36,928
              ReLU-4         [-1, 64, 224, 224]               0
         MaxPool2d-5         [-1, 64, 112, 112]               0
            Conv2d-6        [-1, 128, 112, 112]          73,856
              ReLU-7        [-1, 128, 112, 112]               0
            Conv2d-8        [-1, 128, 112, 112]         147,584
              ReLU-9        [-1, 128, 112, 112]               0
        MaxPool2d-10          [-1, 128, 56, 56]               0
           Conv2d-11          [-1, 256, 56, 56]         295,168
             ReLU-12          [-1, 256, 56, 56]               0
           Conv2d-13          [-1, 256, 56, 56]         590,080
             ReLU-14          [-1, 256, 56, 56]               0
           Conv2d-15          [-1, 256, 56, 56]         590,080
             ReLU-16          [-1, 256, 56, 56]               0
        MaxPool2d-17          [-1, 256, 28, 28]               0
           Conv2d-18          [-1, 512, 28, 28]       1,180,160
             ReLU-19          [-1, 512, 28, 28]               0
           Conv2d-20          [-1, 512, 28, 28]       2,359,808
             ReLU-21          [-1, 512, 28, 28]               0
           Conv2d-22          [-1, 512, 28, 28]       2,359,808
             ReLU-23          [-1, 512, 28, 28]               0
        MaxPool2d-24          [-1, 512, 14, 14]               0
           Conv2d-25          [-1, 512, 14, 14]       2,359,808
             ReLU-26          [-1, 512, 14, 14]               0
           Conv2d-27          [-1, 512, 14, 14]       2,359,808
             ReLU-28          [-1, 512, 14, 14]               0
           Conv2d-29          [-1, 512, 14, 14]       2,359,808
             ReLU-30          [-1, 512, 14, 14]               0
        MaxPool2d-31            [-1, 512, 7, 7]               0
           Conv2d-32           [-1, 4096, 7, 7]       2,101,248
             ReLU-33           [-1, 4096, 7, 7]               0
          Dropout-34           [-1, 4096, 7, 7]               0
           Conv2d-35              [-1, 3, 7, 7]          12,291
             ReLU-36              [-1, 3, 7, 7]               0
          Dropout-37              [-1, 3, 7, 7]               0
        AvgPool2d-38              [-1, 3, 6, 6]               0
          Softmax-39              [-1, 3, 6, 6]               0

Output:
I wrote a function that takes a Pytorch model as input and converts the classification layer to convolution layer. It works for VGG and Alexnet for now, but you can extend it for other models as well.

import torch
import torch.nn as nn
from torchvision.models import alexnet, vgg16

def convolutionize(model, num_classes, input_size=(3, 224, 224)):
    '''Converts the classification layers of VGG & Alexnet to convolutions

    Input:
        model: torch.models
        num_classes: number of output classes
        input_size: size of input tensor to the model

    Returns:
        model: converted model with convolutions
    '''
    features = model.features
    classifier = model.classifier

    # create a dummy input tensor and add a dim for batch-size
    x = torch.zeros(input_size).unsqueeze_(dim=0)

    # change the last layer output to the num_classes
    classifier[-1] = nn.Linear(in_features=classifier[-1].in_features,
                               out_features=num_classes)

    # pass the dummy input tensor through the features layer to compute the output size
    for layer in features:
        x = layer(x)

    conv_classifier = []
    for layer in classifier:
        if isinstance(layer, nn.Linear):
            # create a convolution equivalent of linear layer
            conv_layer = nn.Conv2d(in_channels=x.size(1),
                                   out_channels=layer.weight.size(0),
                                   kernel_size=(x.size(2), x.size(3)))

            # transfer the weights
            conv_layer.weight.data.view(-1).copy_(layer.weight.data.view(-1))
            conv_layer.bias.data.view(-1).copy_(layer.bias.data.view(-1))
            layer = conv_layer

        x = layer(x)
        conv_classifier.append(layer)

    # replace the model.classifier with newly created convolution layers
    model.classifier = nn.Sequential(*conv_classifier)

    return model

def visualize(model, input_size=(3, 224, 224)):
    '''Visualize the input size though the layers of the model'''
    x = torch.zeros(input_size).unsqueeze_(dim=0)
    print(x.size())
    for layer in list(model.features) + list(model.classifier):
        x = layer(x)
        print(x.size())

This is how the input looks when passed through the model

_vgg = vgg16()
vgg = convolutionize(_vgg, 100)
print('\n\nVGG')
visualize(vgg)

...

VGG
torch.Size([1, 3, 224, 224])
torch.Size([1, 64, 224, 224])
torch.Size([1, 64, 224, 224])
torch.Size([1, 64, 224, 224])
torch.Size([1, 64, 224, 224])
torch.Size([1, 64, 112, 112])
torch.Size([1, 128, 112, 112])
torch.Size([1, 128, 112, 112])
torch.Size([1, 128, 112, 112])
torch.Size([1, 128, 112, 112])
torch.Size([1, 128, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 56, 56])
torch.Size([1, 256, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 28, 28])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 14, 14])
torch.Size([1, 512, 7, 7])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 4096, 1, 1])
torch.Size([1, 100, 1, 1])


Instruction: What are the default values of kernel size and padding in pytorch?

Input:
For an instance, take this piece of code :

conv = conv2d(in_channels = 3, out_channels = 64)

What can I expect the padding and kernel size to be, by default?

Output:
  class torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)

So to answer your question, by default padding=0, there is no default value for kernel_size. If you leave it empty, you will get an error


Instruction: Padding in Pytorch

Input:
In PyTorch tensor, I would like to obtain the output from the input as follows:

How can I achieve this padding in Pytroch?

Output:
One way of doing this is

def my_odd_padding(list_of_2d_tensors, pad_value):
  # get the sizes of the matrices
  hs = [t_.shape[0] for t_ in list_of_2d_tensors]
  ws = [t_.shape[1] for t_ in list_of_2d_tensors]
  # allocate space for output
  result = torch.zeros(sum(hs), sum(ws))
  result.add_(pad_value)
  fh = 0
  fw = 0
  for i, t_ in enumerate(list_of_2d_tensors):
    result[fh:fh+hs[i], fw:fw+ws[i]] = t_
    fh += hs[i]
    fw += ws[i]
  return result

assuming all tensors on list_of_2d_tensors are of the same dtype and on the same device you may explicitly set this dtype and device to result when you allocate it using torch.zeros


Instruction: How to make cuda unavailable in pytorch

Input:
i'm running some code with cudas, and I need to test the same code on CPU to compare running time. To decide between regular pytorch tensor and cuda float tensor, the library I use calls torch.cuda.is_available(). Is there an easy method to make this function return false? I tried changing the Cuda visible devices with

os.environ["CUDA_VISIBLE_DEVICES"]=""

but torch.cuda.is_available() still return True. I went into pytorch source code, and in my case, torch.cuda.is_avaible returns

torch._C._cuda_getDeviceCount() > 0

I assume I should be able to "hide" my GPU at the start of my notebook, so the device count is equal to zero, but i didn't get any success so far. Any help is appreciated :)

Output:
  my code

Instead of trying to trick it, why not rewrite your code? For example,

use_gpu = torch.cuda.is_available() and not os.environ['USE_CPU']

Then you can start your program as python runme.py to run on GPU if available, and USE_CPU=1 python3 runme.py to force CPU execution (or make it semi-permanent by export USE_CPU=1).

  I tried changing the Cuda visible devices with

You can also try running your code with CUDA_VISIBLE_DEVICES="" python3 runme.py; if you're setting the environment variable inside your code, it might be set later than PyTorch initialisation, and have no effect.


Instruction: usage about rnn.pack_padded_sequence

Input:
When I use torch.nn.utils.rnn.pack_padded_sequence(), an error occurred.
Here is my code:

import torch
import numpy as np
x = torch.from_numpy(np.array([[1,2,3,4,5,6,0,0],[6,7,8,9,0,0,0,0],[12,83,84,0,0,0,0,0]]))
length =[6,4,3]
print(torch.nn.utils.rnn.pack_padded_sequence(input=x, lengths=length, batch_first=True))

The error is as follows：

Traceback (most recent call last):
  File "/home/pc/PycharmProjects/padded/padded.py", line 112, in <module>
    print(torch.nn.utils.rnn.pack_padded_sequence(input=x, lengths=length, batch_first=True))
  File "/home/pc/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 67, in wrapper
    if not might_trace(args):
  File "/home/pc/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 141, in might_trace
    first_arg = args[0]
IndexError: tuple index out of range

But when I enter the following code:

import torch
import numpy as np
x = torch.from_numpy(np.array([[1,2,3,4,5,6,0,0],[6,7,8,9,0,0,0,0],[12,83,84,0,0,0,0,0]]))
length =[6,4,3]
print(torch.nn.utils.rnn.pack_padded_sequence(x, lengths=length, batch_first=True))

The result is normal. And I don't know why.

Can you help me solve the problem?

Output:
So the reason for this sounds incredibly stupid, but I think it is due to the way how the wrapper/low-level translator function operates on PyTorch functions:

From what I understand about the *args, **kwargs decorators of python (see more here, the problem is that you only have ordered arguments if you pass them without the =.

Meaning, they get stored in a dictionary of keyword/value pairs. Instead, if we look at the relevant part to our error message (this part: File "/home/pc/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py"), we can see the following:

first_arg = args[0]

meaning that it wants to address this with an index. Unfortunately, since the keyword is only stored and passed down in the key/value dictionary, we cannot address this with the index, and subsequently the error is thrown.

I have so far not been able to figure out a way around this, because this is very abstract and (potentially) used by many different classes, specifically since it gets passed down through the intermediate layer (in line 67, see stack trace).


Instruction: Comparing Conv2D with padding between Tensorflow and PyTorch

Input:
I am trying to import weights saved from a Tensorflow model to PyTorch. So far the results have been very similar. I ran into a snag when the model calls for conv2d with stride=2.

To verify the mismatch, I set up a very simple comparison between TF and PyTorch. First, I compare conv2d with stride=1.

import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
# PyTorch adopts [outputC, inputC, kH, kW]
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow defaults to NHWC
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 1, 1, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=1)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))

The result of this execution is:

Mean of Abs Diff: 2.0443112092038973e-08

When I change stride to 2, the results start to vary.

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=1, stride=2)

The result of this execution is:

Mean of Abs Diff: 0.2104552686214447

According to PyTorch documentation, conv2d uses zero-padding defined by the padding argument. Thus, zeros are added to the left, top, right, and bottom of the input in my example.

If PyTorch simply adds padding on both sides based on the input parameter, it should be easy to replicate in Tensorflow.

# Manually add padding - consistent with PyTorch
paddings = tf.constant([[0, 0], [1, 1], [1, 1], [0, 0]])
x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_tf = tf.pad(x_tf, paddings, "CONSTANT")

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, 2, 2, 1],
                         padding="VALID")

The result of this comparison is:

Mean of Abs Diff: 1.6035047067930464e-08

What this tells me is that if I am somehow able to replicate the default padding behavior from Tensorflow into PyTorch, then my results will be similar.

This question inspected the behavior of padding in Tensorflow. TF documentation explains how padding is added for "SAME" convolutions. I discovered these links while writing this question.

Now that I know the padding strategy of Tensorflow, I can implement it in PyTorch.

Output:
To replicate the behavior, padding sizes are calculated as described in the Tensorflow documentation. Here, I test the padding behavior by setting stride=2 and padding the PyTorch input.

import tensorflow as tf
import numpy as np
import torch
import torch.nn.functional as F

np.random.seed(0)
sess = tf.Session()

# Create random weights and input
weights = torch.empty(3, 3, 3, 8)
torch.nn.init.constant_(weights, 5e-2)
x = np.random.randn(1, 3, 10, 10)

weights_tf = tf.convert_to_tensor(weights.numpy(), dtype=tf.float32)
weights_torch = torch.Tensor(weights.permute((3, 2, 0, 1)))

# Tensorflow padding behavior. Assuming that kH == kW to keep this simple.
stride = 2
if x.shape[2] % stride == 0:
    pad = max(weights.shape[0] - stride, 0)
else:
    pad = max(weights.shape[0] - (x.shape[2] % stride), 0)

if pad % 2 == 0:
    pad_val = pad // 2
    padding = (pad_val, pad_val, pad_val, pad_val)
else:
    pad_val_start = pad // 2
    pad_val_end = pad - pad_val_start
    padding = (pad_val_start, pad_val_end, pad_val_start, pad_val_end)

x_tf = tf.convert_to_tensor(x.transpose((0, 2, 3, 1)), dtype=tf.float32)
x_torch = torch.Tensor(x)
x_torch = F.pad(x_torch, padding, "constant", 0)

# TF Conv2D
tf_conv2d = tf.nn.conv2d(x_tf,
                         weights_tf,
                         strides=[1, stride, stride, 1],
                         padding="SAME")

# PyTorch Conv2D
torch_conv2d = F.conv2d(x_torch, weights_torch, padding=0, stride=stride)

sess.run(tf.global_variables_initializer())
tf_result = sess.run(tf_conv2d)

diff = np.mean(np.abs(tf_result.transpose((0, 3, 1, 2)) - torch_conv2d.detach().numpy()))
print('Mean of Abs Diff: {0}'.format(diff))

The output is:

Mean of Abs Diff: 2.2477470551507395e-08

I wasn't quite sure why this was happening when I started writing this question, but a bit of reading clarified this very quickly. I hope this example can help others.


Instruction: PyTorch: create non-fully-connected layer / concatenate output of hidden layers

Input:
In PyTorch, I want to create a hidden layer whose neurons are not fully connected to the output layer. I try to concatenate the output of two linear layers but run into the following error:

  RuntimeError: size mismatch, m1: [2 x 2], m2: [4 x 4]

my current code:

class NeuralNet2(nn.Module):
    def __init__(self):
        super(NeuralNet2, self).__init__()

        self.input = nn.Linear(2, 40)
        self.hiddenLeft = nn.Linear(40, 2)
        self.hiddenRight = nn.Linear(40, 2)
        self.out = nn.Linear(4, 4)

    def forward(self, x):
        x = self.input(x)
        xLeft, xRight = torch.sigmoid(self.hiddenLeft(x)), torch.sigmoid(self.hiddenRight(x))
        x = torch.cat((xLeft, xRight))
        x = self.out(x)

        return x

I don't get why there is a size mismatch? Is there an alternative way to implement non-fully-connected layers in pytorch?

Output:
It turned out to be a simple comprehension problem with the concatenation function.
Changing
    x = torch.cat((xLeft, xRight))
to
    x = torch.cat((xLeft, xRight), dim=1)
did the trick.
Thanks @dennlinger


Instruction: Bidirectional LSTM output question in PyTorch

Input:
Hi I have a question about how to collect the correct result from a BI-LSTM module’s output.

Suppose I have a 10-length sequence feeding into a single-layer LSTM module with 100 hidden units:

lstm = nn.LSTM(5, 100, 1, bidirectional=True)

output will be of shape:

[10 (seq_length), 1 (batch),  200 (num_directions * hidden_size)]
# or according to the doc, can be viewed as
[10 (seq_length), 1 (batch),  2 (num_directions), 100 (hidden_size)]

If I want to get the 3rd (1-index) input’s output at both directions (two 100-dim vectors), how can I do it correctly?

I know output[2, 0] will give me a 200-dim vector. Does this 200 dim vector represent the output of 3rd input at both directions?

A thing bothering me is that when do reverse feeding, the 3rd (1-index) output vector is calculated from the 8th(1-index) input, right?

Will pytorch automatically take care of this and group output considering direction?

Thanks!

Output:
Yes, when using a BiLSTM the hidden states of the directions are just concatenated (the second part after the middle is the hidden state for feeding in the reversed sequence). So splitting up in the middle works just fine.

As reshaping works from the right to the left dimensions you won't have any problems in separating the two directions.

Here is a small example:

# so these are your original hidden states for each direction
# in this case hidden size is 5, but this works for any size
direction_one_out = torch.tensor(range(5))
direction_two_out = torch.tensor(list(reversed(range(5))))
print('Direction one:')
print(direction_one_out)
print('Direction two:')
print(direction_two_out)

# before outputting they will be concatinated
# I'm adding here batch dimension and sequence length, in this case seq length is 1
hidden = torch.cat((direction_one_out, direction_two_out), dim=0).view(1, 1, -1)
print('\nYour hidden output:')
print(hidden, hidden.shape)

# trivial case, reshaping for one hidden state
hidden_reshaped = hidden.view(1, 1, 2, -1)
print('\nReshaped:')
print(hidden_reshaped, hidden_reshaped.shape)

# This works as well for abitrary sequence lengths as you can see here
# I've set sequence length here to 5, but this will work for any other value as well
print('\nThis also works for more multiple hidden states in a tensor:')
multi_hidden = hidden.expand(5, 1, 10)
print(multi_hidden, multi_hidden.shape)
print('Directions can be split up just like this:')
multi_hidden = multi_hidden.view(5, 1, 2, 5)
print(multi_hidden, multi_hidden.shape)

Output:

Direction one:
tensor([0, 1, 2, 3, 4])
Direction two:
tensor([4, 3, 2, 1, 0])

Your hidden output:
tensor([[[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]]]) torch.Size([1, 1, 10])

Reshaped:
tensor([[[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]]]) torch.Size([1, 1, 2, 5])

This also works for more multiple hidden states in a tensor:
tensor([[[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]],

        [[0, 1, 2, 3, 4, 4, 3, 2, 1, 0]]]) torch.Size([5, 1, 10])
Directions can be split up just like this:
tensor([[[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]],

        [[[0, 1, 2, 3, 4],
          [4, 3, 2, 1, 0]]]]) torch.Size([5, 1, 2, 5])

Hope this helps! :)


Instruction: AttributeError: 'tuple' object has no attribute 'dim', when feeding input to Pytorch LSTM network

Input:
I am trying to run the following code:

import matplotlib.pylab as plt
import numpy as np
import torch
import torch.nn as nn

class LSTM(nn.Module):
    def __init__(self, input_shape, n_actions):
        super(LSTM, self).__init__()

        self.lstm = nn.LSTM(input_shape, 12)
        self.hidden2tag = nn.Linear(12, n_actions)

    def forward(self, x):
        out = self.lstm(x)
        out = self.hidden2tag(out)
        return out

state = [(1,2,3,4,5),(2,3,4,5,6),(3,4,5,6,7),(4,5,6,7,8),(5,6,7,8,9),(6,7,8,9,0)]

device = torch.device("cuda")
net = LSTM(5, 3).to(device)

state_v = torch.FloatTensor(state).to(device)

q_vals_v = net(state_v.view(1, state_v.shape[0], state_v.shape[1]))
_, action = int(torch.max(q_vals_v, dim=1).item())

And that returns this error:

Traceback (most recent call last):
  File "/home/dikkerj/Documents/PycharmProjects/LSTMReactor/QuestionStackoverflow.py", line 26, in <module>
    q_vals_v = net(state_v.view(1, state_v.shape[0], state_v.shape[1]))
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/dikkerj/Documents/PycharmProjects/LSTMReactor/QuestionStackoverflow.py", line 15, in forward
    out = self.hidden2tag(out)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/dikkerj/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 1022, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

Anyone knows how to fix this? (to get rid of the tensor being a tuple so that it can be fed into the LSTM network)

Output:
The pytorch LSTM returns a tuple. So you get this error as your linear layer self.hidden2tag can not handle this tuple.

So change:

out = self.lstm(x)

to

out, states = self.lstm(x)

This will fix your error, by splitting up the tuple so that out is just your output tensor.

out then stores the hidden states, while states is another tuple that contains the last hidden and cell state.

You can also take a look here:
https://pytorch.org/docs/stable/nn.html#torch.nn.LSTM

You will get another error for the last line as max() returns a tuple as well. But this should be easy to fix and is yet different error :)


Instruction: How to create a torchtext.data.TabularDataset directly from a list or dict

Input:
torchtext.data.TabularDataset can be created from a TSV/JSON/CSV file and then it can be used for building the vocabulary from Glove, FastText or any other embeddings. But my requirement is to create a torchtext.data.TabularDataset directly, either from a list or a dict.

Current implementation of the code by reading TSV files

self.RAW = data.RawField()
self.TEXT = data.Field(batch_first=True)
self.LABEL = data.Field(sequential=False, unk_token=None)

self.train, self.dev, self.test = data.TabularDataset.splits(
    path='.data/quora',
    train='train.tsv',
    validation='dev.tsv',
    test='test.tsv',
    format='tsv',
    fields=[('label', self.LABEL),
            ('q1', self.TEXT),
            ('q2', self.TEXT),
            ('id', self.RAW)])

self.TEXT.build_vocab(self.train, self.dev, self.test, vectors=GloVe(name='840B', dim=300))
self.LABEL.build_vocab(self.train)

sort_key = lambda x: data.interleave_keys(len(x.q1), len(x.q2))

self.train_iter, self.dev_iter, self.test_iter = \
    data.BucketIterator.splits((self.train, self.dev, self.test),
                               batch_sizes=[args.batch_size] * 3,
                               device=args.gpu,
                               sort_key=sort_key)

This is the current working code for reading data from a file. So in order to create the dataset directly from a List/Dict I tried inbuilt functions like Examples.fromDict or Examples.fromList but then while coming to the last for loop, it throws an error that AttributeError: 'BucketIterator' object has no attribute 'q1'

Output:
It required me to write an own class inheriting the Dataset class and with few modifications in torchtext.data.TabularDataset class.

class TabularDataset_From_List(data.Dataset):

    def __init__(self, input_list, format, fields, skip_header=False, **kwargs):
        make_example = {
            'json': Example.fromJSON, 'dict': Example.fromdict,
            'tsv': Example.fromTSV, 'csv': Example.fromCSV}[format.lower()]

        examples = [make_example(item, fields) for item in input_list]

        if make_example in (Example.fromdict, Example.fromJSON):
            fields, field_dict = [], fields
            for field in field_dict.values():
                if isinstance(field, list):
                    fields.extend(field)
                else:
                    fields.append(field)

        super(TabularDataset_From_List, self).__init__(examples, fields, **kwargs)

    @classmethod
    def splits(cls, path=None, root='.data', train=None, validation=None,
               test=None, **kwargs):
        if path is None:
            path = cls.download(root)
        train_data = None if train is None else cls(
            train, **kwargs)
        val_data = None if validation is None else cls(
            validation, **kwargs)
        test_data = None if test is None else cls(
            test, **kwargs)
        return tuple(d for d in (train_data, val_data, test_data)
                     if d is not None)


Instruction: Modifying a pytorch tensor and then getting the gradient lets the gradient not work

Input:
I am a beginner in pytorch and I face the following issue:

When I get the gradient of the below tensor (note that I use some variable x in some way as you can see below), I get the gradient:

import torch
myTensor = torch.randn(2, 2,requires_grad=True)
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)

Now, if I try to modify an element of myTensor, I get the error of leaf variable has been moved into the graph interior. See this code:

import torch
myTensor = torch.randn(2, 2,requires_grad=True)
myTensor[0,0]*=5
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()
print(myTensor.grad)

What is wrong with my latter code? And how do I correct it?

Any help would be highly appreciated. Thanks a lot!

Output:
The problem here is that this line represents an in-place operation:

myTensor[0,0]*=5

And PyTorch or more precisely autograd is not very good in handling in-place operations, especially on those tensors with the requires_grad flag set to True.

You can also take a look here:
https://pytorch.org/docs/stable/notes/autograd.html#in-place-operations-with-autograd

Generally you should avoid in-place operations where it is possible, in some cases it can work, but you should always avoid in-place operations on tensors where you set requires_grad to True.

Unfortunately there are not many pytorch functions to help out on this problem. So you would have to use a helper tensor to avoid the in-place operation in this case:

Code:

import torch

myTensor = torch.randn(2, 2,requires_grad=True)
helper_tensor = torch.ones(2, 2)
helper_tensor[0, 0] = 5
new_myTensor = myTensor * helper_tensor # new tensor, out-of-place operation
with torch.enable_grad():
    x=new_myTensor.sum() *10 # of course you need to use the new tensor
x.backward()                 # for further calculation and backward
print(myTensor.grad)

Output:

tensor([[50., 10.],
        [10., 10.]])

Unfortunately this is not very nice and I would appreciate if there would be a better or nicer solution out there. But for all I know in the current version (0.4.1) you will have to got with this workaround for tensors with gradient resp. requires_grad=True.

Hopefully for future versions there will be a better solution.

Btw. if you activate the gradient later you can see that it works just fine:

import torch
myTensor = torch.randn(2, 2,requires_grad=False) # no gradient so far
myTensor[0,0]*=5                                 # in-place op not included in gradient
myTensor.requires_grad = True                    # activate gradient here
with torch.enable_grad():
    x=myTensor.sum() *10
x.backward()                                     # no problem here
print(myTensor.grad)

But of course this will yield to a different result:

tensor([[10., 10.],
        [10., 10.]])

Hope this helps!


Instruction: Why does Pytorch expect a DoubleTensor instead of a FloatTensor?

Input:
From everything I see online, FloatTensors are Pytorch's default for everything, and when I create a tensor to pass to my generator module it is a FloatTensor, but when I try to run it through a linear layer it complains that it wants a DoubleTensor.

class Generator(nn.Module):
  def __init__(self):
    super(Generator, self).__init__()
    self.fully_connected = nn.Linear(100, 1024*4*4, bias=False)

  def forward(self, zvec):
    print(zvec.size())
    fc = self.fully_connected(zvec)
    return(fc.size())

gen = Generator();

gen(torch.from_numpy(np.random.normal(size=100)))

Which produces

RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'mat2'

Output:
The Problem here is that your numpy input uses double as data type the same data type is also applied on the resulting tensor.

The weights of your layer self.fully_connected on the other hand are float. When feeding data trough the layer a matrix multiplication is applied and this multiplication requires both matrices to be of same data type.

So you have two solutions:

You can convert your input to float:

By changing:

gen(torch.from_numpy(np.random.normal(size=100)))

To:

gen(torch.from_numpy(np.random.normal(size=100)).float())

Your input which is fed into gen will be converted to float then.

Full working code for converting inputs:

from torch import nn
import torch
import numpy as np
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.fully_connected = nn.Linear(100, 1024*4*4, bias=False)

    def forward(self, zvec):
        print(zvec.size())
        fc = self.fully_connected(zvec)
        return(fc.size())

gen = Generator();
gen(torch.from_numpy(np.random.normal(size=100)).float()) # converting network input to float

Or alternatively you can convert your layer weights to double:

If you need the double precision you can also convert your weights to double.

Change this line:

self.fully_connected = nn.Linear(100, 1024*4*4, bias=False)

Just to:

self.fully_connected = nn.Linear(100, 1024*4*4, bias=False).double()

Full working code for converting weights:

from torch import nn
import torch
import numpy as np
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.fully_connected = nn.Linear(100, 1024*4*4, bias=False).double() # converting layer weights to double()

    def forward(self, zvec):
        print(zvec.size())
        fc = self.fully_connected(zvec)
        return(fc.size())

gen = Generator();
gen(torch.from_numpy(np.random.normal(size=100)))

So both ways should work for you, but if you don't need the extra precision of double you should go with float as double requires more computational power.


Instruction: what is the tensorflow equivalent for pytorch probability function: torch.bernoulli?

Input:
In Pytorch, you can do following:
x = torch.bernoulli(my_data)

Any similar functionality in tensorflow? Can the input be 2-D tensor, such as (batch, len)?
I tried tensorflow.contrib.distributions.Bernoulli:
import numpy as np
tmp_x1 = np.random.rand(20,5)
new_data_2 = tf.convert_to_tensor(tmp_x1)
from tensorflow.contrib.distributions import  Bernoulli
tmp2_x1 = Bernoulli(probs=new_data_2)

got error:
return math_ops.log(probs) - math_ops.log1p(-1. * probs), probs
TypeError: unsupported operand type(s) for *: 'float' and 'Tensor'

Output:
It seems tf.distributions.Bernoulli does what you need.  The input can be an N-D tensor, which includes a 2D tensor.

EDIT: example use

After your comment, I tried the following, which worked for me (using tensorflow 1.11):

import numpy as np
import tensorflow
import tensorflow as tf
from tensorflow.distributions import  Bernoulli

tmp_x1 = np.random.rand(20,5)
new_data_2 = tf.convert_to_tensor(tmp_x1)
tmp2_x1 = Bernoulli(probs=new_data_2)
sess = tf.Session()
print sess.run(tmp2_x1.sample())


Instruction: Pytorch: Modifying VGG16 Architecture

Input:
I'm currently trying to modify the VGG16 network architecture so that it's able to accept 400x400 px images.

Based on literature that I've read, the way to do it would be to covert the fully connected (FC) layers into convolutional (CONV) layers. This would essentially " allow the network to efficiently “slide” across a larger input image and make multiple evaluations of different parts of the image, incorporating all available contextual information." Afterwards, an Average Pooling layer is used to "average the multiple feature vectors into a single feature vector that summarizes the input image".

I've done this using this function, and have come up with the following network architecture:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1         [-1, 64, 400, 400]           1,792
              ReLU-2         [-1, 64, 400, 400]               0
            Conv2d-3         [-1, 64, 400, 400]          36,928
              ReLU-4         [-1, 64, 400, 400]               0
         MaxPool2d-5         [-1, 64, 200, 200]               0
            Conv2d-6        [-1, 128, 200, 200]          73,856
              ReLU-7        [-1, 128, 200, 200]               0
            Conv2d-8        [-1, 128, 200, 200]         147,584
              ReLU-9        [-1, 128, 200, 200]               0
        MaxPool2d-10        [-1, 128, 100, 100]               0
           Conv2d-11        [-1, 256, 100, 100]         295,168
             ReLU-12        [-1, 256, 100, 100]               0
           Conv2d-13        [-1, 256, 100, 100]         590,080
             ReLU-14        [-1, 256, 100, 100]               0
           Conv2d-15        [-1, 256, 100, 100]         590,080
             ReLU-16        [-1, 256, 100, 100]               0
        MaxPool2d-17          [-1, 256, 50, 50]               0
           Conv2d-18          [-1, 512, 50, 50]       1,180,160
             ReLU-19          [-1, 512, 50, 50]               0
           Conv2d-20          [-1, 512, 50, 50]       2,359,808
             ReLU-21          [-1, 512, 50, 50]               0
           Conv2d-22          [-1, 512, 50, 50]       2,359,808
             ReLU-23          [-1, 512, 50, 50]               0
        MaxPool2d-24          [-1, 512, 25, 25]               0
           Conv2d-25          [-1, 512, 25, 25]       2,359,808
             ReLU-26          [-1, 512, 25, 25]               0
           Conv2d-27          [-1, 512, 25, 25]       2,359,808
             ReLU-28          [-1, 512, 25, 25]               0
           Conv2d-29          [-1, 512, 25, 25]       2,359,808
             ReLU-30          [-1, 512, 25, 25]               0
        MaxPool2d-31          [-1, 512, 12, 12]               0
           Conv2d-32           [-1, 4096, 1, 1]     301,993,984
             ReLU-33           [-1, 4096, 1, 1]               0
          Dropout-34           [-1, 4096, 1, 1]               0
           Conv2d-35           [-1, 4096, 1, 1]      16,781,312
             ReLU-36           [-1, 4096, 1, 1]               0
          Dropout-37           [-1, 4096, 1, 1]               0
           Conv2d-38              [-1, 3, 1, 1]          12,291
AdaptiveAvgPool2d-39              [-1, 3, 1, 1]               0
          Softmax-40              [-1, 3, 1, 1]               0
================================================================
Total params: 333,502,275
Trainable params: 318,787,587
Non-trainable params: 14,714,688
----------------------------------------------------------------
Input size (MB): 1.83
Forward/backward pass size (MB): 696.55
Params size (MB): 1272.21
Estimated Total Size (MB): 1970.59
----------------------------------------------------------------

My question is simple: Is the use of the average pooling layer at the end necessary? It seems like by the last convolutional layer, we get a 1x1 image with 3 channels. Doing an average pooling on that would seem to not have any effect.

If there is anything amiss in my logic/ architecture, kindly feel free to point it out.
Thanks!

Output:
  How to convert VGG to except input size of 400 x 400 ?

First Approach

The problem with VGG style architecture is we are hardcoding the number of input & output features in our Linear Layers.
i.e

vgg.classifier[0]: Linear(in_features=25088, out_features=4096, bias=True)

It is expecting 25,088 input features.

If we pass an image of size (3, 224, 224)
through vgg.features the output feature map will be of dimensions:

(512, 7, 7) => 512 * 7 * 7 => 25,088

If we change the input image size to (3, 400, 400) and pass
through vgg.features the output feature map will be of dimensions:

(512, 12, 12) => 512 * 12 * 12 =>  73,728

throws `sizemismatch` error.

One way to fix this issue is by using nn.AdaptiveAvgPool in place of nn.AvgPool. AdaptiveAvgPool helps to define the output size of the layer which remains constant irrespective of the size of the input through the vgg.features layer.

for eg:

vgg.features[30] = nn.AdaptiveAvgPool(output_size=(7,7))

will make sure the final feature maps have a dimension of `(512, 7, 7)`
irrespective of the input size.

You can read more about Adaptive Pooling in here.

Second Approach

If you use the technique here to convert your Linear layers to Convolutional Layers, you don't have to worry about the input dimension, however you have to change the weight initialisation techniques because of the change in number of parameters.

  Is the use of the average pooling layer at the end necessary?

No, in this case. It is not changing the size of the input feature map, hence it not doing an average over a set of nodes.


Instruction: Set higher shared memory to avoid RuntimeError with PyTorch on Google Colab

Input:
Using pytorch 1.0 Preview with fastai v1.0 in Colab.

I often get RuntimeError: DataLoader worker (pid 13) is killed by signal: Bus error. for more memory intensive tasks (nothing huge).

Looks like a shared memory issue:
https://github.com/pytorch/pytorch/issues/5040#issue-294274594

Fix looks like it is to change shared memory in the docker container:

https://github.com/pytorch/pytorch/issues/2244#issuecomment-318864552

  Looks like the shared memory of the docker container wasn't set high enough. Setting a higher amount by adding --shm-size 8G to the docker run command seems to be the trick as mentioned here.

How can I increase the shared memory of the docker container running in Colab or otherwise avoid this error?

Output: It's not possible to modify this setting in colab, but the default was raised to fix this issue already so you should not need to change the setting further: https://github.com/googlecolab/colabtools/issues/329


Instruction: How to learn the embeddings in Pytorch and retrieve it later

Input: I am building a recommendation system where I predict the best item for each user given their purchase history of items. I have userIDs and itemIDs and how much itemID was purchased by userID. I have Millions of users and thousands of products. Not all products are purchased(there are some products that no one has bought them yet). Since the users and items are big I don't want to use one-hot vectors. I am using pytorch and I want to create and train the embeddings so that I can make the predictions for each user-item pair. I followed this tutorial https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html. If it's an accurate assumption that the embedding layer is being trained, then do I retrieve the learned weights through model.parameters() method or should I use the embedding.data.weight option?

Output:
model.parameters() returns all the parameters of your model, including the embeddings.

So all these parameters of your model are handed over to the optimizer (line below) and will be trained later when calling optimizer.step() - so yes your embeddings are trained along with all other parameters of the network.(you can also freeze certain layers by setting i.e. embedding.weight.requires_grad = False, but this is not the case here).

# summing it up:
# this line specifies which parameters are trained with the optimizer
# model.parameters() just returns all parameters
# embedding class weights are also parameters and will thus be trained
optimizer = optim.SGD(model.parameters(), lr=0.001)

You can see that your embedding weights are also of type Parameter by doing so:

import torch
embedding_maxtrix = torch.nn.Embedding(10, 10)
print(type(embedding_maxtrix.weight))

This will output the type of the weights, which is Parameter:

<class 'torch.nn.parameter.Parameter'>

I'm not entirely sure what mean by retrieve. Do you mean getting a single vector, or do you want just the whole matrix to save it, or do something else?

embedding_maxtrix = torch.nn.Embedding(5, 5)
# this will get you a single embedding vector
print('Getting a single vector:\n', embedding_maxtrix(torch.LongTensor([0])))
# of course you can do the same for a seqeunce
print('Getting vectors for a sequence:\n', embedding_maxtrix(torch.LongTensor([1, 2, 3])))
# this will give the the whole embedding matrix
print('Getting weights:\n', embedding_maxtrix.weight.data)

Output:

Getting a single vector:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020]], grad_fn=<EmbeddingBackward>)
Getting vectors for a sequence:
 tensor([[ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502]],
       grad_fn=<EmbeddingBackward>)
Getting weights:
 tensor([[-0.0144, -0.6245,  1.3611, -1.0753,  0.5020],
        [ 0.9277, -0.1879, -1.4999,  0.2895,  0.8367],
        [-0.1167, -2.2139,  1.6918, -0.3483,  0.3508],
        [ 2.3763, -1.3408, -0.9531,  2.2081, -1.5502],
        [-0.5829, -0.1918, -0.8079,  0.6922, -0.2627]])

I hope this answers your question, you can also take a look at the documentation, there you can find some useful examples as well.

https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding


Instruction: Error while running Convolutional Autoencoder RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Input:
I am a noob and am creating a model in PyTorch for the first time. I am trying to create a convolutional autoencoder and am getting the error while running the model. The code I am using is:

class MyDataset(Dataset):
    def __init__(self, image_paths, target_paths, train=True):
        self.image_paths = image_paths
        self.target_paths = target_paths

    def transform(self, image, target):
        # Transform to tensor
        resize = transforms.Resize(size=(2350,1650))
        image = resize(image)
        target = resize(target)
        grayscale = transforms.Grayscale(1)
        image = grayscale(image)
        target = grayscale(target)
        image = TF.to_tensor(image)
        target = TF.to_tensor(target)
        return image, target

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        target = Image.open(self.target_paths[index])
        x, y = self.transform(image, target)
        return x, y

    def __len__(self):
        return len(self.image_paths)

traindata = MyDataset(image_paths=train_data, target_paths=target_data, train=True)
testdata = MyDataset(image_paths=test_data, target_paths=None, train=False)

train_loader = DataLoader(traindata, batch_size=1, shuffle=True, num_workers=4)
test_loader = DataLoader(testdata, batch_size=1, shuffle=False, num_workers=4)

class ConvolutionalAutoEncoder(nn.Module):
    def __init__(self):
        super(ConvolutionalAutoEncoder, self).__init__()
        self.encoder_block1 = nn.Sequential(
            nn.Conv2d(1, 64, 3, stride=1, padding=1),
            nn.ReLU(True),
            nn.Conv2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True)
        )
        self.decoder_block1 = nn.Sequential(
            nn.ConvTranspose2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True)
         )
        self.decoder_block0 = nn.Sequential(
            nn.ConvTranspose2d(64, 1, 3, stride=1, padding=1),
            nn.Sigmoid()
        )
    def forward(self, x):
        x1 = self.encoder_block1(x)
        y1 = self.decoder_block1(x1)
        y0 = self.decoder_block0(y1)
        return x

device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
print(device)

model = ConvolutionalAutoEncoder().to(device)
# Loss and optimizer
learning_rate = 0.001
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

params = list(model.parameters())
print(len(params))
print(params[0].size())  # conv1's .weight

num_epochs = 30
total_step = len(train_loader)
for epoch in range(num_epochs):
    for batch_idx, data in enumerate(train_loader):
        inp, targ = data
        inp = inp.to(device)
        targ = targ.to(device)

        output = model(inp)
        loss = criterion(output, targ)

        model.zero_grad()
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        if (batch_idx+1) % 10 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

The full error I am getting is:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-28fa0c94d845> in <module>
     13
     14         model.zero_grad()
---> 15         loss.backward()
     16         optimizer.step()
     17

~/anaconda3/envs/gautam_new/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

~/anaconda3/envs/gautam_new/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91
     92

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Please help. Also, if possible also advice on how I can make my model deeper. I keep getting CUDA  out of memory error.

Thanks.

Output:
I cannot test your model, but considering the error message it makes sense the cause of your problem lies in the return value of your forward.

Currently you are returning x which is your actual input not the output:

def forward(self, x):
    x1 = self.encoder_block1(x)
    y1 = self.decoder_block1(x1)
    y0 = self.decoder_block0(y1)
    return x

So to return the output you might want to change the return value form x to y0:

def forward(self, x):
    x1 = self.encoder_block1(x)
    y1 = self.decoder_block1(x1)
    y0 = self.decoder_block0(y1)
    return y0

About the memory: Please don't put too many issues in one question. Imagine you have three completely different issues in one question, and there are three people out there where each of them is able to solve one of your issues, you might end up with no answer at all. Because none of them is able to give you a complete answer addressing all of these issues. But if your split your issues into three questions, you might get just three answers, solving all your issues. In many cases it also can improve the question, because one can be more specific to the problem without writing an entire novel in the question.Of course if your issues are very related you can put them into one question, but this doesn't seem to be the case here. I guess there is still a slight chance your forward function had some side effects leading to the memory problem (wild guess - not sure at all about this). So if you're lucky it might solve your memory problem too, but if not you definitely should open a new question about it.


Instruction: coverting roi pooling in pytorch to nn layer

Input:
I have a an mlmodel  using ROI pooling for which I am using this
(adapted from here) (non NN layer version)

def forward(self, features, rois):
        batch_size, num_channels, data_height, data_width = features.size()
        num_rois = rois.size()[0]
        outputs = Variable(torch.zeros(num_rois, num_channels, self.pooled_height, self.pooled_width)).cuda()

        for roi_ind, roi in enumerate(rois):
            batch_ind = int(roi[0].data[0])
            roi_start_w, roi_start_h, roi_end_w, roi_end_h = np.round(
                roi[1:].data.cpu().numpy() * self.spatial_scale).astype(int)
            roi_width = max(roi_end_w - roi_start_w + 1, 1)
            roi_height = max(roi_end_h - roi_start_h + 1, 1)
            bin_size_w = float(roi_width) / float(self.pooled_width)
            bin_size_h = float(roi_height) / float(self.pooled_height)

            for ph in range(self.pooled_height):
                hstart = int(np.floor(ph * bin_size_h))
                hend = int(np.ceil((ph + 1) * bin_size_h))
                hstart = min(data_height, max(0, hstart + roi_start_h))
                hend = min(data_height, max(0, hend + roi_start_h))
                for pw in range(self.pooled_width):
                    wstart = int(np.floor(pw * bin_size_w))
                    wend = int(np.ceil((pw + 1) * bin_size_w))
                    wstart = min(data_width, max(0, wstart + roi_start_w))
                    wend = min(data_width, max(0, wend + roi_start_w))

                    is_empty = (hend <= hstart) or(wend <= wstart)
                    if is_empty:
                        outputs[roi_ind, :, ph, pw] = 0
                    else:
                        data = features[batch_ind]
                        outputs[roi_ind, :, ph, pw] = torch.max(
                            torch.max(data[:, hstart:hend, wstart:wend], 1)[0], 2)[0].view(-1)

        return outputs

I want to convert the pytorch model to caffe and hence I need to convert the above to an NN layer for which I am using the below (adapted from here)

def forward(self, input, rois):
    output = []
    rois = rois.data.float()
    num_rois = rois.size(0)

    rois[:,1:].mul_(self.spatial_scale)
    rois = rois.long()
    for i in range(num_rois):
        roi = rois[i]
        im_idx = roi[0]
        im = input.narrow(0, im_idx, 1)[..., roi[2]:(roi[4]+1), roi[1]:(roi[3]+1)]
        op = nn.functional.adaptive_max_pool2d(input = im, output_size = self.size)
        output.append(op)

    return torch.cat(tuple(output), dim=0)

The output returned doesn't seem to match for the above methods even though they are performing the same function. I seem to hit a deadlock. Could anyone point if I am making any obvious mistake in the above ?

Output:
Found the issue -
The rois after multiplication with spatial scale were being rounded down and had to call round function before calling long like so

rois = rois.data.float()
num_rois = rois.size(0)

rois[:,1:].mul_(self.spatial_scale)
rois = rois.round().long() ## Check this here !!

Hope this helps someone!


Instruction: pytorch freeze weights and update param_groups

Input:
Freezing weights in pytorch for param_groups setting.

So if one wants to freeze weights during training:

for param in child.parameters():
    param.requires_grad = False

the optimizer also has to be updated to not include the non gradient weights:

optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=opt.lr, amsgrad=True)

If one wants to use different weight_decay / learning rates for bias and weights/this also allows for differing learning rates:

param_groups = [{'params': model.module.bias_parameters(), 'weight_decay': args.bias_decay},
                {'params': model.module.weight_parameters(), 'weight_decay': args.weight_decay}]

param_groups a list of dics is defined and passed into SGD as follows:

optimizer = torch.optim.Adam(param_groups, args.lr,
                                 betas=(args.momentum, args.beta))

How can this be achieved with freezing individual weights? Running filter over a list of dics or is there a way of adding tensors to the optimizer separately?

Output:
Actually I think you don't have to update the optimizer. The Parameters handed over to the optimizer are just references.

So when you change the requires_grad flag it will immediately be updated.

But even if that would for some reason not be the case - as soon as you you set the requires_grad flag to be False you cannot anymore calculate gradients any new gradients (see at the bottom with None and zero gradients) for this weight, so the gradient won't change anymore and if you use optimizer.zero_grad() it will just stay zero.

So if there is no gradient, then there is also no need to exclude these from the optimizer. Because without gradient the optimizer will just do nothing, no matter what learning rate you use.

Here is an small example to show this behaviour:

import torch
import torch.nn as nn
import torch.optim as optim

n_dim = 5

p1 = nn.Linear(n_dim, 1)
p2 = nn.Linear(n_dim, 1)

optimizer = optim.Adam(list(p1.parameters())+list(p2.parameters()))
p2.weight.requires_grad = False
for i in range(4):
    dummy_loss = (p1(torch.rand(n_dim)) + p2(torch.rand(n_dim))).squeeze()
    optimizer.zero_grad()
    dummy_loss.backward()
    optimizer.step()
    print('p1: requires_grad =', p1.weight.requires_grad, ', gradient:', p1.weight.grad)
    print('p2: requires_grad =', p2.weight.requires_grad, ', gradient:', p2.weight.grad)
    print()

    if i == 1:
        p1.weight.requires_grad = False
        p2.weight.requires_grad = True

Output:

p1: requires_grad = True , gradient: tensor([[0.8522, 0.0020, 0.1092, 0.8167, 0.2144]])
p2: requires_grad = False , gradient: None

p1: requires_grad = True , gradient: tensor([[0.7635, 0.0652, 0.0902, 0.8549, 0.6273]])
p2: requires_grad = False , gradient: None

p1: requires_grad = False , gradient: tensor([[0., 0., 0., 0., 0.]])
p2: requires_grad = True , gradient: tensor([[0.1343, 0.1323, 0.9590, 0.9937, 0.2270]])

p1: requires_grad = False , gradient: tensor([[0., 0., 0., 0., 0.]])
p2: requires_grad = True , gradient: tensor([[0.0100, 0.0123, 0.8054, 0.9976, 0.6397]])

Here you can see that no gradients are calculated. You may have notice the gradient for p2 is None at the beginning and later on it is tensor([[0., 0., 0., 0., 0.]]) for p1 instead of None after deactivating gradients.

This is the case because p1.weight.grad is just a variable which is modified by backward() and optimizer.zero_grad().

So at the beginning p1.weight.grad is just initialized with None, after the gradients are written or accumulated to this variable they won't be cleared automatically. But because optimizer.zero_grad() is called they are set to zero and stay like this since backward() cannot anymore calculate new gradients with requires_grad=False.

You can also change the code in the if-statement to:

if i == 1:
    p1.weight.requires_grad = False
    p1.weight.grad = None
    p2.weight.requires_grad = True

So once reset to None they are left untouched and stay None:

p1: requires_grad = True , gradient: tensor([[0.2375, 0.7528, 0.1501, 0.3516, 0.3470]])
p2: requires_grad = False , gradient: None

p1: requires_grad = True , gradient: tensor([[0.5181, 0.5178, 0.6590, 0.6950, 0.2743]])
p2: requires_grad = False , gradient: None

p1: requires_grad = False , gradient: None
p2: requires_grad = True , gradient: tensor([[0.4797, 0.7203, 0.2284, 0.9045, 0.6671]])

p1: requires_grad = False , gradient: None
p2: requires_grad = True , gradient: tensor([[0.8344, 0.1245, 0.0295, 0.2968, 0.8816]])

I hope this makes sense to you!