Instruction: PyTorch is installed but not imported

Input:
I am trying to build PyTorch.
Reference site：https://github.com/hughperkins/pytorch

but, When we performed unit test, The following error occur.

ImportError while importing test module '/home/usr2/pytorch/test/testByteTensor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/testByteTensor.py:2: in <module>
    import PyTorch
E   ImportError: No module named 'PyTorch'
__________________ ERROR collecting test/testDoubleTensor.py ___________________

I understand that PyTorch is not imported.
but It is confirmed that pytorch is installed.
Is there a way to solve this problem?

environment

ubuntu 16.04
python3.5
cuda9.2

Output:
In fact, you should do import torch instead of import PyTorch
Here is what's work for me: (I installed it using conda)

>>> import torch

>>> torch.version
>>> <module 'torch.version' from '/home/koke_cacao/miniconda3/envs/ml/lib/python3.6/site-packages/torch/version.py'>

>>> print(torch.__version__)
>>> 0.4.1.post2

>>> a = torch.FloatTensor(2,3)
>>> tensor([[-7.4368e-13,  3.0911e-41, -9.6122e-13],
    [ 3.0911e-41, -7.3734e-13,  3.0911e-41]])

Edit: The version works with no problem at all for me. But if you insist to perform the unit test, maybe other people can solve your problem.


Instruction: Question about terminology used in LSTM inputs - seq_length vs context_size for sliding window approach

Input:
I have time series of a sequence of n vectors that I need to feed to a LSTM with a sliding window approach.

In different resources that I read online, seq_length is often referred to as the window length [or the number of LSTM cells] and context_size is defined as the input size to LSTM at a timestep. (for eg. one input vector at a time step).

At any given point of time t, I want to pass points x_{t-m}, …, x_{t-1},…,x_{t} to LSTM followed by a dense layer and predict a categorical target attribute at every tilmestep.

If I want to follow the sliding window approach, do I need to explicitly split input data into windows of size m ?

For instance:

[x_{t-m}, …, x_{t-1},…,x_{t}], [x_{t-m+1}, …, x_{t},…,x_{t+1}], [x_{t-m+2}, …, x_{t+1}, x_{t+2}] etc.

Or, can I split the input data into non-overlapping chunks [x_{t-m}, …, x_{t-1},…,x_{t}], [x_{t+1}, …, x_{t+m-1},x_{t+m}] etc. and instead resize the context_size ?

     embeds = embeds.unfold(1, context_size, 1)  # Keeping the step size to be 1
     embeds = embeds.view(embeds.size(0), embeds.size(1), -1)

Is there a better way to implement sliding window approach for timeseries data ?

Output: LSTM by default recurses on data, in that the prediction at time t will depend on all of the past(depending on memory). In this case it seems like you want input at time t to depend on m+1 instances into the past. You do not need a recurrent net if so, you can simple use linear and feed in the sliding window at each instant. However, if you're using a recurrent net, you don't need to pass the same inputs again and again.


Instruction: RuntimeError: Attempting to deserialize object on CUDA device 2 but torch.cuda.device_count() is 1

Input:
I've got a snippet of python code for training a model. The problem is that after running:

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname)

to load a pretrained model,I'm getting:

  Traceback (most recent call last):
  File "img_to_text.py", line 480, in <module>
    main()
  File "img_to_text.py", line 475, in main
    r = setup_test()
  File "img_to_text.py", line 259, in setup_test
    s2s_data = s2s.setup_test()
  File "/media/ahrzb/datasets/notebooks/mzh/SemStyle/semstyle/code/seq2seq_pytorch.py", line 220, in setup_test
    loaded_state= torch.load(model_path+seq_to_seq_test_model_fname)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 358, in load
    return _load(f, map_location, pickle_module)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 542, in _load
    result = unpickler.load()
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 508, in persistent_load
    data_type(size), location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 372, in restore_location
    return default_restore_location(storage, location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 104, in default_restore_location
    result = fn(storage, location)
  File "/home/ahrzb/.pyenv/versions/2.7.15/envs/mzh2.7/lib/python2.7/site-packages/torch/serialization.py", line 85, in _cuda_deserialize
    device, torch.cuda.device_count()))

I think this is because they have trained the model on two GPUs and I need to load it in one GPU. I changed this line:

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname)

to

loaded_state = torch.load(model_path+seq_to_seq_test_model_fname, map_location={'cuda:1': 'cuda:0'} )

in order to map data of cuda 1 to cuda 0 but it did not work.

Output:
I just figured it out:

 loaded_state = torch.load(model_path+seq_to_seq_test_model_fname,map_location='cuda:0')

is the solution


Instruction: Pytorch passing PackSequence argument to LSTM

Input:
As per my understanding, pack_sequence and pack_padded_sequence returns a PackedSequence, for which its data attribute should always be 1 dimension.

However, the following code gives the error:
RuntimeError: input must have 2 dimensions, got 1.

import torch
import torch.nn.utils.rnn as rnn_utils
a = torch.Tensor([1, 2, 3])
b = torch.Tensor([4, 5])
c = torch.Tensor([6])
seq = rnn_utils.pack_sequence([a, b, c])
lstm = torch.nn.LSTM(
    input_size=3,
    hidden_size=3
)
lstm(seq)

I've tried batch_first=True, that did not work.

The source file of error code: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/rnn.py#L127

Output:
There are a few things wrong with your code.

Input size should be 1
LSTM takes a vector at each time step, you are passing scalars and hence the dimension error.

Below code should fix the problem. I made each input a (1,) size array and changed input size to 1.

import torch
import torch.nn.utils.rnn as rnn_utils
a = torch.Tensor([[1], [2], [3]])
b = torch.Tensor([[4], [5]])
c = torch.Tensor([[6]])
seq = rnn_utils.pack_sequence([a, b, c])
lstm = torch.nn.LSTM(input_size=1, hidden_size=3)
lstm(seq)

The packed sequence can be multi-dimensional and it depends on the data at each time step. For example in this case seq will be

PackedSequence(data=tensor([[1.],
        [4.],
        [6.],
        [2.],
        [5.],
        [3.]]), batch_sizes=tensor([3, 2, 1]))


Instruction: Calling cuda() with async results in SyntaxError

Input:
I'm trying to run this PyTorch code:

for i, (input, target) in enumerate(train_loader):

    input = input.float().cuda(async=True)
    target = target.cuda(async=True)
    input_var = torch.autograd.Variable(input)
    target_var = torch.autograd.Variable(target)

    output = model(input_var)

But when I try I am getting this error message:

input = input.float().cuda(async=True)
                               ^
SyntaxError: invalid syntax
Process finished with exit code 1

What am I doing wrong? I already installed cuda.

Output:
Your code does not work because:

async is a reserved keyword in python which cannot be used in that way, that is why you get the SyntaxError

cuda() no longer has an argument async. The constructor looks like this:

cuda(device=None, non_blocking=False) → Tensor

Previously there was an argument async but this replaced by non_blocking as async became a reserved keyword in Python 3.7.
 https://github.com/pluskid/fitting-random-labels/pull/5

Use instead non_blocking:
The argument non_blocking has the same effect as async previously had:

non_blocking (bool): If True and the source is in pinned memory, the
copy will be asynchronous with respect to the host. Otherwise, the
argument has no effect. Default: False.
 https://pytorch.org/docs/stable/tensors.html#torch.Tensor.cuda

As an add-on: If you are interested in what async is actually used for you can take a look here:
https://www.python.org/dev/peps/pep-0492/#new-syntax


Instruction: torch.Tensor() with requires_grad parameter

Input:
I can't use torch.Tensor() with requires_grad parameter (torch version : 0.4.1)

without requires_grad :

x = torch.Tensor([[.5, .3, 2.1]])
print(x)
> tensor([[0.5000, 0.3000, 2.1000]])

with requires_grad=True or requires_grad=False :

x = torch.Tensor([[.5, .3, 2.1]], requires_grad=False)
print(x)
Traceback (most recent call last):
  File "D:/_P/dev/ai/pytorch/notes/tensor01.py", line 4, in <module>
    x = torch.Tensor([[.5, .3, 2.1]], requires_grad=False)
TypeError: new() received an invalid combination of arguments - got (list, requires_grad=bool), but expected one of:
 * (torch.device device)
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, torch.device device)
      didn't match because some of the keywords were incorrect: requires_grad
 * (object data, torch.device device)
      didn't match because some of the keywords were incorrect: requires_grad

Output:
You are creating the tensor x by using the torch.Tensor class constructor which doesn't take the requires_grad flag. Instead, you want to use torch.tensor() (lowercase 't') method

x = torch.tensor([[.5, .3, 2.1]], requires_grad=False)

Edit: adding a link to docs: torch.Tensor


Instruction: is there any difference between matmul and usual multiplication of tensors

Input:
I am confused between the multiplication between two tensors using * and matmul.
Below is my code

import torch
torch.manual_seed(7)
features = torch.randn((2, 5))
weights = torch.randn_like(features)

here, i want to multiply weights and features. so, one way to do it is as follows

print(torch.sum(features * weights))

Output:

tensor(-2.6123)

Another way to do is using matmul

print(torch.mm(features,weights.view((5,2))))

but, here output is

tensor([[ 2.8089,  4.6439],
        [-2.3988, -1.9238]])

What i don't understand here is that why matmul and usual multiplication are giving different outputs, when both are same. Am i doing anything wrong here?

Edit: When, i am using feature of shape (1,5) both * and matmul outputs are same.
but, its not the same when the shape is (2,5).

Output:
When you use *, the multiplication is elementwise, when you use torch.mm it is matrix multiplication.

Example:

a = torch.rand(2,5)
b = torch.rand(2,5)
result = a*b

result will be shaped the same as a or b i.e (2,5)
whereas considering operation

result = torch.mm(a,b)

It will give a size mismatch error, as this is proper matrix multiplication (as we study in linear algebra) and a.shape[1] != b.shape[0]. When you apply the view operation in torch.mm you are trying to match the dimensions.

In the special case of the shape in some particular dimension being 1, it becomes a dot product and hence sum (a*b) is same as mm(a, b.view(5,1))


Instruction: How to efficiently retrieve the indices of maximum values in a Torch tensor?

Input:
Assume to have a torch tensor, for example of the following shape:

x = torch.rand(20, 1, 120, 120)

What I would like now, is to get the indices of the maximum values of each 120x120 matrix. To simplify the problem I would first x.squeeze() to work with shape [20, 120, 120]. I would then like to get torch tensor which is a list of indices with shape [20, 2].

How can I do this fast?

Output:
If I get you correctly you don't want the values, but the indices. Unfortunately there is no out of the box solution. There exists an argmax() function, but I cannot see how to get it to do exactly what you want.

So here is a small workaround, the efficiency should also be okay since we're just dividing tensors:

n = torch.tensor(4)
d = torch.tensor(4)
x = torch.rand(n, 1, d, d)
m = x.view(n, -1).argmax(1)
# since argmax() does only return the index of the flattened
# matrix block we have to calculate the indices by ourself
# by using / and % (// would also work, but as we are dealing with
# type torch.long / works as well
indices = torch.cat(((m / d).view(-1, 1), (m % d).view(-1, 1)), dim=1)
print(x)
print(indices)

n represents your first dimension, and d the last two dimensions. I take smaller numbers here to show the result. But of course this will also work for n=20 and d=120:

n = torch.tensor(20)
d = torch.tensor(120)
x = torch.rand(n, 1, d, d)
m = x.view(n, -1).argmax(1)
indices = torch.cat(((m / d).view(-1, 1), (m % d).view(-1, 1)), dim=1)
#print(x)
print(indices)

Here is the output for n=4 and d=4:

tensor([[[[0.3699, 0.3584, 0.4940, 0.8618],
          [0.6767, 0.7439, 0.5984, 0.5499],
          [0.8465, 0.7276, 0.3078, 0.3882],
          [0.1001, 0.0705, 0.2007, 0.4051]]],

        [[[0.7520, 0.4528, 0.0525, 0.9253],
          [0.6946, 0.0318, 0.5650, 0.7385],
          [0.0671, 0.6493, 0.3243, 0.2383],
          [0.6119, 0.7762, 0.9687, 0.0896]]],

        [[[0.3504, 0.7431, 0.8336, 0.0336],
          [0.8208, 0.9051, 0.1681, 0.8722],
          [0.5751, 0.7903, 0.0046, 0.1471],
          [0.4875, 0.1592, 0.2783, 0.6338]]],

        [[[0.9398, 0.7589, 0.6645, 0.8017],
          [0.9469, 0.2822, 0.9042, 0.2516],
          [0.2576, 0.3852, 0.7349, 0.2806],
          [0.7062, 0.1214, 0.0922, 0.1385]]]])
tensor([[0, 3],
        [3, 2],
        [1, 1],
        [1, 0]])

I hope this is what you wanted to get! :)

Edit:

Here is a slightly modified which might be minimally faster (not much I guess :), but it is a bit simpler and prettier:

Instead of this like before:

m = x.view(n, -1).argmax(1)
indices = torch.cat(((m // d).view(-1, 1), (m % d).view(-1, 1)), dim=1)

The necessary reshaping already done on the argmax values:

m = x.view(n, -1).argmax(1).view(-1, 1)
indices = torch.cat((m // d, m % d), dim=1)

But as mentioned in the comments. I don't think it is possible to get much more out of it.

One thing you could do, if it is really important for you to get the last possible bit of performance improvement out of it, is implementing this above function as a low-level extension (like in C++) for pytorch.

This would give you just one function you can call for it and would avoid slow python code.

https://pytorch.org/tutorials/advanced/cpp_extension.html


Instruction: Pytorch - Getting gradient for intermediate variables / tensors

Input:
As an exercice in pytorch framework (0.4.1) , I am trying to display the gradient of X (gX or dSdX) in a simple Linear layer (Z = X.W + B). To simplify my toy example, I backward() from a sum of Z (not a loss).

To sum up, I want gX(dSdX) of S=sum(XW+B).

The problem is that the gradient of Z (dSdZ) is None. As a result, gX is wrong too of course.

import torch
X = torch.tensor([[0.5, 0.3, 2.1], [0.2, 0.1, 1.1]], requires_grad=True)
W = torch.tensor([[2.1, 1.5], [-1.4, 0.5], [0.2, 1.1]])
B = torch.tensor([1.1, -0.3])
Z = torch.nn.functional.linear(X, weight=W.t(), bias=B)
S = torch.sum(Z)
S.backward()
print("Z:\n", Z)
print("gZ:\n", Z.grad)
print("gX:\n", X.grad)

Result:

Z:
 tensor([[2.1500, 2.9100],
        [1.6000, 1.2600]], grad_fn=<ThAddmmBackward>)
gZ:
 None
gX:
 tensor([[ 3.6000, -0.9000,  1.3000],
        [ 3.6000, -0.9000,  1.3000]])

I have exactly the same result if I use nn.Module as below:

class Net1Linear(torch.nn.Module):
    def __init__(self, wi, wo,W,B):
        super(Net1Linear, self).__init__()
        self.linear1 = torch.nn.Linear(wi, wo)
        self.linear1.weight = torch.nn.Parameter(W.t())
        self.linear1.bias = torch.nn.Parameter(B)
    def forward(self, x):
        return self.linear1(x)
net = Net1Linear(3,2,W,B)
Z = net(X)
S = torch.sum(Z)
S.backward()
print("Z:\n", Z)
print("gZ:\n", Z.grad)
print("gX:\n", X.grad)

Output:
First of all you only calculate gradients for tensors where you enable the gradient by setting the requires_grad to True.

So your output is just as one would expect. You get the gradient for X.

PyTorch does not save gradients of intermediate results for performance reasons. So you will just get the gradient for those tensors you set requires_grad to True.

However you can use register_hook to extract the intermediate grad during calculation or to save it manually. Here I just save it to the grad variable of tensor Z:

import torch

# function to extract grad
def set_grad(var):
    def hook(grad):
        var.grad = grad
    return hook

X = torch.tensor([[0.5, 0.3, 2.1], [0.2, 0.1, 1.1]], requires_grad=True)
W = torch.tensor([[2.1, 1.5], [-1.4, 0.5], [0.2, 1.1]])
B = torch.tensor([1.1, -0.3])
Z = torch.nn.functional.linear(X, weight=W.t(), bias=B)

# register_hook for Z
Z.register_hook(set_grad(Z))

S = torch.sum(Z)
S.backward()
print("Z:\n", Z)
print("gZ:\n", Z.grad)
print("gX:\n", X.grad)

This will output:

Z:
 tensor([[2.1500, 2.9100],
        [1.6000, 1.2600]], grad_fn=<ThAddmmBackward>)
gZ:
 tensor([[1., 1.],
        [1., 1.]])
gX:
 tensor([[ 3.6000, -0.9000,  1.3000],
        [ 3.6000, -0.9000,  1.3000]])

Hope this helps!

Btw.: Normally you would want the gradient to be activated for your parameters - so your weights and biases. Because what you would do right now when using the optimizer, is altering your inputs X and not your weights W and bias B. So usually gradient is activated for W and B in such a case.


Instruction: Two-Layer Neural Network in PyTorch does not Converge

Input:
Problem

I am trying to implement 2-layer neural network using different methods (TensorFlow, PyTorch and from scratch) and then compare their performance based on MNIST dataset.

I am not sure what mistakes I have made, but the accuracy in PyTorch is only about 10%, which is basically random guess. I think probably the weights does not get updated at all.

Note that I intentionally use the dataset provided by TensorFlow to keep the data I use through 3 different methods consistent for accurate comparison.

from tensorflow.examples.tutorials.mnist import input_data
import torch

class Net(torch.nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.fc1 =  torch.nn.Linear(784, 100)
      self.fc2 =  torch.nn.Linear(100, 10)

    def forward(self, x):
      # x -> (batch_size, 784)
      x = torch.relu(x)
      # x -> (batch_size, 10)
      x = torch.softmax(x, dim=1)
      return x

net = Net()
net.zero_grad()
Loss = torch.nn.CrossEntropyLoss()
optimizer =  torch.optim.SGD(net.parameters(), lr=0.01)

for epoch in range(1000):  # loop over the dataset multiple times

    batch_xs, batch_ys = mnist_m.train.next_batch(100)
    # convert to appropriate settins
    # note the input to the linear layer should be (n_sample, n_features)
    batch_xs = torch.tensor(batch_xs, requires_grad=True)
    # batch_ys -> (batch_size,)
    batch_ys = torch.tensor(batch_ys, dtype=torch.int64)

    # forward
    # output -> (batch_size, 10)
    output = net(batch_xs)
    # result -> (batch_size,)
    result = torch.argmax(output, dim=1)
    loss = Loss(output, batch_ys)

    # backward
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Output:
The problem here is that you don't apply your fully connected layers fc1 and fc2.

Your forward() currently looks like:

def forward(self, x):
    # x -> (batch_size, 784)
    x = torch.relu(x)
    # x -> (batch_size, 10)
    x = torch.softmax(x, dim=1)
    return x

So if you change it to:

def forward(self, x):
    # x -> (batch_size, 784)
    x = self.fc1(x)             # added layer fc1
    x = torch.relu(x)

    # x -> (batch_size, 10)
    x = self.fc2(x)             # added layer fc2
    x = torch.softmax(x, dim=1)
    return x

It should work.

Regarding Umang Guptas answer: As I see it, calling zero_grad() before calling backward() as Mr.Robot did, is just fine. This shouldn't be a problem.

Edit:

So I did a short test - I set iterations from 1000 to 10000 to see the bigger picture if it is really decreasing. ( Of course I also loaded the data to mnist_m as this wasn't included in the code you've posted )

I added a print condition to the code:

if epoch % 1000 == 0:
    print('Epoch', epoch, '- Loss:', round(loss.item(), 3))

Which prints out the loss every 1000 iterations:

Epoch 0 - Loss: 2.305
Epoch 1000 - Loss: 2.263
Epoch 2000 - Loss: 2.187
Epoch 3000 - Loss: 2.024
Epoch 4000 - Loss: 1.819
Epoch 5000 - Loss: 1.699
Epoch 6000 - Loss: 1.699
Epoch 7000 - Loss: 1.656
Epoch 8000 - Loss: 1.675
Epoch 9000 - Loss: 1.659

Tested with PyTorch version 0.4.1

So you can see that with the changed forward() the network is learning now, the rest of the code I left untouched.

Good luck further!


Instruction: How to tell PyTorch to not use the GPU?

Input: I want to do some timing comparisons between CPU & GPU as well as some profiling and would like to know if there's a way to tell pytorch to not use the GPU and instead use the CPU only? I realize I could install another CPU-only pytorch, but hoping there's an easier way.

Output:
Before running your code, run this shell command to tell torch that there are no GPUs:
export CUDA_VISIBLE_DEVICES=""

This will tell it to use only one GPU (the one with id 0) and so on:
export CUDA_VISIBLE_DEVICES="0"


Instruction: Pytorch. Can autograd be used when the final tensor has more than a single value in it?

Input:
Can autograd be used when the final tensor has more than a single value in it?

I tried the following.

x = torch.tensor([4.0, 5.0], requires_grad=True)
y = x ** 2

print(y)

y.backward()

Throws an error

RuntimeError: grad can be implicitly created only for scalar outputs

The following however works.

x = torch.tensor([4.0, 5.0], requires_grad=True)
y = x ** 2
y = torch.sum(y)
print(y)

y.backward()
print(x.grad)

The output is as

tensor(41., grad_fn=<SumBackward0>)
tensor([ 8., 10.])

Am I missing something here or can I proceed with the assumption that autograd only works when the final tensor has a single value in it?

Output:
See https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients

y.backward() is same as y.backward(torch.tensor(1.0))

Usually, the output is scalar and hence the scalar is passed to backward as a default choice. However, since your output is two dimensional you should call
y.backward(torch.tensor([1.0,1.0]))

This will give expected results with x.grad being tensor([ 8., 10.])


Instruction: PyTorch gradient differs from manually calculated gradient

Input:
I'm trying to compute the gradient of 1/x without using Pytorch's autograd. I use the formula grad(1/x, x) = -1/x**2. When I compare my result with this formula to the gradient given by Pytorch's autograd, they're different.

Here is my code:

a = torch.tensor(np.random.randn(), dtype=dtype, requires_grad=True)
loss = 1/a
loss.backward()
print(a.grad - (-1/(a**2)))

The output is:

tensor(5.9605e-08, grad_fn=<ThAddBackward>)

Can anyone explain to me what the problem is?

Output:
So I guess you expect zero as result. When you take a closer look you see that it is quite close. When deviding numbers on a binary system (computer) then you often get round-off errors.
Lets take a look at your example with an additional print-statement added:
a = torch.tensor(np.random.randn(), requires_grad=True)
loss = 1/a
loss.backward()
print(a.grad, (-1/(a**2)))
print(a.grad - (-1/(a**2)))

Because you use a random input the output is of course random too. (so you won't get this very same numbers, but just repeat this experiment and you will have similar examples).
Sometimes you will get zero as result. But this was not the case in your initial example:
tensor(-0.9074) tensor(-0.9074, grad_fn=<MulBackward>)
tensor(5.9605e-08, grad_fn=<ThSubBackward>)

You see even though both are displayed as the same number but they differ in one of the last decimal places. That is why you get this very small difference when subtracting both.
This problem as a general problem of computers, some fractions just have a large or infinite number of decimal places, but the memory of your computer has not. So they are cut off at some point.
So what you experience here is actually a lack of precision. And the precision is depending on the numerical data type you are using (i.e. torch.float32 or torch.float64).
You can also take a look here more information:
https://en.wikipedia.org/wiki/Double-precision_floating-point_format

But this is not specific to PyTorch or so, here is a Python example:
print(29/100*100)

Results in:
28.999999999999996

Edit:
As @HOANG GIANG pointed out, changing the equation to -(1/a)*(1/a) works well and the result is zero.
This is probably the case because the calculation in done to calculate the gradient is very similar (or the same) to -(1/a)*(1/a) in this case. Therefore it shares the same round-off errors therefore the difference is zero.
So then here is another more fitting example than the one above. Even though -(1/x)*(1/x) is mathematically equivalent to -1/x^2 it is not always the same when calculating it on the computer, depending on the value of x:
import numpy as np
print('e1 == e2','x value', '\t'*2, 'round-off error', sep='\t')
print('='*70)
for i in range(10):
    x = np.random.randn()
    e1 = -(1/x)*(1/x)
    e2 = (-1/(x**2))
    print(e1 == e2, x, e1-e2, sep='\t\t')

Output:
e1 == e2    x value                 round-off error
======================================================================
True        0.2934154339948173      0.0
True        -1.2881863891014191     0.0
True        1.0463038021843876      0.0
True        -0.3388766143622498     0.0
True        -0.6915415747192347     0.0
False       1.3299049850551317      1.1102230246251565e-16
True        -1.2392046539563553     0.0
False       -0.42534236747121645    8.881784197001252e-16
True        1.407198823994324       0.0
False       -0.21798652132356966    3.552713678800501e-15

Even though the round-off error seems to be a bit less (I tried different random values, and rarely more than two out of ten had a round-off error), but still there are already small differences when just calculating 1/x:
import numpy as np
print('e1 == e2','x value', '\t'*2, 'round-off error', sep='\t')
print('='*70)
for i in range(10):
    x = np.random.randn()
    # calculate 1/x
    result = 1/x
    # apply inverse function
    reconstructed_x = 1/result
    # mathematically this should be the same as x
    print(x == reconstructed_x, x, x-reconstructed_x, sep='\t\t')

Output:
e1 == e2    x value             round-off error
======================================================================
False       0.9382823115235075      1.1102230246251565e-16
True        -0.5081217386356917     0.0
True        -0.04229436058156134    0.0
True        1.1121100294357302      0.0
False       0.4974618312372863      -5.551115123125783e-17
True        -0.20409933212316553    0.0
True        -0.6501652554924282     0.0
True        -3.048057937738731      0.0
True        1.6236075700470816      0.0
True        0.4936926651641918      0.0


Instruction: Initialize weights and bias in torch

Input:
What is the equivalent command for the below keras code in Pytorch

Dense(64, kernel_initializer='he_normal', bias_initializer='zeros', name='uhat_digitcaps')(d5)

How to I initialize weights and bias?

Thanks!

Output:
class Net(nn.Module):
    def __init__(self, in_channels, out_channels):
        self.linear = nn.Linear(in_channels, 64)
        nn.init.kaiming_normal_(self.linear.weight, mode='fan_out')
        nn.init.constant_(self.linear.bias, 0)


Instruction: PyTorch next(iter(training_loader)) extremely slow, simple data, can't num_workers?

Input:
Here x_dat and y_dat are just really long 1-dimensional tensors.

class FunctionDataset(Dataset):
    def __init__(self):
        x_dat, y_dat = data_product()

        self.length = len(x_dat)
        self.y_dat = y_dat
        self.x_dat = x_dat

    def __getitem__(self, index):
        sample = self.x_dat[index]
        label = self.y_dat[index]
        return sample, label

    def __len__(self):
        return self.length

...

data_set = FunctionDataset()

...

training_sampler = SubsetRandomSampler(train_indices)
validation_sampler = SubsetRandomSampler(validation_indices)

training_loader = DataLoader(data_set, sampler=training_sampler, batch_size=params['batch_size'], shuffle=False)
validation_loader = DataLoader(data_set, sampler=validation_sampler, batch_size=valid_size, shuffle=False)

I have also  tried pinning the memory for the two loaders. Setting num_workers to > 0 gives me run-time errors between the processes (like EOF error and interruption errors). I get my batch with:

x_val, target = next(iter(training_loader))

The entire data-set would fit into memory/gpu but I would like to emulate batches for this experiment. Profiling my process gives me the following:

16276989 function calls (16254744 primitive calls) in 38.779 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   1745/1    0.028    0.000   38.780   38.780 {built-in method builtins.exec}
        1    0.052    0.052   38.780   38.780 simple aprox.py:3(<module>)
        1    0.000    0.000   36.900   36.900 simple aprox.py:519(exploreHeatmap)
        1    0.000    0.000   36.900   36.900 simple aprox.py:497(optFromSample)
        1    0.033    0.033   36.900   36.900 simple aprox.py:274(train)
  705/483    0.001    0.000   34.495    0.071 {built-in method builtins.next}
      222    1.525    0.007   34.493    0.155 dataloader.py:311(__next__)
      222    0.851    0.004   12.752    0.057 dataloader.py:314(<listcomp>)
  3016001   11.901    0.000   11.901    0.000 simple aprox.py:176(__getitem__)
       21    0.010    0.000   10.891    0.519 simple aprox.py:413(validationError)
      443    1.380    0.003    9.664    0.022 sampler.py:136(__iter__)
  663/221    2.209    0.003    8.652    0.039 dataloader.py:151(default_collate)
      221    0.070    0.000    6.441    0.029 dataloader.py:187(<listcomp>)
      442    6.369    0.014    6.369    0.014 {built-in method stack}
  3060221    2.799    0.000    5.890    0.000 sampler.py:68(<genexpr>)
  3060000    3.091    0.000    3.091    0.000 tensor.py:382(<lambda>)
      222    0.001    0.000    1.985    0.009 sampler.py:67(__iter__)
      222    1.982    0.009    1.982    0.009 {built-in method randperm}
  663/221    0.002    0.000    1.901    0.009 dataloader.py:192(pin_memory_batch)
      221    0.000    0.000    1.899    0.009 dataloader.py:200(<listcomp>)
....

Suggesting the data loader is immensely slow compared to the remaining activity of my experiment (training the model and lots of other computations etc.). What's going wrong and what would be the best way to speed this up?

Output:
When retrieving a batch with

x, y = next(iter(training_loader))

you actually create a new instance of dataloader iterator at each call (!) See this thread for more infotrmation.
What you should do instead is create the iterator once (per epoch):

training_loader_iter = iter(training_loader)

and then call next for each batch on the iterator

for i in range(num_batches_in_epoch):
  x, y = next(training_loader_iter)

I had similar issue before, and this also made the EOF errors you experience when using multiple workers go away.


Instruction: Confusion matrix and test accuracy for PyTorch Transfer Learning tutorial

Input:
Following the Pytorch Transfer learning tutorial, I am interested in reporting only train and test accuracy as well as confusion matrix (say using sklearn confusionmatrix). How can I do that? The current tutorial only reports train/val accuracy and I am having hard time figuring how to incorporate the sklearn confusionmatrix code there. Link to original tutorial here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

%matplotlib inline
from graphviz import Digraph
import torch
from torch.autograd import Variable
# Author: Sasank Chilamkurthy

from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy

plt.ion()
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = "images"
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def imshow(inp, title=None):
    """Imshow for Tensor."""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    inp = np.clip(inp, 0, 1)
    plt.imshow(inp)
    if title is not None:
        plt.title(title)
    plt.pause(0.001)  # pause a bit so that plots are updated

# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))

# Make a grid from batch
out = torchvision.utils.make_grid(inputs)

imshow(out, title=[class_names[x] for x in classes])

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

def visualize_model(model, num_images=6):
    was_training = model.training
    model.eval()
    images_so_far = 0
    fig = plt.figure()

    with torch.no_grad():
        for i, (inputs, labels) in enumerate(dataloaders['val']):
            inputs = inputs.to(device)
            labels = labels.to(device)

            outputs = model(inputs)
            _, preds = torch.max(outputs, 1)

            for j in range(inputs.size()[0]):
                images_so_far += 1
                ax = plt.subplot(num_images//2, 2, images_so_far)
                ax.axis('off')
                ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                imshow(inputs.cpu().data[j])

                if images_so_far == num_images:
                    model.train(mode=was_training)
                    return
        model.train(mode=was_training)

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 9)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

visualize_model(model_ft)

Output:
Answer given by ptrblck of PyTorch community. Thanks a lot!

nb_classes = 9

confusion_matrix = torch.zeros(nb_classes, nb_classes)
with torch.no_grad():
    for i, (inputs, classes) in enumerate(dataloaders['val']):
        inputs = inputs.to(device)
        classes = classes.to(device)
        outputs = model_ft(inputs)
        _, preds = torch.max(outputs, 1)
        for t, p in zip(classes.view(-1), preds.view(-1)):
                confusion_matrix[t.long(), p.long()] += 1

print(confusion_matrix)

To get the per-class accuracy:

print(confusion_matrix.diag()/confusion_matrix.sum(1))


Instruction: Does it differ to use optimizer.step or model.step in pytorch?

Input:
In pytorch, to update the model, should I use optimizer.step() or model.step() ?

Here is a example snippet:

import torch
import torch nn
class SomeNeuralNet(nn.Module):
    def __init__(self,hs,es,dropout):
          SomeNeuralNet(ClaimRecognizer, self).__init__()
          # Some initialization here
    def forward(x):
          # forward propagation here

model = SomeNeuralNet(es,hs,dp)
optimizer = optim.Adam(model.parameters())
loss_function = nn.NLLLoss()
for epoch in N:
   for x in data:
      logp = model(x)
      loss = loss_function(logp,gold_outs)
      loss.backward()
       # Which one I should call ? Optimizer.step() or model.step() or both ?
       optimizer.step()
       model.step()

Output:
To make the gradient descent step, you normally use just optimizer.step(). Here is also an example taken from the documentation (same link at bottom), what it looks like in general:

for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

I don't know where you got this model.step()? Does did you try it?

If your model really possesses some kind of step()-functionality, it probably does something different.

But unless you define something extra, your model gets its functions from nn.Module and this does not have step function!

See this example from the the Pytorch Documentation:

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

model = Model()
model.step()

Trying to call step() result in an AttributeError:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-b032813f7eda> in <module>
     13
     14 model = Model()
---> 15 model.step()

~/miniconda3/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533
    534     def __setattr__(self, name, value):

AttributeError: 'Model' object has no attribute 'step'

To sum it up, normally your model should not have a step() function, optimizer.step() is the way to go if you want to do the optimization step.

See also here:
https://pytorch.org/docs/stable/optim.html#taking-an-optimization-step


Instruction: Using pre-trained word embeddings - how to create vector for unknown / OOV Token?

Input:
I wan't to add pre-trained embeddings to a model. But as it seems there is no out-of-vocabulary (OOV) token resp. no vector for unseen words existent.

So what can I do to handle OOV-tokens I come across? I have some ideas, but none of them seem to be very good:

I could just create a random vector for this token, but ideally I'd like the vector to within the logic of the existing model. If I just create it randomly I'm afraid the vector accidentally could be very similar to a very frequent word like 'the', 'for', 'that' etc. which is not my intention.
Or should I just initialize the vector with plain zeros instead?
Another idea would be averaging the token over other existing vectors. But averaging on what vectors then? On all? This doesn't seem to be very conclusive either.
I also thought about trying to train this vector. However this doesn't come very handy if I want to freeze the rest of the embedding during training.

(A general solution is appreciated, but I wanted to add that I'm using PyTorch - just in case PyTorch already comes with a handy solution to this problem.)

So what would be a good and easy strategy to create such a vector?

Output:
There are multiple ways you can deal with it. I don't think I can cite references about which works better.
Non-trainable option:

Random vector as embedding
You can use an all-zero vector for OOV.
You can use the mean of all the embedding vectors, that way you avoid the risk of being away from the actual distribution.
Also embeddings generally come with "unk" vectors learned during the training you can use that.

Trainable Option:
You can declare a separate embedding vector for OOV and make it trainable keeping other embedding fixed. You might have to over-write the forward method of embedding lookup for this. You can declare a new trainable Variable and in the forward pass use this vector as embedding for OOV instead of doing a look-up.

Addressing the comments of OP:
I am not sure which of the three non-trainable methods may work better and I am not sure if there is some work about this. But method 4) should be working better.
For trainable option, you can create a new embedding layer as below.
class Embeddings_new(torch.nn.Module):
    def __init__(self, dim, vocab):
        super().__init__()
        self.embedding = torch.nn.Embedding(vocab, dim)
        self.embedding.weight.requires_grad = False
        # vector for oov
        self.oov = torch.nn.Parameter(data=torch.rand(1,dim))
        self.oov_index = -1
        self.dim = dim

    def forward(self, arr):
        N = arr.shape[0]
        mask =  (arr==self.oov_index).long()
        mask_ = mask.unsqueeze(dim=1).float()
        embed =(1-mask_)*self.embedding((1-mask)*arr) + mask_*(self.oov.expand((N,self.dim)))
        return embed

Usage:
model = Embeddings_new(10,20000)
out = model.forward(torch.tensor([-1,-1, 100, 1, 0]))
# dummy loss
loss = torch.sum(a**2)
loss.backward()


Instruction: Pytorch 0.4: Should the requires_grad flag of the network input be set to True during training?

Input: The requires_grad flag of input is default False during my last training. I was wondering if I should set it to True.

Output:
So generally:

For all tensors / weights that you want to be trained the requires_grad flag has to be True.
This would be the case for your parameters resp. weights and biases. So there you want the flag to be True. But this is already default value for predefined modules like nn.Linear, nn.Embedding. nn.Conv2d etc. So you don't have to change it.
For things like inputs the requires_grad flag needs to be False since you don't want to train your input data.

I hope this answers your question, if you want to know something more specific just ask!


Instruction: Pytorch 0.4.0: There are three ways to create tensors on CUDA device. Is there some difference between them?

Input:
I failed in the third way. t3 is still on CPU. No idea why.

a = np.random.randn(1, 1, 2, 3)

t1 = torch.tensor(a)
t1 = t3.to(torch.device('cuda'))

t2 = torch.tensor(a)
t2 = t2.cuda()

t3 = torch.tensor(a, device=torch.device('cuda'))

Output:
All three methods worked for me.

In 1 and 2, you create a tensor on CPU and then move it to GPU when you use .to(device) or .cuda(). They are the same here.

However, when you use .to(device) method you can explicitly tell torch to move to specific GPU by setting device=torch.device("cuda:<id>"). with .cuda() you have to do .cuda(<id>) to move to some particular GPU.

Why do these two methods exist then?

.to(device) was introduced in 0.4 because it is easier to declare device variable at top of the code as

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

and use .to(device) everywhere. This makes it quite easy to switch from CPU to GPU and vice-versa

Before this, we had to use .cuda() and your code will have if check for cuda.is_available() everywhere which made it cumbersome to switch between GPU/CPU.

The third method doesn't create a tensor on the CPU and directly copies data to GPU, which is more efficient.


Instruction: Accumulating Gradients

Input:
I want to accumulate the gradients before I do a backward pass. So wondering what the right way of doing it is. According to this article
it's:

model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()

whereas I expected it to be:

model.zero_grad()                                   # Reset gradients tensors
loss = 0
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss += loss_function(predictions, labels)       # Compute loss function
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        loss = loss / accumulation_steps            # Normalize our loss (if averaged)
        loss.backward()                             # Backward pass
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()
        loss = 0

where I accumulate the loss and then divide by the accumulation steps to average it.

Secondary question, if I am right, would you expect my method to be quicker considering I only do the backward pass every accumulation steps?

Output:
So according to the answer here, the first method is memory efficient. The amount of work required is more or less the same in both methods.

The second method keeps accumulating the graph, so would require accumulation_steps times more memory. The first method calculates the gradients straight away (and simply adds gradients) so requires less memory.


Instruction: Torchvision 0.2.1 transforms.Normalize does not work as expected

Input:
I am trying a new code using Pytorch. In this code, to load the dataset (CIFAR10), I am using torchvision's datasets. I define two transform functions ToTensor() and Normalize(). After normalize I expect the data in the dataset should be between 0 and 1. But the max value is still 255. I also inserted a print statement inside the '__call__' function of Normalize class in transforms.py (Lib\site-packages\torchvision\transforms\transforms.py). This print is not printed while running the code too. Not sure what is happening. Every page I visited in the internet, mentions the usage almost the same way as I do. For example some sites I visited
https://github.com/adventuresinML/adventures-in-ml-code/blob/master/pytorch_nn.py
https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/cifar10_tutorial.py

My code is given below. This reads the dataset with and without Normalize, then prints some stats. The min and max printed is an indicator of whether the data is normalized or not.

import torchvision as tv
import numpy as np

dataDir = 'D:\\general\\ML_DL\\datasets\\CIFAR'

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor()])
trainSet        = tv.datasets.CIFAR10(dataDir, train=True, download=False, transform=trainTransform)
print (trainSet.train_data.mean(axis=(0,1,2))/255)
print (trainSet.train_data.min())
print (trainSet.train_data.max())
print (trainSet.train_data.shape)

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor(), tv.transforms.Normalize((0.4914, 0.4822, 0.4466), (0.247, 0.243, 0.261))])
trainSet        = tv.datasets.CIFAR10(dataDir, train=True, download=False, transform=trainTransform)
print (trainSet.train_data.mean(axis=(0,1,2))/255)
print (trainSet.train_data.min())
print (trainSet.train_data.max())
print (trainSet.train_data.shape)

The output looks like,

[ 0.49139968  0.48215841  0.44653091]
0
255
(50000, 32, 32, 3)
[ 0.49139968  0.48215841  0.44653091]
0
255
(50000, 32, 32, 3)

Please help me understand this better. As most of the functions I tried, ends up with similar results - for example Grayscale, CenterCrop too.

Output:
So, in the code you have laid out a plan that how you want to process your data. You have created a data pipeline through which your data will flow and multiple transforms will be applied.

However, You forgot to call torch.utils.data.DataLoader. Until this is called, transformations on your data won't be applied. You can read more about it here.

Now when we add the above to your code like the following -

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor(),
                  tv.transforms.Normalize((0.4914, 0.4822, 0.4466), (0.247, 0.243, 0.261))])

trainSet = tv.datasets.CIFAR10(root=dataDir, train=True,
                                    download=False, transform=trainTransform)

dataloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=False, num_workers=4)

and printed images like the following -

images, labels = iter(dataloader).next()
print images
print images.max()
print images.min()

We get Tensors having transformations we have applied.

A small snippet of the output

[[ 1.8649,  1.8198,  1.8348,  ...,  0.3924,  0.3774,  0.2572],
      [ 1.9701,  1.9550,  1.9851,  ...,  0.7230,  0.6929,  0.6629],
      [ 2.0001,  1.9550,  2.0001,  ...,  0.7831,  0.7530,  0.7079],
      ...,
      [-0.8096, -1.0049, -1.0350,  ..., -1.3355, -1.3655, -1.4256],
      [-0.7796, -0.8697, -0.9749,  ..., -1.2754, -1.4557, -1.5609],
      [-0.7645, -0.7946, -0.9298,  ..., -1.4106, -1.5308, -1.5909]]]])
tensor(2.1309)
tensor(-1.9895)

Secondly, transforms.Normalize(mean,std) applies input[channel] = (input[channel] - mean[channel]) / std[channel] so according to the mean and standard deviation we are providing we can't get values after transformation in the range of (0,1). In case you want values between (-1,1) you can use the following -

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor(),
                  tv.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

I hope it helps! :)


Instruction: pytorch save a minibatch of 4D tensor as images

Input:
I have set of 8 tensors in 4D tensor of shape (B, C, H, W). More specifically, the shape of my tensor is (8,3,64,64) i.e 8 images with [3x64x64] format.

I would like to save these in 1.png, 2.png ..., 8.png etc. When I try torchvision.utils.save_images(my_tensor), this is working fine. However, that is saving the images in a single grid instead of individual images.

How can I fix this issue?

I tried below code but that did not work.

for i in range(tensor.size(0)):

        np_data = tensor.cpu().numpy()
        imgplot = plt.imshow(np_data)

Output:
If you want to save individual images you could use:

for i in range(tensor.size(0)):
           torchvision.utils.save_image(tensor[i, :, :, :], '{}.png'.format(i))

which will save as : 1.png, 2.png ..., 8.png


Instruction: How to do a "element by element in-place inverse" with pytorch?

Input:
Given is an array a:

a = np.arange(1, 11, dtype = 'float32')

With numpy, I can do the following:

np.divide(1.0, a, out = a)

Resulting in:

array([1.        , 0.5       , 0.33333334, 0.25      , 0.2       ,
       0.16666667, 0.14285715, 0.125     , 0.11111111, 0.1       ],
      dtype=float32)

Assuming that a is instead a pytorch tensor, the following operation fails:

torch.div(1.0, a, out = a)

The first parameter of div is expected to be a tensor of matching length/shape.

If I substitute 1.0 with an array b filled with ones, its length equal to the length of a, it works. The downside is that I have to allocate memory for b. I can also do something like a = 1.0 / a which will yet again allocate extra (temporary) memory.

How can I do this operation efficiently "in-place" (without the allocation of extra memory), ideally with broadcasting?

Output:
Pytorch follows the convention of using _ for in-place operations.
for eg

add -> add_  # in-place equivalent
div -> div_  # in-place equivalent
etc

Element-by-element inplace inverse.

>>> a = torch.arange(1, 11, dtype=torch.float32)
>>> a.pow_(-1)
>>> a
>>> tensor([1.0000, 0.5000, 0.3333, 0.2500, 0.2000, 0.1667, 0.1429, 0.1250, 0.1111, 0.1000])

>>> a = torch.arange(1, 11, dtype=torch.float32)
>>> a.div_(a ** a)
>>> a
>>> tensor([1.0000, 0.5000, 0.3333, 0.2500, 0.2000, 0.1667, 0.1429, 0.1250, 0.1111, 0.1000])


Instruction: Why is this function parameter identical in every invocation, despite passing different value? (Creating closures in a loop)

Input:
I'm using PyTorch and trying to register hooks on model parameters. The following code creates lambda functions to add to each model parameter, so I can see in the hook which tensor the gradient belongs to

import torch
import torchvision

# define model and random train batch
model = torchvision.models.alexnet()
input = torch.rand(10, 3, 224, 224)   # batch of 10 images
targets = torch.zeros(10).long()

def grad_hook_template(param, name, grad):
    print(f'Receive grad for {name} w whape {grad.shape}')

# add one lambda hook to each parameter
for name, param in model.named_parameters():
    print(f'Register hook for {name}')

    # use a lambda so we can pass additional information to the hook, which should only take one parameter
    param.register_hook(lambda grad: grad_hook_template(param, name, grad))

loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
optimizer.zero_grad()

prediction = model(input)
loss = loss_fn(prediction, targets)
loss.backward()
optimizer.step()

The result is that the name and param arguments to grad_hook_template always as the same value (and id), but the grad argument is always different (as expected). Why is it that when I register the hook, the lambdas seem to refer to the same local variables each time?

I read e.g. here that loops do not create new scopes and closures are lexical in Python, i.e. the name and param which I'm passing to the lambda are just pointers and whatever value they have at the end of the loop is seen by everyone with this pointer. But what can I do about it? copy.copy() the variables?

Output:
You've run into late binding closures. The variables param and name are looked up at call time, not when the function they're used in is defined. By the time any of these functions are called, name and param are at the last values in the loop. To get around this, you could do this:

for name, param in model.named_parameters():
    print(f'Register hook for {name}')
    param.register_hook(lambda grad, name=name, param=param: grad_hook_template(param, name, grad))

However, I think using functools.partial is the right solution here:

from functools import partial

for name, param in model.named_parameters():
    print(f'Register hook for {name}')
    param.register_hook(partial(grad_hook_template, name=name, param=param))

You can find more information about late binding closures at the Common Gotchas page of the Hitchhiker's Guide to Python as well as in the Python docs.

Note that this applies equally to functions defined with the def keyword.


Instruction: Stratified cross validation with Pytorch

Input:
My goal is to make binary classification, using neural network.
The problem is that dataset is unbalanced, I have 90% of class 1 and 10 of class 0.
To deal with it I want to use Stratified cross-validation.

The problem that is I am working with Pytorch, I can't find any example and documentation doesn't provide it, and I'm student, quite new for neural networks.

Can anybody help?
Thank you!

Output: The easiest way I've found is to do you stratified splits before passing your data to Pytorch Dataset and DataLoader. That lets you avoid having to port all your code to skorch, which can break compatibility with some cluster computing frameworks.


Instruction: Pytorch Index a tensor of [n*n*3] by two tensors of size[n,n]

Input:
In numpy, I can index in the following:

a = np.random.randn(2,2,3)
b = np.eye(2,2).astype(np.uint8)
c = np.eye(2,2).astype(np.uint8)
print(a)
print("diff")
print(a[b,c,:])

, in which a[b, c, :] is a tensor of 2*2.

[[[-1.01338087  0.70149058  0.55268617]
  [ 2.56941124  1.12720312 -0.07219555]]

 [[-0.04084548  0.17018995  2.14229567]
  [-0.68017558 -0.91788125  1.1719151 ]]]
diff
[[[-0.68017558 -0.91788125  1.1719151 ]
  [-1.01338087  0.70149058  0.55268617]]

 [[-1.01338087  0.70149058  0.55268617]
  [-0.68017558 -0.91788125  1.1719151 ]]]

But in Pytorch, I could not do the index in the same way like a[b,c,:]. Who knows how to do this. Thanks~

Output:
Indexing in PyTorch is almost similar to numpy.

a = torch.randn(2, 2, 3)
b = torch.eye(2, 2, dtype=torch.long)
c = torch.eye(2, 2, dtype=torch.long)

print(a)
print(a[b, c, :])

tensor([[[ 1.2471,  1.6571, -2.0504],
         [-1.7502,  0.5747, -0.3451]],

        [[-0.4389,  0.4482,  0.7294],
         [-1.3051,  0.6606, -0.6960]]])
tensor([[[-1.3051,  0.6606, -0.6960],
         [ 1.2471,  1.6571, -2.0504]],

        [[ 1.2471,  1.6571, -2.0504],
         [-1.3051,  0.6606, -0.6960]]])


Instruction: F.relu(self.fc1(x)) is causing RuntimeError problem

Input:
I have implemented the following CNN for my training and validation data sets
that contain 90 and 20 images respectively divided into 3 classes:

def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1 = nn.Linear(16 * 5 * 5, 120)
    self.fc2 = nn.Linear(120, 50)
    self.fc3 = nn.Linear(50, len(classes))

def forward(self, x):
    print(x.shape)
    x = self.pool(F.relu(self.conv1(x)))
    print(x.shape)
    x = self.pool(F.relu(self.conv2(x)))
    print(x.shape)
    x = x.view(x.size(0),-1)
        #x = x.view(-1,x.size(1)*x.size(2)*x.size(3))
        #x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

I am getting the following error while running:

RuntimeError: size mismatch, m1: [1 x 214720], m2: [400 x 120] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:2033 for x = F.relu(self.fc1(x))

Can someone please suggest what I should do to get rid of this problem? I changed the x.view(...) by following some threads. However, it did not help in this case.

Output:
The size of the in_channels to self.fc1 is dependent on the input image size and not on the kernel-size.

In your case,
self.fc1 = nn.Linear(16 * 5 * 5, 120) should be nn.Linear(16 * image_size * image_size)
where, image_size: is the size of the image in the last convolution layer.

Sample Code:

import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

class Net(nn.Module):
    def __init__(self, classes):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5, padding=2)
        self.fc1 = nn.Linear(16 * 25 * 25, 120)
        self.fc2 = nn.Linear(120, 50)
        self.fc3 = nn.Linear(50, classes)

    def forward(self, x):
        print('one', x.shape)
        x = self.pool(F.relu(self.conv1(x)))
        print('two', x.shape)
        x = self.pool(F.relu(self.conv2(x)))
        print('three', x.shape)
        x = x.view(-1, np.product(x.shape[1:]))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

x = torch.rand((32, 3, 100, 100))
net = Net(2)
out= net(x)
print('out', out.shape)

one torch.Size([32, 3, 100, 100])
two torch.Size([32, 6, 50, 50])
three torch.Size([32, 16, 25, 25])
out torch.Size([32, 2])


Instruction: Pytorch on google-colaboratory GPU - Illegal memory access

Input:
I am using pytorch(0.4.0) on google-colaboratory ( NVIDIA-SMI 396.44                 Driver Version: 396.44)

When running my code outside any function, I am able to send pytorch tensors and model to the GPU :

...
model.cuda()
data_tensor = data_tensor.cuda()
...

And my CNN model is trained successfully with 98% accurancy.

But when I put the same code in a function,

def main(...):
    ....
    model.cuda()
    data_tensor= data_tensor.cuda()
    ...

if __name__ == "__main__":
    main('...)

I have the following error:

cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/generic/THCTensorCopy.c:20

UPDATE(18/11/21):

It turned out that being part or not of a function is irrelevant. Usually, I have first a CUDNN_STATUS_EXECUTION_FAILED error then the second time a  cuda runtime error (77) as shown below. But it sometimes works a few times before failing.

CUDNN_STATUS_EXECUTION_FAILED (first try) :

RuntimeError                              Traceback (most recent call last)
<ipython-input-27-53476e08e017> in <module>()
      1 main('mnist', 'to', 'ndd', Xd=16, epo=5, bs=100, tXn=-1, vXn=300,
----> 2      lr=0.05, suf="s1", n_class=10, cuda=True)

<ipython-input-23-918584456207> in main(ds, framework, format, Xd, epo, bs, tXn, vXn, lr, suf, n_class, cuda)
     12     opt = torch.optim.SGD(net.parameters(), lr)
     13
---> 14     train(net, opt, Xd, epo, bs, cuda, tXn, tX, tT, vX, vT,lr)
     15

<ipython-input-26-6b574a9e8af6> in train(model, optimizer, Xd, epo, bs, cuda, Xn, tX, tT, vX, vT, lr)
     26             #t = t.cuda()
     27             optimizer.zero_grad()
---> 28             z = model(x)
     29             bat_loss = criterion(z, t)
     30             bat_loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

<ipython-input-22-b4bc2e0b39b8> in forward(self, X)
     10         H0 = torch.zeros(self.n_H, X.size(0), self.Wh)
     11         C0 = torch.zeros(self.n_H, X.size(0), self.Wh)
---> 12         O, (Hn, Cn), = self.lstm1(X, (H0, C0))
     13         O = self.linear1(O[:, -1, :])
     14         return O

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    190             flat_weight=flat_weight
    191         )
--> 192         output, hidden = func(input, self.all_weights, hx, batch_sizes)
    193         if is_packed:
    194             output = PackedSequence(output, batch_sizes)

/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py in forward(input, *fargs, **fkwargs)
    321             func = decorator(func)
    322
--> 323         return func(input, *fargs, **fkwargs)
    324
    325     return forward

/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py in forward(input, weight, hx, batch_sizes)
    285             batch_first, dropout, train, bool(bidirectional),
    286             list(batch_sizes.data) if variable_length else (),
--> 287             dropout_ts)
    288
    289         if cx is not None:

RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

cuda runtime error (77)  (other tries):

RuntimeError                              Traceback (most recent call last)
<ipython-input-28-53476e08e017> in <module>()
      1 main('mnist', 'to', 'ndd', Xd=16, epo=5, bs=100, tXn=-1, vXn=300,
----> 2      lr=0.05, suf="s1", n_class=10, cuda=True)

<ipython-input-23-918584456207> in main(ds, framework, format, Xd, epo, bs, tXn, vXn, lr, suf, n_class, cuda)
     12     opt = torch.optim.SGD(net.parameters(), lr)
     13
---> 14     train(net, opt, Xd, epo, bs, cuda, tXn, tX, tT, vX, vT,lr)
     15

<ipython-input-26-6b574a9e8af6> in train(model, optimizer, Xd, epo, bs, cuda, Xn, tX, tT, vX, vT, lr)
      4     if cuda and torch.cuda.is_available():
      5         print("tX type (before):", tX.type())
----> 6         model.cuda()
      7         tX = tX.cuda()
      8         tT = tT.cuda()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in cuda(self, device)
    247             Module: self
    248         """
--> 249         return self._apply(lambda t: t.cuda(device))
    250
    251     def cpu(self):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    174     def _apply(self, fn):
    175         for module in self.children():
--> 176             module._apply(fn)
    177
    178         for param in self._parameters.values():

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in _apply(self, fn)
    109
    110     def _apply(self, fn):
--> 111         ret = super(RNNBase, self)._apply(fn)
    112         self.flatten_parameters()
    113         return ret

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    180                 # Tensors stored in modules are graph leaves, and we don't
    181                 # want to create copy nodes, so we have to unpack the data.
--> 182                 param.data = fn(param.data)
    183                 if param._grad is not None:
    184                     param._grad.data = fn(param._grad.data)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in <lambda>(t)
    247             Module: self
    248         """
--> 249         return self._apply(lambda t: t.cuda(device))
    250
    251     def cpu(self):

RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/generic/THCTensorCopy.c:20

Output:
It now works with Pytorch 1.0 using:

!pip3 install https://download.pytorch.org/whl/cu80/torch-1.0.0-cp36-cp36m-linux_x86_64.whl


Instruction: Why Skorch show NAN in the every epoch?

Input:
I want to create my own dataset class based on Dataset class of Skorch because I want to differentiate categorical columns and continuous columns. These categorical columns will be passed through the embedding layers in the model. The result is weird because it show NAN
like this:

  epoch    train_loss    valid_loss     dur
-------  ------------  ------------  ------
      1           nan           nan  0.2187
      2           nan           nan  0.1719
      3           nan           nan  0.1719
      4           nan           nan  0.1562
      5           nan           nan  0.1406

Can you help me fix it ? I am using data from this kaggle:
Here

from skorch import NeuralNetRegressor
from skorch.dataset import Dataset
import torch
import torch.nn as nn
import torch.nn.functional as F
import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelEncoder

class TabularDataset(Dataset):
    def __init__(self, data, cat_cols=None, output_col=None):
        self.n = data.shape[0]

        if output_col:
            self.y = data[output_col].astype(np.float32).values.reshape(-1, 1)
        else:
            self.y = np.zeros((self.n, 1))

        self.cat_cols = cat_cols if cat_cols else []
        self.cont_cols = [col for col in data.columns
                          if col not in self.cat_cols + [output_col]]

        if self.cont_cols:
            self.cont_X = data[self.cont_cols].astype(np.float32).values
        else:
            self.cont_X = np.zeros((self.n, 1))

        if self.cat_cols:
            self.cat_X = data[self.cat_cols].astype(np.int64).values
        else:
            self.cat_X = np.zeros((self.n, 1))

    def __len__(self):
        # Denotes the total number of sampoes
        return self.n

    def __getitem__(self, idx):
        # generates one sample of data
        return [self.cont_X[idx], self.cat_X[idx]], self.y[idx]

class FeedForwardNN(nn.Module):

    def __init__(self, emb_dims, no_of_cont, lin_layer_sizes,
                 output_size, emb_dropout, lin_layer_dropouts):

        """
        Parameters
        ----------
        emb_dims: List of two element tuples
          This list will contain a two element tuple for each
          categorical feature. The first element of a tuple will
          denote the number of unique values of the categorical
          feature. The second element will denote the embedding
          dimension to be used for that feature.
        no_of_cont: Integer
          The number of continuous features in the data.
        lin_layer_sizes: List of integers.
          The size of each linear layer. The length will be equal
          to the total number
          of linear layers in the network.
        output_size: Integer
          The size of the final output.
        emb_dropout: Float
          The dropout to be used after the embedding layers.
        lin_layer_dropouts: List of floats
          The dropouts to be used after each linear layer.
        """

        super().__init__()

        # Embedding layers
        self.emb_layers = nn.ModuleList([nn.Embedding(x, y)
                                         for x, y in emb_dims])

        no_of_embs = sum([y for x, y in emb_dims])
        self.no_of_embs = no_of_embs
        self.no_of_cont = no_of_cont

        # Linear Layers
        first_lin_layer = nn.Linear(self.no_of_embs + self.no_of_cont,
                                    lin_layer_sizes[0])

        self.lin_layers = \
            nn.ModuleList([first_lin_layer] + \
                          [nn.Linear(lin_layer_sizes[i], lin_layer_sizes[i + 1])
                           for i in range(len(lin_layer_sizes) - 1)])

        for lin_layer in self.lin_layers:
            nn.init.kaiming_normal_(lin_layer.weight.data)

        # Output Layer
        self.output_layer = nn.Linear(lin_layer_sizes[-1],
                                      output_size)
        nn.init.kaiming_normal_(self.output_layer.weight.data)

        # Batch Norm Layers
        self.first_bn_layer = nn.BatchNorm1d(self.no_of_cont)
        self.bn_layers = nn.ModuleList([nn.BatchNorm1d(size)
                                        for size in lin_layer_sizes])

        # Dropout Layers
        self.emb_dropout_layer = nn.Dropout(emb_dropout)
        self.droput_layers = nn.ModuleList([nn.Dropout(size)
                                            for size in lin_layer_dropouts])

    def forward(self, X):
        cont_data = X[0]
        cat_data = X[1]
        if self.no_of_embs != 0:
            x = [emb_layer(cat_data[:, i])
                 for i, emb_layer in enumerate(self.emb_layers)]
            x = torch.cat(x, 1)
            x = self.emb_dropout_layer(x)

        if self.no_of_cont != 0:
            normalized_cont_data = self.first_bn_layer(cont_data)

            if self.no_of_embs != 0:
                x = torch.cat([x, normalized_cont_data], 1)
            else:
                x = normalized_cont_data

        for lin_layer, dropout_layer, bn_layer in \
                zip(self.lin_layers, self.droput_layers, self.bn_layers):
            x = F.relu(lin_layer(x))
            x = bn_layer(x)
            x = dropout_layer(x)

        x = self.output_layer(x)

        return x

# Read data
data = pd.read_csv("data/train.csv", usecols=["SalePrice", "MSSubClass", "MSZoning", "LotFrontage", "LotArea",
                                              "Street", "YearBuilt", "LotShape", "1stFlrSF", "2ndFlrSF"]).dropna()

categorical_features = ["MSSubClass", "MSZoning", "Street", "LotShape", "YearBuilt"]
output_feature = "SalePrice"

# Label Encode Categorial Features
label_encoders = {}
for cat_col in categorical_features:
    label_encoders[cat_col] = LabelEncoder()
    data[cat_col] = label_encoders[cat_col].fit_transform(data[cat_col])

# feed Forward NN
cat_dims = [int(data[col].nunique()) for col in categorical_features]

emb_dims = [(x, min(50, (x + 1) // 2)) for x in cat_dims]

net = FeedForwardNN(emb_dims, no_of_cont=4, lin_layer_sizes=[50, 100],
                    output_size=1, emb_dropout=0.04,
                    lin_layer_dropouts=[0.001, 0.01])

# Fit
ds = TabularDataset(data=data, cat_cols=categorical_features,
                    output_col=output_feature)
X = data.drop(['SalePrice'], axis=1)
y = data['SalePrice'].values.reshape(-1, 1)
net = NeuralNetRegressor(
    net,
    max_epochs=5,
    lr=0.1,
    dataset=ds
)
net.fit(X, y)

Output:
The problem is not with skorch but your data. You have to scale your inputs and, in this case, especially the targets to avoid huge losses and exploding gradients. As a start I suggest using, for example, sklearn.preprocessing.StandardScaler:

from sklearn.preprocessing import StandardScaler

class TabularDataset(Dataset):
    def __init__(self, data, cat_cols=None, output_col=None):
        self.n = data.shape[0]
        # [...]
        if output_col:
            scaler_y = StandardScaler()
            self.y = data[output_col].astype(np.float32).values.reshape(-1, 1)

            scaler_y.fit(self.y)
            self.y = scaler_y.transform(self.y)
        # [...]
        if self.cont_cols:
            scaler_X_cont = StandardScaler()
            self.cont_X = data[self.cont_cols].astype(np.float32).values
            scaler_X_cont.fit(self.cont_X)
            self.cont_X = scaler_X_cont.transform(self.cont_X)
        # [...]

As a side note, you don't need X and y when you have a dataset that provides the actual data, you can simply pass it to net.fit (with the exception of using a stratified CV split):

net = NeuralNetRegressor(
    net,
    max_epochs=5,
    lr=0.00001,
)
net.fit(ds, y=None)


Instruction: Is it possible to split the training DataLoader (and dataset) into training and validation datasets?

Input:
The torchvision package provides easy access to commonly used datasets. You would use them like this:

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

Apparently, you can only switch between train=True and train=False. The docs explain:

  train (bool, optional) – If True, creates dataset from training.pt,
  otherwise from test.pt.

But this goes against the common practice of having a three-way split. For serious work, I need another DataLoader with a validation set. Also, it would be nice to specify the split proportions myself. They don't say what percentage of the dataset is reserved for testing, maybe I would like to change that.

I assume that this is a conscious design decision. Everyone working on one of these datasets is supposed to use the same testset. That makes results comparable. But I still need to get a validation set out of the trainloader. Is it possible to split a DataLoader into two separate streams of data?

Output:
Meanwhile, I stumbled upon the method random_split. So, you don't split the DataLoader, but you split the Dataset:

torch.utils.data.random_split(dataset, lengths)


Instruction: PyTorch DataLoader and Parallelism

Input:
I have created a class that extends DataSet to load images for a segmentation task, so one input and one output. Every time the method getitem is called, this class performs the necessary operations for data augmentation on both the input and the output, and it works perfectly.

However, when I use this class with PyTorch DataLoader, the input transformation do not match with the output transformations. To perform the same operations, I have to get/set the states of random operations/classes, and my bet is that the DataLoader does the same, so there is a conflict between them.

How can I fix it?

Output: The solution is to create a local instance of all the Random classes uses, because the DataLoader does not. Doing that, all the random transformations performed are according to random values/states that are not affected by the DataLoader. The common way to do it seems to create a class and put all the transformations inside.


Instruction: list of fastai models

Input:
FastAI uses AWD-LSTM for text processing. They provide pretrained models with get_language_model(). But I can't find proper documentation on what's available.

Their github example page is really a moving target. Model names such as lstm_wt103 and WT103_1 are used. In the forums I found wt103RNN.

Where can I find an updated list of pretrained models and their download URLs ?

Output:
URLs is defined in fastai.datasets, there are constants for two models: WT103, WT103_1.

AWS bucket has just the two models.


Instruction: How to batch convert sentence lengths to masks in PyTorch?

Input:
For example, from

lens = [3, 5, 4]

we want to get

mask = [[1, 1, 1, 0, 0],
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 0]]

Both of which are torch.LongTensors.

Output:
One way that I found is:

torch.arange(max_len).expand(len(lens), max_len) < lens.unsqueeze(1)

Please share if there are better ways!


Instruction: Using nn.ModuleList over Python list dramatically slows down training

Input:
I'm training a very simple model that takes the number of hidden layers as a parameter. I originally stored these hidden layers in a vanilla python list [], however when converting this list to a nn.ModuleList, training slows down dramatically by at least one order of magnitude!

AdderNet

class AdderNet(nn.Module):
    def __init__(self, num_hidden, hidden_width):
        super(AdderNet, self).__init__()
        self.relu = nn.ReLU()

        self.hiddenLayers = []
        self.inputLayer = nn.Linear(2, hidden_width)
        self.outputLayer = nn.Linear(hidden_width, 1)

        for i in range(num_hidden):
            self.hiddenLayers.append(nn.Linear(hidden_width, hidden_width))

        self.hiddenLayers = nn.ModuleList(self.hiddenLayers)  # <--- causes DRAMATIC slowdown!

    def forward(self, x):
        out = self.inputLayer(x)
        out = self.relu(out)

        for layer in self.hiddenLayers:
            out = layer(out)
            out = self.relu(out)

        return self.outputLayer(out)

Training

for epoch in range(num_epochs):
    for i in range(0,len(data)):
        out = model.forward(data[i].x)
        loss = lossFunction(out, data[i].y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Output: That's because when using a normal python list, the parameters are not added to the model's parameter list, but when using a ModuleList, they are. So, in the original scenario, you were never actually training the hidden layers, which is why it was faster. (Print out model.parameters() in each case and see what happens!)


Instruction: How to print the "actual" learning rate in Adadelta in pytorch

Input:
In short:

I can't draw lr/epoch curve when using adadelta optimizer in pytorch because optimizer.param_groups[0]['lr'] always return the same value.

In detail:

Adadelta can dynamically adapts over time using only first order information and
has minimal computational overhead beyond vanilla stochastic gradient descent [1].

In pytorch, the source code of Adadelta is here https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html#Adadelta

Since it requires no manual tuning of learning rate, in my knowledge, we don't have to set any schedular after declare the optimizer

self.optimizer = torch.optim.Adadelta(self.model.parameters(), lr=1)

The way to check learning rate is

current_lr = self.optimizer.param_groups[0]['lr']

The problem is it always return 1 (the initial lr).

Could anyone tell me how can I get the true learning rate so that can I draw a lr/epch curve?

[1] https://arxiv.org/pdf/1212.5701.pdf

Output:
Check: self.optimizer.state. This is optimized with the lr and used in optimization process.

From documentation a lr is just:

  lr (float, optional): coefficient that scale delta before it is
  applied
                  to the parameters (default: 1.0)

https://pytorch.org/docs/stable/_modules/torch/optim/adadelta.html

Edited: you may find acc_delta values in self.optimizer.state values but you need to go through dictionaries contained by this dictionary:

dict_with_acc_delta = [self.optimizer.state[i] for i in self.optimizer.state.keys() if "acc_delta" in self.optimizer.state[i].keys()]
acc_deltas = [i["acc_delta"] for i in dict_with_acc_delta]

I have eight layers and shapes of elements in the acc_deltas list are following

[torch.Size([25088]),
 torch.Size([25088]),
 torch.Size([4096, 25088]),
 torch.Size([4096]),
 torch.Size([1024, 4096]),
 torch.Size([1024]),
 torch.Size([102, 1024]),
 torch.Size([102])]


Instruction: RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[3, 1, 224, 224] to have 3 channels, but got 1 channels instead

Input:
In the code below:

    model_ft.eval()
    test_data, test_target = image_datasets['train'][idx]
    test_data = test_data.cuda()
    #test_target = test_target.cuda()
    test_target = torch.tensor(test_target)
    test_target = test_target.cuda()
    test_data.unsqueeze_(1)
    test_target.unsqueeze_(0)
    print(test_data.shape)
    output = model_ft(test_data)

I get the following error:

Traceback (most recent call last):
  File "test_loocv.py", line 245, in <module>
    output = model_ft(test_data)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torchvision-0.2.1-py3.6.egg/torchvision/models/resnet.py", line 139, in forward
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch/sjn-p3/anaconda/anaconda3/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[3, 1, 224, 224] to have 3 channels, but got 1 channels instead

Also, test_data has the shape: torch.Size([3, 1, 224, 224]).

How should I fix this?

Output:
Here's the fix:

test_data, test_target = image_datasets['train'][idx]
test_data = test_data.cuda()
test_target = torch.tensor(test_target)
test_target = test_target.cuda()
test_data.unsqueeze_(0)
test_target.unsqueeze_(0)
output = model_ft(test_data)

I had to change test_data.unsqueeze_(1) to test_data.unsqueeze_(0)


Instruction: Pytorch: nn.Dropout vs. F.dropout

Input:
There are two ways to perform dropout:

torch.nn.Dropout
torch.nn.functional.Dropout

I ask:

Is there a difference between them?
When should I use one over the other?

I don't see any performance difference when I switched them around.

Output:
The technical differences have already been shown in the other answer. However the main difference is that nn.Dropout is a torch Module itself which bears some convenience:

A short example for illustration of some differences:

import torch
import torch.nn as nn

class Model1(nn.Module):
    # Model 1 using functional dropout
    def __init__(self, p=0.0):
        super().__init__()
        self.p = p

    def forward(self, inputs):
        return nn.functional.dropout(inputs, p=self.p, training=True)

class Model2(nn.Module):
    # Model 2 using dropout module
    def __init__(self, p=0.0):
        super().__init__()
        self.drop_layer = nn.Dropout(p=p)

    def forward(self, inputs):
        return self.drop_layer(inputs)
model1 = Model1(p=0.5) # functional dropout
model2 = Model2(p=0.5) # dropout module

# creating inputs
inputs = torch.rand(10)
# forwarding inputs in train mode
print('Normal (train) model:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
print()

# switching to eval mode
model1.eval()
model2.eval()

# forwarding inputs in evaluation mode
print('Evaluation mode:')
print('Model 1', model1(inputs))
print('Model 2', model2(inputs))
# show model summary
print('Print summary:')
print(model1)
print(model2)

Output:

Normal (train) model:
Model 1 tensor([ 1.5040,  0.0000,  0.0000,  0.8563,  0.0000,  0.0000,  1.5951,
         0.0000,  0.0000,  0.0946])
Model 2 tensor([ 0.0000,  0.3713,  1.9303,  0.0000,  0.0000,  0.3574,  0.0000,
         1.1273,  1.5818,  0.0946])

Evaluation mode:
Model 1 tensor([ 0.0000,  0.3713,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  0.0000])
Model 2 tensor([ 0.7520,  0.1857,  0.9651,  0.4281,  0.7883,  0.1787,  0.7975,
         0.5636,  0.7909,  0.0473])
Print summary:
Model1()
Model2(
  (drop_layer): Dropout(p=0.5)
)

So which should I use?

Both are completely equivalent in terms of applying dropout and even though the differences in usage are not that big, there are some reasons to favour the nn.Dropout over nn.functional.dropout:

Dropout is designed to be only applied during training, so when doing predictions or evaluation of the model you want dropout to be turned off.

The dropout module nn.Dropout conveniently handles this and shuts dropout off as soon as your model enters evaluation mode, while the functional dropout does not care about the evaluation / prediction mode.

Even though you can set functional dropout to training=False to turn it off, it is still not such a convenient solution like with nn.Dropout.

Also the drop rate is stored in the module, so you don't have to save it in an extra variable. In larger networks you might want to create different dropout layers with different drop rates - here nn.Dropout may increase readability and can bear also some convenience when using the layers multiple times.

Finally, all modules which are assigned to your model are registered in your model. So you model class keeps track of them, that is why you can just turn off the dropout module by calling eval(). When using the functional dropout your model is not aware of it, thus it won't appear in any summary.


Instruction: how to save torchtext Dataset?

Input:
I'm working with text and use torchtext.data.Dataset.
Creating the dataset takes a considerable amount of time.
For just running the program this is still acceptable. But I would like to debug the torch code for the neural network. And if python is started in debug mode, the dataset creation takes roughly 20 minutes (!!). That's just to get a working environment where I can debug-step through the neural network code.

I would like to save the Dataset, for example with pickle. This sample code is taken from here, but I removed everything that is not necessary for this example:

from torchtext import data
from fastai.nlp import *

PATH = 'data/aclImdb/'

TRN_PATH = 'train/all/'
VAL_PATH = 'test/all/'
TRN = f'{PATH}{TRN_PATH}'
VAL = f'{PATH}{VAL_PATH}'

TEXT = data.Field(lower=True, tokenize="spacy")

bs = 64;
bptt = 70

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=10)

with open("md.pkl", "wb") as file:
    pickle.dump(md, file)

To run the code, you need the aclImdb dataset, it can be downloaded from here. Extract it into a data/ folder next to this code snippet. The code produces an error in the last line, where pickle is used:

Traceback (most recent call last):
  File "/home/lhk/programming/fastai_sandbox/lesson4-imdb2.py", line 27, in <module>
    pickle.dump(md, file)
TypeError: 'generator' object is not callable

The samples from fastai often use dill instead of pickle. But that doesn't work for me either.

Output:
You can use dill instead of pickle. It works for me.
You can save a torchtext Field like

TEXT = data.Field(sequential=True, tokenize=tokenizer, lower=True,fix_length=200,batch_first=True)
with open("model/TEXT.Field","wb")as f:
     dill.dump(TEXT,f)

And load a Field like

with open("model/TEXT.Field","rb")as f:
     TEXT=dill.load(f)

The offical code suppport is under development，you can follow https://github.com/pytorch/text/issues/451 and https://github.com/pytorch/text/issues/73 .


Instruction: AttributeError: 'Compose' object has no attribute 'Compose' (in Pytorch 0.2.1)

Input:
This is the block of code where am getting this error:

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.5, 0.5, 0.5],
                                                            [0.5, 0.5, 0.5])])

I've tried updating my torchvision but had no luck!

Output:
The problem is that you have a variable called transforms after from torchvision import transforms which has a compose of a certain type. This override the transform you import from the torchvison. Therefore when you run the above code it calls the transforms which is a variable not the one from torchvision module.

It is advisable to rename the variable or if you are using jupyter notebook run the cell where you import transforms before running the cell with the code above.


Instruction: how can I insert a Tensor into another Tensor in pytorch

Input:
I have pytorch Tensor with shape (batch_size, step, vec_size), for example, a Tensor(32, 64, 128), let's call it A.

I have another Tensor(batch_size, vec_size), e.g. Tensor(32, 128), let's call it B.

I want to insert B into a certain position at axis 1 of A. The insert positions are given in a Tensor(batch_size), named P.

I understand there is no Empty tensor(like an empty list) in pytorch, so, I initialize A as zeros, and add B at a certain position at axis 1 of A.

A = Variable(torch.zeros(batch_size, step, vec_size))

What I'm doing is like:

for i in range(batch_size):
    pos = P[i]
    A[i][pos] = A[i][pos] + B[i]

But I get an Error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Then, I make a clone of A each inside the loop:

for i in range(batch_size):
    A_clone =  A.clone()
    pos = P[i]
    A_clone[i][pos] = A_clone[i][pos] + B[i]

This is very slow for autograd, I wonder if there any better solutions? Thank you.

Output:
You can use a mask instead of cloning.

See the code below

# setup
batch, step, vec_size = 64, 10, 128
A = torch.rand((batch, step, vec_size))
B = torch.rand((batch, vec_size))
pos = torch.randint(10, (64,)).long()

# computations
# create a mask where pos is 0 if it is to be replaced
mask = torch.ones( (batch, step)).view(batch,step,1).float()
mask[torch.arange(batch), pos]=0

# expand B to have same dimension as A and compute the result
result = A*mask + B.unsqueeze(dim=1).expand([-1, step, -1])*(1-mask)

This way you avoid using for loops and cloning as well.


Instruction: How to vectorize the following python code

Input:
I'm trying to obtain a matrix, where each element is calculated as follows:

X = torch.ones(batch_size, dim)
X_ = torch.ones(batch_size, dim)
Y = torch.ones(batch_size, dim)
M = torch.zeros(batch_size, batch_size)
for i in range(batch_size):
    for j in range(batch_size):
        M[i, j] = ((X[i] - X_[i] * Y[j])**2).sum()

It's very slow to calculate M element-wise, is there any suggestion about how to use matrix multiplication to replace the for loops?

Thanks.

Output:
If you want to sum() over dim, you can "lift" your 2D problem to 3D and sum there:

M = ((X[:, None, :] - X_[:, None, :] * Y[None, ...])**2).sum(dim=2)

How it works:

X[:, None, :] and X_[:, None, :] are 3D of size (batch_size, 1, dim), and Y[None, ...] is of size (1, batch_size, dim).

When multiplying X_[:, None, :] * Y[None, ...] pytorch broadcasts the dimensions of size 1 to the appropriate dimension to get a result of size (batch_size, batch_size, dim).
Finally, you sum() only over the last dimension (dim=2) to get an output M of size (batch_size, batch_size).

The trick here is done by taking advantage of broadcasting.


Instruction: What is the PyTorch alternative for Keras input_shape, output_shape, get_weights, get_config and summary

Input:
In Keras, after creating a model, we can see its input and output shapes using model.input_shape, model.output_shape. For weights and config  we can use model.get_weights() and model.get_config(), respectively.

What are the similar alternatives for PyTorch? Also is there any other functions we need to know for inspecting a PyTorch model?

To get the summary in PyTorch, we print model print(model) but this gives lesser information than model.summary(). Is there a better summary for PyTorch?

Output:
There is no "model.summary()" method in pytorch. You need to use built_in methods and fields of the model.

For example, I have customized inception_v3 model. To get the information I need to use other many different fields. For instance:

IN:

print(model) # print network architecture

OUT

Inception3(
  (Conv2d_1a_3x3): BasicConv2d(
    (conv): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)
    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (Conv2d_2a_3x3): BasicConv2d(
    (conv): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (Conv2d_2b_3x3): BasicConv2d(
    (conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (Conv2d_3b_1x1): BasicConv2d(
    (conv): Conv2d(64, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(80, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)
  )
  (Conv2d_4a_3x3): BasicConv2d(
    (conv): Conv2d(80, 192, kernel_size=(3, 3), stride=(1, 1), bias=False)
   ...

IN:

for i in model.state_dict().keys():
    print(i)
#print keys of dict with values of learned weights, bias, parameters

OUT:

    Conv2d_1a_3x3.conv.weight
    Conv2d_1a_3x3.bn.weight
    Conv2d_1a_3x3.bn.bias
    Conv2d_1a_3x3.bn.running_mean
    Conv2d_1a_3x3.bn.running_var
    Conv2d_1a_3x3.bn.num_batches_tracked
    Conv2d_2a_3x3.conv.weight
    Conv2d_2a_3x3.bn.weight
    Conv2d_2a_3x3.bn.bias
    Conv2d_2a_3x3.bn.running_mean
    ...

So if I would like to get weights for the layer CNN at Conv2d_1a_3x3, I look for the key "Conv2d_1a_3x3.conv.weight":

print("model.save_dict()["Conv2d_1a_3x3.conv.weight"])

OUT:

tensor([[[[-0.2103, -0.3441, -0.0344],
          [-0.1420, -0.2520, -0.0280],
          [ 0.0736,  0.0183,  0.0381]],

         [[ 0.1417,  0.1593,  0.0506],
          [ 0.0828,  0.0854,  0.0186],
          [ 0.0283,  0.0144,  0.0508]],
...

If you want to see used hyperparameters from optimizer:

optimizer.param_groups

OUT:

[{'dampening': 0,
  'lr': 0.01,
  'momentum': 0.01,
  'nesterov': False,
  'params': [Parameter containing:
   tensor([[[[-0.2103, -0.3441, -0.0344],
             [-0.1420, -0.2520, -0.0280],
             [ 0.0736,  0.0183,  0.0381]],
          ...