Instruction: Pytorch PPO implementation is not learning

Input:
This PPO implementation has a bug somewhere and I can't figure out what's wrong. The network returns a normal distribution and a value estimate from the critic. The last layer of the actor provides four F.tanhed action values, which are used as mean value for the distribution. nn.Parameter(torch.zeros(action_dim)) is the standard deviation.

The trajectories for 20 parallel agents are added to the same memory. Episode length is 1000 and memory.sample() returns a np.random.permutation of the 20k memory entries as tensors with batches of size 64. Before stacking the batch tensors, the values are stored as (1,-1) tensors in collection.deques. The returned tensors are detach()ed.

environment

brain_name = envs.brain_names[0]
env_info = envs.reset(train_mode=True)[brain_name]
env_info = envs.step(actions.cpu().detach().numpy())[brain_name]
next_states = env_info.vector_observations
rewards = env_info.rewards
dones = env_info.local_done

update step

def clipped_surrogate_update(policy, memory, num_epochs=10, clip_param=0.2, gradient_clip=5, beta=0.001, value_loss_coeff=0.5):

    advantages_batch, states_batch, log_probs_old_batch, returns_batch, actions_batch = memory.sample()

    advantages_batch = (advantages_batch - advantages_batch.mean()) / advantages_batch.std()

    for _ in range(num_epochs):
        for i in range(len(advantages_batch)):

            advantages_sample = advantages_batch[i]
            states_sample = states_batch[i]
            log_probs_old_sample = log_probs_old_batch[i]
            returns_sample = returns_batch[i]
            actions_sample = actions_batch[i]

            dist, values = policy(states_sample)

            log_probs_new = dist.log_prob(actions_sample.to(device)).sum(-1).unsqueeze(-1)
            entropy = dist.entropy().sum(-1).unsqueeze(-1).mean()

            ratio = (log_probs_new - log_probs_old_sample).exp()

            clipped_ratio = torch.clamp(ratio, 1-clip_param, 1+clip_param)
            clipped_surrogate_loss = -torch.min(ratio*advantages_sample, clipped_ratio*advantages_sample).mean()
            value_function_loss = (returns_sample - values).pow(2).mean()

            Loss = clipped_surrogate_loss - beta * entropy + value_loss_coeff * value_function_loss

            optimizer_policy.zero_grad()
            Loss.backward()
            torch.nn.utils.clip_grad_norm_(policy.parameters(), gradient_clip)
            optimizer_policy.step()
            del Loss

data sampling

def collect_trajectories(envs, env_info, policy, memory, tmax=200, nrand=0, gae_tau = 0.95, discount = 0.995):

    next_episode = False

    states = env_info.vector_observations
    n_agents = len(env_info.agents)

    state_list=[]
    reward_list=[]
    prob_list=[]
    action_list=[]
    value_list=[]

    if nrand > 0:
        # perform nrand random steps
        for _ in range(nrand):
            actions = np.random.randn(num_agents, action_size)
            actions = np.clip(actions, -1, 1)
            env_info = envs.step(actions)[brain_name]
            states = env_info.vector_observations

    for t in range(tmax):

        states = torch.FloatTensor(states).to(device)
        dist, values = policy(states)

        actions = dist.sample()
        probs = dist.log_prob(actions).sum(-1).unsqueeze(-1)

        env_info = envs.step(actions.cpu().detach().numpy())[brain_name]
        next_states = env_info.vector_observations
        rewards = env_info.rewards
        dones = env_info.local_done

        state_list.append(states)
        reward_list.append(rewards)
        prob_list.append(probs)
        action_list.append(actions)
        value_list.append(values)

        states = next_states

        if np.any(dones):
            next_episode = True
            break

    _, next_value = policy(torch.FloatTensor(states).to(device))

    reward_arr = np.array(reward_list)
    undiscounted_rewards = np.sum(reward_arr, axis=0)

    state_arr = torch.stack(state_list)
    prob_arr = torch.stack(prob_list)
    action_arr = torch.stack(action_list)
    value_arr = torch.stack(value_list)

    reward_arr = torch.FloatTensor(reward_arr[:, :, np.newaxis])

    advantage_list = []
    return_list = []

    returns = next_value.detach()
    advantages = torch.FloatTensor(np.zeros((n_agents, 1)))
    for i in reversed(range(state_arr.shape[0])):

        returns = reward_arr[i] + discount * returns

        td_error = reward_arr[i] + discount * next_value - value_arr[i]
        advantages = advantages * gae_tau * discount + td_error
        next_value = value_arr[i]

        advantage_list.append(advantages.detach())
        return_list.append(returns.detach())

    advantage_arr = torch.stack(advantage_list)
    return_arr = torch.stack(return_list)

    for i in range(state_arr.shape[0]):
        memory.add({'advantages': advantage_arr[i],
                    'states': state_arr[i],
                    'log_probs_old': prob_arr[i],
                    'returns': return_arr[i],
                    'actions': action_arr[i]})

    return undiscounted_rewards, next_episode

Output:
In the Generalized Advantage Estimation loop advantages and returns are added in reversed order.

advantage_list.insert(0, advantages.detach())
return_list.insert(0, returns.detach())


Instruction: Siamese Neural Network in Pytorch

Input:
How can I implement a siamese neural network in PyTorch?

  What is a siamese neural network? A siamese neural network consists in two identical neural networks, each one taking one input. Identical means that the two neural networks have the exact same architecture and share the same weights.

Output:
Implementing siamese neural networks in PyTorch is as simple as calling the network function twice on different inputs.

mynet = torch.nn.Sequential(
        nn.Linear(10, 512),
        nn.ReLU(),
        nn.Linear(512, 2))
...
output1 = mynet(input1)
output2 = mynet(input2)
...
loss.backward()

When invoking loss.backwad(), PyTorch will automatically sum the gradients coming from the two invocations of mynet.

You can find a full-fledged example here.


Instruction: Unexpected result of convolution operation

Input:
Here is code I wrote to perform a single convolution and output the shape.

Using formula from http://cs231n.github.io/convolutional-networks/ to calculate output size :

  You can convince yourself that the correct formula for calculating how
  many neurons “fit” is given by (W−F+2P)/S+1

The formula for computing the output size has been implemented below as

def output_size(w , f , stride , padding) :
        return (((w - f) + (2 * padding)) / stride) + 1

The issue is output_size computes a size of 2690.5 which differs to the result of the convolution which is 1350 :

%reset -f

import torch
import torch.nn.functional as F
import numpy as np
from PIL import Image
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
from pylab import plt
plt.style.use('seaborn')
%matplotlib inline

width = 60
height = 30
kernel_size_param = 5
stride_param = 2
padding_param = 2

img = Image.new('RGB', (width, height), color = 'red')

in_channels = 3
out_channels = 3

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels,
                      out_channels,
                      kernel_size=kernel_size_param,
                      stride=stride_param,
                      padding=padding_param))

    def forward(self, x):
        out = self.layer1(x)

        return out

# w : input volume size
# f : receptive field size of the Conv Layer neurons
# output_size computes spatial size of output volume - spatial dimensions are (width, height)
def output_size(w , f , stride , padding) :
    return (((w - f) + (2 * padding)) / stride) + 1

w = width * height * in_channels
f = kernel_size_param * kernel_size_param

print('output size :' , output_size(w , f , stride_param , padding_param))

model = ConvNet()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=.001)

img_a = np.array(img)
img_pt = torch.tensor(img_a).float()
result = model(img_pt.view(3, width , height).unsqueeze_(0))
an = result.view(30 , 15 , out_channels).data.numpy()

# print(result.shape)
# print(an.shape)

# print(np.amin(an.flatten('F')))

print(30 * 15 * out_channels)

Have I implemented output_size correctly ? How to amend this model so the result of Conv2d has same shape as result of output_size ?

Output:
The problem is that your input image is not a square, so you should apply the formula on the width and the heigth of the input image.
And also you should not use the nb_channels in the formula because we are explicitly defining how many channels we want in the output.
Then you use your f=kernel_size and not f=kernel_size*kernel_size as described in the formula.

w = width
h = height
f = kernel_size_param
output_w =  int(output_size(w , f , stride_param , padding_param))
output_h =  int(output_size(h , f , stride_param , padding_param))
print("Output_size", [out_channels, output_w, output_h]) #--> [1, 3, 30 ,15]

And then output size :

print("Output size", result.shape)  #--> [1, 3, 30 ,15]

Formula source : http://cs231n.github.io/convolutional-networks/


Instruction: IndexError when iterating my dataset using Dataloader in PyTorch

Input:
I iterated my dataset using Dataloader in PyTorch 0.2 like these:

dataloader = torch.utils.data.DataLoader(...)
data_iter = iter(dataloader)
data = data_iter.next()

but IndexError was raised.

Traceback (most recent call last):
  File "main.py", line 193, in <module>
    data_target = data_target_iter.next()
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
IndexError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/asr4/zhuminxian/adversarial/code/dataset/data_loader.py", line 33, in __getitem__
    return self.X_train[idx], self.y_train[idx]
IndexError: index 4196 is out of bounds for axis 0 with size 4135

I am wondering why the index was out of bounds. Is it the bug of Pytorch?

I tried to run my code again, the same error raised, but at different iteration and with different out-of-bound index.

Output:
My guess is that your data.Dataset.__len__ was not overloaded properly and in-fact len(dataloader.dataset) returns a number larger than len(self.X_train).
Check your implementation of the underlying dataset in '/home/asr4/zhuminxian/adversarial/code/dataset/data_loader.py'.


Instruction: vgg probability doesn't add up to 1, pytorch

Input:
I've trained a vgg16 model to predict 102 classes of flowers.
It works however now that I'm trying to understand one of it's predictions I feel it's not acting normally.

model layout

# Imports here
import os
import numpy as np
import torch

import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import json
from pprint import pprint
from scipy import misc

%matplotlib inline

data_dir = 'flower_data'
train_dir = data_dir + '/train'
test_dir = data_dir + '/valid'

json_data=open('cat_to_name.json').read()
main_classes = json.loads(json_data)
main_classes = {int(k):v for k,v in classes.items()}

train_transform_2 = transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.RandomRotation(30),
                                    transforms.RandomHorizontalFlip(),
                                    transforms.ToTensor()])

test_transform_2= transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.ToTensor()])

# TODO: Load the datasets with ImageFolder
train_data = datasets.ImageFolder(train_dir, transform=train_transform_2)
test_data = datasets.ImageFolder(test_dir, transform=test_transform_2)

# define dataloader parameters
batch_size = 20
num_workers=0

# TODO: Using the image datasets and the trainforms, define the dataloaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)

vgg16 = models.vgg16(pretrained=True)

# Freeze training for all "features" layers
for param in vgg16.features.parameters():
    param.requires_grad = False

import torch.nn as nn

n_inputs = vgg16.classifier[6].in_features

# add last linear layer (n_inputs -> 102 flower classes)
# new layers automatically have requires_grad = True
last_layer = nn.Linear(n_inputs, len(classes))

vgg16.classifier[6] = last_layer

import torch.optim as optim

# specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# specify optimizer (stochastic gradient descent) and learning rate = 0.001
optimizer = optim.SGD(vgg16.classifier.parameters(), lr=0.001)

pre_trained_model=torch.load("model.pt")
new=list(pre_trained_model.items())

my_model_kvpair=vgg16.state_dict()
count=0

for key,value in my_model_kvpair.items():
    layer_name, weights = new[count]
    my_model_kvpair[key] = weights
    count+=1

# number of epochs to train the model
n_epochs = 6

# initialize tracker for minimum validation loss
valid_loss_min = np.Inf # set initial "min" to infinity

for epoch in range(1, n_epochs+1):

    # keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    ###################
    # train the model #
    ###################
    # model by default is set to train
    vgg16.train()

    for batch_i, (data, target) in enumerate(train_loader):
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the batch loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # update training loss
        train_loss += loss.item()

        if batch_i % 20 == 19:    # print training loss every specified number of mini-batches
            print('Epoch %d, Batch %d loss: %.16f' %
                (epoch, batch_i + 1, train_loss / 20))
            train_loss = 0.0

    ######################
    # validate the model #
    ######################
    vgg16.eval() # prep model for evaluation
    for data, target in test_loader:
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the loss
        loss = criterion(output, target)
        # update running validation loss
        valid_loss += loss.item()

    # print training/validation statistics
    # calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(test_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch+1,
        train_loss,
        valid_loss
        ))

    # save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
        valid_loss_min,
        valid_loss))
        torch.save(vgg16.state_dict(), 'model.pt')
        valid_loss_min = valid_loss

testing on a single image

tensor = torch.from_numpy(test_image)
reshaped = tensor.permute(2, 0, 1).unsqueeze(0)
floatified = reshaped.to(torch.float32) / 255
vgg16(floatified)
>>>
tensor([[ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291]],
       grad_fn=<ThAddmmBackward>)

sum([ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291])
>>>
5.325799999999998

given this as how I test it on a single image (and the model as usual is trained and tested on batches it returns a prediction matrix that doesn't seem to be normalized or add up to 1.

Is this normal?

Output:
Yes, official network implementations in PyTorch don't apply softmax to the last linear layer. Check the code for VGG. You can use nn.softmax to achieve what you want:

m = nn.Softmax()
out = vgg16(floatified)
out = m(out)

You can also use nn.functional.softmax:

out = nn.functional.softmax(vgg16(floatified))


Instruction: Copying data from one tensor to another using bit masking

Input:
import numpy as np
import torch
a = torch.zeros(5)
b = torch.tensor(tuple((0,1,0,1,0)),dtype=torch.uint8)
c= torch.tensor([7.,9.])
print(a[b].size())
a[b]=c
print(a)

  torch.Size([2])tensor([0., 7., 0., 9., 0.])

I am struggling to understand how this works. I initially thought the above code was using Fancy indexing but I realised that values from c tensors are getting copied corresponding to the indices marked 1. Also, if I don't specify dtype of b as uint8 then the above code does not work. Can someone please explain me the mechanism of the above code.

Output:
Indexing with arrays works the same as in numpy and most other vectorized math packages I am aware of. There are two cases:

When b is of type uint8 (think boolean, pytorch doesn't distinguish bool from uint8), a[b] is a 1-d array containing the subset of values of a (a[i]) for which the corresponding in b (b[i]) was nonzero. These values are aliased to the original a so if you modify them, their corresponding locations will change as well.
The alternative type you can use for indexing is an array of int64, in which case a[b] creates an array of shape (*b.shape, *a.shape[1:]). Its structure is as if each element of b (b[i]) was replaced by a[i]. In other words, you create a new array by specifying from which indexes of a should the data be fetched. Again, the values are aliased to the original a, so if you modify a[b] the values of a[b[i]], for each i, will change. An example usecase is shown in this question.

These two modes are explained for numpy in integer array indexing and boolean array indexing, where for the latter you have to keep in mind that pytorch uses uint8 in place of bool.

Also, if your goal is to copy data from one tensor to another you have to keep in mind that an operation like a[ixs] = b[ixs] is an in-place operation (a is modified in place), which my not play well with autograd. If you want to do out of place masking, use torch.where. An example usecase is shown in this answer.


Instruction: How to assign a new value to a pytorch Variable without breaking backpropagation?

Input:
I have a pytorch variable that is used as a trainable input for a model. At some point I need to manually reassign all values in this variable.
How can I do that without breaking the connections with the loss function?
Suppose the current values are [1.2, 3.2, 43.2] and I simply want them to become [1,2,3].

Edit
At the time I asked this question, I hadn't realized that PyTorch doesn't have a static graph as Tensorflow or Keras do.
In PyTorch, the training loop is made manually and you need to call everything in each training step. (There isn't the notion of placeholder + static graph for later feeding data).
Consequently, we can't "break the graph", since we will use the new variable to perform all the further computations again. I was worried about a problem that happens in Keras, not in PyTorch.

Output:
You can use the data attribute of tensors to modify the values, since modifications on data do not affect the graph. So the graph will still be intact and modifications of the data attribute itself have no influence on the graph. (Operations and changes on data are not tracked by autograd and thus not present in the graph)

Since you haven't given an example, this example is based on your comment statement: 'Suppose I want to change the weights of a layer.'
I used normal tensors here, but this works the same for weight.data and bias.data attributes of a layers.

Here is a short example:

import torch
import torch.nn.functional as F

# Test 1, random vector with CE
w1 = torch.rand(1, 3, requires_grad=True)
loss = F.cross_entropy(w1, torch.tensor([1]))
loss.backward()
print('w1.data', w1)
print('w1.grad', w1.grad)
print()

# Test 2, replacing values of w2 with w1, before CE
# to make sure that everything is exactly like in Test 1 after replacing the values
w2 = torch.zeros(1, 3, requires_grad=True)
w2.data = w1.data
loss = F.cross_entropy(w2, torch.tensor([1]))
loss.backward()
print('w2.data', w2)
print('w2.grad', w2.grad)
print()

# Test 3, replace data after computation
w3 = torch.rand(1, 3, requires_grad=True)
loss = F.cross_entropy(w3, torch.tensor([1]))
# setting values
# the graph of the previous computation is still intact as you can in the below print-outs
w3.data = w1.data
loss.backward()

# data were replaced with values from w1
print('w3.data', w3)
# gradient still shows results from computation with w3
print('w3.grad', w3.grad)

Output:

w1.data tensor([[ 0.9367,  0.6669,  0.3106]])
w1.grad tensor([[ 0.4351, -0.6678,  0.2326]])

w2.data tensor([[ 0.9367,  0.6669,  0.3106]])
w2.grad tensor([[ 0.4351, -0.6678,  0.2326]])

w3.data tensor([[ 0.9367,  0.6669,  0.3106]])
w3.grad tensor([[ 0.3179, -0.7114,  0.3935]])

The most interesting part here is w3. At the time backward is called the values are replaced by values of w1. But the gradients are calculated based on the CE-function with values of original w3. The replaced values have no effect on the graph.
So the graph connection is not broken, replacing had no influence on graph. I hope this is what you were looking for!


Instruction: Error when converting PyTorch model to TorchScript

Input:
I'm trying to follow the PyTorch guide to load models in C++.

The following sample code works:

import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

However, when trying other networks, such as squeezenet (or alexnet), my code fails:

sq = torchvision.models.squeezenet1_0(pretrained=True)
traced_script_module = torch.jit.trace(sq, example)

>> traced_script_module = torch.jit.trace(sq, example)
/home/fabio/.local/lib/python3.6/site-packages/torch/jit/__init__.py:642: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function.
 Detailed error:
Not within tolerance rtol=1e-05 atol=1e-05 at input[0, 785] (3.1476082801818848 vs. 3.945478677749634) and 999 other locations (100.00%)
  _check_trace([example_inputs], func, executor_options, module, check_tolerance, _force_outplace)

Output:
I just figured out that models loaded from torchvision.models are in train mode by default. AlexNet and SqueezeNet both have Dropout layers, making the inference nondeterministic if in train mode. Simply changing to eval mode fixed the issue:

sq = torchvision.models.squeezenet1_0(pretrained=True)
sq.eval()
traced_script_module = torch.jit.trace(sq, example)


Instruction: Pycharm/Pytorch - 'tensor' is not callable

Input:
When creating a pytorch (1.0) tensor :

import torch
W = torch.tensor(([1.0]))

Pycharm (2018.3.1) gives me the following warning :

'tensor' is not callable less... (Ctrl+F1)
Inspection info: This inspection highlights attempts to call objects which are not callable, like, for example, tuples

My code works fine (tensor() is callable) but I'd like to understand and get rid of this warning.

Output:
This has been a known issue to them . The moderator replied with:

  We will fix this in the next release. It’s being tracked at https://github.com/pytorch/pytorch/issues/7318

However, the reported issue was on PyTorch v0.4.1


Instruction: RuntimeError: dimension specified as 0 but tensor has no dimensions

Input:
I was trying to Implement simple NN using the MNIST datasets and I keep getting this error
    import matplotlib.pyplot as plt

import torch
from torchvision import models
from torchvision import datasets, transforms
from torch import nn, optim
import torch.nn.functional as F
import helper

transform = transforms.ToTensor()

train_data = datasets.MNIST(root='data', train=True,
                                   download=True, transform=transform)
test_data = datasets.MNIST(root='data', train=False,
                                  download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size = 20, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size = 20, shuffle=True)
class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.fc1 = nn.Linear(784,10)

    def forward(self,x):

        x = x.view(-1,784)
        x = F.relu(self.fc1(x))
        x = F.log_softmax(x, dim = 1)
        return x

model = Net()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr = 0.003)

epochs = 20
model.train()
for epoch in range(epochs):
    train_loss = 0

    for image, lables in train_data:

        optimizer.zero_grad()

        output = model(image)
        loss = criterion(output, lables)
        loss.backwards()
        optimizer.step()
        train_loss += loss.item()*image.size(0)
    train_loss = train_loss/len(train_data.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f}'.format(epoch+1, train_loss))

Here is the error
    RuntimeError: dimension specified as 0 but tensor has no dimensions

Output:
The issue you're hitting directly is that NLL loss expects a labels (you're spelling it lables btw) tensor of at least 1 dimension and it's getting a 0-dimensional tensor (aka a scalar). If you see this kind of messages, it's good to just print(output.shape, labels.shape) for easier inspection. The source of this error is that you, probably by mistake, run for image, labels in train_data instead of for image, labels in train_loader. The consequence is that your data is not batched - batching the scalars coming out of dataset would create the missing dimension NLLLoss complains about.

Once we fix this, we proceed to fix backwards -> backward and finally len(train_data.dataset) -> len(train_data). Then the loop works (if it's a reasonable net etc, I did not test).

As a side remark, you can combine NLLLoss and log_softmax by using CrossEntropyLoss, which has the benefit of extra numerical stability.


Instruction: Pytorch - inference all images and back-propagate batch by batch

Input:
I have a special use case that I have to separate inference and back-propagation: I have to inference all images and slice outputs into batches followed by back-propagating batches by batches. I don't need to update my network's weights.
I modified snippets of cifar10_tutorial into the following to simulate my problem: j is a variable to represent the index which returns by my own logic and I want the gradient of some variables.
for epoch in range(2):  # loop over the dataset multiple times

    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data
        inputs.requires_grad = True

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)

        for j in range(4): # j is given by external logic in my own case

            loss = criterion(outputs[j, :].unsqueeze(0), labels[j].unsqueeze(0))

            loss.backward()

            print(inputs.grad.data[j, :]) # what I really want

I got the following errors:

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

My questions are:

According to my understanding, the problem arises because the first back-propagate backwards the whole outputs and outputs[1,:].unsqueeze(0) was released so second back-propagate failed. Am I right?

In my case, if I set retain_graph=True, will the code run slower and slower according to this post?

Is there better way to achieve my goal?

Output:
Yes you are correct. When you already back-propagated through outputs the first time (first iteration), the buffers will be freed and it will fail the following time (next iteration of your loop), because then necessary data for this computation have already been removed.
Yes, the graph grows bigger and bigger, so it could be slower depending on GPU (or CPU) usage and your network. I had used this once and it was much slower, however this depends much on your network architecture. But certainly you will need more memory with retain_graph=True than without.
Depending on your outputs and labels shape you should be able to calculate the loss for all your outputs and labels at once:

criterion(outputs, labels)

You have to skip the j-loop then, this would also make your code faster. Maybe you need to reshape (resp. view) your data, but this should work fine.

If you for some reason cannot do that you can manually sum up the loss on a tensor and call backward after the loop. This should work fine too, but is slower than the solution above.

So than your code would look like this:

# init loss tensor
loss = torch.tensor(0.0) # move to GPU if you're using one

for j in range(4):
    # summing up your loss for every j
    loss += criterion(outputs[j, :].unsqueeze(0), labels[j].unsqueeze(0))
    # ...
# calling backward on the summed loss - getting gradients
loss.backward()
# as you call backward now only once on the outputs
# you shouldn't get any error and you don't have to use retain_graph=True

Edit:

The accumulation of the losses and calling later backward is completely equivalent, here is a small example with and without accumulating the losses:

First creating some data data:

# w in this case will represent a very simple model
# I leave out the CE and just use w to map the output to a scalar value
w = torch.nn.Linear(4, 1)
data = [torch.rand(1, 4) for j in range(4)]

data looks like:

[tensor([[0.4593, 0.3410, 0.1009, 0.9787]]),
 tensor([[0.1128, 0.0678, 0.9341, 0.3584]]),
 tensor([[0.7076, 0.9282, 0.0573, 0.6657]]),
 tensor([[0.0960, 0.1055, 0.6877, 0.0406]])]

Let's first do like you're doing it, calling backward for every iteration j separately:

# code for directly applying backward
# zero the weights layer w
w.zero_grad()
for j, inp in enumerate(data):
    # activate grad flag
    inp.requires_grad = True
    # remove / zero previous gradients for inputs
    inp.grad = None
    # apply model (only consists of one layer in our case)
    loss = w(inp)
    # calling backward on every output separately
    loss.backward()
    # print out grad
    print('Input:', inp)
    print('Grad:', inp.grad)
    print()
print('w.weight.grad:', w.weight.grad)

Here is the print-out with every input and the respective gradient and gradients for the model resp. layer w in our simplified case:

Input: tensor([[0.4593, 0.3410, 0.1009, 0.9787]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.1128, 0.0678, 0.9341, 0.3584]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.7076, 0.9282, 0.0573, 0.6657]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.0960, 0.1055, 0.6877, 0.0406]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

w.weight.grad: tensor([[1.3757, 1.4424, 1.7801, 2.0434]])

Now instead of calling backward once for every iteration j we accumulate the values and call backward on the sum and compare the results:

# init tensor for accumulation
loss = torch.tensor(0.0)
# zero layer gradients
w.zero_grad()
for j, inp in enumerate(data):
    # activate grad flag
    inp.requires_grad = True
    # remove / zero previous gradients for inputs
    inp.grad = None
    # apply model (only consists of one layer in our case)
    # accumulating values instead of calling backward
    loss += w(inp).squeeze()
# calling backward on the sum
loss.backward()

# printing out gradients
for j, inp in enumerate(data):
    print('Input:', inp)
    print('Grad:', inp.grad)
    print()
print('w.grad:', w.weight.grad)

Lets take a look at the results:

Input: tensor([[0.4593, 0.3410, 0.1009, 0.9787]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.1128, 0.0678, 0.9341, 0.3584]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.7076, 0.9282, 0.0573, 0.6657]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

Input: tensor([[0.0960, 0.1055, 0.6877, 0.0406]], requires_grad=True)
Grad: tensor([[-0.0999,  0.2665, -0.1506,  0.4214]])

w.grad: tensor([[1.3757, 1.4424, 1.7801, 2.0434]])

When comparing the results we can see that both are the same. This is a very simple example, but nevertheless we can see that calling backward() on every single tensor and summing up tensors and then calling backward() is equivalent in terms of the resulting gradients for both inputs and weights.

When you use CE for all j 's at once as described in 3. you can use the flag reduction='sum' to archive the same behaviour like above with summing up the CE values, default is ‘mean’, which probably leads to slightly different results.


Instruction: How to load and use a pretained PyTorch InceptionV3 model to classify an image

Input:
I have the same problem as How can I load and use a PyTorch (.pth.tar) model which does not have an accepted answer or one I can figure out how to follow the advice given.

I'm new to PyTorch. I am trying to load the pretrained PyTorch model referenced here: https://github.com/macaodha/inat_comp_2018

I'm pretty sure I am missing some glue.

# load the model
import torch
model=torch.load("iNat_2018_InceptionV3.pth.tar",map_location='cpu')

# try to get it to classify an image
imsize = 256
loader = transforms.Compose([transforms.Scale(imsize), transforms.ToTensor()])

def image_loader(image_name):
    """load image, returns cuda tensor"""
    image = Image.open(image_name)
    image = loader(image).float()
    image = Variable(image, requires_grad=True)
    image = image.unsqueeze(0)
    return image.cpu()  #assumes that you're using CPU

image = image_loader("test-image.jpg")

Produces the error:

   in ()
  ----> 1 model.predict(image)

  AttributeError: 'dict' object has no attribute 'predict

Output:
Problem

Your model isn't actually a model. When it is saved, it contains not only the parameters, but also other information about the model as a form somewhat similar to a dict.

Therefore, torch.load("iNat_2018_InceptionV3.pth.tar") simply returns dict, which of course does not have an attribute called predict.

model=torch.load("iNat_2018_InceptionV3.pth.tar",map_location='cpu')
type(model)
# dict

Solution

What you need to do first in this case, and in general cases, is to instantiate your desired model class, as per the official guide "Load models".

# First try
from torchvision.models import Inception3
v3 = Inception3()
v3.load_state_dict(model['state_dict']) # model that was imported in your code.

However, directly inputing the model['state_dict'] will raise some errors regarding mismatching shapes of Inception3's parameters.

It is important to know what was changed to the Inception3 after its instantiation. Luckily, you can find that in the original author's train_inat.py.

# What the author has done
model = inception_v3(pretrained=True)
model.fc = nn.Linear(2048, args.num_classes) #where args.num_classes = 8142
model.aux_logits = False

Now that we know what to change, lets make some modification to our first try.

# Second try
from torchvision.models import Inception3
v3 = Inception3()
v3.fc = nn.Linear(2048, 8142)
v3.aux_logits = False
v3.load_state_dict(model['state_dict']) # model that was imported in your code.

And there you go with successfully loaded model!


Instruction: How do you change the dimension of your input pictures in pytorch?

Input:
i made a convolutional nuralnetwork and i want it to take input pictures and output pictures but when i turn the pictures into tensors they have the wrong dimension :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [20, 3, 5, 5], but got 3-dimensional input of size [900, 1440, 3] instead

how do i change the dimension of the pictures ? and why does it need to be changed?
and how do i make the output an picture?
i tryed to use

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

to normilize the img but it didnt change the dimension .
here is my nuralnet

    def __init__(self):
    super(Net, self).__init__()
    torch.nn.Module.dump_patches = True
    self.conv1 = nn.Conv2d(3, 20, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(20, 16, 5)
    self.fc1 = nn.Linear(16*5*5, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 16*5*5)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 )
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)

    return x

here i get the image and put it into a list:

for i in range(4):
l.append(ImageGrab.grab())

and here is the code that turns the img into an tensor

k=torch.from_numpy(np.asarray(l[1],dtype="int32" ))

Output:
In summary, according to the comments you and I posted:

The error is due to torch.nn only supports mini-batches. The input should be in the form (batch_size, channels, height, width). You seem to be missing the batch dimension. You can add .unsqueeze(0) to add a fake batch dimension in the first position.

In addition to the above, you'll also have to rearrange the dimensions of your image from [HxWxC] to [CxHxW]. This is done by .ToTensor() transformation in PyTorch.

For the size mismatch problem of your input image, you could use transformation like this:

transform = transforms.Compose(
                   [transforms.Resize((32,32)),
                    transforms.ToTensor(),
                    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])


Instruction: Summing contiguous non zero tensor values

Input:
I am trying to find the summation of contiguous non zero tensor values as shown below

Let’s say, I have a tensor A = [1.3, 0.0, 0.6, 0.7, 0.8]. And I want to

1) sum up the contiguous non-zero values of the tensor to output [1.3, 0.0, 2.1] and then choose the maximum which is 2.1.

2) find the indices as well which were used to sum these values. In this case it will be 2, 3, 4.

Output:
My approach is somewhat different from @Anwarvic. I try to do it in one pass. See the function below. It moves through the array and keep a log of max it has seen so far and current sum. Current sum is updated to 0 if we hit a zero or sum up the value to current if non-zero.

def find_continguous_max_sum(t):
    max_,cur=0,0
    max_indices, cur_indices = [], []

    for idx, i in enumerate(t):
        if (i==0.0):
            cur = 0
            cur_indices = []
        else:
            cur = cur+i
            cur_indices.append(idx)
        if (max_ < cur):
            max_ = cur
            max_indices = cur_indices

    return max_, max_indices

#usage
find_contiguous_max_sum(A)


Instruction: How do I modify this PyTorch convolutional neural network to accept a 64 x 64 image and properly output predictions?

Input:
I took this convolutional neural network (CNN) from here. It accepts 32 x 32 images and defaults to 10 classes. However, I have 64 x 64 images with 500 classes. When I pass in 64 x 64 images (batch size held constant at 32), I get the following error.

ValueError: Expected input batch_size (128) to match target batch_size (32).

The stack trace starts at the line loss = loss_fn(outputs, labels). The outputs.shape is [128, 500] and the labels.shape is [32].

The code is listed here for completeness.

class Unit(nn.Module):
    def __init__(self,in_channels,out_channels):
        super(Unit,self).__init__()
        self.conv = nn.Conv2d(in_channels=in_channels,kernel_size=3,out_channels=out_channels,stride=1,padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()

    def forward(self,input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)
        return output

class SimpleNet(nn.Module):
    def __init__(self,num_classes=10):
        super(SimpleNet,self).__init__()

        self.unit1 = Unit(in_channels=3,out_channels=32)
        self.unit2 = Unit(in_channels=32, out_channels=32)
        self.unit3 = Unit(in_channels=32, out_channels=32)

        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.unit4 = Unit(in_channels=32, out_channels=64)
        self.unit5 = Unit(in_channels=64, out_channels=64)
        self.unit6 = Unit(in_channels=64, out_channels=64)
        self.unit7 = Unit(in_channels=64, out_channels=64)

        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.unit8 = Unit(in_channels=64, out_channels=128)
        self.unit9 = Unit(in_channels=128, out_channels=128)
        self.unit10 = Unit(in_channels=128, out_channels=128)
        self.unit11 = Unit(in_channels=128, out_channels=128)

        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.unit12 = Unit(in_channels=128, out_channels=128)
        self.unit13 = Unit(in_channels=128, out_channels=128)
        self.unit14 = Unit(in_channels=128, out_channels=128)

        self.avgpool = nn.AvgPool2d(kernel_size=4)

        self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4, self.unit5, self.unit6
                                 ,self.unit7, self.pool2, self.unit8, self.unit9, self.unit10, self.unit11, self.pool3,
                                 self.unit12, self.unit13, self.unit14, self.avgpool)

        self.fc = nn.Linear(in_features=128,out_features=num_classes)

    def forward(self, input):
        output = self.net(input)
        output = output.view(-1,128)
        output = self.fc(output)
        return output

Any ideas on how to modify this CNN to accept and properly return outputs?

Output:
The problem is an incompatible reshape (view) at the end.
You're using a sort of "flattening" at the end, which is different from a "global pooling". Both are valid for CNNs, but only the global poolings are compatible with any image size.
The flattened net (your case)
In your case, with a flatten, you need to keep track of all image dimensions in order to know how to reshape at the end.
So:

Enter with 64x64
Pool1 to 32x32
Pool2 to 16x16
Pool3 to 8x8
AvgPool to 2x2

Then, at the end you've got a shape of (batch, 128, 2, 2). Four times the final number if the image were 32x32.
Then, your final reshape should be output = output.view(-1,128*2*2).
This is a different net with a different classification layer, though, because in_features=512.
The global pooling net
On the other hand, you could use the same model, same layers and same weights for any image size >= 32 if you replace the last pooling with a global pooling:
def flatChannels(x):
    size = x.size()
    return x.view(size[0],size[1],size[2]*size[3])

def globalAvgPool2D(x):
    return flatChannels(x).mean(dim=-1)

def globalMaxPool2D(x):
    return flatChannels(x).max(dim=-1)

The ending of the model:
    #removed the pool from here to put it in forward
    self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4,
                             self.unit5, self.unit6, self.unit7, self.pool2, self.unit8,
                             self.unit9, self.unit10, self.unit11, self.pool3,
                             self.unit12, self.unit13, self.unit14)

    self.fc = nn.Linear(in_features=128,out_features=num_classes)

def forward(self, input):
    output = self.net(input)
    output = globalAvgPool2D(output) #or globalMaxPool2D
    output = self.fc(output)
    return output


Instruction: PyTorch - How to deactivate dropout in evaluation mode

Input:
This is the model I defined it is a simple lstm with 2 fully connect layers.

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)
    def forward(self, input):
        out,_=self.lstm(input)
        out=nn.Dropout(p=0.3)(out)
        out=self.linear1(out)
        out=nn.Dropout(p=0.3)(out)
        out=self.linear2(out)
        return out

x_train and x_val are float dataframe with shape (4478,30), while y_train and y_val are float df with shape (4478,10)

    x_train.head()
Out[271]:
       0       1       2       3    ...        26      27      28      29
0  1.6110  1.6100  1.6293  1.6370   ...    1.6870  1.6925  1.6950  1.6905
1  1.6100  1.6293  1.6370  1.6530   ...    1.6925  1.6950  1.6905  1.6960
2  1.6293  1.6370  1.6530  1.6537   ...    1.6950  1.6905  1.6960  1.6930
3  1.6370  1.6530  1.6537  1.6620   ...    1.6905  1.6960  1.6930  1.6955
4  1.6530  1.6537  1.6620  1.6568   ...    1.6960  1.6930  1.6955  1.7040

[5 rows x 30 columns]

x_train.shape
Out[272]: (4478, 30)

Define the varible and do one time bp, I can find out the vaildation loss is 1.4941

model=mylstm(30,10,200,100).double()
from torch import optim
optimizer=optim.RMSprop(model.parameters(), lr=0.001, alpha=0.9)
criterion=nn.L1Loss()
input_=torch.autograd.Variable(torch.from_numpy(np.array(x_train)))
target=torch.autograd.Variable(torch.from_numpy(np.array(y_train)))
input2_=torch.autograd.Variable(torch.from_numpy(np.array(x_val)))
target2=torch.autograd.Variable(torch.from_numpy(np.array(y_val)))
optimizer.zero_grad()
output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_),target2)

moniter
Out[274]: tensor(1.4941, dtype=torch.float64, grad_fn=<L1LossBackward>)

But I called forward function again I get a different number due to randomness of dropout

moniter=criterion(model(input2_),target2)
moniter
Out[275]: tensor(1.4943, dtype=torch.float64, grad_fn=<L1LossBackward>)

what should I do that I can eliminate all the dropout in predicting phrase?

I tried eval():

moniter=criterion(model.eval()(input2_),target2)
moniter
Out[282]: tensor(1.4942, dtype=torch.float64, grad_fn=<L1LossBackward>)

moniter=criterion(model.eval()(input2_),target2)
moniter
Out[283]: tensor(1.4945, dtype=torch.float64, grad_fn=<L1LossBackward>)

And pass an addtional parameter p to control dropout:

import copy
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)
    def forward(self, input,p):
        out,_=self.lstm(input)
        out=nn.Dropout(p=p)(out)
        out=self.linear1(out)
        out=nn.Dropout(p=p)(out)
        out=self.linear2(out)
        return out

model=mylstm(30,10,200,100,0.3).double()

output=model(input_)
loss=criterion(output,target)
loss.backward()
optimizer.step()
moniter=criterion(model(input2_,0),target2)
Traceback (most recent call last):

  File "<ipython-input-286-e49b6fac918b>", line 1, in <module>
    output=model(input_)

  File "D:\Users\shan xu\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() missing 1 required positional argument: 'p'

But neither of them worked.

Output:
You have to define your nn.Dropout layer in your __init__ and assign it to your model to be responsive for calling eval().

So changing your model like this should work for you:

class mylstm(nn.Module):
    def __init__(self,input_dim, output_dim, hidden_dim,linear_dim,p):
        super(mylstm, self).__init__()
        self.hidden_dim=hidden_dim
        self.lstm=nn.LSTMCell(input_dim,self.hidden_dim)
        self.linear1=nn.Linear(hidden_dim,linear_dim)
        self.linear2=nn.Linear(linear_dim,output_dim)

        # define dropout layer in __init__
        self.drop_layer = nn.Dropout(p=p)
    def forward(self, input):
        out,_= self.lstm(input)

        # apply model dropout, responsive to eval()
        out= self.drop_layer(out)
        out= self.linear1(out)

        # apply model dropout, responsive to eval()
        out= self.drop_layer(out)
        out= self.linear2(out)
        return out

If you change it like this dropout will be inactive as soon as you call eval().

NOTE: If you want to continue training afterwards you need to call train() on your model to leave evaluation mode.

You can also find a small working example for dropout with eval() for evaluation mode here:
nn.Dropout vs. F.dropout pyTorch


Instruction: Does someone knows the difference between xavier_normal_ and kaiming_normal_?

Input: Like title. Does someone knows the difference between xavier_normal_ and kaiming_normal_?  Only Xavier have an argument 'gain' more than kaiming?

Output:
Read the documentation:

xavier_normal_

  Fills the input Tensor with values according to the method described in “Understanding the difficulty of training deep feedforward neural networks” - Glorot, X. & Bengio, Y. (2010), using a normal distribution.

kaiming_normal_

  Fills the input Tensor with values according to the method described in “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification” - He, K. et al. (2015), using a normal distribution.

The given equations are completely different. For more details you'll have to read those papers.


Instruction: Is there a way to see what's going wrong with a training session in Pytorch?

Input:
I'm training a triplet convolution neural network in Jupyter.
When I execute the cell I just get the * symbol and nothing happens.

I'm not asking for help finding a problem with the code. I would just like to know if there is a troubleshooting possibility that might let me see what is happening.
There is probably something wrong with my data loader, or data format, or model. I will find it myself if Pytorch or somebody has a method to find a clue here. It is not giving me an error. It is just working ad infinitum on something wrong.

I saw a function called 'set_trace()' that could be typed into the block that is supposed to be able to give a clue about the problem. But after putting it in the for loop I get

NameError: name 'set_trace' is not defined

for batch_idx in range(1):
    for batch_idx, (data, target) in enumerate(triplet_train_loader):
        model.train()
        metrics = []
        losses = []
        total_loss = 0
        data = tuple(d.cuda() for d in data)
        optimizer.zero_grad()
        outputs = model(*data)

        loss_outputs = loss_fn(*outputs)
        loss = loss_outputs[0] if type(loss_outputs) in (tuple, list) else loss_outputs
        losses.append(loss.item())
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
        set_trace()
        if batch_idx % log_interval == 0:
            message = 'Train: [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                batch_idx * len(data[0]), len(triplet_train_loader.dataset),
                100. * batch_idx / len(triplet_train_loader), np.mean(losses))
            for metric in metrics:
                message += '\t{}: {}'.format(metric.name(), metric.value())

            print(message)
            losses = []

"Describe expected and actual results"

LOVE FOR IT TO TRAIN. IN ACTUAL RESULT WORLD IT DOES NOT TRAIN.

Output:
NameError: name 'set_trace' is not defined

You mean:

import pdb; pdb.set_trace()


Instruction: Traceback (most recent call last) in Colab when looping through dataloader in pytorch

Input:
I'm working on a project to classify flower images using a pre-trained model vgg19 using pytorch.

I'm relying on the model features only and using a custom classifier.

However on starting a for-loop to feed images to the model classifier and calculate accuracy through epochs I get an error.

I'm not sure what's the problem as the error is a traceback (most recent call last)

Below is my notebook.

The cell throwing the error is below

#training the classifier
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.classifier.parameters(),lr=0.01)

steps = 0
running_loss = 0
epochs = 5
print_every = 5

for epoch in range(epochs):
    for images,labels in train_dataloader:
        steps += 1
        optimizer.zero_grad()

        logps = model.forward(images)
        loss  = criterion(logps,labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()

            with torch.no_grad():
                for images, labels in valid_dataloader:
                    logps = model.forward(images)
                    batch_loss = criterion(logps, labels)

                    test_loss += batch_loss.item()

                    #Calculate accuracy
                    ps = torch.exp(logps)
                    top_p, top_class = ps.topk(5,dim=1)
                    equals = top_class  == labels.view(*top_class.shape)
                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

            print(f"Epoch {epoch+1}/{epochs}.."
             f"Train loss: {running_loss/print_every: .3f}.."
             f"Test loss: {test_loss/len(valid_loader):.3f}.."
             f"Test accuracy: {accuracy/len(valid_loader):.3f}")
            running_loss = 0
            model.train()

The error I get on running the notebook

AttributeError                            Traceback (most recent call last)
<ipython-input-11-c218f8f2b72e> in <module>()
      8
      9 for epoch in range(epochs):
---> 10     for images,labels in train_dataloader:
     11         steps += 1
     12         optimizer.zero_grad()

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--> 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in <listcomp>(.0)
    312         if self.num_workers == 0:  # same-process loading
    313             indices = next(self.sample_iter)  # may raise StopIteration
--> 314             batch = self.collate_fn([self.dataset[i] for i in indices])
    315             if self.pin_memory:
    316                 batch = pin_memory_batch(batch)

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in __getitem__(self, index)
     99         """
    100         path, target = self.samples[index]
--> 101         sample = self.loader(path)
    102         if self.transform is not None:
    103             sample = self.transform(sample)

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in default_loader(path)
    145         return accimage_loader(path)
    146     else:
--> 147         return pil_loader(path)
    148
    149

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in pil_loader(path)
    127     # open path as file to avoid ResourceWarning (https://github.com/python-pillow/Pillow/issues/835)
    128     with open(path, 'rb') as f:
--> 129         img = Image.open(f)
    130         return img.convert('RGB')
    131

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in open(fp, mode)
   2319     return True
   2320
-> 2321
   2322 def new(mode, size, color=0):
   2323     """

/usr/local/lib/python3.6/dist-packages/PIL/Image.py in preinit()
    368
    369
--> 370 def preinit():
    371     """Explicitly load standard file format drivers."""
    372

/usr/local/lib/python3.6/dist-packages/PIL/PpmImagePlugin.py in <module>()
    156 Image.register_save(PpmImageFile.format, _save)
    157
--> 158 Image.register_extensions(PpmImageFile.format, [".pbm", ".pgm", ".ppm"])

AttributeError: module 'PIL.Image' has no attribute 'register_extensions'

Output:
The error is caused because of interference created by the older version of Pillow which is already installed on Colab. You need to upgrade it to the latest version. Use the following code to upgrade to the latest version of Pillow.

!pip uninstall -y Pillow
!pip install Pillow==5.3.0
import PIL.Image

Now, simply restart the Restart the runtime. It'll remove the error.


Instruction: Is my custom loss function correct? (Pytorch)

Input:
I want to do word recognition using a CNN + Classifier, where the input is an image and the output a matrice 10x37. 10 is the maximum number of characters in a word and 37 is the number of letters in my example.

I wrote a custom loss function for this model but I'm not sure if it's correct since I can't get above 80% Test Accuracy.

I'm using Pytorch

class CustomLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.nllloss = nn.NLLLoss()

    def forward(self, output, labels):
        loss = 0
        for i in range(labels.shape[1]):
            loss += self.nllloss(output[:, i, :], labels[:, i])
        loss /= labels.shape[1]
        return loss

Infos:

output.shape = (batch_size, 10, 37)

labels.shape = (batch_size, 10)

Is the loss function correct?
And what my classification problem is called (Multiple Multi class classification) ?

Output:
The loss function is correct.

The problem was in the file containing my training data. It was not correctly created. In fact, I flipped the dimensions in the images (width and height) so the result from my training set was indecipherable for my CNN.

Now that I have solved the problem, I have reached 99.8% test accuracy.


Instruction: Pytorch select tensor

Input: I want to know if Pytorch have a slice function (same as tf). In particular, I want to select the orange color rows.

Output:
You can use slicing as in numpy.
See below

import torch
A = torch.rand((3,5,500))
first_three_rows = A[:, :3, :]

However to get different slices as you asked in the question, you can do

import torch
A = torch.rand((3,5,500))
indices = [2,4,5]
result = torch.cat([A[idx, :index, :] for idx, index in enumerate(indices)] , dim=0)


Instruction: What are Torch Scripts in PyTorch?

Input:
I've just found that PyTorch docs expose something that is called Torch Scripts. However, I do not know:

When they should be used?
How they should be used?
What are their benefits?

Output:
Torch Script is one of two modes of using the PyTorch just in time compiler, the other being tracing. The benefits are explained in the linked documentation:

Torch Script is a way to create serializable and optimizable models from PyTorch code. Any code written in Torch Script can be saved from your Python process and loaded in a process where there is no Python dependency.

The above quote is actually true both of scripting and tracing. So

You gain the ability to serialize your models and later run them outside of Python, via LibTorch, a C++ native module. This allows you to embed your DL models in various production environments like mobile or IoT. There is an official guide on exporting models to C++ here.
PyTorch can compile your jit-able modules rather than running them as an interpreter, allowing for various optimizations and improving performance, both during training and inference. This is equally helpful for development and production.

Regarding Torch Script specifically, in comparison to tracing, it is a subset of Python, specified in detail here, which, when adhered to, can be compiled by PyTorch. It is more laborious to write Torch Script modules instead of tracing regular nn.Module subclasses, but it allows for some extra features over tracing, most notably flow control like if statements or for loops. Tracing treats such flow control as "constant" - in other words, if you have an if model.training clause in your module and trace it with training=True, it will always behave this way, even if you change the training variable to False later on.
To answer your first question, you need to use jit if you want to deploy your models outside Python and otherwise you should use jit if you want to gain some execution performance at the price of extra development effort (as not every model can be straightforwardly made compliant with jit). In particular, you should use Torch Script if your code cannot be jited with tracing alone because it relies on some features such as if statements. For maximum ergonomy, you probably want to mix the two on a case-by-case basis.
Finally, for how they should be used, please refer to all the documentation and tutorial links.


Instruction: Convert PyTorch tensor to python list

Input:
How do I convert a PyTorch Tensor into a python list?
I want to convert a tensor of size [1, 2048, 1, 1] into a list of 2048 elements. My tensor has floating point values. Is there a solution which also works with other data types such as int?

Output:
Use Tensor.tolist() e.g:

>>> import torch
>>> a = torch.randn(2, 2)
>>> a.tolist()
[[0.012766935862600803, 0.5415473580360413],
 [-0.08909505605697632, 0.7729271650314331]]
>>> a[0,0].tolist()
0.012766935862600803

To remove all dimensions of size 1, use a.squeeze().tolist().
Alternatively, if all but one dimension are of size 1 (or you wish to get a list of every element of the tensor) you may use a.flatten().tolist().


Instruction: Problems with LSTM model

Input:
I try to realise LSTM model in PyTorch and got such problem: loss don't reduce.
My task is so: I have sessions with different features. Session length is fixed and equals to 20. My goal is to predict will the last session been skipped or not.
I tried to scale input features, I tried to pass target into features(maybe provided features are absolutely uninformative, I thought this should lead to overfitting and loss should be near 0), but always my loss reduction looks like this:

print(X.shape)
#(82770, 20, 31) where 82770 is count of sessions, 20 is seq_len, 31 is count of features
print(y.shape)
#(82770, 20)

I defined also get_batches function. And yes, I know about problems with last batch in this generator

def get_batches(X, y, batch_size):
'''Create a generator that returns batches of size
   batch_size x seq_length from arr.
'''
assert X.shape[0] == y.shape[0]
assert X.shape[1] == y.shape[1]
assert len(X.shape) == 3
assert len(y.shape) == 2

seq_len = X.shape[1]
n_batches = X.shape[0]//seq_len

for batch_number in range(n_batches):
    #print(batch_number*batch_size, )
    batch_x = X[batch_number*batch_size:(batch_number+1)*batch_size, :, :]
    batch_y = y[batch_number*batch_size:(batch_number+1)*batch_size, :]
    if batch_x.shape[0] == batch_size:
        yield batch_x, batch_y
    else:
        print('batch_x shape: {}'.format(batch_x.shape))
        break

Here is my RNN

class BaseRNN(nn.Module):

def __init__(self, n_features, hidden_size, n_layers, drop_p=0.3, lr=0.001, last_items=10):
    super(BaseRNN, self).__init__()
    # constants
    self.n_features = n_features
    self.hidden_size = hidden_size
    self.n_layers = n_layers
    self.drop_p = drop_p
    self.lr = lr
    self.last_items = last_items

    # layers
    self.lstm = nn.LSTM(
        n_features, n_hidden, n_layers,
        dropout=drop_p, batch_first=True
    )
    self.dropout = nn.Dropout(self.drop_p)
    self.linear_layer = nn.Linear(self.hidden_size, 1)
    self.sigm = nn.Sigmoid()

def forward(self, x, hidden):
    out, hidden = self.lstm(x, hidden)
    batch_size = x.shape[0]
    out = self.dropout(out)
    out = out.contiguous().view(-1, self.hidden_size)
    out = self.linear_layer(out)
    out = self.sigm(out)
    # use only last elements
    out = out.view(batch_size, -1)
    out = out[:, -1]
    return out, hidden

def init_hidden(self, batch_size):
    #initialize with zeros
    weight = next(self.parameters()).data
    hidden = (weight.new(self.n_layers, batch_size, self.hidden_size).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_size).zero_())

    return hidden

Here is my train function:

def train(net, X, y,
      n_epochs=10, batch_size=10, clip=5):
'''
pass
'''
n_features = X.shape[2]
seq_len = X.shape[1]
net.train()
opt = torch.optim.Adam(net.parameters(), lr=net.lr)
criterion = nn.BCELoss()
counter = 0
losses = []
for e in range(n_epochs):
    h = net.init_hidden(batch_size)
    for x, y in get_batches(X=X, y=y, batch_size=batch_size):
        counter += 1
        h = net.init_hidden(batch_size)
        inputs, targets = torch.from_numpy(x).float(), torch.from_numpy(y.astype(int))
        targets = targets[:,-net.last_items:].float().view(net.last_items*batch_size)
        h = tuple([each.data for each in h])
        net.zero_grad()
        output, h = net(inputs, h)
        loss = criterion(output.view(net.last_items*batch_size), targets)
        losses.append(loss.item())
        loss.backward()
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        opt.step()
return losses

Run training:

n_hidden = 100
n_layers = 1
n_features = X.shape[2]
net = BaseRNN(n_features, n_hidden, n_layers,
              lr=0.01, drop_p=0.1, last_items=1)

losses = train(net, X, y, n_epochs=5, batch_size=1000, lr=0.001, clip=5)
plt.plot(losses)

After all these steps I get plot like in the top of my question. I think I get a huge error somewhere because I put target variable in features, but still no loss reduction.
Where I am wrong?

PS.How to generate sample data? I will use real y data and add some noise.

Y = np.array([[0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1],
       [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1],
       [0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0],
       [0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
       [0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
       [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1],
       [1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1]])
print(Y.shape)
#(10, 20)

# add 5 features with random noise
random_noise = np.random.randn(10*20*5).reshape(10,20,5)
X = np.concatenate((Y.reshape(10,20,1), random_noise), axis=2)
print(X.shape)
#(10, 20, 6)

Output: My fail, forgot to scale input features, now works fine.


Instruction: Pytorch not recognizing directory for dataset

Input:
I'm trying to run code for a Deep Convolutional GAN from the official PyTorch site (https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) on my Mac.

When I try loading the data, I keep getting a "FileNotFound" error.

Random Seed:  999
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-1019cc990fb4> in <module>()
 78                                transforms.CenterCrop(image_size),
 79                                transforms.ToTensor(),
---> 80                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
 81                            ]))
 82 # Create the dataloader

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in __init__(self, root, transform, target_transform, loader)
176         super(ImageFolder, self).__init__(root, loader, IMG_EXTENSIONS,
177                                           transform=transform,
--> 178                                           target_transform=target_transform)
179         self.imgs = self.samples

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in __init__(self, root, loader, extensions, transform, target_transform)
 73
 74     def __init__(self, root, loader, extensions, transform=None, target_transform=None):
---> 75         classes, class_to_idx = find_classes(root)
 76         samples = make_dataset(root, class_to_idx, extensions)
 77         if len(samples) == 0:

/anaconda3/lib/python3.6/site-packages/torchvision/datasets/folder.py in find_classes(dir)
 21
 22 def find_classes(dir):
---> 23     classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
 24     classes.sort()
 25     class_to_idx = {classes[i]: i for i in range(len(classes))}

FileNotFoundError: [Errno 2] No such file or directory: 'Users/user1/Downloads/DCGANs/celeba/'

Here is where I tried loading the dataset where dataroot = "Users/user1/Downloads/DCGANs/celeba/"

The dataset is a folder (named celeba) with about 200,000 images.

dataset = dset.ImageFolder(root=dataroot,
                       transform=transforms.Compose([
                           transforms.Resize(image_size),
                           transforms.CenterCrop(image_size),
                           transforms.ToTensor(),
                           transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                       ]))

I tried this on both Atom and Jupyter Notebook, it didn't make a difference.

All help is highly appreciated :)

Output:
The recognizable directory structure starts with /. So I assume, you should be replacing

dataroot = "Users/user1/Downloads/DCGANs/celeba/"

by

dataroot = "/Users/user1/Downloads/DCGANs/celeba/"


Instruction: TypeError: object of type 'numpy.int64' has no len()

Input:
I am making a DataLoader from DataSet in PyTorch.

Start from loading the DataFrame with all dtype as an np.float64

result = pd.read_csv('dummy.csv', header=0, dtype=DTYPE_CLEANED_DF)

Here is my dataset classes.

from torch.utils.data import Dataset, DataLoader
class MyDataset(Dataset):
    def __init__(self, result):
        headers = list(result)
        headers.remove('classes')

        self.x_data = result[headers]
        self.y_data = result['classes']
        self.len = self.x_data.shape[0]

    def __getitem__(self, index):
        x = torch.tensor(self.x_data.iloc[index].values, dtype=torch.float)
        y = torch.tensor(self.y_data.iloc[index], dtype=torch.float)
        return (x, y)

    def __len__(self):
        return self.len

Prepare the train_loader and test_loader

train_size = int(0.5 * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])

train_loader = DataLoader(dataset=train_dataset, batch_size=16, shuffle=True, num_workers=1)
test_loader = DataLoader(dataset=train_dataset)

Here is my csv file

When I try to iterate over the train_loader. It raises the error

for i , (data, target) in enumerate(train_loader):
    print(i)

TypeError                                 Traceback (most recent call last)
<ipython-input-32-0b4921c3fe8c> in <module>
----> 1 for i , (data, target) in enumerate(train_loader):
      2     print(i)

/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
    635                 self.reorder_dict[idx] = batch
    636                 continue
--> 637             return self._process_next_batch(batch)
    638
    639     next = __next__  # Python 2 compatibility

/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _process_next_batch(self, batch)
    656         self._put_indices()
    657         if isinstance(batch, ExceptionWrapper):
--> 658             raise batch.exc_type(batch.exc_msg)
    659         return batch
    660

TypeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 103, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-27-107e03bc3c6a>", line 12, in __getitem__
    x = torch.tensor(self.x_data.iloc[index].values, dtype=torch.float)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 1478, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 2091, in _getitem_axis
    return self._get_list_axis(key, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexing.py", line 2070, in _get_list_axis
    return self.obj._take(key, axis=axis)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 2789, in _take
    verify=True)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/internals.py", line 4537, in take
    new_labels = self.axes[axis].take(indexer)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2195, in take
    return self._shallow_copy(taken)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/range.py", line 267, in _shallow_copy
    return self._int64index._shallow_copy(values, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/numeric.py", line 68, in _shallow_copy
    return self._shallow_copy_with_infer(values=values, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 538, in _shallow_copy_with_infer
    if not len(values) and 'dtype' not in kwargs:
TypeError: object of type 'numpy.int64' has no len()

Related issues:
https://github.com/pytorch/pytorch/issues/10165
https://github.com/pytorch/pytorch/pull/9237
https://github.com/pandas-dev/pandas/issues/21946

Questions:
How to workaround pandas issue here?

Output:
Reference:
https://github.com/pytorch/pytorch/issues/9211

Just add .tolist() to indices line.

def random_split(dataset, lengths):
    """
    Randomly split a dataset into non-overlapping new datasets of given lengths.
    Arguments:
        dataset (Dataset): Dataset to be split
        lengths (sequence): lengths of splits to be produced
    """
    if sum(lengths) != len(dataset):
        raise ValueError("Sum of input lengths does not equal the length of the input dataset!")

    indices = randperm(sum(lengths)).tolist()
    return [Subset(dataset, indices[offset - length:offset]) for offset, length in zip(_accumulate(lengths), lengths)]


Instruction: How to stack 1-dimensional vectors in pytorch

Input:
I am trying to stack 1-dimensional tensors in pytorch but the stack function seems to be interpreting them as 2-d square matrices. Any ideas how to stack 1-d tensors into a new 1-d tensor?

Reproducibility:

a = torch.randn([2])
b = torch.randn([3])
c = torch.stack([a, b]) # want a (5,) tensor

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 2 and 3 in dimension 1 at c:\new-builder_3\win-wheel\pytorch\aten\src\th\generic/THTensorMath.cpp:3616

If I unsqueeze a and b to (2,1) and (3,1) tensors or squeeze them (should have no effect) the error is still present. This seems odd too though because they should both be size 1 in dimension 1 when unsqueezed (and printing their sizes will accurately reflect this), but the error still appears the same, character-for-character.

Additionally, stacking in dimension 1 just makes it say "...2 and 3 in dimension 0...".

This is all in python 3.5.4, pytorch 0.4.1

Output:
You can try cat (official docs)

a = torch.randn([2])
b = torch.randn([3])
c = torch.cat([a, b], dim=0)


Instruction: Conv 1x1 configuration for feature reduction

Input:
I am using 1x1 convolution in the deep network to reduce a feature x: Bx2CxHxW to BxCxHxW. I have three options:

x -> Conv (1x1) ->  Batchnorm-->ReLU. Code will be output = ReLU(BN(Conv(x))). Reference resnet
x -> BN -> ReLU-> Conv. So the code will be output = Conv(ReLU(BN(x))) . Reference densenet
x-> Conv. The code is output = Conv(x)

Which one is most using for feature reduction? Why?

Output:
Since you are going to train your net end-to-end, whatever configuration you are using - the weights will be trained to accommodate them.

BatchNorm?
I guess the first question you need to ask yourself is do you want to use BatchNorm? If your net is deep and you are concerned with covariate shifts then you probably should have a BatchNorm -- and here goes option no. 3

BatchNorm first?
If your x is the output of another conv layer, than there's actually no difference between your first and second alternatives: your net is a cascade of ...-conv-bn-ReLU-conv-BN-ReLU-conv-... so it's only an "artificial" partitioning of the net into triplets of functions conv, bn, relu and up to the very first and last functions you can split things however you wish. Moreover, since Batch norm is a linear operation (scale + bias) it can be "folded" into an adjacent conv layer without changing the net, so you basically left with conv-relu pairs.
So, there's not really a big difference between the first two options you highlighted.

What else to consider?
Do you really need ReLU when changing dimension of features? You can think of the reducing dimensions as a linear mapping - decomposing the weights mapping to x into a lower rank matrix that ultimately maps into c dimensional space instead of 2c space. If you consider a linear mapping, then you might omit the ReLU altogether.
See fast RCNN SVD trick for an example.


Instruction: What particular change of formula in target changes neural network from gradient descent into gradient ascent?

Input: It was weird when I face it in reinforcement learning. A loss is MSE. Everything should be perfect to be gradient descent and now it is a gradient ascent. I wanna know the magic. I did numpy neural network. Change in a derivative lead to gradient ascent. What particular change in a derivative lead to gradient ascent? Is it that simple that autograd sees that it is concave or convex?

Output:
If you're doing gradient ascent, it must mean that you are doing a variant of policy gradients reinforcement learning.
Doing gradient ascent is extremely simple, long story short, you just apply gradient descent, except you put a minus sign in front of the gradient term!
In tensorflow code:
gradients = - tf.compute_gradients(loss)
update = tf.apply_gradients(zip(gradients, vars))

This is the basic gradient descent algorithm, where theta is the weights of the model, alpha is learning rate, and dJ/dtheta is the gradient of the loss function with respect to the weights.

In the above, we descent upon the gradient because we want to minimize the loss.  But in policy gradient methods, we want to maximize the returns, and since we are taking the gradient with respect to the reward (intuitively), we want to maximize it.
Please see the below picture from TowardsDataScience, you can see that naturally, the weights get updated to the direction of lowest J.  (Notice the positive instead of negative)

By simply changing the sign of the update, we can instead go the other way (i.e., to maximize the reward

Below is the formal equation to gradient asent for policy gradient methods.  The gradient of the policy * Vt is essentially dJ/dtheta.


Instruction: Indexing a batched set of images

Input:
Suppose I have two index tensors and an image tensor, how can I sample the (x, y) points from the image?

img.shape # -> (batch x H x W x 3)
x.shape # -> (batch x H x W)
y.shape # -> batch x H x W)

(H x W being height x width)
Basically I want to perform something like a batch "shuffle" of the image pixel intensities.

Output:
I am assuming you want output[a, b, c, d] == img[a, x[a, b, c], y[a, b, c], d], where a, b, c, d are variables which iterate over batch, H, W and 3, respectively. You can solve that by applying torch.gather twice. As you can see in documentation it performs a similar indexing operation for a single dimension, so we would first gather on dim 1 with x as the index parameter and again on dim 2 with y. Unfortunately gather does not broadcast, so to deal with the trailing rgb dimension we have to add an extra dimension and manually repeat it. The code looks like this

import torch

# prepare data as in the example
batch, H, W = 2, 4, 5
img = torch.arange(batch * H * W * 3).reshape(batch, H, W, 3)
x = torch.randint(0, H, (batch, H, W))
y = torch.randint(0, W, (batch, H, W))

# deal with `torch.gather` not broadcasting
x = x.unsqueeze(3).repeat(1, 1, 1, 3)
y = y.unsqueeze(3).repeat(1, 1, 1, 3)

# do the actual indexing
x_shuff = torch.gather(img, dim=1, index=x)
output = torch.gather(x_shuff, dim=2, index=y)


Instruction: pytorch - Where is “conv1d” implemented?

Input:
I wanted to see how the conv1d module is implemented
https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv1d. So I looked at functional.py but still couldn’t find the looping and cross-correlation computation.

Then I searched Github by keyword ‘conv1d’, checked conv.cpp https://github.com/pytorch/pytorch/blob/eb5d28ecefb9d78d4fff5fac099e70e5eb3fbe2e/torch/csrc/api/src/nn/modules/conv.cpp 1 but still couldn’t locate where the computation is happening.

My question is two-fold.

Where is the source code that "conv1d” is implemented?
In general, if I want to check how the modules are implemented, where is the best place to find? Any pointer to the documentation will be appreciated. Thank you.

Output:
It depends on the backend (GPU, CPU, distributed etc) but in the most interesting case of GPU it's pulled from cuDNN which is released in binary format and thus you can't inspect its source code. It's a similar story for CPU MKLDNN. I am not aware of any place where PyTorch would "handroll" it's own convolution kernels, but I may be wrong. EDIT: indeed, I was wrong as pointed out in an answer below.
It's difficult without knowing how PyTorch is structured. A lot of code is actually being autogenerated based on various markup files, as explained here. Figuring this out requires a lot of jumping around. For instance, the conv.cpp file you're linking uses torch::conv1d, which is defined here and uses at::convolution which in turn uses at::_convolution, which dispatches to multiple variants, for instance at::cudnn_convolution. at::cudnn_convolution is, I believe, created here via a markup file and just plugs in directly to cuDNN implementation (though I cannot pinpoint the exact point in code when that happens).


Instruction: how to flatten input in `nn.Sequential` in Pytorch

Input:
how to flatten input inside the nn.Sequential

Model = nn.Sequential(x.view(x.shape[0],-1),
                     nn.Linear(784,256),
                     nn.ReLU(),
                     nn.Linear(256,128),
                     nn.ReLU(),
                     nn.Linear(128,64),
                     nn.ReLU(),
                     nn.Linear(64,10),
                     nn.LogSoftmax(dim=1))

Output:
You can create a new module/class as below and use it in the sequential as you are using other modules (call Flatten()).

class Flatten(torch.nn.Module):
    def forward(self, x):
        batch_size = x.shape[0]
        return x.view(batch_size, -1)

Ref: https://discuss.pytorch.org/t/flatten-layer-of-pytorch-build-by-sequential-container/5983

EDIT: Flatten is part of torch now. See https://pytorch.org/docs/stable/nn.html?highlight=flatten#torch.nn.Flatten


Instruction: How does Pytorch's "Fold" and "Unfold" work?

Input: I've gone through the official doc. I'm having a hard time understanding what this function is used for and how it works. Can someone explain this in layman's terms?

Output:
The unfold and fold are used to facilitate "sliding window" operations (like convolutions). Suppose you want to apply a function foo to every 5x5 window in a feature map/image:
from torch.nn import functional as f
windows = f.unfold(x, kernel_size=5)

Now windows has size of batch-(55x.size(1))-num_windows, you can apply foo on windows:
processed = foo(windows)

Now you need to "fold" processed back to the original size of x:
out = f.fold(processed, x.shape[-2:], kernel_size=5)

You need to take care of padding, and kernel_size that may affect your ability to "fold" back processed to the size of x. Moreover, fold sums over overlapping elements, so you might want to divide the output of fold by patch size.

Please note that torch.unfold performs a different operation than nn.Unfold. See this thread for details.


Instruction: PyTorch Getting Started example not working

Input:
I followed this tutorial in the Getting Started section on the PyTorch website: "Deep Learning with PyTorch: A 60 Minute Blitz" and I downloaded the code for "Training a Classifier" on the bottom of the page and I ran it, and it's not working for me. I'm using the CPU version of PyTorch if that makes a difference. I'm new to Python and basically learning it for Pytorch. Here's the error message, Control + K isn't working for me because I think the editing interface is different for the first few posts and Stack Overflow needs to fix it. Or it could just be my browser:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
prepare(preparation_data)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
run_name="__mp_main__")
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
  File "C:\ProgramData\Anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
  File "C:\Users\Anonymous\PycharmProjects\pytorchHelloWorld\train_network.py", line 100, in <module>
dataiter = iter(trainloader)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
return _DataLoaderIter(self)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
w.start()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.
Traceback (most recent call last):
 File "C:/Users/Anonymous/PycharmProjects/pytorchHelloWorld/train_network.py", line 100, in <module>
dataiter = iter(trainloader)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
return _DataLoaderIter(self)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
w.start()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

Output:
The error is likely due to multiprocessing in DataLoader and Windows since the tutorial is using num_workers=2. Python3 documentation shares some guidelines on this:

  Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

You can either set num_workers=0 or you need to wrap your code within if __name__ == '__main__'

# Safe DataLoader multiprocessing with Windows
if __name__ == '__main__':
    # Code to load the data with num_workers > 1

Check this reply on PyTorch forum for more details and this issue on GitHub.


Instruction: Using expand_dims in pytorch

Input:
I'm trying to tile a length 18 1 hot vector into a 40x40 grid.

Looking at pytorch docs, expand dims seems to be what i need.

But I cannot get it to work.  Any idea what I'm doing wrong?

one_hot = torch.zeros(18).unsqueeze(0)
one_hot[0,1] = 1.0
one_hot
tensor([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
one_hot.expand(-1,-1,40,40)
Traceback (most recent call last):
  File "<input>", line 1, in <module>
RuntimeError: The expanded size of the tensor (40) must match the existing size (18) at non-singleton dimension 3

I'm expecting a tensor of shape (1, 18, 40,40)

Output:
expand works along singleton dimensions of the input tensor. In your example, you are trying to expand a 1-by-18 tensor along its (non-existent) third and fourth dimensions - this is why you are getting an error. The only singleton dimension (=dimension with size==1) you have is the first dimension.

fix

one_hot = torch.zeros(1,18,1,1, dtype=torch.float)  # create the tensor with all singleton dimensions in place
one_hot[0,1,0,0] = 1.
one_hot.expand(-1,-1,40,40)


Instruction: pytorch - connection between loss.backward() and optimizer.step()

Input:
Where is an explicit connection between the optimizer and the loss?

How does the optimizer know where to get the gradients of the loss without a call liks this optimizer.step(loss)?

-More context-

When I minimize the loss, I didn't have to pass the gradients to the optimizer.

loss.backward() # Back Propagation
optimizer.step() # Gardient Descent

Output:
Without delving too deep into the internals of pytorch, I can offer a simplistic answer:
Recall that when initializing optimizer you explicitly tell it what parameters (tensors) of the model it should be updating. The gradients are "stored" by the tensors themselves (they have a grad and a requires_grad attributes) once you call backward() on the loss. After computing the gradients for all tensors in the model, calling optimizer.step() makes the optimizer iterate over all parameters (tensors) it is supposed to update and use their internally stored grad to update their values.
More info on computational graphs and the additional "grad" information stored in pytorch tensors can be found in this answer.
Referencing the parameters by the optimizer can sometimes cause troubles, e.g., when the model is moved to GPU after initializing the optimizer.
Make sure you are done setting up your model before constructing the optimizer. See this answer for more details.


Instruction: PyTorch custom loss function

Input:
How should a custom loss function be implemented ? Using below code is causing error :
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F

num_epochs = 20

x1 = np.array([0,0])
x2 = np.array([0,1])
x3 = np.array([1,0])
x4 = np.array([1,1])

num_epochs = 200

class cus2(torch.nn.Module):

    def __init__(self):
        super(cus2,self).__init__()

    def forward(self, outputs, labels):
        # reshape labels to give a flat vector of length batch_size*seq_len
        labels = labels.view(-1)

        # mask out 'PAD' tokens
        mask = (labels >= 0).float()

        # the number of tokens is the sum of elements in mask
        num_tokens = int(torch.sum(mask).data[0])

        # pick the values corresponding to labels and multiply by mask
        outputs = outputs[range(outputs.shape[0]), labels]*mask

        # cross entropy loss for all non 'PAD' tokens
        return -torch.sum(outputs)/num_tokens

x = torch.tensor([x1,x2,x3,x4]).float()

y = torch.tensor([0,1,1,0]).long()

train = data_utils.TensorDataset(x,y)
train_loader = data_utils.DataLoader(train , batch_size=2 , shuffle=True)

device = 'cpu'

input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = .0001

class NeuralNet(nn.Module) :
    def __init__(self, input_size, hidden_size, num_classes) :
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size , hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size , num_classes)

    def forward(self, x) :
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

for i in range(0 , 1) :

        model = NeuralNet(input_size, hidden_size, num_classes).to(device)

        criterion = nn.CrossEntropyLoss()
#         criterion = Regress_Loss()
#         criterion = cus2()
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

        total_step = len(train_loader)
        for epoch in range(num_epochs) :
            for i,(images , labels) in enumerate(train_loader) :
                images = images.reshape(-1 , 2).to(device)
                labels = labels.to(device)

                outputs = model(images)
                loss = criterion(outputs , labels)

                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
#                 print(loss)

        outputs = model(x)

        print(outputs.data.max(1)[1])

makes perfect predictions on training data :
tensor([0, 1, 1, 0])

Using a custom loss function from here:

is implemented in above code as cus2
Un-commenting code #         criterion = cus2() to use this loss function returns :
tensor([0, 0, 0, 0])

A warning is also returned :

UserWarning: invalid index of a 0-dim tensor. This will be an error in
PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python
number

I've not implemented the custom loss function correctly ?

Output:
Your loss function is programmatically correct except for below:

    # the number of tokens is the sum of elements in mask
    num_tokens = int(torch.sum(mask).data[0])

When you do torch.sum it returns a 0-dimensional tensor and hence the warning that it can't be indexed. To fix this do int(torch.sum(mask).item()) as suggested or int(torch.sum(mask)) will work too.

Now, are you trying to emulate the CE loss using the custom loss? If yes, then you are missing the log_softmax

To fix that add outputs = torch.nn.functional.log_softmax(outputs, dim=1) before  statement 4. Note that in case of tutorial that you have attached, log_softmax is already done in the forward call. You can do that too.

Also, I noticed that the learning rate is slow and even with CE loss, results are not consistent. Increasing the learning rate to 1e-3 works well for me in case of custom as well as CE loss.


Instruction: How to multiply a tensor row-wise by a vector in PyTorch?

Input: When I have a tensor m of shape [12, 10] and a vector s of scalars with shape [12], how can I multiply each row of m with the corresponding scalar in s?

Output:
You need to add a corresponding singleton dimension:
m * s[:, None]

s[:, None] has size of (12, 1) when multiplying a (12, 10) tensor by a (12, 1) tensor pytorch knows to broadcast s along the second singleton dimension and perform the "element-wise" product correctly.