Instruction: Simple policy gradients (REINFORCE) overfits one action when playing Atari Breakout

Input:
Self-contained code: https://colab.research.google.com/drive/1HYEXMpicymPUySkhGOaCJdJ3pN4RzXYd

The problem: I am trying to use a CNN to play Atari breakout from pixels (Breakout-V0 OpenAI gym). I am trying to use the simple policy gradients algorithm, implemented in PyTorch, to do this. There are four possible actions in this game [<NO-OP>, <FIRE> (play), <LEFT>, <RIGHT>].

Expected results: I would expect the policy to learn to play <NO-OP>, <LEFT>, <RIGHT> with about equal probability, and only play <FIRE> on the first frame of the game.

Actual results: After ~4 weight updates, the network predicts ONE action with nearly 100% probability. This means the gradient vanishes and the policy never recovers.

What I've tried:

Only play random actions at the start of the game (take increasingly greedy actions that follow the policy). You can fiddle around with this in the notebook.
I have rewarded only one action, just to prove to myself that it learns to only play that one action. It does, so I think I can rule out any PyTorch specific implementation errors.
Introducing an entropy penalty to the loss to discourage high confidence in actions.

My understanding is that this should not be necessary to explicitly introduce random actions because action = categorical.sample() does this. And if one action becomes dominant, but does not lead to reward, it should be subsequently discouraged.

My thought was that my training batches were skewed so that taking the action <RIGHT> for example lead to more reward per episode than punishment, and hence it's likelihood kept increasing. I would have thought that taking totally random actions at the start of the game and only slowly beginning to listen to the policy would have fixed this, but in my experiments, it didn't.

I'm really very confused as to why this doesn't work. BIG thanks to anyone who can help. I tried to debug this with a PhD in RL for 5 hours yesterday and made no progress.

Extra Questions:

Is it common to play e.g. 1000 rollouts, then sample a batch randomly from this rollout buffer to learn from? My understanding of policy gradients was that the policy is updated after every episode.
Am I correct in thinking that in policy gradients, you only backprop through the neuron that you selected your action from, but the gradient gets distributed to ALL the network weights through the softmax?

Output:
We ran into a similar problem while training Breakout with VPG (Vanilla Policy Gradient). The solution was to enforce entropy loss over the following

Entropy loss over the policy model output (Penalise selecting a specific action with high likelihood)
Scaling the MSE between rewards to go and value function

Not sure if you are aware of this but do use the Atari wrapper from Deepmind

Q 1 : Is it common to play e.g. 1000 rollouts, then sample a batch randomly from this rollout buffer to learn from? My understanding of policy gradients was that the policy is updated after every episode.

A 1 : In practice its better to batch multiple episodes in a single batch.

Q 2 : Am I correct in thinking that in policy gradients, you only backprop through the neuron that you selected your action from, but the gradient gets distributed to ALL the network weights through the softmax?

A 2 : Based on the expected rewards to go, likelihood of selection of specific action is suppressed or encouraged and the model backdrops this information


Instruction: Different filters for 2d convolution

Input: I’m having an input of shape (B(atch), F(features), N(odes), T(timestamps)). Right now if I apply a 2d convolution with a kernel of shape (1,2) I will have a total of (F_out, F_in, 1,2) weights to learn which is alright. I want to extend this so that for each Node in the input I have it’s own filter with shape (1,2). Does any of you have any idea where should I start from? So far I looped through all N and apply the filter to its respective input. Unfortunately this approach is very slow.

Output:
You are looking for "grouped convolution".
The doc for nn.Conv2d regarding the groups parameter:

  At groups=2, the operation becomes equivalent to having two conv layers side by side, each seeing half the input channels, and producing half the output channels, and both subsequently concatenated.

In your case, you want groups= number of nodes.

It's not that simple in your case, because you want to "merge" features and nodes, and to have only 1d grouped convolution over the "feature"+"node" dimension.
Moreover, you need to permute between "node"s and "feature"s in order to group the features according to nodes.

b = 10;
inf = 8;
outf = 13;
n = 3;
t = 50;

x = torch.rand((b, inf, n, t))  # input tensor
gconv = nn.Conv1d(inf, outf, kernel_size=(2), groups=n) #grouped conv

x_ready = x.permute(0, 2, 1, 3).view(b, inf*n, t)
y_grouped = gconv(x_ready)
# "fix" y
y = y_grouped.view(n, n, outf, t).permute(0, 2, 1, 3)  # now y is b-outf-n-t


Instruction: Expected dimension sizes for pytorch models

Input:
I'm struggling with understanding what sort of dimensions my pytorch model needs as input. My model setup is:
import torch
from torch import nn, tensor

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.rnn_b = nn.RNN(input_size=input_size,
                           hidden_size=hidden_size,
                           num_layers=1,
                           bias=False)
        self.hidden_size = hidden_size
        self.linearout = nn.Linear(hidden_size, output_size)

    def forward(self, input, hidden):
        out, hidden = self.rnn_b(input, hidden)
        output = self.linearout(out)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

model = RNN(1, 2, 1)

If I call model(tensor(25.), tensor([[0., 0.]])) then I get the error:
IndexError: dimension specified as 1 but tensor has no dimensions

If I call model(tensor([25.]), tensor([[0., 0.]])) then I get the error
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Can someone please explain what's going on here? What is the correct way to format the data for the input to my model?

Output:
You need to encapsulate more. This is because pytorch automatically allows batches:

model(tensor([[[25.]]]), tensor([[[0., 0.]]]))

Output:

(tensor([[[-0.7704]]], grad_fn=<AddBackward0>),
 tensor([[[1., 1.]]], grad_fn=<StackBackward>))

You can use multiple input, as a batch:

model(tensor([[[25.]], [[25.]]]), tensor([[[0., 0.]]]))

Output:

(tensor([[[0.0341]],

         [[0.0341]]], grad_fn=<AddBackward0>),
 tensor([[[ 0.9999, -1.0000]]], grad_fn=<StackBackward>))


Instruction: How to create upper triangular matrix in Pytorch?

Input: Simple question, but is there a native way to create an upper triangular matrix from an existing matrix in Pytorch? I was thinking of using a mask, but even that requires creating the upper triangular matrix.

Output:
import torch
upper_tri = torch.ones(rol, col).triu()

Eg:
>> mat = torch.ones(3, 3).triu()
>> print(mat)
tensor([[1., 1., 1.],
        [0., 1., 1.],
        [0., 0., 1.]])


Instruction: How to adapt the gpu batch size during training?

Input:
I found surprising that I could not find any resources online on how to dynamically adapt the GPU batch size without halting training.

The idea is the following:

1) Have a training script that is (almost) agnostic to the GPU in use. The batch size will dynamically adjust without interference of the user or need for tunning.

2) Still being able to specifying the desired training batch size, even if too big to fit in the biggest known GPU.

For instance, let's say I want to train a model using a batch size of 4096 images, each image 1024x1024. Let's also say that I have access to a server with different NVidea GPUs, but I don't know which one will be assigned to me in advance. (Or that everybody wants to use the biggest GPU and that I am left waiting a long time before it is my term).

I want my training script to find the max batch size (let's say it is 32 images per GPU batch),  and only update the optimizer when all 4096 images have been processed (one training batch = 128 GPU batches).

Output:
There are different ways of solving this problem. But if specifying the GPU that can do the job, or using multiple GPUs are not an option, then it is handy to dynamically adapt the GPU batch size.

I prepared this repo with an illustrative training example in pytorch (it should work similarly in TensorFlow)

In the code below, the try/except is used to try different GPU batch sizes without halting training. When the batch becomes too large, it is downsized and the adaptation is turned off. Please check the repo for the implementation details and possible bug fixes.

It is also implemented a technique called Batch Spoofing, which performs a number of forward passes before doing the backpropagation. In PyTorch it only requires replacing the optimizer.zero_grad().

import torch
import torchvision
import torch.optim as optim
import torch.nn as nn

# Example of how to use it with Pytorch
if __name__ == "__main__":

    # #############################################################
    # 1) Initialize the dataset, model, optimizer and loss as usual.
    # Initialize a fake dataset

    trainset = torchvision.datasets.FakeData(size=1_000_000,
                                             image_size=(3, 224, 224),
                                             num_classes=1000)

    # initialize the model, loss and SGD-based optimizer
    resnet = torchvision.models.resnet152(pretrained=True,
                                          progress=True)
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(resnet.parameters(), lr=0.01)

    continue_training = True  # criteria to stop the training

    # #############################################################
    # 2) Set parameters for the adaptive batch size
    adapt = True  # while this is true, the algorithm will perform batch adaptation
    gpu_batch_size = 2  # initial gpu batch_size, it can be super small
    train_batch_size = 2048  # the train batch size of desire

    # Modified training loop to allow for adaptive batch size
    while continue_training:

        # #############################################################
        # 3) Initialize dataloader and batch spoofing parameter
        # Dataloader has to be reinicialized for each new batch size.
        trainloader = torch.utils.data.DataLoader(trainset,
                                                  batch_size=int(gpu_batch_size),
                                                  shuffle=True)

        # Number of repetitions for batch spoofing
        repeat = max(1, int(train_batch_size / gpu_batch_size))

        try:  # This will make sure that training is not halted when the batch size is too large

            # #############################################################
            # 4) Epoch loop with batch spoofing
            optimizer.zero_grad()  # done before training because of batch spoofing.

            for i, (x, y) in enumerate(trainloader):

                y_pred = resnet(x)
                loss = criterion(y_pred, y)
                loss.backward()

                # batch spoofing
                if not i % repeat:
                    optimizer.step()
                    optimizer.zero_grad()

                # #############################################################
                # 5) Adapt batch size while no RuntimeError is rased.
                # Increase batch size and get out of the loop
                if adapt:
                    gpu_batch_size *= 2
                    break

                # Stopping criteria for training
                if i > 100:
                    continue_training = False

        # #############################################################
        # 6) After the largest batch size is found, the training progresses with the fixed batch size.
        # CUDA out of memory is a RuntimeError, the moment we will get to it when our batch size is too large.
        except RuntimeError as run_error:
            gpu_batch_size /= 2  # resize the batch size for the biggest that works in memory
            adapt = False  # turn off the batch adaptation

            # Number of repetitions for batch spoofing
            repeat = max(1, int(train_batch_size / gpu_batch_size))

            # Manual check if the RuntimeError was caused by the CUDA or something else.
            print(f"---\nRuntimeError: \n{run_error}\n---\n Is it a cuda error?")

If you have code that can do similarly in Tensorflow, Caffe or others, please share!


Instruction: Why the resnet110 I train on CIFAR10 dataset only get 77% test acc

Input:
I trained the Resnet110 on CIFAR10 dataset, and I got 100% acc on training, but only 77.85% on test dataset. What is the problem probally be?
Otherwise, I use Pytorch framwork.
Thank U very much!

    ------------------------------------------------------------------------
Train Epoch: 200 [0/50000 (0%)] Loss: 0.000811, Accuracy: 100.00
Train Epoch: 200 [12800/50000 (26%)]    Loss: 0.000335, Accuracy: 100.00
Train Epoch: 200 [25600/50000 (51%)]    Loss: 0.000757, Accuracy: 100.00
Train Epoch: 200 [38400/50000 (77%)]    Loss: 0.000334, Accuracy: 100.00
Epoch time:45.98s
Accuracy of plane : 81 %
Accuracy of   car : 88 %
Accuracy of  bird : 65 %
Accuracy of   cat : 60 %
Accuracy of  deer : 73 %
Accuracy of   dog : 69 %
Accuracy of  frog : 81 %
Accuracy of horse : 82 %
Accuracy of  ship : 86 %
Accuracy of truck : 86 %

Test set: Average loss: 1.3605, Accuracy: 77.58%

Output:
ResNet-101 is definitely too big for CIFAR10, go with smaller versions, ResNet-18 from torchvision should be fine.

Furthermore, you could train those really fast using super convergence (e.g. setting learning rate to 5 or 3), see this article or other related. You could do so in 18 epochs or so with torch.optim.AdamW I think. Furthermore, such high learning rate plays a regularizing role as it will only converge to really flat minima regions.

In order to not overfit, use really powerful image augmentation, rotations, flipping, CutOut, maybe MixUp. You could find them inside albumentations third party library.


Instruction: Adapting Pytorch "NLP from Scratch" for bidirectional GRU

Input:
I have taken the code from the tutorial and attempted to modify it to include bi-directionality and any arbitrary numbers of layers for GRU.

Link to the tutorial which uses uni-directional, single layer GRU:
https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

The model works fine, but when i use set bidirectional=True, i get the a dimension mismatch error (shown below). Any thoughts why this is?

Encoder:

import torch.nn.init as init
class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers=1, bidirectional=False):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.hidden_var = hidden_size//2 if bidirectional else hidden_size
        self.n_layers = n_layers
        self.bidirectional = bidirectional
        self.n_directions = 2 if bidirectional else 1

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size,
                          self.hidden_var,
                          num_layers=self.n_layers,
                          bidirectional=self.bidirectional)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output = embedded
        output, hidden = self.gru(output, hidden)
        #output = (output[:, :, :self.hidden_size] +
        #        output[:, :, self.hidden_size:])
        return output, hidden

    def initHidden(self):
        return torch.zeros(self.n_layers*self.n_directions, 1, self.hidden_var, device=device)

AttnDecoder:

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1, max_length=MAX_LENGTH):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p
        self.max_length = max_length
        self.n_layers = n_layers

        self.embedding = nn.Embedding(self.output_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
        self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
        self.dropout = nn.Dropout(self.dropout_p)

        self.gru = nn.GRU(self.hidden_size,
                          self.hidden_size,
                          num_layers = self.n_layers)

        self.out = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        attn_weights = F.softmax(
            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
        attn_applied = torch.bmm(attn_weights.unsqueeze(0),
                                 encoder_outputs.unsqueeze(0))

        output = torch.cat((embedded[0], attn_applied[0]), 1)
        output = self.attn_combine(output).unsqueeze(0)

        output = F.relu(output)
        output, hidden = self.gru(output, hidden)

        output = F.log_softmax(self.out(output[0]), dim=1)

        return output, hidden, attn_weights

    def initHidden(self):
        return torch.zeros(1*self.n_layers, 1, self.hidden_size, device=device)

Everything else from the tutorial is kept exactly the same apart from this code block ( to account for the new parameters):

n_layers=1
bidirectional = True
hidden_size = 256
encoder1 = EncoderRNN(input_lang.n_words, hidden_size, n_layers=n_layers, bidirectional=bidirectional).to(device)
attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1, n_layers=n_layers).to(device)
trainIters(encoder1, attn_decoder1, 75000, print_every=5000)

Error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-133-37084c93a197> in <module>
      5 attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words, dropout_p=0.1, n_layers=n_layers).to(device)
      6
----> 7 trainIters(encoder1, attn_decoder1, 75000, print_every=5000)

<ipython-input-131-774ce8edefa6> in trainIters(encoder, decoder, n_iters, print_every, plot_every, learning_rate)
     16
     17         loss = train(input_tensor, target_tensor, encoder,
---> 18                      decoder, encoder_optimizer, decoder_optimizer, criterion)
     19         print_loss_total += loss
     20         plot_loss_total += loss

<ipython-input-130-67be7e8c2a58> in train(input_tensor, target_tensor, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion, max_length)
     39         for di in range(target_length):
     40             decoder_output, decoder_hidden, decoder_attention = decoder(
---> 41                 decoder_input, decoder_hidden, encoder_outputs)
     42             topv, topi = decoder_output.topk(1)
     43             decoder_input = topi.squeeze().detach()  # detach from history as input

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

<ipython-input-129-6dd1d30fe28f> in forward(self, input, hidden, encoder_outputs)
     24
     25         attn_weights = F.softmax(
---> 26             self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)
     27         attn_applied = torch.bmm(attn_weights.unsqueeze(0),
     28                                  encoder_outputs.unsqueeze(0))

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1367     if input.dim() == 2 and bias is not None:
   1368         # fused op is marginally faster
-> 1369         ret = torch.addmm(bias, input, weight.t())
   1370     else:
   1371         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [1 x 384], m2: [512 x 10] at /tmp/pip-req-build-58y_cjjl/aten/src/TH/generic/THTensorMath.cpp:752

Any help would be appreciated!

Update based on user3923920 comment (encoder-decoder also includes LSTM option & now works with bidirectionality)

New working and adapted Encoder

class EncoderRNN(nn.Module):
        def __init__(self, input_size, hidden_size, n_layers=1, bidirectional=False, method='GRU'):
            super(EncoderRNN, self).__init__()
            self.hidden_size = hidden_size
            self.hidden_var = hidden_size // 2 if bidirectional else hidden_size
            self.n_layers = n_layers
            self.bidirectional = bidirectional
            self.n_directions = 2 if bidirectional else 1
            self.method = method

            self.embedding = nn.Embedding(input_size, hidden_size)
            if self.method == 'GRU':
                self.net = nn.GRU(hidden_size,
                                  self.hidden_var,
                                  num_layers=self.n_layers,
                                  bidirectional=self.bidirectional)
            elif self.method == 'LSTM':
                self.net = nn.LSTM(hidden_size,
                                   self.hidden_var,
                                   num_layers=self.n_layers,
                                   bidirectional=self.bidirectional)

        def forward(self, input, hidden):
            embedded = self.embedding(input).view(1, 1, -1)
            output = embedded
            output, hidden = self.net(output, hidden)
            # output = (output[:, :, :self.hidden_size] +
            #        output[:, :, self.hidden_size:])
            return output, hidden, embedded

        def initHidden(self):
            if self.method == 'GRU':
                return torch.zeros(self.n_layers * self.n_directions, 1, self.hidden_var, device=device)
            elif self.method == 'LSTM':
                h_state = torch.zeros(self.n_layers * self.n_directions, 1, self.hidden_var)
                c_state = torch.zeros(self.n_layers * self.n_directions, 1, self.hidden_var)
                hidden = (h_state, c_state)
                return hidden

New working and adapted Decoder

class AttnDecoderRNN(nn.Module):
        def __init__(self, hidden_size, output_size, n_layers=1, dropout_p=0.1,
                     max_length=MAX_LENGTH, method='GRU', bidirectional=False):

            super(AttnDecoderRNN, self).__init__()
            self.hidden_size = hidden_size
            self.output_size = output_size
            self.dropout_p = dropout_p
            self.max_length = max_length
            self.n_layers = n_layers
            self.method = method
            self.bidirectional = bidirectional

            self.embedding = nn.Embedding(self.output_size, self.hidden_size)
            self.attn = nn.Linear(self.hidden_size * 2, self.max_length)
            self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
            self.dropout = nn.Dropout(self.dropout_p)

            if self.method == 'GRU':
                self.net = nn.GRU(self.hidden_size,
                                  self.hidden_size,
                                  num_layers=self.n_layers)
            elif self.method == 'LSTM':
                self.net = nn.LSTM(self.hidden_size,
                                   self.hidden_size,
                                   num_layers=self.n_layers)

            self.out = nn.Linear(self.hidden_size, self.output_size)

        def forward(self, input, hidden, encoder_outputs):

            # Embed
            embedded = self.embedding(input).view(1, 1, -1)
            embedded = self.dropout(embedded)
            self.hidden = hidden

            # Concatenate all of the layers
            hidden_h_rows = ()
            hidden_c_rows = ()

            if self.method == 'LSTM':
                # hidden is a tuple of h_state and c_state
                decoder_h, decoder_c = hidden
                print(decoder_h.shape)
                hidden_shape = decoder_h.shape[0]

                # h_state
                for x in range(0, hidden_shape):
                    hidden_h_rows += (decoder_h[x],)

                # c_state
                for x in range(0, hidden_shape):
                    hidden_c_rows += (decoder_c[x],)

            elif self.method == "GRU":

                # hidden is not a tuple (GRU)
                decoder_h = hidden
                hidden_shape = decoder_h.shape[0]

                # h_state
                for x in range(0, hidden_shape):
                    hidden_h_rows += (decoder_h[x],)

            if self.bidirectional:
                decoder_h_cat = torch.cat(hidden_h_rows, 1)
                # Make sure the h_dim size is compatible with num_layers with concatenation.
                decoder_h = decoder_h_cat.view((self.n_layers, 1, self.hidden_size))  # hidden_size=256

                if self.method == "LSTM":
                    decoder_c_cat = torch.cat(hidden_c_rows, 1)
                    decoder_c = decoder_c_cat.view((self.n_layers, 1, self.hidden_size))  # hidden_size=256
                    hidden_lstm = (decoder_h, decoder_c)

                elif self.method == "GRU":
                    hidden_gru = decoder_h

            # Attention Block
            attn_weights = F.softmax(
                self.attn(torch.cat((embedded[0], hidden_lstm[0][0] if self.method == "LSTM" else \
                    hidden_gru[0]), 1)), dim=1)
            attn_applied = torch.bmm(attn_weights.unsqueeze(0), encoder_outputs.unsqueeze(0))
            output = torch.cat((embedded[0], attn_applied[0]), 1)
            output = self.attn_combine(output).unsqueeze(0)

            output = F.relu(output)
            output, hidden = self.net(output,
                                      hidden_lstm if self.method == "LSTM" else hidden_gru)  # I am not sure about this!
            output = F.log_softmax(self.out(output[0]), dim=1)
            return output, hidden, attn_weights

        def initHidden(self):

            if self.method == 'GRU':
                return torch.zeros(self.n_layers * 1, 1, self.hidden_var, device=device)
            elif self.method == 'LSTM':
                h_state = torch.zeros(self.n_layers * 1, 1, self.hidden_var)
                c_state = torch.zeros(self.n_layers * 1, 1, self.hidden_var)
                hidden = (h_state, c_state)
                return hidden

Output:
So I'm not sure if this is 100% correct as I'm just learning how to program RNNs, but i changed my code in a couple of extra areas.

For one you'll notice that the error says m1: [1x384] so the result of

torch.cat((embedded[0], hidden[0]), 1))

when putting this through the attn weight layer is not a dimension ending with 512, the expected input size. This is because hidden is a tensor of shape [2, 1, 256] instead of some shape [1, 1, 512] or something. Since your dimensions don't exactly match mine I'm not exactly sure what is different, so in train(...) where it just sets

decoder_hidden = encoder_hidden

I do

decoder_hidden = torch.cat((encoder_hidden[0], encoder_hidden[1]) , 1)
 decoder_hidden = decoder_hidden.view((1, 1, 512))

Hopefully that helps in some way


Instruction: How to extract images, labels from csv file and create a trainset using torch?

Input:
I downloaded a dataset for facial key point detection the image and the labels were in a CSV file I extracted it using pandas but I don't know how to convert it into a tensor and load it into a data loader for training.

dataframe = pd.read_csv("training_facial_keypoints.csv")
dataframe['Image'] = dataframe['Image'].apply(lambda i: np.fromstring(i, sep=' '))
dataframe= dataframe.dropna()
images_array = np.vstack(dataframe['Image'].values)/255.0
images_array = images_array.astype(np.float32)
images_array = images_array.reshape(-1, 96, 96, 1)
print(images_array.shape)
labels_array = dataframe[dataframe.columns[:-1]].values
labels_array = (labels_array-48)/48
labels_array = labels_array.astype(np.float32)

I have the images and labels in two arrays. How do I create a training set from this and use transforms.
Then load it using a dataloader.

Output:
Create a subclass of torch.utils.data.Dataset, fill it with your data.
You can pass desired torchvision.transforms to it and apply them to your data in __getitem__(self, index).

Than you can pass it to torch.utils.data.DataLoader which allows multi-threaded loading of data.

And PyTorch has an overwhelming documentation you should first refer to.


Instruction: error installing pytorch using pip on windows 10

Input:
I am trying to install pytorch with pip using

pip install torch

or

pip3 install torch===1.3.1 torchvision===0.4.2 -f https://download.pytorch.org/whl/torch_stable.html

with python 3.7.4

and with python 3.8 (latest stable release)

both on 32 and 64 bit.

and getting

Collecting torch   Using cached
https://files.pythonhosted.org/packages/f8/02/880b468bd382dc79896eaecbeb8ce95e9c4b99a24902874a2cef0b562cea/torch-0.1.2.post2.tar.gz
Collecting pyyaml (from torch)   Downloading
https://files.pythonhosted.org/packages/bc/3f/4f733cd0b1b675f34beb290d465a65e0f06b492c00b111d1b75125062de1/PyYAML-5.1.2-cp37-cp37m-win_amd64.whl
(215kB)
    100% |████████████████████████████████| 225kB 1.2MB/s Installing collected packages: pyyaml, torch   Running setup.py install for torch
... error
    Complete output from command C:\Noam\Code\threadart\stav-rl\venv\Scripts\python.exe -u -c "import
setuptools,
tokenize;__file__='C:\\Users\\noams\\AppData\\Local\\Temp\\pip-install-djc6s2t8\\torch\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read( ).replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" install
--record C:\Users\noams\AppData\Local\Temp\pip-record-zohv2zo7\install-record.txt
--single-version-externally-managed --compile --install-headers C:\Noam\Code\threadart\stav-rl\venv\inclu de\site\python3.7\torch:
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "C:\Noam\Code\threadart\stav-rl\venv\lib\site-packages\setuptools-40.8.0-py3.7.egg\setuptools\__init__.py",
line 145, in setup
      File "C:\Python37_x64\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Python37_x64\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Python37_x64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 99, in run
        self.run_command('build_deps')
      File "C:\Python37_x64\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Python37_x64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

    ---------------------------------------- Command "C:\Noam\Code\threadart\stav-rl\venv\Scripts\python.exe -u -c "import
setuptools,
tokenize;__file__='C:\\Users\\noams\\AppData\\Local\\Temp\\pip-install-djc6s2t8\\torch\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n', '\n');
f.close();exec(compile(code, __file__, 'exec'))" install --record
C:\Users\noams\AppData\Local\Temp\pip-record-zohv2zo7\install-record.txt
--single-version-externally-managed --compile --install-headers C:\Noam\Code\threadart\stav-rl\venv\include\site\python3.7\torch"
failed with error code 1 in
C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\

clearly, I am doing something wrong.

Please help!

Output:
I had the same problem. Now the problem is fixed. (2020-05-31)

Visited the site pytorch.org
and find "QUICK START LOCALLY" on homepage of pytorch.org. ( it' can find by scroll down little )
Checking the environment form of your system
(ex: Windows, pip, python, ,,)
then, you can see the install command.
"pip install torch===.... "
Copy the install command
and Execute the command at your system.

Good Luck !!


Instruction: Turn CNN model into class

Input:
I am trying to build a CNN for multilabel classification in Pytorch (each image can have more than one label). So far I have built the model as follows:

model.fc = nn.Sequential(nn.Linear(2048, 512),
                                 nn.ReLU(),
                                 nn.Dropout(0.2),
                                 nn.Linear(512, 10),
                                 nn.LogSigmoid())
                                 # nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
# criterion = nn.BCELoss()
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)

But I would like to build it using a class like the following example:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(10, 20, 5)
        self.fc1 = nn.Linear(20 * 22 * 39, 100)
        self.fc2 = nn.Linear(100, 50)
        self.fc3 = nn.Linear(50, 10)
        self.fc4 = nn.Linear(10, 3)

    def forward(self, x):
        x = x.view(-1, 3, 100, 170)
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 20 * 22 * 39)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        return self.fc4(x)

What would be the best way of accomplishing this given that I am dealing with a multilabel classification problem? Any insights I would appreciate it.

Output:
You should use torch.nn.BCEWithLogitsLoss for multilabel classification (and numerical stability), no LogSigmoid or NLLLoss as the output.
You have to output N elements for each element in batch, where 1 on position N in vector would mean an existence of class N on image.
Your network is fine, provided you only got 3 labels to predict (either 0 or 1). You may think about it's design or use something pretrained, other than that it should at least run.


Instruction: Can I specify kernel-weight specific learning rates in PyTorch?

Input:
I would like to set specific learning rates for each parameter on their lowest level. I.e. each value in a kernels weight and biases should have their own learning rate.

I can specify filter-wise learning rates like that:

optim = torch.optim.SGD([{'params': model.conv1.weight, 'lr': 0.1},], lr=0.01)

But when I want to get a level lower, like that:

optim = torch.optim.SGD([{'params': model.conv1.weight[0, 0, 0, 0], 'lr': 0.1},], lr=0.01)

I receive an error: ValueError: can't optimize a non-leaf Tensor
I also tried specifying a learning rate that has the same shape as the filter such as 'lr': torch.ones_like(model.conv1.weight), but that also didn't work out.

Is there even a way to do this using torch.optim?

Output:
I might have found a solution. As one can only input the whole weights and biases of a Conv Layer, we need to insert a learning rate having the same shape as the weight/bias tensor.

Here is an example using torch.optim.Adam:

torch.optim.CustomAdam([{'params': param, 'lr': torch.ones_like(param, requires_grad=False) * lr}
    for name, param in model.named_parameters()])

Then we have to change a line in the optimizer itself. For that I created a custom optimizer:

class CustomAdam(torch.optim.Adam):
   def step(self, closure=None):
       ...
       # change the last line: p.data.addcdiv_(-step_size, exp_avg, denom) to
       p.data.add_((-step_size * (exp_avg / denom)))


Instruction: I define a loss function but backward present error to me could someone tell me how to fix it

Input:
class loss(Function):
    @staticmethod
    def forward(ctx,x,INPUT):

        batch_size = x.shape[0]
        X = x.detach().numpy()
        input = INPUT.detach().numpy()
        Loss = 0
        for i in range(batch_size):
            t_R_r = input[i,0:4]
            R_r = t_R_r[np.newaxis,:]
            t_R_i = input[i,4:8]
            R_i = t_R_i[np.newaxis,:]
            t_H_r = input[i,8:12]
            H_r = t_H_r[np.newaxis,:]
            t_H_i = input[i,12:16]
            H_i = t_H_i[np.newaxis,:]

            t_T_r = input[i, 16:32]
            T_r = t_T_r.reshape(4,4)
            t_T_i = input[i, 32:48]
            T_i = t_T_i.reshape(4,4)

            R = np.concatenate((R_r, R_i), axis=1)
            H = np.concatenate((H_r, H_i), axis=1)

            temp_t1 = np.concatenate((T_r,T_i),axis=1)
            temp_t2 = np.concatenate((-T_i,T_r),axis=1)
            T = np.concatenate((temp_t1,temp_t2),axis=0)
            phi_r = np.zeros((4,4))
            row, col = np.diag_indices(4)
            phi_r[row,col] = X[i,0:4]
            phi_i = np.zeros((4, 4))
            row, col = np.diag_indices(4)
            phi_i[row, col] = 1 - np.power(X[i, 0:4],2)

            temp_phi1 = np.concatenate((phi_r,phi_i),axis=1)
            temp_phi2 = np.concatenate((-phi_i, phi_r), axis=1)
            phi = np.concatenate((temp_phi1,temp_phi2),axis=0)

            temp1 = np.matmul(R,phi)

            temp2 = np.matmul(temp1,T)  # error
            H_hat = H + temp2

            t_Q_r = np.zeros((4,4))
            t_Q_r[np.triu_indices(4,1)] = X[i,4:10]
            Q_r = t_Q_r + t_Q_r.T
            row,col = np.diag_indices(4)
            Q_r[row,col] = X[i,10:14]
            Q_i = np.zeros((4,4))
            Q_i[np.triu_indices(4,1)] = X[i,14:20]
            Q_i = Q_i - Q_i.T

            temp_Q1 = np.concatenate((Q_r,Q_i),axis=1)
            temp_Q2 = np.concatenate((-Q_i,Q_r),axis=1)
            Q = np.concatenate((temp_Q1,temp_Q2),axis=0)

            t_H_hat_r = H_hat[0,0:4]
            H_hat_r = t_H_hat_r[np.newaxis,:]
            t_H_hat_i= H_hat[0,4:8]
            H_hat_i = t_H_hat_i[np.newaxis,:]

            temp_H1 = np.concatenate((-H_hat_i.T,H_hat_r.T),axis=0)
            H_hat_H = np.concatenate((H_hat.T,temp_H1),axis=1)
            temp_result1 = np.matmul(H_hat,Q)
            temp_result2 = np.matmul(temp_result1,H_hat_H)

            Loss += np.log10(1+temp_result2[0][0])
        Loss = t.from_numpy(np.array(Loss / batch_size))
        return Loss
    @staticmethod
    def backward(ctx,grad_output):
        print('gradient')
        return grad_output
def criterion(output,input):
    return loss.apply(output,input)

This is my loss function. But it present the error:
Traceback (most recent call last):

  File "/Users/mrfang/channel_capacity/training.py", line 24, in

      loss.backward()   File "/Users/mrfang/anaconda3/lib/python3.6/site-packages/torch/tensor.py",
  line 150, in backward
      torch.autograd.backward(self, gradient, retain_graph, create_graph)   File
  "/Users/mrfang/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py",
  line 99, in backward
      allow_unreachable=True)  # allow_unreachable flag RuntimeError: function lossBackward returned an incorrect number of gradients
  (expected 2, got 1)

How could I fix it. Thanks very much

Output:
Your forward(ctx,x,INPUT) takes two inputs, x and INPUT, thus backward should output two gradients as well, grad_x and grad_INPUT.

In addition, in your snippet, you're not really computing a custom gradient, so you could compute that with Pytorch's autograd, without having to define a special Function.

If this is working code and you're going to define the custom loss, here's a quick boilerplate of what backward should comprise:

@staticmethod
def forward(ctx, x, INPUT):
    # this is required so they're available during the backwards call
    ctx.save_for_backward(x, INPUT)

    # custom forward

@staticmethod
def backward(ctx, grad_output):
    x, INPUT = ctx.saved_tensors
    grad_x = grad_INPUT = None

    # compute grad here

    return grad_x, grad_INPUT

You don't need to return gradients for inputs that don't require it, thus you can return None for them.

More info here and here.


Instruction: pytorch debugging timeout with PyCharm

Input:
I have a frustrating problem, where I cannot debug my pytorch code while in Pycharm.
While trying to inspect (breakpoint, then print e.g.) the code below, I receive a "Loading time out"

import torch

    tensors = []
    num_tensors = 16
    shape = (1, 3, 512, 512)
    for i in range(num_tensors):
        tensors.append(torch.zeros(shape))

I saw this[1,2] posts, set variable loading policy to syncronyous, disabled Qt debugger options, and all the options specified, But I believe there is something basic I'm missing.
pycharm 2019.2.5, happens both in python2 and python3.

Output: Are you using DataLoader? If yes, you can try reducing the num_workers to 0.


Instruction: .grad() returns None in pytorch

Input:
I am trying to write a simple script for parameter estimation (where parameters are weights here). I am facing problem when .grad() returns None. I have gone through this  and this link also and understood the concept both theoretically and practically. For me following script should work but unfortunately, it is not working.

My 1st attempt: Following script is my first attempt

alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device

                                   )
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)

Above code throws an error message

element 0 of tensors does not require grad and does not have a grad_fn

at line loss.backward()

My Attempt 2: Improvement in 1st attempt is as follows:

gamma = torch.tensor(2.0, device=device, dtype=torch.float, requires_grad=True)
alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        pred = torch.tensor([[x_dt],
                             [y0_dt],
                             [y_dt]],device=device,
                                   dtype=torch.float,
                                   requires_grad=True)
        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)
#        with torch.no_grad():
#            gamma -= leraning_rate * gamma.grad

Now the script is working but except pred.gred other two return None.

I want to update all the parameters after computing loss.backward() and update them but it is not happening due to None. Can anyone suggest me how to improve this script? Thanks.

Output:
You're breaking the computation graph by declaring a new tensor for pred. Instead you can use torch.stack. Also, x_dt and pred are non-leaf tensors so the gradients aren't retained by default. You can override this behavior by using .retain_grad().

gamma = torch.tensor(2.0, device=device, dtype=torch.float, requires_grad=True)
alpha_xy = torch.tensor(3.7, device=device, dtype=torch.float, requires_grad=True)
beta_y = torch.tensor(1.5, device=device, dtype=torch.float, requires_grad=True)
alpha0 = torch.tensor(1.1, device=device, dtype=torch.float, requires_grad=True)
alpha_y = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha1 = torch.tensor(0.1, device=device, dtype=torch.float, requires_grad=True)
alpha2 = torch.tensor(0.9, device=device, dtype=torch.float, requires_grad=True)
alpha3 = torch.tensor(0.001, device=device, dtype=torch.float, requires_grad=True)

learning_rate = 1e-4
total_loss = []

for epoch in tqdm(range(500)):
    loss_1 = 0
    for j in range(x_train.size(0)):
        input = x_train[j:j+1]
        target = y_train[j:j+1]
        input = input.to(device,non_blocking=True)
        target = target.to(device,non_blocking=True)
        x_dt = gamma*input[0][0] + \
               alpha_xy*input[0][0]*input[0][2] + \
               alpha1*input[0][0]

        # retain the gradient for non-leaf tensors
        x_dt.retain_grad()

        y0_dt = beta_y*input[0][0] + \
                alpha2*input[0][1]

        y_dt = alpha0*input[0][1] + \
               alpha_y*input[0][2] + \
               alpha3*input[0][0]*input[0][2]

        # use stack instead of declaring a new tensor
        pred = torch.stack([x_dt, y0_dt, y_dt], dim=0).unsqueeze(1)

        # pred is also a non-leaf tensor so we need to tell pytorch to retain its grad
        pred.retain_grad()

        loss = (pred - target).pow(2).sum()
        loss_1 += loss
        loss.backward()
        print(pred.grad, x_dt.grad, gamma.grad)
        with torch.no_grad():
            gamma -= learning_rate * gamma.grad

Closed form solution

Assuming you want to optimize for the parameters defined at the top of the function gamma, alpha_xy, beta_y, etc... Then what you have here is an example of ordinary least squares. See least squares for a slightly friendlier introduction to the topic. Take a look at the components of pred and you'll notice that x_dt, y0_dt, and y_dt are actually independent of each other with respect to the parameters (in this case it's obvious because they each use totally different parameters). This makes the problem much easier because it means we can actually optimize the terms (x_dt - target[0])**2, (y0_dt - target[1])**2 and (y_dt - target[2])**2 separately!

Without getting into the details the solution (without back-propagation or gradient descent) ends up being

# supposing x_train is [N,3] and y_train is [N,3]
x1 = torch.stack((x_train[:, 0], x_train[:, 0] * x_train[:, 2]), dim=0)
y1 = y_train[:, 0].unsqueeze(1)

# avoid inverses using solve to get p1 = inv(x1 . x1^T) . x1 . y1
p1, _ = torch.solve(x1 @ y1, x1 @ x1.transpose(1, 0))

# gamma and alpha1 are redundant. As long as gamma + alpha1 = p1[0] we get the same optimal value for loss
gamma = p1[0] / 2
alpha_xy = p1[1]
alpha1 = p1[0] / 2

x2 = torch.stack((x_train[:, 0], x_train[:, 1]), dim=0)
y2 = y_train[:, 1].unsqueeze(1)

p2, _ = torch.solve(x2 @ y2, x2 @ x2.transpose(1, 0))

beta_y = p2[0]
alpha2 = p2[1]

x3 = torch.stack((x_train[:, 1], x_train[:, 2], x_train[:, 0] * x_train[:, 2]), dim=0)
y3 = y_train[:, 2].unsqueeze(1)

p3, _ = torch.solve(x3 @ y3, x3 @ x3.transpose(1, 0))

alpha0 = p3[0]
alpha_y = p3[1]
alpha3 = p3[2]

loss_1 = torch.sum((x1.transpose(1, 0) @ p1 - y1)**2 + (x2.transpose(1, 0) @ p2 - y2)**2 + (x3.transpose(1, 0) @ p3 - y3)**2)
mse = loss_1 / x_train.size(0)

To test this code is working I generated some fake data which I knew the underlying model coefficients (there's some noise added so the final result won't exactly match the expected).

def gen_fake_data(samples=50000):
    x_train = torch.randn(samples, 3)
    # define fake data with known minimal solutions
    x1 = torch.stack((x_train[:, 0], x_train[:, 0] * x_train[:, 2]), dim=0)
    x2 = torch.stack((x_train[:, 0], x_train[:, 1]), dim=0)
    x3 = torch.stack((x_train[:, 1], x_train[:, 2], x_train[:, 0] * x_train[:, 2]), dim=0)
    y1 = x1.transpose(1, 0) @ torch.tensor([[1.0], [2.0]])  # gamma + alpha1 = 1.0
    y2 = x2.transpose(1, 0) @ torch.tensor([[3.0], [4.0]])
    y3 = x3.transpose(1, 0) @ torch.tensor([[5.0], [6.0], [7.0]])
    y_train = torch.cat((y1, y2, y3), dim=1) + 0.1 * torch.randn(samples, 3)
    return x_train, y_train

x_train, y_train = gen_fake_data()

# optimization code from above
...

print('loss_1:', loss_1.item())
print('MSE:', mse.item())

print('Expected 0.5, 2.0, 0.5, 3.0, 4.0, 5.0, 6.0, 7.0')
print('Actual', gamma.item(), alpha_xy.item(), alpha1.item(), beta_y.item(), alpha2.item(), alpha0.item(), alpha_y.item(), alpha3.item())

which results in

loss_1: 1491.731201171875
MSE: 0.029834624379873276
Expected 0.5, 2.0, 0.5, 3.0, 4.0, 5.0, 6.0, 7.0
Actual 0.50002 2.0011 0.50002 3.0009 3.9997 5.0000 6.0002 6.9994


Instruction: any similar function like df.mask for tensor in pytorch?

Input:
I want to replace all 0s in a 2-D tensor with -5.

With dataframe, i can easily do this:

df = df.mask(df=0, -5)

but this does not work for tensors. I have tried:

 y = torch.where(y = 0, -5, y)

Output:
There are two general ways.

One, given above by prhmma is to use in-place mutation like y[y == 0] = -5. It is nice and efficient, but will break autograd operation. So if you want gradient to flow through y, you should not do that.

The other is to use torch.where, as you have attempted. The proper incantation is

y = torch.where(y == 0, torch.tensor(-5), y)

or, if you want to be device- and dtype-agnostic

five = torch.tensor(-5, dtype=y.dtype, device=y.device)
y = torch.where(y == 0, five, y)

the fact that where does not accept scalars is an annoying papercut, but that's how it is ATM. Note that while the choice itself is discrete and obviously not differentiable, this operation will let gradients flow through both operands.


Instruction: Pytorch equivalent of Google Seedbank

Input: Does Pytorch have an equivalent of Google Seedbank ? Everything in Seedbank is (unsurprisingly) Tensorflow based, and I want to learn Pytorch.

Output: You can check out Pytorch Hub. Compared to Seedback, it has more emphasis on re-useable models (and less on tutorials), although many entries do have Colab notebooks for reference.


Instruction: Conda package install [Errno 13] Permission denied while installing conda-forge::protobuf-3.8.0

Input:
I have a conda environment with Python 3.6 and something went wrong with my Pytorch installation so I tried to install it again.  Towards the end of the installation I get this error:

ERROR conda.core.link:_execute(700): An error occurred while installing package 'conda-forge::protobuf-3.8.0-py36h6de7cb9_1'.
Rolling back transaction: done

[Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/__init__.py'
()

Also, it says "The environment is inconsistent" which is probably a clue something was already wrong.

Full details:

$ conda install pytorch torchvision -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: -
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - conda-forge/osx-64::tensorboard==1.14.0=py36_0
  - conda-forge/noarch::tensorboardx==1.9=py_0
done

## Package Plan ##

  environment location: /Users/myusername/anaconda3/envs/torch

  added / updated specs:
    - pytorch
    - torchvision

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    intel-openmp-2019.4        |              233         887 KB
    mkl-2019.4                 |              233       101.9 MB
    pytorch-1.3.1              |          py3.6_0        32.7 MB  pytorch
    torchvision-0.4.2          |         py36_cpu         5.9 MB  pytorch
    ------------------------------------------------------------
                                           Total:       141.4 MB

The following NEW packages will be INSTALLED:

  intel-openmp       pkgs/main/osx-64::intel-openmp-2019.4-233
  libprotobuf        conda-forge/osx-64::libprotobuf-3.8.0-hfbae3c0_0
  mkl                pkgs/main/osx-64::mkl-2019.4-233
  protobuf           conda-forge/osx-64::protobuf-3.8.0-py36h6de7cb9_1
  pytorch            pytorch/osx-64::pytorch-1.3.1-py3.6_0
  torchvision        pytorch/osx-64::torchvision-0.4.2-py36_cpu

The following packages will be UPDATED:

  openssl                                 1.1.1c-h01d97ff_0 --> 1.1.1d-h0b31af3_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
pytorch-1.3.1        | 32.7 MB   | ##################################### | 100%
torchvision-0.4.2    | 5.9 MB    | ##################################### | 100%
mkl-2019.4           | 101.9 MB  | ##################################### | 100%
intel-openmp-2019.4  | 887 KB    | ##################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
ERROR conda.core.link:_execute(700): An error occurred while installing package 'conda-forge::protobuf-3.8.0-py36h6de7cb9_1'.
Rolling back transaction: done

[Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/__init__.py'
()

UPDATE:
I tried uninstalling tensorboard and tensorboardx but the same error occurs preventing the uninstall.

ALSO:

Tried pip uninstall protobuf but that caused a similar error:

Uninstalling protobuf-3.10.0:
  Would remove:
...

ERROR: Exception:
Traceback (most recent call last):
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 544, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/' -> '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/~-otobuf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/commands/uninstall.py", line 79, in run
    auto_confirm=options.yes, verbose=self.verbosity > 0,
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 755, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 394, in remove
    moved.stash(path)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 283, in stash
    renames(path, new_path)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 338, in renames
    shutil.move(old, new)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 556, in move
    rmtree(src)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 389, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "/Users/myusername/anaconda3/envs/torch/lib/python3.6/shutil.py", line 387, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [Errno 13] Permission denied: '/Users/myusername/anaconda3/envs/torch/lib/python3.6/site-packages/google/protobuf/descriptor.py'

I have no idea how to fix this other than deleting the whole environment and start again.

Output:
I faced a similar error like this when I was trying to update/uninstall a python package(matplotlib) from my environment. The reason turned out to be that I had another python application which was running and had a matplotlib plot window open, so therefore since a process was accessing the package, it couldn't be deleted.
When i closed all the python programs, I was able to upgrade the package without the permission error.
So moral of the story, if you trying to update/uninstall a package, make sure all your python scripts are not running.


Instruction: somehow my accuracy is very low on cifar10?

Input:
with torch.no_grad():
    for data in test_loader:
        images,labels = data
        images, labels = images.to(device), labels.to(device)
        outputs, features = net(images)
        _ , predicted = torch.max(outputs,1)
        total += labels.size(0)
        correct += (predicted==labels).sum().item()
    print('Accuracy of the network on the 10000 test images: %d %%' % (
            100 * (correct / total)))

How do I obtain GPU results with a label?
I got almost 10% accuracy but my original training is accuracy is 70%.

Output:
The trick training that exact dataset (cifar10) and getting better accuracy is to use data augmentation.

Originally cifar10 has 50.000 images for training and 10.000 for validation.

If you don't augment images while training you will overfit. Training accuracy will be much bigger than validation accuracy.

So your goal is to reduce overfitting.

Best Way to Reduce overfitting is to train on more data (augment your data).
Here is one repo that may help you dealing with augmentation in PyTorch.

And in PyTorch check these to augment your data such as RandomRotation, Resize, RandomVerticalFlip, RandomSizedCrop, ...

One example of a native PyTorch transform may look like:

t = transforms.Compose([transforms.RandomHorizontalFlip(),
                       transforms.ToTensor(),
                       transforms.RandomErasing(),
                       transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))]
                       )

dl_train = DataLoader( torchvision.datasets.CIFAR10('/data/cifar10', download=True, train=True, transform=t),
                batch_size=bs, shuffle=True)
dl_valid = DataLoader( torchvision.datasets.CIFAR10('/data/cifar10', download=True, train=False, transform=t),
                batch_size=bs, shuffle=True)


Instruction: Why multiprocessing suddenly stops after a certain number of tasks?

Input:
I'm trying to write some code to parallelize a bunch of tasks. Basically, the script is organized as the following.

import multiprocessing as mp

def obj_train(x):
    return x.train()

class ServerModel(nn.Module):
    self.S = nn.Parameter(torch.rand(x, y), requires_grad=True)

class ClientModel(nn.Module):
    self.S = nn.Parameter(torch.rand(x, y), requires_grad=True)
    self.U = nn.Parameter(torch.rand(x, y), requires_grad=True)

class Server:
    def __init__(self, model):
        self.model = model
        ...

    def train(clients):
        for i, c in enumerate(clients):
            sd = c.model.state_dict()
            sd['S'] = self.model.S
            c.model.load_state_dict(sd)
        self.c_list = random.sample(clients, 200)
        pool = mp.Pool(mp.cpu_count()-1)
        results = pool.map(obj_train, self.c_list)
        pool.close()
        pool.join()
        print("Training complete")

class Client:
    def __init__(self, client_id, model, train_set):
        self.id = client_id
        self.model = model
        self.train_set = train_set

    def train(self):
        self.optimizer = optim.SGD([self.model.S, self.model.U])
        for i in self.train_set:
            loss = self.model(i)
            loss.backward()
            self.optimizer.step()
        print("Trained client %d", self.id)

        return self.model.S

if __name__ == '__main__':
    ...
    server = Server(server_model)
    clients = [Client(u, ClientModel(), train_set[u]) for u in range(n_clients)]
    server.train(clients)

Ok, the problem is in multiprocessing. I tried with a lot of approaches but all of them gives me the same problem. Server should manage the training of 200 clients, but after a certain number of trainings (it depends on the approach, but approx 50-100), the script completely stucks and cores of the CPU stop working.

Have you any ideas? Other approaches I tried are for example mp.Pool and with ProcessPoolExecutor.

Thank you for your help.

Output:
Could it be that you hit the maximum number of processes/threads your machine is able to handle?
It is common, for example, when moving a web crawler from development to production that the machine does not allow more processes.

I would give a look at the file

/etc/sysctl.d

and in case increase the number of possible processes for the machine to handle.

Another reason might be that you capped RAM limit or something similar, try to give another quick look at the command

htop

followed by

free -m

and see what they tell you. It might be a hardware problem. While from a software it might be that the library you are using https://docs.python.org/2/library/multiprocessing.html has a hard-coded limit. Also here you can easily set it higher within the library parameters.

Last but not least, try to find the problem incrementally. I would test it with with 2 processes and increment slowly to see when the application starts having issues. And at that point it would probably be even clearer what the issue was. Good luck!


Instruction: ModuleNotFoundError: No module named 'torch'

Input:
I try to use pytorch module by conda but I get an error

Traceback (most recent call last):
File "train.py", line 8, in <module>
import torch
ModuleNotFoundError: No module named 'torch'

when I write conda list | findstr torch I see that torch is installed:

What is the problem?

I tried:

conda update conda -n root
conda install mkl=2018

but get:

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

Could Not Find C:\WINDOWS\TEMP\conda-23721-26470.tmp
Could Not Find C:\WINDOWS\TEMP\tmpry_dlvar.bat

and the same error occurrence

Output:
Try the below mentioned one, surely  it will work.

conda install -c pytorch pytorch


Instruction: Use numpy in script class (torch.jit.script)

Input:
I was wondering if I can use numpy APIs in a function which is going to be scripted by torch.jit.script. I have this simple function which does not work:

import torch
import torch.nn as nn

class MyModule(nn.Module):

def __init__(self):
    super(MyModule, self).__init__()

@torch.jit.ignore
def call_np():
    return torch.jit.export(np.random.choice(2, p=[.95,.05]))

def forward(self):
    pass

@torch.jit.export
def func(self):
    done = self.call_np()
    print (done)

  scripted_module = torch.jit.script(MyModule())
  scripted_module.func()

which results in:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-133-ab1ce37d6edc> in <module>()
     18         print (done)
     19
---> 20 scripted_module = torch.jit.script(MyModule())
     21 scripted_module.func()

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\__init__.py in script(obj, optimize, _frames_up, _rcb)
   1201
   1202     if isinstance(obj, torch.nn.Module):
-> 1203         return torch.jit.torch.jit._recursive.recursive_script(obj)
   1204
   1205     qualified_name = _qualified_name(obj)

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\_recursive.py in recursive_script(mod, exclude_methods)
    171     filtered_methods = filter(ignore_overloaded, methods)
    172     stubs = list(map(make_stub, filtered_methods))
--> 173     return copy_to_script_module(mod, overload_stubs + stubs)
    174
    175

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\_recursive.py in copy_to_script_module(original, stubs)
     93             setattr(script_module, name, item)
     94
---> 95     torch.jit._create_methods_from_stubs(script_module, stubs)
     96
     97     # Now that methods have been compiled, take methods that have been compiled

C:\ProgramData\Anaconda3\lib\site-packages\torch\jit\__init__.py in _create_methods_from_stubs(self, stubs)
   1421     rcbs = [m.resolution_callback for m in stubs]
   1422     defaults = [get_default_args(m.original_method) for m in stubs]
-> 1423     self._c._create_methods(self, defs, rcbs, defaults)
   1424
   1425 # For each user-defined class that subclasses ScriptModule this meta-class,

RuntimeError: Unable to cast Python instance of type <class 'int'> to C++ type 'unsigned __int64'

I appreciate any help or comment.

Output:
I got an answered at the pytorch forum:
https://discuss.pytorch.org/t/use-numpy-in-script-class-torch-jit-script/62351


Instruction: Forward Propagate RNN using Pytorch

Input:
I am trying to create an RNN forward pass method that can take a variable input, hidden, and output size and create the rnn cells needed. To me, it seems like I am passing the correct variables to self.rnn_cell -- the input values of x and the previous hidden layer. However, the error I receive is included below.

I have also tried using x[i] and x[:,i,i] (as suggested by my professor) to no avail. I am confused and just looking for guidance as to whether or not I am doing the right thing here. My prof suggested that since I keep receiving errors, I should restart the kernel in jupyter notebook and rerun code. I have, and I receive the same errors...

Please let me know if you need additional context.

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size

        self.rnn_cell = nn.RNNCell(input_size, hidden_size)
        self.fc = nn.Linear(hidden_size, output_size)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        """
        x: size [seq_length, 1, input_size]
        """
        h = torch.zeros(x.size(1), self.hidden_size)

        for i in range(x.size(0)):
            ### START YOUR CODE ###
            h = self.rnn_cell(x[:,:,i], h)
            ### END YOUR CODE ###

        ### START YOUR CODE ###
        # Hint: first call fc, then call softmax
        out = self.softmax(self.fc(self.hidden_size, h.size(0)))
        ### END YOUR CODE ###

        return out
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Output:
I am not an expert at RNNs but giving it a try.

class RNN(nn.Module):
def __init__(self, input_size, hidden_size, output_size):
    super(RNN, self).__init__()
    self.hidden_size = hidden_size

    self.rnn_cell = nn.RNN(input_size, hidden_size)
    self.fc = nn.Linear(hidden_size, output_size)

def forward(self, x):
    """
    x: size [seq_length, 1, input_size]
    """
    h = torch.zeros(num_layers(hidden), x.size(0), self.hidden_size)

        ### START YOUR CODE ###
    out,hidden = self.rnn_cell(x, h)
        ### END YOUR CODE ###

    ### START YOUR CODE ###
    # Hint: first call fc, then call softmax
    out = out.contiguous().view(-1, self.hidden_dim) #You need to reshape the output to fit the FC layer
    out = self.fc(out)
    return F.softmax(out)
    ### END YOUR CODE ###

    return out

Please try running this and let me know in case of errors or any doubts. (Cannot ask you details as I can't comment rn.)

If you got any idea from my answer, do support.


Instruction: NOT using multiprocessing but get CUDA error on google colab while using PyTorch DataLoader

Input:
I've cloned my GitHub repo into google colab and trying to load data using PyTorch's DataLoader.

global gpu, device
if torch.cuda.is_available():
    gpu = True
    device = 'cuda:0'
    torch.set_default_tensor_type('torch.cuda.FloatTensor')
    print("Using GPU")
else:
    gpu = False
    device = 'cpu'
    print("Using CPU")

data_transforms = transforms.Compose([
    #transforms.Resize(224),
    transforms.ToTensor(),
    transforms.Normalize([0.3112, 0.2636, 0.2047], [0.2419, 0.1972, 0.1554])
    ])
train_path = '/content/convLSTM/code/data/train/'
val_path = '/content/convLSTM/code/data/val/'
test_path = '/content/convLSTM/code/data/test/'

train_data = datasets.ImageFolder(root=train_path, transform=data_transforms)
val_data = datasets.ImageFolder(root=val_path, transform=data_transforms)
test_data = datasets.ImageFolder(root=test_path, transform=data_transforms)

train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=18,
    num_workers=4,
    shuffle=False,
    pin_memory=True
    )

val_loader = torch.utils.data.DataLoader(
    val_data,
    batch_size=18,
    shuffle=False,
    num_workers=4,
    pin_memory=True
    )

test_loader = torch.utils.data.DataLoader(
    test_data,
    batch_size=18,
    shuffle=False,
    num_workers=4,
    pin_memory=True
    )
for batch_idx, (data, target) in enumerate(train_loader):
  print(batch_idx)
  if batch_idx==3:
    break

I'm getting the following error when I run the last for loop:

RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

I tried num_workers = 1 instead of 4 but the error persists. I'm not using any multiprocessing.

I also tried without setting torch.set_default_tensor_type('torch.cuda.FloatTensor') but the error persists.

Python  : 3.6.8 | PyTorch : 1.3.1

What seems to be the problem?

Output:
Not sure if you fixed it already but just in case someone else reads this, using n number of works activates pytorch multi processing. To disable it you need to have the default number of workers which is 0, not 1.

Try setting num_workers to 0 or using the Torch Multiprocessing submodule.


Instruction: Running in Google Colab

Input:
Good evening.
I want to run my python program on Google collab, but in what place I should download files and when open in python file?
How do I open this file?

Output:
You can always upload files to Google colab and you can create a directory as well.

You can create a directory named data

And then upload the files which you want to be placed in the directory as shown below.

* Remember the data uploaded or created during runtime will not be saved *

Alternatively, you can save the files to your google drive and mount the drive
->
Google Colab: how to read data from my google drive? to save your runtime files and folder directly to the drive.


Instruction: Flatten Tensor in Pytorch Convolutional Neural Network (size mismatch error)

Input:
I made a reproducible example with random pixels. I'm trying to flatten the tensors for the dense layers after the convolutional layers. The problem is at the intersection of the convolutional layers and the dense layers. I don't know how to put the right number of neurons.

tl;dr I'm looking for the manual equivalent of keras.layers.Flatten() since it doesn't exist in pytorch.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader

x = np.random.rand(1_00, 3, 100, 100)
y = np.random.randint(0, 2, 1_00)

if torch.cuda.is_available():
    x = torch.from_numpy(x.astype('float32')).cuda()
    y = torch.from_numpy(y.astype('float32')).cuda()

class ConvNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(128, 1024) # 128 is wrong here
        self.fc2 = nn.Linear(1024, 1)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        return x

net = ConvNet()
net.cuda()
optimizer = optim.Adam(net.parameters(), lr=0.03)
loss_function = nn.BCELoss()

class Train:

    def __init__(self):
        self.len = x.shape[0]
        self.x_train = x
        self.y_train = y

    def __getitem__(self, index):
        return x[index], y[index].unsqueeze(0)

    def __len__(self):
        return self.len

train = Train()
train_loader = DataLoader(dataset=train, batch_size=64, shuffle=True)

epochs = 1
train_losses = list()
for e in range(epochs):
    running_loss = 0
    for images, labels in train_loader:
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    print('It\'s working.')

Output:
Here's a function I made to automatically fit the right number of neurons while flattening a convolutional tensor:

def flatten(w, k=3, s=1, p=0, m=True):
    """
    Returns the right size of the flattened tensor after
        convolutional transformation
    :param w: width of image
    :param k: kernel size
    :param s: stride
    :param p: padding
    :param m: max pooling (bool)
    :return: proper shape and params: use x * x * previous_out_channels

    Example:
    r = flatten(*flatten(*flatten(w=100, k=3, s=1, p=0, m=True)))[0]
    self.fc1 = nn.Linear(r*r*128, 1024)
    """
    return int((np.floor((w - k + 2 * p) / s) + 1) / 2 if m else 1), k, s, p, m

In your case:

def __init__(self):
    super().__init__()
    self.conv1 = nn.Conv2d(3, 32, 3)
    self.conv2 = nn.Conv2d(32, 64, 3)
    self.conv3 = nn.Conv2d(64, 128, 3)

    r = flatten(*flatten(*flatten(w=100, k=3, s=1, p=0, m=True)))[0]

    self.fc1 = nn.Linear(r*r*128, 1024)
    self.fc2 = nn.Linear(1024, 1)

    def forward(self, x): ...


Instruction: How can I do to evaluate mean and std for a dataset?

Input:
I am using pytorch and the dataset fashion MNIST but I do not know how can I do to evaluate the mean and the std for this dataset. Here is my code :

import torch
from torchvision import datasets, transforms
import torch.nn.functional as F

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((mean), (std))])
batch_size = 32
train_loader = torch.utils.data.DataLoader(datasets.MNIST(
'../data', train=True, download=True, transform=transform)
, batch_size=batch_size, shuffle=True)

Could you help me please ?

Thank you very much !

Output:
Use this to calculate mean and std-

loader = data.DataLoader(dataset,
                         batch_size=10,
                         num_workers=0,
                         shuffle=False)

mean = 0.
std = 0.
for images, _ in loader:
    batch_samples = images.size(0) # batch size (the last batch can have smaller size!)
    images = images.view(batch_samples, images.size(1), -1)
    mean += images.mean(2).sum(0)
    std += images.std(2).sum(0)

mean /= len(loader.dataset)
std /= len(loader.dataset)


Instruction: How to use gensim with pytorch to create an intent classifier (With LSTM NN)?

Input:
The problem to solve: Given a sentence, return the intent behind it (Think chatbot)

Reduced example dataset (Intent on the left of dict):

data_raw    = {"mk_reservation" : ["i want to make a reservation",
                                   "book a table for me"],
               "show_menu"      : ["what's the daily menu",
                                   "do you serve pizza"],
               "payment_method" : ["how can i pay",
                                   "can i use cash"],
               "schedule_info"  : ["when do you open",
                                   "at what time do you close"]}

I have stripped down the sentences with spaCy, and tokenized each word by using the word2vec  algorithm provided by the gensim library.

This is what resulted from the use of word2vec model GoogleNews-vectors-negative300.bin:

[[[ 5.99331968e-02  6.50703311e-02  5.03010787e-02 ... -8.00536275e-02
    1.94782894e-02 -1.83010306e-02]
  [-2.14406010e-02 -1.00447744e-01  6.13847338e-02 ... -6.72588721e-02
    3.03986594e-02 -4.14126664e-02]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  ...
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]]

 [[ 4.48647663e-02 -1.03907576e-02 -1.78682189e-02 ...  3.84555124e-02
   -2.29179319e-02 -2.05144612e-03]
  [-5.39291985e-02 -9.88398306e-03  4.39085700e-02 ... -3.55276838e-02
   -3.66208404e-02 -4.57760505e-03]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  ...
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]]]

This is a List of sentences, and each sentence is a list of words ([sentences[sentence[word]]])
Each sentence (list) must be of size 10 words (I am padding the remaining with zeroes)
Each word (list) has 300 elements (word2vec dimensions)

By following some tutorials i transformed this to a TensorDataset.

At this moment, i am very confused on how to use the word2vec and probably i have just been wasting time, as of now i believe the embeddings layer from an LSTM configuration should be composed by importing the word2vec model weights using:

import gensim
model = gensim.models.KeyedVectors.load_word2vec_format('path/to/file')
weights = torch.FloatTensor(model.vectors)
word_embeddings = nn.Embedding.from_pretrained(weights)

This is not enough as pytorch is saying it does not accept embeddings where indices are not INT type  .

EDIT: I found out that importing the weight matrix from gensim word2vec is not straightforward, one has to import the word_index table as well.

As soon as i fix this issue i'll post it here.

Output:
You don't need neither a neural network nor word embeddings. Use parsed trees with NLTK, where intents are Verbs V acting on entities (N) in a given utterance:

To classify a sentence, then you can use a Neural Net. I personally like BERT in fast.ai. Once again, you won't need embeddings to run the classification, and you can do it in multilanguage:

Fast.ai_BERT_ULMFit

Also, you can use Named Entity Recognition if you are working on a chatbot, to guide conversations.


Instruction: Heroku: slug size too large after installing Pytorch

Input:
I've been getting the slug size too large warning (Compiled slug size: 789.8M is too large (max is 500M)) from Heroku and I can't figure out why, as my model size (cnn.pth below) is fairly small and my file directory is only 1.1mb in total: screenshot of directory.

It seems like the size increase is caused by running pipenv install torch, as the slug size was 89.1mb before installing torch and 798.8mb after.

My Pipfile currently has these packages installed:

[packages]
flask = "*"
flask-sqlalchemy = "*"
psycopg2 = "*"
psycopg2-binary = "*"
requests = "*"
numpy = "*"
gunicorn = "*"
pillow = "*"
torch = "*"

Is there any workaround for this?

Edit: I'm running Mac OSX 10.10.5, using Flask and pipenv.

Output:
The pytorch package that you're installing comes with both cpu and gpu support, thus has a large size. It seems you're using the free version of heroku, and only require the cpu support. The solution is to install the pytorch package for cpu only i.e.

In requirements.txt, write the wheel file path corresponding to the version of pytorch (cpu) that you're interested in. You can find the list of wheel files, which can be installed with pip. For example, for PyTorch 1.3.1, torchvision 0.4.2, Python 3.7, Linux, you can write the following for pytorch and torchvision respectively:

https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.4.2%2Bcpu-cp37-cp37m-linux_x86_64.whl

The above will download torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl (107MB) torchvision-0.4.2+cpu-cp37-cp37m-linux_x86_64.whl (13MB) respectively.


Instruction: How is spectral norm of a parameter calculated?

Input:
when I do,

import torch, torch.nn as nn
x = nn.Linear(3, 3)
y = torch.nn.utils.spectral_norm(x)

then it gives four different weight matrices,

y.weight_u

tensor([ 0.6534, -0.1644,  0.7390])

y.weight_orig

Parameter containing:
tensor([[ 0.2538,  0.3196,  0.3380],
        [ 0.4946,  0.0519,  0.1022],
        [-0.5549, -0.0401,  0.1654]], requires_grad=True)

y.weight_v

tensor([-0.3650,  0.2870,  0.8857])

y.weight

tensor([[ 0.5556,  0.6997,  0.7399],
        [ 1.0827,  0.1137,  0.2237],
        [-1.2149, -0.0878,  0.3622]], grad_fn=<DivBackward0>)

how are these four matrices calculated?

Output:
I just finished reading the paper for this method which can be found on arxiv. If you have the appropriate mathematical background I would recommend reading it. See appendix A for the power algorithm which describes what u and v are.

That said I'll try to summarize here.

First, you should know that the spectral norm of a matrix is the maximum singular value. The authors propose finding the spectral norm of weight matrix W, then dividing W by its spectral norm to make it close to 1 (justification for this decision is in the paper).

While we could just use torch.svd to find a precise estimate of the singular values, they instead use a fast (but imprecise) method called "power iteration". Long story short, the weight_u and weight_v are rough approximations of the left and right singular vectors corresponding to the largest singular value of W. They are useful because the associated singular value, i.e. the spectral norm, of W is equal to u.transpose(1,0) @ W @ v if u and v are the actual left/right singular vectors of W.

y.weight_orig contains the original values in the layer.
y.weight_u is the approximation of the first left singular vector of y.weight_orig.
y.weight_v is the approximation of the first right singular vector of y.weight_orig.
y.weight is the updated weight matrix which is y.weight_orig divided by its approximate spectral norm.

We can verify these claims by showing that the actual left and right singular vectors are nearly parallel to y.weight_u and y.weight_v

import torch
import torch.nn as nn

# pytorch default is 1
n_power_iterations = 1

y = nn.Linear(3,3)
y = nn.utils.spectral_norm(y, n_power_iterations=n_power_iterations)

# spectral normalization is performed during forward pre hook for technical reasons, we
# need to send something through the layer to ensure normalization is applied
# NOTE: After this is performed, x.weight is changed in place!
_ = y(torch.randn(1,3))

# test svd vs. spectral_norm u/v estimates
u,s,v = torch.svd(y.weight_orig)
cos_err_u = 1.0 - torch.abs(torch.dot(y.weight_u, u[:, 0])).item()
cos_err_v = 1.0 - torch.abs(torch.dot(y.weight_v, v[:, 0])).item()
print('u-estimate cosine error:', cos_err_u)
print('v-estimate cosine error:', cos_err_v)

# singular values
actual_orig_sn = s[0].item()
approx_orig_sn = (y.weight_u @ y.weight_orig @ y.weight_v).item()
print('Actual original spectral norm:', actual_orig_sn)
print('Approximate original spectral norm:', approx_orig_sn)

# updated weights singular values
u,s_new,v = torch.svd(y.weight.data, compute_uv=False)
actual_sn = s_new[0].item()
print('Actual updated spectral norm:', actual_sn)
print('Desired updated spectral norm: 1.0')

which results in

u-estimate cosine error: 0.00764310359954834
v-estimate cosine error: 0.034041762351989746
Actual original spectral norm: 0.8086231350898743
Approximate original spectral norm: 0.7871124148368835
Actual updated spectral norm: 1.0273288488388062
Desired updated spectral norm: 1.0

Increasing the n_power_iterations parameter will increase the accuracy of the estimate at the cost of computation time.


Instruction: installing py-torch in anaconda , got error

Input:
C:\WINDOWS\system32>python -m pip install torch
Collecting torch
  Using cached https://files.pythonhosted.org/packages/f8/02/880b468bd382dc79896eaecbeb8ce95e9c4b99a24902874a2cef0b562cea/torch-0.1.2.post2.tar.gz
Requirement already satisfied: pyyaml in c:\programdata\anaconda3\lib\site-packages (from torch) (5.1)
Building wheels for collected packages: torch
  Building wheel for torch (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'C:\ProgramData\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\vinayak\AppData\Local\Temp\pip-wheel-_w68wcvp' --python-tag cp37
       cwd: C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\
  Complete output (30 lines):
  running bdist_wheel
  running build
  running build_deps
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\setup.py", line 265, in <module>
      description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
    File "C:\ProgramData\Anaconda3\lib\site-packages\setuptools\__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "C:\ProgramData\Anaconda3\lib\distutils\core.py", line 148, in setup
      dist.run_commands()
    File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\ProgramData\Anaconda3\lib\site-packages\wheel\bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "C:\ProgramData\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\ProgramData\Anaconda3\lib\distutils\command\build.py", line 135, in run
      self.run_command(cmd_name)
    File "C:\ProgramData\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 985, in run_command
      cmd_obj.run()
    File "C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\setup.py", line 51, in run
      from tools.nnwrap import generate_wrappers as generate_nn_wrappers
  ModuleNotFoundError: No module named 'tools.nnwrap'
  ----------------------------------------
  ERROR: Failed building wheel for torch
  Running setup.py clean for torch
  ERROR: Command errored out with exit status 1:
   command: 'C:\ProgramData\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch
  Complete output (2 lines):
  running clean
  error: [Errno 2] No such file or directory: '.gitignore'
  ----------------------------------------
  ERROR: Failed cleaning build dir for torch
Failed to build torch
Installing collected packages: torch
    Running setup.py install for torch ... error
    ERROR: Command errored out with exit status 1:
     command: 'C:\ProgramData\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\vinayak\AppData\Local\Temp\pip-record-x966ekmg\install-record.txt' --single-version-externally-managed --compile
         cwd: C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\
    Complete output (23 lines):
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\setup.py", line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "C:\ProgramData\Anaconda3\lib\site-packages\setuptools\__init__.py", line 145, in setup
        return distutils.core.setup(**attrs)
      File "C:\ProgramData\Anaconda3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\setup.py", line 99, in run
        self.run_command('build_deps')
      File "C:\ProgramData\Anaconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\ProgramData\Anaconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\vinayak\AppData\Local\Temp\pip-install-67llvrv3\torch\setup.py", line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\ProgramData\Anaconda3\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\vinayak\\AppData\\Local\\Temp\\pip-install-67llvrv3\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\vinayak\AppData\Local\Temp\pip-record-x966ekmg\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.

Output:
Select the right options based on hardware configuration and install it from
https://pytorch.org/get-started/locally/

Should work without any problems.


Instruction: Conv2d and the value of padding

Input:
I would like to ask about the value of padding we set in the Conv2d function.

I know what zero-padding is. However, what does it mean that the padding is 0 for instance, or 1,2, or 3. What do these values mean? Do they represent the number of columns and rows that will be filled with zeros?

Thanks a lot.

Output:
As the documentation states:

  padding controls the amount of implicit zero-paddings on both sides for padding number of points for each dimension.

Therefore, padding=1 means you pad your input with 1 column and row from each size.
If you want more control over the amount of padding and its value you can either use a Padding layer, or the pad functional.


Instruction: How can we calculate receptive field of a network includes transposed convolutional layer?

Input:
A network I designed includes transposed convolutional layer.
(ConvTranspose2d in pytorch)

I want to get receptive field size of my network.

Does the concept of receptive field also hold on with transposed convolutional layer?

If yes, then how can I get it?

Output: You can use the library pytorch-receptive-field to automatically compute all layers' receptive fields.


Instruction: Train a single pytorch model on multiple GPUs with some layers fixed?

Input:
I met some problems when using pytorch DistributedDataParallel. The situation is:

My model is A, and it has been trained on a single GPU as usual. Suppose that there are three layers in A:

class A(nn.module):
    def __init__(self):
        super(A,self).__init__()
        self.layer0 = layer0
        self.layer1 = layer1
        self.layer2 = layer2

    def forward(self,x):
        x=self.layer0(x)
        x=self.layer1(x)
        x=self.layer2(x)
        return x

Now I have some new data. I want to fine-tune A with it on multiple GPUs. I need to wrap A as a multi-GPU model B.
But there are two training stages. In the 1st stage, I want to fix layer0 and layer1 of B. In the 2nd stage, only to fix layer0. Then requires_grad of parameters in layer1 should be changed during training.  However, DistributedDataParallel doc says:

  You should never try to change your model’s parameters after wrapping up your model with DistributedDataParallel.

In fact, I tried to use B.module to refer A wrapped in B. But the test results were abnormal compared to the single-GPU model. Maybe this way is disallowed.

What should I do? Is there any proper way to wrap my model? And what should be take care for when saving and loading the model?

Just run it on a single machine with multiple GPUs so you can ignore the distributed situation using multiple machines. Many thanks.

Update 2019.12.03

As suggested by @jodag, I tried DataParallel, but it didn't work. This time I didn't change anything in B (except training it) after wrapping it. For simplification, My code is like this (and I refered this):

class B(nn.DataParallel):
     def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
a = A()
b = B(a,device_ids=[0,1])
b = b.cuda()
trained_param = b.layer2.parameters()
# trained_param = [{'params':b.layer2.parameters()},{'params':b.layer1.parameters()}]
optimizer = optim.Adam(trained_param)
b.train()
...
for x, label in data_loader:
    optimizer.zero_grad()
    x = x.to(0) # This line can be commented.
    y = b(x)
    l = loss(y, label)
    l.backword()
    optimizer.step()

Output:
If you only try to optimize part of the parameters, why not try controlling this via the optimizer, rather than the model?
You can leave your model as-is (wrapped in a DistributedDataParallel) and pass only part of its parameters to the relevant optimizer.


Instruction: Receptive fields for feed forward network

Input: I am pretty new to artificial intelligence and neural networks. I have implemented a feed-forward neural network in PyTorch for classification on the MNIST data set. Now I want to visualize the receptive fields of (a subset of) hidden neurons. But I am having some problems with understanding the concept of receptive fields and when I google it all results are about CNNs. So can anyone help me with how I could do this in PyTorch and how to interpret the results?

Output:
I have previously described the concept of a receptive field for CNNs in this answer, just to give you some context that might be useful in my actual answer.

It seems that you are also struggling with the idea of receptive fields. Generally, you can best understand it by asking the question "which part of the (previous) layer representation is affecting my current input?"

In Convolutional layers, the formula to compute the current layer only takes part of the image as an input (or at least only changes the outcome based on a this subregion). This is precisely what the receptive field is.
Now, a fully connected layer, as the name implies, has a connection from every previous hidden state to every new hidden state, see the image below:

In that case, the receptive field is simply "every previous state" (e.g., in the image, a cell in the first turquoise layer is affected by all yellow cells), which is not very helpful. The whole idea would be to have a smaller subset instead of all available states.

Therefore, I think your question regarding implementations in PyTorch do not really make much sense, unfortunately, but I hope that the answer still provided some clarity on the topic.

As a follow-up, I also encourage you to think about the implications of this "connectedness", especially when it comes to the number of tunable parameters.