Instruction: Basic DNN with highly imbalanced dataset -- network predicts same labels

Input:
I will try to explain my issue at a high level, and I hope I'd be able to get some better understanding of the ML behind it all.

I am working with aggregated features extracted from audio files, so each feature vector is of size (1xN). The output would be a single sentiment label, Positive, Neutral, or Negative. I mapped these to 2, 1, 0 respectively (the labels are discrete by design, but maybe I could make it continuous?)

The dataset I am using is 90% neutral, 6% negative, and 4% positive, and I split these into train/dev/test. I wrote up a basic DNN in PyTorch, and have been training using CrossEntropyLoss and SGD (with nesterov momentum). The issue I am running into is that the network, after seeing only ~10% of the data, starts to predict only netural labels. The class weights converge to something like

tensor([[-0.9255],
        [ 1.9352],
        [-1.1473]])

no matter what 1xN feature vectors you feed in. I would appreciate guidance on how to address this issue.

For reference, the architecture is

DNNModel(
  (in_layer): Linear(in_features=89, out_features=1024, bias=True)
  (fcs): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): Linear(in_features=512, out_features=256, bias=True)
    (2): Linear(in_features=256, out_features=128, bias=True)
  )
  (out_layer): Sequential(
    (0): SequenceWise (
    Linear(in_features=128, out_features=3, bias=True))
  )
)

def forward(self, x):
    x =  F.relu(self.in_layer(x))
    for fc in self.fcs:
        x = F.relu(fc(x))
    x = self.out_layer(x)
    return x

Not sure if NN actually makes sense -- could it be the relus between each hidden layer or the bias? Or something else. Thanks!

EDIT: Moved to Data Science Stack Exchange, since this is more relevant there. link

Output:
There are various ways to get through this problem.You could try re-sampling your dataset.

This can be done in two ways :

You could either try under-sampling i.e, delete instances of the over represented class ,or
You could try over-sampling i.e, add more instances of the under represented class.

This is probably the simplest way but if you are willing to try this you could try penalised models.

In penalised models we impose an additional cost on the model for making classification mistakes on the minority class during training.This additional cost cost or penalties can make the model pay more attention to the minority class.There are penalised versions of algorithms such as penalised-svm and others.

For more information on the penalised-svm algorithm you could follow this link:
 [https://stats.stackexchange.com/questions/122095/does-support-vector-machine-handle-imbalanced-dataset][1]


Instruction: can't import 'torchtext' module in jupyter notebook while using pytorch

Input:
I installed pytorch using anaconda3 and my created virtual conda environment named 'torchTest'.
I installed all the modules needed but, codes doesn't work in jupyter python.

I installed torchtext using
1.pip install https://github.com/pytorch/text/archive/master.zip
2.and also pip install torchtext too.
all I mentioned successfully downloaded in my MAC OS X, but can't get what's wrong with my Jupyter notebook..

Output:
After having the same issue with torchtext from within my jupyterlab, I opened an issue on Github at the jupyterlab project as well as at the torchtext repository.

My current solution is to add the PYTHONPATH from the Anaconda env.

The Anaconda path is usually like that $HOME/anaconda/bin

You can add it from within Jupyter Lab/Notebook like that:

import sys
sys.path.append("/some/path/to/add")

import torchtext


Instruction: Installing PyTorch under conda fails with permissions error and Rolling back transaction

Input:
I'd like to use PyTorch in a Python program. The instructions for installing it require conda. After installing Conda I ran:
>conda install -c pytorch pytorch (as instructed on the PyTorch [page][1])

It looked promising -- until the end.
Solving environment: done

## Package Plan ##

  environment location: C:\ProgramData\Miniconda3

  added / updated specs:
    - pytorch

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    icc_rt-2017.0.4            |       h97af966_0         8.0 MB
    vs2015_runtime-15.5.2      |                3         2.2 MB
    pytorch-0.4.0              |py36_cuda80_cudnn7he774522_1       529.2 MB      pytorch
    mkl-2018.0.3               |                1       178.1 MB
    numpy-1.14.5               |   py36h9fa60d3_4          35 KB
    intel-openmp-2018.0.3      |                0         1.7 MB
    numpy-base-1.14.5          |   py36h5c71026_4         3.8 MB
    vc-14.1                    |       h0510ff6_3           5 KB
    blas-1.0                   |              mkl           6 KB
    conda-4.5.8                |           py36_0         1.0 MB
    mkl_fft-1.0.2              |   py36hb217b18_0         113 KB
    mkl_random-1.0.1           |   py36h77b88f5_1         268 KB
    ------------------------------------------------------------
                                           Total:       724.4 MB

The following NEW packages will be INSTALLED:

    blas:           1.0-mkl
    icc_rt:         2017.0.4-h97af966_0
    intel-openmp:   2018.0.3-0
    mkl:            2018.0.3-1
    mkl_fft:        1.0.2-py36hb217b18_0
    mkl_random:     1.0.1-py36h77b88f5_1
    numpy:          1.14.5-py36h9fa60d3_4
    numpy-base:     1.14.5-py36h5c71026_4
    pytorch:        0.4.0-py36_cuda80_cudnn7he774522_1 pytorch

The following packages will be UPDATED:

    conda:          4.5.4-py36_0                               --> 4.5.8-py36_0
    vc:             14-h0510ff6_3                              --> 14.1-h0510ff6_3
    vs2015_runtime: 14.0.25123-3                               --> 15.5.2-3

Proceed ([y]/n)? y

Downloading and Extracting Packages
icc_rt-2017.0.4      |  8.0 MB | ############################################################################## | 100%
vs2015_runtime-15.5. |  2.2 MB | ############################################################################## | 100%
pytorch-0.4.0        | 529.2 MB | ############################################################################# | 100%
mkl-2018.0.3         | 178.1 MB | ############################################################################# | 100%
numpy-1.14.5         |   35 KB | ############################################################################## | 100%
intel-openmp-2018.0. |  1.7 MB | ############################################################################## | 100%
numpy-base-1.14.5    |  3.8 MB | ############################################################################## | 100%
vc-14.1              |    5 KB | ############################################################################## | 100%
blas-1.0             |    6 KB | ############################################################################## | 100%
conda-4.5.8          |  1.0 MB | ############################################################################## | 100%
mkl_fft-1.0.2        |  113 KB | ############################################################################## | 100%
mkl_random-1.0.1     |  268 KB | ############################################################################## | 100%
Preparing transaction: done
Verifying transaction: done

But then this.
Executing transaction: failed
ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
PermissionError(13, 'Access is denied')
Attempting to roll back.

Rolling back transaction: done

PermissionError(13, 'Access is denied')

Apparently it was at least partly installed because PyCharm was able to see it. But when I asked PyCharm to install it in an environment, I got this error.
RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

It suggests an alternative way to install PyTorch.  So I tried that.
>conda install pytorch torchvision -c pytorch
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - torchvision

Current channels:

  - https://conda.anaconda.org/pytorch/win-64
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/win-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/win-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/pro/win-64
  - https://repo.anaconda.com/pkgs/pro/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-64
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

But when I do that and search for PyTorch, I eventually find myself back at the original instructions.
When I search for Torchvision, no Windows versions are listed.

Output:
Try the following steps in Windows:

Create a virtual environment using the command :

conda create -n py_env python=3.5
source activate py_env
conda install pytorch-cpu -c pytorch
pip install torchvision

Note: You can use any name instead of py_env

Thanks


Instruction: Dimension of hidden layer LSTM Pytorch

Input:
I was reading the implementation of LSTM in Pytorch.
The code goes like this:

lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [torch.randn(1, 3) for _ in range(5)]  # make a sequence of length 5

# initialize the hidden state.
hidden = (torch.randn(1, 1, 3),
          torch.randn(1, 1, 3))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)

I don't understand why the hidden state is defined by a tuple of two tensors instead of one? Since the hidden layer is simply a layer of the feed-forward neural network which is a vector.

Output:
Apart from the hidden state, LSTM also has cell state, C. Therefore, a tuple is passed I think. See https://pytorch.org/docs/stable/nn.html#lstmcell.

If you don't pass C, it is taken to be all zeros.

Note that this is the case for LSTM, GRU or RNN do not have C.


Instruction: How to use PyTorch to calculate partial derivatives?

Input:
I want to use PyTorch to get the partial derivatives between output and input. Suppose I have a function Y = 5*x1^4 + 3*x2^3 + 7*x1^2 + 9*x2 - 5, and I train a network to replace this function, then I use autograd to calculate dYdx1, dYdx2:

net = torch.load('net_723.pkl')
x = torch.tensor([[1,-1]],requires_grad=True).type(torch.FloatTensor)
y = net(x)
grad_c = torch.autograd.grad(y,x,create_graph=True,retain_graph=True)[0]

Then I get a wrong derivative as:

>>>tensor([[ 7.5583, -5.3173]])

but when I use function to calculate, I get the right answer:

Y = 5*x[0,0]**4 + 3*x[0,1]**3 + 7*x[0,0]**2 + 9*x[0,1] - 5
grad_c = torch.autograd.grad(Y,x,create_graph=True,retain_graph=True)[0]
>>>tensor([[ 34.,  18.]])

Why does this happen?

Output:
A neural network is a universal function approximator. What that means is, that, for enough computational resources, training time, nodes, etc., you can approximate any function.
Without any further information on how you trained your network in the first example, I would suspect that your network simply does not fit properly to the underlying function, meaning that the internal representation of your network actually models a different function!

For the second code snippet, autmatic differentiation does give you the exact partial derivative. It does so via a different method, see another one of my answers on SO, on the topic of AutoDiff/Autograd specifically.


Instruction: How to create sub network reference in pytorch?

Input:
The simple explanation for what I aim to do is the following

Given network O with structure of

---------           ----------            ----------            ----------            ----------
| Input | -> (x) -> | Blob A | -> (xa) -> | Blob B | -> (xb) -> | Blob C | -> (xc) -> | Output |
---------           ----------            ----------            ----------            ----------

I want to create a sub network to calculate a noise loss function for Blob C. The operation is given input xb with original output xc, and pass xb + noise through Blob C again to get xc'. Then the mse_loss is compute between xc and xc'

I have tried creating nn.Sequential from the original model. But I am not sure that it created a new deep copy or a reference.

If I have missed anything, please comment

Thank you

Output:
So after some testing, I found out that if the layer references is kept (like in some variable), and then create a new model using nn.Sequential with that layer architecture, the new model will share the same layer reference. So when the original network is updated, the new model is also updated.

The code I used to test my hypothesis is the following

class TestNN(nn.Module):
    def __init__(self):
        super(TestNN, self).__init__()

        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(64, 64, kernel_size=3, padding=1)
        self.relu2 = nn.ReLU()
        self.conv3 = nn.Conv2d(64, 3, kernel_size=3, padding=1)
        self.relu3 = nn.ReLU()

    def forward(self, x):
        in_x = x
        h = self.relu1(self.conv1(in_x))
        h = self.relu2(self.conv2(h))
        h = self.relu3(self.conv3(h))
        return h

net = TestNN()

testInput = torch.from_numpy(np.random.rand(1, 3, 3, 3)).float()
target = torch.from_numpy(np.random.rand(1, 3, 3, 3)).float()

criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

def subnetwork(model, start_layer_idx, end_layer_idx):
    subnetwork = nn.Sequential()
    for idx, layer in enumerate(list(model)[start_layer_idx: end_layer_idx+1]):
        subnetwork.add_module("layer_{}".format(idx), layer)
    return subnetwork

start = subnetwork(net.children(), 0, 1)
middle = subnetwork(net.children(), 2, 3)
end = subnetwork(net.children(), 4, 5)

print(end(middle(start(testInput))))
print(net(testInput))

for idx in range(5):
    net.zero_grad()
    out = net(testInput)
    loss = criterion(out, target)

    print("[{}] {:4f}".format(idx, loss))
    loss.backward()
    optimizer.step()

print(end(middle(start(testInput))))
print(net(testInput))

The output before and after the training is the same. So I concluded that my hypothesis is correct.

To finish my objective, I created a 'transparent' loss like from this tutorial.

class NoiseLoss(nn.Module):

    def __init__(self, subnet, noise_count = 20, noise_range=0.3):
        super(NoiseLoss, self).__init__()
        self.net = subnet
        self.noise_count = noise_count
        self.noise_range = noise_range

    def add_noise(self, x):
        b, c, h, w = x.size()
        noise = torch.zeros(c, h, w)
        for i in range(self.noise_count):
            row, col = rng.randint(0, h-1), rng.randint(0, w-1)
            for j in range(c):
                noise[j,row,col] = (2*(rng.random()%self.noise_range)) - self.noise_range
        noise = noise.float()
        xp = x.clone()
        for b_idx in range(b):
            xp[b_idx,:,:,:] = xp[b_idx,:,:,:] + noise
        return xp

    def forward(self, x):
        self.loss = F.mse_loss(x, self.add_noise(x))
        print(self.loss)
        return x

noise_losses = []
testLoss = NoiseLoss(subnetwork(net.children(), 2, 3))
middle.add_module('noise_loss_test', testLoss)
noise_losses.append(testLoss)

and modify my loop to

    ...
    print("[{}] {:4f}".format(idx, loss))
    for nl in noise_losses:
        loss += nl.loss
    loss.backward(retain_graph=True)
    ...

If I miss something, please leave a comment


Instruction: How to train neural network over days?

Input:
I need to train a CNN that will take 1-2 days to train on a remotely accessed GPU server.

Will I simply need to leave my laptop on overnight for the training to be complete or is there a way to save the state of the training and resume from there the next day?

(Implementation in pytorch)

Output:
I assume you ssh into you remote server.  When training the model by running your script, say, $ python train.py, simply pre-append nohup:

$ nohup python train.py

This tells your process to disregard the hangup signal when you exit the ssh session and shut down your laptop.


Instruction: Debug Pytorch Optimizer

Input:
When I run optimizer.step on my code, I get this error

RuntimeError: sqrt not implemented for 'torch.LongTensor'

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189
    190         if callable(arg):

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magics\execution.py in time(self, line, cell, local_ns)
   1178         else:
   1179             st = clock2()
-> 1180             exec(code, glob, local_ns)
   1181             end = clock2()
   1182             out = None

<timed exec> in <module>()

C:\Program Files\Anaconda3\lib\site-packages\torch\optim\adam.py in step(self, closure)
     98                     denom = max_exp_avg_sq.sqrt().add_(group['eps'])
     99                 else:
--> 100                     denom = exp_avg_sq.sqrt().add_(group['eps'])
    101
    102                 bias_correction1 = 1 - beta1 ** state['step']

RuntimeError: sqrt not implemented for 'torch.LongTensor'

I am using my own loss function. My question is how will I debug this error? Is there a quick way to see the type of all my variables? I am manually doing it and all of them are type float (including the output of my custom loss). I can't figure out why we are even getting an error related to a LongTensor. How does the optimizer.step function work in PyTorch?

Just in case, below is most of the code.
This is the model:

class LSTM(nn.Module):

    def __init__(self, mel_channels=40, frames=81, hidden_dim=768, proj_dim=256):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.mel_channels = mel_channels
        self.frames = frames
        self.proj_dims = proj_dim

        weight = torch.tensor([10])
        bias = torch.tensor([-5])
        self.w = nn.Parameter(weight)
        self.b = nn.Parameter(bias)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm1 = nn.LSTM(mel_channels, hidden_dim, batch_first=False)
        print("here1")
        self.lstm2 = nn.LSTM(proj_dim, hidden_dim, batch_first=False)
        self.lstm3 = nn.LSTM(proj_dim, hidden_dim, batch_first=False)
        self.lstms = [self.lstm1, self.lstm2, self.lstm3]

        self.proj1 = nn.Linear(hidden_dim, proj_dim)
        self.proj2 = nn.Linear(hidden_dim, proj_dim)
        self.proj3 = nn.Linear(hidden_dim, proj_dim)
        self.projs = [self.proj1, self.proj2, self.proj3]

    def init_states(self, batchsize):
        # Before we've done anything, we dont have any hidden state.
        # Refer to the Pytorch documentation to see exactly
        # why they have this dimensionality.
        # The axes semantics are (num_layers, minibatch_size, hidden_dim)
        return [(torch.zeros(1, batchsize, self.hidden_dim),
                 torch.zeros(1, batchsize, self.hidden_dim)),
                (torch.zeros(1, batchsize, self.hidden_dim),
                 torch.zeros(1, batchsize, self.hidden_dim)),
                (torch.zeros(1, batchsize, self.hidden_dim),
                 torch.zeros(1, batchsize, self.hidden_dim)),
                ]

    def forward(self, inputs, states=None):
        time, batchsize, inputdim = list(inputs.shape)

        if states is None:
            states = self.init_states(batchsize)
        output = inputs

        print(output.type())

        for i in range(3):
            print(output.type())
            output, state = self.lstms[i](output, states[i])
            output = self.projs[i](output)
        # perform normalization on this output here
        output = output[-1]
        print(output.type())
        output = F.normalize(output, p=2, dim=-1)
        print(output.type())

        self.state = state

        print(output.type())

        return output

    def get_w(self):
        print(get_w.type())
        return(self.w)

    def get_b(self):
        print(get_b.type())
        return(self.b)

    def get_state(self):
        print(get_state())
        return(self.state)

This is the custom loss:

class CustomLoss(_Loss):

    def __init__(self,  size_average=True, reduce=True):

        super(CustomLoss, self).__init__(size_average, reduce)

    def forward(self, S, N, M, type='softmax',):

        return self.loss_cal(S, N, M, type)

    def loss_cal(self, S, N, M, type="softmax",):

        self.A = torch.cat([S[i * M:(i + 1) * M, i:(i + 1)]
                           for i in range(N)], dim=0)

        if type == "softmax":
            self.B = torch.log(torch.sum(torch.exp(S.float()), dim=1, keepdim=True) + 1e-8)
            total = torch.abs(torch.sum(self.A - self.B))
        else:
            raise AssertionError("loss type should be softmax or contrast !")
        return total

Finally, this is the main file

model=LSTM()
optimizer = optim.Adam(list(model.parameters()), lr=LEARNING_RATE)
model = model.to(device)

best_loss = 100.
generator = SpeakerVerificationDataset()
dataloader = DataLoader(generator, batch_size=4,
                        shuffle=True, num_workers=0)

loss_history = []
update_counter = 1

for epoch in range(NUM_EPOCHS):
    print("Epoch # : ", epoch + 1)

    for step in range(STEPS_PER_EPOCH):
        # get batch dataset
        for i_batch, sample_batched in enumerate(dataloader):
            print(sample_batched['MelData'].size())
            inputs = sample_batched['MelData'].float()
            inputs=sample_batched['MelData'].view(180, M*N, 40).float()
            print((inputs.size()))
            inputs = inputs
            #print(here)
            # remove previous gradients
            optimizer.zero_grad()

            # get gradients and loss at this iteration
            #predictions,state,w,b = model(inputs)
            predictions = model(inputs)
            w = model.w
            b = model.b
            predictions = similarity(output=predictions,w=w,b=b)
            #loss = CustomLoss()
            S = predictions

            loss_func = CustomLoss()
            loss = loss_func.loss_cal(S=S,N=N,M=M)
            loss.backward()

            # update the weights
            print("start optimizing")
            optimizer.step()

            loss_history.append(loss.item())
            print(update_counter, ":", loss_history[-1])

            update_counter += 1
    print()

    # save the weights
    torch.save(model.state_dict(), CHECKPOINT_PATH)

    print("Saving weights")
    print()
print()

Output:
The error comes from here:

weight = torch.tensor([10])
bias = torch.tensor([-5])
self.w = nn.Parameter(weight)
self.b = nn.Parameter(bias)

Had to change it to

weight = torch.tensor([10.0])
bias = torch.tensor([-5.0])
self.w = nn.Parameter(weight)
self.b = nn.Parameter(bias)


Instruction: tensorflow stop_gradient equivalent in pytorch

Input: What is the tf.stop_gradient() equivalent (provides a way to not compute gradient with respect to some variables during back-propagation) in pytorch?

Output: Could you check with x.detach().


Instruction: Can we deploy Pytorch models to Google Cloud ML Engine? If so, how?

Input:
I searched if anybody has. Somebody has provided a work around but couldn't confirm its working. Here are the links -

How to setup pytorch in google-cloud-ml

https://discuss.pytorch.org/t/pytorch-in-google-cloud-ml-engine/12818

Currently, I am using Floydhub, but it is good to start with. I work mostly with Google Cloud products, so want to keep my deep learning projects as close to it as possible.

Output:
The answer you referred to will likely work (I've also not fully tested it). The Cloud ML Engine team is working hard to simplify this.

In the meantime, if you prefer working with a VM (just remember to shut it off when not in use), follow instructions on this page. For example,

export IMAGE_FAMILY="pytorch-latest-cu91"
export ZONE="us-west1-b"
export INSTANCE_NAME="my-instance"

gcloud compute instances create $INSTANCE_NAME \
  --zone=$ZONE \
  --image-family=$IMAGE_FAMILY \
  --image-project=deeplearning-platform-release \
  --maintenance-policy=TERMINATE \
  --accelerator='type=nvidia-tesla-v100,count=1' \
  --metadata='install-nvidia-driver=True'


Instruction: How to Invert AvgPool2d?

Input:
Is it possible to invert an avgPool2d operation in PyTorch, like maxunpool2d for a maxpool2d operation, and if so, how could that be done?

I've already checked the documentation, and there isn't an option to return the indices, like in the maxpool2d operation, so I assume the unpooling won't be possible in a similar way.

EDIT:
I found a document by Intel which describes how the unpooling works. After checking the math regarding the avgpool2d function the unpooling seems to be pretty straight forward, basically mirroring every input element onto multiple output elements, and apply padding in order to get a correct output size.

Output:
I think you are looking for ConvTransposed2d, aka deconvolution: This function allows you to "upsample" the pooled layer.
Using fixed weights you can replicate the averged pooled values. You can also train this layer hopefully getting something better.


Instruction: Pytorch how to get the gradient of loss function twice

Input:
Here is what I'm trying to implement:

We calculate loss based on F(X), as usual. But we also define "adversarial loss" which is a loss based on F(X + e). e is defined as dF(X)/dX multiplied by some constant. Both loss and adversarial loss are backpropagated for the total loss.

In tensorflow, this part (getting dF(X)/dX) can be coded like below:

  grad, = tf.gradients( loss, X )
  grad = tf.stop_gradient(grad)
  e = constant * grad

Below is my pytorch code:

class DocReaderModel(object):
    def __init__(self, embedding=None, state_dict=None):
        self.train_loss = AverageMeter()
        self.embedding = embedding
        self.network = DNetwork(opt, embedding)
        self.optimizer = optim.SGD(parameters)

    def adversarial_loss(self, batch, loss, embedding, y):
        self.optimizer.zero_grad()
        loss.backward(retain_graph=True)
        grad = embedding.grad
        grad.detach_()

        perturb = F.normalize(grad, p=2)* 0.5
        self.optimizer.zero_grad()
        adv_embedding = embedding + perturb
        network_temp = DNetwork(self.opt, adv_embedding) # This is how to get F(X)
        network_temp.training = False
        network_temp.cuda()
        start, end, _ = network_temp(batch) # This is how to get F(X)
        del network_temp # I even deleted this instance.
        return F.cross_entropy(start, y[0]) + F.cross_entropy(end, y[1])

    def update(self, batch):
        self.network.train()
        start, end, pred = self.network(batch)
        loss = F.cross_entropy(start, y[0]) + F.cross_entropy(end, y[1])
        loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
        loss_total = loss + loss_adv

        self.optimizer.zero_grad()
        loss_total.backward()
        self.optimizer.step()

I have few questions:

1) I substituted tf.stop_gradient with grad.detach_(). Is this correct?

2) I was getting "RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time." so I added retain_graph=True at the loss.backward. That specific error went away.
However now I'm getting a memory error after few epochs (RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1525909934016/work/aten/src/THC/generic/THCStorage.cu:58
). I suspect I'm unnecessarily retaining graph.

Can someone let me know pytorch's best practice on this? Any hint / even short comment will be highly appreciated.

Output:
I think you are trying to implement generative adversarial network (GAN), but from the code, I don't understand and can't follow to what you are trying to achieve as there are a few missing pieces for a GAN to works. I can see there's a discriminator network module, DNetwork but missing the generator network module.

If to guess, when you say 'loss function twice', I assumed you mean you have one loss function for the discriminator net and another for the generator net. If that's the case, let me share how I would implement a basic GAN model.

As an example, let's take a look at this Wasserstein GAN Jupyter notebook

I'll skip the less important bits and zoom into the important ones here:

First, import PyTorch libraries and set up

# Set up batch size, image size, and size of noise vector:
bs, sz, nz = 64, 64, 100 # nz is the size of the latent z vector for creating some random noise later

Build a discriminator module

class DCGAN_D(nn.Module):
    def __init__(self):
        ... truncated, the usual neural nets stuffs, layers, etc ...
    def forward(self, input):
        ... truncated, the usual neural nets stuffs, layers, etc ...

Build a generator module

class DCGAN_G(nn.Module):
    def __init__(self):
        ... truncated, the usual neural nets stuffs, layers, etc ...
    def forward(self, input):
        ... truncated, the usual neural nets stuffs, layers, etc ...

Put them all together

netG = DCGAN_G().cuda()
netD = DCGAN_D().cuda()

Optimizer needs to be told what variables to optimize. A module automatically keeps track of its variables.

optimizerD = optim.RMSprop(netD.parameters(), lr = 1e-4)
optimizerG = optim.RMSprop(netG.parameters(), lr = 1e-4)

One forward step and one backward step for Discriminator

Here, the network can calculate gradient during the backward pass, depends on the input to this function. So, in my case, I have 3 type of losses; generator loss, dicriminator real image loss, dicriminator fake image loss. I can get gradient of loss function three times for 3 different net passes.

def step_D(input, init_grad):
    # input can be from generator's generated image data or input image from dataset
    err = netD(input)
    err.backward(init_grad) # backward pass net to calculate gradient
    return err # loss

Control trainable parameters [IMPORTANT]

Trainable parameters in the model are those that require gradients.

def make_trainable(net, val):
    for p in net.parameters():
        p.requires_grad = val # note, i.e, this is later set to False below in netG update in the train loop.

In TensorFlow, this part can be coded like below:

grad = tf.gradients(loss, X)
grad = tf.stop_gradient(grad)

So, I think this will answer your first question, "I substituted tf.stop_gradient with grad.detach_(). Is this correct?"
Train loop

You can see here how's the 3 different loss functions are being called here.

    def train(niter, first=True):

        for epoch in range(niter):
            # Make iterable from PyTorch DataLoader
            data_iter = iter(dataloader)
            i = 0

            while i < n:
                ###########################
                # (1) Update D network
                ###########################
                make_trainable(netD, True)

                # train the discriminator d_iters times
                d_iters = 100

                j = 0

                while j < d_iters and i < n:
                    j += 1
                    i += 1

                    # clamp parameters to a cube
                    for p in netD.parameters():
                        p.data.clamp_(-0.01, 0.01)

                    data = next(data_iter)

                    ##### train with real #####
                    real_cpu, _ = data
                    real_cpu = real_cpu.cuda()
                    real = Variable( data[0].cuda() )
                    netD.zero_grad()

                    # Real image discriminator loss
                    errD_real = step_D(real, one)

                    ##### train with fake #####
                    fake = netG(create_noise(real.size()[0]))
                    input.data.resize_(real.size()).copy_(fake.data)

                    # Fake image discriminator loss
                    errD_fake = step_D(input, mone)

                    # Discriminator loss
                    errD = errD_real - errD_fake
                    optimizerD.step()

                ###########################
                # (2) Update G network
                ###########################
                make_trainable(netD, False)
                netG.zero_grad()

                # Generator loss
                errG = step_D(netG(create_noise(bs)), one)
                optimizerG.step()

                print('[%d/%d][%d/%d] Loss_D: %f Loss_G: %f Loss_D_real: %f Loss_D_fake %f'
                    % (epoch, niter, i, n,
                    errD.data[0], errG.data[0], errD_real.data[0], errD_fake.data[0]))

  "I was getting "RuntimeError: Trying to backward through the graph a second time..."

PyTorch has this behaviour; to reduce GPU memory usage, during the .backward() call, all the intermediary results (if you have like saved activations, etc.) are deleted when they are not needed anymore. Therefore, if you try to call .backward() again, the intermediary results don't exist and the backward pass cannot be performed (and you get the error you see).

It depends on what you are trying to do. You can call .backward(retain_graph=True) to make a backward pass that will not delete intermediary results, and so you will be able to call .backward() again. All but the last call to backward should have the retain_graph=True option.

  Can someone let me know pytorch's best practice on this

As you can see from the PyTorch code above and from the way things are being done in PyTorch which is trying to stay Pythonic, you can get a sense of PyTorch's best practice there.


Instruction: Pytorch- why is “accumulating” the default mode of .gradient?

Input: Why didn't the authors just make it overwrite the gradient? Is there any specific reason for keeping it accumulated?

Output: Because if you use the same network twice (or same weights) in the forward pass, it should accumulate and not override. Also, since pytorch computation graph is defined by the run, so it makes sense to accumulate. See https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/9


Instruction: How to get backward gradients from sparse matrices in PyTorch networks?

Input:
I am following the PyTorch tutorial example:

https://pytorch.org/tutorials/beginner/pytorch_with_examples.html

The example runs without any problems, but as soon as I switched it to my own dataset (which is a sparse tensor, because it is too big to be used as a dense tensor), I run into this error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-127-8b4999644085> in <module>()
 41     # Backward pass: compute gradient of the loss with respect to model
 42     # parameters
---> 43     loss.backward()
 44
 45     # Calling the step function on an Optimizer makes an update to its

~/miniconda3/envs/py3/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
 91                 products. Defaults to ``False``.
 92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
 94
 95     def register_hook(self, hook):

~/miniconda3/envs/py3/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
 87     Variable._execution_engine.run_backward(
 88         tensors, grad_tensors, retain_graph, create_graph,
---> 89         allow_unreachable=True)  # allow_unreachable flag
 90
 91

RuntimeError: Expected object of type torch.FloatTensor but found type torch.sparse.FloatTensor for argument #2 'mat2'

I tried switching the optimizer (Adagrad, Adam), but it doesn't seem to work.

Edit: Added more of the error output. The error occurs on backward()

Output:
Looks like this functionality is currently being worked on now.

https://github.com/pytorch/pytorch/issues/2389


Instruction: Disparate result after setting requires_grad=True

Input:
I am currently using PyTorch for deep neural network. I wrote a toy neural network shown below and I found that whether or not I set requires_grad=True for label y makes huge difference. When y.requires_grad=True, the neural network diverges. I am wondering why this happens.

import torch

x = torch.unsqueeze(torch.linspace(-1, 1, 100), dim=1)
y = x.pow(2) + 10 * torch.rand(x.size())

x.requires_grad = True
# this is where problem occurs
y.requires_grad = True

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)
        self.predict = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = torch.relu(self.hidden(x))
        x = self.predict(x)
        return x

net = Net(1, 10, 1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.5)
criterion = torch.nn.MSELoss()

for t in range(200):
    y_pred = net(x)

    loss= criterion(y_pred, y)

    optimizer.zero_grad()
    loss.backward()
    print("Epoch {}: {}".format(t, loss))
    optimizer.step()

Output:
It seems that you are using an outdated version of PyTorch. In more recent versions (0.4.0+), this will throw you the following error:

AssertionError: nn criterions don't compute the gradient w.r.t. targets -
                please mark these tensors as not requiring gradients

Essentially, it tells you that it will only work if you set the requires_grad flag to False for your targets. The reason why this works at all in prior versions is indeed very interesting, and also why it causes diverging behavior.

My guess would be that a backwards pass would then also change your targets (instead of only changing your weights), which is obviously something you do not desire.


Instruction: Output of DecoderRNN contains extra dimensions (Pytorch)

Input:
I have developed an Encoder(CNN)-Decoder (RNN) network for image captioning in pytorch.  The decoder network takes in two inputs- Context feature vector from the Encoder and the word embeddings of the caption for training. The context feature vector is of size = embed_size , which is also the embedding size of each word in the caption. My question here is more concerned with the output of the Class DecoderRNN. Please refer to the code below.

class DecoderRNN(nn.Module):
def __init__(self, embed_size, hidden_size, vocab_size, num_layers=1):
    super(DecoderRNN, self).__init__()
    self.embed_size = embed_size
    self.hidden_size = hidden_size
    self.vocab_size = vocab_size
    self.num_layers = num_layers
    self.linear = nn.Linear(hidden_size, vocab_size)
    self.embed = nn.Embedding(vocab_size, embed_size)
    self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first = True)

def forward(self, features, captions):
    embeddings = self.embed(captions)
    embeddings = torch.cat((features.unsqueeze(1), embeddings),1)
    hiddens,_ = self.lstm(embeddings)
    outputs = self.linear(hiddens)
    return outputs

In the forward function , I send in a sequence of (batch_size, caption_length+1, embed_size) (concatenated tensor of context feature vector and the embedded caption) . The output of the sequence should be captions and  of the shape (batch_size, caption_length, vocab_size) , but I am still receiving an output of shape (batch_size, caption_length +1, vocab_size) .  Can anyone please suggest what should I alter in my forward function so that extra 2nd dimension is not received? Thanks in advance

Output: Since in LSTM (or in any RNN) for each time step (or caption length here), there will be one output, I do not see any problem here. What you need to do is make input size (caption_length) at the second dimension to get the required output. (or people usually add a < END of SENTENCE > tag to target. Hence the target length is caption+1)


Instruction: How to use PyTorch to calculate the gradients of outputs w.r.t. the inputs in a neural network?

Input:
I have a trained network. And I want to calculate the gradients of outputs w.r.t. the inputs. By querying the PyTorch Docs, torch.autograd.grad may be useful.
So, I use the following code:

    x_test = torch.randn(D_in,requires_grad=True)
    y_test = model(x_test)
    d = torch.autograd.grad(y_test, x_test)[0]

model is the neural network. x_test is the input of size D_in and y_test is a scalar output.
I want to compare the calculated result with numerical difference by scipy.misc.derivative.
So, I calculated the partial derivate by setting a index.

    idx = 3
    x_test = torch.randn(D_in,requires_grad=True)
    y_test = model(x_test)
    print(x_test[idx].item())
    d = torch.autograd.grad(y_test, x_test)[0]
    print(d[idx].item())
    def fun(x):
        x_input = x_test.detach()
        x_input[idx] = x
        with torch.no_grad():
            y = model(x_input)
        return y.item()
    x0 = x_test[idx].item()
    print(x0)
    print(derivative(fun, x0, dx=1e-6))

But I got totally different results.
The gradient calculated by torch.autograd.grad is -0.009522666223347187,
while that by scipy.misc.derivative is -0.014901161193847656.

Is there anything wrong about the calculation? Or I use torch.autograd.grad wrongly?

Output:
In fact, it is very likely that your given code is completely correct.
Let me explain this by redirecting you to a little background information on backpropagation, or rather in this case Automatic Differentiation (AutoDiff).

The specific implementation of many packages is based on AutoGrad, a common technique to get the exact derivatives of a function/graph. It can do this by essentially "inverting" the forward computational pass to compute piece-wise derivatives of atomic function blocks, like addition, subtraction, multiplication, division, etc., and then "chaining them together".
I explained AutoDiff and its specifics in a more detailed answer in this question.

On the contrary, scipy's derivative function is only an approximation to this derivative by using finite differences. You would take the results of the function at close-by points, and then calculate a derivative based on the difference in function values for those points. This is why you see a slight difference in the two gradients, since this can be an inaccurate representation of the actual derivative.


Instruction: Pytorch embedding RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

Input:
I'm getting this error saying

RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

But what does it mean by argument #3 "index"? I can't find "index" argument in torch.embedding (here source : https://pytorch.org/docs/stable/_modules/torch/nn/modules/sparse.html#Embedding)
It seems like I'm passing the embedding the wrong parameters.

I even changed the data type of my input like below but the error persists.

batch['doc_tok'] = batch['doc_tok'].long()
batch['query_tok'] = batch['query_tok'].long()

Any comment (even though it's short!) or just listing keywords to look at will be highly appreciated!

Here is a full traceback.

Traceback (most recent call last):
  File "train_v2.py", line 110, in <module>
    main()
  File "train_v2.py", line 81, in main
    model.update(batch)
  File "/home/aerin/Desktop/squad_vteam/src/model.py", line 129, in update
    loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
  File "/home/aerin/Desktop/squad_vteam/src/model.py", line 104, in adversarial_loss
    start, end, _ = self.network(batch)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/Desktop/squad_vteam/src/dreader.py", line 78, in forward
    doc_mask, query_mask = self.lexicon_encoder(batch)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/Desktop/squad_vteam/src/encoder.py", line 116, in forward
    doc_emb, query_emb = emb(doc_tok), emb(query_tok)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 108, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/aerin/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/functional.py", line 1076, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.LongTensor but found type torch.cuda.LongTensor for argument #3 'index'

Update: I even sent the whole model.network to cpu but still getting the same error.

batch['doc_tok']=batch['doc_tok'].long().cpu()
batch['query_tok']=batch['query_tok'].long().cpu()
self.network.cpu()

print(batch['doc_tok'].dtype,  batch['query_tok'].dtype) # They are both torch.int64 torch.int64

start, end, _ = self.network(batch)

At this point, I'm suspecting this might be a bug...

model.py code: https://github.com/byorxyz/san_mrc/blob/master/src/model.py

Network defined: https://github.com/byorxyz/san_mrc/blob/master/src/dreader.py

Output:
It seems to me like your input/target tensors (batch['doc_tok'], etc.) and your network and its variables (I believe index is an internal tensor for Embedding layers) are on different devices (CPU for your data, GPU for your model?).

If you want everything to run on GPU, you need both to:

Load your data there,  e.g. batch['doc_tok'].cuda()
Load your model there, e.g. model.network.cuda()

Same goes if you want to run on CPU, replacing with .cpu().


Instruction: import torch error in google Colaboratory

Input:
I get the following error on google colaboratory.

  ImportErrorTraceback (most recent call last)
   in ()
  ----> 1 import torch

  /usr/local/lib/python3.6/site-packages/torch/init.py in ()
       54 except ImportError:
       55     pass
  ---> 56 from torch._C import *
       57
       58 all += [name for name in dir(_C)

  ImportError: No module named _C

I tried importing from a different directory. But, still get the same error. Any help?

Output: Try restarting runtime. Colab sometimes gives errors like this :P


Instruction: How to release temporarily consumed GPU memory after each forward?

Input:
I have a class like this:

class Stem(nn.Module):

    def __init__(self):
        super(Stem, self).__init__()
        self.out_1 = BasicConv2D(3, 32, kernelSize = 3, stride = 2)
        self.out_2 = BasicConv2D(32, 32, kernelSize = 3, stride = 1)
        self.out_3 = BasicConv2D(32, 64, kernelSize = 3, stride = 1, padding = 1)

    def forward(self, x):
        x = self.out_1(x)
        x = self.out_2(x)
        x = self.out_3(x)

        return x

and the attributes out_1,2,3 of Stem are instances of below class:

class BasicConv2D(nn.Module):

    def __init__(self, inChannels, outChannels, kernelSize, stride, padding = 0):
        super(BasicConv2D, self).__init__()
        self.conv = nn.Conv2d(inChannels, outChannels,
                            kernel_size = kernelSize,
                            stride = stride,
                            padding = padding, bias = False)
        self.bn = nn.BatchNorm2d(outChannels,
                                    eps = 0.001,
                                    momentum = 0.1,
                                    affine = True)
        self.relu = nn.ReLU(inplace = False)

    def forward(self, x):
        x = self.conv(x)
        x = self.bn(x)
        y = self.relu(x)
        return y

when training, within Stem.forward(), nvidia-smi tells that each line will consume x MBs GPU memory, but after Stem.forward() finished, the memory won't be released, causing the training to quickly crashed, out of GPU memory.

The question is thus:
how to release the temporarily consumed GPU memory?

Output:
Your model does look good, so you might want to have a general look at how pytorch manages memory allocation. I suspect that you simply keep pointers to your return value (y) alive (e.g. by accumulating a loss or some such). As pytorch stores the entire attached computation graph, you never free the memory.

See this question and in particular this answer for a more detailed discussion.


Instruction: Pytorch installation issue under Anaconda

Input:
I followed the link here to install fastai library using pip install git+https://github.com/fastai/fastai.git

It gave me the following error message. These messages keep the same even I installed Pytorch successfully  using  conda install pytorch-cpu -c pytorch
and pip3 install torchvision. What can be the reason?

Collecting torch<0.4 (from fastai==0.7.0)
  Using cached https://files.pythonhosted.org/packages/5f/e9/bac4204fe9cb1a002ec6140b47f51affda1655379fe302a1caef421f9846/torch-0.1.2.post1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\shuxi\AppData\Local\Temp\pip-install-7sjptuad\torch\setup.py", line 11, in <module>
        raise RuntimeError(README)
    RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

    Please follow the instructions at http://pytorch.org/ to install with miniconda instead.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\shuxi\AppData\Local\Temp\pip-install-7sjptuad\torch\

Output:
To fix this, do:

$ pip install --upgrade git+https://github.com/fastai/fastai.git

OR

$ pip install --no-cache-dir git+https://github.com/fastai/fastai.git

Your command probably failed because you have installed a old version of torch (0.1.2) some time ago. pip was not supported for torch install for that version and pip instead redirected the user to open pytorch.org in the browser. In your case, pip is reusing this cached package. --upgrade forces pip to choose latest version of all depending packages.


Instruction: pytorch Rnn.py RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

Input:
I'm getting an CUDNN_STATUS_INTERNAL_ERROR error like below.

python train_v2.py

Traceback (most recent call last):
  File "train_v2.py", line 113, in <module>
    main()
  File "train_v2.py", line 74, in main
    model.cuda()
  File "/home/ahkim/Desktop/squad_vteam/src/model.py", line 234, in cuda
    self.network.cuda()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/ahkim/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 105, in flatten_parameters
    self.batch_first, bool(self.bidirectional))
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

What should I try to resolve this issue?
I tried deleting .nv but no success.

nvidia-smi

Wed Aug  8 10:56:29 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.67                 Driver Version: 390.67                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 00000000:04:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |    125MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 00000000:05:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 00000000:08:00.0 Off |                  N/A |
| 22%   23C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 00000000:09:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX TIT...  Off  | 00000000:85:00.0 Off |                  N/A |
| 22%   24C    P8    14W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX TIT...  Off  | 00000000:86:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX TIT...  Off  | 00000000:89:00.0 Off |                  N/A |
| 22%   21C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX TIT...  Off  | 00000000:8A:00.0 Off |                  N/A |
| 22%   23C    P8    15W / 250W |     11MiB / 12212MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1603      C   /usr/bin/python                              114MiB |
+-----------------------------------------------------------------------------+

Update:

The same code runs without error using Nvidia
Driver Version: 396.26 (cuda V9.1.85. torch.backends.cudnn.version():
7102). I'm getting an error using Driver Version: 390.67 (cuda V9.1.85. torch.backends.cudnn.version(): 7102)

Output:
solved by below steps.

export LD_LIBRARY_PATH= "/usr/local/cuda-9.1/lib64"
Due to nfs issue, have pytoch cache not in nfs. For example:

$ rm ~/.nv -rf

$ mkdir -p /tmp/$USER/.nv

$ ln -s /tmp/$USER/.nv ~/.nv


Instruction: Change dimensions of image when creating custom dataloader in pytorch

Input:
I was training my CNN on the CIFAR10 dataset. I extracted the 50,000 images of dimensions (32 x 32 x 3) and read them in a list. I converted them into numpy arrays and stored them in a list.  I did the same with the labels for my training and testing.

I then constructed my CNN of two layers and a single FC in pytorch. Before doing this I created my own custom data loader. While doing so, the dimensions of the image that I am feeding in it are changing. The dimensions (32 x 32 x 3) are changing to (3 x 32 x 32) and I am not being able to train my neural network.

tensor_x = torch.stack([torch.Tensor(i) for i in train_images])
tensor_y = torch.stack([torch.Tensor(i) for i in train_labels])
dataset = data_utils.TensorDataset(tensor_x , tensor_y)
train_dataloader = data_utils.DataLoader(dataset=dataset)

tensor_x = torch.stack([torch.Tensor(i) for i in test_images])
tensor_y = torch.stack([torch.Tensor(i) for i in test_labels])
dataset = data_utils.TensorDataset(tensor_x , tensor_y)
test_dataloader = data_utils.DataLoader(dataset=dataset)

RuntimeError: Given groups=1, weight[64, 3, 3, 3], so expected input[1, 32, 32, 3] to have 3 channels, but got 32 channels instead

Output:
In PyTorch image, the channels come first, so your image should be 3, 32, 32 and not 32, 32, 3.

If image is a numpy array, then you can do something like this

image = image.transpose((2, 0, 1))


Instruction: How to convert FloatTensor to ByteTensor with Pytorch?

Input:
I'm new to Pytorch and neural network programming but I've an issue I encountered and I'm not able to solve it on my own. My data are numpy arrays of 1 and 0. But when I try to train my net, I get this error :

  RuntimeError: Expected object of type torch.ByteTensor but found type torch.FloatTensor for argument #2 'mat2'

the line where the error comes from is in the forward method of my net

x = self.fc1(x)

I've tried these to convert my tensors but I still get the error :

  x = x.type('torch.ByteTensor')

and

  x.byte()

Output:
x.byte() returns what you need, but it's not an "inplace" method. Try doing:

x = x.byte()


Instruction: How to use different data augmentation for Subsets in PyTorch

Input:
How to use different data augmentation (transforms) for different Subsets in PyTorch?

For instance:

train, test = torch.utils.data.random_split(dataset, [80000, 2000])

train and test will have the same transforms as dataset. How to use custom transforms for these subsets?

Output:
My current solution is not very elegant, but works:

from copy import copy

train_dataset, test_dataset = random_split(full_dataset, [train_size, test_size])
train_dataset.dataset = copy(full_dataset)

test_dataset.dataset.transform = transforms.Compose([
    transforms.Resize(img_resolution),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

train_dataset.dataset.transform = transforms.Compose([
    transforms.RandomResizedCrop(img_resolution[0]),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])

Basically, I'm defining a new dataset (which is a copy of the original dataset) for one of the splits, and then I define a custom transform for each split.

Note: train_dataset.dataset.transform works since I'm using an ImageFolder dataset, which uses the .tranform attribute to perform the transforms.

If anybody knows a better solution, please share with us!


Instruction: How do I use a saved model in Pytorch to predict the label of a never before seen image?

Input:
I have been trying to use my pretrained model to predict the label on a never before seen image. I have trained a CNN to classify flowers of 5 types using the Kaggle flower recognition dataset. I so far have trained my model to 97% accuracy and saved the model to a directory. I now want to download any image of a flower from those types and be able to use this pretrained model to predict the label. So far this is my code: (A code review for all of this would be very helpful as this is my first ever project)

This is my CNN model that I train:

from multiprocessing import freeze_support

import torch
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np

# Hyperparameters.
num_epochs = 20
num_classes = 5
batch_size = 100
learning_rate = 0.001
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train\\'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test\\'
MODEL_STORE_PATH = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\model'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

# Flowers dataset.
train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
test_dataset = datasets.ImageFolder(root=DATA_PATH_TEST, transform=trans)

# Create custom random sampler class to iter over dataloader.
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False, num_workers=num_of_workers)

# CNN we are going to implement.
class Unit(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(Unit, self).__init__()

        self.conv = nn.Conv2d(in_channels=in_channels, kernel_size=3, out_channels=out_channels, stride=1, padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()

    def forward(self, input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)

        return output

class CNNet(nn.Module):
    def __init__(self, num_class):
        super(CNNet, self).__init__()

        # Create 14 layers of the unit with max pooling in between
        self.unit1 = Unit(in_channels=3, out_channels=32)
        self.unit2 = Unit(in_channels=32, out_channels=32)
        self.unit3 = Unit(in_channels=32, out_channels=32)

        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.unit4 = Unit(in_channels=32, out_channels=64)
        self.unit5 = Unit(in_channels=64, out_channels=64)
        self.unit6 = Unit(in_channels=64, out_channels=64)
        self.unit7 = Unit(in_channels=64, out_channels=64)

        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.unit8 = Unit(in_channels=64, out_channels=128)
        self.unit9 = Unit(in_channels=128, out_channels=128)
        self.unit10 = Unit(in_channels=128, out_channels=128)
        self.unit11 = Unit(in_channels=128, out_channels=128)

        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.unit12 = Unit(in_channels=128, out_channels=128)
        self.unit13 = Unit(in_channels=128, out_channels=128)
        self.unit14 = Unit(in_channels=128, out_channels=128)

        self.avgpool = nn.AvgPool2d(kernel_size=4)

        # Add all the units into the Sequential layer in exact order
        self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4, self.unit5, self.unit6
                                 , self.unit7, self.pool2, self.unit8, self.unit9, self.unit10, self.unit11, self.pool3,
                                 self.unit12, self.unit13, self.unit14, self.avgpool)

        self.fc = nn.Linear(in_features=128, out_features=num_class)

    def forward(self, input):
        output = self.net(input)
        output = output.view(-1, 128)
        output = self.fc(output)
        return output

# Check if gpu support is available
cuda_avail = torch.cuda.is_available()

# Create model, optimizer and loss function
model = CNNet(num_classes)

# if cuda is available, move the model to the GPU
if cuda_avail:
    model.cuda()

# Define the optimizer and loss function
optimizer = Adam(model.parameters(), lr=0.0001, weight_decay=0.0001)
loss_fn = nn.CrossEntropyLoss()

def save_models(epoch):
    torch.save(model.state_dict(), f"flowermodel_{epoch}.model")
    print("Checkpoint saved")

def test():
    model.eval()
    test_acc = 0.0
    for i, (images, labels) in enumerate(test_loader):

        if cuda_avail:
            images = Variable(images.cuda())
            labels = Variable(labels.cuda())

        # Predict classes using images from the test set
        outputs = model(images)
        _, prediction = torch.max(outputs.data, 1)

        test_acc += torch.sum(prediction == labels.data).float()

    # Compute the average acc and loss over all 10000 test images
    test_acc = test_acc / 4242 * 100

    return test_acc

def train(num_epoch):
    best_acc = 0.0

    for epoch in range(num_epoch):
        model.train()
        train_acc = 0.0
        train_loss = 0.0
        for i, (images, labels) in enumerate(train_loader):
            # Move images and labels to gpu if available
            if cuda_avail:
                images = Variable(images.cuda())
                labels = Variable(labels.cuda())

            # Clear all accumulated gradients
            optimizer.zero_grad()
            # Predict classes using images from the test set
            outputs = model(images)
            # Compute the loss based on the predictions and actual labels
            loss = loss_fn(outputs, labels)
            # Backpropagate the loss
            loss.backward()

            # Adjust parameters according to the computed gradients
            optimizer.step()

            train_loss += loss.cpu().data[0] * images.size(0)
            _, prediction = torch.max(outputs.data, 1)

            train_acc += torch.sum(prediction == labels.data).float()

        # Call the learning rate adjustment function
        #adjust_learning_rate(epoch)

        # Compute the average acc and loss over all 50000 training images
        train_acc = train_acc / 4242 * 100
        train_loss = train_loss / 8484

        # Evaluate on the test set
        test_acc = test()

        # Save the model if the test acc is greater than our current best
        if test_acc > best_acc:
            save_models(epoch)
            best_acc = test_acc

        # Print the metrics
        print(f"Epoch {epoch + 1}, Train Accuracy: {train_acc} , TrainLoss: {train_loss} , Test Accuracy: {test_acc}")

if __name__ == '__main__':
    freeze_support()
    train(num_epochs)

This is my image loader to view preprocessed images:

from multiprocessing import freeze_support

import torch
from torch import nn
import torchvision
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np
import PIL

num_classes = 5
batch_size = 100
num_of_workers = 5

DATA_PATH_TRAIN = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\train'
DATA_PATH_TEST = 'C:\\Users\Aeryes\PycharmProjects\simplecnn\images\\test'

trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

train_dataset = datasets.ImageFolder(root=DATA_PATH_TRAIN, transform=trans)
train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=num_of_workers)

def imshow(img):
    img = img / 2 + 0.5 # unnormalize
    #npimg = img.numpy()
    plt.imshow(np.transpose(img[0].numpy(), (1, 2, 0)))
    plt.show()

def main():
    # get some random training images
    dataiter = iter(train_loader)
    images, labels = dataiter.next()

    # show images
    imshow(images)

if __name__ == "__main__":
    main()

So far this is what I have for my classify new image file which will classify new images:

from multiprocessing import freeze_support

import torch
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader, Sampler
from torchvision import datasets
from torchvision.transforms import transforms
from torch.optim import Adam

import matplotlib.pyplot as plt
import numpy as np

def classify_new_image():
    # Classify a new image using a pretrained model from the above training.

    # Location of the image we will classify.
    IMG_PATH = "C:\\Users\\Aeryes\\PycharmProjects\\simplecnn\\images\\pretrain_classify\\"

    # Pre-processing the new image using transform.
    min_img_size = 32
    trans = transforms.Compose([transforms.Resize(min_img_size),
                                         transforms.CenterCrop(32),
                                         transforms.ToTensor(),
                                         transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                              std=[0.229, 0.224, 0.225])])

# Picture dataset.
classify_dataset = datasets.ImageFolder(root=IMG_PATH, transform=trans)
# Create custom random sampler class to iter over dataloader.
classify_loader = DataLoader(dataset=classify_dataset, batch_size=1, shuffle=True, num_workers=5)

# Check if gpu support is available
cuda_avail = torch.cuda.is_available()

model = torch.load('C:\\Users\\Aeryes\\PycharmProjects\\simplecnn\\src\\flowermodel_20.tar')['state_dict']

# if cuda is available, move the model to the GPU
if cuda_avail:
    model.cuda()

if __name__ == "__main__":
    classify_new_image()

A big issue that I am also facing is making sense of the outputs. I printed the prediction variable from the CNN models and it gave me a tensor of numbers ranging from 0-4, which I presume is the 5 classes I have in my data folders. If anyone can help me make sense of this a I would be very grateful.

Here is my direct question: How do I use my pre-trained model to predict never before seen images of flowers?

Output:
You can do like this for a single image,

import torch
from torchvision.transforms import transforms
from PIL import Image
from cnn_main import CNNet
from pathlib import Path

model = CNNet(5)
checkpoint = torch.load(Path('C:/Users/Aeryes/PycharmProjects/simplecnn/src/19.model'))
model.load_state_dict(checkpoint)
trans = transforms.Compose([
    transforms.RandomHorizontalFlip(),
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5),(0.5, 0.5, 0.5))
    ])

image = Image.open(Path('C:/Users/Aeryes/PycharmProjects/simplecnn/images/pretrain_classify/rose_classify.jpg'))

input = trans(image)

input = input.view(1, 3, 32,32)

output = model(input)

prediction = int(torch.max(output.data, 1)[1].numpy())
print(prediction)

if (prediction == 0):
    print ('daisy')
if (prediction == 1):
    print ('dandelion')
if (prediction == 2):
    print ('rose')
if (prediction == 3):
    print ('sunflower')
if (prediction == 4):
    print ('tulip')


Instruction: load a pretrained model pytorch - dict object has no attribute eval

Input:
def save_checkpoint(state, is_best, filename='checkpoint.pth.tar'):
    torch.save(state, filename)
    if is_best:
        shutil.copyfile(filename, 'model_best.pth.tar')

save_checkpoint({
                'epoch': epoch + 1,
                'arch': args.arch,
                'state_dict': model.state_dict(),
                'best_prec1': best_prec1,
                'optimizer': optimizer.state_dict()
            }, is_best)

I am saving my model like this. How can I load back the model so that I can use it in other places, like cnn visualization?

This is how I am loading the model now:

torch.load('model_best.pth.tar')

But when I do this, I get this error:

  AttributeError: 'dict' object has no attribute 'eval'

What am I missing here???

EDIT: I want to use the model that I trained to visualize the filters and grads. I am using this repo to make the vis. I replaced line 179 with torch.load('model_best.pth.tar')

Output:
First, you have stated your model.
And torch.load() gives you a dictionary. That dictionary has not an eval function. So you should upload the weights to your model.

import torch
from modelfolder import yourmodel

model = yourmodel()
checkpoint = torch.load('model_best.pth.tar')
try:
    checkpoint.eval()
except AttributeError as error:
    print error
### 'dict' object has no attribute 'eval'

model.load_state_dict(checkpoint['state_dict'])
### now you can evaluate it
model.eval()


Instruction: In-place operations with PyTorch

Input:
I was wondering how to deal with in-place operations in PyTorch. As I remember using in-place operation with autograd has always been problematic.

And actually I’m surprised that this code below works, even though I haven’t tested it I believe this code would have raised an error in version 0.3.1.

Basically I want do is set a certain position of a tensor vector to a certain value in a like:

my_tensor[i] = 42

Working example code:

# test parameter a
a = torch.rand((2), requires_grad=True)
print('a ', a)
b = torch.rand(2)

# calculation
c = a + b

# performing in-place operation
c[0] = 0
print('c ', c)
s = torch.sum(c)
print('s ', s)

# calling backward()
s.backward()

# optimizer step
optim = torch.optim.Adam(params=[a], lr=0.5)
optim.step()

# changed parameter a
print('changed a', a)

Output:

a  tensor([0.2441, 0.2589], requires_grad=True)
c  tensor([0.0000, 1.1511], grad_fn=<CopySlices>)
s  tensor(1.1511, grad_fn=<SumBackward0>)
changed a tensor([ 0.2441, -0.2411], requires_grad=True)

So obviously in version 0.4.1. this works just fine without warnings or errors.

Referring to this article in the documentation: autograd-mechanics

  Supporting in-place operations in autograd is a hard matter, and we
  discourage their use in most cases. Autograd’s aggressive buffer
  freeing and reuse makes it very efficient and there are very few
  occasions when in-place operations actually lower memory usage by any
  significant amount. Unless you’re operating under heavy memory
  pressure, you might never need to use them.

But even though it works, the use of in-place operations is discouraged in most cases.

So my questions are:

How much does the usage of in-place operations affect performance?
How do I get around using in-place operations in such cases where I want to set one element of a tensor to a certain value?

Thanks in advance!

Output:
I am not sure about how much in-place operation affect performance but I can address the second query. You can use a mask instead of in-place ops.

a = torch.rand((2), requires_grad=True)
print('a ', a)
b = torch.rand(2)

# calculation
c = a + b

# performing in-place operation
mask = np.zeros(2)
mask[1] =1
mask = torch.tensor(mask)
c = c*mask
...


Instruction: How can I load and use a PyTorch (.pth.tar) model

Input:
I am not very familiar with Torch, and I primarily use Tensorflow. I, however, need to use a retrained inception model that was retrained in Torch. Due to the large amount of computing resources required to retrain an inception model for my particular application, I would like to use the model that was already retrained.
This model is saved as a .pth.tar file.
I would like to be able to first load this model. So far, I have been able to figure out that I must use the following:
model = torch.load('iNat_2018_InceptionV3.pth.tar', map_location='cpu')

This seems to work, because print(model) prints out a large set of numbers and other values, which I presume are the values for the weights an biases.
After this, I need to be able to classify an image with it. I haven't been able to figure this out. How must I format the image? Should the image be converted into an array? After this, how must I pass the input data to the network?

Output:
you basically need to do the same as in tensorflow. That is, when you store a network, only the parameters (i.e. the trainable objects in your network) will be stored, but not the "glue", that is all the logic you need to use a trained model.
So if you have a .pth.tar file, you can load it, thereby overriding the parameter values of a model already defined.

That means that the general procedure of saving/loading a model is as follows:

write your network definition (i.e. your nn.Module object)
train or otherwise change the network's parameters in a way you want
save the parameters using torch.save
when you want to use that network, use the same definition of an nn.Module object to first instantiate a pytorch network
then override the values of the network's parameters using torch.load

Here's a discussion with some references on how to do this: pytorch forums

And here's a super short mwe:

# to store
torch.save({
    'state_dict': model.state_dict(),
    'optimizer' : optimizer.state_dict(),
}, 'filename.pth.tar')

# to load
checkpoint = torch.load('filename.pth.tar')
model.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])


Instruction: How to specify os-specific wheel files in the Pipfile

Input:
I'm trying to use the pipenv and pytorch.

To install pytorch in windows, I have to write following codes into the Pipfile:

[packages]
torch = {file = "http://download.pytorch.org/whl/cpu/torch-0.4.1-cp37-cp37m-win_amd64.whl"}

However, the wheel file is different for the linux.

[packages]
torch = {file = "http://download.pytorch.org/whl/cpu/torch-0.4.1.post2-cp37-cp37m-linux_x86_64.whl "}

How to specify both of them in Pipfile?

Output:
This article demonstrates how to structure your Pipfile to use pytorch across multiple platforms. I tweaked their example to deal with whl files on my local file system:

[packages]
pyfoo = {path = "./../pyfoo/dist/pyfoo-1.1.0-cp37-cp37m-linux_x86_64.whl", platform_system = "== 'Linux'"}
pyfoo-win = {path = "./../pyfoo/dist/pyfoo-1.1.0-py3-none-any.whl", platform_system = "== 'Windows'"}

In this case, pyfoo is an internal libarary that has been built using python setup.py build and python setup.py bdist_wheel.

The article uses some kind of pipenv generated hash in place of pyfoo-win. I could not coerce pipenv into generating that hash, so I created my own string. It may just be the prefix of the hash in the lock file.

Note: This approach is working for me, but there is a downer: both whl files need to be present when you install packages. pipenv sync bombs out if one of them is missing, even though it really only needs one of the two. Interestingly, the contents of the irrelevant file does not seem to matter. On my linux machine, I did an echo 'hello' > to the windows whl file, and pipenv was happy with it.


Instruction: Change Image format from NHWC to NCHW for Pytorch

Input:
In pytorch we need images in NCHW format but my images are NHWC.

What is the procedure to feed this image to CNN? (I have found this solution which suggests to use "permute" but where and how should i use it?)

Output:
Using torch.Tensor.permute():

 x = x.permute(0, 3, 1, 2) # from NHWC to NCHW


Instruction: Model modification in Pytorch

Input: I want to change weights in a certain kernels in a my saved CNN model. How can I change the values in specific kernels and save to a new model?

Output: You can torch.load the weights you saved. You should get a state_dict dictionary in which the weights are stored. Use the state_dict keys to locate the weights you wish to change, modify them and then torch.save the modified state_dict (better use different filename ;).


Instruction: How to use backpropagation with a self-defined loss in pytorch?

Input:
I am trying to implement a Siamese network with a ranking loss between two images. If I define my own loss would I be able to do the backpropagation step as follows? When I run it sometimes it seems to me that it is giving the same results that the single network gives.

with torch.set_grad_enabled(phase == 'train'):
    outputs1 = model(inputs1)
    outputs2 = model(inputs2)
    preds1 = outputs1;
    preds2 = outputs2;

    alpha = 0.02;
    w_r = torch.tensor(1).cuda(async=True);

    y_i, y_j, predy_i, predy_j = labels1,labels2,outputs1,outputs2;
    batchRankLoss =  torch.tensor([max(0,alpha - delta(y_i[i], y_j[i])*predy_i[i] - predy_j[i])) for i in range(batchSize)],dtype = torch.float)
    rankLossPrev = torch.mean(batchRankLoss)
    rankLoss = Variable(rankLossPrev,requires_grad=True)

    loss1 = criterion(outputs1, labels1)
    loss2 = criterion(outputs2, labels2)

    #total loss = loss1 + loss2 + w_r*rankLoss
    totalLoss = torch.add(loss1,loss2)
    w_r = w_r.type(torch.LongTensor)
    rankLossPrev = rankLossPrev.type(torch.LongTensor)
    mult = torch.mul(w_r.type(torch.LongTensor),rankLossPrev).type(torch.FloatTensor)
    totalLoss = torch.add(totalLoss,mult.cuda(async = True));

     # backward + optimize only if in training phase
         if phase == 'train':
            totalLoss.backward()
            optimizer.step()

            running_loss += totalLoss.item() * inputs1.size(0)

Output:
You have several lines where you generate new Tensors from a constructor or a cast to another data type. When you do this, you disconnect the chain of operations through which you'd like the backwards() command to differentiate.

This cast disconnects the graph because casting is non-differentiable:

w_r = w_r.type(torch.LongTensor)

Building a Tensor from a constructor will disconnect the graph:

batchRankLoss = torch.tensor([max(0,alpha - delta(y_i[i], y_j[i])*predy_i[i] - predy_j[i])) for i in range(batchSize)],dtype = torch.float)

From the docs, wrapping a Tensor in a Variable will set the grad_fn to None (also disconnecting the graph):

rankLoss = Variable(rankLossPrev,requires_grad=True)

Assuming that your critereon function is differentiable, then gradients are currently flowing backward only through loss1 and loss2. Your other gradients will only flow as far as mult before they are stopped by a call to type(). This is consistent with your observation that your custom loss doesn't change the output of your neural network.

To allow gradients to flow backward through your custom loss, you'll have to code the same logic while avoiding type() casts and calculate rankLoss without using a list comprehension.


Instruction: Reducing batch size in pytorch

Input:
I am new to programming in pytorch. I am getting this error which says cuda out of memory. So I have to reduce the batch size. Can someone tell me how to do it in python code? I also don't know my current batch size.

p.s. I am trying to run the Deep Image Prior's super-resolution. Here's the code.

The error I am getting is when running the optimization. It says

  RuntimeError: Cuda out of memory.

from __future__ import print_function
import matplotlib.pyplot as plt
%matplotlib inline

import argparse
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

import numpy as np
from models import *

import torch
import torch.optim
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

import warnings
warnings.filterwarnings("ignore")

from skimage.measure import compare_psnr
from models.downsampler import Downsampler

from utils.sr_utils import *

torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark =True
dtype = torch.cuda.FloatTensor

imsize = -1
factor = 16 # 8
enforse_div32 = 'CROP' # we usually need the dimensions to be divisible by a power of two (32 in this case)
PLOT = True
path_to_image = '/home/smitha/deep-image-prior/tnew.tif'
imgs = load_LR_HR_imgs_sr(path_to_image , imsize, factor, enforse_div32)
imgs['bicubic_np'], imgs['sharp_np'], imgs['nearest_np'] = get_baselines(imgs['LR_pil'], imgs['HR_pil'])
if PLOT:
    plot_image_grid([imgs['HR_np'], imgs['bicubic_np'], imgs['sharp_np'], imgs['nearest_np']], 4,12);
    print ('PSNR bicubic: %.4f   PSNR nearest: %.4f' %  (
                                    compare_psnr(imgs['HR_np'], imgs['bicubic_np']),
                                    compare_psnr(imgs['HR_np'], imgs['nearest_np'])))
input_depth = 8
INPUT =     'noise'
pad   =     'reflection'
OPT_OVER =  'net'
KERNEL_TYPE='lanczos2'
LR = 5
tv_weight = 0.0
OPTIMIZER = 'adam'
if factor == 16:
    num_iter = 10
    reg_noise_std = 0.01
elif factor == 8:
    num_iter = 40
    reg_noise_std = 0.05
else:
    assert False, 'We did not experiment with other factors'
net_input = get_noise(input_depth, INPUT, (imgs['HR_pil'].size[1],  imgs['HR_pil'].size[0])).type(dtype).detach()
NET_TYPE = 'skip' # UNet, ResNet
net = get_net(input_depth, 'skip', pad,
          skip_n33d=128,
          skip_n33u=128,
          skip_n11=4,
          num_scales=5,
          upsample_mode='bilinear').type(dtype)
mse = torch.nn.MSELoss().type(dtype)

img_LR_var = np_to_torch(imgs['LR_np']).type(dtype)

downsampler = Downsampler(n_planes=3, factor=factor,  kernel_type=KERNEL_TYPE, phase=0.5, preserve_size=True).type(dtype)
def closure():
    global i, net_input
    if reg_noise_std > 0:
        net_input = net_input_saved + (noise.normal_() * reg_noise_std)

    out_HR = net(net_input)
    out_LR = downsampler(out_HR)

    total_loss = mse(out_LR, img_LR_var)

    if tv_weight > 0:
        total_loss += tv_weight * tv_loss(out_HR)

    total_loss.backward()

    # Log
    psnr_LR = compare_psnr(imgs['LR_np'], torch_to_np(out_LR))
    psnr_HR = compare_psnr(imgs['HR_np'], torch_to_np(out_HR))
    print ('Iteration %05d    PSNR_LR %.3f   PSNR_HR %.3f' % (i, psnr_LR, psnr_HR), '\r', end='')

    # History
    psnr_history.append([psnr_LR, psnr_HR])

    if PLOT and i % 100 == 0:
        out_HR_np = torch_to_np(out_HR)
        plot_image_grid([imgs['HR_np'], imgs['bicubic_np'], np.clip(out_HR_np, 0, 1)], factor=13, nrow=3)

    i += 1

    return total_loss

psnr_history = []
volatile=True
net_input_saved = net_input.detach().clone()
noise = net_input.clone()
i = 0
p = get_params(OPT_OVER, net, net_input)
optimize(OPTIMIZER, p, closure, LR, num_iter)
out_HR_np = np.clip(torch_to_np(net(net_input)), 0, 1)
result_deep_prior = put_in_center(out_HR_np, imgs['orig_np'].shape[1:])
plot_image_grid([imgs['HR_np'],
             imgs['bicubic_np'],
             out_HR_np], factor=4, nrow=1);

Output: The batch size depends on the model.  Typically, it's the first dimension of your input tensors.  Your model uses different names than I'm used to, some of which are general terms, so I'm not sure of your model topology or usage.


Instruction: Incorporating dim parameter of torch.topk in tf.nn.top_k

Input:
Pytorch provide torch.topk(input, k, dim=None, largest=True, sorted=True) function to calculate k largest elements of the given input tensor along a given dimension dim.

I have a tensor of shape (16, 512, 4096) and I am using torch.topk in the following manner-

# inputs.shape (16L, 512L, 4096L)
dist, idx = torch.topk(inputs, 64, dim=2, largest=False, sorted=False)
# dist.shape (16L, 512L, 64L), idx.shape (16L, 512L, 64L)

I found similar tensorflow implementaion as following - tf.nn.top_k(input, k=1, sorted=True, name=None).

My question is how to Incorporate dim=2 parameter in tf.nn.top_k so as to achieve the tensor of the same shape as calculated by pytorch?

Output:
tf.nn.top_k works on the last dimension of the input. This means that it should work as is for your example:

dist, idx = tf.nn.top_k(inputs, 64, sorted=False)

In general you can imagine the Tensorflow version to work like the Pytorch version with hardcoded dim=-1, i.e. the last dimension.

However it looks like you actually want the k smallest elements. In this case we could do

dist, idx = tf.nn.top_k(-1*inputs, 64, sorted=False)
dist = -1*dist

So we take the k largest of the negative inputs, which are the k smallest of the original inputs. Then we invert the negative on the values.


Instruction: PyTorch - better way to get back original tensor order after torch.sort

Input:
I want to get back the original tensor order after a torch.sort operation and some other modifications to the sorted tensor, so that the tensor is not anymore sorted. It is better to explain this with an example:

x = torch.tensor([30., 40., 20.])
ordered, indices = torch.sort(x)
# ordered is [20., 30., 40.]
# indices is [2, 0, 1]
ordered = torch.tanh(ordered) # it doesn't matter what operation is
final = original_order(ordered, indices)
# final must be equal to torch.tanh(x)

I have implemented the function in this way:

def original_order(ordered, indices):
    z = torch.empty_like(ordered)
    for i in range(ordered.size(0)):
        z[indices[i]] = ordered[i]
    return z

Is there a better way to do this? In particular, it is possible to avoid the loop and compute the operation more efficiently?

In my case I have a tensor of size torch.Size([B, N]) and I sort each of the B rows separately with a single call of torch.sort. So, I have to call original_order B times with another loop.

Any, more pytorch-ic, ideas?

EDIT 1 - Get rid of inner loop

I solved part of the problem by simply indexing z with indices in this way:

def original_order(ordered, indices):
    z = torch.empty_like(ordered)
    z[indices] = ordered
    return z

Now, I just have to understand how to avoid the outer loop on B dimension.

EDIT 2 - Get rid of outer loop

def original_order(ordered, indices, batch_size):
    # produce a vector to shift indices by lenght of the vector
    # times the batch position
    add = torch.linspace(0, batch_size-1, batch_size) * indices.size(1)

    indices = indices + add.long().view(-1,1)

    # reduce tensor to single dimension.
    # Now the indices take in consideration the new length
    long_ordered = ordered.view(-1)
    long_indices = indices.view(-1)

    # we are in the previous case with one dimensional vector
    z = torch.zeros_like(long_ordered).float()
    z[long_indices] = long_ordered

    # reshape to get back to the correct dimension
    return z.view(batch_size, -1)

Output:
def original_order(ordered, indices):
    return ordered.gather(1, indices.argsort(1))

Example
original = torch.tensor([
    [20, 22, 24, 21],
    [12, 14, 10, 11],
    [34, 31, 30, 32]])
sorted, index = original.sort()
unsorted = sorted.gather(1, index.argsort(1))
assert(torch.all(original == unsorted))

Why it works
For simplicity, imagine t = [30, 10, 20], omitting tensor notation.
t.sort() gives us the sorted tensor s = [10, 20, 30], as well as the sorting index i = [1, 2, 0] for free. i is in fact the output of t.argsort().
i tells us how to go from t to s. "To sort t into s, take element 1, then 2, then 0, from t". Argsorting i gives us another sorting index j = [2, 0, 1], which tells us how to go from i to the canonical sequence of natural numbers [0, 1, 2], in effect reversing the sort. Another way to look at it is that j tells us how to go from s to t. "To sort s into t, take element 2, then 0, then 1, from s". Argsorting a sorting index gives us its "inverse index", going the other way.
Now that we have the inverse index, we dump that into torch.gather() with the correct dim, and that unsorts the tensor.
Sources
torch.gather
torch.argsort
I couldn't find this exact solution when researching this problem, so I think this is an original answer.


Instruction: tensorflow equivalent of torch.gather

Input:
I have a tensor of shape (16, 4096, 3). I have another tensor of indices of shape (16, 32768, 3). I am trying to collect the values along dim=1. This was initially done in pytorch using gather function as shown below-

# a.shape (16L, 4096L, 3L)
# idx.shape (16L, 32768L, 3L)
b = a.gather(1, idx)
# b.shape (16L, 32768L, 3L)

Please note that the size of output b is the same as that of idx. However, when I apply gather function of tensorflow, I get a completely different output. The output dimension was found mismatching as shown below-

b = tf.gather(a, idx, axis=1)
# b.shape (16, 16, 32768, 3, 3)

I also tried using tf.gather_nd but got in vain. See below-

b = tf.gather_nd(a, idx)
# b.shape (16, 32768)

Why am I getting different shapes of tensors? I want to get the tensor of the same shape as calculated by pytorch.

In other words, I want to know the tensorflow equivalent of torch.gather.

Output:
For 2D case,there is a method to do it:

# a.shape (16L, 10L)
# idx.shape (16L,1)
idx = tf.stack([tf.range(tf.shape(idx)[0]),idx[:,0]],axis=-1)
b = tf.gather_nd(a,idx)

However,For ND case,this method maybe very complex


Instruction: Why do we normalize the image to mean=0.5, std=0.5?

Input:
I was lookking for GAN code in Github. The code I found uses pytorch. In this code, we first normalized the image to mean = 0.5, std = 0.5. Normally, normalize to min = 0 and max = 1. Or normal distribution with mean = 0 and std = 1. Why is this normalized to mean = 0.5 and std = 0.5?

transformtransfo  = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
])

Output:
The values of mean and std for transform.normalize are not the desired mean and std, but rather the values to subtract and divide by, i.e., the estimated mean and std.

In your example you subtract 0.5 and then divide by 0.5 yielding an image with mean zero and values in range [-1, 1]


Instruction: Why does the hash value of models change everytime I save it?

Input:
I am using torch.save() to save a model file. However, everytime I save it, it changes. Why so?

netG_1 = torch.load('netG.pth')
netG_2 = torch.load('netG.pth')

torch.save(netG_1, 'netG_1.pth')
torch.save(netG_2, 'netG_2.pth')

Using md5sum *.pth:

779f0fefca47d17a0644033f9b65e594  netG_1.pth
476f502ec2d1186c349cdeba14983d09  netG_2.pth
b0ceec8ac886a11b79f73fc04f51c6f9  netG.pth

The model is an instance of this class:

https://github.com/taoxugit/AttnGAN/blob/master/code/model.py#L397

Output:
A class which does not define the __hash__ method will have its instances hashed according to their id. As for CPython, this means everytime the instance is saved and reloaded, it changes hash since its position in memory changed.

Here is a proof of concept.

class Foo:
    pass

instance = Foo()

print('hash:', hex(hash(instance)))
print('id:  ', hex(id(instance)))

Output

hash: 0x22f3f57608
id:   0x22f3f576080

The exact transformation is hash(o) == id(o) // 16.


Instruction: DataLoader - shuffle implicit pairs

Input:
Is there a way to handle the DataLoader as a list ? The idea is that I want to shuffle implicit pairs of images, without setting the shuffling into True

Basically, I have for example 10 scenes, each containing let's say 100 sequences, so they are represented inside the directory as

'1_1.png', '1_2.png', '1_3.png', '....., '2_1.png', '2_2.png', '2_3.png', ...., '3_1.png', '3_2.png', '3_3.png', ..., ...., '10_1.png', '10_2.png', '10_3.png', ...

I don't want complete shuffling of data, what I want simply is to shuffle but keeping pairs, so they are represented in the data loader as

[ '1_3.png', '1_4.png', '2_2.png', '2_3.png', '10_1.png', '10_2.png', '1_2.png', '1_3.png', ...]
and so on

Please have a look at this question which I have already asked on Stack Overflow concerning shuffling array of implicit pairs, where you can understand what I mean

As an example:
if this is a list

L = [['1_1'],['1_2'],['1_3'],['1_4'],['1_5'],['1_6'],['2_1'],['2_2'],['2_3'],['2_4'],['2_5'],['2_6'],['3_1'],['3_2'],['3_3'],['3_4'],['3_5'],['3_6']]

then this is the output

[['1_2'], ['1_3'], ['2_1'], ['2_2'], ['2_4'], ['2_5'],
 ['2_2'], ['2_3'], ['1_3'], ['1_4'], ['3_4'], ['3_5'],
 ['3_3'], ['3_4'], ['3_2'], ['3_3'], ['1_6'], ['2_1'],
 ['2_5'], ['2_6'], ['2_6'], ['3_1'], ['1_4'], ['1_5'],
 ['1_1'], ['1_2'], ['2_3'], ['2_4'], ['1_5'], ['1_6'],
 ['3_1'], ['3_2'], ['3_5'], ['3_6']]

I want to achieve the same for a DataLoader

The main idea, is that I want to train my network on sequential frames, but it doesn't have to be the complete sequence, but at least I need each step, two sequences are there

Output:
I think you are looking for data.Sampler: instead of the completely radom default shuffle of data.DataLoader, you can provide your own "sampler" that sample examples from your Dataset.
Looking at the input parameters of data.DataLoader:

  sampler (Sampler, optional) – defines the strategy to draw samples
  from the dataset. If specified, shuffle must be False.

I think a good starting point for is too look at the code of data.SubsetRandomSampler.


Instruction: How to load 2D data into an LSTM in pytorch

Input:
I have a series of sine waves that i have loaded in using a custom dataloader. The data is converted to a torch tensor using from_numpy. I then try to load the data using an enumerator over the train_loader. The iterator is shown below.

for epoch in range(epochs):
    for i, data in enumerate(train_loader):
        input = np.array(data)
        train(epoch)

The error i receive is:

RuntimeError: input must have 3 dimensions, got 2

I know i need to have my input data in [sequence length, batch_size, input_size] for an LSTM but i have no idea how to format my array data of 1000 sine waves of length 10000.

Below is my training method.

def train(epoch):
    model.train()
    train_loss = 0
    def closure():
        optimizer.zero_grad()
        print(input.shape)
        output = model(Variable(input))
        loss = loss_function(output)
        print('epoch: ', epoch.item(),'loss:', loss.item())
        loss.backward()
        return loss
    optimizer.step(closure)

I thought i would try add (seq_length, batch_size, input_size) in a tuple but this cant be fed into the network. Further to this my assumption was that the dataloader fed batch size into the system. Any help would be appreciated.

edit:

Here is my sample data:

T = 20
L = 1000
N = 100

x = np.empty((N, L), 'int64')
x[:] = np.array(range(L)) + np.random.randint(-4 * T, 4 * T, N).reshape(N, 1)
data = np.sin(x / 1.0 / T).astype('float64')
torch.save(data, open('traindata.pt', 'wb'))

Output:
Can you share a simple example of your data just to confirm?

Also, you have to have a different order for your shape. Generally, the first dimension is always batch_size, and then afterwards the other dimensions, like  [batch_size, sequence_length, input_dim].

One way to achieve this, if you have a batch size of 1, is to use torch.unsqueeze(). This allows you to create a "fake" dimension:

import torch as t
x = t.Tensor([1,2,3])
print(x.shape)
x = x.unsqueeze(dim=0) # adds a 0-th dimension of size 1
print(x.shape)


Instruction: How to use double as the default type for floating numbers in PyTorch

Input: I want all the floating numbers in my PyTorch code double type by default, how can I do that?

Output:
You are looking for torch.set_default_tensor_type:

torch.set_default_tensor_type(torch.DoubleTensor)

Aletrnatively, you can use torch.set_default_dtype:

torch.set_default_dtype(torch.float64)