Instruction: when training simple code of pytorch, cpu ratio increased. GPU is 0% approximately

Input:
I'm doing tutorial of Pytorch.

Code is clearly completed. but i have one problem.

It is about my CPU use ratio.
If I enter into training, CPU usage ratio is increasıng up to 100%.
but GPU is roughly 0%.

I installed CUDA 9.2 and cudnn.
and I already checked massage about torch.cuda.is_available()==True.

is it OK, or my setup is wrong?

Output:
1.. Did you upload your model and input tensors onto GPU explicitly, showing as follow
https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#training-on-gpu

For example,

# Configure your device
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# Upload your model onto GPU
net.to(device)
# Upload your tensor onto GPU
inputs, labels = inputs.to(device), labels.to(device)

2.. You also can use "gpustat" to check GPU usage.
https://github.com/wookayin/gpustat

After installing, you can type "gpustat" on terminal
If your code runs on GPU, GPU usage will increase.

3.. And check whether you've added following CUDA path into your bashrc file.
Following CUDA path is general path on Ubuntu Linux,
but that path can be different per OS or your setting.

You can open bashrc file by typing vim ./.bashrc
when your current directory is home in case where you use Ubuntu Linux.

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

4.. Also check your graphic driver has been installed
by typing nvidia-smi on terminal if you use Ubuntu Linux.


Instruction: Why does gpytorch seem to be less accurate than scikit-learn?

Input:
I current found gpytorch (https://github.com/cornellius-gp/gpytorch). It seems to be a great package for integrating GPR into pytorch. First tests were also positive. Using gpytorch the GPU-Power as well as intelligent algorithms can used in order to improve performance in comparison to other packages such as scikit-learn.
However, I found that it is much harder to estimate the hyperparameters that are needed. In scikit-learn that happens in the background and is very robust. I would like get some feed from the community about the reasons and to discuss if there might be a better way to estimatethese parameter than provided by the example in the documentation of gpytorch.
For comparisson, I took the code of a provided example on the offcial page of gpytorch (https://github.com/cornellius-gp/gpytorch/blob/master/examples/03_Multitask_GP_Regression/Multitask_GP_Regression.ipynb) and modified it in two parts:

I use a different kernel (gpytorch.kernels.MaternKernel(nu=2.5) in stead of gpytorch.kernels.RBFKernel())
I used a different output function

In the following, I provide first the code using gpytorch. Subsequently, I provide the code for scikit-learn. Finally, I compare the results
Importing (for gpytorch and scikit-learn):
import math
import torch
import numpy as np
import gpytorch

Generating data (for gpytorch and scikit-learn):
n = 20
train_x = torch.zeros(pow(n, 2), 2)
for i in range(n):
    for j in range(n):
        # Each coordinate varies from 0 to 1 in n=100 steps
        train_x[i * n + j][0] = float(i) / (n-1)
        train_x[i * n + j][1] = float(j) / (n-1)

train_y_1 = (torch.sin(train_x[:, 0]) + torch.cos(train_x[:, 1]) * (2 * math.pi) + torch.randn_like(train_x[:, 0]).mul(0.01))/4
train_y_2 = torch.sin(train_x[:, 0]) + torch.cos(train_x[:, 1]) * (2 * math.pi) + torch.randn_like(train_x[:, 0]).mul(0.01)

train_y = torch.stack([train_y_1, train_y_2], -1)

test_x = torch.rand((n, len(train_x.shape)))
test_y_1 = (torch.sin(test_x[:, 0]) + torch.cos(test_x[:, 1]) * (2 * math.pi) + torch.randn_like(test_x[:, 0]).mul(0.01))/4
test_y_2 = torch.sin(test_x[:, 0]) + torch.cos(test_x[:, 1]) * (2 * math.pi) + torch.randn_like(test_x[:, 0]).mul(0.01)
test_y = torch.stack([test_y_1, test_y_2], -1)

Now comes the estimation as described in the provided example from the cited documentation:
torch.manual_seed(2) # For a more robust comparison
class MultitaskGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(MultitaskGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.MultitaskMean(
            gpytorch.means.ConstantMean(), num_tasks=2
        )
        self.covar_module = gpytorch.kernels.MultitaskKernel(
            gpytorch.kernels.MaternKernel(nu=2.5), num_tasks=2, rank=1
        )

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)

likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=2)
model = MultitaskGPModel(train_x, train_y, likelihood)

# Find optimal model hyperparameters
model.train()
likelihood.train()

# Use the adam optimizer
optimizer = torch.optim.Adam([
    {'params': model.parameters()},  # Includes GaussianLikelihood parameters
], lr=0.1)

# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

n_iter = 50
for i in range(n_iter):
    optimizer.zero_grad()
    output = model(train_x)
    loss = -mll(output, train_y)
    loss.backward()
    # print('Iter %d/%d - Loss: %.3f' % (i + 1, n_iter, loss.item()))
    optimizer.step()

# Set into eval mode
model.eval()
likelihood.eval()

# Make predictions
with torch.no_grad(), gpytorch.settings.fast_pred_var():
    predictions = likelihood(model(test_x))
    mean = predictions.mean
    lower, upper = predictions.confidence_region()

test_results_gpytorch = np.median((test_y - mean) / test_y, axis=0)

In the following, I provide the code for scikit-learn. Which is a little bit more convenient^^:
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import WhiteKernel, Matern
kernel = 1.0 * Matern(length_scale=0.1, length_scale_bounds=(1e-5, 1e5), nu=2.5) \
         + WhiteKernel()
gp = GaussianProcessRegressor(kernel=kernel, alpha=0.0).fit(train_x.numpy(),
                                                            train_y.numpy())
# x_interpolation = test_x.detach().numpy()[np.newaxis, :].transpose()
y_mean_interpol, y_std_norm = gp.predict(test_x.numpy(), return_std=True)

test_results_scitlearn = np.median((test_y.numpy() - y_mean_interpol) / test_y.numpy(), axis=0)

Finally I compare the results:
comparisson = (test_results_scitlearn - test_results_gpytorch)/test_results_scitlearn
print('Variable 1: scitkit learn is more accurate my factor: ' + str(abs(comparisson[0]))
print('Variable 2: scitkit learn is more accurate my factor: ' + str(comparisson[1]))

Unfortunatelly, I did not find an easy way to fix the seed for scikit-learn. The last time I have run the code, it returned:

Variable 1: scitkit learn is more accurate my factor: 11.362540360431087
Variable 2: scitkit learn is more accurate my factor: 29.64760087022618

In case of gpytorch, I assume that the optimizer runs in some local optima. But I cannot think of any more robust optimization algorithm that still uses pytorch.
I am looking forward for suggestions!
Lazloo

Output:
(I also answer your question on the GitHub issue you created for it here)

Primarily this happened because you used different models in sklearn and gpytorch. In particular, sklearn learns independent GPs in the multi-output setting by default (see e.g., the discussion here). In GPyTorch, you used the multitask GP method introduced in Bonilla et al, 2008. Correcting for this difference yields:

  test_results_gpytorch = [5.207913e-04 -8.469360e-05]

  test_results_scitlearn = [3.65288816e-04 4.79017145e-05]


Instruction: pretrained object detection models in keras

Input:
There are pretrained object recognition models in keras.applications library. But as far as I know, there is no pretrained object detection model available.

Does anyone know why it is the case? Object detection is a big part of problems when dealing with visual problems.

Output:
That is because vanilla Keras does not include implementation of methods/models for object detection.

There are many approaches to object detection with deep learning (see Object Detection with Deep Learning: A Review for a survey), but none of them are implemented as a part of Keras library, so no official models as well. I have a feeling that François Chollet tries to keep it simple and minimalistic, so bloating the code with something like TensorFlow models will be against its philosophy.

However, Keras is easily extendable, so there are plenty of unofficial implementations (e.g. SSD or Mask R-CNN) supplied with the trained models though. See Keras model zoo for more.


Instruction: RuntimeError: Expected a Tensor of type torch.FloatTensor but found a type torch.IntTensor for sequence element

Input:
I wanna generate some random number with python and transform it to tensor with pytorch. Here is my code for generating the random number and transform it into tensor.

import numpy as np
import torch

P = np.random.uniform(0.5, 1, size=[20, 1])
k = np.random.randint(1, 20, size=[20, 1])
d_k = np.random.uniform(0, np.sqrt(80000), size=[20, 1])

P = torch.from_numpy(P).float()
k = torch.from_numpy(k).int()
d_k = torch.from_numpy(d_k).float()

torch.cat((P, k, d_k), dim=-1)

Afterward, I got some error which showed:

RuntimeError: Expected a Tensor of type torch.FloatTensor but found a type torch.IntTensor for sequence element 1 in sequence argument at position #1 'tensors'

Output:
The error is because k tensor is of dtype torch.int32 while other tensors P and d_k are of dtype torch.float32. But the cat operation requires all the input tensors to be of same type. From the documentation

  torch.cat(tensors, dim=0, out=None) → Tensor

  tensors (sequence of Tensors) – any python sequence of tensors of the
  same type.

One of the solutions is to convert k to float dtype as follows:

k = torch.from_numpy(k).float()


Instruction: name '_C' is not defined pytorch+jupyter notebook

Input:
I have some code that uses pytorch, that runs fine from my IDE (pycharm).

For research, I tried to run it from a jupyter notebook.

The code in the notebook:

from algorithms import Argparser
from algorithms import Session
def main():
    print("main started")
    args = Argparser.parse()
    session = Session(args)
    session.run()

The package looks like:

|-algorithms
|---__init__.py
|---Argparser.py
|---Session.py
|---<many more files that are being used by Session>.py

some of those files do import torch

When running the code in the notebook, I get

  NameError                                 Traceback (most recent call
  last)  in
        1 from algorithms import Argparser
  ----> 2 from algorithms import Session
        3 def main():
        4     print("main started")
        5     args = Argparser.parse()

  D:\git\stav\stav-rl\algorithms\Session.py in
       12
       13
  ---> 14 from algorithms.Episode import Episode
       15 from algorithms.Agent import Agent
       16 import torch

  D:\git\stav\stav-rl\algorithms\Episode.py in
        1 author = 'Noam'
        2
  ----> 3 import torch
        4 import numpy as np
        5 import cv2

  c:\anaconda3\envs\threadartrl\lib\site-packages\torch__init__.py in

       84 from torch._C import *
       85
  ---> 86 all += [name for name in dir(C)
       87             if name[0] != '' and
       88             not name.endswith('Base')]

  NameError: name '_C' is not defined

The error is on from algorithms import Session-->...-->import torch

How can i get the code to run?

Output:
You need Cython for pytorch to work:

pip3 install Cython

See this comment on the issue on github.

My understanding is that there is a library called _C.cpython-37m-x86_64-linux-gnu.so in site-packages/torch which provides the shared object _C and requires Cython. PyCharm provides Cython support whereas the Jupyter environment doesn't.


Instruction: LSTM autoencoder always returns the average of the input sequence

Input:
I'm trying to build a very simple LSTM autoencoder with PyTorch. I always train it with the same data:

x = torch.Tensor([[0.0], [0.1], [0.2], [0.3], [0.4]])

I have built my model following this link:

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(latent_dim)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)
encoder = Model(inputs, encoded)

My code is running with no errors but y_pred converge to:

tensor([[[0.2]],
        [[0.2]],
        [[0.2]],
        [[0.2]],
        [[0.2]]], grad_fn=<StackBackward>)

Here is my code:

import torch
import torch.nn as nn
import torch.optim as optim

class LSTM(nn.Module):

    def __init__(self, input_dim, latent_dim, batch_size, num_layers):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.latent_dim = latent_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        self.encoder = nn.LSTM(self.input_dim, self.latent_dim, self.num_layers)

        self.decoder = nn.LSTM(self.latent_dim, self.input_dim, self.num_layers)

    def init_hidden_encoder(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.latent_dim),
                torch.zeros(self.num_layers, self.batch_size, self.latent_dim))

    def init_hidden_decoder(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.input_dim),
                torch.zeros(self.num_layers, self.batch_size, self.input_dim))

    def forward(self, input):
        # Reset hidden layer
        self.hidden_encoder = self.init_hidden_encoder()
        self.hidden_decoder = self.init_hidden_decoder()

        # Reshape input
        input = input.view(len(input), self.batch_size, -1)

        # Encode
        encoded, self.hidden = self.encoder(input, self.hidden_encoder)
        encoded = encoded[-1].repeat(5, 1, 1)

        # Decode
        y, self.hidden = self.decoder(encoded, self.hidden_decoder)
        return y

model = LSTM(input_dim=1, latent_dim=20, batch_size=1, num_layers=1)
loss_function = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

x = torch.Tensor([[0.0], [0.1], [0.2], [0.3], [0.4]])

while True:
    y_pred = model(x)
    optimizer.zero_grad()
    loss = loss_function(y_pred, x)
    loss.backward()
    optimizer.step()
    print(y_pred)

Output:
1. Initializing hidden states

In your source code you are using init_hidden_encoder and init_hidden_decoder functions to zero hidden states of both recurrent units in every forward pass.

In PyTorch you don't have to do that, if no initial hidden state is passed to RNN-cell (be it LSTM, GRU or RNN from the ones currently available by default in PyTorch), it is implicitly fed with zeroes.

So, to obtain the same code as your initial solution (which simplifies next parts), I will scrap unneeded parts, which leaves us with the model seen below:

class LSTM(nn.Module):
    def __init__(self, input_dim, latent_dim, num_layers):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.latent_dim = latent_dim
        self.num_layers = num_layers

        self.encoder = nn.LSTM(self.input_dim, self.latent_dim, self.num_layers)

        self.decoder = nn.LSTM(self.latent_dim, self.input_dim, self.num_layers)

    def forward(self, input):
        # Encode
        _, (last_hidden, _) = self.encoder(input)
        encoded = last_hidden.repeat(5, 1, 1)

        # Decode
        y, _ = self.decoder(encoded)
        return torch.squeeze(y)

Addition of torch.squeeze

We don't need any superfluous dimensions (like the 1 in [5,1,1]).
Actually, it's the clue to your results equal to 0.2

Furthermore, I left input reshape out of the network (in my opinion, network should be fed with input ready to be processed), to separate strictly both tasks (input preparation and model itself).

This approach gives us the following setup code and training loop:

model = LSTM(input_dim=1, latent_dim=20, num_layers=1)
loss_function = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

y = torch.Tensor([[0.0], [0.1], [0.2], [0.3], [0.4]])
# Sequence x batch x dimension
x = y.view(len(y), 1, -1)

while True:
    y_pred = model(x)
    optimizer.zero_grad()
    loss = loss_function(y_pred, y)
    loss.backward()
    optimizer.step()
    print(y_pred)

Whole network is identical to yours (for now), except it is more succinct and readable.

2. What we want, describing network changes

As your provided Keras code indicates, what we want to do (and actually you are doing it correctly) is to obtain last hiddden state from the encoder (it encodes our entire sequence) and decode the sequence from this state to obtain the original one.

BTW. this approach is called sequence to sequence or seq2seq for short (often used in tasks like language translation). Well, maybe a variation of that approach, but I would classify it as that anyway.

PyTorch provides us the last hidden state as a separate return variable from RNNs family.
I would advise against yours encoded[-1]. The reason for it would be bidirectional and multilayered approach. Say, you wanted to sum bidirectional output, it would mean a code along those lines

# batch_size and hidden_size should be inferred cluttering the code further
encoded[-1].view(batch_size, 2, hidden_size).sum(dim=1)

And that's why the line _, (last_hidden, _) = self.encoder(input) was used.

3. Why does the output converge to 0.2?

Actually, it was a mistake on your side and only in the last part.

Output shapes of your predictions and targets:

# Your output
torch.Size([5, 1, 1])
# Your target
torch.Size([5, 1])

If those shapes are provided, MSELoss, by default, uses argument size_average=True. And yes, it averages your targets and your output, which essentially calculates loss for the average of your tensor (around 2.5 at the beginning) and average of your target which is 0.2.

So the network converges correctly, but your targets are wrong.

3.1 First and wrong solution

Provide MSELoss with argument reduction="sum", though it's really temporary and works accidentally.
Network, at first, will try to get all of the outputs to be equal to sum (0 + 0.1 + 0.2 + 0.3 + 0.4 = 1.0), at first with semi-random outputs, after a while it will converge to what you want, but not for the reasons you want!.

Identity function is the easiest choice here, even for summation (as your input data is really simple).

3.2 Second and correct solution.

Just pass appropriate shapes to loss function, e.g. batch x outputs, in your case, the final part would look like this:

model = LSTM(input_dim=1, latent_dim=20, num_layers=1)
loss_function = nn.MSELoss()
optimizer = optim.Adam(model.parameters())

y = torch.Tensor([0.0, 0.1, 0.2, 0.3, 0.4])
x = y.view(len(y), 1, -1)

while True:
    y_pred = model(x)
    optimizer.zero_grad()
    loss = loss_function(y_pred, y)
    loss.backward()
    optimizer.step()
    print(y_pred)

Your target is one dimensional (as batch is of size 1) and so is your output (after squeezing unnecessary dimensions).

I changed Adam's parameters to defaults as it converges faster that way.

4. Final working code

For brevity, here is the code and results:

import torch
import torch.nn as nn
import torch.optim as optim

class LSTM(nn.Module):
    def __init__(self, input_dim, latent_dim, num_layers):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.latent_dim = latent_dim
        self.num_layers = num_layers

        self.encoder = nn.LSTM(self.input_dim, self.latent_dim, self.num_layers)

        self.decoder = nn.LSTM(self.latent_dim, self.input_dim, self.num_layers)

    def forward(self, input):
        # Encode
        _, (last_hidden, _) = self.encoder(input)
        # It is way more general that way
        encoded = last_hidden.repeat(input.shape)

        # Decode
        y, _ = self.decoder(encoded)
        return torch.squeeze(y)

model = LSTM(input_dim=1, latent_dim=20, num_layers=1)
loss_function = nn.MSELoss()
optimizer = optim.Adam(model.parameters())

y = torch.Tensor([0.0, 0.1, 0.2, 0.3, 0.4])
x = y.view(len(y), 1, -1)

while True:
    y_pred = model(x)
    optimizer.zero_grad()
    loss = loss_function(y_pred, y)
    loss.backward()
    optimizer.step()
    print(y_pred)

And here are the results after ~60k steps (it is stuck after ~20k steps actually, you may want to improve your optimization and play around with hidden size for better results):

step=59682
tensor([0.0260, 0.0886, 0.1976, 0.3079, 0.3962], grad_fn=<SqueezeBackward0>)

Additionally, L1Loss (a.k.a Mean Absolute Error) may get better results in this case:

step=10645
tensor([0.0405, 0.1049, 0.1986, 0.3098, 0.4027], grad_fn=<SqueezeBackward0>)

Tuning and correct batching of this network is left for you, hope you'll have some fun now and you get the idea. :)

PS. I repeat entire shape of input sequence, as it's more general approach and should work with batches and more dimensions out of the box.


Instruction: How to balance (oversampling) unbalanced data in PyTorch (with WeightedRandomSampler)?

Input:
I have a 2-class problem and my data is highly unbalanced. I have 232550 samples from one class and 13498 from the second class. PyTorch docs and the internet tells me to use the class WeightedRandomSampler for my DataLoader.

I have tried using the WeightedRandomSampler but I keep getting errors.

    trainratio = np.bincount(trainset.labels)
    classcount = trainratio.tolist()
    train_weights = 1./torch.tensor(classcount, dtype=torch.float)
    train_sampleweights = train_weights[trainset.labels]
    train_sampler = WeightedRandomSampler(weights=train_sampleweights,
    num_samples = len(train_sampleweights))
    trainloader = DataLoader(trainset, sampler=train_sampler,
    shuffle=False)

I can not see why I am getting this error when initializing the WeightedRandomSampler class?

I have tried other similar workarounds but so far all attempts produce some error.
How should I implement this to balance my train, validation and test data?

Currently getting this error:

  train__sampleweights = train_weights[trainset.labels] ValueError: too
  many dimensions 'str'

Output:
The problem is in the type of trainset.labels
To fix the error it is possible to convert trainset.labels to float


Instruction: PyTorch runtime error : invalid argument 0: Sizes of tensors must match except in dimension 1

Input:
I have a PyTorch model and I'm trying to test it by performing a forward pass. Here is the code:

class ResBlock(nn.Module):
    def __init__(self, inplanes, planes, stride=1):
        super(ResBlock, self).__init__()
        self.conv1x1 = nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, bias=False)
        self.conv1 = nn.Conv2d(inplanes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        #batch normalization
        self.bn1 = nn.BatchNorm2d(planes)
        self.relu = nn.ReLU()
        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(planes)
        self.stride = stride

    def forward(self, x):
        residual = self.conv1x1(x)

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        #adding the skip connection
        out += residual
        out = self.relu(out)

        return out

class ResUnet (nn.Module):

    def __init__(self, in_shape,  num_classes):
        super(ResUnet, self).__init__()
        in_channels, height, width = in_shape
        #
        #self.L1 = IncResBlock(in_channels,64)
        self.e1 = nn.Sequential(
            nn.Conv2d(in_channels, 64, kernel_size=4, stride=2,padding=1),
            ResBlock(64,64))

        self.e2 = nn.Sequential(
            nn.LeakyReLU(0.2,),
            nn.Conv2d(64, 128, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(128),
            ResBlock(128,128))
        #
        self.e2add = nn.Sequential(
            nn.Conv2d(128, 128, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(128))
        #
        ##
        self.e3 = nn.Sequential(
            nn.LeakyReLU(0.2,inplace=True),
            nn.Conv2d(128, 128, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2,),
            nn.Conv2d(128,256, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(256),
            ResBlock(256,256))

        self.e4 = nn.Sequential(
            nn.LeakyReLU(0.2,),
            nn.Conv2d(256,512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            ResBlock(512,512))
        #
        self.e4add = nn.Sequential(
            nn.Conv2d(512,512, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(512))
        #
        self.e5 = nn.Sequential(
            nn.LeakyReLU(0.2,inplace=True),
            nn.Conv2d(512,512, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(512),
            nn.LeakyReLU(0.2,),
            nn.Conv2d(512,512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            ResBlock(512,512))
        #
        #
        self.e6 = nn.Sequential(
            nn.LeakyReLU(0.2,),
            nn.Conv2d(512,512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            ResBlock(512,512))
        #
        self.e6add = nn.Sequential(
            nn.Conv2d(512,512, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(512))
        #
        self.e7 = nn.Sequential(
            nn.LeakyReLU(0.2,inplace=True),
            nn.Conv2d(512,512, kernel_size=3, stride=1,padding=1),
            nn.BatchNorm2d(512),
            nn.LeakyReLU(0.2,),
            nn.Conv2d(512,512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            ResBlock(512,512))
        #
        self.e8 = nn.Sequential(
            nn.LeakyReLU(0.2,),
            nn.Conv2d(512,512, kernel_size=4, stride=2,padding=1))
            #nn.BatchNorm2d(512))

        self.d1 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(512, 512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            nn.Dropout(p=0.5),
            ResBlock(512,512))
        #
        self.d2 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(1024, 512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            nn.Dropout(p=0.5),
            ResBlock(512,512))
        #
        self.d3 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(1024, 512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            nn.Dropout(p=0.5),
            ResBlock(512,512))
        #
        self.d4 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(1024, 512, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(512),
            ResBlock(512,512))

        #
        self.d5 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(1024, 256, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(256),
            ResBlock(256,256))
        #
        self.d6 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(512, 128, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(128),
            ResBlock(128,128))
        #
        self.d7 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(256, 64, kernel_size=4, stride=2,padding=1),
            nn.BatchNorm2d(64),
            ResBlock(64,64))
        #
        self.d8 = nn.Sequential(
            nn.ReLU(),
            nn.ConvTranspose2d(128, 64, kernel_size=4, stride=2,padding=1))
            #nn.BatchNorm2d(64),
            #nn.ReLU())

        self.out_l = nn.Sequential(
            nn.Conv2d(64,num_classes,kernel_size=1,stride=1))
            #nn.ReLU())

    def forward(self, x):

        #Image Encoder

        #### Encoder #####

        en1 = self.e1(x)

        en2 = self.e2(en1)
        en2add = self.e2add(en2)

        en3 = self.e3(en2add)

        en4 = self.e4(en3)
        en4add = self.e4add(en4)

        en5 = self.e5(en4add)

        en6 = self.e6(en5)
        en6add = self.e6add(en6)

        en7 = self.e7(en6add)

        en8 = self.e8(en7)

        #### Decoder ####
        de1_ = self.d1(en8)
        de1 = torch.cat([en7,de1_],1)

        de2_ = self.d2(de1)
        de2 = torch.cat([en6add,de2_],1)

        de3_ = self.d3(de2)
        de3 = torch.cat([en5,de3_],1)

        de4_ = self.d4(de3)
        de4 = torch.cat([en4add,de4_],1)

        de5_ = self.d5(de4)
        de5 = torch.cat([en3,de5_],1)

        de6_ = self.d6(de5)
        de6 = torch.cat([en2add,de6_],1)

        de7_ = self.d7(de6)
        de7 = torch.cat([en1,de7_],1)
        de8 = self.d8(de7)

        out_l_mask = self.out_l(de8)

        return out_l_mask

Here is how I attempt to test it:

modl = ResUnet((1,512,512), 1)
x = torch.rand(1, 1, 512, 512)
modl(x)

This works fine, as does for any size that are multiples of 64.

If I try:

modl = ResUnet((1,320,320), 1)
x = torch.rand(1, 1, 320, 320)
modl(x)

It throws an error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-46-4ddc821c365b> in <module>
----> 1 modl(x)

~/.conda/envs/torch0.4/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

<ipython-input-36-f9eeefa3c0b8> in forward(self, x)
    221         de2_ = self.d2(de1)
    222         #print de2_.size()
--> 223         de2 = torch.cat([en6add,de2_],1)
    224         #print de2.size()
    225

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 5 and 4 in dimension 2 at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/TH/generic/THTensorMath.cpp:3616

I figure the problem is caused by the input size not being a power of 2 but I am not sure how to rectify it for the given input dimenstions (320, 320).

Output:
This issue arises from mismatch in size between the variables in the downsampling (encoder) path and the upsampling (decoder) path. Your code is huge and difficult to understand, but by inserting print statements, we can check that

en6add is of size [1, 512, 5, 5]
en7 is [1, 512, 2, 2]
en8 is [1, 512, 1, 1]
then upsampling goes as powers of two: de1_ is [1, 512, 2, 2]
de1 [1, 1024, 2, 2]
de2_ [1, 512, 4, 4]

at which point you try to concatenate it with en6add, so apparently the code creating de2_ is not "upsampling enough". My strong guess is that you need to pay the attention to the output_padding parameter of nn.ConvTranspose2d and possibly set it to 1 in a couple of places. I would try and fix this error for you, but that example is so far from being minimal that I can't wrap my head around the whole of it.


Instruction: PyTorch Dataloader - List is not callable error when enumerating

Input:
When iterating over a PyTorch dataloader, e.g.

# define dataset, dataloader
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=64)

# define model, optimizer, loss
# not included - irrelevant to the question

for ii, (inputs, labels) in enumerate(trainloader):

    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

    start = time.time()

    outputs = model.forward(inputs)
    loss = criterion(outputs, labels)
    loss.backward()

I get a TypeError: 'list' object is not callable on this line

for ii, (inputs, labels) in enumerate(trainloader):

What dumb thing am I forgetting?

Output:
Did you remember to call transforms.Compose on your list of transforms?

In this line

train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)

the transform parameter is expecting a callable object, not a list.

So, for example, this is wrong:

train_transforms = [
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]

It should look like this

train_transforms = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])


Instruction: net.load_state_dict(torch.load('rnn_x_epoch.net')) not working on cpu

Input:
I am using pytorch to train a Neural Network. When I train and test on GPU, it works fine.
But When I try to load the model parameters on CPU using:

net.load_state_dict(torch.load('rnn_x_epoch.net'))

I get the following error:

RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51

I have searched for the error, it's mainly because of CUDA driver dependency, but since I'm running on CPU when I get this error,it must be something else, or may be I missed something.
Since it's working fine using GPU, I could just run it on GPU but I'm trying to train the network on GPU, store the parameters and then load it on CPU mode for predictions.
I am just looking for a way to load the parameters while on CPU mode.

I tried this as well to load the parameters:

check = torch.load('rnn_x_epoch.net')

It did not work.

I tried to save the model parameters in two ways, to see if any of these would work, but didn't:
1)

checkpoint = {'n_hidden': net.n_hidden,
          'n_layers': net.n_layers,
          'state_dict': net.state_dict(),
          'tokens': net.chars}

with open('rnn_x_epoch.net', 'wb') as f:
    torch.save(checkpoint, f)

2)

torch.save(model.state_dict(), 'rnn_x_epoch.net')

TraceBack:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-e61f28013b35> in <module>()
----> 1 net.load_state_dict(torch.load('rnn_x_epoch.net'))

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    301         f = open(f, 'rb')
    302     try:
--> 303         return _load(f, map_location, pickle_module)
    304     finally:
    305         if new_fd:

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    467     unpickler = pickle_module.Unpickler(f)
    468     unpickler.persistent_load = persistent_load
--> 469     result = unpickler.load()
    470
    471     deserialized_storage_keys = pickle_module.load(f)

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in persistent_load(saved_id)
    435             if root_key not in deserialized_objects:
    436                 deserialized_objects[root_key] = restore_location(
--> 437                     data_type(size), location)
    438             storage = deserialized_objects[root_key]
    439             if view_metadata is not None:

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in default_restore_location(storage, location)
     86 def default_restore_location(storage, location):
     87     for _, _, fn in _package_registry:
---> 88         result = fn(storage, location)
     89         if result is not None:
     90             return result

/opt/conda/lib/python3.6/site-packages/torch/serialization.py in _cuda_deserialize(obj, location)
     68     if location.startswith('cuda'):
     69         device = max(int(location[5:]), 0)
---> 70         return obj.cuda(device)
     71
     72

/opt/conda/lib/python3.6/site-packages/torch/_utils.py in _cuda(self, device, non_blocking, **kwargs)
     66         if device is None:
     67             device = -1
---> 68     with torch.cuda.device(device):
     69         if self.is_sparse:
     70             new_type = getattr(torch.cuda.sparse,
self.__class__.__name__)

/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py in __enter__(self)
    223         if self.idx is -1:
    224             return
--> 225         self.prev_idx = torch._C._cuda_getDevice()
    226         if self.prev_idx != self.idx:
    227             torch._C._cuda_setDevice(self.idx)

RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51

Also may be the save/load operations in Pytorch are only for GPU mode, but I am not really convinced by that.

Output:
From the PyTorch documentation:

  When you call torch.load() on a file which contains GPU tensors, those tensors will be loaded to GPU by default.

To load the model on CPU which was saved on GPU, you need to pass map_location argument as cpu in load function as follows:

# Load all tensors onto the CPU
net.load_state_dict(torch.load('rnn_x_epoch.net', map_location=torch.device('cpu')))

In doing so, the storages underlying the tensors are dynamically remapped to the CPU device using the map_location argument. You can read more on the official PyTorch tutorials.

This can also be done as follows:

# Load all tensors onto the CPU, using a function
net.load_state_dict(torch.load('rnn_x_epoch.net', map_location=lambda storage, loc: storage))


Instruction: Application of nn.Linear layer in pytorch on additional dimentions

Input: How is the fully-connected layer (nn.Linear) in pytorch applied on "additional dimensions"? The documentation says, that it can be applied to connect a tensor (N,*,in_features) to (N,*,out_features), where N in the number of examples in a batch, so it is irrelevant, and * are those "additional" dimensions. Does it mean that a single layer is trained using all possible slices in the additional dimensions or are separate layers trained for each slice or something yet different?

Output:
There are in_features * out_features parameters learned in linear.weight and out_features parameters learned in linear.bias. You can think of nn.Linear working as

reshape the tensor to some (N', in_features), where N' is the product of N and all dimensions described with *: input_2d = input.reshape(-1, in_features)
Apply a standard matrix-matrix multiplication output_2d = linear.weight @ input_2d.
Add the bias output_2d += linear.bias.reshape(1, in_features) (notice we broadcast it across all N' dimensions)
Reshape the output to have the same dimensions as input, aside from the last one: output = output_2d.reshape(*input.shape[:-1], out_features)
return output

So the leading dimension N is treated the same as the * dimensions. The documentation makes N explicit to let you know that the input has to be at least 2d, but can be as many dimensional as you wish.


Instruction: Unsupported Wheel Error when pip installing PyTorch without Conda

Input:
I have been trying to install PyTorch in Windows 10 for Python 3.7.1

I do not have Anaconda on my machine, and do not wish to install it. I believe I have already satisfied all the necessary prerequisites (CUDA v10.0, NumPy). When I run the following installation command in the admin command line, (found on the PyTorch webpage):

pip3 install https://download.pytorch.org/whl/cu100/torch-1.0.0-cp37-cp37m-win_amd64.whl

I received the following error:

torch-1.0.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.

I tried downloading the wheel file in my browser, then running a modified command in my downloads directory.

pip install torch-1.0.0-cp37-cp37m-win_amd64.whl

I received the same error message. My pip version is up to date and I am attempting to install the appropriate wheel file for my Python version. This problem is unique to others, as I do not want to use Conda to install PyTorch. What is causing  this problem?

Output:
The wheel I was trying to install required 32 bit Python, I had 64 bit Python installed. Therefore, the wheel I was trying to install was not compatible with my Python version.
Checking Python Version:
I confirmed my Python version using the following command:
python -c "import struct; print(struct.calcsize('P') * 8)"
Checking Wheel Version:
64 bit wheels typically contain amd64 or similar in their title
32 bit wheels typically contain win32or similar in their title
Switching to a 64 bit Python Installaion:
The default Windows Python installer does not have a 64 bit option. To acquire 64 bit Python, navigate to python.org/downloads/windows/, and select a version that specifies x86-64 (the other versions are all 32 bit).

Credit to phd for the comment that led to this solution.
A Redditor had the same problem here.


Instruction: AND-gate with Pytorch

Input:
I'm new to PyTorch and deep learning generally.
The code I wrote can be seen longer down.
I'm trying to learn the simple 'And' problem, which is linearby separable.
The problem is, that I'm getting poor results. Only around 2/10 times it gets to the correct answer.
Sometimes the loss.item() values is stuck at 0.250.

Just to clear up

Why does it only work 2/10 times?

.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.autograd as autog

data_x = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
data_y = np.array([[0, 0, 0, 1]]).T
data_x = autog.Variable(torch.FloatTensor(data_x))
data_y = autog.Variable(torch.FloatTensor(data_y), requires_grad=False)

in_dim = 2
out_dim = 1
epochs = 15000
epoch_print = epochs / 5
l_rate = 0.001

class NeuralNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(NeuralNet, self).__init__()
        self.lin1 = nn.Linear(input_size, output_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        out = x
        out = self.lin1(out)
        out = self.relu(out)
        return out

model = NeuralNet(in_dim, out_dim)
criterion = nn.L1Loss()
optimizer = optim.Adam(model.parameters(), lr=l_rate)

for epoch in range(epochs):
    pred = model(data_x)
    loss = criterion(pred, data_y)
    loss.backward()
    optimizer.step()
    if (epoch + 1) % epoch_print == 0:
        print("Epoch %d  Loss %.3f" %(epoch + 1, loss.item()))

for x, y in zip(data_x, data_y):
    pred = model(x)
    print("Input", list(map(int, x)), "Pred", int(pred), "Output", int(y))

Output:
1. Using zero_grad with optimizer

You are not using optimizer.zero_grad() to clear the gradient. Your learning loop should look like this:

for epoch in range(epochs):
    optimizer.zero_grad()
    pred = model(data_x)
    loss = criterion(pred, data_y)
    loss.backward()
    optimizer.step()
    if (epoch + 1) % epoch_print == 0:
        print("Epoch %d  Loss %.3f" %(epoch + 1, loss.item()))

In this particular case it will not have any detrimental effect, the gradient is accumulating, but as you have the same dataset looped over and over it makes barely any difference (you should get into this habit though, as you will use it throughout your deep learning journey).

2. Cost Function

You are using Mean Absolute Error which is regression loss function, not a classification one (what you do is binary classification).

Accordingly, you should use BCELoss and sigmoid activation or (I prefer it that way), return logits from the network and use BCEWithLogitsLoss, both of them calculate binary cross entropy (simplified version of cross-entropy).

See below:

class NeuralNet(nn.Module):
    def __init__(self, input_size, output_size):
        super(NeuralNet, self).__init__()
        self.lin1 = nn.Linear(input_size, output_size)

    def forward(self, x):
        # You may want to use torch.nn.functional.sigmoid activation
        return self.lin1(x)

...
# Change your criterion to nn.BCELoss() if using sigmoid
criterion = nn.BCEWithLogitsLoss()
...

3. Predictions

If you used the logits version, classifier learns to assign negative values to 0 label and positive to indicate 1. Your display function has to be modified to incorporate this knowledge:

for x, y in zip(data_x, data_y):
    pred = model(x)
    # See int(pred > 0), that's the only change
    print("Input", list(map(int, x)), "Pred", int(pred > 0), "Output", int(y))

This step does not apply if your forward applies sigmoid to the output. Oh, and it's better to use torch.round instead of casting to int.


Instruction: How to properly update the weights in PyTorch?

Input:
I'm trying to implement the gradient descent with PyTorch according to this schema but can't figure out how to properly update the weights. It is just a toy example with 2 linear layers with 2 nodes in hidden layer and one output.
Learning rate = 0.05;
target output = 1

https://hmkcode.github.io/ai/backpropagation-step-by-step/
Forward
Backward
My code is as following:
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim

    class MyNet(nn.Module):

    def __init__(self):
         super(MyNet, self).__init__()
         self.linear1 = nn.Linear(2, 2,  bias=None)
         self.linear1.weight = torch.nn.Parameter(torch.tensor([[0.11, 0.21], [0.12, 0.08]]))
         self.linear2 = nn.Linear(2, 1,  bias=None)
         self.linear2.weight = torch.nn.Parameter(torch.tensor([[0.14, 0.15]]))

    def forward(self, inputs):
         out = self.linear1(inputs)
         out = self.linear2(out)
         return out

    losses = []
    loss_function = nn.L1Loss()
    model = MyNet()
    optimizer = optim.SGD(model.parameters(), lr=0.05)
    input = torch.tensor([2.0,3.0])
    print('weights before backpropagation = ',   list(model.parameters()))
    for epoch in range(1):
       result = model(input )
       loss = loss_function(result , torch.tensor([1.00],dtype=torch.float))
       print('result = ', result)
       print("loss = ",   loss)
       model.zero_grad()
       loss.backward()
       print('gradients =', [x.grad.data  for x in model.parameters()] )
       optimizer.step()
       print('weights after backpropagation = ',   list(model.parameters()))

The result is following :
    weights before backpropagation =  [Parameter containing:
    tensor([[0.1100, 0.2100],
            [0.1200, 0.0800]], requires_grad=True), Parameter containing:
    tensor([[0.1400, 0.1500]], requires_grad=True)]

    result =  tensor([0.1910], grad_fn=<SqueezeBackward3>)
    loss =  tensor(0.8090, grad_fn=<L1LossBackward>)

    gradients = [tensor([[-0.2800, -0.4200], [-0.3000, -0.4500]]),
                 tensor([[-0.8500, -0.4800]])]

    weights after backpropagation =  [Parameter containing:
    tensor([[0.1240, 0.2310],
            [0.1350, 0.1025]], requires_grad=True), Parameter containing:
    tensor([[0.1825, 0.1740]], requires_grad=True)]

Forward pass values:
2x0.11 + 3*0.21=0.85 ->
2x0.12 + 3*0.08=0.48 -> 0.85x0.14 + 0.48*0.15=0.191 -> loss =0.191-1 = -0.809

Backward pass: let's calculate w5 and w6 (output node weights)
w = w - (prediction-target)x(gradient)x(output of previous node)x(learning rate)
w5= 0.14 -(0.191-1)*1*0.85*0.05= 0.14 + 0.034= 0.174
w6= 0.15 -(0.191-1)*1*0.48*0.05= 0.15 + 0.019= 0.169

In my example Torch doesn't multiply the loss by derivative so we get wrong weights after updating. For the output node we got new weights w5,w6 [0.1825, 0.1740] , when it should be [0.174, 0.169]
Moving backward to update the first weight of the output node (w5) we need to calculate: (prediction-target)x(gradient)x(output of previous node)x(learning rate)=-0.809*1*0.85*0.05=-0.034. Updated weight w5 = 0.14-(-0.034)=0.174. But instead pytorch calculated new weight = 0.1825. It forgot to multiply by (prediction-target)=-0.809. For the output node we got gradients -0.8500 and -0.4800. But we still need to multiply them by loss 0.809 and learning rate 0.05 before we can update the weights.
What is the proper way of doing this?
Should we pass 'loss' as an argument to backward() as following: loss.backward(loss) .
That seems to fix it. But I couldn't find any example on this in documentation.

Output:
You should use .zero_grad() with optimizer, so optimizer.zero_grad(), not loss or model as suggested in the comments (though model is fine, but it is not clear or readable IMO).

Except that your parameters are updated fine, so the error is not on PyTorch's side.

Based on gradient values you provided:

gradients = [tensor([[-0.2800, -0.4200], [-0.3000, -0.4500]]),
             tensor([[-0.8500, -0.4800]])]

Let's multiply all of them by your learning rate (0.05):

gradients_times_lr = [tensor([[-0.014, -0.021], [-0.015, -0.0225]]),
                      tensor([[-0.0425, -0.024]])]

Finally, let's apply ordinary SGD (theta -= gradient * lr), to get exactly the same results as in PyTorch:

parameters = [tensor([[0.1240, 0.2310], [0.1350, 0.1025]]),
              tensor([[0.1825, 0.1740]])]

What you have done is taken the gradients calculated by PyTorch and multiplied them with the output of previous node and that's not how it works!.

What you've done:

w5= 0.14 -(0.191-1)*1*0.85*0.05= 0.14 + 0.034= 0.174

What should of been done (using PyTorch's results):

w5 = 0.14 - (-0.85*0.05) = 0.1825

No multiplication of previous node, it's done behind the scenes (that's what .backprop() does - calculates correct gradients for all of the nodes), no need to multiply them by previous ones.

If you want to calculate them manually, you have to start at the loss (with delta being one) and backprop all the way down (do not use learning rate here, it's a different story!).

After all of them are calculated, you can multiply each weight by optimizers learning rate (or any other formula for that matter, e.g. Momentum) and after this you have your correct update.

How to calculate backprop

Learning rate is not part of backpropagation, leave it alone until you calculate all of the gradients (it confuses separate algorithms together, optimization procedures and backpropagation).

1. Derivative of total error w.r.t. output

Well, I don't know why you are using Mean Absolute Error (while in the tutorial it is Mean Squared Error), and that's why both those results vary. But let's go with your choice.

Derivative of | y_true - y_pred | w.r.t. to y_pred is 1, so IT IS NOT the same as loss. Change to MSE to get equal results (here, the derivative will be (1/2 * y_pred - y_true), but we usually multiply MSE by two in order to remove the first multiplication).

In MSE case you would multiply by the loss value, but it depends entirely on the loss function (it was a bit unfortunate that the tutorial you were using didn't point this out).

2. Derivative of total error w.r.t. w5

You could probably go from here, but... Derivative of total error w.r.t to w5 is the output of h1 (0.85 in this case). We multiply it by derivative of total error w.r.t. output (it is 1!) and obtain 0.85, as done in PyTorch. Same idea goes for w6.

I seriously advise you not to confuse learning rate with backprop, you are making your life harder (and it's not easy with backprop IMO, quite counterintuitive), and those are two separate things (can't stress that one enough).

This source is nice, more step-by-step, with a little more complicated network idea (activations included), so you can get a better grasp if you go through all of it.

Furthermore, if you are really keen (and you seem to be), to know more ins and outs of this, calculate the weight corrections for other optimizers (say, nesterov), so you know why we should keep those ideas separated.


Instruction: Pytorch Inner Product of 3D tensor with 1D Tensor to generate 2D Tensor

Input:
Operation : I have pytorch tensor A of dimension [n x m x c] and B of dimension [1 x 1 x c]. I want to take inner product of each of 1 x 1 x c vector from A with B and hence generate a tensor C of dimension [n x m].

Inside forward function of my network at a specific step I receive tensor of dimension [N, channels, Height, Width] where N is number of images, channels is number of channels in the feature map, and height and width are of the current feature map. I also have an [N x channels] feature map from some other subnetwork. In the next step I want to carry out the above mentioned operation.

Can somebody explain the best way and functions available in pytorch to achieve such step.

I am new to pytorch and was unable to find a proper way. Tensorflow supports NHWC format but I think pytorch doesn't, so one of the way is to reshape it to [N, Height, Width, channels] and then iterate like :

# if img is reshaped to [N, H, W, C]
img
# tensor of dimension [N, C]
aud

ans = torch.empty(N, H, W, dtype=torch.double)

for batches in range(img.shape[0]):
    for i in range(img.shape[1]):
        for j in range(img.shape[2]):
            ans[batches][i][j] = torch.dot(img[batches][i][j], aud[batches])

Any other cleaner API ?

PS : This step is required in DeepMind's paper "Object That Sound" for sound localization step.

Output:
There is a one-liner

 ans = torch.einsum('nhwc,nc->nhw', img, aud)

The API of torch.einsum can be difficult to grasp if you haven't had any experience with it before, but it's extremely powerful and generalizes a great deal of liner algebra operations (transpositions, matrix multiplications and traces).

import torch

N, H, W, C = 10, 11, 12, 13
img = torch.randn(N, H, W, C)
aud = torch.randn(N, C)

ans = torch.empty(N, H, W)
for batches in range(img.shape[0]):
    for i in range(img.shape[1]):
        for j in range(img.shape[2]):
            ans[batches][i][j] = torch.dot(img[batches][i][j], aud[batches])

ans2 = torch.einsum('nhwc,nc->nhw', img, aud)

assert torch.allclose(ans, ans2, atol=1e-6)

Note I had to increase the assertion tolerance above the standard 1e-8 because of numerical precision issues. If einsum becomes a bottleneck in more advanced usecases have a look at opt_einsum which optimizes the order of underlying operations for performance.


Instruction: Error with _DataLoaderIter in torch.utils.data.dataloader

Input:
I want to run a code which needs to import _DataLoaderIter from torch.utils.data.dataloader. By checking the source code for dataloader class, that method exist. However, I get the error:

  Traceback (most recent call last):

  File "main.py", line 4, in

  import data

  File "D:\Hyperspectral Data\RCAN\RCAN_TrainCode\code\data\__init__.py", line 3, in module

  from dataloader import MSDataLoader

  File "D:\Hyperspectral Data\RCAN\RCAN_TrainCode\code\dataloader.py", line 14, in module

  from torch.utils.data.dataloader import _DataLoaderIter

  ImportError: cannot import name '_DataLoaderIter'

Why is this happening?

Output:
Your comment answers the question: the _DataLoaderIter is there in 1.0.0 (for which you are linking documentation) but not in 0.3.1, as you can check here - its name has no preceding _.

This is a textbook example why it is a bad idea to access other packages' private classes/functions (customarily prefixed with an underscore) - you have zero guarantees on the stability of their implementation and behavior. If you need their code, it's usually better to copy-paste the code to your own file, because there it is at least guaranteed to not change between updates and bug fixes to torch.


Instruction: Pytorch - Are gradients transferred on creation of new Variables?

Input:
I have the following code:

A = Tensor of [186,3]

If I create a new empty tensor as follows:

tempTens = torch.tensor(np.zeros((186,3)), requires_grad = True).cuda()

And I apply some operations on a block of A and output it into tempTens, which I use totally for further computation, say like this:

tempTens[20,:] = SomeMatrix * A[20,:]

Will the gradients actually be transferred correctly, lets say I am having a cost function that optimizes for the output of tempTens to some ground truth

Output:
In this case, tempTens[20,:] = SomeMatrix * A[20,:] is an in-place operation with respect to tempTens, which is generally not guaranteed to work with autograd. However, if you create a new variable by applying an operation like concatenation

output = torch.cat([SomeMatrix * A[20, :], torch.zeros(163, 3, device='cuda')], dim=0)

you will get the same result in terms of math (a matrix with first 20 rows from SomeMatrix * A[20, :] and the following 166 rows of 0s), but this will work properly with autograd. This is, generally speaking, the right way to approach this kind of problems.


Instruction: About custom operations in Tensorflow and PyTorch

Input:
I have to implement an energy function, termed Rigidity Energy, as in Eq 7 of this paper here.
The energy function takes as input two 3D object meshes, and returns the energy between them. The first mesh is the source mesh, and the second mesh is the deformed version of the source mesh. In rough psuedo-code, the computation would go like this:

Iterate over all the vertices in the source mesh.

For every vertex, compute its covariance matrix with its neighboring vertices.
Perform SVD on the computed covariance matrix and find the rotation matrix of the vertex.
Use the computed rotation matrix, the point coordinates in the original mesh and the corresponding coordinates in the deformed mesh, to compute the energy deviation of the vertex.

Thus this energy function requires me to iterate over each point in the mesh, and the mesh could have more than 2k such points. In Tensorflow, there are two ways to do this. I can have 2 tensors of shape (N,3), one representing the points of source and the other of the deformed mesh.

Do it purely using Tensorflow tensors. That is, iterate over elements of the above tensors using tf.gather and perform the computation on each point using only existing TF operations. This method, would be extremely slow. I've tried to define loss functions that iterate over 1000s of points before, and the graph construction itself takes too much time to be practical.
Add a new TF OP as explained in the TF documentation here . This involves writing the function in CPP (and Cuda, for GPU support), and registering the new OP with TF.

The first method is easy to write, but impractically slow. The second method is a pain to write.

I've used TF for 3 years, and have never used PyTorch before, but at this point I'm considering switching to it, if it offers a better alternative for such cases.

Does PyTorch have a way of implementing such loss functions both easily and performs as fast as it would on GPU. i.e, A pythonic way of writing my own loss functions that runs on GPU, without any C or Cuda code on my part?

Output:
As far as I understand, you are essentially asking if this operation can be vectorized. The answer is no, at least not fully, because svd implementation in PyTorch is not vectorized.

If you showed the tensorflow implementation, it would help in understanding your starting point. I don't know what you mean by finding the rotation matrix of the vertex, but I would guess this can be vectorized. This would mean that svd is the only non-vectorized operation and you could perhaps get away with writing just a single custom OP, that is the vectorized svd - which is likely quite easy, because it would amount to calling some library routines in a loop in C++.

Two possible sources of problems I see are

if the neighborhoods of N(i) in equation 7 can be of significantly different sizes (which would mean that the covariance matrices are of different sizes and vectorization would require some dirty tricks)
the general problem of dealing with meshes and neighborhoods could be difficult. This is an innate property of irregular meshes, but PyTorch has support for sparse matrices and a dedicated package torch_geometry, which at least helps.


Instruction: how to resolve the error while installing pytorch

Input:
When i am trying to install pytorch getting an error.

All my packages are upgraded to the latest version. The error is

setup.py::build_deps::run()
Failed to run 'bash ../tools/build_pytorch_libs.sh --use-fbgemm --use-nnpack --use-mkldnn --use-qnnpack caffe2'

thank you in Advance

Output:
I also got same error with older python versions. It resolved for me when i tried with latest python version 3.7

Hope this information may help you.


Instruction: Can we use pytorch scatter_ on GPU

Input:
I'm trying to do one hot encoding on some data with pyTorch on GPU mode, however, it keeps giving me an exception. Can anybody help me?

Here's one example:

def char_OneHotEncoding(x):
    coded = torch.zeros(x.shape[0], x.shape[1], 101)
    for i in range(x.shape[1]):
        coded[:,i] = scatter(x[:,i])
    return coded

def scatter(x):
    return torch.zeros(x.shape[0], 101).scatter_(1, x.view(-1,1), 1)

So if I give it an tensor on GPU, it shows like this:

x_train = [[ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [14, 13, 83, 18, 14],
       [ 0,  0,  0,  0,  0]]
print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long).cuda()).shape)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-62-95c0c4ade406> in <module>()
      4        [14, 13, 83, 18, 14],
      5        [ 0,  0,  0,  0,  0]]
----> 6 print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long).cuda()).shape)
      7 x_train[:5, maxlen:maxlen+5]

<ipython-input-53-055f1bf71306> in char_OneHotEncoding(x)
      2     coded = torch.zeros(x.shape[0], x.shape[1], 101)
      3     for i in range(x.shape[1]):
----> 4         coded[:,i] = scatter(x[:,i])
      5     return coded
      6

<ipython-input-53-055f1bf71306> in scatter(x)
      7
      8 def scatter(x):
----> 9     return torch.zeros(x.shape[0], 101).scatter_(1, x.view(-1,1), 1)

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 'index'

BTW, if we simply remove the .cuda() here, everything goes one well

print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long)).shape)

torch.Size([5, 5, 101])

Output:
Yes, it is possible. You have to pay attention that all tensors are on GPU. In particular, by default, constructors like torch.zeros allocate on CPU, which will lead to this kind of mismatches. Your code can be fixed by constructing with device=x.device, as below

import torch

 def char_OneHotEncoding(x):
     coded = torch.zeros(x.shape[0], x.shape[1], 101, device=x.device)
     for i in range(x.shape[1]):
         coded[:,i] = scatter(x[:,i])
     return coded

 def scatter(x):
     return torch.zeros(x.shape[0], 101, device=x.device).scatter_(1, x.view(-1,1), 1)

 x_train = torch.tensor([
     [ 0,  0,  0,  0,  0],
     [ 0,  0,  0,  0,  0],
     [ 0,  0,  0,  0,  0],
     [14, 13, 83, 18, 14],
     [ 0,  0,  0,  0,  0]
 ], dtype=torch.long, device='cuda')

 print(char_OneHotEncoding(x_train).shape)

Another alternative are constructors called xxx_like, for instance zeros_like, though in this case, since you need different shapes than x, I found device=x.device more readable.


Instruction: Why can't I install new version of Pytorch?

Input:
My OS is CentOS 7, and I want to install PyTorch so I did the following:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda -V
conda 4.6.2

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ conda install -c anaconda pytorch-gpu

What's strange is that the installation message shows that it is installing  a very old version of PyTorch:

Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /home/martin/anaconda3/envs/pt_gpu

  added / updated specs:
    - pytorch-gpu

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2018.12.5  |                0         123 KB  anaconda
    certifi-2018.11.29         |           py36_0         146 KB  anaconda
    pytorch-gpu-0.1.12         |           py36_0        16.8 MB  anaconda
    ------------------------------------------------------------
                                           Total:        17.0 MB

The following packages will be UPDATED:

  openssl              pkgs/main::openssl-1.1.1a-h7b6447c_0 --> anaconda::openssl-1.1.1-h7b6447c_0

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates                                 pkgs/main --> anaconda
  certifi                                         pkgs/main --> anaconda
  mkl                    pkgs/main::mkl-2017.0.4-h4c4d0af_0 --> anaconda::mkl-2017.0.1-0
  pytorch-gpu                                     pkgs/free --> anaconda

Proceed ([y]/n)? y

Downloading and Extracting Packages
certifi-2018.11.29   | 146 KB    | ########################################################################################################################## | 100%
ca-certificates-2018 | 123 KB    | ########################################################################################################################## | 100%
pytorch-gpu-0.1.12   | 16.8 MB   | ########################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

To verify what's installed, I did:

(pt_gpu) [martin@A08-R32-I196-3-FZ2LTP2 mlm]$ python -c "import torch; print(torch.__version__)"
0.1.12

Why is it that?

Output:
According to their official website( https://pytorch.org ) ,  they install package named pytorch,  not pytorch-gpu.

conda install pytorch torchvision -c pytorch


Instruction: PyTorch: Is there a way to store model in CPU ram, but run all operations on the GPU for large models?

Input: From what I see, most people seem to be initializing an entire model, and sending the whole thing to the GPU. But I have a neural net model that is too big to fit entirely on my GPU. Is it possible to keep the model saved in ram, but run all the operations on the GPU?

Output:
I do not believe this is possible. However, one easy work around would be to split you model into sections that will fit into gpu memory along with your batch input.

Send the first part(s) of the model to gpu and calculate outputs
Release the former part of the model from gpu memory, and send the next section of the model to the gpu.
Input the output from 1 into the next section of the model and save outputs.

Repeat 1 through 3 until you reach your models final output.


Instruction: Pytorch Argrelmax function (or C++)

Input:
I'm trying to find the equivalent pytorch (or C++) for scipy.signal.argrelmax(), which finds the peaks in a 1D array with some padding. https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.argrelmax.html

Here's what I've come up with and it is faster than scipy.signal.argrelmax - but I'm missing a fast solution to the last step which deletes peaks within some window.

import torch

# initalize an array (not the one in plot below)
gpu_max = torch.rand(100000)

# find peaks and troughs by subtracting shifted versions
gpu_temp1 = gpu_max[1:-1]-gpu_max[:-2]
gpu_temp2 = gpu_max[1:-1]-gpu_max[2:]

# and checking where both shifts are positive;
out1 = torch.where(gpu_temp1>0, gpu_temp1*0+1, gpu_temp1*0)
out2 = torch.where(gpu_temp2>0, out1, gpu_temp2*0)

# argrelmax containing all peaks
argrelmax_gpu = torch.nonzero(out2, out=None)+1

So the top plot which marks every single relmaxpeak is pretty fast. But need the bottom one - which comes from scipy.signal.argrelmax() and uses a 30sample time window (i.e. it only returns maxima within a 60 time point window).

EDIT: I've updated code to reflect updated torch.nonzero() search.  But still need to figure out how to do the last step (in figure below).

Output:
Ok, so someone on pytorch forums did have a pretty good solution: https://discuss.pytorch.org/t/pytorch-argrelmax-or-c-function/36404

Here's the full answer in case it gets deleted:

    a = #1D Array of  your choice
    window_maxima = torch.nn.functional.max_pool1d_with_indices(a.view(1,1,-1), width, 1, padding=width//2)[1].squeeze()
    candidates = window_maxima.unique()
    nice_peaks = candidates[(window_maxima[candidates]==candidates).nonzero()]


Instruction: Get corner of rectangle near to origin in batch of tensor given any two diagonal coordinates in pytorch

Input:
Let's say I have pytorch tensor of batch of coordinates of off diagonal elements and I want to get coordinate of the corner which is near to origin. coordinates are in (x1, y1, x2, y2) form.

a = torch.tensor([[3,2,2,3], [1,1,2,2])
# expected output
[[2,2], [1,1]]

Output:
You can just iterate over all tensors and for each of them calculate distance to four corners and take the corner with minimum distance.

import torch
a = torch.tensor([[3,2,2,3], [1,1,2,2]])
c = torch.zeros(a.shape[0], 2)
for idx, x in enumerate(a):
    d1 = x[0] ** 2 + x[1] ** 2
    d2 = x[2] ** 2 + x[3] ** 2
    d3 = x[0] ** 2 + x[3] ** 2
    d4 = x[2] ** 2 + x[1] ** 2
    dmin = min(d1, d2, d3, d4)
    if d1 == dmin:
        c[idx] = torch.tensor([x[0], x[1]])
    elif d2 == dmin:
        c[idx] = torch.tensor([x[2], x[3]])
    elif d3 == dmin:
        c[idx] = torch.tensor([x[0], x[3]])
    elif d4 == dmin:
        c[idx] = torch.tensor([x[2], x[1]])
print(c)  # tensor([[2., 2.], [1., 1.]])


Instruction: More efficient way of implement this equation in pytorch (or Numpy)

Input:
I'm implementing the analytical form of this function

where k(x,y) is a RBF kernel k(x,y) = exp(-||x-y||^2 / (2h))

My function prototype is

def A(X, Y, grad_log_px，Kxy):
   pass

and X, Y are NxD matrix where N is batch size and D is a dimension. So X is a batch of x with size N in the above equation grad_log_px is some NxD matrix I've computed using autograd.

Kxy is NxN matrix where each entry (i,j) is the RBF kernel K(X[i],Y[j])

The challenge here is that in the above equation, y is just a vector with dimension D. I kind of want to pass into a batch of y. (So to pass matrix Y with NxD size)

The equation is fine using loop through the batch size but I'm having trouble to implement in a more neat way

here is my attempted loop solution:

def A(X, Y, grad_log_px，Kxy):
   res = []
   for i in range(Y.shape[0]):
       temp = 0
       for j in range(X.shape[0]):
           # first term of equation
           temp += grad_log_px[j].reshape(D,1)@(Kxy[j,i] * (X[i] - Y[j]) / h).reshape(1,D)
           temp += Kxy[j,i] * np.identity(D) - ((X[i] - Y[j]) / h).reshape(D,1)@(Kxy[j,i] * (X[i] - Y[j]) / h).reshape(1,D) # second term of equation
       temp /= X.shape[0]

        res.append(temp)
    return np.asarray(res) # return NxDxD array

In the equation: grad_{x} and grad_{y} both dimension D

Output:
Given that that I inferred all the dimensions of the various terms correctly here's a way to go about it. But first a summary of the dimensions (screenshot as it's easier to explain with math type setting; please verify if they are correct):

Also note the double derivative of the second term which gives:

where subscripts denote samples and superscripts denote features.

So we can create the two terms by using np.einsum (similarly torch.einsum) and array broadcasting:

grad_y_K = (X[:, None, :] - Y) / h * K[:, :, None]  # Shape: N_x, N_y, D
term_1 = np.einsum('ij,ikl->ikjl', grad_log_px, grad_y_K)  # Shape: N_x, N_y, D_x, D_y
term_2_h = np.einsum('ij,kl->ijkl', K, np.eye(D)) / h  # Shape: N_x, N_y, D_x, D_y
term_2_h2_xy = np.einsum('ijk,ijl->ijkl', grad_y_K, grad_y_K)  # Shape: N_x, N_y, D_x, D_y
term_2_h2 = K[:, :, None, None] * term_2_h2_xy / h**2  # Shape: N_x, N_y, D_x, D_y
term_2 = term_2_h - term_2_h2  # Shape: N_x, N_y, D_x, D_y

Then the result is given by:

(term_1 + term_2).sum(axis=0) / N  # Shape: N_y, D_x, D_y


Instruction: PyTorch weak_script_method decorator

Input:
I came across some code in an introduction to Word2Vec and PyTorch that I'm not quite familiar with. I haven't seen this type of code structure before.

>>> import torch
>>> from torch import nn

>>> # an Embedding module containing 10 tensors of size 3
>>> embedding = nn.Embedding(10, 3)
>>> # a batch of 2 samples of 4 indices each
>>> input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
>>> embedding(input)

tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])

I'm a little confused about the following line of code.

>>> embedding(input)

I may have inadvertently ignored this syntax in the past, but I don't recall seeing a variable being passed to a class instance before? Referring to the PyTorch documentation where Class Embedding() is defined, is this behaviour enabled with decorator @weak_script_method wrapping def forward()? The code below suggests this may be the case?

>>> torch.manual_seed(2)
>>> torch.eq(embedding(input), embedding.forward(input)).all()

tensor(1, dtype=torch.uint8)

Why is the use of decorator @weak_script_method preferable in this case?

Output:
No, @weak_script_method has nothing to do with it. embedding(input) follows the Python function call syntax, which can be used with both "traditional" functions and with objects which define the __call__(self, *args, **kwargs) magic function. So this code

class Greeter:
    def __init__(self, name):
        self.name = name

    def __call__(self, name):
        print('Hello to ' + name + ' from ' + self.name + '!')

greeter = Greeter('Jatentaki')
greeter('EBB')

will result in Hello to EBB from Jatentaki! being printed to stdout. Similarly, Embedding is an object which you construct by telling it how many embeddings it should contain, what should be their dimensionality, etc, and then, after it is constructed, you can call it like a function, to retrieve the desired part of the embedding.

The reason you do not see __call__ in nn.Embedding source is that it subclasses nn.Module, which provides an automatic __call__ implementation which delegates to forward and calls some extra stuff before and afterwards (see the documentation). So, calling module_instance(arguments) is roughly equivalent to calling module_instance.forward(arguments).

The @weak_script_method decorator has little to do with it. It is related to jit compatibility, and @weak_script_method is a variant of @script_method designed for internal use in PyTorch - the only message for you should be that nn.Embedding is compatible with jit, if you wanted to use it.


Instruction: Best practices for exploration/exploitation in Reinforcement Learning

Input:
My question follows my examination of the code in the PyTorch DQN tutorial, but then refers to Reinforcement Learning in general: what are the best practices for optimal exploration/exploitation in reinforcement learning?

In the DQN tutorial, the steps_done variable is a global variable, and the EPS_DECAY = 200. This means that: after 128 steps, the epsilon threshold = 0.500; after 889 steps, the epsilon threshold = 0.0600; and after 1500 steps, the epsilon threshold = 0.05047.

This might work for the CartPole problem featured in the tutorial – where the early episodes might be very short and the task fairly simple – but what about on more complex problems in which far more exploration is required? For example, if we had a problem with 40,000 episodes, each of which had 10,000 timesteps, how would we set up the epsilon greedy exploration policy? Is there some rule of thumb that’s used in RL work?

Thank you in advance for any help.

Output:
well, for that I guess it is better to use the linear annealed epsilon-greedy policy which updates epsilon based on steps:

EXPLORE = 3000000   #how many time steps to play
FINAL_EPSILON = 0.001 # final value of epsilon
INITIAL_EPSILON = 1.0# # starting value of epsilon

if epsilon > FINAL_EPSILON:
            epsilon -= (INITIAL_EPSILON - FINAL_EPSILON) / EXPLORE


Instruction: I was training the lstm network using pytorch and encountered this error

Input:
I was training the lstm network using pytorch and encountered this error in jupyter notebook.

RuntimeError                              Traceback (most recent call last)
<ipython-input-16-b6b1e0b8cad1> in <module>()
      4
      5 # train the model
----> 6 train(net, encoded, epochs=n_epochs, batch_size=batch_size, seq_length=seq_length, lr=0.001, print_every=10)

<ipython-input-14-43dc0cc515e7> in train(net, data, epochs, batch_size, seq_length, lr, clip, val_frac, print_every)
     55
     56             # calculate the loss and perform backprop
---> 57             loss = criterion(output, targets.view(batch_size*seq_length))
     58             loss.backward()
     59             # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    902     def forward(self, input, target):
    903         return F.cross_entropy(input, target, weight=self.weight,
--> 904                                ignore_index=self.ignore_index, reduction=self.reduction)
    905
    906

~\Anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   1968     if size_average is not None or reduce is not None:
   1969         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 1970     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   1971
   1972

~\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1788                          .format(input.size(0), target.size(0)))
   1789     if dim == 2:
-> 1790         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1791     elif dim == 4:
   1792         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Expected object of scalar type Long but got scalar type Int for argument #2 'target'

Output:
Cast output vector of your network to Long (you have Int) as the error says.

Oh, and please provide Minimal, Complete and Verifiable example next time you ask a question.


Instruction: Differences in SciKit Learn, Keras, or Pytorch

Input:
Are these libraries fairly interchangeable?

Looking here, https://stackshare.io/stackups/keras-vs-pytorch-vs-scikit-learn, it seems the major difference is the underlying framework (at least for PyTorch).

Output:
Yes, there is a major difference.

SciKit Learn is a general machine learning library, built on top of NumPy. It features a lot of machine learning algorithms such as support vector machines, random forests, as well as a lot of utilities for general pre- and postprocessing of data. It is not a neural network framework.

PyTorch is a deep learning framework, consisting of

A vectorized math library similar to NumPy, but with GPU support and a lot of neural network related operations (such as softmax or various kinds of activations)
Autograd - an algorithm which can automatically calculate gradients of your functions, defined in terms of the basic operations
Gradient-based optimization routines for large scale optimization, dedicated to neural network optimization
Neural-network related utility functions

Keras is a higher-level deep learning framework, which abstracts many details away, making code simpler and more concise than in PyTorch or TensorFlow, at the cost of limited hackability. It abstracts away the computation backend, which can be TensorFlow, Theano or CNTK. It does not support a PyTorch backend, but that's not something unfathomable - you can consider it a simplified and streamlined subset of the above.

In short, if you are going with "classic", non-neural algorithms, neither PyTorch nor Keras will be useful for you. If you're doing deep learning, scikit-learn may still be useful for its utility part; aside from it you will need the actual deep learning framework, where you can choose between Keras and PyTorch but you're unlikely to use both at the same time. This is very subjective, but in my view, if you're working on a novel algorithm, you're more likely to go with PyTorch (or TensorFlow or some other lower-level framework) for flexibility. If you're adapting a known and tested algorithm to a new problem setting, you may want to go with Keras for its greater simplicity and lower entry level.


Instruction: How to Tensorize loss of multiple 3D Keypoints

Input:
I have a tensor of ground truth values of 3D points of G=[18000x3], and an output from my network of the same size O=[18000x3].

I need to compute a loss so that I basically have the square root of the distance between each 3D point, summed over all keypoints and normalized over 18000. How do I write this efficiently?

Output:
Just write the expression you propose using the vectorized operations provided by PyTorch. In this case

loss = (O - G).pow(2).sum(axis=1).sqrt().mean()

Check out pow, sum, sqrt and mean.


Instruction: Need to change GPU option to CPU in a python pytorch based code

Input:
The code basically trains the usual MNIST image dataset but it does the training on a GPU. I need to change this option so the code trains the model using my laptop computer. I need to substitute the .cuda() at the second line for the equivalent in CPU.

I know there are many examples online on how to train neural networks using the MNIST database but what is special about this code is that it does the optimization using a PID controller (commonly used in industry) and I need the code as part of my research.

net = Net(input_size, hidden_size, num_classes)
net.cuda()
net.train()
#Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = PIDOptimizer(net.parameters(), lr=learning_rate, weight_decay=0.0001, momentum=0.9, I=I, D=D)
# Train the Model
for epoch in range(num_epochs):
    train_loss_log = AverageMeter()
    train_acc_log = AverageMeter()
    val_loss_log = AverageMeter()
    val_acc_log = AverageMeter()
    for i, (images, labels) in enumerate(train_loader):
        # Convert torch tensor to Variable
        images = Variable(images.view(-1, 28*28).cuda())
        labels = Variable(labels.cuda())

Would need to be able to run the code without using the .cuda() option which is for training using a GPU. Need to run it on my PC.

Here's the source code in case needed.

https://github.com/tensorboy/PIDOptimizer

Many thanks, community!

Output:
It is better to move up to latest pytorch (1.0.x).

With latest pytorch, it is more easy to manage "device".

Below is a simple example.

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#Now send existing model to device.
model_ft = model_ft.to(device)
#Now send input to device and so on.
inputs = inputs.to(device)

With this construct, your code automatically uses appropriate device.

Hope this helps!


Instruction: Element wise calculation breaks autograd

Input:
I am using pytorch to calculate loss for a logistic regression (I know pytorch can do this automatically but I have to make it myself). My function is defined below but the cast to torch.tensor breaks autograd and gives me w.grad = None. Im new to pytorch so Im sorry.

logistic_loss = lambda X,y,w: torch.tensor([torch.log(1 + torch.exp(-y[i] * torch.matmul(w, X[i,:]))) for i in range(X.shape[0])], requires_grad=True)

Output:
Your post isn't very clear on details and this is a monster of a one-liner. I first reworked it to make a minimal, complete, verifiable example. Please correct me if I misunderstood your intentions and please do it yourself next time.

import torch

# unroll the one-liner to have an easier time understanding what's going on
def logistic_loss(X, y, w):
    elementwise = []
    for i in range(X.shape[0]):
        mm = torch.matmul(w, X[i, :])
        exp = torch.exp(-y[i] * mm)
        elementwise.append(torch.log(1 + exp))

    return torch.tensor(elementwise, requires_grad=True)

# I assume that's the excepted dimensions of your input
X = torch.randn(5, 30, requires_grad=True)
y = torch.randn(5)
w = torch.randn(30)

# I assume you backpropagate from a reduced version
# of your sum, because you can't call .backward on multi-dimensional
# tensors
loss = logistic_loss(X, y, w).mean()
loss.mean().backward()
print(X.grad)

The simplest solution to your problem is to replace torch.tensor(elementwise, requires_grad=True) with torch.stack(elementwise). You can think of torch.tensor as a constructor for entirely new tensors, if your tensor is more of a result of some mathematical expression, you should use operations like torch.stack or torch.cat.

That being said, this code is still wildly inefficient because you do manual looping over i. Instead, you could write simply

def logistic_loss_vectorized(X, y, w):
    mm = torch.matmul(X, w)
    exp = torch.exp(-y * mm)

    return torch.log(1 + exp)

which is mathematically equivalent, but will be much faster in practice, because it allows for better parallelization due to lack of explicit looping.

Note that there is still a numerical issue with this code - you're taking a logarithm of an exponential, but the intermediate result, called exp, is likely to attain very high values, causing loss of precision. There are workarounds for that, which is why the loss functions provided by PyTorch are preferable.