Instruction: Pytorch: RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered

Input:
I am running into the following error when trying to train this on this dataset.

Since this is the configuration published in the paper, I am assuming I am doing something incredibly wrong.

This error arrives on a different image every time I try to run training.

C:/w/1/s/windows/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: block: [0,0,0], thread: [6,0,0] Assertion `t >= 0 && t < n_classes` failed.
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1741, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.1.1\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Noam/Code/vision_course/hopenet/deep-head-pose/code/original_code_augmented/train_hopenet_with_validation_holdout.py", line 187, in <module>
    loss_reg_yaw = reg_criterion(yaw_predicted, label_yaw_cont)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\modules\loss.py", line 431, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "C:\Noam\Code\vision_course\hopenet\venv\lib\site-packages\torch\nn\functional.py", line 2204, in mse_loss
    ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
RuntimeError: reduce failed to synchronize: cudaErrorAssert: device-side assert triggered

Any ideas?

Output:
This kind of error generally occurs when using NLLLoss or CrossEntropyLoss, and when your dataset has negative labels (or labels greater than the number of classes). That is also the exact error you are getting Assertion t >= 0 && t < n_classes failed.

This won't occur for MSELoss, but OP mentions that there is a CrossEntropyLoss somewhere and thus the error occurs (the program crashes asynchronously on some other line). The solution is to clean the dataset and ensure that t >= 0 && t < n_classes is satisfied (where t represents the label).

Also, ensure that your network output is in the range 0 to 1 in case you use NLLLoss or BCELoss (then you require softmax or sigmoid activation respectively). Note that this is not required for CrossEntropyLoss or BCEWithLogitsLoss because they implement the activation function inside the loss function. (Thanks to @PouyaB for pointing out).


Instruction: How do I use tensorboard with pytorch?

Input:
I was following the pytorch tensorboard tutorial: https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html.

But I can't even start because of the following error:

from torch.utils.tensorboard import SummaryWriter

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
~/apps/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py in <module>
      1 try:
----> 2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:

ModuleNotFoundError: No module named 'tensorboard'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-4-c8ffdef1cfab> in <module>
----> 1 from torch.utils.tensorboard import SummaryWriter
      2
      3 # default `log_dir` is "runs" - we'll be more specific here
      4 writer = SummaryWriter('runs/fashion_mnist_experiment_1')

~/apps/anaconda3/envs/torch/lib/python3.7/site-packages/torch/utils/tensorboard/__init__.py in <module>
      2     from tensorboard.summary.writer.record_writer import RecordWriter  # noqa F401
      3 except ImportError:
----> 4     raise ImportError('TensorBoard logging requires TensorBoard with Python summary writer installed. '
      5                       'This should be available in 1.14 or above.')
      6 from .writer import FileWriter, SummaryWriter  # noqa F401

ImportError: TensorBoard logging requires TensorBoard with Python summary writer installed. This should be available in 1.14 or above.

I installed pytorch 1.14 via conda. Am I supposed to install something else?

Output:
That tutorial should probably let you know that you need to install tensorboard. Take a look at the pytorch tensorboard docs which explains that you need to install tensorboard first.

Basically you can install tensorboard using

pip install tensorboard

and then start the tensorboard server by running

tensorboard --logdir=runs

The runs directory is where your summary writer will write to and it's where the tensorboard server reads from to know what to visualize.


Instruction: Best way to vectorize generating a batch of randomly rotated matrices in Numpy/PyTorch?

Input:
I’d like to generate batches of randomly rotated matrices based on an initial starting matrix (which has a shape of, for example, (4096, 3)), where the rotation applied to each matrix in the batch is randomly chosen from a group of rotation matrices (in my code in the original post, I only want to randomly select from 8 possible rotation angles). Therefore, what I end up with is a tensor of shape (batch_size, 4096, 3).

My current approach is that I pre-make the possible rotated matrices (since I’m only dealing with 8 possible random rotations), and then use a for loop to generate the batch by randomly picking one of the eight pre-made rotated matrices for each item in the batch. This isn’t super efficient, so I was hoping to vectorize the whole process somehow.

Right now, this is how I loop over a batch to generate a batch of rotated matrices one by one:

for view_i in range(batch_size):
        # Get rotated view grid points randomly
        idx = torch.randint(0, 8, (1,))
        pointsf = rotated_points[idx]

In the code below, I generate a pre-made set of random rotation matrices that get randomly selected from in a for-loop over the batch.

The make_3d_grid function generates a (grid_dim * grid_dim * grid_dim, 3) shaped matrix (basically a 2D array of x, y, z coordinate points). The get_rotation_matrix function returns a (3, 3) rotation matrix, where theta is used for rotation around the x-axis.

rotated_points = []
grid_dim = 16

pointsf = make_3d_grid((-1,)*3, (1,)*3, (grid_dim,)*3)

view_angles = torch.tensor([0, np.pi / 4.0, np.pi / 2.0, 3 * np.pi / 4.0, np.pi, 5 * np.pi / 4.0, 3 * np.pi / 2.0, 7 * np.pi / 4.0])

for i in range(len(view_angles)):
    theta = view_angles[i]

    rot = get_rotation_matrix(theta, torch.tensor(0.0), torch.tensor(0.0))

    pointsf_rot = torch.mm(pointsf, rot)

    rotated_points.append(pointsf_rot)

Any help in vectorizing this would be greatly appreciated! If code for this can be done in Numpy that works fine too, since I can convert it to PyTorch myself.

Output:
You can pre-generate your rotation matrices as a (batch_size, 3, 3) array, and then multiply by your (N, 3) points array broadcasted to (batch_size, N, 3).

rotated_points = np.dot(pointsf, rots)

np.dot will sum-product over the last axis of pointsf and the second-to-last axis of rots, putting the dimensions of pointsf first. This means that your result will be of shape (N, batch_size, 3) rather than (batch_size, N, 3). You can of course fix this with a simple axis swap:

rotated_points = np.dot(pointsf, rots).transpose(1, 0, 2)

OR

rotated_points = np.swapaxes(np.dot(pointsf, rots), 0, 1)

I would suggest, however, that you make rots be the inverse (transposed) rotation matrices from what you had before. In that case, you can just compute:

rotated_points = np.dot(transposed_rots, pointsf.T)

You should be able to convert np.dot to torch.mm fairly trivially.


Instruction: Select specific rows of 2D PyTorch tensor

Input:
Suppose I have a 2D tensor looking something like this:

[[44, 50, 1, 32],
.
.
.
[7, 13, 90, 83]]

and a list of  row indices that I want to select that looks something like this [0, 34, 100, ..., 745]. How can I go through and create a new tensor that contains only the rows whose indices are contained in the array?

Output:
You could select like with numpy

import torch
x = torch.Tensor([[1, 2, 3, 4],
                  [5, 6, 7, 8],
                  [9, 8, 7, 6],
                  [5, 4, 2, 1]])

indices = [0, 3]

print(x[indices])
# tensor([[1., 2., 3., 4.],
#         [5., 4., 2., 1.]])


Instruction: How can I efficiently modify/make pairwise distance matrix?

Input:
    x_norm = (x**2).sum(1).view(-1, 1)
    if y is not None:
        y_norm = (y**2).sum(1).view(1, -1)
    else:
        y = x
        y_norm = x_norm.view(1, -1)
    dist = (x_norm + y_norm - 2.0 * torch.mm(x, torch.transpose(y, 0, 1)))
    return dist

Above is a code used to calculate pairwise distance matrix(M*N) between x (M points) and y (N points).

I hope to make pairwise distance matrix that has 0 element when distance between two points is larger than specific value 'T'.

In this case, what should I do?

Thanks

Output:
I think you are looking for torch.where:

new_dist = troch.where(dist > T, dist, 0.)


Instruction: Operator translate error occurs when I try to convert onnx file to caffe2

Input:
I train a boject detection model on pytorch, and I have exported to onnx file.

And I want to convert it to caffe2 model :

import onnx
import caffe2.python.onnx.backend as onnx_caffe2_backend

# Load the ONNX ModelProto object. model is a standard Python protobuf object
model = onnx.load("CPU4export.onnx")

# prepare the caffe2 backend for executing the model this converts the ONNX model into a
# Caffe2 NetDef that can execute it. Other ONNX backends, like one for CNTK will be
# availiable soon.
prepared_backend = onnx_caffe2_backend.prepare(model)

# run the model in Caffe2

# Construct a map from input names to Tensor data.
# The graph of the model itself contains inputs for all weight parameters, after the input image.
# Since the weights are already embedded, we just need to pass the input image.
# Set the first input.
W = {model.graph.input[0].name: x.data.numpy()}

# Run the Caffe2 net:
c2_out = prepared_backend.run(W)[0]

# Verify the numerical correctness upto 3 decimal places
np.testing.assert_almost_equal(torch_out.data.cpu().numpy(), c2_out, decimal=3)

print("Exported model has been executed on Caffe2 backend, and the result looks good!")

I always got this error :

RuntimeError: ONNX conversion failed, encountered 1 errors:

Error while processing node: input: "90"
input: "91"
output: "92"
op_type: "Resize"
attribute {
  name: "mode"
  s: "nearest"
  type: STRING
}
. Exception: Don't know how to translate op Resize

How can I solve it ?

Output:
The problem is that the Caffe2 ONNX backend does not yet support the export of the Resize operator.

Please raise an issue on the Caffe2 / PyTorch github -- there's an active community of developers who should be able to address this use case.


Instruction: Pytorch Change the learning rate based on number of epochs

Input:
When I set the learning rate and find the accuracy cannot increase after training few epochs
optimizer = optim.Adam(model.parameters(), lr = 1e-4)

n_epochs = 10
for i in range(n_epochs):

    // some training here

If I want to use a step decay: reduce the learning rate by a factor of 10 every 5 epochs, how can I do so?

Output:
You can use learning rate scheduler torch.optim.lr_scheduler.StepLR
import torch.optim.lr_scheduler.StepLR
scheduler = StepLR(optimizer, step_size=5, gamma=0.1)

Decays the learning rate of each parameter group by gamma every step_size epochs see docs here
Example from docs
# Assuming optimizer uses lr = 0.05 for all groups
# lr = 0.05     if epoch < 30
# lr = 0.005    if 30 <= epoch < 60
# lr = 0.0005   if 60 <= epoch < 90
# ...
scheduler = StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(100):
    train(...)
    validate(...)
    scheduler.step()

Example:
import torch
import torch.optim as optim

optimizer = optim.SGD([torch.rand((2,2), requires_grad=True)], lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

for epoch in range(1, 21):
    scheduler.step()
    print('Epoch-{0} lr: {1}'.format(epoch, optimizer.param_groups[0]['lr']))
    if epoch % 5 == 0:print()

Epoch-1 lr: 0.1
Epoch-2 lr: 0.1
Epoch-3 lr: 0.1
Epoch-4 lr: 0.1
Epoch-5 lr: 0.1

Epoch-6 lr: 0.010000000000000002
Epoch-7 lr: 0.010000000000000002
Epoch-8 lr: 0.010000000000000002
Epoch-9 lr: 0.010000000000000002
Epoch-10 lr: 0.010000000000000002

Epoch-11 lr: 0.0010000000000000002
Epoch-12 lr: 0.0010000000000000002
Epoch-13 lr: 0.0010000000000000002
Epoch-14 lr: 0.0010000000000000002
Epoch-15 lr: 0.0010000000000000002

Epoch-16 lr: 0.00010000000000000003
Epoch-17 lr: 0.00010000000000000003
Epoch-18 lr: 0.00010000000000000003
Epoch-19 lr: 0.00010000000000000003
Epoch-20 lr: 0.00010000000000000003

More on How to adjust Learning Rate - torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.


Instruction: How to translate or convert code from Python Pytorch to C++ Libtorch

Input:
I couldn't find the equivalent C++ calls in Libtorch (Pytorch C++ Frontend) for my Python Pytorch code.

The documentation according to my searchs (Pytorch Discuss) does not exist yet for my code. I wonder if someone can guide me with the following  pieces (bellow).

I cutted pieces where I've been having more crashes (wrong usage) of Libtorch C++.

import torch as th

th.set_grad_enabled(False)
...
X = th.zeros((nobs, 3+p), device=dev, dtype=th.float32)
y = th.tensor(indata, device=dev, dtype=th.float32)
diffilter = th.tensor([-1., 1.], device=dev, dtype=th.float32).view(1, 1, 2)
dy = th.conv1d(y.view(1, 1, -1), diffilter).view(-1)
z = dy[p:].clone()
...
# X matrix
X[:, 0] = 1
X[:, 1] = th.arange(p+1, n)
X[:, 2] = y[p:-1]
...
# master X
Xm = th.zeros((nobsadf, 3+p), device=th.device('cpu'), dtype=th.float32)
...
# batch matrix, vector and observations
Xbt = th.zeros(batch_size, adfs_count, nobsadf, (3+p), device=th.device('cpu'), dtype=th.float32)
...
t = 0 # start line for master main X OLS matrix/ z vector
for i in range(nbatchs):
    for j in range(batch_size): # assembly batch_size matrixes
        Xm[:] = X[t:t+nobsadf]
        ...
        Xbt[j, :, :, :] = Xm.repeat(adfs_count, 1).view(adfs_count, nobsadf, (3+p))
        for k in range(adfs_count):
            Xbt[j, k, :k, :] = 0
            nobt[j, k] = float(nobsadf-k-(p+3))

Output:
After suffering a lot!...
I learned to better use the Pytorch Discuss forum for Pytorch and Libtorch information. Using the tag C++ for example.
Unfortunantly, there is the oficial source of information (altough quite messy). This is the reason why I am sharing my answer here in SO.
namespace th = torch;
...
// th.set_grad_enabled(False)
th::NoGradGuard guard; // or same as with torch.no_grad(): block
...
auto dtype_option = th::TensorOptions().dtype(th::kFloat32);
//X = th.zeros((nobs, 3+p), device=dev, dtype=th.float32)
//y = th.tensor(indata, device=dev, dtype=th.float32)
//diffilter = th.tensor([-1., 1.], device=dev, dtype=th.float32).view(1, 1, 2)
//dy = th.conv1d(y.view(1, 1, -1), diffilter).view(-1)
//z = dy[p:].clone()
auto X = th::zeros({nobs, 3+p}, dtype_option);
auto y = th::from_blob(signal, {n}, dtype_option);
auto diffilter = th::tensor({-1, 1}, dtype_option).view({ 1, 1, 2 }); // first difference filter
auto dy = th::conv1d(y.view({ 1, 1, -1 }), diffilter).view({ -1 });
auto z = dy.slice(0, p).clone();
...
// X[:, 0] = 1 # drift
// X[:, 1] = th.arange(p+1, n)
// X[:, 2] = y[p:-1]
// create acessors to fill in the matrix
auto ay = y.accessor<float, 1>(); // <1> dimension
auto aX = X.accessor<float, 2>(); // <2> dimension
for (auto i = 0; i < nobs; i++) {
    aX[i][0] = 1;
    aX[i][1] = p + 1 + i;
    aX[i][2] = ay[p+i];
}
...
// Xm = th.zeros((nobsadf, 3+p), device=th.device('cpu'), dtype=th.float32)
auto Xm = th::zeros({ nobsadf, 3 + p }, dtype_option.device(th::Device(th::kCPU)));
// Xbt = th.zeros(batch_size, adfs_count, nobsadf, (3+p), device=th.device('cpu'), dtype=th.float32)
auto Xbt = th::zeros({ batch_size, adfs_count, nobsadf, (3 + p) }, dtype_option.device(th::Device(th::kCPU)));
...
// this acessor will be used in the inner for loop k
auto anobt = nobt.accessor<float, 2>();
auto tline = 0; // start line for master main X OLS matrix/ z vector
for (int i = 0; i < nbatchs; i++){
    for (int j = 0; j < batch_size; j++){ // assembly batch_size matrixes
        // Xm[:] = X[t:t+nobsadf]
        Xm.copy_(X.narrow(0, tline, nobsadf));
        ...
        // Xbt[j, :, :, :] = Xm.repeat(adfs_count, 1).view(adfs_count, nobsadf, (3+p))
        auto Xbts = Xbt.select(0, j);
        Xbts.copy_(Xm.repeat({ adfs_count, 1 }).view({ adfs_count, nobsadf, (3 + p) }));
        for (int k = 0; k < adfs_count; k++) {
            // Xbt[j, k, :k, :] = 0
            // nobt[j][k] = float(nobsadf - k - (p + 3));
            Xbts.select(0, k).narrow(0, 0, k).fill_(0);
            anobt[j][k] = float(nobsadf - k - (p + 3));
        }
        tline++;
    }
}

Probably there is a better or faster way of coding but the code above fully works. Fell free to make suggestions to improve my code.
C++ Signatures of commom functions above
Tensor Tensor::slice(int64_t dim, int64_t start, int64_t end, int64_t step)

Tensor Tensor::narrow(int64_t dim, int64_t start, int64_t length)

Tensor Tensor::select(int64_t dim, int64_t index)

Tensor & Tensor::copy_(const Tensor & src, bool non_blocking=false)

Further notes:
Almost all C++ function have Pytorch Python equivalent. So here is my golden tip:

Translate your python script using C++ equivalent functions like copy_, narrow, slice testing it (to make sure it works) than just go to C++ replicating everything.


Instruction: Why does roi_align not seem to work in pytorch?

Input:
I am a pytorch beginner. It seems that there is a bug in the RoIAlign module in pytorch. The code is simple but the result is out of my expectation.

code:

import torch
from torchvision.ops import RoIAlign

if __name__ == '__main__':
    output_size = (3,3)
    spatial_scale = 1/4
    sampling_ratio = 2

    #x.shape:(1,1,6,6)
    x = torch.FloatTensor([[
        [[1,2,3,4,5,6],
        [7,8,9,10,11,12],
        [13,14,15,16,17,18],
        [19,20,21,22,23,24],
        [25,26,27,28,29,30],
        [31,32,33,34,35,36],],
    ]])

    rois = torch.tensor([
        [0,0.0,0.0,20.0,20.0],
    ])
    channel_num = x.shape[1]
    roi_num = rois.shape[0]

    a = RoIAlign(output_size, spatial_scale=spatial_scale, sampling_ratio=sampling_ratio)
    ya = a(x, rois)
    print(ya)

output:

tensor([[[[ 6.8333,  8.5000, 10.1667],
          [16.8333, 18.5000, 20.1667],
          [26.8333, 28.5000, 30.1667]]]])

But in this case shouldn't it be an average pooling operation on every 2x2 cell, like:

tensor([[[[ 4.5000,  6.5000, 8.5000],
          [16.5000, 18.5000, 20.5000],
          [28.5000, 30.5000, 32.5000]]]])

My torch version is 1.3.0 with python3.6 and cuda 10.1, on Ubuntu16. I have been troubled for two days and I couldn't appreciate it more if anyone could help me.

Output:
Intuitive Interpretation
There are some complications with image coordinates. We need to take into account the fact that pixels are actually squares and not points in space. We interpret the center of the pixel to be the integer coordinates, so for example (0,0) refers to the center of the first pixel while (-0.5, -0.5) refers to the upper left corner of the first pixel. Basically this is why you aren't getting the results you expect. An roi that goes from (0,0) to (5,5) actually cuts through the border pixels and leads to sampling between pixels when performing roi align. If instead we define our roi from (-0.5, -0.5) to (5.5, 5.5) then we get the expected result. Accounting for the scale factor this translates to an roi from (-2, -2) to (22, 22).
import torch
from torchvision.ops import RoIAlign

output_size = (3, 3)
spatial_scale = 1 / 4
sampling_ratio = 2

x = torch.FloatTensor([[
    [[1,  2,  3,  4,  5,  6 ],
     [7,  8,  9,  10, 11, 12],
     [13, 14, 15, 16, 17, 18],
     [19, 20, 21, 22, 23, 24],
     [25, 26, 27, 28, 29, 30],
     [31, 32, 33, 34, 35, 36]]
]])

rois = torch.tensor([
    [0, -2.0, -2.0, 22.0, 22.0],
])

a = RoIAlign(output_size, spatial_scale=spatial_scale, sampling_ratio=sampling_ratio)
ya = a(x, rois)
print(ya)

which results in
tensor([[[[ 4.5000,  6.5000,  8.5000],
          [16.5000, 18.5000, 20.5000],
          [28.5000, 30.5000, 32.5000]]]])

Alternative interpretation
Partitioning the interval [0, 5] into 3 intervals of equal length gives [0, 1.67], [1.67, 3.33], [3.33, 5]. So the boundaries of the output window will fall into these coordinates. Clearly this won't lead to nice sampling results.


Instruction: forcing pytorch to use gpu

Input:
i've recently followed a tutorial here https://www.provideocoalition.com/automatic-rotoscopingfor-free/

And ended with a functional bit of code that generate masks outlining interestings objects.

But now, i want ot run it on my gpu, since cpu is way too slow.

I have CUDA installed and all, but pytorch refuses to use it. I've used most tricks like setting torch.device and all, but to no avail; pytorch keep using 0 gpu.

here's the code :

from PIL import Image
import torch
import torchvision.transforms as T
from torchvision import models
import numpy as np

fcn = None

device = torch.device('cuda')
torch.cuda.set_device(0)
print('Using device:', device)
print()

if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')

def getRotoModel():
    global fcn
    #fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()
    fcn = models.segmentation.deeplabv3_resnet101(pretrained=True).eval()

# Define the helper function
def decode_segmap(image, nc=21):

    label_colors = np.array([(0, 0, 0),  # 0=background
                           # 1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle
               (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128),
               # 6=bus, 7=car, 8=cat, 9=chair, 10=cow
               (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0),
               # 11=dining table, 12=dog, 13=horse, 14=motorbike, 15=person
               (192, 128, 0), (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128),
               # 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor
               (0, 64, 0), (128, 64, 0), (0, 192, 0), (128, 192, 0), (0, 64, 128)])

    r = np.zeros_like(image).astype(np.uint8)
    g = np.zeros_like(image).astype(np.uint8)
    b = np.zeros_like(image).astype(np.uint8)

    for l in range(0, nc):
        idx = image == l
        r[idx] = label_colors[l, 0]
        g[idx] = label_colors[l, 1]
        b[idx] = label_colors[l, 2]

    rgb = np.stack([r, g, b], axis=2)
    return rgb

def createMatte(filename, matteName, size):
    img = Image.open(filename)
    trf = T.Compose([T.Resize(size),
                     T.ToTensor(),
                     T.Normalize(mean = [0.485, 0.456, 0.406],
                                 std = [0.229, 0.224, 0.225])])
    inp = trf(img).unsqueeze(0)
    if (fcn == None): getRotoModel()
    out = fcn(inp)['out']
    om = torch.argmax(out.squeeze(), dim=0).detach().cpu().numpy()
    rgb = decode_segmap(om)
    im = Image.fromarray(rgb)
    im.save(matteName)

What could i do ? thanks.

Output:
If everything is set up correctly you just have to move the tensors you want to process on the gpu to the gpu. You can try this to make sure it works in general

import torch
t = torch.tensor([1.0]) # create tensor with just a 1 in it
t = t.cuda() # Move t to the gpu
print(t) # Should print something like tensor([1], device='cuda:0')
print(t.mean()) # Test an operation just to be sure

You already have a device variable so instead of .cuda() you can just use .to(device). Which is also the preferable way to do it so you can just switch between cpu and gpu by setting one variable.


Instruction: Tracing back deprecated warning in pytorch

Input:
I am training yolov3 on my data using this code here :
https://github.com/cfotache/pytorch_custom_yolo_training/

But I am getting this annoying deprecation warnings

Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at /pytorch/aten/src/ATen/native/IndexingUtils.h:20)

I tried using python3 -W ignore train.py
I tried adding :

import warnings
warnings.filterwarnings('ignore')

but the warning is still persistent.

I found this piece of code on here on stackoverflow that prints that stack on warnings ,

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None, line=None):

    log = file if hasattr(file,'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(warnings.formatwarning(message, category, filename, lineno, line))

warnings.showwarning = warn_with_traceback

and here's what I get :

  File "/content/pytorch_custom_yolo_training/train.py", line 102, in <module>
   loss = model(imgs, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 267, in forward
    x, *losses = module[0](x, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/content/pytorch_custom_yolo_training/models.py", line 203, in forward
    loss_x = self.mse_loss(x[mask], tx[mask])
  File "/usr/lib/python3.6/warnings.py", line 99, in _showwarnmsg
    msg.file, msg.line)
  File "/content/pytorch_custom_yolo_training/train.py", line 29, in warn_with_traceback
    traceback.print_stack(file=log)
  /pytorch/aten/src/ATen/native/IndexingUtils.h:20: UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead.

Going to the files and functions mentioned in the stack , I don't find any uint8 .
What can I to solve the problem or even stop getting these warnings ?

Output:
Found the problem.
line : loss_x = self.mse_loss(x[mask], tx[mask])
the mask variable was a ByteTensor which is deprecated . Just replaced it with a BoolTensor


Instruction: Number of Conv2d Layers and Filters for Small Image Classification Task

Input:
If I'm working with a dataset where I have ~100,000 training images and ~20,000 validation images, each of size 32 x 32 x 3, how does the size and the dimensions of my dataset affect the number of Conv2d layers I have in my CNN? My intuition is to use fewer Conv2d layers, 2-3, because any more than 3 layers will be working with parts of the image that are too small to gain relevant data from.

In addition, does it make sense to have layers with a large number of filters, >128? My thought is that when dealing with small images, it doesn't make sense to have a large number of parameters.

Output:
Since you have the exact input size like the images in Cifar10 and Cifar100 just have a look what people tried out.

In general you can start with something like a ResNet18. Also I don't quite understand why you say

  because any more than 3 layers will be working with parts of the image that are too small to gain relevant data from.

As long as you don't downsample using something like max pooling or a conv with padding 1 and stride 2. The size of 32x32 will be the same and only the number of channels will change depending on the network.

Designing networks is almost always at looking what did other people do and what worked for them and starting from there. You almost never want to do it from scratch on your own, since the iteration cycles are just to long and models released by researches from Google, Facebook ... had way more resources then you will ever have to find something good.


Instruction: Sharing GPU memory between process on a same GPU with Pytorch

Input:
I'm trying to implement an efficient way of doing concurrent inference in Pytorch.

Right now, I start 2 processes on my GPU (I have only 1 GPU, both process are on the same device). Each process load my Pytorch model and do the inference step.

My problem is that my model takes quite some space on the memory. I have 12Gb of memory on the GPU, and the model takes ~3Gb of memory alone (without the data). Which means together, my 2 processes takes 6Gb of memory just for the model.

Now I was wondering if it's possible to load the model only once, and use this model for inference on 2 different processes. What I want is only 3Gb of memory is consumed by the model, but still have 2 processes.

I came accross this answer mentioning IPC, but as far as I understood it means the process #2 will copy the model from process #1, so I will still end up with 6Gb allocated for the model.

I also checked on the Pytorch documentation, about DataParallel and DistributedDataParallel, but it seems not possible.

This seems to be what I want, but I couldn't find any code example on how to use with Pytorch in inference mode.

I understand this might be difficult to do such a thing for training, but please note I'm only talking about the inference step (the model is in read-only mode, no need to update gradients). With this assumption, I'm not sure if it's possible or not.

Output:
The GPU itself has many threads. When performing an array/tensor operation, it uses each thread on one or more cells of the array. This is why it seems that an op that can fully utilize the GPU should scale efficiently without multiple processes -- a single GPU kernel is already massively parallelized.

In a comment you mentioned seeing better results with multiple processes in a small benchmark. I'd suggest running the  benchmark with more jobs to ensure warmup, ten kernels seems like too small of a test. If you're finding a thorough representative benchmark to run faster consistently though, I'll trust good benchmarks over my intuition.

My understanding is that kernels launched on the default CUDA stream get executed sequentially. If you want them to run in parallel, I think you'd need multiple streams. Looking in the PyTorch code, I see code like getCurrentCUDAStream() in the kernels, which makes me think the GPU will still run any PyTorch code from all processes sequentially.

This NVIDIA discussion suggests this is correct:

https://devtalk.nvidia.com/default/topic/1028054/how-to-launch-cuda-kernel-in-different-processes/

Newer GPUs may be able to run multiple kernels in parallel (using MPI?) but it seems like this is just implemented with time slicing under the hood anyway, so I'm not sure we should expect higher total throughput:

How do I use Nvidia Multi-process Service (MPS) to run multiple non-MPI CUDA applications?

If you do need to share memory from one model across two parallel inference calls, can you just use multiple threads instead of processes, and refer to the same model from both threads?

To actually get the GPU to run multiple kernels in parallel, you may be able to use nn.Parallel in PyTorch. See the discussion here:
https://discuss.pytorch.org/t/how-can-l-run-two-blocks-in-parallel/61618/3


Instruction: How to get indices of multiple elements in a 2D tensor, in a GPU friendly way?

Input:
This question is similar to that already answered here, but that question does not address how to retrieve the indices of multiple elements.

I have a 2D tensor points with many rows and a small number of columns, and would like to get a tensor containing the row indices of all the elements in that tensor. I know what elements are present in points beforehand; It contains integer elements ranging from 0 to 999, and I can make a tensor using the range function to reflect the set of possible elements. The elements may be in any of the columns.

How can I retrieve the row indices where each element appears in my tensor in a way that avoids looping or using numpy, so I can do this quickly on a GPU?

I am looking for something like (points == elements).nonzero()[:,1]

Thanks!

Output:
try torch.cat([(t == i).nonzero() for i in elements_to_compare])

>>> import torch
>>> t = torch.empty((15,4)).random_(0, 999)
>>> t
tensor([[429., 833., 393., 828.],
        [555., 893., 846., 909.],
        [ 11., 861., 586., 222.],
        [232.,  92., 576., 452.],
        [171., 341., 851., 953.],
        [ 94.,  46., 130., 413.],
        [243., 251., 545., 331.],
        [620.,  29., 194., 176.],
        [303., 905., 771., 149.],
        [482., 225.,   7., 315.],
        [ 44., 547., 206., 299.],
        [695.,   7., 645., 385.],
        [225., 898., 677., 693.],
        [746.,  21., 505., 875.],
        [591., 254.,  84., 888.]])

>>> torch.cat([(t == i).nonzero() for i in [7,385]])
tensor([[ 9,  2],
        [11,  1],
        [11,  3]])

>>> torch.cat([(t == i).nonzero()[:,1] for i in [7,385]])
tensor([2, 1, 3])

Numpy:

>>> np.nonzero(np.isin(t, [7,385]))
(array([ 9, 11, 11], dtype=int64), array([2, 1, 3], dtype=int64))

>>> np.nonzero(np.isin(t, [7,385]))[1]
array([2, 1, 3], dtype=int64)


Instruction: Difference between Keras' BatchNormalization and PyTorch's BatchNorm2d?

Input:
I've a sample tiny CNN implemented in both Keras and PyTorch. When I print summary of both the networks, the total number of trainable parameters are same but total number of parameters and number of parameters for Batch Normalization don't match.

Here is the CNN implementation in Keras:

inputs = Input(shape = (64, 64, 1)). # Channel Last: (NHWC)

model = Conv2D(filters=32, kernel_size=(3, 3), padding='SAME', activation='relu', input_shape=(IMG_SIZE, IMG_SIZE, 1))(inputs)
model = BatchNormalization(momentum=0.15, axis=-1)(model)
model = Flatten()(model)

dense = Dense(100, activation = "relu")(model)
head_root = Dense(10, activation = 'softmax')(dense)

And the summary printed for above model is:

Model: "model_8"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_9 (InputLayer)         (None, 64, 64, 1)         0
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 64, 64, 32)        320
_________________________________________________________________
batch_normalization_2 (Batch (None, 64, 64, 32)        128
_________________________________________________________________
flatten_3 (Flatten)          (None, 131072)            0
_________________________________________________________________
dense_11 (Dense)             (None, 100)               13107300
_________________________________________________________________
dense_12 (Dense)             (None, 10)                1010
=================================================================
Total params: 13,108,758
Trainable params: 13,108,694
Non-trainable params: 64
_________________________________________________________________

Here's the implementation of the same model architecture in PyTorch:

# Image format: Channel first (NCHW) in PyTorch
class CustomModel(nn.Module):
def __init__(self):
    super(CustomModel, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(in_channels=1, out_channels=32, kernel_size=(3, 3), padding=1),
        nn.ReLU(True),
        nn.BatchNorm2d(num_features=32),
    )
    self.flatten = nn.Flatten()
    self.fc1 = nn.Linear(in_features=131072, out_features=100)
    self.fc2 = nn.Linear(in_features=100, out_features=10)

def forward(self, x):
    output = self.layer1(x)
    output = self.flatten(output)
    output = self.fc1(output)
    output = self.fc2(output)
    return output

And following is the output of summary of the above model:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 64, 64]             320
              ReLU-2           [-1, 32, 64, 64]               0
       BatchNorm2d-3           [-1, 32, 64, 64]              64
           Flatten-4               [-1, 131072]               0
            Linear-5                  [-1, 100]      13,107,300
            Linear-6                   [-1, 10]           1,010
================================================================
Total params: 13,108,694
Trainable params: 13,108,694
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 0.02
Forward/backward pass size (MB): 4.00
Params size (MB): 50.01
Estimated Total Size (MB): 54.02
----------------------------------------------------------------

As you can see in above results, Batch Normalization in Keras has more number of parameters than PyTorch (2x to be exact). So what's the difference in above CNN architectures? If they are equivalent, then what am I missing here?

Output:
Keras treats as parameters (weights) many things that will be "saved/loaded" in the layer.

While both implementations naturally have the accumulated "mean" and "variance" of the batches, these values are not trainable with backpropagation.

Nevertheless, these values are updated every batch, and Keras treats them as non-trainable weights, while PyTorch simply hides them. The term "non-trainable" here means "not trainable by backpropagation", but doesn't mean the values are frozen.

In total they are 4 groups of "weights" for a BatchNormalization layer. Considering the selected axis (default = -1, size=32 for your layer)

scale (32) - trainable
offset (32) - trainable
accumulated means (32) - non-trainable, but updated every batch
accumulated std   (32) - non-trainable, but updated every batch

The advantage of having it like this in Keras is that when you save the layer, you also save the mean and variance values the same way you save all other weights in the layer automatically. And when you load the layer, these weights are loaded together.


Instruction: Pytorch - Porting @ Operator

Input:
I have the following line of code I want to port to Torch Matmul

rotMat = xmat @ ymat @ zmat

Can I know if this is the correct ordering:

rotMat = torch.matmul(xmat, torch.matmul(ymat, zmat))

Output:
According to the python docs on operator precedence the @ operator has left-to-right associativity

https://docs.python.org/3/reference/expressions.html#operator-precedence

  Operators in the same box group left to right (except for exponentiation, which groups from right to left).

Therefore the equivalent operation is

rotMat = torch.matmul(torch.matmul(xmat, ymat), zmat)

Though keep in mind that matrix multiplication is associative (mathematically) so you shouldn't see much of a difference in the result if you do it the other way. Generally you want to associate in the way that results in the fewest computational steps. For example using the naive matrix multiplication algorithm, if X is 1x10, Y is 10x100 and Z is 100x1000 then the difference between

(X @ Y) @ Z

and

X @ (Y @ Z)

is about 1*10*100 + 1*100*1000 = 101,000 multiplication/addition operations for the first versus 10*100*1000 + 1*10*1000 = 1,001,000 operations for the second. Though these have the same result (ignoring rounding errors) the second version will be about 10 x slower!

As pointed out by @Szymon Maszke pytorch tensors also support the @ operator so you can still use

xmat @ ymat @ zmat

in pytorch.


Instruction: How to solve the RuntimeError when using torch.utils.tensorboard to add a graph

Input:
I am trying to use tensorboard to visualize my pytorch model and encounter a problem. The input tensor's shape is (-1, 1, 20, 15) and the output tensor's shape is (-1, 6). My model combines a list of 5 convolutional networks.

packages:

python: 3.7.6
pytorch: 1.4.0
tensorboard: 2.1.0

The pytorch model is as below:

import torch
from torch import nn
from torch.nn import functional as F
class MyModel(nn.Module):
    """example"""

    def __init__(self, nchunks=[2, 5, 3, 2, 3], resp_size=6):
        super().__init__()
        self.nchunks = nchunks
        self.conv = [nn.Conv2d(1, 2, (2, x)) for x in nchunks]
        self.pool = nn.Sequential(
            nn.AdaptiveMaxPool1d(output_size=10), nn.Flatten(start_dim=1)
        )
        self.bn = nn.BatchNorm1d(100)
        self.fc1 = nn.Linear(100, 100)
        self.fc2 = nn.Linear(100, 100)
        self.fc3 = nn.Linear(100, resp_size)

    def forward(self, x):
        xi = torch.split(x, self.nchunks, dim=3)
        xi = [f(subx.float()).view(-1, 2, 19) for f, subx in zip(self.conv, xi)]
        xi = [self.pool(subx) for subx in xi]
        xi = torch.cat(xi, dim=1)
        xi = self.bn(xi)
        xi = F.relu(self.fc1(xi))
        xi = F.relu(self.fc2(xi))
        xi = self.fc3(xi)
        return xi

Here is the code for the tensorboard summary writer:

from torch.utils.tensorboard import SummaryWriter
x = torch.rand((5,1,20,15))
model = MyModel()
writer = SummaryWriter('logs')
writer.add_graph(model, x)

Such an error is returned:

RuntimeError: Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient
Tensor:
(1,1,.,.) =
 -0.2108 -0.4986
 -0.4009 -0.1910

(2,1,.,.) =
  0.2383 -0.4147
  0.2642  0.0456
[ torch.FloatTensor{2,1,2,2} ]

I guess the model has some issues, but I am not sure what happens.

This similar github issue does not relate to my problem because I am not using multi GPUs.

Output:
I solved the problem by replacing

[nn.Conv2d(1, 2, (2, x)) for x in nchunks]

with

nn.ModuleList([nn.Conv2d(1, 2, (2, x)) for x in nchunks])


Instruction: Error in loading Glove vectors using TorchText on Kaggle Kernel

Input:
I am trying to load the Glove embedding vectors in kaggle kernel using TorchText lib.

from torchtext import vocab
vec = vocab.Vectors('glove.6B.100d.txt', '../input/glove6b100dtxt/')

I am getting the following error:

OSError: [Errno 30] Read-only file system: '../input/glove6b100dtxt/glove.6B.100d.txt.pt'

Adding screenshot for more clarification:

Output:
This happens when you are using glove embeddings as part of the input from other datasets. You don't have the right access to the input folder so the workaround is as follows :
Now we will load the Glove Embedding and move it out to the working directory
!cp -r ../input/glove-embeddings/ ../kaggle/working/glove-embeddings

Once moved, change the location of access
vec = vocab.Vectors('glove.6B.100d.txt', '../kaggle/working/glove-embeddings')


Instruction: How to implement a gaussian renderer with mean and variance values as input in any deep modeling framework (needs to be back-propagable)

Input:
Imagine a typical auto-encoder-decoder model. However, instead of a general decoder where deconvoutions together with upscaling are used to create/synthesize a tensor similar to the model's input, I need to implement a structured/custom decoder.

Here, I need the decoder to take its input, e.g. a 10x2 tensor where each row represents x,y positions or coordinates, and render a fixed predefined size image where there are 10 gaussian distributions generated at the location specified by the input.

In another way, I need to create an empty fixed sized tensor, fill the locations specified by the 10 coordinates a value 1, and then sweep a gaussian kernel over the whole tensor. For example, imagine the following 1-d scenario. let the input to the whole model be a vector of size 10. if the input to the decoder is [3, 7], which are two x-coordinates (0-indexing), and the gaussian kernel of size 3 that we want to use is [0.28, 0.44, 0.28], then the output of the decoder should look like the following (should be the same size as the original input of the model which is 10):

[0, 0, 0.28, 0.44, 0.28, 0, 0.28, 0.44, 0.28, 0]

which is the same as [0, 0, 0, 1, 0, 0, 0, 1, 0, 0]*[0.28, 0.44, 0.28] where * represents the convolution operator. please note that in the first vector, the 1 or located at positions 3 and 7 considering a 0-indexing format.

Finally a typical pixel loss such as MSE will be calculated. The important part is that this rendering module needs to be able to backpropagate the errors from the loss to its inputs which are the coordinates.

This module itself does not have any trainable parameters. Also, I do not want to change the layers coming before this rendering module and they need to stay as they are. In a more advanced setting, I would also like to provide the 4 covariance values as input too, i.e. the input to the renderer would be in the form of [num_points, 5] where each row is [x_coord, y_coord, cov(x,x), cov(x,y), cov(y,y)].

How can I implement such a module in any of the available deep learning frameworks? a hint towards something similar would also be very useful.

Output:
In my experience, punctual things in neural networks will have a bad performance because it cuts the influence of distant pixels.

Thus, instead of using a gaussian kernel, it would be better to have an actual gaussian function applied to all pixels.

So, taking a 2D gaussian distribution function:

We can use it like this:

This means some steps in a custom function:

import keras.backend as K

def coords_to_gaussian(x): #where x is shape (batch, 10, 2), and 2 = x, y

    #pixel coordinates - must match the values of x and y
    #here I suppose from 0 to image size, but you may want it normalized, maybe
    x_pixels = K.reshape(K.arange(image_size), (1,1,image_size,1))
    x_pixels = K.concatenate([x_pixels]*image_size, axis=-1) #shape(1,1,size,size)

    y_pixels = K.permute_dimensions(x_pixels, (0,1,3,2))

    pixels = K.stack([x_pixels, y_pixels], axis=-1) #shape(1,1,size,size,2)

    #adjusting the AE locations to a compatible shape:
    locations = K.reshape(x, (-1, 10, 1, 1, 2))

    #calculating the upper part of the equation
    result = K.square(pixels - locations) #shape (batch, 10, size, size, 2)
    result = - K.sum(result, axis=-1) / (2*square_sigma) #shape (batch, 10, size, size)

    #calculating the E:
    result = K.exp(result) / (2 * pi * square_sigma)

    #sum the 10 channels (principle of superposition)
    result = K.sum(result, axis=1) #shape (batch, size, size)

    #add a channel for future convolutions
    result = K.expand_dims(result, axis=-1) #shape (batch, size, size, 1)

    return result

Use this in a Lambda layer:

from keras.layers import Lambda
Lambda(coords_to_gaussian)(coordinates_tensor_from_encoder)

I'm not considering the covariances here, but you might find a way to put them in the formulas and adjust the code.


Instruction: Display PyTorch model with multiple outputs using torchviz make_dots

Input:
I have a model with multiple outputs, 4 to be exact:

 def forward(self, x):
      outputs = []
      for conv, act in zip(self.Convolutions, self.Activations):
           y = conv(x)
           outputs.append(act(y))
      return outputs

I wanted to display it using make_dot from torchviz:

 from torchviz import make_dot
 generator = ...
 batch = next(iter(generator))
 input, output = batch["input"].to(device, dtype=torch.float), batch["output"].to(device, dtype=torch.float)
 dot = make_dot(model(input), params=dict(model.named_parameters()))

But I get the following error:

 File "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torchviz/dot.py", line 37, in make_dot
 output_nodes = (var.grad_fn,) if not isinstance(var, tuple) else tuple(v.grad_fn for v in var)
 AttributeError: 'list' object has no attribute 'grad_fn'

Obviously a list does not have a grad_fn function, but according to this discussion, I can return a list of outputs.

What am I doing wrong?

Output: Model can return a list, but make_dot wants a Tensor. If output components have similar shape, I suggest to use torch.cat on it.


Instruction: segfault when using xarray, only when 'import torch'

Input:
I get a segfault when I open a file using xarray, but only if I import torch (imagine how long it took me to figure this out):

import xarray as xr
import torch

xr.open_dataset('my/file.nc')

----

[1651963ee602:10863] *** Process received signal ***
[1651963ee602:10863] Signal: Segmentation fault (11)
[1651963ee602:10863] Signal code: Address not mapped (1)
[1651963ee602:10863] Failing at address: 0x440000e9
[1651963ee602:10863] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f26f1e11890]
[1651963ee602:10863] [ 1] /usr/local/mpi/lib/libmpi.so.40(PMPI_Comm_set_errhandler+0x41)[0x7f26dc349691]
[1651963ee602:10863] [ 2] /opt/conda/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so(+0x82370)[0x7f26cadc2370]
[1651963ee602:10863] [ 3] /opt/conda/lib/python3.6/site-packages/mpi4py/MPI.cpython-36m-x86_64-linux-gnu.so(+0x2dd79)[0x7f26cad6dd79]
[1651963ee602:10863] [ 4] /opt/conda/bin/python /opt/conda/bin/ipython(PyModule_ExecDef+0x7a)[0x55cad2262cca]
[1651963ee602:10863] [ 5] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1ecd38)[0x55cad2262d38]
[1651963ee602:10863] [ 6] /opt/conda/bin/python /opt/conda/bin/ipython(PyCFunction_Call+0xf4)[0x55cad218cb54]
[1651963ee602:10863] [ 7] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x539a)[0x55cad224183a]
[1651963ee602:10863] [ 8] /opt/conda/bin/python /opt/conda/bin/ipython(+0x170cf6)[0x55cad21e6cf6]
[1651963ee602:10863] [ 9] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171c91)[0x55cad21e7c91]
[1651963ee602:10863] [10] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [11] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [12] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [13] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [14] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [15] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [16] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [17] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [18] /opt/conda/bin/python /opt/conda/bin/ipython(+0x171a5b)[0x55cad21e7a5b]
[1651963ee602:10863] [19] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1a1635)[0x55cad2217635]
[1651963ee602:10863] [20] /opt/conda/bin/python /opt/conda/bin/ipython(_PyEval_EvalFrameDefault+0x30a)[0x55cad223c7aa]
[1651963ee602:10863] [21] /opt/conda/bin/python /opt/conda/bin/ipython(_PyFunction_FastCallDict+0x11b)[0x55cad21e80cb]
[1651963ee602:10863] [22] /opt/conda/bin/python /opt/conda/bin/ipython(_PyObject_FastCallDict+0x26f)[0x55cad2189f0f]
[1651963ee602:10863] [23] /opt/conda/bin/python /opt/conda/bin/ipython(_PyObject_CallMethodIdObjArgs+0x100)[0x55cad21b3be0]
[1651963ee602:10863] [24] /opt/conda/bin/python /opt/conda/bin/ipython(PyImport_ImportModuleLevelObject+0x280)[0x55cad21804b0]
[1651963ee602:10863] [25] /opt/conda/bin/python /opt/conda/bin/ipython(+0x1abbda)[0x55cad2221bda]
[1651963ee602:10863] [26] /opt/conda/bin/python /opt/conda/bin/ipython(PyCFunction_Call+0xc6)[0x55cad218cb26]
[1651963ee602:10863] [27] /opt/conda/bin/python /opt/conda/bin/ipython(PyObject_Call+0x3e)[0x55cad218994e]
[1651963ee602:10863] [28] /opt/conda/bin/python /opt/conda/bin/ipython(PyObject_CallFunction+0xf4)[0x55cad21e6854]
[1651963ee602:10863] [29] /opt/conda/bin/python /opt/conda/bin/ipython(PyImport_Import+0x9e)[0x55cad2180a9e]
[1651963ee602:10863] *** End of error message ***
[1]    10863 segmentation fault (core dumped)  ipython

This works:

import xarray as xr

xr.open_dataset('my/file.nc')

This works as well:

import xarray as xr
import torch

print(torch.ones(1))

OS       ubuntu 18.04.3
python   3.6.7
xarray   0.15.0
torch    1.3.0a0+24ae9b5

I appreciate any help.

Output: Installing netcdf4 via pip solved the problem.


Instruction: PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

Input:
I am a beginner at PyTorch and I am just trying out some examples on this webpage. But I can't seem to get the 'super_resolution' program running due to this error:

RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

I searched the Internet and found that some people suggest setting num_workers to 0. But if I do that, the program tells me that I am running out of memory (either with CPU or GPU):

RuntimeError: [enforce fail at ..\c10\core\CPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 9663676416 bytes. Buy new RAM!

or

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 0 bytes free; 2.03 GiB reserved in total by PyTorch)

How do I fix this?

I am using python 3.8 on Win10(64bit) and pytorch 1.4.0.

More complete error messages (--cuda means using GPU, --threads x means passing x to the num_worker parameter):

with command line arguments --upscale_factor 1 --cuda

  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "E:\Python38\lib\multiprocessing\queues.py", line 108, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 48, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "E:\Python38\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 16596, 9376, 12756, 9844) exited unexpectedly

with command line arguments --upscale_factor 1 --cuda --threads 0

  File "Z:\super_resolution\main.py", line 81, in <module>
    train(epoch)
  File "Z:\super_resolution\main.py", line 52, in train
    loss = criterion(model(input), target)
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "Z:\super_resolution\model.py", line 21, in forward
    x = self.relu(self.conv2(x))
  File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "E:\Python38\lib\site-packages\torch\nn\modules\conv.py", line 341, in conv2d_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 954.35 MiB free; 2.03 GiB reserved in total by PyTorch)

Output:
There is no "complete" solve for GPU out of memory errors, but there are quite a few things you can do to relieve the memory demand. Also, make sure that you are not passing the trainset and testset to the GPU at the same time!

Decrease batch size to 1
Decrease the dimensionality of the fully-connected layers (they are the most memory-intensive)
(Image data) Apply centre cropping
(Image data) Transform RGB data to greyscale
(Text data) Truncate input at n chars (which probably won't help that much)

Alternatively, you can try running on Google Colaboratory (12 hour usage limit on K80 GPU) and Next Journal, both of which provide up to 12GB for use, free of charge. Worst case scenario, you might have to conduct training on your CPU. Hope this helps!


Instruction: Finding mean and standard deviation across image channels PyTorch

Input:
Say I have a batch of images in the form of tensors with dimensions (B x C x W x H) where B is the batch size, C is the number of channels in the image, and W and H are the width and height of the image respectively. I'm looking to use the transforms.Normalize() function to normalize my images with respect to the mean and standard deviation of the dataset across the C image channels, meaning that I want a resulting tensor in the form 1 x C. Is there a straightforward way to do this?

I tried torch.view(C, -1).mean(1) and torch.view(C, -1).std(1) but I get the error:

view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Edit

After looking into how view() works in PyTorch, I know realize why my approach doesn't work; however, I still can't figure out how to get the per-channel mean and standard deviation.

Output:
You just need to rearrange batch tensor in a right way: from [B, C, W, H] to [B, C, W * H] by:

batch = batch.view(batch.size(0), batch.size(1), -1)

Here is complete usage example on random data:

Code:

import torch
from torch.utils.data import TensorDataset, DataLoader

data = torch.randn(64, 3, 28, 28)
labels = torch.zeros(64, 1)
dataset = TensorDataset(data, labels)
loader = DataLoader(dataset, batch_size=8)

nimages = 0
mean = 0.
std = 0.
for batch, _ in loader:
    # Rearrange batch to be the shape of [B, C, W * H]
    batch = batch.view(batch.size(0), batch.size(1), -1)
    # Update total number of images
    nimages += batch.size(0)
    # Compute mean and std here
    mean += batch.mean(2).sum(0)
    std += batch.std(2).sum(0)

# Final step
mean /= nimages
std /= nimages

print(mean)
print(std)

Output:

tensor([-0.0029, -0.0022, -0.0036])
tensor([0.9942, 0.9939, 0.9923])


Instruction: How to use GPU in pytorch?

Input:
I tried following steps at: https://pytorch.org/get-started/locally/

First I created a conda environment as:

conda create -n facenet37_2 python=3.7

Then on above site I selected:

PyTorch Build: Stable (1.4)
OS: Linux (I am using Ubuntu 18.04)
Package: conda
Language: python
CUDA: 10.1

and it asked me to run following command:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

But after that when I opened python and typed:

import torch
torch.cuda.is_available()

I get False

I have GeForce GT 630M (computeCapability: 2.1). But it is not getting detected. Why? Is it too old and no longer supported? How can I fix the issue?

Edit: Why did I get a negative vote?

Output:
The GeForce GT 630M has compute capability 2.1 and therefore only supports up to CUDA 8.

PyTorch binaries dropped support for compute capability <= 5.0 in PyTorch 0.3.1. It's not clear to me if compute capability 2.1 was ever included in the binaries.
The PyTorch codebase dropped CUDA 8 support in PyTorch 1.1.0.

Due to the second point there's no way short of changing the PyTorch codebase to make your GPU work with the latest version. Your options are:

Install PyTorch without GPU support.
Try compiling PyTorch < 1.1.0 from source (instructions). Make sure to checkout the v1.0.1 tag. This will produce a binary with support for your compute capability.
If acceptable you could try installing a really old version: PyTorch < 0.3.1 using conda or a wheel and see if that works. It may have compute capability 2.1 support though I can't verify this. See pytorch.org for information. Though it looks like the link to https://download.pytorch.org/whl/cu80/torch_stable.html is broken.


Instruction: Pytorch runtimeError "matrices expected, got 1D, 2D tensors at"

Input:
I begin to practice Pytorch ，try to use torch.mm() method。

Below is my code

import torch
import numpy as np
from torch.autograd import Variable
num_x = np.array([[1.0, 2.0]
                  ,[3.0,4.0]])

tensor_x = torch.from_numpy(num_x)
x = Variable(tensor_x,requires_grad = True)
s = Variable(torch.DoubleTensor([0.01,0.02]),requires_grad = True)
print(s)
s = s.mm(x)
print(s)

Unfortunately,there is a runtime error

*RuntimeError                              Traceback (most recent call last)
<ipython-input-58-e8a58ffb2545> in <module>()
      9 s = Variable(torch.DoubleTensor([0.01,0.02]),requires_grad = True)
     10 print(s)
---> 11 s = s.mm(x)
     12 print(s)
RuntimeError: matrices expected, got 1D, 2D tensors at /pytorch/aten/src/TH/generic/THTensorMath.cpp:131*

How can I fix this problem。
your reply is appreciated

Output:
try reshape you need to change shape of s to (1,2) to make possible matrix multiplication operation with (2,2) tensor

>>> s.reshape(1,2).mm(x)
tensor([[0.0700, 0.1000]], dtype=torch.float64, grad_fn=<MmBackward>)

Or give right shape when initializing s

>>> s = Variable(torch.DoubleTensor([[0.01,0.02]]),requires_grad = True)
>>> s.mm(x)
tensor([[0.0700, 0.1000]], dtype=torch.float64, grad_fn=<MmBackward>)


Instruction: Find specific element index from Tensor(Matrix)

Input:
I am a student just beginning to study deep learning.

   x_norm = (x**2).sum(1).view(-1, 1)
   if y is not None:
       y_norm = (y**2).sum(1).view(1, -1)
   else:
       y = x
       y_norm = x_norm.view(1, -1)
   ## NOTICE ##
   dist = torch.exp(-1*(x_norm + y_norm - 2.0 * torch.mm(x, torch.transpose(y, 0, 1))))
   return dist

dist = pairwise_distances(atom_s[:3,-3:])
zero_mat=torch.zeros_like(dist,dtype=torch.float)
dist= torch.where(dist>exp(-8),dist,zero_mat)

Above is my coding to make pairwise distance map. And change some element that satiesfy a condition to 0.

Question is that "how can I get indexes of elements that satiesfy specific condition ( e.g larger than >0.5)?? without using slow 'for' loop.

Output:
If it is normal tensor you can use torch.nonzero

>>> (dist > 0.5).nonzero()

which will return indices of all elements that are more than 0.5

Example:

>>> dist = torch.rand((6,5))
>>> dist
tensor([[0.7549, 0.0962, 0.3198, 0.6868, 0.8117],
        [0.0785, 0.7666, 0.2623, 0.5140, 0.2713],
        [0.5768, 0.8160, 0.8654, 0.6978, 0.0138],
        [0.8147, 0.1394, 0.3204, 0.0104, 0.2872],
        [0.1396, 0.5639, 0.7085, 0.7151, 0.8253],
        [0.6115, 0.0214, 0.6033, 0.1403, 0.1977]])

>>> (dist > 0.5).nonzero()
tensor([[0, 0],
        [0, 3],
        [0, 4],
        [1, 1],
        [1, 3],
        [2, 0],
        [2, 1],
        [2, 2],
        [2, 3],
        [3, 0],
        [4, 1],
        [4, 2],
        [4, 3],
        [4, 4],
        [5, 0],
        [5, 2]])


Instruction: PyTorch flatten doesn't maintain batch size

Input: In Keras, using the Flatten() layer retains the batch size. For eg, if the input shape to Flatten is (32, 100, 100), in Keras output of Flatten is (32, 10000), but in PyTorch it is 320000. Why is it so?

Output:
As OP already pointed out in their answer, the tensor operations do not default to considering a batch dimension. You can use torch.flatten() or Tensor.flatten() with start_dim=1 to start the flattening operation after the batch dimension.

Alternatively since PyTorch 1.2.0 you can define an nn.Flatten() layer in your model which defaults to start_dim=1.


Instruction: How to read from a high IO dataset in pytorch which grows from epoch to epoch

Input:
I use Tensorflow, but I'm writing documentation for users that will typically vary across deep learning frameworks.

When working with datasets that don't fit on the local filesystem (TB+) I sample data from a remote data store and write samples locally to a Tensorflow standardtfrecords format.

During the first epoch of training I will have only sampled a few values, therefore an epoch of local data is very small, I train on it. On epoch 2 I re-examine what data files have been produced by my sampling subprocesses (now more) and train on the expanded set of local data files for the next epoch. Repeat the process each epoch. In this way I build up a local cache of samples and can evict older samples as I fill up the local storage. The local samples cache grows at about the time the model needs the variance the most (towards the latter part of training).

In Python/Tensorflow it's crucial that I not deserialize the data in the Python training loop process because the Python GIL can't support the data transfer rates (300-600 MB/sec, the data is raw scientific uncompressible), and thus GPU performance suffers when the Python GIL can't service the training loop fast.

Writing the samples to a tfrecords file from subprocesses (python multiprocessing) allows tensorflow's native TFRecordsDataset to do deserialization outside of Python and thus we sidestep the Python GIL issues, and I can saturate a GPU with high IO data rates.

  I would like to know how I would address this issue in Pytorch. I'm writing about the sampling strategy that's being used, and want to provide specific recommendations to users of both Tensorflow and PyTorch, but I don't know the PyTorch preprocessing ecosystem well enough to write with sufficient detail.

Side note: the only purely Python based solution to support these data transfer rates may come in Python 3.8 with System V shared memory and multiprocessing, but I haven't tried that yet as support for it isn't quite sufficient (soon it will be). Existing multiprocessing solutions aren't sufficient because they require deserialization in the training loop process and thus lock the GIL during deserialization at high IO rates.

Output:
Actually, you can easily deserialize data in a subprocess by using torch.utils.data.DataLoader. By setting num_workers argument to 1 or a bigger value, you can spawn subprocesses with their own python interpreters and GILs.

loader = torch.utils.data.DataLoader(your_dataset, num_workers=n, **kwargs)
for epoch in range(epochs):
    for batch_idx, data in enumerate(loader):
         # loader in the main process does not claim GIL at this point

A Dataloader requires a torch.utils.data.Dataset to get data from. It may not be a trivial job to implement a proper subclass in your case. In case you need to recreate a Dataset instance for every epoch, you can do something like this.

for epcoh in range(epochs):
    dset = get_new_dataset()
    loader = torch.utils.data.DataLoader(dset, num_workers=n, **kwargs)
    for batch_idx, data in enumerate(loader):
        # Do training

or even better

dset = get_new_dataset()
loader = torch.utils.data.DataLoader(dset, num_workers=n, **kwargs)

for epcoh in range(epochs):
    last_batch_idx =  (len(dset)-1) // loader.batch_size
    for batch_idx, data in enumerate(loader):
        # Prepare next loader in advance to avoid blocking
        if batch_idx == last_batch_idx:
            dset = get_new_dataset()
            loader = torch.utils.data.DataLoader(dset, num_workers=n, **kwargs)
        # Do training

As a side note, please note that it's CPU bound operation that is affected by GIL in most cases, not I/O bound operation, i.e., threading will do for any purely I/O heavy operation and you don't even need subprocess. For more information please refer to this question and this wikipedia article.


Instruction: Optimizer and scheduler for BERT fine-tuning

Input:
I'm trying to fine-tune a model with BERT (using transformers library), and I'm a bit unsure about the optimizer and scheduler.

First, I understand that I should use transformers.AdamW instead of Pytorch's version of it. Also, we should use a warmup scheduler as suggested in the paper, so the scheduler is created using get_linear_scheduler_with_warmup function from transformers package.

The main questions I have are:

get_linear_scheduler_with_warmup should be called with the warm up. Is it ok to use 2 for warmup out of 10 epochs?
When should I call scheduler.step()? If I do after train, the learning rate is zero for the first epoch. Should I call it for each batch?

Am I doing something wrong with this?

from transformers import AdamW
from transformers.optimization import get_linear_scheduler_with_warmup

N_EPOCHS = 10

model = BertGRUModel(finetune_bert=True,...)
num_training_steps = N_EPOCHS+1
num_warmup_steps = 2
warmup_proportion = float(num_warmup_steps) / float(num_training_steps)  # 0.1

optimizer = AdamW(model.parameters())
criterion = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([class_weights[1]]))

scheduler = get_linear_schedule_with_warmup(
    optimizer, num_warmup_steps=num_warmup_steps,
    num_training_steps=num_training_steps
)

for epoch in range(N_EPOCHS):
    scheduler.step() #If I do after train, LR = 0 for the first epoch
    print(optimizer.param_groups[0]["lr"])

    train(...) # here we call optimizer.step()
    evaluate(...)

My model and train routine(quite similar to this notebook)

class BERTGRUSentiment(nn.Module):
    def __init__(self,
                 bert,
                 hidden_dim,
                 output_dim,
                 n_layers=1,
                 bidirectional=False,
                 finetune_bert=False,
                 dropout=0.2):

        super().__init__()

        self.bert = bert

        embedding_dim = bert.config.to_dict()['hidden_size']

        self.finetune_bert = finetune_bert

        self.rnn = nn.GRU(embedding_dim,
                          hidden_dim,
                          num_layers = n_layers,
                          bidirectional = bidirectional,
                          batch_first = True,
                          dropout = 0 if n_layers < 2 else dropout)

        self.out = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, text):
        #text = [batch size, sent len]

        if not self.finetune_bert:
            with torch.no_grad():
                embedded = self.bert(text)[0]
        else:
            embedded = self.bert(text)[0]
        #embedded = [batch size, sent len, emb dim]
        _, hidden = self.rnn(embedded)

        #hidden = [n layers * n directions, batch size, emb dim]

        if self.rnn.bidirectional:
            hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))
        else:
            hidden = self.dropout(hidden[-1,:,:])

        #hidden = [batch size, hid dim]

        output = self.out(hidden)

        #output = [batch size, out dim]

        return output

import torch
from sklearn.metrics import accuracy_score, f1_score

def train(model, iterator, optimizer, criterion, max_grad_norm=None):
    """
    Trains the model for one full epoch
    """
    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for i, batch in enumerate(iterator):
        optimizer.zero_grad()
        text, lens = batch.text

        predictions = model(text)

        target = batch.target

        loss = criterion(predictions.squeeze(1), target)

        prob_predictions = torch.sigmoid(predictions)

        preds = torch.round(prob_predictions).detach().cpu()
        acc = accuracy_score(preds, target.cpu())

        loss.backward()
        # Gradient clipping
        if max_grad_norm:
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Output:
Here you can see a visualization of learning rate changes using get_linear_scheduler_with_warmup.

Referring to this comment: Warm up steps is a parameter which is used to lower the learning rate in order to reduce the impact of deviating the model from learning on sudden new data set exposure.

By default, number of warm up steps is 0.

Then you make bigger steps, because you are probably not near the minima. But as you are approaching the minima, you make smaller steps to converge to it.

Also, note that number of training steps is number of batches * number of epochs, but not just number of epochs. So, basically num_training_steps = N_EPOCHS+1 is not correct, unless your batch_size is equal to the training set size.

You call scheduler.step() every batch, right after optimizer.step(), to update the learning rate.


Instruction: pytorch nllloss function target shape mismatch

Input:
I'm training a LSTM model using pytorch with batch size of 256 and NLLLoss() as loss function.
The loss function is having problem with the data shape.

The softmax output from the forward passing has shape of torch.Size([256, 4, 1181]) where 256 is batch size, 4 is sequence length, and 1181 is vocab size.

The target is in the shape of torch.Size([256, 4]) where 256 is batch size and 4 is the output sequence length.

When I was testing earlier with batch size of 1, the model works fine but when I add batch size, it is breaking. I read that NLLLoss() can take class target as input instead of one hot encoded target.

Am I misunderstanding it? Or did I not format the shape of the target correctly?

class LSTM(nn.Module):

    def __init__(self, embed_size=100, hidden_size=100, vocab_size=1181, embedding_matrix=...):
        super(LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.word_embeddings = nn.Embedding(vocab_size, embed_size)
        self.word_embeddings.load_state_dict({'weight': torch.Tensor(embedding_matrix)})
        self.word_embeddings.weight.requires_grad = False
        self.lstm = nn.LSTM(embed_size, hidden_size)
        self.hidden2out = nn.Linear(hidden_size, vocab_size)

    def forward(self, tokens):
        batch_size, num_steps = tokens.shape
        embeds = self.word_embeddings(tokens)
        lstm_out, _ = self.lstm(embeds.view(batch_size, num_steps, -1))
        out_space = self.hidden2out(lstm_out.view(batch_size, num_steps, -1))
        out_scores = F.log_softmax(out_space, dim=1)
        return out_scores

model = LSTM(self.config.embed_size, self.config.hidden_size, self.config.vocab_size, self.embedding_matrix)
loss_function = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=self.config.lr)

Error:

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1846         if target.size()[1:] != input.size()[2:]:
   1847             raise ValueError('Expected target size {}, got {}'.format(
-> 1848                 out_size, target.size()))
   1849         input = input.contiguous().view(n, c, 1, -1)
   1850         target = target.contiguous().view(n, 1, -1)

ValueError: Expected target size (256, 554), got torch.Size([256, 4])

Output:
Your input shape to the loss function is (N, d, C) = (256, 4, 1181) and your target shape is (N, d) = (256, 4), however, according to the docs on NLLLoss the input should be (N, C, d) for a target of (N, d).
Supposing x is your network output and y is the target then you can compute loss by transposing the incorrect dimensions of x as follows:
loss = loss_function(x.transpose(1, 2), y)

Alternatively, since NLLLoss is just averaging all the responses anyway, you can reshape x and y to be (N*d, C) and (N*d). This gives the same result without creating temporary copies of your tensors.
loss = loss_function(x.reshape(N*d, C), y.reshape(N*d))


Instruction: How to match items in two pytorch tensors

Input:
I have two  pytorch tensors have the same number of rows,
  ( tensor1-size  is (6,4) and  tensor2-size  is (6), The first tensor is bounding-box coordinate points, whereas the second tensor is the prediction scores.

tensor1 =
([[780.8306,  98.1060, 813.8367, 149.8171],
[585.6562, 117.6804, 621.6012, 166.3151],
[ 88.4085, 117.1313, 129.3327, 173.1145],
[223.1263, 239.2682, 255.1892, 270.7897],
[194.4088, 117.9768, 237.3028, 166.1765],
[408.9165, 109.0131, 441.0802, 141.2362]])
tensor2 =
([0.9842, 0.9751, 0.9689, 0.8333, 0.7021, 0.6191])

How could I concatenate  every item in tensor2 with its correspondent row in tensor1)?

I want get predilection-score and the bounding box.
The output to be something like:

prediction  bounding box
0.9842      780.8306,  98.1060, 813.8367, 149.8171
0.9751      585.6562, 117.6804, 621.6012, 166.3151]
0.96898     88.4085, 117.1313, 129.3327, 173.1145
0.8333      223.1263, 239.2682, 255.1892, 270.7897
0.7021      194.4088, 117.9768, 237.3028, 166.1765
0.6191      408.9165, 109.0131, 441.0802, 141.2362

I appreciate If anyone could  help,
Thank you

Output:
You can use torch.cat to concatenate the columns. Since cat requires that all tensors have the same number of dimensions you will first need to insert a unitary dimension to tensor2 to make it a (6, 1). This can be done a few ways, but the most clear is probably Tensor.unsqueeze(1) which reshapes the tensor to have a unitary dimension at dimension 1.

tensor21 = torch.cat((tensor2.unsqueeze(1), tensor1), dim=1)


Instruction: What gets printed when you print a object of some class in python?

Input:
I want to ask about this specific example, taken from the official pytorch tutorial.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

net = Net()
print(net)

And the output is

Net(
  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))
  (fc1): Linear(in_features=576, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

As I understand, this code defines a child class Net of nn.Module, and defines in its initializer the members conv1, conv2, etc. These members are printed when print(net) is called. Based on this obsevation, I thought that if I add the line self.x = 0 to the initializer of Net, there would be an extra line of output, something like: (x): 0. But that didn't happen. So who decides which part of Net gets printed?

Output:
From the Python3 documentation

  repr(object)
  Return a string containing a printable representation of an object. For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object. A class can control what this function returns for its instances by defining a __repr__() method.

Since your class inherits the nn.Module class, it uses its repr method


Instruction: Different results after converting pytorch to torchscript? Converting NSnumber to Float cause any loss?

Input:
I converted the pytorch pretrained-model(.pt) to torchscript model(.pt) for using it in Swift 5(ios-iphone6s, xcode 11). In Swift, the “predict” function of the model gave me its embedding values(Tensor). Since it returned the NSNumber array as a result of prediction, I used type casting [NSNumber] to both [Double] or [Float] to calculate the distance between two embedding values. L2 normalization, dot product, etc.

However, while the pytorch version got the correct answers, the torchscript model got so many wrong answers. Not only the answers are different, the distance calculations of the two embedding pairs in torchscript are also different from the results of the pytorch model on the PC(CPU, Pycharm). In fact, before using type casting for distance calculations, the embedding values in NSNumber(Swift) are so different from the values in float32(pytorch). I used same input images.

I tried to find the reason.. Once, I copied the embedding values( [NSNumber] ) from swift-torchscript and calculated the distance between two embeddings in pytorch, to check if there was a problem with my distance calculation implementation in Swift. I used torch.FloatTensor to use type casting [NSNumber] -> [Float]. I also tried [Double]. As a result of this, I found many infinite numbers. Is this infinite numbers related to the wrong answer?

What does this “inf” mean? Is it a calculation or type casting error? Did I lost information while casting from NSNumber to Float or Double? How could I get the correct value from the torchscript model in swift? What should I check?

I used the following codes to convert. pytorch -> torchscript.

import torch

from models.inception_resnet_v1 import InceptionResnetV1

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

resnet = InceptionResnetV1(pretrained='vggface2').eval().to(device)

example = torch.rand(1, 3, 160, 160)
traced_script_module = torch.jit.trace(resnet, example)
traced_script_module.save("mobile_model.pt")

Output:
Are you using InceptionResnetV1 from:
https://github.com/timesler/facenet-pytorch ?
When you are referring to the pytorch model in your comparison of the outputs, are you referring to the torchscript model when run in pytorch, or the resnet as is?

If it is the latter, did you already check something similar as below?

What do you get when running the following:

print('Original:')
orig_res = resnet(example)
print(orig_res.shape)
print(orig_res[0, 0:10])
print('min abs value:{}'.format(torch.min(torch.abs(orig_res))))
print('Torchscript:')
ts_res = traced_script_module(example)
print(ts_res.shape)
print(ts_res[0, 0:10])
print('min abs value:{}'.format(torch.min(torch.abs(ts_res))))
print('Dif sum:')
abs_diff = torch.abs(orig_res-ts_res)
print(torch.sum(abs_diff))
print('max dif:{}'.format(torch.max(abs_diff)))

after defining 'traced_script_module'.
I get the following:

Original:
torch.Size([1, 512])
tensor([ 0.0347,  0.0145, -0.0124,  0.0723, -0.0102,  0.0653, -0.0574,  0.0004,
        -0.0686,  0.0695], device='cuda:0', grad_fn=<SliceBackward>)
min abs value:0.00034740756382234395
Torchscript:
torch.Size([1, 512])
tensor([ 0.0347,  0.0145, -0.0124,  0.0723, -0.0102,  0.0653, -0.0574,  0.0004,
        -0.0686,  0.0695], device='cuda:0', grad_fn=<SliceBackward>)
min abs value:0.0003474018594715744
Dif sum:
tensor(8.1539e-06, device='cuda:0', grad_fn=<SumBackward0>)
max dif:5.960464477539063e-08

which is not perfect but considering the outputs are in the order of 10^-4 minimum, and that the before last number is the sum of the absolute difference of 512 elements, not the mean, it seems not too far off for me. The maximum difference is at around 10^-8.

By the way, you might want to change to:

example = torch.rand(1, 3, 160, 160).to(device)

If you get something similar for the tests above, what are the kind of values you get for the first 10 output values you get from the swift-torchscript as NSNumber, and then, once casted in float, when compared against both the same slices in the pytorch and torchscript-pytorch model outputs?


Instruction: How do I separate the input and targets from Pytorch Fashion MNIST?

Input:
The Fashion MNIST dataset is implemented pretty weirdly in Pytorch. I want to do something like:

X, y = FashionMNIST

But in reality, it's a little more complicated. This is what I have:

from torchvision.datasets import FashionMNIST
train = FashionMNIST(root='.', download=True, train=True)
print(train)

The output:

Dataset FashionMNIST
    Number of datapoints: 60000
    Root location: c:/users/nicolas/documents/data/fashionmnist
    Split: Train

What one observation looks like:

print(train[0])

(<PIL.Image.Image image mode=L size=28x28 at 0x20868074780>, 9)

I could only do it for one observation.

X, y = train[0]

So how do I separate the input and targets?

Output:
FashionMNIST object has data and targets attributes.

You can simply write

X, y = train.data, train.targets

and then you can see the shapes

X.shape, y.shape

  (torch.Size([60000, 28, 28]), torch.Size([60000]))


Instruction: Loading a model with pytorch

Input:
I'm having a problem loading my model in the Image Classifier project. First, I saved it:

model.class_to_idx = train_data.class_to_idx

checkpoint = {'arch': 'vgg19',
              'learn_rate': learn_rate,
              'epochs': epochs,
              'state_dict': model.state_dict(),
              'class_to_idx': model.class_to_idx,
              'optimizer': optimizer.state_dict(),
              'input_size': 25088,
              'output_size': 102,
              'momentum': momentum,
              'batch_size':64,
              'classifier' : classifier
             }

torch.save(checkpoint, 'checkpoint.pth')

Then I tried to load the project I had saved:

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)

    learn_rate = checkpoint['learn_rate']

    optimizer.load_state_dict(checkpoint['optimizer'])

    model = models.vgg16(pretrained=True)
    model.epochs = checkpoint['epochs']
    model.load_state_dict(checkpoint['state_dict'])
    model.class_to_idx = checkpoint['class_to_idx']
    model.classifier = checkpoint['classifier']

    return learn_rate, optimizer, model

learn_rate, optimizer, model = load_checkpoint('checkpoint.pth')

And I get an error when I try to load:

<ipython-input-75-5bd1aa042c7f> in load_checkpoint(filepath)
      9     model = models.vgg16(pretrained=True)
     10     model.epochs = checkpoint['epochs']
---> 11     model.load_state_dict(checkpoint['state_dict'])
     12     model.class_to_idx = checkpoint['class_to_idx']
     13     model.classifier = checkpoint['classifier']

RuntimeError: Error(s) in loading state_dict for VGG:
    Missing key(s) in state_dict: "classifier.0.weight", "classifier.0.bias", "classifier.3.weight", "classifier.3.bias", "classifier.6.weight", "classifier.6.bias".
    Unexpected key(s) in state_dict: "classifier.fc1.weight", "classifier.fc1.bias", "classifier.fc2.weight", "classifier.fc2.bias".

This seems to be classifier issue. Does anyone know what's going on?

Output:
jodag's comment points at the heart of the issue.
If fc1 fc2 correspond to classifier.0 classifier.3, classifier.6 you can adjust the dictionary to link them.
When loading the weights to the model make sure to add the option strict=False.

You will need to retrain your model for the classifier - because your state dict misses weights for 3 layers but have 2 unused layer weights - but it should converge really quickly (from personal experience).


Instruction: Pytorch or Numpy Batch Matrix Operation

Input:
I am trying to us torch.bmm to do the following matrix operation,

If matrix is a M * N tensor, batch is a N * B tensor, how can i achieve,
In each batch, matrix @ batch_i, which gives M, and put the batch size together, the output tensor looks like M * B

There two questions here,

1.To use torch.bmm, it seems need both matrix need be batch, but my first input is not

The batch size need be the first dimension, while my batch size in the end

I guess it is the same question for Numpy users

Output: It seems that torch.einsum('ij,jbc->ibc', A, B) will solve the question


Instruction: Tensorboard is showing a blank page (Refused to execute script from 'http://localhost:6006/index.js' because its MIME type)

Input:
When Trying to open tesnorflow I just get a plank page:

This is how it looks like in firefox:

I get the error message in the chrome console:

Refused to execute script from 'http://localhost:6006/index.js' because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.

In the firefox console I get the error message:

The resource from “http://localhost:6006/index.js” was blocked due to MIME type (“text/plain”) mismatch (X-Content-Type-Options: nosniff)

and

Loading failed for the <script> with source “http://localhost:6006/index.js”.

I tried:
Unable to open Tensorboard in browser
Tensorboard get blank page

I typed in the console:

tensorboard --logdir=runs --bind_all
tensorboard --logdir=./runs --bind_all
tensorboard --logdir=./runs/ --bind_all
tensorboard --logdir=./runs --host localhost --port 6006
tensorboard --logdir=./runs --host localhost
tensorboard --logdir=./runs --port 6006 --bind_all

I have tensorboard version: 2.1.0
I generated my data like that:

 train_set = torchvision.datasets.FashionMNIST(
        root="./data/FashionMNIST",
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor()
        ])
    )
train_loader = torch.utils.data.DataLoader(train_set, batch_size=1000)
tb = SummaryWriter()

network = Network()
images, labels = next(iter(train_loader))
grid = torchvision.utils.make_grid(images)

tb.add_image("image", grid)
tb.add_graph(network, images)
tb.close()

I followed this tutorial: TensorBoard with PyTorch - Visualize Deep Learning Metrics

Output:
There's a similar error and resolution reported here.

Apparently this has to do with some issue in the windows registry. Based on the comments this seems to be the solution

  In my case following procedure solved the problem:

  windows + r and regedit
  [your computer]\HKEY_LOCAL_MACHINE\SOFTWARE\Classes\.js
  Change content type from 'text/plain' to 'application/javascript'


Instruction: Is there any build in function for making 2D tensor from 1D tensor using specific calculation?

Input:
Hi I'm student who just started for deep learning.

For example, I have 1-D tensor x = [ 1 , 2]. From this one, I hope to make 2D tensor y whose (i,j)th element has value (x[i] - x[j]), i.e y[0,:] = [0  ,  1] , y[1,:]=[ -1 , 0].

Is there built-in function like this in pytorch library?

Thanks.

Output:
Here you need right dim of tensor to get expected result which you can get using torch.unsqueeze

x = torch.tensor([1 , 2])
y = x - x.unsqueeze(1)
y
tensor([[ 0,  1],
        [-1,  0]])


Instruction: Issues installing PyTorch 1.4 - "No matching distribution found for torch===1.4.0"

Input:
Used the install guide on pytorch.org on how to install it and the command I'm using is
pip install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html

But it's coming up with this error;

ERROR: Could not find a version that satisfies the requirement torch===1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.4.0

Is this even a me-related issue? Can other people use this command?
Pip is installed and works for other modules, Python 3.8, CUDA version 10.1, Windows 10 Home 2004

Output:
Looks like this issue is related to virtual environment. Did you try recommended installation line in another/new one virtual environment? If it doesn't help the possible solution might be installing package using direct link to PyTorch and TorchVision builds for your system:

pip install https://download.pytorch.org/whl/cu101/torch-1.4.0-cp38-cp38-win_amd64.whl

pip install https://download.pytorch.org/whl/cu101/torchvision-0.5.0-cp38-cp38-win_amd64.whl


Instruction: HuggingFace Transformers For Text Generation with CTRL with Google Colab's free GPU

Input:
I wanted to test TextGeneration with CTRL using PyTorch-Transformers, before using it for fine-tuning. But it doesn't prompt anything like it does with GPT-2 and other similar language generation models. I'm very new for this and am stuck and can't figure out what's going on.

This is the procedure I followed in my Colab notebook,

!pip install transformers

!git clone https://github.com/huggingface/pytorch-transformers.git

!python pytorch-transformers/examples/run_generation.py \
    --model_type=ctrl \
    --length=100 \
    --model_name_or_path=ctrl \
    --temperature=0.2 \
    --repetition_penalty=1.2 \

And this is what I get after running the script

02/10/2020 01:02:31 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json from cache at /root/.cache/torch/transformers/a858ad854d3847b02da3aac63555142de6a05f2a26d928bb49e881970514e186.285c96a541cf6719677cfb634929022b56b76a0c9a540186ba3d8bbdf02bca42
02/10/2020 01:02:31 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt from cache at /root/.cache/torch/transformers/aa2c569e6648690484ade28535a8157aa415f15202e84a62e82cc36ea0c20fa9.26153bf569b71aaf15ae54be4c1b9254dbeff58ca6fc3e29468c4eed078ac142
02/10/2020 01:02:31 - INFO - transformers.configuration_utils -   loading configuration file https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json from cache at /root/.cache/torch/transformers/d6492ca334c2a4e079f43df30956acf935134081b2b3844dc97457be69b623d0.1ebc47eb44e70492e0c20494a084f108332d20fea7fe5ad408ef5e7a8f2baef4
02/10/2020 01:02:31 - INFO - transformers.configuration_utils -   Model config CTRLConfig {
  "architectures": null,
  "attn_pdrop": 0.1,
  "bos_token_id": 0,
  "dff": 8192,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "from_tf": false,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-06,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "ctrl",
  "n_ctx": 512,
  "n_embd": 1280,
  "n_head": 16,
  "n_layer": 48,
  "n_positions": 50000,
  "num_beams": 1,
  "num_labels": 1,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 246534
}

02/10/2020 01:02:31 - INFO - transformers.modeling_utils -   loading weights file https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin from cache at /root/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
tcmalloc: large alloc 1262256128 bytes == 0x38b92000 @  0x7fe1900bdb6b 0x7fe1900dd379 0x7fe139843b4a 0x7fe1398455fa 0x7fe13bb7578a 0x7fe13bdbe30b 0x7fe13be05b37 0x7fe184c8cad5 0x7fe184c8d17b 0x7fe184c91160 0x7fe184ade496 0x551b15 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x5096b7 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9
tcmalloc: large alloc 1262256128 bytes == 0x19fdda000 @  0x7fe1900bdb6b 0x7fe1900dd379 0x7fe139843b4a 0x7fe1398455fa 0x7fe13bb7578a 0x7fe13bdbe30b 0x7fe13be05b37 0x7fe184c8cad5 0x7fe184c8d17b 0x7fe184c91160 0x7fe184ade496 0x551b15 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50d390 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5a067e 0x50d966 0x508245
^C

and then terminates. Could this be because of a GPU problem?

Any sort of help is appreciated.

Output:
The solution was to increase the RAM. Since I was using the Google Colab's free GPU, I was going through this: GitHub issue
and found this useful: Solution

The following piece of code will crash the session in Colab and select 'Get more RAM', which will increase the RAM up to 25.51GB

d=[]
while(1):
  d.append('1')


Instruction: RuntimeError: cuDNN version mismatch: PyTorch was compiled against 7102 but linked against 7604

Input:
I got this error when running training a deep learning model and although looking at many solutions over the Internet, they did not help me. The log is as follows:

Traceback (most recent call last):
  File "main.py", line 208, in <module>
    main()
  File "main.py", line 100, in main
    model = nn.DataParallel(model).cuda()
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 105, in __init__
    self.module.cuda(device_ids[0])
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 249, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/module.py", line 176, in _apply
    module._apply(fn)
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 112, in _apply
    self.flatten_parameters()
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/nn/modules/rnn.py", line 78, in flatten_parameters
    if not any_param.is_cuda or not torch.backends.cudnn.is_acceptable(any_param):
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 87, in is_acceptable
    if _libcudnn() is None:
  File "/home/dexter/miniconda3/envs/VideoSum/lib/python3.5/site-packages/torch/backends/cudnn/__init__.py", line 58, in _libcudnn
    'but linked against {}'.format(compile_version, __cudnn_version))
RuntimeError: cuDNN version mismatch: PyTorch was compiled against 7102 but linked against 7604

Output:
The question is, no matter the versions showed in the log are, 7.6.4 is my cudnn version and 7.1.2 is the cudnn version that the code is originally compiled. What I need is just downgrade (or upgrade my current cudnn version) by:

conda install cudnn=7.1.2

It works, if any, please correct me.