Instruction: Vectorizing assignment of a tensor to a slice in PyTorch

Input:
I'm trying to vectorize a slice assignment of the form

for i in range(a.shape[1]):
    for j in range(a.shape[2]):
        a[:,i,j,:,i:i+b.shape[2],j:j+b.shape[3]] = b

where b itself is an array. This is because the nested Python loop is too inefficient and is taking up most of the runtime. Is there a way to do this?

For a simpler case, consider the following:

for i in range(a.shape[1]):
    a[:,i,:,i:i+b.shape[2]] = b

This is what b and a might look like:

You can see the diagonal, "sliding" structure of the resulting matrix.

Output:
We can leverage np.lib.stride_tricks.as_strided based scikit-image's view_as_windows to get sliding windowed views into a 0s padded version of the input and being a view would be efficient on memory and performance.  More info on use of as_strided based view_as_windows.

Hence, for the simpler case, it would be -

from skimage.util.shape import view_as_windows

def sliding_2D_windows(b, outshp_axis1):
    # outshp_axis1 is desired output's shape along axis=1
    n = outshp_axis1-1
    b1 = np.pad(b,((0,0),(0,0),(n,n)),'constant')
    w_shp = (1,b1.shape[1],b.shape[2]+n)
    return view_as_windows(b1,w_shp)[...,0,::-1,0,:,:]

Sample run -

In [192]: b
Out[192]:
array([[[54, 57, 74, 77],
        [77, 19, 93, 31],
        [46, 97, 80, 98]],

       [[98, 22, 68, 75],
        [49, 97, 56, 98],
        [91, 47, 35, 87]]])

In [193]: sliding_2D_windows(b, outshp_axis1=3)
Out[193]:
array([[[[54, 57, 74, 77,  0,  0],
         [77, 19, 93, 31,  0,  0],
         [46, 97, 80, 98,  0,  0]],

        [[ 0, 54, 57, 74, 77,  0],
         [ 0, 77, 19, 93, 31,  0],
         [ 0, 46, 97, 80, 98,  0]],

        [[ 0,  0, 54, 57, 74, 77],
         [ 0,  0, 77, 19, 93, 31],
         [ 0,  0, 46, 97, 80, 98]]],

       [[[98, 22, 68, 75,  0,  0],
         [49, 97, 56, 98,  0,  0],
         [91, 47, 35, 87,  0,  0]],

        ....

        [[ 0,  0, 98, 22, 68, 75],
         [ 0,  0, 49, 97, 56, 98],
         [ 0,  0, 91, 47, 35, 87]]]])


Instruction: Run inference on CPU using pytorch and multiprocessing

Input:
I have trained a CNN model on GPU using FastAI (PyTorch backend). I am now trying to use that model for inference on the same machine, but using CPU instead of GPU. Along with that, I am also trying to make use of multiple CPU cores using the multiprocessing module. Now here is the issue,

  Running the code on single CPU (without multiprocessing) takes only 40 seconds to process nearly 50
  images

  Running the code on multiple CPUs using torch multiprocessing takes more than 6 minutes to process the same 50 images

from torch.multiprocessing import Pool, set_start_method
os.environ['CUDA_VISIBLE_DEVICES']=""
from fastai.vision import *
from fastai.text import *
defaults.device = torch.device('cpu')

def process_image_batch(batch):

    learn_cnn  = load_learner(scripts_folder, 'cnn_model.pkl')
    learn_cnn.model.training = False
    learn_cnn.model = learn_cnn.model.eval()
    # for image in batch:
    #     prediction = ... # predicting the image here
    #     return prediction

if __name__ == '__main__':
    #
    # image_batches = ..... # retrieving the image batches (It is a list of 5 lists)
    # n_processes = 5
    set_start_method('spawn', force=True)
    try:
        pool = Pool(n_processes)
        pool.map(process_image_batch, image_batches)
    except Exception as e:
        print('Main Pool Error: ', e)
    except KeyboardInterrupt:
        exit()
    finally:
        pool.terminate()
        pool.join()

I am not sure what's causing this slowdown in multiprocessing mode. I've read a lot of posts discussing similar issue but couldn't find a proper solution anywhere.

Output:
I think you have done a very naive mistake here, you are reading the model object in the function which you are parallelizing.

Meaning for every single image, you are reloading the model from the disk.
Depending on your model object size, IO is gonna be more time consuming then running a forward step.

Please consider reading the model once in the main thread and then make the object available for inference in the parallel function.


Instruction: Assign a tensor to multiple slices

Input:
Let

a = tensor([[0, 0, 0, 0],
            [0, 0, 0, 0],
            [0, 0, 0, 0]])
b = torch.tensor([1, 2])
c = tensor([[1, 2, 0, 0],
            [0, 1, 2, 0],
            [0, 0, 1, 2]])

Is there a way to obtain c by assigning b to slices of a without any loops? That is, a[indices] = b for some indices or something similar?

Output:
You can use scatter method in pytorch.

a = torch.tensor([[0, 0, 0, 0],
                 [0, 0, 0, 0],
                 [0, 0, 0, 0]])

b = torch.tensor([1, 2])

index = torch.tensor([[0,1],[1,2],[2,3]])

a.scatter_(1, index, b.view(-1,2).repeat(3,1))
# tensor([[1, 2, 0, 0],
#         [0, 1, 2, 0],
#         [0, 0, 1, 2]])


Instruction: Custom c++ extension error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token

Input:
I’m trying to add custom c++ extension with pytorch.

I’m following the tutorial at https://pytorch.org/tutorials/advanced/cpp_extension.html

I have created two files.

project/
  main.py
  sigmoid.cpp

main.py

from torch.utils.cpp_extension import load
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)

sigmoid.cpp

#include <torch/extension.h>

#include <iostream>

torch::Tensor d_sigmoid(torch::Tensor z) {
  auto s = torch::sigmoid(z);
  return (1 - s) * s;
}

PYBIND11_MODULE('sigmoid', m) {
  m.def('d_sigmoid', &d_sigmoid, 'sigmoid');
}

when I run main.py I get the following error

  error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid
  preprocessing token

How can I fix the error?

Full error report

Using /tmp/torch_extensions as PyTorch extensions root…
Emitting ninja build file /tmp/torch_extensions/sigmoid/build.ninja…
Building extension module sigmoid…
[1/2] c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
FAILED: sigmoid.o
c++ -MMD -MF sigmoid.o.d -DTORCH_EXTENSION_NAME=sigmoid -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /home/wickrama/anaconda3/envs/pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++11 -c /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp -o sigmoid.o
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “PyInit_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: error: pasting “pybind11_init_” and “‘sigmoid’” does not give a valid preprocessing token
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:1: warning: character constant too long for its type
PYBIND11_MODULE(‘sigmoid’, m) {
^~~~~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:9: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~~~
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:11:34: warning: character constant too long for its type
m.def(‘d_sigmoid’, &d_sigmoid, ‘sigmoid’);
^~~~~~~~~
In file included from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pytypes.h:12:0,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/cast.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/attr.h:13,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/pybind11/pybind11.h:44,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/utils/pybind.h:6,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/python.h:12,
from /home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/include/torch/extension.h:6,
from /home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:1:
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
/home/wickrama/projects/torch_cpp_ext/sigmoid.cpp:10:17: error: expected initializer before ‘\x6d6f6964’
PYBIND11_MODULE(‘sigmoid’, m) {
^
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 949, in _build_extension_module
check=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/subprocess.py”, line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/wickrama/projects/torch_cpp_ext/main.py”, line 3, in
lltm_cpp = load(name=‘sigmoid’, sources=[‘sigmoid.cpp’], verbose=True)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 644, in load
is_python_module)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 813, in _jit_compile
with_cuda=with_cuda)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 866, in _write_ninja_file_and_build
_build_extension_module(name, build_directory, verbose)
File “/home/wickrama/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 962, in _build_extension_module
raise RuntimeError(message)
RuntimeError: Error building extension ‘sigmoid’

Output:
My guess: unlike in Python, single quotes (') are not equivalent with double quotes (") in c++. The former are used to construct char literals, so they can only wrap a single character, and the latter are used to construct string literals (strictly speaking const char * literals). So try replacing

PYBIND11_MODULE('sigmoid', m) {
  m.def('d_sigmoid', &d_sigmoid, 'sigmoid');
}

with

PYBIND11_MODULE("sigmoid", m) {
  m.def("d_sigmoid", &d_sigmoid, "sigmoid");
}

and see if that helps.


Instruction: Pytorch Question from 'Deep Reinforcement Learning: Hands-On'

Input:
I'm reading Maxim Lapan's Deep Learning Hands On. I came across this code in chapter 2 and I don't understand a few things. Could anybody explain why the output of print(out) gives three parameters instead of the single float tensor we put in. Also, why is the super function necessary here? Finally, what is the x parameter that forward is accepting? Thank you.

class OurModule(nn.Module):
    def __init__(self, num_inputs, num_classes, dropout_prob=0.3):  #init
        super(OurModule, self).__init__() #Call OurModule and pass the net instance (Why is this necessary?)
        self.pipe = nn.Sequential( #net.pipe is the nn object now
            nn.Linear(num_inputs, 5),
            nn.ReLU(),
            nn.Linear(5, 20),
            nn.ReLU(),
            nn.Linear(20, num_classes),
            nn.Dropout(p=dropout_prob),
            nn.Softmax(dim=1)
        )

    def forward(self, x): #override the default forward method by passing it our net instance and (return the nn object?). x is the tensor? This is called when 'net' receives a param?
        return self.pipe(x)

if __name__ == "__main__":
    net = OurModule(num_inputs=2, num_classes=3)
    print(net)
    v = torch.FloatTensor([[2, 3]])
    out = net(v)
    print(out) #[2,3] put through the forward method of the nn? Why did we get a third param for the output?
    print("Cuda's availability is %s" % torch.cuda.is_available()) #find if gpu is available
    if torch.cuda.is_available():
        print("Data from cuda: %s" % out.to('cuda'))

OurModule.__mro__

Output:
OurModule defined a PyTorch nn.Module that accepts 2 inputs (num_inputs) and produces 3 outputs (num_classes).

It consists of:

A Linear layers that accepts 2 inputs and produces 5 outputs
A ReLU
A Linear layer that accepts 5 inputs and produces 20 outputs
A ReLU
A Linear layer that accepts 20 inputs and produces 3 (num_classes) outputs
A Dropout layer
A Softmax layer

You create v which consists of 2 inputs and pass it through this network's forward() method when you call net(v). The result of running this network (3 outputs) is then stored in out.

In your example, x takes on the value of v, torch.FloatTensor([[2, 3]])


Instruction: Pytorch - how to extract features of an MLP network (weights, biases, number of nodes, hidden layers)?

Input:
I would be interested to extract the weights, biases, number of nodes and number of hidden layers from an MLP/neural network built in pytorch. I wonder if anyone may be able to point me in the right direction?

Many thanks,

Max

Output:
Yes, we can do what you want by first creating a simple network:

input_dim = 400
hidden_dim = 512

net = nn.Sequential(nn.Linear(input_dim, hidden_dim),
                      nn.Sigmoid())

print(net)

When we print the net, we get to know the number of layers, number of nodes (out_features), and many other details:

Sequential(
  (0): Linear(in_features=400, out_features=512, bias=True)
  (1): Sigmoid()
)

Then if you want to load specific values of each parameter, you can print that too

model = Net(400, 512,10)

bias = model.fc1.bias

print(bias)

the output is:

tensor([ 3.4078e-02,  3.1537e-02,  3.0819e-02,  2.6163e-03,  2.1002e-03,
         4.6842e-05, -1.6454e-02, -2.9456e-02,  2.0646e-02, -3.7626e-02,
         3.5531e-02,  4.7748e-02, -4.6566e-02, -1.3317e-02, -4.6593e-02,
        -8.9996e-03, -2.6568e-02, -2.8191e-02, -1.9806e-02,  4.9720e-02,
        ---------------------------------------------------------------
        -4.6214e-02, -3.2799e-02, -3.3605e-02, -4.9720e-02, -1.0293e-02,
         3.2559e-03, -6.6590e-03, -1.2456e-02, -4.4547e-02,  4.2101e-02,
        -2.4981e-02, -3.6840e-03], requires_grad=True)

Hope it helps


Instruction: Right place to do class imbalance regularisation (data level or batch level)

Input:
I have two binary imbalanced dataset where the labels are ether 0 or 1 and prediction output is between 0 and 1. The positive case has 10000 samples, while the negative case has 90000 samples. I'm using a batch of 100 when training.

when calculating the BinaryCrossEntropyLoss (in pytorch) its possible to supply the per batch element regularisation weight.

My question is:
To calculate the general class weight dose it make more sense to calculate it 1 time at the start (so 1/(10000/(100000) for the positive case) and scale the loss of each sample with this value

or:

Calculate the weight at the batch level, by firstly finding the batch class imbalance (e.g in the batch it might be 25 positives and 75 negatives, hence 1/(25/(25+75) for the positive case)

I'm asking this because the loss is averaged across the batch

Output:
If you wish to do it this way, you should calculate per batch class imbalance.

On the other hand you should probably make sure that each batch preserves label statistics (e.g. for batch 64 and your case, you should have 6 positive samples and the rest negative). This way, it would be enough to calculate class imbalance once and add it to torch.nn.BCELoss on a per-batch basis.

I would suggest the other approach though, e.g. oversampling or undersampling using PyTorch's Sampler class (don't do it by copying examples, it wastes space totally unnecessarily). You can implement it manually or use third party library which did it for you for example torchdata (disclosure: I'm the author) and torchdata.samplers.RandomOverSampler.


Instruction: Where does Ray.Tune create the model vs implementing the perturbed hyperparameters

Input:
I am new to using ray.tune. I already have my network written in a modular format and now I am trying to incorporate ray.tune, but I do not know where to initialize the model (vs updating the perturbed hyperparameters) so that the model and the weights are not re-initialized when a worker is truncated and replaced by a better performing worker.

Background

I am using the PBT scheduler of ray.tune which creates num_samples number of models (workers) each of which are initialized with a different set of sampled hyperparameters. When a model is evaluated, if it is performing poorly, it will be stopped and load the checkpoint of one of the top performing workers. Once it is loaded (this is a deep copy of the network), the hyperparameters are perturbed and then it will train until the next evaluation.
The MyTrainable class should have a _setup, _train, _save, and _restore function. The setup calls for a config variable and this is where the newly sampled hyperparameters are implemented.

My question is where should be the original model be defined? I can easily implement the updated HPs in this section. But I have not seen anywhere in the documentation where I can pass a pre-defined model into the ray.tune.run function. If I keep the create_model() function in the _setup() though, it will eliminate the previously trained weights which is part of the benefit of this method.

Code

Here are the 3 functions I have:

self._hyperparameters(config)    # redefines the self.opt options accoring to the new perturbations
self.model.update_optimizer(self.opt)  # redefines the optimizers using the new learning rates and the beta values for Adam

self.model = create_model(self.opt)   # Original function that defines the initial model and initializes the weights

Output: The create_model should be called in _setup. _restore will be called after _setup, and in restore, the model should be updated to the weights stored in the checkpoint.


Instruction: Understanding TypeError: '<' not supported between instances of 'Example' and 'Example'

Input:
I am working on a project of text simplification using a multi-head attention transformer model. For the same, I am using torchtext for tokenisation and numericalization. The dataset contains two aligned files for training and two aligned files for testing. In the training files, one file contains the complex sentences while the other contains the corresponding simplified sentences.

I read the files as such:

training_sentences = open(path + "train.en" , encoding = "utf-8").read().split("\n")
target_sentences = open(path + "train.sen" , encoding = "utf-8").read().split("\n")

Next, I tokenised them as such:

complicated = spacy.load('en')
simple = spacy.load('en')

def tokenize_complicated(sentence):
   return [tok.text for tok in complicated.tokenizer(sentence)]

def tokenize_simple(sentence):
    return [tok.text for tok in simple.tokenizer(sentence)]

C_TEXT = Field(tokenize=tokenize_complicated, fix_length = 100)
S_TEXT = Field(tokenize=tokenize_simple, fix_length = 100, init_token = "<sos>", eos_token = "<eos>")

I then converted into TabularDataset object of torchtext.

import pandas as pd
raw_data = {'Complicated' : [line for line in training_sentences],
        'Simple': [line for line in target_sentences]}

df = pd.DataFrame(raw_data, columns=["Complicated", "Simple"])

df.to_csv("df.csv", index=False)
data_fields = [('Complicated', C_TEXT), ('Simple', S_TEXT)]

train = torchtext.data.TabularDataset.splits(path='./', train = "df.csv", format='csv', fields=data_fields, skip_header = True)

And then created vocabulary

C_TEXT.build_vocab(train)
S_TEXT.build_vocab(train)

However, on doing so I got this error:

  TypeError: '<' not supported between instances of 'Example' and
  'Example'

On searching, I came across this solution here and the error disappeared. However, I am not understanding whether this makes the model take only one instance or it takes all of the dataset?
I would like to know the significance of the index [0] so that I can manipulate it effectively for my model.

Output:
In my case I solved the issue passing a sort_key and sort_within_batch arg to True as follows:
BATCH_SIZE = 64
train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    device = device,
    batch_size = BATCH_SIZE,
    sort_key = lambda x: len(x.src),
    sort_within_batch=True
)

Good luck


Instruction: How to Fix "RuntimeError: CUDA error: device-side assert triggered" in Pytorch

Input:
I am trying to train the yolo-v3 model from this repo https://github.com/eriklindernoren/PyTorch-YOLOv3
on my custom dataset of shapes, but I keep getting the error "RuntimeError: CUDA error: device-side assert triggered"

I have tried to lookup the solution and tried several things suggested in different answers (like fixing the indexing of the classes in the annotations) but the error persists.

I am following the description in the readme of the repo to train on a custom dataset, and have adjusted custom.data and the data/custom/ accordingly.

I keep receiving this output.

C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [33,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [36,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [37,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [38,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [39,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [40,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [41,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [42,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [43,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [44,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [45,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [3,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [4,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [5,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [6,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [7,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [12,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [13,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [14,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [15,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [20,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [21,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [22,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [23,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [24,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [25,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [26,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [27,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [28,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [29,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
C:/w/1/s/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [0,0,0], thread: [31,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "train.py", line 105, in <module>
    loss, outputs = model(imgs, targets)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\models.py", line 259, in forward
    x, layer_loss = module[0](x, targets, img_dim)
  File "C:\Users\user\AppData\Local\Programs\Python\Python36\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\models.py", line 188, in forward
    ignore_thres=self.ignore_thres,
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\utils\utils.py", line 318, in build_targets
    iou_scores[b, best_n, gj, gi] = bbox_iou(pred_boxes[b, best_n, gj, gi], target_boxes, x1y1x2y2=False)
  File "D:\Documents\GP\Code\TorchYolo\PyTorch-YOLOv3\utils\utils.py", line 199, in bbox_iou
    b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2
RuntimeError: CUDA error: device-side assert triggered

with the only thing changing being the "2" in the array index when messing around with the train.jpg class label index

b1_x1, b1_x2 = box1[:, 0] - box1[:, 2] / 2, box1[:, 0] + box1[:, 2] / 2

Output:
I also had this issue while training resnet model on google colab. So in my case I was training the model on 7 classes but the last layer of my network was set to output 3 classes.
Hence I changed
self.classifier = torch.nn.Sequential(torch.nn.BatchNorm1d(512), torch.nn.Linear(512, 3))

to this
self.classifier = torch.nn.Sequential(torch.nn.BatchNorm1d(512), torch.nn.Linear(512, 7))

After doing that also, i was still getting the error, because I didn't restart the google colab.
Remember whenever you get this error, check two things:

class_labels should start from 0 i.e. in my case [0,1,2,3,4,5,6] for 7 classes.
Check whether the final output layer outputs the exact number of classes.

And after that,
Refresh the notebook to flush all the cuda asserts.
After any of the CUDA error, restart the notebook, otherwise you will keep on getting the CUDA error because the earlier assertion hasn't been flushed out. By restarting the notebook, you will flush out all the cuda assertions.


Instruction: How do I train an LSTM in Pytorch?

Input:
I am having a hard time understand the inner workings of LSTM in Pytorch.

Let me show you a toy example. Maybe the architecture does not make much sense, but I am trying to understand how LSTM works in this context.

The data can be obtained from here. Each row i (total = 1152) is a slice, starting from t = i until t = i + 91, of a longer time series. I will extract the last column of each row to use as labels.

import torch
import numpy as np
import pandas as pd
from torch import nn, optim
from sklearn.metrics import mean_absolute_error

data = pd.read_csv('data.csv', header = None).values
X = torch.tensor(data[:, :90], dtype = torch.float).view(1152, 1, 90)
y = torch.tensor(data[:, 90], dtype = torch.float).view(1152, 1, 1)

dataset = torch.utils.data.TensorDataset(X, y)
loader = torch.utils.data.DataLoader(dataset, batch_size = 50)

Then I am defining an LSTM regressor containing three LSTM layers with different structures.

class regressor_LSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm1 = nn.LSTM(input_size = 49, hidden_size = 100)
        self.lstm2 = nn.LSTM(100, 50)
        self.lstm3 = nn.LSTM(50, 50, dropout = 0.3, num_layers = 2)
        self.dropout = nn.Dropout(p = 0.3)
        self.linear = nn.Linear(in_features = 50, out_features = 1)

    def forward(self, X):
        X, _ = self.lstm1(X)
        X = self.dropout(X)
        X, _ = self.lstm2(X)
        X = self.dropout(X)
        X, _ = self.lstm3(X)
        X = self.dropout(X)
        X = self.linear(X)

        return X

Initializing what needs to be initialized:

regressor = regressor_LSTM()
criterion = nn.MSELoss()
optimizer = optim.RMSprop(regressor.parameters())

Then training:

for epoch in range(25):
    acc_loss = 0.
    acc_mae = 0.
    for i, data in enumerate(loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = regressor(inputs)
        loss = criterion(outputs, labels)
        loss.backward(retain_graph = True)
        optimizer.step()
        acc_loss += loss.item()
        mae = mean_absolute_error(labels.detach().cpu().numpy().flatten(), outputs.detach().cpu().numpy().flatten())
        acc_mae += mae
#       print('\rEPOCH {:3d} - Loop {:3d} of {:3d}: loss {:03.2f} - MAE {:03.2f}'.format(epoch+1, i+1, len(loader), loss, mae), end = '\r')
    print('\nEPOCH %3d FINISHED: loss %.5f - MAE %.5f' % (epoch+1, acc_loss/len(loader), acc_mae/len(loader)))

The thing is, after some initial decrease in both loss and MAE (expected behavior), both seem to get stuck (showing only first 10 epochs below):

EPOCH   1 FINISHED: loss 0.38506 - MAE 0.27322
EPOCH   2 FINISHED: loss 0.02825 - MAE 0.13601
EPOCH   3 FINISHED: loss 0.02593 - MAE 0.13117
EPOCH   4 FINISHED: loss 0.02568 - MAE 0.12705
EPOCH   5 FINISHED: loss 0.02546 - MAE 0.12920
EPOCH   6 FINISHED: loss 0.02502 - MAE 0.12763
EPOCH   7 FINISHED: loss 0.02445 - MAE 0.12659
EPOCH   8 FINISHED: loss 0.02310 - MAE 0.12328
EPOCH   9 FINISHED: loss 0.02277 - MAE 0.12237
EPOCH  10 FINISHED: loss 0.02352 - MAE 0.12476

When run with Keras, both metrics decrease consistently throughout the process. (I also noticed Keras takes much longer.)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM
import pandas as pd

data = pd.read_csv('data.csv', header = None).values
X = data[:, :90].reshape(1152, 90, 1)
y = data[:, 90]

regressor = Sequential()
regressor.add(LSTM(units = 100, return_sequences = True, input_shape = (90, 1)))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.3))
regressor.add(Dense(units = 1, activation = 'linear'))
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['mean_absolute_error'])
regressor.fit(X, y, epochs = 25, batch_size = 32)

[OUTPUT]
Epoch 1/25
1152/1152 - 35s 30ms/sample - loss: 0.0307 - mean_absolute_error: 0.1225
Epoch 2/25
1152/1152 - 32s 28ms/sample - loss: 0.0156 - mean_absolute_error: 0.0978
Epoch 3/25
1152/1152 - 32s 28ms/sample - loss: 0.0126 - mean_absolute_error: 0.0871
Epoch 4/25
1152/1152 - 34s 30ms/sample - loss: 0.0111 - mean_absolute_error: 0.0806
Epoch 5/25
1152/1152 - 29s 25ms/sample - loss: 0.0103 - mean_absolute_error: 0.0785
Epoch 6/25
1152/1152 - 29s 25ms/sample - loss: 0.0088 - mean_absolute_error: 0.0718
Epoch 7/25
1152/1152 - 32s 27ms/sample - loss: 0.0085 - mean_absolute_error: 0.0699
Epoch 8/25
1152/1152 - 30s 26ms/sample - loss: 0.0069 - mean_absolute_error: 0.0640
Epoch 9/25
1152/1152 - 30s 26ms/sample - loss: 0.0077 - mean_absolute_error: 0.0660
Epoch 10/25
1152/1152 - 30s 26ms/sample - loss: 0.0070 - mean_absolute_error: 0.0644

I've been reading about hidden state initialization, I tried to set them to 0 in the beginning of the forward method (which, though, I understood to be the standard behavior), but nothing helped. I must confess that I do not understand what the parameters of an LSTM are, nor which should be reinitialized (if any) after each batch or epoch.

I appreciate any return!

Output:
I am coming back after a few days because I have come to a conclusion. After reading some material on hidden/cell states, (this one was quite useful) it seems that reusing them is a matter of net design choice. Whether doing so, and when, can count as a hyperparameter. I tried many options with my toy dataset, mainly resetting the states after each batch, resetting after each epoch, and not resetting at all, and the results were quite similar. Also, my results were so low because (as I believe) I did not choose shuffle = True in the loader; doing so made them considerably better (loss around 0.003, MAE around 0.047).

In the original code for the LSTM class, line 510, it also seems that hidden/cell states are initiated at zero if no values are explicitly passed.


Instruction: Deriving the structure of a pytorch network

Input:
For my use case, I require to be able to take a pytorch module and interpret the sequence of layers in the module so that I can create a “connection” between the layers in some file format. Now let’s say I have a simple module as below

class mymodel(nn.Module):
    def __init__(self, input_channels):
        super(mymodel, self).__init__()
        self.fc = nn.Linear(input_channels, input_channels)
    def forward(self, x):
        out = self.fc(x)
        out += x
        return out

if __name__ == "__main__":
    net = mymodel(5)

    for mod in net.modules():
        print(mod)

Here the output yields:

mymodel(
  (fc): Linear(in_features=5, out_features=5, bias=True)
)
Linear(in_features=5, out_features=5, bias=True)

as you can see the information about the plus equals operation or plus operation is not captured as it is not a nnmodule in the forward function. My goal is to be able to create a graph connection from the pytorch module object to say something like this in json :

layers {
"fc": {
"inputTensor" : "t0",
"outputTensor": "t1"
}
"addOp" : {
"inputTensor" : "t1",
"outputTensor" : "t2"
}
}

The input tensor names are arbitrary but it captures the essence of the graph and the connections between layers.

My question is, is there a way to extract the information from a pytorch object? I was thinking to use the .modules() but then realized that hand written operations are not captured this way as a module. I guess if everything is an nn.module then the .modules() might give me the network layer arrangement. Looking for some help here. I want to be able to know the connections between tensors to create a format as above.

Output:
The information you are looking for is not stored in the nn.Module, but rather in the grad_fn attribute of the output tensor:

model = mymodel(channels)
pred = model(torch.rand((1, channels))
pred.grad_fn  # all the information is in the computation graph of the output tensor

It is not trivial to extract this information. You might want to look at torchviz package that draws a nice graph from the grad_fn information.


Instruction: tf.function property in pytorch

Input:
I'm a beginner in pytorch, and I have some functions that are needed to implement in network.

My question is: is there any way like tf.function, or should I use "class(nn.Module)" with variable?

For example, let X be a 10x2 matrix . In pseudo-code:

a = Variable(1.0)
b = Variable(1.0)
Y = a*X[:,0]**2 + b*X[:,1]

Output:
In PyTorch you don't need things like tf.function, you just use normal Python code (because of the dynamic graph).

Please give more detailed example (with code) of what you're trying to do if the above doesn't answer your question.


Instruction: Is this how nn.Transformer works?

Input:
If I want to transform an image to another image,
then

transformer_model = nn.Transformer(img_size, n_heads)
transformer_model(source_image, target_image)

is this the correct way to use nn.Transformer?

Output:
No, this is not what the Transformer module does. The Transformer is primarily used for pre-training general use models for NLP on large bodies of text. If you're curious to learn more, I strongly recommend you read the article which introduced the architecture, "Attention is All You Need". If you've heard of models like BERT or GPT-2, these are examples of transformers.

It's not entirely clear what you are trying to accomplish when you ask how to "transform an image into another image." I'm thinking maybe you are looking for something this? https://junyanz.github.io/CycleGAN/

In any event, to re-answer your question: no, that's not how you use nn.Transformer. You should try to clarify what you are trying to accomplish with "transforming one picture into another," and post that description as a separate question.


Instruction: EnsembleVoteClassifier with neural network

Input:
I have a trained neural networks in which I am trying to average their prediction using EnsembleVoteClassifier from mlxtend.classifier. The problem is my neural network don't share the same input, (I performed feature reduction and feature select algorithms randomly and stored the results on new different variables, so I have something like X_test_algo1, X_test_algo2 and X_test_algo3 and Y_test).

I am trying to average the weights, but as I said, I don't have the same X, and I didn't any example on the documentation. How can I average the predictions for my three models model1, model2 and model3

eclf = EnsembleVoteClassifier(clfs=[model1, model2, model3], weights=[1,1,1], refit=False)
names = ['NN1', 'NN2', 'NN2', 'Ensemble']

eclf.fit(X_train_algo1, Ytrain) #????

If it's not possible, that is okay. I am only interested on how to calculate the formulas of Hard Voting, Hard Voting and Weighted Voting, or if there is anther library that is more flexible or the explicit expressions of the formulas could be helpful too.

Output:
Why would you need a library to do that?

Simply pass the same examples through all your neural networks and get the predictions (either logits or probabilities or labels).

Hard voting choose the label predicted most often by classifiers.
Soft voting, average probabilities predicted by classifiers and choose the label having the highest.
Weighted voting - either of the above can be weighted. Just assign weights to each classifier and multiply their predictions by them. Weights are usually normalized to (0, 1] range.

In principle you could also sum logits and choose the label with highest.

Oh, and weight averaging is different technique and requires you to have the same model and usually is done for the same initialization but at different training timesteps. You can read about it in this blog post.


Instruction: Segmentation fault when using the custom activation function

Input:
I am trying to implement a custom activation function (the codes attached below). Before using the custom activation function, everything works well. However, as long as it is used, the server would throw the error:

Segmentation fault

The error always appears at the first epoch.

I am using

Pytorch 1.1.0
Cuda compilation tools, release 9.2, V9.2.148
the codes

def mg(x):

    c = 1.33
    b = 0.4
    p = 6.88
    input_size = x.shape
    num = torch.numel(x) # the element number of the input tensor
    x = x.view(num)

    out = torch.zeros(len(x))

    for i in range(len(x)):
    if x[i] < 0:
            out[i] = 0
        else:
            out[i] = (c * x[i]) / (1 + torch.mul(b * p, torch.pow(x[i], p)))

    out = out.view(input_size[0], input_size[1], input_size[2], input_size[3])
    return out

Output:
You are breaking the gradient with newly created out.

You should modify your code to act upon x inputs. Additionally, you shouldn't use any loops (almost always there is a way to do it without them). Given that, this function should be equivalent to yours but works:

def mg(x, c=1.33, b=0.4, p=6.88):
    input_size = x.shape
    x = x.flatten()

    x[x < 0] = 0
    x[x != 0] *= c
    x[x != 0] /= 1 + b * p * x[x != 0] ** p

    return x.reshape(*input_size)

If you are still getting an error it's probably related to some other part of your program.


Instruction: Restarting with adam

Input:
I am training my network with early stopping strategy. I start with a higher learning rate, and based on validation loss, I need to restart training from an earlier snapshot.

I am able to save/load snapshot with model and optimizer state_dicts. No problem with that.

My question is, once I restart training, how do I set the learning rate of adam again? Should I restart adam fresh instead of using a state_dict or should I use
optimizer.param_groups[0][‘lr’] = lr to adjust learning rate with loaded optimizer state_dict?

For example,
I train my network with lr = 1e-6 for 5 epochs, saved model and optimizer state_dict.
I am now restarting from epoch 6, but I need lr = 1e-7 instead. What is the best approach for this?

Thanks!

Output:
Looking further into the scheduler code, I found the correct way to do it as:

def get_lr(gamma, optimizer):
    return [group['lr'] * gamma
            for group in optimizer.param_groups]

for param_group, lr in zip(optimizer.param_groups, get_lr(gamma, optimizer)):
    param_group['lr'] = lr


Instruction: What is the logic behind this assignment: understanding in-place assignment operations in numpy

Input:
I have two fairly simple codes that give different answer. I understand it is due to the reference shared but I am not very clear what exactly happens in the 2nd case

a = np.ones(5)
b = torch.from_numpy(a)
a=np.add(a, 1, out=a)
print(a)
print(b)

  [out]:

  [2. 2. 2. 2. 2.]

  tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

a = np.ones(5)
b = torch.from_numpy(a)
a=a+1
print(a)
print(b)

  [out]:

  [2. 2. 2. 2. 2.]

  tensor([1., 1., 1., 1., 1.], dtype=torch.float64)

Why isn't b changed in the second case ?

Output:
In the first case both a and b share the same memory (i.e. b is a view of a or in other words, b is pointing to the (array) value where a is also pointing to) and out argument guarantees that the same memory of a is updated after the np.add() operation is completed.  Whereas in the second case, a is a new copy when you do a = a+1 and b is still pointing to the old value of a.

Try the second case with:

a += 1

and observe that both a and b are indeed updated.

In [7]: a = np.ones(5)
   ...: b = torch.from_numpy(a)
   ...: a += 1

In [8]: a
Out[8]: array([2., 2., 2., 2., 2.])

In [9]: b
Out[9]: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

As @hpaulj aptly pointed out in his comment, when we do a = a+1, a new object is created and a would now point to this new (array) object instead of the old one, which is still pointed to by b. And this is the reason the (array) value of b is not updated.

To understand this behavior a bit better, you might wanna refer the excellent article by Ned Batchelder about how names are bind to values in Python


Instruction: Can't see graph using torch.utils.tensorboard

Input:
I'm trying to get used to tensorboard, and I code my models using pytorch.
However when I try to see my model using the add_graph() function, I've got this:

With this as the test code:

import numpy as np
import torch
import torchvision.transforms as transforms
import torch.nn as nn
import torch.optim as optim
from torch.utils.tensorboard import SummaryWriter

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear = nn.Linear(2, 1)

    def forward(self, x):
        x = self.linear(x)
        return x

writer = SummaryWriter('runs_pytorch/test')
net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
writer.add_graph(net, torch.zeros([4, 2], dtype=torch.float))
writer.close()

On the other hand, if I try to see a graph using TensorFlow, everything seems fine:

with this as the test code this time:

import tensorflow as tf
tf.Variable(42, name='foo')
w = tf.summary.FileWriter('runs_tensorflow/test')
w.add_graph(tf.get_default_graph())
w.flush()
w.close()

In case you are wondering, I'm using this command to start tensorboard:

tensorboard --logdir runs_pytorch

Something I noticed is that when I use it on the directory allocated for my tensorflow test, I've got the usual message with the address, but if I do the same thing with  --logdir runs_pytorch I've got something more:

W1010 15:19:24.225109 15308 plugin_event_accumulator.py:294] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1010 15:19:24.226075 15308 plugin_event_accumulator.py:322] Found more than one "run metadata" event with tag step1. Overwriting it with the newest event.

I'm on windows, I tried on different browsers (chrome, firefox...).
I have tensorflow 1.14.0, torch 1.2.0, Python 3.7.3

Thank you very much for your help, it's driving me crazy!

Output:
There are two ways to solve it:

1. update PyTorch to 1.3.0 and above:

conda way:

    conda install pytorch torchvision cudatoolkit=9.2 -c pytorch

pip way:

    pip3 install torch==1.3.0+cu92 torchvision==0.4.1+cu92 -f https://download.pytorch.org/whl/torch_stable.html

2. install tensorboardX instead:

uninstall tensorboard:

if your tensorboard is installed by pip:

    pip uninstall tensorboard

if your tensorboard is installed by anaconda:

    conda uninstall tensorboard

install tensorboardX

    pip install tensorboardX

when writing script,
change

    from torch.utils.tensorboard import SummaryWriter

to

    from tensorboardX import SummaryWriter


Instruction: Pytorch copying inexact value of numpy floating point number

Input:
I'm converting a floating point number (or numpy array) to Pytorch tensor and it seems to be copying the inexact value to the tensor. The error comes in the 8th significant digit and afterwards. This is significant (no-pun intended) for my work as I deal with chaotic dynamics which is very sensitive towards the slight change in the initial conditions.

I'm already using torch.set_printoptions(precision=16) to print 16 significant digits.

np_x = state
print(np_x)
x = torch.tensor(np_x,requires_grad=True,dtype=torch.float32)
print(x.data[0])

and the output is :

0.7575408585008059
tensor(0.7575408816337585)

It would be helpful to know what is going wrong or how it could be resolved ?

Output:
Because you're using float32 dtype. If you convert these two numbers to binary, you will find they are actually the same. Strictly speaking, the most accurate representations of those two numbers in float32 format are the same.

0.7575408585008059
Most accurate representation = 7.57540881633758544921875E-1

0.7575408816337585
Most accurate representation = 7.57540881633758544921875E-1

Binary: 00111111 01000001 11101110 00110011


Instruction: How to fix the error `Process finished with exit code -1073741819 (0xC0000005)`

Input:
My problem is when I running FC network the code works well in both CPU and GPU. But when it comes to CNN, I can only train it on CPU. It raises an error when I try to train it on GPU.

Like that:

  Process finished with exit code -1073741819 (0xC0000005)

I find the error raised when the code goes to loss.backword. The error happened when I use the first column instead of the second.

device = torch.device("cuda:0")

device = torch.device("cuda:0" if opt.cuda else "cpu")

My environment is Python 3.6.9, Windows 10, Torch 1.2.0, Cuda 9.2.

Output:
Finally, I figured out this.

This error occurs just because one of my variables is not loaded in cuda.

When I add this output = Variable(netD(real_cpu),requires_grad=True) the problem solved.


Instruction: How to initialize mean and variance of Pytorch BatchNorm2d?

Input:
I’m transforming a TensorFlow model to Pytorch. And I’d like to initialize the mean and variance of BatchNorm2d using TensorFlow model.
I’m doing it in this way:

bn.running_mean = torch.nn.Parameter(torch.Tensor(TF_param))

And I get this error:

RuntimeError: the derivative for 'running_mean' is not implemented

But is works for bn.weight and bn.bias. Is there any way to initialize the mean and variance using my pre-trained Tensorflow model? Is there anything like moving_mean_initializer and moving_variance_initializer in Pytorch?

Thanks!

Output:
The running mean and variance of a batch norm layer are not nn.Parameters, but rather a buffer of the layer.

I think you can simply assign a torch.tensor, no need to wrap a nn.Parameter around it.


Instruction: How to avoid create a network structure properly

Input:
I tried to define a feedforwad function in my neural network model:

class FeedForward(nn.Module):
    def __init__(self):
        super(FeedForward,self).__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 10)

    def feedforward(self, x):
        x = x.view(x.shape[0], -1)  # make sure inputs are flattened

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x= F.log_softmax(x, dim=1)  # preserve batch dim

        return x

The message says:

  NotImplementedError

I am not sure what I have missing.

Output:
The method name must be forward and not feedforward:

class FeedForward(nn.Module):
    def __init__(self):
        super(FeedForward,self).__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 64)
        self.fc2 = nn.Linear(64, 10)

    def forward(self, x):  # this is what pytorch expects
        x = x.view(x.shape[0], -1)  # make sure inputs are flattened

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x= F.log_softmax(x, dim=1)  # preserve batch dim

        return x


Instruction: Translating LSTM model from Keras to Pytorch

Input:
I am having a hard time translating a quite simple LSTM model from Keras to Pytorch. X (get it here) corresponds to 1152 samples of 90 timesteps, each timestep has only 1 dimension. y (here) is a single prediction at t = 91 for all 1152 samples.

In Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM
import numpy as np
import pandas as pd

X = pd.read_csv('X.csv', header = None).values
X.shape

y = pd.read_csv('y.csv', header = None).values
y.shape

# From Keras documentation [https://keras.io/layers/recurrent/]:
# Input shape 3D tensor with shape (batch_size, timesteps, input_dim).
X = np.reshape(X, (1152, 90, 1))

regressor = Sequential()
regressor.add(LSTM(units = 100, return_sequences = True, input_shape = (90, 1)))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.3))
regressor.add(Dense(units = 1, activation = 'linear'))
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['mean_absolute_error'])
regressor.fit(X, y, epochs = 10, batch_size = 32)

... leads me to:

# Epoch 10/10
# 1152/1152 [==============================] - 33s 29ms/sample - loss: 0.0068 - mean_absolute_error: 0.0628

Then in Pytorch:

import torch
from torch import nn, optim
from sklearn.metrics import mean_absolute_error

X = pd.read_csv('X.csv', header = None).values
y = pd.read_csv('y.csv', header = None).values

X = torch.tensor(X, dtype = torch.float32)
y = torch.tensor(y, dtype = torch.float32)

dataset = torch.utils.data.TensorDataset(X, y)
loader = torch.utils.data.DataLoader(dataset, batch_size = 32, shuffle = True)

class regressor_LSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm1 = nn.LSTM(input_size = 1, hidden_size = 100)
        self.lstm2 = nn.LSTM(100, 50)
        self.lstm3 = nn.LSTM(50, 50, dropout = 0.3, num_layers = 2)
        self.dropout = nn.Dropout(p = 0.3)
        self.linear = nn.Linear(in_features = 50, out_features = 1)

    def forward(self, X):
        # From the Pytorch documentation [https://pytorch.org/docs/stable/_modules/torch/nn/modules/rnn.html]:
        # **input** of shape `(seq_len, batch, input_size)`
        X = X.view(90, 32, 1)
        # I am discarding hidden/cell states since in Keras I am using a stateless approach
        # [https://keras.io/examples/lstm_stateful/]
        X, _ = self.lstm1(X)
        X = self.dropout(X)
        X, _ = self.lstm2(X)
        X = self.dropout(X)
        X, _ = self.lstm3(X)
        X = self.dropout(X)
        X = self.linear(X)

        return X

regressor = regressor_LSTM()
criterion = nn.MSELoss()
optimizer = optim.RMSprop(regressor.parameters())

for epoch in range(10):
    running_loss = 0.
    running_mae = 0.

    for i, data in enumerate(loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = regressor(inputs)
        outputs = outputs[-1].view(*labels.shape)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
        mae = mean_absolute_error(labels.detach().cpu().numpy().flatten(), outputs.detach().cpu().numpy().flatten())
        running_mae += mae

    print('EPOCH %3d: loss %.5f - MAE %.5f' % (epoch+1, running_loss/len(loader), running_mae/len(loader)))

... leads me to:

# EPOCH  10: loss 0.04220 - MAE 0.16762

You can notice that both loss and MAE are quite different (Pytorch's are much higher). If I use Pytorch's model to predict the values, they all return as a constant.

What am I doing wrong?

Output:
Oh I believe I made considerable progress. It seems that the way to represent y is different between Keras and Pytorch. In Keras, we should pass it as a single value representing one timestep in the future (or, at least, for the problem I am trying to solve). But in Pytorch, y must be X shifted one timestep to the future. It is like this:

time_series = [0, 1, 2, 3, 4, 5]

X = [0, 1, 2, 3, 4]
# Keras:
y = [5]
# Pytorch:
y = [1, 2, 3, 4, 5]

This way, Pytorch compares all values in the time slice when calculating loss. I believe Keras rearranges the data under the hood to conform to this approach, as the code works when fed the variables just like that. But in Pytorch, I was estimating loss based only on one value (the one I was trying to predict), not the whole series, therefore I believe it could not correctly capture the time dependency.

When taking this in consideration, I got to:

EPOCH 100: loss 0.00551 - MAE 0.058435

And, most importantly, comparing true and predicted values in a separate dataset got me to

The patterns were clearly captured by the model.

Hooray!


Instruction: Masking and Instance Normalization in PyTorch

Input:
Assume I have a PyTorch tensor, arranged as shape [N, C, L] where N is the batch size, C is the number of channels or features, and L is the length.  In this case, if one wishes to perform instance normalization, one does something like:

N = 20
C = 100
L = 40
m = nn.InstanceNorm1d(C, affine=True)
input = torch.randn(N, C, L)
output = m(input)

This will perform a normalization in the L-wise dimension for each N*C = 2000 slices of data, subtracting 2000 means, scaling by 2000 standard deviations, and re-scaling by 100 learnable weight and bias parameters (one per channel).  The unspoken assumption here is that all of these values exist and are meaningful.

But I have a situation where, for the slice N=1, I would like to exclude all data after (say) L=35.  For the slice N=2 (say) all the data are valid.  For the slice N=3, exclude all data after L=30, etc.  This mimics data which are one dimensional time sequences, having multiple features, but which are not the same length.

How can I perform an instance norm on such data, get correct statistics, and maintain differentiability/AutoGrad information in PyTorch?

Update:  While maintaining GPU performance, or at least not killing it dead.

I cannot...

...Mask with zero values, as this destroys the computer means and variances giving erroneous results
...Mask with np.nan or np.inf, as PyTorch tensors do not ignore such values, but treat them as errors.  They are sticky, and lead to garbage results.  PyTorch currently lacks the equivalent of np.nanmean and np.nanvar.
...Permute or transpose to an amenable arrangement of data; no such approach gives me what I need
...Use a pack_padded_sequence; instance normalization does not operate on that data structure, and one cannot import data into that structure as far as I know.  Also, data re-arrangement would still be necessary, see 3 above.

Am I missing an approach which would give me what I need?  Or perhaps am I missing a method of data re-arrangement which would allow 3 or 4 above to work?

This is an issue faced by recurrent neural networks all the time, hence the pack_padded_sequence functionality, but it isn't quite applicable here.

Output:
I don't think this is directly possible to implement using the existing InstanceNorm1d, the easiest way would probably be implementing it yourself from scratch. I did a quick implementation that should work. To make it a little bit more general this module requires a boolean mask (a boolean tensor of the same size as the input) that specifies which elements should be considered when passing through the instance norm.

import torch

class MaskedInstanceNorm1d(torch.nn.Module):
    def __init__(self, num_features, eps=1e-6, momentum=0.1, affine=True, track_running_stats=False):
        super().__init__()
        self.num_features = num_features
        self.eps = eps
        self.momentum = momentum
        self.affine = affine
        self.track_running_stats = track_running_stats
        self.gamma = None
        self.beta = None
        if self.affine:
            self.gamma = torch.nn.Parameter(torch.ones((1, self.num_features, 1), requires_grad=True))
            self.beta = torch.nn.Parameter(torch.zeros((1, self.num_features, 1), requires_grad=True))

        self.running_mean = None
        self.running_variance = None
        if self.affine:
            self.running_mean = torch.zeros((1, self.num_features, 1), requires_grad=True)
            self.running_variance = torch.zeros((1, self.num_features, 1), requires_grad=True)

    def forward(self, x, mask):
        mean = torch.zeros((1, self.num_features, 1), requires_grad=False)
        variance = torch.ones((1, self.num_features, 1), requires_grad=False)

        # compute masked mean and variance of batch
        for c in range(self.num_features):
            if mask[:, c, :].any():
                mean[0, c, 0] = x[:, c, :][mask[:, c, :]].mean()
                variance[0, c, 0] = (x[:, c, :][mask[:, c, :]] - mean[0, c, 0]).pow(2).mean()

        # update running mean and variance
        if self.training and self.track_running_stats:
            for c in range(self.num_features):
                if mask[:, c, :].any():
                    self.running_mean[0, c, 0] = (1-self.momentum) * self.running_mean[0, c, 0] \
                                                 + self.momentum * mean[0, c, 0]
                    self.running_variance[0, c, 0] = (1-self.momentum) * self.running_variance[0, c, 0] \
                                                     + self.momentum * variance[0, c, 0]

        # compute output
        x = (x - mean)/(self.eps + variance).sqrt()

        if self.affine:
            x = x * self.gamma + self.beta

        return x


Instruction: output:\ntorch-1.1.0-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform- Pytorch /cloud functions

Input:
I'm trying to deploy a function on Cloud functions and I'm having trouble getting pytorch to work. I need either version 1.1, 1.2 or 1.3 (any version that has torch.hub functionality)

Here is what I have been trying in the requirments.txt for my function:

numpy==1.17.2
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp27-cp27mu-linux_x86_64.whl

Which results in the error:

Build failed: {"error": {"canonicalCode": "INVALID_ARGUMENT", "errorMessage": "`pip_download_wheels` had stderr output:\ntorch-1.1.0-cp27-cp27mu-linux_x86_64.whl is not a supported wheel on this platform.\n\nerror: `pip_download_wheels` returned code: 1", "errorType": "InternalError", "errorId": XXX}}

And of course trying the various other url for 1.2 and 1.3 and the same thing happens.

What can I do to fix this/ what am I doing wrong??

Appreciate the help.

-if this affects anything, I'm using Python 3.7 for my function and have been using the linux packages.

__________________________________________________________________________

Edit: I've tried this in my requirments.txt:

numpy==1.17.0
torch==1.3.0
torchvision===0.4.1

And I now get the error:

Build failed: {"cacheStats": [{"status": "MISS", "hash": "1f6ebb5b3667b3d677184dbf04b82666XXX", "type": "docker_layer_cache", "level": "global"}, {"status": "MISS", "hash": "1f6ebb5b3667b3d677184dbf04b826660b67c784608d4e4XXXXX", "type": "docker_layer_cache", "level": "project"}]}

I've never had this error with any other library on cloud functions. If anyone has other suggestions it would be greatly appreciated.

Output:
That cp27 means it's for python2.7. It's actually not a good idea to install from url, use the package name istead(like numpy==1.17.2)

Try something like pip3 install torch torchvision, this will give the latest stable(1.3) version with cuda 10.

Look at the pytorch's home page -  https://pytorch.org/ - for reference


Instruction: What is the gradient of pytorch floor() gradient method?

Input:
I am looking to use floor() method in one of my models. I would like to understand what pytorch does with its gradient propagation since as such floor is a discontinuous method.

If there is no gradient defined, I could override the backward method to define my own gradient as necessary but I would like to understand what the default behavior is and the corresponding source code if possible.

import torch

x = torch.rand(20, requires_grad=True)
y = 20*x
z = y.floor().sum()
z.backward()

x.grad returns zeros.

z has a grad_fn=

So FloorBackward is the gradient method. But there is no reference to the source code of FloorBackward in pytorch repository.

Output:
As the floor function is piece wise constant. This means the gradient must be zero almost everywhere.

While the code doesn't say anything about it, I expect that the gradient is set to a constant zero everywhere.


Instruction: Trouble with nn.embedding in pytorch, expected scalar type Long, but got torch.cuda.FloatTensor (how to fix)?

Input:
so I have an RNN encoder that is part of a larger language model, where the process is encode -> rnn -> decode.
As part of my __init__ for my rnn class I have the following:
self.encode_this = nn.Embedding(self.vocab_size, self.embedded_vocab_dim)

now I am trying to implement a forward class, which takes in batches and performs encoding then decoding,
def f_calc(self, batch):
    #Here, batch.shape[0] is the size of batch while batch.shape[1] is the sequence length

    hidden_states = (torch.zeros(self.num_layers, batch.shape[0], self.hidden_vocab_dim).to(device))
    embedded_states = (torch.zeros(batch.shape[0],batch.shape[1], self.embedded_vocab_dim).to(device))

    o1, h = self.encode_this(embedded_states)

however, my problem is always with the encoder which gives me the following error:
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1465         # remove once script supports set_grad_enabled
   1466         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1467     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1468
   1469

RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)

Anyone have any idea how to fix? I am completely new to pytorch so please excuse me if this is a stupid question. I know there is some form of type casting involved but I am not sure how to go about doing it...
much appreciated!

Output:
Embedding layer expects integers at the input.

import torch as t

emb = t.nn.Embedding(embedding_dim=3, num_embeddings=26)

emb(t.LongTensor([0,1,2]))

Add long() in your code:

embedded_states = (torch.zeros(batch.shape[0],batch.shape[1], self.embedded_vocab_dim).to(device)).long()


Instruction: Run multiple models of an ensemble in parallel with PyTorch

Input:
My neural network has the following architecture:

input -> 128x (separate fully connected layers) -> output averaging

I am using a ModuleList to hold the list of fully connected layers. Here's how it looks at this point:

class MultiHead(nn.Module):
    def __init__(self, dim_state, dim_action, hidden_size=32, nb_heads=1):
        super(MultiHead, self).__init__()

        self.networks = nn.ModuleList()
        for _ in range(nb_heads):
            network = nn.Sequential(
                nn.Linear(dim_state, hidden_size),
                nn.Tanh(),
                nn.Linear(hidden_size, dim_action)
            )
            self.networks.append(network)

        self.cuda()
        self.optimizer = optim.Adam(self.parameters())

Then, when I need to calculate the output, I use a for ... in construct to perform the forward and backward pass through all the layers:

q_values = torch.cat([net(observations) for net in self.networks])

# skipped code which ultimately computes the loss I need

self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()

This works! But I am wondering if I couldn't do this more efficiently. I feel like by doing a for...in, I am actually going through each separate FC layer one by one, while I'd expect this operation could be done in parallel.

Output:
In the case of Convnd in place of Linear you could use the groups argument for "grouped convolutions" (a.k.a. "depthwise convolutions"). This let's you handle all parallel networks simultaneously.

If you use a convolution kernel of size 1, then the convolution does nothing else than applying a Linear layer, where each channel is considered an input dimension. So the rough structure of your network would look like this:

Modify the input tensor of shape B x dim_state as follows: add an additional dimension and replicate by nb_state-times  B x dim_state to B x (dim_state * nb_heads) x 1
replace the two Linear with

nn.Conv1d(in_channels=dim_state * nb_heads, out_channels=hidden_size * nb_heads, kernel_size=1, groups=nb_heads)

and

nn.Conv1d(in_channels=hidden_size * nb_heads, out_channels=dim_action * nb_heads, kernel_size=1, groups=nb_heads)

we now have a tensor of size B x (dim_action x nb_heads) x 1 you can now modify it to whatever shape you want (e.g. B x nb_heads x dim_action)

While CUDA natively supports grouped convolutions, there were some issues in pytorch with the speed of grouped convolutions (see e.g. here) but I think that was solved now.


Instruction: What is the alternative of CUDA GPU for model training with CPU support?

Input:
I dont have CUDA enabled GPU but I have i7 processor and 16GB Ram 1 GB amd graphics card

i want to disable that option and need to train a model with CPU support itself

mycodes are

parser = argparse.ArgumentParser()
parser.add_argument("--gpu", dest='gpu', type=str, default='0',                       help='Set CUDA_VISIBLE_DEVICES environment variable, optional')
os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu
params = vars(args)

how can i change into cpu version

Output:
So, the above is just the argparser, which tells Python which values to accept at the command line. It just sets variable values within the code. Even if we change this, it wouldn't change how the code runs.

It depends on how your code is written (that actually calls the ML) but running on CPU is the default. Your code specifically has to tell it to run on the GPU.

With the line os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu you're setting the environment variable CUDA_VISIBLE_DEVICES to the command-line-passed-in argument gpu ... which your code that calls the GPU will use.

But you need to change the code regarding how the ML processes are called.

Maybe you can post more code?


Instruction: how to specify python==3.6.8 for PyTorch Estimator (conda_packages not sufficient)

Input:
I need to run my python script under Azure Machine Learning, using python=3.6.8 (not the default 3.6.2).  I am using the AML "PyTorch()" Estimator, setting the "conda_packages" arg to ["python==3.6.8"].

I am relying on this doc page for the PyTorch Estimator:

  https://learn.microsoft.com/en-us/python/api/azureml-train-core/azureml.train.dnn.pytorch?view=azure-ml-py

When my script runs, I print out "sys.version" and see that it is still set to python 3.6.2:

python: 3.6.2 | packaged by conda-forge | (default, Jul 23 2017, 22:59:30)
[GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]

I expected to see python 3.6.8, since I specified that in the PyTorch Estimator's conda_packages arg.

I also tried moving the "python==3.6.8" from conda_packages to pip_packages, but received an error saying pip could not locate that package.

FYI, I have another package specified in pip_packages, and that does get installed correctly during this process.  It seems like the value of the "conda_packages" arg is not being used (I can find no mention of a conda or python install error in the AML logs for my job).

Output:
one other option is specifying a conda dependency file conda_dependencies_file_path with the right python version. the below docs outlines detailed documentation on how to do that. once you specify conda_depencies_file_path, it overrides pip_packages, and conda_packages so I recommend putting all your packages in the conda dependency file

https://learn.microsoft.com/en-us/python/api/azureml-core/azureml.core.conda_dependencies.condadependencies?view=azure-ml-py


Instruction: Feed multi resolution images to Neural Network Pytorch

Input:
I am new in this area so pardon if my question seems stupid.
I have created a multiresolution image pyramid using

  skimage.transform.pyramid_gaussian

The images are 2D. Now I want to feed these images to a neural network. The structure of the neural network is not fixed. But I can't do that since the images are not of the same size. Can anyone guide me to any resource regarding if this can be done?

Thank you

Output:
There are two types of Neural Networks: First one that can process variable input size and second that requires fixed input size.

Good example for first kind is Fully Convolutional Network (FCN). They are widely used for object detection and semantic segmentation. Next code snippet is minimal example of testing pre-trained keypointrcnn from PyTorch. This is improvement of previous state of the art Mask R-CNN

import torch
import torchvision
from PIL import Image

model_rcnn = torchvision.models.detection.keypointrcnn_resnet50_fpn(pretrained=True)
model_rcnn.eval()

image1 = Image.open('image122 × 430.jpg')
image2 = Image.open('image448 × 465.jpg')

image_tensor1 = torchvision.transforms.functional.to_tensor(image1)
image_tensor2 = torchvision.transforms.functional.to_tensor(image2)

output1 = model_rcnn([image_tensor1])
output2 = model_rcnn([image_tensor2])
print(output1, output2)

Second kind of Neural Networks require fixed size input, for example ResNet. Standard solution is using Resize transform before feeding images to the network. Minimal example:

import torch
import torchvision
from torchvision import transforms
from PIL import Image

model_imagnet = torchvision.models.resnet50(pretrained=True)
model_imagnet.eval()

# don't forget to use the same normalization as in training,
# if you are using pre-trained model
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
my_transforms = transforms.Compose([transforms.Resize(224),
                                   transforms.ToTensor(),
                                   normalize])

image1 = Image.open('image122 × 430.jpg')
image2 = Image.open('image448 × 465.jpg')

image_tensor1 = my_transforms(image1)
image_tensor2 = my_transforms(image2)

output1 = model_imagnet(torch.unsqueeze(image_tensor1, 0))
output2 = model_imagnet(torch.unsqueeze(image_tensor2, 0))

For more details about the models and there usage you may refer to PyTorch documentation


Instruction: How to extract features from a pytorch pretrained fine-tuned model

Input:
I need to extract features from a pretrained (fine-tuned) BERT model.

I fine-tuned a pretrained BERT model in Pytorch using huggingface transformer. All the training/validation is done on a GPU in cloud.

At the end of the training, I save the model and tokenizer like below:

best_model.save_pretrained('./saved_model/')
tokenizer.save_pretrained('./saved_model/')

This creates below files in the saved_model directory:

config.json
added_token.json
special_tokens_map.json
tokenizer_config.json
vocab.txt
pytorch_model.bin

I save the saved_model directory in my computer and load the model and tokenizer like below

model = torch.load('./saved_model/pytorch_model.bin',map_location=torch.device('cpu'))
tokenizer = BertTokenizer.from_pretrained('./saved_model/')

Now to extract features, I do below

input_ids = torch.tensor([tokenizer.encode("Here is some text to encode", add_special_tokens=True)])
last_hidden_states = model(input_ids)[0][0]

But for the last line, it throws me error TypeError: 'collections.OrderedDict' object is not callable

It seems like I am not loading the model properly. Instead of loading the entire model in itself, I think my model=torch.load(....) line is loading a ordered dictionary.

What am I missing here? Am I even saving the model in the right way? Please suggest.

Output:
torch.load() returns a collections.OrderedDict object. Checkout the recommended way of saving and loading a model's state dict.

Save:

torch.save(model.state_dict(), PATH)

Load:

model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()

So, in your case, it should be:

model = BertModel(config)
model.load_state_dict('./saved_model/pytorch_model.bin',
                       map_location=torch.device('cpu'))
model.eval() # to disable dropouts


Instruction: Compute gradient between a scalar and vector in PyTorch

Input:
I am trying to replicate code which was written using Theano, to PyTorch. In the code, the author computes the gradient using

import theano.tensor as T
gparams = T.grad(cost, params)

and the shape of gparams is (256, 240)

I have tried using backward() but it doesn't seem to return anything. Is there an equivalent to grad within PyTorch?

Assume this is my input,

import torch
from torch.autograd import Variable
cost = torch.tensor(1.6019)
params = Variable(torch.rand(1, 73, 240))

Output:
cost needs to be a result of an operation involving params. You can't compute a gradient just knowing the values of two tensors. You need to know the relationship as well. This is why pytorch builds a computation graph when you perform tensor operations. For example, say the relationship is

cost = torch.sum(params)

then we would expect the gradient of cost with respect to params to be a vector of ones regardless of the value of params.

That could be computed as follows. Notice that you need to add the requires_grad flag to indicate to pytorch that you want backward to update the gradient when called.

# Initialize independent variable. Make sure to set requires_grad=true.
params = torch.tensor((1, 73, 240), requires_grad=True)

# Compute cost, this implicitly builds a computation graph which records
# how cost was computed with respect to params.
cost = torch.sum(params)

# Zero the gradient of params in case it already has something in it.
# This step is optional in this example but good to do in practice to
# ensure you're not adding gradients to existing gradients.
if params.grad is not None:
    params.grad.zero_()

# Perform back propagation. This is where the gradient is actually
# computed. It also resets the computation graph.
cost.backward()

# The gradient of params w.r.t to cost is now stored in params.grad.
print(params.grad)

Result:

tensor([1., 1., 1.])


Instruction: RuntimeError: Error(s) in loading state_dict for BertModel

Input:
I finetune a BERT model using hugging face transformer library and train it in GPU in the cloud. Then I save the model and tokenizer like below:

model.save_pretrained('/saved_model/')
torch.save(best_model.state_dict(), '/saved_model/model')
tokenizer.save_pretrained('/saved_model/')

I download the saved_model directory in my computer. Then I load the model/tokenizer like below in my computer

import torch
from transformers import *
tokenizer = BertTokenizer.from_pretrained('./saved_model/')
config = BertConfig('./saved_model/config.json')
model = BertModel(config)
model.load_state_dict(torch.load('./saved_model/pytorch_model.bin', map_location=torch.device('cpu')))
model.eval()

But it throws below error for the model.load_state_dict line

RuntimeError: Error(s) in loading state_dict for BertModel:
    Missing key(s) in state_dict:

It lists a bunch of keys that are apparently missing from the state_dict.

I am new to pytorch and not sure what is going on. Most likely I am not saving the model the right way.

Please suggest.

Output:
As you may know, the state_dict of a PyTorch module is an OrderedDict. When you tried to load the weights of a module from a state_dict, it complains about missing keys which means the state_dict does not contain those keys. In this situation, I would suggest taking the following actions.

Check which keys are present in the state_dict. It sounds impossible that you save a subset of the keys only.
Also, make sure you have the correct configuration loaded. Otherwise, if your trained BertModel and the new BertModel for which you want to load the weights are different, then you will receive this error.
Finally, if your code passes both the above cases, then saving the model, make sure you save all the layers' parameters in the file. The statement, torch.save(best_model.state_dict(), '/saved_model/model') looks okay to me but make sure the best_model.state_dict() contains all the expected keys.


Instruction: How to view the weight_decay loss in PyTorch during the training?

Input:
I train a model with Adam optimizer in PyTorch and set the weight_decay parameter to 1.0.
optimizer = optim.Adam(model.parameters(), lr=args.lr, weight_decay=1.0)
optimizer.zero_grad()
loss.backward()
optimizer.step()

If I want to compare the number of the weight_decay loss and the model loss, how do I view the value of the loss caused by the weight_decay?

Output:
Are you familiar with L2 regularization? If not, you can study it. I find this tutorial very helpful.

There is a subtle difference between L2 regularization and weight decay and that is:

  Weight decay is usually defined as a term that’s added directly to the update rule. On the other hand, the L2 regularization term is added to the loss function.

You may find this tutorial helpful to study the differences between weight decay and L2 regularization.

[Update] I find the lecture by Prof. Andrew Ng very helpful.


Instruction: pytorch model returns NANs after first round

Input:
This is my first time writing a Pytorch-based CNN.  I've finally gotten the code to run to the point of producing output for the first data batch, but on the second batch produces nans.  I greatly simplified the model for debugging purposes, but it's still not working right.  The model shown here is just a few fully connected layers with a linear output.
I am guessing that the problem is the the back-propagation step, but it's unclear to me where and why.
Here is a very simplified version of the model that still produces the error:
Data loader:
batch_size = 36
device = 'cuda'
# note "rollaxis" to move channel from last to first dimension
# X_train is n input images x 70 width x 70 height x 3 channels
# Y_train is n doubles
torch_train = utils.TensorDataset(torch.from_numpy(np.rollaxis(X_train, 3, 1)).float(), torch.from_numpy(Y_train).float())
train_loader = utils.DataLoader(torch_train, batch_size=batch_size, shuffle=True)

Define and create the model:
def MyCNN(**kwargs):
    return MyCNN_model_simple(**kwargs)

# switched from Sequential() style to assist debugging
class MyCNN_model_simple(nn.Module):
    def __init__(self, **kwargs):
        super(MyCNN_model_simple, self).__init__()
        self.fc1 = FullyConnected( 3 * 70 * 70, 100)
        self.fc2 = FullyConnected( 100, 100)
        self.last = nn.Linear(100, 1)
#         self.net = nn.Sequential(
#             self.fc1,
#             self.fc2,
#             self.last,
#             nn.Flatten()
#         )
    def forward(self, x):
        print(f"x shape A: {x.shape}")
        x = torch.flatten(x, 1)
        print(f"x shape B: {x.shape}")
        x = self.fc1(x)
        print(f"x shape C: {x.shape}")
        x = self.fc2(x)
        print(f"x shape D: {x.shape}")
        x = self.last(x)
        print(f"x shape E: {x.shape}")
        x = torch.flatten(x)
        print(f"x shape F: {x.shape}")
        return x
#        return self.net(x)

class FullyConnected(nn.Module):
    def __init__(self, in_channels, out_channels, dropout=None):
        super(FullyConnected, self).__init__()
        layers = []
        layers.append(nn.Linear(in_channels, out_channels, bias=True))
        layers.append(nn.ReLU())
        if dropout != None:
            layers.append(nn.Dropout(p=dropout))
        self.net = nn.Sequential(*layers)

    def forward(self, x):
        return self.net(x)

model = MyCNN()
# convert to 16-bit half-precision to save memory
model.half()
model.to(torch.device('cuda'))

Run the model:
loss_fn = nn.MSELoss()
dev = torch.device('cuda')
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
losses = []
max_batches = 2

def process_batch():
    inputs = images.half().to(dev)
    values = scores.half().to(dev)

    # clear accumulated gradients
    optimizer.zero_grad()
    # make predictions
    outputs = model(inputs)
    # calculate and save the loss
    model_out = torch.flatten(outputs)
    print(f"Outputs: {model_out}")
    loss = loss_fn(model_out.half(), torch.flatten(values))
    losses.append( loss.item() )
    # backpropogate the loss
    loss.backward()
    # adjust parameters to computed gradients
    optimizer.step()

model.train()
i = 0
for images, scores in train_loader:
    process_batch()
    i += 1
    if i > max_batches: break

Stdout:
x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([0.0406, 0.0367, 0.0446, 0.0529, 0.0406, 0.0391, 0.0397, 0.0391, 0.0415,
        0.0443, 0.0410, 0.0406, 0.0349, 0.0396, 0.0368, 0.0401, 0.0343, 0.0419,
        0.0428, 0.0385, 0.0345, 0.0431, 0.0287, 0.0328, 0.0309, 0.0416, 0.0473,
        0.0352, 0.0422, 0.0375, 0.0428, 0.0345, 0.0368, 0.0319, 0.0365, 0.0382],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

x shape A: torch.Size([36, 3, 70, 70])
x shape B: torch.Size([36, 9800])
x shape C: torch.Size([36, 100])
x shape D: torch.Size([36, 100])
x shape E: torch.Size([36, 1])
x shape F: torch.Size([36])
Outputs: tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       device='cuda:0', dtype=torch.float16, grad_fn=<AsStridedBackward>)

You can see the nans that are coming out of the model starting with the second batch.  Is there anything obviously wrong that I'm doing?  If anyone has tips on best practices for debugging pytorch module runs that I can use to track down the problem, that would be very helpful.
Thanks.

Output:
You should switch to full precision when updating the gradients and to half precision upon training

loss.backward()
model.float() # add this here
optimizer.step()

Switch back to half precission

for images, scores in train_loader:
    model.half()  # add this here
    process_batch()


Instruction: PyTorch - a functional equivalent of nn.Module

Input: As we know we can wrap arbitrary number of stateful building blocks into a class which inherits from nn.Module. But how is it supposed to be done when you want to wrap a bunch of stateless functions (from nn.Functional), in order to fully utilize things which nn.Module allows you to, like automatic moving of tensors between CPU and GPU with just model.to(device)?

Output: I already found the solution: if you have an operation inside of a module which creates a new tensor, then you have to use self.register_buffer in order to fully utilize automating moving between devices.


Instruction: Creating custom dataset in PyTorch

Input:
Problem

In PyTorch, I am trying to write a class that could return the entire data and label separately using syntax like dataset.data and dataset.label. The code skeleton looks like:

class MyDataset(object):
  data = _get_data()
  label = _get_label()
  def __init__(self, dir, transforms):
    self.img_list = ... # all image paths loaded from dir
    # do something

  def __getitem__(self):
    # do something
    return data, label

  def __len__(self):
    return len(self.img_list)

  def _get_data():
    # do something

  def _get_label():
    # do something

However, when I use dataset.data and dataset.label to access the corresponding variables, nothing is returned.

I am wondering why this is the case and how I can fix this.

Edit

Thank you for all of your attention.

I have solved this problem by myself. The solution is pretty straightforward, which just utilizes the property of class variables.

class FaceDataset(object):
    # class variable
    data = None
    label = None

    def __init__(self, root, transforms=None):
        # read img_list from root
        img_list = ...
        self.transforms = ...
        FaceDataset.data = FaceDataset._get_data(self.img_list, self.transforms)
        FaceDataset.label = FaceDataset._get_label(self.img_list)

    @classmethod
    def _get_data(cls, img_list, transforms):
        data_list = []
        for img_path in img_list:
            data_list.append(transforms(Image.open(img_path)).unsqueeze(0))
        return torch.stack(data_list, dim=0)

    @classmethod
    def _get_label(cls, img_list):
        label = torch.zeros(len(img_list))
        for i, img_path in enumerate(img_list):
            label[i] = ...
        return label

    def __getitem__(self, index):
        img_path = self.img_list[index]
        label = ...

        # read image from file
        data = Image.open(img_path)
        # apply transform defined in __init__
        data = self.transforms(data)

        return data, label

    def __len__(self):
        return len(self.img_list)

Output:
The "normal" way to create custom datasets in Python has already been answered here on SO. There happens to be an official PyTorch tutorial for this.

For a simple example, you can read the PyTorch MNIST dataset code here (this dataset is used in this PyTorch example code for further illustration). Finally, you can find other dataset implementations in this torchvision datasets list (click on the dataset name, then on the "source" button in the dataset documentation, to access the dataset's PyTorch implementation).


Instruction: In PyTorch, what is the difference between forward() and an ordinary method?

Input:
How is implementing the forward() method of a custom nn.Module class different from adding an ordinary method to that class?

I heard that the forward() method should only accept and return tensors, because PyTorch has implemented special processing on the input and output of the forward() method. But I tried inputting/outputting non-tensor objects on a forward() method, and implementing a module that doesn't have a forward() method (instead, there are multiple custom-named methods which act like forward() methods). Both ways worked well.

Output:
forward() method does accept any type of parameters. However, the goal of the forward() method is to encapsulate the forward computational steps. forward() is called in the __call__ function. In the forward() method, PyTorch call the nested model itself to perform the forward pass.

It is encouraged to:

  NOT call the forward(x) method. You should call the whole model itself, as in model(x) to perform a forward pass and output predictions.

What happens if you do not do that?

If you call the .forward() method, and have hooks in your model, the hooks won’t have any effect.


Instruction: Can both the GPU and CPU versions of PyTorch be installed in the same Conda environment?

Input:
The PyTorch installation web page shows how to install the GPU and CPU versions of PyTorch:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

and

conda install pytorch torchvision cpuonly -c pytorch

Can both version be installed in the same Conda environment?

In case you might ask why would this be needed, it's because I would like a single Conda environment which I can use on computers which have a GPU and those which don't.

Output: The GPU version of PyTorch is actually a superset of the CPU PyTorch. You can use the GPU PyTorch on a CPU, but you cannot use the CPU PyTorch on a GPU. So in your case, installing just the GPU version of PyTorch would be sufficient.