Instruction: AssertionError: Torch not compiled with CUDA enabled

Input:
From https://pytorch.org/

to install pytorch on MacOS the following is stated :

conda install pytorch torchvision -c pytorch
# MacOS Binaries dont support CUDA, install from source if CUDA is needed

Why would want to install pytorch without cuda enabled ?

Reason I ask is I receive error :

  --------------------------------------------------------------------------- AssertionError                            Traceback (most recent call
  last)  in ()
       78 #     predicted = outputs.data.max(1)[1]
       79
  ---> 80     output = model(torch.tensor([[1,1]]).float().cuda())
       81     predicted = output.data.max(1)[1]
       82

  ~/anaconda3/lib/python3.6/site-packages/torch/cuda/init.py in
  _lazy_init()
      159         raise RuntimeError(
      160             "Cannot re-initialize CUDA in forked subprocess. " + msg)
  --> 161     _check_driver()
      162     torch._C._cuda_init()
      163     _cudart = _load_cudart()

  ~/anaconda3/lib/python3.6/site-packages/torch/cuda/init.py in
  _check_driver()
       73 def _check_driver():
       74     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
  ---> 75         raise AssertionError("Torch not compiled with CUDA enabled")
       76     if not torch._C._cuda_isDriverSufficient():
       77         if torch._C._cuda_getDriverVersion() == 0:

  AssertionError: Torch not compiled with CUDA enabled

when attempting to execute code :

x = torch.tensor([[0,0] , [0,1] , [1,0]]).float()
print(x)

y = torch.tensor([0,1,1]).long()
print(y)

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=2, shuffle=True)

# Device configuration
device = 'cpu'
print(device)

# Hyper-parameters
input_size = 2
hidden_size = 100
num_classes = 2

learning_rate = 0.001

train_dataset = my_train

train_loader = my_train_loader

pred = []

for i in range(0 , model_iters) :
    # Fully connected neural network with one hidden layer
    class NeuralNet(nn.Module):
        def __init__(self, input_size, hidden_size, num_classes):
            super(NeuralNet, self).__init__()
            self.fc1 = nn.Linear(input_size, hidden_size)
            self.relu = nn.ReLU()
            self.fc2 = nn.Linear(hidden_size, num_classes)

        def forward(self, x):
            out = self.fc1(x)
            out = self.relu(out)
            out = self.fc2(out)
            return out

    model = NeuralNet(input_size, hidden_size, num_classes).to(device)

    # Loss and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    # Train the model
    total_step = len(train_loader)
    for epoch in range(num_epochs):
        for i, (images, labels) in enumerate(train_loader):
            # Move tensors to the configured device
            images = images.reshape(-1, 2).to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
{:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

    output = model(torch.tensor([[1,1]]).float().cuda())

To fix this error I need to install pytorch from source with cuda already installed ?

Output:
To summarize and expand on the comments:

CUDA is an Nvidia proprietary (apparently unlicensed) technology to allow general computing on GPU processors.
Very few Macbook Pro's have an Nvidia CUDA-capable GPU. Take a look here to see whether your MBP has an Nvidia GPU.  Then, look at the table here to see if that GPU supports CUDA
Same story for iMac, iMac Pro and Mac Pro.
Therefore, PyTorch is installed without CUDA support by default on MacOS

This PyTorch github issue mentions that very few Macs have Nvidia processors: https://github.com/pytorch/pytorch/issues/30664

IF your Mac does have a CUDA-capable GPU, then to use CUDA commands on MacOS you'll need to recompile pytorch from source with correct command line options.


Instruction: Pylint with pytorch: Is there a way to tell pylint to look for module in different place?

Input:
I am using pytorch and pylint does not recognize few functions for ex: torch.stack however, if I do import torch._C as torch it seems to work fine.

If I do above, actual modules that exist inside torch package like torch.cuda or torch.nn need to imported individually as simply doing torch.cuda would point to torch._C.cuda and hence won't work.

Is there a way to tell pylint to look at both torch and torch._C when I do import torch or even whenever it sees torch? I don't think I would use torch to reference any other thing in my code.

Output:
A solution for now is to add torch to generated-members:

pylint --generated-members="torch.*" ...

or in pylintrc under the [TYPECHECK] section:

generated-members=torch.*

I found this solution in a reply to the github discussion of the pytorch issue [Minor Bug] Pylint E1101 Module 'torch' has no 'from_numpy' member #701. Less satisfying than whitelisting, because I guess it won't catch if you reference something that actually isn't a member, but it's the best solution I've come across so far.


Instruction: Tensorflow finfo (numeric limits)

Input:
How do I access something like numpy.finfo (Or torch.finfo) in the tensorflow Python API? I want to look up things like the smallest increment or largest finite value of the given type (for example tf.float32). Some attributes are accessible directly;

tf.float32.max
>> -3.4028235e+38
tf.float32.min
>> -3.4028235e+38

But how about epsilon, infinity or similar?

I would expect a straightforward interface to c++ std::numeric_limits, but I can't seem to find it.

Output:
You can always check this via PyTorch. Should be the same as in TF.

import torch
print(torch.finfo(torch.float16).eps) #0.0009765625
print(torch.finfo(torch.float32).eps) #1.1920928955078125e-07
print(torch.finfo(torch.float64).eps) #2.220446049250313e-16
print(torch.finfo(torch.double).eps)  #2.220446049250313e-16

Else this will be keras epsilon way:

import tensorflow as tf
tf.keras.backend.epsilon() #1e-07

There is also set_epsilon() you can play with.


Instruction: Speed up datasets loading on Google Colab

Input:
I am working on Image classification on the German Traffic Sign Dataset on Google Colab with Pytorch.
Here is the structure of the dataset:

GTSRB

Training

00000/

*.ppmm

…
00043/

*.ppmm

Test

*.ppmm
…
labels.csv

I have managed to upload the whole dataset to my drive(it took a long time!!!).
I have used ImageFolder class and Dataset class to load respectively training and test images.

However, Training my model is really slow, and GPU is not used efficiently. After many searches, I discovered that file transfer from drive to Colab is at fault here.

Does anyone know how I can use hd5 dataset (or others techniques) to first store all training and test images for later preprocessing?

Output:
If your problem truly is the network speed between Colab and Drive, you should try uploading the files directly to the Google Colab instance, rather than accessing them from Drive.

from google.colab import files
dataset_file_dict = files.upload()

Doing this will save the files directly to your Colab instance, allowing your code to access the files locally.

However, I'd suspect that there might be other problems besides the network latency – perhaps your model has lots of parameters, or somehow there was a bug in the code to get CUDA going. Sometimes I would forget to change my runtime to a GPU runtime under the "Runtime" menu tab, "Change Runtime Type".

Hope this helps!


Instruction: Can't send pytorch tensor to cuda

Input:
I create a torch tensor and I want it to go to GPU but it doesn't. This is so broken. What's wrong?
def test_model_works_on_gpu():
    with torch.cuda.device(0) as cuda:
        some_random_d_model = 2 ** 9
        five_sentences_of_twenty_words = torch.from_numpy(np.random.random((5, 20, T * d))).float()
        five_sentences_of_twenty_words_mask = torch.from_numpy(np.ones((5, 1, 20))).float()
        pytorch_model = make_sentence_model(d_model=some_random_d_model, T_sgnn=T, d_sgnn=d)

        five_sentences_of_twenty_words.to(cuda)
        five_sentences_of_twenty_words_mask.to(cuda)
        print(type(five_sentences_of_twenty_words), type(five_sentences_of_twenty_words_mask))
        print(five_sentences_of_twenty_words.is_cuda, five_sentences_of_twenty_words_mask.is_cuda)
        pytorch_model.to(cuda)
        output_before_match = pytorch_model(five_sentences_of_twenty_words, five_sentences_of_twenty_words_mask)

        assert output_before_match.shape == (5, some_random_d_model)
        print(type(output_before_match))
        print(output_before_match.is_cuda, output_before_match.get_device())

tests/test_model.py:58: RuntimeError

<class 'torch.Tensor'> <class 'torch.Tensor'>
False False
<class 'torch.Tensor'>

>       print(output_before_match.is_cuda, output_before_match.get_device())
E       RuntimeError: get_device is not implemented for tensors with CPU backend

Also:
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
2

And:
pip freeze | grep -i torch
torch==1.0.0
torchvision==0.2.1

Output:
Your issue is the following lines:
five_sentences_of_twenty_words.to(cuda)
five_sentences_of_twenty_words_mask.to(cuda)

.to(device) only operates in place when applied to a model.
When applied to a tensor, it must be assigned:
five_sentences_of_twenty_words = five_sentences_of_twenty_words.to(cuda)
five_sentences_of_twenty_words_mask = five_sentences_of_twenty_words_mask.to(cuda)


Instruction: How do i add an layer to an Nural network in pytorch

Input:
i want to add a layer to a Nural network programicaly it returned this error TypeError: forward() missing 1 required positional argument: 'x'

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(1, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

netz =Net()
print(netz)

netz = nn.Sequential([nn.Linear(10, 120), netz()])

print(netz)
`

The same error happend when i was loading it with netz=torch.load()

the line which seems to cause the error is :netz = nn.Sequential([nn.Linear(10, 120), netz()])

how do i make it work?

Output:
Ok so there are several things.

Beginning with why are you calling netz(), you already instiantiated the object earlier with netz =Net(), so this make no sense.

Second thing, nn.Sequential expects *args as "constructor" argument, so you directly pass subclasses of modules: netz = nn.Sequential(Net(), nn.Linear(100,100)) or you unpack them: nn.Sequential(*[nn.Linear(100,100), Net()]).

You can also add multiple modules using an OrderedDict as is well documented in the PyTorch docs (which you should consult by the way - it's there for a reason!)

model = nn.Sequential(OrderedDict([
          ('conv1', nn.Conv2d(1,20,5)),
          ('relu1', nn.ReLU()),
          ('conv2', nn.Conv2d(20,64,5)),
          ('relu2', nn.ReLU())
        ]))

You can also add a module with my_modules.add_module("my_module_name", Net()) to an existing collection of ordered modules.


Instruction: Pytorch NN and communication between classes

Input:
I am new at python, and pytorch, and I have a problem understanding how it works.

    import torch.nn as nn
    import torch.nn.functional as F
    import torch.optim as optim

    class Net(nn.Module):
        def __init__(self):
            ..
        def forward(self, x):
            ..
            return x
    net = Net()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

So, this is the code, and I draw what I understand from the code in the picture.
I have some questions:

A) Why could not I use nn.CrossEntropy instead of 'criterion' directly in the code? What difference does it make if I assign it to a variable? I get this error: bool value of Tensor with more than one value is ambiguous

B) Why when the class Net gets an object (nn) (I assumed when 'as' is used, an object is created), then class Net can simply use backward afterward? It is supposed to be part of nn, not Net. Could you please make sense of it for me?

C) While optim is a different object, how do the parameters, which were optimized by optim, can affect nn? I don't understand how they pass variable and update each other?

Output:
A) By setting it as a variable in one spot it helps make it easier to change the loss function in one location as opposed to having to type nn.MSELoss in many places as the code increases in size and complexity.  Less likely to make errors basically.
As for the error one would need more information to answer that bool error.  On what line what are the inputs etc.  Too little information to help there.

B) Net(nn.Module) inherits from nn.Module which will add the backwards to all the operations you add to the class.  See the docs for more info.

C) "net" is an object. net.parameters() is an iterator that iterates over all of the parameters in the net object.  So it's passed by reference as opposed to passing the parameters by value.


Instruction: How to customize number of multiple hidden layer units in pytorch LSTM?

Input:
In pytorch LSTM, RNN or GRU models, there is a parameter called "num_layers", which controls the number of hidden layers in an LSTM. I wonder that since there are multiple layers in an LSTM, why the parameter "hidden_size" is only one number instead of a list containing the number of hidden states in multiple layers, like [10, 20, 30].

I came across when I worked on a regression project, in which I feed sequence data of (seq_len, batch, feature) to LSTM, and I want to get the scalar output of every time step.

A helpful link to understand the pytorch LSTM framework, here.
I'd really appreciate it if anyone can answer this.

Output: Seems that I've found a solution to this, which is to use LSTMCell instead. Helpful links: [1], [2]. But is there an easier way?


Instruction: How to set the environment default python in anaconda?

Input:
It makes me crazy, In anaconda I create the environment with the defualt iterpreter python3.4   Next I install pytorch 0.4.1

conda install pytorch=0.4.1 cuda80 -c pytorch

After this I found that the pytorch was installed in python3.6!

And the environment defualt interpreter is chaged from python3.4 to python3.6.

I am very confused what happend ? How shoud I fix it back? change defualt python back to python3.4? Hope some one could help me.

The commands I typed in are as follows:

conda create -n pointgen python=3.4 ipykernel
source activate pointgen
conda install pytorch=0.4.1 cuda80 -c pytorch

Thats all. What Novak said is right, there is remaining question is how could I manually change the python version from 3.6 back to 3.4, is there any config file I can deal with?

Output: As you can see here there is no version of pytorch for python3.4... The default version of pytorch is for python3.6 and that is the version you installed installed. In the process anaconda prompts you that it will have to upgrade/downgrade some package versions and there is probably the the line in which it says it will upgrade python to 3.6


Instruction: Efficient PyTorch DataLoader collate_fn function for inputs of various dimensions

Input:
I'm having trouble writing a custom collate_fn function for the PyTorch DataLoader class. I need the custom function because my inputs have different dimensions.

I'm currently trying to write the baseline implementation of the Stanford MURA paper. The dataset has a set of labeled studies. A study may contain more than one image. I created a custom Datasetclass that stacks these multiple images using torch.stack.

The stacked tensor is then provided as input to the model and the list of outputs is averaged to obtain a single output. This implementation works fine with DataLoader when batch_size=1. However, when I try to set the batch_size to 8, as is the case in the original paper, the DataLoader fails since it uses torch.stack to stack the batch and the inputs in my batch have variable dimensions (since each study can have multiple number of images).

In order to fix this, I tried to implement my custom collate_fn function.

def collate_fn(batch):
    imgs = [item['images'] for item in batch]
    targets = [item['label'] for item in batch]
    targets = torch.LongTensor(targets)
    return imgs, targets

Then in my training epoch loop, I loop through each batch like this:

for image, label in zip(*batch):
    label = label.type(torch.FloatTensor)
    # wrap them in Variable
    image = Variable(image).cuda()
    label = Variable(label).cuda()
    # forward
    output = model(image)
    output = torch.mean(output)
    loss = criterion(output, label, phase)

However, this does not give me any improved timings on the epoch and still takes as long as it did with a batch size of only 1. I've also tried setting the batch size to 32 and that does not improve the timings either.

Am I doing something wrong?
Is there a better approach to this?

Output:
Very interesting problem! If I understand you correctly (and also checking the abstract of the paper), you have 40,561 images from 14,863 studies, where each study is manually labeled by radiologists as either normal or abnormal.
I believe the reason why you had the issue you faced was, say, for example, you created a stack for,

study A - 12 images
study B - 13 images
study C - 7 images
study D - 1 image, etc.

And you try to use a batch size of 8 during training which would fail when it gets to study D.
Therefore, is there a reason why we want to average the list of outputs in a study to fit a single label? Otherwise, I would simply collect all 40,561 images, assign the same label to all images from the same study (such that list of outputs in A is compared with a list of 12 labels).
Therefore, with a single dataloader you can shuffle across studies (if desired) and use the desired batch size during training.
I see this question has been around for a while, I hope it helps someone in the future :)


Instruction: How to train a L2-SVM classifier on top of a flattened vector of representations as per DCGAN paper

Input:
In the original DCGAN paper, the GAN is partly evaluated by being used as a feature extractor to classify CIFAR-10, after having been trained on Imagenet.

From the paper:

  To evaluate the quality of the representations learned by DCGANs for supervised tasks,
  we train on Imagenet-1k and then use the discriminator’s convolutional features from all layers,
  maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then
  flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM
  classifier is trained on top of them.

I have tried to replicate this using PyTorch to train the official PyTorch DCGAN and then use scikit-learn to classify using their linear SVC, but find the wording of the paper confusing and am not sure where to go from here. I've been able to maxpool each layer and then concatenate them, but am stumped on how to proceed with the classification of CIFAR-10.

In e.g. sklearn, you use model.fit(x,y) to fit the model according to the given training data, and then model.predict([X]) to predict the class labels for the samples in X. In model.fit(x,y), x is the (2D) features (e.g. images) and y is the labels. But it feels like they’re saying in the above quote make this 28672 dimensional vector the x. But that’s a 1D vector, and they use it to classify CIFAR-10, which has 50k images, and 50000 > 28672. Am I missing something obvious?

Do I use e.g. model.fit with x being the CIFAR-10 images (using e.g. torchvision.datasets.cifar10) (although how to make 50k Tensors of RGB images a 2D array is another story) and y being their labels, and then somehow predict using the 28672 dimensional vector?

Apologies if this is super obvious; unfortunately that’s all they say about it in the paper, and no one seems to have reproduced it (at least on GitHub etc.). Any help would be greatly appreciated!

Output:
DCGAN would give you a 28672 dimenstional vector for each image. Hence the shape of the output of DCGAN woud be (50000,28672) for a complete CIFAR10 dataset.

you have to take this as input for your sklearn SVM x, which as you mentioned takes a 2D data.


Instruction: Image similarity using Tensorflow or PyTorch

Input:
I want to compare two images for similarity. Since my purpose is to match a given image against a massive collection of images, I want to run the comparisons on GPU.

I came across tf.image.ssim and tf.image.psnr functions but I am unable to find and working examples only. The solutions in PyTorch is also appreciated. Since I don't have a good understanding of CUDA and C language, I am hesitant to try kernels in PyCuda.

Will it be helpful in terms of processing if I read the entire image collection and store as Tensorflow Records for future processing?

Any guidance or solution, greatly appreciated. Thank you.

Edit:- I am matching images of same size only. I don't want to do mere histogram match. I want to do SSIM or PSNR implementation for image similarity. So, I am assuming it would be similar in color, content etc

Output:
There is no implementation of PSNR or SSIM in PyTorch. You can either implement them yourself or use a third-party package, like piqa which I have developed.
Assuming you already have torch and torchvision installed, you can get it with
pip install piqa

Then for the image comparison
import torch

from torchvision import transforms
from PIL import Image

im1 = Image.open('path/to/im1.png')
im2 = Image.open('path/to/im2.png')

transform = transforms.ToTensor()

x = transform(im1).unsqueeze(0).cuda() # .cuda() for GPU
y = transform(im2).unsqueeze(0).cuda()

from piqa import PSNR, SSIM

psnr = PSNR()
ssim = SSIM().cuda()

print('PSNR:', psnr(x, y))
print('SSIM:', ssim(x, y))


Instruction: ModuleNotFoundError: No module named 'torch.utils.serialization'

Input:
When I run a project used Pytorch I came up with this error:

Traceback (most recent call last):
File "train_drnet.py", line 10, in <module>
import utils
File "/home/muse/drnet-py/utils.py", line 18, in <module>
from data.kth import KTH
File "/home/muse/drnet-py/data/kth.py", line 7, in <module>
from torch.utils.serialization import load_lua
ModuleNotFoundError: No module named 'torch.utils.serialization'

how to solve this,please?

Output: I think it was removed from Pytorch about a year ago, you can try tourch file instead - https://github.com/bshillingford/python-torchfile


Instruction: Having issues with neural network training. Loss not decreasing

Input:
I'm largely following this project but am doing a pixel-wise classification.  I have 8 classes and 9 band imagery.  My images are gridded into 9x128x128.  My loss is not reducing and training accuracy doesn't fluctuate much.  I'm guessing I have something wrong with the model.  Any advice is much appreciated!  I get at least 91% accuracy using random forest.

My classes are extremely unbalanced so I attempted to adjust training weights based on the proportion of classes within the training data.

# get model
learning_rate = 0.0001
model = unet.UNetSmall(8)
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

# set up weights based on data proportion
weights = np.array([0.79594768, 0.07181202, 0.02347426, 0.0042031, 0.00366211, 0.00764327, 0.07003923, 0.02321833])
weights = (1 - weights)/7
print('Weights of training data based on proportion of the training labels.  Not compted here')
print(weights)
print(sum(weights))
criterion = nn.CrossEntropyLoss(weight = weight)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)

  Weights of training data based on proportion of the training labels.
  Not compted here [0.02915033 0.13259828 0.13950368 0.1422567
  0.14233398 0.14176525
   0.13285154 0.13954024]
  1.0000000000000002

I've normalized the data using the transforms.functional.normalize function.  I calculated the mean and standard deviation of the training data and added this augmentation to my data loader.

dataset_train = data_utils.SatIn(data_path, 'TrainValTest.csv', 'train', transform=transforms.Compose([aug.ToTensorTarget(), aug.NormalizeTarget(mean=popmean, std=popstd)]))

I augmented my training data in preprocessing by rotating and flipping the imagery.  1 image grid then became 8.

I checked that my training data matched my classes and everything checked out.
Since I'm using  8 classes I chose to use CrossEntropyLoss since it has Softmax built in.

Current model

class UNetSmall(nn.Module):
    """
    Main UNet architecture
    """
    def __init__(self, num_classes=1):
        super().__init__()
        # encoding
        self.conv1 = encoding_block(9, 32)
        self.maxpool1 = nn.MaxPool2d(kernel_size=2)
        self.conv2 = encoding_block(32, 64)
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)
        self.conv3 = encoding_block(64, 128)
        self.maxpool3 = nn.MaxPool2d(kernel_size=2)
        self.conv4 = encoding_block(128, 256)
        self.maxpool4 = nn.MaxPool2d(kernel_size=2)

        # center
        self.center = encoding_block(256, 512)

        # decoding
        self.decode4 = decoding_block(512, 256)
        self.decode3 = decoding_block(256, 128)
        self.decode2 = decoding_block(128, 64)
        self.decode1 = decoding_block(64, 32)

        # final
        self.final = nn.Conv2d(32, num_classes, kernel_size=1)

    def forward(self, input):

        # encoding
        conv1 = self.conv1(input)
        maxpool1 = self.maxpool1(conv1)
        conv2 = self.conv2(maxpool1)
        maxpool2 = self.maxpool2(conv2)
        conv3 = self.conv3(maxpool2)
        maxpool3 = self.maxpool3(conv3)
        conv4 = self.conv4(maxpool3)
        maxpool4 = self.maxpool4(conv4)

        # center
        center = self.center(maxpool4)

        # decoding
        decode4 = self.decode4(conv4, center)
        decode3 = self.decode3(conv3, decode4)
        decode2 = self.decode2(conv2, decode3)
        decode1 = self.decode1(conv1, decode2)
        # final
        final = nn.functional.upsample(self.final(decode1), input.size()[2:], mode='bilinear')
        return final

Training method

def train(train_loader, model, criterion, optimizer, scheduler, epoch_num):

    correct = 0
    totalcount = 0

    scheduler.step()

    # iterate over data
    for idx, data in enumerate(tqdm(train_loader, desc="training")):
        # get the inputs and wrap in Variable
        if torch.cuda.is_available():
            inputs = Variable(data['sat_img'].cuda())
            labels = Variable(data['map_img'].cuda())
        else:
            inputs = Variable(data['sat_img'])
            labels = Variable(data['map_img'])

        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels.long())
        loss.backward()
        optimizer.step()

        test = torch.max(outputs.data, 1)[1] == labels.long()
        correct += test.sum().item()
        totalcount += test.size()[0] * test.size()[1] * test.size()[2]

    print('Training Loss: {:.4f}, Accuracy: {:.2f}'.format(loss.data[0], correct/totalcount))
    return {'train_loss': loss.data[0], 'train_acc' : correct/totalcount}

Training call in epoch loop

lr_scheduler.step()
train_metrics = train(train_dataloader, model, criterion, optimizer, lr_scheduler, epoch)

Some epoch iteration output

  #### Epoch 0/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:17<00:00,  5.77it/s] Training Loss: 0.8901, Accuracy: 0.83
  Current elapsed time 2m 6s

  #### Epoch 1/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:17<00:00,  5.72it/s] Training Loss: 0.7922, Accuracy: 0.83
  Current elapsed time 2m 24s

  #### Epoch 2/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:18<00:00,  5.44it/s] Training Loss: 0.8753, Accuracy: 0.84
  Current elapsed time 2m 42s

  #### Epoch 3/19

  ---------- training: 100%|████████████████████████████████████████████████████████████████████████|
  84/84 [00:18<00:00,  5.53it/s] Training Loss: 0.7741, Accuracy: 0.84
  Current elapsed time 3m 1s

Output:
It's hard to debug your model with those informations, but maybe some of those ideas will help you in some way:

Try to overfit your network on much smaller data and for many epochs without augmenting first, say one-two batches for many epochs. If this one doesn't work, than your model is not capable to model relation between data and desired target or you have an error somewhere. Furthermore it's easier to debug it that way.
I'm not sure about the weights idea, maybe try to upsample underrepresented classes in order to make it more balanced (repeat some underrepresented examples in your dataset). Curious where is this idea from, never heard of it.
Have you tried to run the model from the repo you provided before applying your own customisations? How well it performs, were you able to replicate their findings? Why do you think this architecture would be a good fit for your, from what I understand, different case? Loss function in the link you provided is different, while the architecture is the same. I haven't read this paper, neither have I tried your model, but it seems a little strange.
Link inside GitHub repo points to a blog post, where bigger batches are advised as it stabilizes the training, what is your batch size?
Maybe start with smaller and easier model and work you way up from there?

And the most important coming last; I don't think SO is the best place for such question (especially as it is research oriented), I see you have already asked it on GitHub issues though, maybe try to contact author directly?

If I were you I would start with the last point and thorough understanding of operations and their effect on your goal, good luck.


Instruction: How can I write the below equivalent code of Keras Neural Net in Pytorch?

Input:
How can I write the below equivalent code of Keras Neural Net in Pytorch?

actor = Sequential()
        actor.add(Dense(20, input_dim=9, activation='relu', kernel_initializer='he_uniform'))
        actor.add(Dense(20, activation='relu'))
        actor.add(Dense(27, activation='softmax', kernel_initializer='he_uniform'))
        actor.summary()
        # See note regarding crossentropy in cartpole_reinforce.py
        actor.compile(loss='categorical_crossentropy',
                      optimizer=Adam(lr=self.actor_lr))[Please find the image eq here.][1]

  [1]: https://i.stack.imgur.com/gJviP.png

Output:
Similiar questions have already been asked, but here it goes:

import torch

actor = torch.nn.Sequential(
    torch.nn.Linear(9, 20), # output shape has to be specified
    torch.nn.ReLU(),
    torch.nn.Linear(20, 20), # same goes over here
    torch.nn.ReLU(),
    torch.nn.Linear(20, 27), # and here
    torch.nn.Softmax(),
)

print(actor)

Initialization: By default, from version 1.0 onward, linear layers will be initialized with Kaiming Uniform (see this post). If you want to initialize your weights differently, see most upvoted answer to this question.

You may also use Python's OrderedDict to match certain layers easier, see Pytorch's documentation, you should be able to proceed from there.


Instruction: pytorch grad is None after .backward()

Input:
I just installed torch-1.0.0 on Python 3.7.2 (macOS), and trying the tutorial, but the following code:

import torch
x = torch.ones(2, 2, requires_grad=True)
y = x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(out.grad)

prints None which is not what's expected.

What's the problem?

Output:
This is the expected result.

.backward accumulate gradient only in the leaf nodes. out is not a leaf node, hence grad is None.

autograd.backward also does the same thing

autograd.grad can be used to find the gradient of any tensor w.r.t to any tensor. So if you do autograd.grad (out, out) you get (tensor(1.),) as output which is as expected.

Ref:

Tensor.backward (https://pytorch.org/docs/stable/autograd.html#torch.Tensor.backward)
autograd.backward (https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward)
autograd.grad (https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad)


Instruction: When does PyTorch automatically cast Tensor dtype?

Input:
When does PyTorch automatically cast Tensor dtype? Why does it sometimes do it automatically and other times throws and error?

For example this automatically casts c to be a float:

a = torch.tensor(5)
b = torch.tensor(5.)
c = a*b

a.dtype
>>> torch.int64

b.dtype
>>> torch.float32

c.dtype
>>> torch.float32

But this throws an error:

a = torch.ones(2, dtype=torch.float)
b = torch.ones(2, dtype=torch.long)
c = torch.matmul(a,b)

Traceback (most recent call last):

  File "<ipython-input-128-fbff7a713ff0>", line 1, in <module>
    torch.matmul(a,b)

RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'tensor'

I'm confused since Numpy seems to automatically cast all arrays as necessary e.g.

a = np.ones(2, dtype=np.long)
b = np.ones(2, dtype=np.float)

np.matmul(a,b)
>>> 2.0

a*b
>>> array([1., 1.])

Output:
It looks like the PyTorch team is working on those types of problems, see this issue. It seems like some basic upcasting is already implemented in 1.0.0 as per your example (probably for the overloaded operators, tried some others like '//' or addition and they work fine), did not find any proof of this though (like github issue or info in documentation). If someone finds it (implicit casting of torch.Tensor for various operations), please post a comment or another answer.

This issue is a proposal on type promotion, as you can see all of those are still open.


Instruction: pytorch autoencoder model evaluation fail

Input:
I am literally a beginner of PyTorch.
I trained an autoencoder network so that I can plot the distribution of the latent vectors (the result of encoders).

This is the code that I used for network training.

import torch
import torchvision
from torch import nn
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torch.utils.data import Dataset
from PIL import Image
import os
import glob

dir_img_decoded = '/media/dohyeong/HDD/mouth_autoencoder/dc_img_2'
if not os.path.exists(dir_img_decoded):
    os.mkdir(dir_img_decoded)

dir_check_point = '/media/dohyeong/HDD/mouth_autoencoder/ckpt_2'
if not os.path.exists(dir_check_point):
    os.mkdir(dir_check_point)

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

num_epochs = 200
batch_size = 150  # up -> GPU memory increase
learning_rate = 1e-3

dir_dataset = '/media/dohyeong/HDD/mouth_autoencoder/mouth_crop/dir_normalized_mouth_cropped_images'
images = glob.glob(os.path.join(dir_dataset, '*.png'))
train_images = images[:-113]
test_images = images[-113:]

train_images.sort()
test_images.sort()

class TrumpMouthDataset(Dataset):
    def __init__(self, images):
        super(TrumpMouthDataset, self).__init__()
        self.images = images

        self.transform = transforms.Compose([
            # transforms.Resize((28, 28)),
            transforms.ToTensor(),
            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])

    def __getitem__(self, index):
        image = Image.open(self.images[index])

        return self.transform(image)

    def __len__(self):
        return len(self.images)

train_dataset = TrumpMouthDataset(train_images)
test_dataset = TrumpMouthDataset(test_images)

train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(60000, 60),
            nn.ReLU(True),
            nn.Linear(60, 3),
            nn.ReLU(True),
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 60),
            nn.ReLU(True),
            nn.Linear(60, 60000),
            nn.Tanh()
        )

    def forward(self, x):
        x = x.view(x.size(0), -1)
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)

        return encoded, decoded

model = Autoencoder().cuda()
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)
    model.to(device)

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),
                             lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):

    total_loss = 0

    for index, imgs in enumerate(train_dataloader):
        imgs = imgs.to(device)

        # ===================forward=====================
        outputs = model(imgs)

        imgs_flatten = imgs.view(imgs.size(0), -1)
        loss = criterion(outputs, imgs_flatten)

        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        total_loss += loss.item()

        print('{} Epoch, [{}/{}] batch, loss: {:.4f}'.format(epoch, index + 1, len(train_dataloader), loss.item()))

    avg_loss = total_loss / len(train_dataset)
    print('{} Epoch, avg_loss: {:.4f}'.format(epoch, avg_loss))

    if epoch % 10 == 0:
        check_point_file = os.path.join(dir_check_point, str(epoch) + ".pth")
        torch.save(model.state_dict(), check_point_file)

After training, I tried to get encoded values using this code.

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

check_point = '/media/dohyeong/HDD/mouth_autoencoder/290.pth'
model = torch.load(check_point)

for index, imgs in enumerate(train_dataloader):

    imgs = imgs.to(device)

    # ===================evaluate=====================
    encoded, _ = model(imgs)

It finished with this error message.
"TypeError: 'collections.OrderedDict' object is not callable"
May I get some help?

Output:
Hi and welcome to the PyTorch community :D

TL;DR

Change model = torch.load(check_point) to model.load_state_dict(torch.load(check_point)).

The only problem is with the line:

model = torch.load(check_point)

The way you saved the checkpoint was:

torch.save(model.state_dict(), check_point_file)

That is, you saved the model's state_dict (which is just a dictionary of the various parameters that together describe the current instance of the model) in check_point_file.

Now, in order to load it back, just reverse the process.
check_point_file contains just the state_dict.

It knows nothing about the internals of the model - what it's architecture is, how it's supposed to work etc.

So, load it back:

state_dict = torch.load(check_point)

This state_dict can now be copied onto your Model instance as follows:

model.load_state_dict(state_dict)

Or, more succinctly,

model.load_state_dict(torch.load(check_point))

You got the error because the torch.load(check_point) returned the state_dict which you assigned to model.

When you subsequently called model(imgs), model was an OrderedDict object (not callable).

Hence the error.

See the Serialization Semantics Notes for more details.

Apart from that, your code sure is thorough for a beginner. Great going!

P.S. Your device agnosticity is brilliant! Perhaps you'd want to take a look at:

the line model = Autoencoder().cuda()
The map_location argument of torch.load()


Instruction: How to implement current pytorch activation functions with parameters?

Input:
I am looking for a simple way to use an activation function which exist in the pytorch library, but using some sort of parameter. for example:

Tanh(x/10)

The only way I came up with looking for solution was implementing the custom function completely from scratch. Is there any better/more elegant way to do this?

edit:

I am looking for some way to append to my model the function Tanh(x/10) rather than plain Tanh(x). Here is the relevant code block:

    self.model = nn.Sequential()
    for i in range(len(self.layers)-1):
        self.model.add_module("linear_layer_" + str(i), nn.Linear(self.layers[i], self.layers[i + 1]))
        if activations == None:
            self.model.add_module("activation_" + str(i), nn.Tanh())
        else:
            if activations[i] == "T":
                self.model.add_module("activation_" + str(i), nn.Tanh())
            elif activations[i] == "R":
                self.model.add_module("activation_" + str(i), nn.ReLU())
            else:
                #no activation
                pass

Output:
Instead of defining it as a specific function, you could inline it in a custom layer.

For instance your solution could look like:

import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(4, 10)
        self.fc2 = nn.Linear(10, 3)
        self.fc3 = nn.Softmax()

    def forward(self, x):
        return self.fc3(self.fc2(torch.tanh(self.fc1(x)/10)))

where torch.tanh(output/10) is inlined in the forward function of your module.


Instruction: How to put datasets created by torchvision.datasets in GPU in one operation?

Input:
I’m dealing with CIFAR10 and I use torchvision.datasets to create it. I’m in need of GPU to accelerate the calculation but I can’t find a way to put the whole dataset into GPU at one time. My model need to use mini-batches and it is really time-consuming to deal with each batch separately.

I've tried to put each mini-batch into GPU separately but it seems really time-consuming.

Output:
TL;DR

You won't save time by moving the entire dataset at once.

I don't think you'd necessarily want to do that even if you have the GPU memory to handle the entire dataset (of course, CIFAR10 is tiny by today's standards).

I tried various batch sizes and timed the transfer to GPU as follows:

num_workers = 1 # Set this as needed

def time_gpu_cast(batch_size=1):
    start_time = time()
    for x, y in DataLoader(dataset, batch_size, num_workers=num_workers):
        x.cuda(); y.cuda()
    return time() - start_time

# Try various batch sizes
cast_times = [(2 ** bs, time_gpu_cast(2 ** bs)) for bs in range(15)]
# Try the entire dataset like you want to do
cast_times.append((len(dataset), time_gpu_cast(len(dataset))))

plot(*zip(*cast_times)) # Plot the time taken

For num_workers = 1, this is what I got:

And if we try parallel loading (num_workers = 8), it becomes even clearer:


Instruction: How to iterate over layers in Pytorch

Input:
Let's say I have a network model object called m. Now I have no prior information about the number of layers this network has. How can create a for loop to iterate over its layer?
I am looking for something like:

Weight=[]
for layer in m._modules:
    Weight.append(layer.weight)

Output:
Let's say you have the following neural network.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # define the forward function
        return x

Now, let's print the size of the weight parameters associated with each NN layer.

model = Net()
for name, param in model.named_parameters():
    print(name, param.size())

Output:

conv1.weight torch.Size([6, 1, 5, 5])
conv1.bias torch.Size([6])
conv2.weight torch.Size([16, 6, 5, 5])
conv2.bias torch.Size([16])
fc1.weight torch.Size([120, 400])
fc1.bias torch.Size([120])
fc2.weight torch.Size([84, 120])
fc2.bias torch.Size([84])
fc3.weight torch.Size([10, 84])
fc3.bias torch.Size([10])

I hope you can extend the example to fulfill your needs.


Instruction: Torch C++: API to check NAN

Input:
I am using libtorch C++. In python version we can easily check the value of a tensor by calling its numpy value, and in numpy we have np.isnan(). I was wondering if there is a built in function in libtorch C++ to check whether a tensor has any NAN value?

Thanks,
Afshin

Output:
Adding on to Fábio's answer (my reputation is too low to comment):

If you actually want to use the information about NANs in an assert or if condition you need convert it from a torch::Tensor to a C++ bool like so

torch::Tensor myTensor;
// do something
auto tensorIsNan = at::isnan(myTensor).any().item<bool>(); // will be of type bool


Instruction: How to speed up the "ImageFolder" for ImageNet

Input:
I am in an university, and all the file system are in a remote system, wherever I log in with my account, I could aways access my home directory. even though I log into the GPU servers through SSH command. This is the condition where I employ the GPU servers to read data.

Currently, I use the PyTorch to train ResNet from scratch on ImageNet, my codes only use all the GPUs in the same computer, I found that the "torchvision.datasets.ImageFolder" will take almost two hours.

Would you please provide some experiences in how to speed up "torchvision.datasets.ImageFolder"? Thanks very much.

Output:
Why it takes so long?
Setting up an ImageFolder can take a long time, especially when the images are stored on a slow remote disk. The reason for this latency is that the __init__ function for the dataset goes over all files in the image folders and check whether this file is an image file. For ImageNet that can take quite a while as there are over 1 million files to check.

What can you do?
- As Kevin Sun already pointed out, copying the dataset to a local (and possibly much faster) storage can significantly speed up things.
- Alternatively, you can create a modified dataset class that does not read all the files, but relies on a cached list of files - a cached list that you prepare only once in advance and to be used for all runs.


Instruction: How to use multiple GPUs in pytorch?

Input:
I use this command to use a GPU.
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

But, I want to use two GPUs in jupyter, like this:
device = torch.device("cuda:0,1" if torch.cuda.is_available() else "cpu")

Output:
Assuming that you want to distribute the data across the available GPUs (If you have batch size of 16, and 2 GPUs, you might be looking providing the 8 samples to each of the GPUs), and not really spread out the parts of models across difference GPU's. This can be done as follows:
If you want to use all the available GPUs:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = CreateModel()

model= nn.DataParallel(model)
model.to(device)

If you want to use specific GPUs:
(For example, using 2 out of 4 GPUs)
device = torch.device("cuda:1,3" if torch.cuda.is_available() else "cpu") ## specify the GPU id's, GPU id's start from 0.

model = CreateModel()

model= nn.DataParallel(model,device_ids = [1, 3])
model.to(device)

To use the specific GPU's by setting OS environment variable:
Before executing the program, set CUDA_VISIBLE_DEVICES variable as follows:
export CUDA_VISIBLE_DEVICES=1,3  (Assuming you want to select 2nd and 4th GPU)
Then, within program, you can just use DataParallel() as though you want to use all the GPUs. (similar to 1st case). Here the GPUs available for the program is restricted by the OS environment variable.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = CreateModel()

model= nn.DataParallel(model)
model.to(device)

In all of these cases, the data has to be mapped to the device.
If X and y are the data:
X.to(device)
y.to(device)


Instruction: Unable to optimize function using pytorch

Input:
I am trying to write an estimator for a Structural Equation Model. So basically I start with the random parameters for the model B, gamma, phi_diag, psi. And using this I compute the implied covariance matrix sigma. And my optimization function f_ml is computed based on the sigma and the covariance matrix of the data S. Here's my computation code:

device = torch.device('cpu')
dtype = torch.float

B_s = (4, 4)
gamma_s = (4, 1)
phi_s = (1, 1)
psi_s = (4, 4)

# Covariance matrix of data
S = torch.tensor(data.cov().values, dtype=dtype, device=device, requires_grad=False)

# Defining parameters of the model
B = torch.rand(*B_s, dtype=dtype, device=device, requires_grad=True)
B_lower = B.tril(diagonal=-1)

gamma = torch.rand(*gamma_s, dtype=dtype, device=device, requires_grad=True)

phi_diag = torch.rand(phi_s[0], dtype=dtype, device=device, requires_grad=True)
phi = torch.diag(phi_diag)

psi = torch.rand(*psi_s, dtype=dtype, device=device, requires_grad=True)
psi_sym = psi @ psi.t()

B_inv = (torch.eye(*B_s, dtype=dtype, device=device, requires_grad=False) - B_lower).inverse()
sigma_yy = B_inv @ (gamma @ phi @ gamma.t() + psi_sym) @ B_inv.t()
sigma_xy = phi @ gamma.t() @ B_inv.t()
sigma_yx = sigma_xy.t()
sigma_xx = phi

# Computing the covariance matrix from the parameters
sigma = torch.cat((torch.cat((sigma_yy, sigma_yx), 1), torch.cat((sigma_xy, sigma_xx), 1)), 0)

And I am trying to do the optimization as:

optim = torch.optim.Adam([B, gamma, phi_diag, psi], lr=0.01)
for t in range(5000):
    optim.zero_grad()
    f_ml = sigma.logdet() + (S @ sigma.inverse()).trace() - S.logdet() - (4 + 1)
    f_ml.backward(retain_graph=True)
    optim.step()

The problem I am facing is that the values of my parameters aren't updated during the optimization. I tried to debug the problem a bit and what I noticed is that in the first loop of optimization the gradients get calculated but the values of the parameters don't get updated. Here's an example using pdb (breakpoint set right after the for loop):

> <ipython-input-232-c6a6fda6610b>(14)<module>()
-> optim.zero_grad()
(Pdb) B
tensor([[ 6.0198e-01,  8.7188e-01,  5.4234e-01,  6.0800e-01],
        [-4.9971e+03,  9.3324e-01,  8.1482e-01,  8.3517e-01],
        [-1.4002e+04,  2.6706e+04,  2.6412e-01,  4.7804e-01],
        [ 1.1382e+04, -2.1603e+04, -6.0834e+04,  1.2768e-01]],
       requires_grad=True)
(Pdb) c
> <ipython-input-232-c6a6fda6610b>(13)<module>()
-> import pdb; pdb.set_trace()
(Pdb) B.grad
tensor([[ 0.0000e+00,  0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 1.6332e+04,  0.0000e+00,  0.0000e+00,  0.0000e+00],
        [ 4.6349e+04, -8.8694e+04,  0.0000e+00,  0.0000e+00],
        [-3.7612e+04,  7.1684e+04,  2.0239e+05,  0.0000e+00]])
(Pdb) B
tensor([[ 6.0198e-01,  8.7188e-01,  5.4234e-01,  6.0800e-01],
        [-4.9971e+03,  9.3324e-01,  8.1482e-01,  8.3517e-01],
        [-1.4002e+04,  2.6706e+04,  2.6412e-01,  4.7804e-01],
        [ 1.1382e+04, -2.1603e+04, -6.0834e+04,  1.2768e-01]],
       requires_grad=True)

I can't figure out what am I doing wrong. Any ideas?

Output: The problem is that the value of sigma wasn't getting computed in each iteration. Basically, the computation code needs to be moved in a function and it needs to be computed in every iteration.


Instruction: Saving/Loading models in AllenNLP package

Input:
I am trying to load an AllenNLP model weights. I could not find any documentation on how to save/load a whole model, so playing with weights only.

from allennlp.nn import util
model_state = torch.load(filename_model, map_location=util.device_mapping(-1))
model.load_state_dict(model_state)

I modified my input corpus a bit and I am guessing because of this I am getting corpus-size mismatch:

RuntimeError: Error(s) in loading state_dict for BasicTextFieldEmbedder:

    size mismatch for token_embedder_tokens.weight:
    copying a param with shape torch.Size([2117, 16]) from checkpoint,
    the shape in current model is torch.Size([2129, 16]).

Seemingly there is no official way to save model with corpus vocabulary. Any hacks around it?

Output:
There's a functionality in AllenNLP allowing to load or save a model.
Have you followed the steps outlined in AllenNLP's tutorial? Below I pasted a snippet from the tutorial that might be of interest to you:

# Here's how to save the model.
with open("/tmp/model.th", 'wb') as f:
    torch.save(model.state_dict(), f)

vocab.save_to_files("/tmp/vocabulary")

# And here's how to reload the model.
vocab2 = Vocabulary.from_files("/tmp/vocabulary")

model2 = LstmTagger(word_embeddings, lstm, vocab2)
with open("/tmp/model.th", 'rb') as f:
    model2.load_state_dict(torch.load(f))

If the above somehow doesn't work for you, you can check allennlp.models.archival.archive_model helper function. Using this function you should be able to archive your model's training configuration along with weights and vocabulary to model.tar.gz.
Here you can find more information on the constraints of the two approaches that I discussed


Instruction: Order of layers in hidden states in PyTorch GRU return

Input:
This is the API I am looking at, https://pytorch.org/docs/stable/nn.html#gru

It outputs:

output of shape (seq_len, batch, num_directions * hidden_size)
h_n of shape (num_layers * num_directions, batch, hidden_size)

For GRU with more than one layers, I wonder how to fetch the hidden state of the last layer, should it be h_n[0] or h_n[-1]?

What if it's bidirectional, how to do the slicing to obtain the last hidden layer states of GRUs in both directions?

Output:
The documentation nn.GRU is clear about this. Here is an example to make it more explicit:
For the unidirectional GRU/LSTM (with more than one hidden layer):
output - would contain all the output features of all the timesteps t
h_n - would return the hidden state (at last timestep) of all layers.
To get the hidden state of the last hidden layer and last timestep, use:
first_hidden_layer_last_timestep = h_n[0]
last_hidden_layer_last_timestep = h_n[-1]

where  n is the sequence length.

This is because description says:
num_layers – Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two GRUs together to form a stacked GRU, with the second GRU taking in outputs of the first GRU and computing the final results.
So, it is natural and intuitive to also return the results (i.e. hidden states) accordingly in the same order.


Instruction: Pytorch - Can not slice torchvision MNIST dataset

Input:
In Pytorch, when using torchvision's MNIST dataset, we can get a digit as follows:
from torchvision import datasets, transforms
from torch.utils.data import DataLoader, Dataset, TensorDataset

tsfm = transforms.Compose([transforms.Resize((16, 16)),
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,))])

mnist_ds = datasets.MNIST(root='../../../_data/mnist',train=True,download=True,
                          transform=tsfm)

digit_12 = mnist_ds[12]

Though it is possible to slice on many datasets, we cannot slice on this one:
>>> digit_12_to_14 = mnist_ds[12:15]
ValueError: Too many dimensions: 3 > 2.

This is due to a Image.fromarray() in the getItem().
Is it possible to use MNIST dataset without using a Dataloader?

PS: The reason why I would like to avoid using Dataloader is that sending batches one at a time to the GPU slows down the training. I prefer to send the entire dataset to the GPU at once. For this I need to have access to the whole transformed dataset.

Output:
You can use torch.utils.data.Subset() to get an index based slice of a torch Dataset e.g:
import torch.utils.data as data_utils

indices = torch.arange(12,15)
mnist_12to14 = data_utils.Subset(tr, indices)


Instruction: Transferring pretrained pytorch model to onnx

Input:
I am trying to convert pytorch model to ONNX, in order to use it later for TensorRT. I followed the following tutorial https://pytorch.org/tutorials/advanced/super_resolution_with_caffe2.html, but my kernel dies all the time.

This is the code that I implemented.

 # Some standard imports
import io
import numpy as np

from torch import nn
import torch.onnx
from deepformer.nets.quicknat import quickNAT
param = {
    'num_channels': 64,
    'num_filters': 64,
    'kernel_h': 5,
    'kernel_w': 5,
    'kernel_c': 1,
    'stride_conv': 1,
    'pool': 2,
    'stride_pool': 2,
    'num_classes': 1,
    'padding': 'reflection'
}

net = quickNAT(param)
checkpoint_path = 'checkpoint_epoch36_loss0.78.t7'
checkpoints=torch.load(checkpoint_path)
map_location = lambda storage, loc: storage
if torch.cuda.is_available():
    map_location = None
net.load_state_dict(checkpoints['net'])
net.train(False)
# Input to the modelvcdfx
x = torch.rand(1, 64, 256, 1600, requires_grad=True)

# Export the model
torch_out = torch.onnx._export(net,             # model being run
                               x,                       # model input (or a tuple for multiple inputs)
                               "quicknat.onnx", # where to save the model (can be a file or file-like object)
                               export_params=True)      # store the trained parameter weights inside the model file

Output:
What is the output you get? It seems SuperResolution is supported with the export operators in pytorch as mentioned in the documentation
Are you sure the input to your model is:
x = torch.rand(1, 64, 256, 1600, requires_grad=True)

That could be the variable that you used for training, since for deployment you run the network on one or multiple images the dummy input to export to onnx is usually:
dummy_input = torch.randn(1, 3, 720, 1280, device='cuda')

With 1 being the batch size, 3 being the channels of the image(RGB), and then the size of the image, in this case 720x1280. Check on that input, I guess you don't have a 64 channel image as input right?
Also, it'd be helpful if you post the terminal output to see where it fails.
Good luck!


Instruction: Saving and Loading Pytorch Model Checkpoint for inference not working

Input:
I have a trained model using LSTM. The model is trained on GPU (On Google COLABORATORY).
I have to save the model for inference; which I will run on CPU.
Once trained, I saved the model checkpoint as follows:

torch.save({'model_state_dict': model.state_dict()},'lstmmodelgpu.tar')

And, for inference, I loaded the model as :

# model definition
vocab_size = len(vocab_to_int)+1
output_size = 1
embedding_dim = 300
hidden_dim = 256
n_layers = 2

model = SentimentLSTM(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)

# loading model
device = torch.device('cpu')
checkpoint = torch.load('lstmmodelgpu.tar', map_location=device)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

But, it is raising the following error:

model.load_state_dict(checkpoint['model_state_dict'])
  File "workspace/envs/envdeeplearning/lib/python3.5/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for SentimentLSTM:
    Missing key(s) in state_dict: "embedding.weight".
    Unexpected key(s) in state_dict: "encoder.weight".

Is there anything I missed while saving the checkpoint?

Output:
There are two things to be considered here.

You mentioned that you're training your model on GPU and using it for inference on CPU, so u need to add a parameter map_location in load function passing torch.device('cpu').
There is a mismatch of state_dict keys (indicated in your ouput message), which might be caused by some missing keys or having more keys in state_dict you are loading than the model u are using currently. And for it you have to add a parameter strict  with value False  in the load_state_dict function. This will make method to ignore the mismatch of keys.

Side note : Try to use extension of pt or pth for checkpoint files as it is a convention .


Instruction: ImportError: No module named 'torchvision.datasets.mnist'

Input:
Even after installing pytorch, this error is coming for this line.

from torchvision import datasets

Output:
If you're using anaconda distribution, first install torchvision using:

$ conda install -c conda-forge torchvision

If the package is not installed, then it will be installed. Else, it will throw the message

  # All requested packages already installed.

After this, try to import the torchvision.datasets as you mentioned.

In [1]: from torchvision import datasets

In [2]: dir(datasets)
Out[2]:
['CIFAR10',
 'CIFAR100',
 'CocoCaptions',
 'CocoDetection',
 'DatasetFolder',
 'EMNIST',
 'FakeData',
 'FashionMNIST',
 'ImageFolder',
 'LSUN',
 'LSUNClass',
 'MNIST',
 'Omniglot',
 'PhotoTour',
 'SEMEION',
 'STL10',
 'SVHN',
 ....,
 ....
]

As you can see from above listing of dir(datasets), the dataset class for MNIST is listed which will be the case when the torchvision package is installed correctly.


Instruction: Pytorch softmax along different masks without for loop

Input:
Say I have a vector a , with an index vector b of the same length. The indexs are in range 0~N-1, corresponding to N groups. How can I do softmax for every group without for loop?

I'm doing some sort of attention operation here. The numbers for every group are not the same, so I can't reshape a to a matrix and use the dim in standard Softmax() API.

Toy example:

a = torch.rand(10)
a: tensor([0.3376, 0.0557, 0.3016, 0.5550, 0.5814, 0.1306, 0.2697, 0.9989, 0.4917,
        0.6306])
b = torch.randint(0,3,(1,10), dtype=torch.int64)
b: tensor([[1, 2, 0, 2, 2, 0, 1, 1, 1, 1]])

I want to do softmax like

for index in range(3):
    softmax(a[b == index])

but without the for loop to save time.

Output:
Maybe this answer will have to change slightly based on a potential response to my comment, but I'm just going ahead and throwing in my two cents about Softmax.
Generally, the formula for softmax is explained rather well in the PyTorch documentation, where we can see that this is a exponential of the current value, divided by the sum of all classes.
The reason for doing this is founded in probability theory, and probably a little outside of my comfort zone, but essentially it helps you to maintain a rather simple backpropagation derivative, when using it in combination with a popular loss strategy called "Cross Entropy Loss" (CE) (see the corresponding function in PyTorch here.
Furthermore, you can also see in the description for CE that it automatically combines two functions, namely a (numerically stable) version of the softmax function, as well as the negative log likelihood loss (NLLL).
Now, to tie back to your original question, and hopefully resolving your issue:
For the sake of the question - and the way you asked it - it seems you are playing around with the popular MNIST handrwitten digit dataset, in which we want to predict some values for your current input image.
I am also assuming that your output a will at some point be the output from a layer from a neural network. It does not matter whether this is squashed to a specific range or not (e.g., by applying some form of activation function), as the softmax will be basically a normalization. Specifically, it will give us, as discussed before, some form of distribution across all the predicted values, and sum up to 1 across all classes. To do this, we can simply apply something like
soft_a = softmax(a, dim=0) # otherwise throws error if we don't specify axis
print(torch.sum(soft_a)) # should return "Tensor(1.)"

Now, if we assume that you want to do the "classical" MNIST example, you could then use the argmax() function to predict which value your system thinks is the correct answer, and calculate an error based off that, e.g., with the nn.NLLLoss() function.
If you are indeed predicting values for each position in a single output, you have to think slightly different about this.
First of all, softmax() ceases to make sense here, since you are computing a probability distribution across multiple outputs, and unless you are fairly certain that their distributions are dependent on one another in a very specific way, I would argue that this is not the case here.
Also, keep in mind that you are then looking to calculate a pairwise loss, i.e. something for every index of your output. The function that comes to mind for this specific purpose would be nn.BCELoss(), which calculates a binarized (element-wise) version of Cross-Entropy.
For this, you can then simply "prop in" your original prediction tensor a, as well as your ground truth tensor b. A minimal example for this would look like this:
bce = torch.nn.BCELoss(reduction="none") # to keep losses for each element separate
loss = bce(a,b) # returns tensor with respective pairwise loss

If you are interested in a single loss, you can obviously use BCELoss with a different argument for reduction, as described in the docs.
Let me know if I can clarify some parts of the answer for you.
EDIT: Something else to keep in mind here: The BCELoss() requires you to feed in values that can potentially be close to the value you want to predict. This is especially a problem if you feed in values to an activation function first (e.g., sigmoid or tanh), which can then never reach the value you want to predict, since they are bound by an interval!


Instruction: batch_size not match in torchtext BucketIterator

Input:
I have set batch_size equals to 64, but when i print out the train_batch and val_batch, the size is not equal to 64.

The train data and val data are in the below format:

First, i define TEXT and LABEL field.

tokenize = lambda x: x.split()

TEXT = data.Field(sequential=True, tokenize=tokenize)
LABEL = data.Field(sequential=False)

And then i keep trying follow tutorials, and wrote things below:

train_data, valid_data = data.TabularDataset.splits(
        path='.',
        train='train_intent.csv', validation='val.csv',
        format='csv',
        fields= {'sentences': ('text', TEXT),
                'labels': ('label',LABEL)}
)

test_data = data.TabularDataset(
        path='test.csv',
        format='csv',
        fields={'sentences': ('text', TEXT)}

)
TEXT.build_vocab(train_data)
LABEL.build_vocab(train_data)

BATCH_SIZE = 64

train_iter, val_iter = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_sizes=(BATCH_SIZE, BATCH_SIZE),
    sort_key=lambda x: len(x.text),
    sort_within_batch=False,
    repeat=False,
    device=device
)

But when i want to know the iter is fine or not, i just find the below strange things:

train_batch = next(iter(train_iter))
print(train_batch.text.shape)
print(train_batch.label.shape)
[output]
torch.Size([15, 64])
torch.Size([64])

And the train process output errorValueError: Expected input batch_size (15) to match target batch_size (64).:

def train(model, iterator, optimizer, criterion):

    epoch_loss = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        predictions = model(batch.text)
        loss = criterion(predictions, batch.label)
        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()

    return epoch_loss / len(iterator)

Anyone could give me a hint would be highly appreciated. Thanks!

Output:
The returned batch size not always equal to batch_size. ex: you have 100 train data, batch_size is 64. The returned batch_size should be [64, 36].

Code:  https://github.com/pytorch/text/blob/1c2ae32d67f7f7854542212b229cd95c85cf4026/torchtext/data/iterator.py#L255-L271


Instruction: Modification to Caffe VGG 16 to handle 1 channel images on PyTorch

Input:
I am converting a VGG16 network to be a Fully Convolutional network and also modifying the the input to accept a single channel image. The complete code for reproducibilty is given below.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

import numpy as np
import torchvision.datasets as datasets
import copy

from torch.utils import model_zoo
from torchvision import models
from collections import OrderedDict

def convolutionalize(modules, input_size):
    """
    Recast `modules` into fully convolutional form
    """
    fully_conv_modules = []
    x = Variable(torch.zeros((1, ) + input_size))
    for m in modules:
         if isinstance(m, nn.Linear):
              n = nn.Conv2d(x.size(1), m.weight.size(0), kernel_size=(x.size(2), x.size(3)))
              n.weight.data.view(-1).copy_(m.weight.data.view(-1))
              n.bias.data.view(-1).copy_(m.bias.data.view(-1))
              m = n
         fully_conv_modules.append(m)
         x = m(x)
    return fully_conv_modules

def vgg16(is_caffe=True):
     """
     Load the VGG-16 net for use as a fully convolutional backbone.
     """
     vgg16 = models.vgg16(pretrained=True)
     # cast into fully convolutional form (as list of layers)
     vgg16 = convolutionalize(list(vgg16.features) + list(vgg16.classifier),
                         (3, 224, 224))
     # name layers like the original paper
     names = ['conv1_1', 'relu1_1', 'conv1_2', 'relu1_2', 'pool1',
    'conv2_1', 'relu2_1', 'conv2_2', 'relu2_2', 'pool2',
    'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2', 'conv3_3', 'relu3_3', 'pool3',
    'conv4_1', 'relu4_1', 'conv4_2', 'relu4_2', 'conv4_3', 'relu4_3', 'pool4',
    'conv5_1', 'relu5_1', 'conv5_2', 'relu5_2', 'conv5_3', 'relu5_3', 'pool5',
    'fc6', 'relu6', 'drop6', 'fc7', 'relu7', 'drop7', 'fc8']

    vgg16 = nn.Sequential(OrderedDict(zip(names, vgg16)))

    if is_caffe:
        # substitute original Caffe weights for improved fine-tuning accuracy
        # see https://github.com/jcjohnson/pytorch-vgg
        caffe_params = model_zoo.load_url('https://s3-us-west-2.amazonaws.com/'
                                      'jcjohns-models/vgg16-00b39a1b.pth')
        for new_p, old_p in zip(vgg16.parameters(), caffe_params.values()):
            new_p.data.copy_(old_p.view_as(new_p))
        # surgery: decapitate final classifier
   del vgg16._modules['fc8']  # note: risky use of private interface
   # surgery: keep fuller spatial dims by including incomplete pooling regions
   for m in vgg16.modules():
       if isinstance(m, nn.MaxPool2d):
                m.ceil_mode = True
   return vgg16

class Learner(nn.Module):

     def __init__(self, num_classes, singleChannel=False):
          super().__init__()

          backbone = vgg16(is_caffe=True)
          for k in list(backbone._modules)[-6:]:
                del backbone._modules[k]

          supp_backbone = copy.deepcopy(backbone)

          # Modify conv1_1 of conditioning branch to have 1 input channels
          # Init the weights in the new channels to the channel-wise mean
          # of the pre-trained conv1_1 weights
          if singleChannel==True:
               old_conv1 = backbone._modules['conv1_1'].weight.data
               mean_conv1 = torch.mean(old_conv1, dim=1, keepdim=True)
               new_conv1 = nn.Conv2d(1, old_conv1.size(0), kernel_size=old_conv1.size(2), stride=1, padding=1)
               new_conv1.weight.data = mean_conv1
               new_conv1.bias.data = backbone._modules['conv1_1'].bias.data
               backbone._modules['conv1_1'] = new_conv1

          self.encoder = copy.deepcopy(backbone)
          self.num_classes=num_classes

     def forward(self,im):

          # encode image
          supp_feats = self.encoder(im)

          return supp_feats

 model=Learner(num_classes=2,singleChannel=True).cpu()
 mnist_trainset = datasets.MNIST(root='./data', train=True, download=True, transform=None)
 im2arr = np.array(mnist_trainset[1][0])
 im2arr = im2arr[np.newaxis,:, :,] # shape(1,28,28)

 model.train()
 x=model(torch.from_numpy(im2arr))

I expected x to be a torch tensor output but get an error message
'ValueError: Expected 4D tensor as input, got 3D tensor instead.' on the last line

Output:
You need your input shape to be Batch-Channel-Height-Width, which is 4D. In your case, you only have one channel so you "squeezed out" this singleton dimension, but pytorch does not like it!

try

im2arr = im2arr[np.newaxis, np.newaxis, :, :]  # add singleton for the channles as well


Instruction: ML Model inputs and outputs analogy

Input:
As I learn more and more about ML (I am a mobile DEV) I'm starting to form an analogy in my head. I would like the communities opinion / validation.

As a front end DEV you have a backend and an API that you can make requests to. The standard format for the inputs and outputs to the API is JSON.

I'm running into a problem with ML Models that I am trying to use where I don't know how to read the expected input (API) and I don't know how to decode the expected output.

So far I my experience has been fragmented because some models say "Give me an image of [1,2,120,120]" or something like that.

To analogize, is there a unified way to define inputs and outputs for a ML model like JSON unifies the inputs and outputs for an backend API?

If so, what are some rules one must follow to encode and decode data into this format?

Output:
Assuming this "ML Model" is in the context of running an input through say a trained pytorch model's forward pass to get an output, the unified way to define inputs and outputs for an ML model are through Tensors.  Tensors are essentially a multi-dimensional matrix containing elements of a single data type.  Think multi-dimensional lists with a single data type.

Tensors:MLModels::JSON:WebAPI

An Example using an Object Detector
Model

Let's say your model example with the image is an object detector model that takes in an image as input and outputs either dog or cat
The input would usually be:
A tensor representation of an Image with the shape of [1, 2, 120, 120] where 1 represents the batch size, 2 is the dimension of your rgb channels, and 120x120 is the width and height of an image.
The output would usually be:
A normalized 2 dimensional tensor like [0.7, 0.3] where index 0 represents the probability of the image depicting a dog and index 1 represents the probability it's a cat.

Encoding and Decoding

Decoding the output to a string like "dog" or "cat" is obvious.
Encoding an image is slightly less obvious.  At its heart, the format
of an image is that of a tensor...a multi-dimensional matrix
containing a single datatype.  So is still intuitive to encode an
image in the form of a JPEG or PNG to a tensor representation through
the rgb channel dimensions and the pixel values for each channel.
Typically image files are loaded in using libraries and methods like
the python imaging library and pytorch's
torchvision.transforms.ToTensor().

This example is very specific to an object detector type model, but most supervised ML models will output a tensor like the above or a one-hot label.  Most ML models in general will always have data inputs and outputs that can be represented as Tensors.


Instruction: Confusion in understand python commands for deep learning

Input:
I recently started learning the deep learning with pytorch using this tutorial.
I am having problem with these lines of code.
Parameter train=True means it will take out the training data.
But how much data does it take for the training 50%?
How can we specify the amount of data for training. Similarly, couldn't understand batch_size and num_workers, what that means in loading the data data? Is the batch_size parameter is similar to one used in deep learning for training?
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

Output:
If you don't split your data previously, the trainloader will use the entire train folder. You can specify the amount of training by splitting your data, see:

from torchvision import datasets

# convert data to a normalized torch.FloatTensor
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

# choose the training and test datasets
train_data = datasets.CIFAR10('data', train=True,
                              download=True, transform=transform)
test_data = datasets.CIFAR10('data', train=False,
                             download=True, transform=transform)
valid_size = 0.2

# obtain training indices that will be used for validation
num_train = len(train_data)
indices = list(range(num_train))
np.random.shuffle(indices)
split = int(np.floor(valid_size * num_train))
train_idx, valid_idx = indices[split:], indices[:split]

# define samplers for obtaining training and validation batches
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)

# prepare data loaders (combine dataset and sampler)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    sampler=train_sampler, num_workers=num_workers)
valid_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
    sampler=valid_sampler, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
    num_workers=num_workers)```

The Batch size is the numbers of files that you catch by iteration (epoch). For example, if your training_size is 1000, and you have a batch_size of 10, then each epoch would contain 100 iterations.

The number of workers is used to preprocess the data of batch. More workers will consume more memory usage and workers are helpful to speed up the Input and Output process.
num_workers = 0 means that will do the data loading when needed,
num_workers > 0 means your data will be preprocessed with the number of workers you defined.


Instruction: NumPy is faster than PyTorch for larger cross or outer products

Input:
I'm computing huge outer products between vectors of size (50500,) and found out that NumPy is (much?) faster than PyTorch while doing so.

Here are the tests:

# NumPy

In [64]: a = np.arange(50500)
In [65]: b = a.copy()

In [67]: %timeit np.outer(a, b)
5.81 s ± 56.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

-------------

# PyTorch

In [73]: t1 = torch.arange(50500)
In [76]: t2 = t1.clone()

In [79]: %timeit torch.ger(t1, t2)
7.73 s ± 143 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

I'd ideally like to have the computation done in PyTorch. So, how can I speed things up for computing outer product in PyTorch for such huge vectors?

Note: I tried to move the tensors to GPU but I was treated with MemoryError because it needs around 19 GiB of space. So, I eventually have to do it on the CPU.

Output:
Unfortunately there's really no way to specifically speed up torch's method of computing the outer product torch.ger() without a vast amount of effort.

Explanation and Options

The reason numpy function np.outer() is so fast is because it's written in C, which you can see here: https://github.com/numpy/numpy/blob/7e3d558aeee5a8a5eae5ebb6aef03de892a92ebd/numpy/core/numeric.py#L1123
where the function uses operations from the umath C source code.

Pytorch's torch.ger() function is written in C++ here: https://github.com/pytorch/pytorch/blob/7ce634ebc2943ff11d2ec727b7db83ab9758a6e0/aten/src/ATen/native/LinearAlgebra.cpp#L142 which makes it ever so slightly slower as you can see in your example.

Your options to "speed up computing outer product in PyTorch" would be to add a C implementation for outer product in pytorch's native code, or make your own outer product function while interfacing with C using something like Cython if you really don't want to use numpy (which wouldn't make much sense).

P.S.

Also just as an aside, using GPUs would only improve your parallel computation speed on the GPU which may not outweigh the cost of time required to transfer data between RAM and GPU memory.


Instruction: packed_padded_sequence gives error when used with GPU

Input:
I am trying to set up an RNN capable of utilizing a GPU but packed_padded_sequence gives me a

RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor

here is how I direct gpu computing

parser = argparse.ArgumentParser(description='Trainer')
parser.add_argument('--disable-cuda', action='store_true',
                    help='Disable CUDA')
args = parser.parse_args()
args.device = None
if not args.disable_cuda and torch.cuda.is_available():
    args.device = torch.device('cuda')
    torch.set_default_tensor_type(torch.cuda.FloatTensor)
else:
    args.device = torch.device('cpu')

here is the relevant part of the code.

def Tensor_length(track):
    """Finds the length of the non zero tensor"""
    return int(torch.nonzero(track).shape[0] / track.shape[1])
.
.
.
def forward(self, tracks, leptons):
        self.rnn.flatten_parameters()
        # list of event lengths
        n_tracks = torch.tensor([Tensor_length(tracks[i])
                                 for i in range(len(tracks))])
        sorted_n, indices = torch.sort(n_tracks, descending=True)
        sorted_tracks = tracks[indices].to(args.device)
        sorted_leptons = leptons[indices].to(args.device)
        # import pdb; pdb.set_trace()
        output, hidden = self.rnn(pack_padded_sequence(sorted_tracks,
                                                       lengths=sorted_n.cpu().numpy(),
                                                       batch_first=True)) # this gives the error

        combined_out = torch.cat((sorted_leptons, hidden[-1]), dim=1)
        out = self.fc(combined_out)  # add lepton data to the matrix
        out = self.softmax(out)
        return out, indices  # passing indices for reorganizing truth

I have tried everything from casting sorted_n to a long tensor to having it be a list, but i aways seem to get the same error.
I have not worked with pytorch on gpu before and any advice will be greatly appreciated.

Thanks!

Output:
I assume you are using the GPU and probably on the Google Colab. Check yours device
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device

you may solve this error by downgrading the torch version, if you are using colab the following command will help you:
!pip install torch==1.6.0 torchvision==0.7.0

once you downgrade the torch, this error of padded length will go.


Instruction: Implementing an “infinite loop” Dataset & DataLoader in PyTorch

Input:
I’d like to implement an infinite loop Dataset & DataLoader. Here’s what I tried:

class Infinite(Dataset):
    def __len__(self):
        return HPARAMS.batch_size
#         return 1<<30 # This causes huge memory usage.
    def __getitem__(self, idx):
        """Randomly generates one new example."""
        return sample_func_to_be_parallelized()

infinite_loader = DataLoader(
    dataset=Infinite(),
    batch_size=HPARAMS.batch_size,
    num_workers=16,
    worker_init_fn=lambda worker_id: np.random.seed(worker_id),
)

while True:
    for idx, data in enumerate(infinite_loader):
        # forward + backward on "data"

As you can see, the main challenge here is the __len()__ method. If I put a large enough number there, like 1<<30, the symptom is memory usage will JUMP TO 10+GB on the first iteration of train loop. After a while the workers are killed presumably due to OOM.

If I put a small number there, like 1 or BATCH_SIZE, the sampled “data” in the train loop will be periodically duplicated. This is not what I want as I’d like new data to be generated & trained on at every iteration.

I’m guessing the culprit of the excessive memory usage is somewhere in the stack, a bunch of things are cached. Upon a casual look at the Python side of things I can’t pinpoint where.

Can someone advise what’s the best way to have what I want implemented? (Use DataLoader’s parallel loading, while simultaneously guaranteeing every batch loaded is entirely new.)

Output:
This seems to be working without periodically duplicating the data:

import numpy as np
import torch
from torch.utils.data import Dataset, DataLoader

BATCH_SIZE = 2

class Infinite(Dataset):

    def __len__(self):
        return BATCH_SIZE

    def __getitem__(self, idx):
        return torch.randint(0, 10, (3,))

data_loader = DataLoader(Infinite(), batch_size=BATCH_SIZE, num_workers=16)

batch_count = 0
while True:
    batch_count += 1
    print(f'Batch {batch_count}:')

    data = next(iter(data_loader))
    print(data)
    # forward + backward on "data"

    if batch_count == 5:
        break

Result:

Batch 1:
tensor([[4, 7, 7],
        [0, 8, 0]])
Batch 2:
tensor([[6, 8, 6],
        [2, 6, 7]])
Batch 3:
tensor([[6, 6, 2],
        [8, 7, 0]])
Batch 4:
tensor([[9, 4, 8],
        [2, 4, 1]])
Batch 5:
tensor([[9, 6, 1],
        [2, 7, 5]])

So I think the problem is in your function sample_func_to_be_parallelized().

Edit: If instead of torch.randint(0, 10, (3,)) I use np.random.randint(10, size=3) in __getitem__ (as an example of the sample_func_to_be_parallelized()), then the data is indeed duplicated at each batch. See this issue.

So if you are using numpy's RGN somewhere in your sample_func_to_be_parallelized(), then the workaround is to use

worker_init_fn=lambda worker_id: np.random.seed(np.random.get_state()[1][0] + worker_id)

and to reset the seed by np.random.seed() before each call of data = next(iter(data_loader)).


Instruction: How to fix this strange error: "RuntimeError: CUDA error: out of memory"

Input:
I successfully trained the network but got this error during validation:

RuntimeError: CUDA error: out of memory

Output:
The error occurs because you ran out of memory on your GPU.
One way to solve it is to reduce the batch size until your code runs without this error.


Instruction: How do I extract only subset of classes from torchvision.datasets.CIFAR10?

Input:
How do I extract only 2 or 3 classes from torchvision.datasets.CIFAR10?

Standard way of loading all 10 classes

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
    testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)

Output:
By inspecting the code of CIFAR10, you can see that the data is stored as numpy array and the labels are stored as a list. You can therefore subclass this and filter the two arrays adequately. An example is below:

class SubLoader(torchvision.datasets.CIFAR10):
    def __init__(self, *args, exclude_list=[], **kwargs):
        super(SubLoader, self).__init__(*args, **kwargs)

        if exclude_list == []:
            return

        if self.train:
            labels = np.array(self.train_labels)
            exclude = np.array(exclude_list).reshape(1, -1)
            mask = ~(labels.reshape(-1, 1) == exclude).any(axis=1)

            self.train_data = self.train_data[mask]
            self.train_labels = labels[mask].tolist()
        else:
            labels = np.array(self.test_labels)
            exclude = np.array(exclude_list).reshape(1, -1)
            mask = ~(labels.reshape(-1, 1) == exclude).any(axis=1)

            self.test_data = self.test_data[mask]
            self.test_labels = labels[mask].tolist()


Instruction: Compute the number of epoch from iteration in training?

Input:
I have a Caffe prototxt as follows:

stepsize: 20000
iter_size: 4
batch_size: 10
gamma =0.1

in which, the dataset has 40.000 images. It means after 20000 iters, the learning rate will decrease 10 times. In pytorch, I want to compute the number of the epoch to have the same behavior in caffe (for learning rate). How many epoch should I use to decrease learning rate 10 times (note that, we have iter_size=4 and batch_size=10). Thanks

Ref: Epoch vs Iteration when training neural networks

My answer: Example: if you have 40000 training examples, and batch size is 10, then it will take 40000/10 =4000 iterations to complete 1 epoch. Hence, 20000 iters to reduce learning rate in caffe will same as 5 epochs in pytorch.

Output:
You did not take into account iter_size: 4: when batch is too large to fit into memory, you can "split" it into several iterations.
In your example, the actual batch size is batch_sizexiter_size=10 * 4 = 40. Therefore, an epoch takes only 1,000 iterations and therefore you need to decrease the learning rate after 20 epochs.


Instruction: Preprocessing in image recognition

Input:
I am a beginner in image recognition and need some help about preprocessing images.

I use transfer learning model resnet18 to do the recognition work. And I get:

    In [3]: pretrainedmodels.pretrained_settings['resnet18']
    Out[3]:
    {'imagenet': {'url':
    'https://download.pytorch.org/models/resnet18-
       5c106cde.pth',
      'input_space': 'RGB',
      'input_size': [3, 224, 224],
      'input_range': [0, 1],
      'mean': [0.485, 0.456, 0.406],
      'std': [0.229, 0.224, 0.225],
      'num_classes': 1000}}

I find that the mean and std is quite different from my image dataset's.

How should I normalize my trainset? Use the mean and std above or use the mean and std I calculate myself?

I divided my dataset into train_set, valid_set and test_set.

I have two methods:

A.calculate their mean and std and normalize them individually

B.calculate the whole dataset's mean and std and then do normalization.

Which one is right?

3.When should i do normalization? Before data_augmentation or after data_augmentation?

Output:
If you training a new model with your own dataset, with the pre-trained weights, you will need to a new mean and std for your new dataset.

Basically you will need to repeat the process of how ImageNet did it. Make a script that calculates the general [mean, std] value of your entire dataset.

But remember to keep watch on your dataset distribution as it will definitely affect the model performance.

Then define a transformer method individually for your train/valset. Usually we do not normalize the test set as in real world scenario your model will takes in data of different sort. You should perform the normalization process when building the dataset, together with other augmentation techniques.

For instance, consider this toy example

"transformer": {
    "train": transforms.Compose([
        transforms.Resize(size=299),
        transforms.RandomHorizontalFlip(p=0.2),
        transforms.ToTensor(),
        transforms.Normalize(new_mean, new_std)
    ]),
    "valid": transforms.Compose([
        transforms.Resize(size=299),
        transforms.ToTensor(),
    ])
}

train_ds = CustomDataset(type="train", transformer=transformer["train"])
valid_ds = CustomDataset(type="valid", transformer=transformer["valid"])

Let me know if you have more confusion


Instruction: How to (quickly) extract bilinear-interpolated patches from a 2d image at specific points?

Input:
  Update: The original question formulation was a bit unclear. I am not just cropping the image but applying bilinear interpolation during the patches extraction process. (See the paper reference below). That's why the algorithm is a bit more involved than just taking slices.

I am trying to train a deep learning model to predict face landmarks following this paper. I need to crop parts of the image that contains face into smaller patches around facial landmarks. For example, if we have the image shown below:

The function should generate N=15 "patches", one patch per landmark:

I have the following naïve implementation build on top of torch tensors:

def generate_patch(x, y, w, h, image):
    c = image.size(0)
    patch = torch.zeros((c, h, w), dtype=image.dtype)
    for q in range(h):
        for p in range(w):
            yq = y + q - (h - 1)/2
            xp = x + p - (w - 1)/2
            xd = 1 - (xp - math.floor(xp))
            xu = 1 - (math.ceil(xp) - xp)
            yd = 1 - (yq - math.floor(yq))
            yu = 1 - (math.ceil(yq) - yq)
            for idx in range(c):
                patch[idx, q, p] = (
                    image[idx, math.floor(yq), math.floor(xp)]*yd*xd +
                    image[idx, math.floor(yq),  math.ceil(xp)]*yd*xu +
                    image[idx,  math.ceil(yq), math.floor(xp)]*yu*xd +
                    image[idx,  math.ceil(yq),  math.ceil(xp)]*yu*xu
                ).item()
    return patch

def generate_patches(image, points, n=None, sz=31):
    if n is None:
        n = len(points)//2
    patches = []
    for i in range(n):
        x_val, y_val = points[i], points[i + n]
        patch = generate_patch(x_val, y_val, sz, sz, image)
        patches.append(patch)
    return patches

The code does its work but too slowly. I guess because of all these for-loops and separate pixels indexing. I would like to vectorize this code, or maybe find some C-based implementation that could do it faster.

I know there is the extract_patches_2d function from sklearn package that helps to pick random patches from the image. However, I would like to pick the patches from specific points instead of doing it randomly. I guess that I can somehow adapt the aforementioned function, or convert the implementation shown above into Cython/C code but probably someone has already done something like this before.

Could you please advise some alternative to the code shown above, or maybe a proposal on how to make it faster? (Except using several parallel workers).

Output:
1) use numpy

2) select patches with index extraction. Example:

Patch=img[0:100,0:100]

3) create 3 dimensional body where in 3rd dimension are patches. [15x15xnumber of patches]

4) do your bilinear int. With numpy for all patches in the same time( insted of one pixel calculate with all pixels in 3rd dimension).

That will increase your processing beyond your imagination

If you dont want to get old waiting for your job to be done forget math module. It has no place in datascience.


Instruction: Using CNN for pixel classification,how?

Input:
I have already learn about some classification using CNN like for Mnist. But recently I received a dataset which is consist of a vector set. The normal image dataset(mnist) is like nxcxwxh. The one I received is (w*h)x1xc. The goal is to train a network to classify these pixels(as I understand,is classification for pixels). The label length is groundtruth picture.

I am a little confused about this work. As I understand, for Image processing, we use the CNN with different receiption feld to make the convolution operations so that a feature can be obtained to represent the image. But in this case, the image is already expand to pixel-set. Why is the convolutional neural network still suitable?

Still I am not sure about the work but I started to try.I used the 1d convolution instead of 2d in the network. After 4-Conv1d,the output is connected to a softmax layer then fed to crossentropy loss function. It seems, I have some problems with the output dimensions so the network is not able to train.

I use pytorch to implement the work. Below is the network form I try to build. The dimensions do not match those need for crossentropyloss function. 122500 was set to be the sample numbers. So I think the convolution was performed along the 1-200 directions.

First I want to know,is that right to implement like this using conv1d when I want to classify the pixels?

If this thought was right, how can I continue to feed the features to the loss function?

If this is wrong,can I have some similar examples for this kind of work? I am new to python, so if there were some stupid mistakes, pls point out.

Thanks all.

class network(nn.Module):
"""
    Building network

"""
def __init__(self):
    super(network, self).__init__()
    self.conv1 = nn.Conv1d(in_channels = 1,out_channels = 32,stride = 1,kernel_size = 3)
    self.conv2 = nn.Conv1d(in_channels = 32,out_channels = 64,stride = 1,kernel_size = 3)
    self.conv3 = nn.Conv1d(in_channels = 64,out_channels = 128,stride = 1,kernel_size = 3)
    self.conv4 = nn.Conv1d(in_channels = 128,out_channels = 256,stride = 1,kernel_size = 3)
    self.fc = nn.Linear(13, 2)

def forward(self,s):
    s = self.conv1(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv2(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv3(s)
    s = F.relu(F.max_pool1d(s, 2))
    s = self.conv4(s)
    s = F.relu(F.max_pool1d(s, 2))

    s = self.fc(s)

    s = F.softmax(s,1)

output = model(input)
loss = loss_fn(output, labels)

Output: I guess what you're supposed to do is image segmentation and in the shape of the labels you got, the last dimension of 200 corresponds to 200 possible categories of pixels (that sounds like a lot to me, but without more context I cannot judge). The problem of image segmentation is way too broad to explain in an SO answer, but I suggest you check resources such as this tutorial and check out the influential papers in this field.