Instruction: Why pytorch training on CUDA works much slower than in CPU?

Input:
I guess i have made something in folowing simple neural network with PyTorch, because this runs much slower with CUDA then in CPU, can you find the mistake pls. The using function like

    def backward(ctx, input):

        return backward_sigm(ctx, input)

seems have no real impact on preformance

import torch
import torch.nn as nn
import torch.nn.functional as f

dname = 'cuda:0'
dname = 'cpu'

device = torch.device(dname)

print(torch.version.cuda)

def forward_sigm(ctx, input):

    sigm = 1 / (1 + torch.exp(-input))

    ctx.save_for_backward(sigm)

    return sigm

def forward_step(ctx, input):

    return  torch.tensor(input > 0.5, dtype = torch.float32, device = device)

def backward_sigm(ctx, grad_output):

    sigm, = ctx.saved_tensors

    return grad_output * sigm * (1-sigm)

def backward_step(ctx, grad_output):

    return grad_output

class StepAF(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        return forward_sigm(ctx, input)

    @staticmethod
    def backward(ctx, input):

        return backward_sigm(ctx, input)
    #else return grad_output

class StepNN(torch.nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(StepNN, self).__init__()
        self.linear1 = torch.nn.Linear(input_size, hidden_size)
        #self.linear1.cuda()
        self.linear2 = torch.nn.Linear(hidden_size, output_size)
        #self.linear2.cuda()

        #self.StepAF = StepAF.apply

    def forward(self,x):

        h_line_1 = self.linear1(x)

        h_thrash_1 = StepAF.apply(h_line_1)

        h_line_2 = self.linear2(h_thrash_1)

        output = StepAF.apply(h_line_2)

        return output

inputs = torch.tensor( [[1,0,1,0],[1,0,0,1],[0,1,0,1],[0,1,1,0],[1,0,0,0],[0,0,0,1],[1,1,0,1],[0,1,0,0],], dtype = torch.float32, device = device)

expected = torch.tensor( [[1,0,0],[1,0,0],[0,1,0],[0,1,0],[1,0,0],[0,0,1],[0,1,0],[0,0,1],], dtype = torch.float32, device = device)

nn = StepNN(4,8,3)

#print(*(x for x in nn.parameters()))

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(nn.parameters(), lr=1e-3)

steps = 50000

print_steps = steps // 20

good_loss = 1e-5

for t in range(steps):

    output = nn(inputs)
    loss = criterion(output, expected)

    if t % print_steps == 0:
        print('step ',t, ', loss :' , loss.item())

    if loss < good_loss:
        print('step ',t, ', loss :' , loss.item())
        break

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

test = torch.tensor( [[0,1,0,1],[0,1,1,0],[1,0,1,0],[1,1,0,1],], dtype = torch.float32, device=device)

print(nn(test))

Output:
Unless you have large enough data, you won't see any performance improvement while using GPU. The problem is that GPUs use parallel processing, so unless you have large amounts of data, the CPU can process the samples almost as fast as the GPU.

As far as I can see in your example, you are using 8 samples of size (4, 1). I would imagine maybe when having over hundreds or thousands of samples, then you would see the performance improvement on a GPU. In your case, the sample size is (4, 1), and the hidden layer size is 8, so the CPU can perform the calculations fairly quickly.

There are lots of example notebooks online of people using MNIST data (it has around 60000 images for training), so you could load one in maybe Google Colab and then try training on the CPU and then on GPU and observe the training times. You could try this link for example. It uses TensorFlow instead of PyTorch but it will give you an idea of the performance improvement of a GPU.

Note : If you haven't used Google Colab before, then you need to change the runtime type (None for CPU and GPU for GPU) in the runtime menu at the top.

Also, I will post the results from this notebook here itself (look at the time mentioned in the brackets, and if you run it, you can see firsthand how fast it runs) :

On CPU :

INFO:tensorflow:loss = 294.3736, step = 1
INFO:tensorflow:loss = 28.285727, step = 101 (23.769 sec)
INFO:tensorflow:loss = 23.518856, step = 201 (24.128 sec)

On GPU :

INFO:tensorflow:loss = 295.08328, step = 0
INFO:tensorflow:loss = 47.37291, step = 100 (4.709 sec)
INFO:tensorflow:loss = 23.31364, step = 200 (4.581 sec)
INFO:tensorflow:loss = 9.980572, step = 300 (4.572 sec)
INFO:tensorflow:loss = 17.769928, step = 400 (4.560 sec)
INFO:tensorflow:loss = 16.345463, step = 500 (4.531 sec)


Instruction: How to convert torch int64 to torch LongTensor?

Input:
I am going through a course which uses a deprecated version of PyTorch which does not change torch.int64 to torch.LongTensor as needed. The current section of code which is throwing the error is:

loss = loss_fn(Ypred, Ytrain_) # calc loss on the prediction

I believe the dtype should be changed in thhis section though:

Ytrain_ = torch.from_numpy(y_train.values).view(1, -1)[0].

When testing the data-type by using Ytrain_.dtype it returns torch.int64. I have tried to convert it by applying the long() function as such: Ytrain_ = Ytrain_.long() to no avail.

I have also tried looking for it in the documentation but it seems that it says torch.int64 OR torch.long which I assume means torch.int64 should work.

RuntimeError                              Traceback (most recent call last)
----> 9     loss = loss_fn(Ypred, Ytrain_) # calc loss on the prediction
RuntimeError: Expected object of scalar type Long but got scalar type Int for argument #2 'target'

Output:
As stated by user8426627 you want to change the tensor type, not the data type. Therefore the solution was to add .type(torch.LongTensor) to convert it to a LongTensor.

Final code:

Ytrain_ = torch.from_numpy(Y_train.values).view(1, -1)[0].type(torch.LongTensor)

Test tensor type:

Ytrain_.type()

'torch.LongTensor'


Instruction: converting tensor to one hot encoded tensor of indices

Input:
I have my label tensor of shape (1,1,128,128,128) in which the values might range from 0,24. I want to convert this to one hot encoded tensor, using the nn.fucntional.one_hot function

n = 24
one_hot = torch.nn.functional.one_hot(indices, n)

but this expects a tensor of indices, honestly, I am not sure how to get those. The only tensor I have is the label tensor of the shape described above and it contains values ranging from 1-24, not the indices

How can I get a tensor of indices from my tensor? Thanks in advance.

Output:
If the error you are getting is this one:

Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
RuntimeError: one_hot is only applicable to index tensor.

Maybe you just need to convert to int64:

import torch

# random Tensor with the shape you said
indices = torch.Tensor(1, 1, 128, 128, 128).random_(1, 24)
# indices.shape => torch.Size([1, 1, 128, 128, 128])
# indices.dtype => torch.float32

n = 24
one_hot = torch.nn.functional.one_hot(indices.to(torch.int64), n)
# one_hot.shape => torch.Size([1, 1, 128, 128, 128, 24])
# one_hot.dtype => torch.int64

You can use indices.long() too.


Instruction: best_state changes with the model during training in pytorch

Input:
I want to save the best model and then load it during the test. So I used the following method:

def train():
    #training steps …
    if acc > best_acc:
        best_state = model.state_dict()
        best_acc = acc
    return best_state

Then, in the main function I used:

model.load_state_dict(best_state)

to resume the model.

However, I found that best_state is always the same as the last state during training, not the best state. Is anyone know the reason and how to avoid it?

By the way, I know I can use torch.save(the_model.state_dict(), PATH) and then load the model by
the_model.load_state_dict(torch.load(PATH)).
However, I don’t want to save the parameters to file as train and test functions are in one file.

Output:
model.state_dict() is OrderedDict

from collections import OrderedDict

You can use:

from copy import deepcopy

To fix the problem

Instead:

best_state = model.state_dict()

You should use:

best_state = copy.deepcopy(model.state_dict())

Deep (not shallow) copy makes the mutable OrderedDict instance not to mutate best_state as it goes.

You may check my other answer on saving the state dict in PyTorch.


Instruction: When to use layernorm/batch norm?

Input: Where should you splice the normalization when designing a network? E.g. if you have a stacked Transformer or Attention network, does it make sense to normalize any time after you have a dense layer?

Output:
What the original paper tries to explain is to reduce overfitting use Batch Normalization.

  Where should you splice the normalization when designing a network?

Set the normalization early on inputs. Unbalanced input extreme values can cause instability.

While if you normalize on outputs this will not prevent the inputs to cause the instability all over again.

Here is the little code that explains what the BN do:

import torch
import torch.nn as nn

m = nn.BatchNorm1d(100, affine=False)
input = 1000*torch.randn(3, 100)
print(input)
output = m(input)
print(output)
print(output.mean()) # should be ~ 0
print(output.std()) # should be ~ 1

  Does it make sense to normalize any time after you have a dense layer

Yes, you may do so as matrix multiplication may lead to producing the extremes. Also, after convolution layers, because these are also matrix multiplication, similar but less intense comparing to dense (nn.Linear) layer. If you for instance print the resent model, you will see that batch norms are set every time after the conv layer like this:

(conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
(bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

To print the full resnet you may use this:

import torchvision.models as models
r = models.resnet18()
print(r)


Instruction: How do I install Pytorch offline?

Input:
I need to install Pytorch on a computer with no internet connection.

I've tried finding information about this online but can't find a single piece of documentation.

Do you know how I can do this? Is it even possible?

Output:
An easy way with pip:

Create an empty folder
pip download torch using the connected computer. You'll get the pytorch package and all its dependencies.
Copy the folder to the offline computer. You must be using the same python setup on both computers (this goes for virtual environments as well)
pip install * on the offline computer, in the copied folder. This installs all the packages in the correct order. You can then use pytorch.

Note that this works for (almost) any kind of python package.


Instruction: Pytorch loss of convolution is 0.0 from start

Input:
I am building a conv net that classifies dog and cat. Architecture is pretty simple. 2 Conv(with batch norm, leakyReLU, Maxpooling) to 1 fc. Input image size is resized to 64. The size is good. The problem is loss is 0.0 from the start. I have no clue what the cause is. I couldn't find any answer. I have wrote every detail that might be important. If you need anything else please tell me, I will edit.

  main.py

import torch
import torch.nn as nn
from torchvision import transforms, datasets
import PIL
import matplotlib.pyplot as plt
from Dataset import Dataset
from Network import Network
# Added to avoid     torch._C._cuda_init() \n RuntimeError: CUDA error: unknown error
torch.cuda.current_device()
# Hyper Parameters
batch_size = 1
img_size = 64
learning_rate = 0.001
num_epoch = 1

# Directories
trainDir = "D:/Programming/python/Deep learning/datasets/dogs-vs-cats/train"
testDir = "D:/Programming/python/Deep learning/datasets/dogs-vs-cats/test1"

print("Initializing...")

# Device
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Augmentation
transforms = transforms.Compose([
    transforms.Resize((img_size, img_size)),
    transforms.ColorJitter(hue=.05, saturation=.05),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(20, resample=PIL.Image.BILINEAR) ,
    transforms.ToTensor()
])

trainset = datasets.ImageFolder(root=trainDir, transform=transforms)
testset = datasets.ImageFolder(root=testDir, transform=transforms)
train_loader = torch.utils.data.DataLoader(
    trainset, batch_size=batch_size, shuffle=True)
test_loader = torch.utils.data.DataLoader(
    testset, batch_size=batch_size, shuffle=False)  # test set will not be shuffled

model = Network(img_size,2).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

total_step = len(train_loader)
print("Tranining started")
for epoch in range(num_epoch):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        # forward propagate
        outputs = model(images)
        loss = criterion(outputs, labels)

        # backpropagte and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print(
                "Epoch [{}/{}], Step[{}/{}], Loss: {}".format(
                    epoch+1, num_epoch, i+1, total_step, loss.item()
                )
            )
print("Tranining complete, validation started")

with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

        print('Test Accuracy: {} %'.format(100 * correct / total))

#
torch.save(model.state_dict(), "model.ckpy")

  Network.py

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

"""
    Input size for conv
    l = number of input feature maps
    k = number of output feature maps
    n, m = width and height of kernel
    total parameter = (n*m*l+1)*k
"""

class Network(nn.Module):
    def __init__(self, input_size, num_class):
        super(Network, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ) # output size = (128, 128, 16)
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.LeakyReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        ) # output size = (64, 64, 32)

        self.fc1 = nn.Linear(
            int((input_size/4)**2*32), num_class
        )

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        return out

  Output

Epoch [1/1], Step[5800/25000], Loss: 0.0
Epoch [1/1], Step[5900/25000], Loss: 0.0
Epoch [1/1], Step[6000/25000], Loss: 0.0
Epoch [1/1], Step[6100/25000], Loss: 0.0
Epoch [1/1], Step[6200/25000], Loss: 0.0
Epoch [1/1], Step[6300/25000], Loss: 0.0
Epoch [1/1], Step[6400/25000], Loss: 0.0
Epoch [1/1], Step[6500/25000], Loss: 0.0

  Result after each layer

outputs of conv1,2
         [[ 3.0135e-01,  3.5849e-01,  4.7758e-01,  ...,  3.9759e-01,
            3.7988e-01,  9.7870e-01],
          [ 4.3010e-01,  6.0753e-03,  4.5642e-01,  ..., -8.5486e-04,
            4.4537e-02,  2.9074e-01],
          [ 3.8567e-01,  7.8431e-02,  2.3859e-01,  ..., -3.0013e-03,
           -5.5821e-03,  1.2284e-01],
          ...,
          [ 3.9181e-01,  3.9093e-01,  1.2053e-01,  ..., -4.7156e-03,
            5.6266e-01,  7.7017e-01],
outputs of fc1
          [[-0.0772,  0.2166]]

Output:
loss = criterion(output, target.view(-1))  # Flatten target

try this.

could you remove these two line?
images = images.to(device)
labels = labels.to(device)

self.conv1 and 2 must be sent to cuda : self.conv1(2).cuda()


Instruction: How to solve the run time error "Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment"

Input:
I want to use output variables of NN as an input in another function,but met with error like this 'Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment'.The out variables require gradient.

I tried by changing the output variables to numpy values, but in that case the back propagataion does not work because it see numpy values as variables which does not need gradient.

output = model(SOC[13])

# Three output values of NN
Rs=output[0]
R1=output[1]
C1=output[2]

# Using these variables in another function

num1=[Rs*R1*C1,R1+Rs]
den1=[C1*R1,1]
G = control.tf(num,den)

It should work, but it gives error.

     14             num=[Rs*R1*C1,R1+Rs]
     15             den=[C1*R1,1]
---> 16             G = control.tf(num,den)
~\Anaconda3\lib\site-packages\control\xferfcn.py in __init__(self, *args)
    106
    107         """
--> 108         args = deepcopy(args)
    109         if len(args) == 2:
    110             # The user provided a numerator and a denominator.
~\Anaconda3\lib\site-packages\torch\tensor.py in __deepcopy__(self, memo)
     16     def __deepcopy__(self, memo):
     17         if not self.is_leaf:
---> 18             raise RuntimeError("Only Tensors created explicitly by the user "
     19                                "(graph leaves) support the deepcopy protocol at the moment")
     20         if id(self) in memo:

Output:
In pytorch, you can use the #tensor_name#.detach() function
new_tensor = _tensor_.detach()


Instruction: What is a fused kernel (or fused layer) in deep learning?

Input:
I am reading the Apex AMP documentation:

  A Python-only build omits:

  Fused kernels required to use apex.optimizers.FusedAdam.
  Fused kernels
  required to use apex.normalization.FusedLayerNorm.
  Fused kernels that
  improve the performance and numerical stability of
  apex.parallel.SyncBatchNorm.
  Fused kernels that improve the
  performance of apex.parallel.DistributedDataParallel and apex.amp.
  DistributedDataParallel, amp, and SyncBatchNorm will still be usable,
  but they may be slower.

There also seems to be a "FusedAdam" optimizer:

  The Adam optimizer in Pytorch (like all Pytorch optimizers) carries
  out optimizer.step() by looping over parameters, and launching a
  series of kernels for each parameter. This can require hundreds of
  small launches that are mostly bound by CPU-side Python looping and
  kernel launch overhead, resulting in poor device utilization.
  Currently, the FusedAdam implementation in Apex flattens the
  parameters for the optimization step, then carries out the
  optimization step itself via a fused kernel that combines all the Adam
  operations. In this way, the loop over parameters as well as the
  internal series of Adam operations for each parameter are fused such
  that optimizer.step() requires only a few kernel launches.

  The current implementation (in Apex master) is brittle and only works
  with Amp opt_level O2. I’ve got a WIP branch to make it work for any
  opt_level (https://github.com/NVIDIA/apex/pull/351). I recommend
  waiting until this is merged then trying it.

This partially explains it. I'm left with more questions:

What is meant by kernel? A layer or an optimizer?

Is the idea of fused layer the same as a fused optimizer?

Output:
"Kernel" here is for computation kernels: https://en.wikipedia.org/wiki/Compute_kernel
Operations like convolution are often implemented using compute kernels for better efficiency. Compute kernels can be written using C, CUDA, OpenCL or even assembly for maximum efficiency. It is therefore not surprizing that "a Python-only build" does not support...
"Fusing" means commonalization of computation steps. Basically, it's an implementation trick to run code more efficiently by combining similar operations in a single hardware (GPU, CPU or TPU) operation. Therefore, a "fusedLayer" is a layer where operations benefit from a "fused" implementation.


Instruction: Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

Input:
I have a model in mind, but I'm having a hard time figuring out how to actually implement it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all let me quickly introduce the context:

I'm working on VQA (visual question answering), in which the task is to answer questions about images, for example:

So, letting aside many details, I just want to focus here on the NLP aspect/branch of the model. In order to process the natural language question, I want to  use character-level embeddings (instead of traditional word-level embeddings) because they are more robust in the sense that they can easily accommodate for morphological variations in words (e.g. prefixes, suffixes, plurals, verb conjugations, hyphens, etc.). But at the same time I don't want to lose the inductive bias of reasoning at the word level. Therefore, I came up with the following design:

As you can see in the picture above, I want to use transformers (or even better, universal transformers), but with a little twist. I want to use 2 transformers: the first one will process each word characters in isolation (character-level transformer) to produce an initial word-level embedding for each word in the question. Once we have all these initial word-level embeddings, a second word-level transformer will refine these embeddings to enrich their representation with context, thus obtaining context-aware word-level embeddings.

The full model for the whole VQA task obviously is more complex, but I just want to focus here on this NLP part. So my question is basically about which Pytorch functions should I pay attention to when implementing this. For example, since I'll be using character-level embeddings I have to define a character-level embedding matrix, but then I have to perform lookups on this matrix to generate the inputs for the character-level transformer, repeat this for each word in the question and then feed all these vectors into the word-level transformer. Moreover, words in a single question can have different lengths, and questions within a single mini-batch can have different lengths too. So in my code I have to somehow account for different lengths at the word and the question level simultaneously in a single mini-batch (during training), and I've got no idea how to do that in Pytorch or whether it's even possible at all.

Any tips on how to go about implementing this in Pytorch that could lead me in the right direction will be deeply appreciated.

Output:
A way to implement what  you say in pyTorch would require adapting the Transformer encoder:

1) Define a custom tokenizer that splits words into character embeddings (instead of word or word-piece embeddings)

2) Define a mask for each word (similar to what the original paper used to mask future tokens in the decoder), in order to force the model to be constrained to the word-context (in the first stage)

3) Then use a traditional Transformer with the mask (effectively restricting word-level context).

4) Then discard the mask and apply Transformer again (sentence-level context).

.

Things to be careful about:

1) Remember that Transformer encoder's output length is always the same size as the input (the decoder is the one able to produce longer or shorter sequences). So in your first stage, you will not have word-level embeddings (as shown in your diagram) but character level embeddings. If you want to merge them into word level embeddings, you will need an additional intermediate decoder step or merge the embeddings using a custom strategy (ex: a learnt weighted sum or using something similar to BERT's  token).

2) You may face efficiency issues. Remember that Transformer is O(n^2), so the longer the sequence, the more computationally-expensive it is. In the original Transformer, if you had a sentence of length 10 words, then the Thansformer will have to deal with a 10-token sequence. If you use word-piece embeddings, your model will work at around ~15-token sequences. But if you use character-level embeddings, I estimate that you will be dealing with ~50-token sequences, which may not be feasible for long sentences, so you may need to truncate your input (and you will be losing all the long-term dependency power of attention models).

3) Are you sure that you will have a representational contribution by adding the character-level Transformer? Transformer aims to enrich embeddings based on the context (surrounding embeddings), that's why the original implementation used word-level embeddings. BERT uses word-piece embeddings, to take advantage of language regularities in related words and GPT-2 uses Byte-Pais-Embeddings (BPE), which I don't recommend in your case, because it is more suited for next-token prediction. In your case, what information do you think will be captured at the learnt character embeddings so that it can be effectively shared between the characters of the word? Do you think it will be richer than using a learnt embedding for each word or word-piece? My guess is that this is what you are trying to find out... right?


Instruction: No module named torch.distributed

Input:
ImportError: No module named torch.distributed

File "train.py", line 4, in <module>
    import torch.distributed as dist
ImportError: No module named torch.distributed

I installed CUDA AND cuDNN then created env and installed pip3 install torch torchvision but getting error.

Output:
This works for me:

Create a conda virtual environment:

conda create -n env_pytorch python=3.6

Active this environment create above:

source activate env_pytorch

Install PyTorch with pip or pip3:

pip install torchvision --user


Instruction: What does nnz in mean in the output of torch.sparse_coo_tensor(indices, values, size=None, dtype=None, device=None, requires_grad=False)?

Input:
What does nnz mean in the output of below pytorch function

torch.sparse_coo_tensor(indices, values, size=None, dtype=None, device=None, requires_grad=False)

It can be found at this link https://pytorch.org/docs/stable/torch.html

i = torch.tensor([[0, 1, 1],
                      [2, 0, 2]])

v = torch.tensor([3, 4, 5], dtype=torch.float32)

torch.sparse_coo_tensor(i, v, [2, 4],
                            dtype=torch.float64,
                            device=torch.device('cuda:0'))

tensor(indices=tensor([[0, 1, 1],
                       [2, 0, 2]]),
       values=tensor([3., 4., 5.]),
       device='cuda:0', size=(2, 4), nnz=3, dtype=torch.float64,
       layout=torch.sparse_coo)

Output: nnz mean number non zero elements. In this example nnz = 3.


Instruction: How does DataParallel figure out which gpu I want to use?

Input:
I want to find a simple way to specify the gpus that my experiments run on. Currently, I know I can use prepend my python command with CUDA_VISIBLE_DEVICES=1,2,3,4 to set the gpu, and I am guessing DataParallel will then try to use all the gpu.

Is there a way to tell DataParallel directly the ids, like 4,7,9,12?

Output:
Yeah, DataParallem provides us the feature of directly passing the gpu ids.

As per the official documentation here, Data Parallelism is implemented using torch.nn.DataParallel. One can wrap a Module in DataParallel and it will be parallelized over multiple GPUs in the batch dimension.

torch.nn.DataParallel(module, device_ids=None, output_device=None, dim=0)

In your case, you can simply do something like this:

torch.nn.DataParallel(model, device_ids=[4, 7, 9, 12])
output = net(input_var)  # input_var can be on any device, including CPU

You can know more about how to pass gpu ids directly to DataParallel in below links:

MULTI-GPU EXAMPLES
DataParallel layers


Instruction: How to fix pytorch multi processing issue on cpu?

Input: I'm doing inference of pytorch on CPU. I found pytorch is not utilizing all the cores of CPU for prediction. How to use all cores in pytorch?

Output:
Skeleton

Using the skeleton below I see 4 processes running. You should tweak n_train_processes. I set it to 10 which was 2-much as I have 8 cores. Setting it to 6 work fine.

...
import torch.multiprocessing as mp

class MyModel(nn.Module):
    ...

def train(model, rank):
    ...

def test(model):
    ...

n_train_processes = 3

if __name__ == '__main__':
    model = MyModel()
    model.share_memory()

    processes = []
    for rank in range(n_train_processes + 1):  # + 1 for test process
        if rank == 0:
            p = mp.Process(target=test, args=(model,))
        else:
            p = mp.Process(target=train, args=(model, rank,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

Complete example

This example is taken from https://github.com/seungeunrho/minimalRL which has some other nice RL examples. This is a3c.py.

# a3c.py
import gym
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical
import torch.multiprocessing as mp
import time

n_train_processes = 6

# Hyperparameters
learning_rate = 0.0002
update_interval = 5
gamma = 0.98
max_train_ep = 300
max_test_ep = 400

class ActorCritic(nn.Module):
    def __init__(self):
        super(ActorCritic, self).__init__()
        self.fc1 = nn.Linear(4, 256)
        self.fc_pi = nn.Linear(256, 2)
        self.fc_v = nn.Linear(256, 1)

    def pi(self, x, softmax_dim=0):
        x = F.relu(self.fc1(x))
        x = self.fc_pi(x)
        prob = F.softmax(x, dim=softmax_dim)
        return prob

    def v(self, x):
        x = F.relu(self.fc1(x))
        v = self.fc_v(x)
        return v

def train(model, rank):
    optimizer = optim.Adam(model.parameters(), lr=learning_rate)
    env = gym.make('CartPole-v1')

    for n_epi in range(max_train_ep):
        done = False
        s = env.reset()
        while not done:
            s_lst, a_lst, r_lst = [], [], []
            for t in range(update_interval):
                prob = model.pi(torch.from_numpy(s).float())
                m = Categorical(prob)
                a = m.sample().item()
                s_prime, r, done, info = env.step(a)

                s_lst.append(s)
                a_lst.append([a])
                r_lst.append(r/100.0)

                s = s_prime
                if done:
                    break

            R = 0.0
            R_lst = []
            for reward in r_lst[::-1]:
                R = gamma * R + reward
                R_lst.append([R])
            R_lst.reverse()

            done_mask = 0.0 if done else 1.0
            s_batch, a_batch, R_batch, s_final = \
                torch.tensor(s_lst, dtype=torch.float), torch.tensor(a_lst), \
                torch.tensor(R_lst), torch.tensor(s_prime, dtype=torch.float)

            td_target = R_batch + gamma * model.v(s_final) * done_mask
            advantage = td_target - model.v(s_batch)
            pi = model.pi(s_batch, softmax_dim=1)
            pi_a = pi.gather(1, a_batch)
            loss = -torch.log(pi_a) * advantage.detach() + \
                F.smooth_l1_loss(td_target.detach(), model.v(s_batch))

            optimizer.zero_grad()
            loss.mean().backward()
            optimizer.step()

    env.close()
    print("Training process {} reached maximum episode.".format(rank))

def test(model):
    env = gym.make('CartPole-v1')
    score = 0.0
    print_interval = 20

    for n_epi in range(max_test_ep):
        done = False
        s = env.reset()
        while not done:
            prob = model.pi(torch.from_numpy(s).float())
            a = Categorical(prob).sample().item()
            s_prime, r, done, info = env.step(a)
            s = s_prime
            score += r

        if n_epi % print_interval == 0 and n_epi != 0:
            print("# of episode :{}, avg score : {:.1f}".format(
                n_epi, score/print_interval))
            score = 0.0
            time.sleep(1)
    env.close()

if __name__ == '__main__':
    model = ActorCritic()
    model.share_memory()
    processes = []
    for rank in range(n_train_processes + 1):  # + 1 for test process
        if rank == 0:
            p = mp.Process(target=test, args=(model,))
        else:
            p = mp.Process(target=train, args=(model, rank,))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()


Instruction: fastai.vision Import Error: How to fix the import error so I can use ImageDataBunch.from_folder?

Input:
I'm working on google colab ( python 3.6 and GPU), I import torch (1.2.0) nicely and I use the following to import fastai:

    import fastai
    print(fastai.__version__)
    from fastai import *
    from fastai.vision import *

I get the following error:

ImportError: /usr/local/lib/python3.6/dist-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106Device8validateEv

I have tried to install different torch versions like 1.0.0 and work with earlier python versions. I have also tried to install fastai and its dependencies manually using !pip, but nothing worked.

This is the complete code I used to install torch:

    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag

    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(),
    get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\
    ([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install torch_nightly -f
 https://download.pytorch.org/whl/nightly/{accelerator}/torch_nightly.html
    !pip install fastai

   import torch

   print(torch.__version__)
   print(torch.cuda.is_available())
   print(torch.backends.cudnn.enabled)

And this is the full error message I get:

    ImportError
    Traceback (most recent call last)
    <ipython-input-5-4b8b8d8134df> in <module>()
  2 print(fastai.__version__)
  3 from fastai import *
  ----> 4 from fastai.vision import *
    8 frames
  /usr/local/lib/python3.6/dist-packages/torchvision/ops/boxes.py in
  <module>()
  1 import torch
  ---->  2 from torchvision import _C
  3
  4
  5 def nms(boxes, scores, iou_threshold):

  ImportError: /usr/local/lib/python3.6/dist-
  packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined
  symbol: _ZN3c106Device8validateEv

I'm unable to use ImageDataBunch.from_folder because of this fastai import error. I get the error NameError: name 'ImageDataBunch' is not defined when I do.

Note: I did use the same code before and I was able to use fastai and  ImageDataBunch.from_folder with no import errors , but I'm guessing that an update to fastai or torch happened.

Output:
You don't need to install PyTorch first when you use FastAi, it will do it for you.
If you need latest FastAi do this:

pip3 install git+https://github.com/fastai/fastai.git


Instruction: PyTorch Convolution `in_channels` and `out_channels` meaning?

Input:
From the PyTorch documentation for Convolution, I see the function torch.nn.Conv1d requires users to pass the parameters in_channels and out_channels.
I know these refer to "input channels" and "output channels", but I am not sure what they mean in the context of a convolution. My guess is that in_channels is equivalent to the input features and out_channels is equivalent to the output features, but I am not sure.
Could someone explain what thse arguments refer to?

Output:
Given a convolution of:

length m
over N input channels / signals / variables
outputting P channels / features / filters

you would use:
nn.Conv1d(in_channels=N, out_channels=P, kernel_size=m)

This is illustrated for 2d images below in Deep Learning with PyTorch (where the kernels are of size 3x3xN (where N=3 for an RGB image), and there are 5 such kernels for the 5 outputs desired):


Instruction: How Batch learning in Pytorch is performed?

Input:
When you look at how network architecture is built inside the pytorch code, we need to extend the torch.nn.Module and inside __init__, we define the module of networks and pytorch is going to track the gradients of parameters of these modules. Then inside the forward function, we define how the forward pass should be done for our network.

The thing I do not understand here is how the batch learning is going to occur. In none of the definition above including the forward function, we do not care about the dimension of batch of the input to our network. The only thing we need to set to perform batch learning is to add an extra dimension to the input which corresponds to the batch size but nothing inside the network definition is going to be changed if we are working with batch learning. At least, this is the thing I have seen in the codes here.

So, if all the things I have explained so far is correct (I would really appreciate if you let me know if I have misunderstood something), how batch learning is performed if nothing is declared regarding the batch size inside the definition of our network class (the class that inherits torch.nn.Module)? Specifically, I am interested to know how batch gradient descent algorithm is implemented in pytorch when we just set nn.MSELoss with batch dimension.

Output:
Check this:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        print("Hi ma")
        print(x)
        x = F.relu(x)
        return x

n = Net()
r = n(torch.tensor(-1))
print(r)
r = n.forward(torch.tensor(1)) #not planned to call directly
print(r)

out:

Hi ma
tensor(-1)
tensor(0)
Hi ma
tensor(1)
tensor(1)

Thing to remember is that forward should not be called directly.
The PyTorch made this Module object n callable. They implemented callable like:

 def __call__(self, *input, **kwargs):
    for hook in self._forward_pre_hooks.values():
        hook(self, input)
    if torch._C._get_tracing_state():
        result = self._slow_forward(*input, **kwargs)
    else:
        result = self.forward(*input, **kwargs)
    for hook in self._forward_hooks.values():
        hook_result = hook(self, input, result)
        if hook_result is not None:
            raise RuntimeError(
                "forward hooks should never return any values, but '{}'"
                "didn't return None".format(hook))
    if len(self._backward_hooks) > 0:
        var = result
        while not isinstance(var, torch.Tensor):
            if isinstance(var, dict):
                var = next((v for v in var.values() if isinstance(v, torch.Tensor)))
            else:
                var = var[0]
        grad_fn = var.grad_fn
        if grad_fn is not None:
            for hook in self._backward_hooks.values():
                wrapper = functools.partial(hook, self)
                functools.update_wrapper(wrapper, hook)
                grad_fn.register_hook(wrapper)
    return result

And just n() will call forward automatically.

In general, __init__ defines the module structure and forward() defines operations on a single batch.

That operation may repeat if needed for some structure elements or you may call functions on tensors directly like we did x = F.relu(x).

You got this great, everything in PyTorch will do in batches (mini-batches), since the PyTorch is optimized to work this way.

This means when you read the image, you will not read the single one, but one bs batches of images.


Instruction: What does layout = torch.strided mean?

Input:
As I was going through pytorch documentation I came across a term layout = torch.strided in many of the functions. Can anyone help me in understanding where is it used and how. The description says it's the the desired layout of returned Tensor. What does layout mean and how many types of layout are there ?

torch.rand(*sizes, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

Output:
strides is number of steps (or jumps) that is needed to go from one element to next element, in a given dimension. In computer memory, the data is stored linearly in a contiguous block of memory. What we view is just a (re)presentation.
Let's take an example tensor for understanding this:
# a 2D tensor
In [62]: tensor = torch.arange(1, 16).reshape(3, 5)

In [63]: tensor
Out[63]:
tensor([[ 1,  2,  3,  4,  5],
        [ 6,  7,  8,  9, 10],
        [11, 12, 13, 14, 15]])

With this tensor in place, the strides are:
# get the strides
In [64]: tensor.stride()
Out[64]: (5, 1)

What this resultant tuple (5, 1) says is:

to traverse along the 0th dimension/axis (Y-axis), let's say we want to jump from 1 to 6, we should take 5 steps (or jumps)
to traverse along the 1st dimension/axis (X-axis), let's say we want to jump from 7 to 8, we should take 1 step (or jump)

The order (or index) of 5 & 1 in the tuple represents the dimension/axis. You can also pass the dimension, for which you want the stride, as an argument:
# get stride for axis 0
In [65]: tensor.stride(0)
Out[65]: 5

# get stride for axis 1
In [66]: tensor.stride(1)
Out[66]: 1

With that understanding, we might have to ask why is this extra parameter needed when we create the tensors? The answer to that is for efficiency reasons. (How can we store/read/access the elements in the (sparse) tensor most efficiently?).
With sparse tensors (a tensor where most of the elements are just zeroes), so we don't want to store these values. we only store the non-zero values and their indices. With a desired shape, the rest of the values can then be filled with zeroes, yielding the desired sparse tensor.

For further reading on this, the following articles might be of help:

numpy.ndarray.strides
torch.layout
torch.sparse

P.S: I guess there's a typo in the torch.layout documentation which says

Strides are a list of integers ...

The composite data type returned by tensor.stride() is a tuple, not a list.


Instruction: Meaning of parameters in torch.nn.conv2d

Input:
In the fastai cutting edge deep learning for coders course lecture 7.

 self.conv1 = nn.Conv2d(3,10,kernel_size = 5,stride=1,padding=2)

Does 10 there mean the number of filters or the number activations the filter will give?

Output:
Here is what you may find

torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True, padding_mode='zeros')

Parameters

in_channels (int) – Number of channels in the input image
out_channels (int) – Number of channels produced by the convolution
kernel_size (int or tuple) – Size of the convolving kernel
stride (int or tuple, optional) – Stride of the convolution. (Default: 1)
padding (int or tuple, optional) – Zero-padding added to both sides of the input (Default: 0)
padding_mode (string, optional) – zeros
dilation (int or tuple, optional) – Spacing between kernel elements. (Default: 1)
groups (int, optional) – Number of blocked connections from input to output channels. (Default: 1)
bias (bool, optional) – If True, adds a learnable bias to the output. (Default: True)

And this URL has helpful visualization of the process.
So the in_channels in the beginning is 3 for images with 3 channels (colored images).
For images black and white it should be 1.
Some satellite images should have 4.
The out_channels is the number of filters and you can set this arbitrary.
Let's create an example to "prove" that.
import torch
import torch.nn as nn
c = nn.Conv2d(1,3, stride = 1, kernel_size=(4,5))
print(c.weight.shape)
print(c.weight)

Out
torch.Size([3, 1, 4, 5])
Parameter containing:
tensor([[[[ 0.1571,  0.0723,  0.0900,  0.1573,  0.0537],
          [-0.1213,  0.0579,  0.0009, -0.1750,  0.1616],
          [-0.0427,  0.1968,  0.1861, -0.1787, -0.2035],
          [-0.0796,  0.1741, -0.2231,  0.2020, -0.1762]]],

        [[[ 0.1811,  0.0660,  0.1653,  0.0605,  0.0417],
          [ 0.1885, -0.0440, -0.1638,  0.1429, -0.0606],
          [-0.1395, -0.1202,  0.0498,  0.0432, -0.1132],
          [-0.2073,  0.1480, -0.1296, -0.1661, -0.0633]]],

        [[[ 0.0435, -0.2017,  0.0676, -0.0711, -0.1972],
          [ 0.0968, -0.1157,  0.1012,  0.0863, -0.1844],
          [-0.2080, -0.1355, -0.1842, -0.0017, -0.2123],
          [-0.1495, -0.2196,  0.1811,  0.1672, -0.1817]]]], requires_grad=True)

If we would alter the number of out_channels,
c = nn.Conv2d(1,5, stride = 1, kernel_size=(4,5))
print(c.weight.shape) # torch.Size([5, 1, 4, 5])

We will get 5 filters each filter 4x5 as this is our kernel size.
If we would set 2 channels, (some images may have 2 channels only)
c = nn.Conv2d(2,5, stride = 1, kernel_size=(4,5))
print(c.weight.shape) # torch.Size([5, 2, 4, 5])

our filter will have 2 channels.
I think they have terms from this book and since they haven't called it filters, they haven't used that term.
So you are right; filters are what conv layer is learning and the number of filters is the number of out channels. They are set randomly at the start.
Number of activations is calculated based on bs and image dimension:
bs=16
x = torch.randn(bs, 3, 28, 28)
c = nn.Conv2d(3,10,kernel_size=5,stride=1,padding=2)
out = c(x)
print(out.nelement()) #125440 number of activations


Instruction: Is hidden and output the same for a GRU unit in Pytorch?

Input:
I do understand conceptually what an LSTM or GRU should (thanks to this question What's the difference between "hidden" and "output" in PyTorch LSTM?) BUT when I inspect the output of the GRU h_n and output are NOT the same while they should be...

(Pdb) rnn_output
tensor([[[ 0.2663,  0.3429, -0.0415,  ...,  0.1275,  0.0719,  0.1011],
         [-0.1272,  0.3096, -0.0403,  ...,  0.0589, -0.0556, -0.3039],
         [ 0.1064,  0.2810, -0.1858,  ...,  0.3308,  0.1150, -0.3348],
         ...,
         [-0.0929,  0.2826, -0.0554,  ...,  0.0176, -0.1552, -0.0427],
         [-0.0849,  0.3395, -0.0477,  ...,  0.0172, -0.1429,  0.0153],
         [-0.0212,  0.1257, -0.2670,  ..., -0.0432,  0.2122, -0.1797]]],
       grad_fn=<StackBackward>)
(Pdb) hidden
tensor([[[ 0.1700,  0.2388, -0.4159,  ..., -0.1949,  0.0692, -0.0630],
         [ 0.1304,  0.0426, -0.2874,  ...,  0.0882,  0.1394, -0.1899],
         [-0.0071,  0.1512, -0.1558,  ..., -0.1578,  0.1990, -0.2468],
         ...,
         [ 0.0856,  0.0962, -0.0985,  ...,  0.0081,  0.0906, -0.1234],
         [ 0.1773,  0.2808, -0.0300,  ..., -0.0415, -0.0650, -0.0010],
         [ 0.2207,  0.3573, -0.2493,  ..., -0.2371,  0.1349, -0.2982]],

        [[ 0.2663,  0.3429, -0.0415,  ...,  0.1275,  0.0719,  0.1011],
         [-0.1272,  0.3096, -0.0403,  ...,  0.0589, -0.0556, -0.3039],
         [ 0.1064,  0.2810, -0.1858,  ...,  0.3308,  0.1150, -0.3348],
         ...,
         [-0.0929,  0.2826, -0.0554,  ...,  0.0176, -0.1552, -0.0427],
         [-0.0849,  0.3395, -0.0477,  ...,  0.0172, -0.1429,  0.0153],
         [-0.0212,  0.1257, -0.2670,  ..., -0.0432,  0.2122, -0.1797]]],
       grad_fn=<StackBackward>)

they are some transpose of each other...why?

Output:
They are not really the same. Consider that we have the following Unidirectional GRU model:
import torch.nn as nn
import torch

gru = nn.GRU(input_size = 8, hidden_size = 50, num_layers = 3, batch_first = True)

Please make sure you observe the input shape carefully.
inp = torch.randn(1024, 112, 8)
out, hn = gru(inp)

Definitely,
torch.equal(out, hn)
False

One of the most efficient ways that helped me to understand the output vs. hidden states was to view the hn as hn.view(num_layers, num_directions, batch, hidden_size) where num_directions = 2 for bidirectional recurrent networks (and 1 other wise, i.e., our case). Thus,
hn_conceptual_view = hn.view(3, 1, 1024, 50)

As the doc states (Note the italics and bolds):

h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len (i.e., for the last timestep)

In our case, this contains the hidden vector for the timestep t = 112, where the:

output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features h_t from the last layer of the GRU, for each t. If a torch.nn.utils.rnn.PackedSequence has been given as the input, the output will also be a packed sequence. For the unpacked case, the directions can be separated using output.view(seq_len, batch, num_directions, hidden_size), with forward and backward being direction 0 and 1 respectively.

So, consequently, one can do:
torch.equal(out[:, -1], hn_conceptual_view[-1, 0, :, :])
True

Explanation: I compare the last sequence from all batches in out[:, -1] to the last layer hidden vectors from hn[-1, 0, :, :]

For Bidirectional GRU (requires reading the unidirectional first):
gru = nn.GRU(input_size = 8, hidden_size = 50, num_layers = 3, batch_first = True bidirectional = True)
inp = torch.randn(1024, 112, 8)
out, hn = gru(inp)

View is changed to (since we have two directions):
hn_conceptual_view = hn.view(3, 2, 1024, 50)

If you try the exact code:
torch.equal(out[:, -1], hn_conceptual_view[-1, 0, :, :])
False

Explanation: This is because we are even comparing wrong shapes;
out[:, 0].shape
torch.Size([1024, 100])
hn_conceptual_view[-1, 0, :, :].shape
torch.Size([1024, 50])

Remember that for bidirectional networks, hidden states get concatenated at each time step where the first hidden_state size (i.e., out[:, 0, :50]) are the the hidden states for the forward network, and the other hidden_state size are for the backward (i.e., out[:, 0, 50:]). The correct comparison for the forward network is then:
torch.equal(out[:, -1, :50], hn_conceptual_view[-1, 0, :, :])
True

If you want the hidden states for the backward network, and since a backward network processes the sequence from time step n ... 1. You compare the first timestep of the sequence but the last hidden_state size and changing the hn_conceptual_view direction to 1:
torch.equal(out[:, -1, :50], hn_conceptual_view[-1, 1, :, :])
True

In a nutshell, generally speaking:
Unidirectional:
rnn_module = nn.RECURRENT_MODULE(num_layers = X, hidden_state = H, batch_first = True)
inp = torch.rand(B, S, E)
output, hn = rnn_module(inp)
hn_conceptual_view = hn.view(X, 1, B, H)

Where RECURRENT_MODULE is either GRU or LSTM (at the time of writing this post), B is the batch size, S sequence length, and E embedding size.
torch.equal(output[:, S, :], hn_conceptual_view[-1, 0, :, :])
True

Again we used S since the rnn_module is forward (i.e., unidirectional) and the last timestep is stored at the sequence length S.
Bidirectional:
rnn_module = nn.RECURRENT_MODULE(num_layers = X, hidden_state = H, batch_first = True, bidirectional = True)
inp = torch.rand(B, S, E)
output, hn = rnn_module(inp)
hn_conceptual_view = hn.view(X, 2, B, H)

Comparison
torch.equal(output[:, S, :H], hn_conceptual_view[-1, 0, :, :])
True

Above is the forward network comparison, we used :H because the forward stores its hidden vector in the first H elements for each timestep.
For the backward network:
torch.equal(output[:, 0, H:], hn_conceptual_view[-1, 1, :, :])
True

We changed the direction in hn_conceptual_view to 1 to get hidden vectors for the backward network.

For all examples we used hn_conceptual_view[-1, ...] because we are only interested in the last layer.


Instruction: How to compute the Surface Dice-Sørensen Coefficient in pytorch?

Input:
I would like to compute the Surface Dice-Sørensen Coefficient from this  paper (page 19)in python3/pytorch.

I have to point out, that I do not try to implement the simple standard volumetric Dice-Sørensen Coefficient! This one would look as follows in my implementation:

import torch

def volumetric_DSC(M1, M2):
    M1 = M1.view(-1)
    M2 = M2.view(-1)
    dividend = 2 * (M1 * M2).sum()
    divisor = (M1 * M1).sum() + (M2 * M2).sum()
    return dividend / divisor

if __name__ == "__main__":
    m1 = torch.empty(5, 5, 5).uniform_(0, 1)
    m1 = torch.bernoulli(m1)
    m2 = torch.empty(5, 5, 5).uniform_(0, 1)
    m2 = torch.bernoulli(m2)
    loss = volumetric_DSC(m1, m2)
    print("loss = {0}".format(loss))

How can I extend this code to a Surface Dice-Sørensen Coefficient loss?

Output: A surface dice implementation was provided here as part of this study. You can use it as an evaluation metric but not as a loss function as it contains non-differentiable ops. You will need to provide a "tolerance" distance i.e. a surface dice of 0.9 means that 90% of surfaces lie within the tolerance (which is better calculated from the data itself, such as the inter-observer variation of the task you are solving)


Instruction: How to get the filename of a sample from a DataLoader?

Input:
I need to write a file with the result of the data test of a Convolutional Neural Network that I trained. The data include speech data collection. The file format needs to be "file name, prediction", but I am having a hard time to extract the file name. I load the data like this:

import torchvision
from torchvision import transforms
from torch.utils.data import DataLoader

TEST_DATA_PATH = ...

trans = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])

test_dataset = torchvision.datasets.MNIST(
    root=TEST_DATA_PATH,
    train=False,
    transform=trans,
    download=True
)

test_loader = DataLoader(dataset=test_dataset, batch_size=1, shuffle=False)

and I am trying to write to the file as follows:

f = open("test_y", "w")
with torch.no_grad():
    for i, (images, labels) in enumerate(test_loader, 0):
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        file = os.listdir(TEST_DATA_PATH + "/all")[i]
        format = file + ", " + str(predicted.item()) + '\n'
        f.write(format)
f.close()

The problem with os.listdir(TESTH_DATA_PATH + "/all")[i] is that it is not synchronized with the loaded files order of test_loader. What can I do?

Output:
Well, it depends on how your Dataset is implemented. For instance, in the torchvision.datasets.MNIST(...) case, you cannot retrieve the filename simply because there is no such thing as the filename of a single sample (MNIST samples are loaded in a different way).

As you did not show your Dataset implementation, I'll tell you how this could be done with the torchvision.datasets.ImageFolder(...) (or any torchvision.datasets.DatasetFolder(...)):

f = open("test_y", "w")
with torch.no_grad():
    for i, (images, labels) in enumerate(test_loader, 0):
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        sample_fname, _ = test_loader.dataset.samples[i]
        f.write("{}, {}\n".format(sample_fname, predicted.item()))
f.close()

You can see that the path of the file is retrieved during the __getitem__(self, index), especifically here.

If you implemented your own Dataset (and perhaps would like to support shuffle and batch_size > 1), then I would return the sample_fname on the __getitem__(...) call and do something like this:

for i, (images, labels, sample_fname) in enumerate(test_loader, 0):
    # [...]

This way you wouldn't need to care about shuffle. And if the batch_size is greater than 1, you would need to change the content of the loop for something more generic, e.g.:

f = open("test_y", "w")
for i, (images, labels, samples_fname) in enumerate(test_loader, 0):
    outputs = model(images)
    pred = torch.max(outputs, 1)[1]
    f.write("\n".join([
        ", ".join(x)
        for x in zip(map(str, pred.cpu().tolist()), samples_fname)
    ]) + "\n")
f.close()


Instruction: Is there an function in PyTorch for converting convolutions to fully-connected networks form?

Input:
I'm trying to convert a convolution layer to a fully-connected layer.

For example, there is an example of 3×3 input and 2x2 kernel:

which is equivalent to a vector-matrix multiplication，

Is there a function in PyTorch to get the matrix B?

Output:
I can only partially answer your question:

In your example above, you write the kernel as matrix and the input as a vector. If you are fine with writing the input as a matrix, you can use torch.nn.Unfold which explicitly calculates a convolution in the documentation:

# Convolution is equivalent with Unfold + Matrix Multiplication + Fold (or view to output shape)
inp = torch.randn(1, 3, 10, 12)
w = torch.randn(2, 3, 4, 5)
inp_unf = torch.nn.functional.unfold(inp, (4, 5))
out_unf = inp_unf.transpose(1, 2).matmul(w.view(w.size(0), -1).t()).transpose(1, 2)
out = out_unf.view(1, 2, 7, 8)
(torch.nn.functional.conv2d(inp, w) - out).abs().max()
# tensor(1.9073e-06)

If, however, you need to calculate the matrix for the kernel (the smaller matrix) you can use this function, which is based on Warren Weckessers answer:

def toeplitz_1_ch(kernel, input_size):
    # shapes
    k_h, k_w = kernel.shape
    i_h, i_w = input_size
    o_h, o_w = i_h-k_h+1, i_w-k_w+1

    # construct 1d conv toeplitz matrices for each row of the kernel
    toeplitz = []
    for r in range(k_h):
        toeplitz.append(linalg.toeplitz(c=(kernel[r,0], *np.zeros(i_w-k_w)), r=(*kernel[r], *np.zeros(i_w-k_w))) )

    # construct toeplitz matrix of toeplitz matrices (just for padding=0)
    h_blocks, w_blocks = o_h, i_h
    h_block, w_block = toeplitz[0].shape

    W_conv = np.zeros((h_blocks, h_block, w_blocks, w_block))

    for i, B in enumerate(toeplitz):
        for j in range(o_h):
            W_conv[j, :, i+j, :] = B

    W_conv.shape = (h_blocks*h_block, w_blocks*w_block)

    return W_conv

which is not in pytorch but in numpy. This is for padding = 0 but can easily be adjusted by changing h_blocks and w_blocks and W_conv[i+j, :, j, :].

Update:  Multiple output channels are just multiple of these matrices, as each output has its own kernel. Multiple input channels also have their own kernels - and their own matrices - over which you average after the convolution. This can be implemented as follows:

def conv2d_toeplitz(kernel, input):
    """Compute 2d convolution over multiple channels via toeplitz matrix
    Args:
        kernel: shape=(n_out, n_in, H_k, W_k)
        input: shape=(n_in, H_i, W_i)"""

    kernel_size = kernel.shape
    input_size = input.shape
    output_size = (kernel_size[0], input_size[1] - (kernel_size[1]-1), input_size[2] - (kernel_size[2]-1))
    output = np.zeros(output_size)

    for i,ks in enumerate(kernel):  # loop over output channel
        for j,k in enumerate(ks):  # loop over input channel
            T_k = toeplitz_1_ch(k, input_size[1:])
            output[i] += T_k.dot(input[j].flatten()).reshape(output_size[1:])  # sum over input channels

    return output

To check the correctness:

k = np.random.randn(4*3*3*3).reshape((4,3,3,3))
i = np.random.randn(3,7,9)

out = conv2d_toeplitz(k, i)

# check correctness of convolution via toeplitz matrix
print(np.sum((out - F.conv2d(torch.tensor(i).view(1,3,7,9), torch.tensor(k)).numpy())**2))

>>> 1.0063523219807736e-28

Update 2:

It is also possible to do this without looping in one matrix:

def toeplitz_mult_ch(kernel, input_size):
    """Compute toeplitz matrix for 2d conv with multiple in and out channels.
    Args:
        kernel: shape=(n_out, n_in, H_k, W_k)
        input_size: (n_in, H_i, W_i)"""

    kernel_size = kernel.shape
    output_size = (kernel_size[0], input_size[1] - (kernel_size[1]-1), input_size[2] - (kernel_size[2]-1))
     T = np.zeros((output_size[0], int(np.prod(output_size[1:])), input_size[0], int(np.prod(input_size[1:]))))

    for i,ks in enumerate(kernel):  # loop over output channel
        for j,k in enumerate(ks):  # loop over input channel
            T_k = toeplitz_1_ch(k, input_size[1:])
            T[i, :, j, :] = T_k

    T.shape = (np.prod(output_size), np.prod(input_size))

    return T

The input has to be flattened and the output reshaped after multiplication.
Checking for correctness (using the same i and k as above):

T = toeplitz_mult_ch(k, i.shape)
out = T.dot(i.flatten()).reshape((1,4,5,7))

# check correctness of convolution via toeplitz matrix
print(np.sum((out - F.conv2d(torch.tensor(i).view(1,3,7,9), torch.tensor(k)).numpy())**2))
>>> 1.5486060830252635e-28


Instruction: Is there a way to reduce the time needed to find the indices of the top k largest elements from a 1D vector?

Input:
I am trying to obtain the indices of the of the largest n and smallest n elements of an array in Pytorch, and then concatenate these elements into a return array, but the time needed to do so is extremely unreasonable as the size of the array increases exponentially.

I have tried the code that is included, as well as sorting the vector so I can just take the first and last n elements, but time is not reduced and I lose the indices of the vector, which is what I need.

def draw(n, distr):
    return np.concatenate((np.array(distr.topk(k=int(n), largest=True).indices),
                           np.array(distr.topk(k=int(n), largest=False).indices)),
                          axis=0)

For a series of 1 dimensional arrays, varying in size from length 10 to length 2359296; furthermore, the variable n is an integer valued at 1/10th of the length of the array. My computer can compute the indices of the largest and smallest n elements of all the arrays in about 0.5 seconds. I would prefer to minimize this time as much as possible, preferably to less than 0.2 seconds

Output:
You should use torch.topk(x,k).

k=2
x = torch.arange(0,10).resize_((2,5))
print(x)
print("...")
res, ind = torch.topk(x,k)
print(res)

tensor([[0, 1, 2, 3, 4],
        [5, 6, 7, 8, 9]])
...
tensor([[4, 3],
        [9, 8]])


Instruction: How to implement Pytorch 1D crosscorrelation for long signals in fourier domain?

Input:
I have a series of signals length n = 36,000 which I need to perform crosscorrelation on. Currently, my cpu implementation in numpy is a little slow. I've heard Pytorch can greatly speed up tensor operations, and provides a way to perform computations in parallel on the GPU. I'd like to explore this option, but I'm not quite sure how to accomplish this using the framework.

Because of the length of these signals, I'd prefer to perform the crosscorrelation operation in the frequency domain.

Normally using numpy I'd perform the operation like so:

import numpy as np

signal_length=36000

# make the signals
signal_1 = np.random.uniform(-1,1, signal_length)
signal_2 = np.random.uniform(-1,1, signal_length)

# output target length of crosscorrelation
x_cor_sig_length = signal_length*2 - 1

# get optimized array length for fft computation
fast_length = np.fftpack.next_fast_len(x_cor_sig_length)

# move data into the frequency domain. axis=-1 to perform
# along last dimension
fft_1 = np.fft.rfft(src_data, fast_length, axis=-1)
fft_2 = np.fft.rfft(src_data, fast_length, axis=-1)

# take the complex conjugate of one of the spectrums. Which one you choose depends on domain specific conventions
fft_1 = np.conj(fft_1)

fft_multiplied = fft_1 * fft_2

# back to time domain.
prelim_correlation = np.fft.irfft(result, x_corr_sig_length, axis=-1)

# shift the signal to make it look like a proper crosscorrelation,
# and transform the output to be purely real

final_result = np.real(np.fft.fftshift(prelim_correlation),axes=-1)).astype(np.float64)

Looking at the Pytorch documentation, there doesn't seem to be an equivalent for numpy.conj(). I'm also not sure if/how I need to implement a fftshift after the irfft operation.

So how would you go about writing a 1D crosscorrelation in Pytorch using the fourier method?

Output:
A few things to be considered.
Python interpreter is very slow, what those vectorization libraries do is to move the workload to a native implementation. In order to make any difference you need to be able to give perform many operations in one python instruction. Evaluating things on GPU follows the same principle, while GPU has more compute power it is slower to copy data to/from GPU.
I adapted your example to process multiple signals simulataneously.
import numpy as np
def numpy_xcorr(BATCH=1, signal_length=36000):
    # make the signals
    signal_1 = np.random.uniform(-1,1, (BATCH, signal_length))
    signal_2 = np.random.uniform(-1,1, (BATCH, signal_length))

    # output target length of crosscorrelation
    x_cor_sig_length = signal_length*2 - 1

    # get optimized array length for fft computation
    fast_length = next_fast_len(x_cor_sig_length)

    # move data into the frequency domain. axis=-1 to perform
    # along last dimension
    fft_1 = np.fft.rfft(signal_1, fast_length, axis=-1)
    fft_2 = np.fft.rfft(signal_2 + 0.1 * signal_1, fast_length, axis=-1)

    # take the complex conjugate of one of the spectrums.
    fft_1 = np.conj(fft_1)

    fft_multiplied = fft_1 * fft_2

    # back to time domain.
    prelim_correlation = np.fft.irfft(fft_multiplied, fast_length, axis=-1)

    # shift the signal to make it look like a proper crosscorrelation,
    # and transform the output to be purely real

    final_result = np.fft.fftshift(np.real(prelim_correlation), axes=-1)
    return final_result, np.sum(final_result)

Since torch 1.7 we have the torch.fft module that provides an interface similar to numpy.fft, the fftshift is missing but the same result can be obtained with torch.roll. Another point is that numpy uses by default 64-bit precision and torch will use 32-bit precision.
The fast length consists in choosing smooth numbers (those having that are factorized in to small prime numbers, and I suppose you are familiar with this subject).
def next_fast_len(n, factors=[2, 3, 5, 7]):
    '''
      Returns the minimum integer not smaller than n that can
      be written as a product (possibly with repettitions) of
      the given factors.
    '''
    best = float('inf')
    stack = [1]
    while len(stack):
        a = stack.pop()
        if a >= n:
            if a < best:
                best = a;
                if best == n:
                    break; # no reason to keep searching
        else:
            for p in factors:
                b = a * p;
                if b < best:
                    stack.append(b)
    return best;

Then the torch implementation goes
import torch;
import torch.fft
def torch_xcorr(BATCH=1, signal_length=36000, device='cpu', factors=[2,3,5], dtype=torch.float):
    signal_length=36000
    # torch.rand is random in the range (0, 1)
    signal_1 = 1 - 2*torch.rand((BATCH, signal_length), device=device, dtype=dtype)
    signal_2 = 1 - 2*torch.rand((BATCH, signal_length), device=device, dtype=dtype)

    # just make the cross correlation more interesting
    signal_2 += 0.1 * signal_1;

    # output target length of crosscorrelation
    x_cor_sig_length = signal_length*2 - 1

    # get optimized array length for fft computation
    fast_length = next_fast_len(x_cor_sig_length, [2, 3])

    # the last signal_ndim axes (1,2 or 3) will be transformed
    fft_1 = torch.fft.rfft(signal_1, fast_length, dim=-1)
    fft_2 = torch.fft.rfft(signal_2, fast_length, dim=-1)

    # take the complex conjugate of one of the spectrums. Which one you choose depends on domain specific conventions

    fft_multiplied = torch.conj(fft_1) * fft_2

    # back to time domain.
    prelim_correlation = torch.fft.irfft(fft_multiplied, dim=-1)

    # shift the signal to make it look like a proper crosscorrelation,
    # and transform the output to be purely real

    final_result = torch.roll(prelim_correlation, (fast_length//2,))
    return final_result, torch.sum(final_result);

And here a code to test the results
import time
funcs = {'numpy-f64': lambda b: numpy_xcorr(b, factors=[2,3,5], dtype=np.float64),
         'numpy-f32': lambda b: numpy_xcorr(b, factors=[2,3,5], dtype=np.float32),
         'torch-cpu-f64': lambda b: torch_xcorr(b, device='cpu', factors=[2,3,5], dtype=torch.float64),
         'torch-cpu': lambda b: torch_xcorr(b, device='cpu', factors=[2,3,5], dtype=torch.float32),
         'torch-gpu-f64': lambda b: torch_xcorr(b, device='cuda', factors=[2,3,5], dtype=torch.float64),
         'torch-gpu': lambda b: torch_xcorr(b, device='cuda', factors=[2,3,5], dtype=torch.float32),
         }
times ={}
for batch in [1, 10, 100]:
    times[batch] = {}
    for l, f in funcs.items():
        t0 = time.time()
        t1, t2 = f(batch)
        tf = time.time()
        del t1
        del t2
        times[batch][l] = 1000 * (tf - t0) / batch;

I obtained the following results

And what surprised myself is the result when the numbers are not so smooth e.g. using 17-smooth length the torch implementation is so much better that I used logarithmic scale here (with batch size 100 the torch gpu was 10000 times faster than numpy with batch size 1).

Remember that these functions are generating the data at the GPU in general we want to copy the final results to the CPU, if we consider the time spent copying the final result to CPU I observed times up to 10x higher than the cross correlation computation (random data generation + three FFTs).


Instruction: Why aren't torch.functional.sigmoid and torch.nn.functional.relu deprecated like torch.nn.functional.tanh?

Input:
Now when torch.autograd.Variable is merged with torch.tensor and obsolete, why did they deprecate some functions in torch.nn.functional but not others?  Namely, tanhis deprecated but not sigmoid or relu.

>>> torch.__version__
'1.1.0'
>>> u
tensor(2., grad_fn=<MeanBackward0>)
>>> torch.nn.functional.tanh(u)
C:\Users\mlearning\AppData\Local\Continuum\anaconda3\lib\site-packages\torch\nn\functional.py:1374: UserWarning: nn.functional.tanh is deprecated. Use torch.tanh instead.
  warnings.warn("nn.functional.tanh is deprecated. Use torch.tanh instead.")
tensor(0.9640, grad_fn=<TanhBackward>)
>>> torch.nn.functional.sigmoid(u)
tensor(0.8808, grad_fn=<SigmoidBackward>)
>>> torch.nn.functional.relu(u)
tensor(2., grad_fn=<ReluBackward0>)

Is there any difference between torch.nn.functional.relu and torch.relu, or I can use them interchangeably?

Output:
You can check this thread where one of the few main PyTorch designers (actually a creator) set the directive.

You can also check the reasoning behind. Also, you may propose the same for the other 2 functions.

The other should deprecate as well.


Instruction: BERT Multi-class text classification in Google Colab

Input:
I'm working on a data set of social media comments(including youtube links) as input features and the Myers-Biggs Personality Profile as the target label:

    type    posts
0   INFJ    'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   ENTP    'I'm finding the lack of me in these posts ver...
2   INTP    'Good one _____ https://www.youtube.com/wat...
3   INTJ    'Dear INTP, I enjoyed our conversation the o...
4   ENTJ    'You're fired.|||That's another silly misconce...

but from what I've found, BERT wants DataFrame's to be in this format:

a   label   posts
0   a   8   'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   a   3   'I'm finding the lack of me in these posts ver...
2   a   11  'Good one _____ https://www.youtube.com/wat...
3   a   10  'Dear INTP, I enjoyed our conversation the o...
4   a   2   'You're fired.|||That's another silly misconce...

The resulting output must be a prediction on a test set of comments split into four columns, one for each Personality Profile where, for example, 'Mind' = 1 is the label for Extrovert. Basically splitting a type like INFJ into 'Mind','Energy','Nature','Tactics', like such:

    type    post    Mind    Energy  Nature  Tactics
0   INFJ    'url-web    0   1   0   1
1   INFJ    url-web 0   1   0   1
2   INFJ    enfp and intj moments url-web sportscenter n... 0   1   0   1
3   INFJ    What has been the most life-changing experienc...   0   1   0   1
4   INFJ    url-web url-web On repeat for most of today.    0   1   0   1

I've installed pytorch-pretrained-bert using:

!pip install pytorch-pretrained-bert

I've imported the models and tried to tokenize the 'posts' column using:

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

tokenized_train = tokenizer.tokenize(train)

but receive this error:

TypeError: ord() expected a character, but string of length 5 found

I tried this based off the pytorch-pretrained-bert GitHub Repo and a Youtube vidoe.

I am a Data Science intern with no Deep Learning experience at all. I simply want to experiment with the BERT model in the most simplest way to predict the multi-class classified output so I can compare the results to simpler text-classification models we are currently working on. I am working in Google Colab and the resulting output should be a .csv file.

I understand that this is a complicated model and all documentation and examples surrounding the model are complex (fine tuning layers etc.) but any help for a simple implementation(if in fact there is such a thing) for a beginner Data Scientist with minimal Software Engineering experience, would be much appreciated.

Output:
I recommend you start with a simple BERT classification task, for example following this excellent tutorial: https://mccormickml.com/2019/07/22/BERT-fine-tuning/

Then you can get into multi-label by following: https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d

Only then I would recommend you try your task on your own dataset.


Instruction: How does the groups parameter in torch.nn.conv* influence the convolution process?

Input:
I want to convolve a multichannel tensor with the same single channel weight.
I could repeat the weight along the channel dimension, but I thought there might be an other way.

I thought the groups parameter might do the job. However I don't understand the documentation.
That's why I want to ask how the groups parameter influences the convolution process ?

Output:
Just minor tips since I never used it.

Group parameter multiplies the number of kernels you would normally have.
So if you set group=2, expect 2 times more kernels.

The definition of conv2d in PyTorch states group is 1 by default.

If you increase the group you get the depth-wise convolution, where each input channel is getting specific kernels per se.

The constraint is both in and out channels should be dividable by group number.

I think in Tensorfolow you can read the documentation of SeparableConv2D since this is what is equivalent when group>1.


Instruction: TypeError: mul() argument 'other' (position 1) must be Tensor, not ReLU

Input:
I would like to add a torch.nn.ReLU() layer between fc1 and fc2 layer.

Original code:

model:

# ...
self.fc1 = nn.Linear(4096, 256)
self.fc2 = nn.Linear(256, 4096)
# ...

def forward(...):
    # ...
    x = x.view(-1, 4096)
    x = self.fc1(x))
    if a7 is not None:
        x = x * a7.squeeze()
    # ...

I tried

# ...
x = x.view(-1, 4096)
x = nn.ReLU(self.fc1(x)))
if a7 is not None:
    x = x * a7.squeeze()
# ...

and this error pops out.

Output:
My answer assumes __init__ was a typo and it should be forward. Let me know if that is not the case and I'll delete it.

import torch
from torch import nn

class SimpleModel(nn.Module):
  def __init__(self, with_relu=False):
    super(SimpleModel, self).__init__()
    self.fc1 = nn.Sequential(nn.Linear(3, 10), nn.ReLU(inplace=True)) if with_relu else nn.Linear(3, 10)
    self.fc2 = nn.Linear(10, 3)

  def forward(self, x):
    x = self.fc1(x)
    print(torch.min(x))  # just to show you ReLU is working...
    return self.fc2(x)

# Model without ReLU
net_without_relu = SimpleModel(with_relu=False)
print(net_without_relu)

# Model with ReLU
net_with_relu = SimpleModel(with_relu=True)
print(net_with_relu)

# random input data
x = torch.randn((5, 3))
print(x)

# we expect it to print something < 0
output1 = net_without_relu(x)

# we expect it to print 0.
output2 = net_with_relu(x)

You can check the code below running on the Colab: https://colab.research.google.com/drive/1W3Dh4_KPd3iABx5FSzZm3tilm6tnJh0v

To use as you tried:

x = nn.ReLU(self.fc1(x)))

you can use the functional API:

from torch.nn import functional as F

# ...
x = F.relu(self.fc1(x)))


Instruction: Numerical equivalence of PyTorch backpropagation

Input:
After i 'v written the simple neural network with numpy, i wanted to compare it numerically with PyTorch impementation. Running alone, seems my neural network implementation converges, so it seems to have no errors.
Also i v checked forward pass matches to PyTorch, so basic setup is correct.

But something different happens while backward pass, because the weights after one backpropagation are different.

I dont want to post full code here because its linked over several .py files, and most of the code is irrelevant to the question. I just want to know does PyTorch "basic" gradient descent or something different.

I m viewing the most simle example about full-connected weights of the last layer, cause if it is different, further will be also different:

self.weight +=  self.learning_rate * hidden_layer.T.dot(output_delta )

where

output_delta  = self.expected -  self.output

self.expected  are expected value,
self.output is forward pass result

No activation or further  stuff here.

The torch past is:

optimizer = torch.optim.SGD(nn.parameters() , lr = 1.0)

criterion = torch.nn.MSELoss(reduction='sum')

output = nn.forward(x_train)

loss = criterion(output, y_train)

loss.backward()

optimizer.step()

optimizer.zero_grad()

So it is possible that with SGD optimizer and MSELoss it uses some different delta or backpropagation function, not the basic one mentioned above? If its so i d like to know how to numerically check my numpy solution with pytorch.

Output:
I just want to know does PyTorch "basic" gradient descent or something different.

If you set torch.optim.SGD, this means stochastic gradient descent.
You have different implementations on GD, but the one that is used in PyTorch is applied to mini-batches.
There are GD implementations that will optimize parameters after the full epoch. As you may guess they are very "slow", this may be great for supercomputers to test. There are GD implementations that work for every sample, as you may guess their imperfectness is "huge" gradient fluctuations.
These are all relative terms, so I am using ""
Note you are using too big learning rates like lr = 1.0, which means you haven't normalized your data at first, but this is a skill you may scalp over time.

So it is possible that with SGD optimizer and MSELoss it uses some different delta or backpropagation function, not the basic one mentioned above?

It uses what you told.
Here is a the example in PyTorch and in Python to show detection of gradients works as expected (used in back propagation) :
x = torch.tensor([5.], requires_grad=True);
print(x) # tensor([5.], requires_grad=True)

y = 3*x**2
y.backward()
print(x.grad) # tensor([30.])

How would you get this value 30 in plain python?
def y(x):
    return 3*x**2

x=5
e=0.01 #etha
g=(y(x+e)-y(x))/e
print(g) # 30.0299

As we expect we got ~30, it would be even better with smaller etha.


Instruction: Is there a way to create a graph comparing hyper-parameters vs model accuracy with TRAINS python package?

Input:
I would like to run multiple experiments, then report model accuracy per experiment.

I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the toy problem, to have it integrated with the actual code base.

As I understand the TRAINS python package, with the "two lines of code" all my hyper-parameters are already logged (Command line argparse in my case).

What do I need to do in order to report a final scalar and then be able to sort through all the different training experiments (w/ hyper-parameters) in order to find the best one.

What I'd like to get, is a graph/s where on the X-axis I have hyper-parameter values and on the Y-axis I have the validation accuracy.

Output:
I assume you are referring to: https://pypi.org/project/trains/ (https://github.com/allegroai/trains),
which I'm one of the maintainers.

You can manually create a plot with a single point X-axis for the hyper-parameter value, and Y-Axis for the accuracy.

number_layers = 10
accuracy = 0.95
Task.current_task().get_logger().report_scatter2d(
    "performance", "accuracy", iteration=0,
    mode='markers', scatter=[(number_layers, accuracy)])

Assuming your hyper-parameter is "number_layers" with current value 10, and the accuracy for the trained model is 0.95.

Then when you compare the experiments you get something like that:


Instruction: Pytorch model.train() and a separte train() function written in a tutorial

Input:
I am new to PyTorch and I was wondering if you could explain to me some of the key differences between the default model.train() in PyTorch and the train() function here.

The other train() function is on the official PyTorch tutorial on text classification and was confused as to whether the model weights are being stored at the end of training.

https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

learning_rate = 0.005

criterion = nn.NLLLoss()

def train(category_tensor, line_tensor):
    hidden = rnn.initHidden()
    rnn.zero_grad()
    for i in range(line_tensor.size()[0]):
        output, hidden = rnn(line_tensor[i], hidden)
    loss = criterion(output, category_tensor)
    loss.backward()
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        p.data.add_(-learning_rate, p.grad.data)
    return output, loss.item()

This is the function. This function is then called multiple times in this form:

n_iters = 100000
print_every = 5000
plot_every = 1000
record_every = 500

# Keep track of losses for plotting
current_loss = 0
all_losses = []
predictions = []
true_vals = []

def timeSince(since):
    now = time.time()
    s = now - since
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)

start = time.time()

for iter in range(1, n_iters + 1):
    category, line, category_tensor, line_tensor = randomTrainingExample()
    output, loss = train(category_tensor, line_tensor)
    current_loss += loss

    if iter % print_every == 0:
        guess, guess_i = categoryFromOutput(output)
        correct = 'O' if guess == category else 'X (%s)' % category
        print('%d %d%% (%s) %.4f %s / %s %s' % (iter, iter / n_iters * 100, timeSince(start), loss, line, guess, correct))

    if iter % plot_every == 0:
        all_losses.append(current_loss / plot_every)
        current_loss = 0

    if iter % record_every == 0:
        guess, guess_i = categoryFromOutput(output)
        predictions.append(guess)
        true_vals.append(category)

To me it seems that the model weights are not being saved or updated but rather being overridden at each iteration when written like this. Is this correct? Or does the model appear to be training correctly?

Additionally, if I were to use the default function model.train(), what is the main advantage and does model.train() perform more or less the same functionality as the train() function above?

Output:
As per the source code here, model.train() sets the module in training mode. So,  it basically tells your model that you are training the model. This has any effect only on certain modules like dropout, batchnorm etc. which behave differently in  training/evaluation mode. In case of model.train() the model knows it has to learn the layers.

You can call either model.eval() or model.train(mode=False) to tell the model that it has nothing new to learn and the model is used for testing purpose.

model.train() just sets the mode. It doesn't actually train the model.

train() that you are using above is actually training the model, i.e., calculating gradient and doing backpropagation to learn the weights.

Learn more about model.train() from official pytorch discussion forum, here.


Instruction: How to fix 'The kernel appears to have died. It will restart automatically" caused by pytorch

Input:
I have a strange problem with Pytorch. When I use torch functions with tensors like tensor.reshape  or torch.transpose, I don't have any problems; even when I created networks, it's ok. However, when I want to train network my jupyter crashed.

I find where error is but I don't know why it is there and how to fix it.

I installed pytorch using conda. I have Ubuntu 18.04.
I don't have cuda.

Output:
If you are on Ubuntu you may not install PyTorch just via conda.

It can be:

Conda
Pip
LibTorch
From Source

So you have multiple options.

Go to this page and select Cuda to NONE, LINUX, stable 1.1, CONDA.

conda install pytorch-cpu torchvision-cpu -c pytorch

If you have problems still, you may try also install PIP way.

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
pip3 install https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl

Hopefully some of these ways will work.


Instruction: AttributeError: module 'torch' has no attribute 'hub'

Input:
import torch
model = torch.hub.list('pytorch/vision')

My pytorch version is 1.0.0, but I can't load the hub, why is this?

Output:
You will need torch >= 1.1.0 to use torch.hub attribute.

Alternatively, try by downloading this hub.py file and then try below code:

import hub
model = hub.list('pytorch/vision', force_reload=False)

Arguments:

github: Required, a string with format repo_owner/repo_name[:tag_name] with an optional tag/branch. The default branch is master if not specified.
Example: pytorch/vision[:hub]

force_reload: Optional, whether to discard the existing cache and force a fresh download.
Default is False.


Instruction: Incorrect results obtained on running a model in LibTorch that was trained and exported from PyTorch

Input:
I am trying to export a trained model along with weights for inference in C++ using LibTorch. However, the output tensor results do not match.

The shape of the output tensor is the same.

model = FCN()
state_dict = torch.load('/content/gdrive/My Drive/model/trained_model.pth')
model.load_state_dict(state_dict)
example = torch.randn(1, 3, 768, 1024)
traced_script_module = torch.jit.trace(model, example)
traced_script_module.save('/content/gdrive/My Drive/model/mymodel.pt')

However some warnings are generated which I think maybe causing the incorrect results to be generated.

  /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:137:
  TracerWarning: Converting a tensor to a Python index might cause the
  trace to be incorrect. We can't record the data flow of Python values,
  so this value will be treated as a constant in the future. This means
  that the trace might not generalize to other inputs!
  /usr/local/lib/python3.6/dist-packages/torch/tensor.py:435:
  RuntimeWarning: Iterating over a tensor might cause the trace to be
  incorrect. Passing a tensor of different shape won't change the number
  of iterations executed (and might lead to errors or silently give
  incorrect results).'incorrect results).', category=RuntimeWarning)

Following is the LibTorch code to generate the output tensor

at::Tensor predict(std::shared_ptr<torch::jit::script::Module> model, at::Tensor &image_tensor) {
    std::vector<torch::jit::IValue> inputs;
    inputs.push_back(image_tensor);

    at::Tensor result = model->forward(inputs).toTensor();

    return result;
}

Has anyone tried using a trained PyTorch model in LibTorch?

Output:
Just ran into the same issue, and found a solution:
add

model.eval()

before

traced_script_module = torch.jit.trace(model, example)

and the model gives the same result in c++ as in python


Instruction: Transfer learning in Pytorch using fasterrcnn_resnet50_fpn

Input:
I am looking for Object Detection for custom dataset in PyTorch.

Tutorial here provides a snippet to use pre-trained model for custom object classification

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                   num_epochs=25)

I tried to use similar method for Object Detection using faster rcnn model.

# load a model pre-trained pre-trained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
for param in model.parameters():
    param.requires_grad = False
# replace the classifier with a new one, that has
# num_classes which is user-defined
num_classes = 1  # 1 class (person) + background
print(model)
model = model.to(device)
criterion = nn.CrossEntropyLoss()
# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
model = train_model(model, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

PyTorch throws these errors . Is this approach correct in the first place ?

Epoch 0/24
----------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-69-527ca4db8e5d> in <module>()
----> 1 model = train_model(model, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

2 frames
/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     43         """
     44         if self.training and targets is None:
---> 45             raise ValueError("In training mode, targets should be passed")
     46         original_image_sizes = [img.shape[-2:] for img in images]
     47         images, targets = self.transform(images, targets)

ValueError: In training mode, targets should be passed

Is there a way modify this example for custom object detection ?
https://www.learnopencv.com/faster-r-cnn-object-detection-with-pytorch/

Output:
If you want to detect a person and background, you will have to set num_classes to 2.
To train your custom detection model, you need to pass, images (each pixel between 0 and 1) and targets. You can follow this Kaggle tutorial : https://www.kaggle.com/abhishek/training-fast-rcnn-using-torchvision


Instruction: Adding custom labels to pytorch dataloader/dataset does not work for custom dataset

Input:
I am working on the cactus image competition on Kaggle and I am trying to use the PyTorch dataloader for my CNN. However, I am running into an issue where I cannot set the labels for the training set. The training set images are given in a folder and the labels are in a csv file. This is my code.

 train = torchvision.datasets.ImageFolder(root='../input/train',
 transform=transform)

 train.targets = torch.from_numpy(df['has_cactus'].values)

 train_loader = torch.utils.data.DataLoader(train, batch_size=64, shuffle=True, num_workers=2)

 for i, data in enumerate(train_loader, 0):
     print(data[1])

This code outputs batch tensors of all zeros, which is clearly incorrect as the great majority of the labels(if you were to look at the dataframe) are ones. I believe that this is a problem with assigning the labels to "train.targets". If "train.targets" is printed prior to the assignment of the other labels, it returns a tensor of all zeros which is consistent with the incorrect results that I am getting. How do I fix this issue?

Output:
I typically inherit the builtin DataSet class as follows:

from torch.utils.data import DataLoader
class DataSet:

    def __init__(self, root):
        """Init function should not do any heavy lifting, but
            must initialize how many items are available in this data set.
        """

        self.ROOT = root
        self.images = read_images(root + "/images")
        self.labels = read_labels(root + "/labels")

    def __len__(self):
        """return number of points in our dataset"""

        return len(self.images)

    def __getitem__(self, idx):
        """ Here we have to return the item requested by `idx`
            The PyTorch DataLoader class will use this method to make an iterable for
            our training or validation loop.
        """

        img = images[idx]
        label = labels[idx]

        return img, label

And now, you can create an instance of this class as,

ds = Dataset('../input/train')

Now, you can instantiate the DataLoader:

dl = DataLoader(ds, batch_size=TRAIN_BATCH_SIZE, shuffle=False, num_workers=4, drop_last=True)

This will create batches of your data that you can access as:

for image, label in dl:
    print(label)


Instruction: How to obtain input data from ONNX model?

Input:
I have exported my PyTorch model to ONNX. Now, is there a way for me to obtain the input layer from that ONNX model?

Exporting PyTorch model to ONNX

import torch.onnx
checkpoint = torch.load("./saved_pytorch_model.pth")
model.load_state_dict(checkpoint['state_dict'])
input = torch.tensor(df_X.values).float()
torch.onnx.export(model, input, "onnx_model.onnx")

Loading ONNX model

onnx_model = onnx.load('onnx_model.onnx')

I want to be able to somehow obtain the input layer from onnx_model. Is this possible?

Output: The ONNX model is a protobuf structure, as defined here (https://github.com/onnx/onnx/blob/master/onnx/onnx.in.proto). You can work with it using the standard protobuf methods generated for python (see: https://developers.google.com/protocol-buffers/docs/reference/python-generated). I don't understand what exactly you want to extract. But you can iterate through the nodes that make up the graph (model.graph.node). The first node in the graph may or may not correspond to what you might consider the first layer (it depends on how the translation was done). You can also get the inputs of the model (model.graph.input).


Instruction: Pytorch Build from Source gives Error make: *** No rule to make target 'install'. Stop

Input:
I am following this guide to build Pytorch from scratch on a Raspberry Pi3B. For some reason, there is an error:

Building wheel torch-1.2.0a0+f13fadd
-- Building version 1.2.0a0+f13fadd
cmake --build . --target install --config Release -- -j 4
make: *** No rule to make target 'install'.  Stop.

when I call python3 setup.py build. I am running Python version 3.5 and I am unsure why this seems to be failing.

Output:
Recently I encountered this error so after some research, in
https://stackoverflow.com/a/46987554/12164529
someone mentioned something about cache.
Therefore I guess that's because of some CMake cache behavior, so I run this command:

sudo USE_ROCM=1 USE_LMDB=1 USE_OPENCV=1 MAX_JOBS=15 python setup.py clean

And the error went away.

ps. This is my first answer on stackoverflow, and I'm not sure if this is a good one, but I hope it helps people find here.


Instruction: ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable MASTER_ADDR expected, but not set

Input:
I am not able to initialize the group process in PyTorch for BERT model
I had tried to initialize using following code:

import torch
import datetime

torch.distributed.init_process_group(
    backend='nccl',
    init_method='env://',
    timeout=datetime.timedelta(0, 1800),
    world_size=0,
    rank=0,
    store=None,
    group_name=''
)

and tried to access the get_world_size() function:

num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()

full code:

train_examples = None
    num_train_optimization_steps = None
    if do_train:
        train_examples = processor.get_train_examples(data_dir)
        num_train_optimization_steps = int(
            len(train_examples) / train_batch_size / gradient_accumulation_steps) * num_train_epochs
        if local_rank != -1:
            import datetime
            torch.distributed.init_process_group(backend='nccl',init_method='env://', timeout=datetime.timedelta(0, 1800), world_size=0, rank=0, store=None, group_name='')
            num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
            print(num_train_optimization_steps)

Output:
I solve the problem by referring https://github.com/NVIDIA/apex/issues/99.
Specifically run

python -m torch.distributed.launch xxx.py


Instruction: Difference between "detach()" and "with torch.nograd()" in PyTorch?

Input:
I know about two ways to exclude elements of a computation from the gradient calculation backward

Method 1: using with torch.no_grad()

with torch.no_grad():
    y = reward + gamma * torch.max(net.forward(x))
loss = criterion(net.forward(torch.from_numpy(o)), y)
loss.backward();

Method 2: using .detach()

y = reward + gamma * torch.max(net.forward(x))
loss = criterion(net.forward(torch.from_numpy(o)), y.detach())
loss.backward();

Is there a difference between these two? Are there benefits/downsides to either?

Output:
tensor.detach() creates a tensor that shares storage with tensor that does not require grad. It detaches the output from the computational graph. So no gradient will be backpropagated along this variable.

The wrapper with torch.no_grad() temporarily set all the requires_grad flag to false. torch.no_grad says that no operation should build the graph.

The difference is that one refers to only a given variable on which it is called. The other affects all operations taking place within the with statement. Also, torch.no_grad will use less memory because it knows from the beginning that no gradients are needed so it doesn’t need to keep intermediary results.

Learn more about the differences between these along with examples from here.


Instruction: Why doesn't setting `random.seed(42)` give me identical results in pytorch?

Input:
I am setting random seed for both random and numpy.random at the beginning of my main file:

import random
import numpy as np
np.random.seed(42)
random.seed(42)
import torch

Nevertheless, when I create a Net() object with randomly initialized parameters, it gives a completely different result every time:

net=neuralnet.Net()
print ("initialized params: ", net.fc1.weight)

Note that neuralnet.Net() is in a different file, and is a class that extends torch.nn.Module. it is torch.nn.Module that is randomly initializing net.fc1.weight, not my own code.

How is it possible that when I create a Net() object with randomly initialized parameters, it gives a completely different result every time?

Output:
try:

import torch
torch.manual_seed(0)

For further information:
https://pytorch.org/docs/stable/notes/randomness.html


Instruction: Compute a convolution with weights that where computed by another function

Input:
I would like to perform a conv2d but a special one because I compute the weights of a conv2d layer in a function for each sample. Can I simply assign the weights of a given layer for each sample?

I am implementing the Spatially variant convolution of this paper : https://arxiv.org/abs/1804.00389

Thanks

Output: It is actually very simple : torch.nn.functional.conv2d !