Instruction: About pytorch learning rate scheduler

Input:
here is my code

optimizer = optim.SGD(net.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)

for i in range(15):
    lr = scheduler.get_lr()[0]
    lr1 = optimizer.param_groups[0]["lr"]
    print(i, lr, lr1)
    scheduler.step()

And here is the result

0 0.1 0.1
1 0.1 0.1
2 0.1 0.1
3 0.1 0.1
4 0.1 0.1
5 0.025 0.05
6 0.05 0.05
7 0.05 0.05
8 0.05 0.05
9 0.05 0.05
10 0.0125 0.025
11 0.025 0.025
12 0.025 0.025
13 0.025 0.025
14 0.025 0.025

We can see that the when scheduler.step() is applied, the learning rate first decreases 0.25 times, then bounces back to 0.5 times. Is it the problem of scheduler.get_lr() lr or the problem of scheduler.step()

About the envirioment

python=3.6.9
pytorch=1.1.0

In addition, I can't find this problem when pytorch=0.4.1 is used.

Output:
Yes, the "problem" is in the use of get_lr(). To get the current LR, what you need is actually the get_last_lr().

If you take a look at the implementation:

def get_lr(self):
    if not self._get_lr_called_within_step:
        warnings.warn("To get the last learning rate computed by the scheduler, "
                      "please use `get_last_lr()`.", UserWarning)

    if (self.last_epoch == 0) or (self.last_epoch % self.step_size != 0):
        return [group['lr'] for group in self.optimizer.param_groups]
    return [group['lr'] * self.gamma
            for group in self.optimizer.param_groups]

When it is in the step=5, it does not satisfy the conditions (because step_size=5), and it will return the lr * gamma. The awkward thing is that you should be getting a warning when you call get_lr() out the the step() function (as you can see in the implementation above) and apparently you didn't. The warning was added only 3 months ago, so you won't have it on v1.1.0.

For the sake of completeness, what the step() method does is that it adds 1 to the last_epoch and updates the LR by calling the get_lr() function (see here):

self.last_epoch += 1
values = self.get_lr()


Instruction: Method for feeding multi-class image data-set where folders name can be used as labels in Pytorch?

Input:
I want to feed the multiclass image data-set in Pytorch, in the main folder of data-set I have 15 more folders with different names, I want to use folders names as the labels.
For example, one folder name is Aeroplanes and contain the images (1245 images) other folder name is Cars and contains images of the Cars (997), likewise, each folder has different numbers of images.
Now I want to load them to train my model and to test it, but I don't have separate folders for the training and testing. I want to use folder names as labels and also want to split the data-set into training and testing as an equal ratio.
Your guidance, in this case, will be appreciated. Thanks

Output:
To split your dataset into train and test datasets you could use random_split function:

import torch
from torchvision import datasets, transforms
from torch.utils import data
import numpy as np

dataset = datasets.ImageFolder('path_to_dataset', transform=transforms.ToTensor())

lengths = [int(np.ceil(0.5*len(dataset))),
           int(np.floor(0.5*len(dataset)))]
train_set, test_set = data.random_split(dataset, lengths)

train_dataloader = data.DataLoader(train_set, batch_size=...)
test_dataloader = data.DataLoader(test_set, batch_size=...)

In case you want to perform separate transformations on your train and test datasets look here: How to use different data augmentation for Subsets in PyTorch


Instruction: Pytorch mask tensor with boolean numpy array

Input:
I have a 84x84 pytorch tensor named target. I need to mask it with an 84x84 boolean numpy array which consists of True and False.

When I do target = target[mask], I get the error TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

Surprisingly, I get this error only when running on a GPU. When running on a CPU, everything works fine. How can I fix this?

Output:
I think there is some confusion with the types. But this works.

import torch
tensor = torch.randn(84,84)
c = torch.randn(tensor.size()).bool()
c[1, 2:5] = False
x = tensor[c].size()

For testing I created a tensor with random values. Afterwards 3 elements are set to False. In the last step I look get the size 7053 resulting from 84^2 - 3.

Hope that helps somehow.


Instruction: AttributeError: 'MpoImageFile' object has no attribute 'shape'

Input:
images, labels = next(iter(self.loader))
grid = torchvision.utils.make_grid(images)

  images, labels = next(iter(self.loader))

triggers the error.

I have a custom dataset class where I load each image (RGB) from an url :

  image = Image.open(urllib.request.urlopen(URL))

and I apply some albumentations transforms.

The code works when I read an image for which I have a path using cv2.
However, it doesn't work when I read an image from the url.
Note that I verified that the urls aren't broken.

Here's the traceback:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/content/transform_dataset.py in __getitem__(self, idx)
     49     labels = torch.from_numpy(item[2:].values.astype("float32"))
     50     #print("self.root,item,self.image_transform,self.transform,self.size", self.root,item,self.image_transform,self.transform,self.size)
---> 51     image = load_image(self.root,item.ID,item.URL,self.image_transform)
     52     return image, labels
     53

/content/transform_dataset.py in load_image(root, ID, URL, image_transform)
     81     print(image.shape)
     82     image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
---> 83   image = image_transform(image=image)["image"]
     84   return image

/usr/local/lib/python3.6/dist-packages/albumentations/core/composition.py in __call__(self, **data)
    169                                               convert_keypoints_to_albumentations, data)
    170
--> 171             data = t(**data)
    172
    173             if dual_start_end is not None and idx == dual_start_end[1]:

/usr/local/lib/python3.6/dist-packages/albumentations/core/transforms_interface.py in __call__(self, **kwargs)
     26         if (random.random() < self.p) or self.always_apply:
     27             params = self.get_params()
---> 28             params = self.update_params(params, **kwargs)
     29             if self.targets_as_params:
     30                 targets_as_params = {k: kwargs[k] for k in self.targets_as_params}

/usr/local/lib/python3.6/dist-packages/albumentations/core/transforms_interface.py in update_params(self, params, **kwargs)
     66         if hasattr(self, 'interpolation'):
     67             params['interpolation'] = self.interpolation
---> 68         params.update({'cols': kwargs['image'].shape[1], 'rows': kwargs['image'].shape[0]})
     69         return params
     70

AttributeError: 'MpoImageFile' object has no attribute 'shape'

Output:
In order to work with albumentations, you must pass a numpy array to the transforms not a PIL image. So:

image = Image.open(urllib.request.urlopen(URL))
image = np.array(image)


Instruction: Understanding Pytorch Grid Sample

Input:
I have a input tensor of size [1,32,296,400]

and I have a pixel set of [1, 56000, 400, 2]

After applying grid_sample with mode=‘bilinear’ I have [1, 32, 56000, 400]

Can I know what exactly happened here? I know that grid_sample is suppose to effective transform pixels to a new location in a differentiable manner, but these dimensions don't make it clear what is happening.

Output:
Please look at the documentation of grid_sample.
Your input tensor has a shape of 1x32x296x400, that is, you have a single example in the batch with 32 channels and spatial dimensions of 296x400 pixels.
Additionally, you have a "grid" of size 1x56000x400x2 which PyTorch interprets as new locations for a grid of spatial dimensions of 56000x400 where each new location has the x,y coordinates from which to sample the new grid value. Hence the "grid" information is of shape 1x56000x400x2.
The output is, as expected, a 2D tensor of shape 1x32x56000x400: batch and channel dimensions are unchanged but the spatial coordinates are in accordance with the "grid" information provided to grid_sample.


Instruction: Trying to mask tensor with another tensor of same dimension getting "index 1 is out of bounds for dimension 0 with size 1"

Input:
    attn_weights = F.softmax(self.attn(torch.cat((input, hidden_cat), 2)), dim=2)
    attn_weights[mask] = float('-inf')
    attn_applied = torch.bmm(attn_weights.transpose(0,1),encoder_outputs.transpose(0,1)).transpose(0,1)
    attn_output = torch.cat((input, attn_applied), 2)

So I'm trying to set all the indexes in mask that are equal 1 to negative infinity, but that line

attn_weights[mask] = float('-inf')

keeps throwing this exception "index 1 is out of bounds for dimension 0 with size 1" not really sure what's going on attn_weights and mask both have the same dimension, which is 1 x 2048 x 40.

Output: turns out the dtype for mask tensor had to be torch.uint8 or torch.bool I had it torch.long


Instruction: Pytorch: can we use nn.Module layers directly in forward() function?

Input:
Generally,
In the constructor, we declare all the layers we want to use.
In the forward function, we define how the model is going to be run, from input to output.

My question is what if calling those predefined/built-in nn.Modules directly in forward() function? Is this Keras function API style legal for Pytorch? If not, why?

Update: TestModel constructed in this way did run successfully, without an alarm. But the training loss will descend slowly compared with the conventional way.

import torch.nn as nn
from cnn import CNN

class TestModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.num_embeddings = 2020
        self.embedding_dim = 51

    def forward(self, input):
        x = nn.Embedding(self.num_embeddings, self.embedding_dim)(input)
        # CNN is a customized class and nn.Module subclassed
        # we will ignore the arguments for its instantiation
        x = CNN(...)(x)
        x = nn.ReLu()(x)
        x = nn.Dropout(p=0.2)(x)
        return output = x

Output:
You need to think of the scope of the trainable parameters.

If you define, say, a conv layer in the forward function of your model, then the scope of this  "layer" and its trainable parameters is local to the function and will be discarded after every call to the forward method. You cannot update and train weights that are constantly being discarded after every forward pass.
However, when the conv layer is a member of your model its scope extends beyond the forward method and the trainable parameters persists as long as the model object exists. This way you can update and train the model and its weights.


Instruction: Basic Pytorch tensor multiplication and addition

Input:
I just realize I lack some very basic pytorch tensor math. How do I do the following with a pytorch tensor?

lab_rs = (lab_rs * [100, 255, 255] - [0, 128, 128])

This works well in numpy. It's an image with shape (3, 512, 1024) and I want to multiply and subtract values from each color channel individually

The error I get trying the same with a tensor is:

TypeError: only integer tensors of a single element can be converted to an index

Output:
You need to make sure all your operands can be broadcast to the same dimensions:

lab_rs = lab_rs * torch.tensor([[[100]], [[255]], [[255.]]]) - torch.tensor([[[0]], [[128]], [[128.]]])


Instruction: PyTorch - How to use k-fold cross validation when the data is loaded through ImageFolder?

Input: My data, which is images, is stored on the filesystem, and it is fed into my convolutional neural network through the ImageFolder data loader of PyTorch. Therefore, the training, validation, and test data is manually splitted into different folders on the filesystem. So, how can I apply k-fold cross validation when using ImageFolder?

Output: You can merge the fixed train/val/test folds you currently have using data.ConcatDataset into a single Dataset. Then you can use data.Subset to randomly split the single dataset into different folds over and over.


Instruction: How to get quick documentation working with PyCharm and Pytorch

Input:
I'm running PyCharm on Windows 10, and installed PyTorch following the getting started guide. Where I used Chocolatey and Anaconda to set up everything.

I can run the PyTorch tutorials from inside the PyCharm IDE without any problems. So I feel like I have a proper set up, but there aren't any intellisense documentations for any of the PyTorch APIs.

For example;

import torch

x = torch.randn(128, 20)

If I mouse over randn and press CTRL+Q then PyCharm shows me a popup of the function definition without any documentation.

I'm expecting to see Python comments from the API documentation for that function:

https://pytorch.org/docs/stable/torch.html?highlight=randn#torch.randn

I'm a new beginner with Pytorch and Python, but this is something that I often have access to from inside the IDE with many other languages and libraries. So I feel like this should be possible to get working, but I can't seem to find any instructions on how to fix this.

Output:
I was able to get it working by doing the following:

  PyStorm 2019.3

Open the settings for external documentation:

  File / Settings / Tools / External Documentation

Add the following URL patterns:

Module Name: torch.nn.functional
        URL: https://pytorch.org/docs/stable/nn.functional.html#{element.qname}

Module Name: torch
        URL: https://pytorch.org/docs/stable/{module.basename}.html#{element.qname}

Seems to work for most APIs but you have to trigger the quick documentation tool window. This won't show docs if you CTRL+CLICK something.


Instruction: What does torchvision.transforms.Resize(size, interpolation=2) actually do?

Input: Does it add to the image if too small or crop if too big or just stretch the image to the desired size?

Output:
When you set interpolation=2, then you are using Bilinear interpolation, ti can be either used for upsampling or down sampling. In the case of upsampling you are doing something like

There are several types of upsampling and down-sampling, but bilinear one uses a combination of the neighbouring pixels to cimpute the new pixel.

Look to this links for more informations: link1; link2


Instruction: RoBERTa classification RuntimeError: shape '[-1, 9]' is invalid for input of size 8

Input:
    m = MultiLabelBinarizer()

    X = pd.read_csv('data/data.csv', sep=None, engine='python')
    X = X.dropna()

    Y_train = m.fit_transform(X['labels'])
    Y_train2 = [list(i) for i in Y_train]

    data = pd.DataFrame({'text': pd.Series(X[text_col]), 'labels': Y_train2})
    data = data.dropna()

    train_df, eval_df = train_test_split(data, test_size=0.2)

    numLabels = len(pd.unique(X['labels])) # count of the labels

    model = MultiLabelClassificationModel('roberta', 'roberta-base', num_labels=numLabels, use_cuda=False)

    model.train_model(pd.DataFrame(train_df))

My data-structure for the label column is: [[0,1,0,0,0,1,0,0], [0,1,1,0,0,1,0,0], [0,0,0,0,0,0,0,1]....] for every row there is one label-list like [0,1,0,0,0,1,0,0] in the label-column

And for the texts there is one text (newspaper article) per row.

(got it from that source: https://github.com/ThilinaRajapakse/simpletransformers#minimal-start-for-multilabel-classification)

the model can be trained if i train it with only 4 entries. But when i want to train it with the whole dataset it gives me that: RuntimeError: shape '[-1, 9]' is invalid for input of size 8:

 File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 121, in train_model
    return super().train_model(train_df, multi_label=multi_label, eval_df=eval_df, output_dir=output_dir, show_running_loss=show_running_loss, args=args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py", line 208, in train_model
    global_step, tr_loss = self.train(train_dataset, output_dir, multi_label=multi_label, show_running_loss=show_running_loss, eval_df=eval_df, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simpletransformers/classification/classification_model.py", line 306, in train
    outputs = model(**inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/simpletransformers/custom_models/models.py", line 117, in forward
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1, self.num_labels))
RuntimeError: shape '[-1, 9]' is invalid for input of size 8

I have no idea where that size 8 comes from and what to do now since it works with very few entries.
Can anyone help?

Output:
[0,1,0,0,0,1,0,0] - it is 8 size, but your model expect size 9.
it means that, your numLabels = 9. If you have 9 classes, then the  label-list in the label-colum should be like this: [0,1,0,0,0,1,0,0,0].
But I think you just need to pass the num_labels as 8


Instruction: 'int' object has no attribute 'size'"

Input:
F.nll_loss: I am getting

  AttributeError: 'int' object has no attribute 'size'

when I try to run this code. I also get a snippet of the module code.

  raise ValueError('Expected 2 or more dimensions (got {})'.format(dim))
      if input.size(0) != target.size(0):
        raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
        format(input.size(0), target.size(0)))

import torch
from torchvision import transforms, datasets
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pylab as plt

train_dataset = datasets.MNIST(root = '', train =True, download = True,
                                transform =transforms.Compose([transforms.ToTensor()]))

test_dataset = datasets.MNIST(root ='', download =True, train =False,
                               transform =transforms.Compose([transforms.ToTensor()]))
batch_size = 10
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size, shuffle =True)

test_dataset = torch.utils.data.DataLoader(test_dataset, batch_size, shuffle =True)
class Net(nn.Module):

  def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(28*28, 64)
    self.fc2 = nn.Linear(64,64)
    self.fc3 = nn.Linear(64,64)
    self.fc4 = nn.Linear(64,10)

  def forward(self, x):
      x = F.relu(self.fc1(x))
      x = F.relu(self.fc2(x))
      x = F.relu(self.fc2(x))
      x = self.fc4(x)
      return F.log_softmax(x, dim=1)

x=torch.rand((28,28))
x=x.view(-1,28*28)
net =Net()
out=net(x)
out
import torch.optim as optim

optimizer =optim.Adam(net.parameters(), lr=0.001)

EPOCHS = 3
for epoch in range(EPOCHS):
  for data in train_dataset:
      x, y = data
      net.zero_grad()
      x=x.view(-1, 28*28)
      output = net(x)
      loss = F.nll_loss(output, y)
      loss.backward()
      optimizer.step()
  print(loss)

Output:
Just change the for loop from:

for data in train_dataset:

to

for data in train_loader:


Instruction: torch find indices of matching rows in 2 2D tensors

Input:
I have two 2D tensors, in different length, both are different subsets of the same original 2d tensor and I would like to find all the matching "rows"
e.g

A = [[1,2,3],[4,5,6],[7,8,9],[3,3,3]
B = [[1,2,3],[7,8,9],[4,4,4]]
torch.2dintersect(A,B) -> [0,2] (the indecies of A that B also have)

I've only see numpy solutions, that use dtype as dicts, and does not work for pytorch.

Here is how I do it in numpy

arr1 = edge_index_dense.numpy().view(np.int32)
arr2 = edge_index2_dense.numpy().view(np.int32)
arr1_view = arr1.view([('', arr1.dtype)] * arr1.shape[1])
arr2_view = arr2.view([('', arr2.dtype)] * arr2.shape[1])
intersected = np.intersect1d(arr1_view, arr2_view, return_indices=True)

Output:
This answer was posted before the OP updated the question with other restrictions that changed the problem quite a bit.

TL;DR You can do something like this:

torch.where((A == B).all(dim=1))[0]

First, assuming you have:

import torch
A = torch.Tensor([[1,2,3],[4,5,6],[7,8,9]])
B = torch.Tensor([[1,2,3],[4,4,4],[7,8,9]])

We can check that A == B returns:

>>> A == B
tensor([[ True,  True,  True],
        [ True, False, False],
        [ True,  True,  True]])

So, what we want is: the rows in which they are all True. For that, we can use the .all() operation and specify the dimension of interest, in our case 1:

>>> (A == B).all(dim=1)
tensor([ True, False,  True])

What you actually want to know is where the Trues are. For that, we can get the first output of the torch.where() function:

>>> torch.where((A == B).all(dim=1))[0]
tensor([0, 2])


Instruction: How to calculate geometric mean in a differentiable way?

Input: How to calculate goemetric mean along a dimension using Pytorch? Some numbers can be negative. The function must be differentiable.

Output:
A known (reasonably) numerically-stable version of the geometric mean is:
import torch

def gmean(input_x, dim):
    log_x = torch.log(input_x)
    return torch.exp(torch.mean(log_x, dim=dim))

x = torch.Tensor([2.0] * 1000).requires_grad_(True)
print(gmean(x, dim=0))
# tensor(2.0000, grad_fn=<ExpBackward>)

This kind of implementation can be found, for example, in SciPy (see here), which is a quite stable lib.

The implementation above does not handle zeros and negative numbers. Some will argue that the geometric mean with negative numbers is not well-defined, at least when not all of them are negative.


Instruction: PyTorch - Custom ReLU squared Implementation

Input:
I work on a project and I want to implement the ReLU squared activation function (max{0,x^2}). Is it ok to call it like:

# example code
    def forward(self, x):
        s = torch.relu(x**2)
        return s

Or should I implement the activation function on my own? In the second case could you please provide me an example on how to do so?

Thanks a lot!

Output:
It doesn't make much sense to compute max(0, x**2) because x**2 >= 0 no matter what.
You probably want to compute max(0, x) ** 2 instead:
s = torch.pow(torch.relu(x), 2)


Instruction: How to use pytorch to construct multi-task DNN, e.g., for more than 100 tasks?

Input:
Below is the example code to use pytorch to construct DNN for two regression tasks. The forward function returns two outputs (x1, x2). How about the network for lots of regression/classification tasks? e.g., 100 or 1000 outputs. It definitely not a good idea to hardcode all the outputs (e.g., x1, x2, ..., x100). Is there an simple method to do that? Thank you.

import torch
from torch import nn
import torch.nn.functional as F

class mynet(nn.Module):
    def __init__(self):
        super(mynet, self).__init__()
        self.lin1 = nn.Linear(5, 10)
        self.lin2 = nn.Linear(10, 3)
        self.lin3 = nn.Linear(10, 4)

    def forward(self, x):
        x = self.lin1(x)
        x1 = self.lin2(x)
        x2 = self.lin3(x)
        return x1, x2

if __name__ == '__main__':
    x = torch.randn(1000, 5)
    y1 = torch.randn(1000, 3)
    y2 = torch.randn(1000,  4)
    model = mynet()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=1e-4)
    for epoch in range(100):
        model.train()
        optimizer.zero_grad()
        out1, out2 = model(x)
        loss = 0.2 * F.mse_loss(out1, y1) + 0.8 * F.mse_loss(out2, y2)
        loss.backward()
        optimizer.step()

Output:
You can (and should) use nn containers such as nn.ModuleList or nn.ModuleDict to manage arbitrary number of sub-modules.

For example (using nn.ModuleList):

class MultiHeadNetwork(nn.Module):
    def __init__(self, list_with_number_of_outputs_of_each_head):
        super(MultiHeadNetwork, self).__init__()
        self.backbone = ...  # build the basic "backbone" on top of which all other heads come
        # all other "heads"
        self.heads = nn.ModuleList([])
        for nout in list_with_number_of_outputs_of_each_head:
            self.heads.append(nn.Sequential(
              nn.Linear(10, nout * 2),
              nn.ReLU(inplace=True),
              nn.Linear(nout * 2, nout)))

    def forward(self, x):
        common_features = self.backbone(x)  # compute the shared features
        outputs = []
        for head in self.heads:
            outputs.append(head(common_features))
        return outputs

Note that in this example each head is more complex than a single nn.Linear layer.
The number of different "heads" (and number of outputs) is determined by the length of the argument list_with_number_of_outputs_of_each_head.

Important notice: it is crucial to use nn containers, rather than simple pythonic lists/dictionary to store all sub modules. Otherwise pytorch will have difficulty managing all sub modules.
See, e.g., this answer, this question and this one.


Instruction: TensorFlow / PyTorch: Gradient for loss which is measured externally

Input:
I am relatively new to Machine Learning and Python.

I have a system, which consists of a NN whose output is fed into an unknown nonlinear function F, e.g. some hardware. The idea is to train the NN to be an inverse F^(-1) of that unknown nonlinear function F. This means that a loss L is calculated at the output of F. However, backpropagation cannot be used in a straightforward manner for calculating the gradients and updating the NN weights because the gradient of F is not known either.

Is there any way how to use a loss function L, which is not directly connected to the NN, for the calculation of the gradients in TensorFlow or PyTorch? Or to take a loss that was obtained with any other software (Matlab, C, etc.) use it for backpropagation?

As far as I know, Keras keras.backend.gradients only allows to calculate gradients with respect to connected weights, otherwise the gradient is either zero or NoneType.
I read about the stop_gradient() function in TensorFlow. But I am not sure whether this is what I am looking for. It allows to not compute the gradient with respect to some variables during backpropagation. But I think the operation F is not interpreted as a variable anyway.
Can I define any arbitrary loss function (including a hardware measurement) and use it for backpropagation in TensorFlow or is it required to be connected to the graph as well?

Please, let me know if my question is not specific enough.

Output:
AFAIK, all modern deep learning packages (pytorch, tensorflow, keras etc.) are relaying on gradient descent (and its many variants) to train networks.
As the name suggests, you cannot do gradient descent without gradients.

However, you might circumvent the "non differentiability" of your "given" function F by looking at the problem from a slightly different perspective:
You are trying to learn a model M that "counters" the effect of F. So you have access to F (but not its gradients) and a set of representative inputs X={x_0, x_1, ... x_n}.
For each example x_i you can compute y_i = F(x_i) and your end goal is to have a model M that given y_i will output x_i.
Therefore, you can treat y_i as your model's input and compute a loss between M(y_i) and x_i that produced it. This way you do not need to compute gradients through the "black box" F.

A pseudo code would look something like:

for x in examples:
  y = F(x)  # applying F on x - getting only output WITHOUT any gradients
  pred = M(y)  # apply the trainable model M to the output of F
  loss = ||x - pred||  # loss will propagate gradients through M and stop at F
  loss.backward()


Instruction: How to circumvent AWS package and ephemeral limits for large packages + large models

Input:
We have a production scenario with users invoking expensive NLP functions running for short periods of time (say 30s). Because of the high load and intermittent usage, we're looking into Lambda function deployment. However - our packages are big.

I'm trying to fit AllenNLP in a lambda function, which in turn depends on pytorch, scipy, spacy and numpy and a few other libs.

What I've tried

Following recommendations made here and the example here, tests and additional files are removed. I also use a non-cuda version of Pytorch which gets its' size down. I can package an AllenNLP deployment down to about 512mb. Currently, this is still too big for AWS Lambda.

Possible fixes?

I'm wondering if anyone of has experience with one of the following potential pathways:

Cutting PyTorch out of AllenNLP. Without Pytorch, we're in reach of getting it to 250mb. We only need to load archived models in production, but that does seem to use some of the PyTorch infrastructure. Maybe there are alternatives?
Invoking PyTorch in (a fork of) AllenNLP as a second lambda function.
Using S3 to deliver some of the dependencies: SIMlinking some of the larger .so files and serving them from an S3 bucket might help. This does create an additional problem: the Semnatic Role Labelling we're using from AllenNLP also requires some language models of around 500mb, for which the ephemeral storage could be used - but maybe these can be streamed directly into RAM from S3?

Maybe i'm missing an easy solution. Any direction or experiences would be much appreciated!

Output:
You could deploy your models to SageMaker inside of AWS, and run Lambda -> Sagemaker to avoid having to load up very large functions inside of a Lambda.

Architecture explained here - https://aws.amazon.com/blogs/machine-learning/call-an-amazon-sagemaker-model-endpoint-using-amazon-api-gateway-and-aws-lambda/


Instruction: Pytorch, backprop and composite models

Input:
Just a quick check for a question I have.

I want to build a model that generates its output based on two models F and G like so.

y = G(F(x))

where x is of course the input, and y the output.

However, first I want to update the weights of the F(x) and then later update the weights of both F and G based on the value of y.

I understand that pytorch offers a way to specify your own backprop method.. but since my "method" seems to build out of basic components, could it be that I can do this with a standard solution?

My thoughts would be that I need separate optimizer/loss for the F and G objects. But in addition to that, also some update functionality for the composite model G(F()). Can anyone confirm this as well?

Output:
If as you suggest, the optimizers and losses for F and G can be separated, then I don't think that it will be necessary to implement any different update functionalities since you can specify the set of parameters for each optimizer, e.g.

optimizer_F = optim.SGD(F.parameters(),...)
optimizer_G = optim.SGD(G.parameters(),...)

then when you call optimizer_F.step() it will only update the parameters of F and similarly optimizer_G.step() will only update the parameters of G.


Instruction: How can I load a model in PyTorch without redefining the model?

Input:
I am looking for a way to save a pytorch model, and load it without the model definition. By this I mean that I want to save my model including model definition.

For example, I would like to have two scripts. The first would define, train, and save the model. The second would load and predict the model without including the model definition.

The method using torch.save(), torch.load() requires me to include the model definition in the prediction script, but I want to find a way to load a model without redefining it in the script.

Output:
You can attempt to export your model to TorchScript using tracing. This has limitations. Due to the way PyTorch constructs the model's computation graph on the fly, if you have any control-flow in your model then the exported model may not completely represent your python module. TorchScript is only supported in PyTorch >= 1.0.0, though I would recommend using the latest version possible.

For example, a model without any conditional behavior is fine

from torch import nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(10)
        self.conv2 = nn.Conv2d(10, 20, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(20)
        self.fc = nn.Linear(20 * 4 * 4, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2, 2)
        x = self.bn1(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2, 2)
        x = self.bn2(x)
        x = self.fc(x.flatten(1))
        return x

We can export this as follows

from torch import jit

net = Model()
# ... train your model

# put model in the mode you want to export (see bolded comment below)
net.eval()

# print example output
x = torch.ones(1, 3, 16, 16)
print(net(x))

# create TorchScript by tracing the computation graph with an example input
x = torch.ones(1, 3, 16, 16)
net_trace = jit.trace(net, x)
jit.save(net_trace, 'model.zip')

If successful then we can load our model into a new python script without using Model.

from torch import jit
net = jit.load('model.zip')

# print example output (should be same as during save)
x = torch.ones(1, 3, 16, 16)
print(net(x))

The loaded model is also trainable, however, the loaded model will only behave in the mode it was exported in. For example, in this case we exported our model in eval() mode, so using net.train() on the loaded module will have no effect.

Control-flow

A model like this, which has behavior that changes between passes won't be properly exported. Only the code evaluated during jit.trace will be exported.

from torch import nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 10, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(10)
        self.conv2 = nn.Conv2d(10, 20, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(20)
        self.fca = nn.Linear(20 * 4 * 4, 2)
        self.fcb = nn.Linear(20 * 4 * 4, 2)

        self.use_a = True

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2, 2)
        x = self.bn1(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, 2, 2)
        x = self.bn2(x)
        if self.use_a:
            x = self.fca(x.flatten(1))
        else:
            x = self.fcb(x.flatten(1))
        return x

We can still export the model as follows

import torch
from torch import jit

net = Model()
# ... train your model

net.eval()

# print example input
x = torch.ones(1, 3, 16, 16)
net.use_a = True
print('a:', net(x))
net.use_a = False
print('b:', net(x))

# save model
x = torch.ones(1, 3, 16, 16)
net_trace = jit.trace(net, x)
jit.save(net_trace, "model.ts")

In this case the example outputs are

a: tensor([[-0.0959,  0.0657]], grad_fn=<AddmmBackward>)
b: tensor([[ 0.1437, -0.0033]], grad_fn=<AddmmBackward>)

However, loading

import torch
from torch import jit

net = jit.load("model.ts")

# will not match the output from before
x = torch.ones(1, 3, 16, 16)
net.use_a = True
print('a:', net(x))
net.use_a = False
print('b:', net(x))

results in

a: tensor([[ 0.1437, -0.0033]], grad_fn=<DifferentiableGraphBackward>)
b: tensor([[ 0.1437, -0.0033]], grad_fn=<DifferentiableGraphBackward>)

Notice that the logic of the branch "a" is not present since net.use_a was False when jit.trace was called.

Scripting

These limitations can be overcome but require some effort on your end. You can use the scripting functionality to ensure that all the logic is exported.


Instruction: Inference pytorch C++ with alexnet and cv::imread image

Input:
I am trying to infer with a C++ application an image classification task using an alexnet pre-trained net.I have successfully inferred a dog image loading the net with python:

alexnet = torchvision.models.alexnet(pretrained=True)
img = Image.open("dog.jpg")
transform = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(
 mean=[0.485, 0.456, 0.406],
 std=[0.229, 0.224, 0.225]
 )])
img_t = transform(img)
batch_t = torch.unsqueeze(img_t, 0)
alexnet.forward(batch_t)
_, index = torch.max(out, 1)

Result index is 208, Labrador_retriever, that looks good.
Then I save the net to be loaded from a C++ application

example = torch.rand(1, 3, 224, 224)
traced_script_module_alex = torch.jit.trace(alexnet, example)
traced_script_module.save("alexnet.pt")

When I  load to C++, I get the wrong result:

cv::Mat img = cv::imread("dog.jpg");
cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);

// Convert the image and label to a tensor.
torch::Tensor img_tensor = torch::from_blob(img.data, { 1, img.rows, img.cols, 3 }, torch::kByte);
img_tensor = img_tensor.permute({ 0, 3, 1, 2 }); // convert to CxHxW
img_tensor = img_tensor.to(torch::kFloat);
std::vector<torch::jit::IValue> input;
input.push_back(img_tensor);
torch::jit::script::Module  module = torch::jit::load("alexnet.pt");
at::Tensor output = module.forward(input).toTensor();
std::cout << output.argmax(1) << '\n';

the argmax is 463, bucket.
I think I am not looking at the same image; what am I missing...?

Output:
Your C++ code is missing this part of your Python code:

transform = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(
 mean=[0.485, 0.456, 0.406],
 std=[0.229, 0.224, 0.225]
 )])
img_t = transform(img)


Instruction: Pytorch equivalent of tf.Variable

Input:
I am trying to implement this code in pytorch:

self.scale_var = tf.Variable(
          0.1, name='scale_var',
          trainable=True,
          dtype=tf.float32,
          constraint=lambda x: tf.clip_by_value(x, 0, np.infty))

I want to have a scalar value that is trainable and would like to scale a constant with this value in the loss function. Is the below mentioned code  appropriate?

class pytorch_variable(nn.Module):
    def __init__(self):
        super(pytorch_variable,self).__init__()
        self.var = nn.Parameter(torch.tensor(0.1))

    def forward(self):
        return self.var

What is happening right now is that the gradients flow through this, but the trainable scalar value slowly reduces to zero, decreasing by 0.001 from initial value of 0.1 (till zero because I clip the data after loss.backward() call).

Output:
In PyTorch, Variable and Tensor were merged, so you are correct that a scalar variable should just be a scalar tensor.

In isolation:

>>> x=torch.tensor(5.5, requires_grad=True)
>>> x.grad
>>> x.backward(torch.tensor(12.4))
>>> x.grad
tensor(12.4000)

0.001 is a common learning rate, so I'd suspect that's related to the rate at which your trainable variable is being updated.


Instruction: How to install torch in python

Input:
I tried pip3 install torch --no-cache-dir and, after few seconds, I got this:

Collecting torch
      Downloading https://files.pythonhosted.org/packages/24/19/4804aea17cd136f1705a5e98a00618cb8f6ccc375ad8bfa437408e09d058/torch-1.4.0-cp36-cp36m-manylinux1_x86_64.whl (753.4MB)
        100% |████████████████████████████████| 753.4MB 5.7MB/s
    Exception:
    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
        status = self.run(options, args)
      File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 342, in run
        requirement_set.prepare_files(finder)
      File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
        ignore_dependencies=self.ignore_dependencies))
      File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 620, in _prepare_file
        session=self.session, hashes=hashes)
      File "/usr/lib/python3/dist-packages/pip/download.py", line 821, in unpack_url
        hashes=hashes
      File "/usr/lib/python3/dist-packages/pip/download.py", line 663, in unpack_http_url
        unpack_file(from_path, location, content_type, link)
      File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 617, in unpack_file
        flatten=not filename.endswith('.whl')
      File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 506, in unzip_file
        data = zip.read(name)
      File "/usr/lib/python3.6/zipfile.py", line 1338, in read
        return fp.read()
      File "/usr/lib/python3.6/zipfile.py", line 858, in read
        buf += self._read1(self.MAX_N)
      File "/usr/lib/python3.6/zipfile.py", line 948, in _read1
        data = self._decompressor.decompress(data, n)
    MemoryError

What should I do now to install PyTorch?

I tried almost every method mentioned on google. I am working on Ubuntu, I tried using conda too, but I am unable to use that package outside conda.

Output:
For pip environment use this

pip3 install torchvision

For conda environment use this (run this command on anaconda prompt)

conda install PyTorch -c PyTorch

Update

Use this code to turn off your cache

pip3 --no-cache-dir install torchvision

or

pip3 install torchvision--no-cache-dir

or

pip install --no-cache-dir torchvision

Try one by one


Instruction: Resize RGB Tensor pytorch

Input:
I want to resize a 3-D RBG tensor in pytorch. I know how to resize a 4-D tensor, but unfortunalty this method does not work for 3-D.

The input is:

#input shape: [3, 100, 200]   ---> desired output shape: [3, 80, 120]

if I have a 4-D vector it works fine.

#input shape: [2, 3, 100, 200]
out = torch.nn.functional.interpolate(T,size=(100,80), mode='bilinear')

Any suggestions? Thanks in advance!

Output:
Thanks to jodag I found the answer:

# input shape [3, 200, 120]
T = T.unsqueeze(0)
T = torch.nn.functional.interpolate(T,size=(100,80), mode='bilinear')
T = T.squeeze(0)
# output shape [3, 100, 80]


Instruction: Unable to allocate GPU memory, when there is enough of cached memory

Input:
I am training vgg16 model from scratch on AWS EC2 Deep Learning AMI machine (Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1054-aws x86_64v)) with Python3 (CUDA 10.1 and Intel MKL) (Pytorch 1.3.1) and facing below error while updating model parameters.

  RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 11.17 GiB total capacity; 10.76 GiB already allocated; 4.81 MiB free; 119.92 MiB cached)

Code for updating parameters:

def _update_fisher_params(self, current_ds, batch_size, num_batch):
    dl = DataLoader(current_ds, batch_size, shuffle=True)
    log_liklihoods = []
    for i, (input, target) in enumerate(dl):
        if i > num_batch:
            break
        output = F.log_softmax(self.model(input.cuda().float()), dim=1)
        log_liklihoods.append(output[:, target])
    log_likelihood = torch.cat(log_liklihoods).mean()
    grad_log_liklihood = autograd.grad(log_likelihood, self.model.parameters())
    _buff_param_names = [param[0].replace('.', '__') for param in self.model.named_parameters()]
    for _buff_param_name, param in zip(_buff_param_names, grad_log_liklihood):
        self.model.register_buffer(_buff_param_name+'_estimated_fisher', param.data.clone() ** 2)

After debugging: log_liklihoods.append(output[:, target]) line throws error after 157 iterations

I have the required memory but it does not allocate, I am not getting why updating the gradients is causing the memory problem, as gradients should be de-referenced and released automatically on each iteration. Any idea?

I have tried following solutions but no luck.

Lowering batch size
Freeing cache with torch.cuda.empty_cache()
Reducing the number of filters to reduce the memory footprint

Machine Specs:

Output:
Finally I solved the memory problem! I realized that in each iteration I put the input data in a new tensor, and pytorch generates a new computation graph.
That causes the used RAM to grow forever. Then I used .detach() function, and the RAM always stays at a low level.

self.model(input.cuda().float()).detach().requires_grad_(True)


Instruction: pytorch 1D Dropout leads to unstable learning

Input:
I'm implementing an Inception-like CNN in pytorch.  After the blocks of convolution layers, I have three fully-connected linear layers followed by a sigmoid activation to give me my final regression output.  I'm testing the effects of dropout layers in this network, but it's giving me some unexpected results.

Here is the code:

class MyInception(nn.Module):
    def __init__(self, in_channels, verbose=False):
        super(MyInception, self).__init__()
        self.v = verbose
        ic=in_channels; oc=16
        self.inceptionBlock1 = InceptionBlock(in_channels=ic, out_channels=oc, maxpool=False, verbose=verbose)
        self.inceptionBlock2 = InceptionBlock(in_channels=oc * 6, out_channels=oc, maxpool=False, verbose=verbose)
        self.pool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.regressor = nn.Sequential(
            nn.Linear(oc * 6 * 35 * 35, 1024, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 1
            nn.Linear(1024, 128, bias=True),
            nn.ReLU(inplace=True),
            nn.Dropout(p=0.2, inplace=False),  # <--- Dropout 2
            nn.Linear(128, 1, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        x = self.inceptionBlock1(x)
        x = self.inceptionBlock2(x)
        x = self.pool(x)
        x = torch.flatten(x, 1)
        x = self.regressor(x)
        return x

def train(epochs=10, dot_every=25):
    running = pd.DataFrame(columns=['Epoch','Round','TrainLoss','TestLoss','LearningRate'])
    for epoch in range(epochs):
        train_losses = []
        model.train()
        counter = 0

        for images, targets in train_loader:
            images = images.to(device)
            targets = targets.to(device)

            optimizer.zero_grad()
            outputs = model(images)
            loss = loss_fn(torch.flatten(outputs), targets)
            train_losses.append( loss.item() )
            loss.backward()
            optimizer.step()

            counter += 1
            if counter % dot_every == 0:
                print(".",  end='.', flush=True)
                test_loss = test()
            else:
                test_loss = -1.
            lr = np.squeeze(scheduler.get_lr())
            running = running.append(pd.Series([epoch, counter, loss.item(), test_loss, lr], index=running.columns), ignore_index=True)

        test_loss = test()
        train_loss = np.mean(np.asarray(train_losses))
        running = running.append(pd.Series([epoch, counter, train_loss, test_loss, lr], index=running.columns), ignore_index=True)
        print("")
        print(f"Epoch {epoch+1}, Train Loss: {np.round(train_loss,4)}, Test Loss: {np.round(test_loss, 4)}, Learning Rate: {np.format_float_scientific(lr, precision=4)}")
    return running

def test():
    model.eval()
    test_losses = []
    for i, (images,targets) in enumerate(test_loader):
        images = images.to(device)
        targets = targets.to(device)
        outputs = model(images)
        loss = loss_fn(torch.flatten(outputs), targets)
        test_losses.append( loss.item() )

    mean_loss = np.mean(np.asarray(test_losses))
    return mean_loss

# instantiate the model
model = MyInception(in_channels=4, verbose=False).to(device)
# define the optimizer and loss function
optimizer = Adam(model.parameters(), lr=0.001, weight_decay=0.0001)
loss_fn = nn.MSELoss()

# run it
results = train(epochs=10, dot_every=20)

Here is a plot of the MSE losses for the training data.  (red = no dropout, green = second dropout only, blue = first dropout only, purple = both dropouts)
Runs with dropout have big increases in losses at the epoch boundaries (dashed vertical lines), with the double dropout even having a big jump in loss at the start of epoch 10.

The important thing is the test loss.  That is much more stable and not too different between either condition after the 5th epoch, so maybe I shouldn't care. But I would like to understand what is going on.

Output: I cracked the case.  I realized that I flip model.train() to model.eval() in the test call without setting it back to train() after.  Since Dropout behaves differently in train and eval modes, adding in Dropout revealed the bug.


Instruction: How to remove certain layers from Fastern-RCNN in Pytorch?

Input:
Target: I want to use the pretrained Faster-RCNN model to extract features from image.

What I have tried: I use below code to build the model:

import torchvision.models as models
from PIL import Image
import torchvision.transforms as T
import torch

# download the pretrained fasterrcnn model
model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
model.cuda()

# remove [2:] layers
modules = list(model.children())[:2]
model_t=torch.nn.Sequential(*modules)

# load image and extract features
img = Image.open('data/person.jpg')
transform = T.Compose([T.ToTensor()])
img_t = transform(img)
batch_t = torch.unsqueeze(img_t, 0).cuda()
ft = model_t(batch_t)

Error: But I got the following error:TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple

Please help! Thank you!

Output:
print(model.modules) to get the layer names. Then delete a layer with:
del model.my_layer_name


Instruction: Make PyTorch variables to float64

Input:
How to make all the variables created in a PyTorch file to float64?
Is there a single line of code which can do that?

Output:
You can set the default tensor type using this one-liner:

torch.set_default_tensor_type(torch.DoubleTensor)


Instruction: Multiplying and powering python float and pytorch integer

Input:
Why does a python float multiplied by a torch.long gives a torch.float but powering a float by a torch.long gives a torch.long?

>>> a = 0.9
>>> b = torch.tensor(2, dtype=torch.long)

>>> foo = a * b
>>> print(foo, foo.dtype)
tensor(1.8000) torch.float32

>>> bar = a ** b
>>> print(bar, bar.dtype)
tensor(0) torch.int64

Output:
This looks like a bug, probably in the way pytorch binds ** to __rpow__ or __pow__.

E.g. if you tried 0.9 - torch.tensor(2), since 0.9 isn't a tensor, this gets interpreted as torch.tensor(2).__rsub__(0.9), which works correctly. ** behaves the same way, but torch.tensor(2).__rpow__(0.9) incorrectly returns tensor(0) with dtype int64.

In the meantime you can use torch.tensor(0.9) ** torch.tensor(2).

Filed a bug:
https://github.com/pytorch/pytorch/issues/32436


Instruction: layer Normalization in pytorch?

Input:
shouldn't the layer normalization of x = torch.tensor([[1.5,0,0,0,0]]) be [[1.5,-0.5,-0.5,-0.5]] ? according to this paper paper and the equation from the pytorch doc. But the torch.nn.LayerNorm gives [[ 1.7320, -0.5773, -0.5773, -0.5773]]

Here is the example code:

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)

y1 = layerNorm(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)

where:

y1 == tensor([[ 1.7320, -0.5773, -0.5773, -0.5773]])
y2 == tensor([[ 1.5000, -0.5000, -0.5000, -0.5000]])

Output:
Yet another simplified implementation of a Layer Norm layer with bare PyTorch.
from typing import Tuple
import torch

def layer_norm(
    x: torch.Tensor, dim: Tuple[int], eps: float = 0.00001
) -> torch.Tensor:
    mean = torch.mean(x, dim=dim, keepdim=True)
    var = torch.square(x - mean).mean(dim=dim, keepdim=True)
    return (x - mean) / torch.sqrt(var + eps)

def test_that_results_match() -> None:
    dims = (1, 2)
    X = torch.normal(0, 1, size=(3, 3, 3))

    indices = torch.tensor(dims)
    normalized_shape = torch.tensor(X.size()).index_select(0, indices)
    orig_layer_norm = torch.nn.LayerNorm(normalized_shape)

    y = orig_layer_norm(X)
    y_hat = layer_norm(X, dim=dims)

    assert torch.allclose(y, y_hat)

Note, that original implementation also has trainable parameters  and β (see the docs).


Instruction: Bitwise operations in Pytorch

Input:
Could someone help me how to perform bitwise  AND operations on two tensors in Pytorch 1.4?

Apparently I could only find NOT and XOR operations in official document

Output:
I don't see them in the docs, but it looks like &, |, __and__, __or__, __xor__, etc are bit-wise:

>>> torch.tensor([1, 2, 3, 4]).__xor__(torch.tensor([1, 1, 1, 1]))
tensor([0, 3, 2, 5])
>>> torch.tensor([1, 2, 3, 4]) | torch.tensor([1, 1, 1, 1])
tensor([1, 3, 3, 5])
>>> torch.tensor([1, 2, 3, 4]) & torch.tensor([1, 1, 1, 1])
tensor([1, 0, 1, 0])
>>> torch.tensor([1, 2, 3, 4]).__and__(torch.tensor([1, 1, 1, 1]))
tensor([1, 0, 1, 0])

See https://github.com/pytorch/pytorch/pull/1556


Instruction: How to specify pytorch as a package requirement on windows?

Input:
I have a python package which depends on pytorch and which I’d like windows users to be able to install via pip (the specific package is: https://github.com/mindsdb/lightwood, but I don’t think this is very relevant to my question).

What are the best practices for going about this ?

Are there some project I could use as examples ?

It seems like the pypi hosted version of torch & torchvision aren’t windows compatible and the “getting started” section suggests installing from the custom pytorch repository, but beyond that I’m not sure what the ideal solution would be to incorporate this as part of a setup script.

Output:
  What are the best practices for going about this ?

If your project depends on other projects that are not distributed through PyPI then you have to inform the users of your project one way or another. I recommend the following combination:

clearly specify (in your project's documentation pages, or in the project's long description, or in the README, or anything like this) which dependencies are not available through PyPI (and possibly the reason why, with the appropriate links) as well as the possible locations to get them from;
to facilitate the user experience, publish alongside your project a pre-prepared requirements.txt file with the appropriate --find-links options.

The reason why (or main reason, there are others), is that anyone using pip assumes that (by default) everything will be downloaded from PyPI and nowhere else. In other words anyone using pip puts some trust into pypi.org as a source for Python project distributions. If pip were suddenly to download artifacts from other sources, it would breach this trust. It should be the user's decision to download from other sources.

So you could provide in your project's documentation an example of requirements.txt file like the following:

# ...
torch===1.4.0  --find-links https://download.pytorch.org/whl/torch_stable.html
torchvision===0.5.0  --find-links https://download.pytorch.org/whl/torch_stable.html
# ...

Update

The best solution would be to help the maintainers of the projects in question to publish Windows wheels on PyPI directly:

https://github.com/pytorch/pytorch/issues/24310
https://github.com/pytorch/vision/issues/1774
https://pypi.org/help/#file-size-limit


Instruction: PyTorch: How to define a new neural network that utilizes transfer learning

Input:
I am migrating from Keras/TF frameworks and I have litte troubles understanding the transfer learning process in PyTorch.

I want to use pytorch-lightning framework and I want to switch between different neural networks in one script.

Per this example we can switch between different neural networks in their implementation:

class BERT(pl.LightningModule):
def __init__(self, model_name, task):
    self.task = task

    if model_name == 'transformer':
        self.net = Transformer()
    elif model_name == 'my_cool_version':
        self.net = MyCoolVersion()

The question is: how to create a new neural network that extends the nn.Module and utilizes transfer learning process?

My own implementation looks like this: I am using vgg16 network and replaced the classifier layer with only one fc with two output neurons.

class VGGNetwork(nn.Module):
    def __init__(self):
        super(VGGNetwork, self).__init__()
        # vgg16 is the default model here, we can use bn etc...
        self.model = vgg16(pretrained=True)

        # removing the last three layers of classifier only 2 ...
        self.model.classifier = nn.Sequential(nn.Linear(512 * 7 * 7, 2))

def forward(self, x):
    return self.model.forward(x)

Is this the correct way how to do that?

Output:
you can freeze weights and bais for the neural network layer except for the last layer.

you can use requires_grad = False

for param in model_conv.parameters():
    param.requires_grad = False

you can find more about this at the following link
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html


Instruction: PyTorch allocates more memory on the first available GPU (cuda:0)

Input:
As a part of the reinforcement learning training system, I am training four policies in parallel using four GPUs. For each model, there are two processes - the actor and the learner, which only use their specific GPU (e.g. actor and learner corresponding to model #2 only use GPU #2 for all their tensors). Actor and learner share the model layers via torch's share_memory_().
Since the four training "subsystems" are completely symmetric, I would expect them to use the exact same amount of GPU memory on each of the four GPUs. In practice, however, I see a lot more GPU memory allocated on the first GPU (cuda:0).

It feels like all the memory sharing is somehow done via GPU #0. Is there a way to fix this?

So far, I tried setting CUDA_VISIBLE_DEVICES in child processes by explicitly altering os.environ in the process start function. This does not seem to have any effect, probably because child processes are forked from the main process, where PyTorch CUDA is already initialized, and envvars just seem to be ignored at this point.

Output:
Ok, so far I came up with a workaround. My hypothesis was right, if PyTorch CUDA subsystem is already initialized before the child process is forked, setting CUDA_VISIBLE_DEVICES to a different value for a subprocess does not do anything.

Even worse, calling torch.cuda.device_count() is enough to initialize CUDA, so we can't even query the number of GPUs from PyTorch. Solution is either hardcode it, pass as a parameter, or query PyTorch API in a separate process. My implementation for the latter:

import sys

def get_available_gpus_without_triggering_pytorch_cuda_initialization(envvars):
    import subprocess
    out = subprocess.run([sys.executable, '-m', 'utils.get_available_gpus'], capture_output=True, env=envvars)
    text_output = out.stdout.decode()
    from utils.utils import log
    log.debug('Queried available GPUs: %s', text_output)
    return text_output

def main():
    import torch
    device_count = torch.cuda.device_count()
    available_gpus = ','.join(str(g) for g in range(device_count))
    print(available_gpus)
    return 0

if __name__ == '__main__':
    sys.exit(main())

Basically this function calls its own script as separate python process and reads stdout.

I won't mark this answer as accepted because I would like to learn a proper solution if it exists.


Instruction: Python process never finishing when called from Java

Input:
I've tried to setup an AI with PyTorch. Everything is fine when I call my script from the console. But when I call the script in a Java `ProcessBuildera, it will finish but never terminate...

Here is the ProcessBuilder code

String[] cmd = {"python3", "-i" , "AI/Home-System.py",
                data.getName().replace(".csv", ""),
                "true",
                "false"};
ProcessBuilder pb = new ProcessBuilder(cmd);
Process p = pb.start();

Hope that you can help me

Edit:

I found another solution. I call this script in a linux screen with

String[] cmd = {"screen", "-dmS", "AI-" + device,
                "python3", "AI/Home-System.py",
                data.getName().replace(".csv", ""),
                "true",
                "false"};
Runtime.getRuntime().exec(cmd);

Output:
Read  the process' output stream, as the end of this stream allows your ProcessBuilder to exit. Or else call the ProcessBuilder's inheritIO().

Then waitFor() the process.

Here is some sample code showing these steps.


Instruction: Trying to learn how to implement a single Neuron

Input:
I have this code in Pytorch but cant get it to work. I have it working with Numpy as return (X.T * W).sum(axis=1) + B

But with Pytorch I keep getting this error...

def neural_network_neurons(W, B, X):

    W = W.view(W.size(0), -1)
    z1 =  torch.matmul(X, W) + B
    return ReLU(z1)

# --------------------------------------------------
W = torch.tensor([[1.2, 0.3, 0.1], [.01, 2.1, 0.7]])
B = torch.tensor([2.1, 0.89])
X = torch.tensor([0.3, 6.8, 0.59])

neural_network_neurons(W, B, X)

---------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-21-8a5d3a425c16> in <module>
      4 X = torch.tensor([0.3, 6.8, 0.59])
      5
----> 6 neural_network_neurons(W, B, X)

<ipython-input-20-7450924eb4a5> in neural_network_neurons(W, B, X)
      5     ###
      6     W = W.view(W.size(0), -1)
----> 7     z1 =  torch.matmul(X, W) + B
      8     return ReLU(z1)
      9     #return (X.T * W).sum(axis=1) + B

RuntimeError: size mismatch, m1: [1 x 3], m2: [2 x 3] at
/pytorch/aten/src/TH/generic/THTensorMath.cpp:197

Output: You have the wrong orientation for W: you defined a 2x3 matrix, but your algorithm requires a 3x2.  Try W.T instead?


Instruction: In PyTorch's "MaxPool2D", is padding added depending on "ceil_mode"?

Input:
In MaxPool2D the padding is by default set to 0 and the ceil_mode is also set to False. Now, if I have an input of size 7x7 with kernel=2,stride=2 the output shape becomes 3x3, but when I use ceil_mode=True, it becomes 4x4, which makes sense because (if the following formula is correct), for 7x7 with output_shape would be 3.5x3.5 and depending on the ceil_mode it would be either 3x3 or 4x4.

Now, my question is, if the ceil_mode=True, does it change the default padding?

If it does, then how is it adding the padding i.e. is it adding the padding on left first or right, up first or down?

Output:
Ceil_mode=True changes the padding.

  In the case of ceil mode, additional columns and rows are added at the right as well as at the down. (Not top and not left). It does not need to be one extra column. It depends on the stride value as well. I just wrote small code snippet where you can check how the populated values are pooled in either modes.

Before I found the post referenced above, I experimented the same way with your problem, it also seems as though the zero-padding is not used during the pooling operation, as in my following example the zeros would have been the maximum elements to be taken, but this does not seem to be the case.

    test_tensor = torch.FloatTensor(2,7,7).random_(-10,-5)
    print(test_tensor)
    max_pool = nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=True)
    print(max_pool(test_tensor))
    max_pool = nn.MaxPool2d(kernel_size=2, stride=2, ceil_mode=False)
    print(max_pool(test_tensor))

Random sample tensor:

    tensor([[[ -6.,  -9.,  -7., -10.,  -6.,  -8.,  -6.],
             [-10., -10., -10.,  -6., -10.,  -9.,  -6.],
             [-10.,  -7.,  -7.,  -8., -10., -10.,  -9.],
             [ -8., -10., -10.,  -9.,  -9., -10.,  -9.],
             [ -8.,  -6.,  -8.,  -6.,  -7.,  -7.,  -9.],
             [-10.,  -8.,  -7., -10.,  -9.,  -6.,  -8.],
             [-10.,  -6.,  -9., -10.,  -9.,  -9., -10.]],

            [[-10.,  -8.,  -6., -10.,  -9.,  -6.,  -7.],
             [ -7.,  -7., -10., -10.,  -6.,  -9.,  -7.],
             [ -6., -10.,  -7.,  -8.,  -8., -10.,  -9.],
             [ -8.,  -8.,  -6.,  -7.,  -6.,  -8.,  -6.],
             [ -9.,  -8.,  -7., -10.,  -8.,  -8.,  -7.],
             [-10., -10.,  -6.,  -9.,  -8.,  -8.,  -8.],
             [-10.,  -6.,  -9.,  -9.,  -7.,  -9., -10.]]])

ceil_mode=True

    tensor([[[ -6.,  -6.,  -6.,  -6.],
             [ -7.,  -7.,  -9.,  -9.],
             [ -6.,  -6.,  -6.,  -8.],
             [ -6.,  -9.,  -9., -10.]],

            [[ -7.,  -6.,  -6.,  -7.],
             [ -6.,  -6.,  -6.,  -6.],
             [ -8.,  -6.,  -8.,  -7.],
             [ -6.,  -9.,  -7., -10.]]])

ceil_mode=False

    tensor([[[-6., -6., -6.],
             [-7., -7., -9.],
             [-6., -6., -6.]],

            [[-7., -6., -6.],
             [-6., -6., -6.],
             [-8., -6., -8.]]])


Instruction: Image deconvolution with a CNN

Input:
I have an input tensor of shape (C,H,W), where H=W and C=W^2.  This tensor contains non-linearly transformed information for an image of shape (1,H,W) squeezed to (H,W).  The exact form of the transformation is not important (plus, there is no closed-form expression for it anyway).  I would like to design a CNN to estimate images from such tensors.  I realize that I will have to experiment with CNN architectures (since I don't have the exact form of the transformation), but I'm not exactly sure how to proceed.

The input tensor has both positive and negative values which are important for the image reconstruction, so a ReLU layer probably should not be implemented near the beginning of CNN.  I don't think that pooling layers would be useful, either, at least in the H and W dimensions.  Clearly, I have to collapse the C dimension to get the image, but I don't think that it should be done all at once, e.g., torch.nn.Conv2d( C, 1, kernel_size ) is probably not a good idea.

It seems to me that I should first use a Conv2D layer which produces the same size tensor as the input tensor (to partially unscramble the non-linear transformation), but if the kernel size is greater than one the H and W dimensions will be reduced in size, which I don't want (unless this can be addressed later in the CNN).  On the other hand, if the kernel size is one the shape will stay the same but I don't think that anything happens to the tensor in this case.  Also, I will probably have to include linear layers, but I'm not sure how to use them with 3D tensors.

Any suggestions would be welcome.

Output:
There's no problem with applying a ReLU layer near the beginning, as long as you apply a weighted linear layer first. If the net learns that it needs the values there, it can apply a negative weight to preserve the information (roughly speaking).

In fact, a useful thing to do in some networks is to normalize the input to fit a N(0, 1) normal distribution. See https://www.researchgate.net/post/Which_data_normalization_method_should_be_used_in_this_artificial_neural_network

As to the problem of "reducing" the H/W dimensions because of kernel sizes - you can probably use 0-padding on the borders to avoid this problem. In my experience the networks usually handle this relatively well. However, if performance is an issue, usually you might want to reduce resolution significantly and then do upscaling of some sort at the end. You can find an example of such network here: Create image of Neural Network structure

As for pooling/feature layers: Because the depth of the tensor is very big (W^2) I would suggest that you in fact do reduce a lot of it right away. The complexity of your network is quadratic in the depth of your tensors and in your pixels count, because of weights from/into each layer in the tensor. So, my basic strategy would be to reduce the information space fast in the beginning, do some layers of calculations, and then upscaling.

What I've learned over the years is that CNNs are pretty resilient, and that architectural ideas that might seem good on paper do very little in reality - the best factors are pretty much always more layers (done in a good way, but since ResNet it's gotten way easier) and more/better data. So I would start experimenting and try to assess given a working PoC what blocks the network or try variations.

I hope this makes enough sense :) Good luck!


Instruction: where could I find training.pt / test.pt

Input:
In the Pytorch docs for MNIST I read:

root (string): Root directory of dataset where MNIST/processed/training.pt
and  MNIST/processed/test.pt exist.

Where could I find these two files traing.pt, test.pt? And what are their format?

Output:
Assuming pytorch 1.x+, The constructor of torchvision.datasets.MNIST follows this signature:

torchvision.datasets.MNIST(root, train=True, transform=None, target_transform=None, download=False)

The easiest way to get the dataset is to set download=True, that way it will automatically download and store training.pt and test.pt. Assuming a local install, it will by default store them somewhere like .local/lib/python3.6/site-packages/torchvision/, although you don't have to worry about that.


Instruction: Famous neural networks for regression

Input: I have come across many neural network architecture for classification problems. AlexNet, ResNet, VGGNet, GoogLeNet etc... Is there similar networks for regression problems which can be used for transfer learning?

Output:
Alright, all those architecture are not only for classification, the only shift you have to make for modifying a DL model from classification to regression is to change the top layer. For example in the VGGNET the last layer could be :

Dense(25, activation='softmax')

That means that we want to predict 25 outputs with a probability distribution (classification)

But it could be

Dense(1, activation='linear')

With the exact same architecture, it will output a number (regression)

So in the case of transfer learning you can just take an existing architecture that is pretrained on classification tasks, remove the top layer and do whatever you want.


Instruction: Autograd function in Pytorch documentation

Input:
In the Pytorch documentation https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py
In the image, I am unable to understand what y.backward(v) means and why do we need to define another tensor v to do the backward operation and also how we got the results of x.grad
Thanks in advance

Output:
y.backward() computes dy/dz where z are all the leaf nodes in the computation graph. And it stores dy/dz in z.grad.

For example: In the above case, leaf nodes are x.

y.backward() works when y is a scalar which is the case for most of the deep-learning. When y is a vector you have to pass another vector (v in the above case). You can see this as computing d(v^Ty)/dx.

To answer how we got x.grad note that you raise x by the power of 2 unless norm exceeds 1000, so x.grad will be v*k*x**(k-1) where k is 2**i and i is the number of times the loop was executed.

To have a less complicated example, consider this:

x = torch.randn(3,requires_grad=True)
print(x)
Out: tensor([-0.0952, -0.4544, -0.7430], requires_grad=True)

y = x**2
v = torch.tensor([1.0,0.1,0.01])
y.backward(v)

print(x.grad)
Out[15]: tensor([-0.1903, -0.0909, -0.0149])

print(2*v*x)
Out: tensor([-0.1903, -0.0909, -0.0149], grad_fn=<MulBackward0>)


Instruction: while installing apex extension for pytorch(python environment) the following error is showing, am unable to solve this problem

Input:
I want to install apex extension for my pytorch environment, my system is windows 10 and am using python version 3.8.1 and pip version is 20.0.2

I read the instructions from this https://github.com/NVIDIA/apex and I executed the command

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext

This error is showing.

  c:\users\dell\appdata\local\programs\python\python38\lib\site-packages\pip_internal\commands\install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
        cmdoptions.check_install_build_global(options)
      Non-user install because site-packages writeable
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-ephem-wheel-cache-ehoqwpvf
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Initialized build tracking at C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Created build tracker: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Entered build tracker: C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi
      Created temporary directory: C:\Users\Dell\AppData\Local\Temp\pip-install-rivnsaa9
      Cleaning up...
      Removed build tracker: 'C:\Users\Dell\AppData\Local\Temp\pip-req-tracker-uowlsjqi'
      ERROR: You must give at least one requirement to install (see "pip help install")
      Exception information:
      Traceback (most recent call last):
        File "c:\users\dell\appdata\local\programs\python\python38\lib\site-packages\pip_internal\cli\base_command.py", line 186, in _main
          status = self.run(options, args)

Please solve this problem

Output:
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext

The line specified in your link is

$ pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Note that you're missing the final ./, which is why pip tells you that

  You must give at least one requirement to install (see "pip help install")

you're telling it to install, but you're not telling it what to install.


Instruction: How can I convert Tensor into Bitmap on PyTorch Mobile?

Input:
I found that solution (https://itnext.io/converting-pytorch-float-tensor-to-android-rgba-bitmap-with-kotlin-ffd4602a16b6) but when I tried to convert that way I found that the size of inputTensor.dataAsFloatArray is more than bitmap.width*bitmap.height. How works converting tensor to float array or is there any other possible method to convert pytorch tensor to bitmap?

val inputTensor = TensorImageUtils.bitmapToFloat32Tensor(
    bitmap,
    TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB
)

// Float array size is 196608 when width and height are 256x256 = 65536

val res = floatArrayToGrayscaleBitmap(inputTensor.dataAsFloatArray, bitmap.width, bitmap.height)

fun floatArrayToGrayscaleBitmap (
    floatArray: FloatArray,
    width: Int,
    height: Int,
    alpha :Byte = (255).toByte(),
    reverseScale :Boolean = false
) : Bitmap {

    // Create empty bitmap in RGBA format (even though it says ARGB but channels are RGBA)
    val bmp = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
    val byteBuffer = ByteBuffer.allocate(width*height*4)
    Log.d("App", floatArray.size.toString() + " " + (width * height * 4).toString())

    // mapping smallest value to 0 and largest value to 255
    val maxValue = floatArray.max() ?: 1.0f
    val minValue = floatArray.min() ?: 0.0f
    val delta = maxValue-minValue
    var tempValue :Byte

    // Define if float min..max will be mapped to 0..255 or 255..0
    val conversion = when(reverseScale) {
        false -> { v: Float -> ((v-minValue)/delta*255).toByte() }
        true -> { v: Float -> (255-(v-minValue)/delta*255).toByte() }
    }

    // copy each value from float array to RGB channels and set alpha channel
    floatArray.forEachIndexed { i, value ->
        tempValue = conversion(value)
        byteBuffer.put(4*i, tempValue)
        byteBuffer.put(4*i+1, tempValue)
        byteBuffer.put(4*i+2, tempValue)
        byteBuffer.put(4*i+3, alpha)
    }

    bmp.copyPixelsFromBuffer(byteBuffer)

    return bmp
}

Output:
None of the answers were able to produce the output I wanted, so this is what I came up with - it is basically only reverse engineered version of what happenes in TensorImageUtils.bitmapToFloat32Tensor().
Please note that this function only works if you are using MemoryFormat.CONTIGUOUS (which is default) in TensorImageUtils.bitmapToFloat32Tensor().
fun tensor2Bitmap(input: FloatArray, width: Int, height: Int, normMeanRGB: FloatArray, normStdRGB: FloatArray): Bitmap? {
    val pixelsCount = height * width
    val pixels = IntArray(pixelsCount)
    val output = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)

    val conversion = { v: Float -> ((v.coerceIn(0.0f, 1.0f))*255.0f).roundToInt()}

    val offset_g = pixelsCount
    val offset_b = 2 * pixelsCount
    for (i in 0 until pixelsCount) {
        val r = conversion(input[i] * normStdRGB[0] + normMeanRGB[0])
        val g = conversion(input[i + offset_g] * normStdRGB[1] + normMeanRGB[1])
        val b = conversion(input[i + offset_b] * normStdRGB[2] + normMeanRGB[2])
        pixels[i] = 255 shl 24 or (r.toInt() and 0xff shl 16) or (g.toInt() and 0xff shl 8) or (b.toInt() and 0xff)
    }
    output.setPixels(pixels, 0, width, 0, 0, width, height)
    return output
}

Example usage then could be as follows:
tensor2Bitmap(outputTensor.dataAsFloatArray, bitmap.width, bitmap.height, TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB)


Instruction: PyTorch-YOLOv3 Generating Training and Validation Curves

Input:
Hello again stackoverflow!  I greatly appreciate this community and the helpful feedback.

I have some other questions that I hope someone can help me with.  I am working with an implementation of PyTorch-YOLOv3 from https://github.com/eriklindernoren/PyTorch-YOLOv3

I have been able to train the model, but now I would like to generate training/validation curves. During training, I get back metrics on each epoch that look like this:

Epoch 1/3001 batch 0/8
Epoch 1/3001 batch 7/8

I'm trying to generate a graph of a metric (loss, recall, precision, accuacy, mAP) versus epoch by logging these metrics to an external .csv file and plotting those values.

Question 1: Does anyone with experience with this YOLOv3 know where the relevant information is?  I know the AP and mAP for each epoch are at the bottom of the second image (epoch 1/3001 batch 7/8).  I'm not sure where to look for the relevant loss, recall, and precision metrics.

Question 2: Does anyone know the difference between the YOLO layer 0 and layer 1.  Would plotting the metrics from each of these layers yield the training and validation curves, respectively?

Question 3: As of two to three months ago, I started receiving the warning the following warning instead of getting the epoch outputs.

/pytorch/aten/src/ATen/native/IndexingUtils.h:20: UserWarning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead.

It doesn't affect the training, but I would like to update the code so that this warning disappears.  Any suggestions?

Thank you all in advance.

Output:
The solution is mentioned in the issues. The exact link is: https://github.com/eriklindernoren/PyTorch-YOLOv3/issues/283

Replace in utils:269

ByteTensor = torch.cuda.ByteTensor if pred_boxes.is_cuda else torch.ByteTensor

With:
BoolTensor = torch.cuda.BoolTensor if pred_boxes.is_cuda else torch.BoolTensor

And it usage in lines 278, 279
obj_mask = BoolTensor(nB, nA, nG, nG).fill_(0)
noobj_mask = BoolTensor(nB, nA, nG, nG).fill_(1)


Instruction: Make GPU available again after numba.cuda.close()?

Input:
So when I run cuda.select_device(0) and then cuda.close(). Pytorch cannot access the GPU again, I know that there is way so that PyTorch can utilize the GPU again without having to restart the kernel. But I forgot how. Does anyone else know?
from  numba import cuda as cu
import torch

# random tensor
a=torch.rand(100,100)

#tensor can be loaded onto the gpu()
a.cuda()

device = cu.get_current_device()
device.reset()

# thows error "RuntimeError: CUDA error: invalid argument"
a.cuda()

cu.close()
# thows error "RuntimeError: CUDA error: invalid argument"
a.cuda()

torch.cuda.is_available()
#True

And then trying to run cuda-based pytorch code yields:
RuntimeError: CUDA error: invalid argument

Output:
I had the same issue but with TensorFlow and Keras when iterating through a for loop to tune hyperparamenters. It did not free up the GPU memory used by older models. The cuda solution did not work for me. The following did:
import gc
gc.collect()


Instruction: How to use Pytorch OneCycleLR in a training loop (and optimizer/scheduler interactions)?

Input:
I'm training an NN and using RMSprop as an optimizer and OneCycleLR as a scheduler.  I've been running it like this (in slightly simplified code):

optimizer = torch.optim.RMSprop(model.parameters(), lr=0.00001,
                              alpha=0.99, eps=1e-08, weight_decay=0.0001, momentum=0.0001, centered=False)
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.0005, epochs=epochs)

    for epoch in range(epochs):
        model.train()
        for counter, (images, targets) in enumerate(train_loader):

            # clear gradients from last run
            optimizer.zero_grad()

            # Run forward pass through the mini-batch
            outputs = model(images)

            # Calculate the losses
            loss = loss_fn(outputs, targets)

            # Calculate the gradients
            loss.backward()

            # Update parameters
            optimizer.step()   # Optimizer before scheduler????
            scheduler.step()

            # Check loss on training set
            test()

Note the optimizer and scheduler calls in each mini-batch.  This is working, though when I plot the learning rates through the training, the curve is very bumpy.  I checked the docs again, and this is the example shown for torch.optim.lr_scheduler.OneCycleLR

>>> data_loader = torch.utils.data.DataLoader(...)
>>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
>>> scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.01, steps_per_epoch=len(data_loader), epochs=10)
>>> for epoch in range(10):
>>>     for batch in data_loader:
>>>         train_batch(...)
>>>         scheduler.step()

Here, they omit the optimizer.step() in the training loop. And I thought, that makes sense since the optimizer is provided to OneCycleLR in its initialization, so it must be taking care of that on the back end.  But doing so gets me the warning:

UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.

Do I ignore that and trust the pseudocode in the docs?  Well, I did, and the model didn't do any learning, so the warning is correct and I put optimizer.step() back in.

This gets to the point that I don't really understand how the optimizer and scheduler interact (edit: how the Learning Rate in the optimizer interacts with the Learning Rate in the scheduler).  I see that generally the optimizer is run every mini-batch and the scheduler every epoch, though for OneCycleLR, they want you to run it every mini-batch too.

Any guidance (or a good tutorial article) would be appreciated!

Output:
Use optimizer.step() before scheduler.step(). Also, for OneCycleLR, you need to run scheduler.step() after every step - source (PyTorch docs). So, your training code is correct (as far as calling step() on optimizer and schedulers is concerned).

Also, in the example you mentioned, they have passed steps_per_epoch parameter, but you haven't done so in your training code. This is also mentioned in the docs. This might be causing the issue in your code.


Instruction: How exactly should the input file be formatted for the language model finetuning (BERT through Huggingface Transformers)?

Input:
I wanted to employ the examples/run_lm_finetuning.py from the Huggingface Transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus file should be structured (apart from referencing the Wiki-2 dataset). I've tried

One document per line (multiple sentences)
One sentence per line. Documents are separated by a blank line (this I found in some older pytorch-transformers documentation)

By looking at the code of examples/run_lm_finetuning.py it is not directly evident how sequence pairs for the Next Sentence Prediction objective are formed. Would the --line-by-line option help here? I'd be grateful, if someone could give me some hints how a text corpus file should look like.

Many thanks and cheers,

nminds

Output:
First of all, I strongly suggest to also open this as an issue in the huggingface library, as they have probably the strongest interest to answer this, and may take it as a sign that they should update/clarify their documentation.
But to answer your question, it seems that this specific sample script is basically returning either a LineByLineTextDataset (if you pass --line_by_line to the training), and otherwise a TextDataset, see ll. 144-149 in the script (formatted slightly for better visibility):
def load_and_cache_examples(args, tokenizer, evaluate=False):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    if args.line_by_line:
        return LineByLineTextDataset(tokenizer, args,
                           file_path=file_path, block_size=args.block_size)
    else:
        return TextDataset(tokenizer, args,
                           file_path=file_path, block_size=args.block_size)

A TextDataset simply splits the text into consecutive "blocks" of certain (token) length, e.g., it will cut your text every 512 tokens (default value).
The Next Sentence Prediction task is only implemented for the default BERT model, if I recall that correctly (seems to be consistent with what I found in the documentation), and is unfortunately not part of this specific finetuning script.
None of the utilized BERT models in the lm_finetuning script make use of that particular task, as far as I can see.


Instruction: Error in Tensorboard's(PyTorch) add_graph

Input:
I'm following this Pytorch's Tensorboard documentation.
I have the following code:
model = torchvision.models.resnet50(False)
writer.add_graph(model)

It throws the following error:

_ = model(*args)  # don't catch, just print the error message
TypeError: ResNet object argument after * must be an iterable, not NoneType

I don't know what I'm doing wrong here!

Output:
I had this problem too..

Passing an input_to_model parameter different from None solved the problem. However, I though it should be optional

dataiter = iter(trainloader)
images, labels = dataiter.next()
writer.add_graph(model, images)


Instruction: Computing Linear Layer in Tensor/Outer-Product space in PyTorch is Very Slow

Input:
I would like to make a PyTorch model that takes the outer product of the input with itself and then does a linear regression on that. As an example, consider the input vector [1,2,3], then I would like to compute w and b to optimize [1*1, 1*2, 1*3, 2*1, 2*2, 2*3, 3*1, 3*2, 3*3] @ w + b.

For a batch input with r rows and c columns, I can do this in PyTorch with

(input.reshape(r,c,1) @ input.reshape(r,1,c)).reshape(r,c**2) @ weigts + b

My problem is that it is extraordinarily slow. Like a factor 1000 times slower and more memory consumptious than Adding a fully connected c*c RELU layer, even though it has the same number of weights.

My question is why this happens?

Is reshape a very expensive operation for PyTorch? Could I reformulate it in a different way that would make things more efficient?

Another equivalent formulation I know is torch.diag(input @ weights @ input.T) + b, but now we are computing way more values than we need (r*r) just to throw them away again.

Output:
When you have to reshape a tensor during the training loop of a model it's always best to use view instead of reshape. There doesn't appear to be any performance overhead with a view, but it does require that the tensor data is contiguous.

If your tensors at the beginning aren't contiguous you can recopy the tensor and make it contiguous.


Instruction: pytorch train function Varibles and Tensors (read my introduction i dont know my problem as well it just dont work )

Input:
I started learning pytorch and started with videos about MNIST handwriting and learnt it with an video but the video is 2 years old and some things have changen since then i guess because it dosent work as in the video and i seriously dont know anything so i dont know whats my error or what i do wrong i just type everything the dude says in the video and want to understand and learn it this way(maybe you know better ways how to learn machine learning/deep learning would appreciate it) my code looks like this:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.optim as optim
import os
from torchvision import datasets, transforms

kwargs = {'num_workers': 1, 'pin_memory': True}
train_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=True, download=True, transform=transforms.Compose([transforms.ToTensor(),
                                                    transforms.Normalize((0.1307,),(0.3081,))])),
                                                    batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=False, transform=transforms.Compose([transforms.ToTensor(),
                                                    transforms.Normalize((0.1307,),(0.3081,))])),
                                                    batch_size=64, shuffle=True, **kwargs)

above everything works like in the video and i find the data in an folder now comes the class and it doesnt looks like theres an error but i dont know.

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size= 4)
        self.conv2 = nn.Conv2d(10, 20, kernel_size= 4)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 4)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 4)
        x = F.relu(x)
        print(x.size())
        exit()

model = Netz()
model.cuda()

something with this Varibale function is wrong it just dont works and pycharm also shows me there has to be something wrong but i dont know what so i ask here maybe you can help i also googled abit about it and it looks like this varible thing got removed or so but i dont know what to write else

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)
def train(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(train_data):
        data = data.cuda()
        target = target.cuda()
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = F.nll_loss
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()

for epoch in range(1, 30):
    train(epoch)

the error code looks like this :

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\mnist handwriting.py", line 60, in <module>
    train(epoch)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\mnist handwriting.py", line 46, in train
    for batch_id, (data, target) in enumerate(train_data):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    w.start()
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Finnw/PycharmProjects/pytorch 3.7/mnist handwriting.py", line 60, in <module>
    train(epoch)
  File "C:/Users/Finnw/PycharmProjects/pytorch 3.7/mnist handwriting.py", line 46, in train
    for batch_id, (data, target) in enumerate(train_data):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 798, in _get_data
    success, data = self._try_get_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 10444) exited unexpectedly

Process finished with exit code 1

Output:
I believe just setting num_workers to zero would solve your problem. One other thing that would solve your problem is to place your code in a main function.

The reasons for this can be found here:
https://docs.python.org/2/library/multiprocessing.html#multiprocessing-programming . The reason for this is that num_workers tells PyTorch to generate data samples in a multithreaded way, launching num_workers threads, such that they can be served as fast as possible to your training loop.

The error code you gave actually tells you pretty much the same thing:

An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...