Instruction: Way too much resources are used with Pytorch

Input:
I am using pytorch to train a DQN model. With ubuntu, if I use htop, I get

As you can see, all resources are used and I am a bit worried about that.
Here is my code.

Is there a way to use less resource? Do I have to add my requirement using pytorch?

Be aware that there's no GPUs on my machine, just CPUs

Output:
Yes, there is. You can use torch.set_num_threads(...) to specify the number of threads. Depending on the PyTorch version you use, maybe this function will not work correctly. See why in this issue. In there, you'll see that if needed you can use environment variables to limit OpenMP or MKL threads usage via OMP_NUM_THREADS=? and MKL_NUM_THREADS=? respectively, where ? is the number of threads.

Keep in mind that these things are expected to run on GPUs with thousands of cores, so I would limit CPU usage only when extremely necessary.


Instruction: How to use ITK to convert PNG into tensor for PyTorch

Input:
I'm trying to run a C++ PyTorch framework and ran into the following problem.

I successfully scripted a model and is now ready to run. Now I have to feed in a png image to the model.

I found someone with simmilar issue on the internet, his idea was to use ITK module to read in a PNG file and convert it to a array, then make it to Tensor.

PNG -> RGBPixel[] -> tensor

So the following is what I am trying out right now.

using PixelTyupe = itk::RGBPixel<unsinged char>;
const unsigned int Dimension = 3;
typedef itk::Image<PixelType, Dimension>      ImageType;
typedef itk::ImageFileReader<ImageType>       ReaderType;
typedef itk::ImageRegionIterator<ImageType>   IteratorType;

typename ImageType::RegionType region = itk_img->GetLargestPossibleRegion();
const typename ImageType::SizeType size = region.GetSize();

int len = size[0] * size[1] * size[2]; // This ends up 1920 * 1080 * 1
PixelType rowdata[len];
int count = 0;
IteratorType iter(itk_img, itk_img->GetRequestedRegion());

// convert itk to array
for (iter.GoToBegin(); !iter.IsAtEnd(); ++iter) {
   rowdata[count] = iter.Get();
   count++;
} // count = 1920 * 1080

// convert array to tensor
tensor_img = torch::from_blob(rowdata, {3, (int)size[0], (int)size[1]}, torch::kShort). clone(); // Segmenation Fault

When I try to print log data, it holds three numbers like 84 85 83, so I suppose the PNG file is successfully read.

However, I can't get the last part of the code to work. What I need is 3:1920:1080 tensor, but I don't think the three RGBPixel value is understood properly by the function.

And apart from that, I don't see why the dimension is set to 3.

I would appreciate any kind of help.

Output:
You don't need dimension 3, Dimension = 2 is sufficient. If the layout you need is RGBx1920x1080, then PixelType* rowdata = itk_img->GetBufferPointer(); would get you that layout without further processing. As torch::from_blob does not take ownership of the buffer, the other person was trying to use .clone(). You don't have to do that either, assuming you keep the itk_img in scope, or muck around with its reference count and deleter.

The crash probably comes from saying the buffer has short pixels (torch::kShort), when it has uchar.


Instruction: How to deserialize a PyTorch saved model with private methods inside a class?

Input:
I used the PyTorch saving method to serialize a bunch of essential objects. Among those, there was one class referencing a private method inside the __init__ of that same class. Now, after the serialization, I can't deserialize (unpickle) files because the private method is not accessible outside the class. Any idea how to solve or bypass it? I need to recover the data saved into the attributes of that class.

  File ".conda/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-a5666d77c70f>", line 1, in <module>
    torch.load("snapshots/model.pth", map_location='cpu')
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File ".conda/envs/py37/lib/python3.7/site-packages/torch/serialization.py", line 702, in _legacy_load
    result = unpickler.load()
AttributeError: 'Trainer' object has no attribute '__iterator'

EDIT-1:

Here there is a piece of code that will generate the problem I’m facing right now.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.__private  # buggy

    def __private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")

However, if you remove the private attribute from the method, you won’t get any error.

import torch

class Test:
    def __init__(self):
        self.a = min
        self.b = max
        self.c = self.private  # not buggy

    def private(self):
        return None

test = Test()

torch.save({"test": test}, "file.pkl")
torch.load("file.pkl")

Output:
This question is similar to Python multiprocessing - mapping private method, but can not be marked as duplicate because of the open bounty.

The issue arises from this open issue on the Python bug tracker: Objects referencing private-mangled names do not roundtrip properly under pickling, and is related to the way pickle handles name-mangling. More details on this answer: https://stackoverflow.com/a/57497698/6352677.

At this point, the only workaround is not using private methods in __init__.


Instruction: Difficulty in understanding the tokenizer used in Roberta model

Input:
from transformers import AutoModel, AutoTokenizer

tokenizer1 = AutoTokenizer.from_pretrained("roberta-base")
tokenizer2 = AutoTokenizer.from_pretrained("bert-base-cased")

sequence = "A Titan RTX has 24GB of VRAM"
print(tokenizer1.tokenize(sequence))
print(tokenizer2.tokenize(sequence))

Output:

['A', 'ĠTitan', 'ĠRTX', 'Ġhas', 'Ġ24', 'GB', 'Ġof', 'ĠVR', 'AM']

['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']

Bert model uses WordPiece tokenizer. Any word that does not occur in the WordPiece vocabulary is broken down into sub-words greedily. For example, 'RTX' is broken into 'R', '##T' and '##X' where ## indicates it is a subtoken.

Roberta uses BPE tokenizer but I'm unable to understand

a) how BPE tokenizer works?

b) what does G represents in each of tokens?

Output:
This question is extremely broad, so I'm trying to give an answer that focuses on the main problem at hand. If you feel the need to have other questions answered, please open another question focusing on one question at a time, see the [help/on-topic] rules for Stackoverflow.

Essentially, as you've correctly identified, BPE is central to any tokenization in modern deep networks. I highly recommend you to read the original BPE paper by Sennrich et al., in which they also highlight a bit more of the history of BPEs.
In any case, the tokenizers for any of the huggingface models are pretrained, meaning that they are usually generated from the training set of the algorithm beforehand. Common implementations such as SentencePiece also give a bit better understanding of it, but essentially the task is framed as a constrained optimization problem, where you specify a maximum number of k allowed vocabulary words (the constraint), and the algorithm tries to then keep as many words intact without exceeding k.

if there are not enough words to cover the whole vocabulary, smaller units are used to approximate the vocabulary, which results in the splits observed in the example you gave.
RoBERTa uses a variant called "byte-level BPE", the best explanation is probably given in this study by Wang et al.. The main benefit is, that it results in a smaller vocabulary while maintaining the quality of splits, from what I understand.

The second part of your question is easier to explain; while BERT highlights the merging of two subsequent tokens (with ##), RoBERTa's tokenizer instead highlights the start of a new token with a specific unicode character (in this case, \u0120, the G with a dot). The best reason I could find for this was this thread, which argues that it basically avoids the use of whitespaces in training.


Instruction: Error in Transforming custom Dataset in Pytorch

Input:
I was following this tutorial: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html for making my own custom data loader for MuNuSeg Dataset but I am stuck a point. The dataloader is working fine but when I add transforms to it, I get errors.

I am facing an issue similar to what is mentioned here :
Error Utilizing Pytorch Transforms and Custom Dataset
According to the answer there I have made custom transform for each of the inbuilt transforms so that one whole sample is transformed at the same time. Below are my custom transforms

class AffineTrans(object):
def __init__(self, degrees, translate):
    self.degrees = degrees
    self.translate = translate

def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF = transforms.RandomAffine(degrees = self.degrees, translate=self.translate)
    image = TF(image)
    contour = TF(contour)
    clrmask = (clrmask)

class Flip(object):
def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF1 = transforms.RandomHorizontalFlip()
    TF2 = transforms.RandomVerticalFlip()
    image = TF1(image)
    contour = TF1(contour)
    clrmask = TF1(clrmask)
    image = TF2(image)
    contour = TF2(contour)
    clrmask = TF2(clrmask)

class ClrJitter(object):
def __init__(self, brightness, contrast, saturation, hue):
    self.brightness = brightness
    self.contrast = contrast
    self.saturation = saturation
    self.hue = hue

def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF = transforms.ColorJitter(self.brightness, self.contrast, self.saturation, self.hue)
    image = TF(image)
    contour = TF(contour)
    clrmask = TF(clrmask)

And composed them in the following way

composed = transforms.Compose([RandomCrop(256),
                           AffineTrans(15.0,(0.1,0.1)),
                           Flip(),
                           ClrJitter(10, 10, 10, 0.01),
                           ToTensor()])

And here's the trainLoader Code

class trainLoader(Dataset):
def __init__(self, transform=None):
    """
    Args:
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    [self.train_data , self.test_data1, self.test_data2] = dirload()
    self.transform = transform

def __len__(self):
    return(len(self.train_data[0]))

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    img_name = self.train_data[0][idx]
    contour_name = self.train_data[1][idx]
    color_name = self.train_data[2][idx]
    image = cv2.imread(img_name)
    contour = cv2.imread(contour_name)
    clrmask = cv2.imread(color_name)
    sample = {'image': image, 'contour': contour, 'clrmask': clrmask}

    if self.transform:
        sample = self.transform(sample)

    return sample

To check the working of the above code I am doing the following

    train_dat = trainLoader(composed)

for i in range(len(train_dat)):
    sample = train_dat[i]

    print(i, sample['image'].shape, sample['contour'].shape, sample['clrmask'].shape)
    cv2.imshow('sample',sample['image'])
    cv2.waitKey()
    if i == 3:
        break

But I am still, again and again, Encountering the following error

runcell(0, 'F:/Moodle/SEM 8/SRE/code/MoNuSeg/main.py')

runcell(1, 'F:/Moodle/SEM 8/SRE/code/MoNuSeg/main.py')
Traceback (most recent call last):

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 212, in <module>
    sample = train_dat[i]

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 109, in __getitem__
    sample = self.transform(sample)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 60, in __call__
    img = t(img)

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 156, in __call__
    image = TF(image)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 1018, in __call__
    ret = self.get_params(self.degrees, self.translate, self.scale, self.shear, img.size)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 992, in get_params
    max_dx = translate[0] * img_size[0]

TypeError: 'int' object is not subscriptable

This is quite an ambiguous error as I exactly don't get what is the error is

Any help would be really appreciated

Output:
The problem is that you're passing a NumPy array, whereas the transform expects a PIL Image. You can fix that by adding transforms.ToPILImage() as the first transform:

composed = transforms.Compose([
    transforms.ToPILImage(),
    RandomCrop(256),
    AffineTrans(15.0,(0.1,0.1)),
    Flip(),
    ClrJitter(10, 10, 10, 0.01),
    ToTensor()
])

Assuming you have a from torchvision import transforms at the beginning.

The root of the problem is that you're using OpenCV to load the images:

def __getitem__(self, idx):
    # [...]
    image = cv2.imread(img_name)

and you could replace these loading calls from OpenCV to PIL to solve the problem as well.

Just so you know, for NumPy array, the .size() returns an int, which is causing you the problem. Check in the following code the difference:

import numpy as np
from PIL import Image

# NumPy
img = np.zeros((30, 30))
print(img.size)  # output: 900

# PIL
pil_img = Image.fromarray(img)
print(pil_img.size)  # output: (30, 30)


Instruction: Fine-tuning Resnet using pre-convoluted features

Input:
As we know that convolution layers are expensive to calculate , I would like to compute the output of the convolution layers once and use them to train the fully connected layer of my Resnet, in order to speed up the process.

In the case of a VGG model, we can compute the output of the first convolutional part as follows

x = model_vgg.features(inputs)

But how can I do to extract features from a Resnet?

Thanks in advance

Output:
I guess you can try hacking through the net. I'll use resnet18 as an example:

import torch
from torch import nn
from torchvision.models import resnet18

net = resnet18(pretrained=False)
print(net)

You'll see something like:

....
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=1000, bias=True)

Let's store the Linear layer somewhere, and in its place, put a dummy layer. Then the output of the whole net is actually the output of the conv layers.

x = torch.randn(4,3,32,32) # dummy input of batch size 4 and 32x32 rgb images
out = net(x)
print(out.shape)
>>> 4, 1000 # batch size 4, 1000 default class predictions

store_fc = net.fc      # Save the actual Linear layer for later
net.fc = nn.Identity() # Add a layer which actually does nothing
out = net(x)
print(out.shape)
>>> 4, 512 # batch size 4, 512 features that are the input to the actual fc layer.


Instruction: PyTorch's Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same but my data has been pushed to GPU

Input:
I receive the error:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

yet I made sure to send my data as well as my model on the GPU. Anyone can help ?

My code is:

net.cuda()

'''
print('pyTorch style  summay: ',net)
print('Keras style summary:\n')
summary(net,(2,128,128))
'''

criterion=nn.MSELoss()
#optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
learning_rate = 1e-4
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
print('\nLossFun=',str(criterion))

hf=h5py.File(fn,'r')
print(hf['trainingset'])
tr=np.array(hf['trainingset'])
trtg=np.array(hf['targetsTraining'])
hf.close()

tr = np.moveaxis(tr,3,2)
trtg = np.moveaxis(trtg,3,2)

tr = torch.FloatTensor(tr)
trtg = torch.FloatTensor(trtg)

tr.cuda()
trtg.cuda()

batch_size=16
epochs=2
# run the main training loop
for epoch in range(epochs):
    for batch_idx in range(batch_size):#batch_idx, (data, target) in enumerate(train_loader):
        data =  tr[batch_idx:batch_idx+batch_size-1,:,:,:]
        target =  trtg[batch_idx:batch_idx+batch_size-1,:,:,:]
        data, target = Variable(data), Variable(target)
        # resize data from (batch_size, 1, 28, 28) to (batch_size, 28*28)
        #data = data.view(-1, 28*28)
        optimizer.zero_grad()
        net_out = net(data)
        loss = criterion(net_out, target)
        loss.backward()
        optimizer.step()
        batch_idx += 1
        if batch_idx % log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, batch_idx * len(data), len(train_loader.dataset),100. * batch_idx / len(train_loader), loss.data[0]))

I don't understand why tr.cuda() and trtg.cuda() is not enough! How can I force them to go to cuda?

Output:
Tensor.cuda() doesn't actually change the tensor in place. It creates a copy of the tensor on the GPU and returns the copy. With this in mind you probably want

tr = tr.cuda()
trtg = trtg.cuda()

This is actually different than nn.Module.cuda() which is performed in-place and actually modifies the registered parameters and buffers of the module.


Instruction: How to set the next n elements of a Numpy / PyTorch array to some value?

Input:
If I have a variable side_inputs, with .shape of [48, 161]. I want to set the last n values to some other value.

My other variable, side_input has a .shape of [45, 161].

I want to set the last 45 elements of side_inputs to side_input with:

side_inputs[:45,:] = side_input

But I get an error:

RuntimeError: The expanded size of the tensor (48) must match the existing size (45) at non-singleton dimension 0.  Target sizes: [48, 161].  Tensor sizes: [45, 161]

Output: side_inputs[:-45, :] = side_input


Instruction: Understanding cdist() function

Input:
What does this new_cdist() function actually do? More specifically:

Why is there a sqrt() operation when the AdderNet paper does not use it in its backward propagation equation?
How is needs_input_grad[] used?

def new_cdist(p, eta):
    class cdist(torch.autograd.Function):
        @staticmethod
        def forward(ctx, W, X):
            ctx.save_for_backward(W, X)
            out = -torch.cdist(W, X, p)
            return out

        @staticmethod
        def backward(ctx, grad_output):
            W, X = ctx.saved_tensors
            grad_W = grad_X = None
            if ctx.needs_input_grad[0]:
                _temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                grad_W = torch.matmul(grad_output, _temp)
                # print('before norm: ', torch.norm(grad_W))
                grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W
                print('after norm: ', torch.norm(grad_W))
            if ctx.needs_input_grad[1]:
                _temp1 = torch.unsqueeze(W, 2).expand(W.shape[0], W.shape[1], X.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(X.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                _temp = torch.nn.functional.hardtanh(_temp, min_val=-1., max_val=1.)
                grad_X = torch.matmul(grad_output.transpose(0, 1), _temp)
            return grad_W, grad_X
    return cdist().apply

I mean that it seems to be related to a new type of back-propagation equation and adaptive learning rate.

Output:
Actually, the AdderNet paper does use the sqrt. It is in the adaptive learning rate computation (Algorithm 1, line 6). More specifically, you can see that Eq. 12:

is what is written in this line:

# alpha_l = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W)
grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W

and the sqrt() comes from Eq. 13:

  where k denotes the number of elements in F_l to average the l2-norm, and η is a hyper-parameter to control the learning rate of adder filters.

About your second question: needs_input_grad is just a variable to check if the inputs really require gradients. [0] in this case would refer to W, and [1] to X. You can read more about it here.


Instruction: PyTorch torch.max over multiple dimensions

Input:
Have tensor like :x.shape = [3, 2, 2].

import torch

x = torch.tensor([
    [[-0.3000, -0.2926],[-0.2705, -0.2632]],
    [[-0.1821, -0.1747],[-0.1526, -0.1453]],
    [[-0.0642, -0.0568],[-0.0347, -0.0274]]
])

I need to take .max() over the 2nd and 3rd dimensions. I expect some like this [-0.2632, -0.1453, -0.0274] as output. I tried to use: x.max(dim=(1,2)), but this causes an error.

Output:
Now, you can do this. The PR was merged (Aug 28 2020) and it is now available in the nightly release.
Simply use torch.amax():
import torch

x = torch.tensor([
    [[-0.3000, -0.2926],[-0.2705, -0.2632]],
    [[-0.1821, -0.1747],[-0.1526, -0.1453]],
    [[-0.0642, -0.0568],[-0.0347, -0.0274]]
])

print(torch.amax(x, dim=(1, 2)))

# Output:
# >>> tensor([-0.2632, -0.1453, -0.0274])

Original Answer
As of today (April 11, 2020), there is no way to do .min() or .max() over multiple dimensions in PyTorch. There is an open issue about it that you can follow and see if it ever gets implemented. A workaround in your case would be:
import torch

x = torch.tensor([
    [[-0.3000, -0.2926],[-0.2705, -0.2632]],
    [[-0.1821, -0.1747],[-0.1526, -0.1453]],
    [[-0.0642, -0.0568],[-0.0347, -0.0274]]
])

print(x.view(x.size(0), -1).max(dim=-1))

# output:
# >>> values=tensor([-0.2632, -0.1453, -0.0274]),
# >>> indices=tensor([3, 3, 3]))

So, if you need only the values: x.view(x.size(0), -1).max(dim=-1).values.
If x is not a contiguous tensor, then .view() will fail. In this case, you should use .reshape() instead.

Update August 26, 2020
This feature is being implemented in PR#43092 and the functions will be called amin and amax. They will return only the values. This is probably being merged soon, so you might be able to access these functions on the nightly build by the time you're reading this :) Have fun.


Instruction: RuntimeError: Unknown device when trying to run AlbertForMaskedLM on colab tpu

Input:
I am running the following code on colab taken from the example here: https://huggingface.co/transformers/model_doc/albert.html#albertformaskedlm

import os
import torch
import torch_xla
import torch_xla.core.xla_model as xm

assert os.environ['COLAB_TPU_ADDR']

dev = xm.xla_device()

from transformers import AlbertTokenizer, AlbertForMaskedLM
import torch

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForMaskedLM.from_pretrained('albert-base-v2').to(dev)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1

data = input_ids.to(dev)

outputs = model(data, masked_lm_labels=data)
loss, prediction_scores = outputs[:2]

I haven't done anything to the example code except move input_ids and model onto the TPU device using .to(dev). It seems everything is moved to the TPU no problem as when I input data I get the following output: tensor([[    2, 10975,    15,    51,  1952,    25, 10901,     3]], device='xla:1')

However when I run this code I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-f756487db8f7> in <module>()
      1
----> 2 outputs = model(data, masked_lm_labels=data)
      3 loss, prediction_scores = outputs[:2]

9 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py in forward(self, hidden_states, attention_mask, head_mask)
    277         attention_output = self.attention(hidden_states, attention_mask, head_mask)
    278         ffn_output = self.ffn(attention_output[0])
--> 279         ffn_output = self.activation(ffn_output)
    280         ffn_output = self.ffn_output(ffn_output)
    281         hidden_states = self.full_layer_layer_norm(ffn_output + attention_output[0])

RuntimeError: Unknown device

Anyone know what's going on?

Output:
Solution is here: https://github.com/pytorch/xla/issues/1909

Before calling model.to(dev), you need to call xm.send_cpu_data_to_device(model, xm.xla_device()):

model = AlbertForMaskedLM.from_pretrained('albert-base-v2')
model = xm.send_cpu_data_to_device(model, dev)
model = model.to(dev)

There are also some issues with getting the gelu activation function ALBERT uses to work on the TPU, so you need to use the following branch of transformers when working on TPU: https://github.com/huggingface/transformers/tree/fix-jit-tpu

See the following colab notebook (by https://github.com/jysohn23) for full solution: https://colab.research.google.com/gist/jysohn23/68d620cda395eab66289115169f43900/getting-started-with-pytorch-on-cloud-tpus.ipynb


Instruction: How to apply image transform to a list of images and maintain the right dimensions?

Input:
I'm using the Omniglot dataset, which is a set of 19,280 images, each which is 105 x 105 (grayscale).

I defined a custom Dataset class with the following transform:

class OmniglotDataset(Dataset):

    def __init__(self, X, transform=None):
        self.X = X
        self.transform = transform

    def __len__(self):
        return self.X.shape[0]

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        img = self.X[idx]
        if self.transform:
            img = self.transform(img)
        return img

img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

X_train.shape
(19280, 105, 105)
train_dataset = OmniglotDataset(X_train, transform=img_transform)

When I index a single image, it returns the right dimensions:

train_dataset[0].shape
torch.Size([1, 105, 105])

But when I index several images, it returns the dimensions in the wrong order (I expect 3 x 105 x 105):

train_dataset[[1,2,3]].shape
torch.Size([105, 3, 105])

Output:
You got the error because try apply transformation of single image to list:

A more convenient way to get a batch of any size is to use Dataloader:

from torch.utils.data import DataLoader
from torchvision import datasets, transforms

img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

omniglot = datasets.Omniglot(root='./data', background=True, download=True, transform = img_transform)

data_loader = DataLoader(omniglot, shuffle=False, batch_size = 8)
for image_batch in data_loader:
  # now image_batch contain first eight samples
  print(image_batch.shape) # torch.Size([8, 1, 105, 105])
  break

If you really need to get images in arbitrary order:

from operator import itemgetter

indexes = [1,3,5]
selected_samples = itemgetter(*b)(omniglot)


Instruction: Should we train the original data point when we do data augmentation?

Input:
I am confused about the definition of data augmentation. Should we train the original data points and the transformed ones or just the transformed? If we train both, then we will increase the size of the dataset while the second approach won't.

I got this question when using the function RandomResizedCrop.

'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),

If we resize and crop some of the dataset randomly, we don't actually increase the size of the dataset for data augmentation. Is that correct? Or data augmentation just requires the change/modification of original dataset rather than increase the size of it?

Thanks.

Output:
By definition, or at least the influential paper AlexNet from 2012 that popularized data augmentation in computer vision, increases the size of the training set. Hence the word augmentation. Go ahead and have a look at Section 4.1 from the AlexNet paper. But, here is the gist of it, which I'm quoting from the paper:

  The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations. The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random 224 × 224 patches (and their horizontal reflections) from the 256×256 images and training our network on these extracted patches. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent.

As for the specific implementation, it depends on your use case and most importantly the size of your training data. If you're short in quantity in the latter case, you should consider training on original and transformed images, by sufficiently taking care to preserve the labels.


Instruction: cant see android app after ./gradlew installDebug on device

Input:
I have recently started learning to build android apps, and I am trying to follow the instructions from the Pytorch website, which says:

git clone https://github.com/pytorch/android-demo-app.git
cd HelloWorldApp
./gradlew installDebug

I have my device plugged in and I ran the above and got:

BUILD SUCCESSFUL in 8s
26 actionable tasks: 1 executed, 25 up-to-date

But.. I cannot find the application on the phone..
I am baffled as to why this is.

I would REALLY appreciate any directions to troubleshoot this - spent the last two hours on this!

Output: The app is being installed normally, but not automatically opened after installation. You just need to find it amongst other apps on your device/emulator.


Instruction: what does padding_idx do in nn.embeddings()

Input:
I'm learning pytorch and
I'm wondering what does the padding_idx attribute do in torch.nn.Embedding(n1, d1, padding_idx=0)?
I have looked everywhere and couldn't find something I can get.
Can you show example to illustrate this?

Output:
As per the docs, padding_idx pads the output with the embedding vector at padding_idx (initialized to zeros) whenever it encounters the index.

What this means is that wherever you have an item equal to padding_idx, the output of the embedding layer at that index will be all zeros.

Here is an example:
Let us say you have word embeddings of 1000 words, each 50-dimensional ie num_embeddingss=1000, embedding_dim=50. Then torch.nn.Embedding works like a lookup table (lookup table is trainable though):

emb_layer = torch.nn.Embedding(1000,50)
x = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
y = emb_layer(x)

y will be a tensor of shape 2x4x50. I hope this part is clear to you.

Now if I specify padding_idx=2, ie

emb_layer = torch.nn.Embedding(1000,50, padding_idx=2)
x = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
y = emb_layer(x)

then output will still be 2x4x50 but the 50-dim vector at (1,2)  and (2,3) will be all zeros since x[1,2] and x[2,3] values are 2 which is equal to the padding_idx.
You can think of it as 3rd word in the lookup table (since lookup table would be 0-indexed) is not being used for training.


Instruction: Choosing the learning_rate using fastai's learn.lr_find()

Input:
I am going over this Heroes Recognition ResNet34 notebook published on Kaggle.

The author uses fastai's learn.lr_find() method to find the optimal learning rate.

Plotting the loss function against the learning rate yields the following figure:

It seems that the loss reaches a minimum for 1e-1, yet in the next step the author passes 1e-2 as the max_lr in fit_one_cycle in order to train his model:

learn.fit_one_cycle(6,1e-2)

Why use 1e-2 over 1e-1 in this example? Wouldn't this only make the training slower?

Output:
The idea for a learning rate range test as done in lr_find comes from this paper by Leslie Smith: https://arxiv.org/abs/1803.09820  That has a lot of other useful tuning tips; it's worth studying closely.
In lr_find, the learning rate is slowly ramped up (in a log-linear way).  You don't want to pick the point at which loss is lowest; you want to pick the point at which it is dropping fastest per step (=net is learning as fast as possible).  That does happen somewhere around the middle of the downward slope or 1e-2, so the guy who wrote the notebook has it about right.  Anything between 0.5e-2 and 3e-2 has roughly the same slope and would be a reasonable choice; the smaller values would correspond to a bit slower learning (=more epochs needed, also less regularization) but with a bit less risk of reaching a plateau too early.
I'll try to add a bit of intuition about what is happening when loss is the lowest in this test, say learning rate=1e-1.  At this point, the gradient descent algorithm is taking large steps in the direction of the gradient, but loss is not decreasing.  How can this happen?  Well, it would happen if the steps are consistently too large.  Think of trying to get into a well (or canyon) in the loss landscape.  If your step size is larger than the size of the well, you can consistently step over it every time and end up on the other side.
This picture from a nice blog post by Jeremy Jordan shows it visually:

In the picture, it shows the gradient descent climbing out of a well by taking too large steps (maybe lr=1+0 in your test).  I think this rarely happens exactly like that unless lr is truly excessive; more likely, the well is in a relatively flat landscape, and the gradient descent can step over it, not being able to get into the well in the first place.  High-dimensional loss landscapes are hard to visualize, and may be very irregular, but in a sense the lr_find test is looking for the scale of the typical features in the landscape and then picking a learning rate that gives you a step which is similar sized but a bit smaller.


Instruction: I am installing the Detecto library in anaconda environment but it fails while importing

Input:
As a source, I apply the guide here:
Object Detection

I applied the pip3 install detecto command as shown here and it was successfully installed. But when I run python and try to run the detecto command, I get an error. How can I solve it?

Operation:

Output: I don't think you can import detecto directly. Use from detecto import <lib> . Try from detecto.core import Model and see if it works in your terminal


Instruction: How do I convert 'the array saved as string to csv' back to float array?

Input:
I had to merge a lot of files (containing word embeddings and othe real valued vectors) based on some common attributes so I used Pandas DataFrame and saved the intermediate files as csv.

Currently I have a dataframe whose columns look something like this:

I want to merge all last 4 columns (t-1embedding1a,t-1embedding7b,t-2embedding1a,t-2embedding7b) into a single vector to pass to neural network.

I planned to iterate over the current dataframe and take 4 temporary tensors with value of each column and concatenate and write to new dataframe.
However torch.tensor doesn't work as it says:

torch_tensor = torch.tensor(final['t-1embedding1a'].astype(float).values)

could not convert string to float: '[-6.12873614e-01 -5.58319509e-01 -9.73452032e-01  3.66993636e-01\n

I also tried np.fromstring() but the original values are lost in this case.

Sorry, if the question is unnecessarily complicated, I am a newbie to pytorch. Any help is appreciated!

Output:
First of all, the data type for columns with "t-lembeddingXX" is string that look like "[-6.12873614e-01 -5.58319509e-01 -9.73452032e-01 3.66993636e-01]". You have to convert them to a list of float.

final["t-lembeddingXX"] = final["t-lembeddingXX"].apply(lambda x : [float(x) for x in x.replace("[", "").replace ("]", "").split()])

Then, you have to check that each list of final.loc[i,"t-lembeddingXX"] has the same lengths.

If I haven't mistaken, you want to merge the 4 columns into one verctor.

all_values = list(df["t-lembeddingX1"]) + list(df["t-lembeddingX2"]) + list(df["t-lembeddingX3"]) + list(df["t-lembeddingX4"])
# there is sureliy a better way

Then pass to tensor:

torch_tensor = torch.tensor(all_values)

------------

Finally, I advise you to take a look at the function of torch.cat. You can convert each column to a vector and then use this function to concatenate them together.


Instruction: test accuracy fluctuate even train and test are always same

Input:
Even though my train, validation and test set are always the same, test accuracy fluctuate. The only reason I can think of is weight initialization. I am using PyTorch and I guess they use advance initialization technique (kaiming initialization).

What might be the reason for accuracy fluctuation even though train, validation and test data are the same?

Output:
In addition to weight initialisation, dropout between layers also involves randomness and these can lead to different results on running again.

These random numbers are generally based on a seed value and fixing it can help reproduce the results. You can take a look here on how to fix seed value.


Instruction: How to solve size mismatch error in Python (Pytorch)

Input:
I am currently working on multivariate linear regression using PyTorch and I am getting the following error, I did search a lot about this error and the only thing I got to know is that there is a size mismatch between data and labels. But how to solve this error. Please help me or show me the right way to solve this problem.

size mismatch, m1: [824 x 1], m2: [8 x 8]

import torch
import torch.nn as nn
import numpy as np

Xtr = np.loadtxt("TrainData.csv")
Ytr = np.loadtxt("TrainLabels.csv")

X_train = torch.FloatTensor(Xtr)
Y_train = torch.FloatTensor(Ytr)

#### MODEL ARCHITECTURE ####

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = torch.nn.Linear(8,8)
        self.lin2 = torch.nn.Linear(8,1)

    def forward(self, x):
        x = self.lin2(x)
        y_pred = self.linear(x)
        return y_pred

model = Model()

loss_func = nn.MSELoss(size_average=False)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
#print(len(list(model.parameters())))
def count_params(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

### TRAINING
for epoch in range(2):
    y_pred = model(X_train)

    loss = loss_func(y_pred, Y_train)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    count = count_params(model)
    print(count)

test_exp = torch.FloatTensor([[6.0]])

Output:
It looks like the order of operations in your forward pass is incorrect. The short answer is swap them as shown below. More context on the various shapes below.

    def forward(self, x):
        x = self.lin2(x)
        y_pred = self.linear(x)
        return y_pred

Should be:

    def forward(self, x):
        x = self.linear(x)
        y_pred = self.lin2(x)
        return y_pred

Assuming that you have 8 features and some batch size N your input data to the forward pass will have size (N x 8). After you pass it through lin2 it will have shape (N x 1).  The linear node expects an input with shape (N x 8) but it's getting (N x 1) hence the error.


Instruction: Should Decoder Prediction Be Detached in PyTorch Training?

Input:
Hi guys I have recently started to use PyTorch for my research that needs the encoder-decoder framework. PyTorch's tutorials on this are wonderful, but there's a little problem: when training the decoder without teacher forcing, which means the prediction of the current time step is used as the input to the next, should the prediction be detached?

In this PyTorch tutorial, detach is used (decoder_input = topi.squeeze().detach()  # detach from history as input
), but it is not the case in this one (top1 = output.max(1)[1]; output = (trg[t] if teacher_force else top1)).

Both tutorials are RNN-based, so I am not sure about Transformer-based architectures. Would be grateful if someone could point out which one is the better practice :).

Output:
Yes, you should detach it. Detaching a tensor removes it from the computational graph, so it's no longer tracked in respect to the gradient calculations, which is exactly what you want. Since the previous token can be seen as a constant defining the starting point, it will be discarded after one time step. However if you don't detach it, it will still be hanging around since it's tracked in the computational graph, which consumes unnecessary memory.

Realistically, the memory overhead is usually rather small, so you would only notice it if you have a lot of time steps and are at the upper limit of your GPU memory usage. Just regard it as a micro-optimisation.

There are instances where you absolutely need to detach a tensor to avoid unwanted backpropagation, but that generally happens when the same input is used in two different models, since backward consumes the graph by default and if two different backpropagations try to go through the same path it won't be available anymore and fail.


Instruction: How can we provide the output of a Linear layer to a Conv2D in PyTorch?

Input:
I am building an Autoencoder where I need to encode an image into a latent representation of length 100. I am using the following architecture for my model.

        self.conv1 = nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size=3)
        self.conv2 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3,stride=2)
        self.conv3 = nn.Conv2d(in_channels=64,out_channels=128,kernel_size=3,stride=2)

        self.linear = nn.Linear(in_features=128*30*30,out_features=100)

        self.conv1_transpose = nn.ConvTranspose2d(in_channels=128,out_channels=64,kernel_size=3,stride=2,output_padding=1)
        self.conv2_transpose = nn.ConvTranspose2d(in_channels=64,out_channels=32,kernel_size=3,stride=2,output_padding=1)
        self.conv3_transpose = nn.ConvTranspose2d(in_channels=32,out_channels=3,kernel_size=3,stride=1)

Is there any way I could give my Linear layer's output to a Conv2D or a ConvTranspose2D layer so that I can reconstruct my image? The output is restored if I remove the Linear layer. I want to know how I can reconstruct my image keeping the Linear layer

Any help would be appreciated. Thanks!

Output:
You could use another linear layer:

self.linear2 = nn.Linear(in_features=100, out_features=128*30*30)

And then reshape the output into a 3D volume and pass it into your de-convolution layers.


Instruction: how to initialize Tensor for torch.cat

Input:
import torch

#Y_pred = ?

for xi in X_iter:
    y_pred = net(xi).argmax(dim=1)
    Y_pred = torch.cat([Y_pred, y_pred])

How do you initialize this tensor, or is there a better way to write it？

Output:
You could do this instead:

Y_pred = torch.cat([net(xi).argmax(dim=1) for xi in X_iter])


Instruction: Resize torch tensor channels

Input:
I have a torch tensor with 3 channels, and I want it to be 1 channel (all other dimensions should stay the same).
So if my current dimensions are torch.Size([6, 3, 512, 512]) I want it to be torch.Size([6, 1, 512, 512])

How can I do that?

Output:
Does this solve your problem?
a = torch.ones(6, 3, 512, 512)
b = a[:, 0:1, :, :]
print(b.size()) # torch.Size([6, 1, 512, 512])


Instruction: Torch optimisers with different scaled parameters

Input:
I am trying to optimise parameter values using a torch optimiser but the parameters are on vastly different scales. i.e., one parameter has values in the thousands while others are between 0 and 1. For example in this made up case there are two parameters - one has an optimal value of 0.1 and the other an optimal value of 20. How can I modify this code so it applies a sensible learning rate to each parameter say 1e-3 and 0.1?

import torch as pt
# Objective function
def f(x, y):
    return (10 - 100 * x) ** 2 + (y - 20) ** 2
# Optimal parameters
print("Optimal value:", f(0.1, 20))
# Initial parameters
hp = pt.Tensor([1, 10])
print("Initial value", f(*hp))
# Optimiser
hp.requires_grad = True
optimizer = pt.optim.Adam([hp])
n = 5
for i in range(n):
    optimizer.zero_grad()
    loss = f(*hp)
    loss.backward()
    optimizer.step()
hp.requires_grad = False
print("Final parameters:", hp)
print("Final value:", f(*hp))

Output:
torch.optim.Optimizer class accepts a list of dictionaries in the params argument as the parameter groups. In each dictionary, you need to define params and other arguments used for this parameter group. If you do not provide a specific argument in the dictionary, the original arguments passed to the Optimizer will be used instead. Refer to the official documentation for more information.

Here is the updated code:

import torch as pt

# Objective function
def f(x, y):
    return (10 - 100 * x) ** 2 + (y - 20) ** 2

# Optimal parameters
print("Optimal value:", f(0.1, 20))
# Initial parameters
hp = pt.Tensor([1]), pt.Tensor([10])
print("Initial value", f(*hp))
# Optimiser
for param in hp:
    param.requires_grad = True
# eps and betas are shared between the two groups

optimizer = pt.optim.Adam([{"params": [hp[0]], "lr": 1e-3}, {"params": [hp[1]], "lr": 0.1}])
# optimizer = pt.optim.Adam([{"params": [hp[0]], "lr": 1}, {"params": [hp[1]], "lr": 2.2}])

n = 5
for i in range(n):
    optimizer.zero_grad()
    loss = f(*hp)
    loss.backward()
    optimizer.step()
for param in hp:
    param.requires_grad = False
print("Final parameters:", hp)
print("Final value:", f(*hp))

Try using {"lr": 1} and {"lr": 2.2} for the first and second parameters, respectively. It will result in the final value of 19.9713.


Instruction: AWS EC2 Deep Learning instance cuda 3.0

Input:
I just launched (and paid for) the Deep Learning AMI (Ubuntu 18.04) Version 27.0 (ami-0dbb717f493016a1a) instance type g2.2xlarge.  I activated

for PyTorch with Python3 (CUDA 10.1 and Intel MKL) ____________source activate pytorch_p36

When I run my pytorch network I see a warning

/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/cuda/__init__.py:134: UserWarning:
    Found GPU0 GRID K520 which is of cuda capability 3.0.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability that we support is 3.5.

Is this real?

This is my code to put my neural net on the gpu

if torch.cuda.is_available():
        device = torch.device("cuda:0")  # you can continue going on here, like cuda:1 cuda:2....etc.
        print("Running on the GPU")
    else:
        device = torch.device("cpu")
        print("Running on the CPU")

    net = Net(image_height, image_width)
    net.to(device)

Output:
I had to use a g3s.xlarge instance.  I guess the g2 instances use older GPUs.

Also I had to make num_workers=0 on my dataloaders following this https://discuss.pytorch.org/t/oserror-errno-12-cannot-allocate-memory-but-memory-usage-is-actually-normal/56027.

And this is another pytorch gotcha https://stackoverflow.com/a/51606286/3614578 when adding tensors to a device.


Instruction: Reduce the dimension of a tensor using max-pooling layer

Input:
My question is very simple:

How can I reduce the dimension of a list or a tensor using max-pooling layer to 512 elements in the list:

I'm trying the following code:

    input_ids = tokenizer.encode(question, text)
    print(input_ids) # input_ids is a list of 700 elements
    m = nn.AdaptiveMaxPool1d(512)
    input_ids = m(torch.tensor([[input_ids]])) # convert the list to tensor and apply max-pooling layer

But I get the following error:

RuntimeError: "adaptive_max_pool2d_cpu" not implemented for 'Long'

So, please help to figure out where is the error

Output:
The problem is with your input_ids. You are passing a tensor of type long to AdaptiveMaxPool1d, just convert it to float.

    input_ids = tokenizer.encode(question, text)
    print(input_ids) # input_ids is a list of 700 elements
    m = nn.AdaptiveMaxPool1d(512)
    input_ids = m(torch.tensor([[input_ids]]).float()) #


Instruction: PyTorch LSTM for multiclass classification: TypeError: '<' not supported between instances of 'Example' and 'Example'

Input:
I am trying to modify the code in this Tutorial to adapt it to a multiclass data (I have 55 distinct classes). An error is triggered and I am uncertain of the root cause. The changes I made to this tutorial have been annotated in same-line comments.

One of two solutions would satisfy this questions:

(A) Help identifying the root cause of the error, OR

(B) A boilerplate script for multiclass classification using PyTorch LSTM

import spacy
import torchtext
from torchtext import data
import re

TEXT = data.Field(tokenize = 'spacy', include_lengths = True)
LABEL = data.LabelField(dtype = torch.float)
fields = [(None,None),('text', TEXT), ('wage_label', LABEL)]

train_torch, test_torch = data.TabularDataset.splits(path='/Users/jdmoore7/Desktop/Python Projects/560_capstone/',
                                            format='csv',
                                            train='train_text_target.csv',
                                            test='test_text_target.csv',
                                            fields=fields,
                                            skip_header=True)

import random
train_data, valid_data = train_torch.split(random_state = random.seed(SEED))

MAX_VOCAB_SIZE = 25_000

TEXT.build_vocab(train_data,
                 max_size = MAX_VOCAB_SIZE,
                 vectors = "glove.6B.100d",
                 unk_init = torch.Tensor.normal_)

LABEL.build_vocab(train_data)

BATCH_SIZE = 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_torch),
    batch_size = BATCH_SIZE,
    sort_within_batch = True,
    device = device)

import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,
                 bidirectional, dropout, pad_idx):

        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx = pad_idx)

        self.rnn = nn.LSTM(embedding_dim,
                           hidden_dim,
                           num_layers=n_layers,
                           bidirectional=bidirectional,
                           dropout=dropout)

        self.fc = nn.Linear(hidden_dim * 2, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text, text_lengths):

        #text = [sent len, batch size]

        embedded = self.dropout(self.embedding(text))

        #embedded = [sent len, batch size, emb dim]

        #pack sequence
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)

        packed_output, (hidden, cell) = self.rnn(packed_embedded)

        #unpack sequence
        output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)

        #output = [sent len, batch size, hid dim * num directions]
        #output over padding tokens are zero tensors

        #hidden = [num layers * num directions, batch size, hid dim]
        #cell = [num layers * num directions, batch size, hid dim]

        #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
        #and apply dropout

        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))

        #hidden = [batch size, hid dim * num directions]

        return self.fc(hidden)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = len(LABEL.vocab) ### changed from previous value (1)
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = RNN(INPUT_DIM,
            EMBEDDING_DIM,
            HIDDEN_DIM,
            OUTPUT_DIM,
            N_LAYERS,
            BIDIRECTIONAL,
            DROPOUT,
            PAD_IDX)

import torch.optim as optim
optimizer = optim.Adam(model.parameters())

criterion = nn.CrossEntropyLoss() # Previously: criterion = nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)

def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    #round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float() #convert into float for division
    acc = correct.sum() / len(correct)
    return acc
def train(model, iterator, optimizer, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        text, text_lengths = batch.text

        predictions = model(text, text_lengths).squeeze(1)

        loss = criterion(predictions, batch.label)

        acc = binary_accuracy(predictions, batch.label)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):

    epoch_loss = 0
    epoch_acc = 0

    model.eval()

    with torch.no_grad():

        for batch in iterator:

            text, text_lengths = batch.text

            predictions = model(text, text_lengths).squeeze(1)

            loss = criterion(predictions, batch.label)

            acc = binary_accuracy(predictions, batch.label)

            epoch_loss += loss.item()
            epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

All the above ran smoothly, it's the next code block which triggers the error:

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()

    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut2-model.pt')

    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-888-c1b298b1eeea> in <module>
      7     start_time = time.time()
      8
----> 9     train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
     10     valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
     11

<ipython-input-885-9a57198441ec> in train(model, iterator, optimizer, criterion)
      6     model.train()
      7
----> 8     for batch in iterator:
      9
     10         optimizer.zero_grad()

~/opt/anaconda3/lib/python3.7/site-packages/torchtext/data/iterator.py in __iter__(self)
    140         while True:
    141             self.init_epoch()
--> 142             for idx, minibatch in enumerate(self.batches):
    143                 # fast-forward if loaded from state
    144                 if self._iterations_this_epoch > idx:

~/opt/anaconda3/lib/python3.7/site-packages/torchtext/data/iterator.py in pool(data, batch_size, key, batch_size_fn, random_shuffler, shuffle, sort_within_batch)
    284     for p in batch(data, batch_size * 100, batch_size_fn):
    285         p_batch = batch(sorted(p, key=key), batch_size, batch_size_fn) \
--> 286             if sort_within_batch \
    287             else batch(p, batch_size, batch_size_fn)
    288         if shuffle:

TypeError: '<' not supported between instances of 'Example' and 'Example'

Lastly, the PyTorch forum has an issue opened for this error, however, the code that produced it is not similar so I understand that to be a separate issue.

Output:
The BucketIterator sorts the data to make batches with examples of similar length to avoid having too much padding. For that it needs to know what the sorting criterion is, which should be the text length. Since it is not fixed to a specific data layout, you can freely choose which field it should use, but that also means you must provide that information to sort_key.

In your case, there are two possible fields, text and wage_label, and you want to sort it based on the length of the text.

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_torch),
    batch_size = BATCH_SIZE,
    sort_within_batch = True,
    sort_key = lambda x: len(x.text),
    device = device)

You might be wondering why it worked in the tutorial but doesn't in your example. The reason is that if sort_key is not specified, it defers it to the underlying dataset. In the tutorial they used the IMDB dataset, which defines the sort_key to be x.text. Your custom dataset did not define that, so you need to specify it manually.


Instruction: Speed of L2 Regularization on Pytorch

Input:
I'm trying to manually implement L2 regularisation and a couple of its variations in a neural network. What I'm doing is the following:

for name, param in model.state_dict():
    if 'weight' in name:
        l2_reg += torch.sum(param**2)

loss = cross_entropy(outputs, labels) + 0.0001*l2_reg

Is this equivalent to adding 'weight_decay = 0.0001' inside my optimizer? i.e.:

torch.optim.SGD(model.parameters(), lr=learning_rate , momentum=0.9, weight_decay = 0.0001)

My problem is that I thought they were equivalent, but the manual procedure is about 100x slower than adding 'weight_decay = 0.0001'. Why is that? How can I fix it?

Note that I need to also implement my own variation of L2 regularization, so just adding 'weight_decay = 0.0001' won't help.

Output:
You can check PyTorch implementation of SGD to get some tips and base off of that code.

There are a few things going on which should speed up your custom regularization.
Below is a cleaned version (a little pseudo-code, refer to original) of the parts we are interested in:

for p in group['params']:
    if p.grad is None:
        continue
    d_p = p.grad.data
    if weight_decay != 0:
        d_p.add_(weight_decay, p.data)

    p.data.add_(-group['lr'], d_p)

return loss

BTW. It seems your implementation is mathematically sound (correct me if I missed anything) and equivalent to PyTorch but will be slow indeed.

Modify only gradient

Please notice you perform regularization explicitly during forward pass. This takes a lot of time, more or less because:

take parameters and iterate over them
take it to the power of 2
sum all of them
add to variable containing all previous parameters (all this while creating graph dynamically and creating new nodes).

What pytorch does is it only focuses on backward pass as that's all is needed. This is pretty handy because:

parameters have to be loaded and iterated over once anyway during corrections performed by optimizer (in your case they are taken out twice)
no power of 2 because gradient of w**2 is simply 2*w (2 is further left out and L2 is often expressed as 1/2 * w **2 to make it simpler and a little faster)
no accumulation and creation of additional graph nodes

Essentially, this line:

d_p.add_(weight_decay, p.data)

Modifies the gradient adding p.data (weight) multiplied by weight_decay all done in-place (notice d_p.add_), which is all you have to do to perform L2 regularization.

Finally this line:

p.data.add_(-group['lr'], d_p)

Updates weights with gradient (modified by weight decay) using standard SGD formula (once again, in-place to be as fast as possible, at least on Python level).

Your own implementation

I would advise you to follow similar logic for your own regularization if you want to make it faster.

You can copy PyTorch implementation of SGD and only change this one relevant line. This would also gives you functionality of PyTorch optimizer in case you need it in your experiments.

For L1 regularization (|w| instead of w**2) you would have to calculate the derivative of it (which is 1 for positive case, -1 for negative and undefined for 0 (we can't have that so it should be zero)).

With that in mind we can write the weight_decay like this:

if weight_decay != 0:
    d_p.add_(weight_decay, torch.sign(p.data))

torch.sign returns 1 for positive values and -1 for negative and 0 for... yeah, 0.

Hope this helps, exact implementation is left for you (hit me up in the comments in case you have any questions or troubles).


Instruction: EOFError: Ran out of input, using torch.load()

Input:
I saw this error being posted a lot and often it was due to the file not being closed properly after opening. But since I'm using the integrated torch.load() function, I'm not sure what I could do different.

First the saving part:

    torch.save({
            'model_state_dict': agent.dqn.state_dict(),
             ...
            'loss_history': agent.losshistory
            }, modelpath)

and here the loading part, where I also get the error message:

if os.path.exists(modelpath):
    checkpoint = torch.load(modelpath)
    agent.dqn.load_state_dict(checkpoint['model_state_dict'])
    ...
    agent.losshistory = checkpoint['loss_history']

and here the error:

Traceback (most recent call last):
  File "c:/Users/levin/Desktop/programming/main.py", line 33, in <module>
    checkpoint = torch.load(modelpath)
  File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\levin\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\serialization.py", line 702, in _legacy_load
    result = unpickler.load()
EOFError: Ran out of input

One more thing I want to mention is that I used this exact code several times without a problem. I can't remember changing anything that could have caused the error.

Output: According to this thread it seems to raise an exception when reading an empty file, so please check the size of the document before reading it and post a response if it is not solved.


Instruction: Why doesn't torch.autograd compute the gradient in this case?

Input:
Why doesn't torch.autograd compute the gradient in this case?

import torch
x = torch.tensor([1., 2., ], requires_grad=True)
y = torch.tensor([x[0] + x[1], x[1] - x[0], ], requires_grad=True)
z = y[0] + y[1]
z.backward()
x.grad

Output is a blank line (None). The same occurs for x[0].grad. Why?

PS: In retrospect, I realize the motivation for making y a tensor with requires_grad was so I could examine its own gradient. I learned that one must use retain_grad for that here: Why does autograd not produce gradient for intermediate variables?

Output:
When you use torch.tensor for y, it just uses the values of x to initialize the tensor, the gradient chain is lost.

This works:

x = torch.tensor([1., 2., ], requires_grad=True)
y = [x[0] + x[1], x[1] - x[0], ]
z = y[0] + y[1]
z.backward()
x.grad

The result is tensor([0., 2.])


Instruction: Getting the following error while using scikit-image to read images "AttributeError: 'PngImageFile' object has no attribute '_PngImageFile__frame' "

Input:
I am using scikit-image to load a random image from a folder. OpenCV is being used for operations later on..

Code is as follows (only relevant parts included)

import imageio
import cv2 as cv
import fileinput
from collections import Counter

from data.apple_dataset import AppleDataset
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
from torchvision.transforms import functional as F

import utility.utils as utils
import utility.transforms as T

from PIL import Image
import skimage.io
from skimage.viewer import ImageViewer
from matplotlib import pyplot as plt
%matplotlib inline

 APPLE_IMAGE_PATH = r"__mypath__\samples\apples\images"

# Load a random image from the images folder
FILE_NAMES = next(os.walk(APPLE_IMAGE_PATH))[2]
random_apple_in_folder = os.path.join(APPLE_IMAGE_PATH, random.choice(FILE_NAMES))
apple_image = skimage.io.imread(random_apple_in_folder)
apple_image_cv = cv.imread(random_apple_in_folder)
apple_image_cv = cv.cvtColor(apple_image_cv, cv.COLOR_BGR2RGB)

Error is as follows

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-9575eed18f18> in <module>
     11 FILE_NAMES = next(os.walk(APPLE_IMAGE_PATH))[2]
     12 random_apple_in_folder = os.path.join(APPLE_IMAGE_PATH, random.choice(FILE_NAMES))
---> 13 apple_image = skimage.io.imread(random_apple_in_folder)
     14 apple_image_cv = cv.imread(random_apple_in_folder)

    AttributeError: 'PngImageFile' object has no attribute '_PngImageFile__frame'

How do i proceed from here? What should i change???

Output:
This is a bug in Pillow 7.1.0. You can upgrade Pillow with pip install -U pillow. See this bug report for more information:

https://github.com/scikit-image/scikit-image/issues/4548


Instruction: Confusion in Pre-processing text for Roberta Model

Input:
I want to apply Roberta model for  text similarity. Given a pair of sentences,the input should be in the format <s> A </s></s> B </s>. I figure out two possible ways to generate the input ids namely

a)

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')

list1 = tokenizer.encode('Very severe pain in hands')

list2 = tokenizer.encode('Numbness of upper limb')

sequence = list1+[2]+list2[1:]

In this case, sequence is [0, 12178, 3814, 2400, 11, 1420, 2, 2, 234, 4179, 1825, 9, 2853, 29654, 2]

b)

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')

list1 = tokenizer.encode('Very severe pain in hands', add_special_tokens=False)

list2 = tokenizer.encode('Numbness of upper limb', add_special_tokens=False)

sequence = [0]+list1+[2,2]+list2+[2]

In this case, sequence is [0, 25101, 3814, 2400, 11, 1420, 2, 2, 487, 4179, 1825, 9, 2853, 29654, 2]

Here 0 represents <s> token and 2 represents </s> token. I'm not sure which is the correct way to encode the given two sentences for calculating sentence similarity using Roberta model.

Output:
The easiest way is probably to directly use the provided function by HuggingFace's Tokenizers themselves, namely the text_pair argument in the encode function, see here. This allows you to directly feed in two sentences, which will be giving you the desired output:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
sequence = tokenizer.encode(text='Very severe pain in hands',
                            text_pair='Numbness of upper limb',
                            add_special_tokens=True)

This is especially convenient if you are dealing with very long sequences, as the encode function automatically reduces your lengths according to the truncaction_strategy argument. You obviously don't have to worry about this, if it is only short sequences.

Alternatively, you can also make use of the more explicit build_inputs_with_special_tokens() function of the RobertaTokenizer, specifically, which could be added to your example like so:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')

list1 = tokenizer.encode('Very severe pain in hands', add_special_tokens=False)
list2 = tokenizer.encode('Numbness of upper limb', add_special_tokens=False)

sequence = tokenizer.build_inputs_with_special_tokens(list1, list2)

Note that in that case, you have to generate the sequences list1 and list2 still without any special tokens, as you have already done correctly.


Instruction: Python : convert a float encoded as a byte string (from PyTorch) to an int

Input:
I have converted an output from PyTorch using .detach().numpy() which produces this kind of data :

b'0.06722715'

which is a byte type according to type() from Python. How can I convert this to an integer ?

Output:
Try this (explaination in code comments). You can convert 0.06 to an integer but you'll get a zero. Did you mean float?

#byte
b = b'0.06722715'
# to string
s = b.decode()
# to float
f = float(s)
# to integer
i = int(f)
print("Float", f)
print("Integer", i)

or simply

be_float = float(b.decode())
print (be_float)


Instruction: Model behaves differently after saving and loading

Input:
I want to use torch.save() to save a trained model for inference. However, with either torch.load_state_dict() or torch.load(), I can't get the saved model. The loss computed by the loaded model is just different from the loss computed by the saved model.

The relevant Libraries:

import numpy as np
import torch
import torch.nn as nn
from torch.autograd import Variable
from torch.nn import functional as F

The model:

class nn_block(nn.Module):

    def __init__(self, feats_dim):

        super(nn_block, self).__init__()

        self.linear = nn.Linear(feats_dim, feats_dim)
        self.bn = nn.BatchNorm1d(feats_dim)
        self.softplus1 = nn.Softplus()
        self.softplus2 = nn.Softplus()

    def forward(self, rep_mat):

        transformed_mat = self.linear(rep_mat)
        transformed_mat = self.bn(transformed_mat)
        transformed_mat = self.softplus1(transformed_mat)
        transformed_mat = self.softplus2(transformed_mat + rep_mat)

        return transformed_mat

class test_nn(nn.Module):

    def __init__(self, in_feats, feats_dim, num_conv, num_classes):

        super(test_nn, self).__init__()

        self.linear1 = nn.Linear(in_feats, feats_dim)
        self.convs = [nn_block(feats_dim) for _ in range(num_conv)]
        self.linear2 = nn.Linear(feats_dim, num_classes)
        self.softmax = nn.Softmax()

    def forward(self, rep_mat):

        h = self.linear1(rep_mat)
        for conv_func in self.convs:
            h = conv_func(h)
        h = self.linear2(h)
        h = self.softmax(h)

        return h

Train, save, and reload a model:

# fake a classification task
num_classes = 2; input_dim = 8
one = np.random.multivariate_normal(np.zeros(input_dim),np.eye(input_dim),20)
two = np.random.multivariate_normal(np.ones(input_dim),np.eye(input_dim),20)
inputs = np.concatenate([one, two], axis=0)
labels = np.concatenate([np.zeros(20), np.ones(20)])

inputs = Variable(torch.Tensor(inputs))
labels = torch.LongTensor(labels)

# build a model
net = test_nn(input_dim, 5, 2, num_classes)
optimizer = torch.optim.Adam(net.parameters(), lr=0.01)
net.train()

losses = []
best_score = 1e10
for epoch in range(25):

    preds = net(inputs)
    loss = F.cross_entropy(preds, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    state_dict = {'state_dict': net.state_dict()}

    if loss.item()-best_score<-1e-4:
        # save only parameters
        torch.save(state_dict, 'model_params.torch')
        # save the whole model
        torch.save(net, 'whole_model.torch')
    best_score = np.min([best_score, loss.item()])

    losses.append(loss.item())

net_params = test_nn(input_dim, 5, 2, num_classes)
net_params.load_state_dict(torch.load('model_params.torch')['state_dict'])
net_params.eval()
preds_params = net_params(inputs)
loss_params = F.cross_entropy(preds_params, labels)
print('reloaded params %.4f %.4f' % (loss_params.item(), np.min(losses)))

net_whole = torch.load('whole_model.torch')
net_whole.eval()
preds_whole = net_whole(inputs)
loss_whole = F.cross_entropy(preds_whole, labels)
print('reloaded whole %.4f %.4f' % (loss_whole.item(), np.min(losses)))

As you can see by running the code, the losses computed by the two loaded models are different, while the two loaded models are exactly the same. Not just the two losses are different, they are also different from the loss computed by the best model that was saved in the first place.

Why this can happen?

Output:
The state dict contains every parameter (nn.Parameter
) and buffer (similar to parameter, but which should not be trained/optimised) that has been registered on the module and all of its submodules. Everything else will not be included in that state dict.

Your test_nn module uses a list for convs, therefore it is not included in the state dict:

self.convs = [nn_block(feats_dim) for _ in range(num_conv)]

Not only are they not contained in the state dict, they are also not visible to net.parameters(), which means they are not trained/optimised at all.

To register the modules from the list you can wrap it in nn.ModuleList, which is a module that acts like a list, while correctly registering the modules it contains:

self.convs = nn.ModuleList([nn_block(feats_dim) for _ in range(num_conv)])

With that change both models produce the same result.

Since you are calling the convs modules sequentially in the for-loop (output of one module is the input of the next), you may consider using nn.Sequential, which you can call directly instead of having to use the for-loop. Sequencing is used a lot and it just makes it a little simpler, for example if you want to replace the sequence of modules with a single module, you don't need to change anything in the forward method.

  Not just the two losses are different, they are also different from the loss computed by the best model that was saved in the first place.

When you are training, you calculate the loss for the current input (batch) and then you optimise the parameters based on that input. This means your parameters differ from the ones used to calculate the loss. Because you are saving the model after that, it will also have a different loss (the one that would occur in the next iteration).

preds = net(inputs)
# Calculating the loss of the current model
loss = F.cross_entropy(preds, labels)

optimizer.zero_grad()
loss.backward()
# Updating the model's parameters based on the loss
optimizer.step()

# State of the model after it has been updated
state_dict = {'state_dict': net.state_dict()}

# Comparing the loss from BEFORE the update
# But saving the model from AFTER the update
if loss.item()-best_score<-1e-4:
    # save only parameters
    torch.save(state_dict, 'model_params.torch')
    # save the whole model
    torch.save(net, 'whole_model.torch')

It's important to evaluate the model after the updates have been made. For this reason a validation set should be used, which is run after each epoch to assess the model's accuracy.


Instruction: How to output the accuracy alongside with the loss when training the MNIST dataset after each epoch

Input:
from __future__ import print_function
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
from tensorflow.examples.tutorials.mnist import input_data
import torch.optim as optim
import tensorflow.python.util.deprecation as deprecation
deprecation._PRINT_DEPRECATION_WARNINGS = False
import matplotlib.pyplot as plt
%matplotlib inline
from plot import plot_loss_and_acc

mnist = input_data.read_data_sets("MNIST_data", one_hot=False)
batch_size = 250
epoch_num = 10
lr = 0.0001
disp_freq = 20

def next_batch(train=True):
    # Reads the next batch of MNIST images and labels and returns them

    if train:
        batch_img, batch_label = mnist.train.next_batch(batch_size)
    else:
        batch_img, batch_label = mnist.test.next_batch(batch_size)

    batch_label = torch.from_numpy(batch_label).long()  # convert the numpy array into torch tensor
    batch_label = Variable(batch_label)         # create a torch variable

    batch_img = torch.from_numpy(batch_img).float()     # convert the numpy array into torch tensor
    batch_img = Variable(batch_img)             # create a torch variable
    return batch_img, batch_label

class MLP(nn.Module):
    def __init__(self, n_features, n_classes):
        super(MLP, self).__init__()
        self.layer1 = nn.Linear(n_features, 128)
        self.layer2 = nn.Linear(128, 128)
        self.layer3 = nn.Linear(128, n_classes)

    def forward(self, x, training=True):
        # a neural network with 2 hidden layers
        # x -> FC -> relu -> dropout -> FC -> relu -> dropout -> FC -> output
        x = F.relu(self.layer1(x))
        x = F.dropout(x, 0.5, training=training)
        x = F.relu(self.layer2(x))
        x = F.dropout(x, 0.5, training=training)
        x = self.layer3(x)
        return x

    def predict(self, x):
        # a function to predict the labels of a batch of inputs
        x = F.softmax(self.forward(x, training=False))
        return x

    def accuracy(self, x, y):
        # a function to calculate the accuracy of label prediction for a batch of inputs
        #   x: a batch of inputs
        #   y: the true labels associated with x
        prediction = self.predict(x)
        maxs, indices = torch.max(prediction, 1)
        acc = 100 * torch.sum(torch.eq(indices.float(), y.float()).float())/y.size()[0]
        print(acc.data)
        return acc.data

# define the neural network (multilayer perceptron)
net = MLP(784, 10)

# calculate the number of batches per epoch
batch_per_ep = mnist.train.num_examples // batch_size

# define the loss (criterion) and create an optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=lr)
print(' ')
print("__________Training__________________")

xArray = []
yLoss = []
yAcc = []
for ep in range(epoch_num):  # epochs loop
    for batch_n in range(batch_per_ep):  # batches loop
        features, labels = next_batch()

        # Reset gradients
        optimizer.zero_grad()

        # Forward pass
        output = net(features)
        loss = criterion(output, labels)
        # Backward pass and updates
        loss.backward()                 # calculate the gradients (backpropagation)
        optimizer.step()                # update the weights
        if batch_n % disp_freq == 0:
            print('epoch: {} - batch: {}/{} '.format(ep, batch_n, batch_per_ep))
    xArray.append(ep)
    yLoss.append(loss.data)
    #yAcc.append(acc.data)

    print('loss: ', loss.data)
    print('__________________________________')

# test the accuracy on a batch of test data
features, labels = next_batch(train=False)
print("Result")
print('Test accuracy: ', net.accuracy(features, labels))
print('loss: ', loss.data)
accuracy = net.accuracy(features, labels)

#Loss Plot

# plotting the points
plt.plot(xArray, yLoss)

# naming the x axis
plt.xlabel('epoch')
# naming the y axis
plt.ylabel('loss')

# giving a title to my graph
plt.title('Loss Plot')

# function to show the plot
plt.show()

#Accuracy Plot

# plotting the points
plt.plot(xArray, yAcc)

# naming the x axis
plt.xlabel('epoch')
# naming the y axis
plt.ylabel(' accuracy')

# giving a title to my graph
plt.title('Accuracy Plot ')

# function to show the plot
plt.show()

I want to display the accuracy of my training dataset. I have managed to display and plot the loss but I didn't manage to do it for accuracy. I know I am missing 1 or 2 lines of code and I don't know how to do it.

I mean if I can display the accuracy alongside each epoch like the loss I can do the plotting myself.

Output:
Hi replace this code print('epoch: {} - batch: {}/{} '.format(ep, batch_n, batch_per_ep)) with
print('epoch: {} - batch: {}/{} - accuracy: {}'.format(ep, batch_n, batch_per_ep, net.accuracy(features,labels)))

Hope this helps.


Instruction: Using annoy with Torchtext for nearest neighbor search

Input:
I'm using Torchtext for some NLP tasks, specifically using the built-in embeddings.

I want to be able to do a inverse vector search: Generate a noisy vector, find the vector that is closest to it, then get back the word that is "closest" to the noisy vector.

From the torchtext docs, here's how to attach embeddings to a built-in dataset:

from torchtext.vocab import GloVe
from torchtext import data

embedding = GloVe(name='6B', dim=100)

# Set up fields
TEXT = data.Field(lower=True, include_lengths=True, batch_first=True)
LABEL = data.Field(sequential=False, is_target=True)

# make splits for data
train, test = datasets.IMDB.splits(TEXT, LABEL)

# build the vocabulary
TEXT.build_vocab(train, vectors=embedding, max_size=100000)
LABEL.build_vocab(train)

# Get an example vector
embedding.get_vecs_by_tokens("germany")

Then we can build the annoy index:

from annoy import AnnoyIndex

num_trees = 50

ann_index = AnnoyIndex(embedding_dims, 'angular')

# Iterate through each vector in the embedding and add it to the index
for vector_num, vector in enumerate(TEXT.vocab.vectors):
    ann_index.add_item(vector_num, vector) # Here's the catch: will vector_num correspond to torchtext.vocab.Vocab.itos?

ann_index.build(num_trees)

Then say I want to retrieve a word using a noisy vector:

# Get an existing vector
original_vec = embedding.get_vecs_by_tokens("germany")
# Add some noise to it
noise = generate_noise_vector(ndims=100)
noisy_vector = original_vec + noise
# Get the vector closest to the noisy vector
closest_item_idx = ann_index.get_nns_by_vector(noisy_vector, 1)[0]
# Get word from noisy item
noisy_word = TEXT.vocab.itos[closest_item_idx]

My question comes in for the last two lines above: The ann_index was built using enumerate over the embedding object, which is a Torch tensor.

The [vocab][2] object has its own itos list that given an index returns a word.

My question is this: Can I be certain that the order in which words appear in the itos list, is the same as the order in TEXT.vocab.vectors? How can I map one index to the other?

Output:
  Can I be certain that the order in which words appear in the itos list, is the same as the order in TEXT.vocab.vectors?

Yes.

The Field class will always instantiate a Vocab object (source), and since you are passing the pre-trained vectors to TEXT.build_vocab, the Vocab constructor will call the load_vectors function.

if vectors is not None:
    self.load_vectors(vectors, unk_init=unk_init, cache=vectors_cache)

In the load_vectors, the vectors are filled by enumerating the words in the itos.

for i, token in enumerate(self.itos):
    start_dim = 0
    for v in vectors:
        end_dim = start_dim + v.dim
        self.vectors[i][start_dim:end_dim] = v[token.strip()]
        start_dim = end_dim
    assert(start_dim == tot_dim)

Therefore, you can be certain that itos and vectors will have the same order.


Instruction: TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first (fastai)

Input:
I am following the code here:

https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-with-resnet50-oversampling

However, during the metrics calculation, I am getting the following error:

File "main.py", line 50, in <module>
 learn.fit_one_cycle(4,max_lr = 2e-3)
...
File "main.py", line 39, in quadratic_kappa
    return torch.tensor(cohen_kappa_score(torch.argmax(y_hat,1), y, weights='quadratic'),device='cuda:0')
...
File "/pfs/work7/workspace/scratch/ul_dco32-conda-0/conda/envs/resnet50/lib/python3.8/site-packages/torch/tensor.py", line 486, in __array__
    return self.numpy()
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Here are the metrics and the model:

def quadratic_kappa(y_hat, y):
    return torch.tensor(cohen_kappa_score(torch.argmax(y_hat,1), y, weights='quadratic'),device='cuda:0')

learn = cnn_learner(data, models.resnet50, metrics = [accuracy,quadratic_kappa])
learn.fit_one_cycle(4,max_lr = 2e-3)

As it is being said in the discussion https://discuss.pytorch.org/t/typeerror-can-t-convert-cuda-tensor-to-numpy-use-tensor-cpu-to-copy-the-tensor-to-host-memory-first/32850/6, I have to bring the data back to cpu. But I am slightly lost how to do it.

I tried to add .cpu() all over the metrics but could not solve it so far.

Output:
I'm assuming that both y and y_hat are CUDA tensors, that means that you need to bring them both to the CPU for the cohen_kappa_score, not just one.

def quadratic_kappa(y_hat, y):
    return torch.tensor(cohen_kappa_score(torch.argmax(y_hat.cpu(),1), y.cpu(), weights='quadratic'),device='cuda:0')
    #                                                        ^^^         ^^^

Calling .cpu() on a tensor that is already on the CPU has no effect, so it's safe to use in any case.


Instruction: Pytorch Custom Optimizer got an empty parameter list

Input:
new here. I am trying to create a custom optimizer in PyTorch, where the backprop takes place in a meta RL policy, with the policy receiving the model parameters, and outputting the desired model parameters. However, I am seeing the above error. My models work fine on Adam and SGD, but not my optimizer.

Code:

class MetaBackProp(torch.optim.Optimizer):
    def __init__(self, params):

        self.param_shape_list = np.array([])
        for param in list(params):
            np.append(self.param_shape_list, list(param.size()))

        pseudo_lr = 1e-4
        pseudo_defaults = dict(lr=pseudo_lr)
        length = 100 #TODO: get shape, flatten, multiply...
        self.policy = AEPolicy(length)
        self.policy_optim = torch.optim.Adam(self.policy.parameters(), lr=pseudo_lr)
        super(MetaBackProp, self).__init__(params, pseudo_defaults)

    def step(self, closure=None):
        params = torch.cat([p.view(-1) for p in self.param_groups])
        self.policy_optim.zero_grad()
        quit()

Traceback:

Traceback (most recent call last):
  File "main.py", line 6, in <module>
    gan = CycleGAN()
  File "/home/ai/Projects_v2/R/cycle_gan.py", line 32, in __init__
    self.discriminator2_optim = MetaBackProp(self.discriminator2.parameters())
  File "/home/ai/Projects_v2/R/lr_schedule.py", line 34, in __init__
    super(MetaBackProp, self).__init__(params, pseudo_defaults)
  File "/home/ai/anaconda3/lib/python3.7/site-packages/torch/optim/optimizer.py", line 46, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list

Output:
You retrieve the parameters with self.discriminator2.parameters(), which returns an iterator. In your constructor you are converting them to a list for the for loop:

for param in list(params):

This consumes the iterator, but you are passing that same iterator to the constructor of the base class, hence it does not contain any parameter at all.

super(MetaBackProp, self).__init__(params, pseudo_defaults)

Instead of passing the iterator, you can use the list you created from the iterator, since the parameters just need to be iterable, which lists are.

# Convert parameters to a list to allow multiple iterations
params = list(params)
for param in params:


Instruction: Add More Metrics to Ray Tune Status Table (Python, PyTorch)

Input:
When running tune.run() on a set of configs to search, is it possible to add more metrics columns (i.e. a, b, etc) to the status table being printed out?

tune.track.log(a=metric1, b=metric2)

will give the following table without columns for the metrics a and b:

== Status ==
Memory usage on this node: 22.1/125.8 GiB
Using FIFO scheduling algorithm.
Resources requested: 1/32 CPUs, 1/4 GPUs, 0.0/65.59 GiB heap, 0.0/22.13 GiB objects
Result logdir: /home/nyxynyx/ray_results/fooba
Number of trials: 4 (3 PENDING, 1 RUNNING)
+--------------+----------+-------+------+-----+
| Trial name   | status   | loc   |   lr |   x |
|--------------+----------+-------+------+-----|
| fooba_00000  | RUNNING  |       | 0.01 |   1 |
| fooba_00001  | PENDING  |       | 0.1  |   1 |
| fooba_00002  | PENDING  |       | 0.01 |   5 |
| fooba_00003  | PENDING  |       | 0.1  |   5 |
+--------------+----------+-------+------+-----+

How can we include a column for every metric that we pass to tune.track.log() other than mean_accuracy?

Using Python 3.7.3 and Ray 0.8.4

Output: Yep! You should be able to do this with a reporter object: https://ray.readthedocs.io/en/latest/tune/api_docs/reporters.html


Instruction: How can I show multiple predictions of the next word in a sentence?

Input:
I am using the GPT-2 pre trained model. the code I am working on will get a sentence and generate the next word for that sentence. I want to print multiple predictions, like the three first predictions with best probabilities!
for example if I put in the sentence "I's an interesting ...."
predictions:  "Books"    "story"    "news"

is there a way I can modify this code to show these predictions instead of one?!

also there are two parts in the code, I do not understand, what is the meaning of the numbers in  (predictions[0, -1, :])?  and why do we put [0] after predictions = output[0]?

import torch
from pytorch_transformers import GPT2Tokenizer, GPT2LMHeadModel

# Load pre-trained model tokenizer (vocabulary)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# Encode a text inputs
text = "The fastest car in the  "
indexed_tokens = tokenizer.encode(text)

# Convert indexed tokens in a PyTorch tensor
tokens_tensor = torch.tensor([indexed_tokens])

# Load pre-trained model (weights)
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Set the model in evaluation mode to deactivate the DropOut modules
model.eval()

# If you have a GPU, put everything on cuda
#tokens_tensor = tokens_tensor.to('cuda')
#model.to('cuda')

# Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor)
    predictions = outputs[0]
    #print(predictions)

# Get the predicted next sub-word
predicted_index = torch.argmax(predictions[0, -1, :]).item()
predicted_text = tokenizer.decode(indexed_tokens + [predicted_index])

# Print the predicted word
#print(predicted_index)
print(predicted_text)

The result for the above code will be :

The fastest car in the world.

Output:
You can use torch.topk as follows:

predicted_indices = [x.item() for x in torch.topk(predictions[0, -1, :],k=3)]


Instruction: Pytorch: AttributeError: 'function' object has no attribute 'copy'

Input:
I am trying to load a model state_dict I trained on Google Colab GPU, here is my code to load the model:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = models.resnet50()
num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, n_classes)
model.load_state_dict(copy.deepcopy(torch.load("./models/model.pth",device)))
model = model.to(device)
model.eval()

Here is the error:

  state_dict = state_dict.copy()

  AttributeError: 'function' object has no attribute 'copy'

Pytorch :

>>> import torch
>>> print (torch.__version__)
1.4.0
>>> import torchvision
>>> print (torchvision.__version__)
0.5.0

Please help I have searched everywhere to no avail

[full error details][1] https://i.stack.imgur.com/s22DL.png

Output:
I am guessing this is what you did by mistake.
You saved the function

torch.save(model.state_dict, 'model_state.pth')

instead of the state_dict()

torch.save(model.state_dict(), 'model_state.pth')

Otherwise, everything should work as expected. (I tested the following code on Colab)

Replace model.state_dict() with model.state_dict to reproduce error

import copy
model = TheModelClass()
torch.save(model.state_dict(), 'model_state.pth')
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model.load_state_dict(copy.deepcopy(torch.load("model_state.pth",device)))


Instruction: Saving vocabulary object from pytorch's torchtext library

Input:
Building a text classification model using pytorch's torchtext . The vocabulary object is in the data.field :

def create_tabularDataset_object(self,csv_path):
   self.TEXT = data.Field(tokenize=self.tokenizer,batch_first=True,include_lengths=True)
   self.LABEL = data.LabelField(dtype = torch.float,batch_first=True)

def get_vocab_with_glov(self,data):
   # initialize glove embeddings
   self.TEXT.build_vocab(data,min_freq=100,vectors = "glove.6B.100d")

After training , when serving the model in production how do i hold the TEXT object ? at prediction time  i need it for indexing the words tokens

[TEXT.vocab.stoi[t] for t in tokenizedׁ_sentence]

am i missing something and it is not necessary to hold  that object ? Do i need any other file other then the model weights ?

Output:
I've found that i can save it as a pkl:
Saving the TEXT.vocab as a pkl worked :

def save_vocab(vocab, path):
    import pickle
    output = open(path, 'wb')
    pickle.dump(vocab, output)
    output.close()

Where

vocab = TEXT.vocab

and reading it as usual.


Instruction: How to partition a neural network into sub-networks in Pytorch?

Input:
I'd like to partition a neural network into two sub-networks using Pytorch. To make things concrete, consider this image:

In 1, I've a 3x4x1 neural network. What I want is, for example during epoch 1, I'd only like to update the weights in the sub-network 1, i.e., the weights that appear in the sub-network 2 must be frozen. Then again, in epoch 2, I'd like to train the weights that appear in sub-network 2 while the rest should be frozen.

How can I do that?

Output:
You can do this easily if your subnet is a subset of layers. That is, you do not need to freeze any partial layers. It is all or nothing.

For your example that would mean dividing the hidden layer into two different 2-node layers. Each would belong to exactly one of the subnetworks, which gets us back to all or nothing.

With that done, you can toggle individual layers using requires_grad. Setting this to False on the parameters will disable training and freeze the weights. To do this for an entire model, sub-model, or Module, you loop through the model.parameters().

For your example, with 3 inputs, 1 output, and a now split 2x2 hidden layer, it might look something like this:

import torch.nn as nn
import torch.nn.functional as F

def set_grad(model, grad):
    for param in model.parameters():
        param.requires_grad = grad

class HalfFrozenModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.hid1 = torch.nn.Linear(3, 2)
        self.hid2 = torch.nn.Linear(3, 2)
        self.out = torch.nn.Linear(4, 1)

    def set_freeze(self, hid1=False, hid2=False):
        set_grad(self.hid1, not hid1)
        set_grad(self.hid2, not hid2)

    def forward(self, inp):
        hid1 = self.hid1(inp)
        hid2 = self.hid2(inp)
        hidden = torch.cat([hid1, hid2], 1)
        return self.out(F.relu(hidden))

Then you can train one half or the other like so:

model = HalfFrozenModel()
model.set_freeze(hid1=True)
# Do some training.
model.set_freeze(hid2=True)
# Do some more training.
# ...

If you happen to use fastai, then there is a concept of layer groups that is also used for this. The fastai documentation goes into some detail about how that works.


Instruction: Expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

Input:
I read this question but it doesnt seem to answer my question :(.

So basically I'm trying to vectorize the game snake so it can run faster.
Here is my code till now:

import torch
import torch.nn.functional as F
device = torch.device("cpu")
class SnakeBoard:
  def __init__(self, board=None):
    if board != None:
      self.channels = board
    else:
      # 0 - Food, 1 - Head, 2 - Body
      self.channels = torch.zeros(1, 3, 15, 17,
                                  device=device)

      # Initialize game channels
      self.channels[:, 0, 7, 12] = 1
      self.channels[:, 1, 7, 5] = 1
      self.channels[:, 2, 7, 2:6] = torch.arange(1, 5)
    self.move()

  def move(self):
    self.channels[:, 2] -= 1
    F.relu(self.channels[:, 2], inplace=True)
    # Up movement test
    F.conv2d(self.channels[:, 1], torch.tensor([[[0,1,0],[0,0,0],[0,0,0]]]), padding=1)

SnakeBoard()

The first dimension in channels represents batch size, second dimension represent the 3 channels of the snake game: food, head, and body, and finally the third and fourth dimensions represent the height and width of the board.

Unfortunately when running the code I get error: Expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

How can I fix that?

Output:
The dimensions of the inputs for the convolution are not correct for a 2D convolution. Let's have a look at the dimensions you're passing to F.conv2d:

self.channels[:, 1].size()
# => torch.Size([1, 15, 17])
torch.tensor([[[0,1,0],[0,0,0],[0,0,0]]]).size()
# => torch.Size([1, 3, 3])

The correct sizes should be

input: (batch_size, in_channels , height, width)
weight: (out_channels, in_channels , kernel_height, kernel_width)

Because your weight has only 3 dimensions, it is considered to be a 1D convolution, but since you called F.conv2d the stride and padding will be tuples and therefore it won't work.

For the input you indexed the second dimension, which selects that particular element across that dimensions and eliminates that dimensions. To keep that dimension you can index it with a slice of just one element.
And for the weight you are missing one dimension as well, which can just be added directly. Also your weight is of type torch.long, since you are only using integers in the tensor creation, but the weight needs to be of type torch.float.

F.conv2d(self.channels[:, 1:2], torch.tensor([[[[0,1,0],[0,0,0],[0,0,0]]]], dtype=torch.float), padding=1)

On a different note, I don't think that convolutions are appropriate for this use case, because you're not using a key property of the convolution, which is to capture the surroundings. Those are just too many unnecessary computations to achieve what you want, most of them are multiplications with 0.

For example, a move up is much easier to achieve by removing the first row and adding a new row of zeros at the end, so everything is shifted up (assuming that the first row is the top and the last row is the bottom of the board).

head = self.channels[:, 1:2]
batch_size, channels, height, width = head.size()
# Take everything but the first row of the head
# Add a row of zeros to the end by concatenating them across the height (dimension 2)
new_head = torch.cat([head[:, :, 1:], torch.zeros(batch_size, channels, 1, width)], dim=2)

# Or if you want to wrap it around the board, it's even simpler.
# Move the first row to the end
wrap_around_head = torch.cat([head[:, :, 1:], head[:, :, 0:1]], dim=2)