Instruction: Full Page Text Recognition Dataset Creation

Input:
I have been reading OCR papers such as this one https://arxiv.org/pdf/1704.08628.pdf , and I am have trouble finding out how these datasets are actually generated.

In the linked paper, they use a regressor to predict the start location (a point) and height of a line of text. Then, based on that starting point and height, a second network performs OCR and end of line detection. I realize this is a very simplified explanation, but it follows that their dataset consists (at least in part) of full page text 'images' annotated with where each line begins, and then a transcription of the text on a given line. Alternatively, they could have just used the lower left point of bounding boxes as the start point and the height of the box as the word height (avoiding the need to re-annotate if the data was previously prepared using bounding boxes).

So how is a dataset like this actually created? Looking at other datasets it seems like there is some software that can create XML files containing the ground truths relevant to each image, can someone point me in the right direction? I've been googling around and finding lots of tools for annotating text with sentiment etc and other tools for annotating images for segmentation (for something like a YOLO network), but I'm coming up empty for the creation something like the Maurdoor dataset used in the linked paper.

Thank you

Output: So after submitting this, the related threads window showed me many threads that my googling did not turn up. This http://www.prima.cse.salford.ac.uk/tools software seems to be what I was looking for, but I would still love to hear other ideas.


Instruction: PyTorch : How to properly create a list of nn.Linear()

Input:
I have created a class that has nn.Module as subclass.

In my class, I have to create N number of linear transformation, where N is given as class parameters.

I therefore proceed as follow :

    self.list_1 = []

    for i in range(N):
        self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias))

In the forward method, i call these matrices (with list_1[i]) and concat the results.

Two things :

1)

Even though I use model.cuda(), these Linear transform are used on cpu and i get the following error :

  RuntimeError: Expected object of type Variable[torch.cuda.FloatTensor] but found type Variable[torch.FloatTensor] for argument #1 'mat2'

I have to do

self.list_1.append(nn.Linear(self.x, 1, bias=mlp_bias).cuda())

This is not required if instead, i do :

self.nn = nn.Linear(self.x, 1, bias=mlp_bias)

and then use self.nn directly.

2)

For more obvious reason, when I print(model) in my main, the Linear matrices in my list arent printed.

Is there any other way. maybe using bmm ? I find it less easy, and i actually want to have my N results separately.

Thank you in advance,

M

Output:
You can use nn.ModuleList to wrap your list of linear layers as explained here

self.list_1 = nn.ModuleList(self.list_1)


Instruction: Input 3 and 1 channel input to the network in pytorch?

Input:
My dataset consists mostly of 3 channel images, but i also have a few 1 channel images,Is it possible to train a network that takes in both 3 channels and 1 channels as inputs?

Any suggestions are welcome,Thanks in advance,

Output:
You can detect the grayscale images by checking the size and apply some transformation to have 3 channels.

It seems to be better to convert images from grayscale to RGB than simply copying the image three times on the channels.

You can do that by cv2.cvtColor(gray_img, cv.CV_GRAY2RGB) if you have opencv-python installed.

If you want a clean implementation you can extend torchvision.transform with a new Transform that does this job automatically.


Instruction: PyTorch - Torchvision - BrokenPipeError: [Errno 32] Broken pipe

Input:
I'm trying to carry out the tutorial named "Training a classifier" with PyTorch.
WHen trying to debug this part of the code :

import matplotlib.pyplot as plt
import numpy as np

# functions to show an image

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

I get this error message :

Files already downloaded and verified Files already downloaded and verified
Files already downloaded and verified Files already downloaded and verified Traceback (most recent call last):
File "<string>", line 1, in <module>
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 277, in
_fixup_main_from_path
    run_name="__mp_main__")
File "D:\Anaconda\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
File "D:\Anaconda\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
File "D:\Anaconda\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
    dataiter = iter(trainloader)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
File "D:\Anaconda\lib\multiprocessing\spawn.py", line 136, in
_check_not_importing_main
    is not going to be frozen to produce an executable.)
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
 Traceback (most recent call last):
 File "d:\Yggdrasil\Programmation\PyTorch\TutorialCIFAR10.py", line 36, in <module>
    dataiter = iter(trainloader)
 File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
 File "D:\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
 File "D:\Anaconda\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
 File "D:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)   File "D:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
 File "D:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 65, in
__init__
    reduction.dump(process_obj, to_child)
 File "D:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
 BrokenPipeError: [Errno 32] Broken pipe

All the previous lines in the tutorial are working perfectly.

Does someone know how to solve this, please ?
Thanks a lot in advance

Output: This doesn't look to be a PyTorch problem. Try executing the code in Jupyter notebooks and other environment troubleshooting.


Instruction: Can't install torch on linux box using pip

Input:
As the title states I am trying to install torch on linux using pip.
I run the command pip install torch==0.3.1

And I get the following output:

Collecting torch==0.3.1
  Could not find a version that satisfies the requirement torch==0.3.1 (from versions: 0.1.2, 0.1.2.post1)
No matching distribution found for torch==0.3.1

Any ideas what the issue might be?

Output:
Try update pip itself, using

pip install --upgrade pip

then,

pip install torch==0.3.1


Instruction: If I'm not specifying to use CPU/GPU, which one is my script using?

Input:
In pytorch, if I'm not writing anything about using CPU/GPU, and my machine supports CUDA (torch.cuda.is_available() == True):

What is my script using, CPU or GPU?
If CPU, what should I do to make it run on GPU? Do I need to rewrite everything?
If GPU, will this script crash if torch.cuda.is_available() == False?
Does this do anything about making the training faster?
I'm aware of Porting PyTorch code from CPU to GPU but this is old. Does this situation change in v0.4 or the upcoming v1.0?

Output:
My way is like this (below pytorch 0.4):

dtype = torch.cuda.float if torch.cuda.is_available() else torch.float
torch.zeros(2, 2, dtype=dtype)

UPDATE pytorch 0.4:

device = torch.device("cuda" if use_cuda else "cpu")
model = MyRNN().to(device)

from PyTorch 0.4.0 Migration Guide.


Instruction: Tips or patterns for reshaping 4D/5D arrays, (videos to frames)

Input:
I find it really hard to visualize reshaping 4D 5D arrays in numpy/pytorch. (I assume both reshape in similar patter, I am using pytorch currently!).

Like suppose I have videos with dimension [N x C x D x H x W]

(num videos x channels video x frames video x height video x width video)

Suppose I want to reshape video into frames as [N x C x H x W], how should I proceed in reshape.

Simple applying x = x.reshape(N*D, C, H, W) doesn't actually do it, it gives wrong order of elements.

Can you help me with how to do this, and any slight of intuition of pattern you used?

  On a sidenote, if i have one video (i.e suppose 1x3x100x256x256 I use :

  the following code approach:

  x = x.squeeze(0).T.reshape((100,3,256,256))[:,:,None,:,:] and it works

  great. Couldnt figure out for more than 1 video.

Thanks!

As per the request :

input = np.random.randn(N,C,D,H,W)
output = np.zeros((N*D,C,H,W))

As per the request, a for loop based code to show what I want

for h in range(N):
    for i in range(D):
        for j in range(C):
            for k in range(H):
                for l in range(W):
                    output[h*D + i,j,k,l] = input[h,j,i,k,l]

Output:
Simply swap the second and third axes, and then merge the new second axis (old third one) with the first one with reshaping -

output = input_array.swapaxes(1,2).reshape(N*D,C,H,W)

We can also use transpose : input_array.transpose(0,2,1,3,4) to get the same swapping axes effect.

For a general intuitive method, please refer to Intuition and idea behind reshaping 4D array to 2D array in NumPy.


Instruction: Pytorch simple text generator not working and loss keeps diverging

Input:
I am new to pytorch and deep learning in general , and am trying to build a simple text generator. For reasons I don't understand, the loss keeps diverging and the model doesn't. Here's the code.

class RNN(nn.Module):
    def __init__(self, embed_size, hidden_size):
        super(RNN, self).__init__()
        self.embeds = nn.Embedding(num_chars, embed_size)
        self.l1 = nn.Linear(embed_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, num_chars)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax()

    def forward(self, inp):
        out = self.embeds(inp)
        out = self.l3(self.relu(self.l2(self.relu(self.l1(out)))))
        return self.softmax(out)

rnn = RNN(10, 50)
optimizer = torch.optim.Adam(rnn.parameters(), lr = 0.002)
criterion = nn.NLLLoss()

def charTensor(x):
    out = torch.zeros(1, num_chars)
    out[0][all_chars.index(x)] = 1
    return out.long()

for epoch in range(5):
    epoch_loss = 0
    rnn.zero_grad()
    for i in range(len(train_str[:400])-1):

        inp = charTensor(train_str[i])
        output = rnn(inp)
        loss = criterion(output, charTensor(train_str[i+1]))
        epoch_loss += loss

        loss.backward(retain_graph=True)
        optimizer.step()

    print("Epoch Loss:", epoch_loss) //epoch loss is always tensor(-399)

first_char = 'c'
inp_t = charTensor(first_char)
fin = first_char
for i in range(10):
    next_t = rnn(inp_t)
    next_char = all_chars[torch.argmax(next_t).numpy()]   //always ends up as 0, which is the char for space
    fin += next_char
    inp_t = charTensor(next_char)
print(fin)    //prints new line

Output:
Are you trying to implement an RNN? Because I see you are naming your model as RNN but the implementation doesn't seem to take signals from previous time steps.
It seems that you are not implementing batches and are training based on inputting 1 character and then backpropagating on that. This is known to cause instability. You may want to iterate over a few characters and accumulated the loss and average it before backpropagating.

In generating text, what you want to do is train the model by having prepared sequence of data for example "the fox jumped over the lazy dog". In a word level prediction, your input would be:

["the","fox","jumped", "over","the","lazy"]

and the target would be:

["fox","jumped", "over","the","lazy", "dog"]

What model does is try to predict the next word given the previous words.
In character level, then simply change the list to each character withing the sentence. That way you will have a model that learns the probability distribution.

For PyTorch specific implementation check here: https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html

Also, you don't need retain_graph=True as it will build up memory. Instead just type:

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()


Instruction: How to do softmax for pixelwise classification

Input:
My goal is to do grey scale image segmentation using pixelwise classification. So I have two labels 0 and 1. I made a network in pytorch which looks like the following.

class Net(nn.Module):

def __init__(self):
    super(Net, self).__init__()

    self.up = nn.Upsample(scale_factor=2, mode='nearest')

    self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
    self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
    self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

def forward(self, x):
    in_size = x.size(0)

    x = F.relu(self.conv11(x))
    x = F.relu(self.conv12(x))
    x = F.relu(self.conv13(x))

    x = F.softmax(x, 2)

    return x

In the last layer I designed the conv13 in such that it produces 2 channels one for each class.

Since I was using the softmax I was expecting that summation of value of same index on 2 separate channel would equal to 1.

For example assume the output image is ( 2{channel}, 4, 4). So I was expecting that

  image[ channel 1 ][0][0] + image[ channel 2 ][0][0] = 1

But the output I get is 0.0015 which is not even close to 1. How can i use the softmax to predict channelwise ?

To check this I used the following code

for batch, data in enumerate(trainloader, 0):
    inputs , labels = data
    inputs, labels = inputs.to(device), labels.to(device)

    optimizer.zero_grad()
    outputs = net(inputs)
    loss = rmse(outputs, labels)
    loss.backward()
    optimizer.step()
    running_loss += loss.item()

    predicted = outputs.data
    predicted = predicted.to('cpu')
    predicted_img = predicted.numpy()

    predicted_img = np.reshape(predicted_img,(2, 4, 4))

    print(predicted_img[0])
    print(predicted_img[1])

Those prints showed this

[[**0.2762002**  0.13305853 0.2510342  0.23114938]
 [0.26812425 0.28500515 0.05682982 0.15851443]
 [0.1640967  0.5409352  0.43547812 0.44782472]
 [0.29157883 0.0410011  0.2566578  0.16251141]]

[[**0.23052207** 0.868455   0.43436486 0.0684725 ]
 [0.18001427 0.02341573 0.0727293  0.2525512 ]
 [0.06587404 0.04974682 0.3773188  0.6559266 ]
 [0.5235896  0.05838248 0.11558701 0.02304965]]

It is clear that the corresponding elements are not summing up to 1 like

  0.2762002 (index 0, 0) + 0.23052207 (index 0, 0) != 1

How can I fix it ?

Output:
Please check last line of my code .. basically your dimension for softmax was wrong.

class Net(nn.Module):

  def __init__(self):
        super(Net, self).__init__()

        self.up = nn.Upsample(scale_factor=2, mode='nearest')

        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)

        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x))

        x = F.softmax(x, 1) #this line is changed

        return x

net = Net()
inputs = torch.rand(1,1,4,4)
out = net (Variable(inputs))
print (out)
out.sum(dim=1)

Hope that helps.


Instruction: How to do CIFAR-10 with PyTorch on CUDA?

Input:
I'm following the CIFAR-10 PyTorch tutorial at this pytorch page , and can't get PyTorch running on the GPU. The code is exactly as in the tutorial.

The error I get is

Traceback (most recent call last):
  File "(file path)/CIFAR10_tutorial.py", line 116, in <module>
   outputs = net(images)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
  File "(file path)/CIFAR10_tutorial.py", line 65, in forward
x = self.pool(F.relu(self.conv1(x)).cuda())
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 491, in __call__
result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)

My CUDA version is 9.0, Pytorch 0.4.0. I have used tensorflow-gpu on the machine, so I know CUDA is set up correctly. Where exactly must I use .cuda() and .to(device) as suggested in the tutorial?

Output:
I'm leaving an answer, in case anyone else is stuck on the same.

First, configure Pytorch to use the GPU if available

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)

Then, in the init function, cast to gpu by calling .cuda() on every element of the NN, e.g.

self.conv1 = nn.Conv2d(3, 24, 5).cuda()
self.pool = nn.MaxPool2d(2, 2).cuda()

If you're not sure about the GPU, call .to(device) on every element.

In the forward(self, x) function, before the steps, I did

x = x.to(device)

Right after net object is created, cast it to device by

net.to(device)

All inputs and labels should be cast to device before any operation is performed on them.

inputs, labels = inputs.to(device), labels.to(device)

I am, skipping writing the entire code as the link has already been mentioned in the question. If there are seem to be a few redundant casts to gpu, they're not breaking anything. I might also put together an ipynb with the changes.


Instruction: Getting multiprocessing lock error when running vizdoom and pytorch program on Windows Subsystem for Linux

Input:
whenever I try to run my  program on WSL, I get the following error. I'm pretty new to pytorch and vizdoom, so I don't know how to solve this problem.

Setup
- Windows 10 x64
- Ubuntu 14 (on WSL)
- Python 2.7.14  (Anaconda 2)
- OpenAI Gym 0.9.5
- Vizdoom 1.1.4
- doom-py 0.0.14
- ppaquette/gym-doom
- pytorch 0.0.12

(doomenv) hybridsyntax@Blacklynx:/mnt/f/_TUTORIALS/ai/doom/code$ python ai.py  > wsl.log
[2018-05-25 18:21:44,354] Making new env: ppaquette/DoomCorridor-v0
[2018-05-25 18:21:44,365] Clearing 2 monitor files from previous run (because force=True was provided)
Assertion 'pthread_mutex_unlock(&m->mutex) == 0' failed at pulsecore/mutex-posix.c:108, function pa_mutex_unlock(). Aborting.
Traceback (most recent call last):
  File "ai.py", line 155, in <module>
    memory.run_steps(200)
  File "/mnt/f/_TUTORIALS/ai/doom/code/experience_replay.py", line 70, in run_steps
    entry = next(self.n_steps_iter) # 10 consecutive steps
  File "/mnt/f/_TUTORIALS/ai/doom/code/experience_replay.py", line 21, in __iter__
    state = self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/monitoring.py", line 39, in _reset
    observation = self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 311, in _reset
    observation = self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/frame_skipping.py", line 33, in _reset
    return self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 283, in _reset
    return self.env.reset(**kwargs)
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/wrappers/time_limit.py", line 49, in _reset
    return self.env.reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym/gym/core.py", line 104, in reset
    return self._reset()
  File "/mnt/f/_TUTORIALS/ai/doom/gym-doom/ppaquette_gym_doom/doom_env.py", line 244, in _reset
    return self._load_level()
  File "/mnt/f/_TUTORIALS/ai/doom/gym-doom/ppaquette_gym_doom/doom_env.py", line 157, in _load_level
    'singleton lock in memory.')
gym.error.Error: [ViZDoomUnexpectedExitException, ViZDoomErrorException] VizDoom exited unexpectedly. This is likely caused by a missing multiprocessing lock. To run VizDoom across multiple processes, you need to pass a lock when you configure the env [e.g. env.configure(lock=my_multiprocessing_lock)], or create and close an env before starting your processes [e.g. env = gym.make("DoomBasic-v0"); env.close()] to cache a singleton lock in memory.
[2018-05-25 18:21:44,696] Finished writing results. You can upload them to the scoreboard via gym.upload('/mnt/f/_TUTORIALS/ai/doom/code/videos')
(doomenv) hybridsyntax@Blacklynx:/mnt/f/_TUTORIALS/ai/doom/code$

Thanks in advance

Output:
Upgrading the Ubuntu on WSL to the latest version (18.04) solved the problem for me.

For me it was running the following commands on WSL.

sudo -S env RELEASE_UPGRADER_NO_SCREEN=1 do-release-upgrade
sudo apt-get update
sudo apt-get upgrade -y


Instruction: Is there a function in google.colab module to close the runtime

Input:
Sometimes when I give run in google.colab I cant stay infront of the computer to manually disconnect from the server when the run is complete and the connection stays on even when my run is complete occupying the node for no reason.

Is there a function in google.colab so that say I can insert the function to close the connection after some epochs? I am looking for something like colab.disconnect() etc...

Output:
import sys
sys.exit()

This will end the runtime, freeing up the GPU.

EDIT: Apparently my last answer doesn't work any more.
The thing to do now is !kill -9 -1.


Instruction: How do I split a custom dataset into training and test datasets?

Input:
import pandas as pd
import numpy as np
import cv2
from torch.utils.data.dataset import Dataset

class CustomDatasetFromCSV(Dataset):
    def __init__(self, csv_path, transform=None):
        self.data = pd.read_csv(csv_path)
        self.labels = pd.get_dummies(self.data['emotion']).as_matrix()
        self.height = 48
        self.width = 48
        self.transform = transform

    def __getitem__(self, index):
        pixels = self.data['pixels'].tolist()
        faces = []
        for pixel_sequence in pixels:
            face = [int(pixel) for pixel in pixel_sequence.split(' ')]
            # print(np.asarray(face).shape)
            face = np.asarray(face).reshape(self.width, self.height)
            face = cv2.resize(face.astype('uint8'), (self.width, self.height))
            faces.append(face.astype('float32'))
        faces = np.asarray(faces)
        faces = np.expand_dims(faces, -1)
        return faces, self.labels

    def __len__(self):
        return len(self.data)

This is what I could manage to do by using references from other repositories.
However, I want to split this dataset into train and test.

How can I do that inside this class? Or do I need to make a separate class to do that?

Output:
Using Pytorch's SubsetRandomSampler:

import torch
import numpy as np
from torchvision import datasets
from torchvision import transforms
from torch.utils.data.sampler import SubsetRandomSampler

class CustomDatasetFromCSV(Dataset):
    def __init__(self, csv_path, transform=None):
        self.data = pd.read_csv(csv_path)
        self.labels = pd.get_dummies(self.data['emotion']).as_matrix()
        self.height = 48
        self.width = 48
        self.transform = transform

    def __getitem__(self, index):
        # This method should return only 1 sample and label
        # (according to "index"), not the whole dataset
        # So probably something like this for you:
        pixel_sequence = self.data['pixels'][index]
        face = [int(pixel) for pixel in pixel_sequence.split(' ')]
        face = np.asarray(face).reshape(self.width, self.height)
        face = cv2.resize(face.astype('uint8'), (self.width, self.height))
        label = self.labels[index]

        return face, label

    def __len__(self):
        return len(self.labels)

dataset = CustomDatasetFromCSV(my_path)
batch_size = 16
validation_split = .2
shuffle_dataset = True
random_seed= 42

# Creating data indices for training and validation splits:
dataset_size = len(dataset)
indices = list(range(dataset_size))
split = int(np.floor(validation_split * dataset_size))
if shuffle_dataset :
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, val_indices = indices[split:], indices[:split]

# Creating PT data samplers and loaders:
train_sampler = SubsetRandomSampler(train_indices)
valid_sampler = SubsetRandomSampler(val_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                           sampler=train_sampler)
validation_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                                sampler=valid_sampler)

# Usage Example:
num_epochs = 10
for epoch in range(num_epochs):
    # Train:
    for batch_index, (faces, labels) in enumerate(train_loader):
        # ...


Instruction: Do I need to make multiple instances of a neural network in PyTorch to test multiple loss functions?

Input:
I have written out a neural network in PyTorch and I would like to compare the results of two different loss functions on this one network

Should I go about making two different instances of the network and test one loss function per network like this

network_w_loss_1 = ANN().cuda()
network_w_loss_2 = ANN().cuda()

crit_loss_1 = loss_1()
crit_loss_2 = loss_2()

opt_loss_1 = optim.SGD('params')
opt_loss_2 = optim.SGD('params')

for epoch in range(num_epochs):
    for i, dat in enumerate(data_loader):
        #unpack data
        opt_loss_1.zero_grad()
        opt_loss_2.zero_grad()
        output1 = network_w_loss_1('params')
        output2 = network_w_loss_2('params')
        los_1 = crit_loss_1(output1)
        los_2 = crit_loss_2(output2)
        los_1.backward()
        los_2.backward()
        opt_loss_1.step()
        opt_loss_2.step()

or can I get away with doing this?

network = ANN().cuda()

crit_loss_1 = loss_1()
crit_loss_2 = loss_2()

opt = optim.SGD('params')

for epoch in range(num_epochs):
    for i, dat in enumerate(data_loader):
        #unpack data
        opt.zero_grad()
        output1 = network('params')
        output2 = network('params')
        los_1 = crit_loss_1(output1)
        los_2 = crit_loss_2(output2)
        los_1.backward()
        los_2.backward()
        opt.step()

I am using Python 3.6.5 and PyTorch 0.4.0

Output: You have to make 2 different instances. Otherwise you are just training one network alternating between 2 losses (both losses would update its parameters).


Instruction: Issue with torch.cuda() function

Input:
I'm new to PyTorch. It is working when I use gpu running my program with TensorFlow. But this is a problem with PyTorch.
I search a lot, but cannot find any useful answer. Who can help me?
Error:

My environment:

windows10 64bit
python3.6
cuda9.0
cudnn64
gpu:GTX965m

Output:
As you can see for yourself in the release notes here, the PyTorch developers have decided to deprecate compute capability 3.0 and 5.0 devices from their builds starting with PyTorch 0.3.1.
Your device is a compute 5.0 device and is, therefore, not supported in the most recent versions of PyTorch.
You can read about what alternatives are open to you here (basically use an older version of PyTorch, or build your own from source with support for your GPU).


Instruction: Fast way to multiple 3D tensors of shape (1, 1, 256) and (10, 1, 256) in PyTorch and Numpy

Input:
I am trying to adapt the seq2seq model for my own task, https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

I have two tensors at the decoder stage

rnn_output: (1, 1, 256)       # time_step x batch_size x hidden_dimension
encoder_inputs: (10, 1, 256)  # seq_len   x batch_size x hidden_dimension

They should be multipied to get attention score (before softmax) of shape

attn_score: (10, 1, 1)

What's the best way to do so? The notebook seems to use a for loop, is there are better via a matrix multiplication kind of operation?

Output:
Example using torch.bmm():

import torch
from torch.autograd import Variable
import numpy as np

seq_len = 10
rnn_output = torch.rand((1, 1, 256))
encoder_outputs = torch.rand((seq_len, 1, 256))

# As computed in the tutorial:
attn_score = Variable(torch.zeros(seq_len))
for i in range(seq_len):
    attn_score[i] = rnn_output.squeeze().dot(encoder_outputs[i].squeeze())
    # note: the code would fail without the "squeeze()". I would assume the tensors in
    # the tutorial are actually (,256) and (10, 256)

# Alternative using batched matrix multiplication (bmm) with some data reformatting first:
attn_score_v2 = torch.bmm(rnn_output.expand(seq_len, 1, 256),
                          encoder_outputs.view(seq_len, 256, 1)).squeeze()

# ... Interestingly though, there are some numerical discrepancies between the 2 methods:
np.testing.assert_array_almost_equal(attn_score.data.numpy(),
                                     attn_score_v2.data.numpy(), decimal=5)
# AssertionError:
# Arrays are not almost equal to 5 decimals
#
# (mismatch 30.0%)
#  x: array([60.32436, 69.04288, 72.04784, 70.19503, 71.75543, 67.45459,
#        63.01708, 71.70189, 63.07552, 67.48799], dtype=float32)
#  y: array([60.32434, 69.04287, 72.0478 , 70.19504, 71.7554 , 67.4546 ,
#        63.01709, 71.7019 , 63.07553, 67.488  ], dtype=float32)


Instruction: Implementing Luong Attention in PyTorch

Input:
I am trying to implement the attention described in Luong et al. 2015 in PyTorch myself, but I couldn't get it work. Below is my code, I am only interested in the "general" attention case for now. I wonder if I am missing any obvious error. It runs, but doesn't seem to learn.

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p

        self.embedding = nn.Embedding(
            num_embeddings=self.output_size,
            embedding_dim=self.hidden_size
        )
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size, self.hidden_size)
        # hc: [hidden, context]
        self.Whc = nn.Linear(self.hidden_size * 2, self.hidden_size)
        # s: softmax
        self.Ws = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        gru_out, hidden = self.gru(embedded, hidden)

        # [0] remove the dimension of directions x layers for now
        attn_prod = torch.mm(self.attn(hidden)[0], encoder_outputs.t())
        attn_weights = F.softmax(attn_prod, dim=1) # eq. 7/8
        context = torch.mm(attn_weights, encoder_outputs)

        # hc: [hidden: context]
        out_hc = F.tanh(self.Whc(torch.cat([hidden[0], context], dim=1)) # eq.5
        output = F.log_softmax(self.Ws(out_hc), dim=1) eq. 6

        return output, hidden, attn_weights

I have studied the attention implemented in

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

and

https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

The first one isn't the exact attention mechanism I am looking for. A major disadvantage is that its attention depends on the sequence length (self.attn = nn.Linear(self.hidden_size * 2, self.max_length)), which could be expensive for long sequences.
The second one is more similar to what's described in the paper, but still not the same as there is not tanh. Besides, it is really slow after updating it to latest version of pytorch (ref). Also I don't know why it takes the last context (ref).

Output:
This version works, and it follows the definition of Luong Attention (general), closely. The main difference from that in the question is the separation of embedding_size and hidden_size, which appears to be important for training after experimentation. Previously, I made both of them the same size (256), which creates trouble for learning, and it seems that the network could only learn half the sequence.

class EncoderRNN(nn.Module):
    def __init__(self, input_size, embedding_size, hidden_size,
                 num_layers=1, bidirectional=False, batch_size=1):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.bidirectional = bidirectional
        self.batch_size = batch_size

        self.embedding = nn.Embedding(input_size, embedding_size)

        self.gru = nn.GRU(embedding_size, hidden_size, num_layers,
                          bidirectional=bidirectional)

    def forward(self, input, hidden):
        embedded = self.embedding(input).view(1, 1, -1)
        output, hidden = self.gru(embedded, hidden)
        return output, hidden

    def initHidden(self):
        directions = 2 if self.bidirectional else 1
        return torch.zeros(
            self.num_layers * directions,
            self.batch_size,
            self.hidden_size,
            device=DEVICE
        )

class AttnDecoderRNN(nn.Module):
    def __init__(self, embedding_size, hidden_size, output_size, dropout_p=0):
        super(AttnDecoderRNN, self).__init__()
        self.embedding_size = embedding_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p

        self.embedding = nn.Embedding(
            num_embeddings=output_size,
            embedding_dim=embedding_size
        )
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(embedding_size, hidden_size)
        self.attn = nn.Linear(hidden_size, hidden_size)
        # hc: [hidden, context]
        self.Whc = nn.Linear(hidden_size * 2, hidden_size)
        # s: softmax
        self.Ws = nn.Linear(hidden_size, output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        gru_out, hidden = self.gru(embedded, hidden)

        attn_prod = torch.mm(self.attn(hidden)[0], encoder_outputs.t())
        attn_weights = F.softmax(attn_prod, dim=1)
        context = torch.mm(attn_weights, encoder_outputs)

        # hc: [hidden: context]
        hc = torch.cat([hidden[0], context], dim=1)
        out_hc = F.tanh(self.Whc(hc))
        output = F.log_softmax(self.Ws(out_hc), dim=1)

        return output, hidden, attn_weights


Instruction: Why pytorch isn't minimizing x*x for me?

Input:
I expect x to converge to 0, which is minimum of x*x. But this doesn't happen. What am I doing wrong in this small sample code:

import torch
from torch.autograd import Variable
tns = torch.FloatTensor([3])
x = Variable(tns, requires_grad=True)
z = x*x
opt = torch.optim.Adam([x], lr=.01, betas=(0.5, 0.999))
for i in range(3000):
    z.backward(retain_graph=True) # Calculate gradients
    opt.step()
    print(x)

Output:
The problem you have is that you don't zero the gradients when you are calculating each loop. Instead, by setting retain_graph=True and not calling opt.zero_grad() at each step of the loop you are actually adding the gradients calculated to ALL previous gradients calculated. So instead of taking a step in gradient descent, you are taking a step with respect to all accumulated gradients which is certainly NOT what you want.

You should instead make sure to call opt.zero_grad() at the beginning of your loop, and move the z=x*x inside your loop so that you don't have to retain_graph.

I made these slight modifications:

import torch
from torch.autograd import Variable
tns = torch.FloatTensor([3])
x = Variable(tns, requires_grad=True)
opt = torch.optim.Adam([x], lr=.01, betas=(0.5, 0.999))
for i in range(3000):
    opt.zero_grad()
    z = x*x
    z.backward() # Calculate gradients
    opt.step()
    print(x)

And my final x is 1e-25.


Instruction: PyTorch equivalent of index_add_ that takes the maximum instead

Input:
In PyTorch, the index_add_ method of a Tensor does a summation using a provided index tensor:

idx = torch.LongTensor([0,0,0,0,1,1])
child = torch.FloatTensor([1, 3, 5, 10, 8, 1])
parent = torch.FloatTensor([0, 0])
parent.index_add_(0, idx, child)

The first four child values sum into parent[0] and the next two go into parent[1], so the result is tensor([ 19.,   9.])

However, I need to do index_max_ instead, which doesn't exist in the API. Is there a way to do it efficiently (without having to loop or allocate more memory)? One (bad) loop solution would be:

for i in range(max(idx)+1):
    parent[i] = torch.max(child[idx == i])

This produces the desired result of tensor([ 10.,   8.]), but very slowly.

Output:
A solution playing with the indices:

def index_max(child, idx, num_partitions):
    # Building a num_partition x num_samples matrix `idx_tiled`:
    partition_idx = torch.range(0, num_partitions - 1, dtype=torch.long)
    partition_idx = partition_idx.view(-1, 1).expand(num_partitions, idx.shape[0])
    idx_tiled = idx.view(1, -1).repeat(num_partitions, 1)
    idx_tiled = (idx_tiled == partition_idx).float()
    # i.e. idx_tiled[i,j] == 1 if idx[j] == i, else 0

    parent = idx_tiled * child
    parent, _ = torch.max(parent, dim=1)
    return parent

Benchmarking:

import timeit

setup = '''
import torch

def index_max_v0(child, idx, num_partitions):
    parent = torch.zeros(num_partitions)
    for i in range(max(idx) + 1):
        parent[i] = torch.max(child[idx == i])
    return parent

def index_max(child, idx, num_partitions):

    # Building a num_partition x num_samples matrix `idx_tiled`
    # containing for each row indices of
    partition_idx = torch.range(0, num_partitions - 1, dtype=torch.long)
    partition_idx = partition_idx.view(-1, 1).expand(num_partitions, idx.shape[0])
    idx_tiled = idx.view(1, -1).repeat(num_partitions, 1)
    idx_tiled = (idx_tiled == partition_idx).float()

    parent = idx_tiled * child
    parent, _ = torch.max(parent, dim=1)
    return parent

idx = torch.LongTensor([0,0,0,0,1,1])
child = torch.FloatTensor([1, 3, 5, 10, 8, 1])
num_partitions = torch.unique(idx).shape[0]

'''
print(min(timeit.Timer('index_max_v0(child, idx, num_partitions)', setup=setup).repeat(5, 1000)))
# > 0.05308796599274501
print(min(timeit.Timer('index_max(child, idx, num_partitions)', setup=setup).repeat(5, 1000)))
# > 0.024736385996220633


Instruction: Variational Autoencoder gives same output image for every input mnist image when using KL divergence

Input:
When not using KL divergence term, the VAE reconstructs mnist images almost perfectly but fails to generate new ones properly when provided with random noise.

When using KL divergence term, the VAE gives the same weird output both when reconstructing and generating images.

Here's the pytorch code for the loss function:

def loss_function(recon_x, x, mu, logvar):
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), size_average=True)
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (BCE+KLD)

recon_x is the reconstructed image, x is the original_image, mu is the mean vector while logvar is the vector containing the log of variance.

What is going wrong here? Thanks in advance :)

Output: A possible reason is the numerical unbalance between the two losses, with your BCE loss computed as an average over the batch (c.f. size_average=True) while the KLD one is summed.


Instruction: How can I assign pytorch tensor a matrix from numpy?

Input:
Create WxW tensor:

x = Variable(torch.FloatTensor(W,W).zero_(), requires_grad=True)

Do some calculations:

x_copy = x0=np.copy(x.data.numpy())
x_upd = handleArray(x_copy)

How can I assign x data from x_upd ?

Output:
Ok, the solution was to do:

x.data.from_numpy(x_upd)


Instruction: Overflow when unpacking long - Pytorch

Input:
I am running the following code

import torch
from __future__ import print_function
x = torch.empty(5, 3)
print(x)

on an Ubuntu machine in CPU mode, which gives me following error, what would be the reason and how to fix

      x = torch.empty(5, 3)
----> print(x)

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in __repr__(self)
     55         # characters to replace unicode characters with.
     56         if sys.version_info > (3,):
---> 57             return torch._tensor_str._str(self)
     58         else:
     59             if hasattr(sys.stdout, 'encoding'):

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _str(self)
    216             suffix = ', dtype=' + str(self.dtype) + suffix
    217
--> 218         fmt, scale, sz = _number_format(self)
    219         if scale != 1:
    220             prefix = prefix + SCALE_FORMAT.format(scale) + ' ' * indent

/usr/local/lib/python3.6/dist-packages/torch/_tensor_str.py in _number_format(tensor, min_sz)
     94     # TODO: use fmod?
     95     for value in tensor:
---> 96         if value != math.ceil(value.item()):
     97             int_mode = False
     98             break

RuntimeError: Overflow when unpacking long

Output:
Since, torch.empty() gives uninitialized memory, so you may or may not get a large value from it. Try

x = torch.rand(5, 3)
print(x)

this would give the response.


Instruction: Custom convolution kernel and toroidal convolution in PyTorch

Input:
I want to do two things with a PyTorch convolution which aren't mentioned in the documentation or code:

I want to create a convolution with a fixed kernel like this:

000010000
000010000
100010001
000010000
000010000

The horizontal aspect is like dilation, I guess, but the vertical part is different. I see that dilation is available as a parameter in the code, but it has to be a scalar or single-element tuple (not one element per dimension), so I don't think it can do what I want here.
I would like my convolutions to "wrap around" like a toroid, rather than use padding.

EDIT TO ADD: I see that there is an open issue for this , which also provides a suboptimal workaround. So, I guess that there's no "right" way to do it, yet.

Output:
Unlike torch.nn.conv2d() (which instantiates its own trainable kernel), torch.nn.functional.conv2d() takes as parameters both your matrix and kernel, so you can pass it whatever custom kernel you want.
As suggested by @zou3519 in a Github issue (linked to the issue you mentioned yourself), you could implement yourself a 2D circular padding by "repeating the tensor in a nxn grid, then cropping out the part you need.":

def circular_pad_2d(x, pad=(1, 1)):
   # Snipped by @zou3519 (https://github.com/zou3519)
   return x.repeat(*x_shape[:2])[
        (x.shape[0]-pad[0]):(2*x.shape[0]+pad[0]),
        (x.shape[1]-pad[1]):(2*x.shape[1]+pad[1])
   ]

# Example:
x = torch.tensor([[1,2,3],[4,5,6]])
y = circular_pad_2d(x, pad=(2, 3))
print(y)
#     1     2     3     1     2     3     1     2     3
#     4     5     6     4     5     6     4     5     6
#     1     2     3     1     2     3     1     2     3
#     4     5     6     4     5     6     4     5     6

(previous) In the torch.nn.functional module too, torch.nn.functional.pad() can take as parameter mode=reflect, which is what you want I believe (?). You could use this method to manually pad your input matrix before performing the convolution. (note: you also have the torch.nn.ReflectionPad2d layer specifically tailored for fixed 2D padding by reflection)


Instruction: Convert image to proper dimension PyTorch

Input:
I have an input image, as numpy array of shape [H, W, C] where H - height, W - width and C - channels.
I want to convert it into [B, C, H, W] where B - batch size, which should be equal to 1 every time, and changing the place for C.

_image = np.array(_image)
h, w, c = _image.shape
image = torch.from_numpy(_image).unsqueeze_(0).view(1, c, h, w)

So, will this preserve the image properly i.e without displacing the original image pixel values?

Output:
I'd prefer the following, which leaves the original image unmodified and simply adds a new axis as desired:

_image = np.array(_image)
image = torch.from_numpy(_image)
image = image[np.newaxis, :]
# _unsqueeze works fine here too

Then to swap the axes as desired:

image = image.permute(0, 3, 1, 2)
# permutation applies the following mapping
# axis0 -> axis0
# axis1 -> axis3
# axis2 -> axis1
# axis3 -> axis2


Instruction: Finding euclidean distance given an index array and a pytorch tensor

Input:
I got a pytorch tensor:

Z = np.random.rand(100,2)
tZ = autograd.Variable(torch.cuda.FloatTensor(Z), requires_grad=True)

and an index array:

idx = (np.array([0, 0, 0, 4, 3, 8], dtype="int64"),
       np.array([0, 1, 2, 3, 7, 4], dtype="int64"))

I need to find the distances of all the pairs of points in my tZ tensor using the idx array as indexes.

Right now I am doing it using numpy, but it would be nice if it could all be done using torch

dist = np.linalg.norm(tZ.cpu().data.numpy()[idx[0]]-tZ.cpu().data.numpy()[idx[1]], axis=1)

If anyone knows a way of doing it utilizing pytorch, to speed it up, it would be a great help!

Output:
Using torch.index_select():

Z = np.random.rand(100,2)
tZ = autograd.Variable(torch.cuda.FloatTensor(Z), requires_grad=True)

idx = (np.array([0, 0, 0, 4, 3, 8], dtype="int64"),
       np.array([0, 1, 2, 3, 7, 4], dtype="int64"))

tZ_gathered = [torch.index_select(tZ, dim=0,
                                  index=torch.cuda.LongTensor(idx[i]))
                                  # note: you may have to wrap it in a Variable too
                                  # (thanks @rvd for the comment):
                                  # index = autograd.Variable(torch.cuda.LongTensor(idx[i])))
               for i in range(len(idx))]

print(tZ_gathered[0].shape)
# > torch.Size([6, 2])

dist = torch.norm(tZ_gathered[0] - tZ_gathered[1])


Instruction: Training on minibatches of varying size

Input:
I'm trying to train a deep learning model in PyTorch on images that have been bucketed to particular dimensions. I'd like to train my model using mini-batches, but the mini-batch size does not neatly divide the number of examples in each bucket.

One solution I saw in a previous post was to pad the images with additional whitespace (either on the fly or all at once at the beginning of training), but I do not want to do this. Instead, I would like to allow the batch size to be flexible during training.

Specifically, if N is the number of images in a bucket and B is the batch size, then for that bucket I would like to get N // B batches if B divides N, and N // B + 1 batches otherwise. The last batch can have fewer than B examples.

As an example, suppose I have indexes [0, 1, ..., 19], inclusive and I'd like to use a batch size of 3.

The indexes [0, 9] correspond to images in bucket 0 (shape (C, W1, H1))
The indexes [10, 19] correspond to images in bucket 1 (shape (C, W2, H2))

(The channel depth is the same for all images). Then an acceptable partitioning of the indexes would be

batches = [
    [0, 1, 2],
    [3, 4, 5],
    [6, 7, 8],
    [9],
    [10, 11, 12],
    [13, 14, 15],
    [16, 17, 18],
    [19]
]

I would prefer to process the images indexed at 9 and 19 separately because they have different dimensions.

Looking through PyTorch's documentation, I found the BatchSampler class that generates lists of mini-batch indexes. I made a custom Sampler class that emulates the partitioning of indexes described above. If it helps, here's my implementation for this:

class CustomSampler(Sampler):

    def __init__(self, dataset, batch_size):
        self.batch_size = batch_size
        self.buckets = self._get_buckets(dataset)
        self.num_examples = len(dataset)

    def __iter__(self):
        batch = []
        # Process buckets in random order
        dims = random.sample(list(self.buckets), len(self.buckets))
        for dim in dims:
            # Process images in buckets in random order
            bucket = self.buckets[dim]
            bucket = random.sample(bucket, len(bucket))
            for idx in bucket:
                batch.append(idx)
                if len(batch) == self.batch_size:
                    yield batch
                    batch = []
            # Yield half-full batch before moving to next bucket
            if len(batch) > 0:
                yield batch
                batch = []

    def __len__(self):
        return self.num_examples

    def _get_buckets(self, dataset):
        buckets = defaultdict(list)
        for i in range(len(dataset)):
            img, _ = dataset[i]
            dims = img.shape
            buckets[dims].append(i)
        return buckets

However, when I use my custom Sampler class I generate the following error:

Traceback (most recent call last):
    File "sampler.py", line 143, in <module>
        for i, batch in enumerate(dataloader):
    File "/home/roflcakzorz/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 263, in __next__
        indices = next(self.sample_iter)  # may raise StopIteration
    File "/home/roflcakzorz/anaconda3/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 139, in __iter__
        batch.append(int(idx))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'list'

The DataLoader class seems to expect to be passed indexes, not list of indexes.

Should I not be using a custom Sampler class for this task? I also considered making a custom collate_fn to pass to the DataLoader, but with that approach I don't believe I can control which indexes are allowed to be in the same mini-batch. Any guidance would be greatly appreciated.

Output: Do you have 2 networks for each of the samples(A cnn kernel size has to be fix). If yes just pass the above custom_sampler to the batch_sampler args of DataLoader class. That would fix the issue.


Instruction: Converting a scipy coo_matrix to pytorch sparse tensor

Input:
I have a coo_matrix:

from scipy.sparse import coo_matrix
coo = coo_matrix((3, 4), dtype = "int8")

That I want converted to a pytorch sparse tensor. According to the documentation https://pytorch.org/docs/master/sparse.html it should follow the coo format, but I cannot find a simple way to do the conversion. Any help would be greatly appreciated!

Output:
Using the data as in the Pytorch docs, it can be done simply using the attributes of the Numpy coo_matrix:

import torch
import numpy as np
from scipy.sparse import coo_matrix

coo = coo_matrix(([3,4,5], ([0,1,1], [2,0,2])), shape=(2,3))

values = coo.data
indices = np.vstack((coo.row, coo.col))

i = torch.LongTensor(indices)
v = torch.FloatTensor(values)
shape = coo.shape

torch.sparse.FloatTensor(i, v, torch.Size(shape)).to_dense()

Output

0 0 3
4 0 5
[torch.FloatTensor of size 2x3]


Instruction: Google Colab: "Unable to connect to the runtime" after uploading Pytorch model from local

Input:
I am using a simple (not necessarily efficient) method for Pytorch model saving.

import torch
from google.colab import files

torch.save(model, filename) # save a trained model on the VM
files.download(filename) # download the model to local

best_model = files.upload() # select the model just downloaded
best_model[filename] # access the model

Colab disconnects during execution of the last line, and hitting RECONNECT tab always shows ALLOCATING -> CONNECTING (fails, with "unable to connect to the runtime" message in the left bottom corner) -> RECONNECT. At the same time, executing any one of the cells gives Error message "Failed to execute cell, Could not send execute message to runtime: [object CloseEvent]"

I know it is related to the last line, because I can successfully connect with my other google accounts which doesn't execute that.

Why does it happen? It seems the google accounts which have executed the last line can no longer connect to the runtime.

Edit:

One night later, I can reconnect with the google account after session expiration. I just attempted the approach in the comment, and found that just files.upload() the Pytorch model would lead to the problem. Once the upload completes, Colab disconnects.

Output:
(I wrote this answer before reading your update. Think it may help.)

files.upload() is just for uploading files. We have no reason to expect it to return some pytorch type/model.

When you call a = files.upload(), a is a dictionary of filename - a big bytes array.

{'my_image.png': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR....' }
type(a['my_image.png'])

Just like when you do open('my_image', 'b').read()

So, I think the next line best_model[filename] try to print the whole huge bytes array, which bugs the colab.


Instruction: Beta Distribution in PyTorch for a, b>1?

Input:
PyTorch supports Beta distributions however, when alpha or beta is greater than 1, it doesn't work:

m = Beta(torch.tensor([2]), torch.tensor([2]))
m.sample()

Output:
It works as expected using FloatTensor with torch==0.4.0:

import torch
from torch.distributions import Beta
m = Beta(torch.FloatTensor([2]), torch.FloatTensor([2]))
m.sample()


Instruction: Missing Weight Vectors when converting from PyTorch to CoreML via ONNX

Input:
I am trying to convert a PyTorch model to CoreML via ONNX, but the ONNX-->CoreML conversion is missing weight vectors?

I am following the tutorial here which makes this statement:

  Step 3: Converting the model to CoreML

  It's as easy as running the convert function. The resulting object is a coremltools MLModel object that you can save to a file and import in XCode later.
  cml = onnx_coreml.convert(model)

Unfortunately when I try to do this it fails horribly.

Here's my code:

# convert.py
import torch
import torch.onnx
from torch.autograd import Variable

import onnx
from onnx_coreml import convert

from hourglass_model import Hourglass

model_no = 1
torch_model = Hourglass(joint_count=14, size=256)
state_dict = torch.load("hourglass_model_{}.model".format(model_no))
torch_model.load_state_dict(state_dict)
torch_model.train(False)
torch_model.eval()

# Dummy Input to the model
x = Variable(torch.randn(1,3,256,256,dtype=torch.float32))

# Export the model
onnx_filename = "test_hourglass.onnx"
torch_out = torch.onnx.export(torch_model, x, onnx_filename, export_params=False)

# Load back in ONNX model
onnx_model = onnx.load(onnx_filename)

# Check that the IR is well formed
onnx.checker.check_model(onnx_model)

# Print a human readable representation of the graph
graph = onnx.helper.printable_graph(onnx_model.graph)
print(graph)

coreml_model = convert(onnx_model,
    add_custom_layers=True,
    image_input_names=["input"],
    image_output_names=["output"])
coreml_model.save('test_hourglass.mlmodel')

Here's what the print(graph) line gives.

graph torch-jit-export (
  %0[FLOAT, 1x3x256x256]
  %1[FLOAT, 64x3x5x5]
  %2[FLOAT, 64]
  %3[FLOAT, 64x64x5x5]
  %4[FLOAT, 64]
  %5[FLOAT, 64x64x5x5]
  %6[FLOAT, 64]
  %7[FLOAT, 64x64x5x5]
  %8[FLOAT, 64]
  %9[FLOAT, 64x64x5x5]
  %10[FLOAT, 64]
  %11[FLOAT, 64x64x5x5]
  %12[FLOAT, 64]
  %13[FLOAT, 64x64x5x5]
  %14[FLOAT, 64]
  %15[FLOAT, 64x64x1x1]
  %16[FLOAT, 64]
  %17[FLOAT, 14x64x1x1]
  %18[FLOAT, 14]
) {
  %19 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%0, %1, %2)
  %20 = Relu(%19)
  %21 = MaxPool[kernel_shape = [4, 4], pads = [0, 0, 0, 0], strides = [4, 4]](%20)
  %22 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%21, %3, %4)
  %23 = Relu(%22)
  %24 = MaxPool[kernel_shape = [4, 4], pads = [0, 0, 0, 0], strides = [4, 4]](%23)
  %25 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%24, %5, %6)
  %26 = Relu(%25)
  %27 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%26, %7, %8)
  %28 = Relu(%27)
  %29 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%28, %9, %10)
  %30 = Relu(%29)
  %31 = Upsample[height_scale = 4, mode = 'nearest', width_scale = 4](%30)
  %32 = Add(%31, %23)
  %33 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%32, %11, %12)
  %34 = Relu(%33)
  %35 = Upsample[height_scale = 4, mode = 'nearest', width_scale = 4](%34)
  %36 = Add(%35, %20)
  %37 = Conv[dilations = [1, 1], group = 1, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]](%36, %13, %14)
  %38 = Relu(%37)
  %39 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%38, %15, %16)
  %40 = Relu(%39)
  %41 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%40, %17, %18)
  %42 = Relu(%41)
  return %42
}

And this is the error message:

1/24: Converting Node Type Conv
Traceback (most recent call last):
  File "convert.py", line 38, in <module>
    image_output_names=["output"])
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/converter.py", line 396, in convert
    _convert_node(builder, node, graph, err)
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_operators.py", line 994, in _convert_node
    return converter_fn(builder, node, graph, err)
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_operators.py", line 31, in _convert_conv
    "Weight tensor: {} not found in the graph initializer".format(weight_name,))
  File "/Users/stephenf/Developer/miniconda3/envs/pytorch/lib/python3.6/site-packages/onnx_coreml/_error_utils.py", line 71, in missing_initializer
    format(node.op_type, node.inputs[0], node.outputs[0], err_message)
ValueError: Missing initializer error in op of type Conv, with input name = 0, output name = 19. Error message: Weight tensor: 1 not found in the graph initializer

From what I can gather, it says the weight tensor %1[FLOAT, 64x3x5x5] is missing. This is how I'm saving the model:

torch.save(model.state_dict(), "hourglass_model_{}.model".format(epoch))

ONNX loads it fine - it's just the step where I'm converting from ONNX to CoreML.

Any help in figuring this out would be greatly appreciated. I'm sure I've done a bunch of other things wrong, but I just need this thing to export for now.

Thanks,

Output:
You are calling torch.onnx.export with export_params=False, which, as the 0.3.1 doc reads, is saving the model architecture without the actual parameter tensors. The more recent documentation doesn't specify this, but we can assume that due to the Weight tensor not found error that you are getting.

Try it with export_params=True, you should see how the saved model's size increases notably.

Glad it helped!
Andres


Instruction: PyTorch Tutorial Error Training a Classifier

Input:
I just started the PyTorch-Tutorial Deep Learning with PyTorch: A 60 Minute Blitz and I should add, that I haven't programmed any python (but other languages like Java) before.

Right now, my Code looks like

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

print("\n-------------------Backpropagation-------------------\n")
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)

testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

dataiter = iter(trainloader)
images, labels = dataiter.next()

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

imshow(torchvision.utils.make_grid(images))

print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

which should be consistent with the tutorial.
If I execute this, I'll get the following error:

"C:\Program Files\Anaconda3\python.exe" C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py

-------------------Backpropagation-------------------

Files already downloaded and verified
Files already downloaded and verified

-------------------Backpropagation-------------------

Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\MA\pytorch\deepLearningWithPytorchTutorial\trainingClassifier.py", line 23, in <module>
    dataiter = iter(trainloader)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py", line 23, in <module>
    dataiter = iter(trainloader)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

Process finished with exit code 1

I already downloaded the *.py and *.ipynb.
Running the *.ipynb with jupyter works fine (but I don't want to programm in the juniper web-interface, I prefer pyCharm) while the *.py in the console (Anaconda prompt and cmd) fails with the same error.

Does anyone know how to fix this?
(I'm using Python 3.6.5 (from Anaconda) and pyCharm, OS: Win10 64-bit)

Thanks!
Bene

Update:
If it is relevant, I just set num_workers=2 to num_workers=0 (both) and then it'll work.. .

Output:
Check out the documentation for multiprocessing: programming guidelines for windows. You should wrap all operations in functions and then call them inside an if __name__ == '__main__' clause:

# required imports

def load_datasets(...):
    # Code to load the datasets with multiple workers

def train(...):
    # Code to train the model

if __name__ == '__main__':
    load_datasets()
    train()

In short, the the idea here is to wrap the example code inside an if __name__ == '__main__' statement.


Instruction: How reshape 3D tensor of shape (3, 1, 2) to (1, 2, 3)

Input:
I intended

(Pdb) aa = torch.tensor([[[1,2]], [[3,4]], [[5,6]]])
(Pdb) aa.shape
torch.Size([3, 1, 2])
(Pdb) aa
tensor([[[ 1,  2]],

        [[ 3,  4]],

        [[ 5,  6]]])
(Pdb) aa.view(1, 2, 3)
tensor([[[ 1,  2,  3],
         [ 4,  5,  6]]])

But what I really want is

tensor([[[ 1,  3,  5],
         [ 2,  4,  6]]])

How?

In my application, I am trying to transform my input data of shape (L, N, C_in) to (N, C_in, L) in order to use Conv1d, where

L: sequence length
N: batch size
C_in: number of channels in the input, I also understand it as the dimensionality of the input at each position of a sequence.

I am also wondering the input of Conv1d doesn't have the same input shape as GRU?

Output:
You can permute the axes to the desired shape. (This is similar to numpy.moveaxis() operation).
In [90]: aa
Out[90]:
tensor([[[ 1,  2]],

        [[ 3,  4]],

        [[ 5,  6]]])

In [91]: aa.shape
Out[91]: torch.Size([3, 1, 2])

# pass the desired ordering of the axes as argument
# assign the result back to some tensor since permute returns a "view"
In [97]: permuted = aa.permute(1, 2, 0)

In [98]: permuted.shape
Out[98]: torch.Size([1, 2, 3])

In [99]: permuted
Out[99]:
tensor([[[ 1,  3,  5],
         [ 2,  4,  6]]])


Instruction: CIFAR-10 Meaningless Normalization Values

Input:
I tried to build a neural network for a CIFAR-10 database. I used Pytorch Framework.

I have a question about of data loading step.

transform_train = T.Compose([
    T.RandomCrop(32, padding=4),
    T.RandomHorizontalFlip(),
    T.ToTensor(),
    T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

trainset = tv.datasets.CIFAR10(root=root, train=True, download=True, transform=transform_train)

This is a normal step for data loading step. While loading data, I am normalizing values. At the beginning of the my project I found below row.

T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))

After I searched better transform values, I found this values.

T.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))

I did not find an explanation why these values were used. Do you have a description of these values?

Output:
I think you can have a look here:

The first three values are the means over each channel, while the second triple are the standard deviations.


Instruction: Resize PyTorch Tensor

Input:
I am currently using the tensor.resize() function to resize a tensor to a new shape t = t.resize(1, 2, 3).

This gives me a deprecation warning:

  non-inplace resize is deprecated

Hence, I wanted to switch over to the tensor.resize_() function, which seems to be the appropriate in-place replacement. However, this leaves me with an

  cannot resize variables that require grad

error.
I can fall back to

from torch.autograd._functions import Resize
Resize.apply(t, (1, 2, 3))

which is what tensor.resize() does in order to avoid the deprecation warning.
This doesn't seem like an appropriate solution but rather a hack to me.
How do I correctly make use of tensor.resize_() in this case?

Output:
You can instead choose to go with tensor.reshape(new_shape) or torch.reshape(tensor, new_shape) as in:

# a `Variable` tensor
In [15]: ten = torch.randn(6, requires_grad=True)

# this would throw RuntimeError error
In [16]: ten.resize_(2, 3)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-16-094491c46baa> in <module>()
----> 1 ten.resize_(2, 3)

RuntimeError: cannot resize variables that require grad

The above RuntimeError can be resolved or avoided by using tensor.reshape(new_shape)

In [17]: ten.reshape(2, 3)
Out[17]:
tensor([[-0.2185, -0.6335, -0.0041],
        [-1.0147, -1.6359,  0.6965]])

# yet another way of changing tensor shape
In [18]: torch.reshape(ten, (2, 3))
Out[18]:
tensor([[-0.2185, -0.6335, -0.0041],
        [-1.0147, -1.6359,  0.6965]])


Instruction: How to share a list of tensors in PyTorch multiprocessing?

Input:
I am programming with PyTorch multiprocessing. I want all the subprocesses can read/write the same list of tensors (no resize). For example the variable can be

m = list(torch.randn(3), torch.randn(5))

Because each tensor has different sizes, I cannot organize them into a single tensor.

A python list has no share_memory_() function, and multiprocessing.Manager cannot handle a list of tensors. How can I share the variable m among multiple subprocesses?

Output:
I find the solution by myself. It is pretty straightforward. Just call share_memory_() for each list elements. The list itself is not in the shared memory, but the list elements are.

Demo code

import torch.multiprocessing as mp
import torch

def foo(worker,tl):
    tl[worker] += (worker+1) * 1000

if __name__ == '__main__':
    tl = [torch.randn(2), torch.randn(3)]

    for t in tl:
        t.share_memory_()

    print("before mp: tl=")
    print(tl)

    p0 = mp.Process(target=foo, args=(0, tl))
    p1 = mp.Process(target=foo, args=(1, tl))
    p0.start()
    p1.start()
    p0.join()
    p1.join()

    print("after mp: tl=")
    print(tl)

Output

before mp: tl=
[
 1.5999
 2.2733
[torch.FloatTensor of size 2]
,
 0.0586
 0.6377
-0.9631
[torch.FloatTensor of size 3]
]
after mp: tl=
[
 1001.5999
 1002.2733
[torch.FloatTensor of size 2]
,
 2000.0586
 2000.6377
 1999.0370
[torch.FloatTensor of size 3]
]


Instruction: PyTorch how to implement disconnection (connections and corresponding gradients are masked)?

Input:
I try to implement the following graph. As you can see, the neurons are not fully connected, i.e., the weights are masked and so are their corresponding gradients.

import torch
import numpy as np

x = torch.rand((3, 1))
# tensor([[ 0.8525],
#         [ 0.1509],
#         [ 0.9724]])

weights = torch.rand((2, 3), requires_grad=True)
# tensor([[ 0.3240,  0.0792,  0.6858],
#         [ 0.5248,  0.4565,  0.3625]])

mask = torch.Tensor([[0,1,0],[1,0,1]])
# tensor([[ 0.,  1.,  0.],
#         [ 1.,  0.,  1.]])

mask_weights = weights * mask
# tensor([[ 0.0000,  0.0792,  0.0000],
#         [ 0.5248,  0.0000,  0.3625]])

y = torch.mm(mask_weights, x)
# tensor([[ 0.0120],
#         [ 0.7999]])

This question is originally posted at Pytorch Forum. Note the above way

  mask_weights = weights * mask

is NOT suitable since corresponding gradients are not 0.

Is there an elegant way to do that please?

Thank you in advance.

Output:
Actually, the above method is correct. The disconnections essentially block feed-forward and back-propogation on corresponding connections. In other words, weights and gradients are masked. The codes in question reveal the first while this answer reveals the latter.

mask_weights.register_hook(print)

z = torch.Tensor([[1], [1]])
# tensor([[ 1.],
#         [ 1.]])

out = (y-z).mean()
# tensor(-0.6595)

out.backward()
# tensor([[ 0.1920,  0.1757,  0.0046],
#         [ 0.1920,  0.1757,  0.0046]])

weights.grad
# tensor([[ 0.0000,  0.1757,  0.0000],
#         [ 0.1920,  0.0000,  0.0046]])

As you can see, the gradients of weights are masked automatically.


Instruction: PyTorch: When using backward(), how can I retain only part of the graph?

Input:
I have a PyTorch computational graph, which consists of a sub-graph performing some calculation, and the result of this calculation (let's call it x) is then branched into two other sub-graphs. Each of these two sub-graphs yields some scalar results (lets call them y1 and y2). I want to do a backward pass for each of these two results (that is, I want to accumulate the gradients of the two sub-graphs. I do not want to perform the actual optimization step).

Now, since memory is an issue here, I would like to perform the operations in the following order:
First, calculate x. Then, calculate y1, and perform y1.backward() while (and this is the key point) retaining the graph that leads to x, but freeing the graph from x to y1. Then, calculate y2, and perform y2.backward().

In other words, to save memory without sacrificing too much speed, I want to keep x without needing to recalculate it, but I want to drop all calculations leading from x to y1 after I don't need them anymore.

The problem is that the argument retain_graph of the function backward() will retain the entire graph leading to y1, whereas I need to retain only the part of the graph leading to x.

Here is an example of what I would ideally want:

import torch

w = torch.tensor(1.0)
w.requires_grad_(True)

# sub-graph for calculating `x`
x = w+10

# sub-graph for calculating `y1`
x1 = x*x
y1 = x1*x1
y1.backward(retain_graph=x) # this would not work, since retain_graph is a boolean and can either retain the entire graph or free it.

# sub-graph for calculating `y2`
x2 = torch.sqrt(x)
y2 = x2/2
y2.backward()

How can this be done?

Output:
The argument retain_graph will retain the entire graph, not just a sub-graph. However, we can use garbage collection to free unneeded parts of the graph. By removing all references to the sub-graph from x to y1, this sub-graph will be freed:

import torch

w = torch.tensor(1.0)
w.requires_grad_(True)

# sub-graph for calculating `x`
x = w+10

# sub-graph for calculating `y1`
x1 = x*x
y1 = x1*x1
y1.backward(retain_graph=True) # all graph is retained

# remove unneeded parts of graph. Note that these parts will be freed from memory (even if they were on GPU), due to python's garbage collection
y1 = None
x1 = None

# sub-graph for calculating `y2`
x2 = torch.sqrt(x)
y2 = x2/2
y2.backward()


Instruction: Gradient error when calculating - pytorch

Input:
I am learning to use pytorch (0.4.0) to automate the gradient calculation, however I did not quite understand how to use the backward () and grad, as I'm doing an exercise I need to calculate df / dw using pytorch and
 making the derivative analytically, returning respectively auto_grad, user_grad, but I did not quite understand the use of automatic differentiation, in the code I made f.backward () and did w.grad to find df / dw, in addition the two calculations are not corresponding, if I even erred the derivative, it follows the graph that I am using and the code that I am trying to do:

import numpy as np
import torch
import torch.nn.functional as F

def graph2(W_np, x_np, b_np):
    W = torch.Tensor(W_np)
    W.requires_grad = True
    x = torch.Tensor(x_np)
    b = torch.Tensor(b_np)
    u = torch.matmul(W, x) + b
    g = F.sigmoid(u)
    f = torch.sum(g)
    user_grad = (sigmoid(W_np*x_np + b_np)*(1 - sigmoid(W_np*x_np + b_np))).T*x_np
    f.backward(retain_graph=True)
    auto_grad = W.grad

    print(auto_grad)
    print(user_grad)

 #   raise NotImplementedError("falta completar a função graph2")
    # END YOUR CODE
    return f, auto_grad, user_grad

test:

iterations = 1000
sizes = np.random.randint(2,10, size=(iterations))
for i in range(iterations):
    size = sizes[i]
    W_np = np.random.rand(size, size)
    x_np = np.random.rand(size, 1)
    b_np = np.random.rand(size, 1)
    f, auto_grad, user_grad = graph2(W_np, x_np, b_np)
    manual_f = np.sum(sigmoid(np.matmul(W_np, x_np) + b_np))
    assert np.isclose(f.data.numpy(), manual_f, atol=1e-4), "f not correct"
    assert np.allclose(auto_grad.numpy(), user_grad), "Gradient not correct"

Output:
I think you computed the gradients in the wrong way. Try this.

import numpy as np
import torch
from torch.autograd import Variable
import torch.nn.functional as F

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

def graph2(W_np, x_np, b_np):
    W = Variable(torch.Tensor(W_np), requires_grad=True)
    x = torch.tensor(x_np, requires_grad=True).type(torch.FloatTensor)
    b = torch.tensor(b_np, requires_grad=True).type(torch.FloatTensor)
    u = torch.matmul(W, x) + b
    g = F.sigmoid(u)
    f = torch.sum(g)
    user_grad = (sigmoid(np.matmul(W_np, x_np) + b_np)*(1 - sigmoid(np.matmul(W_np, x_np) + b_np)))*x_np.T
    f.backward(retain_graph=True)
    auto_grad = W.grad
    print("auto_grad", auto_grad)
    print("user_grad", user_grad)
    # END YOUR CODE
    return f, auto_grad, user_grad

iterations = 1000
sizes = np.random.randint(2,10, size=(iterations))
for i in range(iterations):
    size = sizes[i]
    print("i, size", i, size)
    W_np = np.random.rand(size, size)
    x_np = np.random.rand(size, 1)
    b_np = np.random.rand(size, 1)
    f, auto_grad, user_grad = graph2(W_np, x_np, b_np)
    manual_f = np.sum(sigmoid(np.matmul(W_np, x_np) + b_np))
    assert np.isclose(f.data.numpy(), manual_f, atol=1e-4), "f not correct"
    assert np.allclose(auto_grad.numpy(), user_grad), "Gradient not correct"


Instruction: Pytorch : How .grad() function returning result?

Input:
I am trying to understand grad() function in python, I know about backpropagation but having some doubt in .grad() function result.

So if i have a very simple network say with one single input and one single weight :

import torch
from torch.autograd import Variable
from torch import FloatTensor

a_tensor=Variable(FloatTensor([1]))
weight=Variable(FloatTensor([1]),requires_grad=True)

Now i am running this in ipython cell:

net_out=a_tensor*weight
loss=5-net_out
loss.backward()
print("atensor",a_tensor)
print('weight',weight)
print('net_out',net_out)
print('loss',loss)
print(weight.grad)

During first run it returns :

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-1.])

Which is correct because if i am right then computing gradient equation would be here :

Now netout/w would be (w*a) w.r.t to w ==> 1*a
And loss/netout  (5-netout) w.r.t to netout ==> (0-1)

Which would be 1*a*-1 ==> -1

But problem is if i press same cell again without modifying anything then i am getting grad -2 , -3 ,-4 ...etc

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-2.])

next run:

atensor tensor([ 1.])
weight tensor([ 1.])
net_out tensor([ 1.])
loss tensor([ 4.])
tensor([-3.])

so on..

I am not getting what's happening there why and how the value of grad is increasing?

Output:
This is because you are not zeroing the gradients. What loss.backward() does is accumulate gradients - it adds gradients to existing ones. If you don't zero the gradient, then running loss.backward() over and over just keep adding the gradients to each other. What you want to do is zero the gradients after each step and you will see that the gradients are calculated correctly.

If you have built a network net( which should be a nn.Module class object), you can zero the gradients simply by calling net.zero_grad(). If you haven't built a net (or an torch.optim object) you will have to zero the gradients yourself manually.

Use weight.grad.data.zero_() method there.


Instruction: Adding modules in Pytorch Custom Module

Input:
Is it considered bad practice to add modules to a custom pytorch nn.Module using self.add_module()? All of the documentation seems to assign the layers to properties, then access them in the forward() method.

For example:

class ConvLayer(nn.Module):
    def __init__(self):
        self.add_module('conv',nn.Conv2d(...))
        self.add_module('bn',nn.BatchNorm2d(...))

Output: Calling add_module will add an entry to the _modules dict. The Module class also overwrites __getattr__ so that when you try to access a layer, it will look inside the _modules dict, despite the fact that the layer is not actually an attribute of the object. But from the user's perspective, it doesn't make a difference whether module.layer returns an actual property or simply an element from some private dict.


Instruction: What does data.norm() < 1000 do in PyTorch?

Input:
I am following the PyTorch tutorial here.
It says that

x = torch.randn(3, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

print(y)

Out:
tensor([-590.4467,   97.6760,  921.0221])

Could someone explain what data.norm() does here?
When I change .randn to .ones its output is  tensor([ 1024.,  1024.,  1024.]).

Output:
It's simply the L2 norm (a.k.a Euclidean norm) of the tensor. Below is a reproducible illustration:

In [15]: x = torch.randn(3, requires_grad=True)

In [16]: y = x * 2

In [17]: y.data
Out[17]: tensor([-1.2510, -0.6302,  1.2898])

In [18]: y.data.norm()
Out[18]: tensor(1.9041)

# computing the norm using elementary operations
In [19]: torch.sqrt(torch.sum(torch.pow(y, 2)))
Out[19]: tensor(1.9041)

Explanation: First, it takes a square of every element in the input tensor x, then it sums them together, and finally it takes a square root of the resulting sum. All in all, these operations compute the so-called L2 or Euclidean norm.


Instruction: Debugging GAN covergence error

Input:
Building a GAN to generate images. The images have 3 color channels, 96 x 96.

The images that are generated by the generator at the beginning are all black, which is an issue given that is statistically highly unlikely.

Also, the loss for both networks is not improving.

I have posted the entire code below, and commented to allow it to be easily read. This is my first time building a GAN and I am new to Pytorch so any help is very appreciated!

Thanks.

import torch
from torch.optim import Adam
from torch.utils.data import DataLoader
from torch.autograd import Variable

import numpy as np
import os
import cv2
from collections import deque

# training params
batch_size = 100
epochs = 1000

# loss function
loss_fx = torch.nn.BCELoss()

# processing images
X = deque()
for img in os.listdir('pokemon_images'):
    if img.endswith('.png'):
        pokemon_image = cv2.imread(r'./pokemon_images/{}'.format(img))
        if pokemon_image.shape != (96, 96, 3):
            pass
        else:
            X.append(pokemon_image)

# data loader for processing in batches
data_loader = DataLoader(X, batch_size=batch_size)

# covert output vectors to images if flag is true, else input images to vectors
def images_to_vectors(data, reverse=False):
    if reverse:
        return data.view(data.size(0), 3, 96, 96)
    else:
        return data.view(data.size(0), 27648)

# Generator model
class Generator(torch.nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        n_features = 1000
        n_out = 27648

        self.model = torch.nn.Sequential(
                torch.nn.Linear(n_features, 128),
                torch.nn.ReLU(),
                torch.nn.Linear(128, 256),
                torch.nn.ReLU(),
                torch.nn.Linear(256, 512),
                torch.nn.ReLU(),
                torch.nn.Linear(512, 1024),
                torch.nn.ReLU(),
                torch.nn.Linear(1024, n_out),
                torch.nn.Tanh()
        )

    def forward(self, x):
        img = self.model(x)
        return img

    def noise(self, s):
       x = Variable(torch.randn(s, 1000))
       return x

# Discriminator model
class Discriminator(torch.nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        n_features = 27648
        n_out = 1

        self.model = torch.nn.Sequential(
                torch.nn.Linear(n_features, 512),
                torch.nn.ReLU(),
                torch.nn.Linear(512, 256),
                torch.nn.ReLU(),
                torch.nn.Linear(256, n_out),
                torch.nn.Sigmoid()
        )

    def forward(self, img):
        output = self.model(img)
        return output

# discriminator training
def train_discriminator(discriminator, optimizer, real_data, fake_data):
    N = real_data.size(0)
    optimizer.zero_grad()

    # train on real
    # get prediction
    pred_real = discriminator(real_data)

    # calculate loss
    error_real = loss_fx(pred_real, Variable(torch.ones(N, 1)))

    # calculate gradients
    error_real.backward()

    # train on fake
    # get prediction
    pred_fake = discriminator(fake_data)

    # calculate loss
    error_fake = loss_fx(pred_fake, Variable(torch.ones(N, 0)))

    # calculate gradients
    error_fake.backward()

    # update weights
    optimizer.step()

    return error_real + error_fake, pred_real, pred_fake

# generator training
def train_generator(generator, optimizer, fake_data):
    N = fake_data.size(0)

    # zero gradients
    optimizer.zero_grad()

    # get prediction
    pred = discriminator(generator(fake_data))

    # get loss
    error = loss_fx(pred, Variable(torch.ones(N, 0)))

    # compute gradients
    error.backward()

    # update weights
    optimizer.step()

    return error

# Instance of generator and discriminator
generator = Generator()
discriminator = Discriminator()

# optimizers
g_optimizer = torch.optim.Adam(generator.parameters(), lr=0.001)
d_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.001)

# training loop
for epoch in range(epochs):
     for n_batch, batch in enumerate(data_loader, 0):
         N = batch.size(0)

         # Train Discriminator

         # REAL
         real_images = Variable(images_to_vectors(batch)).float()

         # FAKE
         fake_images = generator(generator.noise(N)).detach()

         # TRAIN
         d_error, d_pred_real, d_pred_fake = train_discriminator(
                 discriminator,
                 d_optimizer,
                 real_images,
                 fake_images
         )

         # Train Generator

         # generate noise
         fake_data = generator.noise(N)

         # get error based on discriminator
         g_error = train_generator(generator, g_optimizer, fake_data)

         # convert generator output to image and preprocess to show
         test_img = np.array(images_to_vectors(generator(fake_data), reverse=True).detach())
         test_img = test_img[0, :, :, :]
         test_img = test_img[..., ::-1]

         # show example of generated image
         cv2.imshow('GENERATED', test_img[0])
         if cv2.waitKey(1) & 0xFF == ord('q'):
             break

     print('EPOCH: {0}, D error: {1}, G error: {2}'.format(epoch, d_error, g_error))

cv2.destroyAllWindows()

# save weights
# torch.save('weights.pth')

Output:
One can't really easily debug your training without the data and so on, but a possible problem is that your generator's last layer is a Tanh(), which means output values between -1 and 1. You probably want:

To have your real images normalized to the same range, e.g. in train_discriminator():

# train on real
pred_real = discriminator(real_data * 2. - 1.) # supposing real_data in [0, 1]

To re-normalize your generated data to [0, 1] before visualization/use.

# convert generator output to image and preprocess to show
test_img = np.array(
    images_to_vectors(generator(fake_data), reverse=True).detach())
test_img = test_img[0, :, :, :]
test_img = test_img[..., ::-1]
test_img = (test_img + 1.) / 2.


Instruction: Pytorch not using cuda device

Input:
I have the following code:

from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import scipy.io

folder = 'small/'
mat = scipy.io.loadmat(folder+'INISTATE.mat');
ini_state = np.float32(mat['ini_state']);
ini_state = torch.from_numpy(ini_state);
ini_state = ini_state.cuda();

mat = scipy.io.loadmat(folder+'TARGET.mat');
target = np.float32(mat['target']);
target = torch.from_numpy(target);
target = target.cuda();

class MLPNet(nn.Module):
    def __init__(self):
        super(MLPNet, self).__init__()
        self.fc1 = nn.Linear(3, 64)
        self.fc2 = nn.Linear(64, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 41)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x

    def name(self):
        return "MLP"

model = MLPNet();
model = model.cuda();

criterion = nn.MSELoss();
criterion = criterion.cuda();
learning_rate = 0.001;
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

batch_size = 20
iter_size = int(target.size(0)/batch_size)
print(iter_size)

for epoch in range(50):
    for i in range(iter_size):
        start = i*batch_size;
        end = (i+1)*batch_size-1;
        samples = ini_state[start:end,:];
        labels = target[start:end,:];

        optimizer.zero_grad()  # zero the gradient buffer
        outputs = model(samples)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if (i+1) % 500 == 0:
            print("Epoch %s, batch %s, loss %s" % (epoch, i, loss))
    if (epoch+1) % 7 == 0:
        for g in optimizer.param_groups:
            g['lr'] = g['lr']*0.1;

But when I train the simple MLP, the CPU usage is around 100% while the gpu is only around 10%. What is the problem that prevents using the GPU?

Output:
Actually your model indeed runs on GPU instead of CPU. The reason of low GPU usage is that both your model and batch size are small, which demands low computational cost. You may try increasing the batch size to around 1000, and the GPU usage should be higher. In fact PyTorch prevents operations that mix CPU and GPU data, e.g., you can't multiply a GPU tensor and a CPU tensor. So usually it is unlikely that part of your network runs on CPU and the other part runs on GPU, unless you deliberately design it.

By the way, data shuffling is necessary for neural networks. As your are using mini-batch training, in each iteration you are hoping that the mini batch approximates the whole dataset. Without data shuffling, it is likely that samples in a mini batch are highly correlated, which leads to biased estimation of parameter update. The data loader provided by PyTorch can help you do the data shuffling.


Instruction: Pytorch - handling picutres and .jpeg files (beginner's questions)

Input:
I am new at Pytorch, and have a couple of questions regarding the way pictures are being handled:

1) In the "training a classifier" tutorial, the pictures are PIL files, and are being handled via the following commands (where "transform" also turns the PIL format into a tensor format):

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                    download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

It seems like trainset[1] (and also for the other indices) consists of a tensor, and a number.  I want to define a new variable "image" that will consist of the tensor part of trainset[ 1 ] and then print it - how can I do it?

2) Assume that I have a different dataset that I want to classify. It consists of .jpeg images that are located in the folder "C:/temp/dataset". How can I define the variable "trainset" to consist of these images?

Thanks a lot in advance!

Output:
For your first question:

image = trainset[1][0]

print(image)

For your second question:

from PIL import Image
import numpy as np
import os

def load_image(infilename):
    """This function loads an image into memory when you give it
       the path of the image
    """
    img = Image.open(infilename)
    img.load()
    data = np.asarray(img, dtype="float32")
    return data

def create_npy_from_image(images_folder, output_name, num_images, image_dim):
    """Loops through the images in a folder and saves all of them
       as a numpy array in output_name
    """
    image_matrix = np.empty((num_images, image_dim, image_dim, 3), dtype=np.float32)
    for i, filename in enumerate(os.listdir(images_folder)):
        if filename.endswith(".jpg"):
            data = load_image(images_folder + filename)
            image_matrix[i] = data
        else:
            continue
    np.save(output_name, image_matrix)

So I would write something like this:

create_npy_from_image(path_to_images_folder, "trainset.npy", numer_of_images_in_your_folder, DIM)

DIM is 64 for example if your images are 64x64x3

You can then load the saved array with np.load and then convert it to a pytorch tensor using from_numpy function.

Let me know if this works. Good luck!


Instruction: Understanding the Input Parameters in RNN

Input:
I'm having a hard time to understand the different "jargons" used in RNN. They are the following:
batch_size, time_steps, inputs and instances.

Let me go through my understanding of each input parameters & please correct me where I'm wrong.
Suppose I've got a sequence of numbers and I want to predict the next number. The numbers are the following:
[1,2,3,4,5,....,100]

time_steps: This parameter means how far RNN will look into past before it predicts the future. For simplicity, I want to predict 1 number ahead. And want to do after I see 10 numbers in the past. So, in this case, time_steps will be 10.

inputs: These are the values at each time_steps. In first time_step (t) the inputs are
t0: [1]

t1: [2]

.
.
.

t10: [10]`

batch_size: This helps in efficient computation of RNN model. Suppose my batch_size is 2. In that case, at time_step 2, the RNN input will be

t0: [1]
t0: [11]

Then what's the usage of instances? E.g. in this post, instances have been used. And there are multiple cases where instances are used. Is it means each loop over batch? E.g. there are 5 batches, each of size 2. Then there will be 5 instances.

Please help me correct my understanding.

Thanks!

Output: Alright pal, you did good learning those concepts. I had a hard time learning those correctly. Everything you know seems to be in order and as for "instances". They're basically a set of data. There's no fixed term of usage of "instances" in a deep learning community. Some people use it for referring for a different set of data or batches of data. I rarely hear it in papers.


Instruction: How to calculate Pixel wise accuracy in pytorch?

Input:
My code looks like the following and I get accuracy from 0 to 9000, which means its clearly not working.

optimizer.zero_grad()
outputs = net(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()

predicted = outputs.data
predicted = predicted.to('cpu')
predicted_img = predicted.numpy()

labels_data = labels.data
labels_data = labels_data.to('cpu')
labels_data = labels_data.numpy()
labels = labels.to(device)

_, predicted = torch.max(outputs.data, 1)
total = labels.size(0) * labels.size(1) * labels.size(2)
correct = (predicted_img == labels_data).sum().item()
accuracy += ( correct / total)
avg_accuracy = accuracy/(batch)

What am I doing wrong ?

Output:
I am assuming the following line accumulates accuracy over mini-batches.

accuracy += (correct/total)

And avg_accuracy = accuracy/batch gives average accuracy over the entire dataset where batch represents the total number of mini-batches representing the whole dataset.

If you are getting accuracy greater than 100, then you should check if in any mini-batch, you get correct > total? Also check if total = labels_data.size gives you the same value as the following line.

total = labels.size(0) * labels.size(1) * labels.size(2)