Instruction: CUDA out of memory with matrix multiply

Input:
I'm trying to multiply 3 matrices, but am running out of CUDA memory.

# A: 3000 x 100 (~2MB)
# B: 100 x 100  (~0.05MB)
# C: 100 x 3MM  (~2GB)

A = np.random.randn(3000, 100)
B = np.random.randn(100, 100)
C = np.random.randn(100, 3e6)

A_gpu = torch.from_numpy(A).cuda()
B_gpu = torch.from_numpy(B).cuda()
C_gpu = torch.from_numpy(C).cuda()

R_gpu = (A_gpu @ B_gpu @ C_gpu)

Cuda is requesting about 90GB of memory for this operation.  I don't understand why.

Output: Multiplying matrices, your output size is going to be 3,000 x 3,000,000 matrix! so despite A and B being relatively small, the output R is HUGE: 9G elements. Moreover, I suspect dtype of your matrices is float64 and not float32 (because you used numpy to init them). Therefore, each of the 9G elements of R_gpu requires 8 bytes; bringing you to size of at least 72GB GPU memory only for R_gpu. I suspect intermediate results and some other stuff occupies a little more of you GPU memory.


Instruction: 'NoneType' object has no attribute 'add_summary'

Input:
I'm having trouble with visualizing the weights and bias of my model using tensorboardX.
Here is my model (it's pretty simple anyway):

    self.pipe = nn.Sequential(nn.Linear(9, 128),
                             nn.ReLU(),
                             nn.Linear(128, 256),
                             nn.ReLU(),
                             nn.Linear(256,2),
                             nn.Softmax()
                             )

     def forward(self, x):
     return self.pipe(x)

And here is where I get error in pytorch

for name, param in net.named_parameters():
    writer.add_histogram(name, param, epoch_size, bins='auto')

and the error is

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-70-d060d2df4423> in <module>()
      1 for name, param in net.named_parameters():
----> 2     writer.add_histogram(name, param, epoch_size, bins='auto')

~\Anaconda3\lib\site-packages\tensorboardX\writer.py in add_histogram(self, tag, values, global_step, bins, walltime)
    403         if isinstance(bins, six.string_types) and bins == 'tensorflow':
    404             bins = self.default_bins
--> 405         self.file_writer.add_summary(
    406             histogram(tag, values, bins), global_step, walltime)
    407

AttributeError: 'NoneType' object has no attribute 'add_summary'

but I really have to see the histogram where the weights stuck in suboptimal.
so I changed code little bit to proceed step by step

param = np.array(list(net.parameters()))
print(param[0].data)
writer.add_histogram('weight', param[0].data)

BOOM!  still same error, maybe that doesn't change at all.

Output:
The posted code snippet is insufficient to root cause the issue.

The member variable file_writer is set to None when the close() method is invoked on writer. Please check if the close() method was invoked on writer. The close() method is also invoked when the writer object is used as a Context manager and the with block is exited.

with SummaryWriter() as writer:
   writer.add_scalar...

writer.add_histogram # this will cause a crash


Instruction: Data Augmentation with torchvision.transforms in pytorch

Input:
I found out data augmentation can be done in PyTorch by using torchvision.transforms. I also read that transformations are apllied at each epoch. So I'm wondering whether or not the effect of copying each sample multiple times and then applying random transformation to them is same as using torchvision.transforms on original data set(unique images) and just training it for a longer time(more epochs).
Thanks in advance.

Output:
This is a question to be answered in a broad scale. don't get misunderstood that the TorchVision Transforms doesn't increase your dataset. It applies random or non-random transforms to your current data set at runtime. (hence unique each time and each epoch).

  the effect of copying each sample multiple times and then applying random transformation to them is same as using torchvision.transforms on original data set(unique images) and just training it for a longer time(more epochs).

Answer-
To increase your dataset, you can copy paste, also use pyTorch or WEKA software. However, more epochs are a totally different concept to this. Of course, the more epochs you use, the better the model will be (only till the validation loss and training loss intersect each other)
Hope this helps.


Instruction: "function 'AddDllDirectory' not found" while importing pytorch

Input:
I keep getting this error while trying to import pytorch in jupyter notebook

import torch

  AttributeError: function 'AddDllDirectory' not found

this error occur only when I try installing pytorch in anaconda (I used both command to install pytorch pip and conda, I also tried installing only the cpu version and nothing changed, i still get the same error while trying to import it)

I searched the internet and found nothing remotely close to my problem.
is there's a way I can fix this error?

thanks in advance

Output:
I fixed the issue by installing the file (KB2533623) from
Here (for windows 7 users). this file apparently contains the missing "AddDllDirectory" function.

Posting this answe in case someone ran into the same problem.


Instruction: Calling forward function without .forward()

Input:
While looking at some pytorch code on pose estimation AlphaPose I noticed some unfamiliar syntax:

Basically, we define a Darknet class which inherits nn.Module properties like so: class Darknet(nn.Module)

This re-constructs the neural net from some config file and also defines functions to load pre-trained weights and a forward pass

Now, forward pass takes the following parameters:

def forward(self, x, CUDA)

I should note that in class definition forward is the only method that has a CUDA attribute (this will become important later on)

In the forward pass we get the predictions:

for i in range(number_of_modules):
     x = self.module[i](x)

where module[i] was constructed as:

module = nn.Sequential()
conv = nn.Conv2d(prev_fileters, filters, kernel_size, stride, pad, bias=bias)
module.add_module("conv_{0}".format(index), conv)

We then call invoke this model and (I presume) a forward method like so:

self.det_model = Darknet("yolo/cfg/yolov3-spp.cfg")
self.det_model.load_weights('models/yolo/yolov3-spp.weights')
self.det_model.cpu()
self.det_model.eval()

image = image.cpu()
prediction = self.det_model(img, CUDA = False)

I assume that the last line is the calling of the forward pass but why not use the .forward? Is this a pytorch specific syntax or am I missing some basic python principles?

Output:
This is nothing torch specific. When you call something as class_object(fn params) it invokes the __call__ method of that class.

If you dig the code of torch, specifically nn.Module you will see that __call__ internally invokes forward but taking care of hooks and states that pytorch allows. So when you are calling self.det_model(img, cuda) you are still calling forward.

See the code for nn.module here.


Instruction: how sort randomly images and its mask?

Input:
before splitting the dataset I need to randomly load the data and then do splitting . This is the snippet for splitting the dataset which is not randomly. I am wondering how can I do this for images and corresponding mask in folder_mask?

folder_data = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\imagesResized\\*.png")
folder_mask = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\labelsResized\\*.png")

 # split these path using a certain percentage
  len_data = len(folder_data)
  print("count of dataset: ", len_data)
  # count of dataset:  992

  split_1 = int(0.6 * len(folder_data))
  split_2 = int(0.8 * len(folder_data))

  #folder_data.sort()

  train_image_paths = folder_data[:split_1]
  print("count of train images is: ", len(train_image_paths))

  valid_image_paths = folder_data[split_1:split_2]
  print("count of validation image is: ", len(valid_image_paths))

  test_image_paths = folder_data[split_2:]
  print("count of test images is: ", len(test_image_paths))

   train_mask_paths = folder_mask[:split_1]
   valid_mask_paths = folder_mask[split_1:split_2]
   test_mask_paths = folder_mask[split_2:]

   train_dataset = CustomDataset(train_image_paths, train_mask_paths)
   train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=1,
   shuffle=True, num_workers=2)

   valid_dataset = CustomDataset(valid_image_paths, valid_mask_paths)
   valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=1,
   shuffle=True, num_workers=2)

    test_dataset = CustomDataset(test_image_paths, test_mask_paths)
    test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=1,
     shuffle=False, num_workers=2)

     dataLoaders = {
          'train': train_loader,
          'valid': valid_loader,
           'test': test_loader,
                 }

Output:
As far as I understood, you want to randomize the order of the pictures, so that with each rerun there are different photos in the train and test set. Assuming you want to do this in more or less plain Python you can do the following.

The easiest way to use shuffle a list of elements in python is:

import random
random.shuffle(list)  // shuffles in place

So you have to list and want to still keep the link between data and masks. So if you can accept a rather quick hack, I'd propose something like this.

import random

folder_data = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\imagesResized\\*.png")
folder_mask = glob.glob("D:\\Neda\\Pytorch\\U-net\\my_data\\labelsResized\\*.png")

assert len(folder_data) == len(folder_mask) // everything else would be bad

indices = list(range(len(folder_data)))
random.shuffle(indices)

Now you have a list of indices you can split and then use the indices from the splitted list to access the other lists.

split_1 = int(0.6 * len(folder_data))
split_2 = int(0.8 * len(folder_data))

train_image_paths = [folder_data[i] for i in indices]
// and so on...

This would be the plain Python way. But there are functions to do this in packages like sklearn. So you might consider using those. They'll gonna save you from doing a lot of work. (Usually it's way better to reuse code then to implement it yourself.)


Instruction: shall I apply softmax before cross entropy?

Input:
The pytorch tutorial (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) trains a convolutional neural network (CNN) on a CIFAR dataset.

    class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1, 16 * 5 * 5)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

The network looks good except that the very last layer fc3, which predicts the probability of belonging to 10 classes without a softmax. Shouldn't we apply a softmax first to make sure the output of the fc layer is between 0 and 1 and sum before calculating cross-entropy loss?

I tested this by applying the softmax and rerunning, butvthe accuracy dropped to around 35%. This seems counterintuitive. What is the explanation?

Output:
CrossEntropyLoss in PyTorch is already implemented with Softmax:
https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss

This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

The answer to the second part of your question is a little more complicated. There can be multiple causes for reduction in accuracy. Theoretically speaking, since the softmax layer you added can predict the correct answer in a reasonable accuracy, the following layer should be able to do the same by preserving the maximum value with identity between the last two layers. Although the softmax normalizes those bounded outputs (between 0 and 1) again, it may change the way those are distributed, but still can preserve the maximum and therefore the class that is predicted.
However, in practice, things are a little bit different. When you have a double softmax in the output layer, you basically change the output function in such way that it changes the gradients that are propagated to your network. The softmax with cross entropy is a preferred loss function due to the gradients it produces. You can prove it to yourself by computing the gradients of the cost function, and account for the fact that each "activation" (softmax) is bounded between 0 and 1. The additional softmax "behind" the original one just multiplies the gradients with values between 0 and 1 and thus reducing the value. This affects the updates to the weights. Maybe it can be fixed by changing the learning rate but this is strongly not suggested. Just have one softmax and you're done.
See Michael Nielsen's book, chapter 3 for more profound explanation on that.


Instruction: CycleGAN with 1-channel tiffs both as input and as output

Input:
I am running CycleGAN with different types of tiffs in trainA and trainB. The tiffs are 256x256 pixels in size and have 1 channel per pixel. I am using tiffs to have a wide range of values.

I changed the code as suggested in the pytorch-CycleGAN-and-pix2pix
repo (https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/320 and similar), but what I got out during the training in ./checkpoints are three-channels PNGs. Do you think it would be possible to change the code so that it goes from 1 channel tiff to 1 channel tiff with no information loss? As far as I understand, at present the code is converting the imported files to PNGs along the way. In other words: I would like my tensors to be [256*256*int_range,1]. Thanks for the help!

Output: Have you tried adding the parameters --input_nc 1 --output_nc 1 during training? It will convert the number of channels from 3 to 1.


Instruction: cuDNN error: CUDNN_STATUS_BAD_PARAM.Can someone explain why i am getting this error and how can i correct it?

Input:
I am trying to implement a Character LSTM using Pytorch.But I am getting cudnn_status_bad_params errors.This is the training loop.I getting error on line output = model(input_seq).

for epoch in tqdm(range(epochs)):
  for i in range(len(seq)//batch_size):
   sidx = i*batch_size
   eidx = sidx + batch_size
   x = seq[sidx:eidx]
   x = torch.tensor(x).cuda()
   input_seq =torch.nn.utils.rnn.pack_padded_sequence(x,seq_lengths,batch_first = True)
   y = out_seq[sidx:eidx]
   output = model(input_seq)
   loss = criterion(output,y)
   loss.backward()
   optimizer.step()
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    180         else:
    181             result = _impl(input, batch_sizes, hx, self._flat_weights, self.bias,
--> 182                            self.num_layers, self.dropout, self.training, self.bidirectional)
    183         output = result[0]
    184         hidden = result[1:] if self.mode == 'LSTM' else result[1]

 RuntimeError: cuDNN error: CUDNN_STATUS_BAD_PARAM

Output:
I got the same error, if you switch to CPU, you'll get a much better description of the error. In my case the problem was in type of input that I was giving to the network. I was sending I guess long, while the model needed float. I made the following changes and the code worked. Basically switching to cpu gives better error descriptions.

input_seq = input_seq.float().cuda()


Instruction: Torchtext AttributeError: 'Example' object has no attribute 'text_content'

Input:
I'm working with RNN and using Pytorch & Torchtext. I've got a problem with building vocab in my RNN. My code is as follows:

TEXT = Field(tokenize=tokenizer, lower=True)
LABEL = LabelField(dtype=torch.float)

trainds = TabularDataset(
    path='drive/{}'.format(TRAIN_PATH), format='tsv',
    fields=[
        ('label_start', LABEL),
        ('label_end', None),
        ('title', None),
        ('symbol', None),
        ('text_content', TEXT),
    ])

testds = TabularDataset(
    path='drive/{}'.format(TEST_PATH), format='tsv',
    fields=[
        ('text_content', TEXT),
    ])

TEXT.build_vocab(trainds, testds)

When I want to build vocab, I'm getting this annoying error:

AttributeError: 'Example' object has no attribute 'text_content'

I'm sure, that there is no missing text_content attr. I made try-catch in order to display this specific case:

try:
    print(len(trainds[i]))
except:
    print(trainds[i].text_content)

Surprisingly, I don't get any error and this specific print command shows:

['znana', 'okresie', 'masarni', 'walc', 'y', 'myśl', 'programie', 'sprawy', ...]

So it indicates, that there is text_content attr. When I perform this on a smaller dataset, it works like a charm. This problem occurs when I want to work with proper data. I ran out of ideas. Maybe someone had a similar case and can explain it.

My full traceback:

AttributeError                            Traceback (most recent call last)
<ipython-input-16-cf31866a07e7> in <module>()
    155
    156 if __name__ == "__main__":
--> 157     main()
    158

<ipython-input-16-cf31866a07e7> in main()
    117             break
    118
--> 119     TEXT.build_vocab(trainds, testds)
    120     print('zbudowano dla text')
    121     LABEL.build_vocab(trainds)

/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in build_vocab(self, *args, **kwargs)
    260                 sources.append(arg)
    261         for data in sources:
--> 262             for x in data:
    263                 if not self.sequential:
    264                     x = [x]

/usr/local/lib/python3.6/dist-packages/torchtext/data/dataset.py in __getattr__(self, attr)
    152         if attr in self.fields:
    153             for x in self.examples:
--> 154                 yield getattr(x, attr)
    155
    156     @classmethod

AttributeError: 'Example' object has no attribute 'text_content'

Output: This problem arises when the fields are not passed in the same order as they are in the csv/tsv file. Order must be same. Also check if no extra or less fields are mentioned than there are in the csv/tsv file..


Instruction: convert cv2.umat to numpy array

Input:
Processed_image() function returns a cv2.Umat type value which is to be reshaped from
3 dimensions(h, ch, w) to 4 dimensions(h, ch, w, 1) so i need it to be converted
to numpy array or also if possible help me to directally rehshape cv2.umat
type variable to be directally reshaped and converted to a pytorch tensor and can be
assigned to reshaped_image_tensor.

img_w=640
img_h=640
img_ch=3
umat_img = cv2.UMat(img)
display_one(umat_img, "RESPONSE")    #function created by me to display image
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
with torch.no_grad():
    processed_img = preprocess_image(umat_img, model_image_size = (img_h, img_ch, img_w))
    #___________write YOUR CODE here________
    reshaped_images_tensor = torch.from_numpy(processed_img.reshape(img_h, img_ch, img_w, 1)).float().to(device)      #images_tensor.reshape(img_h, img_ch, img_w, 1)
    outputs = model(reshaped_images_tensor)
    _, predicted = torch.max(outputs, 1)
    c = predicted.squeeze()
    output_probability(predicted, processed_img, umat_img)
if ord('q')==cv2.waitKey(10):
    exit(0)

Output:
I didn't quite catch your question, but you can get numpy data of an opencv's umat with "get()" like this

and you should probably permute your tensor before feeding it into your model.


Instruction: How to load images with multiple JSON annotation in PyTorch

Input:
I would like to know how I can use the data loader in PyTorch for the custom file structure of mine. I have gone through PyTorch documentation, but all those are with separate folders with class.

My folder structure consists of 2 folders(called training and validation), each with 2 subfolders(called images and json_annotations). Each image in the "images" folder has multiple objects(like cars, cycles, man etc) and each is annotated and have separate JSON files. Standard coco annotation is followed. My intention is to make a neural network which can do real-time classification from videos.

Edit 1:
I have done the coding as suggested by Fábio Perez.

class lDataSet(data.Dataset):
    def __init__(self, path_to_imgs, path_to_json):
        self.path_to_imgs = path_to_imgs
        self.path_to_json = path_to_json
        self.img_ids = os.listdir(path_to_imgs)

    def __getitem__(self, idx):
        img_id = self.img_ids[idx]
        img_id = os.path.splitext(img_id)[0]
        img = cv2.imread(os.path.join(self.path_to_imgs, img_id + ".jpg"))
        load_json = json.load(open(os.path.join(self.path_to_json, img_id + ".json")))
        #n = len(load_json)
        #bboxes = load_json['annotation'][n]['segmentation']
        return img, load_json

    def __len__(self):
        return len(self.image_ids)

When I try this

l_data = lDataSet(path_to_imgs = '/home/training/images', path_to_json = '/home/training/json_annotations')

I'm getting l_data with l_data[][0] - images and l_data with json. Now I'm confused. How will I use it with finetuning example availalbe in PyTorch? In that example, dataset and dataloader is done as shown below.
https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html

# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['train', 'val']}
# Create training and validation dataloaders
dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['train', 'val']}

Output:
You should be able to implement your own dataset with data.Dataset. You just need to implement __len__ and __getitem__ methods.

In your case, you can iterate through all images in the image folder (then you can store the image ids in a list in your Dataset). Then, you use the index passed to __getitem__ to get the corresponding image id. With this image id, you can read the corresponding JSON file and return the target data that you need.

Something like this:

class YourDataLoader(data.Dataset):
    def __init__(self, path_to_imgs, path_to_json):
        self.path_to_imags = path_to_imgs
        self.path_to_json = path_to_json
        self.image_ids = iterate_through_images(path_to_images)

    def __getitem__(self, idx):
        img_id = self.image_ids[idx]
        img = load_image(os.path.join(self.path_to_images, img_id)
        bboxes = load_bboxes(os.path.join(self.path_to_json, img_id)
        return img, bboxes

    def __len__(self):
        return len(self.image_ids)

In iterate_through_images you get all the ids (e.g. filenames) of images in a directory.
In load_bboxes you read the JSON and get the information you need.

I have a JSON loader implementation here if you want a reference.


Instruction: Display a tensor image in matplotlib

Input:
I'm doing a project for Udacity's AI with Python nanodegree.

I'm trying to display a torch.cuda.FloatTensor that I obtained from an image file path. Below that image will be a bar chart showing the top 5 most likely flower names with their associated probabilities.

plt.figure(figsize=(3,3))
path = 'flowers/test/1/image_06743.jpg'

top5_probs, top5_class_names = predict(path, model,5)

print(top5_probs)
print(top5_class_names)

flower_np_image = process_image(Image.open(path))
flower_tensor_image = torch.from_numpy(flower_np_image).type(torch.cuda.FloatTensor)
flower_tensor_image = flower_tensor_image.unsqueeze_(0)

axs = imshow(flower_tensor_image, ax = plt)
axs.axis('off')
axs.title(top5_class_names[0])
axs.show()
fig, ax = plt.subplots()
y_pos = np.arange(len(top5_class_names))
plt.barh(y_pos, list(reversed(top5_probs)))
plt.yticks(y_pos, list(reversed(top5_class_names)))
plt.ylabel('Flower Type')
plt.xlabel('Class Probability')

The imshow function was given to me as

def imshow(image, ax=None, title=None):
    if ax is None:
        fig, ax = plt.subplots()

    # PyTorch tensors assume the color channel is the first dimension
    # but matplotlib assumes is the third dimension
    image = image.transpose((1, 2, 0))

    # Undo preprocessing
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    image = std * image + mean

    # Image needs to be clipped between 0 and 1 or it looks like noise when displayed
    image = np.clip(image, 0, 1)

    ax.imshow(image)

    return ax

But I get this output

[0.8310797810554504, 0.14590543508529663, 0.013837042264640331, 0.005048676859587431, 0.0027143193874508142]
['petunia', 'pink primrose', 'balloon flower', 'hibiscus', 'tree mallow']

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-f54be68feb7a> in <module>()
     12 flower_tensor_image = flower_tensor_image.unsqueeze_(0)
     13
---> 14 axs = imshow(flower_tensor_image, ax = plt)
     15 axs.axis('off')
     16 axs.title(top5_class_names[0])

<ipython-input-15-9c543acc89cc> in imshow(image, ax, title)
      5     # PyTorch tensors assume the color channel is the first dimension
      6     # but matplotlib assumes is the third dimension
----> 7     image = image.transpose((1, 2, 0))
      8
      9     # Undo preprocessing

TypeError: transpose(): argument 'dim0' (position 1) must be int, not tuple

<matplotlib.figure.Figure at 0x7f5855792160>

My predict function works, but the imshow just chokes with the call to transpose. Any ideas on how to fix this? I think it vaguely has something to do with converting back to a numpy array.

The notebook that I'm working on can be found at https://github.com/BozSteinkalt/ImageClassificationProject

Thanks!

Output:
You are trying to apply numpy.transpose to a torch.Tensor object, thus calling tensor.transpose instead.

You should convert flower_tensor_image to numpy first, using .numpy()

axs = imshow(flower_tensor_image.detach().cpu().numpy(), ax = plt)


Instruction: RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got NoneType

Input:
My code is

a=torch.randn(1,80,100,requires_grad=True)
torch.onnx.export(waveglow,a, "waveglow.onnx")

I am trying to export a PyTorch model to ONNX format so i can use it in TensorRT. while testing my model in PyTorch the input tensor dimension is (1,80,x) where x varies depending on the input text length(the model i am using is TTS model named waveglow)

when i try to run the above code for exporting the model to onnx, I am always getting this error

RuntimeError: Only tuples, lists and Variables supported as JIT inputs, but got NoneType

Please help

Output:
Given that you have NoneType, perhaps you could check if there is an actual input, because the fact is, you actually got None.

Also, any reason to not use Variable? Variable converts your inputs to a tensor that can be accepted as an input for torch.onnx.export.


Instruction: Model returns a Nan value

Input:
I was trying to build a neural network with 4 input nodes/ features and just one output feature(0/1). I wrote this code and it runs but while training the model returns NaN. I debugged too and weights and biases are fine until they go through the model.

From what I've searched so far, this could be a problem in the way I am passing the data.
My input data is : tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00, 1.5340e+00],
        [1.5000e+01, 1.0000e-01, 2.4210e+00, 3.0000e+01],
        [3.0000e+00, 2.2000e-01, 2.2000e-01, 4.5000e+01],
        ...,
        [1.0000e+00, 2.0000e-02, 2.0000e-02, 1.5000e+01],
        [6.0000e+00, 2.0000e-01, 2.0000e-01, 1.5000e+01],
        [1.7000e+01, 5.2400e-01, 5.2400e-01, 2.0000e+00]], dtype=torch.float64)

import torch
from torchvision import datasets, transforms
import pandas as pd
import numpy as np
from torch.autograd import Variable
# Import tensor dataset & data loader
from torch.utils.data import TensorDataset, DataLoader
from torch import nn, optim
import torch.nn.functional as F

file = pd.read_csv('ks-projects-201801.csv')
array = np.array(file.values)
result = np.empty(len(array))
input_data = np.empty((len(array), 4))
for i in range(len(array)):
    input_data[i] = np.array([array[i][10], array[i][12]/1000, array[i][13]/1000, array[i][14]/1000])
    if array[i][9] == 'successful':
        result[i] = 1
    else:
        result[i] = 0

input_node = Variable(torch.from_numpy(input_data))
output = torch.from_numpy(result)
print(input_node)
print(output)

train_ds = TensorDataset(input_node.squeeze(), output.squeeze())

batch_size = 5
train_dl = DataLoader(train_ds, batch_size, shuffle=True)

This is the actual model and training

model = nn.Linear(4, 1)
print(model.weight)
print(model.bias)
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.003)

epochs = 5
model = model.double()
for e in range(epochs):
    running_loss = 0
    for xb, yb in train_dl:
        optimizer.zero_grad()
        res = model(xb)
        loss = criterion(res, yb)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"model : {loss}")

This prints out model: nan for every epoch and terminates. I am very new to pytorch and I'm not sure how to handle this problem.

Output: If you see NaN's in loss try gradient clipping and data normalisation. Normalising data is a must (i.e normalize input data such that mean  = 0 and variance =1)


Instruction: A Basic Python Question about Defining Function

Input:
I have a basic question regarding Python code.
For example,
import torch
import torch.nn as nn

loss = nn.MSELoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)
output = loss(input, target)
output.backward()

Why do I need to define the loss function at the first line? I can't replace loss() at the 4th line with nn.MSELoss().

Output:
As a few others have pointed out, nn.MSELoss is a class and not a function. In line 1 you are creating an object of type torch.nn.modules.loss.MSELoss. And because it inherits from nn.Module, you can call this object like you would call a function, like you do in line 4.
If you don't want to use the MSELoss class, you can also import torch.nn.functional as F and then use F.mse_loss(input, target) directly (this is what pytorch normally calls for you).


Instruction: How to handle large JSON file in Pytorch?

Input: I am working on a time series problem. Different training time series data is stored in a large JSON file with the size of 30GB. In tensorflow I know how to use TF records. Is there a similar way in pytorch?

Output:
I suppose IterableDataset (docs) is what you need, because:

you probably want to traverse files without random access;
number of samples in jsons is not pre-computed.

I've made a minimal usage example with an assumption that every line of dataset file is a json itself, but you can change the logic.
import json
from torch.utils.data import DataLoader, IterableDataset

class JsonDataset(IterableDataset):
    def __init__(self, files):
        self.files = files

    def __iter__(self):
        for json_file in self.files:
            with open(json_file) as f:
                for sample_line in f:
                    sample = json.loads(sample_line)
                    yield sample['x'], sample['time'], ...

...

dataset = JsonDataset(['data/1.json', 'data/2.json', ...])
dataloader = DataLoader(dataset, batch_size=32)

for batch in dataloader:
    y = model(batch)


Instruction: How to visualize my training history in pytorch?

Input:
How do you guys visualize the training history of your pytorch model like in keras here.

I have a pytorch trained model and I want to see the graph of its training.
Can I do this using only matplotlib? If yes, can someone give me resources to follow.

Output:
You have to save the loss while training. A trained model won't have history of its loss. You need to train again.

Save the loss while training then plot it against the epochs using matplotlib. In your training function, where loss is being calculated save that to a file and visualize it later.
Also, you can use tensorboardX if you want to visualize in realtime.

This is a tutorial for tensorboardX: http://www.erogol.com/use-tensorboard-pytorch/


Instruction: How to calculate Batch Pairwise Distance in PyTorch efficiently

Input:
I have tensors X of shape BxNxD and Y of shape BxNxD.

I want to compute the pairwise distances for each element in the batch, i.e. I a BxMxN tensor.

How do I do this?

There is some discussion on this topic here: https://github.com/pytorch/pytorch/issues/9406, but I don't understand it as there are many implementation details while no actual solution is highlighted.

A naive approach would be to use the answer for non-batched pairwise distances as discussed here: https://discuss.pytorch.org/t/efficient-distance-matrix-computation/9065, i.e.

import torch
import numpy as np

B = 32
N = 128
M = 256
D = 3

X = torch.from_numpy(np.random.normal(size=(B, N, D)))
Y = torch.from_numpy(np.random.normal(size=(B, M, D)))

def pairwise_distances(x, y=None):
    x_norm = (x**2).sum(1).view(-1, 1)
    if y is not None:
        y_t = torch.transpose(y, 0, 1)
        y_norm = (y**2).sum(1).view(1, -1)
    else:
        y_t = torch.transpose(x, 0, 1)
        y_norm = x_norm.view(1, -1)

    dist = x_norm + y_norm - 2.0 * torch.mm(x, y_t)
    return torch.clamp(dist, 0.0, np.inf)

out = []
for b in range(B):
    out.append(pairwise_distances(X[b], Y[b]))
print(torch.stack(out).shape)

How can I do this without looping over B?
Thanks

Output:
I had a similar issue and spent some time to find the easiest and fastest solution. Now you can compute batched distance by using PyTorch cdist which will give you BxMxN tensor:

torch.cdist(Y, X)

Also, it works well if you just want to compute distances between each pair of rows of two matrixes.


Instruction: .py not reading from the content folder

Input:
|-content
  |-utils
    |- parse_config.py
  |-models.py

folder structure
This is my folder structure in google colab.

I have already installed Pytorch and all other requirments for the project.
Here models.py file is not able to access the files in utils folder.

In my models.py I'm importing utils.parse_config which is inside the utils folder but it shows the following error.

ImportErrorTraceback (most recent call last) <ipython-input-6-77f4a3369184> in <module>()
----> 1 from models import *
      2 from utils import *
      3
      4 import os, sys, time, datetime, random
      5 import torch

/content/models.py in <module>()
      9 from PIL import Image
     10
---> 11 from utils.parse_config import *
     12 from utils.utils import build_targets
     13 from collections import defaultdict

ImportError: No module named utils.parse_config

--------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the "Open Examples" button below.
---------------------------------------------------------------------------

Error

How do I make models.py get access to files in utils folder?

Output: Models.py seem not in utils folder. Please recheck


Instruction: GRU Language Model not Training Properly

Input:
I’ve tried reimplementing a simple GRU language model using just a GRU and a linear layer (the full code is also at https://www.kaggle.com/alvations/gru-language-model-not-training-properly):

class Generator(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, num_layers):
        super(Generator, self).__init__()

        # Initialize the embedding layer with the
        # - size of input (i.e. no. of words in input vocab)
        # - no. of hidden nodes in the embedding layer
        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=0)

        # Initialize the GRU with the
        # - size of the input (i.e. embedding layer)
        # - size of the hidden layer
        self.gru = nn.GRU(embedding_size, hidden_size, num_layers)

        # Initialize the "classifier" layer to map the RNN outputs
        # to the vocabulary. Remember we need to -1 because the
        # vectorized sentence we left out one token for both x and y:
        # - size of hidden_size of the GRU output.
        # - size of vocabulary
        self.classifier = nn.Linear(hidden_size, vocab_size)

    def forward(self, inputs, use_softmax=False, hidden=None):
        # Look up for the embeddings for the input word indices.
        embedded = self.embedding(inputs)
        # Put the embedded inputs into the GRU.
        output, hidden = self.gru(embedded, hidden)

        # Matrix manipulation magic.
        batch_size, sequence_len, hidden_size = output.shape
        # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
        output = output.contiguous().view(batch_size * sequence_len, hidden_size)
        # Put it through the classifier
        # And reshape it to [batch_size x sequence_len x vocab_size]
        output = self.classifier(output).view(batch_size, sequence_len, -1)

        return (F.softmax(output,dim=2), hidden) if use_softmax else (output, hidden)

    def generate(self, max_len, temperature=1.0):
        pass

And the training routine:

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Set the hidden_size of the GRU
embed_size = 100
hidden_size = 100
num_layers = 1

# Setup the data.
batch_size=50
kilgariff_data = KilgariffDataset(tokenized_text)
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

criterion = nn.CrossEntropyLoss(ignore_index=kilgariff_data.vocab.token2id['<pad>'], size_average=True)
model = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers).to(device)

learning_rate = 0.003
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

#model = nn.DataParallel(model)

losses = []

def train(num_epochs, dataloader, model, criterion, optimizer):
    plt.ion()
    for _e in range(num_epochs):
        for batch in tqdm(dataloader):
            x = batch['x'].to(device)
            x_len = batch['x_len'].to(device)
            y = batch['y'].to(device)
            # Zero gradient.
            optimizer.zero_grad()
            # Feed forward.
            output, hidden = model(x, use_softmax=True)
            # Compute loss:
            # Shape of the `output` is [batch_size x sequence_len x vocab_size]
            # Shape of `y` is [batch_size x sequence_len]
            # CrossEntropyLoss expects `output` to be [batch_size x vocab_size x sequence_len]

            _, prediction = torch.max(output, dim=2)
            loss = criterion(output.permute(0, 2, 1), y)
            loss.backward()
            optimizer.step()
            losses.append(loss.float().data)

            clear_output(wait=True)
            plt.plot(losses)
            plt.pause(0.05)

train(50, dataloader, model, criterion, optimizer)

#learning_rate = 0.05
#optimizer = optim.SGD(model.parameters(), lr=learning_rate)
#train(4, dataloader, model, criterion, optimizer)

But when the model is predicting, we see that it’s only predicting “the” and comma “,”.

Anyone spot something wrong with my code? Or hyperparameters?

The full code:

# coding: utf-8

# In[1]:

# IPython candies...
from IPython.display import Image
from IPython.core.display import HTML

from IPython.display import clear_output

# In[2]:

import numpy as np
from tqdm import tqdm

import pandas as pd

from gensim.corpora import Dictionary

import torch
from torch import nn, optim, tensor, autograd
from torch.nn import functional as F
from torch.utils.data import Dataset, DataLoader

from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# In[3]:

import matplotlib.pyplot as plt
import seaborn as sns

sns.set_style("darkgrid")
sns.set(rc={'figure.figsize':(12, 8)})

torch.manual_seed(42)

# In[4]:

try: # Use the default NLTK tokenizer.
    from nltk import word_tokenize, sent_tokenize
    # Testing whether it works.
    # Sometimes it doesn't work on some machines because of setup issues.
    word_tokenize(sent_tokenize("This is a foobar sentence. Yes it is.")[0])
except: # Use a naive sentence tokenizer and toktok.
    import re
    from nltk.tokenize import ToktokTokenizer
    # See https://stackoverflow.com/a/25736515/610569
    sent_tokenize = lambda x: re.split(r'(?<=[^A-Z].[.?]) +(?=[A-Z])', x)
    # Use the toktok tokenizer that requires no dependencies.
    toktok = ToktokTokenizer()
    word_tokenize = word_tokenize = toktok.tokenize

# In[5]:

import os
import requests
import io #codecs

# Text version of https://kilgarriff.co.uk/Publications/2005-K-lineer.pdf
if os.path.isfile('language-never-random.txt'):
    with io.open('language-never-random.txt', encoding='utf8') as fin:
        text = fin.read()
else:
    url = "https://gist.githubusercontent.com/alvations/53b01e4076573fea47c6057120bb017a/raw/b01ff96a5f76848450e648f35da6497ca9454e4a/language-never-random.txt"
    text = requests.get(url).content.decode('utf8')
    with io.open('language-never-random.txt', 'w', encoding='utf8') as fout:
        fout.write(text)

# In[6]:

# Tokenize the text.
tokenized_text = [list(map(str.lower, word_tokenize(sent)))
                  for sent in sent_tokenize(text)]

# In[7]:

class KilgariffDataset(nn.Module):
    def __init__(self, texts):
        self.texts = texts

        # Initialize the vocab
        special_tokens = {'<pad>': 0, '<unk>':1, '<s>':2, '</s>':3}
        self.vocab = Dictionary(texts)
        self.vocab.patch_with_special_tokens(special_tokens)

        # Keep track of the vocab size.
        self.vocab_size = len(self.vocab)

        # Keep track of how many data points.
        self._len = len(texts)

        # Find the longest text in the data.
        self.max_len = max(len(txt) for txt in texts)

    def __getitem__(self, index):
        vectorized_sent = self.vectorize(self.texts[index])
        x_len = len(vectorized_sent)
        # To pad the sentence:
        # Pad left = 0; Pad right = max_len - len of sent.
        pad_dim = (0, self.max_len - len(vectorized_sent))
        vectorized_sent = F.pad(vectorized_sent, pad_dim, 'constant')
        return {'x':vectorized_sent[:-1],
                'y':vectorized_sent[1:],
                'x_len':x_len}

    def __len__(self):
        return self._len

    def vectorize(self, tokens, start_idx=2, end_idx=3):
        """
        :param tokens: Tokens that should be vectorized.
        :type tokens: list(str)
        """
        # See https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary.doc2idx
        # Lets just cast list of indices into torch tensors directly =)

        vectorized_sent = [start_idx] + self.vocab.doc2idx(tokens) + [end_idx]
        return torch.tensor(vectorized_sent)

    def unvectorize(self, indices):
        """
        :param indices: Converts the indices back to tokens.
        :type tokens: list(int)
        """
        return [self.vocab[i] for i in indices]

# In[8]:

kilgariff_data = KilgariffDataset(tokenized_text)
len(kilgariff_data.vocab)

# In[9]:

batch_size = 10
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

for data_dict in dataloader:
    # Sort indices of data in batch by lengths.
    sorted_indices = np.array(data_dict['x_len']).argsort()[::-1].tolist()
    data_batch = {name:_tensor[sorted_indices]
                  for name, _tensor in data_dict.items()}
    print(data_batch)
    break

# In[97]:

class Generator(nn.Module):
    def __init__(self, vocab_size, embedding_size, hidden_size, num_layers):
        super(Generator, self).__init__()

        # Initialize the embedding layer with the
        # - size of input (i.e. no. of words in input vocab)
        # - no. of hidden nodes in the embedding layer
        self.embedding = nn.Embedding(vocab_size, embedding_size, padding_idx=0)

        # Initialize the GRU with the
        # - size of the input (i.e. embedding layer)
        # - size of the hidden layer
        self.gru = nn.GRU(embedding_size, hidden_size, num_layers)

        # Initialize the "classifier" layer to map the RNN outputs
        # to the vocabulary. Remember we need to -1 because the
        # vectorized sentence we left out one token for both x and y:
        # - size of hidden_size of the GRU output.
        # - size of vocabulary
        self.classifier = nn.Linear(hidden_size, vocab_size)

    def forward(self, inputs, use_softmax=False, hidden=None):
        # Look up for the embeddings for the input word indices.
        embedded = self.embedding(inputs)
        # Put the embedded inputs into the GRU.
        output, hidden = self.gru(embedded, hidden)

        # Matrix manipulation magic.
        batch_size, sequence_len, hidden_size = output.shape
        # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
        output = output.contiguous().view(batch_size * sequence_len, hidden_size)
        # Put it through the classifier
        # And reshape it to [batch_size x sequence_len x vocab_size]
        output = self.classifier(output).view(batch_size, sequence_len, -1)

        return (F.softmax(output,dim=2), hidden) if use_softmax else (output, hidden)

    def generate(self, max_len, temperature=1.0):
        pass

# In[98]:

# Set the hidden_size of the GRU
embed_size = 12
hidden_size = 10
num_layers = 4

_encoder = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers)

# In[99]:

# Take a batch.
_batch = next(iter(dataloader))
_inputs, _lengths = _batch['x'], _batch['x_len']
_targets = _batch['y']
max(_lengths)

# In[100]:

_output, _hidden = _encoder(_inputs)
print('Output sizes:\t', _output.shape)
print('Input sizes:\t', batch_size, kilgariff_data.max_len -1, len(kilgariff_data.vocab))
print('Target sizes:\t', _targets.shape)

# In[101]:

_, predicted_indices = torch.max(_output, dim=2)
print(predicted_indices.shape)
predicted_indices

# In[103]:

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# Set the hidden_size of the GRU
embed_size = 100
hidden_size = 100
num_layers = 1

# Setup the data.
batch_size=50
kilgariff_data = KilgariffDataset(tokenized_text)
dataloader = DataLoader(dataset=kilgariff_data, batch_size=batch_size, shuffle=True)

criterion = nn.CrossEntropyLoss(ignore_index=kilgariff_data.vocab.token2id['<pad>'], size_average=True)
model = Generator(len(kilgariff_data.vocab), embed_size, hidden_size, num_layers).to(device)

learning_rate = 0.003
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

#model = nn.DataParallel(model)

losses = []

def train(num_epochs, dataloader, model, criterion, optimizer):
    plt.ion()
    for _e in range(num_epochs):
        for batch in tqdm(dataloader):
            x = batch['x'].to(device)
            x_len = batch['x_len'].to(device)
            y = batch['y'].to(device)
            # Zero gradient.
            optimizer.zero_grad()
            # Feed forward.
            output, hidden = model(x, use_softmax=True)
            # Compute loss:
            # Shape of the `output` is [batch_size x sequence_len x vocab_size]
            # Shape of `y` is [batch_size x sequence_len]
            # CrossEntropyLoss expects `output` to be [batch_size x vocab_size x sequence_len]

            _, prediction = torch.max(output, dim=2)
            loss = criterion(output.permute(0, 2, 1), y)
            loss.backward()
            optimizer.step()
            losses.append(loss.float().data)

            clear_output(wait=True)
            plt.plot(losses)
            plt.pause(0.05)

train(50, dataloader, model, criterion, optimizer)

#learning_rate = 0.05
#optimizer = optim.SGD(model.parameters(), lr=learning_rate)
#train(4, dataloader, model, criterion, optimizer)

# In[ ]:

list(kilgariff_data.vocab.items())

# In[105]:

start_token = '<s>'
hidden_state = None
max_len = 20
temperature=0.8

i = 0

while start_token not in ['</s>', '<pad>'] and i < max_len:
    i += 1
    start_state = torch.tensor(kilgariff_data.vocab.token2id[start_token]).unsqueeze(0).unsqueeze(0).to(device)
    model.embedding(start_state)
    output, hidden_state = model.gru(model.embedding(start_state), hidden_state)

    batch_size, sequence_len, hidden_size = output.shape
    output = output.contiguous().view(batch_size * sequence_len, hidden_size)

    output = model.classifier(output).view(batch_size, sequence_len, -1)
    _, prediction = torch.max(F.softmax(output, dim=2), dim=2)

    start_token = kilgariff_data.vocab[int(prediction.squeeze(0).squeeze(0))]

    print(start_token, end=' ')

Output:
I'm by no means a PyTorch expert, but that snippet looks fishy to me:

    # Put the embedded inputs into the GRU.
    output, hidden = self.gru(embedded, hidden)
    # Matrix manipulation magic.
    batch_size, sequence_len, hidden_size = output.shape
    # Technically, linear layer takes a 2-D matrix as input, so more manipulation...
    output = output.contiguous().view(batch_size * sequence_len, hidden_size)

When GRU is not instantiated with batch_first=True, then the output shape is (seq_len, batch, num_directions * hidden_size) -- not that seq_len and batch_size are flipped. For the view command it actually doesn't technically matter, but that's my main issue here.
view(batch_size * sequence_len, hidden_size) doesn't look right at all. Say you start with a batch of size 32, but after that you have size of 32*seq_len. Usually, only the output of the last step is used (or the average or the max over all steps)

Something like this should work:

    # Put the embedded inputs into the GRU.
    output, hidden = self.gru(embedded, hidden)
    # Not needed, just to show the true output shape order
    seq_len, batch_size, hidden_size = output.shape
    # Given the shape of output, this is the last step
    output = output[-1]
    # output.shape = (batch_size, hidden_size) <-- What you want

Two personal words of warning:

view() is a dangerous command! PyTorch or any other framework only throws errors when the dimensions of the tensors do not match up. But just because the dimensions fit after view() does not mean the reshaping was done correctly, i.e., that the values are in the right spot of the output tensor. For example, if you have to flatten a shape (seq_len, batch_size, hidden_size) to (batch_size, seq_len*hidden_size), you cannot simply do view(batch_size, -1), but first have to do transpose(1,0) to get a shape of (batch_size, seq_len, hidden_size). With out without transpose(), view() will work and the dimensions will be correct. But only with transpose(), the values are at the right position after view()
Since this is such an easy mistake to make, I saw many examples on GitHub and such where in my opinion it's no done correctly. The problem is that the network often still learns something. In short, I'm not much more careful when looking and adopting code snippets and the view() command is in my opinion the biggest trap.

If it helps, here's the forward method of a GRU classifier network:

def forward(self, batch, method='last_step'):
    embeds = self.word_embeddings(batch)
    x = torch.transpose(embeds, 0, 1)
    x, self.hidden = self.gru(x, self.hidden)

    if method == 'last_step':
        x = x[-1]
    elif method == 'average_pooling':
        x = torch.sum(x, dim=0) / len(batch[0])
    elif method == 'max_pooling':
        x, _ = torch.max(x, dim=0)
    else:
        raise Exception('Unknown method.')
    # A series of Linear layers with ReLU and Dropout
    for l in self.linears:
        x = l(x)
    log_probs = F.log_softmax(x, dim=1)
    return log_probs


Instruction: Curious on how to use some basic machine learning in a web application

Input:
A co-worker and I had an idea to create a little web game where a user enters a chunk of data about themselves and then the application would write for them to sound like them in certain structures. (Trying to leave the idea a little vague.) We are both new to ML and thought this could be a fun first dive.

We have a decent bit of background with PHP, JavaScript (FE and Node), Ruby a little bit of other languages, and have had interest in learning Python for ML. Curious if you can run a cost efficient ML library for text well with a web app, being most servers lack GPUs?

Perhaps you have to pay for one of the cloud based systems, but wanted to find the best entry point for this idea without racking up too much cost. (So far I have been reading about running Pytorch or TensorFlow, but it sounds like you lose a lot of efficiency running with CPUs.)

Thank you!

(My other thought is doing it via an iOS app and trying Apple's ML setup.)

Output: It sounds like you are looking for something like  Tensorflow JS


Instruction: Implement dropout to fully connected layer in PyTorch

Input:
How to apply dropout to the following fully connected network in Pytorch:

class NetworkRelu(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784,128)
        self.fc2 = nn.Linear(128,64)
        self.fc3 = nn.Linear(64,10)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x),dim=1)
        return x

Output:
Since there is functional code in the forward method, you could use functional dropout, however, it would be better to use nn.Module in __init__() so that the model when set to model.eval() evaluate mode automatically turns off the dropout.

Here is the code to implement dropout:

class NetworkRelu(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784,128)
        self.fc2 = nn.Linear(128,64)
        self.fc3 = nn.Linear(64,10)
        self.dropout = nn.Dropout(p=0.5)

    def forward(self,x):
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = F.softmax(self.fc3(x),dim=1)
        return x


Instruction: Pytorch sum over a list of tensors along an axis

Input:
I have a list of tensors of the same shape.
I would like to sum the entire list of tensors along an axis.
Does torch.cumsum perform this op along a dim?
If so it requires the list to be converted to a single tensor and summed over?

Output:
you don't need cumsum, sum is your friend
and yes you should first convert them into a single tensor with stack or cat based on your needs, something like this:

import torch
my_list = [torch.randn(3, 5), torch.randn(3, 5)]
result = torch.stack(my_list, dim=0).sum(dim=0).sum(dim=0)
print(result.shape) #torch.Size([5])


Instruction: Faster pytorch dataset file

Input:
I have the following problem, I have many files of 3D volumes that I open to extract a bunch of numpy arrays.
I want to get those arrays randomly, i.e. in the worst case I open as many 3D volumes as numpy arrays I want to get, if all those arrays are in separate files.
The IO here isn't great, I open a big file only to get a small numpy array from it.
Any idea how I can store all these arrays so that the IO is better?
I can't pre-read all the arrays and save them all in one file because then that file would be too big to open for RAM.

I looked up LMDB but it all seems to be about Caffe.
Any idea how I can achieve this?

Output:
I iterated through my dataset, created an hdf5 file and stored elements in the hdf5. Turns out, when the hdf5 is opened, it doesn't load all data in ram, it loads the header instead.
The header is then used to fetch the data on request, that's how I solved my problem.

Reference:
http://www.machinelearninguru.com/deep_learning/data_preparation/hdf5/hdf5.html


Instruction: Need very different learning rate for manual updates vs. using model

Input:
I am currently just trying to write some pedagogical material, in which I borrow from some common examples that have been reworked numerous times on the web.

I have a simple bit of code where I manually create tensors for layers, and update them within a loop.  E.g.:

w1 = torch.randn(D_in, H, dtype=torch.float, requires_grad=True)
w2 = torch.randn(H, D_out, dtype=torch.float, requires_grad=True)

learning_rate = 1e-6
for t in range(501):
    y_pred = x.mm(w1).clamp(min=0).mm(w2)
    loss = (y_pred - y).pow(2).sum()
    loss.backward()
    w1 -= learning_rate * w1.grad
    w2 -= learning_rate * w2.grad
    w1.grad.zero_()
    w2.grad.zero_()

This works great.  Then I construct similar code using actual modules:

model = torch.nn.Sequential(
          torch.nn.Linear(D_in, H),
          torch.nn.ReLU(),
          torch.nn.Linear(H, D_out),
        )
loss_fn = torch.nn.MSELoss(reduction='sum')
learning_rate = 1e-4
for t in range(501):
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    model.zero_grad()
    loss.backward()
    for param in model.parameters():
        param.data -= learning_rate * param.grad

This also works great.

BUT there is a difference here.  If I use a 1e-4 LR in the manual case, the loss explodes, become large, then inf, then nan.  So that's no good.  If I use a 1e-6 LR in the model case, the loss decreases far too slowly.

Basically I'm just trying to understand why learning rate means something very different in these two snippets which are otherwise equivalent.

Output:
The crucial difference is the initialization of the weights. The weight matrix in a nn.Linear is initialized smart. I'm pretty sure that if you construct both the models and copy the weight matrices in one way or the other, you'll get consistent behavior.

Additionally, please note that the two models are not equivalent, as your handcrafted model lacks biases. Which matters.


Instruction: Convert a python list of python lists to pytorch tensor

Input:
What is the conventional way to convert python list of lists to PyTorch tensors?

a = [0,0]
b = [1,1]
c = [2]
c = [a, b, c]

I want c to be converted to a flattened Torch tensor as below:

torch([ 0, 0, 1, 1, 2])

Output:
You can flatten your list first in Python:

flat_list = [item for sublist in c for item in sublist]

And create your Tensor:

flattened_tensor = torch.FloatTensor(flat_list)


Instruction: Integrating ROS with pycharm

Input:
I wanted to run ROS in PyCharm, but could not find that .desktop file as mentioned here in which changes should be made. Moreover, I want to use the same environment that is created for PyTorch, do not want to change the interpreter. Can someone help me out with this?

Regards.

Output:
You could add a virtual environment with the following instructions, then you should add ROS distpackages (roslib) on it with this instruction.

File > Settings (or Ctrl+Alt+s as shortcut)> Project:  > Project interpreter.
In the project interpreter dropdown list, you can specify ROS Python interpreter by selecting the appropriate from the list.

ROS distpackages path that you need: /opt/ros/kinetic/lib/python2.7/distpackages


Instruction: What is the best way to use multiprocessing CPU inference for PyTorch models?

Input:
I have to productionize a PyTorch BERT Question Answer model. The CPU inference is very slow for me as for every query the model needs to evaluate 30 samples. Out of the result of these 30 samples, I pick the answer with the maximum score. GPU would be too costly for me to use for inference.

Can I leverage multiprocessing / parallel CPU inference for this?
If Yes, what is the best practice to do so?
If No, is there a cloud option that bills me only for the GPU queries I make and not for continuously running the GPU instance?

Output:
Another possible way to get better performance would be to reduce the model as much as possible.

One of the most promising techniques is quantized and binarized neural networks. Here are some references:

https://arxiv.org/abs/1603.05279
https://arxiv.org/abs/1602.02505


Instruction: How to integrate LIME with PyTorch?

Input:
Using this mnist image classification model :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader_subset):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % print_progress_every_n_epochs == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

predicted_test = []
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
probs_l = []

predicted_values = []
actual_values = []
labels_l = []

with torch.no_grad():
    for images, labels in test_loader_subset:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        predicted_test.append(predicted.cpu().numpy())

        sm = torch.nn.Softmax()
        probabilities = sm(outputs)
        probs_l.append(probabilities)
        labels_l.append(labels.cpu().numpy())

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

if (epoch) % 1 == 0:
    print('test accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

I'm to attempting to integrate 'Local Interpretable Model-Agnostic Explanations for machine learning classifiers' : https://marcotcr.github.io/lime/

It appears PyTorch support is not enabled as it is not mentioned in doc and following tutorial :

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20images.html

With my updated code for PyTorch :

from lime import lime_image
import time

explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(images[0].reshape(28,28), model(images[0]), top_labels=5, hide_color=0, num_samples=1000)

Causes error :

/opt/conda/lib/python3.6/site-packages/skimage/color/colorconv.py in gray2rgb(image, alpha)
    830     is_rgb = False
    831     is_alpha = False
--> 832     dims = np.squeeze(image).ndim
    833
    834     if dims == 3:

AttributeError: 'Tensor' object has no attribute 'ndim'

So appears tensorflow object is expected here ?

How to integrate LIME with PyTorch image classification ?

Output:
Here's my solution:

Lime expects an image input of type numpy. This is why you get the attribute error and a solution would be to convert the image (from Tensor) to numpy before passing it to the explainer object. Another solution would be to select a specific image with the test_loader_subset and convert it with img = img.numpy().

Secondly, in order to make LIME work with pytorch (or any other framework), you'll need to specify a batch prediction function which outputs the prediction scores of each class for each image. The name of this function (here I've called it batch_predict) is then passed to explainer.explain_instance(img, batch_predict, ...). The batch_predict needs to loop through all images passed to it, convert them to Tensor, make a prediction and finally return the prediction score list (with numpy values). This is how I got it working.
Note also that the images need to have shape (... ,... ,3) or (... ,... ,1) in order to be properly segmented by the default segmentation algorithm. This means that you might have to use np.transpose(img, (...)). You may specify the segmentation algorithm as well if the results are poor.

Finally you'll need to display the LIME image mask on top of the original image. This snippet shows how this may be done:

from skimage.segmentation import mark_boundaries
temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=False, num_features=5, hide_rest=False)
img_boundry = mark_boundaries(temp, mask)
plt.imshow(img_boundry)
plt.show()

This notebook is a good reference:
https://github.com/marcotcr/lime/blob/master/doc/notebooks/Tutorial%20-%20images%20-%20Pytorch.ipynb


Instruction: PyTorch specify model parameters

Input:
I am trying to create a convolutional model in PyTorch where

one layer is fixed (initialized to prescribed values)
another layer is learned (but initial guess taken from prescribed values).

Here is a sample code for model definition:

import torch.nn as nn

class Net(nn.Module):
    def __init__(self, weights_fixed, weights_guess):
        super(Net, self).__init__()
        self.convL1 = nn.Conv1d(1, 3, 3, bias=False)
        self.convL1.weight = weights_fixed # I want to keep these weights fixed

        self.convL2 = nn.Conv1d(3, 1, 1, bias=False)
        self.convL1.weight = weights_guess # I want to learn these weights

    def forward(self, inp_batch):
        out1 = self.convL1(inp_batch)
        out2 = self.convL2(out1)

        return out2

and the sample use:

weights_fixed = ...
weights_guess = ...

model = Net(weights_fixed, weights_guess)

loss_fn = nn.CrossEntropyLoss()
optim = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

train_dataset = ... #define training set here

for (X, y) in train_dataset:
    optim.zero_grad()
    out = model(X)
    loss = loss_fn(out, y)
    loss.backward()
    optim.step()

How can I make the weights
weights_fixed - fixed and
weights_guess - learnable?

My guess would be
    weights_fixed = nn.Parameter(W1,requires_grad=False)
    weights_guess = nn.Parameter(W2,requires_grad=True)
where for the sake of completeness
    import numpy as np
    import torch

krnl = np.zeros((5,order+1))
krnl[:,0] = [ 0. , 1., 0. ]
krnl[:,1] = [-0.5, 0., 0.5]
krnl[:,2] = [ 1. ,-2., 1. ]
W1 = torch.tensor(krnl)

a = np.array((1.,2.,3.))
W2 = torch.tensor(a)

But I am utterly confused. Any suggestions or references would be greatly appreciated. Of course I went over PyTorch docs, but it did not add clarity to my understanding.

Output:
Just wrap the learnable parameter with nn.Parameter (requires_grad=True is the default, no need to specify this), and have the fixed weight as a Tensor without nn.Parameter wrapper.

All nn.Parameter weights are automatically added to net.parameters(), so when you do training like optimizer = optim.SGD(net.parameters(), lr=0.01), the fixed weight will not be changed.

So basically this:

weights_fixed = W1
weights_guess = nn.Parameter(W2)


Instruction: How to return intermideate gradients (for non-leaf nodes) in pytorch?

Input:
My question is concerning the syntax of pytorch register_hook.

x = torch.tensor([1.], requires_grad=True)
y = x**2
z = 2*y

x.register_hook(print)
y.register_hook(print)

z.backward()

outputs:

tensor([2.])
tensor([4.])

this snippet simply prints the gradient of z w.r.t x and y, respectively.

Now my (most likely trivial) question is how to return the intermediate gradients (rather than only printing)?

UPDATE:

It appears that calling retain_grad() solves the issue for leaf nodes. ex. y.retain_grad().

However, retain_grad does not seem to solve it for non-leaf nodes. Any suggestions?

Output:
I think you can use those hooks to store the gradients in a global variable:

grads = []
x = torch.tensor([1.], requires_grad=True)
y = x**2 + 1
z = 2*y

x.register_hook(lambda d:grads.append(d))
y.register_hook(lambda d:grads.append(d))

z.backward()

But you most likely also need to remember the corresponding tensor these gradients were computed for. In that case, we slightly extend above using a dict instead of list:

grads = {}
x = torch.tensor([1.,2.], requires_grad=True)
y = x**2 + 1
z = 2*y

def store(grad,parent):
    print(grad,parent)
    grads[parent] = grad.clone()

x.register_hook(lambda grad:store(grad,x))
y.register_hook(lambda grad:store(grad,y))

z.sum().backward()

Now you can, for example, access tensor y's grad simply using grads[y]


Instruction: TypeError: forward() missing 1 required positional argument: 'hidden'

Input:
I'm trying to visualize my GRU model using PyTorchViz but every time I run this code it gives me an error.

I want something like this in the

import torch
from torch import nn
from torchviz import make_dot, make_dot_from_trace

model = IC_V6(f.tokens)
x = torch.randn(1,8)

make_dot(model(x), params=dict(model.named_parameters()))

here is my Class for holding the data

class Flickr8KImageCaptionDataset:

    def __init__(self):

        all_data = json.load(open('caption_datasets/dataset_flickr8k.json', 'r'))
        all_data=all_data['images']

        self.training_data = []
        self.test_data = []
        self.w2i = {ENDWORD: 0, STARTWORD: 1}
        self.word_frequency = {ENDWORD: 0, STARTWORD: 0}
        self.i2w = {0: ENDWORD, 1: STARTWORD}
        self.tokens = 2 #END is default
        self.batch_index = 0

        for data in all_data:
            if(data['split']=='train'):
                self.training_data.append(data)
            else:
                self.test_data.append(data)

            for sentence in data['sentences']:
                for token in sentence['tokens']:
                    if(token not in self.w2i.keys()):
                        self.w2i[token] = self.tokens
                        self.i2w[self.tokens] = token
                        self.tokens +=1
                        self.word_frequency[token] = 1
                    else:
                        self.word_frequency[token] += 1

    def image_to_tensor(self,filename):
        image = Image.open(filename)
        image = TF.resize(img=image, size=(HEIGHT,WIDTH))

        image = TF.to_tensor(pic=image)
        image = TF.normalize(image, mean=[0.485, 0.456, 0.406],
                                     std=[0.229, 0.224, 0.225])

        return torch.unsqueeze(image,0)

    def return_train_batch(self): #size of 1 always

        #np.random.shuffle(self.training_data)
        for index in range(len(self.training_data)):

        #index = np.random.randint(len(self.training_data))
            sentence_index = np.random.randint(len(self.training_data[index]['sentences']))
            output_sentence_tokens = deepcopy(self.training_data[index]['sentences'][sentence_index]['tokens'])
            output_sentence_tokens.append(ENDWORD) #corresponds to end word

            image = self.image_to_tensor('/home/vincent/Documents/Final Code/Flicker8k_Dataset/'+self.training_data[index]['filename'])

            yield image, list(map(lambda x: self.w2i[x], output_sentence_tokens)), output_sentence_tokens, index

    def convert_tensor_to_word(self, output_tensor):
        output = F.log_softmax(output_tensor.detach().squeeze(), dim=0).numpy()
        return self.i2w[np.argmax(output)]

    def convert_sentence_to_tokens(self, sentence):

        tokens = sentence.split(" ")
        converted_tokens= list(map(lambda x: self.w2i[x], tokens))
        converted_tokens.append(self.w2i[ENDWORD])
        return converted_tokens

    def caption_image_greedy(self, net, image_filename, max_words=15): #non beam search, no temperature implemented
        net.eval()
        inception.eval()
        image_tensor = self.image_to_tensor(image_filename)
        hidden=None
        embedding=None
        words = []

        input_token = STARTWORD
        input_tensor = torch.tensor(self.w2i[input_token]).type(torch.LongTensor)

        for i in range(max_words):

            if(i==0):
                out, hidden=net(input_tensor, hidden=image_tensor, process_image=True)
            else:
                out, hidden=net(input_tensor, hidden)

            word = self.convert_tensor_to_word(out)

            input_token = self.w2i[word]
            input_tensor = torch.tensor(input_token).type(torch.LongTensor)

            if(word==ENDWORD):
                break
            else:
                words.append(word)

        return ' '.join(words)

    def forward_beam(self, net, hidden, process_image, partial_sentences, sentences, topn_words=5, max_sentences=10):

        max_words = 50

        hidden_index = {}

        while(sentences<max_sentences):

            #print("Sentences: ",sentences)

            new_partial_sentences = []
            new_partial_sentences_logp = []
            new_partial_avg_logp= []

            if(len(partial_sentences[-1][0])>max_words):
                break

            for partial_sentence in partial_sentences:

                input_token = partial_sentence[0][-1]

                input_tensor = torch.tensor(self.w2i[input_token]).type(torch.FloatTensor)

                if(partial_sentence[0][-1]==STARTWORD):
                    out, hidden=net(input_tensor, hidden, process_image=True)
                else:
                    out, hidden=net(input_tensor, torch.tensor(hidden_index[input_token]))

                #take first topn words and add as children to root

                out = F.log_softmax(out.detach().squeeze(), dim=0).numpy()
                out_indexes = np.argsort(out)[::-1][:topn_words]

                for out_index in out_indexes:

                    if(self.i2w[out_index]==ENDWORD):
                        sentences=sentences+1

                    else:

                        total_logp = float(out[out_index]) + partial_sentence[1]
                        new_partial_sentences_logp.append(total_logp)
                        new_partial_sentences.append([np.concatenate((partial_sentence[0], [self.i2w[out_index]])),total_logp])

                        len_words = len(new_partial_sentences[-1][0])

                        new_partial_avg_logp.append(total_logp/len_words)

                        #print(self.i2w[out_index])

                        hidden_index[self.i2w[out_index]] = deepcopy(hidden.detach().numpy())

            #select topn partial sentences

            top_indexes = np.argsort(new_partial_sentences_logp)[::-1][:topn_words]

            new_partial_sentences = np.array(new_partial_sentences)[top_indexes]

            #print("New partial sentences (topn):", new_partial_sentences)

            partial_sentences = new_partial_sentences

        return partial_sentences

    def caption_image_beam_search(self, net, image_filename, topn_words=10, max_sentences=10):

        net.eval()
        inception.eval()
        image_tensor = self.image_to_tensor(image_filename)
        hidden=None
        embedding=None
        words = []

        sentences = 0
        partial_sentences = [[[STARTWORD], 0.0]]
        #root_id = hash(input_token) #for start word
        #nodes = {}

        #nodes[root_id] = Node(root_id, [STARTWORD, 0], None)

        partial_sentences = self.forward_beam(net, image_tensor, True, partial_sentences, sentences,  topn_words, max_sentences)

        logp = []

        joined_sentences = []

        for partial_sentence in partial_sentences:

            joined_sentences.append([' '.join(partial_sentence[0][1:]),partial_sentence[1]])

        return joined_sentences

    def print_beam_caption(self, net, train_filename,num_captions=0):
        beam_sentences = f.caption_image_beam_search(net,train_filename)
        if(num_captions==0):
            num_captions=len(beam_sentences)
        for sentence in beam_sentences[:num_captions]:
            print(sentence[0]+" [",sentence[1], "]")

and here is my GRU model

class IC_V6(nn.Module):

    #V2: Fed image vector directly as hidden and fed words generated as iputs back to LSTM
    #V3: Added an embedding layer between words input and GRU/LSTM

    def __init__(self, token_dict_size):
        super(IC_V6, self).__init__()

        #Input is an image of height 500, and width 500

        self.embedding_size = INPUT_EMBEDDING
        self.hidden_state_size = HIDDEN_SIZE
        self.token_dict_size = token_dict_size
        self.output_size = OUTPUT_EMBEDDING

        self.batchnorm = nn.BatchNorm1d(self.embedding_size)

        self.input_embedding = nn.Embedding(self.token_dict_size, self.embedding_size)

        self.embedding_dropout = nn.Dropout(p=0.22)
        self.gru_layers = 3

        self.gru = nn.GRU(input_size=self.embedding_size, hidden_size=self.hidden_state_size, num_layers=self.gru_layers, dropout=0.22)
        self.linear = nn.Linear(self.hidden_state_size, self.output_size)
        self.out = nn.Linear(self.output_size, token_dict_size)

    def forward(self, input_tokens, hidden, process_image=False, use_inception=True):

        if(USE_GPU):
            device = torch.device('cuda')
        else:
            device = torch.device('cpu')

        if(process_image):
            if(use_inception):
                inp=self.embedding_dropout(inception(hidden))
            else:
                inp=hidden
            #inp=self.batchnorm(inp)
            hidden=torch.zeros((self.gru_layers,1, self.hidden_state_size))
        else:
            inp=self.embedding_dropout(self.input_embedding(input_tokens.view(1).type(torch.LongTensor).to(device)))
            #inp=self.batchnorm(inp)

        hidden = hidden.view(self.gru_layers,1,-1)
        inp = inp.view(1,1,-1)

        out, hidden = self.gru(inp, hidden)
        out = self.out(self.linear(out))

        return out, hidden

this is how I called them:

f = Flickr8KImageCaptionDataset()
net = IC_V6(f.tokens)

the error is:

TypeError Traceback (most recent call last)
<ipython-input-42-7993fc1a032f> in <module>
      6 x = torch.randn(1,8)
      7
----> 8 make_dot(model(x), params=dict(model.named_parameters()))

~/anaconda3/envs/Thesis/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

  TypeError: forward() missing 1 required positional argument: 'hidden'

What should I do to solve this problem? Any help will be much appreciated.

Output:
I think the error message is pretty straight forward. You have two positional arguments input_tokens and hidden for your forward().

Python complains that one of them (hidden)  is missing when you call your forward() function.

Looking at your code, you call your forward like this:

model(x)

So x is mapped to input_tokens, but you need to hand over a second argument hidden.

So you need to call it like this, providing your hidden state:

model(x, hidden)


Instruction: Error when trying to send neural net generated image in flask

Input:
I'm making a basic api to request for generated images from a generator model from pytorch. I've done this using flask and I'm running it locally on MacOS. Everything works and the image returns but then python quits unexpectedly. Here is the code and the error:

Error:

   2019-03-25 16:21:23.514 Python[78776:1407049] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (
    0   AppKit                              0x00007fff4f96fccc -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 386
    1   AppKit                              0x00007fff4f96d07c -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1488
    2   AppKit                              0x00007fff4f96caa6 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45
    3   _macosx.cpython-37m-darwin.so       0x000000010fe634c0 -[Window initWithContentRect:styleMask:backing:defer:withManager:] + 80
    4   _macosx.cpython-37m-darwin.so       0x000000010fe66a17 FigureManager_init + 327
    5   Python                              0x00000001023780bc wrap_init + 12
    6   Python                              0x000000010232fe09 wrapperdescr_call + 121
    7   Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    8   Python                              0x00000001023e76d4 call_function + 420
    9   Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    10  Python                              0x0000000102329100 function_code_fastcall + 128
    11  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    12  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    13  Python                              0x0000000102378001 slot_tp_init + 145
    14  Python                              0x0000000102373959 type_call + 297
    15  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    16  Python                              0x00000001023e76d4 call_function + 420
    17  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    18  Python                              0x0000000102329100 function_code_fastcall + 128
    19  Python                              0x00000001023e7812 call_function + 738
    20  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    21  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    22  Python                              0x000000010232886b _PyFunction_FastCallDict + 523
    23  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    24  Python                              0x0000000102328df7 PyObject_Call + 135
    25  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    26  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    27  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    28  Python                              0x00000001023e7812 call_function + 738
    29  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    30  Python                              0x0000000102329100 function_code_fastcall + 128
    31  Python                              0x00000001023e7812 call_function + 738
    32  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    33  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    34  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    35  Python                              0x00000001023e7812 call_function + 738
    36  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    37  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    38  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    39  Python                              0x00000001023e7812 call_function + 738
    40  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    41  Python                              0x0000000102329100 function_code_fastcall + 128
    42  Python                              0x00000001023e7812 call_function + 738
    43  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    44  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    45  Python                              0x000000010232886b _PyFunction_FastCallDict + 523
    46  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    47  Python                              0x0000000102329100 function_code_fastcall + 128
    48  Python                              0x00000001023e7812 call_function + 738
    49  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    50  Python                              0x0000000102329100 function_code_fastcall + 128
    51  Python                              0x00000001023e7812 call_function + 738
    52  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    53  Python                              0x0000000102329100 function_code_fastcall + 128
    54  Python                              0x00000001023e7812 call_function + 738
    55  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    56  Python                              0x0000000102329100 function_code_fastcall + 128
    57  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    58  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    59  Python                              0x0000000102376bc6 slot_tp_call + 150
    60  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    61  Python                              0x00000001023e76d4 call_function + 420
    62  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    63  Python                              0x00000001023370de gen_send_ex + 206
    64  Python                              0x00000001023e3fb8 _PyEval_EvalFrameDefault + 23112
    65  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    66  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    67  Python                              0x00000001023e7812 call_function + 738
    68  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    69  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    70  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    71  Python                              0x00000001023e7812 call_function + 738
    72  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    73  Python                              0x0000000102329100 function_code_fastcall + 128
    74  Python                              0x00000001023e7812 call_function + 738
    75  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    76  Python                              0x0000000102329100 function_code_fastcall + 128
    77  Python                              0x00000001023e7812 call_function + 738
    78  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    79  Python                              0x0000000102329100 function_code_fastcall + 128
    80  Python                              0x00000001023e7812 call_function + 738
    81  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    82  Python                              0x0000000102329100 function_code_fastcall + 128
    83  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    84  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    85  Python                              0x0000000102378001 slot_tp_init + 145
    86  Python                              0x0000000102373959 type_call + 297
    87  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    88  Python                              0x00000001023e76d4 call_function + 420
    89  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    90  Python                              0x0000000102329100 function_code_fastcall + 128
    91  Python                              0x00000001023e7812 call_function + 738
    92  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    93  Python                              0x0000000102329100 function_code_fastcall + 128
    94  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    95  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    96  Python                              0x0000000102328df7 PyObject_Call + 135
    97  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    98  Python                              0x0000000102329100 function_code_fastcall + 128
    99  Python                              0x00000001023e7812 call_function + 738
    100 Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    101 Python                              0x0000000102329100 function_code_fastcall + 128
    102 Python                              0x00000001023e7812 call_function + 738
    103 Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    104 Python                              0x0000000102329100 function_code_fastcall + 128
    105 Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    106 Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    107 Python                              0x0000000102328df7 PyObject_Call + 135
    108 Python                              0x000000010246fbd7 t_bootstrap + 71
    109 Python                              0x0000000102426819 pythread_wrapper + 25
    110 libsystem_pthread.dylib             0x00007fff7f7ce305 _pthread_body + 126
    111 libsystem_pthread.dylib             0x00007fff7f7d126f _pthread_start + 70
    112 libsystem_pthread.dylib             0x00007fff7f7cd415 thread_start + 13
)
127.0.0.1 - - [25/Mar/2019 16:21:23] "GET /sdfa HTTP/1.1" 200 -
Assertion failed: (NSViewIsCurrentlyBuildingLayerTreeForDisplay() != currentlyBuildingLayerTree), function NSViewSetCurrentlyBuildingLayerTreeForDisplay, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1671.20.108/AppKit.subproj/NSView.m, line 14143

Main:

from flask import Flask, request, send_file
import loadModel

app = Flask(__name__)

@app.route('/')
def index():
    return 'this is the homepage'

@app.route('/<ganType>')
def generate(ganType):
    loadModel.loadModel()
    return send_file('fakes.jpg')

if __name__ == "__main__":
    app.run(debug=True)

Load Model:

import torch
import torch.nn as nn
import torchvision.utils as vutils
import matplotlib.pyplot as plt
import numpy as np

batch_size = 100
image_size = 64
nc = 3
nz = 100
ngf = 64
ndf = 64
num_epochs = 1
lr = 0.0002
beta1 = 0.5

class Generator(nn.Module): # nn.Module is the base class for all neural net modules.
    def __init__(self):
        super(Generator, self).__init__() # Calls the parent's initialization method

        self.main = nn.Sequential(
             # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)

def loadModel():

    # device = torch.device("cpu")
    model = Generator()
    model.load_state_dict(torch.load('generator.pt', map_location='cpu'))

    model.eval()
    first_list = []
    img_list = []
    # Generate some images to test the model.

    noise = torch.randn(64, nz, 1, 1)
    fake = model(noise).detach().cpu()
    # inception_img_list.append(np.transpose(fake[0],(1,2,0)))
    img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

    # img_list.append(first_list)
    # plt.subplot(1,2,2)
    plt.axis("off")
    plt.title("Fake Images")
    plt.imsave('fakes.jpg', np.transpose(img_list[-1],(1,2,0)))

    return 'hello'

if __name__ == "__main__":

    loadModel()

As I said, it actually works and returns the generated image. It just also crashes python. I'm not sure if I am doing something incorrectly or if its a MacOs problem. Any help would be appreciated.

Output:
I worked out from people having the same problem: https://github.com/matplotlib/matplotlib/issues/11094

Closing plt before returning fixes the error.


Instruction: In language modeling, why do I have to init_hidden weights before every new epoch of training? (pytorch)

Input:
I have a question about the following code in pytorch language modeling:

print("Training and generating...")
    for epoch in range(1, config.num_epochs + 1):
        total_loss = 0.0
        model.train()
        hidden = model.init_hidden(config.batch_size)

        for ibatch, i in enumerate(range(0, train_len - 1, seq_len)):
            data, targets = get_batch(train_data, i, seq_len)
            hidden = repackage_hidden(hidden)
            model.zero_grad()

            output, hidden = model(data, hidden)
            loss = criterion(output.view(-1, config.vocab_size), targets)
            loss.backward()

Please check line 5.

And the init_hidden function is as follows:

def init_hidden(self, bsz):
    weight = next(self.parameters()).data
    if self.rnn_type == 'LSTM':  # lstm：(h0, c0)
        return (Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()),
                Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_()))
    else:  # gru & rnn：h0
        return Variable(weight.new(self.n_layers, bsz, self.hi_dim).zero_())

My question is:

Why do we need to init_hidden every epoch? Shouldn't it be that the model inherit the hidden parameters from last epoch and continue training on them.

Output:
The hidden state stores the internal state of the RNN from predictions made on previous tokens in the current sequence, this allows RNNs to understand context. The hidden state is determined by the output of the previous token.

When you predict for the first token of any sequence, if you were to retain the hidden state from the previous sequence your model would perform as if the new sequence was a continuation of the old sequence which would give worse results. Instead for the first token you initialise an empty hidden state, which will then be filled with the model state and used for the second token.

Think about it this way: if someone asked you to classify a sentence and handed you the US constitution (irrelevant information) vs. if someone gave you some background context about the sentence and then asked you to classify the sentence.