Instruction: Pytorch - Concatenating Datasets before using Dataloader

Input:
I am trying to load two datasets and use them both for training.

Package versions: python 3.7;
pytorch 1.3.1

It is possible to create data_loaders seperately and train on them sequentially:

from torch.utils.data import DataLoader, ConcatDataset

train_loader_modelnet = DataLoader(ModelNet(args.modelnet_root, categories=args.modelnet_categories,split='train', transform=transform_modelnet, device=args.device),batch_size=args.batch_size, shuffle=True)

train_loader_mydata = DataLoader(MyDataset(args.customdata_root, categories=args.mydata_categories, split='train', device=args.device),batch_size=args.batch_size, shuffle=True)

for e in range(args.epochs):
    for idx, batch in enumerate(tqdm(train_loader_modelnet)):
        # training on dataset1
    for idx, batch in enumerate(tqdm(train_loader_custom)):
        # training on dataset2

Note: MyDataset is a custom dataset class which has def __len__(self): def __getitem__(self, index): implemented. As the above configuration works it seems that this is implementation is OK.

But I would ideally like to combine them into a single dataloader object. I attempted this as per the pytorch documentation:

train_modelnet = ModelNet(args.modelnet_root, categories=args.modelnet_categories,
                          split='train', transform=transform_modelnet, device=args.device)
train_mydata = CloudDataset(args.customdata_root, categories=args.mydata_categories,
                             split='train', device=args.device)
train_loader = torch.utils.data.ConcatDataset(train_modelnet, train_customdata)

for e in range(args.epochs):
    for idx, batch in enumerate(tqdm(train_loader)):
        # training on combined

However, on random batches I get the following 'expected a tensor as element X in argument 0, but got a tuple instead' type of error. Any help would be much appreciated!

>   40%|████      | 53/131 [01:03<02:00,  1.55s/it]
>  Traceback (mostrecent call last):   File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/pydevd.py",
> line 1434, in _exec
>     pydev_imports.execfile(file, globals, locals)  # execute the script   File
> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
>     exec(compile(contents+"\n", file, 'exec'), glob, loc)   File "/home/chris/Documents/4yp/Data/my_kaolin/Classification/pointcloud_classification_combinedset.py",
> line 83, in <module>
>     for idx, batch in enumerate(tqdm(train_loader)):   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/tqdm/std.py",
> line 1107, in __iter__
>     for obj in iterable:   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/dataloader.py",
> line 346, in __next__
>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration   File
> "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py",
> line 47, in fetch
>     return self.collate_fn(data)   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in default_collate
>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 79, in <listcomp>
>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",
> line 55, in default_collate
>     return torch.stack(batch, 0, out=out) TypeError: expected Tensor as element 3 in argument 0, but got tuple

Output:
If I got your question right, you have train and dev sets (and their corresponding loaders) as follows:
train_set = CustomDataset(...)
train_loader = DataLoader(dataset=train_set, ...)
dev_set = CustomDataset(...)
dev_loader = DataLoader(dataset=dev_set, ...)

And you want to concatenate them in order to use train+dev as the training data, right? If so, you just simply call:
train_dev_sets = torch.utils.data.ConcatDataset([train_set, dev_set])
train_dev_loader = DataLoader(dataset=train_dev_sets, ...)

The train_dev_loader is the loader containing data from both sets.
Now, be sure your data has the same shapes and the same types, that is, the same number of features, or the same categories/numbers, etc.


Instruction: Compute grads of cloned tensor Pytorch

Input:
I am having a hard time with gradient computation using PyTorch.

I have the outputs and the hidden states of the last time step T of an RNN.

I would like to clone my hidden states and compute its grad after backpropagation but it doesn't work.

After reading pytorch how to compute grad after clone a tensor, I used  retain_grad() without any success.

Here's my code

        hidden_copy = hidden.clone()
        hidden.retain_grad()
        hidden_copy.retain_grad()
        outputs_T = outputs[T]
        targets_T = targets[T]
        loss_T = loss(outputs_T,targets_T)
        loss_T.backward()
        print(hidden.grad)
        print(hidden_copy.grad)

hidden_grad gives an array while hidden_copy.grad gives None.

Why does hidden_copy.grad give None ? Is there any way to compute the gradients of a cloned tensor ?

Output:
Based on the comments the problem is that hidden_copy is never visited during the backward pass.

When you perform backward pytorch follows the computation graph backwards starting at loss_T and works backwards to all the leaf nodes. It only visits the tensors which were used to compute loss_T. If a tensor isn't part of that backward path then it won't be visited and it's grad member will not be updated. Basically by creating a copy of the tensor and then not using it to compute loss_T this results in a "dead-end" in the computation graph.

To illustrate take a look at this diagram representing a simplified view of a computation graph. Each node in the graph is a tensor where the edges point back to direct descendants.

Notice if we follow the path back from loss_T to the leaves then we never visit hidden_conv. Note that a leaf is a tensor with no descendants and in this case input is the only leaf.

This is an extremely simplified computation graph used to demonstrate a point. Of course in reality there are probably many more nodes between input and hidden and between hidden and output_T as well as other leaf tensors since the weights of layers are almost certainly leaves.


Instruction: Slice 4d tensor into 4D tensor of smaller subtensors (slice in last 2 dimensions only)

Input:
The question is analogous to Slice 2d array into smaller 2d arrays except for the fact that I use tensors (torch) & I have a 4D, not 2D, tensor of the shape eg. (3, 1, 32, 32) - in my case, it is 3 images of size 32x32.

I want to split each tensor of form [i, 0, :, :] into smaller subarrays, so the output would have a shape eg. (3, 16, 8, 8), where each [:, j, :, :] is a small square cut from the original image. I cannot find a way to modify the proposed solution for 4D tensor.

I also tried to just use

subx = x.reshape(3, 16, 8, 8)

but this does not reshape it as I want.

Output:
reshape will not work for this purpose. You could look into skimage's view_as_blocks, where the resulting blocks are non-overlapping views of the input array:

from skimage.util.shape import view_as_blocks
view_as_blocks(a, block_shape=(3,1,8,8)).reshape(3, 16, 8, 8)


Instruction: k-fold cross validation using DataLoaders in PyTorch

Input:
I have splitted my training dataset into 80% train and 20% validation data and created DataLoaders as shown below. However I do not want to limit my model's training. So I thought of splitting my data into K(maybe 5) folds and performing cross-validation. However I do not know how to combine the datasets to my dataloader after splitting them.
train_size = int(0.8 * len(full_dataset))
validation_size = len(full_dataset) - train_size
train_dataset, validation_dataset = random_split(full_dataset, [train_size, validation_size])

full_loader = DataLoader(full_dataset, batch_size=4,sampler = sampler_(full_dataset), pin_memory=True)
train_loader = DataLoader(train_dataset, batch_size=4, sampler = sampler_(train_dataset))
val_loader = DataLoader(validation_dataset, batch_size=1, sampler = sampler_(validation_dataset))

Output:
I just wrote a cross validation function work with dataloader and dataset.
Here is my code, hope this is helpful.
# define a cross validation function
def crossvalid(model=None,criterion=None,optimizer=None,dataset=None,k_fold=5):

    train_score = pd.Series()
    val_score = pd.Series()

    total_size = len(dataset)
    fraction = 1/k_fold
    seg = int(total_size * fraction)
    # tr:train,val:valid; r:right,l:left;  eg: trrr: right index of right side train subset
    # index: [trll,trlr],[vall,valr],[trrl,trrr]
    for i in range(k_fold):
        trll = 0
        trlr = i * seg
        vall = trlr
        valr = i * seg + seg
        trrl = valr
        trrr = total_size
        # msg
#         print("train indices: [%d,%d),[%d,%d), test indices: [%d,%d)"
#               % (trll,trlr,trrl,trrr,vall,valr))

        train_left_indices = list(range(trll,trlr))
        train_right_indices = list(range(trrl,trrr))

        train_indices = train_left_indices + train_right_indices
        val_indices = list(range(vall,valr))

        train_set = torch.utils.data.dataset.Subset(dataset,train_indices)
        val_set = torch.utils.data.dataset.Subset(dataset,val_indices)

#         print(len(train_set),len(val_set))
#         print()

        train_loader = torch.utils.data.DataLoader(train_set, batch_size=50,
                                          shuffle=True, num_workers=4)
        val_loader = torch.utils.data.DataLoader(val_set, batch_size=50,
                                          shuffle=True, num_workers=4)
        train_acc = train(res_model,criterion,optimizer,train_loader,epoch=1)
        train_score.at[i] = train_acc
        val_acc = valid(res_model,criterion,optimizer,val_loader)
        val_score.at[i] = val_acc

    return train_score,val_score

train_score,val_score = crossvalid(res_model,criterion,optimizer,dataset=tiny_dataset)

In order to give an intuition of correctness for what we are doing, see the output below:
train indices: [0,0),[3600,18000), test indices: [0,3600)
14400 3600

train indices: [0,3600),[7200,18000), test indices: [3600,7200)
14400 3600

train indices: [0,7200),[10800,18000), test indices: [7200,10800)
14400 3600

train indices: [0,10800),[14400,18000), test indices: [10800,14400)
14400 3600

train indices: [0,14400),[18000,18000), test indices: [14400,18000)
14400 3600


Instruction: pytorch .cuda() can't get the tensor to cuda

Input:
I try to get my data onto gpu,but it doesn't work;

in my train.py

if __name__ == '__main__':
ten = torch.FloatTensor(2)
ten = ten.cuda()
print(ten)

args = config()
train_net(args, args.train_net, loss_config=net_loss_config[args.train_net])

when it runs ,it prints this

tensor([0., 0.])

the tensor is not on cuda

but in test.py

import torch
ten=torch.FloatTensor(2)
ten=ten.cuda()
print(ten)

it prints this

tensor([1.4013e-45, 0.0000e+00], device='cuda:0')

now the tensor is on cuda

Output:
The error means that the ten variable in your model is of type torch.FloatTensor (CPU), while the input you provide to the model is of type torch.cuda.FloatTensor (GPU).

The most likely scenario is that you have nn.Parameter or other modules such as nn.Conv2d defined in the __init__() method of your model, and additional weights or layers defined in the forward() method of your model.

In this case, the layers defined in the forward() method are not modules of the model, and they won’t be mapped to GPU when you call cuda().

You also need to explicitly add the parameters to your optimizer if you want them to be updated with gradient descent.


Instruction: LSTM in PyTorch Classifying Names

Input:
I am trying the example presented in https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html but I am using a LSTM model instead of a RNN. The dataset is composed by different names (of different sizes) and their corresponding language (total number of languages is 18), and the objective is to train a model that given a certain name outputs the language it belongs to.

My problems right now are:

How to deal with variable size names, i.e. Hector and Kim, in the LSTM
A whole name (secuence of character) is processed every time in the LSTM so the output of the softmax function has shape (#characters of name, #target classes) but I would like just to obtain (1,#target of classes) in order to decide each name to which class does it correspond to. I have tried to just get the last row but results are very bad.

class LSTM(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim

        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)
        self.softmax = nn.LogSoftmax(dim = 1)

    def forward(self, word):
        embeds = self.word_embeddings(word)
        lstm_out, _ = self.lstm(embeds.view(len(word), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(word), -1))
        tag_scores = self.softmax(tag_space)
        return tag_scores

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_dim))

    lstm = LSTM(n_embedding_dim,n_hidden,n_characters,n_categories)
    optimizer = torch.optim.SGD(lstm.parameters(), lr=learning_rate)
    criterion = nn.NLLLoss()
    def train(category_tensor, line_tensor):
        # i.e. line_tensor = tensor([37,  4, 14, 13, 19,  0, 17,  0, 10,  8, 18]) and category_tensor = tensor([11])
        optimizer.zero_grad()
        output = lstm(line_tensor)

        loss = criterion(output[-1:], category_tensor) # VERY BAD
        loss.backward()

        optimizer.step()

        return output, loss.data.item()

Where line_tensor  is of variable size (depending the size of each name) and is a mapping between character and their index in the dictionary

Output:
Lets dig into the solution step by step

Frame the problem

Given your problem statement, you will have to use LSTM for making a classification rather then its typical use of tagging. The LSTM is unrolled for certain timestep and this is the reason why input and ouput dimensions of a recurrent models are

Input: batch size X time steps X input size
Output: batch size X time steps X hidden size

Now since you want to use it for classification, you have two options:

Put a dense layer over the output of all the timesteps/unrollings [My example below uses this]
Ignore the all timestep outputs except the last, and put a dense layer over the last timestep

So the input to our LSTM model are the names fed as one character per LSTM timestep and output will be the class corresponding to its language.

How to handle variable length inputs/names

We have two options again here.

Batch same length names together. This is called bucketing
Fix max length based on the average size of names you have. Pad the smaller names and chop off the longer names [My example below uses max length of 10]

Do we need Embedding layer ?

No. Embedding layers are typically used to learn a good vector representations of words. But in the case of character model, the input is a character not a word so adding an embedding layers does not help. Character can be directly encoded to number and embedding layers does very little in capturing relationship between different characters. You can still use embedding layer, but I strongly believe it will not help.

Toy character LSTM model code

import numpy as np
import torch
import torch.nn as nn

# Model architecture
class Recurrent_Model(nn.Module):
    def __init__(self, output_size, time_steps=10):
        super(Recurrent_Model, self).__init__()
        self.time_steps = time_steps
        self.lstm = nn.LSTM(1,32, bidirectional=True, num_layers=2)
        self.linear = nn.Linear(32*2*time_steps, output_size)

    def forward(self, x):
        lstm_out, _ = self.lstm(x)
        return self.linear(lstm_out.view(-1,32*2*self.time_steps))

# Sample input and output
names = ['apple', 'dog', 'donkey', "elephant", "hippopotamus"]
lang = [0,1,2,1,0]

def pad_sequence(name, max_len=10):
    x = np.zeros((len(name), max_len))
    for i, name in enumerate(names):
        for j, c in enumerate(name):
            if j >= max_len:
                break
            x[i,j] = ord(c)
    return torch.FloatTensor(x)

x = pad_sequence(names)
x = torch.unsqueeze(x, dim=2)
y = torch.LongTensor(lang)

model = Recurrent_Model(3)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), 0.01)

for epoch in range(500):
    model.train()
    output = model(x)
    loss = criterion(output, y)
    print (f"Train Loss: {loss.item()}")
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Note

All the tensors are loaded into memory so if you have huge dataset, you will have to use a dataset and dataloader to avoid OOM error.
You will have to split data into train test and validate on the test dataset (the standard model building stuff)
You will have to normalize the input tensors before passing it to the model (again the standard model building stuff)

Finally

so how do you make sure your model architecture does not have bugs or is learning. As Andrej karpathy says, overfit the model on a small dataset and if it is overfitting then we are fine.


Instruction: How to prevent inf while working with exponential

Input:
I'm trying to create  a function in a network with trainable parameters. In my function I have an exponential that for large tensor values goes to infinity. What would the best way to avoid this be?

The function is as follows:

step1 = Pss-(k*Pvv)
step2 = step1*s
step3 = torch.exp(step2)
step4 = torch.log10(1+step3)
step5 = step4/s

#or equivalently
# train_curve = torch.log(1+torch.exp((Pss-k*Pvv)*s))/s

If it makes it easier to understand, the basic function is log10(1+e^(x-const)*10)/10. The exponential inside the log gets too big and goes to inf.

I think I might have to normalize my tensor x, and this would mean normalizing the constants and the rest of the function also. Would someone have any thoughts on the best way to go about this?

Thanks so much.

Output:
One solution is to just use a more stable computation. Notice that log(1 + exp(x)) is approximately equal to x when x is large enough. Intuitively this can be observed by noting that, for example, exp(50) is approximately 5.18e+21 for which adding 1 will have no effect when using 32-bit floating point arithmetic like PyTorch does. Further verification using an arbitrary precision calculator shows that the error in this approximation at 50 is far outside the maximum 32-bit floating point precision (which is about 7 decimal digits).
Using this information we can implement a simple piecewise function in PyTorch for which we use log1p(exp(x)) for values less than 50 and x for values greater than 50. Also note that this function is autograd compatible
def log1pexp(x):
    # more stable version of log(1 + exp(x))
    return torch.where(x < 50, torch.log1p(torch.exp(x)), x)

This get's us most of the way to a solution since you actually want to evaluate torch.log10(1+torch.exp((Pss-k*Pvv)*s))/s
Now we can use our new log1pexp function to compute this expression without worrying about infinities
(log1pexp((Pss - k*Pvv)*s) / math.log(10)) / s

and mind the conversion from natural log to log base-10 by dividing by log(10).


Instruction: How do I add an image title to tensorboardX?

Input:
I am currently using tensorboardX to visualize input images while training a ResNet image classifier. Is there a way to add the image title along with the added image? I would like to have the image name (as stored in the dataset) displayed below the image in the tensorboard display.

So far I have tried passing a comment parameter into my tensorboard writer, which does not seem to do the job.
Currently, the relevant lines of my code are:

pretrain_train_writer = SummaryWriter('log/pretrain_train')
img_grid = vutils.make_grid(inputs[tp_idx_0], normalize=True, scale_each=True, nrow=8)
pretrain_val_writer.add_image('true_positive_class_0', img_grid, global_step=epoch, comment = img_path)

Output:
there is no way of doing it directly with tensorboard, instead you have to create images with titles using matplotlib and then supply them to tensorboard. Here is a sample code from the tensorboard documentation:
def plot_to_image(figure):
  """Converts the matplotlib plot specified by 'figure' to a PNG image and
  returns it. The supplied figure is closed and inaccessible after this call."""
  # Save the plot to a PNG in memory.
  buf = io.BytesIO()
  plt.savefig(buf, format='png')
  # Closing the figure prevents it from being displayed directly inside
  # the notebook.
  plt.close(figure)
  buf.seek(0)
  # Convert PNG buffer to TF image
  image = tf.image.decode_png(buf.getvalue(), channels=4)
  # Add the batch dimension
  image = tf.expand_dims(image, 0)
  return image

def image_grid():
  """Return a 5x5 grid of the MNIST images as a matplotlib figure."""
  # Create a figure to contain the plot.
  figure = plt.figure(figsize=(10,10))
  for i in range(25):
    # Start next subplot.
    plt.subplot(5, 5, i + 1, title=class_names[train_labels[i]])
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)

  return figure

# Prepare the plot
figure = image_grid()
# Convert to image and log
with file_writer.as_default():
  tf.summary.image("Training data", plot_to_image(figure), step=0)

here is the link to the doc: https://www.tensorflow.org/tensorboard/image_summaries


Instruction: Why PyTorch nn.Module.cuda() not moving Module tensor but only parameters and buffers to GPU?

Input:
nn.Module.cuda() moves all model parameters and buffers to the GPU.

But why not the model member tensor?

class ToyModule(torch.nn.Module):
    def __init__(self) -> None:
        super(ToyModule, self).__init__()
        self.layer = torch.nn.Linear(2, 2)
        self.expected_moved_cuda_tensor = torch.tensor([0, 2, 3])

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        return self.layer(input)

toy_module = ToyModule()
toy_module.cuda()

next(toy_module.layer.parameters()).device
>>> device(type='cuda', index=0)

for the model member tensor, the device stays unchanged.

>>> toy_module.expected_moved_cuda_tensor.device
device(type='cpu')

Output:
If you define a tensor inside the module it needs to be registered as either a parameter or a buffer so that the module is aware of it.

Parameters are tensors that are to be trained and will be returned by model.parameters(). They are easy to register, all you need to do is wrap the tensor in the nn.Parameter type and it will be automatically registered. Note that only floating point tensors can be parameters.

class ToyModule(torch.nn.Module):
    def __init__(self) -> None:
        super(ToyModule, self).__init__()
        self.layer = torch.nn.Linear(2, 2)
        # registering expected_moved_cuda_tensor as a trainable parameter
        self.expected_moved_cuda_tensor = torch.nn.Parameter(torch.tensor([0., 2., 3.]))

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        return self.layer(input)

Buffers are tensors that will be registered in the module so methods like .cuda() will affect them but they will not be returned by model.parameters(). Buffers are not restricted to a particular data type.

class ToyModule(torch.nn.Module):
    def __init__(self) -> None:
        super(ToyModule, self).__init__()
        self.layer = torch.nn.Linear(2, 2)
        # registering expected_moved_cuda_tensor as a buffer
        # Note: this creates a new member variable named expected_moved_cuda_tensor
        self.register_buffer('expected_moved_cuda_tensor', torch.tensor([0, 2, 3])))

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        return self.layer(input)

In both of the above cases the following code behaves the same

>>> toy_module = ToyModule()
>>> toy_module.cuda()
>>> next(toy_module.layer.parameters()).device
device(type='cuda', index=0)
>>> toy_module.expected_moved_cuda_tensor.device
device(type='cuda', index=0)


Instruction: Error trying to convert simple convolutional model to CoreML

Input:
I'm trying to convert a simple GAN generator (from ClusterGAN):

self.name = 'generator'
self.latent_dim = latent_dim
self.n_c = n_c
self.x_shape = x_shape
self.ishape = (128, 7, 7)
self.iels = int(np.prod(self.ishape))
self.verbose = verbose

self.model = nn.Sequential(
    # Fully connected layers
    torch.nn.Linear(self.latent_dim + self.n_c, 1024),
    nn.BatchNorm1d(1024),
    nn.LeakyReLU(0.2, inplace=True),
    torch.nn.Linear(1024, self.iels),
    nn.BatchNorm1d(self.iels),
    nn.LeakyReLU(0.2, inplace=True),

    # Reshape to 128 x (7x7)
    Reshape(self.ishape),

    # Upconvolution layers
    nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1, bias=True),
    nn.BatchNorm2d(64),
    nn.LeakyReLU(0.2, inplace=True),

    nn.ConvTranspose2d(64, 1, 4, stride=2, padding=1, bias=True),
    nn.Sigmoid()
)

But onnx-coreml fails with Error while converting op of type: BatchNormalization. Error message: provided number axes 2 not supported

I thought it was the BatchNorm2d, so I tried reshaping and applying BatchNorm1d, but I get the same error. Any thoughts? I'm very surprised that I'm having problems converting such a simple model, so I'm assuming that I must be missing something obvious.

I'm targeting iOS 13 and using Opset v10 for the onnx conversion.

Output:
Core ML does not have 1-dimensional batch norm. The tensor must have at least rank 3.

If you want to convert this model, you should fold the batch norm weights into those of the preceding layer and remove the batch norm layer. (I don't think PyTorch has a way to automatically do this for you.)


Instruction: Pytorch cuda get_device_name and current_device() hang and are killed?

Input:
I've just installed a new GPU (RTX 2070) in my machine alongside the old GPU. I wanted to see if PyTorch picked up it, so following the instructions here: How to check if pytorch is using the GPU?, I ran the following commands (Python3.6.9, Linux Mint Tricia 19.3)

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
Killed
>>> torch.cuda.get_device_name(0)
Killed

Both of the two Killed processes took some time and one of them froze the machine for half a minute or so. Does anyone have any experience with this? Are there some setup steps I'm missing?

Output: If I understand correctly, you would like to list the available cuda devices. This can be done via nvidia-smi (not a PyTorch function), and both your old GPU and the RTX 2070 should show up, as devices 0 and 1. In PyTorch, if you want to pass data to one specific device, you can do device = torch.device("cuda:0") for GPU 0 and device = torch.device("cuda:1") for GPU 1. While running, you can do nvidia-smi to check the memory usage & running processes for each GPU.


Instruction: KeyError: 'answers' error when using BioASQ dataset using Huggingface Transformers

Input:
I am using run_squad.py https://github.com/huggingface/transformers/blob/master/examples/run_squad.py from Huggingface Transformers for fine-tuning on BioASQ Question Answering dataset.

I have converted the tensorflow weights provided by the authors of BioBERT https://github.com/dmis-lab/bioasq-biobert to Pytorch as discussed here https://github.com/huggingface/transformers/issues/312.

Further, I am using the preprocessed data of BioASQ https://github.com/dmis-lab/bioasq-biobert which is converted to the SQuAD form.
However, when I am running the run_squad.py script with the below parameters

 --model_type bert \
  --model_name_or_path /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/BioBERTv1.1-SQuADv1.1-Factoid-PyTorch/ \
  --do_train \
  --do_eval \
  --save_steps 1000 \
  --train_file $data/BioASQ-train-factoid-6b.json \
  --predict_file $data/BioASQ-test-factoid-6b-1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/QA_output_squad/BioASQ-factoid-6b/BioASQ-factoid-6b-1-issue-23mar/

I get the below error:

03/23/2020 12:53:12 - INFO - transformers.modeling_utils -   loading weights file /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/QA_output_squad/BioASQ-factoid-6b/BioASQ-factoid-6b-1-issue-23mar/pytorch_model.bin
03/23/2020 12:53:15 - INFO - __main__ -   Creating features from dataset file at .

  0%|          | 0/1 [00:00<?, ?it/s]
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_squad.py", line 856, in <module>
    main()
  File "run_squad.py", line 845, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 299, in evaluate
    dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)
  File "run_squad.py", line 475, in load_and_cache_examples
    examples = processor.get_dev_examples(args.data_dir, filename=args.predict_file)
  File "/scratch/oe7/uk1594/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 522, in get_dev_examples
    return self._create_examples(input_data, "dev")
  File "/scratch/oe7/uk1594/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 549, in _create_examples
    answers = qa["answers"]
KeyError: 'answers'

Really appreciate your help.

Thanks a lot for your guidance.

The evaluaton dataset is looks like this:

{
  "version": "BioASQ6b",
  "data": [
    {
      "title": "BioASQ6b",
      "paragraphs": [
        {
          "context": "emMAW: computing minimal absent words in external memory. Motivation: The biological significance of minimal absent words has been investigated in genomes of organisms from all domains of life. For instance, three minimal absent words of the human genome were found in Ebola virus genomes",
          "qas": [
            {
              "question": "Which algorithm is available for computing minimal absent words using external memory?",
              "id": "5a6a3335b750ff4455000025_000"
            }
          ]
        }
    ]
}
]
}

Output: The BioASQ evaluation files are test files that don't contain answers, only used for predictions. for evaluation during training you can use a portion of the training files


Instruction: BERT training with character embeddings

Input: Does it make sense to change the tokenization paradigm in the BERT model, to something else? Maybe just a simple word tokenization or character level tokenization?

Output:
That is one motivation behind the paper "CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters" where BERT's wordpiece system is discarded and replaced with a CharacterCNN (just like in ELMo). This way, a word-level tokenization can be used without any OOV issues (since the model attends to each token's characters) and the model produces a single embedding for any arbitrary input token.
Performance-wise, the paper shows that CharacterBERT is generally at least as good BERT while at the same time being more robust to noisy texts.


Instruction: How to install torch 0.4.1 on Windows 10?

Input:
I have windows 10 on a Lenovo Thinkpad P72 with a Nvidia Quadro P5200, and I absolutely need to install (py)torch v0.4.1 to use a 3D Mask R-CNN. So I tried the following link:
https://github.com/pytorch/pytorch/issues/19457

However, when I finish with "python setup.py install", I obtain:

C:\Users\...\pytorch-0.4.1\build>msbuild INSTALL.vcxproj /p:Configuration=Release
Microsoft (R) Build Engine, version 4.8.3752.0
[Microsoft .NET Framework, Version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. Tous droits réservés.

La génération a démarré 31/03/2020 07:03:00.
Projet "C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj" sur le noud 1 (cibles par défaut).
C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj(32,3): error MSB4019: Le projet importé     "C:\Microsoft.Cpp.Default.props" est introuvable. Vérifiez que le chemin d'accès dans la     déclaration <Import> est correct et que le fichier existe sur le disque.
Génération du projet "C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj" terminée (cibles par défaut) -- ÉCHEC.

ÉCHEC de la build.

"C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj" (cible par défaut) (1) -> C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj(32,3): error MSB4019: Le projet importé     "C:\Microsoft.Cpp.Default.props" est introuvable. Vérifiez que le chemin d'accès dans la déclaration <Import> est correct et que le fichier existe sur le disque.

0 Avertissement(s)
1 Erreur(s)

Temps écoulé 00:00:00.28

C:\Users\...\pytorch-0.4.1\build>IF ERRORLEVEL 1 exit 1
Failed to run 'tools\build_pytorch_libs.bat --use-cuda --use-nnpack caffe2 nanopb libshm_windows'

Since I wasn't able to solve this issue, I copied all the missing files there, and then I obtained (even if C:\Microsoft.Build.CppTasks.Common.dll exists):

ÉCHEC de la build.

"C:\Users\...\pytorch-0.4.1\build\INSTALL.vcxproj" (cible par défaut) (1) ->
"C:\Users\...\pytorch-0.4.1\build\ZERO_CHECK.vcxproj" (cible par défaut) (2) ->
(SetBuildDefaultEnvironmentVariables cible) -> C:\Microsoft.Cpp.Current.targets(64,5): error MSB4062: Impossible de charger la tâche "SetEnv" à partir de l'assembly
C:\Microsoft.Build.CppTasks.Common.dll. Impossible de charger le fichier ou l'assembly         'Microsoft.Build.Utilities.Core, Version=14.0.0.0, Culture=neutral, PublicKeyToken=...' ou une     de ses dépendances. Le fichier spécifié est introuvable. Assurez-vous que la déclaration     <UsingTask> est correcte, que l'assembly et toutes ses dépendances sont
disponibles et que la tâche contient une classe publique qui implémente     Microsoft.Build.Framework.ITask. [C:\Users\...\pytorch-0.4.1\build\ZERO_CHECK.vcxproj]

Someone has an idea?

Output:
for pip

pip install torch===0.4.1 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: Use a generator to perform operation on a matrix in Python

Input:
I have a similarity matrix (torch tensor) which is a cosine similarity matrix between two matrix (source and target). From the matrix I need to obtain the sum of the distance between the top nearest neighbor of each source and target.

Then fillup two defaultdicts using the computed values above as shown in the code snippet below

import torch
from collections import defaultdict
src2tgt = defaultdict(dict)
tgt2src = defaultdict(dict)

#similarity matrix between source and target matrix
matx = torch.Tensor([[3,2,1,7],[1,1,0,8],[0,7,1,0],[2,0,0,0],[1,5,2,1]])

#the src and tgt
src = torch.LongTensor([[1,1],[1,2],[1,3],[1,4],[1,5]])
tgt = torch.LongTensor([[2,1],[2,2],[2,3],[2,4]])
#the data above are dummy, in my actual code, they are generated by a process
similarities=[]

#I need a kinda of the sum of nearest neighbor
k = 2
nearestSrc = torch.topk(matx, k, dim=1, largest=True, sorted=False, out=None)
sumDistSource = torch.sum(nearestSrc[0], 1)
nearestTgt = torch.topk(matx, k, dim=0, largest=True, sorted=False, out=None)
sumDistTarget = torch.sum(nearestTgt[0], 0)

#finally fill default dictionary of source2target and target2source
for i in range(matx.shape[0]):
  for j in range(matx.shape[1]):
    src2tgt[src[i]][tgt[j]] = matx[i][j].tolist() / (sumDistSource[i].tolist() + sumDistTarget[j].tolist())
    tgt2src[tgt[j]][src[i]] = matx[i][j].tolist() / (sumDistTarget[j].tolist() + sumDistSource[i].tolist())
    similarities.append(matx[i][j].tolist() )

Is there a way I can optimize the above code, either using a generator without having to create nearestSrc, sumDistSource, nearestTgt, sumDistTarget explicitly thus requiring less memory?  Or can I also reduce the double loop?

Output:
I don't think it's necessary to save memory here. Let's say the shape of matx is [n x m], then the nearestSrc/Tgt and sumDistSource/Target tensors will contain no more than 2 * (n + m), memory consumption of which is almost ignorable compared to matx. Besides, I don't think PyTorch provides an API to generate top-k elements on-the-fly, and it would be hard to implement a differentiable or GPU-optimized version of that.

The double-loop can be optimized, although I'm a bit confused by what you're doing here. It seems that the values you're computing for src2tgt[src[i]][tgt[j]] and tgt2src[tgt[j]][src[i]] are exactly the same. Also, I don't think storing these in a nested dictionary is a good idea, for two reasons:

src[i] and tgt[j] are floating-point tensors, and interestingly, the hash value for PyTorch tensors does not depend on tensor values. In fact, it is equivalent to the id function. See more discussions here. This means that two tensors with the same value would still be stored as different keys in the dictionary. Besides, it's also probably not a good idea to use floating-point numbers as keys, because testing equality for floating-point numbers often require special care. For more information on this topic, consider reading this very helpful blog.
Taking values out of a tensor and storing them in another structure would prevent further optimizations. Operations on tensors can benefit from the highly-optimized PyTorch functions, which scales sublinearly on GPUs (i.e., a 2x increase in data size leads to a <2x increase in compute time). It's often more desirable to do stuff in tensor form as much as you can.

Thus, we can optimize your code by first computing all the values you need to store in src2tgt by a batch tensor operation, and then storing them into the dictionary.

norm = (sumDistSource.unsqueeze(-1).expand(-1, matx.size(1)) +
        sumDistTarget.unsqueeze(0).expand(matx.size(0), -1))
s2t = (matx / norm).tolist()

src_vals = src.tolist()
tgt_vals = tgt.tolist()
for i, s in enumerate(src_vals):
    s = tuple(s)  # convert to tuples because lists are not hashable
    for j, t in enumerate(tgt_vals):
        t = tuple(t)
        src2tgt[s][t] = s2t[i][j]
        tgt2src[t][s] = s2t[i][j]
similarities = matx.flatten().tolist()

However, the speedup would be small as long as you still use a dictionary. I would encourage you to consider other ways to store the data.


Instruction: Tensorflow and PyTorch hang on initializing with CUDA

Input:
When I try to run a very minimal Tensorflow example:

import tensorflow as tf

c = tf.constant([1,2,3])

The system hangs forever (at least for ten minutes) with no sign of what it is doing. It uses 100% of one virtual CPU core when in this state. When run in a Juypter notebook the kernel outputs this to the console:

2020-03-31 11:12:04.840507: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840576: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:12:04.840589: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2020-03-31 11:12:05.521172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-03-31 11:12:05.539193: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.539639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1070 computeCapability: 6.1
coreClock: 1.7845GHz coreCount: 15 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 238.66GiB/s
2020-03-31 11:12:05.539841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-03-31 11:12:05.541113: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-03-31 11:12:05.542119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-03-31 11:12:05.542324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-03-31 11:12:05.543632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-03-31 11:12:05.544401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-03-31 11:12:05.547212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-03-31 11:12:05.547337: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-31 11:12:05.548512: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-03-31 11:12:05.567845: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3393550000 Hz
2020-03-31 11:12:05.568364: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564107e16440 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-03-31 11:12:05.568395: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version

I did have Tensorflow working previously on this system, so I think this might be some sort of library issue that got caused by a system update.

The GPU is an Nvidia GTX 1070. Tensorflow version is 2.1.0, and hasn't changed since when it was working. Running Arch Linux, if that matters.

I tried downgrading from CUDA 10.2 to 10.1, but the issue still occurs.

I can also reproduce this with PyTorch:

import torch
import transformers

t = torch.tensor([1,2,3])
t.cuda()

(import transformers prevents a "CUDA: Out of memory" issue - there must be something it does that initializes PyTorch that I don't know how to do.)

This has the same issue, where it freezes pegging one CPU core, though it produces less output:

020-03-31 11:13:41.428483: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428571: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-31 11:13:41.428587: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

I'm pretty sure the complaints about TensorRT are not relevant, because when I had this working previously it would output those as well.

How can I resolve this issue? Or at least, what else can I do to determine what it is doing while frozen?

Output: My issue was caused by a ulimit I had set on the amount of virtual memory the Python process was aloud to consume (with ulimit -Sv 12000000 in zsh). I don't know why that would cause it to hang, but if anyone else encounters a similar issue, make sure you aren't limiting virtual memory.


Instruction: Pytorch: multi-target error with CrossEntropyLoss

Input:
So I was training a Conv. Neural Network. Following are the essential details:

original label dim  = torch.Size([64, 1])
output from the net dim = torch.Size([64, 2])
loss type = nn.CrossEntropyLoss()
error = RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

WHERE AM I WRONG..?

training:

EPOCHS        = 5
LEARNING_RATE = 0.0001
BATCH_SIZE    = 64

net = Net().to(device)
optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

loss_log = []
loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)

train function:

def train(net, train_set, loss_log=[], EPOCHS=5, LEARNING_RATE=0.001, BATCH_SIZE=32):
  print('Initiating Training..')
  loss_func = nn.CrossEntropyLoss()

  # Iteration Begins
  for epoch in tqdm(range(EPOCHS)):
    # Iterate over every sample in the batch
    for data in tqdm(trainSet, desc=f'Iteration > {epoch+1}/{EPOCHS} : ', leave=False):
        x, y = data
        net.zero_grad()

        #Compute the output
        output, sm = net(x)

        # Compute Train Loss
        loss = loss_func(output, y.to(device))

        # Backpropagate
        loss.backward()

        # Update Parameters
        optimizer.step()

        # LEARNING_RATE -= LEARNING_RATE*0.0005

    loss_log.append(loss)
    lr_log.append(LEARNING_RATE)

  return loss_log, lr_log

FULL ERROR:

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-20-8deb9a27d3b4> in <module>()
     13
     14 total_epochs += EPOCHS
---> 15 loss_log = train(net, trainSet, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     16
     17 plt.plot(loss_log)

4 frames

<ipython-input-9-59e1d2cf0c84> in train(net, train_set, loss_log, EPOCHS, LEARNING_RATE, BATCH_SIZE)
     21         # Compute Train Loss
     22         # print(output, y.to(device))
---> 23         loss = loss_func(output, y.to(device))
     24
     25         # Backpropagate

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2019     if size_average is not None or reduce is not None:
   2020         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2021     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2022
   2023

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1836                          .format(input.size(0), target.size(0)))
   1837     if dim == 2:
-> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1839     elif dim == 4:
   1840         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:15

Output: The problem is that your target tensor is 2-dimensional ([64,1] instead of [64]), which makes PyTorch think that you have more than 1 ground truth label per data. This is easily fixed via loss_func(output, y.flatten().to(device)). Hope this helps!


Instruction: Importance weighted autoencoder doing worse than VAE

Input:
I've been implementing VAE and IWAE models on the caltech silhouettes dataset and am having an issue where the VAE outperforms IWAE by a modest margin (test LL ~120 for VAE, ~133 for IWAE!). I don't believe this should be the case, according to both theory and experiments produced here.

I'm hoping someone can find some issue in how I'm implementing that's causing this to be the case.

The network I'm using to approximate q and p is the same as that detailed in the appendix of the paper above. The calculation part of the model is below:

data_k_vec = data.repeat_interleave(K,0) # Generate K samples (in my case K=50 is producing this behavior)

mu, log_std = model.encode(data_k_vec)
z = model.reparameterize(mu, log_std) # z = mu + torch.exp(log_std)*epsilon (epsilon ~ N(0,1))
decoded = model.decode(z) # this is the sigmoid output of the model

log_prior_z = torch.sum(-0.5 * z ** 2, 1)-.5*z.shape[1]*T.log(torch.tensor(2*np.pi))
log_q_z = compute_log_probability_gaussian(z, mu, log_std) # Definitions below
log_p_x = compute_log_probability_bernoulli(decoded,data_k_vec)

if model_type == 'iwae':
    log_w_matrix = (log_prior_z + log_p_x  - log_q_z).view(-1, K)
elif model_type =='vae':
    log_w_matrix = (log_prior_z + log_p_x  - log_q_z).view(-1, 1)*1/K

log_w_minus_max = log_w_matrix - torch.max(log_w_matrix, 1, keepdim=True)[0]
ws_matrix = torch.exp(log_w_minus_max)
ws_norm = ws_matrix / torch.sum(ws_matrix, 1, keepdim=True)

ws_sum_per_datapoint = torch.sum(log_w_matrix * ws_norm, 1)

loss = -torch.sum(ws_sum_per_datapoint) # value of loss that gets returned to training function. loss.backward() will get called on this value

Here are the likelihood functions. I had to fuss with the bernoulli LL in order to not get nan during training

def compute_log_probability_gaussian(obs, mu, logstd, axis=1):
    return torch.sum(-0.5 * ((obs-mu) / torch.exp(logstd)) ** 2 - logstd, axis)-.5*obs.shape[1]*T.log(torch.tensor(2*np.pi))

def compute_log_probability_bernoulli(theta, obs, axis=1): # Add 1e-18 to avoid nan appearances in training
    return torch.sum(obs*torch.log(theta+1e-18) + (1-obs)*torch.log(1-theta+1e-18), axis)

In this code there's a "shortcut" being used in that the row-wise importance weights are being calculated in the model_type=='iwae' case for the K=50 samples in each row, while in the model_type=='vae' case the importance weights are being calculated for the single value left in each row, so that it just ends up calculating a weight of 1. Maybe this is the issue?

Any and all help is huge - I thought that addressing the nan issue would permanently get me out of the weeds but now I have this new problem.

EDIT:
Should add that the training scheme is the same as that in the paper linked above. That is, for each of i=0....7 rounds train for 2**i epochs with a learning rate of 1e-4 * 10**(-i/7)

Output:
The K-sample importance weighted ELBO is
$$ \textrm{IW-ELBO}(x,K) = \log \sum_{k=1}^K \frac{p(x \vert z_k) p(z_k)}{q(z_k;x)}$$
For the IWAE there are K samples originating from each datapoint x, so you want to have the same latent statistics mu_z, Sigma_z obtained through the amortized inference network, but sample multiple z K times for each x.
So its computationally wasteful to compute the forward pass for data_k_vec = data.repeat_interleave(K,0), you should compute the forward pass once for each original datapoint, then repeat the statistics output by the inference network for sampling:
mu = torch.repeat_interleave(mu,K,0)
log_std = torch.repeat_interleave(log_std,K,0)

Then sample z_k. And now repeat your datapoints data_k_vec = data.repeat_interleave(K,0), and use the resulting tensor to efficiently evaluate the conditional p(x |z_k) for each importance sample z_k.
Note you may also want to use the logsumexp operation when calculating the IW-ELBO for numerical stability. I can't quite figure out what's going on with the log_w_matrix calculation in your post, but this is what I would do:
log_pz = ...
log_qzCx = ....
log_pxCz = ...

log_iw = log_pxCz + log_pz - log_qzCx
log_iw = log_iw.reshape(-1, K)
iwelbo = torch.logsumexp(log_iw, dim=1) - np.log(K)

EDIT: Actually after thinking about it a bit and using the score function identity, you can interpret the IWAE gradient as an importance weighted estimate of the standard single-sample gradient, so the method in the OP for calculation of the importance weights is equivalent (if a bit wasteful), provided you place a stop_gradient operator around the normalized importance weights, which you call w_norm. So I the main problem is the absence of this stop_gradient operator.


Instruction: optimized_execution() takes 1 positional argument but 2 were given

Input:
I'm following the pytorch sagemaker docs here and I'm stuck on this line

torch.jit.optimized_execution(True, {'target_device': 'eia:device ordinal'})

When I run it, I get the error optimized_execution() takes 1 positional argument but 2 were given. I'm using pytorch 1.3.1, but I tried with 1.4.0 and was running into similar problems. Can I use optimized execution without this second argument? How can I specify the accelerator?

Output:
(I'll refer to the Elastic Inference enabled PyTorch framework as "PyTorch-EI" for convenience)

Are you using SageMaker through notebook or hosting? SageMaker notebook support is not currently released, so there's no official notebook kernel / Conda environment that you can activate that will have the Elastic Inference enabled PyTorch framework. You could however create your own by activating the pytorch_p36 environment (which has standard PyTorch), uninstalling PyTorch, and then installing using the PyTorch-EI 1.3.1 framework wheel - linked from here.

SageMaker hosting does currently support PyTorch-EI out of the box. If you're currently using SageMaker hosting and having issues, share some of your inference code + what container you're using. List of containers can be found at here.

Also note that EC2 currently supports Elastic Inference through the DLAMI v27.0. The Conda environment name is amazonei_pytorch_p36.


Instruction: Reading h5py files into tensors

Input:
So I have a training set and a test set both in h5py format. I also have a data_load function that loads the files and returns NumPy arrays. The main problem is I don't need NumPy as I am working with Tensors. I am expecting to have an x&y tensor of size N(batch size) and D_in(input size for each image) and D_out(Output size of each tensor).

The problem:

x&y do not get converted to tensors of dimensions mentioned below.If anything their types remain to be numpy.ndarray. Any help is appreciated.

def load_data(train_file, test_file):
# Load the training data

train_dataset =h5py.File(train_file, 'r')

# Separate features(x) and labels(y) for training set
train_set_x_orig =np.array(train_dataset["train_set_x"][:])
train_set_y_orig =np.array(train_dataset["train_set_y"][:])

# Load the test data
test_dataset =h5py.File(test_file,'r')

# Separate features(x) and labels(y) for training set
test_set_x_orig =np.array(test_dataset["test_set_x"][:])
test_set_y_orig =np.array(test_dataset["test_set_y"][:])

classes = np.array(test_dataset["list_classes"][:]) # the list of classes

train_set_y_orig = torch.from_numpy(train_set_y_orig.reshape((1, train_set_y_orig.shape[0])))
test_set_y_orig = torch.from_numpy(test_set_y_orig.reshape((1, test_set_y_orig.shape[0])))

return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes

x = torch.Tensor(N, D_in)
y = torch.Tensor(N, D_out)
train_file="data/train_catvnoncat.h5"
test_file="data/test_catvnoncat.h5"
x,y,_,_,_=load_data(train_file,test_file)

Output:
Because you did not convert train_set_x_orig to a torch tensor before returning.
Either use torch.from_numpy() on train_set_x_orig before returning as you do with train_set_y_orig or cast it to a tensor before assigning to x.

However, y should be of type torch.tensor.

Below is a demonstration that explains the issue:

# some sample tensor
In [27]: x = torch.Tensor(3, 2)

# check its type
In [28]: type(x)
Out[28]: torch.Tensor

# some sample ndarray
In [29]: arrx = np.arange(6).reshape(3, -1)

# assign array to tensor
# note that now the object `x` refers to the numpy array object
In [30]: x = arrx

# see that the type() of `x` is now numpy ndarray
In [31]: type(x)
Out[31]: numpy.ndarray

Also, as hpaulj pointed out in the comments, there is no need to wrap the sliced objects from h5py in np.array() since the sliced objects are already of type numpy ndarrays. So, you can just get rid of them and the code will look more cleaner!


Instruction: How to optimize pip imports for Dockerfile layers caching

Input:
I have a Dockerfile for ML/DL stack that needs a lot of requirements that could be logically split into python standard libraries and python ml libraries at least:

Python libraries (requirements.txt):

Cython
python-dateutil==2.8.0
setuptools>=41.0.0
progressbar2
argparse
smart_open
backoff
boto3
botocore
google
protobuf
tornado==5.1.1

Python ml libraries (requirements.lib.txt):

numpy==1.15.1
intel-numpy
matplotlib
pandas
scipy==1.2.1
scikit-learn==0.21.3
torch
tensorflow==1.14.0
keras==2.1.1

Now when I build my Docker image it turns out that the whole image size is ~5-7GB.
The other point is that I have, in terms of size in MB, small layers (< 100MB), big layers (~100MB-500MB) and huge layers (>500MB up to 1-2GB).

Of course ml python libraries cached to layers do not help since torch itself is about 800MB, tensorflow is ~500MB, intel mkl is about 300MB, etc.

Currently to prevent packages version override I do like

COPY $ROOT_APPLICATION/src/requirements.txt /tmp/requirements.txt
RUN cat /tmp/requirements.txt | xargs -n 1 -L 1 pip3 install

COPY $ROOT_APPLICATION/src/requirements.lib.txt /tmp/requirements.lib.txt
RUN cat /tmp/requirements.lib.txt | xargs -n 1 -L 1 pip3 install

where I do a copy to /tmp before RUN for caching reasons.
To keep everything compact I start from the good python slim buster:

FROM python:3.7.4-slim-buster

while the missing part of the Dockerfile is the apt-get of required libraries like:

########################################  BASE SYSTEM
RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
RUN apt-get update && apt-get install -y --no-install-recommends \
    software-properties-common \
    build-essential \
    pkg-config \
    libjemalloc2 \
    libgmp3-dev \
    libicu-dev \
    python3.7-icu \
    swig \
    curl \
    unzip \
    cron \
    jq

######################################## AUDIO
RUN apt-get update && apt-get install -y \
    libyaml-0-2 \
    libfftw3-3 \
    libtag1v5 \
    libsamplerate0 \
    libasound-dev \
    portaudio19-dev \
    libportaudio2 \
    libportaudiocpp0 \
    ffmpeg \
    espeak

How to optimize layers caching to 1) keep the docker images as small as possible 2) optimize layer's size for bandwidth when using docker push (then avoiding to push to registry huge layers over the network)?

Output:
You're going to want pip install --no-cache-dir, so it doesn't keep copies of the downloads around.
You don't want to keep the toolchain (compiler etc.) installed, but you need them to build the image. So what you do is, you use multi-stage builds: you use one image to build everything, and then a second image that just copies the built packages over and omits all the build tools and artifacts. You can find a guide to multi-stage builds for Python here (it's three parts, this is part 1): https://pythonspeed.com/articles/smaller-python-docker-images/


Instruction: How to revert BERT/XLNet embeddings?

Input:
I've been experimenting with stacking language models recently and noticed something interesting: the output embeddings of BERT and XLNet are not the same as the input embeddings. For example, this code snippet:

bert = transformers.BertForMaskedLM.from_pretrained("bert-base-cased")
tok = transformers.BertTokenizer.from_pretrained("bert-base-cased")

sent = torch.tensor(tok.encode("I went to the store the other day, it was very rewarding."))
enc = bert.get_input_embeddings()(sent)
dec = bert.get_output_embeddings()(enc)

print(tok.decode(dec.softmax(-1).argmax(-1)))

Outputs this for me:

,,,,,,,,,,,,,,,,,

I would have expected the (formatted) input sequence to be returned since I was under the impression that the input and output token embeddings were tied.

What's interesting is that most other models do not exhibit this behavior. For example, if you run the same code snippet on GPT2, Albert or Roberta, it outputs the input sequence.

Is this a bug? Or is it expected for BERT/XLNet?

Output:
Not sure if it's too late, but I've experimented a bit with your code and it can be reverted. :)
bert = transformers.BertForMaskedLM.from_pretrained("bert-base-cased")
tok = transformers.BertTokenizer.from_pretrained("bert-base-cased")

sent = torch.tensor(tok.encode("I went to the store the other day, it was very rewarding."))
print("Initial sentence:", sent)
enc = bert.get_input_embeddings()(sent)
dec = bert.get_output_embeddings()(enc)

print("Decoded sentence:", tok.decode(dec.softmax(0).argmax(1)))

For this, you get the following output:
Initial sentence: tensor([  101,   146,  1355,  1106,  1103,  2984,  1103,  1168,  1285,   117,
         1122,  1108,  1304, 10703,  1158,   119,   102])
Decoded sentence: [CLS] I went to the store the other day, it was very rewarding. [SEP]


Instruction: Google Colab become slower with the same code sometimes. What is the possible reasons?

Input:
I am training a CNN model with Google Colab's GPU through pytorch.  My question is, even though running with the same code, it gets about three times slower sometimes(30s -> 90s in my case).

I've tried restart runtime(it clears all local variable but keep files), it doesn't work

I have seen this    post, however, I've checked my GPU status, it works well with 11.5GB.

Sometimes it goes normal after disconnect it for a while though, I still want to figure out what can be the possible reason of it.

Output:
Guys I think I found the possible answer here

So it might be the limit of Google Colab itself. Due to their policy, sometimes you'll get fewer computation resources, which slower down the process even though no change in any code.


Instruction: Text classification using BERT - how to handle misspelled words

Input:
I am not sure if this is the best place to submit that kind of question, perhaps CrossValdation would be a better place.

I am working on a text multiclass classification problem.
I built a model based on BERT concept implemented in PyTorch (huggingface transformer library). The model performs pretty well, except when the input sentence has an OCR error or equivalently it is misspelled.

For instance, if the input is "NALIBU DRINK" the Bert tokenizer generates ['na', '##lib', '##u', 'drink'] and model's prediction is completely wrong. On the other hand, if I correct the first character, so my input is "MALIBU DRINK", the Bert tokenizer generates two tokens ['malibu', 'drink'] and the model makes a correct prediction with very high confidence.

Is there any way to enhance Bert tokenizer to be able to work with misspelled words?

Output:
You can leverage BERT's power to rectify the misspelled word.
The article linked below beautifully explains the process with code snippets
https://web.archive.org/web/20220507023114/https://www.statestitle.com/resource/using-nlp-bert-to-improve-ocr-accuracy/
To summarize, you can identify misspelled words via a SpellChecker function and get replacement suggestions. Then, find the most appropriate replacement using BERT.


Instruction: How to index a 3-d tensor with 2-d tensor in pytorch?

Input:
import torch
a = torch.rand(5,256,120)
min_values, indices = torch.min(a,dim=0)
aa = torch.zeros(256,120)
for i in range(256):
    for j in range(120):
        aa[i,j] = a[indices[i,j],i,j]

print((aa==min_values).sum()==256*120)

I want to know how to avoid to using the for-for loop to get the aa values? (I want to use the indices to select elements in another 3-d tensors so I can't use the values return by min directly)

Output:
You can use torch.gather

aa = torch.gather(a, 0, indices.unsqueeze(0))

as explained here: Slicing a 4D tensor with a 3D tensor-index in PyTorch


Instruction: Pytorch - Indexing a range of multiple Indices?

Input:
Lets say I have a tensor of size [100, 100] and I have a set of start_indices and end_indices of size [100]

I want to be able to do something like this:

tensor[start_indices:end_indices, :] = 0

Unfortunately, I get an error saying

TypeError: only integer tensors of a single element can be converted to an index

So is this actually possible without a for loop?

Output:
To the best of my knowledge this is not possible without some sort of loop or list comprehension.

Below are some alternatives which may be useful depending on your use-case. Specifically if you are looking to reuse the same start_indices and end_indices for multiple assignments, or if you are looking have only one in-place assignment to tensor then the solutions below would be useful.

If instead of start_indices and end_indices you were given a list of indices, for example

row_indices = torch.cat([torch.arange(s, e, dtype=torch.int64) for s, e in zip(start_indices, end_indices)])

Then this would be possible using

tensor[row_indices, :] = 0

Or if you were given a mask

mask = torch.zeros(tensor.shape, dtype=torch.bool, device=tensor.device)
for s, e in zip(start_indices, end_indices):
    mask[s:e, :] = True

then this would be possible using

tensor[mask] = 0


Instruction: Maxpool of an image in pytorch

Input:
I'm trying to just apply maxpool2d (from torch.nn) on a single image (not as a maxpool layer). Here is my code right now:

name = 'astronaut'
imshow(images[name], name)
img = images[name]
# pool of square window of size=3, stride=1
m = nn.MaxPool2d(3,stride = 1)
img_transform = torch.Tensor(images[name])
plt.imshow(m(img_transform).view((512,510)))

The issue is, this code gives me a very green image as a result. I am sure the problem is with the dimensions of view, but I was unable to find how to apply maxpool to just one image so I couldn't fix it. The dimension of the image I'm considering is 512x512. The arguments for view make no sense for me right now, it's just the only number that gives a result...

If for example, I gave 512,512 as the argument for view, I get the following error:

RuntimeError: shape '[512, 512]' is invalid for input of size 261120

If anyone can tell me how to apply maxpool, avgpool, or minpool to an image and display the result I would be super grateful!

Thanks (:

Output:
Assuming your image is a numpy.array upon loading (please see comments for explanation of each step):

import numpy as np
import torch

# Assuming you have 3 color channels in your image
# Assuming your data is in Width, Height, Channels format
numpy_img = np.random.randint(low=0, high=255, size=(512, 512, 3))

# Transform to tensor
tensor_img = torch.from_numpy(numpy_img)
# PyTorch takes images in format Channels, Width, Height
# We have to switch their dimensions using `permute`
tensor_img = tensor_img.permute(2, 0, 1)
tensor_img.shape # Shape [3, 512, 512]

# Layers always need batch as first dimension (even for one image)
# unsqueeze will add it for you
ready_tensor_img = tensor_img.unsqueeze(dim=0)
ready_tensor_img.shape # Shape [1, 3, 512, 512]

pooling = torch.nn.MaxPool2d(kernel_size=3, stride=1)

# You need to cast your image to float as
# pooling is not implemented for Tensors of type long
new_img = pooling(ready_tensor_img.float())

If your image is black and white you would need shape [1, 1, 512, 512] (single channel only), you can't leave/squeeze those dimensions, they always have to be there for any torch.nn.Module!

To transform tensor into image again you could use similar steps:

# Cast to long and squeeze batch dimension
no_batch = new_img.long().squeeze(dim=0)

# Unpermute
width_height_channels = no_batch.permute(1, 2, 0)
width_height_channels.shape  # Shape: [510, 510, 3]

# Cast to numpy and you have your image
final_image = width_height_channels.numpy()


Instruction: why cannot cuda model be initialized under the __init__ method in a class that inherits multiprocessing.process?

Input:
Here is my code:

from MyDetector import Helmet_Detector
from multiprocessing import Process

class Processor(Process):
    def __init__(self):
        super().__init__()
        self.helmet_detector = Helmet_Detector()

    def run(self):
         print(111)

    if __name__ == '__main__':
         p=Processor()
         p.start()

As you can see, the class 'Processor' inherits multiprocessing.Process, and Helmet_Detector is a YOLO model using cuda. But when I ran it, the error occurred as follow:

THCudaCheck FAIL file=C:\w\1\s\tmp_conda_3.7_075911\conda\conda-bld\pytorch_1579075223148\work\torch/csrc/generic/StorageSharing.cpp line=245 error=71 : operation not supported
Traceback (most recent call last):
  File "E:/python-tasks/WHU-CSTECH/Processor.py", line 17, in <module>
    p.start()
  File "C:\Anaconda\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Anaconda\lib\site-packages\torch\multiprocessing\reductions.py", line 242, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
RuntimeError: cuda runtime error (71) : operation not supported at C:\w\1\s\tmp_conda_3.7_075911\conda\conda-bld\pytorch_1579075223148\work\torch/csrc/generic/StorageSharing.cpp:245

then I tried to intialize the Helmet_Detector in run method:

def run(self):
     print(111)
     self.helmet_detector = Helmet_Detector()

No error occurred. Could anyone please tell me the reason for this and how could I solve this problem? Thank you!

Output:
Error occurs because in python multiprocessing requires Process class objects to be pickelable so that data can be transferred to the process being created i.e. Serialisation and deserialization of the object. Suggestion to overcome the issue, lazy instantiate the Helmet_Detector object (hint: try property in python).

Edit:

As per the comment by @jodag, you should use pytorch's multiprocessing library instead of standard multiprocessing library

Example:

import torch.multiprocessing as mp

class Processor(mp.Process):
.
.
.


Instruction: how does the neural netwok definition in pytorch use pyton classes

Input:
in order to understand how this code works, I have written a small reproducer. How does the self.hidden variable use a variable x in the forward method?

enter code class Network(nn.Module):
def __init__(self):
    super().__init__()

    # Inputs to hidden layer linear transformation
    self.hidden = nn.Linear(784, 256)
    # Output layer, 10 units - one for each digit
    self.output = nn.Linear(256, 10)

    # Define sigmoid activation and softmax output
    self.sigmoid = nn.Sigmoid()
    self.softmax = nn.Softmax(dim=1)

def forward(self, x):
    # Pass the input tensor through each of our operations
    x = self.hidden(x)
    x = self.sigmoid(x)
    x = self.output(x)
    x = self.softmax(x)

    return x

Output:
You misunderstood what self.hidden = nn.Linear(784, 256) does. You wrote that:

  hidden is defined as a function

but this is not true. self.hidden is an object of the class nn.Linear. And when you call self.hidden(...), you are not passing arguments to nn.Linear; you are passing arguments to __call__ (defined in the nn.Linear class).

If you want more details on that, I have expanded on how it works in PyTorch: see this answer.


Instruction: Anaconda Integration with Cuda 9.0 shows Incompatible Package Error

Input:
I am trying to install CUDA 9.0  with NVIDIA-SMI: 445.75 in Windows 10.

My Cuda 9.0 installation is successful, as shown from Command-prompt

*(DL) C:\Users\User>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:32_Central_Daylight_Time_2017
**Cuda compilation tools, release **9.0**, V9.0.176***

(1) I downloaded cudnn-9.0-windows10-x64-v7.zip, extracted it, and moved it to the fold, which was created when Cuda was installed.

(2) In the terminal prompt of the Anaconda,
I input conda install pytorch=1.1.0 torchvision=0.3.0 cudatoolkit=9.0 –c pytorch.

However, Anaconda prompt gives the following error

**Error messages**
*Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation in your environment:

Specifications:

  - pytorch=1.1.0 -> python[version='>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']
  - torchvision=0.3.0 -> python[version='>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0']

Your python: python=3.8

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow not available for the python version you are constrained to. Note that conda will not change your python version to a different minor version unless you explicitly specify that.

The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package cudatoolkit conflicts for:

    torchvision=0.3.0 -> cudatoolkit[version='>=10.0,<10.1|>=9.0,<9.1']
    pytorch=1.1.0 -> cudatoolkit[version='>=10.0,<10.1|>=9.0,<9.1']
    torchvision=0.3.0 -> pytorch[version='>=1.1.0'] -> cudatoolkit[version='>=10.1,<10.2|>=9.2,<9.3']The following specifications were found to be incompatible with your CUDA driver:

      - feature:/win-64::__cuda==11.0=0

    Your installed CUDA driver is: 11.0*

Output:
I got solved this issue as follows.

Open Anaconda Powershell Prompt by searching it on the start menu.
then run conda install -c anaconda tensorflow-gpu command.
it may be asked to your acceptance.

finally tensorflow-gpu listed on the installed list.

Reference: https://anaconda.org/anaconda/tensorflow-gpu


Instruction: Pytorch "NCCL error": unhandled system error, NCCL version 2.4.8"

Input:
I use pytorch to distributed training my model.I have two nodes and two gpu for each node, and I run the code for one node:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 0 --dist-url tcp://192.168.**.***:8000

and the other:
python train_net.py  --config-file configs/InstanceSegmentation/pointrend_rcnn_R_50_FPN_1x_coco.yaml  --num-gpu 2  --num-machines 2 --machine-rank 1 --dist-url tcp://192.168.**.***:8000

However the other has RuntimeError problem
global_rank 3 machine_rank 1 num_gpus_per_machine 2 local_rank 1
global_rank 2 machine_rank 1 num_gpus_per_machine 2 local_rank 0
Traceback (most recent call last):
  File "train_net.py", line 109, in <module>
    args=(args,),
  File "/root/detectron2_repo/detectron2/engine/launch.py", line 49, in launch
    daemon=False,
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 171, in spawn
    while not spawn_context.join():
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 118, in join
    raise Exception(msg)
Exception:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/root/detectron2_repo/detectron2/engine/launch.py", line 72, in _distributed_worker
    comm.synchronize()
  File "/root/detectron2_repo/detectron2/utils/comm.py", line 79, in synchronize
    dist.barrier()
  File "/root/anaconda3/envs/PointRend/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 1489, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:410, unhandled system error, NCCL version 2.4.8

IF I change mask-rank = 1 to mask-rank = 0, then no error will be reported, but can't distributed training,Does anyone know why this error may occur?

Output:
A number of things can cause this issue, see for example 1, 2. Adding the line
import os
os.environ["NCCL_DEBUG"] = "INFO"

to your script will log more specific debug info leading up to the error, giving you a more helpful error message to google.


Instruction: Kornia rotation not quite rotating as expected

Input:
I'm trying to use the kornia.geometry.transform.rotate function, in Python, to rotate a PyTorch tensor by arbitrary angles. However if I do a simple 90 degree rotation, the resulting tensor doesn't look like it's been fully rotated.

Here's some sample code:

import torch
from kornia.geometry.transform import rotate
import matplotlib.pyplot as plt

a = torch.ones((1,64,64))
a[0,:,2] += 1

angle = torch.tensor([90])

c = rotate(a,angle)

plt.figure()
plt.subplot(121)
plt.imshow(a[0].detach().numpy())
plt.subplot(122)
plt.imshow(c[0].detach().numpy())

And the results before and after the rotation:

Am I missing a subtlety due to the tensor being too coarse here, which causes interpolation issues or something that would be alleviated with a much finer grained tensor?

Many thanks in advance!

Note I am using:

python 3.6.10
kornia 0.2.0
pytorch 1.4.0

Output:
to use Kornia, you can use the Rotate class.
Below is an example to rotating all tensors in a mini-batch by a fixed 45 degrees:
import kornia as tgm

# set the rotation angles - assume batch size is N;
angle = torch.tensor([45]*N).cuda()

# do the rotation:
tensor_rotated = tgm.Rotate(angle)(tensor_input)

The only caveat right now is that it seems super-slow..
Hope this helps!


Instruction: Image classification in Pytorch

Input:
I'm working on facenet-pytorch library in Pytorch, I want to know

the data augmentation should be in train dataset or test data set?

how many images should I put to test data set at least (I've used 2% of images in test data set)

I have 21 classes(21 persons face) and with (vggface2 dataset ) with evaluation mode , does it enough for training and test data set?

how to visualize the images in test dataset to display if a face matched or not
I tried this but it will rise this error :

TypeError: Invalid shape (3, 160, 160) for image data

The shape of images are : (10, 3, 160, 160)
dataiter = iter(test_loader)
images, labels = dataiter.next()
# get predictions
preds = np.squeeze(net(images).data.max(1, keepdim=True)[1].numpy())
images = images.numpy()

# plot the images in the batch, along with predicted and true labels
fig = plt.figure(figsize=(25, 4))
for idx in np.arange(batch_size):
    ax = fig.add_subplot(2, batch_size/2, idx+1, xticks=[], yticks=[])
    ax.imshow(np.squeeze(images[idx]), cmap='gray')
    ax.set_title("{} ({})".format(classes[preds[idx]], classes[labels[idx]]),
             color=("green" if preds[idx]==labels[idx] else "red"))

how to take input faces from webcam after detected the face (prediction function)?

cap = cv.VideoCapture(0)
while True:
    ret, frame = cap.read()
    frame = cv.resize(frame, None, fx=0.5, fy=0.5, interpolation=cv2.INTER_AREA)
    image = predict_draw_bounding_box(frame)
    cv.imshow('Output', image)
    c = cv.waitKey(1)
    if c == 27:
        break
cap.release()
cv.destroyAllWindows()

But I don't know to implement predict_draw_bounding_box function?
Thanks for any advice

Output:
That's a lot of questions; you should probably split those up into multiple questions. In any case, I'll try answering some.

Data augmentation should generally be done on the train dataset
only. Typical augmentations include random rotation, resized crops,
horizontal flips, cutout etc. All of these only go on the train set.

Other than this, off the top of my head, I can only think of channel
normalization as the only augmentation you usually apply to both
training and testing set. You compute x-x_mean/sigma channelwise
for all images in a dataset.
The percentage of images in your test dataset is entirely empirical, and depends on how many images you actually have. For very large datasets with a million plus images, small percentages like 2% is okay. However if your number of images is in the ten thousands, thousands, or even less, it's good practice to keep around 20% as the test set.
Can't understand your question.
Your images are in the shape (3, 160, 160). It's the channel-first syntax used by pytorch's nn.Module system, but plotting an RGB image in matplotlib requires it to have the channel in the last dimension, ie, (160,160,3). If images is a batch of images of shape (10,3,160,160), then do:

...
images = images.numpy()
images = images.swapaxes(1,2).swapaxes(2,3)
...

This will reshape it to (10,160,160,3), without harming the axes order.
No clue.


Instruction: pytorch element intersection

Input:
When I calculate the Hit Ratio, I need to calculate the number of elements of predict tensor in the target tensor, I wanna calculate the number of elements in their intersection.

For example: [# of classes: 20, # of samples: 2]

target:
tensor([[14, 13,  8, 11, 18, 12,  5,  1,  0, 10],
        [ 8, 10,  2, 10,  7, 17,  6, 12, 13, 14]])

pred_idx: (HR@5):
tensor([[14, 11,  8, 19,  4],
        [ 6,  9,  8, 13, 18]])

now when I do >>> (pred_idx & target).sum((1,2))

RuntimeError: The size of tensor a (5) must match the size of tensor b (10) at non-singleton dimension 1.

But the thing is, the prediction and target have different size, how can I calculate the number of elements in the pred@5 that are also in the target?

Output:
Perhaps you could convert to numpy and then use its set operations.

import torch
import numpy as np

target = torch.tensor([[14, 13, 8, 11, 18, 12, 5, 1, 0, 10], [ 8, 10, 2, 10, 7, 17, 6, 12, 13, 14]])
pred_idx = torch.tensor([[14, 11, 8, 19, 4], [ 6, 9, 8, 13, 18]])

Find elements of p@5 in target:

[np.intersect1d(t,p) for t,p in zip(target.cpu().numpy(),pred_idx.cpu().numpy())]

Find number of elements in p@5 also in target:

[len(np.intersect1d(t,p)) for t,p in zip(target.cpu().numpy(),pred_idx.cpu().numpy())]


Instruction: Cannot improve model accuracy

Input:
I am building a general-purpose NN that would classify images (Dog/No Dog) and movie reviews(Good/Bad). I have to stick to a very specific architecture and loss function so changing these two seems out of the equation. My architecture is a two-layer network with relu followed by a sigmoid and a cross-entropy loss function. With 1000 epochs and a learning rate of around .001 I am getting 100 percent training accuracy and .72 testing accuracy.I was looking for suggestions to improve my testing accuracy.This is the layout of what I have:

def train_net(epochs,batch_size,train_x,train_y,model_size,lr):
  n_x,n_h,n_y=model_size
  model = Net(n_x, n_h, n_y)
  optim = torch.optim.Adam(model.parameters(),lr=0.005)
  loss_function = nn.BCELoss()
  train_losses = []
  accuracy = []
  for epoch in range(epochs):
    count=0
    model.train()
    train_loss = []
    batch_accuracy = []
    for idx in range(0, train_x.shape[0], batch_size):

      batch_x = torch.from_numpy(train_x[idx : idx + batch_size]).float()
      batch_y = torch.from_numpy(train_y[:,idx : idx + batch_size]).float()
      model_output = model(batch_x)
      batch_accuracy=[]
      loss = loss_function(model_output, batch_y)
      train_loss.append(loss.item())

      preds = model_output > 0.5
      nb_correct = (preds == batch_y).sum()
      count+=nb_correct.item()
      optim.zero_grad()
      loss.backward()
# Scheduler made it worse
# scheduler.step(loss.item())
  optim.step()

    if epoch % 100 == 1:
      train_losses.append(train_loss)
      print("Iteration : {}, Training loss: {} ,Accuracy %: {}".format(epoch,np.mean(train_loss),(count/train_x.shape[0])*100))
  plt.plot(np.squeeze(train_losses))
  plt.ylabel('loss')
  plt.xlabel('iterations (per tens)')
  plt.title("Learning rate =" + str(lr))
  plt.show()
  return model

My model parameters:

batch_size = 32
lr = 0.0001
epochs = 1500

n_x = 12288     # num_px * num_px * 3
n_h = 7
n_y = 1
model_size=n_x,n_h,n_y
model=train_net(epochs,batch_size,train_x,train_y,model_size,or)

and this is the testing phase.

model.eval()  #Setting the model to eval mode, hence making it deterministic.
test_loss = []
count=0;
loss_function = nn.BCELoss()

for idx in range(0, test_x.shape[0], batch_size):
  with torch.no_grad():
    batch_x = torch.from_numpy(test_x[idx : idx + batch_size]).float()
    batch_y = torch.from_numpy(test_y[:,idx : idx + batch_size]).float()
    model_output = model(batch_x)
    preds = model_output > 0.5
    loss = loss_function(model_output, batch_y)
    test_loss.append(loss.item())
    nb_correct = (preds == batch_y).sum()
    count+=nb_correct.item()

print("test loss: {},test accuracy: {}".format(np.mean(test_loss),count/test_x.shape[0]))

Things I have tried:
Messing around with the learning rate, having momentum, using schedulers and changing batch sizes.Of course these were mainly guesses and not based on any valid assumptions.

Output:
The issue you're facing is overfitting. With 100% accuracy on the training set, your model is effectively memorizing the training set, then failing to generalize to unseen samples. The good news is this is a very common major challenge!

You need regularization. One method is dropout, whereby on different training epochs a random set of the NN connections are dropped, forcing the network to "learn" alternate pathways and weights, and softening sharp peaks in parameter space. Since you need to keep your architecture and loss function the same, you won't be able to add such an option in (though for completeness, read this article for a description and implementation of dropout in PyTorch).

Given your constraints, you'll want to use something like L2 or L1 weight regularization. This typically shows up in the way of adding an additional term to the cost/loss function, which penalizes large weights. In PyTorch, L2 regularization is implemented via the torch.optim construct, with the option weight_decay. (See documentation: torch.optim, search for 'L2')

For your code, try something like:

def train_net(epochs,batch_size,train_x,train_y,model_size,lr):
  ...
  optim = torch.optim.Adam(model.parameters(),...,weight_decay=0.01)
  ...


Instruction: PyTorch Model throwing error for list of layers

Input:
I have designed the following torch model with 2 conv2d layers. It works without any error.
import torch.nn as nn
from torchsummary import summary

class mini_unet(nn.Module):
    def __init__(self):
        super(mini_unet, self).__init__()
        self.c1 = nn.Conv2d(1, 1, 3, padding = 1)
        self.r1 = nn.ReLU()
        self.c2 = nn.Conv2d(1, 1, 3, padding = 1)
        self.r2 = nn.ReLU()

    def forward(self, x):
        x = self.c1(x)
        x = self.r1(x)
        x = self.c2(x)
        x = self.r2(x)
        return x

a = mini_unet().cuda()

print(a)

But, let's say I have too many layers, I don't want to explicitly write each of them in the forward function. So, I used a list to automate it like below.
import torch.nn as nn
from torchsummary import summary

class mini_unet2(nn.Module):
    def __init__(self):
        super(mini_unet2, self).__init__()
        self.layers = [nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU(),
        nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU()]

    def forward(self, x):
        for l in self.layers:
            x = l(x)
        return x

a2 = mini_unet2().cuda()
print(a2)
summary(a2, (1,4,4))

This gives me the following error which is strange, I have used cuda() why it doesn't work?
RuntimeError                              Traceback (most recent call last)
<ipython-input-36-1d71e75b96e0> in <module>
     17 a2 = mini_unet2().cuda()
     18 print(a2)
---> 19 summary(a2, (1,4,4))

~/anaconda3/envs/torch/lib/python3.6/site-packages/torchsummary/torchsummary.py in summary(model, input_size, batch_size, device)
     70     # make a forward pass
     71     # print(x.shape)
---> 72     model(*x)
     73
     74     # remove these hooks

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-36-1d71e75b96e0> in forward(self, x)
     12     def forward(self, x):
     13         for l in self.layers:
---> 14             x = l(x)
     15         return x
     16

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/anaconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/conv.py in forward(self, input)
    318     def forward(self, input):
    319         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 320                         self.padding, self.dilation, self.groups)
    321
    322

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Output:
The error is maybe a little counter-intuitive but the error originates from you using python lists for the layers.

From the documentation, you need to use torch.nn.ModuleList to contain the submodules, not a python list.

So, just changing the list with nn.Modulelist(list) will solve the error.

import torch.nn as nn
from torchsummary import summary

class mini_unet2(nn.Module):
    def __init__(self):
        super(mini_unet2, self).__init__()
        self.layers = nn.ModuleList([nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU(),
        nn.Conv2d(1, 1, 3, padding = 1),
        nn.ReLU()])

    def forward(self, x):
        for l in self.layers:
            x = l(x)
        return x

a2 = mini_unet2().cuda()
print(a2)
summary(a2, (1,4,4))


Instruction: Maybe I found something strange on pytorch, which result in property setter not working

Input:
Maybe I found something strange on pytorch, which result in property setter not working. Below is a minimal example that demonstrates this:

import torch.nn as nn

class A(nn.Module):
    def __init__(self):
        super(A, self).__init__()
        self.aa = 1
        self.oobj = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    @property
    def obj(self):
        print('get attr [obj]: {0}'.format(self.oobj))
        return self.oobj
    @obj.setter
    def obj(self, val):
        print('set attr [obj] to {0}'.format(val))
        self.oobj = val

class B(nn.Module):
    def get_attr(self):
        print('no any attr.')

class C:
    def get_attr(self):
        print('no any attr.')

b = A()             # set obj, and prints my setter message

b.obj               # get obj using my getter

# respectively run the following 3 lines, only the last line not call the setter I defined explicitly.
b.obj = C()         # set obj, and prints my setter message

# b.obj = [1, 2, 3]   # set obj, and prints my setter message

# b.obj = B()         # set obj, but it doesn't print my setter message

The last line doesn't call property setter I defined on class A, but call setter on torch.nn.Module. Because A
    regard B as a nn.Module, call the setter on nn.Module to set attr [obj] as a Module, but it still strange, why
    not call the setter I explicitly defined on class A?

And my project needs to set a nn.Module attribute via setter I defined explicitly, which causes BUG( because it failed). Now I change my code solved the BUG, but still puzzle with the problem.

Output:
It may not look obvious at first, but up until you set b.obj as a nn.Module object, you are defining a normal attribute; but once you set b.obj as a nn.Module object, then you can "only" replace b.obj with another nn.Module, because you registered it to _modules. Let me walk you through the code and you'll get it.

nn.Module()'s __setattr__ implementation can be found here.

First, you defined a new nn.Module:

b = A()  # btw, why not a = A() :)

Then, you set (I'll skip unnecessary steps to reproduce the behavior):

b.obj = [1, 2, 3]

In this case, because

[1,2,3] is not a nn.Parameter;
You haven't set a nn.Parameter as attribute before;
[1,2,3] is not a nn.Module;
You haven't set a nn.Module as attribute before;
You haven't registered a buffer before;

Then, this line will be execute:

object.__setattr__(self, name, value)

which is nothing but a normal attribute set, which calls your setter.

Now, when you set:

b.obj = B()

Then, because B() is a nn.Module, the following block will be executed instead:

modules = self.__dict__.get('_modules')
if isinstance(value, Module):
    if modules is None:
        raise AttributeError(
            "cannot assign module before Module.__init__() call")
    remove_from(self.__dict__, self._parameters, self._buffers)
    modules[name] = value

So, now you are actually registering a nn.Module to self.__dict__.get('_modules') (print it before and after and you'll see... do it before and after setting [1,2,3] as well).

After this point, if you are not setting a nn.Parameter, and you try to set .obj again, then it will fall into this block:

elif modules is not None and name in modules:
    if value is not None:
        raise TypeError("cannot assign '{}' as child module '{}' "
                        "(torch.nn.Module or None expected)"
                        .format(torch.typename(value), name))
    modules[name] = value

That is: you already have modules['obj'] set to something and from now on you need to provide another nn.Module or None if you want to set it again. And, as you can see, because you are providing a list if you try to set b.obj = [1,2,3] again, you'll get the error message in the block above, and that is what you get.

If you really want set it to something else, then you have to delete it before:

b.obj = B()
del b.obj
b.obj = [1,2,3]


Instruction: Differentiable image compression operations in PyTorch

Input:
During a CNN classification model training while calculating the loss I am applying the encoding jpeg compression on the image in PyTorch. While I call loss.backward() it must also backpropagate through encoding and compression operation performed on the images.

Are those compression algorithms (e.g. encoding and JPEG compression) are differentiable otherwise how to backpropagate the loss gradient through those operations?

If those operations are not differentiable is there any differentiable compression algorithm that exists in PyTorch which performs H.264 encoding and JPEG compression?

Any suggestions will be highly helpful.

Output:
To start with, carefully consider whether you need to differentiate across the JPEG compression step. The vast majority of projects do not differentiate across this step, and if you're unsure if you need to, you probably don't.

If you really need to differentiate across an image compressor, you might consider a codec that is easier to implement than JPEG. Wavelett-based compression (the technology behind the ill-fated JPEG 2000 format) is mathematically elegant and easy to differentiate across. In a recent application of this technique, Thies et al. 2019 represent an image as a laplacian pyramid, with a loss component that serves to force sparsity in the higher resolution levels.

Now, as a thought experiment, we can look at the different steps within JPEG compression and determine if they could be implemented in a differentiable way.

Color transform (RBG to YCbCr): We can represent this as a point-wise convolution.
Chroma downsampling: Easy enough with torch.nn.functional.interpolate on the chroma channels.
Discrete Cosine Transform (DCT): Now things are getting interesting. Here is a Pytorch implementation of DCT that might work: https://github.com/zh217/torch-dct.
Quantization table: Easy again. This should just be multiplying output of the DCT with the values in the table.
Huffman encoding: Hard; I'm not sure this is possible. The number of output elements is going to vary based on the image entropy, which rules out many differentiable building blocks. Depending on your application, you might be able to skip this step (this step is lossless compression; so if you're trying to differentiate across the compression artifacts introduced by JPEG, the previous steps should be sufficient).

For an interesting related work on inputting JPEG DCT components directly into a neural net, see Faster Neural Networks Straight from JPEG.