Instruction: Visualizing Neural Network Layer Activation

Input:
Feature visualizing in tensor flow or keras is easy and can be found here. https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/ or Convolutional Neural Network visualization - weights or activations?

how to do this in pytorch?

I am using PyTorch with pretrained resnet18 model. All i need to input the image and get activation for specific layer(e.g. Layer2.0.conv2). Layer2.0.conv2 is specified in the pretrained model.

In simple words; how to convert link one code to PyTorch? how to get the specific layers in resnet18 PyTorch and how to get the activation for input image.
I tried this in tensorflow and it worked but not PyTorch.

Output:
You would have to register PyTorch's hooks on specific layer. See this tutorial for intro about hooks.

Basically, it allows to capture input/output of forward/backward going into the torch.nn.Module. Whole thing could be a bit complicated, there exists a library with similar goal to your (disclaimer I'm the author), called torchfunc. Especially torchfunc.hooks.recorder allows you to do what you want, see code snippet and comments below:

import torchvision
import torchfunc

my_network = torchvision.resnet18(pretrained=True)
# Recorder saving inputs to all submodules
recorder = torchfunc.hooks.recorders.ForwardPre()

# Will register hook for all submodules of resnet18
# You could specify some submodules by index or by layer type, see docs
recorder.modules(my_networks)

# Push example image through network
my_network(torch.randn(1, 3, 224, 224))

You could register recorder only for some layers (submodule) specified by index or layer type, to get necessary info, run:

# Zero image before going into the third submodule of this network
recorder.data[3][0]

# You can see all submodules and their positions by running this:
for i, submodule in enumerate(my_network.modules()):
    print(i, submodule)

# Or you can just print the network to get this info
print(my_network)


Instruction: Transforming every training points without using dataloaders

Input:
I just found out that even though torchvision.dataset.MNIST accepts the transformer parameter, ...

transform = transforms.compose(
    [transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))]
)
mnist_trainset = datasets.mnist(
    root="mnist", train=True, download=True, transform=transform
)

...the value obtained from the mnist_trainset.data variable is still not transformed (please observe that the data in the range of (0, 255) should be normalised to (-1, 1) regarding the transformer's behaviour).

[102] mnist_testset.data[0].min()
tensor(0, dtype=torch.uint8)

[103] mnist_testset.data[0].max()
tensor(255, dtype=torch.uint8)

I tried calling mnist_trainset.transform over mnist_trainset.data, but the output shape is not what I intended

[104] mnist_testset.data.shape
torch.Size([10000, 28, 28])

[105] transform(mnist_testset.data).shape
torch.Size([3, 28, 28])

# Should be [10000, 28, 28] as identical to the original data.

I can use the DataLoader to load the entire training set and set the shuffling to False, but I think it's too overkilling. What is the best way to transform the entire mnist_testset using the defined transformer object, in order to obtain the intended transformed image, without having to manually transform it one-by-one?

Output:
Transforms are invoked when you sample the dataset using its __getitem__ method. So you could do something like the following to get all the transformed data.

imgs_transformed = []
for img, label in mnist_testset:
    imgs_transformed.append(img[0,:,:])

or using list comprehension

imgs_transformed = [img[0,:,:] for img, label in mnist_testset]

If you want to turn this into one big tensor you can use torch.stack

data_transformed = torch.stack(imgs_transformed, dim=0)


Instruction: PyTorch: Learning rate scheduler

Input:
How do I use a learning rate scheduler with the following optimizer?
optimizer = torch.optim.Adam(optim_params,betas=(args.momentum, args.beta), weight_decay=args.weight_decay)

I have written the following scheduler:
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)

I am not sure about whether I should step the scheduler or the optimizer. Which order should I take to perform the following?
optimizer.zero_grad()
scheduler.step()
optimizer.step()

Output:
Since 1.3 the behaviour was changed, see releases and this issue especially.

Before this version, you should step scheduler before optimizer, which IMO wasn't reasonable. There was some back and forth (actually it breaks backward compatibility and IMO it's not a good idea to break it for such a minor inconvenience), but currently you should step scheduler after optimizer.

optimizer.zero_grad()
optimizer.step()
scheduler.step()


Instruction: loading pretrained (CNN) model from .ckpt file using Pytorch

Input:
I am using Pytorch for image classification. I am looking for CNN models pretrained on a dataset other than ImageNet, I have found a link to a ".ckpt" file. I also found tutorials on loading this file with Tenserflow, but not using pytorch.
How can I load pretrained model using Pytorch from ".ckpt" file ?

Output:
I agree with @jodag that in general, PyTorch and Tensorflow are not interoperable. There are some special cases in which you may be able to do this. For example, HuggingFace provides support for converting the transformer model from TensorFlow to PyTorch.

There is a related (though closed) question on DataScience StackExchange where the idea is to rewrite the Tensorflow model into PyTorch and then loads the weights from the checkpoint file. Note that this can get tricky at times.


Instruction: What is nuclear norm of a tensor?

Input:
When I do

import torch
x = torch.ones(3, 4)
x.norm(p='nuc')

it gives

tensor(3.4641)

how is this value calculated?

Output:
The nuclear norm, aka the trace norm, is the sum of singular values of x or equivalently one of the expressions below (assuming x is real)
u,s,v = torch.svd(x, compute_uv=False)
print(torch.sum(s))

eigs, eigvecs = torch.symeig(x.transpose(1,0) @ x)
print(torch.sum(torch.sqrt(torch.abs(eigs))))

Finding the matrix square root isn't supported natively in pytorch (you could use symeig but then this would reduce to the previous expression). If you use something like this sqrtm implementation then you can compute the nuclear norm using
print(torch.trace(sqrtm(x.transpose(1,0) @ x))

From the above expression, it should be clear that if x is positive semi-definite then the trace norm is just
# use this only if you know x is positive semi-definite
print(torch.trace(x))


Instruction: NLP Transformers: Best way to get a fixed sentence embedding-vector shape?

Input:
I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some french sentences:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)

def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   embeddings = all_layers[0]
   return embeddings

# Here we see that the shape of the embedding vector depends on the number of tokens in the sentence

u = embed(sentence="Bonjour, ça va ?")
u.shape # torch.Size([1, 7, 768])
v = embed(sentence="Salut, comment vas-tu ?")
v.shape # torch.Size([1, 9, 768])

Imagine now in order to do some semantic search, I want to calculate the cosine distance between the vectors (tensors in our case) u and v :

cos = torch.nn.CosineSimilarity(dim=1)
cos(u, v) # will throw an error since the shape of `u` is different from the shape of `v`

I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of its tokens?

=> The first solution I'm thinking of is calculating the mean on axis=1 (embedding of a sentence is the mean embedding its tokens) since axis=0 and axis=2 have always the same size:

cos = torch.nn.CosineSimilarity(dim=1)
cos(u.mean(axis=1), v.mean(axis=1)) # works now and gives 0.7269

But, I'm afraid that I'm hurting the embedding of the sentence when calculating the mean since it gives the same weight for each token (maybe multiplying by TF-IDF?).

=> The second solution is to pad shorter sentences out. That means:

giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
look up for the sentence with the longest tokens and embed it, get its shape S
for the rest of sentences embed then pad zero to get the same shape S (the sentence has 0 in the rest of dimensions)

What are your thoughts?
What other techniques would you use and why?

Thanks in advance!

Output:
Bert-as-service is a great example of doing exactly what you are asking about.

They use padding. But read the FAQ, in terms of which layer to get the representation from how to pool it: long story short, depends on the task.

EDIT: I am not saying "use Bert-as-service"; I am saying "rip off what Bert-as-service does."

In your example, you are getting word embeddings (because of the layer you are extracting from). Here is how Bert-as-service does that. So, it actually shouldn't surprise you that this depends on sentence length.

You then talk about getting sentence embeddings by mean pooling over word embeddings. That is... a way to do it. But, using Bert-as-service as a guide for how to get a fixed-length representation from Bert...

  Q: How do you get the fixed representation? Did you do pooling or something?

  A: Yes, pooling is required to get a fixed representation of a sentence. In the default strategy REDUCE_MEAN, I take the second-to-last hidden layer of all of the tokens in the sentence and do average pooling.

So, to do Bert-as-service's default behavior, you'd do

def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   pooling_layer = all_layers[-2]
   embedded = pooling_layer.mean(1)  # 1 is the dimension you want to average ovber
   # note, using numpy to take the mean is bad if you want to stay on GPU
   return embedded


Instruction: MultiLabel Soft Margin Loss in PyTorch

Input:
I want to implement a classifier which can have 1 of 10 possible classes. I am trying to use the MultiClass Softmax Loss Function to do this. Going through the documentation I'm not clear with what input is required for the function.

The documentation says it needs two matrices of [N, C] of which one is input and the other is target. As much as I understand, input matrix would be the one my Neural Network would calculate, which would have probabilities given by the neural network to each of the 10 classes. The target is the one that I have from my dataset.

The documentation says - "Target(N, C) - label targets padded by -1 ensuring same shape as the input." What does this mean? Do I pass zeros in incorrect classes and -1 for the correct one?

It would be great if someone could elaborate on this and show even a sample 2d matrix which could be passed as a target matrix.

Output:
If you know that for each example you only have 1 of 10 possible classes, you should be using CrossEntropyLoss, to which you pass your networks predictions, of shape [batch, n_classes], and labels of shape [batch] (each element of labels is an integer between 0 and n_classes-1).

The loss you're looking at is designed for situations where each example can belong to multiple classes (say a person can be classified as both female and old). I think it's this "multi" that confuses you - it stands for the multiple possible classifications per example, not just multiple potential labels in the whole "universe".

In the sense of two/more labels in the universe, in which you seem to have been thinking, the counterpart to CrossEntropyLoss would be BCELoss (BCE stands for Binary Cross Entropy), which is just a simplification of CrossEntropyLoss for the case of two labels.


Instruction: convLSTM size mismatch

Input:
I'm trying to implement just one cell of Convolutional LSTM and passing a tensor (1,3,128,128) in it. I get a size mismatch error.

    class ConvLSTMCell(nn.Module):
        def __init__(self, input_size, input_dim, hidden_dim, kernel_size, bias):
            """
            Parameters
            ----------
            input_size: (int, int)
                Height and width of input tensor as (height, width).
            input_dim: int
                Number of channels of input tensor.
            hidden_dim: int
                Number of channels of hidden state.
            kernel_size: (int, int)
                Size of the convolutional kernel.
            bias: bool
                Whether or not to add the bias.
            """
            super(ConvLSTMCell, self).__init__()

            self.height, self.width = input_size
            self.input_dim  = input_dim
            self.hidden_dim = hidden_dim

            self.kernel_size = kernel_size
            # self.padding     = kernel_size[0] // 2, kernel_size[1] // 2
            self.bias        = bias

            self.conv = nn.Conv2d(in_channels=self.input_dim + self.hidden_dim,
                              out_channels=4 * self.hidden_dim,
                              kernel_size=self.kernel_size,
                              #padding=self.padding,
                              bias=self.bias)

        def forward(self, input, prev_state):
            h_prev, c_prev = prev_state
            print('x: {}\nh_prev: {}\nc_prev: {}'.format(x.size(), h_prev.size(), c_prev.size()))
            combined = torch.cat((input, h_prev), dim=1) # concatenate along channel axis
            print('combined: {}'.format(combined.size()))

            combined_conv = self.conv(combined)
            print('combined_conv: {}'.format(combined_conv.size()))
            cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)
            print('cc_i: {}\ncc_f: {}\ncc_o: {}\ncc_g: {}'.format(cc_i.size(), cc_f.size(),   cc_o.size(), cc_g.size()))

            i = torch.sigmoid(cc_i)
            f = torch.sigmoid(cc_f)
            o = torch.sigmoid(cc_o)
            g = torch.tanh(cc_g)
            print('i: {}\nf: {}\no: {}\ng: {}'.format(i.size(), f.size(), o.size(), g.size()))

            c_cur = f * c_prev + i * g
            h_cur = o * F.tanh(c_cur)
            print('c_cur: {}\nh_cur: {}'.format(c_cur.size(), h_cur.size()))

            return h_cur, c_cur

        def init_hidden(self, batch_size):
            return (Variable(torch.zeros(batch_size, self.hidden_dim, self.height, self.width)),
                    Variable(torch.zeros(batch_size, self.hidden_dim, self.height, self.width)))

    x = torch.randn(1,3,128,128)
    model = ConvLSTMCell(input_size=(128,128), input_dim=3, hidden_dim=3, kernel_size=(5,5),
                         bias=True)
    hc = model.init_hidden(batch_size=1)
    if gpu:
        x.cuda()
        model.cuda()
        hc.cuda()

    out = model(x, hc)
    print(out.size())

I get the following error:

x: torch.Size([1, 3, 128, 128])
h_prev: torch.Size([1, 3, 128, 128])
c_prev: torch.Size([1, 3, 128, 128])
combined: torch.Size([1, 6, 128, 128])
combined_conv: torch.Size([1, 12, 124, 124])
cc_i: torch.Size([1, 3, 124, 124])
cc_f: torch.Size([1, 3, 124, 124])
cc_o: torch.Size([1, 3, 124, 124])
cc_g: torch.Size([1, 3, 124, 124])
i: torch.Size([1, 3, 124, 124])
f: torch.Size([1, 3, 124, 124])
o: torch.Size([1, 3, 124, 124])
g: torch.Size([1, 3, 124, 124])
Traceback (most recent call last):
File "trial.py", line 87, in
out = model(x, hc)
File "/Users/abcde/opt/anaconda3/envs/matrix/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "trial.py", line 66, in forward
c_cur = f * c_prev + i * g
RuntimeError: The size of tensor a (124) must match the size of tensor b (128) at non-singleton >dimension 3

I'm wish to build a network of 17 cells using this and I want to use the output of every cell to calculate loss against the ground truth. Ground truth are 18 (3,128,128) images.
How do I get my network to output hidden states of the same size?

Output: Your outputs are smaller because of border effects - the convolution operation only computes values at coordinates where the kernel can fully fit within input shape. The simplest solution would be to apply padding to your convolution layer (which you seem to have already tried, was there anything wrong with that?). If your kernel is of size 5, you should pad by 2 and then the convolution output will be of the same shape as input.


Instruction: Is there an equivalent function pytorch's loss fuction nn.crossEntropyLoss() in keras?

Input:
I'm converting conv2d model written by pytorch to keras(Tensorflow 2.0).

I've almost done, but last question is delaying my work.

The title says it all: is there an equivalent function pytorch's loss fuction nn.crossEntropyLoss() in keras?

Thanks for your help.

Output:
Depending on type of your labels and outputs, there are currently three different entropy losses supported in Keras.

categorical_crossentropy:

keras.losses.categorical_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)

sparse_categorical_crossentropy:

keras.losses.sparse_categorical_crossentropy(y_true, y_pred, from_logits=False, axis=-1)

binary_crossentropy:

keras.losses.binary_crossentropy(y_true, y_pred, from_logits=False, label_smoothing=0)

You can find a full list of Keras losses here


Instruction: Pytorch BCELoss not accepting lists

Input:
My convLSTM model returns a list of hidden states (17 total, size (1,3,128,128)) and my target is a list of 17 images( all tensors size: (3,128,128)
When the loss function is called, I get the following error:

  File "/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site->packages/torch/nn/modules/loss.py", line 498, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, >reduction=self.reduction)
  File "/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site->packages/torch/nn/functional.py", line 2052, in binary_cross_entropy
     if target.size() != input.size():
  AttributeError: 'list' object has no attribute 'size'

Part of the training loop:

    hc = model.init_hidden(batch_size=1)
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        # Set target, images 2 to 18
        target = data[1:]
        if gpu:
            data = data.cuda()
            target = target.cuda()
            hc.cuda()
        # Get outputs of LSTM
        output = model(data, hc)
        # Calculate loss
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

I was expecting a size mismatch error but got this instead. How can I fix this?

Output: Hi I solved it by using torch.stack. Could have used torch.cat but wanted a tensor with a list of tensors to pass to the loss function to match the target format so used torch.stack.


Instruction: How to sample from normal distribution?

Input:
If I have n sized mean vector, n sized variance vector, then how do I do this?

z ∼ N (μ, σ)

import torch
x = torch.randn(3, 3)
mu = x.mean()
sigma = x.var()

What do I do to get z?

Output:
If you want to sample from a normal distribution with mean mu and std sigma then you can simply

z = torch.randn_like(mu) * sigma + mu

If you sample many such z their mean and std will converge to sigma and mu:

mu = torch.arange(10.)

Out[]: tensor([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])

sigma = 5. - 0.5 * torch.arange(10.)

Out[]: tensor([5.0000, 4.5000, 4.0000, 3.5000, 3.0000, 2.5000, 2.0000, 1.5000, 1.0000, 0.5000])

z = torch.randn(10, 1000000) * sigma[:, None] + mu[:, None]

z.mean(dim=1)
Out[]:
tensor([-5.4823e-03,  1.0011e+00,  1.9982e+00,  2.9985e+00,  4.0017e+00,
         4.9972e+00,  6.0010e+00,  7.0004e+00,  7.9996e+00,  9.0006e+00])

z.std(dim=1)
Out[]:
tensor([4.9930, 4.4945, 4.0021, 3.5013, 3.0005, 2.4986, 1.9997, 1.4998, 0.9990,
        0.5001])

As you can see when you sample 1,000,000 elements from the distribution the sample mean and std are close to the original mu and sigma you started with.


Instruction: what is the default weight initialization used in Pytorch embedding layer?

Input: When we create an embedding layer using the class torch.nn.Embedding, how are the weights initialized ? Is uniform, normal or initialization techniques like He or Xavier used by default?

Output:
In Embedding, by default, the weights are initialization from the Normal distribution. You can check it from the reset_parameters() method:

def reset_parameters(self):
        init.normal_(self.weight)
        ...


Instruction: Use of torchvision.utils.save_image twice on the same image tensor makes the second save not work. What's going on?

Input:
(Fast Gradient Sign Attack method detailed here: https://pytorch.org/tutorials/beginner/fgsm_tutorial.html)

I have a trained classifier with >90% accuracy which I am using to create these adversarial examples, then I am using torchvision.utils.save_image to save the images to different folders.

The folder hierarchy is as follows:

FOLDER_1

original_image.jpg (1)
perturbed_image.jpg (2)

FOLDER_2

perturbed_image.jpg (3)

Here (2) and (3) are the same image tensor, which is the sum of the original image tensor and a perturbation image tensor---I just want to save images that fooled the classifier twice. What I'm finding is that (1) and (2) print O.K., but (3) only prints the perturbation image tensor (it subtracted the original image tensor!). So when I open up (2) I see my original picture with all the noise on top (random RGB pixel switches from the FGSM attack), but when I open up the (3) I see a blank canvas with those random RGB pixel switches ONLY.

Since I am printing the same variable (perturbed_data) twice, I don't understand why torchvision.utils.save_image is choosing to subtract the perturbation image tensor the second time I call it. The code for what I'm describing is below, and data is the original image tensor.

epsilon = 0.5
# Collect datagrad
data_grad = data.grad.data

# Call FGSM Attack
perturbed_data = fgsm_attack(data, epsilon, data_grad)

# Re-classify the perturbed image
perturbed_output = model(perturbed_data)
perturbed_output = torch.sigmoid(perturbed_output)
perturbed_output = perturbed_output.max(1, keepdim=True)[1]
max_pred  = perturbed_output.item()
final_pred = torch.tensor([0, 0]).to(device)
final_pred[max_pred] = 1

# Store all original and perturbed images, regardless of model prediction
torchvision.utils.save_image(data, "./FOLDER_1/original.jpg")
torchvision.utils.save_image(perturbed_data, "./FOLDER_1/perturbed_image.jpg")

# If the perturbed image fools our classifier, put a copy of it in FOLDER_2
if !torch.all(torch.eq(final_pred, target)):
    torchvision.utils.save_image(perturbed_data, "./FOLDER_2/perturbed_image.jpg")

I'm almost sure that this is a torchvision bug, but I thought I would ask here before submitting a bug report. Maybe someone sees something I don't. I've also attached an example of (2) and (3) for visualization. The first image is in the correct format, but the second one prints without the original image tensor.

Output:
It turns out torchvision.utils.save_image modifies the input tensor. A workaround to this is to add a line somewhere before calling torchvision.utils.save_image that's similar to this:

perturbed_data_copy = perturbed_data

Then you can safely save the perturbed image twice if on the second call you use perturbed_data_copy instead of the perturbed_data (which was modified by torchvision.utils.save_image). I will be submitting a bug report and tagging this post. Thanks @Mat for pointing this out!


Instruction: Transform flair language model tensors for viewing in TensorBoard Projector

Input:
I want to convert "vectors,"

vectors = [token.embedding for token in sentence]
print(type(vectors))
<class 'list'>

print(vectors)
[tensor([ 0.0077, -0.0227, -0.0004,  ...,  0.1377, -0.0003,  0.0028]),
...
tensor([ 0.0003, -0.0461,  0.0043,  ..., -0.0126, -0.0004,  0.0142])]

to

0.0077 -0.0227 -0.0004 ... 0.1377 -0.0003 0.0028
...
0.0003 -0.0461 0.0043 ... -0.0126 -0.0004 0.0142

and write that to a TSV.

Aside: those embeddings are from flair (https://github.com/zalandoresearch/flair): how can I get the full output, not the -0.0004 ... 0.1377 abbreviated output?

Output:
OK, I dug around ...

It turns out those are PyTorch tensors (Flair uses PyTorch).  For a simple conversion to NumPy arrays (per the PyTorch docs at https://pytorch.org/docs/stable/tensors.html#torch.Tensor.tolist and this StackOverFlow answer use tolist(), a PyTorch method.

>>> import torch
>>> a = torch.randn(2, 2)
>>> print(a)
tensor([[-2.1693,  0.7698],
        [ 0.0497,  0.8462]])

>>> a.tolist()
[[-2.1692984104156494, 0.7698001265525818],
 [0.049718063324689865, 0.8462421298027039]]

Per my original question, here's how to convert those data to plain text and write them to TSV files.

from flair.embeddings import FlairEmbeddings, Sentence
from flair.models import SequenceTagger
from flair.embeddings import StackedEmbeddings

embeddings_f = FlairEmbeddings('pubmed-forward')
embeddings_b = FlairEmbeddings('pubmed-backward')

sentence = Sentence('The RAS-MAPK signalling cascade serves as a central node in transducing signals from membrane receptors to the nucleus.')

tagger = SequenceTagger.load('ner')

tagger.predict(sentence)
embeddings_f.embed(sentence)

stacked_embeddings = StackedEmbeddings([
    embeddings_f,
    embeddings_b,
])

stacked_embeddings.embed(sentence)

# for token in sentence:
#     print(token)
#     print(token.embedding)
#     print(token.embedding.shape)

tokens = [token for token in sentence]
print(tokens)
'''
  [Token: 1 The, Token: 2 RAS-MAPK, Token: 3 signalling, Token: 4 cascade, Token: 5 serves, Token: 6 as, Token: 7 a, Token: 8 central, Token: 9 node, Token: 10 in, Token: 11 transducing, Token: 12 signals, Token: 13 from, Token: 14 membrane, Token: 15 receptors, Token: 16 to, Token: 17 the, Token: 18 nucleus.]
'''

## https://www.geeksforgeeks.org/python-string-split/

tokens = [str(token).split()[2] for token in sentence]
print(tokens)
'''
  ['The', 'RAS-MAPK', 'signalling', 'cascade', 'serves', 'as', 'a', 'central', 'node', 'in', 'transducing', 'signals', 'from', 'membrane', 'receptors', 'to', 'the', 'nucleus.']
'''

tensors = [token.embedding for token in sentence]
print(tensors)
'''
  [tensor([ 0.0077, -0.0227, -0.0004,  ...,  0.1377, -0.0003,  0.0028]),
  tensor([-0.0007, -0.1601, -0.0274,  ...,  0.1982,  0.0013,  0.0042]),
  tensor([ 4.2534e-03, -3.1018e-01, -3.9660e-01,  ...,  5.9336e-02, -9.4445e-05,  1.0025e-02]),
  tensor([ 0.0026, -0.0087, -0.1398,  ..., -0.0037,  0.0012,  0.0274]),
  tensor([-0.0005, -0.0164, -0.0233,  ..., -0.0013,  0.0039,  0.0004]),
  tensor([ 3.8261e-03, -7.6409e-02, -1.8632e-02,  ..., -2.8906e-03, -4.4556e-04,  5.6909e-05]),
  tensor([ 0.0035, -0.0207,  0.1700,  ..., -0.0193,  0.0017,  0.0006]),
  tensor([ 0.0159, -0.4097, -0.0489,  ...,  0.0743,  0.0005,  0.0012]),
  tensor([ 9.7725e-03, -3.3817e-01, -2.2848e-02,  ..., -6.6284e-02, 2.3646e-04,  1.0505e-02]),
  tensor([ 0.0219, -0.0677, -0.0154,  ...,  0.0102,  0.0066,  0.0016]),
  tensor([ 0.0092, -0.0431, -0.0450,  ...,  0.0060,  0.0002,  0.0005]),
  tensor([ 0.0047, -0.2732, -0.0408,  ...,  0.0136,  0.0005,  0.0072]),
  tensor([ 0.0072, -0.0173, -0.0149,  ..., -0.0013, -0.0004,  0.0056]),
  tensor([ 0.0086, -0.1151, -0.0629,  ...,  0.0043,  0.0050,  0.0016]),
  tensor([ 7.6452e-03, -2.3825e-01, -1.5683e-02,  ..., -5.4974e-04, -1.4646e-04,  6.6120e-03]),
  tensor([ 0.0038, -0.0354, -0.1337,  ...,  0.0060, -0.0004,  0.0102]),
  tensor([ 0.0186, -0.0151, -0.0641,  ...,  0.0188,  0.0391,  0.0069]),
  tensor([ 0.0003, -0.0461,  0.0043,  ..., -0.0126, -0.0004,  0.0142])]
'''

# ----------------------------------------
## Write those data to TSV files.

## https://stackoverflow.com/a/29896136/1904943

import csv

metadata_f = 'metadata.tsv'
tensors_f = 'tensors.tsv'

with open(metadata_f, 'w', encoding='utf8', newline='') as tsv_file:
    tsv_writer = csv.writer(tsv_file, delimiter='\t', lineterminator='\n')
    for token in tokens:
        ## Assign to a dummy variable ( _ ) to suppress character counts;
        ## if I use (token), rather than ([token]), I get spaces between all characters:
        _ = tsv_writer.writerow([token])

## metadata.tsv :
'''
  The
  RAS-MAPK
  signalling
  cascade
  serves
  as
  a
  central
  node
  in
  transducing
  signals
  from
  membrane
  receptors
  to
  the
  nucleus.
'''

with open(metadata_f, 'w', encoding='utf8', newline='') as tsv_file:
    tsv_writer = csv.writer(tsv_file, delimiter='\t', lineterminator='\n')
    _ = tsv_writer.writerow(tokens)

## metadata.tsv :
'''
  The   RAS-MAPK    signalling  cascade serves  as  a   central node    in  transducing signals from    membrane    receptors   to  the nucleus.
'''

tensors = [token.embedding for token in sentence]
print(tensors)
'''
  [tensor([ 0.0077, -0.0227, -0.0004,  ...,  0.1377, -0.0003,  0.0028]),
  tensor([-0.0007, -0.1601, -0.0274,  ...,  0.1982,  0.0013,  0.0042]),
  tensor([ 4.2534e-03, -3.1018e-01, -3.9660e-01,  ...,  5.9336e-02, -9.4445e-05,  1.0025e-02]),
  tensor([ 0.0026, -0.0087, -0.1398,  ..., -0.0037,  0.0012,  0.0274]),
  tensor([-0.0005, -0.0164, -0.0233,  ..., -0.0013,  0.0039,  0.0004]),
  tensor([ 3.8261e-03, -7.6409e-02, -1.8632e-02,  ..., -2.8906e-03, -4.4556e-04,  5.6909e-05]),
  tensor([ 0.0035, -0.0207,  0.1700,  ..., -0.0193,  0.0017,  0.0006]),
  tensor([ 0.0159, -0.4097, -0.0489,  ...,  0.0743,  0.0005,  0.0012]),
  tensor([ 9.7725e-03, -3.3817e-01, -2.2848e-02,  ..., -6.6284e-02, 2.3646e-04,  1.0505e-02]),
  tensor([ 0.0219, -0.0677, -0.0154,  ...,  0.0102,  0.0066,  0.0016]),
  tensor([ 0.0092, -0.0431, -0.0450,  ...,  0.0060,  0.0002,  0.0005]),
  tensor([ 0.0047, -0.2732, -0.0408,  ...,  0.0136,  0.0005,  0.0072]),
  tensor([ 0.0072, -0.0173, -0.0149,  ..., -0.0013, -0.0004,  0.0056]),
  tensor([ 0.0086, -0.1151, -0.0629,  ...,  0.0043,  0.0050,  0.0016]),
  tensor([ 7.6452e-03, -2.3825e-01, -1.5683e-02,  ..., -5.4974e-04, -1.4646e-04,  6.6120e-03]),
  tensor([ 0.0038, -0.0354, -0.1337,  ...,  0.0060, -0.0004,  0.0102]),
  tensor([ 0.0186, -0.0151, -0.0641,  ...,  0.0188,  0.0391,  0.0069]),
  tensor([ 0.0003, -0.0461,  0.0043,  ..., -0.0126, -0.0004,  0.0142])]
'''

with open(tensors_f, 'w', encoding='utf8', newline='') as tsv_file:
    tsv_writer = csv.writer(tsv_file, delimiter='\t', lineterminator='\n')
    for token in sentence:
        embedding = token.embedding
        _ = tsv_writer.writerow(embedding.tolist())

## tensors.tsv (18 lines: one embedding per token in metadata.tsv):
## note: enormous output, even for this simple sentence.
'''
  0.007691788021475077  -0.02268664352595806    -0.0004340760060586035  ...
'''

Last, my intention for all of that was to load contextual language embeddings (Flair, etc.) into TensorFlow's Embedding Projector.  It turns out all I needed to to was to convert (here, Flair data) to NumPy arrays, and load them into a TensorFlow TensorBoard instance (no need for TSV files!).

I describe that in detail in my blog post, here: Visualizing Language Model Tensors (Embeddings) in TensorFlow's TensorBoard [TensorBoard Projector: PCA; t-SNE; ...].


Instruction: Is this the way to create a PyTorch scalar?

Input:
I'm new to PyTorch, and just wanted to kindly confirm if the following creates scalars with values, 1, 2, and 3, respectively?

import torch

a = torch.tensor(1)
b = torch.tensor(2)
c = torch.tensor(3)

Thanks.

Output:
From the example in the documentation for torch.tensor:

>>> torch.tensor(3.14159)  # Create a scalar (zero-dimensional tensor)
tensor(3.1416)

Therefore, it appears they endorse that passing a number creates the corresponding scalar.


Instruction: How to get rid of every column that are filled with zero from a Pytorch tensor?

Input:
I have a pytorch tensor A like below:

A =
tensor([[  4,   3,   3,  ...,   0,   0,   0],
        [ 13,   4,  13,  ...,   0,   0,   0],
        [707, 707,   4,  ...,   0,   0,   0],
        ...,
        [  7,   7,   7,  ...,   0,   0,   0],
        [  0,   0,   0,  ...,   0,   0,   0],
        [195, 195, 195,  ...,   0,   0,   0]], dtype=torch.int32)

I would like to:

identify all the columns whose all of its entries are equal to 0
delete only those columns that has all of their entries equal to 0

I can imagine doing:

zero_list = []
for j in range(A.size()[1]):
    if torch.sum(A[:,j]) == 0:
         zero_list = zero_list.append(j)

to identify the columns that only has 0 for its elements
but I am not sure how to delete such columns filled with 0 from the original tensor.

How can I delete the columns with zero from a pytorch tensor based on the index number?

Thank you,

Output:
It makes more sense to index the columns you want to keep instead of what you want to delete.

valid_cols = []
for col_idx in range(A.size(1)):
    if not torch.all(A[:, col_idx] == 0):
        valid_cols.append(col_idx)
A = A[:, valid_cols]

Or a little more cryptically

valid_cols = [col_idx for col_idx, col in enumerate(torch.split(A, 1, dim=1)) if not torch.all(col == 0)]
A = A[:, valid_cols]


Instruction: Fail to load a .pth file (pre-trained neural network) using torch.load() on google colab

Input:
My google drive is linked to my google colab notebook. Using the pytorch library torch.load($PATH)  fails to load this 219 Mo file (pre-trained neural network) (https://drive.google.com/drive/folders/1-9m4aVg8Hze0IsZRyxvm5gLybuRLJHv-) which is in my google drive. However it works fine when I do it locally on my computer. The error i get on google collab is: (settings: Python 3.6, pytorch 1.3.1):

state_dict = torch.load(model_path)['state_dict']
File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 454, in _load
return legacy_load(f)
File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 380, in legacy_load
with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar,
File "/usr/lib/python3.6/tarfile.py", line 1589, in open
return func(name, filemode, fileobj, **kwargs)
File "/usr/lib/python3.6/tarfile.py", line 1619, in taropen
return cls(name, mode, fileobj, **kwargs)
File "/usr/lib/python3.6/tarfile.py", line 1482, in init
self.firstmember = self.next()
File "/usr/lib/python3.6/tarfile.py", line 2297, in next
tarinfo = self.tarinfo.fromtarfile(self)
File "/usr/lib/python3.6/tarfile.py", line 1092, in fromtarfile
buf = tarfile.fileobj.read(BLOCKSIZE)
OSError: [Errno 5] Input/output error```

Any help would be much appreciated!

Output:
It worked by uploading directly the file to google colab instead of loading it from google drive using:

from google.colab import files
uploaded= files.upload()

I guess this solution is similar to the one proposed by @Yuri


Instruction: Compare Number of Equal Elements in Tensors

Input: I have two tensors of dimension 1000 * 1. I want to check how many of the 1000 elements are equal in the two tensors. I think I should be able to do this in one line like Numpy but couldn't find a similar function.

Output:
You can just use the == operator to check for equality and then sum the resulting tensor:

# Import torch and create dummy tensors
>>> import torch
>>> A = torch.randint(2, (10,))
>>> A
tensor([0, 0, 0, 1, 0, 1, 0, 0, 1, 1])
>>> B = torch.randint(2, (10,))
>>> B
tensor([0, 1, 1, 0, 1, 0, 0, 1, 1, 0])

# Checking for number of equal values
>>> (A == B).sum()
tensor(3)

Edit:

torch.eq yields to the same result. So if you for some reason prefer that:

>>> torch.eq(A, B).sum()
tensor(3)


Instruction: Faster method of computing confusion matrix?

Input:
I am computing my confusion matrix as shown below for image semantic segmentation which is a pretty verbose approach:

def confusion_matrix(preds, labels, conf_m, sample_size):
    preds = normalize(preds,0.9) # returns [0,1] tensor
    preds = preds.flatten()
    labels = labels.flatten()
    for i in range(len(preds)):
        if preds[i]==1 and labels[i]==1:
            conf_m[0,0] += 1/(len(preds)*sample_size) # TP
        elif preds[i]==1 and labels[i]==0:
            conf_m[0,1] += 1/(len(preds)*sample_size) # FP
        elif preds[i]==0 and labels[i]==0:
            conf_m[1,0] += 1/(len(preds)*sample_size) # TN
        elif preds[i]==0 and labels[i]==1:
            conf_m[1,1] += 1/(len(preds)*sample_size) # FN
    return conf_m

In the prediction loop:

conf_m = torch.zeros(2,2) # two classes (object or no-object)
for img,label in enumerate(data):
    ...
    out = Net(img)
    conf_m = confusion_matrix(out, label, len(data))
    ...

Is there a faster approach (in PyTorch) to efficiently calculate the confusion matrix for a sample of inputs for image semantic segmentation?

Output:
I use these 2 functions to calc confusion matrix (as it defined in sklearn):
# rewrite sklearn method to torch
def confusion_matrix_1(y_true, y_pred):
    N = max(max(y_true), max(y_pred)) + 1
    y_true = torch.tensor(y_true, dtype=torch.long)
    y_pred = torch.tensor(y_pred, dtype=torch.long)
    return torch.sparse.LongTensor(
        torch.stack([y_true, y_pred]),
        torch.ones_like(y_true, dtype=torch.long),
        torch.Size([N, N])).to_dense()

# weird trick with bincount
def confusion_matrix_2(y_true, y_pred):
    N = max(max(y_true), max(y_pred)) + 1
    y_true = torch.tensor(y_true, dtype=torch.long)
    y_pred = torch.tensor(y_pred, dtype=torch.long)
    y = N * y_true + y_pred
    y = torch.bincount(y)
    if len(y) < N * N:
        y = torch.cat(y, torch.zeros(N * N - len(y), dtype=torch.long))
    y = y.reshape(N, N)
    return y

y_true = [2, 0, 2, 2, 0, 1]
y_pred = [0, 0, 2, 2, 0, 2]

confusion_matrix_1(y_true, y_pred)
# tensor([[2, 0, 0],
#         [0, 0, 1],
#         [1, 0, 2]])

Second function is faster in case of small number of classes.
%%timeit
confusion_matrix_1(y_true, y_pred)
# 102 µs ± 30.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
confusion_matrix_2(y_true, y_pred)
# 25 µs ± 149 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)


Instruction: What is the best way to handle the background pixel classes (ignore_label), when training deep learning models for semantic segmentation?

Input:
I am trying to train a UNET model on the cityscapes dataset which has 20 'useful' semantic classes and a bunch of background classes that can be ignored (ex. sky, ego vehicle, mountains, street lights). To train the model to ignore these background pixels I am using the following popular solution on the internet :

I assign a common ignore_label (ex: ignore_label=255) for all
the pixels belonging to the ignore classes
Train the model using the cross_entropy loss for each pixel prediction
Provide the ignore_label parameter in the cross_entropy loss, therefore the
loss computed ignores the pixels with the unnecessary classes.

But this approach has a problem. Once trained, the model ends up classifying these background pixels as belonging to one of the 20 classes instead. This is expected as in the loss we do not penalize the model for whatever classification it makes for the background pixels.

The second obvious solution is therefore to use a extra class for all the background pixels. Therefore it is the 21st class in cityscapes. However, here I am worried that I will 'waste' my model's capacity by teaching it to classify this additional unnecessary class.

What is the most accurate way of handling the background pixel classes ?

Output:
Definitely the second solution is the better one. This is the best solution, the background class is definitely and additional class but not an unnecessary one, since in this way there is a clear differentiation between the classes you want to detect and the background.
In fact, this is a standard procedure recommended in segmentation, to assign a class to a background, where background of course represents everything else apart from your specific classes.


Instruction: what is the inputs to a torch.nn.gru function in pytorch?

Input:
I am using a gru function to implement a RNN. This RNN (GRU) is used after some CNN layers. Can someone please tell me what is the input to a GRU function here? Especially, is the hidden size fixed?

self.gru = torch.nn.GRU(
            input_size=input_size,
            hidden_size=128,
            num_layers=1,
            batch_first=True,
            bidirectional=True)

According to my understanding the input size will be the number of features and the hidden size for GRU is always fixed as 128? Can some one please correct me. Or give their feedback

Output:
First, GRU is not a function but a class and you are calling its constructor. You are creating an instance of class GRU here, which is a layer (or Module in pytorch).
The input_size must match the out_channels of the previous CNN layer.
None of the parameters you see is fixed. Just put another value there and it will be something else, i.e. replace the 128 with anything you like.
Even though it is called hidden_size, for a GRU this parameter also determines the output features. In other words, if you have another layer after GRU, this layer's input_size (or in_features or in_channels or whatever it is called) must match the GRU's hidden_size.
Also, have a look at the documentation. This tells you exactly what the parameters you pass to the constructor are good for. Also, it tells you what will be the expected input once you actually use your layer (via self.gru(...)) and what will be the output of that call.


Instruction: PyTorch DataLoader adding extra dimension for TorchVision MNIST

Input:
I am fairly new to PyTorch and have been experimenting with the DataLoader class.
When I attempt to load the MNIST dataset, the DataLoader appears to add an additional dimension after the batch dimension. I am not sure what is causing this to occur.

import torch
from torchvision.datasets import MNIST
from torchvision import transforms

if __name__ == '__main__':
    mnist_train = MNIST(root='./data', train=True, download=True, transform=transforms.Compose([transforms.ToTensor()]))
    first_x = mnist_train.data[0]
    print(first_x.shape)  # expect to see [28, 28], actual [28, 28]

    train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=200)
    batch_x, batch_y = next(iter(train_loader))  # get first batch
    print(batch_x.shape)  # expect to see [200, 28, 28], actual [200, 1, 28, 28]
    # Where is the extra dimension of 1 from?

Can anyone shed some light on the issue?

Output:
I guess that is the number of channels of the input image. So basically it is

batch_x.shape = Batch-size, No of channels, Height of the image, Width of the image


Instruction: How do I add some Gaussian noise to a tensor in PyTorch?

Input:
I have a tensor I created using

    temp = torch.zeros(5, 10, 20, dtype=torch.float64)
    ## some values I set in temp

Now I want to add to each temp[i,j,k] a Gaussian noise (sampled from normal distribution with mean 0 and variance 0.1). How do I do it? I would expect there is a function to noise a tensor, but couldn't find anything. I did find this:

How to add Poisson noise and Gaussian noise?

but it seems to be related to images.

Output:
The function torch.randn produces a tensor with elements drawn from a Gaussian distribution of zero mean and unit variance. Multiply by sqrt(0.1) to have the desired variance.

x = torch.zeros(5, 10, 20, dtype=torch.float64)
x = x + (0.1**0.5)*torch.randn(5, 10, 20)


Instruction: In python3: strange behaviour of list(iterables)

Input:
I have a specific question regarding the behaviour of iterables in python. My iterable is a custom built Dataset class in pytorch:

import torch
from torch.utils.data import Dataset
class datasetTest(Dataset):
    def __init__(self, X):
        self.X = X

    def __len__(self):
        return len(self.X)

    def __getitem__(self, x):
        print('***********')
        print('getitem x = ', x)
        print('###########')
        y = self.X[x]
        print('getitem y = ', y)
        return y

The weird behaviour now comes about when I initialize a specific instance of that datasetTest class. Depending on what data structure I pass as an argument X, it behaves differently when I call list(datasetTestInstance). In particular, when passing a torch.tensor as argument there is no problem, however when passing a dict as argument it will throw a KeyError. The reason for this is that list(iterable) not just calls i=0, ..., len(iterable)-1, but it calls i=0, ..., len(iterable). That is, it will iterate until (inclusive) the index equal to the length of the iterable. Obviously, this index is not definied in any python datastructure, as the last element has always the index len(datastructure)-1 and not len(datastructure). If X is a torch.tensor or a list, no error will be risen, even though I think the should be an error. It will still call getitem even for the (non-existent) element with index len(datasetTestinstance) but it will not compute y=self.X[len(datasetTestInstance]. Does anyone know if pytorch handels this somehow gracefully internally?

When passing a dict as data it will throw an error in the last iteration, when x=len(datasetTestInstance). This is actually the expected behaviour I guess. But why does this only happen for a dict and not for a list or torch.tensor?

if __name__ == "__main__":
    a = datasetTest(torch.randn(5,2))
    print(len(a))
    print('++++++++++++')
    for i in range(len(a)):
        print(i)
        print(a[i])
    print('++++++++++++')
    print(list(a))

    print('++++++++++++')
    b = datasetTest({0: 12, 1:35, 2:99, 3:27, 4:33})
    print(len(b))
    print('++++++++++++')
    for i in range(len(b)):
        print(i)
        print(b[i])
    print('++++++++++++')
    print(list(b))

You could try out that snippet of code if you want to understand better what I have observed.

My questions are:

1.) Why does list(iterable) iterate until (including) the len(iterable)? A for loop doesnt do that.

2.) In case of a torch.tensor or a list passed as data X: Why does it not throw an error even when calling the getitem method for the index len(datasetTestInstance) which should actually be out of range since it is not defined as an index in the tensor/list? Or, in other words, when having reached the index len(datasetTestInstance) and then going into the getitem method, what happens exactly? It obviously doesnt make the call 'y = self.X[x]' anymore (otherwiese there would be an IndexError) but it DOES enter the getitem method which I can see as it prints the index x from within the getitem method. So what happens in that method? And why does it behave different depending on whether having a torch.tensor/list or a dict?

Output:
This isn't really a pytorch specific issue, it's a general python question.

You're constructing a list using list(iterable) where an iterable class is one which implements sequence semantics.

Take a look here at the expected behavior of __getitem__ for sequence types (most relevant parts are in bold)

  object.__getitem__(self, key)

  Called to implement evaluation of
  self[key]. For sequence types, the accepted keys should be integers
  and slice objects. Note that the special interpretation of negative
  indexes (if the class wishes to emulate a sequence type) is up to the
  __getitem__() method. If key is of an inappropriate type, TypeError may be raised; if of a value outside the set of indexes for the
  sequence (after any special interpretation of negative values),
  IndexError should be raised. For mapping types, if key is missing (not
  in the container), KeyError should be raised.

  Note: for loops expect that an IndexError will be raised for illegal
  indexes to allow proper detection of the end of the sequence.

The problem here is that for sequence types python expects an IndexError in the case where __getitem__ is invoked with an invalid index. It appears the list constructor relies on this behavior. In your example when X is a dict, attempting to access an invalid key causes __getitem__ to raise KeyError instead which isn't expected so isn't caught and causes the construction of the list to fail.

Based on this information you could do something like the following

class datasetTest:
    def __init__(self):
        self.X = {0: 12, 1:35, 2:99, 3:27, 4:33}

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        if index < 0 or index >= len(self):
            raise IndexError
        return self.X[index]

d = datasetTest()
print(list(d))

I can't recommend doing this in practice since it relies on your dictionary X containing only the integer keys 0, 1, ..., len(X)-1 which means it end up behaving just like a list for most cases, so you're probably better off just using a list.


Instruction: What is the difference between .pt, .pth and .pwf extentions in PyTorch?

Input:
I have seen in some code examples, that people use .pwf as model file saving format. But in PyTorch documentation .pt and .pth are recommended. I used .pwf and worked fine for small 1->16->16 convolutional network.

My question is what is the difference between these formats? Why is .pwf extension not even recommended in PyTorch documentation and why do people still use it?

Output:
There are no differences between the extensions that were listed: .pt, .pth, .pwf. One can use whatever extension (s)he wants. So, if you're using torch.save() for saving models, then it by default uses python pickle (pickle_module=pickle) to save the objects and some metadata. Thus, you have the liberty to choose the extension you want, as long as it doesn't cause collisions with any other standardized extensions.
Having said that, it is however not recommended to use .pth extension when checkpointing models because it collides with Python path (.pth) configuration files. Because of this, I myself use .pth.tar or .pt but not .pth, or any other extensions.

The standard way of checkpointing models in PyTorch is not finalized yet. Here is an open issue, as of this writing: Recommend a different file extension for models (.PTH is a special extension for Python) - issues/14864
It's been suggested by @soumith to use:

.pt for checkpointing models in pickle format
.ptc for checkpointing models in pytorch compiled (for JIT)


Instruction: In Pytorch, how to test simple image with my loaded model?

Input:
I made a alphabet classification CNN model using Pytorch, and then use that model to test it with a single image that I've never seen before. I extracted a bounding box in my handwriting image with opencv, but I don't know how to apply it to the model.

bounded my_image

this is custom dataset

class CustomDatasetFromCSV(Dataset):
    def __init__(self, csv_path, height, width, transforms=None):
        """
        Args:
            csv_path (string): path to csv file
            height (int): image height
            width (int): image width
            transform: pytorch transforms for transforms and tensor conversion
        """
        self.data = pd.read_csv(csv_path)
        self.labels = np.asarray(self.data.iloc[:, 0])
        self.height = height
        self.width = width
        self.transforms = transforms

    def __getitem__(self, index):
        single_image_label = self.labels[index]
        # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28])
        img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')
        # Convert image from numpy array to PIL image, mode 'L' is for grayscale
        img_as_img = Image.fromarray(img_as_np)
        img_as_img = img_as_img.convert('L')
        # Transform image to tensor
        if self.transforms is not None:
            img_as_tensor = self.transforms(img_as_img)
        # Return image and the label
        return (img_as_tensor, single_image_label)

    def __len__(self):
        return len(self.data.index)

    transformations = transforms.Compose([
                            transforms.ToTensor()
                        ])

    alphabet_from_csv = CustomDatasetFromCSV("/content/drive/My Drive/A_Z Handwritten Data.csv",
                                                28, 28, transformations)

random_seed = 50
data_size = len(alphabet_from_csv)
indices = list(range(data_size))
split = int(np.floor(0.2 * data_size))
if True:
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, test_indices = indices[split:], indices[:split]
train_dataset = SubsetRandomSampler(train_indices)
test_dataset = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(dataset = alphabet_from_csv,
                                           batch_size = batch_size,
                                           sampler = train_dataset)
test_loader = torch.utils.data.DataLoader(dataset = alphabet_from_csv,
                                          batch_size = batch_size,
                                          sampler = test_dataset)

this is my model

class ConvNet3(nn.Module):
    def __init__(self, num_classes=26):
        super().__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 28, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(28),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

        self.layer2 = nn.Sequential(
            nn.Conv2d(28, 56, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(56),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

        self.fc = nn.Sequential(
            nn.Dropout(p = 0.5),
            nn.Linear(56 * 7 * 7, 512),
            nn.BatchNorm1d(512),
            nn.ReLU(),
            nn.Dropout(p = 0.5),
            nn.Linear(512, 26),
        )

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out
model = ConvNet3(num_classes).to(device)
loss_func = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
def train():
    # train phase
    model.train()

    # create a progress bar
    batch_loss_list = []
    progress = ProgressMonitor(length=len(train_dataset))

    for batch, target in train_loader:
        # Move the training data to the GPU
        batch, target = batch.to(device), target.to(device)

        # forward propagation
        output = model( batch )

        # calculate the loss
        loss = loss_func( output, target )

        # clear previous gradient computation
        optimizer.zero_grad()

        # backpropagate to compute gradients
        loss.backward()

        # update model weights
        optimizer.step()

        # update progress bar
        batch_loss_list.append(loss.item())
        progress.update(batch.shape[0], sum(batch_loss_list)/len(batch_loss_list) )

def test():
    # test phase
    model.eval()

    correct = 0

    # We don't need gradients for test, so wrap in
    # no_grad to save memory
    with torch.no_grad():
        for batch, target in test_loader:
            # Move the training batch to the GPU
            batch, target = batch.to(device), target.to(device)

            # forward propagation
            output = model( batch )

            # get prediction
            output = torch.argmax(output, 1)

            # accumulate correct number
            correct += (output == target).sum().item()

    # Calculate test accuracy
    acc = 100 * float(correct) / len(test_dataset)
    print( 'Test accuracy: {}/{} ({:.2f}%)'.format( correct, len(test_dataset), acc ) )

for epoch in range(num_epochs):

    print("{}'s try".format(int(epoch)+1))
    train()
    test()
    print("-----------------------------------------------------------------------------")

this is my image to bound

import cv2
import matplotlib.image as mpimg
im = cv2.imread('/content/drive/My Drive/my_handwritten.jpg')

gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (5, 5), 0)
thresh = cv2.adaptiveThreshold(blur, 255, 1, 1, 11, 2)

contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[1]
rects=[]

for cnt in contours:
  x, y, w, h = cv2.boundingRect(cnt)
  if h < 20: continue
  red = (0, 0, 255)
  cv2.rectangle(im, (x, y), (x+w, y+h), red, 2)
  rects.append((x,y,w,h))

cv2.imwrite('my_handwritten_bounding.png', im)

img_result = []
img_for_class = im.copy()

margin_pixel = 60

for rect in rects:
    #[y:y+h, x:x+w]
    img_result.append(
        img_for_class[rect[1]-margin_pixel : rect[1]+rect[3]+margin_pixel,
                      rect[0]-margin_pixel : rect[0]+rect[2]+margin_pixel])

    # Draw the rectangles
    cv2.rectangle(im, (rect[0], rect[1]),
                  (rect[0] + rect[2], rect[1] + rect[3]), (0, 0, 255), 2)

count = 0
nrows = 4
ncols = 7

plt.figure(figsize=(12,8))

for n in img_result:
    count += 1
    plt.subplot(nrows, ncols, count)
    plt.imshow(cv2.resize(n,(28,28)), cmap='Greys', interpolation='nearest')

plt.tight_layout()
plt.show()

Output:
You have already written the function test to test your net. The only thing you should do — create batch with one image with same preprocessing as images in your dataset.

def test_one_image(I, model):
    '''
    I - 28x28 uint8 numpy array
    '''

    # test phase
    model.eval()

    # convert image to torch tensor and add batch dim
    batch = torch.tensor(I / 255).unsqueeze(0)

    # We don't need gradients for test, so wrap in
    # no_grad to save memory
    with torch.no_grad():
        batch = batch.to(device)

        # forward propagation
        output = model( batch )

        # get prediction
        output = torch.argmax(output, 1)

    return output


Instruction: Facing error in pytorch v-1.1: "RuntimeError: all tensors must be on devices[0]"

Input:
I am using nn.DataParallel() for my model but facing an error.

I am doing something like

    self.model = self.model.to(device)
    self.model = nn.DataParallel(self.model)

If the device is cuda:1 then I get RuntimeError: all tensors must be on devices[0].
But if I change the device to cuda:0 the parallel training on multiple GPUs works with no error. I am wondering what the problem is.

Output:
I changed this: self.model = DataParallel(self.model) to this: self.model = DataParallel(self.model, device_ids=[1,0])
Now it is working fine. But if I write: self.model = DataParallel(self.model, device_ids=[0,1]) the error pops up. If there are more GPUs, say 4, then write: self.model = DataParallel(self.model, device_ids=[1,2,3,0])


Instruction: How to get rid of checkerboard artifacts

Input:
Hello fellow coders,

I am using a fully convolutional autoencoder to color black and white images, however, the output has a checkerboard pattern and I want to get rid of it. The checkerboard artifacts I have seen so far allways have been far smaller than mine and the usual way to get rid of them is replacing all unpooling operations with bilinear upsampling (I have been told that).

But I can not simply replace the unpooling operation because I work with different sized images, thus the unpooling operation is needed, else the output tensor could have a different size than the original.

TLDR:

How can I get rid of these checkerboard-artifacts without replacing the unpooling operations?

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()
        self.leaky_reLU = nn.LeakyReLU(0.2)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=1, return_indices=True)
        self.unpool = nn.MaxUnpool2d(kernel_size=2, stride=2, padding=1)
        self.softmax = nn.Softmax2d()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv5 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.ConvTranspose2d(in_channels=1024, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv7 = nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv8 = nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv9 = nn.ConvTranspose2d(in_channels=128, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv10 = nn.ConvTranspose2d(in_channels=64, out_channels=2, kernel_size=3, stride=1, padding=1)

    def forward(self, x):

        # encoder
        x = self.conv1(x)
        x = self.leaky_reLU(x)
        size1 = x.size()
        x, indices1 = self.pool(x)

        x = self.conv2(x)
        x = self.leaky_reLU(x)
        size2 = x.size()
        x, indices2 = self.pool(x)

        x = self.conv3(x)
        x = self.leaky_reLU(x)
        size3 = x.size()
        x, indices3 = self.pool(x)

        x = self.conv4(x)
        x = self.leaky_reLU(x)
        size4 = x.size()
        x, indices4 = self.pool(x)

        ######################
        x = self.conv5(x)
        x = self.leaky_reLU(x)

        x = self.conv6(x)
        x = self.leaky_reLU(x)
        ######################

        # decoder
        x = self.unpool(x, indices4, output_size=size4)
        x = self.conv7(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices3, output_size=size3)
        x = self.conv8(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices2, output_size=size2)
        x = self.conv9(x)
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices1, output_size=size1)
        x = self.conv10(x)
        x = self.softmax(x)

        return x

.

EDIT - Solution:

Skip-Connections are the way to go!

class AE(nn.Module):
    def __init__(self):
        super(AE, self).__init__()
        self.leaky_reLU = nn.LeakyReLU(0.2)
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2, padding=1, return_indices=True)
        self.unpool = nn.MaxUnpool2d(kernel_size=2, stride=2, padding=1)
        self.softmax = nn.Softmax2d()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv5 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.Conv2d(in_channels=1024, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv7 = nn.Conv2d(in_channels=1024, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv8 = nn.Conv2d(in_channels=512, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv9 = nn.Conv2d(in_channels=256, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv10 = nn.Conv2d(in_channels=128, out_channels=2, kernel_size=3, stride=1, padding=1)

    def forward(self, x):

        # encoder
        x = self.conv1(x)
        out1 = self.leaky_reLU(x)
        x = out1
        size1 = x.size()
        x, indices1 = self.pool(x)

        x = self.conv2(x)
        out2 = self.leaky_reLU(x)
        x = out2
        size2 = x.size()
        x, indices2 = self.pool(x)

        x = self.conv3(x)
        out3 = self.leaky_reLU(x)
        x = out3
        size3 = x.size()
        x, indices3 = self.pool(x)

        x = self.conv4(x)
        out4 = self.leaky_reLU(x)
        x = out4
        size4 = x.size()
        x, indices4 = self.pool(x)

        ######################
        x = self.conv5(x)
        x = self.leaky_reLU(x)

        x = self.conv6(x)
        x = self.leaky_reLU(x)
        ######################

        # decoder
        x = self.unpool(x, indices4, output_size=size4)
        x = self.conv7(torch.cat((x, out4), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices3, output_size=size3)
        x = self.conv8(torch.cat((x, out3), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices2, output_size=size2)
        x = self.conv9(torch.cat((x, out2), 1))
        x = self.leaky_reLU(x)

        x = self.unpool(x, indices1, output_size=size1)
        x = self.conv10(torch.cat((x, out1), 1))
        x = self.softmax(x)

        return x

Output:
Skip connection is commonly used in Encoder-Decoder architecture and it helps to produce accurate result by passing appearance information from shallow layer of encoder (discriminator) to corresponding deeper layer of decoder (generator). Unet is the widely used Encoder-Decoder type architecture. Linknet is also very popular and it differs with Unet in the way of fusing appearance information of encoder layer with the decoder layer. In case of Unet, incoming features (from encoder) are concatenated in the corresponding decoder layer. On the other hand, Linknet performs addition and that why Linknet requires fewer number of operations in a single forward pass and significantly faster than the Unet.

Your each convolution block in Decoder might looks like following:

Additionally, i'm attaching a figure bellow depicting architecture of Unet and LinkNet. Hope using skip connection will help.


Instruction: PyTorch: why is the loss unchanging, in this simple example?

Input:
I'm writing a code example to do a simple linear projection (like PCA) in PyTorch. Everything appears to be OK except that the loss does not change as training progresses. Changing the learning rate doesn't affect this, and it's a simple one-dimensional problem so the loss should certainly be changing. What am I missing here?

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as nnF

class PCArot2D(nn.Module):
    "2D PCA rotation, expressed as a gradient-descent problem"
    def __init__(self):
        super(PCArot2D, self).__init__()
        self.theta = nn.Parameter(torch.tensor(np.random.random() * 2 * np.pi))

    def getrotation(self):
        sintheta = torch.sin(self.theta)
        costheta = torch.cos(self.theta)
        return torch.tensor([[costheta, -sintheta], [sintheta, costheta]], requires_grad=True, dtype=torch.double)

    def forward(self, x):
        xmeans = torch.mean(x, dim=1, keepdim=True)
        rot = self.getrotation()

        return torch.mm(rot, x - xmeans)

def covariance(y):
    "Calculates the covariance matrix of its input (as torch variables)"
    ymeans = torch.mean(y, dim=1, keepdim=True)
    ycentred = y - ymeans
    return torch.mm(ycentred, ycentred.T) / ycentred.shape[1]

net = PCArot2D()

example2 = torch.tensor(np.random.randn(2, 33))

# define a loss function and an optimiser
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.001, momentum=0.1)

# train the network
num_epochs = 1000
for epoch in range(num_epochs):
    optimizer.zero_grad()
    # forward + backward + optimize
    outputs = net(torch.DoubleTensor(example2))
    # the covariance between output channels is the measure that we wish to minimise
    covariance = (outputs[0, :] * outputs[1, :]).mean()
    loss = criterion(covariance, torch.tensor(0, dtype=torch.double))

    loss.backward()
    optimizer.step()

    running_loss = loss.item()
    if ((epoch & (epoch - 1)) == 0) or epoch==(num_epochs-1): # don't print on all epochs
        # print statistics
        print('[%d] loss: %.8f' %
            (epoch, running_loss))

print('Finished Training')

Output:

[0] loss: 0.00629047
[1] loss: 0.00629047
[2] loss: 0.00629047
[4] loss: 0.00629047
[8] loss: 0.00629047
etc

Output:
It seems the problem is in your getrotation function. When creating a new tensor from the other tensors it is not back-probable anymore:

def getrotation(self):
    sintheta = torch.sin(self.theta)
    costheta = torch.cos(self.theta)
    return torch.tensor([[costheta, -sintheta], [sintheta, costheta]], requires_grad=True, dtype=torch.double)

So you need to find some other way to construct your return tensor.

Here is one suggestion that seems to work using torch.cat:

def getrotation(self):
    sintheta = torch.sin(self.theta)
    costheta = torch.cos(self.theta)
    #return torch.tensor([[costheta, -sintheta], [sintheta, costheta]], requires_grad=True, dtype=torch.double)
    A = torch.cat([costheta.unsqueeze(0), -sintheta.unsqueeze(0)], dim=0)
    B = torch.cat([sintheta.unsqueeze(0), costheta.unsqueeze(0)], dim=0)
    return torch.cat([A.unsqueeze(0), B.unsqueeze(0)], dim=0).double()

After implementing this change the loss changes:

[0] loss: 0.00765365
[1] loss: 0.00764726
[2] loss: 0.00764023
[4] loss: 0.00762607
[8] loss: 0.00759777
[16] loss: 0.00754148
[32] loss: 0.00742997
[64] loss: 0.00721117
[128] loss: 0.00679025
[256] loss: 0.00601233
[512] loss: 0.00469085
[999] loss: 0.00288501
Finished Training

I hope this helps!

Edit:
A simpler and prettier version by @DanStowell:

def getrotation(self):
    sintheta = torch.sin(net.theta).double().unsqueeze(0)
    costheta = torch.cos(net.theta).double().unsqueeze(0)
    return torch.cat([costheta, -sintheta, sintheta, costheta]).reshape((2,2))


Instruction: Convolution Neural Network for regression using pytorch

Input:
I am trying to do create CNN for regression purpose. Input is image data.
For learning purpose , i have 10 image of shape (10,3,448,448), where 10 are images, 3 are channel and 448 are hieght and width.
Output lables are (10,245).
Here is my architecture

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 32, kernel_size=5)
        self.conv3 = nn.Conv2d(32,64, kernel_size=5)
        self.fc1 = nn.Linear(3*3*64, 256)
        self.fc2 = nn.Linear(256, 245)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        #x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(F.max_pool2d(self.conv3(x),2))
        x = F.dropout(x, p=0.5, training=self.training)
        x = x.view(-1,3*3*64 )
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return x

cnn = CNN()
print(cnn)

it = iter(train_loader)
X_batch, y_batch = next(it)
print(cnn.forward(X_batch).shape)

Using batch size 2 i am expecting data shape produced by model is (2,245). But it is producing data of shape (2592, 245)

Output:
after self.conv3 you have tensors of shape [2, 64, 108, 108] which produces [2592, 576] after reshape. So this is where 2592 comes from.
Change the lines:
"self.fc1 = nn.Linear(3*3*64, 256)"
and
"x = x.view(-1,3*3*64)"
so that they use proper image size after the layers.

below is the fixed code:

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 32, kernel_size=5)
        self.conv3 = nn.Conv2d(32,64, kernel_size=5)
        self.fc1 = nn.Linear(108*108*64, 256)
        self.fc2 = nn.Linear(256, 245)

    def forward(self, x):
        print (x.shape)
        x = F.relu(self.conv1(x))
        print (x.shape)
        #x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(F.max_pool2d(self.conv2(x), 2))
        print (x.shape)
        x = F.dropout(x, p=0.5, training=self.training)
        print (x.shape)
        x = F.relu(F.max_pool2d(self.conv3(x),2))
        print (x.shape)
        x = F.dropout(x, p=0.5, training=self.training)
        print (x.shape)
        x = x.view(-1,108*108*64 )
        print (x.shape)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return x

cnn = CNN()
print(cnn)

# X_batch, y_batch = next(it)
print(cnn.forward(X_batch).shape)


Instruction: Constructing Key:Value pair from list comprehension in Python

Input:
I am trying to extend some code.

What works:

pretrained_dict = {k: v for k, v pretrained_dict.items if k in model_dict}

However, if I extend it to:

pretrained_dict = {k: v if k in model_dict else k1:v1 for k, v, k1, v1 in zip(pretrained_dict.items(), model_dict.items()) }

The code fails, If I put the else at the end it still fails:

 pretrained_dict = {k: v if k in model_dict for k, v, k1, v1 in zip(pretrained_dict.items(), model_dict.items()) else k1:v1}
                                                 ^
SyntaxError: invalid syntax

How can I construct the key value pair using an if else condition over two lists?

Output:
You can use a ChainMap to achieve what you want without having to use comprehensions at all

from collections import ChainMap
pretrained_dict = ChainMap(pretrained_dict, model_dict)

This returns a dictionary-like object that will lookup keys in pretrained_dict first and if it is not present then lookup the key in model_dict


Instruction: transforms.Normalize() between 0 and 1 when using Lab

Input:
which mean, std should I use when I want to normalize a tensor to a range of 0 to 1? But I work with images with 2 channels (a, b channel -> -128 to 127) only instead of 3 channels. Thus, the usual mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] will not do the job.
transform = transforms.Compose([
   transforms.ToTensor(),
   transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
])

This leads to this error message:

tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
RuntimeError: The size of tensor a (2) must match the size of tensor b
(3) at non-singleton dimension 0

Output:
As you can see, PyTorch complains about the Tensor size, since you lack a channel.

Additionally, the "usual" mean and std values are computed on ImageNet dataset, and are useful if the statistics of your data match the ones of that dataset.

As you work with two channels only, I assume that your domain might be fairly different from 3-channels natural images. In that case I would simply use 0.5 for both mean and std, such that the minimum value 0 will be converted to (0 - 0.5) / 0.5 = -1 and the maximum value of 1 to (1 - 0.5) / 0.5 = 1.

transform = transforms.Compose([
   transforms.ToTensor(),
   transforms.Normalize(mean=[0.5, 0.5],
                        std=[0.5, 0.5])
])

Edit: I would recommend zero-centering of the input.

However, if for some reason you must have it in range [0, 1], calling only ToTensor() would suffice.

In this case, a word of caution. I think ToTensor() assumes your input to lie in range [0, 255] prior to the transform, so it basically divides it by 255. If that is not the case in your domain (e.g. your input is always in range [1, 50] for some reason) I would simply create a custom transform to divide for the actual upper bound for your data.


Instruction: How to create properly shaped input using Dataloader?

Input:
I am trying to input an image and a vector as input to the model. The image has the correct shape of 4d, but the vector that I input doesn't have such shape. The image size is 424x512 while the vector is of shape (18,). After using dataloader, I get batches of shape (50x1x424x512) and (50x18). Model gives error as it needs the vector shape to be 4d too. How do I do that?
Here is my code :

def loadTrainingData_B(args):
    fdm = []
    tdm = []
    parameters = []
    for i in image_files[:4]:
        try:
            false_dm = np.fromfile(join(ref, i), dtype=np.int32)
            false_dm = Image.fromarray(false_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            fdm.append(false_dm)
            true_dm = np.fromfile(join(ref, i), dtype=np.int32)
            true_dm = Image.fromarray(true_dm.reshape((424, 512, 9)).astype(np.uint8)[:,:,1])
            tdm.append(true_dm)
            pos = param_filenames.index(i)
            param = np.array(params[pos, 1:])
            param = np.where(param == '-point-light-source', 1, param).astype(np.float64)
            parameters.append(param)
        except:
            print('[!] File {} not found'.format(i))
    return (fdm, parameters, tdm)

class Flat_ModelB(Dataset):
    def __init__(self, args, train=True, transform=None):
        self.args = args
        if train == True:
            self.fdm, self.parameters, self.tdm = loadTrainingData_B(self.args)
        else:
            self.fdm, self.parameters, self.tdm = loadTestData_B(self.args)
        self.data_size = len(self.parameters)
        self.transform = transforms.Compose([transforms.ToTensor()])

    def __getitem__(self, index):
        return (self.transform(self.fdm[index]).double(), torch.from_numpy(self.parameters[index]).double(), self.transform(self.tdm[index]).double())

    def __len__(self):
        return self.data_size

The error I get is :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 2-dimensional input of size [50, 18] instead

Here is the model :

class Model_B(nn.Module):
    def __init__(self, config):
        super(Model_B, self).__init__()
        self.config = config
        # CNN layers for fdm
        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer2 = nn.Sequential(
            nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer3 = nn.Sequential(
            nn.Conv2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer4 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=32, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(32))
        self.layer5 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=32, out_channels=16, kernel_size=5, stride=2, padding=2,output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(16))
        self.layer6 = nn.Sequential(
            nn.ConvTranspose2d(in_channels=16, out_channels=1, kernel_size=5, stride=2, padding=2, output_padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(1))
        # CNN layer for parameters
        self.param_layer1 = nn.Sequential(
            nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5, stride=2, padding=2),
            nn.ReLU(),
            nn.BatchNorm2d(32))

    def forward(self, x, y):
        out = self.layer1(x)
        out_param = self.param_layer1(y)
        print("LayerParam 1 Output Shape : {}".format(out_param.shape))
        print("Layer 1 Output Shape : {}".format(out.shape))
        out = self.layer2(out)
        print("Layer 2 Output Shape : {}".format(out.shape))
        out = self.layer3(out)
        # out = torch.cat((out, out_param), dim=2)
        print("Layer 3 Output Shape : {}".format(out.shape))
        out = self.layer4(out)
        print("Layer 4 Output Shape : {}".format(out.shape))
        out = self.layer5(out)
        print("Layer 5 Output Shape : {}".format(out.shape))
        out = self.layer6(out)
        print("Layer 6 Output Shape : {}".format(out.shape))
        return out

and the method by which I access the data :

for batch_idx, (fdm, parameters) in enumerate(self.data):
            if self.config.gpu:
                fdm = fdm.to(device)
                parameters = parameters.to(device)
                print('shape of parameters for model a : {}'.format(parameters.shape))

            output = self.model(fdm)
            loss = self.criterion(output, parameters)

Edit :
I think my code is incorrect as I am trying to apply convolutions over a vector of (18). I tried to copy the vector and make it (18x64) and then input it. It still doesnt work and gives this output :

RuntimeError: Expected 4-dimensional input for 4-dimensional weight 32 1 5 5, but got 3-dimensional input of size [4, 18, 64] instead

I am not sure how to concatenate an 18 length vector to the output of layer 3, if I can't do any of these things.

Output:
Looks like you are training an autoencoder model and want to parameterize it with some additional vector input in the bottleneck layer. If you want to perform some transformations on it then you have to decide whether if you need any spatial dependencies. Given the constant input size (N, 1, 424, 512), the output of layer3 will have a shape (N, 32, 53, 64). You have a lot of options, depending on you desired model performance:

Use a nn.Linear with activations to transform the parameter vector. Then you might add extra spatial dimensions and repeat this vector in all spatial locations:

img = torch.rand((1, 1, 424, 512))
vec = torch.rand(1, 19)

layer3_out = model(img)
N, C, H, W = layer3_out.shape

param_encoder = nn.Sequential(nn.Linear(19, 30), nn.ReLU(), nn.Linear(30, 10))
param = param_encoder(vec)
param = param.unsqueeze(-1).unsqueeze(-1).expand(N, -1, H, W)
encoding = torch.cat([param, layer3_out], dim=1)

Use transposed convolutions to upsample your parameter vector to the size of layer3 output. But that would be harder to implement as you have to calculate exact output shape to fit with  (N, 32, 53, 64)
Transform input vector with MLP using nn.Linear to the size 2x of the channels in layer3 output. Then use so called Feature-wise transformations to scale and shift feature maps from layer3.

I would recomend to start with the first option since this is the simplest one to implement and then try others.


Instruction: Why is the variable `input` often marked special in Python syntax markers?

Input:
In many syntax highlighters (e.g. jupyter notebooks) the variable input is highlighted in python. Is there a reason for this? I've avoided using input as a variablename because I thought it did something special internally.

I'm asking because pytorch often uses "input" as a variable in their tutorials.

Output:
input is a function.
This will happen as well with for example list.

It is recommended not to overwrite the build-in functions with own variables.


Instruction: How to properly implement 1D CNN for numerical data in PyTorch?

Input:
I have a 500x2000 matrix, where each row represents an individual and each column is a measurement of some particular quality about that individual. I'm using a batch size of 64, so the input for each cycle of the network is actually a 64x2000 matrix. I'm trying to build a CNN in PyTorch to classify individuals given a set of these measurements. However, I've stumbled on the parameters for the convolutional layer.

Below is my current definition for a simple convolutional neural network.

class CNNnet(nn.Module)
    def __init__(self):
        self.conv1 = nn.Conv1d(2000, 200, (1,2), stride=10)
        self.pool = nn.MaxPool1d(kernel_size = (1, 2), stride = 2)

        self.fc1 = nn.Linear(64, 30)
        self.fc2 = nn.Linear(30, 7)

    def forward(self, x):
        x = x.view(64, 2000, 1)
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

Attempting to train this model produces the following error:

  "RuntimeError: Expected 4-dimensional input for 4-dimensional weight
  200 2000 1 2, but got 3-dimensional input of size [64, 2000, 1]
  instead".

I'm confused on why it's expecting a 4D 200x2000x1x2 matrix (shouldn't the number of output channels be irrelevant to the input? And why is there a 2 at the end?).

My question is what would be the proper syntax or approach for writing a CNN (specifically the convolutional layer) when dealing with 1D data. Any help is greatly appreciated.

Output:
So the kernel size in the 1 dimensional case is simply a vector. So if you’ll want a kernel of size ‘1X2’ you need to specify the ‘2’
In the 2 dimensional case 2 will mean a ‘2X2’ kernel size.

You gave a tuple of 2 values so you use 2 kernel types each will create its own channel


Instruction: Where is the source code of pytorch conv2d?

Input:
Where do I find the source code of the pytorch function conv2d?

It should be in torch.nn.functional but I only find _add_docstr lines,
if i search for conv2d. I looked here:

https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py

Update:
It's is not my typing, I do mean the function.
Conv2d class uses conv2d function from the nn.functional

Here:

https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py

On line 338:

return F.conv2d(F.pad(input, expanded_padding, mode='circular')

F is how they import functional

So then I went there, but I don't find the code.

Output: The functional code is all implemented in C++. The entry point into the C++ pytorch code for conv2d is here.


Instruction: What's different between convolution layer in `Torch`(i.e `nn.SpatialConvolution`) and convolution layer in `Pytorch`(i.e `torch.nn.Conv2d`)

Input:
I would like to know the difference between convolution layer in Torch(i.e nn.SpatialConvolution) and convolution layer in Pytorch(i.e torch.nn.Conv2d)

In Torch's Docs, I found the output shape of SpatialConvolution

It says "If the input image is a 3D tensor nInputPlane x height x width, the output image size will be nOutputPlane x oheight x owidth where

owidth  = floor((width  + 2*padW - kW) / dW + 1)
oheight = floor((height + 2*padH - kH) / dH + 1)

"

which is different from torch.nn.Conv2d's in Pytorch Docs.

Does it mean they are different operation?

Output:
Yes, they are different as torch does not have dilation parameter (for dilation explanation see here, basically the kernel has "spaces" between each kernel element width and height wise and this is what slides over the image).

Except for dilation both equations are the same (set dilation to one in pytorch's version and it's equal).

If you want to use dilation in torch there is a separate class for that called nn.SpatialDilatedConvolution.


Instruction: How to avoid "CUDA out of memory" in PyTorch

Input:
I think it's a pretty common message for PyTorch users with low GPU memory:
RuntimeError: CUDA out of memory. Tried to allocate  MiB (GPU ;  GiB total capacity;  GiB already allocated;  MiB free;  cached)

I tried to process an image by loading each layer to GPU and then loading it back:
for m in self.children():
    m.cuda()
    x = m(x)
    m.cpu()
    torch.cuda.empty_cache()

But it doesn't seem to be very effective. I'm wondering is there any tips and tricks to train large deep learning models while using little GPU memory.

Output:
Although
import torch
torch.cuda.empty_cache()

provides a good alternative for clearing the occupied cuda memory and we can also manually clear the not in use variables by using,
import gc
del variables
gc.collect()

But still after using these commands, the error might appear again because pytorch doesn't actually clears the memory instead clears the reference to the memory occupied by the variables.
So reducing the batch_size after restarting the kernel and finding the optimum batch_size is the best possible option (but sometimes not a very feasible one).
Another way to get a deeper insight into the alloaction of memory in gpu is to use:
torch.cuda.memory_summary(device=None, abbreviated=False)

wherein, both the arguments are optional. This gives a readable summary of memory allocation and allows you to figure the reason of CUDA running out of memory and restart the kernel to avoid the error from happening again (Just like I did in my case).
Passing the data iteratively might help but changing the size of layers of your network or breaking them down would also prove effective (as sometimes the model also occupies a significant memory for example, while doing transfer learning).


Instruction: Simple way to evaluate Pytorch torchvision on a single image

Input:
I have a pre-trained model on Pytorch v1.3, torchvision v0.4.2  as the following:

import PIL, torch, torchvision
# Load and normalize the image
img_file = "./robot_image.jpg"
img = PIL.Image.open(img_file)
img = torchvision.transforms.ToTensor()((img))
img = 0.5 + 0.5 * (img - img.mean()) / img.std()

# Load a pre-trained network and compute its prediction
alexnet = torchvision.models.alexnet(pretrained=True)

I want to test this single image, but I get an error:

alexnet(img)
RuntimeError: Expected 4-dimensional input for 4-dimensional weight 64 3 11 11, but got 3-dimensional input of size [3, 741, 435] instead

what is the most simple and idiomatic way of getting the model to evaluate a single data point?

Output:
AlexNet is expecting a 4-dimensional tensor of size (batch_size x channels x height x width). You are providing a 3-dimensional tensor.

To change your tensor to size (1, 3, 741, 435) simply add the line:

img = img.unsqueeze(0)

You will also need to downsample your image as AlexNet expects inputs of height and width 224x224.


Instruction: tf.cast equivalent in pytorch?

Input:
I am new to PyTorch. TensorFlow has an API tf.cast() and tf.shape(). the tf.cast has specific purpose in TensorFlow, is there anything equivalent in torch?
i have tensor x= tensor(shape(128,64,32,32)): tf.shape(x) create tensor of dimension 1 x.shape create the true dimension. i need to use tf.shape(x) in torch.

tf.cast has a different role than just changing tensor dtype in torch.

did anyone have equivalent API in torch/PyTorch.

Output:
Check out the PyTorch Documentation

As they mentioned:

print(x.dtype) # Prints "torch.int64", currently 64-bit integer type
x = x.type(torch.FloatTensor)
print(x.dtype) # Prints "torch.float32", now 32-bit float
print(x.float()) # Still "torch.float32"
print(x.type(torch.DoubleTensor)) # Prints "tensor([0., 1., 2., 3.], dtype=torch.float64)"
print(x.type(torch.LongTensor)) # Cast back to int-64, prints "tensor([0, 1, 2, 3])"


Instruction: How to modify path where Torch Hub models are downloaded

Input:
When I download models through Torch Hub, models are automatically downloaded in /home/me/.cache/torch.

How can I modify this behavior ?

Output:
From official documentation, there is several ways to modify this path.
In priority order :

Calling hub.set_dir()
$TORCH_HOME/hub, if environment variable TORCH_HOME is set.
$XDG_CACHE_HOME/torch/hub, if environment variable XDG_CACHE_HOME is set.
~/.cache/torch/hub

So I just had to do :

export TORCH_HUB=/my/path/

Edit

TORCH_HUB appear to be deprecated, use TORCH_HOME instead


Instruction: How do you invert a tensor of boolean values in Pytorch?

Input:
With NumPy, you can do it with np.invert(array), but there's no invert function in Pytorch. Let's say I have a 2D tensor of boolean values:
import torch

ts = torch.rand((10, 4)) < .5

tensor([[ True,  True, False,  True],
        [ True,  True,  True,  True],
        [ True, False,  True,  True],
        [False,  True,  True, False],
        [False,  True,  True,  True],
        [ True,  True,  True,  True],
        [ True, False,  True,  True],
        [False,  True, False,  True],
        [ True,  True, False,  True],
        [False, False,  True, False]])

How do I transform the False into True and vice versa?

Output:
Literally just use the tilde to transform all True into False and vice versa.

ts = ~ts


Instruction: Custom conv2d operation Pytorch

Input:
I have tried a custom Conv2d function which has to work similar to nn.Conv2d but the multiplication and addition used inside nn.Conv2d are replaced with mymult(num1,num2) and myadd(num1,num2).

As per insight from very helpful forums 1,2 what i can do is try unfolding it and then do matrix multiplication. That @ part given in the code below can be done using loops with mymult() and myadd() as i believe this @ is doing matmul.

def convcheck():
    torch.manual_seed(123)
    batch_size = 2
    channels = 2

    h, w = 2, 2
    image = torch.randn(batch_size, channels, h, w) # input image
    out_channels = 3
    kh, kw = 1, 1# kernel size
    dh, dw = 1, 1 # stride
    size = int((h-kh+2*0)/dh+1)    #include padding in place of zero

    conv = nn.Conv2d(in_channels=channels, out_channels=out_channels, kernel_size=kw, padding=0,stride=dh ,bias=False)

    out = conv (image)
    #print('out', out)
    #print('out.size()', out.size())
    #print('')
    filt = conv.weight.data

    imageunfold = F.unfold(image,kernel_size=kh,padding=0,stride=dh)

    print("Unfolded image","\n",imageunfold,"\n",imageunfold.shape)
    kernels_flat = filt.view(out_channels,-1)
    print("Kernel Flat=","\n",kernels_flat,"\n",kernels_flat.shape)
    res = kernels_flat @ imageunfold        # I have to replace this operation with mymult() and myadd()
    print(res,"\n",res.shape)
    #print(res.size(2),"\n",res.shape)
    res = res.view(-1, out_channels, size, size)
    #print("Same answer as buitlin function",res)

res = kernels_flat @ imageunfold can be replaced with this. although there can be some other efficient implementation which i am looking to get help for.

     for m_batch in range(len(imageunfold)):
        #iterate through rows of X
        # iterate through columns of Y
        for j in range(imageunfold.size(2)):
            # iterate through rows of Y
            for k in range(imageunfold.size(1)):
                #print(result[m_batch][i][j]," +=",   kernels_flat[i][k], "*", imageunfold[m_batch][k][j])
                result[m_batch][i][j] +=   kernels_flat[i][k] * imageunfold[m_batch][k][j]

Can someone please help me vectorize these three loops for faster execution.

Output:
The problem was with the dimesions as kernels_flat[dim0_1,dim1_1] and imageunfold[batch,dim0_2,dim1_2] the resultant should have [batch,dim0_1,dim1_2]
res = kernels_flat @ imageunfold can be replaced with this. although there can be some other efficient implementation.
     for m_batch in range(len(imageunfold)):
            #iterate through rows of X
            # iterate through columns of Y
            for j in range(imageunfold.size(2)):
                # iterate through rows of Y
                for k in range(imageunfold.size(1)):
                    #print(result[m_batch][i][j]," +=",   kernels_flat[i][k], "*", imageunfold[m_batch][k][j])
                    result[m_batch][i][j] +=   kernels_flat[i][k] * imageunfold[m_batch][k][j]


Instruction: CNN to first make feature key-points prediction then classify image on the base of these keypoints using Pytorch or TensorFlow

Input:
I have already trained an Image Classifier using MobileNet in Pytorch to classify between close eyes image and open eyes image and also deployed it to mobile using Tensorflow

But problem is that the dataset is not big enough and it also doesn't work when the face object is far away or zoomed out.
I want to classify face with predefined key points like following

I want to make a CNN to first make feature key-points prediction and then classify image on the base of these keypoints.

Please guide me to any research paper or guide to predict feature keypoints using CNN and classify keypoints to two classes using Deep Learning techniques. The more Deep Learning used is better

I have already read about unsupervised machine learning but it is not working for me. I want to used deeplearning and pytorch or tensorflow

Output: It seems to me you have enough data. The key is preprocessing. I'll suggest to use MTCNN (implementations: one, two, three) for lightweight face and eyes detection, crop eyes and pass them through your net. Of course you should learn on cropped eyes (not whole images). You can get more precise eyes keypoints by libs like OpenPose, FAN or Seeta.