Instruction: how to use custom python object in torchscript

Input:
I am ready to convert a pytorch module to ScriptModule and then load it in c++，but I am blocked by this error This attribute exists on the Python module, but we failed to convert Python type: 'Vocab' to a TorchScript type, the Vocab is a python object I define.
the demo code is here:
import torch
class Vocab(object):
    def __init__(self, name):
        self.name = name

    def show(self):
        print("dict:" + self.name)

class Model(torch.nn.Module):
    def __init__(self, ):
        super(Model, self).__init__()
        self.layers = torch.nn.Linear(2, 3)
        self.encoder = 4
        self.vocab = Vocab("vocab")

    def forward(self, x):
        name = self.vocab.name
        print("forward show encoder:" + str(self.encoder))
        print("vocab:" + name)
        enc_hidden = []
        step = len(x) // 2
        for i in range(step):
            enc_hidden.append((x[2*i] + x[2*i + 1])/2)
        enc_hidden = torch.stack(enc_hidden, 0)
        enc_hidden = self.__show(enc_hidden)
        return self.layers(enc_hidden)

    @torch.jit.export
    def __show(self, x):
        return x + 1

model = Model()
data = torch.randn(10, 2)
script_model = torch.jit.script(model)
print(script_model)
r1 = model(data)
print(r1)

the error msg:
Traceback (most recent call last):
  File "/mnt/d/python_projects/pytorch_deploy/model4.py", line 47, in <module>
    script_model = torch.jit.script(model)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 1261, in script
    return torch.jit._recursive.create_script_module(obj, torch.jit._recursive.infer_methods_to_compile)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 305, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 361, in create_script_module_impl
    create_methods_from_stubs(concrete_type, stubs)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 279, in create_methods_from_stubs
    concrete_type._create_methods(defs, rcbs, defaults)
RuntimeError:
  Module 'Model' has no attribute 'vocab' (This attribute exists on the Python module, but we failed to convert Python type: 'Vocab' to a TorchScript type.):
  File "/mnt/d/python_projects/pytorch_deploy/model4.py", line 26
  def forward(self, x):
    name = self.vocab.name
           ~~~~~~~~~~ <--- HERE
    print("forward show encoder:" + str(self.encoder))
    print("vocab:" + name)

so how can I use my own python object in torchscript?

Output:
You have to annotate your Vocab with torchscript.jit like this:

@torch.jit.script
class Vocab(object):
    def __init__(self, name: str):
        self.name = name

    def show(self):
        print("dict:" + self.name)

Also note specification name: str as it's also needed for torchscript to infer it's type (PyTorch supports >=Python3.6 type annotations, you could use a comment as well, but it's way less clear).

Please see Torchscript classes and Default Types and other related torchscript info over there.


Instruction: Pytorch BatchNorm2d RuntimeError: running_mean should contain 64 elements not 0

Input:
I'm using Octave Convolutions and have set up a BatchNorm2d adaptation that for some reasen is giving me

RuntimeError: running_mean should contain 64 elements not 0

I've set up some debugging prints to check what was wrong with my Tensors' dimensions, but was unable to find it.
Here is my class:

class _BatchNorm2d(nn.Module):
  def __init__(self, num_features, alpha_in=0, alpha_out=0, eps=1e-5, momentum=0.1, affine=True,
               track_running_stats=True):
    super(_BatchNorm2d, self).__init__()
    hf_ch = int(num_features * (1 - alpha_out))
    lf_ch = num_features - hf_ch
    self.bnh = nn.BatchNorm2d(hf_ch)
    self.bnl = nn.BatchNorm2d(lf_ch)
  def forward(self, x):
    if isinstance(x, tuple):
        hf, lf = x
        print("IN ON BN: ",lf.shape if lf is not None else None) #DEBUGGING PRINT
        print(self.bnl)  #DEBUGGING PRINT
        hf = self.bnh(hf) if type(hf) == torch.Tensor else hf
        lf = self.bnh(lf) if type(lf) == torch.Tensor else lf #THIS IS THE LINE ACCUSING THE ERROR
        print("ENDED BN")
        return hf, lf
    else:
        return self.bnh(x)

Here is the printing error:

IN ON BN:  torch.Size([32, 64, 3, 3])
BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

It seems to me the function should have worked, since x has 64 channels and bn expects 64 channels.

EDIT:
May also be important to mention the error only happens on alpha value of 1. However, I do not understand it, since the volumes are still the same.

Output:
Solved. It was a typo on the call for the low frequency BN.

    hf = self.bnh(hf) if type(hf) == torch.Tensor else hf
    lf = self.bnh(lf) if type(lf) == torch.Tensor else lf

Should have been

    hf = self.bnh(hf) if type(hf) == torch.Tensor else hf
    lf = self.bnl(lf) if type(lf) == torch.Tensor else lf


Instruction: How to use scripting to convert pytorch transformer?

Input:
I am trying to compile pytorch transformer to run it in C++:

from torch.nn import TransformerEncoder, TransformerEncoderLayer
encoder_layers = TransformerEncoderLayer(1000, 8, 512, 0.1)
transf = TransformerEncoder(encoder_layers, 6)
sm = torch.jit.script(transf)

But I am getting an error:

RuntimeError:  Expected a default value of type Tensor on parameter
"src_mask":   File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python36_64\lib\site-packages\torch\nn\modules\transformer.py",
line 271
    def forward(self, src, src_mask=None, src_key_padding_mask=None):
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~... <--- HERE
        r"""Pass the input through the encoder layer.

It looks like something wrong with pytorch transformer module.

Is there any way to run pytorch transformer in C++ ?

Output:
You need to upgrade to PyTorch 1.5.0, older versions did not support converting Transformers to TorchScript (JIT) modules.

pip install torch===1.5.0 -f https://download.pytorch.org/whl/torch_stable.html

In 1.5.0 you will see some warnings about the parameters being declared as constants, such as:

UserWarning: 'q_proj_weight' was found in ScriptModule constants,  but it is a non-constant parameter. Consider removing it.

These can be safely ignored.


Instruction: Combine Pytorch ImageFolder dataset with custom Pytorch dataset

Input:
I am loading data from multiple datasets using Pytorch. I have some images stored in properly labeled folders (e.g., \0 and \1), and in those cases I can use torch.utils.data.ConcatDataset after loading the lists, for example (where trans is a set of pre-defined Pytorch transformations):

l = []
l.append(datasets.ImageFolder(file_path, trans))
l.append(datasets.ImageFolder(file_path2, trans))
image_datasets = torch.utils.data.ConcatDataset(l)

img_datasets = dict()
img_datasets['train'], img_datasets['val'] = torch.utils.data.random_split(image_datasets, (round(0.8*len(image_datasets)), round(0.2*len(image_datasets)) ))

However, I am also loading images from other disparate locations using a csv file. The process here looks like this:

class MyData(Dataset):
  def __init__(self, df):
      self.df = df

  def __len__(self):
      return self.df.shape[0]

  def __getitem__(self, index):
      image = trans(PIL.Image.open(self.df.file_path[index]))
      label = self.df.label[index]

      return image, label

df = pd.read_csv(image_file_paths), names=["file_path", "label"])
mydata = MyData(df)

my_datasets = dict()
my_datasets['train'], my_datasets['val'] = torch.utils.data.random_split(mydata, (round(0.8*len(mydata)), round(0.2*len(mydata))))

So I'd like to be able to combine these datasets into a single dataloader. Any ideas for how I should go about doing this? Thanks!

Output:
Found the solution; just need to use multiple passes of ConcatDataset:

l = []
l.append(datasets.ImageFolder(file_path, trans))
l.append(datasets.ImageFolder(file_path2, trans))
image_datasets = torch.utils.data.ConcatDataset(l)

df = pd.read_csv(image_file_paths), names=["file_path", "label"])
mydata = MyData(df)

image_datasets = torch.utils.data.ConcatDataset([image_datasets, mydata])

img_datasets = dict()
img_datasets['train'], img_datasets['val'] = torch.utils.data.random_split(image_datasets, (round(0.8*len(image_datasets)), round(0.2*len(image_datasets))))

Good to go from there.


Instruction: PyTorch: Loading word vectors into Field vocabulary vs. Embedding layer

Input:
I'm coming from Keras to PyTorch. I would like to create a PyTorch Embedding layer (a matrix of size V x D, where V is over vocabulary word indices and D is the embedding vector dimension) with GloVe vectors but am confused by the needed steps.

In Keras, you can load the GloVe vectors by having the Embedding layer constructor take a weights argument:

# Keras code.
embedding_layer = Embedding(..., weights=[embedding_matrix])

When looking at PyTorch and the TorchText library, I see that the embeddings should be loaded twice, once in a Field and then again in an Embedding layer. Here is sample code that I found:

# PyTorch code.

# Create a field for text and build a vocabulary with 'glove.6B.100d'
# pretrained embeddings.
TEXT = data.Field(tokenize = 'spacy', include_lengths = True)

TEXT.build_vocab(train_data, vectors='glove.6B.100d')

# Build an RNN model with an Embedding layer.
class RNN(nn.Module):
    def __init__(self, ...):

        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim)

        ...

# Initialize the embedding layer with the Glove embeddings from the
# vocabulary. Why are two steps needed???
model = RNN(...)
pretrained_embeddings = TEXT.vocab.vectors
model.embedding.weight.data.copy_(pretrained_embeddings)

Specifically:

Why are the GloVe embeddings loaded in a Field in addition to the Embedding?
I thought the Field function build_vocab() just builds its vocabulary from the training data. How are the GloVe embeddings involved here during this step?

Here are other StackOverflow questions that did not answer my questions:

PyTorch / Gensim - How to load pre-trained word embeddings

Embedding in pytorch

PyTorch LSTM - using word embeddings instead of nn.Embedding()

Thanks for any help.

Output:
When torchtext builds the vocabulary, it aligns the the token indices with the embedding. If your vocabulary doesn't have the same size and ordering as the pre-trained embeddings, the indices wouldn't be guaranteed to match, therefore you might look up incorrect embeddings. build_vocab() creates the vocabulary for your dataset with the corresponding embeddings and discards the rest of the embeddings, because those are unused.

The GloVe-6B embeddings includes a vocabulary of size 400K. For example the IMDB dataset only uses about 120K of these, the other 280K are unused.

import torch
from torchtext import data, datasets, vocab

TEXT = data.Field(tokenize='spacy', include_lengths=True)
LABEL = data.LabelField()

train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)
TEXT.build_vocab(train_data, vectors='glove.6B.100d')

TEXT.vocab.vectors.size() # => torch.Size([121417, 100])

# For comparison the full GloVe
glove = vocab.GloVe(name="6B", dim=100)
glove.vectors.size() # => torch.Size([400000, 100])

# Embedding of the first token is not the same
torch.equal(TEXT.vocab.vectors[0], glove.vectors[0]) # => False

# Index of the word "the"
TEXT.vocab.stoi["the"] # => 2
glove.stoi["the"] # => 0

# Same embedding when using the respective index of the same word
torch.equal(TEXT.vocab.vectors[2], glove.vectors[0]) # => True

After having built the vocabulary with its embeddings, the input sequences will be given in the tokenised version where each token is represented by its index. In the model you want to use the embedding of these, so you need to create the embedding layer, but with the embeddings of your vocabulary. The easiest and recommended way is nn.Embedding.from_pretrained, which is essentially the same as the Keras version.

embedding_layer = nn.Embedding.from_pretrained(TEXT.vocab.vectors)

# Or if you want to make it trainable
trainable_embedding_layer = nn.Embedding.from_pretrained(TEXT.vocab.vectors, freeze=False)

You didn't mention how the embedding_matrix is created in the Keras version, nor how the vocabulary is built such that it can be used with the embedding_matrix. If you do that by hand (or with any other utility), you don't need torchtext at all, and you can initialise the embeddings just like in Keras. torchtext is purely for convenience for common data related tasks.


Instruction: Resnet18 first layer output dimensions

Input: I am looking at the model implementation in PyTorch. The 1st layer is a convolutional layer with filter size = 7, stride = 2, pad = 3. The standard input size to the network is 224x224x3. Based on these numbers, the output dimensions are (224 + 3*2 - 7)/2 + 1, which is not an integer. Does the original implementation contain non-integer dimensions? I see that the network has adaptive pooling before the FC layer, so the variable input dimensions aren't a problem (I tested this by varying the input size). Am I doing something wrong, or why would the authors choose a non-integer dimension while designing the ResNet?

Output:
The dimensions always have to be integers. From nn.Conv2d - Shape:

The brackets that are only closed towards the bottom denote the floor operation (round down). The calculation becomes:

import math

math.floor((224 + 3*2 - 7)/2 + 1) # => 112

# Or using the integer division (two slashes //)
(224 + 3*2 - 7) // 2 + 1 # => 112

Using an integer division has the same effect, since that always rounds it down to the nearest integer.


Instruction: Error in Calculating neural network Test Accuracy

Input:
I tried to train my neural network, and then evaluate it's testing accuracy. I am using the code at the bottom of this post to train. The fact is that for other neural networks, I can evaluate the testing accuracy with my code without issue. However, for this neural network (which I constructed correctly according to the description of the neural network paper), I can't evaluate the testing accuracy properly and its giving me the traceback below. So maybe something's wrong in my forward pass?

Here is the training and testing code:

//imports including import deepnet.py
cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = deepnet.cifar10_deep()
#model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 4500, 5000]

# parametric plot (i.e., don't train the network if set to True)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        #model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0

            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/DeepNetC2BatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl])
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts)
                average_loss_over_epoch += loss_fn.data.numpy() / (X_train.shape[0] // batch_size)
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

grid_size = 18 #How many points of interpolation between [0, 5000]
data_for_plotting = np.zeros((grid_size, 3)) #Uncomment this line if running entire code from scratch
sharpnesses1eNeg3 = []
sharpnesses5eNeg4 = []
#data_for_plotting = np.load("DeepNetCIFAR10-intermediate-values.npy") #Uncomment this line to use an existing NumPy array
print(data_for_plotting)
i = 0

# Fill in test accuracy values for `grid_size' points in the interpolation
for batch_size in batch_range:
    mydict = {}
    batchmodel = torch.load("./models/DeepNetC2BatchSize" + str(batch_size) + ".pth")
    for key, value in batchmodel.items():
        mydict[key] = value
    model.load_state_dict(mydict)

    j = 0
    for datatype in [(X_train, y_train), (X_test, y_test)]:
        dataX = datatype[0]
        datay = datatype[1]
        for smpl in np.split(np.random.permutation(range(dataX.shape[0])), 10):
            ops = opfun(dataX[smpl])
            tgts = Variable(torch.from_numpy(datay[smpl]).long().squeeze())
            var = F.nll_loss(ops, tgts).data.numpy() / 10
            if j == 1:
                data_for_plotting[i, j-1] += accfun(ops, datay[smpl]) / 10.
        j += 1
    print(data_for_plotting[i])
    np.save('DeepNetCIFAR10-intermediate-values', data_for_plotting)
    i += 1

And the model code is here and includes the forward pass

import torch
import torch.nn as nn
F = nn.functional
__all__ = ['cifar10_deepnet', 'cifar100_deepnet']

class VGG(nn.Module):
    def __init__(self, num_classes=10):
        super(VGG, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(64, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(128, 128, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(128, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(256, 256, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(256, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),

            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.4),
            nn.Conv2d(512, 512, kernel_size=3, padding = 1, bias=False),
            nn.BatchNorm2d(512),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),
        )
        self.classifier = nn.Sequential(
            nn.Linear(512, 512, bias=False),
            nn.Dropout(0.5),
            nn.BatchNorm1d(512),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(512, num_classes)
        )

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 512)
        x = self.classifier(x)
        return F.log_softmax(x)

def cifar10_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 10)
    return VGG(num_classes)

def cifar100_deep(**kwargs):
    num_classes = getattr(kwargs, 'num_classes', 100)
    return VGG(num_classes)

Output:
You are trying to load a state dict that belongs to another model.

The error shows that your model is the class AlexNet.

RunTimeError: Error(s) in loading state_dict for AlexNet:

But the state dict you are trying to load is from the VGG you posted, which doesn't have the same modules as AlexNet.

You need to use the same model whose state dict you saved before.


Instruction: how to solve the values changed when converting a pytorch Variable to numpy?

Input:
I am trying to convert a parameter of resnet34 to numpy , but I found that the values would change after converting, as shown in the figure.
Why would this happen? What can I do to get precise values in numpy format?
enter image description here

( I am trying to get parameter in torch pretrained models and put them in tensorflow 1.x model , since I can’t find a pretrained resnet34 models in tensorflow1 afteing searching for several days. I am afraid this changed of values would affect the accuracy of the model. )

(BTW, is there someway to download tensorflow1.x resnet34 pretrained modles with basic block instead of bottleneck block?
I have searched in github for several days but failed to find one. I hate tensorflow.)

Output:
The values are not changed, they are identical, but PyTorch limits the default output to 4 decimal places (which gets rounded) to make it easier to inspect.

You can change that behaviour with torch.set_printoptions to show more decimal places.

value = torch.tensor(0.0052872747)

print(value) # => tensor(0.0053)

# Show 10 decimal places
torch.set_printoptions(precision=10)

print(value) # => tensor(0.0052872747)


Instruction: How can I zero out duplicate values in each row of a PyTorch tensor?

Input:
I would like to write a function that achieves the behavior described in this question.
That is, I want to zero out duplicate values in each row of a matrix in PyTorch. For example, given a matrix
torch.Tensor(([1, 2, 3, 4, 3, 3, 4],
              [1, 6, 3, 5, 3, 5, 4]])

I would like to get
torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 0, 0, 4]])

or
torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 4, 0, 0]])

According to the linked question, torch.unique() alone is not sufficient. I want to know how to implement this function without a loop.

Output:
x = torch.tensor([
    [1, 2, 3, 4, 3, 3, 4],
    [1, 6, 3, 5, 3, 5, 4]
], dtype=torch.long)

# sorting the rows so that duplicate values appear together
# e.g., first row: [1, 2, 3, 3, 3, 4, 4]
y, indices = x.sort(dim=-1)

# subtracting, so duplicate values will become 0
# e.g., first row: [1, 2, 3, 0, 0, 4, 0]
y[:, 1:] *= ((y[:, 1:] - y[:, :-1]) !=0).long()

# retrieving the original indices of elements
indices = indices.sort(dim=-1)[1]

# re-organizing the rows following original order
# e.g., first row: [1, 2, 3, 4, 0, 0, 0]
result = torch.gather(y, 1, indices)

print(result) # => output

Output

tensor([[1, 2, 3, 4, 0, 0, 0],
        [1, 6, 3, 5, 0, 0, 4]])


Instruction: Extracting labels after applying softmax

Input:
I have a multi class classification neural network. I apply softmax at the end to get probabilities for my classes. However, now I want to pick the maximum probability and get the corresponding label for it. I am able to extract the maximum probability but I'm confused how to get the label based on that. This is what I have:

labels = {'id1':0,'id2':2,'id3':1,'id4':3} ### labels

x_t = F.softmax(z,dim=-1)
#print(x_t)
y = torch.argmax(x_t, dim=1)
print(y.detach())

Can someone tell me how to get labels now. For example

y = tensor([3, 1, 3])

final_label = [id4,id3,id4]

Output:
You can try storing the map for index to label like this:
labels = {'id1':0,'id2':2,'id3':1,'id4':3} ### labels
idx_to_label = {v:k for k,v in labels.items()}

x_t = F.softmax(z,dim=-1)
#print(x_t)
y = torch.argmax(x_t, dim=1)
print(y.detach())
final_label = [idx_to_label[i] for i in y.detach().cpu().numpy()]
print(final_label)

Let me know if it helps.


Instruction: PyTorch path generation with RNN - confusion with input, output, hidden and batch sizes

Input:
I'm new to pytorch, I followed a tutorial on sentence generation with RNN and I'm trying to modify it to generate sequences of positions, however I'm having trouble with defining the correct model parameters such as input_size, output_size, hidden_dim, batch_size.

Background:
I have 596 sequences of x,y positions, each looking like [[x1,y1],[x2,y2],...,[xn,yn]]. Each sequence represents the 2D path of a vehicle. I would like to to train a model that, given a starting point (or a partial sequence), could generate one of these sequences.

-I have padded/truncated the sequences so that they all have length 50, meaning each sequence is an array of shape [50,2]

-I then divided this data into input_seq and target_seq:

input_seq: tensor of torch.Size([596, 49, 2]). contains all the 596 sequences, each without its last position.

target_seq: tensor of torch.Size([596, 49, 2]). contains all the 596 sequences, each without its first position.

The model class:

class Model(nn.Module):
def __init__(self, input_size, output_size, hidden_dim, n_layers):
    super(Model, self).__init__()
    # Defining some parameters
    self.hidden_dim = hidden_dim
    self.n_layers = n_layers
    #Defining the layers
    # RNN Layer
    self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
    # Fully connected layer
    self.fc = nn.Linear(hidden_dim, output_size)

def forward(self, x):
    batch_size = x.size(0)
    # Initializing hidden state for first input using method defined below
    hidden = self.init_hidden(batch_size)
    # Passing in the input and hidden state into the model and obtaining outputs
    out, hidden = self.rnn(x, hidden)
    # Reshaping the outputs such that it can be fit into the fully connected layer
    out = out.contiguous().view(-1, self.hidden_dim)
    out = self.fc(out)
    return out, hidden

def init_hidden(self, batch_size):
    # This method generates the first hidden state of zeros which we'll use in the forward pass
    # We'll send the tensor holding the hidden state to the device we specified earlier as well
    hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
    return hidden

I instantiate the model with the following parameters:

input_size of 2 (an [x,y] position)

output_size of 2 (an [x,y] position)

hidden_dim of 2 (an [x,y] position) (or should this be 50 as in the length of a full sequence?)

model = Model(input_size=2, output_size=2, hidden_dim=2, n_layers=1)
n_epochs = 100
lr=0.01
# Define Loss, Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

# Training Run
for epoch in range(1, n_epochs + 1):
    optimizer.zero_grad() # Clears existing gradients from previous epoch
    output, hidden = model(input_seq)
    loss = criterion(output, target_seq.view(-1).long())
    loss.backward() # Does backpropagation and calculates gradients
    optimizer.step() # Updates the weights accordingly
    if epoch%10 == 0:
        print('Epoch: {}/{}.............'.format(epoch, n_epochs), end=' ')
        print("Loss: {:.4f}".format(loss.item()))

When I run the training loop, it fails with this error:

ValueError                                Traceback (most recent call last)
<ipython-input-9-ad1575e0914b> in <module>
      3     optimizer.zero_grad() # Clears existing gradients from previous epoch
      4     output, hidden = model(input_seq)
----> 5     loss = criterion(output, target_seq.view(-1).long())
      6     loss.backward() # Does backpropagation and calculates gradients
      7     optimizer.step() # Updates the weights accordingly
...

ValueError: Expected input batch_size (29204) to match target batch_size (58408).

I tried modifying input_size, output_size, hidden_dim and batch_size and reshaping the tensors, but the more I try the more confused I get. Could someone point out what I am doing wrong?

Furthermore, since batch size is defined as x.size(0) in Model.forward(self,x), this means I only have a single batch of size 596 right? What would be the correct way to have multiple smaller batches?

Output:
The output has size [batch_size * seq_len, 2] = [29204, 2], and you flatten the target_seq, which  has size [batch_size * seq_len * 2] = [58408]. They don't have the same number of dimensions, while having the same number of total elements, therefore the first dimensions are not identical.

Regardless of the dimension mismatch, nn.CrossEntropyLoss is a categorical loss function, which means it would only predict a class from the output. You don't have any classes, but you are trying to predict coordinates, which are continuous values. For this you need to use a regression loss function, such as nn.MSELoss, which calculates the squared error/distance between the predicted and target coordinates.

criterion = nn.MSELoss()

# .flatten() does the same thing as .view(-1) but is more descriptive
loss = criterion(output.flatten(), target_seq.flatten())

The flattening can be avoided as the loss functions as well as the linear layer can operate on multidimensional inputs, which removes the potential risk of getting lost with the flattening and restoring of the dimensions, and the output is more comprehensible to inspect or use later outside of the training. For the linear layer, only the last dimension of the input needs to match the in_features of nn.Linear, which is hidden_dim in your case.

def forward(self, x):
    batch_size = x.size(0)
    # Initializing hidden state for first input using method defined below
    hidden = self.init_hidden(batch_size)
    # Passing in the input and hidden state into the model and obtaining outputs
    # out size: [batch_size, seq_len, hidden_dim]
    out, hidden = self.rnn(x, hidden)
    # out size: [batch_size, seq_len, output_size]
    out = self.fc(out)
    return out, hidden

Now the output of the model has the same size as the target_seq and you can directly call the loss function without flattening:

loss = criterion(output, target_seq)

  hidden_dim of 2 (an [x,y] position) (or should this be 50 as in the length of a full sequence?)

The hidden_dim is not a pair of [x, y] and is completely unrelated to both the input_size and output_size. It defines the number of hidden features of the RNN, which is kind of its complexity, and bigger sizes potentially have more room to retain essential information, but also require more computations. There is no perfect hidden size and it largely depends on the use case. You can experiment with different sizes, e.g. 100, 256, etc. and see whether that improves your results.

  Furthermore, since batch size is defined as x.size(0) in Model.forward(self,x), this means I only have a single batch of size 596 right? What would be the correct way to have multiple smaller batches?

Yes, you only have a single batch of size 596. If you want to use smaller batches, for example if you cannot fit all of them into a more complex model, you could easily use slices of them, but it would be better to use PyTorch's data utilities: torch.utils.data.TensorDataset to get a dataset, where each sequence of the input has a corresponding target, in combination with torch.utils.data.DataLoader to create batches for you.

from torch.utils.data import DataLoader, TensorDataset

# Match each sequence of the input_seq to the corresponding target_seq.
# e.g. dataset[0] == (input_seq[0], target_seq[0])
dataset = TensorDataset(input_seq, target_seq)

# Randomly shuffle the data and load it in batches of 16
data_loader = DataLoader(dataset, batch_size=16, shuffle=True)

# Process one batch at a time
for input, target in data_loader:
    output, hidden = model(input)
    loss = criterion(output, target)


Instruction: Pytorch, cant' run CNN on GPU. Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Input:
I'm building a simple image recognition convolutional neural network and trying to run this on my GPU but I haven't done something important apparently.

I checked If GPU is available at the beginning and in train, set batches to the device (cuda:0).

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# Checks if GPU is available otherwise uses CPU
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    print("Running on the GPU!")
else:
    device = torch.device("cpu")
    print("Running on the CPU!")

REBUILD_DATA = False

# Data clean up and format
class DogsVsCats():
    IMG_SIZE = 50
    CATS = "PetImages/Cat"
    DOGS = "PetImages/Dog"

    LABELS = {CATS: 0, DOGS: 1}

    training_data = []

    catcount =  0
    dogcount = 0

    def make_training_data(self):
        for label in self.LABELS:
            print(label)
            for f in tqdm(os.listdir(label)):
                try:
                    path = os.path.join(label, f)
                    img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
                    img = cv2.resize(img, (self.IMG_SIZE, self.IMG_SIZE))

                    self.training_data.append([np.array(img), np.eye(2)[self.LABELS[label]] ])

                    if label == self.CATS:
                        self.catcount += 1
                    elif label == self.DOGS:
                        self.dogcount += 1
                except Exception as e:
                    pass

        np.random.shuffle(self.training_data)
        np.save("training_data.npy", self.training_data)

        print("Cats: ", self.catcount)
        print("Dogs: ", self.dogcount)

if REBUILD_DATA:
    dogsvcats = DogsVsCats()
    dogsvcats.make_training_data()

training_data = np.load("training_data.npy", allow_pickle=True)

# print(len("training_data.npy"))

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(50,50).view(-1,1,50,50)

        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512)
        self.fc2 = nn.Linear(512, 2)

    def convs(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2,2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2,2))

        print(x[0].shape)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim = 1)

net = Net().to(device)

optimizer = optim.Adam(net.parameters(), lr = 0.001)
loss_function =  nn.MSELoss()

X = torch.Tensor([i[0] for i in training_data]).view(-1, 50, 50)
X = X/255.0
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1
val_size = int(len(X)*VAL_PCT)

print(val_size)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]

BATCH_SIZE = 100

EPOCHS = 1

def train(net):
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_X), BATCH_SIZE)):
            # print(i, i+BATCH_SIZE)
            batch_X = train_X[i:i+BATCH_SIZE].view(-1,1,50,50).to(device)
            batch_y = train_y[i:i+BATCH_SIZE].to(device)

            net.zero_grad()

            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()

        print(loss)

correct = 0
total = 0

with torch.no_grad():
    for i in tqdm(range(len(test_X))):
        real_class = torch.argmax(test_y[i])
        net_out = net(test_X[i].view(-1, 1, 50, 50))[0]
        predicted_class = torch.argmax(net_out)
        if predicted_class == real_class:
            correct += 1
        total += 1

print("Accuracy: ", round(correct/total,3))

train(net)

Sorry if it's too simple of a question. Thank you in advance!

Output:
You should post the line number of the error, but I'm thinking its from this snipit:

    with torch.no_grad():
        for i in tqdm(range(len(test_X))):
            real_class = torch.argmax(test_y[i])
            net_out = net(test_X[i].view(-1, 1, 50, 50))[0]
            predicted_class = torch.argmax(net_out)
            if predicted_class == real_class:
                correct += 1
            total += 1

You have to put the input to your net must be put to the device, so maybe change the line

     net_out = net(test_X[i].view(-1, 1, 50, 50))[0]

to

     net_out = net(test_X[i].view(-1, 1, 50, 50).to(device)[0]


Instruction: Compute matrix multiplication only for given coordinates

Input:
In PyTorch, I would like to compute

E * A.mm(B)

where E can be a very sparse matrix consisting of 0's and 1's. In other words, I want to compute A.mm(B) and then leave only the certain coordinates. Is there a way to compute such a sparse matrix efficiently? I have full control over matrix representations.

Also, in most cases, E consists only of 1's, so I would like this case also to be handled efficiently.

Output:
You don't need an element-wise multiplication for that, as E is essentially a boolean matrix, that is used as a mask to select the values, where E is 1, and discard the values where E is 0.

C = A.mm(B)

# Ensure that E is a boolean matrix to only keep values where E is True,
# otherwise the 0s and 1s would be treated as indices to select the values.
C = C[E.to(torch.bool)]

If you want to avoid the entire matrix multiplication and only compute the values you would be masking afterwards, you need to manually select the values for A and B that produce the desired values in C.

The matrix multiplication C = AB, where A is an m x n matrix and B  an n x p matrix, produces an m x p matrix C, whose values are obtained by multiplying the i-th row of A with the j-th column of B element-wise and taking the sum of these n products. Formally:

Given E, an m x p matrix, that determines which elements of C are required, the index pairs of the required elements are given as follows:

# Indices of required elements (i.e. indices of non-zero elements of E)
# Separate the tensor of (i, j) pairs, into a pair of tensors,
# containing the indices i and j respectively.
indices_i, indices_j = E.nonzero().unbind(dim=1)

# Select all needed rows of A and the needed columns of B
A = A[indices_i]
B = B[:, indices_j]

# Calculate the values
# B is transposed to change the column vectors to row vectors
# such that the two can be multiplied element-wise.
C = torch.sum(A * B.transpose(0, 1), dim=1)

Is it more efficient to selectively calculate the values you want compared to performing the entire matrix multiplication and then only keep the values you want?

The answer is a resounding No. The matrix multiplication is highly optimised, much more optimised than doing the steps manually with operations that themselves are well optimised. Especially, when E contains mostly 1s, then you're basically re-implementing the matrix multiplication, which is guaranteed to be less efficient. Even for the case where E contains mostly 0s, the matrix multiplication is just faster.

To support my claims, I've timed them. For convenience I did it in IPython, which has the built-in %timeit command.

In [1]: import torch
   ...:
   ...:
   ...: def masked(A, B, E):
   ...:     C = A.mm(B)
   ...:     return C[E]
   ...:
   ...:
   ...: def selective(A, B, E):
   ...:     indices_i, indices_j = E.nonzero().unbind(dim=1)
   ...:     return torch.sum(A[indices_i] * B[:, indices_j].transpose(0, 1), dim=1)
   ...:
   ...:
   ...: A = torch.rand(1200, 1000)
   ...: B = torch.rand(1000, 1100)
   ...: # Only 10% of the elements are 1
   ...: E_mostly_zeros = torch.rand(1200, 1100) < 0.1
   ...: # 90% of the elements are 1
   ...: E_mostly_ones = torch.rand(1200, 1100) < 0.9

In [2]: # All close instead of equal to account for floating point errors
   ...: torch.allclose(masked(A, B, E_mostly_ones), selective(A, B, E_mostly_ones))
Out[2]: True

In [3]: # All close instead of equal to account for floating point errors
   ...: torch.allclose(masked(A, B, E_mostly_zeros), selective(A, B, E_mostly_zeros))
Out[3]: True

In [4]: %timeit masked(A, B, E_mostly_ones)
8.16 ms ± 20.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [5]: %timeit selective(A, B, E_mostly_ones)
2.09 s ± 11.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [6]: %timeit masked(A, B, E_mostly_zeros)
5.73 ms ± 24.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

] In [7]: %timeit selective(A, B, E_mostly_zeros)
266 ms ± 3.36 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The matrix multiplication is staggeringly fast, being over 256x faster when E contains 90% ones (8.16ms vs 2090ms), and over 46x faster when E contains only 10% ones (5.73ms vs 266ms).


Instruction: Bert + Resnet joint learning, pytorch model is empty after instantiation

Input:
I'm writing a simple joint model, which has two branches, one branch is a resnet50 another one is a bert. I concatenate the two outputs and pass that to a simple linear layer with 2 output neurons.

I implemented the following model :

import torch
from torch import nn
import torchvision.models as models
import torch.nn as nn
from collections import OrderedDict
from transformers import BertModel

class BertResNet(nn.Module):
    def __init__(self):
        super(BertResNet, self).__init__()
        # resnet
        resnet50 = models.resnet50(pretrained=True)
        n_inputs = resnet50.fc.in_features
        # compressed embedding space
        classifier = nn.Sequential(OrderedDict([
            ('fc1', nn.Linear(n_inputs, 512))
        ]))

        resnet50.fc = classifier # 512 out resnet

        bert = BertModel.from_pretrained('bert-base-uncased')

        # final classification layer

        classification = nn.Linear(512 + 768, 2)
        #print(resnet50)
        #print(bert)

    def forward(self, img, text):
        res_emb = self.resnet50(img)
        bert_emb = self.bert(text)

        combined = torch.cat(res_emb,
                              bet_emb, dim=1)
        out = self.classification(combined)
        return out

But when I instantiate, I get an empty model:

bert_resnet = BertResNet()

print(bert_resnet)

Out:
BertResNet()

list(bert_resnet.parameters()) also returns []

Output:
You never assigned the models to any attribute of the object of the BertResNet class. There are in temporary variables in the __init__ method, but once that finishes, these variables are discarded. They should be assigned to self:

def __init__(self):
    super(BertResNet, self).__init__()
    # resnet
    self.resnet50 = models.resnet50(pretrained=True)
    n_inputs = self.resnet50.fc.in_features
    # compressed embedding space
    self.classifier = nn.Sequential(OrderedDict([
        ('fc1', nn.Linear(n_inputs, 512))
    ]))

    self.resnet50.fc = classifier # 512 out resnet

    self.bert = BertModel.from_pretrained('bert-base-uncased')

    # final classification layer

    self.classification = nn.Linear(512 + 768, 2)


Instruction: Implement x=T if abs(x)>T as an activation function in pytorch

Input:
I would like to implement the following activation function in pytorch:

x = T if abs(x)>T else x

I could do something close with torch.clamp(min=-T, max=T) but it's not exactly the behavior I want (this would behave the same as above for x>-T but would return -T for x<-T). Is there any torch function that could help me achieve this?

Output:
torch.where does exactly that:

x = torch.where(torch.abs(x) > T, T, x)


Instruction: Tokens to Words mapping in the tokenizer decode step huggingface?

Input:
Is there a way to know the mapping from the tokens back to the original words in the tokenizer.decode() function?
For example:

from transformers.tokenization_roberta import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)

str = "This is a tokenization example"
tokenized = tokenizer.tokenize(str)
## ['this', 'Ġis', 'Ġa', 'Ġtoken', 'ization', 'Ġexample']

encoded = tokenizer.encode_plus(str)
## encoded['input_ids']=[0, 42, 16, 10, 19233, 1938, 1246, 2]

decoded = tokenizer.decode(encoded['input_ids'])
## '<s> this is a tokenization example</s>'

And the objective is to have a function that maps each token in the decode process to the correct input word, for here it will be:
desired_output = [[1],[2],[3],[4,5],[6]] As this corresponds to id 42, while token and ization corresponds to ids [19244,1938] which are at indexes 4,5 of the input_ids array.

Output:
If you use the fast tokenizers, i.e. the rust backed versions from the tokenizers library the encoding contains a word_ids method that can be used to map sub-words back to their original word. What constitutes a word vs a subword depends on the tokenizer, a word is something generated by the pre-tokenization stage, i.e. split by whitespace, a subword is generated by the actual model (BPE or Unigram for example).
The code below should work in general, even if the pre-tokenization performs additional splitting. For example I created my own custom step that splits based on PascalCase - the words here are Pascal and Case, the accepted answer wont work in this case since it assumes words are whitespace delimited.
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('roberta-large', do_lower_case=True)

example = "This is a tokenization example"

encoded = tokenizer(example)

desired_output = []
for word_id in encoded.word_ids():
    if word_id is not None:
        start, end = encoded.word_to_tokens(word_id)
        if start == end - 1:
            tokens = [start]
        else:
            tokens = [start, end-1]
        if len(desired_output) == 0 or desired_output[-1] != tokens:
            desired_output.append(tokens)
desired_output


Instruction: Number of instances per class in pytorch dataset

Input:
I'm trying to make a simple image classifier using PyTorch.
This is how I load the data into a dataset and dataLoader:

batch_size = 64
validation_split = 0.2
data_dir = PROJECT_PATH+"/categorized_products"
transform = transforms.Compose([transforms.Grayscale(), CustomToTensor()])

dataset = ImageFolder(data_dir, transform=transform)

indices = list(range(len(dataset)))

train_indices = indices[:int(len(indices)*0.8)]
test_indices = indices[int(len(indices)*0.8):]

train_sampler = SubsetRandomSampler(train_indices)
test_sampler = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=train_sampler, num_workers=16)
test_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, sampler=test_sampler, num_workers=16)

I want to print out the number of images in each class in training and test data separately, something like this:

In train data:

shoes: 20
shirts: 14

In test data:

shoes: 4
shirts: 3

I tried this:

from collections import Counter
print(dict(Counter(sample_tup[1] for sample_tup in dataset.imgs)))

but I got this error:

AttributeError: 'MyDataset' object has no attribute 'img'

Output:
You need to use .targets to access the labels of data i.e.

print(dict(Counter(dataset.targets)))

It'll print something like this (e.g. in MNIST dataset):

{5: 5421, 0: 5923, 4: 5842, 1: 6742, 9: 5949, 2: 5958, 3: 6131, 6: 5918, 7: 6265, 8: 5851}

Also, you can use .classes or .class_to_idx to get mapping of label id to classes:

print(dataset.class_to_idx)
{'0 - zero': 0,
 '1 - one': 1,
 '2 - two': 2,
 '3 - three': 3,
 '4 - four': 4,
 '5 - five': 5,
 '6 - six': 6,
 '7 - seven': 7,
 '8 - eight': 8,
 '9 - nine': 9}

Edit: Method 1

From the comments, in order to get class distribution of training and testing set separately, you can simply iterate over subset as below:

train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])

# labels in training set
train_classes = [label for _, label in train_dataset]
Counter(train_classes)
Counter({0: 4757,
         1: 5363,
         2: 4782,
         3: 4874,
         4: 4678,
         5: 4321,
         6: 4747,
         7: 5024,
         8: 4684,
         9: 4770})

Edit (2): Method 2

Since you've a large dataset, and as you said it takes considerable time to iterate over all training set, there is another way:

You can use .indices of subset, which referes to indices in the original dataset selected for subset.

i.e.

train_classes = [dataset.targets[i] for i in train_dataset.indices]
Counter(train_classes) # if doesn' work: Counter(i.item() for i in train_classes)


Instruction: Pytorch Dataloader not spliting data into batch

Input:
I have dataset class like this:

class LoadDataset(Dataset):
    def __init__(self, data, label):
        self.data = data
        self.label = label
    def __len__(self):
        dlen = len(self.data)
        return dlen
    def __getitem__(self, index):
        return self.data, self.label

then i load my image dataset which have [485, 1, 32, 32] shape

train_dataset = LoadDataset(xtrain, ytrain)
print(len(train_dataset))
# output 485

then i load the data with DataLoader

train_loader = DataLoader(train_dataset, batch_size=32)

and then i iterate the data:

for epoch in range(num_epoch):
        for inputs, labels in train_loader:
            print(inputs.shape)

the output prints torch.Size([32, 485, 1, 32, 32]), it should be torch.Size([32, 1, 32, 32]),

Can anyone help me?

Output:
The __getitem__ method should return 1 data piece, you returned all of them.

Try this:

class LoadDataset(Dataset):
    def __init__(self, data, label):
        self.data = data
        self.label = label
    def __len__(self):
        dlen = len(self.data)
        llen = len(self.label)  # different here
        return min(dlen, llen)  # different here
    def __getitem__(self, index):
        return self.data[index], self.label[index]  # different here


Instruction: Can anyone explain about the "non-sliding window" statement in Feature Pyramid Networks for Object Detection paper?

Input:
Feature Pyramid Networks for Object Detection adopt RPN technique to create the detector, and it use sliding window technique to classify. How come there is a statement for "non-sliding window" in 5.2 section?

The extended statement in the paper :
5.2. Object Detection with Fast/Faster R-CNN
Next we investigate FPN for region-based (non-sliding window) detectors.

In my understanding, FPN using sliding window in detection task. This is also mentioned in
https://medium.com/@jonathan_hui/understanding-feature-pyramid-networks-for-object-detection-fpn-45b227b9106c the statement is

"FPN extracts feature maps and later feeds into a detector, says RPN, for object detection. RPN applies a sliding window over the feature maps to make predictions on the objectness (has an object or not) and the object boundary box at each location."

Thank you in advanced.

Output:
Feature Pyramid Networks(FPN) for Object Detection is not an RPN.

FPN is just a better way to do feature extraction. It incorporates features from several stages together which gives better features for the rest of the object detection pipeline (specifically because it incorporates features from the first stages which gives better features for detection of small/medium size objects).

As the original paper states: "Our goal is to leverage a ConvNet’s pyramidal feature
hierarchy, which has semantics from low to high levels, and
build a feature pyramid with high-level semantics throughout. The resulting Feature Pyramid Network is general purpose and in this paper we focus on sliding window proposers (Region Proposal Network, RPN for short) [29] and
region-based detectors (Fast R-CNN)"

So they use it to check "Two stage" object detection pipeline. The first stage is the RPN and this is what they check in section 5.1 and then they check it for the classification stage in section 5.2.

Fast R-CNN Faster R-CNN etc.. are region based object detectors and not sliding window detectors. They get a fixed set of regions from the RPN to classify and thats it.

A good explanation on the differences you can see at https://medium.com/@jonathan_hui/what-do-we-learn-from-region-based-object-detectors-faster-r-cnn-r-fcn-fpn-7e354377a7c9.


Instruction: version `GLIBC_2.28' not found

Input:
I'm trying to install PyTorch on ARMv7(32-bit) architecture but PyTorch doesn’t have official ARMv7 builds so i tried this unofficial build.

It installed successfully but when I import torch I get the following error

>>import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

I tried the following

sudo apt-get update
sudo apt-get install libc6

but it seams like that i have the newest version of libc6

Reading package lists... Done
Building dependency tree
Reading state information... Done
libc6 is already the newest version (2.23-0ubuntu11).
The following packages were automatically installed and are no longer required:
  busybox-initramfs cpio initramfs-tools initramfs-tools-bin initramfs-tools-core klibc-utils libdbusmenu-gtk4 libklibc
  libllvm3.8 libmircommon5 linux-base
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.

Here is my GLIBCXX and GLIBC versions that i have:

strings /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 | grep GLIBC
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_3.4.20
GLIBCXX_3.4.21
GLIBCXX_3.4.22
GLIBCXX_3.4.23
GLIBCXX_3.4.24
GLIBCXX_3.4.25
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28
GLIBC_2.4
GLIBC_2.6
GLIBC_2.18
GLIBC_2.16
GLIBC_2.17

Ldd version:

ldd --version

ldd (Ubuntu GLIBC 2.23-0ubuntu11) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

My OS:

cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

So is it possible to install GLIBC_2.28 on my machine?

Output:
  So is it possible to install GLIBC_2.28 on my machine?

It is possible, but the chances of you making a mistake and rendering your system un-bootable are quite high. It is also very likely that doing so will break something else on your system (this is the reason distributions do not usually update the version of GLIBC from the one they originally shipped with).

A much better solution is to built PyTorch targeting your system (i.e. using your "normal" toolchain).

P.S. GLIBCXX has nothing to do with your problem, and just adds noise to your question.


Instruction: New to PyTorch, having trouble making predictions once data is loaded using Data Loader

Input:
I'm completely new to PyTorch (have previously used tensor flow) and I'm stuck on something I'm working on. I've been tasked with using a pretrained model to extract the features from application documents and then compute similarity scores to identify duplicates. I have all of the pdf's converted to .jpg's, and I've loaded the pretrained model and modified the last layer to extract features. The folder structure is like this:

root
|- Application 1
|   |- image 1
|   |- image 2...
|- Application 2
|   |- image 1
|   |- image 2...

What I'm trying to do is extract features from the images in every sub-directory and calculate the euclidean distance between them and output a similarity matrix. Where I'm having an issue, and this may seem really basic, is actually making the predictions once the data is loaded. Below is the code I have so far, any help would be greatly appreciated.

def get_pretrained_model_notop(model_name): #pull the model and change last layer
    pretrained_model = model_name(pretrained=True) #downloads pretrained model weights
    for param in pretrained_model.parameters():
        param.requires_grad = False #freezes layers
    pretrained_model = nn.Sequential(*list(pretrained_model.children())[:-1]) #drops final layer, because we aren't classifying 1000 imagenet classes
    pretrained_model.fc = nn.Sequential(
    nn.Flatten() #adds flatten layer at end of model
                            )
    if torch.cuda.is_available(): #uses GPU if available
        pretrained_model = pretrained_model.cuda()
    return pretrained_model

def get_similarity(pretrained_model,train_imgs):   #function to extract features from the model and compute similarity scores
    bottleneck_feature_example = pretrained_model(train_imgs)
    similarity = euclidean_distances(bottleneck_feature_example)
    similarity=similarity/similarity.max()
    similarity_df = pd.DataFrame(similarity)
    similarity_df=1-similarity_df
    return np.round(similarity_df,4)

transforms =           transforms.Compose([transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])])

img_dir='path'
images = datasets.ImageFolder(img_dir,transform=transforms)
data_loader = torch.utils.data.DataLoader(images,
            batch_size=32,
            shuffle=True,
            num_workers=4)

model_list=[models.densenet201]
model_name=['densenet201']
pretrained_model=[get_pretrained_model_notop(selected_model) for selected_model in model_list]
for data in data_loader:
    pred=[get_similarity(pretrained,data) for pretrained in pretrained_model]
    pred_label_ensemble=sum(pred) / len(pred)
    pred_label_ensemble.columns=page_numbers
    prob_output_folder = unzipped.replace('MF_loan_document', 'MF_loan_document_results')
    pred_label_ensemble.to_csv(prob_output_folder+'/'+'results.csv',index=False)

Output:
You didn't specify the problem, but I would assume that it is because you're expecting images but passing data to the get_similarity. So, you'd need to change:

pred=[get_similarity(pretrained,data) for pretrained in pretrained_model]

to:

pred = [get_similarity(pretrained, data[0]) for pretrained in pretrained_model]

because data is a tuple with (images, labels).


Instruction: How did Tensorflow process images of ImageNet in Tfrecord format when training resnet?

Input:
emm, converting imagenet to Tfrecord format is quite complicated, so I downloaded the processed Tfrecord imagenet from somewhere.

I applied it to resnet34 in tensorflow with parameters from Pytorch, but found that the accuracy is just 55%, too low. I guess the reason may be the different methods processing imagenet between Pytorch.models and this Tfrecord. And a nice bro has told me the way how Pytorch processing data, but I still need to know how tensorflow processing it.

I found that the values of Tfrecord pictures range from -1 to 1, can you tell me the processing method of this Tfrecord so that I can try to improve the accuracy?

Thanks a lot! I am just too new_flesh, your kind help is so so important

Output:
OK, Let me give out my first anser in stackoverflow to myself.

The different way processing images effect the accuracy, that' right.

Pytorch scales pictures to [0,1] and then normalizes them using mean and std.

And the Tfrecord file I found scales pictures to [0,1], and then scale them to [-1, 1] by:

image = tf.subtract(image, 0.5)
image = tf.multiply(image, 2.0)

(which is a strange way as I see).

So I  add comments to thes two lines, and got a 66% accuracy. By fine_tune, got 72%(But I stiil don't understand the accuracy decline when transfer parameters from Pytorch to Tensoflow).

PS: In this period, I found that parameters set in operator would be saved when saving models in tensorflow, so don't worry about how to set parameter of operator the same.


Instruction: What's the difference between dim in Pytorch and axis in Tensorflow?

Input:
I have two lines and I want to understand whether they will produce the same output or not?
In tensorflow: tf.norm(my_tensor, ord=2, axis=1)
In pytorch: torch.norm(my_tensor, p=2, dim=1)
Say the shape of my_tensor is [100,2]

Will the above two lines give the same result? Or is the axis attribute different from dim?

Output:
Yes, they are the same!
import tensorflow as tf
tensor = [[1., 2.], [4., 5.], [3., 6.], [7., 8.], [5., 2.]]
tensor = tf.convert_to_tensor(tensor, dtype=tf.float32)
t_norm = tf.norm(tensor, ord=2, axis=1)
print(t_norm)

Output
tf.Tensor([ 2.236068   6.4031243  6.708204
            10.630146   5.3851647], shape=(5,), dtype=float32)

import torch
tensor = [[1., 2.], [4., 5.], [3., 6.], [7., 8.], [5., 2.]]
tensor = torch.tensor(tensor, dtype=torch.float32)
t_norm = torch.norm(tensor, p=2, dim=1)
print(t_norm)

Output
tensor([ 2.2361,  6.4031,  6.7082, 10.6301,  5.3852])


Instruction: Why does this tensor.unfold method in this example add a dimension?

Input:
I am familiarizing myself with the Pytorch unfold method from https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold

I looked at their example which is

>>> x = torch.arange(1., 8)
>>> x
tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.])
>>> x.unfold(0, 2, 1)
tensor([[ 1.,  2.],
        [ 2.,  3.],
        [ 3.,  4.],
        [ 4.,  5.],
        [ 5.,  6.],
        [ 6.,  7.]])

I understand above that when we unfold in dimension 0, we take chunks of size 2 at a time with stride 1 and therefore, the result is an arrangement of different chunks, which are [1., 2.], [2., 3.] and so on. As we have 6 chunks at the end, the chunks will be put together and the final shape is (6,2).

However, I have another example I ran as shown below.

In [115]: s = torch.arange(20).view(1,10,2)

In [116]: s
Out[116]:
tensor([[[ 0,  1],
         [ 2,  3],
         [ 4,  5],
         [ 6,  7],
         [ 8,  9],
         [10, 11],
         [12, 13],
         [14, 15],
         [16, 17],
         [18, 19]]])

In [117]: s.unfold(0,1,1)
Out[117]:
tensor([[[[ 0],
          [ 1]],

         [[ 2],
          [ 3]],

         [[ 4],
          [ 5]],

         [[ 6],
          [ 7]],

         [[ 8],
          [ 9]],

         [[10],
          [11]],

         [[12],
          [13]],

         [[14],
          [15]],

         [[16],
          [17]],

         [[18],
          [19]]]])

In [119]: s.unfold(0,1,1).shape
Out[119]: torch.Size([1, 10, 2, 1])

So you see my original tensor was of shape (1,10,2) and I asked for an unfolding operation with parameters s.unfold(0, 1, 1).

Going by original understanding from the previous example, I assumed this means in the dimension 0, we take 1 chunk at a time with stride 1. Thus, as we have go into dimension 0, we see that we have only one chunk of size (10,2). So the output should have just taken this chunk and may be it should have just added a dimension to wrap this chunk and given me an output of size (1, 10, 2).

However, it gives me an output of size (1, 10, 2, 1). Why does it have an extra dimension at the last? Can someone elaborate intuitively please?

Output:
The documentation states:

  An additional dimension of size size is appended in the returned tensor.

where size is the size of the chunks you specified (second argument). By definition, it always adds an additional dimension, which makes it consistent no matter what size you choose. Just because a dimension has size 1, doesn't mean it should be omitted automatically.

Regarding the intuition behind this, let's consider that instead of returning a tensor where the last dimension represents the chunks, we create a list of all chunks. For simplicity, we'll limit it to the first dimension with a step of 1.

import torch
from typing import List

def list_chunks(tensor: torch.Tensor, size: int) -> List[torch.Tensor]:
    chunks = []
    for i in range(tensor.size(0) - size + 1):
        chunks.append(tensor[i : i + size])
    return chunks

x = torch.arange(1.0, 8)
s = torch.arange(20).view(1, 10, 2)

# As expected, a list with 6 elements, as there are 6 chunks.
list_chunks(x, 2)
# => [tensor([1., 2.]),
#     tensor([2., 3.]),
#     tensor([3., 4.]),
#     tensor([4., 5.]),
#     tensor([5., 6.]),
#     tensor([6., 7.])]

# The list has only a single element, as there is only a single chunk.
# But it's still a list.
list_chunks(s, 1)
# => [tensor([[[ 0,  1],
#              [ 2,  3],
#              [ 4,  5],
#              [ 6,  7],
#              [ 8,  9],
#              [10, 11],
#              [12, 13],
#              [14, 15],
#              [16, 17],
#              [18, 19]]])]

I've deliberately included type annotations to make it clearer what we are expecting from the function. If there is only a single chunk, it will be a list with one element, as it is always a list of chunks.

You were expecting a different behaviour, namely when there is a single chunk, you want the single chunk instead of a list. Which would change the implementation as follows.

from typing import List, Union

def list_chunks(tensor: torch.Tensor, size: int) -> Union[List[torch.Tensor], torch.Tensor]:
    chunks = []
    for i in range(tensor.size(0) - size + 1):
        chunks.append(tensor[i : i + size])
    # If it's a single chunk, return just the chunk itself
    if len(chunks) == 1:
        return chunks[0]
    else:
        return chunks

With that change, anyone that uses this function, now needs to take two cases into consideration. If you don't distinguish between a list and a single chunk (tensor), you will get unexpected results, e.g. looping over the chunks would instead loop over the first dimension of the tensor.

The programmatically intuitive approach is to always return a list of chunks and torch.unfold does the same, but instead of a list of chunks, it's a tensor where the last dimension can be seen as the listing of the chunks.


Instruction: Best way to output prediction result for a test set from a model with 1 output (binary classification)

Input:
This seems simple, but I'm looking for the best way to output the prediction results for a model with 1 output (binary classification model). My labels are 0 and 1 in this example. Now I can just say if model output > 0.5 label is 1. But I'm sort of guessing this is correct. I am hence wondering if there is a better approach, like

prediction = np.argmax(y_hat.detach().numpy, axis=0) for multiple classes.

Currently I'm doing this to test:

test = [[0,0],[0,1],[1,1],[1,0]]

for trial in test:
  Xtest = torch.Tensor(trial)
  y_hat = logreg_clf(Xtest)

  if y_hat > 0.5:
    prediction = 1
  else:
    prediction = 0
  # prediction = np.argmax(y_hat.detach().numpy, axis=0)
  print("{0} xor {1} = {2}".format(int(Xtest[0]), int(Xtest[1]), prediction))

Was also wondering if there is a function to automatically get the accuracy or confusion matrix for test set 'test'.

FYI my model:

import torch.nn as nn
import torch.nn.functional as F

class LogisticRegression(nn.Module):
  # input_size: Dimensionality of input feature vector.
  # num_classes: The number of classes in the classification problem.
  def __init__(self, input_size, num_classes):
    # Always call the superclass (nn.Module) constructor first!
    super(LogisticRegression, self).__init__()
    # Set up the linear transform
    self.linear = nn.Linear(input_size, num_classes)
    # I do not yet include the sigmoid activation after the linear
    # layer because our loss function will include this as you will see later

  # Forward's sole argument is the input.
  # input is of shape (batch_size, input_size)
  def forward(self, x):
    # Apply the linear transform.
    # out is of shape (batch_size, num_classes).
    out = self.linear(x)
    out = F.sigmoid(out)
    # Softmax the out tensor to get a log-probability distribution
    # over classes for each example.
    return out

# Binary classifiation
num_outputs = 1
num_input_features = 2

# Create the logistic regression model
logreg_clf = LogisticRegression(num_input_features, num_outputs)

print(logreg_clf)

import torch
lr_rate = 0.001  # alpha

# training set of input X and labels Y
X = torch.Tensor([[0,0],[0,1], [1,0], [1,1]])
Y = torch.Tensor([0,1,1,0]).view(-1,1) #view is similar to numpy.reshape() here it makes it into a column

# Run the forward pass of the logistic regression model
sample_output = logreg_clf(X) #completely random at the moment
print(X)

loss_function = nn.BCELoss()
# SGD: stochastic gradient descent is used to train/fit the model
optimizer = torch.optim.SGD(logreg_clf.parameters(), lr=lr_rate)

import numpy as np
# from torch.autograd import Variable

#training loop:

epochs = 2001 #how many times we go through the training set
steps = X.size(0) #steps = 4; we have 4 training examples (I know, tiny training set :)

for i in range(epochs):
    for j in range(steps):
        # randomly sample from the training set:
        data_point = np.random.randint(X.size(0))
        # store the retrieved datapoint into 2 separate variables of the right shape
        x_var = torch.Tensor(X[data_point]).unsqueeze(0)
        y_var = torch.Tensor(Y[data_point])

        optimizer.zero_grad() # empty (zero) the gradient buffers
        y_hat = logreg_clf(x_var) #get the output from the model
        print(y_hat)
        print(y_var)
        loss = loss_function(y_hat, y_var) #calculate the loss
        loss.backward() #backprop
        optimizer.step() #does the update

    if i % 500 == 0:
        print ("Epoch: {0}, Loss: {1}, ".format(i, loss.data.numpy()))

Output:
test_data = torch.Tensor(test)
raw_preds = logreg_clf(test_data)
preds = (raw_preds > 0.5).long()

To get the predictions for all test data at once, we first convert the test data to a tensor, then we can make a forward pass with that tensor.
The raw predictions are something like tensor([[0.4795], [0.4749], [0.4960], [0.5006]].
Then we apply the "risk neutral strategy" i.e. 0.5 as the threshold to get the results (which will be of type torch.bool so we convert it to long).

Equivalently in one line:

preds = (logreg_clf(torch.Tensor(test)) > 0.5).long()

So  the predictions are:

tensor([[0], [0], [0], [1]])

  wondering if there is a function to automatically get the accuracy or confusion matrix

You can use respective functions from sklearn.metrics:

from sklearn.metrics import accuracy_score, confusion_matrix

# the ground truth for the given test data
truth = torch.Tensor([0, 1, 0, 1]).view(-1, 1)

conf_mat = confusion_matrix(truth, preds)
acc = accuracy_score(truth, preds)

which gives the confusion matrix as

array([[2, 0],
       [1, 1]], dtype=int64)

and the accuracy as

0.75

So you have one false negative i.e. the second sample in the test data.


Instruction: Cuda version issue while using Detectron2 in Google Colab

Input:
I am trying to run the Detectron2 module on Colab using CUDA version 10.0 but since today there have been some issues regarding the versions of Cuda Compiler.

The output I get after running !nvidia-smi is :

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   36C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |
|                               |                      |                 ERR! |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And what I get after running !nvcc --version is :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

I am not able to understand the reason for the mismatch. Also the output from detectron after running !python -m detectron2.utils.collect_env is :

----------------------  ----------------------------------------------------------------------------
sys.platform            linux
Python                  3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
numpy                   1.18.5
detectron2              0.1.3 @/content/gdrive/My Drive/Data/Table_Struct/detectron2_repo/detectron2
Compiler                GCC 7.5
CUDA compiler           CUDA 10.1
detectron2 arch flags   sm_60
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.4.0+cu100 @/usr/local/lib/python3.6/dist-packages/torch
PyTorch debug build     False
GPU available           True
GPU 0                   Tesla K80
CUDA_HOME               /usr/local/cuda
Pillow                  7.0.0
torchvision             0.5.0+cu100 @/usr/local/lib/python3.6/dist-packages/torchvision
torchvision arch flags  sm_35, sm_50, sm_60, sm_70, sm_75
fvcore                  0.1.1
cv2                     4.1.2
----------------------  ----------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

My guess is that the version of CUDA on Colab doesn't match the Detectron2 I am using. IF so how can I change something to make this work on Google Colab.

Output:
The problem was with the compiled Detectron2 Cuda runtime version and once I recompiled Detectron2 the error was solved.

Here is the result from !python -m detectron2.utils.collect_env command:

----------------------  ----------------------------------------------------------------------------
sys.platform            linux
Python                  3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]
numpy                   1.18.5
detectron2              0.1.3 @/content/gdrive/My Drive/Data/Table_Struct/detectron2_repo/detectron2
Compiler                GCC 7.5
CUDA compiler           CUDA 10.0
detectron2 arch flags   sm_75
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.4.0+cu100 @/usr/local/lib/python3.6/dist-packages/torch
PyTorch debug build     False
GPU available           True
GPU 0                   Tesla T4
CUDA_HOME               /usr/local/cuda
Pillow                  7.0.0
torchvision             0.5.0+cu100 @/usr/local/lib/python3.6/dist-packages/torchvision
torchvision arch flags  sm_35, sm_50, sm_60, sm_70, sm_75
fvcore                  0.1.1
cv2                     4.1.2
----------------------  ----------------------------------------------------------------------------
PyTorch built with:
  - GCC 7.3
  - Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v0.21.1 (Git Hash 7d2fd500bc78936d1d648ca713b901012f470dbc)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CUDA Runtime 10.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.1
  - Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,


Instruction: Pytorch DataLoader doesn't work with remote interpreter

Input:
I have the following error.

Expected: /home/ubuntu/.pycharm_helpers/pydev/pydevd_attach_to_process/attach_linux_amd64.so to exist.

And Here is the code:

import torch_geometric.transforms as T
category = 'Airplane'
path = osp.join(osp.dirname(osp.realpath(__file__)), '..', 'data', 'ShapeNet')
transform = T.Compose([
    T.RandomTranslate(0.01),
    T.RandomRotate(15, axis=0),
    T.RandomRotate(15, axis=1),
    T.RandomRotate(15, axis=2)
])
pre_transform = T.NormalizeScale()
train_dataset = ShapeNet(path, category, split='trainval', transform=transform,
                         pre_transform=pre_transform)
test_dataset = ShapeNet(path, category, split='test',
                        pre_transform=pre_transform)
train_loader = DataLoader(train_dataset, batch_size=12, shuffle=True, num_workers=6)
test_loader = DataLoader(test_dataset, batch_size=12, shuffle=False,
                         num_workers=6)

When I'm trying sample from dataset using the dataloader, the debugger crashes and returns this erros.

Have tried deleting remote helpers, but it didn't solve my problem.
My local machine in running on windows 10 and the remote one is running on Ubuntu 18.04

Output:
Answering to myself here,

I've tried to follow this : https://intellij-support.jetbrains.com/hc/en-us/community/posts/360006791159--pycharm-helpers-pydev-pydevd-attach-to-process-attach-linux-amd64-so-undefined-symbol-AttachDebuggerTracing

But it has proven inefficient.

On the contrary, deleting the parameter 'num_workers' from the Dataloader solved this issue. So this is an easy workaround.

However, I haven't wrap my head around the problem yet. It's probably that multiprocessing is buggy. But if anyone has any insight of what's going on. I'll gladly take it.


Instruction: tensorboard --logdir=runs not working: Abort trap: 6

Input:
I am trying to run tensorboard: tensorboard --logdir=runs.
I have also tried: tensorboard --logdir=runs --host=127.0.0.1.
I am running the command from the terminal from within the the directory, which contains the runs folder.

I get the following error:

[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1367]
CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
libc++abi.dylib: terminating with uncaught exception of type google::protobuf::FatalException:
CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
Abort trap: 6

My Python code contains the following lines:

tb_path = './runs/SimpleLSTM_MNIST'
if os.path.isdir(tb_path):
    shutil.rmtree(tb_path)

writer = tb.SummaryWriter(log_dir=tb_path)

My runs folder contains the folder SimpleLSTM_MNIST, which contains events.out.tfevents.1591953948.computername.local.29440.0.

Operating System: MacOS Catalina

How can I resolve the problem?

Output:
Apparently this is a specific issue that occurs when running macOS Catalina, and can be solved by switching to protobuf version 3.8.0 and tensorflow version 2.0.0.
So basically uninstalling tensorflow and protobuf and re-installing with pip3 install protobuf==3.8.0 and pip3 install tensorflow==2.0.0.


Instruction: Missing/unexpected keys in resnet50 with pytorch

Input:
I get the following errors and no idea why:

Missing key(s) in state_dict: "layer2.0.layer.0.inplace.0.weight", "layer2.0.layer.0.inplace.0.bias",...

Unexpected key(s) in state_dict: "layer2.0.layer.0.0.weight", "layer2.0.layer.0.0.bias",...

The channel sizes I set seem to be what I wanted. however, I don't see where the mistake is

import torch.nn as nn

#1x1 convolution
def conv1x1(in_channels: object, out_channels: object, stride: object, padding: object) -> object:

    model = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, padding=padding),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True)
    )
    return model

# 3x3 convolution
def conv3x3(in_channels: object, out_channels: object, stride: object, padding: object) -> object:
    model = nn.Sequential(
        nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=padding),
        nn.BatchNorm2d(out_channels),
        nn.ReLU(inplace=True)
    )
    return model

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels, downsample=False):
        super(ResidualBlock, self).__init__()
        self.downsample = downsample

        if self.downsample:
            self.layer = nn.Sequential(
                nn.ReLU(conv1x1(in_channels, middle_channels,1,0)),
                nn.ReLU(conv3x3(middle_channels, middle_channels,1,0)),
                nn.ReLU(conv1x1(middle_channels,out_channels,1,0))
            )
            self.downsize = conv1x1(in_channels, out_channels, 2, 0)

        else:
            self.layer = nn.Sequential(
                nn.ReLU(conv1x1(in_channels,middle_channels,2,0)),
                nn.ReLU(conv3x3(middle_channels,middle_channels,2,0)),
                nn.ReLU(conv1x1(middle_channels,out_channels,2,0))
            )
            self.make_equal_channel = conv1x1(in_channels, out_channels, 1, 0)

    def forward(self, x):
        if self.downsample:
            out = self.layer(x)
            x = self.downsize(x)
            return out + x
        else:
            out = self.layer(x)
            if x.size() is not out.size():
                x = self.make_equal_channel(x)
            return out + x

class ResNet50_layer4(nn.Module):
    def __init__(self, num_classes= 10 ): # Hint : How many classes in Cifar-10 dataset?
        super(ResNet50_layer4, self).__init__()
        self.layer1 = nn.Sequential(
            #in_channels, out_channels, kernel_size, stride, padding
            nn.Conv2d(in_channels=3, out_channels=64, kernel_size=7, stride=2, padding=3),
                # Hint : Through this conv-layer, the input image size is halved.
                #        Consider stride, kernel size, padding and input & output channel sizes.
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2, padding=0)
        )
        self.layer2 = nn.Sequential(
            #in_channels, middle_channels, out_channels, downsample=False
            ResidualBlock(in_channels=64, middle_channels=64, out_channels=256, downsample=False),
            ResidualBlock(in_channels=256, middle_channels=64, out_channels=256, downsample=False),
            ResidualBlock(in_channels=256, middle_channels=64,out_channels=256, downsample=True)
        )
        self.layer3 = nn.Sequential(
            ResidualBlock(in_channels=256, middle_channels=128, out_channels=512, downsample=False),
            ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=False),
            ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=False),
            ResidualBlock(in_channels=512, middle_channels=128, out_channels=512, downsample=True)
        )
        self.layer4 = nn.Sequential(
            ResidualBlock(in_channels=512, middle_channels=256, out_channels=1024, downsample=False),
            ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
            ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
            ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
            ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False),
            ResidualBlock(in_channels=1024, middle_channels=256, out_channels=1024, downsample=False)

        )

        self.fc = nn.Linear(1024, 10)
        self.avgpool = nn.AvgPool2d(7, stride=1)

        for m in self.modules():
            if isinstance(m, nn.Linear):
                nn.init.xavier_uniform_(m.weight.data)
            elif isinstance(m, nn.Conv2d):
                nn.init.xavier_uniform_(m.weight.data)

    def forward(self, x):

        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.avgpool(out)
        out = out.view(out.size()[0], -1)
        out = self.fc(out)

        return out

Output:
You have changed your model, and as a result, the keys have changed. So, you are getting a mismatch error. I think you have added nn.ReLU() in the sequential wrappers in ResidualBlock.

In your ResidualBlock, you have:

self.layer = nn.Sequential(
    nn.ReLU(conv1x1(in_channels, middle_channels, 2, 0)),
    nn.ReLU(conv3x3(middle_channels, middle_channels, 2, 0)),
    nn.ReLU(conv1x1(middle_channels, out_channels, 2, 0))
)

However, in your conv1x1 and conv3x3, you already have nn.ReLU(inplace=True) as the last layer in nn.Sequential. Hence, having another nn.ReLU() in

nn.ReLU(conv1x1(in_channels, middle_channels, 2, 0))

seems unnecessary. If you remove the nn.ReLU(), then the keys will match.

I revise ResidualBlock as follows.

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels, downsample=False):
        super(ResidualBlock, self).__init__()
        self.downsample = downsample

        if self.downsample:
            self.layer = nn.Sequential(
                conv1x1(in_channels, middle_channels, 1, 0),
                conv3x3(middle_channels, middle_channels, 1, 0),
                conv1x1(middle_channels, out_channels, 1, 0)
            )
            self.downsize = conv1x1(in_channels, out_channels, 2, 0)

        else:
            self.layer = nn.Sequential(
                conv1x1(in_channels, middle_channels, 2, 0),
                conv3x3(middle_channels, middle_channels, 2, 0),
                conv1x1(middle_channels, out_channels, 2, 0)
            )
            self.make_equal_channel = conv1x1(in_channels, out_channels, 1, 0)

    def forward(self, x):
        '''Your forward method description'''

Now, let's test.

model = ResNet50_layer4()
for k, v in model.named_parameters():
    print(k)

Output:

layer1.0.weight
layer1.0.bias
layer1.1.weight
layer1.1.bias
layer2.0.layer.0.0.weight
layer2.0.layer.0.0.bias
layer2.0.layer.0.1.weight
layer2.0.layer.0.1.bias
...
...

If you want to still use an additional nn.ReLU(), you can train the revised model and save weights, and then try to load the weights back, it will work.


Instruction: “AttributeError: classificadorFinal' object has no attribute 'log_softmax” when trying to train a neural network using pytorch

Input:
I'm learning to use pytorch and I got an error that won't let me continue programming.

My code:

import torch.nn as nn
from skorch import NeuralNetClassifier #integracao com sklearn
from sklearn.model_selection import cross_val_score,GridSearchCV
from sklearn.preprocessing import LabelEncoder, MinMaxScaler
import torch
import torch.nn.functional as F
from torch import nn,optim

class classificadorFinal(nn.Module):
    def __init__(self, activation=F.tanh, neurons=16, initializer=torch.nn.init.uniform_, dropout=0.3):
        ##from melhores_parametros
        super().__init__()
        self.dense0 = nn.Linear(4, neurons)
        initializer(self.dense0.weight)
        self.activation0 = activation
        self.dense1 = nn.Linear(neurons, neurons)
        initializer(self.dense1.weight)
        self.activation1 = activation
        self.dense2 = nn.Linear(neurons, 3)

        self.dropout = nn.Dropout(dropout)

    def forward(self, X):
        X = self.dense0(X)
        X = self.activation0(X)
        X = self.dropout(X)
        X = self.dense1(X)
        X = self.activation1(X)
        X = self.dropout(X)
        X = self.dense2(X)
        return X

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(classificador.parameters(), lr = 0.001, weight_decay = 0.0001)

#treino
for epoch in range(200):##from melhores_parametros

    running_loss = 0.
    running_accuracy = 0.

    for data in train_loader:
        inputs, labels = data

        optimizer.zero_grad()

        outputs = classificadorFinal(inputs)

        loss = criterion(outputs, labels)###erro
        loss.backward()

        optimizer.step()

        running_loss += loss.item()

        ps = F.softmax(outputs)

        top_p, top_class = ps.topk(k = 1, dim = 1)

        equals = top_class == labels.view(*top_class.shape)

        running_accuracy += torch.mean(equals.type(torch.float))

    print('Época {:3d}: perda {:3.5f} - precisão {:3.5f}'.format(epoch + 1, running_loss/len(train_loader), running_accuracy/len(train_loader)))

The error occurs exactly on loss = criterion(outputs, labels):

AttributeError: 'classificadorFinal' object has no attribute 'log_softmax'

I found out this error is well known, but I did not understand the proposed solution:

disable aux_logits when the model is created  aux_logits=False.

A little help, please!

Output:
The outputs are not actually the output of the model, but rather the model itself. classificadorFinal is the class, calling it creates an object/instance of that class, and inputs will be the first argument to the __init__ method, namely  activation.

# Creates an instance of the model
outputs = classificadorFinal(inputs)

You first have to create the model (an instance), which should be done once, then call that model with the inputs. It looks like you have already created the model before, as you are using classificador.parameters() for the optimiser, hence classificador is presumably the instance of the model. You need to call classificador (instance) not classificadorFinal (class) to create the outputs.

# Call the instance of the model, not the class
outputs = classificador(inputs)


Instruction: Data format and actual shape

Input:
I'm trying to migrate TensorFlow checkpoint weights to PyTorch.

When I extract some weights with cp.load_variable(<CKPT>, <FIELD_NAME>), I get a 4D list ordered as HWCN, for example [1, 1, 512, 1024] which is clearly HWCN.
However, all convolution blocks data_format are set to NHWC.

So, the question is, why there's mismatch?
what should I believe? does the 4D list from cp.load_variable is correct and all left to do is permute the dimensions?

Thanks!

Output:
The weights are not given as HWCN, as the weights do not have any batch dimension (N), otherwise that would apply a different weight for each sample in the batch. The shape is [kernel_height, kernel_width, in_channels, out_channels]. There is no mismatch, because data_format specifies which format the input and output use.

In PyTorch the weight of convolutions is given as [out_channels, in_channels, kernel_height, kernel_width], therefore you only need to permute the dimensions.


Instruction: Scatter tensor in pytorch along the rows

Input:
I want to scatter tensors in granularities of rows.

For example consider,

Input = torch.tensor([[2, 3], [3, 4], [4, 5]])

I want to scatter

S = torch.tensor([[1,2],[1,2]])

to indices

I = torch.tensor([0,2])

I expect the output to be torch.tensor([[1, 2], [3, 4], [1, 2]]).

Here S[0] is scattered to Input[I[0]], similarly S[1] is scattered to Input[I[1]]

How can I achieve this? Instead of looping over the row in S, I am looking for a more efficient way.

Output:
Do  input[I] = S

Example:

input = torch.tensor([[2, 3], [3, 4], [4, 5]])
S = torch.tensor([[1,2],[1,2]])
I = torch.tensor([0,2])

input[I] = S
input
tensor([[1, 2],
        [3, 4],
        [1, 2]])


Instruction: Convert weight and bias to sparse tensor pytorch

Input:
I'm trying to convert torch.nn.Parameters to sparse tensor. Pytorch documents say that Parameters is a Tensor's subclass. Tensor support to_sparse method but if I convert a Parameters to sparse, it will give me:
TypeError: cannot assign 'torch.cuda.sparse.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
Is there a way to bypass this and use sparse tensor for Parameters?
Here is example code to produce the problem:

for name, module in net.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        module.weight = module.weight.data.to_sparse()
        module.bias = module.bias.data.to_sparse()

Output:
torch.Tensor.to_sparse() returns a sparse copy of the tensor which cannot be assigned to module.weight since this is an instance of torch.nn.Parameter. So, you should rather do:

module.weight = torch.nn.Parameter(module.weight.data.to_sparse())
module.bias = torch.nn.Parameter(module.bias.data.to_sparse())

Please note that Parameters are a specific type of Tensor that is marked as being a parameter from an nn.Module, so they are different from ordinary Tensors.


Instruction: Instance Normalization with batch size 1

Input:
I am really confused with the meaning of Instance Norm and whether I can use it with a batch size of 1. I am using PyTorch and nothing in the documentation says that batch size should be greater than 1.

I know that for BatchNorm the performance is adversely affected when batch size is less than 8 and hence it puts a sort of soft bound on the batch size. However, I did not see any such analysis on Instance Norm and am a bit confused now. Should I remove the norm layer if my batch size is 1 then?

Output:
A good overview of the different norms is shown in the Group Normalization paper.

Instance normalisation is summarised as:

  [...] IN computes µ and σ along the (H, W) axes for each sample and each channel.

The mean and standard deviation are computed on the spatial dimensions (H, W) only and are independent of the batch size and channels (there are N x C different norms). Hence, you can use it with a batch size of 1.