Instruction: Image size of 256x256 (not 299x299) fed into Inception v3 model (PyTorch) and works?

Input:
I am testing out the pretrained inception v3 model on Pytorch. I fed it an image size 256x256 and also resized it up to 299x299. In both cases, the image was classified correctly.

Can someone explain why the PyTorch pretrained model can accept an image that's not 299x299?

Output:
It's because the pytorch implementation of inception v3 uses an adaptive average pooling layer right before the fully-connected layer.

If you take a look at the Inception3 class in torchvision/models/inception.py, the operation of most interest with respect to your question is x = F.adaptive_avg_pool2d(x, (1, 1)). Since the average pooling is adaptive the height and width of x before pooling are independent of the output shape. In other words, after this operation we always get a tensor of size [b,c,1,1] where b and c are the batch size and number of channels respectively. This way the input to the fully connected layer is always the same size so no exceptions are raised.

That said, if you're using the pretrained inception v3 weights then the model was originally trained for input of size 299x299. Using inputs of different sizes may have a negative impact on loss/accuracy, although smaller input images will almost certainly decrease computational time and memory footprint since the feature maps will be smaller.


Instruction: CNN: taking the most confident prediction among many

Input:
I'm training a CNN for image classification. The same object (with the same label then) is present in the test set twice (like two view-point). I'd like to take advantage of this when predicting the class.

Right now the final layer is a Linear layer (PyTorch) and I'm using cross-entropy as loss function. I was wondering what is the best way to take the most confident prediction for each object. Should I first compute the LogSoftMax and take the class with the highest probability (among both arrays of predictions), or should I take the logits directly?

Output:
Since LogSoftMax preserves order, the largest logit will always correspond to the highest confidence. Therefore there's no need to perform the operation if all you're interested in is finding the index of most confident class.

Probably the easiest way to get the index of the most confident class is by using torch.argmax.

e.g.

batch_size = 5
num_logits = 10
y = torch.randn(batch_size, num_logits)
preds = torch.argmax(y, dim=1)

which in this case results in

>>> print(preds)
tensor([9, 7, 2, 4, 6])


Instruction: I have a question about TRANSLATION WITH A SEQUENCE TO SEQUENCE in the pytorch tutorials

Input:
I am currently learning about the Seq2seq translation. I am trying to understand and following PyTorch tutorial from this website "https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#attention-decoder".

In the website, They talk about the Attention technique. I would like to know which technique are they used between Luong and Bahdanau?
Another question, Why do they apply Relu layer before GRU cell?
Finally, the red box in the figure is called a context vector, right?

Output:
  I would like to know which technique are they used between Luong and Bahdanau?

Loung is multiplicative, so it should be using Bahdanau (additive attention) as it concats then applies linearity. See http://ruder.io/deep-learning-nlp-best-practices/index.html#attention for more about attention types

  Why do they apply RelU layer before GRU cell?

This is the activation after Linear layer. I think tanh was used originally but ReLU became preferrable.
I think the other ReLU after the embeddings in plain Decoder is there by mistake though
https://github.com/spro/practical-pytorch/issues/4

  the red box in the figure is called a context vector, right?

yes


Instruction: axes don't match array / size mismatch, m1: [132096 x 344], m2: [118336 x 128]

Input:
This is a linear auto-encoder code, the original picture is 344*344 RGB, after the training process is over, I want to show the decoded picture using the code below, but it has ValueError: axes don't match array

pytorch, google colab(GPU)

enter code here:
EPOCH = 20
BATCH_SIZE = 128
LR = 0.005  # learning rate

torch.cuda.empty_cache()
data_transforms = torchvision.transforms.Compose([
    torchvision.transforms.RandomResizedCrop(344),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor()])
path1 = 'drive/My Drive/Colab/image/test/'
train_data = torchvision.datasets.ImageFolder(path1,
transform=data_transforms)

train_loader = Data.DataLoader(dataset=train_data, batch_size=BATCH_SIZE,
shuffle=True)

class AutoEncoder(nn.Module):
    def __init__(self):
        super(AutoEncoder, self).__init__()

        self.encoder = nn.Sequential(
            nn.Linear(3*344*344, 128),
            nn.Tanh(), # 激活
            nn.Linear(128, 64),
            nn.Tanh(),
            nn.Linear(64, 12),
            nn.Tanh(),
            nn.Linear(12, 3),   # compress to 3 features which can be
visualized in plt
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.Tanh(),
            nn.Linear(12, 64),
            nn.Tanh(),
            nn.Linear(64, 128),
            nn.Tanh(),
            nn.Linear(128, 3*344*344),
            nn.Sigmoid(),       # compress to a range (0, 1)
        )

    def forward(self, x):
        x = x.view(x.size(0), -1)

        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return encoded, decoded

autoencoder = AutoEncoder()

optimizer = torch.optim.Adam(autoencoder.parameters(), lr=LR)
loss_func = nn.MSELoss()

for epoch in range(EPOCH):
    for step, (x, b_label) in enumerate(train_loader):
        b_x = x.view(-1, 3*344*344)   # batch x, shape (batch, 28*28)
        b_y = x.view(-1, 3*344*344)   # batch y, shape (batch, 28*28)

        encoded, decoded = autoencoder(b_x)

        loss = loss_func(decoded, b_y)      # mean square error
        optimizer.zero_grad()               # clear gradients for this
training step
        loss.backward()                     # backpropagation, compute
gradients
        optimizer.step()                    # apply gradients

###################################################
######## below is used to plot decoded pic ########

with torch.no_grad():
   for img, label in train_loader :
    fig = plt.figure()

)
    imggg = np.transpose(img[0],(1,2,0))
    ax1 = fig.add_subplot(121)
    ax1.imshow(imggg)

    if torch.cuda.is_available():
      img = Variable(img.to())
    else:
      img = Variable(img)

    encoded, decoded = autoencoder(img)

    decodeddd = np.transpose(decoded.cpu()[0],(1,2,0))

    ax2 = fig.add_subplot(122)
    ax2.imshow(decodeddd)

I expect the output of 2 pics but now only the original one shows, the decoded one doesn't show.

The training process works well, but I don't know what's the problem with picture's size.

Output:
decoder is returning a linear output of shape BATCH_SIZE x 355008. First, we need to reshape the second dimension to 3 dimensions of shape 3 x 344 x 344 before applying transpose on it. Replacing decodeddd with following should do the trick:

decodeddd = np.transpose(decoded.cpu()[0].view(3, 344, 344),(1,2,0))


Instruction: Add sum of squares of model parameters to loss function

Input:
Adding sum of squares of model weights is known to be a regularization method that prevents overfitting of the deep learning models.
Currently I have this loss function for my model which is implemented in pytorch:

lossfunc = nn.NLLLoss(ignore_index=0)

Then, I compute the loss during training as follows:

 ...
loss = lossfunc(out_perm, dec_pp[:,1:])

I want the regularized loss to look like:

loss = lossfunc(out_perm, dec_pp[:,1:])+sum_square_wights(enc)+sum_square_wights(dec)

But don't know how to implement sum_square_wights function. Any help will be appreciated.

Output:
I believe you are referring to  L2 regularization. If that is indeed the case, L2 regularization is already added to optimizer  in torch (SGD, Adam etc) and you can control it using a non-zero value for weight_decay in optimizer's parameter.

As for as L1 regularization is concerned, something like this should do the job:

l1_criterion = nn.L1Loss(size_average=False)
l1_reg_loss = 0
for param in model.parameters():
    l1_reg_loss += l1_criterion (param)

lambda = 0.0005
loss += lambda * l1_reg_loss


Instruction: Faster-RCNN Pytorch problem at prediction time with image dimensions

Input:
I am finetuning Faster-RCNN using PyTorch according to this tutorial: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

The results are pretty good but making predictions only work when feeding a single tensor to the model. For example:

# This works well
>>> img, _ = dataset_test[3]
>>> img.shape
torch.Size([3, 1200, 1600])
>>> model.eval()
>>> with torch.no_grad():
    .. preds = model([img.to(device)])

But when I feed multiple tensors at once I get that error:

>>> random_idx = torch.randint(high=50, size=(4,))
>>> images = torch.stack([dataset_test[idx][0] for idx in random_idx])
>>> images.shape
torch.Size([4, 3, 1200, 1600])
>>> with torch.no_grad():
    .. preds = model(images.to(device))
RuntimeError                              Traceback (most recent call last)
<ipython-input-101-52caf8fee7a4> in <module>()
      5 model.eval()
      6 with torch.no_grad():
----> 7   prediction =  model(images.to(device))

...

RuntimeError: The expanded size of the tensor (1600) must match the existing size (1066) at non-singleton dimension 2.  Target sizes: [3, 1200, 1600].  Tensor sizes: [3, 800, 1066]

Edit

Works when feeding a list of 3D tensors (IMO this behaviour is a bit weird, I cannot understand why it is not working with a 4D tensor):

>>> random_idx = torch.randint(high=50, size=(4,))
>>> images = [dataset_test[idx][0].to(device) for idx in random_idx]
>>> images.shape
torch.Size([4, 3, 1200, 1600])
>>> with torch.no_grad():
    .. preds = model(images)

Output:
MaskRCNN expects a list of tensors as 'input images' and a list of dictionaries as 'target' during training mode. This particular design choice is due to the fact that each image can have variable number of objects, i.e. target tensor of each image will be of variable dimensions, hence we are forced to use a list instead of a batch tensor of targets.

However, it is still not entirely necessary to use a list of image tensors instead of using a batch tensor. My guess is that they have gone with a list of tensors for images as well, for the sake of consistency. Also, this gives an added advantage of being able able to use images of variable sizes as input rather than a fixed size.

Due to this particular design choice, the model expects a list of tensors as input during evaluation mode as well.

As for as the speed performance of the model is concerned, this design choice might have some negative impact during evaluation, but I can not say with hundred percent conviction. However, during training, since we have variable dimension of target tensors for each image, we are forced to iterate over all images one by one for loss calculation. So, there would be no speed gain of using a batch tensor of images over a list of image tensors during training.


Instruction: Adding batch normalization decreases the performance

Input:
I'm using PyTorch to implement a classification network for skeleton-based action recognition. The model consists of three convolutional layers and two fully connected layers. This base model gave me an accuracy of around 70% in the NTU-RGB+D dataset. I wanted to learn more about batch normalization, so I added a batch normalization for all the layers except for the last one. To my surprise, the evaluation accuracy dropped to 60% rather than increasing But the training accuracy has increased from 80% to 90%. Can anyone say what am I doing wrong? or Adding batch normalization need not increase the accuracy?

The model with batch normalization

class BaseModelV0p2(nn.Module):

    def __init__(self, num_person, num_joint, num_class, num_coords):
        super().__init__()
        self.name = 'BaseModelV0p2'
        self.num_person = num_person
        self.num_joint = num_joint
        self.num_class = num_class
        self.channels = num_coords
        self.out_channel = [32, 64, 128]
        self.loss = loss
        self.metric = metric
        self.bn_momentum = 0.01

        self.bn_cv1 = nn.BatchNorm2d(self.out_channel[0], momentum=self.bn_momentum)
        self.conv1 = nn.Sequential(nn.Conv2d(in_channels=self.channels, out_channels=self.out_channel[0],
                                             kernel_size=3, stride=1, padding=1),
                                   self.bn_cv1,
                                    nn.ReLU(),
                                    nn.MaxPool2d(kernel_size=2, stride=2))

        self.bn_cv2 = nn.BatchNorm2d(self.out_channel[1], momentum=self.bn_momentum)
        self.conv2 = nn.Sequential(nn.Conv2d(in_channels=self.out_channel[0], out_channels=self.out_channel[1],
                                            kernel_size=3, stride=1, padding=1),
                                   self.bn_cv2,
                                nn.ReLU(),
                                nn.MaxPool2d(kernel_size=2, stride=2))

        self.bn_cv3 = nn.BatchNorm2d(self.out_channel[2], momentum=self.bn_momentum)
        self.conv3 = nn.Sequential(nn.Conv2d(in_channels=self.out_channel[1], out_channels=self.out_channel[2],
                                            kernel_size=3, stride=1, padding=1),
                                   self.bn_cv3,
                                  nn.ReLU(),
                                  nn.MaxPool2d(kernel_size=2, stride=2))

        self.bn_fc1 = nn.BatchNorm1d(256 * 2, momentum=self.bn_momentum)
        self.fc1 = nn.Sequential(nn.Linear(self.out_channel[2]*8*3, 256*2),
                                 self.bn_fc1,
                                 nn.ReLU(),
                                 nn.Dropout2d(p=0.5))  # TO check

        self.fc2 = nn.Sequential(nn.Linear(256*2, self.num_class))

    def forward(self, input):
        list_bn_layers = [self.bn_fc1, self.bn_cv3, self.bn_cv2, self.bn_cv1]
        # set the momentum  of the batch norm layers to given momentum value during trianing and 0 during evaluation
        # ref: https://discuss.pytorch.org/t/model-eval-gives-incorrect-loss-for-model-with-batchnorm-layers/7561
        # ref: https://github.com/pytorch/pytorch/issues/4741
        for bn_layer in list_bn_layers:
            if self.training:
                bn_layer.momentum = self.bn_momentum
            else:
                bn_layer.momentum = 0

        logits = []
        for i in range(self.num_person):
            out = self.conv1(input[:, :, :, :, i])

            out = self.conv2(out)

            out = self.conv3(out)

            logits.append(out)

        out = torch.max(logits[0], logits[1])
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)

        t = out

        assert not ((t != t).any())  # find out nan in tensor
        assert not (t.abs().sum() == 0)  # find out 0 tensor

        return out

Output:
My interpretation of the phenomenon you are observing,, is that instead of reducing the covariance shift, which is what the Batch Normalization is meant for, you are increasing it. In other words, instead of decrease the distribution differences between train and test, you are increasing it and that's what it is causing you to have a bigger difference in the accuracies between train and test. Batch Normalization does not assure better performance always, but for some problems it doesn't work well. I have several ideas that could lead to an improvement:

Increase the batch size if it is small, what would help the mean and std calculated in the Batch Norm layers to be more robust estimates of the population parameters.
Decrease the bn_momentum parameter a bit, to see if that also stabilizes the Batch Norm parameters.
I am not sure you should set bn_momentum to zero when test, I think you should just call model.train() when you want to train and model.eval() when you want to use your trained model to perform inference.
You could alternatively try Layer Normalization instead of Batch Normalization, cause it does not require accumulating any statistic and usually works well
Try regularizing a bit your model using dropout
Make sure you shuffle your training set in every epoch. Not shuffling the data set may lead to correlated batches that make the statistics in batch normalization cycle. That may impact your generalization
I hope any of these ideas work for you


Instruction: Backward algorithm Hidden Markov Model, 0th index (termination step) yields wrong result

Input:
I'm implementing a backward HMM algorithm in PyTorch. I used this link as reference. This link contains the results of the numerical example used (I am attempting to implement that and compare my generated results to it). Page 3, section 2. Backward probability, there is a table containing the calculated results.

Here is my code:

# Initial Transition matrix as shown in page 2 of above link
A = np.array([[0.6, 0.4], [0.3, 0.7]])
A = torch.from_numpy(A)
# Initial State Probability (page 2)
pi = np.array([0.8, 0.2])
pi = torch.from_numpy(pi)
# Output probabilities (page 2)
emission_matrix = np.array([[0.3, 0.4, 0.3, 0.3], [0.4, 0.3, 0.3, 0.3]])
emission_matrix = torch.from_numpy(emission_matrix)
# Initialize empty 2x4 matrix (dimensions of emission matrix)
backward = torch.zeros(emission_matrix.shape, dtype=torch.float64)

# Backward algorithm
def _backward(emission_matrix):
    # Initialization: A(i, j) * B(T, i) * B(Ot+1, j) , where B(Ot+1, j)  = 1
    backward[:, -1] = torch.matmul(A, emission_matrix[:, -1])
    # I reversed the emission matrix so as to start from the last column
    rev_emission_mat = torch.flip(emission_matrix[:, :-1], [1])
    # I transposed the reversed emission matrix such that each iterable in the for
    # loop is the observation sequence probability
    T_rev_emission_mat = torch.transpose(rev_emission_mat, 1, 0)
    # This step is so that I assign a reverse index enumeration to each iterable in the
    # emission matrix starts from time T to 0, rather than the opposite
    zipped_cols = list(zip(range(len(T_rev_emission_mat)-1, -1, -1), T_rev_emission_mat))

    for i, obs_prob in zipped_cols:
        # Induction: Σ A(i, j) * B(j)(Ot+1) * β(t+1, j)
        if i != 0:
            backward[:, i] = torch.matmul(A * obs_prob, backward[:, i+1])
    # Termination: Σ π(i) * bi * β(1, i)
    backward[:, 0] = torch.matmul(pi * obs_prob, backward[:, 1])

# run backward algorithm
_backward(emission_matrix)
# check results, backward is an all zero matrix that was initialized above
print(backward)
>>> tensor([[0.0102, 0.0324, 0.0900, 0.3000],
           [0.0102, 0.0297, 0.0900, 0.3000]], dtype=torch.float64)

As you can see, the 0-th index does not match the result in page 3 of the previous link. What did I do wrong? If there is anything I can clarify, please let me know. Thanks in advance!

Output: backward[:, 0] = pi * obs_prob * backward[:, 1]


Instruction: Reshaping Image for PyTorch

Input: I used to use keras and the image format it followed is [Height x Width x Channels x Samples]. i decided to switch to PyTorch. But i didn’t switch out my data loading schemes. So now i have numpy arrays of shape HxWxCxS, instead of SxCxHxW which is required for PyTorch. Does anyone have any idea to convert this ?

Output:
First, Keras format is (samples, height, width, channels).
All you need to do is a moved = numpy.moveaxis(data, -1,1)

If by luck you were using the non-default config "channels_first", then the config is identical to that of PyTorch, which is (samples, channels, height, width).

And when transforming to torch: data = torch.from_numpy(moved)


Instruction: Could I use BERT to Cluster phrases with pre-trained model

Input:
I found it was a failure that I had used Gensim with GoogleNews pre-trained model to cluster phrases like:

knitting
knit loom
loom knitting
weaving loom
rainbow loom
home decoration accessories
loom knit/knitting loom
...

I am advised that GoogleNews model does't have the phrases in it. The phrases I have are a little specific to GoogleNews model while I don't have corpus to train a new model. I have only the phrases. And now I am considering to turn to BERT. But could BERT do that as I expected as above? Thank you.

Output: You can feed a phrase into the pretrained BERT model and get an embedding, i.e. a fixed-dimension vector. So BERT can embed your phrases in a space. Then you can use a clustering algorithm (such as k-means) to cluster the phrases. The phrases do not need to occur in the training corpus of BERT, as long as the words they consist of are in the vocabulary. You will have to try to see if the embeddings give you relevant results.


Instruction: Model() got multiple values for argument 'nr_class' - SpaCy multi-classification model (BERT integration)

Input:
Hi I am working on implementing a multi-classification model (5 classes) with the new SpaCy Model en_pytt_bertbaseuncased_lg. The code for the new pipe is here:

nlp = spacy.load('en_pytt_bertbaseuncased_lg')
textcat = nlp.create_pipe(
    'pytt_textcat',
    config={
        "nr_class":5,
        "exclusive_classes": True,
    }
)
nlp.add_pipe(textcat, last = True)

textcat.add_label("class1")
textcat.add_label("class2")
textcat.add_label("class3")
textcat.add_label("class4")
textcat.add_label("class5")

The code for the training is as follows and is based on the example from here(https://pypi.org/project/spacy-pytorch-transformers/):

def extract_cat(x):
    for key in x.keys():
        if x[key]:
            return key

# get names of other pipes to disable them during training
n_iter = 250 # number of epochs

train_data = list(zip(train_texts, [{"cats": cats} for cats in train_cats]))

dev_cats_single   = [extract_cat(x) for x in dev_cats]
train_cats_single = [extract_cat(x) for x in train_cats]
cats = list(set(train_cats_single))
recall = {}
for c in cats:
    if c is not None:
        recall['dev_'+c] = []
        recall['train_'+c] = []

optimizer = nlp.resume_training()
batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)

for i in range(n_iter):
    random.shuffle(train_data)
    losses = {}
    batches = minibatch(train_data, size=batch_sizes)
    for batch in batches:
        texts, annotations = zip(*batch)
        nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)
    print(i, losses)

So the structure of my data looks like this:

[('TEXT TEXT TEXT',
  {'cats': {'class1': False,
    'class2': False,
    'class3': False,
    'class4': True,
    'class5': False}}), ... ]

I am not sure why I get the following error:

TypeError                                 Traceback (most recent call last)
<ipython-input-32-1588a4eadc8d> in <module>
     21
     22
---> 23 optimizer = nlp.resume_training()
     24 batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)
     25

TypeError: Model() got multiple values for argument 'nr_class'

EDIT:

if I take out the nr_class argument, I get this error here:

ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

I actually thought this would happen because I didn't specify the nr_class argument. Is that correct?

Output:
This is a regression in the most recent version we released of spacy-pytorch-transformers. Sorry about this!

The root cause is, this is another case of the evils of **kwargs. I'm looking forward to refining the spaCy API to prevent these issues in future.

You can see the offending line here: https://github.com/explosion/spacy-pytorch-transformers/blob/c1def95e1df783c69bff9bc8b40b5461800e9231/spacy_pytorch_transformers/pipeline/textcat.py#L71 . We provide the nr_class positional argument, which overlaps with the explicit argument you passed in during the config.

In order to workaround the problem, you can simply remove the nr_class key from your the config dict you're passing into spacy.create_pipe().


Instruction: SpaCy - ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

Input:
In relation to the previous post on stackoverflow
Model() got multiple values for argument 'nr_class' - SpaCy multi-classification model (BERT integration) in which my problem partialy have beed resolved I wanted to share the issue which comes up after implementing the solution.

if I take out the nr_class argument, I get this error here:

ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

I actually thought this would happen because I didn't specify the nr_class
 argument. Is this correct?

one more time my code for the multi-class model:

nlp = spacy.load('en_pytt_bertbaseuncased_lg')
textcat = nlp.create_pipe(
    'pytt_textcat',
    config={
        "nr_class":5,
        "exclusive_classes": True,
    }
)
nlp.add_pipe(textcat, last = True)

textcat.add_label("class1")
textcat.add_label("class2")
textcat.add_label("class3")
textcat.add_label("class4")
textcat.add_label("class5")

The code for the training is as follows and is based on the example from here(https://pypi.org/project/spacy-pytorch-transformers/):

def extract_cat(x):
    for key in x.keys():
        if x[key]:
            return key

# get names of other pipes to disable them during training
n_iter = 250 # number of epochs

train_data = list(zip(train_texts, [{"cats": cats} for cats in train_cats]))

dev_cats_single   = [extract_cat(x) for x in dev_cats]
train_cats_single = [extract_cat(x) for x in train_cats]
cats = list(set(train_cats_single))
recall = {}
for c in cats:
    if c is not None:
        recall['dev_'+c] = []
        recall['train_'+c] = []

optimizer = nlp.resume_training()
batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)

for i in range(n_iter):
    random.shuffle(train_data)
    losses = {}
    batches = minibatch(train_data, size=batch_sizes)
    for batch in batches:
        texts, annotations = zip(*batch)
        nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)
    print(i, losses)

So the structure of my data looks like this:

[('TEXT TEXT TEXT',
  {'cats': {'class1': False,
    'class2': False,
    'class3': False,
    'class4': True,
    'class5': False}}), ... ]

Output: As @Milla Well already commented the answer can be found here (the bug fix on github from @syllogism_)


Instruction: Pytorch Train & Eval Different Sample Sizes

Input:
I am learning pytorch, and have the following (abbreviated) code to setup for modeling:

# define the model class for a neural net with 1 hidden layer
class myNN(nn.Module):
    def __init__(self, D_in, H, D_out):
        super(myNN, self).__init__()
        self.lin1 = nn.Linear(D_in,H)
        self.lin2 = nn.Linear(H,D_out)
    def forward(self,X):
        return torch.sigmoid(self.lin2(torch.sigmoid(self.lin1(x))))

# now make the datasets & dataloaders
batchSize = 5
# Create the data class
class Data(Dataset):
    def __init__(self, x, y):
        self.x = torch.FloatTensor(x)
        self.y = torch.Tensor(y.astype(int))
        self.len = self.x.shape[0]
        self.p = self.x.shape[1]
    def __getitem__(self, index):
        return self.x[index], self.y[index]
    def __len__(self):
        return self.len
trainData = Data(trnX, trnY)
trainLoad = DataLoader(dataset = trainData, batch_size = batchSize)
testData = Data(tstX, tstY)
testLoad = DataLoader(dataset = testData, batch_size = len(testData))

# define the modeling objects
hiddenLayers = 30
learningRate = 0.1
model = myNN(p,hiddenLayers,1)
print(model)
optimizer = torch.optim.SGD(model.parameters(), lr = learningRate)
loss = nn.BCELoss()

with trnX.shape=(70, 2), trnY.shape=(70,), tstX.shape=(30,2), and tstY.shape=(30,). The code for training is:

# train!
epochs = 1000
talkFreq = 0.2
trnLoss = [np.inf]*epochs
tstLoss = [np.inf]*epochs
for i in range(epochs):
    # train with minibatch gradient descent
    for x, y in trainLoad:
        # forward step
        yhat = model(x)
        # compute loss (not storing for now, will do after minibatching)
        l = loss(yhat, y)
        # backward step
        optimizer.zero_grad()
        l.backward()
        optimizer.step()
    # evaluate loss on training set
    yhat = model(trainData.x)
    trnLoss[i] = loss(yhat, trainData.y)
    # evaluate loss on testing set
    yhat = model(testData.x)
    tstLoss[i] = loss(yhat, testData.y)

The datasets trainData and testData have 70 and 30 observations, respectively. This is probably just a newbie problem, but when I run the training cell, it errors on the trnLoss[i] = loss(yhat, trainData.y) line with the error

ValueError: Target and input must have the same number of elements. target nelement (70) != input nelement (5)

When I inspect the output of the yhat=model(trainData.x) line, I see that yhat is a tensor with batchSize elements, despite the fact that trainData.x.shape = torch.Size([70, 2]).

How can I iteratively train the model with the minibatch gradient descent, then use the model to compute the loss & accuracy on the full training & testing sets? I tried setting model.train() just before the mini batch iteration, then model.eval() just before the evaluation code, to no avail.

Output:
In myNN.forward(), you are passing lower case x as input to self.lin1 while input parameter to the function is named as capital case X. Small case x is kind of global variable defined at for loop for trainload hence you are not getting any syntax error, but the value that you intend to pass is not being passed to self.lin1.

Might I also suggest that you should consider using model.eval() and with torch.no_grad() for your testing code. It is not absolutely necessary here but it will make more sense.


Instruction: Can't install pytorch with pip on Windows

Input:
I'm trying to install Pytorch with Windows and I'm using the commands of the official site
https://pytorch.org/get-started/locally/

pip3 install torch==1.2.0 torchvision==0.4.0 -f https://download.pytorch.org/whl/torch_stable.html

This is the command if I choose Windows, Cuda 10.0, and Python 3.7
But if I run this I get the error message:

ERROR: Could not find a version that satisfies the requirement torch==1.2.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.2.0

So why does this happen?
My pip is version 19.2 and I am in a newly installed python 3.7 environment

Output:
The most likely reason for Your issue is a 32-bit installation of python, while the torch libraries rely on having a 64-bit version. I had exactly the same issue.

Just start python from command line and observe

C:\Users\marci>python
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32

My installation now shows 64 bits. If Yours shows 32, then install 64-bit python. I used this link: Official python 64-bit Windows installer


Instruction: Getting error while trying to plot ROC for multiclass classification

Input:
I am trying to plot ROC Curve for multiclass classification.I followed https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html.
I used the below code to calculate y_test,and y_score

def test_epoch(net,test_loader):
 y_test =[]
 y_score =[]
 with torch.no_grad():
  for batch in test_loader:
    images, labels = batch['image'], batch['grade']
    images =Variable(images)
    labels= Variable(labels)
    target =F.one_hot(labels,5)
    outputs = net(images)
    _, predicted = torch.max(outputs.data, 1)
    c = (predicted == labels).squeeze().numpy()
    y_score.append(outputs.numpy())
    y_test.append(labels.numpy())
    return y_test,y_score

I seen that my y_test is array like below

y_test data>> [array([[0, 0, 1, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 1, 0, 0, 0],
       [1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 0, 0, 1, 0],
       [0, 0, 1, 0, 0],
       [1, 0, 0, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 1, 0, 0, 0]

And y_score is like

[array([[ 0.30480504, -0.12213976,  0.09632117, -0.16465648, -0.44081157],[ 0.21797988, -0.09650452,  0.07616544, -0.12001953, -0.34972644],[ 0.3230184 , -0.13098559,  0.10277118, -0.17656785, -0.45888817],[ 0.38143447, -0.15880316,  0.12123139, -0.21719441, -0.5281661 ],[ 0.3427343 , -0.13945231,  0.11076729, -0.19657779, -0.4913683 ]

Whenever I called the function for plotting ROC curve

def plot_roc(y_test, y_score, N_classes):
    """
    compute ROC curve and ROC area for each class in each fold

    """

    fpr = dict()
    tpr = dict()
    roc_auc = dict()
    for i in range(N_classes):
        fpr[i], tpr[i], _ = roc_curve(np.array(y_test[:, i]),np.array(y_score[:, i]))
        roc_auc[i] = auc(fpr[i], tpr[i])
    # Compute micro-average ROC curve and ROC area
    fpr["micro"], tpr["micro"], _ = roc_curve(y_test.ravel(), y_score.ravel())
    roc_auc["micro"] = auc(fpr["micro"], tpr["micro"])

    # Compute macro-average ROC curve and ROC area

    # First aggregate all false positive rates
    all_fpr = np.unique(np.concatenate([fpr[i] for i in range(N_classes)]))

    # Then interpolate all ROC curves at this points
    mean_tpr = np.zeros_like(all_fpr)
    for i in range(N_classes):
        mean_tpr += interp(all_fpr, fpr[i], tpr[i])

    # Finally average it and compute AUC
    mean_tpr /= N_classes

    fpr["macro"] = all_fpr
    tpr["macro"] = mean_tpr
    roc_auc["macro"] = auc(fpr["macro"], tpr["macro"])

    # Plot all ROC curves
    plt.figure()
    plt.plot(fpr["micro"], tpr["micro"],
             label='micro-average ROC curve (area = {0:0.2f})'
                   ''.format(roc_auc["micro"]),
             color='deeppink', linestyle=':', linewidth=4)

    plt.plot(fpr["macro"], tpr["macro"],
             label='macro-average ROC curve (area = {0:0.2f})'
                   ''.format(roc_auc["macro"]),
             color='navy', linestyle=':', linewidth=4)

    colors = cycle(['aqua', 'darkorange', 'cornflowerblue'])
    for i, color in zip(range(N_classes), colors):
        plt.plot(fpr[i], tpr[i], color=color, lw=2,
                 label='ROC curve of class {0} (area = {1:0.2f})'
                       ''.format(i, roc_auc[i]))

    plt.plot([0, 1], [0, 1], 'k--', lw=2)
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.title('Some extension of Receiver operating characteristic to multi-class')
    plt.legend(loc="lower right")
    plt.show()

I am getting this error massages,

 Traceback (most recent call last):
  File "/home/Downloads/demo 3.py", line 405, in <module>
    plot_roc(y_test, y_score, 5)
  File "/home/Downloads/demo 3.py", line 225, in plot_roc
    fpr[i], tpr[i], _ = roc_curve(np.array(y_test[:, i]),np.array(y_score[:, i]))
TypeError: list indices must be integers or slices, not tuple

I could not understand how I will solve this problem.
I am highly appreciate any kind of help regarding this issue.

Output: In your code you have a previously defined variable (a list) called roc_curve, and this shadows the scikit-learn function sklearn.metrics.roc_curve in your code, you should prefer not to name variables the same as a well known function, in order to prevent problems like these.


Instruction: How to get sentence embeddings from encoder in Fastai learner language model

Input:
I was able to fine tune a language model using fast ai.  I would like to extract sentence embeddings from the fine-tuned model for sentence similarity.  How do I get the encoder model embeddings? Also can embeddings be compared with dot product like other embeddings from other models such as USE?

data_lm = TextLMDataBunch.from_df(train_df = se1, valid_df = se2, path = "",text_cols='text')
learn = language_model_learner(data_lm,drop_mult=0.7,pretrained=True,arch=AWD_LSTM)
learn.fit_one_cycle(3, 1e-01)

My code is above how can I get encodings from learn?

Output:
This should give you the encoder(Which is an embedding layer) :

learn.model[0].encoder


Instruction: Unable to figure out inplace operation in the pytorch code?

Input:
I have the following implementation in PyTorch for learning using LSTM:

https://gist.github.com/rahulbhadani/f1d64042cc5a80280755cac262aa48aa

However, the code is experiencing in-place operation error

where my error output is:

/home/ivory/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:10: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  # Remove the CWD from sys.path while we load stuff.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-86-560ec78f2b64> in <module>
     27 linear = torch.nn.Linear(hidden_nums, output_dim)
     28
---> 29 global_loss_list = global_training(lstm2)

<ipython-input-84-152890a3028c> in global_training(optimizee)
      3     adam_global_optimizer = torch.optim.Adam([{'params': optimizee.parameters()},
      4                                      {'params':linear.parameters()}], lr = 0.0001)
----> 5     _, global_loss_1 = learn2(LSTM_Optimizee, training_steps, retain_graph_flag=True, reset_theta=True)
      6
      7     print(global_loss_1)

<ipython-input-83-0357a528b94d> in learn2(optimizee, unroll_train_steps, retain_graph_flag, reset_theta)
     43             # requires_grad=True. These are accumulated into x.grad for every
     44             # parameter x. In pseudo-code: x.grad += dloss/dx
---> 45             loss.backward(retain_graph = retain_graph_flag) #The default is False, when the optimized LSTM is set to True
     46
     47             print('x.grad: {}'.format(x.grad))

~/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--> 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119
    120     def register_hook(self, hook):

~/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 10]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I have tried to trace the error but didn't succeed. Any help in this regard will be appreciated.

Thanks.

Output:
I think the issue is with the following line:

global_loss_list.append(global_loss.detach_())

The convention in PyTorch for in-place operations is using _ at the end of the function name (as in detach_). I believe you shouldn't be detaching in-place. In other words, change detach_ to detach


Instruction: How to take a transpose for each matrix in a batch in Pytorch?

Input: Say I have a 4 batch of 5x3 matrixes. So the dimensions of these tensor are 4x5x3. How do I take the transpose of each matrix within each batch. So converting it to 4x3x5?

Output:
I will drop some benchmarks here for the sake of performance. Using the same tensor proposed in the OP's answer.

In[2]: import torch
In[3]: x = torch.randn(2, 3, 5)
In[4]: x.size()
Out[4]: torch.Size([2, 3, 5])
In[5]: %timeit x.permute(1, 0, 2)
1.03 µs ± 41.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In[6]: %timeit torch.transpose(x, 0, 1)
892 ns ± 9.61 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In[7]: torch.transpose(x, 0, 1).equal(x.permute(1, 0, 2))
Out[7]: True

It is clear that torch.transpose is faster, so It is advised to use it when possible.


Instruction: Can the coordinates of a 2 dimensional pytorch tensor be remapped?

Input: I'm trying to remap a 2d image to new coordinates. Previously I was using OpenCV's cv.remap method. I'd like to use CUDA with compiling C directly. Is there a Pytorch version of remap that will interpolate values?

Output:
Are you looking for grid_sample?
Look for make_grid function for additional information


Instruction: Does pytorch apply softmax automatically in nn.Linear

Input:
In pytorch a classification network model is defined as this,

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        self.out = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.out(x)
        return x

Is softmax applied here? In my understanding, things should be like,

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        self.relu =  torch.nn.ReLu(inplace=True)
        self.out = torch.nn.Linear(n_hidden, n_output)   # output layer
        self.softmax = torch.nn.Softmax(dim=n_output)
    def forward(self, x):
        x = self.hidden(x)      # activation function for hidden layer
        x = self.relu(x)
        x = self.out(x)
        x = self.softmax(x)
        return x

I understand that F.relu(self.relu(x)) is also applying relu, but the first block of code doesn't apply softmax, right?

Output:
Latching on to what @jodag was already saying in his comment, and extending it a bit to form a full answer:

No, PyTorch does not automatically apply softmax, and you can at any point apply torch.nn.Softmax() as you want. But, softmax has some issues with numerical stability, which we want to avoid as much as we can. One solution is to use log-softmax, but this tends to be slower than a direct computation.

Especially when we are using Negative Log Likelihood as a loss function (in PyTorch, this is torch.nn.NLLLoss, we can utilize the fact that the derivative of (log-)softmax+NLLL is actually mathematically quite nice and simple, which is why it makes sense to combine the both into a single function/element. The result is then torch.nn.CrossEntropyLoss. Again, note that this only applies directly to the last layer of your network, any other computation is not affected by any of this.


Instruction: How can I do the centercrop of 3D volumes inside the network model with pytorch

Input:
In keras, there is Cropping3D layer for centercropping tensors of 3D volumnes inside the neural network. However, I failed to find out anything similar in pytorch, though they have torchvision.transforms.CenterCrop(size) for 2D images.

How can I do the cropping inside the network? Otherwise I need to do it in preprocessing which is the last thing I want to do for specific reasons.

Do I need to write a custom layer like slicing the input tensors along each axices? Hope to get some inspiration for this

Output:
In PyTorch you don't necessarily need to write layers for everything, often you can just do what you want directly during the forward pass. The basic rules you need to keep in mind when operating on torch tensors for which you will need to compute gradients are

Don't convert torch tensors to other types for computation (e.g. use torch.sum instead of converting to numpy and using numpy.sum).
Don't perform in-place operations (e.g. changing one element of a tensor or using inplace operators, so use x = x + ... instead of x += ...).

That said, you can just use slicing, maybe it would look something like this

def forward(self, x):
    ...
    x = self.conv3(x)
    x = x[:, :, 5:20, 5:20]    # crop out part of the feature map
    x = self.relu3(x)
    ...


Instruction: How to perform element-wise product in PyTorch?

Input:
I have two torch tensors a and b. Tensor a has the shape of [batch_size, emb_size] and Tensor b has the shape of [num_of_words, emb_size]. I want to do the element-wise product on these two tensors instead of dot product.

I noticed that "*" can perform element-wise product but it doesn't fit my case.

For example, batch_size = 3, emb_size = 2, num_of_words = 5.

a = torch.rand((3,2))
b = torch.rand((5,2))

I want to get something like:

torch.cat([a[0]*b, a[1]*b, a[2]*b]).view(3, 5, 2)

but I want to do this in an efficient and elegant way.

Output:
You can use

a.unsqueeze(1) * b

PyTorch supports broadcasting semantics but you need to make sure the singleton dimensions are in the correct locations.


Instruction: Libtorch: cannot load traced lstm scriptmodel

Input:
I save a pytorch ScriptModule and load it using libtorch. However I encountered the following problem

I use linux subsystem under win10 and I use pytorch 1.2.

To reproduce my problem, you could run this piece of python code to save a pt model

import torch
import torch.nn as nn

# TODO: https://github.com/pytorch/pytorch/issues/23930
class test(torch.jit.ScriptModule):

    def __init__(self, vocab_size=10, rnn_dims=512):
        super().__init__()
        self.word_embeds = nn.Embedding(vocab_size, rnn_dims)
        self.emb_drop = nn.Dropout(0.1)
        self.rnn = nn.LSTM(input_size=rnn_dims, hidden_size=rnn_dims, batch_first=True,
                           num_layers=2, dropout=0.1)
        # delattr(self.rnn, 'forward_packed')

    @torch.jit.script_method
    def forward(self, x):
        h1 = (torch.zeros(2, 1, 512), torch.zeros(2, 1, 512))
        embeds = self.emb_drop(self.word_embeds(x))
        out, h1 = self.rnn(embeds, h1)

        return h1

model = test()

input = torch.ones((1,3)).long()
output = model(input)
print('output', output)

# torch.onnx.export(model,  # model being run
#                   input,
#                   'test.onnx',
#                   example_outputs=output)
#torch.jit.trace(model, (torch.ones((1,3)).long(), torch.ones((3,1))), check_trace=False)
model.save('lstm_test.pt')

Then load the model in libtorch.

I don't know why I have this error. I don't use PackedSequence at all. Hope some could help me out.

Output: I know what's wrong now. The libtorch version is of wrong version on the official website. It's ok now when I use the correct libtorch 1.2. Refer to issue https://github.com/pytorch/pytorch/issues/24382


Instruction: Concatenation step of U-Net for unequal number of channels

Input:
I am trying to implement the U-NET architecture for image segmentation while implementing the crop and concatenation step in the expansive path, I am unable to understand how the unequal number of channels are concatenated.

According to the architecture, the input from the first upsampling step has to be concatenated from the corresponding output from contracting path but the problem is number of channels in contracting path is 512 while after upsampling step they are 1024, how they are supposed to be concatenated.My code for the crop and concatenate is -

def crop_and_concat(self, upsampled, bypass, crop=False):
    if crop:
        c = (bypass.size()[2] - upsampled.size()[2]) // 2
        bypass = F.pad(bypass, (-c, -c, -c, -c))
    return torch.cat((upsampled, bypass), 1)

The error I am receiving-
RuntimeError: Given groups=1, weight of size 128 256 5 5, expected input[4, 384, 64, 64] to have 256 channels, but got 384 channels instead
Where I am doing wrong?

Output:
First of all, you don't have to be so strict when it comes to U-Net like architectures, there were many derivatives afterwards (see for example fastai variation with PixelShuffle).
In the case of encoder, in the basic version, your channels go (per block):
1 - 64 - 128 - 256 - 512

Standard convolutional encoder.
After that is a shared layer of  1024.
In decoder, it goes downwards, but has more channels as you are concatenating encoder states from each block.
It would be:

1024 -> 512 -> 512 (decoder) + 512 (encoder), 1024 total -> 512
512 -> 256 -> 256 (decoder) + 256 (encoder), 512 total -> 256

and so on.
You were at the case where 256 from decoder was taken in the account, but 128 added from encoder wasn't. Just up your channels to 256 + 128 and follow the above scheme for each block of your UNet.


Instruction: How do they know mean and std, the input value of transforms.Normalize

Input:
The question is about the data loading tutorial from the PyTorch website. I don't know how they write the value of mean_pix and std_pix of the in transforms.Normalize without calculation

I'm unable to find any explanation relevant to this question on StackOverflow.

import torch
from torchvision import transforms, datasets

data_transform = transforms.Compose([
        transforms.RandomSizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])
hymenoptera_dataset = datasets.ImageFolder(root='hymenoptera_data/train',
                                           transform=data_transform)
dataset_loader = torch.utils.data.DataLoader(hymenoptera_dataset,
                                             batch_size=4, shuffle=True,
                                             num_workers=4)

The value mean=[0.485,0.456, 0.406] and std=[0.229, 0.224, 0.225] is not obvious to me. How do they get them? And why are they equal to these?

Output:
For normalization input[channel] = (input[channel] - mean[channel]) / std[channel], the mean and standard deviation values are to be taken from the training dataset.

Here, mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225] are the mean and std of Imagenet dataset.

  On Imagenet, we’ve done a pass on the dataset and calculated per-channel mean/std.  check here

The pre-trained models available in torchvision for transfer learning were pretrained on Imagenet, so using its mean and std deviation would be fine for fine-tuning your model.

If you're trying to train your model from scratch, it would be better to use the mean and std deviation of your training dataset (face dataset in this case). Other than that, in most of the cases, the mean and std of Imagenet suffice for your problem.


Instruction: How To Fix: RuntimeError: size mismatch in pyTorch

Input:
I am new to pyTorch and getting following Size Mismatch error:

RuntimeError: size mismatch, m1: [7 x 2092500], m2: [180 x 120] at ..\aten\src\TH/generic/THTensorMath.cpp:961

Model:

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 200, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(200, 180, 5)
    self.fc1 = nn.Linear(180, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84,5)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(x.shape[0], -1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

How ever I tried changing x = x.view(x.shape[0], -1) to x = x.view(x.size(0), -1) but that also did'nt work. Dimension of images is 512x384. and have used following transformation:

def load_dataset():
data_path = './dataset/training'

transform = transforms.Compose(
               [transforms.Resize((512,384)),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_dataset = torchvision.datasets.ImageFolder(root=data_path,transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=7,num_workers=0,shuffle=True)

return train_loader

Output:
The problem is that the dimensions of the output of your last max pooling layer don't match the input of the first fully connected layer. This is the network structure until the last max pool layer for input shape (3, 512, 384):

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1        [-1, 200, 508, 380]          15,200
         MaxPool2d-2        [-1, 200, 254, 190]               0
            Conv2d-3        [-1, 180, 250, 186]         900,180
         MaxPool2d-4         [-1, 180, 125, 93]               0
================================================================

The last row of the table means that MaxPool2d-4 outputs 180 channels (filter outputs) of 125 width and 93 height. So you need your first fully connected layer to have 180 * 125 * 93 = 2092500 input size. This is a lot, so I'd advise you to refine your architecture. In any case, if you change the input size of the first fully connected layer to 2092500, it works:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 200, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(200, 180, 5)
        #self.fc1 = nn.Linear(180, 120)
        self.fc1 = nn.Linear(2092500, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84,5)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(x.shape[0], -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

Giving the following architecture:

----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1        [-1, 200, 508, 380]          15,200
         MaxPool2d-2        [-1, 200, 254, 190]               0
            Conv2d-3        [-1, 180, 250, 186]         900,180
         MaxPool2d-4         [-1, 180, 125, 93]               0
            Linear-5                  [-1, 120]     251,100,120
            Linear-6                   [-1, 84]          10,164
            Linear-7                    [-1, 5]             425
================================================================
Total params: 252,026,089
Trainable params: 252,026,089
Non-trainable params: 0

(You can use the torchsummary package to generate these tables.)


Instruction: Augmenting only the training set in K-folds cross validation

Input:
I am trying to create a binary CNN classifier for an unbalanced dataset (class 0 = 4000 images, class 1 = around 250 images), which I want to perform 5-fold cross validation on. Currently I am loading my training set into an ImageLoader that applies my transformations/augmentations(?) and loads it into a DataLoader. However, this results in both my training splits and validation splits containing the augmented data.
I originally applied transformations offline (offline augmentation?) to balance my dataset, but from this thread (https://stats.stackexchange.com/questions/175504/how-to-do-data-augmentation-and-train-validate-split), it seems it would be ideal to only augment the training set. I would also prefer to train my model on solely augmented training data and then validate it on non-augmented data in a 5-fold cross validation
My data is organized as root/label/images, where there are 2 label folders (0 and 1) and images sorted into the respective labels.
My Code So Far
total_set = datasets.ImageFolder(ROOT, transform = data_transforms['my_transforms'])

//Eventually I plan to run cross-validation as such:
splits = KFold(cv = 5, shuffle = True, random_state = 42)

for train_idx, valid_idx in splits.split(total_set):
    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(total_set, batch_size=32, sampler=train_sampler)
    val_loader = torch.utils.data.DataLoader(total_set, batch_size=32, sampler=valid_sampler)

model.train()
//Model train/eval works but may be overpredict

I'm sure I'm doing something sub-optimally or wrong in this code, but I can't seem to find any documentation on specifically augmenting only the training splits in cross-validation!
Any help would be appreciated!

Output:
One approach is to implement a wrapper Dataset class that applies transforms to the output of your ImageFolder dataset. For example

class WrapperDataset:
    def __init__(self, dataset, transform=None, target_transform=None):
        self.dataset = dataset
        self.transform = transform
        self.target_transform = target_transform

    def __getitem__(self, index):
        image, label = self.dataset[index]
        if self.transform is not None:
            image = self.transform(image)
        if self.target_transform is not None:
            label = self.target_transform(label)
        return image, label

    def __len__(self):
        return len(self.dataset)

Then you could use this in your code by wrapping the larger dataset with different transforms.

total_set = datasets.ImageFolder(ROOT)

# Eventually I plan to run cross-validation as such:
splits = KFold(cv = 5, shuffle = True, random_state = 42)

for train_idx, valid_idx in splits.split(total_set):
    train_sampler = SubsetRandomSampler(train_idx)
    valid_sampler = SubsetRandomSampler(valid_idx)

    train_loader = torch.utils.data.DataLoader(
        WrapperDataset(total_set, transform=data_transforms['train_transforms']),
        batch_size=32, sampler=train_sampler)
    valid_loader = torch.utils.data.DataLoader(
        WrapperDataset(total_set, transform=data_transforms['valid_transforms']),
        batch_size=32, sampler=valid_sampler)

    # train/validate now

I haven't tested this code since I don't have your full code/models but the concept should be clear.


Instruction: RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Input:
I'm working with the project 'lda2vec-pytorch' on Google CoLab,
runnin pytorch 1.1.0

https://github.com/TropComplique/lda2vec-pytorch

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
cuda:0

I'm getting an exception in the forward method adding 'noise' in my
class negative_sampling_loss(nn.Module):

        noise = self.multinomial.draw(batch_size*window_size*self.num_sampled)
        noise = Variable(noise).view(batch_size, window_size*self.num_sampled)

        device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.embedding = self.embedding.to(device)
        #print("negative_sampling_loss::forward() self.embedding", self.embedding.is_cuda)  This line get's an error.

        # shape: [batch_size, window_size*num_sampled, embedding_dim]
        noise = self.embedding(noise)  # Exception HERE

Here's the stack trace:

Traceback (most recent call last):
  File "train.py", line 36, in <module>
    main()
  File "train.py", line 32, in main
    save_every=20, grad_clip=5.0
  File "../utils/training.py", line 138, in train
    neg_loss, dirichlet_loss = model(doc_indices, pivot_words, target_words)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../utils/lda2vec_loss.py", line 82, in forward
    neg_loss = self.neg(pivot_words, target_words, doc_vectors, w)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "../utils/lda2vec_loss.py", line 167, in forward
    noise = self.embedding(noise)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

Any ideas?

Output:
Variable noise is available on CPU while self.embedding is on GPU. We can send noise to GPU as well:

noise = noise.to(device)


Instruction: Plotting predicted values causes error: 'Tensor' object has no attribute 'ndim'

Input:
How do I properly write the code to plot the predicted values in this linear regression model?

I am using this tutorial to learn Linear regression:https://www.deeplearningwizard.com/deep_learning/practical_pytorch/pytorch_linear_regression/

I was able to successfully implement GPU. My issue is plotting the predicted values. I tried searching for solutions to learn how to keep the values as a tensor, but it seems that I don't have the syntactical knowledge to do this.

Here is where I'll start

epochs = 100
for epoch in range(epochs):
    epoch += 1

    # Convert numpy array to torch Variable
    if torch.cuda.is_available():
        inputs = (torch.from_numpy(x_train).cuda())
        labels = (torch.from_numpy(y_train).cuda())
    else:
        inputs = (torch.from_numpy(x_train))
        labels = (torch.from_numpy(y_train))

    # Clear gradients w.r.t. parameters
    optimizer.zero_grad()

    # Forward to get output
    outputs = model(inputs)

    # Calculate Loss
    loss = criterion(outputs, labels)

    # Getting gradients w.r.t. parameters
    loss.backward()

    # Updating parameters
    optimizer.step()

    # Logging
    print('epoch {}, loss {}'.format(epoch, loss.item()))

Prediction is made here and I choose to use cuda

predicted = model(Variable(torch.from_numpy(x_train).requires_grad_().cuda()))

print("Predicted")
print(predicted)
print("Output")
print(y_train)

plt.clf()

# Get predictions

#predicted = model(Variable(torch.from_numpy(x_train).requires_grad_().cuda()))

# Plot true data
plt.plot(x_train, y_train, 'go', label='True data', alpha=0.5)

The error is called here after it is unable to plot

# Plot predictions
plt.plot(x_train, predicted, '--', label='Predictions', alpha=0.5)

# Legend and plot
plt.legend(loc='best')
plt.show()

Error Given:

Traceback (most recent call last):
  File "D:/Test with GPU/Linear regression.py", line 101, in <module>
    plt.plot(x_train, predicted, '--', label='Predictions', alpha=0.5)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\pyplot.py", line 2795, in plot
    is not None else {}), **kwargs)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_axes.py", line 1666, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 225, in __call__
    yield from self._plot_args(this, kwargs)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 391, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 271, in _xy_from_xy
    if x.ndim > 2 or y.ndim > 2:
AttributeError: 'Tensor' object has no attribute 'ndim'

Output:
plt.plot function expects its inputs to be numpy arrays, rather than torch.tensor.
You can use .numpy() to view the internal data of the tensor as a numpy array.

Try

with torch.no_grad():
  plt.plot(x_train, predicted.cpu().numpy(), '--', label='Predictions', alpha=0.5)


Instruction: Does torch.utils.tensorboard need installation of Tensorflow?

Input: I'm new to pytorch and I wonder if using tensorboard on pytorch needs tensorflow as a dependency. Moreover, except for tensorboard, what are other options for training curve visualization?

Output:
Indeed tensorboard is part of tensorflow, but it does not depend on it.
You can safely install only tensorboard (e.g, pip install tensorboard) or conda install -c conda-forge tensorboard) and use torch.utils.tensorboard.


Instruction: Filling torch tensor with zeros after certain index

Input:
Given a 3d tenzor, say:
batch x sentence length x embedding dim

a = torch.rand((10, 1000, 96))

and an array(or tensor) of actual lengths for each sentence

lengths =  torch .randint(1000,(10,))

outputs tensor([ 370., 502., 652., 859., 545., 964., 566., 576.,1000., 803.])

How to fill tensor ‘a’ with zeros after certain index along dimension 1 (sentence length) according to tensor ‘lengths’ ?

I want smth like that :

a[ : , lengths : , : ]  = 0

One way of doing it (slow if batch size is big enough):

for i_batch in range(10):
    a[ i_batch  , lengths[i_batch ] : , : ]  = 0

Output:
You can do it using a binary mask.
Using lengths as column-indices to mask we indicate where each sequence ends (note that we make mask longer than a.size(1) to allow for sequences with full length).
Using cumsum() we set all entries in mask after the seq len to 1.
mask = torch.zeros(a.shape[0], a.shape[1] + 1, dtype=a.dtype, device=a.device)
mask[(torch.arange(a.shape[0]), lengths)] = 1
mask = mask.cumsum(dim=1)[:, :-1]  # remove the superfluous column
a = a * (1. - mask[..., None])     # use mask to zero after each column

For a.shape = (10, 5, 96), and lengths = [1, 2, 1, 1, 3, 0, 4, 4, 1, 3].
Assigning 1 to respective lengths at each row, mask looks like:
mask =
tensor([[0., 1., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0., 0.],
        [0., 1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0., 0.],
        [0., 0., 0., 1., 0., 0.],
        [1., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1., 0.],
        [0., 1., 0., 0., 0., 0.],
        [0., 0., 0., 1., 0., 0.]])

After cumsum you get
mask =
tensor([[0., 1., 1., 1., 1.],
        [0., 0., 1., 1., 1.],
        [0., 1., 1., 1., 1.],
        [0., 1., 1., 1., 1.],
        [0., 0., 0., 1., 1.],
        [1., 1., 1., 1., 1.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 1.],
        [0., 1., 1., 1., 1.],
        [0., 0., 0., 1., 1.]])

Note that it exactly has zeros where the valid sequence entries are and ones beyond the lengths of the sequences. Taking 1 - mask gives you exactly what you want.
Enjoy ;)


Instruction: Pytorch gradients has not calculated

Input:
I create a NN. I'm having a problem with recounting gradients. The problem is that I scalarly multiply 2 tensors u @ v and normalize one of them. It is important that gradients cannot be calculated for h. Therefore, I use detach(). In addition, during the recalculation of gradients, normalization should not be taken into account (I do not know how to do this).

import torch
from torch import nn

class Nn(nn.Module):
    def __init__(self):
        super(Nn, self).__init__()
        self.ln = nn.Linear(5, 5)

    def forward(self, x):
        v = self.ln(x)

        u = v.clone()
        h = v.clone()

        u /= u.norm()
        h = h.detach()
        h /= h.norm()

        res = torch.stack([torch.stack([u @ h, u @ h])])

        return res

def patches_generator():
    while True:
        decoder = torch.rand((5, ))
        target = torch.randint(2, (1,))
        yield decoder, target

net = Nn()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters())

net.train()
torch.autograd.set_detect_anomaly(True)
for decoder, targets in patches_generator():
    optimizer.zero_grad()
    outputs = net(decoder)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()

As a result, I get the following error:

  RuntimeError: one of the variables needed for gradient computation has
  been modified by an inplace operation: [torch.FloatTensor [9, 512, 1,
  1]], which is output 0 of ReluBackward1, is at version 3; expected
  version 2 instead. Hint: the backtrace further above shows the
  operation that failed to compute its gradient. The variable in
  question was changed in there or anywhere later. Good luck!

Output:
The problem is the in-place division operator applied to u in this line:

u /= u.norm()

changing it to

u = u / u.norm()

makes the code run. The reason is that the in-place operator overwrites the intermediate result from this line

u = v.clone()

which makes it impossible for Pytorch to compute the gradient.

(The error message in the question contains a reference to a ReluBackward1 layer which is not in the reduced code example. Pytorch ReLU layers have an optional in_place argument which makes the operation in place while supporting backprop. This often works, because in a sequential network there is no need to distinguish between the output of the ReLU activation and the output of the weights to compute the gradient, but in more complex architectures it might be necessary to retain the output of the weights.)


Instruction: Pytorch batch matrix-matrix outer product

Input:
Similarly to the question in Pytorch batch matrix vector outer product I have two matrices and would like to compute their outer product, or in other words the pairwise elementwise product.

Shape example:
If we have X1 and X2 of shapes of torch.Size([32, 300, 8])
The result should be of size torch.Size([32, 300, 300, 8])

Output:
You can add singleton dimensions:

X1[:, None, ...] * X1[..., None, :]

But Usman Ali's comment is also a good idea. Use torch.einsum:

torch.einsum('bik,bjk->bijk', X1, X2)


Instruction: ImportError: No module named nibabel in python 2.7

Input:
I am using python 2.7 and I used

pip install nibabel

but I constantly get the error

ImportError: No module named nibabel.

when I type

pip show nibabel

I get this result so it seems that it is installed but still get the aforementioned error when using import nibabel as nib

    Name: nibabel
Version: 2.5.0
Summary: Access a multitude of neuroimaging data formats
Home-page: https://nipy.org/nibabel
Author: nibabel developers
Author-email: neuroimaging@python.org
License: MIT License
Location: /usr/local/lib/python3.5/dist-packages
Requires: six, numpy
Required-by:

Here the address shows it is installed in python3 path! How can I address this problem?

Output:
Install it using pip2:

pip2 install nibabel

Or explicitly call the pip module for Python 2.7:

python2.7 -m pip install nibabel

for example.


Instruction: Why my tensor defined in forward function can not be transformed in to cuda variable autonomously?

Input:
In PyTorch, in the forward function of a model

class my_model(torch.nn.module):

    ......

    def forward():
        self.a=torch.zeros(1)
        #blabla

After model.cuda(), why self.a is still a cpu variable?

Output:
This is so by design.

Only the tensors which are a part of the model will move with model.cuda() or model.to("cuda").

These tensors are registered with register_parameter or register_buffer. This also includes child modules, parameters and buffers registered with the aforementioned functions.

Even though self.a=torch.zeros(1) is actually a part of the class itself, by design it will not be moved to CUDA, instead you would need to do a.to("cuda"), if you haven't used the register* methods.


Instruction: What's the purpose of torch.autograd.Variable?

Input:
I load features and labels from my training dataset. Both of them are originally numpy arrays, but I change them to the torch tensor using torch.from _numpy(features.copy()) and torch.tensor(labels.astype(np.bool)).

And I notice that torch.autograd.Variable is something like placeholder in tensorflow.

When I train my network, first I tried

features = features.cuda()
labels = labels.cuda()

outputs = Config.MODEL(features)
loss = Config.LOSS(outputs, labels)

Then I tried

features = features.cuda()
labels = labels.cuda()

input_var = Variable(features)
target_var = Variable(labels)
outputs = Config.MODEL(input_var)
loss = Config.LOSS(outputs, target_var)

Both blocks succeed in activating training, but I worried that there might be trivial difference.

Output:
According to this question you no longer need variables to use Pytorch Autograd.
Thanks to @skytree, we can make this even more explizit: Variables have been deprecated, i.e. you're not supposed to use them anymore.

Autograd automatically supports Tensors with requires_grad set to True.

And more importantly

Variable(tensor) and Variable(tensor, requires_grad) still work as expected, but they return Tensors instead of Variables.

This means that if your features and labels are tensors already (which they seem to be in your example) your Variable(features) and Variable(labels) does only return a tensor again.
The original purpose of Variables was to be able to use automatic differentiation (Source):

Variables are just wrappers for the tensors so you can now easily auto compute the gradients.


Instruction: pytorch "log_softmax_lastdim_kernel_impl" not implemented for 'torch.LongTensor'

Input:
I am trying to use my own dataset to classify text according to https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb. My dataset is a csv of sentences and a class associated with it. there are 6 different classes:

sent                      class
'the fox is brown'        animal
'the house is big'        object
'one water is drinkable'  water
...

When running:

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    print(start_time)
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    print(train_loss.type())
    print(train_acc.type())
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)

    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)

    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut5-model.pt')

    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

, I receive the following error

RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'torch.LongTensor'

pointing to:

<ipython-input-38-9c6cff70d2aa> in train(model, iterator, optimizer, criterion)
     14         print('pred'+ predictions.type())
     15         #batch.label = batch.label.type(torch.LongTensor)
---> 16         loss = criterion(predictions.long(), batch.label)**

The solution posted here https://github.com/pytorch/pytorch/issues/14224 suggests I need to use long/int.

I had to add .long() at line ** in order to fix this earlier error:

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target'

The specific lines of code are:

  def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for batch in iterator:

        optimizer.zero_grad()

        predictions = model(batch.text)
        print('pred'+ predictions.type())
        #batch.label = batch.label.type(torch.LongTensor)
        loss = criterion(predictions.long(), batch.label)**

        acc = categorical_accuracy(predictions, batch.label)

        loss.backward()

        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Note, the ** was originally loss = criterion(predictions, batch.label)

Any other suggestions to fix this issue?

Output:
criterion is defined as torch.nn.CrossEntropyLoss() in your notebook. As mentioned in documentation of CrossEntropyLoss, it expects probability values returned by model for each of the 'K' classes and corresponding value for ground-truth label as input. Now, probability values are float tensors, while ground-truth label should be a long tensor representing a class (class can not be a float, e.g. 2.3 can not represent a class). hence:

loss = criterion(predictions, batch.label.long())

should work.


Instruction: The training loss of vgg16 implemented in pytorch does not decrease

Input:
I want to try some toy examples in pytorch, but the training loss does not decrease in the training.

Some info is provided here:

The model is vgg16, consisted of 13 conv layers and 3 dense layers.
The data is cifar100 in pytorch.
I choose cross entropy as the loss function.

The code is as follows

# encoding: utf-8

import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as transforms
import torchvision

import numpy as np

class VGG16(torch.nn.Module):
    def __init__(self, n_classes):
        super(VGG16, self).__init__()

        # construct model
        self.conv1_1 = nn.Conv2d(3, 64, 3, padding=1)
        self.conv1_2 = nn.Conv2d(64, 64, 3, padding=1)
        self.conv2_1 = nn.Conv2d(64, 128, 3, padding=1)
        self.conv2_2 = nn.Conv2d(128, 128, 3, padding=1)
        self.conv3_1 = nn.Conv2d(128, 256, 3, padding=1)
        self.conv3_2 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv3_3 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv4_1 = nn.Conv2d(256, 512, 3, padding=1)
        self.conv4_2 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv4_3 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_1 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_2 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_3 = nn.Conv2d(512, 512, 3, padding=1)

        self.fc6 = nn.Linear(512, 512)
        self.fc7 = nn.Linear(512, 512)
        self.fc8 = nn.Linear(512, n_classes)

    def forward(self, x):
        x = F.relu(self.conv1_1(x))
        x = F.relu(self.conv1_2(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv2_1(x))
        x = F.relu(self.conv2_2(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv3_1(x))
        x = F.relu(self.conv3_2(x))
        x = F.relu(self.conv3_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv4_1(x))
        x = F.relu(self.conv4_2(x))
        x = F.relu(self.conv4_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv5_1(x))
        x = F.relu(self.conv5_2(x))
        x = F.relu(self.conv5_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = x.view(-1, self.num_flat_features(x))

        x = F.relu(self.fc6(x))
        x = F.relu(self.fc7(x))
        x = self.fc8(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]

        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == '__main__':

    BATCH_SIZE = 128
    LOG_INTERVAL = 5

    # data
    transform = transforms.Compose([
        transforms.ToTensor()
    ])

    trainset = torchvision.datasets.CIFAR100(
        root='./data',
        train=True,
        download=True,
        transform=transform
    )

    testset = torchvision.datasets.CIFAR100(
        root='./data',
        train=False,
        download=True,
        transform=transform
    )

    trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=True)
    testloader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, shuffle=False)

    # model
    vgg16 = VGG16(100)
    vgg16.cuda()

    # optimizer
    optimizer = optim.SGD(vgg16.parameters(), lr=0.01)

    # loss
    criterion = nn.CrossEntropyLoss()

    print('———— Train Start —————')
    for epoch in range(20):
        running_loss = 0.
        for step, (batch_x, batch_y) in enumerate(trainloader):
            batch_x, batch_y = batch_x.cuda(), batch_y.cuda()

            #
            optimizer.zero_grad()

            output = vgg16(batch_x)
            loss = criterion(output, batch_y)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()

            if step % LOG_INTERVAL == 0:
                print('[%d, %4d] loss: %.4f' % (epoch, step, running_loss / LOG_INTERVAL))
                running_loss = 0.

        def test():
            print('———— Test Start ————')
            correct = 0
            total = 0

            #
            with torch.no_grad():
                for test_x, test_y in testloader:
                    images, labels = test_x.cuda(), test_y.cuda()
                    output = vgg16(images)
                    _, predicted = torch.max(output.data, 1)
                    total += labels.size(0)
                    correct += (predicted == labels).sum().item()

            accuracy = 100 * correct / total
            print('Accuracy of the network is: %.4f %%' % accuracy)
            print('———— Test Finish ————')

        test()

    print('———— Train Finish —————')

The loss stays around 4.6060 and never decrease. I have tried different learning rate but does not work.

Output:
I have noticed that you are not using Batch normalization in between your convolution layers. I have added batch normalization layers and it seems to work. Following is the modified code:

class VGG16(torch.nn.Module):
def __init__(self, n_classes):
    super(VGG16, self).__init__()

    # construct model
    self.conv1_1 = nn.Conv2d(3, 64, 3, padding=1)
    self.conv11_bn = nn.BatchNorm2d(64)
    self.conv1_2 = nn.Conv2d(64, 64, 3, padding=1)
    self.conv12_bn = nn.BatchNorm2d(64)
    self.conv2_1 = nn.Conv2d(64, 128, 3, padding=1)
    self.conv21_bn = nn.BatchNorm2d(128)
    self.conv2_2 = nn.Conv2d(128, 128, 3, padding=1)
    self.conv22_bn = nn.BatchNorm2d(128)
    self.conv3_1 = nn.Conv2d(128, 256, 3, padding=1)
    self.conv31_bn = nn.BatchNorm2d(256)
    self.conv3_2 = nn.Conv2d(256, 256, 3, padding=1)
    self.conv32_bn = nn.BatchNorm2d(256)
    self.conv3_3 = nn.Conv2d(256, 256, 3, padding=1)
    self.conv33_bn = nn.BatchNorm2d(256)
    self.conv4_1 = nn.Conv2d(256, 512, 3, padding=1)
    self.conv41_bn = nn.BatchNorm2d(512)
    self.conv4_2 = nn.Conv2d(512, 512, 3, padding=1)
    self.conv42_bn = nn.BatchNorm2d(512)
    self.conv4_3 = nn.Conv2d(512, 512, 3, padding=1)
    self.conv43_bn = nn.BatchNorm2d(512)
    self.conv5_1 = nn.Conv2d(512, 512, 3, padding=1)
    self.conv51_bn = nn.BatchNorm2d(512)
    self.conv5_2 = nn.Conv2d(512, 512, 3, padding=1)
    self.conv52_bn = nn.BatchNorm2d(512)
    self.conv5_3 = nn.Conv2d(512, 512, 3, padding=1)
    self.conv53_bn = nn.BatchNorm2d(512)

    self.fc6 = nn.Linear(512, 512)
    self.fc7 = nn.Linear(512, 512)
    self.fc8 = nn.Linear(512, n_classes)

def forward(self, x):
    x = F.relu(self.conv11_bn(self.conv1_1(x)))
    x = F.relu(self.conv12_bn(self.conv1_2(x)))
    x = F.max_pool2d(x, (2, 2))

    x = F.relu(self.conv22_bn(self.conv2_1(x)))
    x = F.relu(self.conv21_bn(self.conv2_2(x)))
    x = F.max_pool2d(x, (2, 2))

    x = F.relu(self.conv31_bn(self.conv3_1(x)))
    x = F.relu(self.conv32_bn(self.conv3_2(x)))
    x = F.relu(self.conv33_bn(self.conv3_3(x)))
    x = F.max_pool2d(x, (2, 2))

    x = F.relu(self.conv41_bn(self.conv4_1(x)))
    x = F.relu(self.conv42_bn(self.conv4_2(x)))
    x = F.relu(self.conv43_bn(self.conv4_3(x)))
    x = F.max_pool2d(x, (2, 2))

    x = F.relu(self.conv51_bn(self.conv5_1(x)))
    x = F.relu(self.conv52_bn(self.conv5_2(x)))
    x = F.relu(self.conv53_bn(self.conv5_3(x)))
    x = F.max_pool2d(x, (2, 2))

    x = x.view(-1, self.num_flat_features(x))

    x = F.relu(self.fc6(x))
    x = F.relu(self.fc7(x))
    x = self.fc8(x)
    return x

However, a more elegant version of the same could be found here


Instruction: Prevent GPU usage in SLURM when --gpus is not set

Input:
We're using SLURM to manage a small on-premise cluster.  A key resource we are managing is GPUs.  When a user requests GPUs via --gpus=2 the CUDA_VISIBLE_DEVICES environment variable is set with the GPUs SLURM allocates to the user.

$ srun --gpus=2 bash -c 'echo $CUDA_VISIBLE_DEVICES'
0,1

We have a small team and can trust our users to not abuse the system (they could easily overwrite the environment variable) so this works great.  However, it's a bit too easy to bypass this accidentally because when --gpus isn't specified $CUDA_VISIBLE_DEVICES is left unset so the user can use any GPU (we're typically using PyTorch).

In other words, the following command works fine (so long as it lands on a GPU node) but I would prefer that it fails (because no GPU was requested).

srun sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'

It would fail if $CUDA_VISIBLE_DEVICES were set to -1.

$ CUDA_VISIBLE_DEVICES=-1 sudo docker run -e CUDA_VISIBLE_DEVICES --runtime=nvidia pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime python -c 'import torch; print(torch.tensor([1., 2.], device=torch.device("cuda:0")))'
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp line=51 error=38 : no CUDA-capable device is detected
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/lib/python3.6/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (38) : no CUDA-capable device is detected at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp:51

How can I configure SLURM to set CUDA_VISIBLE_DEVICES to -1 when --gpus is not specified?

Output:
You can use the TaskProlog script to set the $CUDA_VISIBLE_DEVICES variable to -1 if it was not set by Slurm.

In slurm.conf, configure TaskProlog=/path/to/prolog.sh and set the following content for prolog.sh.

#! /bin/bash

if [[ -z $CUDA_VISIBLE_DEVICES]]; then
echo export CUDA_VISIBLE_DEVICES=-1
fi

The echo export ... part will inject CUDA_VISIBLE_DEVICES=-1 in the job environment.

Make sure /path/to is visible from all compute nodes.
But this will not prevent a user from playing the system and redefining the variable from within the Python script. Really preventing access would require configuring cgroups.


Instruction: fastai error predicting with exported/reloaded model: "Input type and weight type should be the same"

Input:
Whenever I export a fastai model and reload it, I get this error (or a very similar one) when I try and use the reloaded model to generate predictions on a new test set:

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same

Minimal reprodudeable code example below, you just need to update your FILES_DIR variable to where the MNIST data gets deposited on your system:

from fastai import *
from fastai.vision import *

# download data for reproduceable example
untar_data(URLs.MNIST_SAMPLE)
FILES_DIR = '/home/mepstein/.fastai/data/mnist_sample'  # this is where command above deposits the MNIST data for me

# Create FastAI databunch for model training
tfms = get_transforms()
tr_val_databunch = ImageDataBunch.from_folder(path=FILES_DIR,  # location of downloaded data shown in log of prev command
                                train = 'train',
                                valid_pct = 0.2,
                                ds_tfms = tfms).normalize()

# Create Model
conv_learner = cnn_learner(tr_val_databunch,
                           models.resnet34,
                           metrics=[error_rate]).to_fp16()

# Train Model
conv_learner.fit_one_cycle(4)

# Export Model
conv_learner.export()  # saves model as 'export.pkl' in path associated with the learner

# Reload Model and use it for inference on new hold-out set
reloaded_model = load_learner(path = FILES_DIR,
                              test = ImageList.from_folder(path = f'{FILES_DIR}/valid'))

preds = reloaded_model.get_preds(ds_type=DatasetType.Test)

Output:

  "RuntimeError: Input type (torch.cuda.FloatTensor) and weight type
  (torch.cuda.HalfTensor) should be the same"

Stepping through the code statement by statement, everything works fine until the last line pred = ... which is where the torch error above pops up.

Relevant software versions:

Python 3.7.3
fastai           1.0.57
torch            1.2.0
torchvision      0.4.0

Output:
So the answer to this ended up being relatively simple:

1) As noted in my comment, training in mixed precision mode (setting conv_learner to_fp16()) caused the error with the exported/reloaded model

2) To train in mixed precision mode (which is faster than regular training) and enable export/reload of the model without errors, simply set the model back to default precision before exporting.

...In code, simply changing the example above:

# Export Model
conv_learner.export()

to:

# Export Model (after converting back to default precision for safe export/reload
conv_learner = conv_learner.to_fp32()
conv_learner.export()

...and now the full (reproduceable) code example above runs without errors, including the prediction after model reload.


Instruction: How to fix 'RuntimeError: Address already in use' in PyTorch?

Input:
I am trying to run a distributive application with PyTorch distributive trainer. I thought I would first try the example they have, found here. I set up two AWS EC2 instances and configured them according to the description in the link, but when I try to run the code I get two different errors: in the first terminal window for node0 I get the error message: RuntimeError: Address already in use

Under the other three windows I get the same error message:

  RuntimeError: NCCL error in:
  /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system
  error

I followed the code in the link, and terminated the instances an redid but it didn't help/

This is using python 3.6 with the nightly build Cuda 9.0. I tried changing the MASTER_ADDR to the ip for node0 on both nodes, as well as using the same MASTER_PORT (which is an available, unused port). However I still get the same error message.

After running this, my goal is to the adjust this StyleGan implementation so that I can train it across multiple GPUs in two different nodes.

Output:
So after a lot of failed attempts I found out what the problem is. Note that this solution applies to using ASW deep learning instances.

After creating two instances I had to adjust the security group. Add two rules: The first rule should be ALL_TCP, and set the source to the Private IPs of the leader. The second rule should be the same (ALL_TCP), but with the source as the Private IPs of the slave node.

Previously, I had the setting security rule set as: Type SSH, which only had a single available port (22). For some reason I was not able to use this port to allow the nodes to communicate. After changing these settings the code worked fine. I was also able to run this with the above mentioned settings.


Instruction: How to use autograd.gradcheck in PyTorch?

Input: The documentation does not include any example use case of gradcheck, where would it be useful?

Output:
There's an example use case provided in the documentation here:

https://pytorch.org/docs/master/notes/extending.html

  You probably want to check if the backward method you implemented
  actually computes the derivatives of your function. It is possible by
  comparing with numerical approximations using small finite
  differences:

  from torch.autograd import gradcheck

# gradcheck takes a tuple of tensors as input, check if your gradient
# evaluated with these tensors are close enough to numerical
# approximations and returns True if they all verify this condition.
input = (torch.randn(20,20,dtype=torch.double,requires_grad=True), torch.randn(30,20,dtype=torch.double,requires_grad=True))
test = gradcheck(linear, input, eps=1e-6, atol=1e-4)
print(test)

As the quote above suggests, the purpose of the gradcheck function is to verify that a custom backward function agrees with a numerical approximation of the gradient. The primary use case is when you're implementing a custom backward operation. In very few cases should you be implementing your own backward function in PyTorch. This is because PyTorch's autograd functionality takes care of computing gradients for the vast majority of operations.

The most obvious exceptions are

You have a function which can't be expressed as a finite combination of other differentiable functions (for example, if you needed the incomplete gamma function, you might want to write your own forward and backward which used numpy and/or lookup tables).
You're looking to speed up the computation of a particularly complicated expression for which the gradient could be drastically simplified after applying the chain rule.


Instruction: Invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325

Input:
I have a PyTorch tensor called out_probs which is produced like this:

out_probs=F.softmax(out_dec[:,0],dim=0)

Also, the shape of out_probs is [128,20004]

out_probs is the result of a softmax operation and it's not supposed to contain any negative value so naturally the result of out_probs[out_probs<0 is going to be an empty tensor(actually I checked and it was empty)
But when I'm running

torch.multinomial(out_probs, 1)

I'm getting :

RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325

That implies my tensor has a negative entry and I don't know why this is happening?

Output:
I believe you've found a bug in the error reporting for torch.multinomial.

For example

x = torch.ones(128, 1)
x[0] *= 1e100
out_probs = F.softmax(x, dim=0)
print('Negative values:', torch.sum(out_probs < 0).item())
y = torch.multinomial(out_probs, 1)

results in the following output

Negative values: 0
RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:298

It turns out this is getting triggered because out_probs contains nan entries.

print('nan values:', torch.sum(torch.isnan(out_probs)).item())

gives

nan values: 128

Which are caused by mathematical instabilities in softmax.

Strangely, when the values in out_probs are infinite you get the proper error message

RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry = infinity or NaN) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:302

This bug should probably be reported at https://github.com/pytorch/pytorch/issues if it hasn't been fixed in the most recent version.

By the way I'm using PyTorch 1.0.1.post2


Instruction: padding a list of torch tensors (or numpy arrays)

Input:
Let's say I have a list as the following:

l = [torch.randn(2,3), torch.randn(2,4),torch.randn(2,5)]

I want to zero pad all of them in the second dimension, so they will extend as far as 5 elements (5 being the max number between the three of the elements in the second dimension). How can I do this. I tried this but failed:

from torch.nn.utils.rnn import pad_sequence
pad_sequence(l, batch_first=True, padding_value=0)

which caused the following error:

RuntimeError: The expanded size of the tensor (3) must match the existing size (4) at non-singleton dimension 1.  Target sizes: [2, 3].  Tensor sizes: [2, 4]

The equivalent answer in Numpy would also be appreciated.

Output:
One option is to use np.pad.

Example:

import numpy as np
a = np.random.randn(2, 3)
b = np.pad(a, [(0, 0), (0, 2)], mode='constant')

Print a gives

[[ 1.22721163  1.23456672  0.51948003]
 [ 0.16545496  0.06609003 -0.32071653]]

Print b gives

[[ 1.22721163  1.23456672  0.51948003  0.          0.        ]
 [ 0.16545496  0.06609003 -0.32071653  0.          0.        ]]

The second argument of pad is pad_width which is a list of before/after paddings for each dimension. So in this example no padding in the first dimension and two paddings at the end of the second dimension.

There are lots of other mode options you can use so check out the docs.

For your particular problem you'll need to add an extra step to work out the padding for each array.

Edit

For pytorch I think you want torch.nn.functional.pad e.g.

import torch
t = torch.randn(2, 3)
torch.nn.functional.pad(t, (0, 2))

Edit 2

The torch.nn.utils.rnn.pad_sequence requires the trailing dimensions of all the tensors in the list to be the same so you need to some transposing for it to work nicely

import torch
# l = [torch.randn(2,3), torch.randn(2,4),torch.randn(2,5)]
# l = [i.transpose(0, 1) for i in l]
# or simply make you tensors with switched dimensions
l = [torch.randn(3,2), torch.randn(4,2),torch.randn(5,2)]
out = torch.nn.utils.rnn.pad_sequence(l, batch_first=True)
# out will now be a tensor with shape (3, 5, 2)
# You can transpose it back to (3, 2, 5) with
out = out.transpose(1, 2)