Instruction: PyTorch .to(torch.device("cuda")) speed differs vastly depending on execution state - How to achieve a speed up?

Input:
My question is concerned with the speed of the to Method of PyTorch tensors and how it depends on the "execution state" (not sure if thats the correct name, feel free to edit).
My setup is as follows (RTX 2060 Super):
python version: 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
pytorch version: 1.7.0+cu110

Firstly, a file that contains the code for what I'm talking about:
import torch
import time
import sys

gpu = torch.device("cuda")

def ver():
    print("python version:", sys.version)
    print("pytorch version:", torch.__version__)
    print("\n")

def test():
    start = time.time()
    torch.cuda.init()
    print("cuda init:", time.time()-start)
    x = torch.randn(15000,3).float()
    print("randn initialized:", time.time()-start)
    x.to(gpu)
    print("to(gpu):", time.time()-start)
    torch.cuda.synchronize()
    print("time after sync:", time.time()-start)
    print("\n")

if __name__ == "__main__":
    ver()
    test()
    test()

Running this yields following console output:
python version: 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]
pytorch version: 1.7.0+cu110

cuda init: 0.002934694290161133
randn initialized: 0.0033266544342041016
to(gpu): 1.5724568367004395
time after sync: 1.5725233554840088

cuda init: 9.5367431640625e-07
randn initialized: 0.00030875205993652344
to(gpu): 0.00037860870361328125
time after sync: 0.00039458274841308594

The speed of the second to call is so much faster than the first one. Why does this occur?
And more importantly, how can I achieve consistently fast speeds?
In a bigger project I'm working on I should have enough time to maybe initalize the GPU in parallel before to gets called for the first time. Is that possible? If so, how? torch.cuda.init() doesn't seem to change the speed of the first to.
I cannot use torch.randn(x,y, device=gpu) because in the original setup the data comes from torch.from_numpy().
Thanks.

Output: The first run is always the slowest because it is loading everything on to the gpu. After the first run the times get much more consistent. If you run your test a few more times you should see the times are much closer to each other.


Instruction: How to update classification layers without changing weights for convolutional layers

Input:
I have a CNN with numerous convolutional layers. To each of these convolutional layers I have attached a classifier, to check outputs of intermediary layers. After losses have been produced for each of these classifiers, I want to update the weights for the classifier without touching the weights for the convolutional layers. This code:
for i in range(len(loss_per_layer)):
    loss_per_layer[i].backward(retain_graph=True)
    self.classifiers[i].weight.data -= self.learning_rate * self.alpha[i] * self.classifiers[i].weight.grad.data
    self.classifiers[i].bias.data -= self.learning_rate * self.alpha[i] * self.classifiers[i].bias.grad.data

allows me to do so if the classifier consists of a singular nn.Linear layer. However, my classifiers are of the shape:
self.classifiers.append(nn.Sequential(
    nn.Linear(int(feature_map * input_shape[1] * input_shape[2]), 100),
    nn.ReLU(True),
    nn.Dropout(),
    nn.Linear(100, self.num_classes),
    nn.Sigmoid(),
    ))

How can I update the weights of the Sequential block without touching the rest of the network? I have recently changed from keras to pytorch and so am unsure on how exactly to utilize the optimizer.step() function for this situation, but I have a suspicion it can be done using that.
Please note, I need a generic solution for a Sequential block of any shape, as it will change in future iterations of the model.
Any help is much appreciated.

Output:
You can implement your model as below:
class Model(nn.Module):
       def __init__(self, conv_layers, classifier):
           super().__init__()
           self.conv_layers = conv_layers
           self.classifier = classifier

       def forward(self,x):
           x = self.conv_layers(x)
           return self.classifier(x)

When declaring optimizer, only pass the parameters that you want to be updated.
       model = Model(conv_layers, classifier)
       optimizer = torch.optim.Adam(model.classifier.parameters(), lr=lr)

Now when you will do
       loss.backward()
       optimizer.step()
       model.zero_grad()

only classifier params will be updated.
EDIT: After OP's comment, I am adding below for more generic use cases.
A more generic scenario
   class Model(nn.Module):
       def __init__(self, modules):
           super().__init__()
           # supposing you have multiple modules declared like below.
           # You can also keep them as an array or dict too.
           # For this see nn.ModuleList or nn.ModuleDict in pytorch
           self.module0 = modules[0]
           self.module1 = modules[1]
           #..... and so on

       def forward(self,x):
           # implement forward

   # model and optimizer declarations
   model = Model(modules)
   # assuming we want to update module0 and module1
   optimizer = torch.optim.Adam([
       model.module0.parameters(),
       model.module1.parameters()
   ], lr=lr)
   # you can also provide different learning rate for different modules.
   # See [documentation][https://pytorch.org/docs/stable/optim.html]

   # when training
   loss.backward()
   optimizer.step()
   model.zero_grad()
   # use model.zero_grad to remove gradient computed for all the modules.
   # optimizer.zero_grad only removes gradient for parameters that were passed to it.


Instruction: Error while implenting the Faster R-CNN Object detection algorithm

Input:
I am trying to implement the Faster R-CNN object detection algorithm and I have an unusual error.
While trying to call the train_one_epoch function in this colab tutorial I had an error in the loss_dict = model(images, targets)which is mentioned here .
The exact error that i had is :
    101         cell_anchors = self.cell_anchors
    102         assert cell_anchors is not None
--> 103         assert len(grid_sizes) == len(strides) == len(cell_anchors)
    104
    105         for size, stride, base_anchors in zip(

AssertionError:

Anyone have an idea ? And thanks in advance !

Output:
Finally, I was able to solve this problem and it's just by adding adjusting the size of the AnchorGenerator and their corresponding aspect ratios in the Faster R-CNN function
ft_anchor_generator = AnchorGenerator(
    sizes=((32, 64, 128),), aspect_ratios=((0.5, 1.0, 2.0),)
)
ft_model = FasterRCNN(
    backbone=ft_backbone,
    num_classes=num_classes,
    rpn_anchor_generator=ft_anchor_generator)


Instruction: Translating 3D CNN from Keras to Pytorch

Input:
I'm trying to translate the below 3D CNN architecture from keras to pytorch. The 3D images all have the following dimensions: 193 x 229 x 193.
Network architecture in Keras:
def test_model(size):
    model = Sequential()
    model.add(Conv3D(filters=8,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     input_shape=size,
                     name="conv_1_1"))
    model.add(Conv3D(filters=8,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     name="conv_1_2"))
    model.add(MaxPooling3D(pool_size=(2, 2, 2),
                           strides=(2, 2, 2)))
    model.add(Conv3D(filters=16,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     input_shape=size,
                     name="conv_2_1"))
    model.add(Conv3D(filters=16,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     name="conv_2_2"))
    model.add(MaxPooling3D(pool_size=(2, 2, 2),
                           strides=(2, 2, 2)))
    model.add(Flatten())
    model.add(Dense(units=1,
                    name="d_2"))
    return model

My attempt at translating this to Pytorch:
class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv3d(input_channel=1, output_channel=8, kernel_size=3)
        self.conv2 = nn.Conv3d(input_channel=8, output_channel=8, kernel_size=3)
        self.conv3 = nn.Conv3d(input_channel=8, output_channel=16, kernel_size=3)
        self.conv4 = nn.Conv3d(input_channel=16, output_channel=16, kernel_size=3)
        self.fc1 = nn.Linear( ???? , 1)

     def forward (self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(F.max_pool3d(self.conv2(x), kernel_size=2, stride=2))
        x = F.relu(self.conv3(x))
        x = F.relu(F.max_pool3d(self.conv4(x), kernel_size=2, stride=2))
        x = self.fc1

        return x

net = Model()

Please could you let me know where I've made mistakes and also clarify how to determine the input for the nn.Linear( ???? , 2) layer? Thank you for your help!

Output:
Inspecting your last nn.Conv3d's output, you have a tensor shape of (-1, 16, 45, 54, 45). Therefore, you need a total of 16*45*54*45=1749600 connections on your dense layer (this is tremendously large!).
Some other things to point out:

input_channel and output_channels should be in_channels and out_channels, respectively.
You can either use torch.flatten(x, start_dim=1) or a nn.Flatten() layer (which will flatten from axis=1 to axis=-1 by default).
you have misplaced an F.relu activation as your overall structure is [conv3d, relu, conv3d, relu, maxpool3d] + [conv3d, relu, conv3d, relu, maxpool3d] + [flatten, dense].

Resulting code:
class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv3d(in_channels=1, out_channels=8, kernel_size=3)
        self.conv2 = nn.Conv3d(in_channels=8, out_channels=8, kernel_size=3)
        self.conv3 = nn.Conv3d(in_channels=8, out_channels=16, kernel_size=3)
        self.conv4 = nn.Conv3d(in_channels=16, out_channels=16, kernel_size=3)
        self.fc1 = nn.Linear(1749600, 1)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.max_pool3d(x, kernel_size=2, stride=2)

        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        x = F.max_pool3d(x, kernel_size=2, stride=2)

        x = torch.flatten(x, start_dim=1)
        x = self.fc1(x)
        return x


Instruction: How to introduce a loss to get two matrices similar

Input:
I am training a neural network and I want two matrices to be similar (the covariance matrices). My naive approach was to use a loss based on the difference, such as the L1 loss. But this also forced the matrices to become small, which is not what I want.
Does anyone have an idea here?
Thanks a lot!

Output:
There are many metrics you can use (euclidian distances, cosine similarity, the Bhattacharyya similarity for non-negative features, the Jensen-Shannon divergence).
The cosine similarity seems like a good place to start. You can achieve this by considering both n x m matrices in a n*m dimensional space. And, compare those two vectors with the cosine similarity.
In practice, this can be done using torch.flatten and torch.nn.functionnal.cosine_similarity. Or equivalently with a nn.Flatten layer and nn.CosineSimilarity.
Here I've taken the functional route:
>>> x = torch.rand(1, 10, 10)
>>> y = torch.rand(1, 10, 10)

>>> F.cosine_similarity(torch.flatten(x, 1), torch.flatten(y, 1))
tensor([0.6220])

Notice you will need an extra dimension for the batch: axis=0.

Edit - if you're not working with batches, you could simply broadcast both tensors to 1D tensors:
>>> F.cosine_similarity(x.reshape(1, -1), y.reshape(1, -1))


Instruction: Add Samples after Partial Training in PyTorch

Input:
I have trained a model in PyTorch - an RCNN for text classification. The model has very high precision and recall, but I may eventually receive new documents with text unlike what I used to train, validate, or test the model.
I would like to add new text samples to the model without retraining the model from the beginning. This is desirable because I may lose access to some of the text used for initial training.
If it is not possible to add samples (documents), is it possible to train a new model on only the new samples and then somehow combine the original model and the new model? How?
Here is what my model looks like.
RCNN(
  (embeddings): Embedding(10661, 300)
  (lstm): LSTM(300, 64, bidirectional=True)
  (dropout): Dropout(p=0.0, inplace=False)
  (W): Linear(in_features=428, out_features=64, bias=True)
  (tanh): Tanh()
  (fc): Linear(in_features=64, out_features=3, bias=True)
  (softmax): Softmax(dim=1)
  (loss_op): NLLLoss()
)

I am aware of techniques for saving the model and the corresponding load techniques.

State Dictionary:torch.save(model.state_dict(), PATH)
Model:torch.save(model, PATH)
Checkpoint:torch.save({'epoch': EPOCH, 'model_state_dict': net.state_dict(), 'optimizer_state_dict': optimizer.state_dict(), 'loss': LOSS,}, PATH)

I can continue training based on the original samples, but I do not know how to add samples.
If this is something TensorFlow can do by PyTorch cannot, I might switch to TensorFlow.

Output:
Assuming you have your model's state saved in some file PATH, you can load it back in memory with torch.load. Either on CPU or CUDA device, by default it will be loaded on the device it was on when torch.save was called).
state_dict = torch.load(PATH)
model.load_state_dict(state_dict)

Assuming model is an instance of the same nn.Module class that was used to save the state on PATH. Now model will have an identical state (same parameter weights/biases) as when it was saved on PATH with torch.save. From there you can call model and finetune on new data.
Note: You can load it directly on the desired device by passing a torch.device to torch.load's map_location argument.


Instruction: Perceptron on multi-dimensional tensor

Input:
I'm trying to use Perceptron to reduce a tensor of size: [1, 24, 768] to another tensor with size of [1, 1, 768]. The only way I could use was to first reshape the input tensor to [1, 1, 24*768] and then pass it through linear layers. I'm wondering if there's a more elegant way of this transformation --other than using RNNs cause I do not want to use that. Are there other methods generally for the transformation that I want to make? Here is my code for doing the above operation:
lin = nn.Linear(24*768, 768)

# x is in shape of [1, 24, 768]
# out is in shape of [1, 1, 768]
x = x.view(1,1,-1)
out = lin(x)

Output:
If the broadcasting is what's bothering you, you could use a nn.Flatten to do it:
>>> m = nn.Sequential(
...    nn.Flatten(),
...    nn.Linear(24*768, 768))

>>> x = torch.rand(1, 24, 768)

>>> m(x).shape
torch.Size([1, 768])

If you really want the extra dimension you can unsqueeze the tensor on axis=1:
>>> m(x).unsqueeze(1).shape
torch.Size([1, 1, 768])


Instruction: Conditionally apply tensor operations in PyTorch

Input:
I know PyTorch doesn't have a map-like function to apply a function to each element of a tensor. So, could I do something like the following without a map-like function in PyTorch?
if tensor_a * tensor_b.matmul(tensor_c) < 1:
    return -tensor_a*tensor_b
else:
    return 0

This would work if the tensors were 1D. However, I need this to work when tensor_b is 2D (tensor_a needs to be unsqueezed in the return statement). This means a 2D tensor should be returned where some of the rows will be 0 vectors.
Happy to use the latest features of the most recent Python version.

Output:
If I understand correctly, you are looking to return a tensor either way (hence the mapping) but by checking the condition element-wise. Assuming the shapes of  tensor_a, tensor_b, and tensor_c are all two dimensional, as in "simple matrices", here is a possible solution.
What you're looking for is probably torch.where, it's fairly close to a mapping where based on a condition, it will return one value or another element-wise.
It works like torch.where(condition, value_if, value_else) where all three tensors have the same shape (value_if and value_else can actually be floats which will be cast to tensors, filled with the same value). Also, condition is a bool tensor which defines which value to assign to the outputted tensor: it's a boolean mask.
For the purpose of this example, I have used random tensors:
>>> a = torch.rand(2, 2, dtype=float)*100
>>> b = torch.rand(2, 2, dtype=float)*0.01
>>> c = torch.rand(2, 2, dtype=float)*10

>>> torch.where(a*(b@c) < 1, -a*b, 0.)
tensor([[ 0.0000,  0.0000],
        [ 0.0000, -0.0183]], dtype=torch.float64)

More generally though, this will work if tensor_a and tensor_b have a shape of (m, n), and tensor_c has a shape of (n, m) because of the operation constraints. In your experiment I'm guessing you only had columns.


Instruction: PyTorch - indices of corresponding values in another tensor

Input:
I have a tensor, where I want to copy only some of the values (columnwise). The same values are in another tensor but in a random order. What I want, are the column indices from tensor2 of the values of tensor1. Here is an example:
copy_ind = torch.tensor([0, 1, 3], dtype=torch.long)
tensor1 = torch.tensor([[4, 6, 5, 1, 8],[10, 0, 8, 2, 1]])
temp = torch.index_select(tensor1, 1, copy_ind) # values to copy
tensor2 = torch.tensor([[1, 4, 5, 6, 8],[2, 10, 8, 0, 1]], dtype=torch.long)
_, t_ind = torch.sort(temp[0], dim=0)
t2_ind = copy_ind[t_ind] # indices of tensor2

The output should be:
t2_ind = [1, 3, 0]

Here is another example where I want to get the values of the tensor according to c1_new:
c1 = torch.tensor([[6, 7, 7, 8, 6, 8, 9, 4, 7, 6, 1, 3],[5, 11, 5, 7, 2, 9, 5, 5, 7, 11, 10, 7]], dtype=torch.long)
copy_ind = torch.tensor([1, 2, 3, 5, 7, 8], dtype=torch.long)
c1_new = torch.index_select(c1, 1, copy_ind)

indices = torch.as_tensor([[1, 3, 4, 6, 6, 6, 7, 7, 7, 8, 8, 9], [10, 7, 5, 2, 5, 11, 5, 7, 11, 7, 9, 5]])
values = torch.randn(12)
tensor = torch.sparse.FloatTensor(indices, values, (12, 12))

_, t_ind = torch.sort(c1[0], dim=0)
ind = t_ind[copy_ind] # should be [8, 6, 9, 10, 2, 7]

Unfortunately, the indices ind are not correct. Can someone please help me?

Output:
If you're ok with using a for loop you can use something like this: checking each column of your temp tensor against the columns of tensor2:
Edit: using torch.prod across dimension 1 to make sure both rows match
[torch.prod((temp.T[i] == tesnor2.T), dim=1).nonzero()[0] for i in range(temp.size(1))]

My output for your first example is [tensor(1), tensor(3), tensor(0)]


Instruction: PyTorch torch_sparse installation without CUDA

Input:
I am new in PyTorch and I have faced one issue, namely I cannot get my torch_sparse module properly installed.
In general, I wanted to use module torch_geometric - this I have installed. However, when during the execution of the program I keep receiving the error ModuleNotFoundError: No module named ‘torch_sparse’ .
I try to intall it, but when I use the command pip install torch-sparse in anaconda, I get an error:
UserWarning: CUDA initialization:Found no NVIDIA driver on your system.
My system does not have a CUDA. So how could I install torch_sparse module without it?
Thank you in advance!
Kind regards
Rostyslav

Output:
As outlined in in pytorch_geometric installation instructions you have to install dependencies first and torch_geometric after that.
For PyTorch 1.7.0 and CPU:
pip install --no-index torch-scatter -f https://pytorch-geometric.com/whl/torch-1.7.0+cpu.html
pip install --no-index torch-sparse -f https://pytorch-geometric.com/whl/torch-1.7.0+cpu.html
pip install --no-index torch-cluster -f https://pytorch-geometric.com/whl/torch-1.7.0+cpu.html
pip install --no-index torch-spline-conv -f https://pytorch-geometric.com/whl/torch-1.7.0+cpu.html
pip install torch-geometric

Please notice torch-1.7.0+cpu at the very end of each page


Instruction: VGG16 Model Outputs Incorrect dimension - Transfer Learning

Input:
I am trying to use a pre-trained VGG16 model to classify CIFAR10 on pyTorch. The model was originally trained on ImageNet.
Here is how I imported and modified the model:
from torchvision import models

model = models.vgg16(pretrained=True).cuda()
model.classifier[6].out_features = 10

and this is the summary of the model
print(model)

VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=10, bias=True)
  )
)

Now I wanted to train the model on CIFAR10, I created the train loader with batch size = 128.
Below is the training function and I added some printing statements to check if everything is working fine or not. The problem is the model outputs for each data point a 1000 prediction (as the original - unmodified version).
def train(model, optimizer, train_loader, epoch=5):
    """
    This function updates/trains client model on client data
    """
    model.train()
    for e in range(epoch):
        for batch_idx, (data, target) in enumerate(train_loader):
            data, target = data.to(device), target.to(device)
            optimizer.zero_grad()
            output = model(data)
            print("shape output: ", output.shape)
            probs = F.softmax(output, dim=1) #get the entropy
            print("entropy: ", probs)
            _, predicted = torch.max(output.data, 1)
            loss = criterion(output, target)
            loss.backward()
            if batch_idx % 100 == 0:    # print every 100 mini-batches
                print('[%d, %5d] loss: %.3f' %
                  (e + 1, batch_idx + 1, loss.item()))
            optimizer.step()
    return loss.item()

Here is part of the the output which shows the output shape:
shape output:  torch.Size([128, 1000])

I don't understand why the model outputs the results in this way. Is there something that I am missing?

Output:
Error
It is pretty simple, all you do here:
model = models.vgg16(pretrained=True).cuda()
model.classifier[6].out_features = 10

is changing attribute out_features of torch.nn.Linear layer, not changing weights which actually carry out the computation!
In simplest case (see comments for outputted shape):
import torch

layer = torch.nn.Linear(20, 10)
layer.out_features # 10
layer.weight.shape # (10, 20)
layer(torch.randn(64, 20)).shape # (64, 10)

layer.out_features = 100

layer.out_features # 100
layer.weight.shape # (10, 20)
layer(torch.randn(64, 20)).shape # (64, 10)

Weights are of the same shape, as those are created during __init__ (and altering out_features doesn't change anything).
Fix
You have to recreate the last layer a-new (it will be randomly initialized), like this:
model = torchvision.models.vgg16(pretrained=True)
# New layer
model.classifier[6] = torch.nn.Linear(4096, 10)


Instruction: How to convert TensorFlow tensor to PyTorch tensor without converting to Numpy array?

Input:
I want to use a pre-trained Pytorch model in Tensorflow and I need to convert the tensorflow tensors to pytorch tensors. But I don't want to convert the pytorch tensor to a numpy array and convert that to a tensorflow tensor since I'm getting the error of " You must feed a value for placeholder tensor".
I need this conversion when I'm making the graph so the tensorflow tensor doesn't have value and cannot be converted to numpy!
Any solution for that?

Output:
Operations you do to Tensorflow tensors are "remembered" in order to calculate and back-propagate gradients. Same is true for PyTorch tensors. All this is ultimately required to train the model in both frameworks. This also is the reason why you can't convert tensors between the two frameworks: They have different ops and gradient calculation systems. They are incapable of capturing any operation that happens beyond their framework. For example, you can't (as of Jan 2021) have python for loops in custom loss functions. It has to be implemented into the framework in order to work. Similarly, there is no implementation of converting pytorch operations to Tensorflow operations.
This answer shows how it's done when your tensor is well-defined (not a placeholder). But there is currently no way to propagate gradients from Tensorflow to PyTorch or vice-versa. Maybe in the future there will be some kind of massive update to both frameworks that lets them inter-operate, but I doubt that. It's best to use them both separately.
So, in short, you can't convert placeholder tensors between two frameworks. You have to stick to one of the libraries or use concrete tensors + numpy mediator to communicate in-between frameworks.


Instruction: How to implement an epoch-dependent parameter inside a neural network in Pytorch?

Input:
For example, let's say I want to use Softmax with temperature after a layer and I also want to decrease the temperature after 5 epochs.
for epoch in range(EPOCHS):
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    inputs, labels = data[0].to(torch_device), data[1].to(torch_device)
    optimizer.zero_grad()
    outputs = my_model(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()
    my_model.adjust_softmax_temperature(epoch)
    # ...

How should I implement the adjust_softmax_temperature method in order to make my code to work?

Output:
as far as I know, the model is loaded BEFORE thre training, and the forward step passes data through the already-defined model. Am I missing something here?

You can mutate the state of your model even after it has been defined. In this case, we're interested in changing a property of my_model during training.
Here is a broad overview of how you could implement this.
First, expose a function to update the model's temperature based on the provided epoch number:
class MyModel(nn.Module):
    def __init__(self, t0):
        self.t = t0

    def forward(self, x):
        # compute forward pass using self.t
        pass

    def adjust_softmax_temperature(self, epoch):
        if epoch % 5:
            self.t /= 10 # update policy of temperature

Then, after having defined your model, you can call the update function at the end of each epoch: it will decide based on epoch wether to update self.t or not.
my_model = MyModel(t0=100)

for epoch in range(EPOCHS):
    for i, data in enumerate(trainloader, 0):
        # train iteration
    my_model.adjust_softmax_temperature(epoch)


Instruction: Understanding the usage of nn.Linear for Forward Propagation in PyTorch

Input:
The purpose of this study is to build a simplified forward propagation model that reproduces the code structure in PyTorch, yet does not use any of the PyTorch libraries. The idea is to do a matrix multiplication while emulating the code structure, including class definitions as in pyTorch.
PyTorch code for Forward Propagation
from torch import nn
#DEFINE THE REQUIRED CLASS

class Network(nn.Module):
    def __init__(self):
        super().__init__()

        # Inputs to hidden layer linear transformation
        self.hidden = nn.Linear(784, 256)
        # Output layer, 10 units - one for each digit
        self.output = nn.Linear(256, 10)

        # Define sigmoid activation and softmax output
        self.sigmoid = nn.Sigmoid()
        self.softmax = nn.Softmax(dim=1)

    def forward(self, x):
        # Pass the input tensor through each of our operations
        x = self.hidden(x)
        x = self.sigmoid(x)
        x = self.output(x)
        x = self.softmax(x)

        return x
#use
model = Network()
...
ps = model.forward(some_tensor)

here is my reproducer code:
class my_mul:
        def __init__(self, h, w):
                self.dim1 = h
                self.dim2 = w
                self.layer = LAYER(self.dim1, self.dim2)
        def forward(self, X):
                X =  self.layer.doit( X)
                return X

class LAYER:
        def __init__(self, h, w):
                self.dim1 = h
                self.dim2 = w
        def __call__(self, Z):
                self.matrix2 = Z
        def doit(self, X):
                self.matrix1 = np.random.rand(self.dim1, self.dim2)
                print('matrix 1 in class LAYER ',self.matrix1)
                X =  np.matmul(self.matrix1, self.matrix2)

import numpy as np
#
np.random.seed(0)
# initialize matrix2 which emulates the tensor being passed down to the forward
# propagation step within a deep network training

matrix2 = np.random.rand(2,2)
print('matrix2 ',matrix2)
#
# use the __call__ method of LAYER to pass matrix2 to LL
LL = LAYER(2,2)
LL(matrix2)

MM = my_mul(2,2)
P = MM.forward(matrix2)
print('the product of the 2 matrices is ', P)

The above 'reproducer code' fails as follows. The problem is that the data is not passed correctly and I don't know how to make it work.
matrix2  [[0.5488135  0.71518937]
 [0.60276338 0.54488318]]
matrix 1 in class LAYER  [[0.4236548  0.64589411]
 [0.43758721 0.891773  ]]
Traceback (most recent call last):
  File "all_s.py", line 30, in <module>
    P = MM.forward(matrix2)
  File "all_s.py", line 7, in forward
    X =  self.layer.doit( X)
  File "all_s.py", line 19, in doit
    X =  np.matmul(self.matrix1, self.matrix2)
AttributeError: 'LAYER' object has no attribute 'matrix2'

Output:
The issue is, when you are calling the forward function on MM your initialized model, self.matrix2 hasn't been defined as the error suggests:

AttributeError: 'LAYER' object has no attribute 'matrix2'

Just to be clear: MM() is equivalent to calling MM.__call__(). You haven't called it, hence the error.
I'm not sure why you would have both an implementation in forward and __call__. In PyTorch the high level API call is made through __call__ which is what you might expect. And, __call__ will call forward as well as trigger registered hooks on the module.
A quick fix would be to define your matrix (the underlying component of your linear layer) in the initialization (i.e. inside __init__). Then, when called the matrix multiplication between the input and that matrix is performed.
class LAYER:
    def __init__(self, h, w):
        self.dim1 = h
        self.dim2 = w
        self.matrix1 = np.random.rand(self.dim1, self.dim2)

    def __call__(self, Z):
        return self.doit(Z)

    def doit(self, X):
        return np.matmul(X, self.matrix1)

Although, something like this would be clearer:
class Model:
    def __init__(self, h, w):
        self.layer = Linear(h, w)

    def __call__(self, x):
        return self.forward(x)

    def forward(self, x):
        x =  self.layer(x)
        return x

class Linear:
    def __init__(self, h, w):
        self.weights = np.random.rand(h, w)

    def __call__(self, x):
        return self.forward(x)

    def forward(self, x):
        return np.matmul(x, self.weights)

x = np.random.rand(2, 2)
model = Model(2, 2)
model(x)

Now you can use additional layers in your Model class. And a backward pass on Linear!


Instruction: Feeding Classifier data from LSTM Autoencoder

Input:
Goal:
I have built a LSTM autoencoder for the purpose of feature reduction. My plan is to encode some input and feed it to a classifier in the future. The encoder takes data of the shape [batch_size, timesteps, features_of_timesteps However in the output layer of the encoder portion I am returning just the last hidden state in the form [1, timesteps, features_of_timesteps].
class Encoder(nn.Module):
    def __init__(self, input_size, first_layer, second_layer, n_layers):
        super(Encoder, self).__init__()
        self.n_layers = n_layers
        self.encode = nn.Sequential(nn.LSTM(input_size, first_layer, batch_first=True),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, second_layer),
                                    getLast())
        self.decode = nn.Sequential(nn.LSTM(second_layer, first_layer, batch_first=True),
                                    getSequence(),
                                    nn.ReLU(True),
                                    nn.LSTM(first_layer, input_size),
                                    getSequence())

    def forward(self, x):
        x = x.float()
        x = self.encode(x)
        x = x.repeat(batch_size, 1, 1)
        x = self.decode(x)
        return x

Worry:
I am afraid that the last hidden state of the my second LSTM layer in the encoding portion of the model is summarizing the entire batch along with decreasing the feature dimensionality. This feels wrong because I am trying to reduce a single timeseries into a smaller vector, not an entire batch of timeseries into one vector. Am I correct in my worries?

Output:
There have multiple problem in your code, for simplicity, I just give you one well defined model instead, following code build a LSTM Autoencoder that reconstruct the inputs with shape (batch_size, timesteps, number_of_features_at_each_timesteps):
import torch
from torch import nn

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Encoder(nn.Module):
  def __init__(self, seq_len, n_features, embedding_dim=64):
    super(Encoder, self).__init__()

    self.seq_len, self.n_features = seq_len, n_features
    self.embedding_dim, self.hidden_dim = embedding_dim, 2 * embedding_dim

    self.rnn1 = nn.LSTM(
      input_size=n_features,
      hidden_size=self.hidden_dim,
      num_layers=1,
      batch_first=True
    )
    self.rnn2 = nn.LSTM(
      input_size=self.hidden_dim,
      hidden_size=self.embedding_dim,
      num_layers=1,
      batch_first=True
    )

  def forward(self, x):
    x, (_, _) = self.rnn1(x)
    x, (hidden_n, _) = self.rnn2(x)
    return hidden_n

class Decoder(nn.Module):
  def __init__(self, seq_len, input_dim=64, n_features=1):
    super(Decoder, self).__init__()

    self.seq_len, self.input_dim = seq_len, input_dim
    self.hidden_dim, self.n_features = 2 * input_dim, n_features

    self.rnn1 = nn.LSTM(
      input_size=input_dim,
      hidden_size=input_dim,
      num_layers=1,
      batch_first=True
    )
    self.rnn2 = nn.LSTM(
      input_size=input_dim,
      hidden_size=self.hidden_dim,
      num_layers=1,
      batch_first=True
    )
    self.output_layer = nn.Linear(self.hidden_dim, n_features)

  def forward(self, x):
    x = x.repeat(self.seq_len, 1, 1)
    x = x.permute(1, 0, 2)
    x, (hidden_n, cell_n) = self.rnn1(x)
    x, (hidden_n, cell_n) = self.rnn2(x)
    return self.output_layer(x)

class RecurrentAutoencoder(nn.Module):
  def __init__(self, seq_len, n_features, embedding_dim=64):
    super(RecurrentAutoencoder, self).__init__()

    self.encoder = Encoder(seq_len, n_features, embedding_dim).to(device)
    self.decoder = Decoder(seq_len, embedding_dim, n_features).to(device)

  def forward(self, x):
    print("Inputs size:", x.size())
    x = self.encoder(x)
    print("Representation size: ", x.size())
    x = self.decoder(x)
    print("Outputs size: ", x.size())
    return x

batch_n = 5
seq_len = 10
n_features = 3
inputs = torch.randn(batch_n, seq_len, n_features).to(device)

model = RecurrentAutoencoder(seq_len, n_features).to(device)
y = model(inputs)

Outputs:
Inputs size: torch.Size([5, 10, 3])
Representation size:  torch.Size([1, 5, 64])
Outputs size:  torch.Size([5, 10, 3])

Beware the representation (i.e outputs of encoder) have shape (1, batch_size, embedding_dim)


Instruction: MNIST plot first test figure after transform

Input:
Here are some of my code, I want to know how to add code to plot the first figure from the test dataset after transforms?
transform=transforms.Compose([
        transforms.ToTensor(),
        AddGaussianNoise(0, 1),
        transforms.Normalize((0.1307,), (0.3081,))
        ])
    dataset1 = datasets.MNIST('../data', train=True, download=True,
                       transform=transform)
    dataset2 = datasets.MNIST('../data', train=False,
                       transform=transform)
    train_loader = torch.utils.data.DataLoader(dataset1,**train_kwargs)
    test_loader = torch.utils.data.DataLoader(dataset2, **test_kwargs)

    model = Net().to(device)
    optimizer = optim.Adadelta(model.parameters(), lr=args.lr)

    scheduler = StepLR(optimizer, step_size=1, gamma=args.gamma)
    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(model, device, test_loader)
        scheduler.step()

    if args.save_model:
        torch.save(model.state_dict(), "mnist_cnn.pt")

Output:
Try this:
import torchvision.utils as vutils
import matplotlib.pyplot as plt

# get batch from dataloader -> (bs, ch, h, w)
imgs, lbls = next(iter(test_loader))

# make grid using images of the batch
img  = vutils.make_grid(imgs)

fig = plt.figure()
plt.axis("off")
# channels first to channels last conversion using permute
plt.imshow(img.permute(1, 2, 0))
plt.show()


Instruction: Expected more than 1 value per channel when training, got input size torch.Size([1, **])

Input:
I met an error when I use BatchNorm1d, code:
##% first I set a model
class net(nn.Module):
    def __init__(self, max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num__rnn_layers, bidirectional, batch_first=True, p=0.2):
        super(net, self).__init__()
        self.max_len = max_len
        self.feature_linear = feature_linear
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bidirectional = bidirectional
        self.num_directions = 2 if bidirectional == True else 1
        self.p = p
        self.batch_first = batch_first
        self.linear1 = nn.Linear(max_len, feature_linear)
        init.kaiming_normal_(self.linear1.weight, mode='fan_in')
        self.BN1 = BN(feature_linear)

    def forward(self, xb, seq_len_crt):
        rnn_input = torch.zeros(xb.shape[0], self.feature_linear, self.input_size)
        for i in range(self.input_size):
            out = self.linear1(xb[:, :, i]) # xb[:,:,i].shape:(1,34), out.shape(1,100)
            out = F.relu(out) # 输入：out.shape(1,100), 输出：out.shape(1,100)
            out = self.BN1(out) # 输入：out.shape(1,100)，输出：out.shape(1,100)

        return y_hat.squeeze(-1)

##% make the model as a function and optimize it
input_size = 5
hidden_size = 32
output_dim = 1
num_rnn_layers = 2
bidirectional = True
rnn = nn.LSTM
batch_size = batch_size
feature_linear = 60
BN = nn.BatchNorm1d

model = net(max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num_rnn_layers, bidirectional, p=0.1)
loss_func = nn.MSELoss(reduction='none')
# optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# optimizer = optim.Adam(model.parameters(), lr=0.01)
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.05)

##% use this model to predict data
def predict(xb, model, seq_len):
    # xb's shape should be (batch_size, seq_len, n_features)
    if xb.ndim == 2:  # suitable for both ndarray and Tensor
       # add a {batch_size} dim
       xb = xb[None, ]

    if not isinstance(xb, torch.Tensor):
       xb = torch.Tensor(xb)
    return model(xb, seq_len) # xb.shape(1,34,5)

##% create training/valid/test data
seq_len_train_iter = []
for i in range(0, len(seq_len_train), batch_size):
    if i + batch_size <= len(seq_len_train):
       seq_len_train_iter.append(seq_len_train[i:i+batch_size])
    else:
       seq_len_train_iter.append(seq_len_train[i:])

seq_len_valid_iter = []
for i in range(0, len(seq_len_valid), batch_size):
    if i + batch_size <= len(seq_len_valid):
        seq_len_valid_iter.append(seq_len_valid[i:i+batch_size])
    else:
        seq_len_valid_iter.append(seq_len_valid[i:])

seq_len_test_iter = []
for i in range(0, len(seq_len_test), batch_size):
    if i + batch_size <= len(seq_len_test):
        seq_len_test_iter.append(seq_len_test[i:i+batch_size])
    else:
        seq_len_test_iter.append(seq_len_test[i:])

##% fit model
def fit(epochs, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter):
    train_loss_record = []
    valid_loss_record = []
    mean_pct_final = []
    mean_abs_final = []

    is_better = False
    last_epoch_abs_error = 0
    last_epoch_pct_error = 0

    mean_pct_final_train = []
    mean_abs_final_train = []

    for epoch in range(epochs):
        # seq_len_crt: current batch seq len
        for batches, ((xb, yb), seq_len_crt) in enumerate(zip(train_dl, seq_len_train_iter)):
            if isinstance(seq_len_crt, np.int64):
                seq_len_crt = [seq_len_crt]
            y_hat = model(xb, seq_len_crt)
            packed_yb = nn.utils.rnn.pack_padded_sequence(yb, seq_len_crt, batch_first=True, enforce_sorted=False)
            final_yb, input_sizes = nn.utils.rnn.pad_packed_sequence(packed_yb)
            final_yb = final_yb.permute(1, 0)
            # assert torch.all(torch.tensor(seq_len_crt).eq(input_sizes))
            loss = loss_func(y_hat, final_yb)

            batch_size_crt = final_yb.shape[0]
            loss = (loss.sum(-1) / input_sizes).sum() / batch_size_crt

            loss.backward()
            optimizer.step()
            # scheduler.step()
            optimizer.zero_grad()
#             print(i)

            with torch.no_grad():
                train_loss_record.append(loss.item())
                if batches % 50 == 0 and epoch % 1 == 0:
#                     print(f'Epoch {epoch}, batch {i} training loss: {loss.item()}')

                    y_hat = predict(xb[0], model, torch.tensor([seq_len_crt[0]])).detach().numpy().squeeze() # xb[0].shape(34,5)
                    label = yb[0][:len(y_hat)]
                    # plt.ion()
                    plt.plot(y_hat, label='predicted')
                    plt.plot(label, label='label')
                    plt.legend(loc='upper right')
                    plt.title('training mode')
                    plt.text(len(y_hat)+1, max(y_hat.max(), label.max()), f'Epoch {epoch}, batch {batches} training loss: {loss.item()}')
                    plt.show()
    return train_loss_record

but I met:Expected more than 1 value per channel when training, got input size torch.Size([1, 60])
the error message is:
ValueError                                Traceback (most recent call last)
<ipython-input-119-fb062ad3f20e> in <module>
----> 1 fit(500, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)

<ipython-input-118-2eb946c379bf> in fit(epochs, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)
     38 #                     print(f'Epoch {epoch}, batch {i} training loss: {loss.item()}')
     39
---> 40                     y_hat = predict(xb[0], model, torch.tensor([seq_len_crt[0]])).detach().numpy().squeeze() # xb[0].shape(34,5)
     41                     label = yb[0][:len(y_hat)]
     42                     # plt.ion()

<ipython-input-116-28afce77e325> in predict(xb, model, seq_len)
      7     if not isinstance(xb, torch.Tensor):
      8         xb = torch.Tensor(xb)
----> 9     return model(xb, seq_len) # xb.shape(None,34,5)

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-114-3e9c30d20ed6> in forward(self, xb, seq_len_crt)
     50             out = self.linear1(xb[:, :, i]) # xb[:,:,i].shape:(None,34), out.shape(None,100)
     51             out = F.relu(out) # 输入：out.shape(None,100), 输出：out.shape(None,100)
---> 52             out = self.BN1(out) # 输入：out.shape(None,100)，输出：out.shape(None,100)
     53
     54             out = self.linear2(out)

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\batchnorm.py in forward(self, input)
    129         used for normalization (i.e. in eval mode when buffers are not None).
    130         """
--> 131         return F.batch_norm(
    132             input,
    133             # If buffers are not to be tracked, ensure that they won't be updated

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
   2052                 bias=bias, training=training, momentum=momentum, eps=eps)
   2053     if training:
-> 2054         _verify_batch_size(input.size())
   2055
   2056     return torch.batch_norm(

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\functional.py in _verify_batch_size(size)
   2035         size_prods *= size[i + 2]
   2036     if size_prods == 1:
-> 2037         raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
   2038
   2039

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 60])

I have checked and I found that in out = self.BN1(out),out.shape = (1,60),it seems that batchsize=1 is not permitted in BatchNorm1d .But I don't know how to modify it.

Output:
what does BatchNorm1d do mathematically?
try and write down the equation for the case of batch_size=1 and you'll understand why pytorch is angry with you.
How to solve it?
It is simple: BatchNorm has two "modes of operation": one is for training where it estimates the current batch's mean and variance (this is why you must have batch_size>1 for training).
The other "mode" is for evaluation: it uses accumulated mean and variance to normalize new inputs without re-estimating the mean and variance. In this mode there is no problem processing samples one by one.
When evaluating your model use model.eval() before and model.train() after.


Instruction: Underfitting a single batch: Can't cause autoencoder to overfit multi-sample batches of 1d data. How to debug?

Input:
TL;DR
I am unable to overfit batches with multiple samples using autoencoder.
Fully connected decoder seems to handle more samples per batch than conv decoder, but then also fails when number of samples increases.
Why is this happening, and how to debug this?

In depth
I am trying to use an auto encoder on 1d data points of size (n, 1, 1024), where n is the number of samples in the batch.
I am trying to overfit to that single batch.
Using a convolutional decoder, I am only able to fit a single sample (n=1), and when n>1 I am unable to drop the loss (MSE) below 0.2.
In blue: expected output (=input), in orange: reconstruction.
Single sample, single batch:

Multiple samples, single batch, loss won't go down:

Using more than one sample, we can see the net learns the general shape of the input (=output) signal, but greatly misses the bias.

Using a fully connected decoder does manage to reconstruct batches of multiple samples:

Relevant code:
class Conv1DBlock(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size):
        super().__init__()
        self._in_channels = in_channels
        self._out_channels = out_channels
        self._kernel_size = kernel_size

        self._block = nn.Sequential(
                nn.Conv1d(
                        in_channels=self._in_channels,
                        out_channels=self._out_channels,
                        kernel_size=self._kernel_size,
                        stride=1,
                        padding=(self._kernel_size - 1) // 2,
                ),
                # nn.BatchNorm1d(num_features=out_channels),
                nn.ReLU(True),
                nn.MaxPool1d(kernel_size=2, stride=2),
        )

    def forward(self, x):
        for layer in self._block:
            x = layer(x)
        return x

class Upsample1DBlock(nn.Module):
    def __init__(self, in_channels, out_channels, factor):
        super().__init__()
        self._in_channels = in_channels
        self._out_channels = out_channels
        self._factor = factor

        self._block = nn.Sequential(
                nn.Conv1d(
                        in_channels=self._in_channels,
                        out_channels=self._out_channels,
                        kernel_size=3,
                        stride=1,
                        padding=1
                ),  # 'same'
                nn.ReLU(True),
                nn.Upsample(scale_factor=self._factor, mode='linear', align_corners=True),
        )

    def forward(self, x):
        x_tag = x
        for layer in self._block:
            x_tag = layer(x_tag)
        # interpolated = F.interpolate(x, scale_factor=0.5, mode='linear') # resnet idea
        return x_tag

encoder:
self._encoder = nn.Sequential(
            # n, 1024
            nn.Unflatten(dim=1, unflattened_size=(1, 1024)),
            # n, 1, 1024
            Conv1DBlock(in_channels=1, out_channels=8, kernel_size=15),
            # n, 8, 512
            Conv1DBlock(in_channels=8, out_channels=16, kernel_size=11),
            # n, 16, 256
            Conv1DBlock(in_channels=16, out_channels=32, kernel_size=7),
            # n, 32, 128
            Conv1DBlock(in_channels=32, out_channels=64, kernel_size=5),
            # n, 64, 64
            Conv1DBlock(in_channels=64, out_channels=128, kernel_size=3),
            # n, 128, 32
            nn.Conv1d(in_channels=128, out_channels=128, kernel_size=32, stride=1, padding=0),  # FC
            # n, 128, 1
            nn.Flatten(start_dim=1, end_dim=-1),
            # n, 128
        )

conv decoder:
self._decoder = nn.Sequential(
    nn.Unflatten(dim=1, unflattened_size=(128, 1)),  # 1
    Upsample1DBlock(in_channels=128, out_channels=64, factor=4),  # 4
    Upsample1DBlock(in_channels=64, out_channels=32, factor=4),  # 16
    Upsample1DBlock(in_channels=32, out_channels=16, factor=4),  # 64
    Upsample1DBlock(in_channels=16, out_channels=8, factor=4),  # 256
    Upsample1DBlock(in_channels=8, out_channels=1, factor=4),  # 1024
    nn.ReLU(True),
    nn.Conv1d(in_channels=1, out_channels=1, kernel_size=3, stride=1, padding=1),
    nn.ReLU(True),
    nn.Flatten(start_dim=1, end_dim=-1),
    nn.Linear(1024, 1024)
)

FC decoder:
self._decoder = nn.Sequential(
    nn.Linear(128, 256),
    nn.ReLU(True),
    nn.Linear(256, 512),
    nn.ReLU(True),
    nn.Linear(512, 1024),
    nn.ReLU(True),
    nn.Flatten(start_dim=1, end_dim=-1),
    nn.Linear(1024, 1024)
)

Another observation is that when the batch size increases more, to say, 16, the FC decoder also starts to fail.
In the image, 4 samples of a 16 sample batch I am trying to overfit

What could be wrong with the conv decoder?
How to debug this or make the conv decoder work?

Output:
Dying ReLU

I think the main reason for underfitting in your case is Dying Relu problem. Your network is simple Autoencoder with no skip/residual connections. So Code in the bottleneck should encode enough information about bias in the data to make Decoder to learn.
So if ReLU activation function is used Negative Biased data information can be lost due to Dying ReLU problem. The solution is to to use better activation functions like LeakyReLU, ELU, MISH, etc.

Linear vs Conv.
In your case, you are overfitting on a single batch. As Linear layers will have more parameters than that of Convolution layers maybe they are Memorising given small data easily.
Batch Size
As you are overfitting on a single batch, a small-batch of data will make it very easy to memorise on the other hand for large batch with single Update of network per batch(during overfitting) make network to learn Generalized abstract features. (This works better if more batches are there with a lot of variety of data)
I tried to reproduce your problem using simple Gaussian data. Just by using LeakyReLU in place of ReLU with proper learning rate solved the problem.
Same architecture given by you is used.
Hyper parameters:
batch_size = 16
epochs = 100
lr = 1e-3
optimizer = Adam
loss(after training with ReLU) = 0.27265918254852295
loss(after training with LeakyReLU) = 0.0004763789474964142
With Relu

With Leaky Relu


Instruction: Implementing BandRNN with pytorch and tensorflow

Input:
So I am trying to figure out how to train my matrix in a way that I will get a BandRNN.
BandRnn is a diagonalRNN model with a different number of connections per neuron.
For example:

C is the number of connections per neuron.
I found out that there is a way to turn off some of the gradients in a for loop, in a way that prevents them from being trained as follows:
for p in model.input.parameters():
         p.requires_grad = False

But I can't find a proper way to do so, in a way that will make my matrix become a BandRNN.
Hopefully, someone will be able to help me with this issue.

Output:
As far as I know you can only activate/deactivate requires_grad on a tensor, and not on distinct components of that tensor. Instead what you could do is zero out the values outside the band.
First create a mask for the band, you could use torch.ones with torch.diagflat:
>>> torch.diagflat(torch.ones(5), offset=1)

By setting the right dimension for torch.ones as well as the right offset you can generate offset diagonal matrices with consistent shapes.
>>> N = 10; i = -1
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[0., 0., 0., 0., 0.],
        [1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.]])

>>> N = 10; i = 0
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]])

>>> N = 10; i = 1
>>> torch.diagflat(torch.ones(N-abs(i)), offset=i)
tensor([[0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.],
        [0., 0., 0., 0., 0.]])

You get the point, summing these matrices element-wise allows use to get a mask:
>>> N = 10; b = 3
>>> mask = sum(torch.diagflat(torch.ones(N-abs(i)), i) for i in range(-b//2,b//2+1))

>>> mask
tensor([[1., 1., 0., 0., 0.],
        [1., 1., 1., 0., 0.],
        [1., 1., 1., 1., 0.],
        [0., 1., 1., 1., 1.],
        [0., 0., 1., 1., 1.]])

Then you can zero out the values outside the band on your nn.Linear:
>>> m = nn.Linear(N, N)
>>> m.weight.data = m.weight * mask

>>> m.weight
Parameter containing:
tensor([[-0.3321, -0.3377, -0.0000, -0.0000, -0.0000],
        [-0.4197,  0.1729,  0.2101,  0.0000,  0.0000],
        [ 0.3467,  0.2857, -0.3919, -0.0659,  0.0000],
        [ 0.0000, -0.4060,  0.0908,  0.0729, -0.1318],
        [ 0.0000, -0.0000, -0.4449, -0.0029, -0.1498]], requires_grad=True)

Note, you might need to perform this on each forward pass as the parameters outside the band might get updated to non-zero values during the training. Of course, you can initialize mask once and keep it in memory.
It would be more convenient to wrap everything into a custom nn.Module.


Instruction: Pytorch: copy.deepcopy vs torch.tensor.contiguous()?

Input:
In python torch, it seems copy.deepcopy method is generally used to create deep-copies of torch tensors instead of creating views of existing tensors.
Meanwhile,
as far as I understood, the torch.tensor.contiguous() method turns a non-contiguous tensor into a contiguous tensor, or a view into a deeply copied tensor.
Then, do the two code lines below work equivalently if I want to deepcopy src_tensor into dst_tensor?
org_tensor = torch.rand(4)
src_tensor = org_tensor

dst_tensor = copy.deepcopy(src_tensor) # 1
dst_tensor = src_tensor.contiguous() # 2

If the two work equivalent, which method is better in deepcopying tensors?

Output:
torch.tensor.contiguous() and copy.deepcopy() methods are different. Here's illustration:
>>> x = torch.arange(6).view(2, 3)
>>> x
tensor([[0, 1, 2],
        [3, 4, 5]])
>>> x.stride()
(3, 1)
>>> x.is_contiguous()
True
>>> x = x.t()
>>> x.stride()
(1, 3)
>>> x.is_contiguous()
False
>>> y = x.contiguous()
>>> y.stride()
(2, 1)
>>> y.is_contiguous()
True
>>> z = copy.deepcopy(x)
>>> z.stride()
(1, 3)
>>> z.is_contiguous()
False
>>>

Here we can easily see that .contiguous() method created contiguous tensor from non-contiguous tensor while deepcopy method just copied the data without converting it to contiguous tensor.
One more thing contiguous creates new tensor only if old tensor is non-contiguous while deepcopy always creates new tensor.
>>> x = torch.arange(10).view(2, 5)
>>> x.is_contiguous()
True
>>> y = x.contiguous()
>>> z = copy.deepcopy(x)
>>> id(x)
2891710987432
>>> id(y)
2891710987432
>>> id(z)
2891710987720

contiguous()
Use this method to convert non-contiguous tensors to contiguous tensors.
deepcopy()
Use this to copy nn.Module i.e. mostly neural network objects not tensors.
clone()
Use this method to copy tensors.


Instruction: Is there a Numpy or pyTorch function for this code?

Input:
Basically is there a Numpy or PyTorch function that does this:
vp_sa_s=mdp_data['sa_s'].detach().clone()
dims = vp_sa_s.size()
for i in range(dims[0]):
    for j in range(dims[1]):
        for k in range(dims[2]):
            # to mimic matlab functionality: vp(mdp_data.sa_s)
            try:
                vp_sa_s[i,j,k] = vp[mdp_data['sa_s'][i,j,k]]
            except:
                pass

Given that vp_sa_s is size (10,5,5) and each value is a valid index vp i.e in range 0-9. vp is size (10,1) with a bunch of random values.
Matlab do it elegantly and quickly with vp(mdp_data.sa_s) which will form a new (10,5,5) matrix. If all values in mdp_data.sa_s are 1, the result would be a (10,5,5) tensor with each value being the 1st value in vp.
Does a function or method that exists that can achieve this in less than O(N^3) time as the above code is terribly inefficient.
Thanks!

Output:
What is wrong with
result = vp[vp_sa_s, 0]

note that since your vp is of shape (10, 1) (it has a trailing singleton dimension) you need to add the , 0] index in the assignment to get rid of this extra dimension.


Instruction: Torch is installed but I'm unable to import it in a computer vision python project in Jupyter notebook

Input: I'm working on a computer vision project using detectron2. I'm having trouble installing torch or importing it into my jupyter notebook. I'm working on a mac that is running MACOS Catalina, Python3 version in 3.8.2 and I'm using Anaconda for my Development environment. In the screenshot, it says torch is installed but it throws error while importing it

Output:
You have installed torch for python at /usr/local/lib/python3.9/site-packages
but as you said yourself, you are

using Anaconda for my Development environment

So if your jupyter notebook is really configured to use your python installation that came alongside anaconda, then you need to do
conda install pytorch torchvision torchaudio cudatoolkit=10.1 -c pytorch

Note:

Make sure which python interpreter your jupyter is actually using by running
import sys; print(sys.executable)

Make sure that you install torch for the correct conda environment (above command will also indicate which one that is)

Set the cudatoolkit version according to your needs. There is also a simple interface on the official website to get the correct command


Instruction: Does pytorch broadcast consume less memory than expand?

Input:
Does pytorch operations using broadcast consume less memory than expand? For example, are the following two programs different in memory usage?
import torch
x = torch.randn(20,1)
y = torch.randn(1,20)
z = x*y

import torch
x = torch.randn(20,1).expand(-1,20)
y = torch.randn(1,20).expand(20,-1)
z = x*y

Output:
According to the documentation page of torch.expand:

Expanding a tensor does not allocate new memory, but only creates a new view on the existing tensor

You can experiment it yourself by profiling the calls (here in Colab):
>>> x = torch.randn(200,1)
>>> y = torch.randn(1,200)
>>> %memit z = x*y
peak memory: 286.85 MiB, increment: 0.31 MiB

>>> x = torch.randn(200,1).expand(-1,200)
>>> y = torch.randn(1,200).expand(200,-1)
>>> %memit z = x*y
peak memory: 286.86 MiB, increment: 0.00 MiB

%memit is a magic function provided by memory_profiler:
pip install memory_profiler

%load_ext memory_profiler


Instruction: Error: optimizer got an empty parameter list. How can i modify my code?

Input:
Here is code:
class MLP(nn.Module):
    def __ini__(self):
         super (MLP, self). __init__()
         self.model=nn.nn.Sequential(
            nn.Linear(784, 200),
            nn.LeakyReLU(inplace=True),
            nn.Linear(200, 200),
            nn.LeakyReLU(inplace=True),
            nn.Linear(200, 10),
            nn.LeakyReLU(inplace=True),
            )

    def forward(self,x):
                x=self.model(x)
                return
device= torch.device('cuda:0')
net = MLP().to(device)

when running these codes
optimizer = optim.SGD(net.parameters(),lr=learning_rate)

I get

ValueError: optimizer got an empty parameter list"

I am trying to imitate this notebook.

Output:
There are several issues with your code. Currently your code is equivalent to:
class MLP(nn.Module):
    def forward(self, x):
        pass

Here are the issues:

Your initializer is named __init__.

forward should return a value, here x.

Replace nn.nn.Sequential with nn.Sequential.

class MLP(nn.Module):
    def __init__(self):
        super (MLP, self).__init__()
        self.model = nn.Sequential(
            nn.Linear(784, 200),
            nn.LeakyReLU(inplace=True),
            nn.Linear(200, 200),
            nn.LeakyReLU(inplace=True),
            nn.Linear(200, 10),
            nn.LeakyReLU(inplace=True))

    def forward(self,x):
        x = self.model(x)
        return x

device= torch.device('cuda:0')
net = MLP().to(device)

optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)


Instruction: Torch: Why is this collate function so much faster than this other one?

Input:
I have developed two collate functions to read in data from h5py files (I tried to create some synthetic data for a MWE here but it is not going to plan).
The difference between the two in processing my data is about 10x -- a very large increase and I am unsure as to why and I am curious for insights for my future collate functions.
def slow(batch):
    '''
    This function retrieves the data emitted from the H5 torch data set.
    It alters the emitted dimensions from the dataloader
    from: [batch_sz, layers, tokens, features], to:
    [layers, batch_sz, tokens, features]
    '''
    embeddings = []
    start_ids = []
    end_ids = []
    idxs = []
    for i in range(len(batch)):
        embeddings.append(batch[i]['embeddings'])
        start_ids.append(batch[i]['start_ids'])
        end_ids.append(batch[i]['end_ids'])
        idxs.append(batch[i]['idx'])
    # package data;    # swap to expected [layers, batch_sz, tokens, features]
    sample = {'embeddings': torch.as_tensor(embeddings).permute(1, 0, 2, 3),
              'start_ids': torch.as_tensor(start_ids),
              'end_ids': torch.as_tensor(end_ids),
              'idx': torch.as_tensor(idxs)}
    return sample

I thought the one below, with more loops, would be slower, but it is far from the case.
def fast(batch):
    ''' This function alters the emitted dimensions from the dataloader
    from: [batch_sz, layers, tokens, features]
    to: [layers, batch_sz, tokens, features] for the embeddings
    '''
    # turn data to tensors
    embeddings = torch.stack([torch.as_tensor(item['embeddings']) for item in batch])
    # swap to expected [layers, batch_sz, tokens, features]
    embeddings = embeddings.permute(1, 0, 2, 3)
    # get start ids
    start_ids = torch.stack([torch.as_tensor(item['start_ids']) for item in batch])
    # get end ids
    end_ids = torch.stack([torch.as_tensor(item['end_ids']) for item in batch])
    # get idxs
    idxs = torch.stack([torch.as_tensor(item['idx']) for item in batch])
    # repackage
    sample = {'embeddings': embeddings,
              'start_ids': start_ids,
              'end_ids': end_ids}
    return sample

Edit: I tried swapping to this:
It still is about 10x slower compared to 'fast'.
def slow(batch):
    '''
    This function retrieves the data emitted from the H5 torch data set.
    It alters the emitted dimensions from the dataloader
    from: [batch_sz, layers, tokens, features], to:
    [layers, batch_sz, tokens, features]
    '''
    embeddings = []
    start_ids = []
    end_ids = []
    idxs = []
    for item in batch:
        embeddings.append(item['embeddings'])
        start_ids.append(item['start_ids'])
        end_ids.append(item['end_ids'])
        idxs.append(item['idx'])
    # package data;    # swap to expected [layers, batch_sz, tokens, features]
    sample = {'embeddings': torch.as_tensor(embeddings).permute(1, 0, 2, 3),
              'start_ids': torch.as_tensor(start_ids),
              'end_ids': torch.as_tensor(end_ids),
              'idx': torch.as_tensor(idxs)}
    return sample

Output:
See this answer (and give it an upvote):
https://stackoverflow.com/a/30245465/10475762
Particularly the line: "In other words and in general, list comprehensions perform faster because suspending and resuming a function's frame, or multiple functions in other cases, is slower than creating a list on demand."
So in your case, you're calling append multiple times each collate, which is called quite a few times in your training/testing/evaluation steps which all adds up. IMO, always avoid for loops whenever you can as it seems to somehow invariably lead to slowdowns.


Instruction: Is there a way to retrieve the specific parameters used in a random torchvision transform?

Input:
I can augment my data during training by applying a random transform (rotation/translation/rescaling) but I don't know the value that was selected.
I need to know what values were applied. I can manually set these values, but then I lose a lot of the benefits that torch vision transforms provide.
Is there an easy way to get these values are implement them in a sensible way to apply during training?
Here is an example. I would love to be able print out the rotation angle, translation/rescaling being applied at each image:
import numpy as np
import matplotlib.pyplot as plt
from torchvision import transforms

RandAffine = transforms.RandomAffine(degrees=0, translate=(0.1, 0.1), scale=(0.8, 1.2))

rotate = transforms.RandomRotation(degrees=45)
shift = RandAffine
composed = transforms.Compose([rotate,
                               shift])

# Apply each of the above transforms on sample.
fig = plt.figure()
sample = np.zeros((28,28))
sample[5:15,7:20] = 255
sample = transforms.ToPILImage()(sample.astype(np.uint8))
title = ['None', 'Rot','Aff','Comp']
for i, tsfrm in enumerate([None,rotate, shift, composed]):
    if tsfrm:
        t_sample = tsfrm(sample)
    else:
        t_sample = sample
    ax = plt.subplot(1, 5, i + 2)
    plt.tight_layout()
    ax.set_title(title[i])
    ax.imshow(np.reshape(np.array(list(t_sample.getdata())), (-1,28)), cmap='gray')

plt.show()

Output:
I'm afraid there is no easy way around it: Torchvision's random transforms utilities are built in such a way that the transform parameters will be sampled when called. They are unique random transforms, in the sense that (1) parameters used are not accessible by the user and (2) the same random transformation is not repeatable.
As of Torchvision 0.8.0, random transforms are generally built with two main functions:

get_params: which will sample based on the transform's hyperparameters (what you have provided when you initialized the transform operator, namely the parameters' range of values)

forward: the function that gets executed when applying the transform. The important part is it gets its parameters from get_params then applies it to the input using the associated deterministic function. For RandomRotation, F.rotate will get called. Similarly, RandomAffine will use F.affine.

One solution to your problem is sampling the parameters from get_params yourself and calling the functional - deterministic - API instead. So you wouldn't be using RandomRotation, RandomAffine, nor any other Random* transformation for that matter.

For instance, let's look at T.RandomRotation (I have removed the comments for conciseness).
class RandomRotation(torch.nn.Module):
    def __init__(
        self, degrees, interpolation=InterpolationMode.NEAREST, expand=False,
        center=None, fill=None, resample=None):
        # ...

    @staticmethod
    def get_params(degrees: List[float]) -> float:
        angle = float(torch.empty(1).uniform_(float(degrees[0]), \
            float(degrees[1])).item())
        return angle

    def forward(self, img):
        fill = self.fill
        if isinstance(img, Tensor):
            if isinstance(fill, (int, float)):
                fill = [float(fill)] * F._get_image_num_channels(img)
            else:
                fill = [float(f) for f in fill]
        angle = self.get_params(self.degrees)

        return F.rotate(img, angle, self.resample, self.expand, self.center, fill)

    def __repr__(self):
        # ...

With that in mind, here is a possible override to modify T.RandomRotation:
class RandomRotation(T.RandomRotation):
    def __init__(*args, **kwargs):
        super(RandomRotation, self).__init__(*args, **kwargs) # let super do all the work

        self.angle = self.get_params(self.degrees) # initialize your random parameters

    def forward(self): # override T.RandomRotation's forward
        fill = self.fill
        if isinstance(img, Tensor):
            if isinstance(fill, (int, float)):
                fill = [float(fill)] * F._get_image_num_channels(img)
            else:
                fill = [float(f) for f in fill]

        return F.rotate(img, self.angle, self.resample, self.expand, self.center, fill)

I've essentially copied T.RandomRotation's forward function, the only difference being that the parameters are sampled in __init__ (i.e. once) instead of inside the forward (i.e. on every call). Torchvision's implementation covers all cases, you generally won't need to copy the full forward. In some cases, you can just call the functional version pretty much straight away. For example, if you don't need to set the fill parameters, you can just discard that part and only use:
class RandomRotation(T.RandomRotation):
    def __init__(*args, **kwargs):
        super(RandomRotation, self).__init__(*args, **kwargs) # let super do all the work

        self.angle = self.get_params(self.degrees) # initialize your random parameters

    def forward(self): # override T.RandomRotation's forward
        return F.rotate(img, self.angle, self.resample, self.expand, self.center)

If you want to override other random transforms you can look at the source code. The API is fairly self-explanatory and you shouldn't have too many issues implementing an override for each transform.


Instruction: LSTM error: AttributeError: 'tuple' object has no attribute 'dim'

Input:
I have the following code:
import torch
import torch.nn as nn

model = nn.Sequential(
          nn.LSTM(300, 300),
          nn.Linear(300, 100),
          nn.ReLU(),
          nn.Linear(300, 7),
          )

s = torch.ones(1, 50, 300)
a = model(s)

And I get:
My-MBP:Desktop myname$ python3 testmodel.py
Traceback (most recent call last):
  File "testmodel.py", line 12, in <module>
    a = model(s)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/functional.py", line 1688, in linear
    if input.dim() == 2 and bias is not None:
AttributeError: 'tuple' object has no attribute 'dim'

Why? The dimensions should be fine. I saw related fixes to this issue when *input is defined in model.forward, but I don't even have anything implemented yet.
/edit: WAIT, there IS a *input!? How can I override this?

Output:
You won't be able to use a nn.RNN inside a nn.Sequential since nn.LSTM layers will output a tuple containing (1) the output features and (2) hidden states and cell states.
The output must first be unpacked in order to use the output features in your subsequent layer: nn.Linear. Something as, if your interested in the hidden states and cell states:
rnn = nn.LSTM(300, 300)
output, (h_n, c_n) = rnn(x)

You could define a custom nn.Module and implement a simple forward function:
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        self.rnn = nn.LSTM(300, 300)

        self.body = nn.Sequential(
          nn.Linear(300, 100),
          nn.ReLU(),
          nn.Linear(100, 7)) # <- had it set to in_features=300

    def forward(self, x):
        x, _ = self.rnn(x) # <- ignore second output
        x = self.body(x)
        return x

Such that:
>>> model = Model()
>>> s = torch.ones(1, 50, 300)

>>> model(s).shape
torch.Size([1, 50, 7])


Instruction: slice tensor of tensors using boolean tensor

Input:
Having two tensors :inputs_tokens is a batch of 20x300 of token ids
and seq_A is my model output with size of [20, 300, 512] (512 vector for each of the tokens in the batch)
seq_A.size()
Out[1]: torch.Size([20, 300, 512])

inputs_tokens.size()
torch.Size([20, 300])

I would like to get only the vectors of the token 101 (CLS) as follow:
cls_tokens = (inputs_tokens == 101)
cls_tokens
Out[4]:
tensor([[ True, False, False,  ..., False, False, False],
       [ True, False, False,  ..., False, False, False],
       [ True, False, False,  ..., False, False, False], ...

How do I slice seq_A to get only the vectors which are true in cls_tokens for each batch?
when I do
seq_A[cls_tokens].size()
Out[7]: torch.Size([278, 512])

but I still need it to bee in the size of [20 x N x 512 ] (otherwise I don't know to which sample it belongs)

Output:
TLDR; You can't, all sequences must have the same size along a given axis.

Take this simplified example:
>>> inputs_tokens = torch.tensor([[  1, 101,  18, 101,   9],
                                  [  1,   2, 101, 101, 101]])
>>> inputs_tokens.shape
torch.Size([2, 5])

>>> cls_tokens = inputs_tokens == 101
tensor([[False,  True, False,  True, False],
        [False, False,  True,  True,  True]])

Indexing inputs_tokens with the cls_tokens mask comes down to reducing inputs_tokens to cls_tokens's true values. In a general case where there is a different number of true values per batch, keeping the shape is impossible.
Following the above example, here is seq_A:
>>> seq_A = torch.rand(2, 5, 1)
tensor([[[0.4644],
         [0.7656],
         [0.3951],
         [0.6384],
         [0.1090]],

        [[0.6754],
         [0.0144],
         [0.7154],
         [0.5805],
         [0.5274]]])

According to your example, you would expect to have an output shape of  (2, N, 1). What would N be? 3? What about the first batch which only as 2 true values? The resulting tensor can't have different sizes (2 and 3 on axis=1). Hence: "all sequences on axis=1 must have the same size".

If however, you are expecting each batch to have the same number of tokens 101, then you could get away with a broadcast of your indexed tensor:
>>> inputs_tokens = torch.tensor([[  1, 101, 101, 101,   9],
                                  [  1,   2, 101, 101, 101]])
>>> inputs_tokens.shape

>>> N = cls_tokens[0].sum()
3

Here remember, I'm assuming you have:
>>> assert all(cls_tokens.sum(axis=1) == N)

Therefore the desired output (with shape (2, 3, 1)) is:
>>> seq_A[cls_tokens].reshape(seq_A.size(0), N, -1)
tensor([[[0.7656],
         [0.3951],
         [0.6384]],

        [[0.7154],
         [0.5805],
         [0.5274]]])

Edit - if you really want to do this though you would require the use of a list comprehension:
>>> [seq_A[i, cls_tokens[i]] for i in range(cls_tokens.size(0))]
[ tensor([[0.7656],
          [0.6384]]),
  tensor([[0.7154],
          [0.5805],
          [0.5274]]) ]


Instruction: Found 0 Defected examples, But In the directory there are 10 images | Found 0 Perfect examples |

Input:
I am trying to read images from a directory. There are some images, but the code is saying 0 images.
I checked with os.listdir(r'C:\Users\hafizurr\Documents\classifier\dataset\Perfect'), which is showing all of the images. But my code is now showing. I am putting my codes below. Please take a look and help me to figure out the issue. Hoping for your help:
class_names = ['Perfect', 'Defected']
root_dir = r'C:\Users\hafizurr\Documents\classifier\dataset'
source_dirs = ['Perfect', 'Defected']`

`if os.path.isdir(os.path.join(root_dir, source_dirs[1])):
    os.mkdir(os.path.join(root_dir, 'test'))`

    for i, d in enumerate(source_dirs):
        os.rename(os.path.join(root_dir, d), os.path.join(root_dir, class_names[i]))

    for c in class_names:
        os.mkdir(os.path.join(root_dir, 'test', c))

    for c in class_names:
        images = [x for x in os.listdir(os.path.join(root_dir, c)) if x.lower().endswith('.JPG')]
        selected_images = random.sample(images, 0)
        for image in selected_images:
            source_path = os.path.join(root_dir, c, image)
            target_path = os.path.join(root_dir, 'test', c, image)
            shutil.move(source_path, target_path)

`class ImgDataset(torch.utils.data.Dataset):
    def __init__(self, image_dirs, transform):
        def get_images(class_name):
            images = [x for x in os.listdir(image_dirs[class_name]) if x.lower().endswith('.JPG')]
            print(f'Found {len(images)} {class_name} examples')
            return images
        self.images = {}
        self.class_names = ['Perfect', 'Defected']
        for class_name in self.class_names:
            self.images[class_name] = get_images(class_name)
        self.image_dirs = image_dirs
        self.transform = transform`

    def __len__(self):
        return sum([len(self.images[class_name]) for class_name in self.class_names])

    def __getitem__(self, index):
        class_name = random.choice(self.class_names)
        index = index % len(self.images[class_name])
        image_name = self.images[class_name][index]
        image_path = os.path.join(self.image_dirs[class_name], image_name)
        image = Image.open(image_path).convert('RGB')
        return self.transform(image), self.class_names.index(class_name)

train_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.RandomHorizontalFlip(),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

test_transform = torchvision.transforms.Compose([
    torchvision.transforms.Resize(size=(224, 224)),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

train_dirs = {
    'Perfect': f'{root_dir}/Perfect',
    'Defected': f'{root_dir}/Defected'
}

train_dataset = ImgDataset(train_dirs, train_transform)

Output:
That's because x.lower().endswith('.JPG') will always be false!
You could do x.lower().endswith('.jpg') instead.
Alternatively, use imghdr.what to check the image type.


Instruction: Image classification Using Pytorch

Input:
this is the code where I was working on Image Classification using Pytorch and I'm not able to get the  accuracy right.
the accuracy is exceeding 100 ,can anyone help me to find the error.
     def trained_model(criterion, optimizer, epochs=5):

      epoch_loss = 0.0
      epoch_accuracy = 0
      running_loss = 0
      running_accuracy = 0
      total = 0

      for epoch in range(epochs):
        print('epoch : {}/{}'.format(epoch+1, epochs))

        for images, labels in train_loader:
          images, labels = images.to(device), labels.to(device)

          optimizer.zero_grad()

          outputs = model(images)
          loss = criterion(outputs, labels)

          _, predictions = torch.max(outputs, dim=1)

          loss.backward()
          optimizer.step()

          running_loss += loss.item()

          running_accuracy += torch.sum(predictions == labels.data)

        epoch_loss = running_loss / len(train_dataset)
        epoch_accuracy = running_accuracy / len(train_dataset)

        print('Loss:{:.4f} , Accuracy : {:.4f} '.format(epoch_loss, epoch_accuracy))

      return model

Output:
You should probably use torch.argmax  to get the class predictions from your model output, instead of torch.max.
Assuming you are working with indices as labels. Something like the following will get you the average accuracy of the current batch:
>>> outputs = torch.rand(16, 5)

>>> pred = torch.argmax(outputs, axis=0)
tensor([14, 11, 13, 15,  7])

>>> labels = torch.tensor([14, 6, 13, 5, 8])

>>> accuracy = (pred == labels).float().mean()
tensor(0.4000)


Instruction: What is the LibTorch equivalent to PyTorch's torch.no_grad?

Input:
When testing a network in PyTorch one can use with torch.no_grad():. What is the Libtorch (C++) equivalent?
Thanks!

Output: The equivalent in LibTorch is torch::NoGradGuard no_grad, see documentation.


Instruction: Custom dataset and dataloader

Input:
I am new to pytorch .
I have big dataset consist of two txt files one for data and other for target data .
In training file each line is list of length 340, In target each line is list of 136.
I would like to ask how i can define my dataset so I can use Dataloader to load my data to train pytorch model?
I apricate you answers

Output:
Dataset from torch.utils.data is an abstract class representing a dataset. Your custom dataset should inherit Dataset and override the following methods:
__len__() so that len(dataset) returns the size of the dataset.
__getitem__() to support the indexing such that dataset[i] can be used to get ith sample
Eg of writing custom Dataset
i have written a general custom dataloader for you as your problem statement.
here data.txt has data and label.txt has labels.
import torch
from torch.utils.data import Dataset

class CustomDataset(Dataset):
    def __init__(self):

        with open('data.txt', 'r') as f:
                self.data_info = f.readlines()

        with open('label.txt', 'r') as f:
                self.label_info = f.readlines()

    def __getitem__(self, index):

        single_data = self.data_info[index].rstrip('\n')

        single_label = self.label_info[index].rstrip('\n')

        return ( single_data , single_label)

    def __len__(self):
        return len(self.data_info)
# Testing
d = CustomDataset()
print(d[1]) # should output data along with label

This will be a basic for your case but have to do some changes that matches your case.
Note : you have to make required changes as per your dataset


Instruction: PyTorch - Convert CIFAR dataset to `TensorDataset`

Input:
I train ResNet34 on CIFAR dataset. For a certain reason, I need to convert the dataset into TensorDataset.
My solution is based on this: https://stackoverflow.com/a/44475689/15072863 with some differences (maybe they are critical, but I don't see why).
It looks I'm not doing this correctly.
Train loader:
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))])

train_ds = torchvision.datasets.CIFAR10('/files/', train=True, transform=transform_train, download=True)

xs, ys = [], []
for x, y in train_ds:
  xs.append(x)
  ys.append(y)

# 1) Standard Version
# cifar_train_loader = DataLoader(train_ds, batch_size=batch_size_train, shuffle=True, num_workers=num_workers)

# 2) TensorDataset version, seems to be incorrect
cifar_tensor_ds = TensorDataset(torch.stack(xs), torch.tensor(ys, dtype=torch.long))
cifar_train_loader = DataLoader(cifar_tensor_ds, batch_size=batch_size_train, shuffle=True, num_workers=num_workers)

I don't think it matters, but test loader is defined as usual:
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010)),
])

cifar_test_loader = DataLoader(
  torchvision.datasets.CIFAR10('/files/', train=False, transform=transform_test, download=True),
  batch_size=batch_size_test, shuffle=False, num_workers=num_workers)

I know that something is wrong with how I use TensorDataset, since;

With TensorDataset I achieve 100% train accuracy, 80% test accuracy
With standard Dataset I achieve 99% train accuracy (never 100%), 90% test accuracy.

So, what am I doing wrong?
P.S.: My final goal is to split the dataset into 10 datasets based on their class. Is there a better way to do this? Of course, I can define my subclass of DataSet, but manually splitting it and creating TensorDataset's seemed to be simpler.

Output:
When using the "standard" dataset, each time you load an image, a random transform (flip + crop) is applied to it. As a consequence, virtually every image of every epoch is unique, seen only once. So you kind of have nb_epochs * len(dataset) different inputs.
With your custom dataset, you first read all the images of the CIFAR dset (each of them with a random transform), store them all, and then use the stored tensor as your training inputs. Thus at each epoch, the network sees exactly the same inputs
Since the network was already able to achieve great accuracy with the random transformations, removing it makes it even easier and thus it further improves the accuracy
Oh, and you should definitely redefine you own subclass of Dataset. It's not even complicated, and it will be much easier to work with. You just need to extract the 10 different datasets, either by manually moving the images in their folders or using some reindexing arrays or something like that. Either way, you will only have to do it once, so not big deal


Instruction: 2-D Vectorized Function with pytorch

Input:
I have a function that I need to compute at every entry of a 2-D torch tensor, and it is depends on the value of the index of both axes. Right now, I have only been able to implement this as a nested for loop, iterating over both axes. This is quite slow (and needs to be executed >10^5 times) and I want to speed it up for better scaling.
vs = 200
nt = 12
b = torch.ones(vs)/vs
n_kw = torch.rand((nt, vs))
n_k = torch.rand((nt,))

def estimate_p(nt, vs, n_kw, n_k):
    p = torch.zeros((nt, vs))

    for i in range(0, nt):
        for j in range(0, vs):
            p[i,j] = (n_kw[i, j] + b[j])/(n_k[i] + torch.sum(b))
    return p

Is there a way to vectorize this/map based in the i,j index?

Output:
Try playing with broadcasting:
def estimate_p(nt, vs, n_kw, n_k):
    return (n_kw + b) / (n_k + b.sum()).unsqueeze(-1)


Instruction: pytorch OSError: [Errno 28] No space left on device

Input:
I'm using an ubuntu 18 docker container.
$cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

When I try training a resnext101 model from torchvision, I get the following error.
Downloading: "https://download.pytorch.org/models/resnext101_32x8d-8ba56ff5.pth" to /home/vmuser/.cache/torch/hub/checkpoints/resnext101_32x8d-8ba56ff5.pth
  0%|                                                          | 0.00/340M [00:00<?, ?B/s]
Traceback (most recent call last):
  File "train_attn_best_config.py", line 377, in <module>
    tabct = TabCT(cnn = model, fc_dim = fd, attn_filters = af, n_attn_layers = nal).to(gpu)
  File "train_attn_best_config.py", line 219, in __init__
    self.ct_cnn = cnn_dict[cnn](pretrained = True)
  File "/home/vmuser/anaconda3/envs/pulmo/lib/python3.7/site-packages/torchvision/models/resnet.py", line 317, in resnext101_32x8d
    pretrained, progress, **kwargs)
  File "/home/vmuser/anaconda3/envs/pulmo/lib/python3.7/site-packages/torchvision/models/resnet.py", line 227, in _resnet
    progress=progress)
  File "/home/vmuser/anaconda3/envs/pulmo/lib/python3.7/site-packages/torch/hub.py", line 481, in load_state_dict_from_url
    download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "/home/vmuser/anaconda3/envs/pulmo/lib/python3.7/site-packages/torch/hub.py", line 404, in download_url_to_file
    f.write(buffer)
  File "/home/vmuser/anaconda3/envs/pulmo/lib/python3.7/tempfile.py", line 481, in func_wrapper
    return func(*args, **kwargs)
OSError: [Errno 28] No space left on device

When I run df, I get this, one of my tmpfs is only 65 mb. I tried running export TMPDIR=/var/tmp and export TMPDIR=~/Data/tmp
$df
Filesystem      1K-blocks       Used Available Use% Mounted on
overlay        1797272568 1705953392         0 100% /
tmpfs               65536          0     65536   0% /dev
tmpfs            98346264          0  98346264   0% /sys/fs/cgroup
/dev/sda6      1797272568 1705953392         0 100% /etc/hosts
shm                 65536          0     65536   0% /dev/shm
/dev/sdb1      1845816492 1362932848 389098592  78% /home/vmuser/Data
tmpfs            98346264         12  98346252   1% /proc/driver/nvidia
tmpfs            19669256      93256  19576000   1% /run/nvidia-persistenced/socket
udev             98318592          0  98318592   0% /dev/nvidia1
tmpfs            98346264          0  98346264   0% /proc/acpi
tmpfs            98346264          0  98346264   0% /proc/scsi
tmpfs            98346264          0  98346264   0% /sys/firmware

But the error is still there.

Output:
This seems like a shm issue.
Try running docker with ipc=host flag.
For more details, see this thread.


Instruction: Is partial 3D convolution or transpose+2D convolution faster?

Input:
I have some data of shape B*C*T*H*W. I want to apply 2d convolutions on the H*W dimension.
There are two options (that I see):

Apply partial 3D convolution with shape (1, 3, 3). 3D convolution accepts data with shape B*C*T*H*W which is exactly what I have. This is however a pseudo 3d conv that might be under-optimized (despite its heavy use in P3D networks).

Transpose the data, apply 2D convolutions, and transpose the result back. This requires the overhead of data reshaping, but it makes use of the heavily optimized 2D convolutions.

data = raw_data.transpose(1,2).reshape(b*t, c, h, w).detach()
out = conv2d(data)
out = out.view(b, t, c, h, w).transpose(1, 2).contiguous()

Which one is faster?
(Note: I have a self-answer below. This aims to be a quick note for people who are googling, aka me 20 minutes ago)

Output:
Environment: PyTorch 1.7.1, CUDA 11.0, RTX 2080 TI.
TL;DR: Transpose + 2D conv is faster (in this environment, and for the tested data shapes).
Code (modified from here):
import torch
import torch.nn as nn
import time

b = 4
c = 64
t = 4
h = 256
w = 256
raw_data = torch.randn(b, c, t, h, w).cuda()

def time2D():
        conv2d = nn.Conv2d(c, c, kernel_size=3, padding=1).cuda()
        torch.cuda.synchronize()
        start = time.time()
        for _ in range(100):
            data = raw_data.transpose(1,2).reshape(b*t, c, h, w).detach()
            out = conv2d(data)
            out = out.view(b, t, c, h, w).transpose(1, 2).contiguous()
            out.mean().backward()
        torch.cuda.synchronize()
        end = time.time()
        print("  --- %s ---  " %(end - start))

def time3D():
        conv3d = nn.Conv3d(c, c, kernel_size=(1,3,3), padding=(0,1,1)).cuda()
        torch.cuda.synchronize()
        start = time.time()
        for _ in range(100):
            out = conv3d(raw_data.detach())
            out.mean().backward()
        torch.cuda.synchronize()
        end = time.time()
        print("  --- %s ---  " %(end - start))

print("Initializing cuda state")
time2D()

print("going to time2D")
time2D()
print("going to time3D")
time3D()

For shape = 4*64*4*256*256:
2D: 1.8675172328948975
3D: 4.384545087814331

For shape = 8*512*16*64*64:
2D: 37.95961904525757
3D: 49.730860471725464

For shape = 4*128*128*16*16:
2D: 0.6455907821655273
3D: 1.8380646705627441


Instruction: while importing torch- shows - [WinError 126] The specified module could not be found

Input:
I have tried to install python torch by using
  !pip install torch

But I got the error OSError: [WinError 126] The specified module could not be found
Then I tried with
pip install torch -f https://download.pytorch.org/whl/torch_stable.html

The running log is as follows,

Then I have installed the torch by the below command to install the CPU version of the torch.
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

Then, when I try to test by the command,
import torch
print(torch.__version__)

In the Spyder, it still could not recognize,

However, in python.exe it could recognize and provides the correct output.

Output:
Ok the problem is that you are trying to install the cuda version of Pytorch on a non cuda enabled computer. You can find a similar problem here. You have to download the cpu version of pytorch like this
pip install torch==1.7.1+cpu torchvision==0.8.2+cpu torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: PyTorch: while loading batched data using Dataloader, how to transfer the data to GPU automatically

Input:
If we use a combination of the Dataset and Dataloader classes (as shown below), I have to explicitly load the data onto the GPU using .to() or .cuda(). Is there a way to instruct the dataloader to do it automatically/implicitly?
Code to understand/reproduce the scenario:
from torch.utils.data import Dataset, DataLoader
import numpy as np

class DemoData(Dataset):
    def __init__(self, limit):
        super(DemoData, self).__init__()
        self.data = np.arange(limit)

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self, idx):
        return (self.data[idx], self.data[idx]*100)

demo = DemoData(100)

loader = DataLoader(demo, batch_size=50, shuffle=True)

for i, (i1, i2) in enumerate(loader):
    print('Batch Index: {}'.format(i))
    print('Shape of data item 1: {}; shape of data item 2: {}'.format(i1.shape, i2.shape))
    # i1, i2 = i1.to('cuda:0'), i2.to('cuda:0')
    print('Device of data item 1: {}; device of data item 2: {}\n'.format(i1.device, i2.device))

Which will output the following; note - without explicit device transfer instruction, the data is loaded onto CPU:
Batch Index: 0
Shape of data item 1: torch.Size([50]); shape of data item 2: torch.Size([50])
Device of data item 1: cpu; device of data item 2: cpu

Batch Index: 1
Shape of data item 1: torch.Size([50]); shape of data item 2: torch.Size([50])
Device of data item 1: cpu; device of data item 2: cpu

A possible solution is at this PyTorch GitHub repo. Issue(still open at the time this question was posted), but, I am unable to make it to work when the dataloader has to return multiple data-items!

Output:
You can modify the collate_fn to handle several items at once:
from torch.utils.data.dataloader import default_collate

device = torch.device('cuda:0')  # or whatever device/cpu you like

# the new collate function is quite generic
loader = DataLoader(demo, batch_size=50, shuffle=True,
                    collate_fn=lambda x: tuple(x_.to(device) for x_ in default_collate(x)))

Note that if you want to have multiple workers for the dataloader, you'll need to add
torch.multiprocessing.set_start_method('spawn')

after your if __name__ == '__main__' (see this issue).
Having said that, it seems like using pin_memory=True in your DataLoader would be much more efficient. Have you tried this option?
See memory pinning for more information.

Update (Feb 8th, 2021)
This post made me look at my "data-to-model" time spent during training.
I compared three alternatives:

DataLoader works on CPU and only after the batch is retrieved data is moved to GPU.
Same as (1) but with pin_memory=True in DataLoader.
The proposed method of using collate_fn to move data to GPU.

From my limited experimentation it seems like the second option performs best (but not by a big margin).
The third option required fussing about the start_method of the data loader processes, and it seems to incur an overhead at the beginning of each epoch.