Instruction: RuntimeError: expected scalar type Long but found Int in loss = criterion(outputs, y_train)

Input:
I built this acoustic model with features dim = [1124823,13] and labels dim = [1124823,1] and I split both to train, test, and dev. The problem that when I try to run my model I get this error

RuntimeError: expected scalar type Long but found Int in
  loss = criterion(outputs, y_train)

import torch
import torch.nn as nn
from fela import feat, labels
from Dataloader import train_loader, test_loader, X_train, X_test, X_val, y_train, y_test, y_val

################################################################################################
input_size = 13
hidden1_size = 13
hidden2_size = 128
hidden3_size = 64
output_size = 50

################################################################################################

class DNN(nn.Module):
   def __init__(self, input_size, hidden2_size, hidden3_size, output_size):
       super(DNN, self).__init__()
       self.fc1 = nn.Linear(input_size, hidden1_size)
       self.relu1 = nn.ReLU()
       self.fc2 = nn.Linear(hidden1_size, hidden2_size)
       self.relu2 = nn.ReLU()
       self.fc3 = nn.Linear(hidden2_size, hidden3_size)
       self.relu3 = nn.ReLU()
       self.fc4 = nn.Linear(hidden3_size, output_size)
       self.relu4 = nn.ReLU()

   def forward(self, x):
       out = self.fc1(x)
       out = self.relu1(out)
       out = self.fc2(out)
       out = self.relu2(out)
       out = self.fc3(out)
       out = self.relu3(out)
       out = self.fc4(out)
       out = self.relu4(out)
       return out
################################################################################################
# Instantiate the model
batch_size = 50
n_iterations = 50
no_epochs = 80
model = DNN(input_size, hidden2_size, hidden3_size, output_size)

################################################################################################
# Define the loss criterion and optimizer
criterion = nn.CrossEntropyLoss()
learning_rate = 0.01
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
print(model)
########################################################################################################################
# train the network
iter = 0
for epoch in range(no_epochs):
   for i, (X_train, y_train) in enumerate(train_loader):
       optimizer.zero_grad()
       outputs = model(X_train)
       loss = criterion(outputs, torch.max(labels, 1)[1])
       loss.backward()
       optimizer.step()
       iter += 1
       if iter % 500 == 0:
           correct = 0
           total = 0
           for X_test, y_test in test_loader:
               outputs = model(X_test)
               _, predicted = torch.max(outputs.data, 1)
               total += labels.size(0)
               correct += (predicted == labels).sum()
           accuracy = 100 * correct / total
           print(iter, loss.data[0], accuracy)

Output:
I think no_epochs=0 with this initialization. Possibly (len(train_loader) / batch_size) > n_iterations. Then int(no_eps) = 0. Try to change no_epochs to 100 manually, for example.

no_eps = n_iterations / (len(train_loader) / batch_size)
no_epochs = int(no_eps)
for epoch in range(no_epochs):


Instruction: What does the copy_initial_weights documentation mean in the higher library for Pytorch?

Input:
I was trying to use the higher library for meta-learning and I was having issues understanding what the copy_initial_weights mean. The docs say:

  copy_initial_weights – if true, the weights of the patched module are copied to form the initial weights of the patched module, and thus are not part of the gradient tape when unrolling the patched module. If this is set to False, the actual module weights will be the initial weights of the patched module. This is useful when doing MAML, for example.

but that doesn't make much sense to me because of the following:

For example, "the weights of the patched module are copied to form the initial weights of the patched module" doesn't make sense to me because when the context manager is initiated a patched module does not exist yet. So it is unclear what we are copying from and to where (and why copying is something we want to do).

Also, "unrolling the patched module" does not make sense to me. We usually unroll a computaiton graph caused by a for loop. A patched module is just a neural net that has been modified by this library. Unrolling is ambiguous.

Also, there isn't a technical definition for "gradient tape".

Also, when describing what false is, saying that it's useful for MAML isn't actually useful because it doesn't even hint why it's useful for MAML.

Overall, it's impossible to use the context manager.

Any explanations and examples of what the that flag does in more precise terms would be really valuable.

Related:

gitissue: https://github.com/facebookresearch/higher/issues/30
new gitissue: https://github.com/facebookresearch/higher/issues/54
pytorch forum: https://discuss.pytorch.org/t/why-does-maml-need-copy-initial-weights-false/70387
pytorch forum: https://discuss.pytorch.org/t/what-does-copy-initial-weights-do-in-the-higher-library/70384
important question related to this on how the fmodel parameters are copied so that the optimizers work (and the use of deep copy): Why does higher need to deep copy the parameters of the base model to create a functional model?

Output:
I think it's more or less clear what this means now to me.

First I'd like to make some notation clear, specially with respect to indices wrt inner time step and outer time step (also known as episodes):

W^<inner_i, outer_i> = denotes the value a tensor has at time step inner_i, outer_i.

At the beginning of training a neural net has params:

W^<0,0>

and are held inside it's module. For the sake of explanation the specific tensor (for the base model) will be denoted:

W = the weight holding the weights for the model. This can be thought as the initialization of the model.

and will be updated with with an in-place operation (this is important since W is the placeholder for all W^<0,outer_i> for all outer step values during "normal" meta-learning) by the outer optimizer. I want to emphasize that W is the tensor for the normal Pytorch neural net base model. By changing this in-place with an outer optimizer (like Adam) we are effectively training the initialization. The outer optimizer will use the gradients wrt this tensor to do the update through the whole unrolled inner loop process.

When we say copy_initial_weights=False we mean that we will have a gradient path directly to W with whatever value it currently has. Usually the context manager is done before a inner loop after an outer step has been done so W will have W^<0,outer_i> for the current step. In particular the code that does this is this one for copy_initial_weight=False:

params = [ p.clone() if device is None else p.clone().to(device) for p in module.parameters() ]

this might look confusing if you're not familiar with clone but what it's doing is making a copy of the current weight of W. The unusual thing is that clone also remembers the gradient history from the tensor it came from (.clone() is as identity). It's main use it to add an extra layer of safety from the user doing dangerous in-place ops in it's differentiable optimizer. Assuming the user never did anything crazy with in-place ops one could in theory remove the .clone(). the reason this is confusing imho is because "copying in Pytorch" (clinging) does not automatically block gradient flows, which is what a "real" copy would do (i.e. create a 100% totally separate tensor). This is not what clone does and that is not what copy_initial_weights does.

When copy_initial_weights=True what really happens is that the weights are cloned and detached. See the code it eventually runs (here and here):

params = [_copy_tensor(p, safe_copy, device) for p in module.parameters()]

which runs copy tensor (assuming they are doing a safe copy i.e. doing the extra clone):

 t = t.clone().detach().requires_grad_(t.requires_grad)

Note that .detach() does not allocate new memory. It shares the memory with the original tensor, which is why the .clone() is needed to have this op be "safe" (usually wrt in-place ops).

So when copy_initial_weights they are copying and detaching the current value of W. This is usually W^<0,outer_i> if it's doing usual meta-learning in the inner adaptation loop. So the intended semantics of copy_initial_weight is that and the initial_weight they simply mean W. The important thing to note is that the intermediate tensors for the net in the inner loop are not denoted in my notation but they are fmodel.parameters(t=inner_i). Also if things are usually meta-learning we have fmodel.parameters(t=0) = W and it gets update in-place by the outer optimizer.

Note that because of the outer optimizer's in-place op and the freeing of the graphs we never take the derivate Grad_{W^<0,0>} with respect to the initial value of W. Which was something I initially thought we were doing.


Instruction: How does one reset the dataloader in pytorch?

Input: I was trying to reset the dataloader manually but was unable. I tried everything here https://discuss.pytorch.org/t/how-could-i-reset-dataloader-or-count-data-batch-with-iter-instead-of-epoch/22902/4 but no luck. Anyone know how to reset the data loader AND also have the suffle/randomness of the batches not be broken?

Output:
To reset a DataLoader then just enumerate the loader again. Each call to enumerate(loader) starts from the beginning.

To not break transformers that use random values, then reset the random seed each time the DataLoader is initialized.

def seed_init_fn(x):
   seed = args.seed + x
   np.random.seed(seed)
   random.seed(seed)
   torch.manual_seed(seed)
   return

loader = torch.utils.data.DataLoader(...., worker_init_fn = seed_init_fn)

while True:
   for i,data in enumerate(loader):
      # will always yield same data

See worker_init_fn in the documents:

https://pytorch.org/docs/master/data.html#torch.utils.data.DataLoader

Here is a better example:

https://github.com/pytorch/pytorch/issues/5059#issuecomment-404232359


Instruction: Pytorch - how to undersample using weightedrandomsampler

Input:
I have an unbalanced dataset and would like to undersample the class that is overrepresented.How do I go about it. I would like to use to weightedrandomsampler but I am also open to other suggestions.

So far I am assuming that my code will have to be structured kind of like the following. But I dont know how to exaclty do it.

trainset = datasets.ImageFolder(path_train,transform=transform)
...
sampler = data.WeightedRandomSampler(weights=..., num_samples=..., replacement=...)
...
trainloader = data.DataLoader(trainset, batchsize = batchsize, sampler=sampler)

I hope someone can help. Thanks a lot

Output:
From my understanding, pytorch WeightedRandomSampler 'weights' argument is somewhat similar to numpy.random.choice 'p' argument which is the probability that a sample will get randomly selected. Pytorch uses weights instead to random sample training examples and they state in the doc that the weights don't have to sum to 1 so that's what I mean that it's not exactly like numpy's random choice. The stronger the weight, the more likely that sample will get sampled.

When you have replacement=True, it means that training examples can be drawn more than once which means you can have copies of training examples in your train set that get used to train your model; oversampling. Alongside, if the weights are low COMPARED TO THE OTHER TRAINING SAMPLE WEIGHTS the opposite occurs which means that those samples have a lower chance of being selected for random sampling; undersampling.

I have no clue how the num_samples argument works when using it with the train loader but I can warn you to NOT put your batch size there. Today, I tried putting the batch size and it gave horrible results. My co-worker put the number of classes*100 and his results were much better. All I know is that you should not put the batch size there. I also tried putting the size of all my training data for num_samples and it had better results but took forever to train. Either way, play around with it and see what works best for you. I would guess that the safe bet is to use the number of training examples for the num_samples argument.

Here's the example I saw somebody else use and I use it as well for binary classification. It seems to work just fine. You take the inverse of the number of training examples for each class and you set all training examples with that class its respective weight.

A quick example using your trainset object

labels = np.array(trainset.samples)[:,1] # turn to array and take all of column index 1 which are the labels

labels = labels.astype(int) # change to int

majority_weight = 1/num_of_majority_class_training_examples

minority_weight = 1/num_of_minority_class_training_examples

sample_weights = np.array([majority_weight, minority_weight]) # This is assuming that your minority class is the integer 1 in the labels object. If not, switch places so it's minority_weight, majority_weight.

weights = samples_weights[labels] # this goes through each training example and uses the labels 0 and 1 as the index in sample_weights object which is the weight you want for that class.

sampler = WeightedRandomSampler(weights=weights, num_samples=, replacement=True)

trainloader = data.DataLoader(trainset, batchsize = batchsize, sampler=sampler)

Since the pytorch doc says that the weights don't have to sum to 1, I think you can also just use the ratio which between the imbalanced classes. For example, if you had 100 training examples of the majority class and 50 training examples of the minority class, it would be a 2:1 ratio. To counterbalance this, I think you can just use a weight of 1.0 for each majority class training example and a weight 2.0 for all minority class training examples because technically you want the minority class to be 2 times more likely to be selected which would balance your classes during random selection.

I hope this helped a little bit. Sorry for the sloppy writing, I was in a huge rush and saw that nobody answered. I struggled through this myself without being able to find any help for it either. If it doesn't make sense just say so and I'll re-edit it and make it more clear when I get free time.


Instruction: How can I matrix-multiply two PyTorch quantized Tensors?

Input:
I am new to tensor quantization, and tried doing something as simple as
import torch
x = torch.rand(10, 3)
y = torch.rand(10, 3)

x@y.T

with PyTorch quantized tensors running on CPU. I thus tried
scale, zero_point = 1e-4, 2
dtype = torch.qint32
qx = torch.quantize_per_tensor(x, scale, zero_point, dtype)
qy = torch.quantize_per_tensor(y, scale, zero_point, dtype)

qx@qy.T # I tried...

..and got as error

RuntimeError: Could not run 'aten::mm' with arguments from the
'QuantizedCPUTensorId' backend. 'aten::mm' is only available for these
backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId,
CPUTensorId, SparseCUDATensorId].

Is matrix multiplication just not supported, or am I doing something wrong?

Output:
It is not straight forward to implement matrix multiplication for quantized matrices. Therefore, the "conventional" matrix multiplication (@) does not support it (as your error message suggests).

You should look at quantized operations, e.g., torch.nn.quantized.functional.linear:

torch.nn.quantized.functional.linear(qx[None,...], qy.T)


Instruction: bias dimension definition in coding neural network

Input:
In the following Figure showing the code for defining the dimension of the bias b1 term, I wonder why the first dimension of bias b1 is not the batch size? Does it mean then it just assumes this bias is applied to all batches then?

If I specify the bias b1 dimension to be (batch_size, 256) then does it mean i am applying a different b1 to different batch? But theoretically it should still work right? Also what is the difference between tensor (256), (256,) and (256,1)...?

Figure: dimension definition of nn

Output:
The weights and biases of your neural network layer are not specified in terms of batch size.

eg: w1 = torch.randn(784,256) :
This is a 2D matrix you're going to use for a matrix multiply.
784 is the dimension of your input image without considering batch size. (I'm guessing this is for mnist? it looks like you're flattening the 2d images to a 1d vector so 28*28=784).

256 is your output dimension of your output (how many logits you're using)

Similarly, b1 = torch.randn(256):
This is an 1D vector you're just adding to the logits.

256 is the dimension of the logits

Pytorch automaticallly broadcasts (repeats) these over the batch dimension for all your operations, so it doesn't matter what the batch size was.

Eg. eg in adding, b1 is automatically repeated over the first dimension, so it's actual shape for the add is (batch_size, 256).

By convention, pytorch "aligns" dimensions from right to left.

if any are missing, it then repeats the tensor over the missing dimension
If any dimension is 1, it repeats the tensor over that dimension to match the other operand.
Eg (copied from the docs on broadcasting)

>>> x=torch.empty(5,7,3)
>>> y=torch.empty(5,7,3)
# same shapes are always broadcastable (i.e. the above rules always hold)

>>> x=torch.empty((0,))
>>> y=torch.empty(2,2)
# x and y are not broadcastable, because x does not have at least 1 dimension

# can line up trailing dimensions
>>> x=torch.empty(5,3,4,1)
>>> y=torch.empty(  3,1,1)
# x and y are broadcastable.
# 1st trailing dimension: both have size 1
# 2nd trailing dimension: y has size 1
# 3rd trailing dimension: x size == y size
# 4th trailing dimension: y dimension doesn't exist

# but:
>>> x=torch.empty(5,2,4,1)
>>> y=torch.empty(  3,1,1)
# x and y are not broadcastable, because in the 3rd trailing dimension 2 != 3

This is really convenient because it means you don't have to redefine your neural net every time you want to use a different batch_size

Here's a link if you want to learn more about broadcasting in pytorch

  Also what is the difference between tensor (256), (256,) and (256,1)

the first two are exactly the same; python generally allows for trailing commas in tuple expressions. You are creating a 1D vector of 256 elements.

The last one is different; you are creating a 2D tensor where the first dimension is 256 and the second dimension is 1. The underlying data is the same, and it doesn't matter as long as you're consistent about which you're using, but if you mix them, it often leads to undesired behavior:

Eg:

a = torch.randn(256)
b = torch.randn(256)
c = a + b
c.shape
>>> torch.Size([256])

Simple: they just add element-wise.

But notice what happens when one of them is shaped (-1,1):

b = b.view(-1,1) # -1 here means torch will infer the shape of this dimension based on the known size of the tensor and all other specified dimensions
b.shape
>>> torch.Size([256, 1])

c = a + b

Now because of broadcasting rules

a is repeated over the first dimension so it has the same number of dimensions as b, so it automatically interpretes a as tensor(256,256)
b is repeated so it's last dimension (1) now matches the dimension of a (256)

so:

c.shape
>>> torch.Size([256, 256])

Hint: The broadcasting rules can be hard to remember, and are often the source of bugs. When in doubt about tensor shapes, it's worth running your code in an interpreter line by line with dummy data and just checking what the shape of each tensor is eg print(torch.mm(input,w1).shape)


Instruction: Why is there an error (numpy.float64 cannot be interpreted as in integer) in pytorch sample code

Input:
I was trying to run the sample code found here:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I get a crash in the class CocoEvaluator() constructor in coco_eval.py where the following line of code:

for iou_type in iou_types:
        self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type)

will crash with the warning "object of type class 'numpy.float64' cannot be safely interpreted as an integer."

iou_type is a string 'bbox'

COCOeval is a class from pycocotools (pycocotools.cocoeval.COCOeval)

coco_gt is the return value from get_coco_api_from_dataset(data_loader.dataset)

its not clear to me where the numpy.float64 value is being used here, or what I can change to fix this

Output:
The problem most likely lies in the numpy version. Numpy version 1.18.+ usually throws this error. However when changing to numpy 1.17.4 the problem is fixed.
as shown here
-> https://github.com/pytorch/vision/issues/1700
-> https://www.kaggle.com/questions-and-answers/90865
#check for version number
np.version.version

#downgrade version
!pip install numpy==1.17.4

This fixed it for me, hope it helps.


Instruction: Gradient Computation broken by Sigmoid function in Pytorch

Input:
Hey I have been struggling with this weird problem. Here is my code for the Neural Net:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv_3d_=nn.Sequential(
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU(),
            nn.Conv3d(1,1,9,1,4),
            nn.LeakyReLU()
        )

        self.linear_layers_ = nn.Sequential(

            nn.Linear(batch_size*32*32*32,batch_size*32*32*3),
            nn.LeakyReLU(),
            nn.Linear(batch_size*32*32*3,batch_size*32*32*3),
            nn.Sigmoid()
        )

    def forward(self,x,y,z):
        conv_layer = x + y + z
        conv_layer = self.conv_3d_(conv_layer)
        conv_layer = torch.flatten(conv_layer)
        conv_layer = self.linear_layers_(conv_layer)
        conv_layer = conv_layer.view((batch_size,3,input_sizes,input_sizes))
        return conv_layer

The weird problem I am facing is that running this NN gives me an error

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3072]], which is output 0 of SigmoidBackward, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The stack trace shows that the issue is in line

conv_layer = self.linear_layers_(conv_layer)

However, if I replace the last activation function of my FCN from nn.Sigmoid() to nn.LeakyRelu(), the NN executes properly.

Can anyone tell me why Sigmoid activation function is causing my backward computation to break?

Output:
I found the problem with my code. I delved deeper into what in-place actually meant. So, if you check the line

conv_layer = self.linear_layers_(conv_layer)

linear_layers_ of the assignment is changing the values of conv_layer in-place and as a result the values are getting overwritten and because of this, gradient computation fails. Easy solution for this problem is to use the clone() function

i.e.

conv_layer = self.linear_layers_(conv_layer).clone()

This creates a copy of the right hand computation and Autograd is able to store the reference of the computation graph.


Instruction: Validation loss for pytorch Faster-RCNN

Input:
I’m currently doing object detection on a custom dataset using transfer learning from a pytorch pretrained Faster-RCNN model (like in torchvision tutorial).
I would like to compute validation loss dict (as in train mode)  at the end of each epoch.
I can just run model in train mode for validation like this:

model.train()
for images, targets in data_loader_val:
    images = [image.to(device) for image in images]
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

    with torch.no_grad():
         val_loss_dict = model(images, targets)
         print(val_loss_dict)

but I don't think, that it's "correct" way to validate (cause some special layers like dropout and batch norm works different in eval/train mode). And in eval mode model returns predicted bboxes (as expected).  Can I use some build-in function for this?

Thanks.

Output:
There was some discussion about this issue here. The conclusion there is that it is absolutely valid to calculate validation loss in train mode. The numerical value of the val loss in itself is not meaningful, only the trend is important to prevent overfitting. Therefore while train mode does alter the numerical value of the loss, it's still valid to use.

There is however another issue with efficiency here, in case you also need the model outputs in the validation process (for calculating IoU, accuracy, etc. as is often the case). Right now RCNN in torchvision gives you either losses or outputs, depending on training/eval mode.
UPDATE:
I realized this fix is not working unfortunately. All submodules would have to be patched to calculate both losses and outputs. Too bad.

My dirty solution was patching the GeneralizedRCNN class from which
FasterRCNN inherits. The problem is in
this
line, in eager_outputs(). The workaround:
    return losses, detections

model = fasterrcnn_resnet50_fpn() model.eager_outputs =
eager_outputs_patch

Now you can get both outputs after a single inference run: model.train() with torch.no_grad(): loss_dict, outputs = model(images, targets). # yaay, now we have both! Note that you still need to put your model to train mode in
order to have the losses too. In eval mode GeneralizedRCNN's
submodules (rpn, roi_heads) don't calculate any loss, and loss_dict
is empty.


Instruction: challenging special numpy operation

Input:
I have a NumPy array that is full of indices of numbers input. I want to check if certain indices indices are in it. Say that the i'th row of input, input[i] has entries j_1<...<j_n that their values belong to indices. I would like to switch the value of input[i,j_n] with a random value from indices. How can I do it elegantly and efficiently?

Example:

input = [[1,2,3],[4,5,6],[7,8,9]]

indices = [2,4,6]

input --> [[1,4,3],[4,5,4],[7,8,9]]

Output:
Something like that:

import random

input = [[i if i not in indices else random.choice(indices) for i in x] for x in input]

Better, of course, to check in set instead of list:

import random

d = {*indices}
input = [[i if i not in d else random.choice(indices) for i in x] for x in input]


Instruction: Best way to run a trained PyTorch LSTM/GRU model fully in the browser

Input:
I'm looking into running a trained PyTorch model (containing LSTM/GRU layers) fully in the browser (no backend) as part of an interactive blog post. I've looked at ONNX.js, and that works great, but not for a model containing a GRU layer. I saw someone comment on the ONNX.js github that Gated RNN's are not supported yet, but that was over half a year ago and I can't find any other information about this.

Other than that, it seems like the best option would be to just rewrite the model in Tensorflow and export to Tensorflow.js.

Is there an easier and more direct solution?

Output:
There is this thread, that describes the options but is not receiving a lot of attention.

In summary, as of May 2020, there are only two options:

1) ONNX.js but its development is currently stale.

2) Converting the model to Tensorflow.

Technically there is a third one, that includes no server. And that is to run the model in a Mobile Application.


Instruction: Pytorch Runtime Error - The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension

Input:
I am trying to train a Faster RCNN Network on a custom dataset consisting of images for object detection. However, I don't want to directly give an RGB image as input, I actually need to pass it through another network (a feature extractor) along with the corresponding thermal image and give the extracted features as the input to the FRCNN Network. The feature extractor combines these two images into a 4 channel tensor and the output is a 5 channel tensor. It is this 5 channel tensor that I wish to give as input to the Faster RCNN Network.

I followed the PyTorch docs for Object Detection Finetuning (link here) and came up with the following code to suit my dataset.

class CustomDataset(torch.utils.data.Dataset):

    def __getitem__(self, idx):
        self.num_classes = 5
        img_rgb_path = os.path.join(self.root, "rgb/", self.rgb_imgs[idx])
        img_thermal_path = os.path.join(self.root, "thermal/", self.thermal_imgs[idx])

        img_rgb = Image.open(img_rgb_path)
        img_rgb = np.array(img_rgb)
        x_rgb = TF.to_tensor(img_rgb)
        x_rgb.unsqueeze_(0)

        img_thermal = Image.open(img_thermal_path)
        img_thermal = np.array(img_thermal)
        img_thermal = np.expand_dims(img_thermal,-1)
        x_th = TF.to_tensor(img_thermal)
        x_th.unsqueeze_(0)

        print(x_rgb.shape)  # shape of [3,640,512]
        print(x_th.shape) # shape of [1,640,512]

        input = torch.cat((x_rgb,x_th),dim=1) # shape of [4,640,512]

        img = self.feature_extractor(input) #  My custom feature extractor which returns a 5 dimensional tensor

        print(img.shape) # shape of [5,640,512]

        filename = os.path.join(self.root,'annotations',self.annotations[idx])
        tree = ET.parse(filename)
        objs = tree.findall('object')

        num_objs = len(objs)
        boxes = np.zeros((num_objs, 4), dtype=np.uint16)
        labels = np.zeros((num_objs), dtype=np.float32)
        seg_areas = np.zeros((num_objs), dtype=np.float32)

        boxes = []
        for ix, obj in enumerate(objs):
            bbox = obj.find('bndbox')
            x1 = float(bbox.find('xmin').text)
            y1 = float(bbox.find('ymin').text)
            x2 = float(bbox.find('xmax').text)
            y2 = float(bbox.find('ymax').text)

            cls = self._class_to_ind[obj.find('name').text.lower().strip()]
            boxes.append([x1, y1, x2, y2])
            labels[ix] = cls
            seg_areas[ix] = (x2 - x1 + 1) * (y2 - y1 + 1)

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        seg_areas = torch.as_tensor(seg_areas, dtype=torch.float32)
        labels = torch.as_tensor(labels, dtype=torch.float32)

        target =  {'boxes': boxes,
                'labels': labels,
                'seg_areas': seg_areas,
                }

        return img,target

My main function code is as follows

import utils

def train_model(model, criterion,dataloader,num_epochs):
    since = time.time()

    best_model = model
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        optimizer = torch.optim.SGD(params, lr=0.005,
                            momentum=0.9, weight_decay=0.0005)

        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                               step_size=3,
                                               gamma=0.1)

        # optimizer = lr_scheduler(optimizer, epoch)
        model.train()  # Set model to training mode

        running_loss = 0.0
        running_corrects = 0

        for data in dataloader:
            inputs, labels = data[0][0], data[1]

            inputs = inputs.to(device)
            # zero the parameter gradients

            optimizer.zero_grad()

            # forward
            outputs = model(inputs, labels)
            _, preds = torch.max(outputs.data, 1)
            loss = criterion(outputs, labels)

            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            running_corrects += torch.sum(preds == labels).item()

        epoch_loss = running_loss / len(dataloader)
        epoch_acc = running_corrects / len(dataloader)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))

backbone = torchvision.models.mobilenet_v2(pretrained=True).features
backbone.out_channels = 1280

anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),
                                   aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                output_size=7,
                                                sampling_ratio=2)
num_classes = 5

model = FasterRCNN(backbone = backbone,num_classes=5,rpn_anchor_generator=anchor_generator,box_roi_pool=roi_pooler)

dataset = CustomDataset('train_folder/')
data_loader_train = torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=True,collate_fn=utils.collate_fn)

train_model(model, criterion, data_loader_train, num_epochs=10)

The collate_fn defined in the utils.py file is the following

def collate_fn(batch):
    return tuple(zip(*batch))

I, however, get the following error while training

Traceback (most recent call last):
  File "train.py", line 147, in <module>
    train_model(model, criterion, data_loader_train, num_epochs)
  File "train.py", line 58, in train_model
    outputs = model(inputs, labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py", line 66, in forward
    images, targets = self.transform(images, targets)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py", line 46, in forward
    image = self.normalize(image)
  File "/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/transform.py", line 66, in normalize
    return (image - mean[:, None, None]) / std[:, None, None]
RuntimeError: The size of tensor a (5) must match the size of tensor b (3) at non-singleton dimension 0

I am a newbie in Pytorch.

Output:
The backbone network you are using for the FasterRCNN is a pretrained mobilenet_v2.
The input channel of a network is decided by the number of channels of the input data. Since the (backbone) model is pretrained (on natural images?) with 3 channels 3xNxM, you cannot use it for tensors of dimension 5xPxQ (skipping the singleton <batch_size> dimension).

Basically, you have 2 options,
1. Reduce the output channel dimension of the 1st network to 3 (better if you are training it from scratch)
2. Make a new backbone for the FasterRCNN with 5 channels in input and train it from scratch.

As for explaining the error message,

return (image - mean[:, None, None]) / std[:, None, None]

Pytorch is trying to normalize the input image where your input image has dimension (5,M,N)  and teh tensors mean and std have 3 channels instead of 5


Instruction: Torch.sort and argsort sorting randomly in case of same element

Input:
When same elements are encountered, torch.sort and argsort sort the tensor in random manner.
This is not the case in numpy.
I have a list of elements already sorted according to the second column and now i want to sort it using the first column but preserve the earlier sort in case of tie in the new sorting.

import torch

a = torch.tensor(
        [[ 0., 3.],
        [ 2., 3.],
        [ 2., 2.],
        [10., 2.],
        [ 0., 2.],
        [ 6., 2.],
        [10., 1.],
        [ 2., 1.],
        [ 0., 1.],
        [ 6., 1.],
        [10., 0.],
        [12., 0.]]
)
print(a[torch.argsort(a[:, 0])])

Output:

tensor([[ 0.,  3.],
        [ 0.,  2.],
        [ 0.,  1.],
        [ 2.,  1.],
        [ 2.,  2.],
        [ 2.,  3.],
        [ 6.,  1.],
        [ 6.,  2.],
        [10.,  1.],
        [10.,  2.],
        [10.,  0.],
        [12.,  0.]])

Numpy:

import numpy as np

a = np.array(
        [[ 0., 3.],
        [ 2., 3.],
        [ 2., 2.],
        [10., 2.],
        [ 0., 2.],
        [ 6., 2.],
        [10., 1.],
        [ 2., 1.],
        [ 0., 1.],
        [ 6., 1.],
        [10., 0.],
        [12., 0.]]
)
print(a[np.argsort(a[:, 0])])

Output:

[[ 0.  3.]
 [ 0.  2.]
 [ 0.  1.]
 [ 2.  3.]
 [ 2.  2.]
 [ 2.  1.]
 [ 6.  2.]
 [ 6.  1.]
 [10.  2.]
 [10.  1.]
 [10.  0.]
 [12.  0.]]

What could be the reason for this? And what can I do to avoid it?

Output:
As per torch 1.9.0 you can run the sort with option stable=True. See https://pytorch.org/docs/1.9.0/generated/torch.sort.html?highlight=sort#torch.sort
>>> x = torch.tensor([0, 1] * 9)
>>> x.sort()
torch.return_types.sort(
    values=tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
    indices=tensor([ 2, 16,  4,  6, 14,  8,  0, 10, 12,  9, 17, 15, 13, 11,  7,  5,  3,  1]))
>>> x.sort(stable=True)
torch.return_types.sort(
    values=tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1]),
    indices=tensor([ 0,  2,  4,  6,  8, 10, 12, 14, 16,  1,  3,  5,  7,  9, 11, 13, 15, 17]))

The documentation says this is only on the CPU, but it will come to GPU sorting soon, since that documentation warning has been removed in the master branch at github (as per https://github.com/pytorch/pytorch/pull/61685)


Instruction: Install Pytorch GPU with pre-installed CUDA and cudnn

Input:
As the title suggests, I have pre-installed CUDA and cudnn (my Tensorflow is using them).

The version of CUDA is 10.0 from nvcc --version.

The versiuon of cudnn is 7.4.

I am trying to install pytorch in a conda environment using conda install pytorch torchvision cudatoolkit=10.0 -c pytorch.

However, the installed pytorch does not detect my GPU successfully.

Does anyone know if there is a way to install GPU-version pytorch with a specific CUDA and cudnn version? I do not want to change CUDA and cudnn version because my Tensorflow is using them.

Any ideas would be appreciated!

Output: So I solved this myself finally. The issue is that I didn't reboot my system after installing pytorch. After rebooting, torch.cuda.is_available() returns True as expected.


Instruction: Pythonic Nested for - loops in Python

Input:
I am working on this code where I have nested for loops. a_list and b_list are list of tuples, where each tuple is made up of two tensors [(tens1, tens2), ...]. I am trying to compute the similarity of every tens1 in a_list to every tens1 in b_list. Below is the code I have. And the nested loop appears to be a bottleneck. Is there a better way(pythonic) that I can re-write the loops?

a2b= defaultdict(dict)
b2a= defaultdict(dict)
ab_sim = []

for a, vec_a in a_list:
    for b, vec_b in b_list:
        # Ignore combination if the first element in both a and b are same
        if a[0] == b[0]:
            continue
        # Calculate cosine similarity of combination
        sim = self.calculate_similarity(vec_a, vec_b )
        a2b[a][b] = sim
        b2a[b][a] = sim
        ab_sim.append(sim)

The calculate_similarity is just a method computing cosine similarity. a_list and b_list could be of any size. I have b2a and a2b because I need them for other computations.

Output:
You could use a dictionary comprehension:

a2b = {a: {b: self.calculate_similarity(vec_a, vec_b )
       for (b, vec_b) in b_list if a[0] != b[0]} for (a, vec_a) in a_list}


Instruction: Indexing in two dimensional PyTorch Tensor using another Tensor

Input:
Suppose that tensor A is defined as:

 1  2  3  4
 5  6  7  8
 9 10 11 12
13 14 15 16

I'm trying to extract a flat array out of this matrix by using another tensor as indices. For example, if the second tensor is defined as:

0
1
2
3

I want the result of the indexing to be 1-D tensor with the contents:

1
6
11
16

It doesn't seem to behave like NumPy; I've tried A[:, B] but it just throws an error for not being able to allocate an insane amount of memory and I've no idea why!

Output:
1st Approach: using torch.gather
torch.gather(A, 1, B.unsqueeze_(dim=1))

if you want one-dimensional vector, you can add squeeze to the end:
torch.gather(A, 1, B.unsqueeze_(dim=1)).squeeze_()

2nd Approach: using list comprehensions
You can use list comprehensions to select the items at specific indexes, then they can be concatenated using the torch.stack. An importat point here is that you should not use torch.tensor to create a new tensor from a list, if you do, you will break the chain (you cannot calculate gradient through that node):
torch.stack([A[i, B[i]] for i in range(A.size()[0])])


Instruction: TypeError: can't multiply sequence by non-int type of 'tuple' in pytorch

Input:
The code as below:

class L2Norm(nn.Module):
    def __init__(self):
        super(L2Norm, self).__init__()
        self.eps = 1e-10
    def forward(self, x):
        norm = torch.sqrt(torch.sum(x * x, dim = 1) + self.eps)
        x = x / norm.unsqueeze(-1).expand_as(x)
        return x

I want to normalize the features. The input x is the output of nn.Linear() in FC layer (x = self.fc1(x)).

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        if self.feature:
            return x
#        x = self.last_bn(x)
        x = self.fc2(x)

        return x

I print the x as below:

(tensor([[-0.8409,  8.6126, -1.6639,  ..., -3.3563, 10.0872,  2.4730],
        [-1.3959,  0.5608, -0.9233,  ...,  0.4385, -0.7089, -1.3401],
        [ 0.5742, -3.8479,  1.7756,  ..., -4.2798, -5.0684, -0.9032],
        ...,
        [ 0.9205,  3.1602, -3.9247,  ..., -2.1396,  4.0262,  2.8075],
        [-0.2024,  0.5603,  0.0491,  ..., -0.1716, -0.2513,  0.1179],
        [ 4.8053,  0.3062, -1.6867,  ..., -1.5749,  0.5193,  0.8671]],
       device='cuda:0', grad_fn=<GatherBackward>), tensor([[-156.3423, -145.1505, -156.6586,  ..., -157.9570, -141.1895,
         -155.2964],
        [ -31.9854,  -30.2333,  -31.1459,  ...,  -30.3290,  -30.8740,
          -31.8696],
        [-141.5926, -144.1404, -141.1151,  ..., -144.7264, -145.9508,
         -141.7867],
        ...,
        [-193.6224, -192.4931, -195.6285,  ..., -194.2939, -191.7269,
         -192.7527],
        [  -5.8791,   -4.5035,   -5.6987,  ...,   -5.8316,   -5.9696,
           -5.6506],
        [ -77.5002,  -83.8829,  -84.7204,  ...,  -84.6169,  -83.8326,
          -83.6949]], device='cuda:0', grad_fn=<GatherBackward>))

However, the mistake occurs in the torch.sum(x*x, dim=1). I have any solution for the mistake.

Output:
x is a tuple of two tensors, as shown in your output. x * x would require a way to multiply two tuples.

If I simply define x as a tuple of ints, e.g. x=(1, 1), and tried the same code: x * x, I get the same error:

>>> x=(1,1)
>>> x * x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't multiply sequence by non-int of type 'tuple'

# though tuple times an int does work:
>>> x * 3
(1, 1, 1, 1, 1, 1)

In your case, x should probably be a tensor, not a tuple.


Instruction: how visualize multi channel of feature from PyTorch?

Input:
I'm almost newbie at PyTorch

One of my output size from conv is [1, 25, 8, 32]
(25=channel, 8=height, 32=width)

I can use squeeze and make it to [25, 8, 32].

But I'm confused with 25 channel.

When I want to visualize sum of 25 channel and make to one GRAYorRGB image(1or3x8x32),How can i deal with in code??

I can use matplot or tensorboardX for visualizing..

Output:
It is difficult to visualize images with more than 3 channels and it is unclear what a feature vector in 25 dimensional space actually looks like.

The most straight forward approach would be to visualize the 8x32 feature maps you have as separate 25 gray scale images of size 8x32. Each image will show how how "sensitive" is a specific neuron/conv filter/channel (these are all equivalent) to the input at a certain spatial location.

There are more intricate methods for feature visualization, you can find more details about them in this blog post.


Instruction: How to change certain values in a torch tensor based on an index in another torch tensor?

Input:
This is an issue I'm running while convertinf DQN to Double DQN for the cartpole problem. I'm getting close to figuring it out.

tensor([0.1205, 0.1207, 0.1197, 0.1195, 0.1204, 0.1205, 0.1208, 0.1199, 0.1206,
        0.1199, 0.1204, 0.1205, 0.1199, 0.1204, 0.1204, 0.1203, 0.1198, 0.1198,
        0.1205, 0.1204, 0.1201, 0.1205, 0.1208, 0.1202, 0.1205, 0.1203, 0.1204,
        0.1205, 0.1206, 0.1206, 0.1205, 0.1204, 0.1201, 0.1206, 0.1206, 0.1199,
        0.1198, 0.1200, 0.1206, 0.1207, 0.1208, 0.1202, 0.1201, 0.1210, 0.1208,
        0.1205, 0.1205, 0.1201, 0.1193, 0.1201, 0.1205, 0.1207, 0.1207, 0.1195,
        0.1210, 0.1204, 0.1209, 0.1207, 0.1187, 0.1202, 0.1198, 0.1202])
tensor([ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True, False,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True, False,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         True,  True,  True,  True])

As you can see here two tensors.
The first has the q values I want
but,
some values need to be changed to zeros because of it an end state.
The second tensor shows where it will be zeros.

At the index where the Boolean value is false is the equivalent spot for where the upper tensor needs to be zeros.
I am not sure how to do that.

Output:
You can use torch.where - torch.where(condition, x, y)

Ex.:

>>> x = tensor([0.2853, 0.5010, 0.9933, 0.5880, 0.3915, 0.0141, 0.7745,
                0.0588, 0.4939, 0.0849])
>>> condition = tensor([False,  True,  True,  True, False, False,  True,
                        False, False, False])

>>> # It's equivalent to `torch.where(condition, x, tensor(0.0))`
>>> x.where(condition, tensor(0.0))
tensor([0.0000, 0.5010, 0.9933, 0.5880, 0.0000, 0.0000, 0.7745,
        0.0000, 0.0000,0.0000])


Instruction: How forward() method is used when it have more than one two input parameters in pytorch

Input:
Can someone tell me the concept behind the multiple parameters in forward() method?
Generally, the implementation of forward() method has two parameters

self
input

if a forward method has more than these parameters how PyTorch is using the forward method.

Let's consider this codebase:
https://github.com/bamps53/kaggle-autonomous-driving2019/blob/master/models/centernet.py
here online 236 authors have used forward method with two more parameters:

centers
return_embeddings

I am unable to find a single article that can answer my query on what condition Line 254(return_embeddings:) and Line 257(if centers is not None:) will execute. As per my knowledge forward, the method is internally called by nn module. Can someone please put some lights on this?

Output:
Forward function set by you. That means you can add more parameters as you want. For example, you could add inputs as shown below
def forward(self, input1, input2, input3):
    x = self.layer1(input1)
    y = self.layer2(input2)
    z = self.layer3(input3)

    net = torch.cat((x,y,z),1)

    return net

You have to control your parameters while feeding the network. Layers couldn't be feed with more than a parameter. Therefore, you need to extract features from your input one by one and concatenate with torch.cat((x,y),1)(1 for dimension) them.


Instruction: Training TFBertForSequenceClassification with custom X and Y data

Input:
I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library.

I followed the example given on their github page, I am able to run the sample code with given sample data using tensorflow_datasets.load('glue/mrpc').
However, I am unable to find an example on how to load my own custom data and pass it in
model.fit(train_dataset, epochs=2, steps_per_epoch=115, validation_data=valid_dataset, validation_steps=7).

How can I define my own X, do tokenization of my X and prepare train_dataset with my X and Y. Where X represents my input text and Y represents classification category of given X.

Sample Training dataframe :

    text    category_index
0   Assorted Print Joggers - Pack of 2 ,/ Gray Pri...   0
1   "Buckle" ( Matt ) for 35 mm Width Belt  0
2   (Gagam 07) Barcelona Football Jersey Home 17 1...   2
3   (Pack of 3 Pair) Flocklined Reusable Rubber Ha...   1
4   (Summer special Offer)Firststep new born baby ...   0

Output:
Fine Tuning Approach
There are multiple approaches to fine-tune BERT for the target tasks.

Further Pre-training the base BERT model
Custom classification layer(s) on top of the base BERT model  being trainable
Custom classification layer(s) on top of the base BERT model being non-trainable (frozen)

Note that the BERT base model has been pre-trained only for two tasks as in the original paper.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

3.1 Pre-training BERT ...we pre-train BERT using two unsupervised tasks

Task #1: Masked LM
Task #2: Next Sentence Prediction (NSP)

Hence, the base BERT model is like half-baked which can be fully baked for the target domain (1st way). We can use it as part of our custom model training with the base trainable (2nd) or not-trainable (3rd).

1st approach
How to Fine-Tune BERT for Text Classification? demonstrated the 1st approach of Further Pre-training, and pointed out the learning rate is the key to avoid Catastrophic Forgetting where the pre-trained knowledge is erased during learning of new knowledge.

We find that a lower learning rate, such as 2e-5,
is necessary to make BERT overcome the catastrophic forgetting problem. With an aggressive learn rate of 4e-4, the training set fails to converge.

Probably this is the reason why the BERT paper used 5e-5, 4e-5, 3e-5, and 2e-5 for fine-tuning.

We use a batch size of 32 and fine-tune for 3 epochs over the data for all GLUE tasks. For each task, we selected the best fine-tuning learning rate (among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set

Note that the base model pre-training itself used higher learning rate.

bert-base-uncased - pretraining

The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size of 256. The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%. The optimizer used is Adam with a learning rate of 1e-4, β1=0.9 and β2=0.999, a weight decay of 0.01, learning rate warmup for 10,000 steps and linear decay of the learning rate after.

Will describe the 1st way as part of the 3rd approach below.
FYI:
TFDistilBertModel is the bare base model with the name distilbert.
Model: "tf_distil_bert_model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
distilbert (TFDistilBertMain multiple                  66362880
=================================================================
Total params: 66,362,880
Trainable params: 66,362,880
Non-trainable params: 0

2nd approach
Huggingface takes the 2nd approach as in Fine-tuning with native PyTorch/TensorFlow where TFDistilBertForSequenceClassification has added the custom classification layer classifier on top of the base distilbert model being trainable. The small learning rate requirement will apply as well to avoid the catastrophic forgetting.
from transformers import TFDistilBertForSequenceClassification

model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')
optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)
model.compile(optimizer=optimizer, loss=model.compute_loss) # can also use any keras loss fn
model.fit(train_dataset.shuffle(1000).batch(16), epochs=3, batch_size=16)

Model: "tf_distil_bert_for_sequence_classification_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
distilbert (TFDistilBertMain multiple                  66362880
_________________________________________________________________
pre_classifier (Dense)       multiple                  590592
_________________________________________________________________
classifier (Dense)           multiple                  1538
_________________________________________________________________
dropout_59 (Dropout)         multiple                  0
=================================================================
Total params: 66,955,010
Trainable params: 66,955,010  <--- All parameters are trainable
Non-trainable params: 0

Implementation of the 2nd approach
import pandas as pd
import tensorflow as tf
from sklearn.model_selection import train_test_split
from transformers import (
    DistilBertTokenizerFast,
    TFDistilBertForSequenceClassification,
)

DATA_COLUMN = 'text'
LABEL_COLUMN = 'category_index'
MAX_SEQUENCE_LENGTH = 512
LEARNING_RATE = 5e-5
BATCH_SIZE = 16
NUM_EPOCHS = 3

# --------------------------------------------------------------------------------
# Tokenizer
# --------------------------------------------------------------------------------
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
def tokenize(sentences, max_length=MAX_SEQUENCE_LENGTH, padding='max_length'):
    """Tokenize using the Huggingface tokenizer
    Args:
        sentences: String or list of string to tokenize
        padding: Padding method ['do_not_pad'|'longest'|'max_length']
    """
    return tokenizer(
        sentences,
        truncation=True,
        padding=padding,
        max_length=max_length,
        return_tensors="tf"
    )

# --------------------------------------------------------------------------------
# Load data
# --------------------------------------------------------------------------------
raw_train = pd.read_csv("./train.csv")
train_data, validation_data, train_label, validation_label = train_test_split(
    raw_train[DATA_COLUMN].tolist(),
    raw_train[LABEL_COLUMN].tolist(),
    test_size=.2,
    shuffle=True
)

# --------------------------------------------------------------------------------
# Prepare TF dataset
# --------------------------------------------------------------------------------
train_dataset = tf.data.Dataset.from_tensor_slices((
    dict(tokenize(train_data)),  # Convert BatchEncoding instance to dictionary
    train_label
)).shuffle(1000).batch(BATCH_SIZE).prefetch(1)
validation_dataset = tf.data.Dataset.from_tensor_slices((
    dict(tokenize(validation_data)),
    validation_label
)).batch(BATCH_SIZE).prefetch(1)

# --------------------------------------------------------------------------------
# training
# --------------------------------------------------------------------------------
model = TFDistilBertForSequenceClassification.from_pretrained(
    'distilbert-base-uncased',
    num_labels=NUM_LABELS
)
optimizer = tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE)
model.compile(
    optimizer=optimizer,
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)
model.fit(
    x=train_dataset,
    y=None,
    validation_data=validation_dataset,
    batch_size=BATCH_SIZE,
    epochs=NUM_EPOCHS,
)

3rd approach
Basics
Please note that the images are taken from A Visual Guide to Using BERT for the First Time and modified.
Tokenizer
Tokenizer generates the instance of BatchEncoding which can be used like a Python dictionary and the input to the BERT model.

BatchEncoding

Holds the output of the encode_plus() and batch_encode() methods (tokens, attention_masks, etc).

This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space.
Parameters

data (dict) – Dictionary of lists/arrays/tensors returned by the encode/batch_encode methods (‘input_ids’, ‘attention_mask’, etc.).

The data attribute of the class is the tokens generated which has input_ids and attention_mask elements.
input_ids

input_ids

The input ids are often the only required parameters to be passed to the model as input. They are token indices, numerical representations of tokens building the sequences that will be used as input by the model.

attention_mask

Attention mask

This argument indicates to the model which tokens should be attended to, and which should not.

If the attention_mask is 0, the token id is ignored. For instance if a sequence is padded to adjust the sequence length, the padded words should be ignored hence their attention_mask are 0.
Special Tokens
BertTokenizer addes special tokens, enclosing a sequence with [CLS] and [SEP]. [CLS] represents Classification and [SEP] separates sequences. For Question Answer or Paraphrase tasks, [SEP] separates the two sentences to compare.
BertTokenizer

cls_token (str, optional, defaults to "[CLS]")The Classifier Token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens.
sep_token (str, optional, defaults to "[SEP]")The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens.

A Visual Guide to Using BERT for the First Time show the tokenization.

[CLS]
The embedding vector for [CLS] in the output from the base model final layer represents the classification that has been learned by the base model. Hence feed the embedding vector of  [CLS] token into the classification layer added on top of the base model.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

The first token of every sequence is always a special classification token ([CLS]). The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks. Sentence pairs are packed together into a single sequence. We differentiate the sentences in two ways. First, we separate them with a special token ([SEP]). Second, we add a learned embedding to every token indicating whether it belongs to sentence A or sentence B.

The model structure will be illustrated as below.

Vector size
In the model distilbert-base-uncased, each token is embedded into a vector of size 768. The shape of the output from the base model is (batch_size, max_sequence_length, embedding_vector_size=768). This accords with the BERT paper about the BERT/BASE model (as indicated in distilbert-base-uncased).

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT/BASE (L=12, H=768, A=12, Total Parameters=110M) and BERT/LARGE (L=24, H=1024, A=16, Total Parameters=340M).

Base Model - TFDistilBertModel

Hugging Face Transformers: Fine-tuning DistilBERT for Binary Classification Tasks

TFDistilBertModel class to instantiate the base DistilBERT model without any specific head on top (as opposed to other classes such as TFDistilBertForSequenceClassification that do have an added classification head).
We do not want any task-specific head attached because we simply want the pre-trained weights of the base model to provide a general understanding of the English language, and it will be our job to add our own classification head during the fine-tuning process in order to help the model distinguish between toxic comments.

TFDistilBertModel generates an instance of TFBaseModelOutput whose last_hidden_state parameter is the output from the model last layer.
TFBaseModelOutput([(
    'last_hidden_state',
    <tf.Tensor: shape=(batch_size, sequence_lendgth, 768), dtype=float32, numpy=array([[[...]]], dtype=float32)>
)])

TFBaseModelOutput

Parameters

last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) – Sequence of hidden-states at the output of the last layer of the model.

Implementation
Python modules
import pandas as pd
import tensorflow as tf
from sklearn.model_selection import train_test_split
from transformers import (
    DistilBertTokenizerFast,
    TFDistilBertModel,
)

Configuration
TIMESTAMP = datetime.datetime.now().strftime("%Y%b%d%H%M").upper()

DATA_COLUMN = 'text'
LABEL_COLUMN = 'category_index'

MAX_SEQUENCE_LENGTH = 512   # Max length allowed for BERT is 512.
NUM_LABELS = len(raw_train[LABEL_COLUMN].unique())

MODEL_NAME = 'distilbert-base-uncased'
NUM_BASE_MODEL_OUTPUT = 768

# Flag to freeze base model
FREEZE_BASE = True

# Flag to add custom classification heads
USE_CUSTOM_HEAD = True
if USE_CUSTOM_HEAD == False:
    # Make the base trainable when no classification head exists.
    FREEZE_BASE = False

BATCH_SIZE = 16
LEARNING_RATE = 1e-2 if FREEZE_BASE else 5e-5
L2 = 0.01

Tokenizer
tokenizer = DistilBertTokenizerFast.from_pretrained(MODEL_NAME)
def tokenize(sentences, max_length=MAX_SEQUENCE_LENGTH, padding='max_length'):
    """Tokenize using the Huggingface tokenizer
    Args:
        sentences: String or list of string to tokenize
        padding: Padding method ['do_not_pad'|'longest'|'max_length']
    """
    return tokenizer(
        sentences,
        truncation=True,
        padding=padding,
        max_length=max_length,
        return_tensors="tf"
    )

Input layer
The base model expects input_ids and attention_mask whose shape is (max_sequence_length,). Generate Keras Tensors for them with Input layer respectively.
# Inputs for token indices and attention masks
input_ids = tf.keras.layers.Input(shape=(MAX_SEQUENCE_LENGTH,), dtype=tf.int32, name='input_ids')
attention_mask = tf.keras.layers.Input((MAX_SEQUENCE_LENGTH,), dtype=tf.int32, name='attention_mask')

Base model layer
Generate the output from the base model. The base model generates TFBaseModelOutput. Feed the embedding of [CLS] to the next layer.
base = TFDistilBertModel.from_pretrained(
    MODEL_NAME,
    num_labels=NUM_LABELS
)

# Freeze the base model weights.
if FREEZE_BASE:
    for layer in base.layers:
        layer.trainable = False
    base.summary()

# [CLS] embedding is last_hidden_state[:, 0, :]
output = base([input_ids, attention_mask]).last_hidden_state[:, 0, :]

Classification layers
if USE_CUSTOM_HEAD:
    # -------------------------------------------------------------------------------
    # Classifiation leayer 01
    # --------------------------------------------------------------------------------
    output = tf.keras.layers.Dropout(
        rate=0.15,
        name="01_dropout",
    )(output)

    output = tf.keras.layers.Dense(
        units=NUM_BASE_MODEL_OUTPUT,
        kernel_initializer='glorot_uniform',
        activation=None,
        name="01_dense_relu_no_regularizer",
    )(output)
    output = tf.keras.layers.BatchNormalization(
        name="01_bn"
    )(output)
    output = tf.keras.layers.Activation(
        "relu",
        name="01_relu"
    )(output)

    # --------------------------------------------------------------------------------
    # Classifiation leayer 02
    # --------------------------------------------------------------------------------
    output = tf.keras.layers.Dense(
        units=NUM_BASE_MODEL_OUTPUT,
        kernel_initializer='glorot_uniform',
        activation=None,
        name="02_dense_relu_no_regularizer",
    )(output)
    output = tf.keras.layers.BatchNormalization(
        name="02_bn"
    )(output)
    output = tf.keras.layers.Activation(
        "relu",
        name="02_relu"
    )(output)

Softmax Layer
output = tf.keras.layers.Dense(
    units=NUM_LABELS,
    kernel_initializer='glorot_uniform',
    kernel_regularizer=tf.keras.regularizers.l2(l2=L2),
    activation='softmax',
    name="softmax"
)(output)

Final Custom Model
name = f"{TIMESTAMP}_{MODEL_NAME.upper()}"
model = tf.keras.models.Model(inputs=[input_ids, attention_mask], outputs=output, name=name)
model.compile(
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
    optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE),
    metrics=['accuracy']
)
model.summary()
---
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_ids (InputLayer)          [(None, 256)]        0
__________________________________________________________________________________________________
attention_mask (InputLayer)     [(None, 256)]        0
__________________________________________________________________________________________________
tf_distil_bert_model (TFDistilB TFBaseModelOutput(la 66362880    input_ids[0][0]
                                                                 attention_mask[0][0]
__________________________________________________________________________________________________
tf.__operators__.getitem_1 (Sli (None, 768)          0           tf_distil_bert_model[1][0]
__________________________________________________________________________________________________
01_dropout (Dropout)            (None, 768)          0           tf.__operators__.getitem_1[0][0]
__________________________________________________________________________________________________
01_dense_relu_no_regularizer (D (None, 768)          590592      01_dropout[0][0]
__________________________________________________________________________________________________
01_bn (BatchNormalization)      (None, 768)          3072        01_dense_relu_no_regularizer[0][0
__________________________________________________________________________________________________
01_relu (Activation)            (None, 768)          0           01_bn[0][0]
__________________________________________________________________________________________________
02_dense_relu_no_regularizer (D (None, 768)          590592      01_relu[0][0]
__________________________________________________________________________________________________
02_bn (BatchNormalization)      (None, 768)          3072        02_dense_relu_no_regularizer[0][0
__________________________________________________________________________________________________
02_relu (Activation)            (None, 768)          0           02_bn[0][0]
__________________________________________________________________________________________________
softmax (Dense)                 (None, 2)            1538        02_relu[0][0]
==================================================================================================
Total params: 67,551,746
Trainable params: 1,185,794
Non-trainable params: 66,365,952   <--- Base BERT model is frozen

Data allocation
# --------------------------------------------------------------------------------
# Split data into training and validation
# --------------------------------------------------------------------------------
raw_train = pd.read_csv("./train.csv")
train_data, validation_data, train_label, validation_label = train_test_split(
    raw_train[DATA_COLUMN].tolist(),
    raw_train[LABEL_COLUMN].tolist(),
    test_size=.2,
    shuffle=True
)

# X = dict(tokenize(train_data))
# Y = tf.convert_to_tensor(train_label)
X = tf.data.Dataset.from_tensor_slices((
    dict(tokenize(train_data)),  # Convert BatchEncoding instance to dictionary
    train_label
)).batch(BATCH_SIZE).prefetch(1)

V = tf.data.Dataset.from_tensor_slices((
    dict(tokenize(validation_data)),  # Convert BatchEncoding instance to dictionary
    validation_label
)).batch(BATCH_SIZE).prefetch(1)

Train
# --------------------------------------------------------------------------------
# Train the model
# https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit
# Input data x can be a dict mapping input names to the corresponding array/tensors,
# if the model has named inputs. Beware of the "names". y should be consistent with x
# (you cannot have Numpy inputs and tensor targets, or inversely).
# --------------------------------------------------------------------------------
history = model.fit(
    x=X,    # dictionary
    # y=Y,
    y=None,
    epochs=NUM_EPOCHS,
    batch_size=BATCH_SIZE,
    validation_data=V,
)

To implement the 1st approach, change the configuration as below.
USE_CUSTOM_HEAD = False

Then FREEZE_BASE is changed to False and LEARNING_RATE is changed to 5e-5 which will run Further Pre-training on the base BERT model.
Saving the model
For the 3rd approach, saving the model will cause issues. The save_pretrained method of the Huggingface Model cannot be used as the model is not a direct sub class from of Huggingface PreTrainedModel.
Keras save_model causes an error with the default save_traces=True, or causes a different error with save_traces=True when loading the model with Keras load_model.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-71-01d66991d115> in <module>()
----> 1 tf.keras.models.load_model(MODEL_DIRECTORY)

11 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/saving/saved_model/load.py in _unable_to_call_layer_due_to_serialization_issue(layer, *unused_args, **unused_kwargs)
    865       'recorded when the object is called, and used when saving. To manually '
    866       'specify the input shape/dtype, decorate the call function with '
--> 867       '`@tf.function(input_signature=...)`.'.format(layer.name, type(layer)))
    868
    869

ValueError: Cannot call custom layer tf_distil_bert_model of type <class 'tensorflow.python.keras.saving.saved_model.load.TFDistilBertModel'>, because the call function was not serialized to the SavedModel.Please try one of the following methods to fix this issue:

(1) Implement `get_config` and `from_config` in the layer/model class, and pass the object to the `custom_objects` argument when loading the model. For more details, see: https://www.tensorflow.org/guide/keras/save_and_serialize

(2) Ensure that the subclassed model or layer overwrites `call` and not `__call__`. The input shape and dtype will be automatically recorded when the object is called, and used when saving. To manually specify the input shape/dtype, decorate the call function with `@tf.function(input_signature=...)`.

Only Keras Model save_weights worked as far as I tested.
Experiments
As far as I tested with Toxic Comment Classification Challenge, the 1st approach gave better recall (identify true toxic comment, true non-toxic comment). Code can be accessed as below. Please provide correction/suggestion if anything.

Code for 1st and 3rd approach

Related

BERT Document Classification Tutorial with Code - Fine tuning using TFDistilBertForSequenceClassification and Pytorch
Hugging Face Transformers: Fine-tuning DistilBERT for Binary Classification Tasks - Fine tuning using TFDistilBertModel


Instruction: Pairwise similarity matrix between a set of vectors in PyTorch

Input:
Let's suppose that we have a 3D PyTorch tensor, where the first dimension represents the batch_size, as follows:

import torch
import torch.nn as nn
x = torch.randn(32, 100, 25)

That is, for each i, x[i] is a set of 100 25-dimensional vectors. I would like to compute the similarity (e.g., the cosine similarity -- but in general any such pairwise distance/similarity matrix) of these vectors for each batch item.

That is, for each x[i] I need to compute a [100, 100] matrix which will contain the pairwise similarities of the above vectors. More specifically, the (i,j)-th element of this matrix should contain the similarity (or the distance) between the i-th and the j-th row of (the 100x25) x[t], for all t=1, ..., batch_size.

If I use torch.nn.CosineSimilarity(), no matter what dim I'm using, the result is either [100, 25] (dim=0), or [32, 25] (dim=1) , where I need a tensor of size [32, 100, 100]. I would expect torch.nn.CosineSimilarity() to work this way (since, at least to me, it looks more intuitive), but it doesn't.

Could that be done using something like below?

torch.matmul(x, x.permute(0, 2, 1))

I guess that this could give a distance matrix, but what if I need an arbitrary pairwise operation? Should I build this operation using the above?

Or maybe should I repeat x in a way so I can use the built-in torch.nn.CosineSimilarity()?

Thank you.

Output:
The documentation implies that the shapes of the inputs to cosine_similarity must be equal but this is not the case. Internally PyTorch broadcasts via torch.mul, inserting a dimension with a slice (or torch.unsqueeze) will give you the desired result. This is not optimal due to duplicate computations and memory for the upper and lower triangles but it's simple:
import torch
from torch.nn import functional as F
from scipy.spatial import distance

# compute once in pytorch
x = torch.randn(32, 100, 25)
y = F.cosine_similarity(x[..., None, :, :], x[..., :, None, :], dim=-1)

assert y.shape == torch.Size([32, 100, 100])

# test against scipy by iterating over each batch element
z = []
for i in range(x.shape[0]):
    slice = x[i, ...].numpy()
    z.append(torch.tensor(distance.cdist(slice, slice, metric='cosine'), dtype=torch.float32))

# convert similarity to distance and ensure they're reasonably close
assert torch.allclose(torch.stack(z), 1.0-y)


Instruction: pytorch model summary - forward func has more than one argument

Input:
I am using torch summary

from torchsummary import summary

I want to pass more than one argument when printing the model summary, but the examples mentioned here: Model summary in pytorch taken only one argument. for e.g.:

model = Network().to(device)
summary(model,(1,28,28))

The reason is that the forward function takes two arguments as input, e.g.:

def forward(self, img1, img2):

How do I pass two arguments here?

Output:
You can use the example given here: pytorch summary multiple inputs

summary(model, [(1, 16, 16), (1, 28, 28)])


Instruction: creating a common embedding for two languages

Input:
My task deals with multi-language like (english and hindi). For that I need a common embedding to represent both languages.

I know there are methods for learning multilingual embedding like 'MUSE', but this represents those two embeddings in a common vector space, obviously they are similar, but not the same.

So I wanted to know if there is any method or approach that can learn to represent both embedding in form of a single embedding that represents the both the language.

Any lead is strongly appreciated!!!

Output:
I think a good lead would be to look at past work that has been done in the field. A good overview to start with is Sebastian Ruder's talk, which gives you a multitude of approaches, depending on the level of information you have about your source/target language. This is basically what MUSE is doing, and I'm relatively sure that it is considered state-of-the-art.

The basic idea in most approaches is to map embedding spaces such that you minimize some (usually Euclidean) distance between the both (see p. 16 of the link). This obviously works best if you have a known dictionary and can precisely map the different translations, and works even better if the two languages have similar linguistic properties (not so sure about Hindi and English, to be honest).

Another recent approach is the one by Multilingual-BERT (mBERT), or similarly, XLM-RoBERTa, but those learn embeddings based on a shared vocabulary. This might again be less desirable if you have morphologically dissimilar languages, and also has the drawback that they incorporate a bunch of other, unrelated, languages.

Otherwise, I'm unclear on what exactly you are expecting from a "common embedding", but happy to extend the answer once clarified.


Instruction: How to normalize images in PyTorch

Input:
transform = transforms.Compose([
    transforms.ToTensor()
])

trainset = torchvision.datasets.ImageFolder(root='C:/Users/beomseokpark/Desktop/CNN/train_data', transform = transform)
data_loader = DataLoader(dataset = trainset, batch_size = 8, shuffle = True, num_workers=2)

with torch.no_grad():
    for num, data in enumerate(trainset):
        imgs, label = data

I loaded images with ImageFolder in torchvision library, and how can I get mean and std from each channel of my images?

Can anyone please help me out?

Output:
There's the "lazy man" approach: You can simply plug a nn.BatchNorm2d as the very first layer of your network. With the appropriate momentum, and track_running_stats=True this layer will estimate your data's mean and variance for you.

Alternatively, you can compute the mean and variance using

mu = torch.zeros((3,), dtype=torch.float)
sig = torch.zeros((3,), dtype=torch.float)
n = 0
with torch.no_grad():
    for num, data in enumerate(trainset):
        imgs, _ = data
        mu += torch.sum(imgs, dim=(0, 2, 3))
        sig += torch.sum(imgs**2, dim=(0, 2, 3))
        n += imgs.numel() // imgs.shape[0]
n  = float(n)
mu = mu / n  # mean
sig = sig / n - (mu ** 2)


Instruction: PyTorch FasterRCNN TypeError: forward() takes 2 positional arguments but 3 were given

Input:
I am working on object detection and I have a dataset containing images and their corresponding bounding boxes (ground-truth values).

I actually have built my own feature extractor which takes an image as input and outputs a feature map(basically an encoder-decoder system where the final output of the decoder is the same as the image size and has 3 channels). Now, I want to feed this feature map as an input to a FasterRCNN model for detection instead of the original image. I am using the following code to add the feature map(using RTFNet to generate feature map -  code at this link) on top the FRCNN detection module

frcnn_model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = frcnn_model.roi_heads.box_predictor.cls_score.in_features
frcnn_model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
fpn_block = frcnn_model.backbone.fpn
rpn_block = frcnn_model.rpn

backbone = RTFNet(num_classes) RTFNet is a feature extractor taking as input, an image having 4 channels(fused RGB and thermal image) ,
model = nn.Sequential(backbone, nn.ReLU(inplace=True))
model = nn.Sequential(model,fpn_block)
model = nn.Sequential(model,rpn_block)
model = nn.Sequential(model,FastRCNNPredictor(in_features, num_classes))

I am just trying to test and see if it is working by using the following code which generates random images and bounding boxes

images, boxes = torch.rand(1, 4, 512, 640), torch.rand(4, 11, 4)
labels = torch.randint(1, num_classes, (4, 11))
images = list(image for image in images)
targets = []
for i in range(len(images)):
  d = {}
  d['boxes'] = boxes[i]
  d['labels'] = labels[i]
  targets.append(d)
output = model(images, targets)

Running this gives me the following error

TypeError                                 Traceback (most recent call last)
<ipython-input-22-2637b8c27ad2> in <module>()
----> 1 output = model(images, targets)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

TypeError: forward() takes 2 positional arguments but 3 were given

However, when I replace my model with a normal FasterRCNN Model with the following,

model = fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

there is no error and it works fine

Can anyone let me know where I am going wrong? Thanks in advance

Output:
This is because only the image inputs should be passed into the models, instead of both images and the ground truth targets. So instead of doing output = model(images, targets), you can do output = model(images).

As for why the error message talks about being given 3 positional arguments, this is because forward is initiated with a default self keyword, which represents the class instance. So in addition to self, you should only give 1 more argument, which would be the input image.


Instruction: Why torch.FloatTensor([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]]) size is [2,2,3]?

Input:
>>> ft = torch.FloatTensor([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]])
>>> print(ft.shape)
torch.Size([2, 2, 3])

I can't understand this result.
I think the torch size should be [2,3,2], but the result is [2,2,3].

Output:
Because

len([[[0,1,2],[3,4,5]],[[6,7,8],[9,10,11]]]) = 2

This is the first 2.
and each item inside:

len([[0,1,2],[3,4,5]]) = 2

This is the second 2.

and each item inside:

len([0,1,2]) = 3

This is the 3.


Instruction: Pytorch - select region of a tensor using torch function

Input:
I am looking for a way to select a region of a PyTorch tensor using a torch function (without using numpy). Do you have suggestions on how to proceed?

In other words, I'm looking for a way to crop a region of a matrix. Using numpy, it would be something like

import numpy as np
A = np.random.rand(16,16)
B = A[0:8, 0:8]

The approach I am trying is the following:

from torchvision import transforms
A = torch.randn([1,3,64,64])
B = torch.split(A, [16,32,16], dim =2)
C = torch.split(B, [16,32,16], dim =3)

Which gives the error

  'tuple' object has no attribute 'split'

Output:
What's wrong with regular slicing?

import torch

A = torch.randn([1,3,64,64])
B = A[..., 16:32, 16:32]


Instruction: What is the difference between sample() and rsample()?

Input:
When I sample from a distribution in PyTorch, both sample and rsample appear to give similar results:
import torch, seaborn as sns

x = torch.distributions.Normal(torch.tensor([0.0]), torch.tensor([1.0]))

sns.distplot(x.sample((100000,)))
sns.distplot(x.rsample((100000,)))

When should I use sample(), and when should I use rsample()?

Output:
Using rsample allows for pathwise derivatives:

  The other way to implement these stochastic/policy gradients would be to use the reparameterization trick from the rsample() method, where the parameterized random variable can be constructed via a parameterized deterministic function of a parameter-free random variable. The reparameterized sample therefore becomes differentiable.


Instruction: getting the classification labels for torchvision's pretrained networks

Input:
Pytorch's torchvision package provides pre-trained neural networks for image classification. I've been using the following code to classify an image using Alexnet (note: some of this code is from this webpage):

from PIL import Image
import torch
from torchvision import transforms
from torchvision import models

# function to transform image
transform = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225])])

# image
img = Image.open('/path/to/image.jpg')
img = transform(img)
img = torch.unsqueeze(img, 0)

# alexnet
alexnet = models.alexnet(pretrained=True)
alexnet.eval()
out = alexnet(img)
percents = torch.nn.functional.softmax(out, dim=1)[0] * 100
top5_vals, top5_inds = percents.topk(5)

There are 1,000 total classes, and the top5_inds variable gives me the indices of the top 5 classes. But how do I get the associated labels (e.g. snail, basketball, banana)? I can't seem to find any sort of list as part of Pytorch's documentation or the alexnet variable.

Output: Torchvision models are pretrained on the ImageNet dataset. Due to its comprehensiveness and size, ImageNet is the most commonly used dataset for pretraining & transfer learning. As you noted, it has 1000 classes. The complete class list can be searched, or you can refer to this listing on GitHub: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a


Instruction: Tensor output from final layer is of the wrong shape in PyTorch

Input:
I am building a sequence-to-label classifier, where the input data are text sequences and output labels are binary. The model is very simple, with GRU hidden layers and a Word Embeddings input layer. I want a [n, 60] input to output a [n, 1] label, but the Torch model returns a [n, 60] output.

My model, with minimal layers:

class Model(nn.Module):
    def __init__(self, weights_matrix, hidden_size, num_layers):
        super(Model, self).__init__()
        self.embedding, num_embeddings, embedding_dim = create_emb_layer(weights_matrix, True)
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.gru = nn.GRU(embedding_dim, hidden_size, num_layers, batch_first=True)
        self.out = nn.Linear(hidden_size, 1)
    def forward(self, inp, hidden):
        emb = self.embedding(inp);
        out, hidden = self.gru(emb, hidden)
        out = self.out(out);
        return out, hidden;

    def init_hidden(self, batch_size):
        return torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device);

Model Layers:

Model(
  (embedding): Embedding(184901, 100)
  (gru): GRU(100, 60, num_layers=3, batch_first=True)
  (out): Linear(in_features=60, out_features=1, bias=True)
)

Input shapes of my data are: X : torch.Size([64, 60]), and Y : torch.Size([64, 1]), for a single batch of size 64.

When I run the X tensor through the model, it should output a single label, however, the output from the classifier is torch.Size([64, 60, 1]). To run the model, I do the following:

for epoch in range(1):
    running_loss = 0.0;

    batch_size = 64;
    hidden = model.init_hidden(batch_size)
    for ite, data in enumerate(train_loader, 0):
        x, y = data[:,:-1], data[:,-1].reshape(-1,1)

        optimizer.zero_grad();

        outputs, hidden = model(x, hidden);

        hidden = Variable(hidden.data).to(device);
        loss = criterion(outputs, y);

        loss.backward();
        optimizer.step();

        running_loss = running_loss + loss.item();
        if ite % 2000 == 1999:
            print('[%d %5d] loss: %.3f'%(epoch+1, ite+1, running_loss / 2000))
        running_loss = 0.0;

When I print the shape of outputs, it is 64x60x1 rather than 64x1. What I also don't get is how the criterion function is able to calculate the loss when the shapes of outputs and labels are inconsistent. With Tensorflow, this would always throw an error, but it doesn't with Torch.

Output:
The output from your model is of shape torch.Size([64, 60, 1]) i.e. 64 is the batch size, and (60, 1) corresponds [n, 1] as expected.

Assuming you're using nn.CrossEntropy(input, target), it expected the input to be (N,C) and target to be (N), where C is number of classes.

Your output is consistent, and hence loss is evaluated.

For example,

outputs = torch.randn(3, 2, 1)
target = torch.empty(3, 1, dtype=torch.long).random_(2)

criterion = nn.CrossEntropyLoss(reduction='mean')
print(outputs)
print(target)
loss = criterion(outputs, target)
print(loss)

# outputs
tensor([[[ 0.5187],
         [ 1.0320]],

        [[ 0.2169],
         [ 2.4480]],

        [[-0.4895],
         [-0.6096]]])
tensor([[0],
        [1],
        [0]])
tensor(0.5731)

Read more here.


Instruction: Pytorch dataset and shared memory?

Input: I would want to cache data in a torch.utils.data.Dataset. The simple solution is to just persist certain tensors in a member of the dataset. However, since the torch.utils.data.DataLoader class spawns multiple processes, the cache would only be local to each instance and would cause me to possibly cache multiple copies of the same tensors. Is there a way to use Python's multiprocessing library to share data between the different loader processes?

Output:
The answer depends on your OS and settings. If you are using Linux with the default process start method, you don't have to worry about duplicates or process communication, because worker processes share memory! This is efficiently implemented as Inter Process Communication (IPC) through shared memory (some more details here).
For Windows, things are more complicated. From the documentation:

Since workers rely on Python multiprocessing, worker launch behavior
is different on Windows compared to Unix.
On Unix, fork() is the default multiprocessing start method. Using
fork(), child workers typically can access the dataset and Python
argument functions directly through the cloned address space.
On Windows, spawn() is the default multiprocessing start method. Using
spawn(), another interpreter is launched which runs your main script,
followed by the internal worker function that receives the dataset,
collate_fn and other arguments through pickle serialization.

This means that your dynamically cached Dataset members would be automatically shared between all processes on Linux. That's great! However, on Windows, processes will not have received copies of them (they only received the Dataset upon spawning), so you should use a process communication scheme, e.g. through multiprocessing Pipe, Queue or Manager (preferred for broadcasting to multiple processes, but you would have to convert tensors to lists). This is not very efficient, besides rather bothersome to implement.
Nevertheless, there is another method: memory mapping (memmaping). This means that your objects will be written to virtual memory, and again all processes will have access to it, while a respective "shadow copy" of these objects will at some point be flushed and exist on your hard drive (can be placed in a /tmp directory). You can use memmaping with the mmap module, in which case your objects will have to be serialized as a binary file, or you can use numpy.memmap. You can find more details here.


Instruction: torch.nn.functional.conv2d for several channels/batches

Input:
I have an image with I want to pad (to maintain the same shape) and then perform a convolution with a given kernel. It works ok if I have only one channel and one image in the batch. But how to properly rewrite it for several batches & channels? I suppose, for batches I can just duplicate the kernel along dimension 0. But what about channels? What is the proper way to do it in torch? See the toy example below.

import torch.nn.functional as f

x = torch.zeros((1,1,16,16))
x[...,6:10,6:10] = 1.
ker = torch.ones(3,3)
ker[1,1] = -4
padding = (ker.shape[1] // 2, ker.shape[1] // 2,
           ker.shape[0] // 2, ker.shape[0] // 2)
x = f.pad(x, padding, mode='replicate')
kernel = ker.reshape((1, 1, ker.shape[0], -1))
result = f.conv2d(x, ker, groups=1)

How to rewrite this piece of code to deal with multiple channels and batches? I read documentation but to be honest it did not seem very detailed to me.

Output:
The documentation at https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.conv2d seems to answer your question:

 input – input tensor of shape (minibatch,in_channels,iH,iW)
 weight – filters of shape (out_channels,in_channels/groups,kH,kW)

so your x must be size (batch_size, in_channels, 16, 16)

and your kernel (batch_size, out_channels/groups, kH, kW)


Instruction: How do I crop a Landsat image into smaller chunks for training and then predict on the original image

Input:
I am looking at using Landsat imagery to train a CNN for unsupervised pixel-wise semantic segmentation classification.  That said, I have been unable to find a method that allows me to crop images from the larger Landsat image for training and then predict on the original image. Essentially here is what I am trying to do:

Original Landsat image (5,000 x 5,000 - this is an arbitrary size, not exactly sure of the actual dimensions off-hand) -> crop the image into (100 x 100) chunks -> train the model on these cropped images -> output a prediction for each pixel in the original (uncropped) image.

That said, I am not sure if I should predict on the cropped images and stitch them together after they are predicted or if I can predict on the original image.

Any clarification/code examples would be greatly appreciated.  For reference, I use both pytorch and tensorflow.

Thank you!
Lance D

Output:
Borrowing from Ronneberger et al., what we have been doing is to split the input Landsat scene and corresponding  ground truth mask into overlapping tiles.  Take the original image and pad it by the overlap margin (we use reflection for the padding) then split into tiles. Here is a code snippet using scikit-image:

import skimage as sk
patches = sk.util.view_as_windows(image,
  (self.tile_height+2*self.image_margin,
  self.tile_width+2*self.image_margin,raster_value['channels']),
  (self.tile_height,self.tile_width,raster_value['channels'])

I don't know what you are using for a loss function for unsupervised segmentation. In our case with supervised learning, we crop the final segmentation prediction to match the ground truth output shape. In the Ronneberger paper they relied on shrinkage due to the use of valid padding.
For predictions you would do the same (split into overlapping tiles) and stitch the result.


Instruction: padding and attention mask does not work as intended in batch input in GPT language model

Input:
The following code is without batch:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
context=torch.tensor([tokenizer.encode("This is")])
output, past = model(context)
token = torch.argmax(output[..., -1, :])
print(tokenizer.decode(token.item()))

output: ' a'

This is working fine. Now, I extended this to batch setting:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is ")),torch.tensor(tokenizer.encode("Hello How are "))]
context=pad_sequence(context,batch_first=True)

mask=torch.tensor([[1,1,0],[1,1,1]])
output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: '\n you'

Here \n is next token for the first context and you is next token for second context of the batch.
But The expected next token for the first context is a, since all the settings are same. Furthermore, if you reduce the second context to 2 token you will get a in this batch setting. So clearly, model can not understand the padding.
Also, the attention mask does not work. Because,
after padding the next token of sequence this is is 0 (zero). And according to the attention mask ([1,1,0]), this zero should be avoided and only the tokens this and is should be attended. The proofs that this attention masking is not working are:

Use attention mask [1,1,1], that means attend even on the padding zero, you get the same output
which is \n.
Use the the string this is!. Here ! has the zero index in the vocabulary matrix. Again you get the same output which is \n.

Only time, it is possible to get desirable output is without the batch settings and attention mask ( now it seems, it does not matter because it has no effect anyway)

Then I found this, which suggested to use pad_token. So I used like following:

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
from torch.nn.utils.rnn import pad_sequence

tokenizer = GPT2Tokenizer.from_pretrained("gpt2",pad_token="<PAD>")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is <PAD> ")),torch.tensor(tokenizer.encode("Hello How are"))]
context=torch.stack(context)
print(context)
mask=torch.tensor([[1,1,0],[1,1,1]])

output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: 'The you'

Here The is next token for the first context and you is next token for second context of the batch. This is also not working. Because The is not expected for the first context.

How do I use variable length sequence in batch setting in gpt/gpt2 model?

Output: I'm not sure if this helps, but you don't need to implement you own attention masking and padding. The Transformers library provides the encode_plus() and batch_encode_plus() functions that will perform tokenization, generate the attention masks, and do padding for you. The results come out as Python dictionaries.


Instruction: How is the number of channels adjusted in efficientnet

Input:
I was reading the code at efficientnet and was shocked by its clever ideas. But I don't quite understand how it adjusts the number of channels.

def round_filters(filters, width_coefficient, depth_divisor):

  filters *= width_coefficient
  new_filters = int(filters + depth_divisor / 2) // depth_divisor * depth_divisor
  new_filters = max(depth_divisor, new_filters)
  # Make sure that round down does not go down by more than 10%.
  if new_filters < 0.9 * filters:
     new_filters += depth_divisor
  return int(new_filters)

I know the number of channels has to be adjusted by the width factor, but why do I do the following? What is depth_divisor?

Output:
It's making the scaled width divisible by depth_divisor. You can view as rounding the scaled width to the nearest multiple of depth_divisor with some additional consideration (round up when going down more than 10%). In almost all applications of this function in various MobileNets and EfficientNets the depth_divisor is 8.

Why 8? Mostly due to the constraints of common hardware accelerators like GPU and TPU. See a recent cuDNN developer guide and count the number of 'multiple of 8', 'multiple of 32', etc guidelines for optimal performance of certain operations:

https://docs.nvidia.com/deeplearning/sdk/cudnn-developer-guide/index.html

  For algorithms other than *_ALGO_WINOGRAD_NONFUSED, when the following
  requirements are met, the cuDNN library will trigger the Tensor Core
  operations:

  Input, filter, and output descriptors (xDesc, yDesc,
  wDesc, dxDesc, dyDesc and dwDesc as applicable) are of the dataType =
  CUDNN_DATA_HALF (i.e., FP16). For FP32 dataType see FP32-to-FP16
  Conversion.
  The number of input and output feature maps (i.e., channel dimension
  C) is a multiple of 8. When the channel dimension is not a multiple of
  8, see Padding.
  The filter is of type CUDNN_TENSOR_NCHW or CUDNN_TENSOR_NHWC.
  If using a filter of type CUDNN_TENSOR_NHWC, then the input, filter,
  and output data pointers (X, Y, W, dX, dY, and dW as applicable) are
  aligned to 128-bit boundaries.