Instruction: huggingface sequence classification unfreezing layers

Input:
I am using longformer for sequence classification - binary problem
I have downloaded required files
# load model and tokenizer and define length of the text sequence
model = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096',
                                                           gradient_checkpointing=False,
                                                           attention_window = 512)
tokenizer = LongformerTokenizerFast.from_pretrained('allenai/longformer-base-4096', max_length = 1024)

then as shown here I ran the below code
for name, param in model.named_parameters():
     print(name, param.requires_grad)

longformer.embeddings.word_embeddings.weight True
longformer.embeddings.position_embeddings.weight True
longformer.embeddings.token_type_embeddings.weight True
longformer.embeddings.LayerNorm.weight True
longformer.embeddings.LayerNorm.bias True
longformer.encoder.layer.0.attention.self.query.weight True
longformer.encoder.layer.0.attention.self.query.bias True
longformer.encoder.layer.0.attention.self.key.weight True
longformer.encoder.layer.0.attention.self.key.bias True
longformer.encoder.layer.0.attention.self.value.weight True
longformer.encoder.layer.0.attention.self.value.bias True
longformer.encoder.layer.0.attention.self.query_global.weight True
longformer.encoder.layer.0.attention.self.query_global.bias True
longformer.encoder.layer.0.attention.self.key_global.weight True
longformer.encoder.layer.0.attention.self.key_global.bias True
longformer.encoder.layer.0.attention.self.value_global.weight True
longformer.encoder.layer.0.attention.self.value_global.bias True
longformer.encoder.layer.0.attention.output.dense.weight True
longformer.encoder.layer.0.attention.output.dense.bias True
longformer.encoder.layer.0.attention.output.LayerNorm.weight True
longformer.encoder.layer.0.attention.output.LayerNorm.bias True
longformer.encoder.layer.0.intermediate.dense.weight True
longformer.encoder.layer.0.intermediate.dense.bias True
longformer.encoder.layer.0.output.dense.weight True
longformer.encoder.layer.0.output.dense.bias True
longformer.encoder.layer.0.output.LayerNorm.weight True
longformer.encoder.layer.0.output.LayerNorm.bias True
longformer.encoder.layer.1.attention.self.query.weight True
longformer.encoder.layer.1.attention.self.query.bias True
longformer.encoder.layer.1.attention.self.key.weight True
longformer.encoder.layer.1.attention.self.key.bias True
longformer.encoder.layer.1.attention.self.value.weight True
longformer.encoder.layer.1.attention.self.value.bias True
longformer.encoder.layer.1.attention.self.query_global.weight True
longformer.encoder.layer.1.attention.self.query_global.bias True
longformer.encoder.layer.1.attention.self.key_global.weight True
longformer.encoder.layer.1.attention.self.key_global.bias True
longformer.encoder.layer.1.attention.self.value_global.weight True
longformer.encoder.layer.1.attention.self.value_global.bias True
longformer.encoder.layer.1.attention.output.dense.weight True
longformer.encoder.layer.1.attention.output.dense.bias True
longformer.encoder.layer.1.attention.output.LayerNorm.weight True
longformer.encoder.layer.1.attention.output.LayerNorm.bias True
longformer.encoder.layer.1.intermediate.dense.weight True
longformer.encoder.layer.1.intermediate.dense.bias True
longformer.encoder.layer.1.output.dense.weight True
longformer.encoder.layer.1.output.dense.bias True
longformer.encoder.layer.1.output.LayerNorm.weight True
longformer.encoder.layer.1.output.LayerNorm.bias True
longformer.encoder.layer.2.attention.self.query.weight True
longformer.encoder.layer.2.attention.self.query.bias True
longformer.encoder.layer.2.attention.self.key.weight True
longformer.encoder.layer.2.attention.self.key.bias True
longformer.encoder.layer.2.attention.self.value.weight True
longformer.encoder.layer.2.attention.self.value.bias True
longformer.encoder.layer.2.attention.self.query_global.weight True
longformer.encoder.layer.2.attention.self.query_global.bias True
longformer.encoder.layer.2.attention.self.key_global.weight True
longformer.encoder.layer.2.attention.self.key_global.bias True
longformer.encoder.layer.2.attention.self.value_global.weight True
longformer.encoder.layer.2.attention.self.value_global.bias True
longformer.encoder.layer.2.attention.output.dense.weight True
longformer.encoder.layer.2.attention.output.dense.bias True
longformer.encoder.layer.2.attention.output.LayerNorm.weight True
longformer.encoder.layer.2.attention.output.LayerNorm.bias True
longformer.encoder.layer.2.intermediate.dense.weight True
longformer.encoder.layer.2.intermediate.dense.bias True
longformer.encoder.layer.2.output.dense.weight True
longformer.encoder.layer.2.output.dense.bias True
longformer.encoder.layer.2.output.LayerNorm.weight True
longformer.encoder.layer.2.output.LayerNorm.bias True
longformer.encoder.layer.3.attention.self.query.weight True
longformer.encoder.layer.3.attention.self.query.bias True
longformer.encoder.layer.3.attention.self.key.weight True
longformer.encoder.layer.3.attention.self.key.bias True
longformer.encoder.layer.3.attention.self.value.weight True
longformer.encoder.layer.3.attention.self.value.bias True
longformer.encoder.layer.3.attention.self.query_global.weight True
longformer.encoder.layer.3.attention.self.query_global.bias True
longformer.encoder.layer.3.attention.self.key_global.weight True
longformer.encoder.layer.3.attention.self.key_global.bias True
longformer.encoder.layer.3.attention.self.value_global.weight True
longformer.encoder.layer.3.attention.self.value_global.bias True
longformer.encoder.layer.3.attention.output.dense.weight True
longformer.encoder.layer.3.attention.output.dense.bias True
longformer.encoder.layer.3.attention.output.LayerNorm.weight True
longformer.encoder.layer.3.attention.output.LayerNorm.bias True
longformer.encoder.layer.3.intermediate.dense.weight True
longformer.encoder.layer.3.intermediate.dense.bias True
longformer.encoder.layer.3.output.dense.weight True
longformer.encoder.layer.3.output.dense.bias True
longformer.encoder.layer.3.output.LayerNorm.weight True
longformer.encoder.layer.3.output.LayerNorm.bias True
longformer.encoder.layer.4.attention.self.query.weight True
longformer.encoder.layer.4.attention.self.query.bias True
longformer.encoder.layer.4.attention.self.key.weight True
longformer.encoder.layer.4.attention.self.key.bias True
longformer.encoder.layer.4.attention.self.value.weight True
longformer.encoder.layer.4.attention.self.value.bias True
longformer.encoder.layer.4.attention.self.query_global.weight True
longformer.encoder.layer.4.attention.self.query_global.bias True
longformer.encoder.layer.4.attention.self.key_global.weight True
longformer.encoder.layer.4.attention.self.key_global.bias True
longformer.encoder.layer.4.attention.self.value_global.weight True
longformer.encoder.layer.4.attention.self.value_global.bias True
longformer.encoder.layer.4.attention.output.dense.weight True
longformer.encoder.layer.4.attention.output.dense.bias True
longformer.encoder.layer.4.attention.output.LayerNorm.weight True
longformer.encoder.layer.4.attention.output.LayerNorm.bias True
longformer.encoder.layer.4.intermediate.dense.weight True
longformer.encoder.layer.4.intermediate.dense.bias True
longformer.encoder.layer.4.output.dense.weight True
longformer.encoder.layer.4.output.dense.bias True
longformer.encoder.layer.4.output.LayerNorm.weight True
longformer.encoder.layer.4.output.LayerNorm.bias True
longformer.encoder.layer.5.attention.self.query.weight True
longformer.encoder.layer.5.attention.self.query.bias True
longformer.encoder.layer.5.attention.self.key.weight True
longformer.encoder.layer.5.attention.self.key.bias True
longformer.encoder.layer.5.attention.self.value.weight True
longformer.encoder.layer.5.attention.self.value.bias True
longformer.encoder.layer.5.attention.self.query_global.weight True
longformer.encoder.layer.5.attention.self.query_global.bias True
longformer.encoder.layer.5.attention.self.key_global.weight True
longformer.encoder.layer.5.attention.self.key_global.bias True
longformer.encoder.layer.5.attention.self.value_global.weight True
longformer.encoder.layer.5.attention.self.value_global.bias True
longformer.encoder.layer.5.attention.output.dense.weight True
longformer.encoder.layer.5.attention.output.dense.bias True
longformer.encoder.layer.5.attention.output.LayerNorm.weight True
longformer.encoder.layer.5.attention.output.LayerNorm.bias True
longformer.encoder.layer.5.intermediate.dense.weight True
longformer.encoder.layer.5.intermediate.dense.bias True
longformer.encoder.layer.5.output.dense.weight True
longformer.encoder.layer.5.output.dense.bias True
longformer.encoder.layer.5.output.LayerNorm.weight True
longformer.encoder.layer.5.output.LayerNorm.bias True
longformer.encoder.layer.6.attention.self.query.weight True
longformer.encoder.layer.6.attention.self.query.bias True
longformer.encoder.layer.6.attention.self.key.weight True
longformer.encoder.layer.6.attention.self.key.bias True
longformer.encoder.layer.6.attention.self.value.weight True
longformer.encoder.layer.6.attention.self.value.bias True
longformer.encoder.layer.6.attention.self.query_global.weight True
longformer.encoder.layer.6.attention.self.query_global.bias True
longformer.encoder.layer.6.attention.self.key_global.weight True
longformer.encoder.layer.6.attention.self.key_global.bias True
longformer.encoder.layer.6.attention.self.value_global.weight True
longformer.encoder.layer.6.attention.self.value_global.bias True
longformer.encoder.layer.6.attention.output.dense.weight True
longformer.encoder.layer.6.attention.output.dense.bias True
longformer.encoder.layer.6.attention.output.LayerNorm.weight True
longformer.encoder.layer.6.attention.output.LayerNorm.bias True
longformer.encoder.layer.6.intermediate.dense.weight True
longformer.encoder.layer.6.intermediate.dense.bias True
longformer.encoder.layer.6.output.dense.weight True
longformer.encoder.layer.6.output.dense.bias True
longformer.encoder.layer.6.output.LayerNorm.weight True
longformer.encoder.layer.6.output.LayerNorm.bias True
longformer.encoder.layer.7.attention.self.query.weight True
longformer.encoder.layer.7.attention.self.query.bias True
longformer.encoder.layer.7.attention.self.key.weight True
longformer.encoder.layer.7.attention.self.key.bias True
longformer.encoder.layer.7.attention.self.value.weight True
longformer.encoder.layer.7.attention.self.value.bias True
longformer.encoder.layer.7.attention.self.query_global.weight True
longformer.encoder.layer.7.attention.self.query_global.bias True
longformer.encoder.layer.7.attention.self.key_global.weight True
longformer.encoder.layer.7.attention.self.key_global.bias True
longformer.encoder.layer.7.attention.self.value_global.weight True
longformer.encoder.layer.7.attention.self.value_global.bias True
longformer.encoder.layer.7.attention.output.dense.weight True
longformer.encoder.layer.7.attention.output.dense.bias True
longformer.encoder.layer.7.attention.output.LayerNorm.weight True
longformer.encoder.layer.7.attention.output.LayerNorm.bias True
longformer.encoder.layer.7.intermediate.dense.weight True
longformer.encoder.layer.7.intermediate.dense.bias True
longformer.encoder.layer.7.output.dense.weight True
longformer.encoder.layer.7.output.dense.bias True
longformer.encoder.layer.7.output.LayerNorm.weight True
longformer.encoder.layer.7.output.LayerNorm.bias True
longformer.encoder.layer.8.attention.self.query.weight True
longformer.encoder.layer.8.attention.self.query.bias True
longformer.encoder.layer.8.attention.self.key.weight True
longformer.encoder.layer.8.attention.self.key.bias True
longformer.encoder.layer.8.attention.self.value.weight True
longformer.encoder.layer.8.attention.self.value.bias True
longformer.encoder.layer.8.attention.self.query_global.weight True
longformer.encoder.layer.8.attention.self.query_global.bias True
longformer.encoder.layer.8.attention.self.key_global.weight True
longformer.encoder.layer.8.attention.self.key_global.bias True
longformer.encoder.layer.8.attention.self.value_global.weight True
longformer.encoder.layer.8.attention.self.value_global.bias True
longformer.encoder.layer.8.attention.output.dense.weight True
longformer.encoder.layer.8.attention.output.dense.bias True
longformer.encoder.layer.8.attention.output.LayerNorm.weight True
longformer.encoder.layer.8.attention.output.LayerNorm.bias True
longformer.encoder.layer.8.intermediate.dense.weight True
longformer.encoder.layer.8.intermediate.dense.bias True
longformer.encoder.layer.8.output.dense.weight True
longformer.encoder.layer.8.output.dense.bias True
longformer.encoder.layer.8.output.LayerNorm.weight True
longformer.encoder.layer.8.output.LayerNorm.bias True
longformer.encoder.layer.9.attention.self.query.weight True
longformer.encoder.layer.9.attention.self.query.bias True
longformer.encoder.layer.9.attention.self.key.weight True
longformer.encoder.layer.9.attention.self.key.bias True
longformer.encoder.layer.9.attention.self.value.weight True
longformer.encoder.layer.9.attention.self.value.bias True
longformer.encoder.layer.9.attention.self.query_global.weight True
longformer.encoder.layer.9.attention.self.query_global.bias True
longformer.encoder.layer.9.attention.self.key_global.weight True
longformer.encoder.layer.9.attention.self.key_global.bias True
longformer.encoder.layer.9.attention.self.value_global.weight True
longformer.encoder.layer.9.attention.self.value_global.bias True
longformer.encoder.layer.9.attention.output.dense.weight True
longformer.encoder.layer.9.attention.output.dense.bias True
longformer.encoder.layer.9.attention.output.LayerNorm.weight True
longformer.encoder.layer.9.attention.output.LayerNorm.bias True
longformer.encoder.layer.9.intermediate.dense.weight True
longformer.encoder.layer.9.intermediate.dense.bias True
longformer.encoder.layer.9.output.dense.weight True
longformer.encoder.layer.9.output.dense.bias True
longformer.encoder.layer.9.output.LayerNorm.weight True
longformer.encoder.layer.9.output.LayerNorm.bias True
longformer.encoder.layer.10.attention.self.query.weight True
longformer.encoder.layer.10.attention.self.query.bias True
longformer.encoder.layer.10.attention.self.key.weight True
longformer.encoder.layer.10.attention.self.key.bias True
longformer.encoder.layer.10.attention.self.value.weight True
longformer.encoder.layer.10.attention.self.value.bias True
longformer.encoder.layer.10.attention.self.query_global.weight True
longformer.encoder.layer.10.attention.self.query_global.bias True
longformer.encoder.layer.10.attention.self.key_global.weight True
longformer.encoder.layer.10.attention.self.key_global.bias True
longformer.encoder.layer.10.attention.self.value_global.weight True
longformer.encoder.layer.10.attention.self.value_global.bias True
longformer.encoder.layer.10.attention.output.dense.weight True
longformer.encoder.layer.10.attention.output.dense.bias True
longformer.encoder.layer.10.attention.output.LayerNorm.weight True
longformer.encoder.layer.10.attention.output.LayerNorm.bias True
longformer.encoder.layer.10.intermediate.dense.weight True
longformer.encoder.layer.10.intermediate.dense.bias True
longformer.encoder.layer.10.output.dense.weight True
longformer.encoder.layer.10.output.dense.bias True
longformer.encoder.layer.10.output.LayerNorm.weight True
longformer.encoder.layer.10.output.LayerNorm.bias True
longformer.encoder.layer.11.attention.self.query.weight True
longformer.encoder.layer.11.attention.self.query.bias True
longformer.encoder.layer.11.attention.self.key.weight True
longformer.encoder.layer.11.attention.self.key.bias True
longformer.encoder.layer.11.attention.self.value.weight True
longformer.encoder.layer.11.attention.self.value.bias True
longformer.encoder.layer.11.attention.self.query_global.weight True
longformer.encoder.layer.11.attention.self.query_global.bias True
longformer.encoder.layer.11.attention.self.key_global.weight True
longformer.encoder.layer.11.attention.self.key_global.bias True
longformer.encoder.layer.11.attention.self.value_global.weight True
longformer.encoder.layer.11.attention.self.value_global.bias True
longformer.encoder.layer.11.attention.output.dense.weight True
longformer.encoder.layer.11.attention.output.dense.bias True
longformer.encoder.layer.11.attention.output.LayerNorm.weight True
longformer.encoder.layer.11.attention.output.LayerNorm.bias True
longformer.encoder.layer.11.intermediate.dense.weight True
longformer.encoder.layer.11.intermediate.dense.bias True
longformer.encoder.layer.11.output.dense.weight True
longformer.encoder.layer.11.output.dense.bias True
longformer.encoder.layer.11.output.LayerNorm.weight True
longformer.encoder.layer.11.output.LayerNorm.bias True
classifier.dense.weight True
classifier.dense.bias True
classifier.out_proj.weight True
classifier.out_proj.bias True

My questions

why for all layers param.requires_grad is True? Shouldnt it be False at least for classifier. layers? Aren't we training them?
Does param.requires_grad==True mean that particular layer is freezed? I am confused with wording requires_grad. Does it mean freezed?
if i want to train some of the previous layers as shown here , should I use below code?

    for name, param in model.named_parameters():
         if name.startswith("..."): # choose whatever you like here
            param.requires_grad = False

considering it takes a lot of time to train, is there specific recommendation regarding layers that I should train? To begin with I am planning to train -

all layers starting with  longformer.encoder.layer.11. and
`classifier.dense.weight`
`classifier.dense.bias`
`classifier.out_proj.weight`
`classifier.out_proj.bias`

Do i need add any additional layers such as dropout or is that already taken care by LongformerForSequenceClassification.from_pretrained? I am not seeing any dropout layers in the above output and that's why asking the question

#------------------
update 1
How could I know which layers are frozen by using below code from the answer given by @joe32140 ? My guess is everything except last 4 layers from my output shown in my original question gets frozen. But is there any easier way to check?
for param in model.base_model.parameters():
    param.requires_grad = False

Output:
requires_grad==True means that we will compute the gradient of this tensor, so the default setting is we will train/finetune all layers.
You can only train the output layer by freezing the encoder with

for param in model.base_model.parameters():
    param.requires_grad = False

Yes, dropout is used in huggingface output layer implementation. See here: https://github.com/huggingface/transformers/blob/198c335d219a5eb4d3f124fdd1ce1a9cd9f78a9b/src/transformers/models/longformer/modeling_longformer.py#L1938

As for update 1, yes, base_model refers to layers excluding the output classification head. However, it's actually two layers instead of four where each layer has a weight and a bias tensors.


Instruction: Keras Upsampling2d vs PyTorch Upsampling

Input:
I am trying to convert a Keras Model to PyTorch. Now, it involves the UpSampling2D from keras. When I used torch.nn.UpsamplingNearest2d in pytorch, as default value of UpSampling2D in keras is nearest, I got different inconsistent results. The example is as follows:
Keras behaviour
In [3]: t1 = tf.random_normal([32, 8, 8, 512]) # as we have channels last in keras

In [4]: u_s = tf.keras.layers.UpSampling2D(2)(t1)

In [5]: u_s.shape
Out[5]: TensorShape([Dimension(32), Dimension(16), Dimension(16), Dimension(512)])

So the output shape is (32,16,16,512). Now let's do the same thing with PyTorch.
PyTorch Behaviour
In [2]: t1 = torch.randn([32,512,8,8]) # as channels first in pytorch

In [3]: u_s = torch.nn.UpsamplingNearest2d(2)(t1)

In [4]: u_s.shape
Out[4]: torch.Size([32, 512, 2, 2])

Here output shape is (32,512,2,2) as compared to (32,512,16,16) from keras.
So how do I get equvivlent results of Keras in PyTorch. Thanks

Output:
In keras, it uses a scaling factor to upsample. SOURCE.
tf.keras.layers.UpSampling2D(size, interpolation='nearest')

size: Int, or tuple of 2 integers. The upsampling factors for rows and columns.

And, PyTorch provides, both, direct output size and scaling factor. SOURCE.
torch.nn.UpsamplingNearest2d(size=None, scale_factor=None)

To specify the scale, it takes either the size or the scale_factor as its constructor argument.

So, in your case
# scaling factor in keras
t1 = tf.random.normal([32, 8, 8, 512])
tf.keras.layers.UpSampling2D(2)(t1).shape
TensorShape([32, 16, 16, 512])

# direct output size in pytorch
t1 = torch.randn([32,512,8,8]) # as channels first in pytorch
torch.nn.UpsamplingNearest2d(size=(16, 16))(t1).shape
# or torch.nn.UpsamplingNearest2d(size=16)(t1).shape
torch.Size([32, 512, 16, 16])

# scaling factor in pytorch.
torch.nn.UpsamplingNearest2d(scale_factor=2)(t1).shape
torch.Size([32, 512, 16, 16])


Instruction: How to combine an image tensor (4D) and a depth tensor (4D) to create a 5D tensor [batch size, channels, depth, height, width] in PyTorch?

Input:
During training, I load image and disparity data. The image tensor is of shape: [2, 3, 256, 256], and disparity/depth tensor is of shape: [2, 1, 256, 256] (batch size, channels, height, width).
I want to use Conv3D, so I need to combine these two tensors and create a new tensor of shape: [2, 3, 256, 256, 256] (batch size, channels, depth, height, width).
The depth values range from 0-400, and a possibility is to divide that into intervals, e.g., 4 intervals of 100. I want the resulting tensor to look like a voxel, similarly to the technique used in this paper. The training loop that iterates over the data is below:
 for batch_id, sample in enumerate(train_loader):

        sample = {name: tensor.cuda() for name, tensor in sample.items()}

        # image tensor [2, 3, 256, 256]
        rgb_image = transforms.Lambda(lambda x: x.mul(255))(sample["frame"])

        # translate disparity to depth
        depth_from_disparity_frame = 132.28 / sample["disparity_frame"]
        # depth tensor [2, 1, 256, 256]
        depth_image = depth_from_disparity_frame.unsqueeze(1)

Output:
From the article your linked:

We create a
3D voxel representation, with the same height and width as
the original image, and with a depth determined by the difference between the maximum and minimum depth values
found in the images. Each RGB-D pixel of an image is then
placed at the same position in the voxel grid but at its corresponding depth.

This is what Ivan suggested, more or less. If you know that your depth will always be 0-400 and I imagine that you can skip the first part of "depth determined by the difference between the maximum and minimum depth values". This could always be normalized before-hand or later.
Code using dummy data:
import torch
import torch.nn.functional as F

# Declarations (dummy tensors)
rgb_im = torch.randint(0, 255, [1, 3, 256, 256])
depth = torch.randint(0, 400, [1, 1, 256, 256])

# Calculations
depth_ohe = F.one_hot(depth, num_classes=400)       # of shape (batch, channel, height, width, binary)
bchwd_tensor = rgb_im.unsqueeze(-1)*depth_ohe       # of shape (batch, channel, height, width, depth)
bcdhw_tensor = bchwd_tensor.permute(0, 1, 4, 2, 3)  # of shape (batch, channel, depth, height, width)


Instruction: torch matrix equaity sum operation

Input:
I want to do an operation similar to matrix multiplication, except instead of multiplying I want to check equality. The effect that I want to achieve is similar to the following:
a = torch.Tensor([[1, 2, 3], [4, 5, 6]]).to(torch.uint8)
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]]).to(torch.uint8)
result = [[sum(a[i] == b [j]) for j in range(len(b))] for i in range(len(a))]

Is there a way that I can use einsum, or any other function in pytorch to achieve the above efficiently?

Output:
You can make use of the broadcasting to do the same, for instance with
result = (a[:, None, :] == b[None, :, :]).sum(dim=2)

Here None just introduces a dummy dimensions - alternatively you can use the less visual .unsqueeze() instead.


Instruction: Restructure code to avoid for loops in training loop?

Input:
I am defining a train function which I pass in a
data_loader as a dict.

data_loader['train']: consists of train data
data_loader['val'] consists of validation data.

I created a loop which iterates through which phase I am in (either train or val) and sets the model to either model.train() or model.eval() accordingly. However I feel I have too many nested for loops here making it computationally expensive. Could anyone recommend a better way of going about constructing my train function? Should I create a separate function for validating instead?
Below is what I have so far:
#Make train function (simple at first)
def train_network(model, optimizer, data_loader, no_epochs):

  total_epochs = notebook.tqdm(range(no_epochs))

  for epoch in total_epochs:

    for phase in ['train', 'val']:
      if phase == 'train':
        model.train()
      else:
        model.eval()

      for i, (images, g_truth) in enumerate(data_loader[phase]):
        images = images.to(device)
        g_truth = g_truth.to(device)

Output:
The outer-most and inner-most for loops are common when writing training scripts.
The most common pattern I see is to do:
total_epochs = notebook.tqdm(range(no_epochs))

for epoch in total_epochs:
    # Training
    for i, (images, g_truth) in enumerate(train_data_loader):
        model.train()
        images = images.to(device)
        g_truth = g_truth.to(device)
        ...

    # Validating
    for i, (images, g_truth) in enumerate(val_data_loader):
        model.eval()
        images = images.to(device)
        g_truth = g_truth.to(device)
        ...

If you need to use your previous variable data_loader, you can replace train_data_loader with data_loader["train"] and val_data_loader with data_loader["val"]
This layout is common because we generally want to do some things differently when validating as opposed to training. This also structures the code better and avoids a lot of if phase == "train" that you might need at different parts of your inner-most loop. This does however mean that you might need to duplicate some code. The trade off is generally accepted and your original code might be considered if we had 3 or more phases, like multiple validation phases or an evaluation phase as well.


Instruction: Dataloader() takes no arguments Error while executing a code to run a file in python using OOPS methodology

Input:
Hi Guys,
I am trying to create a training, validation and test dataset in python for bike sharing dataset using Object oriented programming.
I have first created a method called "DATALOADER" to read the file and then split the data into train, validation and test set. However, I am facing some challenges while executing the code.
Pasting the code above  and error response below. Need some help with that.
Error Message:
**---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-29-29ec681918b0> in <module>
----> 1 dataloader = Dataloader('C:/Users/pbhal/Downloads/hour.csv/hour.csv')
2 train, val, test = dataloader.getData()
3 fullData = dataloader.getFullData()
4
5 category_features = ['season', 'holiday', 'mnth', 'hr', 'weekday', 'workingday', 'weathersit']
TypeError: Dataloader() takes no arguments**
I was trying to create a train, validation and test set out of hour.csv datafile. However , it did not work out

Output:
You have a typo, you should use __init__ (two underscores) instead of _init_. This means that your class does not have an initialization method defined, and the fallback (the Python class object I think) does not receive any argument.
You can validate that this is the issue with a small empty class:
class Example:
    pass

a = Example()  # Works
b = Example(1)  # Fails with "TypeError: Example() takes no arguments"


Instruction: Error Training Custom COCO Dataset with Detectron2

Input:
I'm trying to train a custom COCO-format dataset with Detectron2 on PyTorch. My datasets are json files with the aforementioned COCO-format, with each item in the "annotations" section looking like this:

The code for setting up Detectron2 and registering the training & validation datasets are as follows:
from detectron2.data.datasets import register_coco_instances
for d in ["train", "validation"]:
    register_coco_instances(f"segmentation_{d}", {}, f"/content/drive/MyDrive/Segmentation Annotations/{d}.json", f"/content/drive/MyDrive/Segmentation Annotations/imgs")

from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("segmentation_train",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")
cfg.SOLVER.IMS_PER_BATCH = 2
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 1000
cfg.SOLVER.STEPS = []
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 20

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = DefaultTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()

However, when I run the training, I get the following error after the first iteration:
KeyError                                  Traceback (most recent call last)
<ipython-input-12-2aaec108c313> in <module>()
     17 trainer = DefaultTrainer(cfg)
     18 trainer.resume_or_load(resume=False)
---> 19 trainer.train()

8 frames
/usr/local/lib/python3.7/dist-packages/detectron2/engine/defaults.py in train(self)
    482             OrderedDict of results, if evaluation is enabled. Otherwise None.
    483         """
--> 484         super().train(self.start_iter, self.max_iter)
    485         if len(self.cfg.TEST.EXPECTED_RESULTS) and comm.is_main_process():
    486             assert hasattr(

/usr/local/lib/python3.7/dist-packages/detectron2/engine/train_loop.py in train(self, start_iter, max_iter)
    147                 for self.iter in range(start_iter, max_iter):
    148                     self.before_step()
--> 149                     self.run_step()
    150                     self.after_step()
    151                 # self.iter == max_iter can be used by `after_train` to

/usr/local/lib/python3.7/dist-packages/detectron2/engine/defaults.py in run_step(self)
    492     def run_step(self):
    493         self._trainer.iter = self.iter
--> 494         self._trainer.run_step()
    495
    496     @classmethod

/usr/local/lib/python3.7/dist-packages/detectron2/engine/train_loop.py in run_step(self)
    265         If you want to do something with the data, you can wrap the dataloader.
    266         """
--> 267         data = next(self._data_loader_iter)
    268         data_time = time.perf_counter() - start
    269

/usr/local/lib/python3.7/dist-packages/detectron2/data/common.py in __iter__(self)
    232
    233     def __iter__(self):
--> 234         for d in self.dataset:
    235             w, h = d["width"], d["height"]
    236             bucket_id = 0 if w > h else 1

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    519             if self._sampler_iter is None:
    520                 self._reset()
--> 521             data = self._next_data()
    522             self._num_yielded += 1
    523             if self._dataset_kind == _DatasetKind.Iterable and \

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _next_data(self)
   1181             if len(self._task_info[self._rcvd_idx]) == 2:
   1182                 data = self._task_info.pop(self._rcvd_idx)[1]
-> 1183                 return self._process_data(data)
   1184
   1185             assert not self._shutdown and self._tasks_outstanding > 0

/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in _process_data(self, data)
   1227         self._try_put_index()
   1228         if isinstance(data, ExceptionWrapper):
-> 1229             data.reraise()
   1230         return data
   1231

/usr/local/lib/python3.7/dist-packages/torch/_utils.py in reraise(self)
    423             # have message field
    424             raise self.exc_type(message=msg)
--> 425         raise self.exc_type(msg)
    426
    427

KeyError: Caught KeyError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 28, in fetch
    data.append(next(self.dataset_iter))
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/common.py", line 201, in __iter__
    yield self.dataset[idx]
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/common.py", line 90, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/usr/local/lib/python3.7/dist-packages/detectron2/utils/serialize.py", line 26, in __call__
    return self._obj(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/dataset_mapper.py", line 189, in __call__
    self._transform_annotations(dataset_dict, transforms, image_shape)
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/dataset_mapper.py", line 132, in _transform_annotations
    annos, image_shape, mask_format=self.instance_mask_format
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/detection_utils.py", line 400, in annotations_to_instances
    segms = [obj["segmentation"] for obj in annos]
  File "/usr/local/lib/python3.7/dist-packages/detectron2/data/detection_utils.py", line 400, in <listcomp>
    segms = [obj["segmentation"] for obj in annos]
KeyError: 'segmentation'

You all have any idea why this might be happening, and if so, what can be done to fix it? Any input is appreciated.
Thanks!

Output:
It's difficult to give a concrete answer without looking at the full annotation file, but a KeyError exception is raised when trying to access a key that is not in a dictionary. From the error message you've posted, this key seems to be 'segmentation'.
This is not in your code snippet, but before even getting into network training, have you done any exploration/inspections using the registered datasets? Doing some basic exploration or inspections would expose any problems with your dataset so you can fix them early in your development process (as opposed to letting the trainer catch them, in which case the error messages could get long and confounding).
In any case, for your specific issue, you can take the registered training dataset and check if all annotations have the 'segmentation' field. A simple code snippet to do this below.
# Register datasets
from detectron2.data.datasets import register_coco_instances
for d in ["train", "validation"]:
    register_coco_instances(f"segmentation_{d}", {}, f"/content/drive/MyDrive/Segmentation Annotations/{d}.json", f"/content/drive/MyDrive/Segmentation Annotations/imgs")

# Check if all annotations in the registered training set have the segmentation field
from detectron2.data import DatasetCatalog

dataset_dicts_train = DatasetCatalog.get('segmentation_train')

for d in dataset_dicts_train:
    for obj in d['annotations']:
        if 'segmentation' not in obj:
            print(f'{d["file_name"]} has an annotation with no segmentation field')

It would be strange if some images have annotations with no 'segmentation' fields in them, but it would indicate that there's some problem in your upstream annotation process.


Instruction: For loops in a dictionary, pytorch

Input:
Hi guys I have a question, for the variable "image_datasets" there is a for loop for x in ['train', 'val']. I have never seen the implementation of a for loop in a dict before.
data_transforms = {
'train': transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)
]),
'val': transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean, std)
]),
}

data_dir = 'data/hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=0)
              for x in ['train', 'val']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

Output:
This is called dictionary comprehension and it is iterating over a list.
The code
dataloaders = {
    x: torch.utils.data.DataLoader(
        image_datasets[x],
        batch_size=4,
        shuffle=True,
        num_workers=0
    )
    for x in ['train', 'val']
}

is equivalent to
dataloaders = {
    'train': torch.utils.data.DataLoader(
        image_datasets['train'],
        batch_size=4,
        shuffle=True,
        num_workers=0
    ),
    'val': torch.utils.data.DataLoader(
        image_datasets['val'],
        batch_size=4,
        shuffle=True,
        num_workers=0
    )
}


Instruction: PyTorch clip_grad_norm vs clip_grad_norm_, what is the differece when it has underline?

Input:
When coding PyTorch in torch.nn.utils I see two functions, clip_grad_norm and clip_grad_norm_.
I want to know the difference so I went to check the documentation but when I searched I only found the clip_grad_norm_ and not clip_grad_norm.
So I'm here to ask if anyone knows the difference.

Output: Pytorch uses the trailing underscore convention for in-place operations. So the difference is that the one with an underscore modifies the tensor in place and the other one leaves the original tensor unmodified and returns a new tensor.


Instruction: Python assigning attribute changing length

Input:
I have a class that accepts a parameter X. This parameter X is a numpy array, containing lists that contain ints.
array([[  101,  2002,  8542, ...,     0,     0,     0],
   [  101,  2002,  8974, ...,     0,     0,     0],
   [  101,  5076,  2743, ...,     0,     0,     0],
   ...,
   [  101,  4302,  2253, ...,     0,     0,     0],
   [  101, 13875,  2003, ...,     0,     0,     0],
   [  101,  1045,  2031, ...,     0,     0,     0]])

I have a class that takes this X and assigns it to an attribute.
class TaskADataset(Dataset):
  def __init__(self, X, y):
    self.X = X,
    self.y = y

But the parameter X and the attribute X now have different lengths.
dataset = TaskADataset(X, y)
print(len(dataset.X), len(X))

1 10000

Why is this occurring?
Thank you for the help.

Output: As shriakhilc pointed out, I included a "," and this turned it into a tuple with one element.


Instruction: Validation accuracy and loss is the same after each epoch

Input:
My validation accuracy is the same after every epoch. Not sure what i'm doing wrong here? I have added my CNN network and my training function below. I initialise the CNN once. The training function however, works perfectly fine, the loss goes down and the accuracy increases per epoch. I made a test function the same structure as my validation function and the same thing happens. My train/val split is 40000/10000. I am using cifar 10.
Below is my code:

#Make train function (simple at first)
def train_network(model, optimizer, train_loader, num_epochs=10):

  total_epochs = notebook.tqdm(range(num_epochs))
  model.train()

  for epoch in total_epochs:
    train_acc = 0.0
    running_loss = 0.0

    for i, (x_train, y_train) in enumerate(train_loader):
      x_train, y_train = x_train.to(device), y_train.to(device)

      y_pred = model(x_train)
      loss = criterion(y_pred, y_train)

      loss.backward()
      optimizer.step()
      optimizer.zero_grad()

      running_loss += loss.item()
      train_acc += accuracy(y_pred, y_train)

    running_loss /= len(train_loader)
    train_acc /= len(train_loader)

    print('Evaluation Loss: %.3f | Evaluation Accuracy: %.3f'%(running_loss, train_acc))

@torch.no_grad()
def validate_network(model, optimizer, val_loader, num_epochs=10):
  model.eval()
  total_epochs = notebook.tqdm(range(num_epochs))

  for epoch in total_epochs:
    accu = 0.0
    running_loss = 0.0

    for i, (x_val, y_val) in enumerate(val_loader):
      x_val, y_val = x_val.to(device), y_val.to(device)

      val_pred = model(x_val)
      loss = criterion(val_pred, y_val)

      running_loss += loss.item()
      accu += accuracy(val_pred, y_val)

    running_loss /= len(val_loader)
    accu /= len(val_loader)

    print('Val Loss: %.3f | Val Accuracy: %.3f'%(running_loss,accu))

OUTPUT:
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786
Val Loss: 0.623 | Val Accuracy: 0.786

So I guess my question is, how do I get a  a representative output for my accuracy and loss per epoch when validating.

Output:
What happens here is that you run a loop for number_of_epochs where you just trst the same network multiple times. I would recommend you calling the validation function during training at the end of each epoch to test the improvement of the epoch to the model's performance. This means that the training function should look something like:
def train_network(model, optimizer, train_loader, val_loader, num_epochs=10):

  total_epochs = notebook.tqdm(range(num_epochs))
  model.train()

  for epoch in total_epochs:
    train_acc = 0.0
    running_loss = 0.0

    for i, (x_train, y_train) in enumerate(train_loader):
      x_train, y_train = x_train.to(device), y_train.to(device)

      y_pred = model(x_train)
      loss = criterion(y_pred, y_train)

      loss.backward()
      optimizer.step()
      optimizer.zero_grad()

      running_loss += loss.item()
      train_acc += accuracy(y_pred, y_train)

    running_loss /= len(train_loader)
    train_acc /= len(train_loader)

    print('Evaluation Loss: %.3f | Evaluation Accuracy: %.3f'%(running_loss, train_acc))
    validate_network(model, optimizer, val_loader, num_epochs=1)

Notice that I added the validation loader as input and called the validation function at the end of each epoch, setting the validation number of epochs to 1. A small additional change will be to remove the epochs loop from the validation function.


Instruction: How to place the dataset for training Yolov5?

Input:
I’m currently working on object detection using yolov5. I trained a model with a custom dataset which has 3 classes = [‘Car’,‘Motorcycle’,‘Person’]
I have many questions related to yolov5.

All the custom images are labelled using Roboflow.
question1 : As you can see from the table that my dataset has mix of images with different sizes. Will this be a problem in training? And also assume that i’ve trained the model and got ‘best.pt’. Will that model work efficiently in any dimensions of images/videos.
question 2:

Is this directory model correct for training. Even i have ‘test’ directory but it seems that the directory is not at all used. The images in the ‘test’ folder is useless. ( I know that i’m asking dumb questions, please bare with me.)
Is it ok if place all my images like this

And should i need a ‘test’ folder?
question3: What is the ‘imgsz’ in detect.py? Is it downsampling the input source?
I’ve spent more than 3 weeks in yolo. I love it but i find some parts difficult to grasp. kindly provide suggestion for this questions. Thanks in advance.

Output:
"question1 : As you can see from the table that my dataset has mix of images with different sizes. Will this be a problem in training? And also assume that i’ve trained the model and got ‘best.pt’. Will that model work efficiently in any dimensions of images/videos."

As long as you've resized/normalized all of your images to be the same square size, then you should be fine. YOLO trains on square images. You can use a platform like Roboflow to process your images so they not only come out in the right structure (for your images and annotation files) but also resize them while generating your dataset so they are all the same size. http://roboflow.com/ - you just need to make a public workspace to upload your images to and you can use the platform free. Here's a video that covers custom training with YOLOv5: https://www.youtube.com/watch?v=x0ThXHbtqCQ

Roboflow's python package can also be used to extract your images programmatically: https://docs.roboflow.com/python
"Is this directory model correct for training. Even i have ‘test’ directory but it seems that the directory is not at all used. The images in the ‘test’ folder is useless. ( I know that i’m asking dumb questions, please bare with me.)"

Yes that directory model is correct from training. Its what I have whenever I run YOLOv5 training too.

You do need a test folder if you want to run inference against the test folder images to learn more about your model's performance.
The 'imgsz' parameter in detect.py is for setting the height/width of the images for inference. You set it at the value you used for --img when you ran train.py.
For example: Resized images to 640 by 640 when generating your images for training? Use (640, 640) for the 'imgsz' parameter (that is the default value). And that would also mean you set --img to 640 when you ran train.py

detect.py parameters (YOLOv5 Github repo)

train.py parameters (YOLOv5 Github repo)

YOLOv5's Github: Tips for Best Training Results https://github.com/ultralytics/yolov5/wiki/Tips-for-Best-Training-Results
Roboflow's Model Production Tips: https://docs.roboflow.com/model-tips


Instruction: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend

Input:
I am trying to save the pytorch model into .ptl file and loading it in android but it keeps throwing this error and driving me nuts.

Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [CPU, Vulkan, BackendSelect, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC].

But the thing is that I am tranferring my model to cpu before saving it. So the error doesn't even make sense.
example = torch.rand(1, 3, 224, 224)
model_conv = model_conv.to("cpu")

for param in model_conv.parameters():
    if param.is_cuda:
        print("Tensor on cuda")
        break
else:
    print("No tensor on cuda.")

# move model back to cpu, do tracing, and optimize
traced_script_module = torch.jit.trace(model_conv, example)
torchscript_model_optimized = optimize_for_mobile(traced_script_module)

# save optimized model for mobile
PATH = 'model.ptl'
torchscript_model_optimized._save_for_lite_interpreter(PATH)
print(f"optimized model saved to {PATH}")

The output of for loop is No tensor on cuda. This is how I am loading the model in android. I loaded a sample model from their github and it works so I doubt there is issue with android code.
module = LiteModuleLoader.load(MainActivity.assetFilePath(getApplicationContext(), "model.ptl"));

Side node: There are so many ways to save a model. Why is there not a good documentation for Pytorch Mobile. Tflite has better doucmentation than this.

Output:
Turned out it was Android issue. Android does not update the asset files even if they are changed and for me it was using an old model which was not converted to cpu.
If you are having similar issues or your accuracy is not improving, try clearing the app data either in similation or app setting.
Phone/emulator settings -> apps menu
this github issue that end up helping me: https://github.com/pytorch/pytorch/issues/53650


Instruction: pytorch reads frames from video for image classification, all predicting the same category

Input:
I have 3 categories of fish images, each category has 1000 images, when I trained the model, the classification accuracy was 97%, and when I used the folder images for prediction, the accuracy was no problem.
But when I replace the picture with a video, and cut out each frame from the video for image classification, no matter which category of video it is, all frames are predicted to be category 1:"stingray".Why?
#!/usr/bin/env python
# coding: utf-8

import torch
from torchvision import transforms
import torchvision.models as models
import cv2
import torch.nn.functional as F

CLASSES = {0:"goldfish", 1:"stingray", 2:"tench"}
BATCH_SIZE = 4
IMG_SIZE = (400, 400)
TRANSFORM_IMG = transforms.Compose([
    transforms.ToTensor(),
    transforms.Resize(IMG_SIZE),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

# model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = models.vgg19_bn(pretrained=False, num_classes=3)
model.to(device)
model.load_state_dict(torch.load('checkpoint.pt'))
model.eval()

videoCapture = cv2.VideoCapture(r'D:/video/Goldfish.mp4')
fps = videoCapture.get(cv2.CAP_PROP_FPS)
size = (int(videoCapture.get(cv2.CAP_PROP_FRAME_WIDTH)), int(videoCapture.get(cv2.CAP_PROP_FRAME_HEIGHT)))

ps = 25
fourcc = cv2.VideoWriter_fourcc(*'DIVX')
videoWriter = cv2.VideoWriter("D:/goldfish.mp4", fourcc, fps, size)

with torch.no_grad():
    success, frame = videoCapture.read()
    while success:
        # frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        image_tensor = TRANSFORM_IMG(frame)
        image_tensor = image_tensor.unsqueeze(0)
        test_input = image_tensor.to(device)
        outputs = model(test_input)
        _, predicted = torch.max(outputs, 1)
        probability =  F.softmax(outputs, dim=1)
        top_probability, top_class = probability.topk(1, dim=1)
        predicted = predicted.cpu().detach().numpy()
        predicted = predicted.tolist()[0]
        label = CLASSES[predicted]
        top_probability = top_probability.cpu().detach().numpy()
        top_probability = top_probability.tolist()[0][0]
        top_probability = '%.2f%%' % (top_probability * 100)
        print(top_probability)
        print(label) #all the label is stingray############################################
        frame = cv2.putText(frame, label+': '+top_probability, (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0, 0, 255), 2)
        videoWriter.write(frame)
        success, frame = videoCapture.read()
    videoWriter.release()

Output:
When I use this code,there is no problem.I convert image channel from BGR to RGB for pytorch.
import torch
from torchvision import transforms
import torchvision.models as models
import cv2
import torch.nn.functional as F
import copy

CLASSES = {0:"goldfish", 1:"stingray", 2:"tench"}
BATCH_SIZE = 4
IMG_SIZE = (400, 400)
TRANSFORM_IMG = transforms.Compose([
    transforms.ToTensor(),
    transforms.Resize(IMG_SIZE),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

# model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = models.vgg19_bn(pretrained=False, num_classes=3)
model.to(device)
model.load_state_dict(torch.load('checkpoint.pt'))
model.eval()

videoCapture = cv2.VideoCapture(r'video/Goldfish.mp4')
fps = videoCapture.get(cv2.CAP_PROP_FPS)
size = (int(videoCapture.get(cv2.CAP_PROP_FRAME_WIDTH)), int(videoCapture.get(cv2.CAP_PROP_FRAME_HEIGHT)))

ps = 25
fourcc = cv2.VideoWriter_fourcc(*'DIVX')
videoWriter = cv2.VideoWriter(r"D:/goldfish.mp4", fourcc, fps, size)

with torch.no_grad():
    success, frame = videoCapture.read()
    while success:
        frame_copy = copy.deepcopy(frame)
        frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_BGR2RGB)
        image_tensor = TRANSFORM_IMG(frame_copy)
        image_tensor = image_tensor.unsqueeze(0)
        test_input = image_tensor.to(device)
        outputs = model(test_input)
        _, predicted = torch.max(outputs, 1)
        probability =  F.softmax(outputs, dim=1)
        top_probability, top_class = probability.topk(1, dim=1)
        predicted = predicted.cpu().detach().numpy()
        predicted = predicted.tolist()[0]
        label = CLASSES[predicted]
        top_probability = top_probability.cpu().detach().numpy()
        top_probability = top_probability.tolist()[0][0]
        top_probability = '%.2f%%' % (top_probability * 100)
        print(top_probability)
        print(label)
        frame = cv2.putText(frame, label+': '+top_probability, (50, 50), cv2.FONT_HERSHEY_SIMPLEX, 1.5, (0, 0, 255), 2)
        videoWriter.write(frame)
        success, frame = videoCapture.read()
    videoWriter.release()


Instruction: Replace every element in each channel if the channel has any value greater than 0 without looping

Input:
I have a batch tensor of size (4, 100, 56, 56), where some channels have a certain values in it, and some only have all zeros. I wanted to make each elements in the channels has any value greater than 0 to be of 100, whereas if it has all zeros, it should be made to has 1 in each element. Any idea how to achieve this without looping?
t = torch.zeros((4, 100, 56, 56))
t[:, 5, 15:20, 15:20] = 0.07
new_t = torch.ones((4, 100, 56, 56))

for b in range(t.size(0))
    for c in range(t.size(1)):
        if t[b, c, :,:].max() > 0:
             new_t[b, c, :, :] = 100

My code above is inefficient for large batches and channels, and it create memory overhead due to new_t, is there a way to use view() or similar functions to achieve this?

Output:
You can perform the following:
mask = torch.any(t.flatten(2, 3) > 0., dim=2)
t[mask] = 100.  # or t[mask] *= 100. for differentiability


Instruction: Couldn't convert MobileNet V2 PyTorch to mlmodel using CoreML tools

Input:
I want to convert PyTorch MobileNet V2 pre-trained model to .mlmodel using coremltools. here is my code:
    import torchvision
    import torch
    import coremltools as ct

    # Load a pre-trained version of MobileNetV2
    torch_model = torchvision.models.mobilenet_v2(pretrained=True)
    # Set the model in evaluation mode.
    torch_model.eval()

    # Trace the model with random data.
    example_input = torch.rand(1, 3, 224, 224)
    traced_model = torch.jit.trace(torch_model, example_input)
    out = traced_model(example_input)

    # Using image_input in the inputs parameter:
    # Convert to Core ML using the Unified Conversion API.
    model = ct.convert(
        traced_model,
        inputs=[ct.TensorType(shape=example_input.shape)]
     )

    # Save the converted model.
    model.save("mobilenet_v2.mlmodel")

It worked well on Google Colab, but when I run it my local machine (MacBook), I've got the following error
Converting Frontend ==> MIL Ops: 100%|▉| 390/391 [00:00<00:00, 647.56
Running MIL Common passes:   0%|         | 0/34 [00:00<?, ? passes/s]anaconda3/lib/python3.8/site-packages/coremltools/converters/mil/mil/passes/name_sanitization_utils.py:101: UserWarning: Input, 'input.1', of the source model, has been renamed to 'input_1' in the Core ML model.
  warnings.warn(msg.format(var.name, new_name))
anaconda3/lib/python3.8/site-packages/coremltools/converters/mil/mil/passes/name_sanitization_utils.py:129: UserWarning: Output, '830', of the source model, has been renamed to 'var_830' in the Core ML model.
  warnings.warn(msg.format(var.name, new_name))
Running MIL Common passes: 100%|█| 34/34 [00:00<00:00, 41.87 passes/s
Running MIL Clean up passes: 100%|█| 9/9 [00:00<00:00, 80.15 passes/s
Translating MIL ==> NeuralNetwork Ops: 100%|█| 495/495 [00:00<00:00,
Traceback (most recent call last):
  File "convert_models.py", line 24, in <module>
    model = ct.convert(
  File "anaconda3/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 352, in convert
    mlmodel = mil_convert(
  File "anaconda3/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "anaconda3/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 231, in _mil_convert
    return modelClass(proto,
  File "anaconda3/lib/python3.8/site-packages/coremltools/models/model.py", line 346, in __init__
    self.__proxy__, self._spec, self._framework_error = _get_proxy_and_spec(
  File "anaconda3/lib/python3.8/site-packages/coremltools/models/model.py", line 123, in _get_proxy_and_spec
    specification = _load_spec(filename)
  File "/anaconda3/lib/python3.8/site-packages/coremltools/models/utils.py", line 210, in load_spec
    raise Exception(
Exception: Unable to load libmodelpackage. Cannot make save spec.

I am using the following versions of libraries:

torchvision: '0.9.0'
torch: '1.8.0'
coremltools: '5.2.0'

Output: I solved this problem by updating the macOS from 10.14.6 Mojave to 11.5.2 MacOS Big Sur. From the Github issue that I have created, they explained to me that the version "5.2" of coremltools is not supported in my previous MacOS version. So, by updating your macOS it should work.


Instruction: Save PyTorch model for conversion to ONNX

Input:
I'm brand new to Pytorch (and Python), I've followed this guide which trains a model and then saves the weights into a pth file:
https://medium.com/@alexppppp/how-to-create-synthetic-dataset-for-computer-vision-keypoint-detection-78ba481cdafd
My understanding is that to convert a model to ONNX, you need to save the entire thing and not just the weights.
I think the relevant code is this:
for epoch in range(num_epochs):
    train_one_epoch(model, optimizer, data_loader_train, device, epoch, print_freq=1000)
    lr_scheduler.step()
    evaluate(model, data_loader_test, device)

# Save model weights after training
torch.save(model.state_dict(), 'keypointsrcnn_weights.pth')

Is there a simple way to save the "entire" model rather than just the weights? I've seen this in the docs but this looks like it would need to go within the epoch loop rather than after it's trained?
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)

Please forgive my total lack of understanding.  My intention is to try convert a PyTorch model to ONNX.

Output:
use torch.onnx.export. Should look something like
  arch = models.alexnet();      pic_x = 227
  dummy_input = torch.zeros((1,3, pic_x, pic_x))
  torch.onnx.export(arch, dummy_input, "alexnet.onnx", verbose=True, export_params=True, )

graph(%input.1 : Float(1, 3, 227, 227, strides=[154587, 51529, 227, 1], requires_grad=0, device=cpu),
      %features.0.weight : Float(64, 3, 11, 11, strides=[363, 121, 11, 1], requires_grad=1, device=cpu),
      %features.0.bias : Float(64, strides=[1], requires_grad=1, device=cpu),
      %features.3.weight : Float(192, 64, 5, 5, strides=[1600, 25, 5, 1], requires_grad=1, device=cpu),
      %features.3.bias : Float(192, strides=[1], requires_grad=1, device=cpu),
      ...
      %classifier.6.weight : Float(1000, 4096, strides=[4096, 1], requires_grad=1, device=cpu),
      %classifier.6.bias : Float(1000, strides=[1], requires_grad=1, device=cpu)):
  %17 : Float(1, 64, 56, 56, strides=[200704, 3136, 56, 1], requires_grad=1, device=cpu) = onnx::Conv[dilations=[1, 1], group=1, kernel_shape=[11, 11], pads=[2, 2, 2, 2], strides=[4, 4]](%input.1, %features.0.weight, %features.0.bias) # c:\python39\lib\site-packages\torch\nn\modules\conv.py:442:0
  %18 : Float(1, 64, 56, 56, strides=[200704, 3136, 56, 1], requires_grad=1, device=cpu) = onnx::Relu(%17) # c:\python39\lib\site-packages\torch\nn\functional.py:1297:0
  ...


Instruction: Multiply a [3, 2, 3] by a [3, 2] tensor in pytorch (dot product along dimension)

Input:
Given the following tensors x and y with shapes  [3,2,3] and [3,2]. I want to multiply the tensors along the 2nd dimension, this is expected to be a kind of dot product and scaling along the axis and return a [3,2,3] tensor.
import torch
a  = [[[0.2,0.3,0.5],[-0.5,0.02,1.0]],[[0.01,0.13,0.06],[0.35,0.12,0.0]], [[1.0,-0.3,1.0],[1.0,0.02, 0.03]] ]
b = [[1,2],[1,3],[0,2]]
x = torch.FloatTensor(a) # shape [3,2,3]
y = torch.FloatTensor(b) # shape [3,2]

The expected output :
Expected output shape should be [3,2,3]
#output = [[[0.2,0.3,0.5],[-1.0,0.04,2.0]],[[0.01,0.13,0.06],[1.05,0.36,0.0]], [[0.0,0.0,0.0],[2.0,0.04, 0.06]] ]

I have tried the two below but none of them is giving the desired output and output shape.
torch.matmul(x,y)
torch.matmul(x,y.unsqueeze(1).shape)

What is the best way to fix this?

Output:
This is just broadcasted multiply. So you can insert a unitary dimension on the end of y to make it a [3,2,1] tensor and then multiply by x. There are multiple ways to insert unitary dimensions.
# all equivalent
x * y.unsqueeze(2)
x * y[..., None]
x * y[:, :, None]
x * y.reshape(3, 2, 1)

You could also use torch.einsum.
torch.einsum('abc,ab->abc', x, y)


Instruction: Simple Neural Network in Pytorch with 3 inputs (Numerical Values)

Input:
Having a hard time setting up a neural network most of the examples are images. My problem has 3 inputs each of size N X M where N are the samples and M are the features. I have a separate file (CSV) with 1 x N binary target (0,1).
The network i'm trying to configure should have two hidden layers with 100 and 50 neurons, respectively. Sigmoid activation function and cross-entropy to check performance. The result should just be a single probability output.
Please help?
EDIT:
import torch
import torch.nn as nn
import torch.optim as optim
import torch.autograd as autograd
import torch.nn.functional as F
#from torch.autograd import Variable
import pandas as pd

# Import Data
Input1 = pd.read_csv(r'...')
Input2 = pd.read_csv(r'...')
Input3 = pd.read_csv(r'...')
Target = pd.read_csv(r'...' )

# Convert to Tensor
Input1_tensor = torch.tensor(Input1.to_numpy()).float()
Input2_tensor = torch.tensor(Input2.to_numpy()).float()
Input3_tensor = torch.tensor(Input3.to_numpy()).float()
Target_tensor = torch.tensor(Target.to_numpy()).float()

# Transpose to have signal as columns instead of rows
input1 = Input1_tensor
input2 = Input2_tensor
input3 = Input3_tensor
y = Target_tensor

# Define the model
class Net(nn.Module):
    def __init__(self, num_inputs, hidden1_size, hidden2_size, num_classes):
        # Initialize super class
        super(Net, self).__init__()
        #self.criterion = nn.CrossEntropyLoss()

        # Add hidden layer
        self.layer1 = nn.Linear(num_inputs,hidden1_size)
        # Activation
        self.sigmoid = torch.nn.Sigmoid()
        # Add output layer
        self.layer2 = nn.Linear(hidden1_size,hidden2_size)
        # Activation
        self.sigmoid2 = torch.nn.Sigmoid()
        self.layer3 = nn.Linear(hidden2_size, num_classes)

    def forward(self, x1, x2, x3):
        # implement the forward pass

        in1 = self.layer1(x1)
        in2 = self.layer1(x2)
        in3 = self.layer1(x3)

        xyz = torch.cat((in1,in2,in3),1)

        return xyz

# Define loss function
loss_function = nn.CrossEntropyLoss()

# Define optimizer
optimizer = optim.SGD(model.parameters(), lr=1e-4)

for t in range(num_epochs):

    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(input1, input2, input3)

    # Compute and print loss
    loss = loss_function(y_pred, y)
    print(t, loss.item())

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()

    # Calculate gradient using backward pass
    loss.backward()

    # Update model parameters (weights)
    optimizer.step()

Here I am getting an error of "
RuntimeError: 0D or 1D target tensor expected, multi-target not supported"
for line "loss = loss_function(y_pred, y)"
Where y_pred is [20000,375] and y is [20000,1]

Output:
you can refer to pytorch, a python library for deep learning and neural networks.
and you can use code that defines network below:
from torch import nn
import torch.nn.functional as F

def network(nn.Module):
    def __init__(self, M):
        # M is the dimension of input feature
        super(network, self).__init__()
        self.layer1 = nn.Linear(M, 100)
        self.layer2 = nn.Linear(100, 50)
        self.out = nn.Linear(50,1)

    def forward(self,x):
        return F.sigmoid(self.out(self.layer2(self.layer1(x))))

----------

You can then refer to the pytorch documentation and finish the rest training code.
Edit:
As for RuntimeError, you can squeeze the target tensor by y.squeeze(). This will remove redundant dimension in your tensor, e.g. [20000,1] -> [20000]


Instruction: Embedding Layer - torch.nn.Embedding in Pytorch

Input:
I'm quite new to NN and sorry if my question is quite dumb. I was just reading codes on github and found the pros use embedding (in that case not a word embedding) but may I please just ask in general:

Does Embedding Layer has trainable variables that learn over time as to improve in embedding?
May you please provide an intuition on it and what circumstances to use, like would the house price regression benefit from it ?
If so (that it learns) what is the difference than just using Linear Layers?

>>> embedding = nn.Embedding(10, 3)
>>> input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
>>> input
tensor([[1, 2, 4, 5],
        [4, 3, 2, 9]])

>>> embedding(input)
tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])

Output:
In short, the embedding layer has learnable parameters and the usefulness of the layer depends on what inductive bias you want on the data.

Does Embedding Layer has trainable variables that learn over time as to improve in embedding?

Yes, as stated in the docs under the Variables section, it has an embedding weight that is altered during the training process.

May you please provide an intuition on it and what circumstances to use, like would the house price regression benefit from it?

An embedding layer is commonly used in NLP tasks where the input is tokenized. This means that the input is discrete in a sense and can be used for indexing the weight (which is basically what the embedding layer is in forward mode). This discrete attribution implies that inputs like 1, 2, 42 are entirely different (until the semantic correlation has been learnt). House price regression has continuous input space and values such as 1.0 and 1.1 might be more correlated than the values 1.0 and 42.0. This kind of assumption about the hypothesis space is called an inductive bias and pretty much every machine learning architecture conforms to some sort of inductive bias. I believe it is possible to use embedding layers for regression problems which would require some kind of discretization, but it would not benefit from it.

If so (that it learns) what is the difference than just using Linear Layers?

There is a big difference, the linear layer performs matrix multiplication with the weight as opposed to using it as a lookup table. During backpropagation for the embedding layer, the gradients will only propagate threw the corresponding indices used in the lookup and duplicate indices are accumulated.


Instruction: How to alternatively concatenate pytorch tensors?

Input:
Pytorch provides API to concatenate tensors, like cat, stack.
But does it provide any API to concatenate pytorch tensors alternatively?
For example，
suppose input1.shape = C*H*W, a1.shape = H\*W, and output.shape = (3C)*H*W
This can be achieved using a loop, but I am wondering if any Pytorch API can do this

Output:
I will try to do it with small example:
input1 = torch.full((3, 3), 1)
input2 = torch.full((3, 3), 2)
input3 = torch.full((3, 3), 3)

out = torch.concat((input1,input2, input3)).T.flatten()
torch.stack(torch.split(out, 3), dim=1).reshape(3,-1)

#output

tensor([[1, 2, 3, 1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3, 1, 2, 3],
        [1, 2, 3, 1, 2, 3, 1, 2, 3]])


Instruction: Gradio - Pytorch MNIST Digit Recognizer

Input:
I watched the following video on YouTube https://www.youtube.com/watch?v=jx9iyQZhSwI where it was shown that it is possible to use Gradio and the learned model of MNIST dataset in Tensorflow. I have read and written that it is possible to use Pytorch in Gradio, but I have problems with its implementation. Does anyone have an idea how to do this?
My Pytorch code of cnn
import torch.nn as nn
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(
                in_channels=1,
                out_channels=16,
                kernel_size=5,
                stride=1,
                padding=2,
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        # fully connected layer, output 10 classes
        self.out = nn.Linear(32 * 7 * 7, 10)
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        # flatten the output of conv2 to (batch_size, 32 * 7 * 7)
        x = x.view(x.size(0), -1)
        output = self.out(x)
        return output, x    # return x for visualization

By watching I find that  I need to change function that Gradio use
def predict_image(img):
  img_3d=img.reshape(-1,28,28)
  im_resize=img_3d/255.0
  prediction=CNN(im_resize)
  pred=np.argmax(prediction)
  return pred

Output:
Im sorry if I got your question wrong, but from what I understand you are getting an error when trying to predict the digit using your function predict image.
So here are two possible hints. Maybe you have implemented them already, but I don't know because of the very small code snippet.
First of all. Have you set your model into evaluation mode using
CNN.eval()

Do after you finished training your model and want to evaluate inputs without training the model.
Second of all, maybe you need to add a fourth dimension to your input tensor "im_resize". Normally your model expects a dimension for the number of channels, the batch size, the height and the width of your input.
In addition I can not tell if your input is a of the datatype torch.tensor . If not transform your array into a tensor first.
You can add a batch dimension to your input tensor by using
im_resize = im_resize.unsqueeze(0)

I hope that I understand your question correctly and was able to help you.


Instruction: Syntax for making objects callable in python

Input:
I understand that in python user-defined objects can be made callable by defining a __call__() method in the class definition. For example,
class MyClass:
  def __init__(self):
    pass

  def __call__(self, input1):
    self.my_function(input1)

  def my_function(self, input1):
    print(f"MyClass - print {input1}")

my_obj = MyClass()
# same as calling my_obj.my_function("haha")
my_obj("haha") # prints "MyClass - print haha"

I was looking at how pytorch makes the forward() method of a nn.Module object be called implicitly when the object is called and saw some syntax I didn't understand.
In the line that supposedly defines the __call__ method the syntax used is,
__call__ : Callable[..., Any] = _call_impl

This seemed like a combination of an annotation (keyword Callable[ following : ignored by python) and a value of _call_impl which we want to be called when __call__ is invoked, and my guess is that this is a shorthand for,
def __call__(self, *args, **kwargs):
    return self._call_impl(*args, **kwargs)

but wanted to understand clearly how this method of defining functions worked.
My question is: When would we want to use such a definition of callable attributes of a class instead of the usual def myfunc(self, *args, **kwargs)

Output:
Functions are normal first-class objects in python. The name to with which you define a function object, e.g. with a def statement, is not set in stone, any more than it would be for an int or list. Just as you can do
a = [1, 2, 3]
b = a

to access the elements of a through the name b, you can do the same with functions. In your first example, you could replace
def __call__(self, input1):
    self.my_function(input1)

with the much simpler
__call__ = my_function

You would need to put this line after the definition of my_function.
The key differences between the two implementations is that def __call__(... creates a new function. __call__ = ... simply binds the name __call__ to the same object as my_function. The noticeable difference is that if you do __call__.__name__, the first version will show __call__, while the second will show my_function, since that's what gets assigned by a def statement.


Instruction: send torch tensor or image via localhost as JSON to another app

Input:
I have a basic flask server with a Generator model loaded. I'm sending an input vector via JSON which is hitting the Generator, which spits out a prediction. This works. I want to then send this image (I would settle for sending it as any kind of data I can reconstruct on the other end) to another application running on the same machine. From what I gather, it might be best to encode the image as base64, but all of my attempts have failed. Any guidance is appreciated.
@app.route("/json", methods=['GET', 'POST', 'PUT'])
def getjsondata():

if request.method=='POST':
    print("received POST")

    data = request.get_json()

    #print(format(data['z']))
    jzf = [float(i) for i in data['z']]
    jzft = torch.FloatTensor(jzf)
    jzftr = jzft.reshape([1, 512])

    z = jzftr.cuda()
    c = None                   # class labels (not used in this example)
    trunc = 1
    img = G(z, c, trunc)
    img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)

    # now what?

Output:
I managed a solution. The Python Flask server looks like this and expects to receive a JSON object containing a Z array of 512 floats and truncation of a single float.
from flask import Flask, jsonify, request, send_file # initialize our Flask application
import json
from flask_cors import CORS
import base64
from torchvision import transforms
import dnnlib
import torch
import PIL.Image
from io import BytesIO
from datetime import datetime
import legacy

device = torch.device('cuda')
with dnnlib.util.open_url("snapshot.pkl") as f:
    G = legacy.load_network_pkl(f)['G_ema'].to(device) # type: ignore

app = Flask(__name__)
CORS(app)

@app.route("/test", methods=['GET', 'POST', 'PUT'])
def test():
    return "OK"

@app.route("/query", methods=['GET', 'POST', 'PUT'])
def getjsondata():

    if request.method=='POST':
        # print("received POST")

        data = request.get_json()

        #print(format(data['z']))
        jzf = [float(i) for i in data['z']]
        jzft = torch.FloatTensor(jzf)
        jzftr = jzft.reshape([1, 512])

        z = jzftr.cuda()
        c = None                   # class labels (not used in this example)
        trunc = data['truncation']
        img = G(z, c, trunc)

        #img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)
        img = (img * 127.5 + 128).clamp(0, 255).to(torch.uint8)

        # turn into PIL image
        pil_img = transforms.ToPILImage()(img[0]).convert("RGB")
        #pil_img = PIL.Image.fromarray(img[0].cpu().numpy(), 'RGB')

        # SAVING...
        #fn = datetime.today().strftime('%Y-%m-%d-%H:%M:%S')
        #pil_img.save('saved_images/' + fn + '.jpg')

        response = serve_pil_image64(pil_img)
        response.headers.add('Access-Control-Allow-Origin', '*')
        # response.headers.add('Content-Transfer-Encoding', 'base64')
        return response

    return 'OK'

def serve_pil_image64(pil_img):
    img_io = BytesIO()
    pil_img.save(img_io, 'JPEG', quality=70)
    img_str = base64.b64encode(img_io.getvalue()).decode("utf-8")
    return jsonify({'status': True, 'image': img_str})

if __name__ == '__main__':
    app.run(host='localhost', port=9000, debug=True)

As it stands, I am sending said JSON array from a simple JavaScript/HTML site. It then listens for the response, also JSON.
// construct an HTTP request
var xhr = new XMLHttpRequest();

// upon successful completion of request...
xhr.onreadystatechange = function() {
    if (xhr.readyState == XMLHttpRequest.DONE) {
        var json = JSON.parse(xhr.responseText);
        // console.log(json);
        document.getElementById("image_output").src = "data:image/jpeg;base64," + json.image;

    }
}

xhr.open("POST", "http://localhost:9000/json");
xhr.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');


Instruction: How to rearrange the sample order of a torch dataloader?

Input: I have a "torch.utils.data.DataLoader". I want to rearrange the order of the samples. Is it possible?

Output:
Yes, you can use torch.utils.data.Subset and specify the indices.
import numpy as np
import torch
from torch.utils.data import DataLoader, Subset, TensorDataset

data = np.arange(5) ** 2
dataset = TensorDataset(torch.tensor(data))

# Subset with entire Dataset in rearranged order
dataset_ordered = Subset(dataset, indices=[2, 1, 3, 4, 0])

for x in DataLoader(dataset_ordered):
    print(x)

# [tensor([4])]
# [tensor([1])]
# [tensor([9])]
# [tensor([16])]
# [tensor([0])]


Instruction: RuntimeError: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu"

Input:
Earlier I have configured the following project
https://github.com/zllrunning/face-makeup.PyTorch
using Pytorch with CUDA=10.2, Now Pytorch with CUDA=10.2 support is not available for Windows.
So, when I am configuring the same project using Pytorch with CUDA=11.3, then I am getting the following error:
RuntimeError: Attempted to set the storage of a tensor on device "cuda:0" to a storage on different device "cpu".  This is no longer allowed; the devices must match.

Please help me in solving this problem.

Output:
I solved this by adding map_location=lambda storage, loc: storage.cuda() in the model_zoo.load_url method. I think in torch 1.12 they have changed the default location from GPU to CPU (which does not make any sense).
Edit:
In the file resnet.py, under function def init_weight(self):, the following line
state_dict = modelzoo.load_url(resnet18_url)
is changed with
state_dict = modelzoo.load_url(resnet18_url, map_location=lambda storage, loc: storage.cuda())


Instruction: Compute metrics/loss every n batches Pytorch Lightning

Input:
I'm trying to use Pytorch lighning but I don't have clear all the steps. Anyway I'm trying to calculate the train_loss (for example) not only for each step(=batch) but every n bacthes (i.e. 500) but I'm not sure how to compute it (compute, reset etc). I tried this approach but this is not working. Can you help me? thanks
def training_step(self, batch: tuple, batch_nb: int, *args, **kwargs) -> dict:
        """
        Runs one training step. This usually consists in the forward function followed
            by the loss function.

        :param batch: The output of your dataloader.
        :param batch_nb: Integer displaying which batch this is
        Returns:
            - dictionary containing the loss and the metrics to be added to the lightning logger.
        """
        inputs, targets = batch
        model_out = self.forward(**inputs)
        loss_val = self.loss(model_out, targets)

        y = targets["labels"]
        y_hat = model_out["logits"]

        labels_hat = torch.argmax(y_hat, dim=1)
        val_acc = self.metric_acc(labels_hat, y)
        tqdm_dict = {"train_loss": loss_val, 'batch_nb': batch_nb}
        self.log('train_loss', loss_val, on_step=True, on_epoch=True,prog_bar=True)
        self.log('train_acc', val_acc, on_step=True, prog_bar=True,on_epoch=True)

        # reset the metric to restart accumulating
        self.loss_val_bn = self.loss(model_out, targets) #accumulate state
        if batch_nb % 500 == 0:
            self.log("x batches test loss_train", self.loss_val_bn.compute(),batch_nb) # perform a compute every 10 batches
            self.loss_val_bn.reset()

        #output = OrderedDict(
            #{"loss": loss_val, "progress_bar": tqdm_dict, "log": tqdm_dict})

        # can also return just a scalar instead of a dict (return loss_val)
        #return output
        return loss_val

Output:
Write your custom logger following (https://pytorch-lightning.readthedocs.io/en/stable/extensions/logging.html#make-a-custom-logger). The one I present here stores the values generated in every step in a dict where each metric name is a key.

class History_dict(LightningLoggerBase):
    def __init__(self):
        super().__init__()

        self.history = collections.defaultdict(list) # copy not necessary here
        # The defaultdict in contrast will simply create any items that you try to access

    @property
    def name(self):
        return "Logger_custom_plot"

    @property
    def version(self):
        return "1.0"

    @property
    @rank_zero_experiment
    def experiment(self):
        # Return the experiment object associated with this logger.
        pass

    @rank_zero_only
    def log_metrics(self, metrics, step):
        # metrics is a dictionary of metric names and values
        # your code to record metrics goes here
        for metric_name, metric_value in metrics.items():
            if metric_name != 'epoch':
                self.history[metric_name].append(metric_value)
            else: # case epoch. We want to avoid adding multiple times the same. It happens for multiple losses.
                if (not len(self.history['epoch']) or    # len == 0:
                    not self.history['epoch'][-1] == metric_value) : # the last values of epochs is not the one we are currently trying to add.
                    self.history['epoch'].append(metric_value)
                else:
                    pass
        return

    def log_hyperparams(self, params):
        pass

Make the model reduce the stored metrics every n steps. I assume that taking the mean is how you want to reduce here. Empty the list after reducing.

class MNISTModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_nb):
        x, y = batch
        loss = F.cross_entropy(self(x), y)
        self.log('loss_epoch', loss, on_step=False, on_epoch=True)
        self.log('loss_step', loss, on_step=True, on_epoch=False)
        print(self.global_step)
        if batch_nb % 50 == 0 and self.global_step != 0:
            step_metrics = self.logger.history['loss_step']
            # I am assuming that the reduction function you want over the saved step values are mean
            reduced = sum(step_metrics) / len(step_metrics)
            print(reduced)
            # Empty the loss list
            self.logger.history['loss_step'] = []
        return loss

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)

Pass this custom logger to your trainer

hd = History_dict()

# Initialize a trainer
trainer = Trainer(
    accelerator="auto",
    devices=1 if torch.cuda.is_available() else None,  # limiting got iPython runs
    max_epochs=3,
    callbacks=[TQDMProgressBar(refresh_rate=20)],
    log_every_n_steps=10,
    logger=[hd],
)

You should see the following being printed with this exact configuration:
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
0.9309097290039062
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
0.9098988473415375
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
0.8920584758122762
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
0.8698084503412247
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
0.8622475385665893
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
0.8433656434218089
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
0.8188161773341043
...


Instruction: Adding Intel MKL and MKL-DNN in Docker

Input:
I have ML code (e.g. Numpy, Scipy, LightGBM, PyTorch) deployed with Docker. I am using Python with Poetry, installing packages with pip.
What should I do in order to use MKL and MKL-DNN? I know that the most standard way is to use Anaconda, but I cannot (large business, without commercial Anaconda license).
Will pip install mkl suffice?
How to install MKL-DNN, so that PyTorch will use it?

Output:
Will pip install mkl suffice?

No, it will not, see the section in the numpy install docs:

The NumPy wheels on PyPI, which is what pip installs, are built with OpenBLAS. The OpenBLAS libraries are included in the wheel. This makes the wheel larger, and if a user installs (for example) SciPy as well, they will now have two copies of OpenBLAS on disk.

So you will need to built numpy from source.

I know that the most standard way is to use Anaconda, but I cannot (large business, without commercial Anaconda license).

Have you considered using miniforge and miniconda? IANAL, but I am quite certain that you are just not allowed to use the ana-/miniconda distributions and the anaconda channel in large scale commercial products, but conda-forge can still be used free of charge. You should be able to set up all the requirements that you mentioned from conda-forge. At least you would probably have an easier time compiling pytorch from source


Instruction: matmul to every row in pytorch tensor

Input:
I currently have a tensor that looks like this (numbers randomly selected), which I will call x:
tensor([[ 1., -5.],
        [ 2., -4.],
        [ 3.,  2.],
        [ 4.,  1.],
        [ 5.,  2.]])

I also have another 2D tensor (call it i)
tensor([[-1.,  1.],
        [ 1., -1.]], requires_grad=True)

I hope to pytorch.matmul i to each row in x. Is there a way for me to achieve this?
Below is my attempt:
apply_i = lambda x: torch.matmul(x, i)
final = pytorch.tensor([apply_i(a) for a in x])

It throws an error saying "only one element tensors can be converted to Python scalars". Does not work even when I remove the square bracket.
Any help would be appreciated!

Output:
import torch
x = torch.tensor([[ 1., -5.],
        [ 2., -4.],
        [ 3.,  2.],
        [ 4.,  1.],
        [ 5.,  2.]])

change your code:
i = torch.tensor([[-1.,  1.],
        [ 1., -1.]], requires_grad=True)
apply_i = lambda x: torch.matmul(x, i)
# final = torch.tensor([apply_i(a) for a in x])
final = [apply_i(a) for a in x]
final = torch. stack(final)


Instruction: Pytorch DataLoader is not dividing the dataset into batches

Input:
I am trying to load training data in the DataLoader with following code
class Dataset(Dataset):
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __getitem__(self, index):
        x = torch.Tensor(self.x[index])
        y = torch.Tensor(self.y[index])
        return (x, y)

    def __len__(self):
        count = self.x.shape[0]
        return count

X_train = np.reshape(X_train,(-1,1,X_train.shape[0],X_train.shape[1]))
y_train = np.reshape(y_train,(-1,1,y_train.shape[0],y_train.shape[1]))
train_dataset = Dataset(X_train, y_train)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=128,shuffle=True)

Now, when I check the length of the DataLoader, I get one dataset everytime. The loader is not splitting the dataset into batches. What am I doing wrong here?

Output:
After testing your code, it seems to work perfectly if you remove the reshape steps. You're introducing a new dimension, so the new shape of X_train is (1, something, something), but you're indexing your items using self.x[index], so you're always accessing the batch dimension. You make the same mistake when calculating the length of your dataset: is always 1.
Solution: do not reshape.
X_train = np.random.rand(12_000, 1280)
y_train = np.random.rand(12_000, 1)
train_dataset = Dataset(X_train, y_train)
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,batch_size=128,shuffle=True)

for x, y in train_loader:
    print(x.shape)
    print(y.shape)
    break


Instruction: pytorch tensor from pandas columns of vectors

Input:
I want to convert a panda's columns to a PyTorch tensor. Each cell of the column has a 300 dim NumPy vector (an embedding).
I have tried this:
torch.from_numpy(g_list[1]['sentence_vector'].to_numpy())

but it throws this error:

TypeError: can't convert np.ndarray of type numpy.object_. The only
supported types are: float64, float32, float16, complex64, complex128,
int64, int32, int16, int8, uint8, and bool.

Output:
If you have this dataframe which each column is a vector of 2 numbers:
import torch
import pandas as pd

df = pd.DataFrame({'a':    [[ 3,  29],[ 3,  29]],
                   'b': [[94, 170],[ 3,  29]],
                   'c': [[31, 115],[ 3,  29]]})

To convert this dataframe to a pytorch tensor, you only need to convert the values of dataframe to list and then a tensor:
t = torch.Tensor(list(df.values))

#output

tensor([[[  3.,  29.],
         [ 94., 170.],
         [ 31., 115.]],

        [[  3.,  29.],
         [  3.,  29.],
         [  3.,  29.]]])

The shape of t is [2,3,2] is 2 rows, 3 columns, 2 elements inside each list.


Instruction: Plot derivatives of sin(x) using pytorch

Input:
Am unsure why my code does not plot cos(x) (yes, am aware pytorch has cos(x) function)
import math
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
import random

x = torch.linspace(-math.pi, math.pi, 5000, requires_grad=True)
y = torch.sin(x)
y.backward(x)
x.grad == torch.cos(x) # assert x.grad same as cos(x)
plt.plot(x.detach().numpy(), y.detach().numpy(), label='sin(x)')
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label='cos(x)') # print derivative of sin(x)

Output:
You need to feed the upstream gradient (equals to all ones in your case) instead of x as input to y.backward().
Thus
import math
import torch
import matplotlib.pyplot as plt

x = torch.linspace(-math.pi, math.pi, 5000, requires_grad=True)
y = torch.sin(x)
y.backward(torch.ones_like(x))
plt.plot(x.detach().numpy(), y.detach().numpy(), label='sin(x)')
plt.plot(x.detach().numpy(), x.grad.detach().numpy(), label='cos(x)') # print derivative of sin(x)
plt.show()


Instruction: Prepare for Binary Masks used for the image segmentation

Input:
I am trying to prepare the masks for image segmentation with Pytorch. I have three questions about data preparation.

What is the appropriate data format to save the binary mask in general? PNG? JPEG?

Is the mask size needed to be set square such as (224x224), not a rectangle such as (224x448)?

Is the mask value fixed when the size is converted from rectangle to square?

For example, the original mask image size is (600x900), which is binary [0,1]. However, when I applied
import torchvision.transforms as transforms
transforms.Compose([
                        transforms.Resize((300, 300)),
                        transforms.ToTensor(),
                        ])

to the mask, the output had other values: 0.01, 0.0156, 0.22... except for 0 and 1, since the mask size was converted.
I applied the below code to convert the mask into the binary again if the value is less than 0.3, the value is 0, otherwise, 1.
def __getitem__(self, idx):
    img, mask = self.load_data(idx)
    if self.img_transforms is not None:
        img = self.img_transforms(img)
    if self.mask_transforms is not None:
        mask = self.mask_transforms(mask)
        mask = torch.where(mask<=0.3,0,1)
    return img, mask

but I wonder the process is a common approach and efficient.

Output:
PNG, because it is lossless by design.
It depends. More convenient is to use standard resolution, (224x224), I would start with that.
Use resize without interpolation transforms.Resize((300, 300), interpolation=InterpolationMode.NEAREST)


Instruction: Is there a way to create a train dataset for multiple pytorch files in a folder?

Input:
I have a folder with the name processed_data.
In this file I have multiple .pt files which are named data_0.pt, data_1.pt, data_2.pt, data_3.pt, data_4.pt, data_5.pt, ....., data_998.pt, data_999.pt, data_1000.pt, data_1001.pt.
All these .pt files are representing a graph which was created using pytorch-geometric.
My question is, how do I save load all these files to create my training dataset so that I can use them in DataLoader?

Output:
A torch DataLoader needs a Dataset object. When defining your Dataset class, you need to implement __init__, __len__, and __getitem__.
__init__ is easy, but also dependent on your exact use case/context. Assuming the simplest possible situation, I'd define init to take in the data folder and a file which contains names of the training set files (one per line). Then, I'd store each file name in a list as a member of the class. So we'd have:
def __init__(self, data_folder, data_list_filename):
    self.data_folder = data_folder
    with open(data_list_filename, 'r') as f:
        self.data_file_list = f.read().splitlines()

Ok, now we have two things stored in your Dataloader: 1) the data folder and 2) a list of data filenames. That makes __len__ especially easy:
def __len__(self):
    return len(self.data_file_list)

And lastly, we just need to deal with __get_item__:
def __getitem__(self, idx):
    filename = self.data_file_list[idx]
    data, label = extract_data_from_file(filename) # this is arbitrary because I don't know how you need to do this
    return data, label

Then put all of this together under a class:
class MyDataset(Dataset):
    def __init__(self, data_folder, data_list_filename):
        self.data_folder = data_folder
        self.data_file_list = open(data_list_filename, 'r').read().splitlines()

    def __len__(self):
        return len(self.data_file_list)

    def __getitem__(self, idx):
        filename = self.data_file_list[idx]
        data, label = extract_data_from_file(filename) # idk how you plan to do this
        return data, label

Obviously, your exact use will look different. But this should get you started.


Instruction: Is loss.backward() meant to be called on each sample or on each batch?

Input:
I have a training dataset which contains features of different sizes. I understand the implications of this in terms of network architecture and have designed my network accordingly to handle these heterogeneous shapes. When it comes to my training loop, though, I'm confused as to the order/placement of optimizer.zero_grad(), loss.backward(), and optimizer.step().
Because of the unequal feature sizes, I cannot do forward pass upon features of a batch at the same time. So, my training loop loops through samples of a batch manually, like this:
for epoch in range(NUM_EPOCHS):
    for bidx, batch in enumerate(train_loader):
        optimizer.zero_grad()
        batch_loss = 0
        for sample in batch:
            feature1 = sample['feature1']
            feature2 = sample['feature2']
            label1 = sample['label1']
            label2 = sample['label2']

            pred_l1, pred_l2 = model(feature1, feature2)

            sample_loss = compute_loss(label1, pred_l1)
            sample_loss += compute_loss(label2, pred_l2)
            sample_loss.backward() # CHOICE 1
            batch_loss += sample_loss.item()
        # batch_loss.backward() # CHOICE 2
        optimizer.step()

I'm wondering if it makes sense here that backward is called upon each sample_loss with the optimizer step called every BATCH_SIZE samples (CHOICE 1). The alternative, I think, would be to call backward upon batch_loss (CHOICE 2) and I'm not so sure which is the right choice.

Output:
Differentiation is a linear operation, so in theory it should not matter whether you first differentiate the different losses and add their derivatives or whether you first add the losses and then compute the derivative of their sum.
So for practical purposes both of them should lead to the same results (disregarding to the usual floating point issues).
You might get a slightly different memory requirements and computation speeds (I'd guess the second version might be slightly faster.), but that is hard to predict but something that you can easily find out by timing the two versions.