Instruction: Restoring mask to image

Input:
My PyTorch model outputs a segmented image with values (0,1,2) for each one of the three classes. During the preparation of the set, I mapped black to 0, red to 1 and white to 2. I have two questions:

How can I show what each class represents? for example take a look at the image:

I am currently using the following method to show each class:
     output = net(input)

     input = input.cpu().squeeze()
     input = transforms.ToPILImage()(input)

     probs = F.softmax(output, dim=1)
     probs = probs.squeeze(0)

     full_mask = probs.squeeze().cpu().numpy()

     fig, (ax0, ax1, ax2, ax3, ax4) = plt.subplots(1, 5, figsize=(20,10), sharey=True)

     ax0.set_title('Input Image')
     ax1.set_title('Background Class')
     ax2.set_title('Neuron Class')
     ax3.set_title('Dendrite Class')
     ax4.set_title('Predicted Mask')

     ax0.imshow(input)
     ax1.imshow(full_mask[0, :, :].squeeze())
     ax2.imshow(full_mask[1, :, :].squeeze())
     ax3.imshow(full_mask[2, :, :].squeeze())

     full_mask = np.argmax(full_mask, 0)
     img = mask_to_image(full_mask)

But there appears to be shared pixels between the classes, is there a better way to show this (I want the first image to only of the background class, the the second only of the neuron class and the third only of the dendrite class)?
2.My second question is about generating a black, red and white image from the mask, currently the mask is of shape (512,512) and has the following values:
[[0 0 0 ... 0 0 0]
 [0 0 0 ... 2 0 0]
 [0 0 0 ... 2 2 0]
 ...
 [2 1 2 ... 2 2 2]
 [2 1 2 ... 2 2 2]
 [0 2 0 ... 2 2 2]]

And the results look like this:

Since I am using this code to convert to image:
def mask_to_image(mask):
   return Image.fromarray((mask).astype(np.uint8))

Output:
But there appears to be shared pixels between the classes, is there a
better way to show this (I want the first image to only of the
background class, the the second only of the neuron class and the
third only of the dendrite class)?

Yes, you can take argmax along 0th dimension so the one with highest logit (unnormalized probability) will be 1, rest will be zero:
output = net(input)

binary_mask = torch.argmax(output, dim=0).cpu().numpy()
ax.set_title('Neuron Class')
ax.imshow(binary_mask == 0)

My second question is about generating a black, red and white image
from the mask, currently the mask is of shape (512,512) and has the
following values

You can spread [0, 1, 2] values into the zero-th axis making it channel-wise. Now [0, 0, 0] values across all channels for single pixel will be black, [255, 255, 255] would be white and [255, 0, 0] would be red (as PIL is RGB format):
import torch

tensor = torch.randint(high=3, size=(512, 512))

red = tensor == 0
white = tensor == 2

zero_channel = red & white

image = torch.stack([zero_channel, white, white]).int().numpy() * 255
Image.fromarray((image).astype(np.uint8))


Instruction: Pytorch why is .float() needed here for RuntimeError: expected scalar type Float but found Double

Input:
Simple question, i wanted to experiment with the simplest possible network, but i kept running into RuntimeError: expected scalar type Float but found Double unless i casted data into .float() (see below code with comment)
What i dont understand is, why is this casting needed? data is already a torch.float64 type. Whys the explicit re-casting in the output = model(data.float()) line needed?
Code
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.optim.lr_scheduler import StepLR
from sklearn.datasets import make_classification
from torch.utils.data import TensorDataset, DataLoader

# =============================================================================
# Simplest Example
# =============================================================================
X, y = make_classification()
X, y = torch.tensor(X), torch.tensor(y)
print("X Shape :{}".format(X.shape))
print("y Shape :{}".format(y.shape))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(X.shape[1], 128)
        self.fc2 = nn.Linear(128, 10)
        self.fc3 = nn.Linear(10, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

device = torch.device("cuda")
lr = 1
batch_size = 32
gamma = 0.7
epochs = 14
args = {'log_interval': 10, 'dry_run':False}

kwargs = {'batch_size': batch_size}
kwargs.update({'num_workers': 1,
               'pin_memory': True,
               'shuffle': True},
                 )

model = Net().to(device)
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = StepLR(optimizer, step_size=1, gamma=gamma)

my_dataset = TensorDataset(X,y) # create dataset
train_loader = DataLoader(my_dataset,**kwargs) #generate dataloader

cross_entropy_loss = torch.nn.CrossEntropyLoss()

for epoch in range(1, epochs + 1):
    ## Train step ##
    model.train()

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data.float()) #HERE: why is .float() needed here?
        loss = cross_entropy_loss(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % args['log_interval'] == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
            if args['dry_run']:
                break

    scheduler.step()

Output:
In PyTorch, 64-bit floating point corresponds to torch.float64 or torch.double.
While, 32-bit floating point corresponds to torch.float32 or torch.float.
Thus,

data is already a torch.float64 type

i.e. data is a 64 floating point type (torch.double).
By casting it using .float(), you convert it into 32-bit floating point.
a = torch.tensor([[1., -1.], [1., -1.]], dtype=torch.double)
print(a.dtype)
# torch.float64
print(a.float().dtype)
# torch.float32

Check different data types in PyTorch.


Instruction: Understanding the reason behind 'RuntimeError: leaf variable has been moved into the graph interior'

Input:
I am trying to understand pytorch and how autograd works in it. I tried creating a tensor by filling it with values from other tensors and then checking the gradients. However, I am running into RuntimeError: leaf variable has been moved into the graph interior if I don't set requires_grad equal to False.
code:
x = torch.ones(3,5,requires_grad=True)

y = x+2

z = y*y*3

out1 = z.mean()
out2 = 2*z.mean()

outi = torch.empty(2,requires_grad=True)

outi[0] = out1
outi[1] = out2

outi.backward(torch.tensor([0.,1.]))

output:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-1000fc52a64c> in <module>
     13 outi[1] = out2
     14
---> 15 outi.backward(torch.tensor([0.,1.]))

~/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    183                 products. Defaults to ``False``.
    184         """
--> 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    186
    187     def register_hook(self, hook):

~/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    123         retain_graph = create_graph
    124
--> 125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
    127         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: leaf variable has been moved into the graph interior

however, I can change the requires_grad to False and it will all work just fine
x = torch.ones(3,5,requires_grad=True)

y = x+2

z = y*y*3

out1 = z.mean()
out2 = 2*z.mean()

outi = torch.empty(2,requires_grad=False)

outi[0] = out1
outi[1] = out2

outi.backward(torch.tensor([0.,1.]))

output:
empty. it worked

Can somebody please help me understand what happened under the hood and, what changed with setting require_grad to True that resulted in this behavior? Thank you for reading

Output:
Intro
First, definition of what a leaf variable in PyTorch is, you can check official documentation for tensor.is_leaf (emphasis mine):

All Tensors that have requires_grad which is False will be leaf
Tensors by convention.
For Tensors that have requires_grad which is True, they will be
leaf Tensors if they were created by the user. This means that they
are not the result of an operation and so grad_fn is None.

So let's see how this looks for outi variable in original code. Immediately after creation, running this snippet:
outi = torch.empty(2, requires_grad=True)
print(outi.is_leaf, outi.grad_fn, outi.requires_grad)

gives:
True, None, True

as it was created by user and there is no previous operation creating it so it should be the second bolded case from the above citation.
Now this line:
outi[0] = out1
outi[1] = out2

Uses two nodes which are not leafs and are part of the graph which goes back to x (which is the only leaf in it). By doing this outi is also part of the original x graph and would have to be backpropagated through, yet you specified it as a leaf (more on that later), which cannot be backpropagated through (by the definition they either don't require gradient or are created by user). Version of outi as leaf was already put on graph, after above assignment, this snippet:
print(outi.is_leaf, outi.grad_fn, outi.requires_grad)

changes to:
False <CopySlices object at 0x7f2dfa83a3d0> True

Error
Now, I agree it's a pretty uninformative error given that changing requires_grad=False does not make it non-leaf variable (requires_grad=False is implicit):
outi = torch.empty(2)
print(outi.is_leaf, outi.grad_fn, outi.requires_grad)
# > True None False

But this tensor could be "upgraded" to non-leaf tensor if you use assignment as you did without breaking the expected behaviour.
Why? Because you implicitly (or explicitly in case of your code) said you don't need gradient for this variable and PyTorch retains gradient only for leaf variables (unless you specify .retain_grad for specific tensor) due to memory optimization. So the only change here would be it will no longer be a leaf, but this would not break promises as .grad would be None anyway.
If you were to have requires_grad=True as you originally did you could, reasonably, according to PyTorch semantics, think that this:
outi.grad

Will give you a tensor with gradient. But if this requires_grad=True tensor were to be changed to non-leaf tensor, then, by definition it wouldn't have this field (as non-leaf tensors have .grad=None).
To me it seems like a design decision on their part to avoid confusion with requires_grad=True and breaking expected user experience.
BTW. If they were to disallow leaf variables inside graph then operation which works fine now (requires_grad=False) should be disallowed as well. But as requires_grad=False is implicit and often used (creating tensors or something like you did) it seems not to be to much of a stretch to allow it. Disallowing it would be much more severe. On the other hand if you specify requires_grad=True it could be assumed you know better what you are doing and really need that gradient here.
BTW2. This explanation might be a stretch but hopefully will shed some light. I haven't found anything official regarding this error (admittedly though I didn't dig too deep).
Some resources here, here (this one is important, someone was asking for justification of some design decisions though didn't get one AFAIK).
Comments
Comment 1

I think the requires_grad is getting inherited from the slice and also
.grad is available.

Yes, it has requires_grad as True also as it's part of the graph now, BUT grad is not available as it is no longer a leaf. Printing outi.grad after backward gives you None and the following warning:

UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor
is being accessed. Its .grad attribute won't be populated during
autograd.backward(). If you indeed want the gradient for a non-leaf
Tensor, use .retain_grad() on the non-leaf Tensor. If you access the
non-leaf Tensor by mistake, make sure you access the leaf Tensor
instead. See github.com/pytorch/pytorch/pull/30531 for more
informations.

So the .grad attribute is None anyway as user would expect giving requires_grad=False as creation argument. User could expect gradient to be not None if he was to set requires_grad=True and that's when PyTorch raises the error, IMO due to possible inconsistency with user expectation in that case.
Comment 2
For example:
a = torch.ones(2,requires_grad=False)
b = 2*a
b.requires_grad=True
print(b.is_leaf) #True

I have changed your code a little to go through it step by step:
a = torch.ones(2, requires_grad=False)
print(a.is_leaf) # True

We should start with a here, a is a leaf according to docs as:

All Tensors that have requires_grad which is False will be leaf
Tensors by convention.

b = a * 2
print(b.is_leaf)

Now b is leaf as it does not require gradient (because a does not need a gradient it doesn't have to be backpropagated through this branch). Manipulating tensors with requires_grad=False creates tensors which do not require_grad otherwise it would be wasteful and non-sensical to turn it on.
b.requires_grad = True
print(b.is_leaf)

Now this one returns True as well. Once again, docs wording might not be the best here (as I've stated before), but (my additions in bold):

For Tensors that have requires_grad which is True (our case now)
they will be leaf Tensors if they were created by the user (debatable about creation here as you have modified the existing one, indeed). This means that they are not the result of an operation and so grad_fn is None (this one IMO clarifies the previous point)

About clarification- as this tensor is not a result of any mathematical operation, you simply said you want this b tensor to require_grad.
IMO it is a user created tensor as it was placed (created) on graph for the first time (before there was no need for that as it didn't require gradient).
And it does have it's requires_grad set to True, you did it explicitly here.
Comment 3 & 4

Everything with requires_grad=True is on the graph

Yes, but something with requires_grad=False can be on a graph as well if it is a leaf. Actually every PyTorch operation is created and added dynamically onto computational graph, here we use simplification: it's on graph if it takes part in backpropagation. For example neural network parameters are leafs yet they are on graph as the last part during backpropagation and they have their gradients (as they are optimized they need to be in graph in order to backprop through it).

Everything not on the graph is a leaf

Yes, essentially

everything on the graph that is not the product of operation on graph
tensors is a leaf

Yes, if you add some tensor to it (e.g. created by torch.randn or such) is a leaf

every leaf on the graph and non-leaf where I set retain_grad=True
manually will get .grad attribute populated.

Yes, it if it is part of backpropagation which is almost always the case in our "mental graph" case (I think at least). Unless it already has requires_grad=True, in this case it will be populated with gradient. Basically, except for creation, you shouldn't tinker with setting requires_grad=True as it is prone to fail (as you saw) and will definitely raise some eyebrows for other people reading your code.

every non-leaf on the graph has a grad_fn associated with it

Yes, that follows as some operation had to create it (and if it was created by some operation and this operation is differentiable, grad_fn is registered to be used during backward() call).


Instruction: Segmentation Fault when exporting to onnx a quantized Pytorch model

Input:
I am trying to export a model to the onnx format. The architecture is complicated so I won't share it here, but basically, I have the network weights in a .pth file. I'm able to load them, create the network and perform inference with it.
It's  important to note that I have adapted the code to be able to quantize the network. I have added quantize and dequantize operators as well as some torch.nn.quantized.FloatFunctional() operators.
However, whenever I try to export it with
torch.onnx.export(torch_model,               # model being run
                  input_example,             # model input
                  model_name,                # where to save the model
                  export_params=True,        # store the trained parameter
                  opset_version=11,          # the ONNX version to export
                  # the model to
                  do_constant_folding=True,  # whether to execute constant
                  # folding for optimization
                 )

I get Segmentation fault (core dumped)
I am working on Ubuntu 20, with the following packages installed :
torch==1.6.0
torchvision==0.7.0
onnx==1.7.0
onnxruntime==1.4.0

Note that the according to some prints I have left in the code, the inference part of the exporting completes. The segmentation fault happens afterward.
Does anyone see any reason why this may happen ?
[Edit] : I can export my network when it is not adapted for quantized operations. Therefore, the problem is not a broken installation but more a problem of some quantized operators for onnx saving.

Output: Well, it turns out that ONNX does not support quantized models (but does not warn you in anyway when running, it just throws out a segfault). It does not seem to be on the agenda yet, so a solution can be to use TensorRT.


Instruction: Trying to download the CIFAR10 dataset using 'torchvision.datasets' on my computer

Input:
i am trying to download the CIFAR10 dataset on my computer using the last two lines of the following code:
import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
[transforms.ToTensor(),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./Users/Sohrab/Downloads', train=True,
                                    download=True, transform=transform)

I execute it in Google Colab,but after that,when i look in my Downloads file on my computer,i do not see
the CIFAR10 dataset.
Is it actually stored on my computer or does Colab store it somewhere else(in a cloud architecture for instance)?
Thank you in advance.

Output: colab stores the data in running instance's disk. If it resets you have to redownload. You could also save it in your Google Drive but that's some trouble and IMO not worth it for CIFAR10.


Instruction: What is the upsampling method called 'area' used for?

Input:
The PyTorch function torch.nn.functional.interpolate contains several modes for upsampling, such as: nearest, linear, bilinear, bicubic, trilinear, area.
What is the area upsampling modes used for?

Output:
As jodag said, it is resizing using adaptive average pooling. While the answer at the link aims to explain what adaptive average pooling is, I find the explanation a bit vague.
TL;DR the area mode of torch.nn.functional.interpolate is probably one of the most intuitive ways to think of when one wants to downsample an image.
You can think of it as applying an averaging Low-Pass Filter(LPF) to the original image and then sampling. Applying an LPF before sampling is to prevent potential aliasing in the downsampled image. Aliasing can result in Moiré patterns in the downscaled image.
It is probably called "area" because it (roughly) preserves the area ratio between the input and output shapes when averaging the input pixels. More specifically, every pixel in the output image will be the average of a respective region in the input image where the 1/area of this region will be roughly the ratio between output image's area and input image's area.
Furthermore, the interpolate function with mode = 'area' calls the source function adaptie_avg_pool2d (implemented in C++) which assigns each pixel in the output tensor the average of all pixel intensities within a computed region of the input. That region is computed per pixel and can vary in size for different pixels. The way it is computed is by multiplying the output pixel's height and width by the ratio between the input and output (in that order) height and width (respectively) and then taking once the floor (for the region's starting index) and once the ceil (for the region's ending index) of the resulting value.
Here's an in-depth analysis of what happens in nn.AdaptiveAvgPool2d:
First of all, as stated there you can find the source code for adaptive average pooling (in C++) here: source
Taking a look at the function where the magic happens (or at least the magic on CPU for a single frame), static void adaptive_avg_pool2d_single_out_frame, we have 5 nested loops, running over channel dimension, then width, then height and within the body of the 3rd loop the magic happens:
First compute the region within the input image which is used to calculate the value of the current pixel (recall we had width and height loop to run over all pixels in the output).
How is this done?
Using a simple computation of start and end indices for height and width as follows: floor((input_height/output_height) * current_output_pixel_height) for the start and ceil((input_height/output_height) * (current_output_pixel_height+1)) and similarly for the width.
Then, all that is done is to simply average the intensities of all pixels in that region and current channel and place the result in the current output pixel.
I wrote a simple Python snippet that does the same thing, in the same fashion (loops, naive) and produces equivalent results. It takes tensor a and uses adaptive average pool to resize a to shape output_shape in 2 ways - once using the built-in nn.AdaptiveAvgPool2d and once with my translation into Python of the source function in C++: static void adaptive_avg_pool2d_single_out_frame. Built-in function's result is saved into b and my translation is saved into b_hat. You can see that the results are equivalent (you can further play with the spatial shapes and validate this):
import torch
from math import floor, ceil
from torch import nn
a = torch.randn(1, 3, 15, 17)
out_shape = (10, 11)
b = nn.AdaptiveAvgPool2d(out_shape)(a)

b_hat = torch.zeros(b.shape)
for d in range(a.shape[1]):
    for w in range(b_hat.shape[3]):
        for h in range(b_hat.shape[2]):
            startW = floor(w * a.shape[3] / out_shape[1])
            endW = ceil((w + 1) * a.shape[3] / out_shape[1])

            startH = floor(h * a.shape[2] / out_shape[0])
            endH = ceil((h + 1) * a.shape[2] / out_shape[0])

            b_hat[0, d, h, w] = torch.mean(a[0, d, startH: endH, startW: endW])

'''
Prints Mean Squared Error =  0 (or a very small number, due to precision error)
as both outputs are the same, proof of output equivalence:
'''
print(nn.MSELoss()(b_hat, b))


Instruction: PyTorch network with customized layer works fine on CPU but get cudaErrorIllegalAddress when moved to GPU

Input:
I'm trying to implement my own version of the Graph-Attention-network
The customized GAT layer is as following
class GATLayer(nn.Module):
    def __init__(self, input_dim: int, output_dim: int, adj: torch.tensor):
        super().__init__()
        self.W = nn.Parameter(torch.zeros(size=(output_dim, input_dim)))
        self.a = nn.Parameter(torch.zeros(size=(2 * output_dim,)))
        self.adj = adj
        self.n_points = adj.shape[0]
        #print(f"input dim:{input_dim}")

    def forward(self, h: torch.Tensor):
        B, T, N, F = h.size()
        hh = functional.linear(h, self.W)
        output = torch.zeros_like(hh)
        for i in range(self.n_points):
          #  print(i)
            hhj = hh[:, :, self.adj[i], :]
            hhi = torch.cat([hh[:, :, i:i + 1, :]] * hhj.size(2), 2)
            hhij = torch.cat([hhi, hhj], 3)
            e = torch.mm(hhij.reshape(B * T * hhj.size(2), -1), self.a.reshape(self.a.size(0), 1)).reshape(B, T, -1)
            alpha = functional.softmax(e, dim=2)
            output[:, :, i, :] = torch.sum(hhj * torch.cat([torch.unsqueeze(alpha, 3)] * hhj.size(3), 3), dim=2)
        return output

And the whole network is defined as:
class AQIP(nn.Module):
    def __init__(self, adj: torch.tensor, seq_len: int, with_aqi: bool = True):
        super().__init__()
        self.hid_size = 128
        self.seq_len = seq_len
        self.gat_layers = [
            GATLayer(input_dim=16 + int(with_aqi), output_dim=128, adj=adj),
            GATLayer(input_dim=128, output_dim=128, adj=adj),
        ]
        self.rnns = [
            nn.LSTM(input_size=128, hidden_size=128, num_layers=4, bias=True, batch_first=True),
        ]
        self.linear = nn.Linear(in_features=128 * 4, out_features=1, bias=True)

    def forward(self, x: torch.Tensor, site_idx: int):
        h = torch.zeros(size=(4, x.size(0), 128))
        c = torch.zeros(size=(4, x.size(0), 128))
        for gat in self.gat_layers:
            x = gat(x)
        for rnn in self.rnns:
            x[:, :, site_idx, :], (h, c) = rnn(x[:, :, site_idx, :], (h, c))
        h = h.permute(1, 0, 2)
        h = h.reshape(h.size(0), -1)
        return self.linear(h).squeeze()

When I independently test the customized GAT layer with the following code, it turned out that the GAT layer worked fine even on GPU
model = GATLayer(3, 1024, torch.tensor(np.array([[1, 0, 1], [0, 0, 1], [1, 0, 1]], dtype='bool')))
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = model.to(device)
print(model(torch.randn(5, 5, 3, 3)).shape)

Which outputs torch.Size([5, 5, 3, 1024])
When I test the whole network with CPU and the following code, it worked fine as well
#device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
adj = torch.tensor(np.array([[1, 0, 0], [0, 1, 1], [1, 1, 1]], dtype="bool"))
exp = torch.randn(3, 8, 3, 17)
gpus = [0]
model = AQIP(adj, seq_len=8)
#model = model.to(device, non_blocking=True)
print(model(exp, 1))

Which outputs tensor([-0.0320, -0.0320, -0.0320], grad_fn=<SqueezeBackward0>)
But as long as I try to move the model to GPU and de-comment the device and to lines, I get the following error with trace back to some Formatter class that is irrelevant to my code:
RuntimeError: copy_if failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

When using CUDA_LAUNCH_BLOCKING=1 to run the code, I get:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Which did not help me to locate error at all
I had also run official examples on the same machine and tried to move them to GPU, it turned out that they all worked fine. So I guess it's not about the CUDA and the Cudnn or the GPU driver version incompatible. But I can not locate the problem in my code as well. PLEASE HELP! I'll be much appreciated if you could save me out of this.

Output:
After numerous endeavors, I finally found the problem. It turned out that if you put layers into list like
self.gat_layers = [
           GATLayer(input_dim=16 + int(with_aqi), output_dim=128, adj=adj).cuda(),
           GATLayer(input_dim=128, output_dim=128, adj=adj).cuda(),
         ]

Then the PyTorch won't automatically identify those layers so that when .to(device) is called, their parameters won't be transferred to the GPU. So one of the solutions is to declare the layers one by one.
While the better solution is to use the nn.ModuleList to contain all the layers you want, so the code could be changed to
self.gat_layers = nn.ModuleList([
            GATLayer(input_dim=16 + int(with_aqi), output_dim=128, adj=adj).cuda(),
            GATLayer(input_dim=128, output_dim=128, adj=adj).cuda(),
        ])


Instruction: The bounding box's position and size is incorrect, how to improve it's accuracy?

Input:
I'm using detectron2 for solving a segmentation task,
I'm trying to classify an object into 4 classes,
so I have used COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml.
I have applied 4 kind of augmentation transforms and after training I get about 0.1
total loss.
But for some reason the accuracy of the bbox is not great on some images on the test set,
the bbox is drawn either larger or smaller or doesn't cover the whole object.
Moreover sometimes the predictor draws few bboxes, it assumes there are few different objects although there is only a single object.
Are there any suggestions how to improve it's accuracy?
Are there any good practice approaches how to resolve this issue?
Any suggestion or reference material will be helpful.

Output:
I would suggest the following:

Ensure that your training set has the object you want to detect in all sizes: in this way, the network learns that the size of the object can be different and less prone to overfitting (detector could assume your object should be only big for example).
Add data. Rather than applying all types of augmentations, try adding much more data. The phenomenon of detecting different objects although there is only one object leads me to believe that your network does not generalize well. Personally I would opt for at least 500 annotations per class.

The biggest step towards improvement will be achieved by means of (2).
Once you have a decent baseline, you could also experiment with augmentations.


Instruction: pytorch view tensor and reduce one dimension

Input:
So I have a 4d tensor with shape [4,1,128,678] and I would like to view/reshape it as [4,678,128].
I have to do this for multiple tensors where the last shape value 678 is not always know and could be different, so [4,1,128,575]should also go to [4,575,128]
Any idea on what is the optimal operation to transform the tensor? view/reshape? and how?
Thanks

Output:
You could also use (less to write and IMO cleaner):
# x.shape == (4, 1, 128, 678)
x.squeeze().permute(0, 2, 1)

If you were to use view you would lose dimension information (but maybe that is what you want), in this case it would be:
x.squeeze().view(4, -1, 128)

permute reorders tensors, while shape only gives a different view without restructuring underlying memory. You can see the difference between those two operations in this StackOverflow answer.


Instruction: Pytorch tensor get the index of the element with specific values?

Input:
I have two tensores, tensor a and tensor b.
I want to get all indexes of values in tensor b.
For example.
a = torch.Tensor([1,2,2,3,4,4,4,5])

b = torch.Tensor([1,2,4])

I want the index of 1, 2, 4 in tensor a. I can do this by the following code.
a = torch.Tensor([1,2,2,3,4,4,4,5])
b = torch.Tensor([1,2,4])
mask = torch.zeros(a.shape).type(torch.bool)
print(mask)
for e in b:
    mask = mask + (a == e)
    print(mask)

How can I do it without for?

Output:
Update:
As @zaydh kindly pointed out in the comments, since PyTorch 1.10, isin() and isinf()(and many other numpy equivalents) are available as well, thus you can simply do:
torch.isin(a, b)

which would give you :
Out[4]: tensor([ True,  True,  True, False,  True,  True,  True, False])

Old answer:
Is this what you want? :
np.in1d(a.numpy(), b.numpy())

will result in :
array([ True,  True,  True, False,  True,  True,  True, False])


Instruction: How to check if an object is a certain PyTorch optimizer?

Input:
I want to check the class of objects like this:
from torch.optim import Adam, SGD, AdamW

adam_range = (0.8, 1.0)
adamw_range = (0.6, 0.7)
sgd_range = (0.0, 0.5)

targets = []
for cfg in configs:
    if isinstance(cfg["optimizer"], Adam):
        sample = np.random.uniform(low=adam_range[0], high=adam_range[1], size=1)
    elif isinstance(cfg["optimizer"], AdamW):
        sample = np.random.uniform(low=adamw_range[0], high=adamw_range[1], size=1)
    elif isinstance(cfg["optimizer"], SGD):
        sample = np.random.uniform(low=sgd_range[0], high=sgd_range[1], size=1)

where configs is a list of dicts and cfg["optimizer"] are uninitialized references of from torch.optim import Adam, SGD, AdamW. However, none of the if/elif statements evaluate to True.
How do I properly check the type of these references? Is there a better way than checking cfg["optimizer"].__name__? The latter seems inelegant.
Thank you.

Output:
Your cfg["optimizer"] is not an instance of any optimizer, but the type itself.
therefore, you should test it like this:
for cfg in configs:
    if cfg["optimizer"] is Adam:
        sample = np.random.uniform(low=adam_range[0], high=adam_range[1], size=1)
    elif cfg["optimizer"] is AdamW:
        sample = np.random.uniform(low=adamw_range[0], high=adamw_range[1], size=1)
    elif cfg["optimizer"] is SGD:
        sample = np.random.uniform(low=sgd_range[0], high=sgd_range[1], size=1)

To emphasize the difference between an instance of Adam and type Adam, try the following:
opt = Adam(model.parameters(), lr=0.1)  # make an _instance_ of Adam

isinstance(opt, Adam)  # True  - opt is an instance of Adam optimizer

isinstance(Adam, Adam) # False - Adam is a type, not an instance of Adam

isinstnace(Adam, type) # True  - "Adam" is a type not an instance of Adam

type(opt) is Adam      # True  - the type of opt is Adam

type(opt) == Adam      # True

I find this page to summarize this difference quite well.


Instruction: PyTorch Conv2D returns non-zero output for an input tensor of zeros?

Input:
If you input an array consisting only of zeros to a Conv2D layer, the output should also consist only of zeros. In TensorFlow, this is the case. But, in PyTorch, it isn't. Here is some very simple sample Python code to demonstrate this. Why does PyTorch output non-zero numbers in this situation?
import torch
import numpy as np

image = np.zeros((3,3,3), dtype=np.float32)
batch = np.asarray([image])

a = torch.nn.Conv2d(3,3,1)
b = a(torch.tensor(batch).permute(0,3,1,2))

print(b.permute(0,2,3,1))

Output:
Unlike Tensorflow, PyTorch initializes the bias with non-zero values (see the source-code):
def reset_parameters(self) -> None:
    init.kaiming_uniform_(self.weight, a=math.sqrt(5))
    if self.bias is not None:
        fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
        bound = 1 / math.sqrt(fan_in)
        init.uniform_(self.bias, -bound, bound)


Instruction: How to define the loss function using the output of intermediate layers?

Input:
class Model(nn.Module):
def __init__(self):
    super(Model, self).__init__()
    self.encoder = nn.Linear(300, 100)
    self.dense1 = nn.Sequential(nn.Linear(100, 10),nn.ReLU())
    self.dense2 = nn.Sequential(nn.Linear(10, 5),nn.ReLU())
    self.dense3 = nn.Sequential(nn.Linear(5, 1))
def forward(self, x):
    x = self.encoder(x)
    x = self.dense1(x)
    x = self.dense2(x)
    x = self.dense3(x)
    return x

I am working on a regression problem, and I need to use the output of the dense2 layer to calculate the loss.
output of dense2 layer is 5 dimensional (5x1).
I am using PyTorch.
Dataset: Suppose i am using 300 features and i need to predict some score(a floating value).
Input: 300 Features
Output: Some Floating Value

Output:
In general, your nn.Module can return as many elements as you like. Moreover, you don't have to use them anywhere - there is no mechanism that checks that. Pytorch philosophy is to compute computational graph on-the-run.
class Model(nn.Module):
def __init__(self):
    super(Model, self).__init__()
    self.encoder = nn.Linear(300, 100)
    self.dense1 = nn.Sequential(nn.Linear(100, 10),nn.ReLU())
    self.dense2 = nn.Sequential(nn.Linear(10, 5),nn.ReLU())
    self.dense3 = nn.Sequential(nn.Linear(5, 1))

def forward(self, x):
    enc_output = self.encoder(x)
    dense1_output = self.dense1(enc_output)
    dense2_output = self.dense2(dense1_output)
    dense3_output = self.dense3(dense2_output)
    return dense3_output, dense2_output


Instruction: pytorch multilabel classification network not training

Input:
I'm trying a simple multi label classification example but the network does not seem to be training correctly as the loss is stagnant.
I've used multilabel_soft_margin_loss as the pytorch docs suggest, but there isn't much else to go on..can't find any proper examples in the docs.
Can anyone peer into this and point out whats wrong with it? Fully working example below (also question on prediction below)
Fully working example code
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR
from sklearn.datasets import make_multilabel_classification
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.metrics import accuracy_score
num_classes = 3
X, y = make_multilabel_classification(n_samples=1000,n_classes=num_classes)
X_tensor, y_tensor = torch.tensor(X), torch.tensor(y)
print("X Shape :{}".format(X_tensor.shape))
print("y Shape :{}".format(y_tensor.shape))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(X.shape[1], 300)
        self.fc2 = nn.Linear(300, 10)
        self.fc3 = nn.Linear(10, num_classes)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

device = torch.device("cpu")
lr = 1
batch_size = 128
gamma = 0.9
epochs = 100
args = {'log_interval': 10, 'dry_run':False}
kwargs = {'batch_size': batch_size}
kwargs.update({'num_workers': 1,
               'pin_memory': True,
               'shuffle': True},
                 )

model = Net().to(device)
optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=0.1)
scheduler = StepLR(optimizer, step_size=1, gamma=gamma)

# data loader
my_dataset = TensorDataset(X_tensor,y_tensor) # create tensor dataset

train_dataset, test_dataset, = train_test_split(
    my_dataset, test_size=0.2, random_state=42)

train_loader = DataLoader(train_dataset,**kwargs)
test_loader = DataLoader(test_dataset,**kwargs)

## Train step ##
for epoch in range(1, epochs + 1):
    model.train() # set model to train

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data.float())
        loss = F.multilabel_soft_margin_loss(output,target)
        loss.backward()
        optimizer.step()
        if batch_idx % args['log_interval'] == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
            if args['dry_run']:
                break

    scheduler.step()

Training Loss Progress
Train Epoch: 1 [0/800 (0%)] Loss: 0.694400
Train Epoch: 2 [0/800 (0%)] Loss: 0.697095
Train Epoch: 3 [0/800 (0%)] Loss: 0.705593
Train Epoch: 4 [0/800 (0%)] Loss: 0.651981
Train Epoch: 5 [0/800 (0%)] Loss: 0.704895
Train Epoch: 6 [0/800 (0%)] Loss: 0.650302
Train Epoch: 7 [0/800 (0%)] Loss: 0.658809
Train Epoch: 8 [0/800 (0%)] Loss: 0.904834
Train Epoch: 9 [0/800 (0%)] Loss: 0.655516
Train Epoch: 10 [0/800 (0%)]    Loss: 0.662808
Train Epoch: 11 [0/800 (0%)]    Loss: 0.664752
Train Epoch: 12 [0/800 (0%)]    Loss: 0.656390
Train Epoch: 13 [0/800 (0%)]    Loss: 0.664982
Train Epoch: 14 [0/800 (0%)]    Loss: 0.664430
Train Epoch: 15 [0/800 (0%)]    Loss: 0.664603 # stagnates

On top of that, how would I obtain predictions for this? It's not the same as taking the argmax anymore as its a multi label problem right? (Example output of the network below)
Output
tensor([[ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354]]

Thanks!

Output:
On top of that, how would I obtain predictions for this?

If it's a multilabel task and you are outputting logits (as you are) then simply do:
output = model(data.float())
labels = output > 0

point out whats wrong with it?

It is hard and opinionated, what I would do in order:

validate your data. Your neural network response is the same for every input (given your example output is real). Maybe you are passing the same single sample (though seems unlikely as it's sklearn created data)
start simple; no LR scheduler, no weight decay, simple neural network and optimizer only (Adam can stay). Use weight decay if your model is overfitting, it clearly isn't right now.
fix your learning rate; it is one of the most important hyperparameters. 1 is probably too high, start with something like 3e-4 or 1e-3.
try to overfit (loss ~0.0) on small amount of samples (say 32 samples). If you can't, your neural network probably doesn't have enough capacity or there is an error in your code (didn't spot it from quick glance, besides what I've mentioned above). You should verify input and output shapes are correct and returned values manually (it seems for each sample network returns the same logits?).
if you are sure there is no error increase network capacity. Add new hidden layer or two (there is only one) and overfit on single batch. If it's capable go with more data

I've used multilabel_soft_margin_loss as the pytorch docs suggest,

It is the same thing as using torch.nn.BCEWithLogitsLoss which I think is more common, but that's an addendum.


Instruction: Fixing error img should be PIL Image. Got

Input:
I tried to create Custom dataset but when show the some images it had the error. Here is my Dataset class and tranforms:
transforms = transforms.Compose([transforms.Resize(224,224)])

class MyDataset(Dataset):
    def __init__(self, path, label,  transform=None):
        self.path = glob.glob(os.path.join(path, '*.jpg'))
        self.transform = transform
        self.label = label

    def __getitem__(self, index):
        img = io.imread(self.path[index])
        img = torch.tensor(img)
        labels = torch.tensor(int(self.label))
        if self.transform:
          img = self.transform(img)
        return (img,labels)

    def __len__(self):
        return len(self.path)

And here error line:
images, labels = next(iter(train_loader))

Output:
transforms.Resize requires PIL.Image instance as input while your img is a torch.Tensor.
This will solve your issue (see comments in source code):
import torchvision
from PIL import Image

# In your transform you should cast PIL Image to tensor
# When no transforms on PIL Image are needed anymore
transforms = transforms.Compose([transforms.Resize(224, 224), transforms.ToTensor()])

class MyDataset(Dataset):
    def __init__(self, path, label, transform=None):
        self.path = glob.glob(os.path.join(path, "*.jpg"))
        self.transform = transform
        self.label = label

    def __getitem__(self, index):
        img = Image.open(self.path[index])
        labels = torch.tensor(int(self.label))
        if self.transform is not None:
            # Now you have PIL.Image instance for transforms
            img = self.transform(img)
        return (img, labels)

    def __len__(self):
        return len(self.path)


Instruction: Docker full with URL path

Input:
Good day all,
Anyone knows if it's possible to just pull a single container from github? I do have this link https://github.com/aws/sagemaker-pytorch-training-toolkit and I will like to pull the container in this link https://github.com/aws/sagemaker-pytorch-training-toolkit/tree/master/src/sagemaker_pytorch_container.
I did try using build docker build -t https://github.com/abc/sagemaker-pytorch-training-toolkit.git to just build an image of one file but there's an init.py file which i'm not sure if its necessary.
Thanks

Output:
You are on a wrong path.

Github does not store docker images, so there is no way you can pull it from there.
AWS Sagemaker provides pre-built images, you just need to select the one you want to use when creating an instance. see https://docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html
If you need a docker with pytorch, just run docker pull pytorch/pytorch


Instruction: How to use the func like torch.nn.functional.conv2d() in mxnet?

Input:
I want to do some convolution calculation with input data and a kernel.
In torch, I can write a func:
import torch
def torch_conv_func(x, num_groups):
    batch_size, num_channels, height, width = x.size()
    conv_kernel = torch.ones(num_channels, num_channels, 1, 1)

    return torch.nn.functional.conv2d(x, conv_kernel)

It works well and now I need rebuild in MXnet,so I write this:

from mxnet import nd
from mxnet.gluon import nn

def mxnet_conv_func(x, num_groups):
    batch_size, num_channels, height, width = x.shape
    conv_kernel = nd.ones((num_channels, num_channels, 1, 1))

    return nd.Convolution(x, conv_kernel)

And I got the error
mxnet.base.MXNetError: Required parameter kernel of Shape(tuple) is not presented, in operator Convolution(name="")

How to fix it?

Output:
You're missing some extra arguments to mxnet.nd.Convolution. You can do it like this:
from mxnet import nd

def mxnet_convolve(x):
    B, C, H, W = x.shape
    weight = nd.ones((C, C, 1, 1))
    return nd.Convolution(x, weight, no_bias=True, kernel=(1,1), num_filter=C)

x = nd.ones((16, 3, 32, 32))
mxnet_convolve(x)

Since you're not using a bias, you need to set no_bias to True. Also, mxnet requires you to specify the kernel dimensions with the kernel and num_filter argument.


Instruction: How to solve "RuntimeError: CUDA error: invalid device ordinal"?

Input:
I'm trying to run this code. I don't know what is wrong with it, but this code is not running. and I don't know how to solve this problem.
import cv2
from facial_emotion_recognition import EmotionRecognition

emotion_detector = EmotionRecognition(device='gpu', gpu_id=1)
camera = cv2.VideoCapture(0)

while True:
    image = camera.read()[1]
    image = emotion_detector.recognise_emotion(image, return_type='BGR')
    cv2.imshow('Camera', image)

    key = cv2.waitKey(1)
    if key == 27:
        break

camera.release()
cv2.destroyAllWindows()

but I'm getting this error:
Traceback (most recent call last):
  File "/home/fahim/Documents/Python_projects/Python tutorials/pantech AI Master/Computer_Vision/Day 8 Face emotion recognition/emotion.py", line 4, in <module>
    emotion_detector = EmotionRecognition(device='gpu', gpu_id=1)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/facial_emotion_recognition/facial_emotion_recognition.py", line 25, in __init__
    self.network = NetworkV2(in_c=1, nl=32, out_f=7).to(self.device)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 607, in to
    return self._apply(convert)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 376, in _apply
    param_applied = fn(param)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 605, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA error: invalid device ordinal

Process finished with exit code 1

This is my the configuration of my computer:
GPU: NVIDIA GeForce MX130
CPU: Intel i5-10210U (8) @ 4.200GHz
Help me to solve this please.

Output:
Try changing:
emotion_detector = EmotionRecognition(device='gpu', gpu_id=1)

To:
emotion_detector = EmotionRecognition(device='gpu', gpu_id=0)

gpu_id is only effective when more than one GPU is detected, you only seem to have one GPU, so it throws an error since you tell the function to get GPU 2 (since we count from 0).


Instruction: How can I get Class label from Mosaic augmentation in Object Detection Dataloader?

Input:
I'm trying to train an object detection model for a multi-class problem. In my training, I am using the Mosaic augmentation, Paper, for this task.
In my training mechanism, I'm a bit stuck to properly retrieving the class labels of each category, as the augmentation mechanism randomly picks the sub-portion of a sample. However, below is a result of a mosaic augmentation that we've achieved with a relevant bounding box until now.

Data Set
I've created a dummy data set. The df.head():

It has 4 class in total and df.object.value_counts():
human    23
car      13
cat       5
dog       3

Data Loader and Mosaic Augmentation
The data loader is defined as follows. However, the mosaic augmentation should be defined inside but for now, I'll create a separate code snippet for better demonstration:

IMG_SIZE = 2000

class DatasetRetriever(Dataset):

    def __init__(self, main_df, image_ids, transforms=None, test=False):
        super().__init__()

        self.image_ids = image_ids
        self.main_df = main_df
        self.transforms = transforms
        self.size_limit = 1
        self.test = test

    def __getitem__(self, index: int):
        image_id = self.image_ids[index]
        image, boxes, labels = self.load_mosaic_image_and_boxes(index)

        # labels = torch.tensor(labels, dtype=torch.int64) # for multi-class
        labels = torch.ones((boxes.shape[0],), dtype=torch.int64) # for single-class

        target = {}
        target['boxes'] = boxes
        target['cls'] = labels
        target['image_id'] = torch.tensor([index])

        if self.transforms:
            for i in range(10):
                sample = self.transforms(**{
                    'image' : image,
                    'bboxes': target['boxes'],
                    'labels': target['cls']
                })

                assert len(sample['bboxes']) == target['cls'].shape[0], 'not equal!'
                if len(sample['bboxes']) > 0:
                    # image
                    image = sample['image']

                    # box
                    target['boxes'] = torch.tensor(sample['bboxes'])
                    target['boxes'][:,[0,1,2,3]] = target['boxes'][:,[1,0,3,2]]

                    # label
                    target['cls'] = torch.stack(sample['labels'])
                    break

        return image, target

    def __len__(self) -> int:
        return self.image_ids.shape[0]

Basic Transform
def get_transforms():
    return A.Compose(
        [
            A.Resize(height=IMG_SIZE, width=IMG_SIZE, p=1.0),
            ToTensorV2(p=1.0),
        ],
        p=1.0,
        bbox_params=A.BboxParams(
            format='pascal_voc',
            min_area=0,
            min_visibility=0,
            label_fields=['labels']
        )
    )

Mosaic Augmentation
Note, It should be defined inside the data loader. The main issue is, in this augmentation, while iterating will all 4 samples to create such augmentation, image and bounding_box is rescaled as follows:
mosaic_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]

offset_x = x1a - x1b
offset_y = y1a - y1b
boxes[:, 0] += offset_x
boxes[:, 1] += offset_y
boxes[:, 2] += offset_x
boxes[:, 3] += offset_y

In this way, how would I select the relevant class labels for those selected bounding_box? Please, see the full code below:
def load_mosaic_image_and_boxes(self, index, s=3000,
                                    minfrac=0.25, maxfrac=0.75):
        self.mosaic_size = s
        xc, yc = np.random.randint(s * minfrac, s * maxfrac, (2,))

        # random other 3 sample
        indices = [index] + random.sample(range(len(self.image_ids)), 3)

        mosaic_image = np.zeros((s, s, 3), dtype=np.float32)
        final_boxes  = [] # box for the sub-region
        final_labels = [] # relevant class labels

        for i, index in enumerate(indices):
            image, boxes, labels = self.load_image_and_boxes(index)

            if i == 0:    # top left
                x1a, y1a, x2a, y2a =  0,  0, xc, yc
                x1b, y1b, x2b, y2b = s - xc, s - yc, s, s # from bottom right
            elif i == 1:  # top right
                x1a, y1a, x2a, y2a = xc, 0, s , yc
                x1b, y1b, x2b, y2b = 0, s - yc, s - xc, s # from bottom left
            elif i == 2:  # bottom left
                x1a, y1a, x2a, y2a = 0, yc, xc, s
                x1b, y1b, x2b, y2b = s - xc, 0, s, s-yc   # from top right
            elif i == 3:  # bottom right
                x1a, y1a, x2a, y2a = xc, yc,  s, s
                x1b, y1b, x2b, y2b = 0, 0, s-xc, s-yc    # from top left

            # calculate and apply box offsets due to replacement
            offset_x = x1a - x1b
            offset_y = y1a - y1b
            boxes[:, 0] += offset_x
            boxes[:, 1] += offset_y
            boxes[:, 2] += offset_x
            boxes[:, 3] += offset_y

            # cut image, save boxes
            mosaic_image[y1a:y2a, x1a:x2a] = image[y1b:y2b, x1b:x2b]
            final_boxes.append(boxes)

            '''
            ATTENTION:
            Need some mechanism to get relevant class labels
            '''
            final_labels.append(labels)

        # collect boxes
        final_boxes  = np.vstack(final_boxes)
        final_labels = np.hstack(final_labels)

        # clip boxes to the image area
        final_boxes[:, 0:] = np.clip(final_boxes[:, 0:], 0, s).astype(np.int32)
        w = (final_boxes[:,2] - final_boxes[:,0])
        h = (final_boxes[:,3] - final_boxes[:,1])

        # discard boxes where w or h <10
        final_boxes = final_boxes[(w>=self.size_limit) & (h>=self.size_limit)]

        return mosaic_image, final_boxes, final_labels

Output:
I parsed the bounding box and class label information at the same time.
Below is the output that we've achieved. To try it with your own data set,  for a starter.


Instruction: How can I make this PyTorch tensor (B, C, H, W) tiling & blending code simpler and more efficient?

Input:
So, I wrote the code below many months ago, and it's worked pretty well. Though I am struggling on how I can simplify it and make it more efficient.
The functions below split an image tensor (B, C, H, W) into equal sized tiles (B, C, H, W) and then you can do stuff individually to the tiles in order to save memory. Then when rebuilding the tensor from the tiles, it uses masks to ensure that the tiles are seamlessly blended back together. The 'special masks' in the masking function handle when tiles in the right most column or tiles in the bottom row can't use the same overlap as the other tiles. This means that the right edge tiles and the bottom tiles may sometimes have almost none of their content visible. This is done to ensure that the tiles are always the exact specified size, regardless of the original image/tensor's size (important for visualization/DeepDream, neural style transfer, etc...). The adjacent row/column to the edge row/column also has special masks as well for where they overlap with the edge row/column.
There are 8 possible masks for every tile, and 4 of those masks can be used at once. The 4 possible masks are left, right, top, and bottom, with a special version for each mask.
# Improved version of: https://github.com/ProGamerGov/neural-dream/blob/master/neural_dream/dream_tile.py
import torch

# Apply blend masks to tiles
def mask_tile(tile, overlap, side='bottom'):
    c, h, w = tile.size(1), tile.size(2), tile.size(3)
    top_overlap, bottom_overlap, right_overlap, left_overlap = overlap[0], overlap[1], overlap[2], overlap[3]

    base_mask = torch.ones_like(tile)

    if 'left' in side and 'left-special' not in side:
        lin_mask_left = torch.linspace(0,1,left_overlap, device=tile.device).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:,:left_overlap] = base_mask[:,:,:,:left_overlap] * lin_mask_left
    if 'right' in side and 'right-special' not in side:
        lin_mask_right = torch.linspace(1,0,right_overlap, device=tile.device).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:,w-right_overlap:] = base_mask[:,:,:,w-right_overlap:] * lin_mask_right
    if 'top' in side and 'top-special' not in side:
        lin_mask_top = torch.linspace(0,1,top_overlap, device=tile.device).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,:top_overlap,:] = base_mask[:,:,:top_overlap,:] * lin_mask_top
    if 'bottom' in side and 'bottom-special' not in side:
        lin_mask_bottom = torch.linspace(1,0,bottom_overlap, device=tile.device).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask[:,:,h-bottom_overlap:,:] = base_mask[:,:,h-bottom_overlap:,:] * lin_mask_bottom

    if 'left-special' in side:
        lin_mask_left = torch.linspace(0,1,left_overlap, device=tile.device)
        zeros_mask = torch.zeros(w-(left_overlap*2), device=tile.device)
        ones_mask = torch.ones(left_overlap, device=tile.device)
        lin_mask_left = torch.cat([zeros_mask, lin_mask_left, ones_mask], 0).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_left
    if 'right-special' in side:
        lin_mask_right = torch.linspace(1,0,right_overlap, device=tile.device)
        ones_mask = torch.ones(w-right_overlap, device=tile.device)
        lin_mask_right = torch.cat([ones_mask, lin_mask_right], 0).repeat(h,1).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_right
    if 'top-special' in side:
        lin_mask_top = torch.linspace(0,1,top_overlap, device=tile.device)
        zeros_mask = torch.zeros(h-(top_overlap*2), device=tile.device)
        ones_mask = torch.ones(top_overlap, device=tile.device)
        lin_mask_top = torch.cat([zeros_mask, lin_mask_top, ones_mask], 0).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_top
    if 'bottom-special' in side:
        lin_mask_bottom = torch.linspace(1,0,bottom_overlap, device=tile.device)
        ones_mask = torch.ones(h-bottom_overlap, device=tile.device)
        lin_mask_bottom = torch.cat([ones_mask, lin_mask_bottom], 0).repeat(w,1).rot90(3).repeat(c,1,1).unsqueeze(0)
        base_mask = base_mask * lin_mask_bottom

    # Apply mask to tile and return masked tile
    return tile * base_mask

def add_tiles(tiles, base_img, tile_coords, tile_size, overlap):

    # Check for any tiles that need different overlap values
    r, c = len(tile_coords[0]), len(tile_coords[1])
    f_ovlp = (tile_coords[0][r-1] - tile_coords[0][r-2], tile_coords[1][c-1] - tile_coords[1][c-2])

    h, w = tiles[0].size(2), tiles[0].size(3)
    t=0
    column, row, = 0, 0
    for y in tile_coords[0]:
        for x in tile_coords[1]:
            mask_sides=''
            c_overlap = overlap.copy()
            if row == 0:
                if row == len(tile_coords[0]) - 2:
                    mask_sides += 'bottom-special'
                    c_overlap[1] = f_ovlp[0] # Change bottom overlap
                else:
                    mask_sides += 'bottom'
            elif row > 0 and row < len(tile_coords[0]) -2:
                mask_sides += 'bottom,top'
            elif row == len(tile_coords[0]) - 2:
                if f_ovlp[0] > 0:
                    mask_sides += 'bottom-special,top'
                    c_overlap[1] = f_ovlp[0] # Change bottom overlap
                elif f_ovlp[0] <= 0:
                    mask_sides += 'bottom,top'
            elif row == len(tile_coords[0]) -1:
                if f_ovlp[0] > 0:
                    mask_sides += 'top-special'
                    c_overlap[0] = f_ovlp[0] # Change top overlap
                elif f_ovlp[0] <= 0:
                    mask_sides += 'top'

            if column == 0:
                if column == len(tile_coords[1]) -2:
                    mask_sides += ',right-special'
                    c_overlap[2] = f_ovlp[1] # Change right overlap
                else:
                    mask_sides += ',right'
            elif column > 0 and column < len(tile_coords[1]) -2:
                mask_sides += ',right,left'
            elif column == len(tile_coords[1]) -2:
                if f_ovlp[1] > 0:
                    mask_sides += ',right-special,left'
                    c_overlap[2] = f_ovlp[1] # Change right overlap
                elif f_ovlp[1] <= 0:
                    mask_sides += ',right,left'
            elif column == len(tile_coords[1]) -1:
                if f_ovlp[1] > 0:
                    mask_sides += ',left-special'
                    c_overlap[3] = f_ovlp[1] # Change left overlap
                elif f_ovlp[1] <= 0:
                    mask_sides += ',left'

            tile = mask_tile(tiles[t], c_overlap, side=mask_sides)
            base_img[:, :, y:y+tile_size[0], x:x+tile_size[1]] = base_img[:, :, y:y+tile_size[0], x:x+tile_size[1]] + tile
            t+=1
            column+=1
        row+=1
        column=0
    return base_img

# Calculate the coordinates for tiles
def get_tile_coords(d, tile_dim, overlap=0):
    move = int(tile_dim * (1-overlap))
    c, tile_start, coords = 1, 0, [0]
    while tile_start + tile_dim < d:
        tile_start = move * c
        if tile_start + tile_dim >= d:
            coords.append(d - tile_dim)
        else:
            coords.append(tile_start)
        c += 1
    return coords

# Calculates info required for tiling
def tile_setup(tile_size, overlap_percent, base_size):
    if type(tile_size) is not tuple and type(tile_size) is not list:
        tile_size = (tile_size, tile_size)
    if type(overlap_percent) is not tuple and type(overlap_percent) is not list:
        overlap_percent = (overlap_percent, overlap_percent)
    x_coords = get_tile_coords(base_size[1], tile_size[1], overlap_percent[1])
    y_coords = get_tile_coords(base_size[0], tile_size[0], overlap_percent[0])
    y_ovlp, x_ovlp = int(tile_size[0] * overlap_percent[0]), int(tile_size[1] * overlap_percent[1])
    return (y_coords, x_coords), tile_size, [y_ovlp, y_ovlp, x_ovlp, x_ovlp]

# Split tensor into tiles
def tile_image(img, tile_size, overlap_percent, info_only=False):
    tile_coords, tile_size, _ = tile_setup(tile_size, overlap_percent, (img.size(2), img.size(3)))

    # Cut out tiles
    tile_list = []
    for y in tile_coords[0]:
        for x in tile_coords[1]:
            tile = img[:, :, y:y + tile_size[0], x:x + tile_size[1]]
            tile_list.append(tile)
    return tile_list

# Put tiles back into the original tensor
def rebuild_image(tiles, image_size, tile_size, overlap_percent):
    base_img = torch.zeros(image_size, device=tiles[0].device)
    tile_coords, tile_size, overlap = tile_setup(tile_size, overlap_percent, (base_img.size(2), base_img.size(3)))
    return add_tiles(tiles, base_img, tile_coords, tile_size, overlap)

The above code can be tested with the code below:
import torchvision.transforms as transforms
from PIL import Image
import random

# Load image
def preprocess_simple(image_name, image_size):
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    image = Image.open(image_name).convert('RGB')
    return Loader(image).unsqueeze(0)

# Save image
def deprocess_simple(output_tensor, output_name):
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.squeeze(0))
    image.save(output_name)

test_input = preprocess_simple('tubingen.jpg', (1024,1024))
tile_size=260
overlap_percent=0.5

img_tiles = tile_image(test_input, tile_size=tile_size, overlap_percent=overlap_percent)

random.shuffle(img_tiles) # Comment this out to not randomize tile positions

output_tensor = rebuild_image(img_tiles, test_input.size(), tile_size=tile_size, overlap_percent=overlap_percent)
deprocess_simple(output_tensor, 'tiled_image.jpg')

I've included an example of what it does below (top is the original image, and the bottom is when I place the tiles back randomly to show off the blending system):

Output:
I was able to remove all the bugs and simplify the code here: https://github.com/ProGamerGov/dream-creator/blob/master/utils/tile_utils.py
The special masks were really only needed for 2 situations, and their were bugs in rebuild_tensor that I had to fix. Overlap percentages should be equal to or less than 50%.


Instruction: Normalizing images passed to torch.transforms.Compose function

Input:
How to find the values to pass to the transforms.Normalize function in PyTorch? Also, where in my code, should I exactly do the transforms.Normalize?
Since normalizing the dataset is a pretty well-known task, I was hoping there should be some sort of script for doing that automatically. At least I couldn't find it in PyTorch forum.
transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose([
                                           Rescale(256),
                                           RandomCrop(224),
                                           transforms.Normalize(mean = [ 0.485, 0.456, 0.406 ],
                                           std = [ 0.229, 0.224, 0.225 ]),
                                           ToTensor()
                                               ]))

for i in range(len(transformed_dataset)):
    sample = transformed_dataset[i]
    print(i, sample['image'].size(), sample['landmarks'].size())
    if i == 3:
       break

I know these current values don't pertain to my dataset and pertain to ImageNet but using them I actually get an error:
    TypeError                                 Traceback (most recent call last)
    <ipython-input-81-eb8dc46e0284> in <module>
         10
         11 for i in range(len(transformed_dataset)):
    ---> 12     sample = transformed_dataset[i]
         13
         14     print(i, sample['image'].size(), sample['landmarks'].size())

    <ipython-input-48-9d04158922fb> in __getitem__(self, idx)
         30
         31         if self.transform:
    ---> 32             sample = self.transform(sample)
         33
         34         return sample

    ~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
         59     def __call__(self, img):
         60         for t in self.transforms:
    ---> 61             img = t(img)
         62         return img
         63

    ~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
        210             Tensor: Normalized Tensor image.
        211         """
    --> 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
        213
        214     def __repr__(self):

    ~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
        278     """
        279     if not torch.is_tensor(tensor):
    --> 280         raise TypeError('tensor should be a torch tensor. Got {}.'.format(type(tensor)))
        281
        282     if tensor.ndimension() != 3:

    TypeError: tensor should be a torch tensor. Got <class 'dict'>.

So basically three questions:

How can I find the similar values as in ImageNet mean and std for my own custom dataset?
How to pass these values and where? I assume I should do it in transforms.Compose method but I might be wrong.
I assume I should apply Normalize to my entire dataset not just the training set, am I right?

Update:
Trying the provided solution here didn't work for me: https://discuss.pytorch.org/t/about-normalization-using-pre-trained-vgg16-networks/23560/6?u=mona_jalal
mean = 0.
std = 0.
nb_samples = 0.
for data in dataloader:
    print(type(data))
    batch_samples = data.size(0)

    data.shape(0)
    data = data.view(batch_samples, data.size(1), -1)
    mean += data.mean(2).sum(0)
    std += data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

error is:
<class 'dict'>

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-51-e8ba3c8718bb> in <module>
      5 for data in dataloader:
      6     print(type(data))
----> 7     batch_samples = data.size(0)
      8
      9     data.shape(0)

AttributeError: 'dict' object has no attribute 'size'

this is print(data) result:
{'image': tensor([[[[0.2961, 0.2941, 0.2941,  ..., 0.2460, 0.2456, 0.2431],
          [0.2953, 0.2977, 0.2980,  ..., 0.2442, 0.2431, 0.2431],
          [0.2941, 0.2941, 0.2980,  ..., 0.2471, 0.2471, 0.2448],
          ...,
          [0.3216, 0.3216, 0.3216,  ..., 0.2482, 0.2471, 0.2471],
          [0.3216, 0.3241, 0.3253,  ..., 0.2471, 0.2471, 0.2450],
          [0.3216, 0.3216, 0.3216,  ..., 0.2471, 0.2452, 0.2431]],

         [[0.2961, 0.2941, 0.2941,  ..., 0.2460, 0.2456, 0.2431],
          [0.2953, 0.2977, 0.2980,  ..., 0.2442, 0.2431, 0.2431],
          [0.2941, 0.2941, 0.2980,  ..., 0.2471, 0.2471, 0.2448],
          ...,
          [0.3216, 0.3216, 0.3216,  ..., 0.2482, 0.2471, 0.2471],
          [0.3216, 0.3241, 0.3253,  ..., 0.2471, 0.2471, 0.2450],
          [0.3216, 0.3216, 0.3216,  ..., 0.2471, 0.2452, 0.2431]],

         [[0.2961, 0.2941, 0.2941,  ..., 0.2460, 0.2456, 0.2431],
          [0.2953, 0.2977, 0.2980,  ..., 0.2442, 0.2431, 0.2431],
          [0.2941, 0.2941, 0.2980,  ..., 0.2471, 0.2471, 0.2448],
          ...,
          [0.3216, 0.3216, 0.3216,  ..., 0.2482, 0.2471, 0.2471],
          [0.3216, 0.3241, 0.3253,  ..., 0.2471, 0.2471, 0.2450],
          [0.3216, 0.3216, 0.3216,  ..., 0.2471, 0.2452, 0.2431]]],

        [[[0.3059, 0.3093, 0.3140,  ..., 0.3373, 0.3363, 0.3345],
          [0.3059, 0.3093, 0.3165,  ..., 0.3412, 0.3389, 0.3373],
          [0.3098, 0.3131, 0.3176,  ..., 0.3450, 0.3412, 0.3412],
          ...,
          [0.2931, 0.2966, 0.2931,  ..., 0.2549, 0.2539, 0.2510],
          [0.2902, 0.2902, 0.2902,  ..., 0.2510, 0.2510, 0.2502],
          [0.2864, 0.2900, 0.2863,  ..., 0.2510, 0.2510, 0.2510]],

         [[0.3059, 0.3093, 0.3140,  ..., 0.3373, 0.3363, 0.3345],
          [0.3059, 0.3093, 0.3165,  ..., 0.3412, 0.3389, 0.3373],
          [0.3098, 0.3131, 0.3176,  ..., 0.3450, 0.3412, 0.3412],
          ...,
          [0.2931, 0.2966, 0.2931,  ..., 0.2549, 0.2539, 0.2510],
          [0.2902, 0.2902, 0.2902,  ..., 0.2510, 0.2510, 0.2502],
          [0.2864, 0.2900, 0.2863,  ..., 0.2510, 0.2510, 0.2510]],

         [[0.3059, 0.3093, 0.3140,  ..., 0.3373, 0.3363, 0.3345],
          [0.3059, 0.3093, 0.3165,  ..., 0.3412, 0.3389, 0.3373],
          [0.3098, 0.3131, 0.3176,  ..., 0.3450, 0.3412, 0.3412],
          ...,
          [0.2931, 0.2966, 0.2931,  ..., 0.2549, 0.2539, 0.2510],
          [0.2902, 0.2902, 0.2902,  ..., 0.2510, 0.2510, 0.2502],
          [0.2864, 0.2900, 0.2863,  ..., 0.2510, 0.2510, 0.2510]]],

        [[[0.2979, 0.2980, 0.3015,  ..., 0.2825, 0.2784, 0.2784],
          [0.2980, 0.2980, 0.2980,  ..., 0.2830, 0.2764, 0.2795],
          [0.2980, 0.2980, 0.3012,  ..., 0.2827, 0.2814, 0.2797],
          ...,
          [0.3282, 0.3293, 0.3294,  ..., 0.2238, 0.2235, 0.2235],
          [0.3255, 0.3255, 0.3255,  ..., 0.2240, 0.2235, 0.2229],
          [0.3225, 0.3255, 0.3255,  ..., 0.2216, 0.2235, 0.2223]],

         [[0.2979, 0.2980, 0.3015,  ..., 0.2825, 0.2784, 0.2784],
          [0.2980, 0.2980, 0.2980,  ..., 0.2830, 0.2764, 0.2795],
          [0.2980, 0.2980, 0.3012,  ..., 0.2827, 0.2814, 0.2797],
          ...,
          [0.3282, 0.3293, 0.3294,  ..., 0.2238, 0.2235, 0.2235],
          [0.3255, 0.3255, 0.3255,  ..., 0.2240, 0.2235, 0.2229],
          [0.3225, 0.3255, 0.3255,  ..., 0.2216, 0.2235, 0.2223]],

         [[0.2979, 0.2980, 0.3015,  ..., 0.2825, 0.2784, 0.2784],
          [0.2980, 0.2980, 0.2980,  ..., 0.2830, 0.2764, 0.2795],
          [0.2980, 0.2980, 0.3012,  ..., 0.2827, 0.2814, 0.2797],
          ...,
          [0.3282, 0.3293, 0.3294,  ..., 0.2238, 0.2235, 0.2235],
          [0.3255, 0.3255, 0.3255,  ..., 0.2240, 0.2235, 0.2229],
          [0.3225, 0.3255, 0.3255,  ..., 0.2216, 0.2235, 0.2223]]]],
       dtype=torch.float64), 'landmarks': tensor([[[160.2964,  98.7339],
         [223.0788,  72.5067],
         [ 82.4163,  70.3733],
         [152.3213, 137.7867]],

        [[198.3194,  74.4341],
         [273.7188, 118.7733],
         [117.7113,  80.8000],
         [182.0750, 107.2533]],

        [[137.4789,  92.8523],
         [174.9463,  40.3467],
         [ 57.3013,  59.1200],
         [129.3375, 131.6533]]], dtype=torch.float64)}

dataloader = DataLoader(transformed_dataset, batch_size=3,
                        shuffle=True, num_workers=4)

and
transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose(
                                               [
                                               Rescale(256),
                                               RandomCrop(224),

                                               ToTensor()#,
                                               ##transforms.Normalize(mean = [ 0.485, 0.456, 0.406 ],
                                               ##         std = [ 0.229, 0.224, 0.225 ])
                                               ]
                                                                        )
                                           )

and
class MothLandmarksDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        img_name = os.path.join(self.root_dir, self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:]
        landmarks = np.array([landmarks])
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample

Output:
Source code errors

How to pass these values and where? I assume I should do it in
transforms.Compose method but I might be wrong.

In MothLandmarksDataset it is no wonder it is not working as you are trying to pass Dict (sample) to torchvision.transforms which require either torch.Tensor or PIL.Image as input. here to be exact:
sample = {'image': image, 'landmarks': landmarks}

if self.transform:
    sample = self.transform(sample)

You could pass sample["image"] into it although you shouldn't. Applying this operation only to sample["image"] would break its relation to landmarks. What you should be after is something like albumentations library (see here) which can transform image and landmarks in the same way to preserve their relations.
Also there is no Rescale transform in torchvision, maybe you meant Resize?
Mean and variance for normalization
Provided code is fine, but you have to unpack your data into torch.Tensor like this:
mean = 0.0
std = 0.0
nb_samples = 0.0
for data in dataloader:
    images, landmarks = data["image"], data["landmarks"]
    batch_samples = images.size(0)

    images_data = images.view(batch_samples, images.size(1), -1)
    mean += images_data.mean(2).sum(0)
    std += images_data.std(2).sum(0)
    nb_samples += batch_samples

mean /= nb_samples
std /= nb_samples

How to pass these values and where? I assume I should do it in
transforms.Compose method but I might be wrong.

Those values should be passed to torchvision.transforms.Normalize applied only to sample["images"], not to sample["landmarks"].

I assume I should apply Normalize to my entire dataset not just the
training set, am I right?

You should calculate normalization values across training dataset and apply those calculated values to validation and test as well.


Instruction: 5.51 GiB already allocated; 417.00 MiB free; 5.53 GiB reserved in total by PyTorch CUDA out of memory

Input:
I am not sure why running this cell throws CUDA out of memory error and how I could fix it? Everytime I have to do a kill -9 PID from the list of $ nvidi-smi jupyter notebook from which this cell is being run. And still after a fresh start I have the same problem.
#torch.autograd.set_detect_anomaly(True)
network = Network()
network.cuda()

criterion = nn.MSELoss()
optimizer = optim.Adam(network.parameters(), lr=0.0001)

loss_min = np.inf
num_epochs = 10

start_time = time.time()
for epoch in range(1,num_epochs+1):

    loss_train = 0
    loss_test = 0
    running_loss = 0

    network.train()
    print('size of train loader is: ', len(train_loader))

    for step in range(1,len(train_loader)+1):

        ##images, landmarks = next(iter(train_loader))
        ##print(type(images))

        batch = next(iter(train_loader))
        images, landmarks = batch['image'], batch['landmarks']
        images = images.permute(0,3,1,2)

        images = images.cuda()

        #RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[64, 600, 800, 3] to have 3 channels, but got 600 channels instead

        landmarks = landmarks.view(landmarks.size(0),-1).cuda()

        print('images shape: ', images.shape)
        print('landmarks shape: ', landmarks.shape)

        ##images = torchvision.transforms.Normalize(images)
        ##landmarks = torchvision.transforms.Normalize(landmarks)

        predictions = network(images)

        # clear all the gradients before calculating them
        optimizer.zero_grad()

        # find the loss for the current step
        loss_train_step = criterion(predictions.float(), landmarks.float())

        print("type(loss_train_step) is: ", type(loss_train_step))

        print("loss_train_step.dtype is: ",loss_train_step.dtype)

        ##loss_train_step = loss_train_step.to(torch.float32)

        # calculate the gradients
        loss_train_step.backward()

        # update the parameters
        optimizer.step()

        loss_train += loss_train_step.item()
        running_loss = loss_train/step

        print_overwrite(step, len(train_loader), running_loss, 'train')

    network.eval()
    with torch.no_grad():

        for step in range(1,len(test_loader)+1):

            batch = next(iter(train_loader))
            images, landmarks = batch['image'], batch['landmarks']
            images = images.cuda()
            landmarks = landmarks.view(landmarks.size(0),-1).cuda()

            predictions = network(images)

            # find the loss for the current step
            loss_test_step = criterion(predictions, landmarks)

            loss_test += loss_test_step.item()
            running_loss = loss_test/step

            print_overwrite(step, len(test_loader), running_loss, 'Testing')

    loss_train /= len(train_loader)
    loss_test /= len(test_loader)

    print('\n--------------------------------------------------')
    print('Epoch: {}  Train Loss: {:.4f}  Test Loss: {:.4f}'.format(epoch, loss_train, loss_test))
    print('--------------------------------------------------')

    if loss_test < loss_min:
        loss_min = loss_test
        torch.save(network.state_dict(), '../moth_landmarks.pth')
        print("\nMinimum Test Loss of {:.4f} at epoch {}/{}".format(loss_min, epoch, num_epochs))
        print('Model Saved\n')

print('Training Complete')
print("Total Elapsed Time : {} s".format(time.time()-start_time))

The complete log is:
size of train loader is:  12
images shape:  torch.Size([64, 3, 600, 800])
landmarks shape:  torch.Size([64, 8])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-efa8f1a4056e> in <module>
     44         ##landmarks = torchvision.transforms.Normalize(landmarks)
     45
---> 46         predictions = network(images)
     47
     48         # clear all the gradients before calculating them

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

<ipython-input-11-46116d2a7101> in forward(self, x)
     10     def forward(self, x):
     11         x = x.float()
---> 12         out = self.model(x)
     13         return out

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

~/anaconda3/lib/python3.7/site-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    206         x = self.maxpool(x)
    207
--> 208         x = self.layer1(x)
    209         x = self.layer2(x)
    210         x = self.layer3(x)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torchvision/models/resnet.py in forward(self, x)
     57         identity = x
     58
---> 59         out = self.conv1(x)
     60         out = self.bn1(out)
     61         out = self.relu(out)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    417
    418     def forward(self, input: Tensor) -> Tensor:
--> 419         return self._conv_forward(input, self.weight)
    420
    421 class Conv3d(_ConvNd):

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
    414                             _pair(0), self.dilation, self.groups)
    415         return F.conv2d(input, weight, self.bias, self.stride,
--> 416                         self.padding, self.dilation, self.groups)
    417
    418     def forward(self, input: Tensor) -> Tensor:

RuntimeError: CUDA out of memory. Tried to allocate 470.00 MiB (GPU 0; 7.80 GiB total capacity; 5.51 GiB already allocated; 417.00 MiB free; 5.53 GiB reserved in total by PyTorch)

Here's $ nvidia-smi output right after running this cell:
$ nvidia-smi
Tue Oct 13 23:14:01 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   47C    P8    13W /  N/A |   7609MiB /  7982MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1424      G   /usr/lib/xorg/Xorg                733MiB |
|    0   N/A  N/A      1767      G   /usr/bin/gnome-shell              426MiB |
|    0   N/A  N/A      6420      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      6949      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      8888      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     10610      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     14943      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A     16181      C   ...mona/anaconda3/bin/python     6429MiB |
+-----------------------------------------------------------------------------+

I have an NVIDIA GeForce RTX 2070 GPU.
I also switched these two lines and still no chance and still same error:
images = images.permute(0,3,1,2)
images = images.cuda()

I also checked the nvidia-smi exactly after running all the cells above this cell, and none of them are causing this CUDA out of memory error.

Output:
Changing the batch_size from 64 to 2 here solved the problem:
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=2, shuffle=True, num_workers=4)

*Thanks a lot to Sepehr Janghorbani for helping me fix this issue.


Instruction: How can I add zeroes to the end of a PyTorch tensor based on a modulo operation?

Input:
I have a tensor with size: torch.Size([1, 305760])
Since 305760 is not divisible by 400, I want to change the size to: [1, 306000] filling the remaining spaces with 0. How can I do this with PyTorch?

Output:
a = torch.randn(1,305760)
N = a.shape[1]
M = 400
b = torch.zeros([a.shape[0], (N // M + 1 ) * 400 - N])
a = torch.cat((a, b), 1)


Instruction: retain_graph problem with GRU in Pytorch 1.6

Input:
I am aware that, while employing loss.backward() we need to specify retain_graph=True if there are multiple networks and multiple loss functions to optimize each network separately. But even with (or without) specifying this parameter I am getting errors. Following is an MWE to reproduce the issue (on PyTorch 1.6).
import torch
from torch import nn
from torch import optim
torch.autograd.set_detect_anomaly(True)

class GRU1(nn.Module):
    def __init__(self):
        super(GRU1, self).__init__()
        self.brnn = nn.GRU(input_size=2, bidirectional=True, num_layers=1, hidden_size=100)

    def forward(self, x):
        return self.brnn(x)

class GRU2(nn.Module):
    def __init__(self):
        super(GRU2, self).__init__()
        self.brnn = nn.GRU(input_size=200, bidirectional=True, num_layers=1, hidden_size=1)

    def forward(self, x):
        return self.brnn(x)

gru1 = GRU1()
gru2 = GRU2()
gru1_opt = optim.Adam(gru1.parameters())
gru2_opt = optim.Adam(gru2.parameters())
criterion = nn.MSELoss()

for i in range(100):
    gru1_opt.zero_grad()
    gru2_opt.zero_grad()
    vector = torch.randn((15, 100, 2))
    gru1_output, _ = gru1(vector)  # (15, 100, 200)
    loss_gru1 = criterion(gru1_output, torch.randn((15, 100, 200)))
    loss_gru1.backward(retain_graph=True)
    gru1_opt.step()
    gru1_output, _ = gru1(vector)  # (15, 100, 200)
    gru2_output, _ = gru2(gru1_output)  # (15, 100, 2)
    loss_gru2 = criterion(gru2_output, torch.randn((15, 100, 2)))
    loss_gru2.backward(retain_graph=True)
    gru2_opt.step()
    print(f"GRU1 loss: {loss_gru1.item()}, GRU2 loss: {loss_gru2.item()}")

With retain_graph set to True I get the error
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [100, 300]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The error without the parameter is
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

which is expected.
Please point at what needs to be changed in the above code for it to begin training. Any help is appreciated.

Output:
In such a case, one can detach the computation graph to exclude the parameters that don't need to be optimized. In this case, the computation graph should be detached after the second forward pass with gru1 i.e.
....
gru1_opt.step()
gru1_output, _ = gru1(vector)
gru1_output = gru1_output.detach()
....

This way, you won't "try to backward through the graph a second time" as the error mentioned.


Instruction: How can I chunk a PyTorch tensor into a specified bucket size with overlap?

Input:
Specifically, I have a tensor of shape: torch.Size([1, 16])
I want to bucket this into 7 buckets (of 4 each). Example:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
should become:
[[1, 2, 3, 4],
 [3, 4, 5, 6],
 [5, 6, 7, 8],
 [7, 8, 9, 10],
 [9, 10, 11, 12],
 [11, 12, 13, 14],
 [13, 14, 15, 16],
]

How can I achieve this with PyTorch?

Output:
Looks like an unfold:
t.unfold(0,4,2)

Output:
tensor([[ 1.,  2.,  3.,  4.],
        [ 3.,  4.,  5.,  6.],
        [ 5.,  6.,  7.,  8.],
        [ 7.,  8.,  9., 10.],
        [ 9., 10., 11., 12.],
        [11., 12., 13., 14.],
        [13., 14., 15., 16.]])


Instruction: Pytorch NN error: Expected input batch_size (64) to match target batch_size (30)

Input:
I'm currently training a neural network to classify food groups of food images, resulting in 5 output classes. However, whenever I begin training the network, I get this error:
ValueError: Expected input batch_size (64) to match target batch_size (30).

Here's my neural network definition and training code. I'd really appriciate help, I'm relatively new to pytorch and can't figure out exactly what the problem is in my code. Thanks!
#Define the Network Architechture

model = nn.Sequential(nn.Linear(7500, 4950),
                      nn.ReLU(),
                      nn.Linear(4950, 1000),
                      nn.ReLU(),
                      nn.Linear(1000, 250),
                      nn.ReLU(),
                      nn.Linear(250, 5),
                      nn.LogSoftmax(dim = 1))

#Define loss
criterion = nn.NLLLoss()

#Initial forward pass
images, labels = next(iter(trainloader))
images = images.view(images.shape[0], -1)
print(images.shape)

logits = model(images)
print(logits.size)
loss = criterion(logits, labels)
print(loss)

#Define Optimizer
optimizer = optim.SGD(model.parameters(), lr = 0.01)

Training the Network:
epochs = 10

for e in range(epochs):
    running_loss = 0
    for image, labels in trainloader:
        #Flatten Images
        images = images.view(images.shape[0], -1)
        #Set gradients to 0
        optimizer.zero_grad()

        #Output
        output = model(images)
        loss = criterion(output, labels) #Where the error occurs
        loss.backward()

        #Gradient Descent Step
        optimizer.step()
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss/len(trainloader)}")

Output:
Not 100% sure but I think that the error is in this line:
nn.Linear(7500, 4950)

Put 1 instead of 7500 unless your absolutely sure that your input is 7500. Remember that the first value will always be your input size. By putting 1, you'll ensure that your model can work with any size of images.
By the way, PyTorch has a flatten function. Use nn.Flatten instead of using images.view() because you don't wanna make any shape errors and waste more time necessarily.
Another small error that you made was that you keep on using images and image as variables and parameters in the for loop. This is really bad practice because you're going to confuse someone whenever they read your code. Make sure that you don't reuse the same variables over and over again.
Also, could you give more info on your data? Like is it greyscale, image_size, etc.


Instruction: How to implement Batchnorm2d in Pytorch myself?

Input:
I'm trying to implement Batchnorm2d() layer with:
class BatchNorm2d(nn.Module):

    def __init__(self, num_features):
        super(BatchNorm2d, self).__init__()
        self.num_features = num_features
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.eps = 1e-5
        self.momentum = 0.1
        self.first_run = True

    def forward(self, input):
        # input: [batch_size, num_feature_map, height, width]
        device = input.device
        if self.training:
            mean = torch.mean(input, dim=0, keepdim=True).to(device)  # [1, num_feature, height, width]
            var = torch.var(input, dim=0, unbiased=False, keepdim=True).to(device)  # [1, num_feature, height, width]
            if self.first_run:
                self.weight = Parameter(torch.randn(input.shape, dtype=torch.float32, device=device), requires_grad=True)
                self.bias = Parameter(torch.randn(input.shape, dtype=torch.float32, device=device), requires_grad=True)
                self.register_buffer('running_mean', torch.zeros(input.shape).to(input.device))
                self.register_buffer('running_var', torch.ones(input.shape).to(input.device))
                self.first_run = False
            self.running_mean = (1 - self.momentum) * self.running_mean + self.momentum * mean
            self.running_var = (1 - self.momentum) * self.running_var + self.momentum * var
            bn_init = (input - mean) / torch.sqrt(var + self.eps)
        else:
            bn_init = (input - self.running_mean) / torch.sqrt(self.running_var + self.eps)
        return self.weight * bn_init + self.bias

But after training & testing I found that the results using my layer is incomparable with the results using nn.Batchnorm2d(). There must be something wrong with it, and I guess the problem relates to initializing parameters in forward()? I did that because I don't know how to know the shape of input in __init__(), maybe there is a better way.  I don't know how to fix it, please help. Thanks!!

Output:
Got answers from HERE!\
So the shape of weight(bias) is (1, num_features, 1, 1), not (1, num_features, width, height).


Instruction: Finding means and stds of a bunch of torch.Tensors (that are converted from ndarray images)

Input:
to_tensor = transforms.ToTensor()
img = to_tensor(train_dataset[0]['image'])
img

Converts my images values between 0 and 1 which is expected.  It also converts img which is an ndarray to a torch.Tensor.
Previously, without using to_tensor (which I need it now), the following code snippet worked (not sure if this is the best way to find means and stds of the train set, however now doesn't work. How can I make it work?
image_arr = []

for i in range(len(train_dataset)):
    image_arr.append(to_tensor(train_dataset[i]['image']))

print(np.mean(image_arr, axis=(0, 1, 2)))
print(np.std(image_arr, axis=(0, 1, 2)))

The error is:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-147-0e007c030629> in <module>
      4     image_arr.append(to_tensor(train_dataset[i]['image']))
      5
----> 6 print(np.mean(image_arr, axis=(0, 1, 2)))
      7 print(np.std(image_arr, axis=(0, 1, 2)))

<__array_function__ internals> in mean(*args, **kwargs)

~/anaconda3/lib/python3.7/site-packages/numpy/core/fromnumeric.py in mean(a, axis, dtype, out, keepdims)
   3333
   3334     return _methods._mean(a, axis=axis, dtype=dtype,
-> 3335                           out=out, **kwargs)
   3336
   3337

~/anaconda3/lib/python3.7/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims)
    133
    134 def _mean(a, axis=None, dtype=None, out=None, keepdims=False):
--> 135     arr = asanyarray(a)
    136
    137     is_float16_result = False

~/anaconda3/lib/python3.7/site-packages/numpy/core/_asarray.py in asanyarray(a, dtype, order)
    136
    137     """
--> 138     return array(a, dtype, copy=False, order=order, subok=True)
    139
    140

ValueError: only one element tensors can be converted to Python scalars

Output:
Here is a working example:
import torch
from torchvision import transforms
train_dataset = torch.rand(100, 32, 32, 3)
image_arr = []
to_tensor = transforms.ToTensor()

for i in range(len(train_dataset)):
    # to tensor will give you a tensor which is emulated here by reading the tensor at i
    image_arr.append(train_dataset[i])

print(torch.mean(torch.stack(image_arr, dim=0), dim=(0, 1, 2)))
print(torch.std(torch.stack(image_arr, dim=0), dim=(0, 1, 2)))

What did I do?
I used torch.stack to concatenate image array into a single torch tensor and use torch.mean and torch.std to compute stats. I would not recommend converting back to numpy for the purpose of evaluating stats as it can lead to unnecessary conversion from GPU to CPU.
More information on which dimension is the channel:
The above example assumes the last dimension is the channel and the image is 32x32x3 with 100 batch size. This is usually the case when the image is loaded using PIL (pillow) or numpy. Images are loaded as HWC (height width channel) in that case. This also seems to be the dimension in the question asked looking at the code example.
If the image tensor is CHW format, then you should use
print(torch.mean(torch.stack(image_arr, dim=0), dim=(0, 2, 3)))
print(torch.std(torch.stack(image_arr, dim=0), dim=(0, 2, 3)))

Torch tensors are usually CHW format as Conv layers expect CHW format. This is done automatically when the toTensor transform is applied to an image (PIL image). For complete rules see documentation of toTensor here.