Instruction: No such file or directory: 'docker': 'docker' when running sagemaker studio in local mode

Input:
I try to train a pytorch model on amazon sagemaker studio.

It's working when I use an EC2 for training with:

estimator = PyTorch(entry_point='train_script.py',
                role=role,
                sagemaker_session = sess,
                train_instance_count=1,
                train_instance_type='ml.c5.xlarge',
                framework_version='1.4.0',
                source_dir='.',
                git_config=git_config,
               )
estimator.fit({'stockdata': data_path})

and it's work on local mode in classic sagemaker notebook (non studio) with:

 estimator = PyTorch(entry_point='train_script.py',
                role=role,
                train_instance_count=1,
                train_instance_type='local',
                framework_version='1.4.0',
                source_dir='.',
                git_config=git_config,
               )
estimator.fit({'stockdata': data_path})

But when I use it the same code (with train_instance_type='local') on sagemaker studio it doesn't work and I have the following error: No such file or directory: 'docker': 'docker'

I tried to install docker with pip install but the docker command is not found if use it in terminal

Output:
This indicates that there is a problem finding the Docker service.
By default, the Docker is not installed in the SageMaker Studio  (confirming github ticket response).


Instruction: pytorch F.cross_entropy does not apply gradient to weights

Input:
I'm trying to train an MLP from scratch using torch tensors and some of the built-in loss functions. I have IRIS-data downloaded and stored in tensor (100, 4) and labels (100) (integers 0-2) in data_tr and targets_tr.  I have enabled gradients on the input data data_tr.requires_grad=True

I have a 2-layer MLP initialized like this:

W1 = torch.randn([4, 64], requires_grad=True)
W2 = torch.randn([64, 3], requires_grad=True)
b1 = torch.tensor([1.0], requires_grad=True)
b2 = torch.tensor([1.0], requires_grad=True

I understand that I should train like this:

for epoch in range(num_epochs):
    W1.grad = None
    W2.grad = None
    b1.grad = None
    b2.grad = None

    f = torch.relu(data_tr @ W1 + b1) @ W2 + b2
    error = torch.nn.functional.cross_entropy(f, targets_tr)
    error.backward()

    W1 = W1 - lr * W1.grad
    W2 = W2 - lr * W2.grad
    b1 = b1 - lr * b1.grad
    b2 = b2 - lr * b2.grad

Which makes a 2 layer MLP and cross_entropy applies softmax.
The problem now is that none of the weights or biases (W1, W2, b1, b2) has any gradients after the backward pass. So I get a TypeError: unsupported operand type(s) for *: 'float' and 'NoneType' on the first attempt to updating a weight.

Output:
If you want to update the weights without using an optimizer, you have to either use torch.no_grad() or update their data directly to ensure autograd is not tracking the update operations.

with torch.no_grad():
    W1 -= lr * W1.grad
    W2 -= lr * W2.grad

b1.data = b1 - lr * b1.grad
b2.data = b2 - lr * b2.grad

Note in the first case, if you do not subtract assign, the requires_grad will be set to False for the weights which again will result in None values for the gradients.

with torch.no_grad():
    W1 = W1 - lr * W1.grad
    W2 = W2 - lr * W2.grad
print(W1.requires_grad, W2.requires_grad)
>>> False False


Instruction: Pytorch: How to find accuracy for Multi Label Classification?

Input:
I am using vgg16, where number of classes is 3, and I can have multiple labels predicted for a data point.

vgg16 = models.vgg16(pretrained=True)
vgg16.classifier[6]= nn.Linear(4096, 3)

using loss function : nn.BCEWithLogitsLoss()

I am able to find find accuracy in case of a single label problem, as

 `images, labels = data
 images, labels = images.to(device), labels.to(device)
 labels = Encode(labels)
 outputs = vgg16(images)
 _, predicted = torch.max(outputs.data, 1)
 total += labels.size(0)
 correct += (predicted == labels).sum().item()
 acc = (100 * correct / total)`

How can I find accuracy for multi label classification?

Output:
From your question, vgg16 is returning raw logits. So here's what you can do:

labels = Encode(labels)  # torch.Size([N, C]) e.g. tensor([[1., 1., 1.]])
outputs = vgg16(images)  # torch.Size([N, C])
outputs = torch.sigmoid(outputs)  # torch.Size([N, C]) e.g. tensor([[0., 0.5, 0.]])
outputs[outputs >= 0.5] = 1
accuracy = (outputs == labels).sum()/(N*C)*100


Instruction: I am trying to build a neural network with one neuron using the pytorch library. It keeps giving me an error

Input:
I am trying to build a neural network with one neuron using the pytorch library. This is my code (the error is at the bottom)

import numpy as np
import random
import matplotlib.pyplot as plt

x_train = np.array([random.randint(1,1000) for x in range(1000)], dtype = np.float32)
y_train = np.array([int(num*3+1) for num in x_train], dtype = np.float32)

x_test = np.array([random.randint(1,1000) for x in range(1000)], dtype = np.float32)
y_test = np.array([int(num*3+1) for num in x_train], dtype = np.float32)

X_train = torch.from_numpy(x_train)
Y_train = torch.from_numpy(y_train)

plt.figure(figsize = (8,8))
plt.scatter(X_train, Y_train)
plt.show()

X_test = torch.from_numpy(x_test)
Y_test = torch.from_numpy(y_test)

input_size = 1
hidden_size = 1
output_size = 1
learning_rate = 0.1

w1 = torch.rand(input_size, hidden_size, requires_grad = True)
b1 = torch.rand(hidden_size, output_size, requires_grad = True)

for i in range(100):
    y_pred = X_train.mm(w1).clamp(min = 0).add(b1)
    loss = (Y_train-y_pred).pow(2).sum()

    loss.backward()

    with torch.no_grad():
        w1-=w1.grad*learning_rate
        b1 -= b1.grad*learning_rate
        w1.grad.zero_()
        b1.grad.zero_()

When I run this code, it gives me a runtime error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-84-5142b17ecfff> in <module>
     32
     33 for i in range(100):
---> 34     y_pred = X_train.mm(w1).clamp(min = 0).add(b1)
     35     loss = (Y_train-y_pred).pow(2).sum()
     36

RuntimeError: matrices expected, got 1D, 2D tensors at C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:192

What is wrong with that line of code and how to make it work as planned.

Output:
torch expects 2D input, so you need to add a new dimension to your inputs tensors.

X_train = torch.from_numpy(x_train[..., np.newaxis])
X_test = torch.from_numpy(x_test[..., np.newaxis])

As someone commented above, you can also use torch.unsqueeze:

for i in range(100):
    y_pred = torch.unsqueeze(X_train, 1).mm(w1).clamp(min = 0).add(b1)
    loss = (Y_train-y_pred).pow(2).sum()

Both do the same. The former applies it to the numpy arrays, and the latter does it on the torch tensors. Both will result in this shape, which is the correct format for torch:

Out[13]: torch.Size([1000, 1])


Instruction: Pytorch memory model: how does "torch.from_numpy()" work?

Input:
I'm trying to have an in-depth understanding of how torch.from_numpy() works.

import numpy as np
import torch

arr = np.zeros((3, 3), dtype=np.float32)
t = torch.from_numpy(arr)
print("arr: {0}\nt: {1}\n".format(arr, t))

arr[0,0]=1
print("arr: {0}\nt: {1}\n".format(arr, t))

print("id(arr): {0}\nid(t): {1}".format(id(arr), id(t)))

The output looks like this:

arr: [[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
t: tensor([[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])

arr: [[1. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
t: tensor([[1., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])

id(arr): 2360964353040
id(t): 2360964352984

This is part of the doc from torch.from_numpy():

  from_numpy(ndarray) -> Tensor

  Creates a :class:Tensor from a :class:numpy.ndarray.

  The returned tensor and :attr:ndarray share the same memory. Modifications to
  the tensor will be reflected in the :attr:ndarray and vice versa. The returned
  tensor is not resizable.

And this is taken from the doc of id():

  Return the identity of an object.

  This is guaranteed to be unique among simultaneously existing objects.
  (CPython uses the object's memory address.)

So here comes the question:
Since the ndarray arr and tensor t share the same memory, why do they have different memory addresses?

Any ideas/suggestions?

Output:
Yes, t and arr are different Python objects at different regions of memory (hence different id) but both point to the same memory address which contains the data (contiguous (usually) C array).

numpy operates on this region using C code binded to Python functions, same goes for torch (but using C++). id() doesn't know anything about the memory address of data itself, only of it's "wrappers".

EDIT: When you assign b = a (assuming a is np.array), both are references to the same Python wrapper (np.ndarray). In other words they are the same object with different name.

It's just how Python's assignment works, see documentation. All of the cases below would return True as well:

import torch
import numpy as np

tensor = torch.tensor([1,2,3])
tensor2 = tensor
id(tensor) == id(tensor2)

arr = np.array([1, 2, 3, 4, 5])
arr2 = arr
id(arr) == id(arr2)

some_str = "abba"
other_str = some_str
id(some_str) == id(other_str)

value = 0
value2 = value
id(value) == id(value2)

Now, when you use torch.from_numpy on np.ndarray you have two objects of different classes (torch.Tensor and original np.ndarray). As those are of different types they can't have the same id. One could see this case as analogous to the one below:

value = 3
string_value = str(3)

id(value) == id(string_value)

Here it's intuitive both string_value and value are two different objects at different memory locations.

EDIT 2:

All in all concepts of Python object and underlying C array have to be separated. id() doesn't know about C bindings (how could it?), but it knows about memory addresses of Python structures (torch.Tensor, np.ndarray).

In case of numpy and torch.tensor you can have following situations:

separate on Python level but using same memory region for array (torch.from_numpy)
separate on Python level and underlying memory region (one torch.tensor and another np.array). Could be created by from_numpy followed by clone() or a-like deep copy operation.
same on Python level and underlying memory region (e.g. two torch.tensor objects, one referencing another as provided above)


Instruction: code to print information about only one batch

Input:
I want to view information about my variables (type, min,max,std etc). While I write code below, it shows me information about all images in all batches. Please, tell me code, that will show me information about only one batch (something like "for batch [0], print(type(input))"

My code:

for t,(input, image_id) in enumerate(loader):
    print(image_id)

    input = input.cuda()
    with torch.no_grad():
        logit = net(input)
        probabilityy= torch.sigmoid(logit)

    probability = probabilityy.data.cpu().numpy()
    batch_size = len(image_id)
    for b in range(batch_size):
        p = probability[b]
        for c in range(4):
            predict, num = post_process(p[c], threshold, min_size)

            rle = run_length_encode(predict)

    print('input shape: ',input.shape)
    print('input ndim qty osi: ',input.ndim)
    print('input type: ',input.type)
    print('input mean std max min: ',input.mean(),input.std(),input.max(),input.min(),'\n')
    print('input: ',input[0,0:3,100:105,100:105])`

Output:
Usually, torch.DataLoader objects have a __getitem__ dunder method, so you should be able to use indexing to select one batch. But, I haven't seen how you load the data, so a universal solution would be to just break the loop:

# ...
print('input: ',input[0,0:3,100:105,100:105])
break

Like that, it won't move on to the next loop iteration and will effectively only execute this once, i.e., on the first batch.


Instruction: Wrap two tensors in pytorch to get size of new tensor as 2

Input:
I have two tensors say x and y:

x has shape: [21314, 3, 128, 128]
y has shape: [21314]

Can I get new tensor of shape : [ [21314, 3, 128, 128], [21314] ], basically of shape 2

Output:
I believe it's not possible, if you require to save it as a tensor object. Of course, you can use a list or a tuple for this case, but I guess that was not what you meant.

First, a tensor is simply a generalization of a matrix for n dimentions instead of two. But let's simplify this for a matrix for now, for example 4x3. The first dimention is of size 4, that means 4 entries. A second dimention of 3 means that each of the 4 first dimention entries will have exactly (and not less then) 3 entries. That is, you must have full list of 3 elements in each nested list. In this simple example, note that you cannot have a matrix like that one:

[[1,2,3]

 [1,2]

 [1]    ]

while this is a nested list it's not a matrix and also not a tensor of 2d. What i'm trying to say is that the shape your requested - [ [21314, 3, 128, 128], [21314] ] - is actually not a tensor.

But, you could have think of it as a tensor of size two, with data type of tensor in each entry (what you probably ment when asking the question). Though this is not possible since tensors in pytorch holds only numbers of types: float32, float64, float16, uint8, int8, int16, int32, int64, bool.

Nevertheless, in most cases you can achieve what you need with assigning two tensors to a list or tuple.


Instruction: PyTorch - unexpected shape of model parameters weights

Input:
I created a fully connected network in Pytorch with an input layer of shape (1,784) and a first hidden layer of shape (1,256).
To be short: nn.Linear(in_features=784, out_features=256, bias=True)

Method 1 : model.fc1.weight.data.shape gives me torch.Size([128, 256]), while

Method 2 : list(model.parameters())[0].shape gives me torch.Size([256, 784])

In fact, between an input layer of size 784 and a hidden layer of size 256, I was expecting a matrix of shape (784,256).
So, in the first case, I see the shape of the next hidden layer (128), which does not make sense for the weights between the input and first hidden layer, and, in the second case, it looks like Pytorch took the transform of the weight matrix.

I don't really understand how Pytorch shapes the different weight matrices, and how can I access individual weights after the training. Should I use method 1 or 2? When I display the corresponding tensors, the displays look totally similar, while the shapes are different.

Output:
In Pytorch, the weights of model parameters are transposed before applying the matmul operation on the input matrix. That's why the weight matrix dimensions are flipped, and is different from what you expect; i.e., instead of being [784, 256], you observe that it is [256, 784].

You can see the Pytorch source documentation for nn.Linear, where we have:

...

self.weight = Parameter(torch.Tensor(out_features, in_features))

...

def forward(self, input):
        return F.linear(input, self.weight, self.bias)

When looking at the implementation of F.linear, we see the corresponding line that multiplies the input matrix with the transpose of the weight matrix:

output = input.matmul(weight.t())


Instruction: Pytorch Matrix element wise multiplication

Input:
Let's say I have two tensors A of shape : [32 , 512] and B of shape : [32 , 512],
And I want to do element-wise multiplication between the vectors of the matrix to get a new matrix of shape : [32 , 1] (first row of A with first row of B, and second row of A with second row of B and so on.. ) , current methods I have tried simply mul the matrix values and not rows and gives a matrix of shape : [32 , 512].
Thanks!

Output:
I guess this is equivalent of what you're looking for, first a matrix multiplication, then sum over each row. [associative law]

import torch

a = torch.zeros(31,512)
b = torch.zeros(31,512)

c = torch.sum(torch.mm(a,torch.t(b)), dim=1).unsqueeze(-1)
print(c.shape)

Out:

torch.Size([31, 1])


Instruction: totally clear GPU memory

Input:
I can't seem to clear the GPU memory after sending a single variable to the GPU.

import torch
tm = torch.Tensor([1,2]).to("cuda")
!nvidia-smi

|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   37C    P2    52W / 250W |    730MiB / 10989MiB |      0%      Default

So I use 730MiB...
Now no matter what I try I can not make the 730MiB go to zero:

del tm
torch.cuda.empty_cache()
import sys;sys.modules[__name__].__dict__.clear()
%reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
!nvidia-smi
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   35C    P8     1W / 250W |    728MiB / 10989MiB |      0%      Default |

I would be happy to hear any suggestions,
Thanks

Output: Ok, it is not possible, this memory is torch drivers, and it can not be released. I've opened a ticket in pytorch GitHub - https://github.com/pytorch/pytorch/issues/37664


Instruction: GAN does not learn when using symmetric outputs from generator to disciminator

Input:
I'm currently trying to implement the paper Generative modeling for protein structures and I have succesfully been able to train a model following Pytorch's DCGAN Tutorial which has a similar model structure to the paper. The two implementations differ when it comes to output of the generator.
In the tutorial's model, the generator simply passes a normal output matrix to the discriminator. This works fine when I implement the paper's model (ommiting the symmetry and clamping) but the paper specifies:

During training, we enforce that G(z) be
positive by clamping output values above zero and symmetric

when I put this into my training loop I receive a loss graph that indicates that the generator isn't learning.
Here is my training loop:
# Training Loop
# Lists to keep track of progress
img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
# For each epoch
for epoch in range(num_epochs):
    # For each batch in the dataloader
    for i, data in enumerate(dataloader, 0):
        ############################
        # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))
        ###########################
        ## Train with all-real batch
        netD.zero_grad()
        # Format batch
        # Unsqueezed dim one to convert [128, 64, 64] to [128, 1, 64, 64] to conform to D architecture
        real_cpu = (data.unsqueeze(dim=1).type(torch.FloatTensor)).to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, device=device)
        # Forward pass real batch through D
        output = netD(real_cpu).view(-1)
        # Calculate loss on all-real batch
        errD_real = criterion(output, label)
        # Calculate gradients for D in backward pass
        errD_real.backward()
        D_x = output.mean().item()

        ## Train with all-fake batch
        # Generate batch of latent vectors
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        # Generate fake image batch with G
        fake = netG(noise)
        label.fill_(fake_label)
        # Make Symmetric
        sym_fake = (fake.detach().clamp(min=0) + fake.detach().clamp(min=0).permute(0, 1, 3, 2)) / 2
        # Classify all fake batch with D
        output = netD(sym_fake).view(-1)
        # Calculate D's loss on the all-fake batch
        errD_fake = criterion(output, label)
        # Calculate the gradients for this batch
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        # Add the gradients from the all-real and all-fake batches
        errD = errD_real + errD_fake
        # Update D
        optimizerD.step()
        #adjust_optim(optimizerD, iters)
        ############################
        # (2) Update G network: maximize log(D(G(z)))
        ###########################
        netG.zero_grad()
        label.fill_(real_label)  # fake labels are real for generator cost
        # Since we just updated D, perform another forward pass of all-fake batch through D
        output = netD(fake.detach()).view(-1)
        # Calculate G's loss based on this output
        errG = criterion(output, label)
        # Calculate gradients for G
        errG.backward()
        D_G_z2 = output.mean().item()
        # Update G
        optimizerG.step()
        adjust_optim(optimizerG, iters)

        # Output training stats
        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        # Save Losses for plotting later
        G_losses.append(errG.item())
        D_losses.append(errD.item())

        # Check how the generator is doing by saving G's output on fixed_noise
        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

Here is the training loss.

Here is my expected loss.

I make the output symmetric with the following line
sym_fake = (fake.detach().clamp(min=0) + fake.detach().clamp(min=0).permute(0, 1, 3, 2)) / 2

and I pass it to the discriminator on the lines that call sym_fake
Question
Is my implementation in pytorch wrong or is there something I'm missing? I don't understand why the paper makes the matrix symmetric and clamps if the network is capable of generating images without the need for symmetry and clamping.

Output: It could be because after the criterion for netG obtains an output that was detached from the parameters of netG thus the optimizer is not / can't be updating the parameters for netG.


Instruction: Pytorch merging list of tensors together

Input:
Let's say I have a list of tensors ([A , B , C ] where each tensor of is of shape [batch_size X 1024].
I want to merge all the tensors into a single tensor in the following way :
The first row in A is the first row in the new tensor, and the first row of B is the seocnd row in the new tensor, and the first row of C is the third row of the new tensor and so on and so forth.
So far I did it with for loops and this is not effictive at all.
Would love to hear about more efficent ways.

Thanks

Output:
Here's a minimal example that works:

import torch
a = torch.tensor([[1,1],[1,1]])
b = torch.tensor([[2,2],[2,2]])
c = torch.tensor([[3,3],[3,3]])
torch.stack([a,b,c],dim=0).view(6,2).t().contiguous().view(6,2)

The output is:

tensor([[1, 1],
        [2, 2],
        [3, 3],
        [1, 1],
        [2, 2],
        [3, 3]])

In your case, view(6,2) should change to batch_size*3, 1024.

Solution adapted from PyTorch forums
 where an example was shown with two tensors.


Instruction: How do i add a dimension to a numpy array and copy the dimension from another numpy array

Input:
I have a numpy array with the shape (128, 8)
I want to add an extra dimension so it has the shape (128, 168, 8)
And add the content of a 168 dimension from another array that has the shape (128, 168, 8).
I can always permute the positions of the dimensions if I can somehow add it.
Is this possible somehow? I have seen the append and concatenation methods but to no luck.

Output:
np.expand_dims(smaller_array, axis=1) + bigger_array

Is the correct solution, thanks!


Instruction: How can I monitor both training and eval loss when finetuning BERT on a GLUE task?

Input:
I am running https://github.com/huggingface/transformers/blob/master/examples/run_glue.py to perform finetuning on a binary classification task (CoLA).
I'd like to monitor both the training and evaluation losses to prevent overfitting.

Currently the library is at 2.8.0, and I did the install from source.

When I run the example with

python run_glue.py --model_name_or_path bert-base-uncased
                   --task_name CoLA
                   --do_train
                   --do_eval
                   --data_dir my_dir
                   --max_seq_length 128
                   --per_gpu_train_batch_size 8
                   --per_gpu_eval_batch_size 8
                   --learning_rate 2e-5
                   --num_train_epochs 3.0
                   --output_dir ./outputs
                   --logging_steps 5

In the stdout logs I see lines with one single value for the loss, such as

  {"learning_rate": 3.3333333333333333e-06, "loss": 0.47537623047828675,
  "step": 25}

By peeking in https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py I see that training and evaluation losses are computed there (looks to me that code was recently refactored).

I have thus replaced https://github.com/huggingface/transformers/blob/abb1fa3f374811ea09d0bc3440d820c50735008d/src/transformers/trainer.py#L314 with

 cr_loss = self._training_step(model, inputs, optimizer)
 tr_loss += cr_loss

and added after line https://github.com/huggingface/transformers/blob/abb1fa3f374811ea09d0bc3440d820c50735008d/src/transformers/trainer.py#L345

logs["training loss"] = cr_loss

with this I get:

0502 14:12:18.644119 23632 summary.py:47] Summary name training loss is illegal; using training_loss instead.
                          | 4/10  [00:02<00:04,  1.49it/s]
{"learning_rate":  3.3333333333333333e-06, "loss": 0.47537623047828675, "training loss": 0.5451719760894775, "step": 25}

Is this OK, or am I doing anything wrong here?

What's the best way to monitor in stdout both the averaged training and evaluation loss for a given logging interval during finetuning?

Output:
There's likely no change needed in the code if installing a more recent version (I tried 2.9.0 via pip): just fire the finetuning with the additional flag --evaluate_during_training and output will be OK
0506 12:11:30.021593 34540 trainer.py:551] ***** Running Evaluation *****
I0506 12:11:30.022596 34540 trainer.py:552]   Num examples = 140
I0506 12:11:30.023634 34540 trainer.py:553]   Batch size = 8 Evaluation:
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:19<00:00,  1.10s/it]
{"eval_mcc": 0.0, "eval_loss": 0.6600487811697854, "learning_rate": 3.3333333333333333e-06, "loss": 0.50044886469841, "step": 25}

beware that the example scripts change quite frequently, so flags to accomplish this may change names... see also here https://discuss.huggingface.co/t/how-to-monitor-both-train-and-validation-metrics-at-the-same-step/1301


Instruction: Reverse Image search (for image duplicates) on local computer

Input:
I have a bunch of poor quality photos that I extracted from a pdf. Somebody I know has the good quality photo's somewhere on her computer(Mac), but it's my understanding that it will be difficult to find them.
I would like to

loop through each poor quality photo
perform a reverse image search using each poor quality photo as the query image and using this persons computer as the database to search for the higher quality images
and create a copy of each high quality image in one destination folder.

Example pseudocode
for each image in poorQualityImages:
    search ./macComputer for a higherQualityImage of image
    copy higherQualityImage to ./higherQualityImages

I need to perform this action once.
I am looking for a tool, github repo or library which can perform this functionality more so than a deep understanding of content based image retrieval.

There's a post on reddit where someone was trying to do something similar
imgdupes is a program which seems like it almost achieves this, but I do not want to delete the duplicates, I want to copy the highest quality duplicate to a destination folder

Update
Emailed my previous image processing prof and he sent me this

Off the top of my head, nothing out of the box.
No guaranteed solution here, but you can narrow the search space.
You’d need a little program that outputs the MSE or SSIM similarity
index between two images, and then write another program or shell
script that scans the hard drive and computes the MSE between each
image on the hard drive and each query image, then check the images
with the top X percent similarity score.
Something like that.  Still not maybe guaranteed to find everything
you want.  And if the low quality images are of different pixel
dimensions than the high quality images, you’d have to do some image
scaling to get the similarity index.   If the poor quality images have
different aspect ratios, that’s even worse.
So I think it’s not hard but not trivial either.  The degree of
difficulty is partly dependent on the nature of the corruption in the
low quality images.

UPDATE
Github project I wrote which achieves what I want

Output:
  What you are looking for is called image hashing
  . In this answer you will find a basic explanation of the concept, as well as a go-to github repo for plug-and-play application.

Basic concept of Hashing

From the repo page: "We have developed a new image hash based on the Marr wavelet that computes a perceptual hash based on edge information with particular emphasis on corners. It has been shown that the human visual system makes special use of certain retinal cells to distinguish corner-like stimuli. It is the belief that this corner information can be used to distinguish digital images that motivates this approach. Basically, the edge information attained from the wavelet is compressed into a fixed length hash of 72 bytes. Binary quantization allows for relatively fast hamming distance computation between hashes. The following scatter plot shows the results on our standard corpus of images. The first plot shows the distances between each image and its attacked counterpart (e.g. the intra distances). The second plot shows the inter distances between altogether different images. While the hash is not designed to handle rotated images, notice how slight rotations still generally fall within a threshold range and thus can usually be matched as identical. However, the real advantage of this hash is for use with our mvp tree indexing structure. Since it is more descriptive than the dct hash (being 72 bytes in length vs. 8 bytes for the dct hash), there are much fewer false matches retrieved for image queries.
"

Another blogpost for an in-depth read, with an application example.

Available Code and Usage

A github repo can be found here. There are obviously more to be found.
After importing the package you can use it to generate and compare hashes:

>>> from PIL import Image
>>> import imagehash
>>> hash = imagehash.average_hash(Image.open('test.png'))
>>> print(hash)
d879f8f89b1bbf
>>> otherhash = imagehash.average_hash(Image.open('other.bmp'))
>>> print(otherhash)
ffff3720200ffff
>>> print(hash == otherhash)
False
>>> print(hash - otherhash)
36

The demo script find_similar_images also on the mentioned github, illustrates how to find similar images in a directory.


Instruction: Callbacks in Fastai

Input:
I am working on a deep learning project in Fastai and wish to use EarlyStoppingCallback with ReduceLROnPlateauCallback in it. Read callbacks.fastai but struggling to understand how to implement both and couldn't find any relevant example. Any help would be appreciated.

learn = cnn_learner(data, models.resnet50, metrics = [accuracy,quadratic_kappa])
learn.fit(50,2e-6)

Output:
The way I normally do it is this way....

First create the learner Object

    learn = Learner(data,
                    model,
                    loss_func=....,
                    opt_func=....,
                    metrics=..... )

    learn.unfreeze()

Then you call any callbacks on your fit_one_cycle

    learn.fit_one_cycle(16,
                        max_lr=1e-3,
                        div_factor=100,
                        pct_start=0.0,
                        callbacks = [SaveModelCallback(learn,
                                     name=f'model',monitor='kappa_score')])


Instruction: How to use two seperate dataloaders together?

Input:
I have two tensors of shape (16384,3,224,224) each. I need to multiply these two together. Obviously these two tensors are too big to fit in GPU ram. So I want to know, how should I go about this, divide them smaller batches using slicing or should I use two separate dataloaders?(I am confused, how to use two different dataloader together)
What would be the best way to do this?

Output:
I'm still not sure I totally understand the problem, but under the assumption that you have two big tensors t1 and t2 of shape [16384, 3, 224, 224] already loaded in RAM and want to perform element-wise multiplication then the easiest approach is

result = t1 * t2

Alternatively you could break these into smaller tensors and multiply them that way. There are lots of ways to do this.

One very PyTorch like way is to use a TensorDataset and operate on corresponding mini-batches of both tensors. If all you want to do is element-wise multiplication then the overhead of transferring tensors to and from the GPU is likely more expensive than the actual time saved during the computation. If you want to try it you can use something like this

import torch
from torch.utils import data

batch_size = 100
device = 'cuda:0'

dataset = data.TensorDataset(t1, t2)
dataloader = data.DataLoader(dataset, num_workers=1, batch_size=batch_size)

result = []
for d1, d2 in dataloader:
    d1, d2 = d1.to(device=device), d2.to(device=device)
    d12 = d1 * d2

    result.append(d12.cpu())

result = torch.cat(result, dim=0)

Or you could just do some slicing, which will probably be faster and more memory efficient since it avoids data copying on the CPU side.

import torch

batch_size = 100
device = 'cuda:0'

index = 0
result = []
while index < t1.shape[0]:
    d1 = t1[index:index + batch_size].to(device=device)
    d2 = t2[index:index + batch_size].to(device=device)
    d12 = d1 * d2

    result.append(d12.cpu())
    index += batch_size

result = torch.cat(result, dim=0)

Note that for both of these examples most of the time is spent copying data back to the CPU and concatenating the final results. Ideally, you would just do whatever you need to do with the d12 batch within the loop and avoid ever sending the final multiplied result back to the CPU.


Instruction: Issue training pytorch model on gpu

Input:
I am trying to implement the discriminator of a basic MNIST GAN in PyTorch. When I run training on the CPU, it works without any issue giving the desired output. However, when I run it on the GPU it shows a runtime error. I am pasting the code for my model as well as my training below along with the modifications I made to try and run training on the GPU.

dev = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

def preprocess(x):
    return x.view(-1,1,28,28).to(dev)

discriminator = nn.Sequential(
    Lambda(preprocess),
    nn.Conv2d(1,64,3,stride=2,padding=1),
    nn.LeakyReLU(negative_slope=0.2),
    nn.Dropout(0.4),
    nn.Conv2d(64,64,3,stride=2,padding=1),
    nn.LeakyReLU(negative_slope=0.2),
    nn.Dropout(0.4),
    Lambda(lambda x:x.view(x.size(0),-1)),
    nn.Linear(3136,1),
    nn.Sigmoid()
)
loss = nn.BCELoss()
opt = optim.Adam(discriminator.parameters(),lr = 0.002)

discriminator.to(dev)

def train_discriminator(model, dataset,opt, n_iter=100, n_batch=256):
    half_batch = int(n_batch / 2)
    for i in range(n_iter):
        X_real, y_real = generate_real_samples(dataset, half_batch)
        error = loss(model(X_real),y_real)
        error.backward()
        X_fake, y_fake = generate_fake_samples(half_batch)
        error = loss(model(X_fake),y_fake)
        error.backward()
        opt.step()

Now on running train_discriminator(discriminator,dataset,opt)
I get the following error which I cannot make sense of.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-ee20eb2a8e55> in <module>
----> 1 train_discriminator(discriminator,dataset,opt)

<ipython-input-13-9e6f9b4874c8> in train_discriminator(model, dataset, opt, n_iter, n_batch)
      3     for i in range(n_iter):
      4         X_real, y_real = generate_real_samples(dataset, half_batch)
----> 5         error = loss(model(X_real),y_real)
      6         error.backward()
      7         X_fake, y_fake = generate_fake_samples(half_batch)

~/environments/workspace/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/environments/workspace/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    496
    497     def forward(self, input, target):
--> 498         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    499
    500

~/environments/workspace/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2075
   2076     return torch._C._nn.binary_cross_entropy(
-> 2077         input, target, weight, reduction_enum)
   2078
   2079

RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'target' in call to _thnn_binary_cross_entropy_forward

Would really appreciate it if anyone could suggest any changes that are needed to be made that will solve this issue.

Output:
According to the error message, the ground truths are not in GPU:

  RuntimeError: Expected object of device type cuda but got device type cpu for argument #2 'target' in call to _thnn_binary_cross_entropy_forward


Instruction: Loss on dev set is always increasing unlike training set loss

Input:
I designed a network for a text classification problem. To do this, I'm using huggingface transformet's BERT model with a linear layer above that for fine-tuning. My problem is that the loss on the training set is decreasing which is fine, but when it comes to do the evaluation after each epoch on the development set, the loss is increasing with epochs. I'm posting my code to investigate if there's something wrong with it.

for epoch in range(1, args.epochs + 1):
    total_train_loss = 0
    trainer.set_train()

    for step, batch in enumerate(train_dataloader):
        loss = trainer.step(batch)
        total_train_loss += loss

    avg_train_loss = total_train_loss / len(train_dataloader)

    logger.info(('Training loss for epoch %d/%d: %4.2f') % (epoch, args.epochs, avg_train_loss))

    print("\n-------------------------------")
    logger.info('Start validation ...')
    trainer.set_eval()
    y_hat = list()
    y = list()
    total_dev_loss = 0
    for step, batch_val in enumerate(dev_dataloader):
        true_labels_ids, predicted_labels_ids, loss = trainer.validate(batch_val)
        total_dev_loss += loss
        y.extend(true_labels_ids)
        y_hat.extend(predicted_labels_ids)
    avg_dev_loss = total_dev_loss / len(dev_dataloader)
    print(("\n-Total dev loss: %4.2f on epoch %d/%d\n") % (avg_dev_loss, epoch, args.epochs))

print("Training terminated!")

Following is the trainer file, which I use for doing a forward pass on a given batch and then backpropagate accordingly.

class Trainer(object):
    def __init__(self, args, model, device, data_points, is_test=False, train_stats=None):
        self.args = args
        self.model = model
        self.device = device
        self.loss = nn.CrossEntropyLoss(reduction='none')

        if is_test:
            # Should load the model from checkpoint
            self.model.eval()
            self.model.load_state_dict(torch.load(args.saved_model))
            logger.info('Loaded saved model from %s' % args.saved_model)

        else:
            self.model.train()
            self.optim = AdamW(model.parameters(), lr=2e-5, eps=1e-8)
            total_steps = data_points * self.args.epochs
            self.scheduler = get_linear_schedule_with_warmup(self.optim, num_warmup_steps=0,
                                                             num_training_steps=total_steps)

    def step(self, batch):
        batch = tuple(t.to(self.device) for t in batch)
        batch_input_ids, batch_input_masks, batch_labels = batch
        self.model.zero_grad()
        outputs = self.model(batch_input_ids,
                             attention_mask=batch_input_masks,
                             labels=batch_labels)
        loss = self.loss(outputs, batch_labels)
        loss = loss.sum()
        (loss / loss.numel()).backward()
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)
        self.optim.step()
        self.scheduler.step()
        return loss

    def validate(self, batch):
        batch = tuple(t.to(self.device) for t in batch)
        batch_input_ids, batch_input_masks, batch_labels = batch
        with torch.no_grad():
            model_output = self.model(batch_input_ids,
                                 attention_mask=batch_input_masks,
                                 labels=batch_labels)

        predicted_label_ids = self._predict(model_output)
        label_ids = batch_labels.to('cpu').numpy()

        loss = self.loss(model_output, batch_labels)
        loss = loss.sum()

        return label_ids, predicted_label_ids, loss

    def _predict(self, logits):
        return np.argmax(logits.to('cpu').numpy(), axis=1)

Finally, the following is my model (i.e., Classifier) class:

import torch.nn as nn
from transformers import BertModel

class Classifier(nn.Module):

    def __init__(self, args, is_eval=False):
        super(Classifier, self).__init__()

        self.bert_model = BertModel.from_pretrained(
            args.init_checkpoint,
            output_attentions=False,
            output_hidden_states=True,
        )
        self.is_eval_mode = is_eval
        self.linear = nn.Linear(768, 2) # binary classification

    def switch_state(self):
        self.is_eval_mode = not self.is_eval_mode

    def forward(self, input_ids, attention_mask=None, labels=None):

        bert_outputs = self.bert_model(input_ids,
                                       token_type_ids=None,
                                       attention_mask=attention_mask)

        # Should give the logits to the the linear layer
        model_output = self.linear(bert_outputs[1])

        return model_output

For visualization the loss throughout the epochs:

Output:
When I've used Bert for text classification my model has generally behaved as you tell. In part this is expected because pre-trained models tend to require few epochs to fine-tune, actually if you check Bert's paper the number of epochs recommended for fine-tuning is between 2 and 4.
On the other hand, I've usually found the optimum at just 1 or 2 epochs, which coincides with your case also. My guess is: there is a trade-off when fine-tuning pre-trained models between fitting to your downstream task and forgetting the weights learned at pre-training. Depending on the data you have, the equilibrium point may happen sooner or later and overfitting starts after that. But this paragraph is speculation based on my experience.


Instruction: example of doing simple prediction with pytorch-lightning

Input:
I have an existing model where I load some pre-trained weights and then do prediction (one image at a time) in pytorch. I am trying to basically convert it to a pytorch lightning module and am confused about a few things.
So currently, my __init__ method for the model looks like this:
self._load_config_file(cfg_file)
# just creates the pytorch network
self.create_network()

self.load_weights(weights_file)

self.cuda(device=0)  # assumes GPU and uses one. This is probably suboptimal
self.eval()  # prediction mode

What I can gather from the lightning docs, I can pretty much do the same, except not to do the cuda() call. So something like:
self.create_network()

self.load_weights(weights_file)
self.freeze()  # prediction mode

So, my first question is whether this is the correct way to use lightning? How would lightning know if it needs to use the GPU? I am guessing this needs to be specified somewhere.
Now, for the prediction, I have the following setup:
def infer(frame):
    img = transform(frame)  # apply some transformation to the input
    img = torch.from_numpy(img).float().unsqueeze(0).cuda(device=0)
    with torch.no_grad():
        output = self.__call__(Variable(img)).data.cpu().numpy()
    return output

This is the bit that has me confused. Which functions do I need to override to make a lightning compatible prediction?
Also, at the moment, the input comes as a numpy array. Is that something that would be possible from the lightning module or do things always have to use some sort of a dataloader?
At some point, I want to extend this model implementation to do training as well, so want to make sure I do it right but while most examples focus on training models, a simple example of just doing prediction at production time on a single image/data point might be useful.
I am using 0.7.5 with pytorch 1.4.0 on GPU with cuda 10.1

Output:
LightningModule is a subclass of torch.nn.Module so the same model class will work for both inference and training. For that reason, you should probably call the cuda() and eval() methods outside of __init__.
Since it's just a nn.Module under the hood, once you've loaded your weights you don't need to override any methods to perform inference, simply call the model instance. Here's a toy example you can use:
import torchvision.models as models
from pytorch_lightning.core import LightningModule

class MyModel(LightningModule):
    def __init__(self):
        super().__init__()
        self.resnet = models.resnet18(pretrained=True, progress=False)

    def forward(self, x):
        return self.resnet(x)

model = MyModel().eval().cuda(device=0)

And then to actually run inference you don't need a method, just do something like:
for frame in video:
    img = transform(frame)
    img = torch.from_numpy(img).float().unsqueeze(0).cuda(0)
    output = model(img).data.cpu().numpy()
    # Do something with the output

The main benefit of PyTorchLighting is that you can also use the same class for training by implementing training_step(), configure_optimizers() and train_dataloader() on that class. You can find a simple example of that in the PyTorchLightning docs.


Instruction: FastAI throwing a runtime error when using custom train & test sets

Input:
I’m working on the Food-101 dataset and as you may know, the dataset comes with both train and test parts. Because the dataset could no longer be found on the ETH Zurich link, I had to divide them into partitions < 1GB each and clone them into Colab and reassemble. Its very tedious work but I got it working. I will omit the Python code but the file structure looks like this:

Food-101
      images
            train
               ...75750 train images
            test
               ...25250 test images
      meta
            classes.txt
            labes.txt
            test.json
            test.txt
            train.json
            train.txt
      README.txt
      license_agreement.txt

The following code is what’s throwing the runtime error

train_image_path = Path('images/train/')
test_image_path = Path('images/test/')
path = Path('../Food-101')

food_names = get_image_files(train_image_path)

file_parse = r'/([^/]+)_\d+\.(png|jpg|jpeg)'

data = ImageDataBunch.from_folder(train_image_path, test_image_path, valid_pct=0.2, ds_tfms=get_transforms(), size=224)
data.normalize(imagenet_stats)

My guess is that  ImageDataBunch.from_folder()  is what’s throwing the error but I don’t know why its getting caught up on the data types as (I don’t think) I’m supplying it with any data that has a specific type.

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:2854: UserWarning: The default behavior for interpolate/upsample with float scale_factor will change in 1.6.0 to align with other frameworks/libraries, and use scale_factor directly, instead of relying on the computed output size. If you wish to keep the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor will change "
You can deactivate this warning by passing `no_check=True`.
/usr/local/lib/python3.6/dist-packages/fastai/basic_data.py:262: UserWarning: There seems to be something wrong with your dataset, for example, in the first batch can't access these elements in self.train_ds: 9600,37233,16116,38249,1826...
  warn(warn_msg)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/IPython/core/formatters.py in __call__(self, obj)
    697                 type_pprinters=self.type_printers,
    698                 deferred_pprinters=self.deferred_printers)
--> 699             printer.pretty(obj)
    700             printer.flush()
    701             return stream.getvalue()

11 frames
/usr/local/lib/python3.6/dist-packages/fastai/vision/image.py in affine(self, func, *args, **kwargs)
    181         "Equivalent to `image.affine_mat = image.affine_mat @ func()`."
    182         m = tensor(func(*args, **kwargs)).to(self.device)
--> 183         self.affine_mat = self.affine_mat @ m
    184         return self
    185

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #3 'mat2' in call to _th_addmm_out

Output:
I had also faced the same error and used no_check=True in your ImageDataBunch arguments.
Try using this before you create ImageDataBunch
import warnings
warnings.filterwarnings("ignore", category=UserWarning, module="torch.nn.functional")

Make sure that you downgrade your torch version to 1.0.0,


Instruction: How is log_softmax() implemented to compute its value (and gradient) with better speed and numerical stability?

Input:
Both MXNet and PyTorch provide special implementation for computing log(softmax()), which is faster and numerically more stable. However, I cannot find the actual Python implementation for this function, log_softmax(), in either package.

Can anyone explain how this is implemented, or better, point me to the relevant source code?

Output:
The numerical error:

>>> x = np.array([1, -10, 1000])
>>> np.exp(x) / np.exp(x).sum()
RuntimeWarning: overflow encountered in exp
RuntimeWarning: invalid value encountered in true_divide
Out[4]: array([ 0.,  0., nan])

There are 2 methods to avoid the numerical error while compute the softmax:

Exp Normalization:

def exp_normalize(x):
    b = x.max()
    y = np.exp(x - b)
    return y / y.sum()

>>> exp_normalize(x)
array([0., 0., 1.])

Log Sum Exp

def log_softmax(x):
    c = x.max()
    logsumexp = np.log(np.exp(x - c).sum())
    return x - c - logsumexp

Please note that, a reasonable choice for both b, c in above formula is max(x). With this choice, overflow due to exp is impossible. The largest number exponentiated after shifting is 0.


Instruction: How to use pytorch's grid_sample()?

Input:
I am having some trouble getting torch.nn.functional working as I would like, illustrated by the below example:

import torch
import torch.nn.functional as F
import numpy as np

sz = 5
input_arr = torch.from_numpy(np.arange(sz*sz).reshape(1,1,sz,sz)).float()
indices = torch.from_numpy(np.array([-1,-1, -0.5,-0.5, 0,0, 0.5,0.5, 1,1]).reshape(1, 1, 5, 2)).float()

out = F.grid_sample(input_arr, indices)
print(input_arr)
print(out)

Since the indices are just the diagonals of the input, I'd expect to get something like tensor([[[[0., 6., 12., 18., 24.]]]]) (since (-1,-1) should give the top left and (1,1) should give the bottom right, according to the docs). However, I am getting this as output to the console:

tensor([[[[ 0.,  1.,  2.,  3.,  4.],
          [ 5.,  6.,  7.,  8.,  9.],
          [10., 11., 12., 13., 14.],
          [15., 16., 17., 18., 19.],
          [20., 21., 22., 23., 24.]]]])
tensor([[[[ 0.0000,  4.5000, 12.0000, 19.5000,  6.0000]]]])

What am I doing wrong? Many thanks!

Output:
Have you tried passing the argument align_corners = True? If you read the documentation it states that :

WARNING
When align_corners = True, the grid positions depend on the pixel size
relative to the input image size, and so the locations sampled by
grid_sample() will differ for the same input given at different
resolutions (that is, after being upsampled or downsampled). The
default behavior up to version 1.2.0 was align_corners = True. Since
then, the default behavior has been changed to align_corners = False,
in order to bring it in line with the default for interpolate().

And to double check, I ran the code with and without align_corners = True, to get both the correct output you required and the incorrect output you described.
# align_corners = False
out = F.grid_sample(input_arr, indices, align_corners = False)
print(out) # tensor([[[[ 0.0000,  4.5000, 12.0000, 19.5000,  6.0000]]]])

And
# align_corners = True
out = F.grid_sample(input_arr, indices, align_corners = True)
print(out) # tensor([[[[ 0.,  6., 12., 18., 24.]]]])


Instruction: Vector-Tensor element-wise multiplication in Pytorch

Input:
I am trying to extract the luminance from a tensor representing an image in Pytorch, and so I need to multiply element-wise a vector of size 3 (for the three RGB value weights) by a 3xNxN tensor representing the image such that I obtain a NxN matrix in the end where the three channels of the tensor have been summed with the weights given in the vector.

I guess there exists Pytorch operations that would help me do that without loops but I haven't found them.

Output:
You have to reshape your 3 dimensional RGB vector to be broadcastable to 3xNxN like this:

rgb = rgb.reshape(-1, 1, 1)

So it will have shape (3, 1, 1)

Now you can multiply it with original image and sum along first dimension:

result = torch.sum(rgb * image, dim=0)


Instruction: MSELoss when mask is used

Input:
I'm trying to calculate MSELoss when mask is used. Suppose that I have tensor with batch_size of 2: [2, 33, 1] as my target, and another input tensor with the same shape. Since sequence length might differ for each instance, I have also a binary mask indicating the existence of each element in the input sequence. So here is what I'm doing:

mse_loss = nn.MSELoss(reduction='none')

loss = mse_loss(input, target)
loss = (loss * mask.float()).sum() # gives \sigma_euclidean over unmasked elements

mse_loss_val = loss / loss.numel()

# now doing backpropagation
mse_loss_val.backward()

Is loss / loss.numel() a good practice? I'm skeptical, as I have to use reduction='none' and when calculating final loss value, I think I should calculate the loss only considering those loss elements that are nonzero (i.e., unmasked), however, I'm taking the average over all tensor elements with torch.numel(). I'm actually trying to take 1/n factor of MSELoss into account. Any thoughts?

Output:
There are some issues in the code. I think correct code should be:

mse_loss = nn.MSELoss(reduction='none')

loss = mse_loss(input, target)
loss = (loss * mask.float()).sum() # gives \sigma_euclidean over unmasked elements

non_zero_elements = mask.sum()
mse_loss_val = loss / non_zero_elements

# now doing backpropagation
mse_loss_val.backward()

This is only slightly worse than using .mean() if you are worried about numerical errors.


Instruction: Broadcast error HDF5: Can't broadcast (3, 2048, 1, 1) -> (4, 2048, 1, 1)

Input:
I have received the following error:

TypeError: Can't broadcast (3, 2048, 1, 1) -> (4, 2048, 1, 1)

I am extracting features and placing them into a hdf5 dataset like this:

array_40 = hdf5_file.create_dataset(
                    f'{phase}_40x_arrays',  shape, maxshape=(None, args.batch_size, 2048, 1, 1))

In (None, args.batch_size, 2048, 1, 1), None is specified due to the unknown nature of the size of dataset. args.batch_size is 4 in this case, 2048, 1 and 1 are the number of features extracted and their spatial dimensions.

shape is defined as:

shape = (dataset_length, args.batch_size, 2048, 1, 1)

However, I'm not sure what I can do with the args.batch_size, which in this case is 4. I can't leave this as None as it comes up with an illegal error:

ValueError: Illegal value in chunk tuple

EDIT: Yes, you're absolutley right. I'm trying to incrementally write to a hdf5 dataset. I've shown more of the code below. I'm extracting features and storing them incrementally into a hdf5 dataset. Despite a batch size of 4, it would be ideal to save each item from the batch, incrementally as its own instance/row.

shape = (dataset_length, 2048, 1, 1)
            all_shape = (dataset_length, 6144, 1, 1)
            labels_shape = (dataset_length)
            batch_shape = (1,)

            path = args.HDF5_dataset + f'{phase}.hdf5'

            #hdf5_file = h5py.File(path, mode='w')
            with h5py.File(path, mode='a') as hdf5_file:

                array_40 = hdf5_file.create_dataset(
                    f'{phase}_40x_arrays',  shape, maxshape=(None, 2048, 1, 1)
                )
                array_labels = hdf5_file.create_dataset(
                    f'{phase}_labels', labels_shape, maxshape=(None), dtype=string_type
                )
                array_batch_idx = hdf5_file.create_dataset(
                    f'{phase}_batch_idx', data=np.array([-1, ])
                )

                hdf5_file.close()

        # either new or checkpionted file exists
        # load file and create references to exisitng h5 datasets
        with h5py.File(path, mode='r+') as hdf5_file:
            array_40 = hdf5_file[f'{phase}_40x_arrays']
            array_labels = hdf5_file[f'{phase}_labels']
            array_batch_idx = hdf5_file[f'{phase}_batch_idx']

            batch_idx = int(array_batch_idx[0]+1)

            print("Batch ID is restarting from {}".format(batch_idx))

            dataloaders_dict = torch.utils.data.DataLoader(datasets_dict, batch_size=args.batch_size, sampler=SequentialSampler2(
                datasets_dict, batch_idx, args.batch_size),drop_last=True, num_workers=args.num_workers, shuffle=False)  # make sure shuffling is false for sampler to work and incase you restart

            for i, (inputs40x, paths40x, labels) in enumerate(dataloaders_dict):

                print(f'Batch ID: {batch_idx}')

                inputs40x = inputs40x.to(device)
                labels = labels.to(device)
                paths = paths40x

                x40 = resnet(inputs40x)

                # torch.Size([1, 2048, 1, 1]) batch, feats, 1l, 1l
                array_40[...] = x40.cpu()
                array_labels[batch_idx, ...] = labels[:].cpu()
                array_batch_idx[:,...] = batch_idx

                batch_idx +=1
                hdf5_file.flush()

Output:
I think you are confused on use of maxshape=() parameter. It sets maximum allocated dataset size in each dimension. The first dataset dimension is set to dataset_length at creation with maxshape[0]=None which allows for unlimited growth in size. The size of second dataset dimension at creation is args.batch_size. You specified the same size for maxshape, so you can't increase this dimension.

I'm a little confused by your example. It sounds like you trying to incrementally write data to the dataset in rows/instances of args.batch_size. Your example has 51 rows/instances of data, and you want to write in batches of args.batch_size=4. With 51 rows, you can write the first 48 rows (0-3, 4-7...44-47), then stuck with the 3 remaining rows. Can't you address this by adding a counter (call it nrows_left), and changing the batch size argument to min(args.batch_size, rows_left)? Seems like the easiest solution to me.

Without more info, I can't write a complete example.
I will attempt to show what I mean below:

# args.batch_size = 4
shape = (dataset_length, 2048, 1, 1)
array_40 = hdf5_file.create_dataset(
           f'{phase}_40x_arrays', shape, maxshape=(None, 2048, 1, 1))
nrows_left= dataset_length
rcnt = 0
loopcnt = dataset_length/args.batch_size
if dataset_length%args.batch_size != 0:
    loopcnt += 1
for loop in range(loopcnt) :
    nload = min(nrows_left, args.batch_size)
    array_40[rcnt :row+nload] = img_data[rcnt:row+nload ]
    rcnt += nload
    nrows_left -= nload


Instruction: What is the point of multinomial vs argmax evaluation of accuracy?

Input:
What is the purpose of evaluating prediction accuracy using multinomial instead of the straight up argmax?

probs_Y = torch.softmax(model(test_batch, feature_1, feature_2), 1)

sampled_Y = torch.multinomial(probs_Y, 1)
argmax_Y = torch.max(probs_Y, 1)[1].view(-1, 1)

print('Accuracy of sampled predictions on the test set: {:.4f}%'.format(
    (test_Y == sampled_Y.float()).sum().item() / len(test_Y) * 100))
print('Accuracy of argmax predictions on the test set: {:4f}%'.format(
    (test_Y == argmax_Y.float()).sum().item() / len(test_Y) * 100))

Result:

Accuracy of sampled predictions on the test set: 88.8889%

Accuracy of argmax predictions on the test set: 97.777778%

Reading the pytorch docs it looks like multinomial is sampling according to some distribution - just not sure how that is relevant in assessing accuracy.

I've noticed that the multinomial is non-deterministic - meaning that it is outputting a different accuracy, presumably by including different samples, each time it runs.

Output:
Here, with multinomial - we're sampling the classes with the multinomial distribution.

From the Wikipedia example,

  Suppose that in a three-way election for a large country, candidate A
  received 20% of the votes, candidate B received 30% of the votes, and
  candidate C received 50% of the votes. If six voters are selected
  randomly, what is the probability that there will be exactly one
  supporter for candidate A, two supporters for candidate B and three
  supporters for candidate C in the sample?

  Note: Since we’re assuming that the voting population is large, it is
  reasonable and permissible to think of the probabilities as unchanging
  once a voter is selected for the sample. Technically speaking this is
  sampling without replacement, so the correct distribution is the
  multivariate hypergeometric distribution, but the distributions
  converge as the population grows large.

If we look closely, we're doing the same, sampling without replacement.

torch.multinomial(input, num_samples, replacement=False, *, generator=None, out=None)

ref: https://pytorch.org/docs/stable/torch.html?highlight=torch%20multinomial#torch.multinomial

A multinomial experiment is a statistical experiment and it consists of n repeated trials. Each trial has a discrete number of possible outcomes. On any given trial, the probability that a particular outcome will occur is constant (that was the initial assumption).

So, the voting count can be simulated with the probabilities. Classification from soft outputs is a similar voting process.

If we repeat the experiment enough times, we'll reach as close as the actual probability.

For example, let's start with an initial probs = [0.1, 0.1, 0.3, 0.5].

We can repeat the experiment n times and count how many times an index was selected by torch.multinomial.

import torch

cnt = [0, 0, 0, 0]
for _ in range(5000):
  sampled_Y = torch.multinomial(torch.tensor([0.1, 0.1, 0.3, 0.5]), 1)
  cnt[sampled_Y[0]] += 1

print(cnt)

After 50 iterations: [6, 3, 14, 27]

After 5000 iterations: [480, 486, 1525, 2509]

After 50000 iterations: [4988, 4967, 15062, 24983]

But, this is avoided in model evaluation since it's not deterministic and requires a random generator to simulate the experiment. This is particularly useful for monte-carlo simulations, prior-posterior calculation. I have seen a graph classification example, where such evaluation was used. But I think it's not common and (even useful) in most classification tasks in machine learning.


Instruction: The loss effects in multitask learning framework

Input:
I have designed a multi-task network where the first layers are shared between two output layers. Through investigating multi-task learning principles, I got to know that there should be a weight scalar parameter such as alpha that dampens the two losses outputted from two output layers. My question is about this parameter itself. Does it have effect on the model's final performance? probably yes.

This is the part of my code snippet for computation of losses:

...

mtl_loss = (alpha) * loss_1 + (1-alpha) * loss_2
mtl_loss.backward()

...

Above, loss_1 is MSELoss, and loss_2 is CrossEntropyLoss. As such, picking alpha=0.9, I'm getting the following loss values during training steps:

[2020-05-03 04:46:55,398 INFO] Step 50/150000; loss_1: 0.90 + loss_2: 1.48 = mtl_loss: 2.43 (RMSE: 2.03, F1score: 0.07); lr: 0.0000001;  29 docs/s;     28 sec
[2020-05-03 04:47:23,238 INFO] Step 100/150000; loss_1: 0.40 + loss_2: 1.27 = mtl_loss: 1.72 (RMSE: 1.38, F1score: 0.07); lr: 0.0000002;  29 docs/s;     56 sec
[2020-05-03 04:47:51,117 INFO] Step 150/150000; loss_1: 0.12 + loss_2: 1.19 = mtl_loss: 1.37 (RMSE: 0.81, F1score: 0.08); lr: 0.0000003;  29 docs/s;     84 sec
[2020-05-03 04:48:19,034 INFO] Step 200/150000; loss_1: 0.04 + loss_2: 1.10 = mtl_loss: 1.20 (RMSE: 0.55, F1score: 0.07); lr: 0.0000004;  29 docs/s;    112 sec
[2020-05-03 04:48:46,927 INFO] Step 250/150000; loss_1: 0.02 + loss_2: 0.96 = mtl_loss: 1.03 (RMSE: 0.46, F1score: 0.08); lr: 0.0000005;  29 docs/s;    140 sec
[2020-05-03 04:49:14,851 INFO] Step 300/150000; loss_1: 0.02 + loss_2: 0.99 = mtl_loss: 1.05 (RMSE: 0.43, F1score: 0.08); lr: 0.0000006;  29 docs/s;    167 sec
[2020-05-03 04:49:42,793 INFO] Step 350/150000; loss_1: 0.02 + loss_2: 0.97 = mtl_loss: 1.04 (RMSE: 0.43, F1score: 0.08); lr: 0.0000007;  29 docs/s;    195 sec
[2020-05-03 04:50:10,821 INFO] Step 400/150000; loss_1: 0.01 + loss_2: 0.94 = mtl_loss: 1.00 (RMSE: 0.41, F1score: 0.08); lr: 0.0000008;  29 docs/s;    223 sec
[2020-05-03 04:50:38,943 INFO] Step 450/150000; loss_1: 0.01 + loss_2: 0.86 = mtl_loss: 0.92 (RMSE: 0.40, F1score: 0.08); lr: 0.0000009;  29 docs/s;    252 sec

As training loss shows, it seems that my first network that uses MSELoss converges super fast, while the second network has not been converged yet. RMSE, and F1score are two metrics that I'm using to track the progress of first, and second network, respectively.

I know that picking the optimal alpha is somewhat experimental, but are there hints to make the process of picking it easier? Specifically, I want the networks being trained in line with each other, not like above that the first network converges super duper fast. Can alpha parameter help controlling this?

Output:
With that alpha, loss_1 is contributing more to the result and due backpropagation updates weights proportionally to error it improves faster. Try using more equilibrated alpha to balance the performance in both tasks.

You also can try change alpha during training.


Instruction: How to make pip run a specific command from requirements file?

Input:
I would like to add lines for torch and torchvision on my requirements.txt file, to allow for easy clone, and I will be moving from computer to computer and to cloud in the near future.

I want an easy pip install -r requirements.txt and be done with it for my project.

> pip freeze > requirements.txt

gives something like

...
torch==1.5.0
torchvision==0.6.0
...

However, pip install -r requirements.txt (which is in fact pip install torch) doesn't work, and instead, as the official torch site, clearly says the command should be:

pip install torch===1.5.0 torchvision===0.6.0 -f https://download.pytorch.org/whl/torch_stable.html

How do I make the requirements file reflect this?

My desktop is Windows 10.

Bonus question:

My cloud is Linux based.
How do I make the requirements file fit both desktop and cloud?

Output:
You can use the same options in your requirements.txt file, e.g.

torch===1.5.0
torchvision===0.6.0
-f https://download.pytorch.org/whl/torch_stable.html

Then simply run pip install -r requirements.txt


Instruction: In pytorch, is there a built-in method to extract rows with given indexes?

Input:
Suppose I have a torch tensor

import torch
a = torch.tensor([[1,2,3],
                  [4,5,6],
                  [7,8,9]])

and a list

b = [0,2]

Is there a built-in method to extract the rows 0 and 2 and put them in a new tensor:

tensor([[1,2,3],
        [7,8,9]])

In particular, is there a function that look likes this:

extract_rows(a,b) -> c

where c contains desired rows. Sure, this can done by a for loop, but a built-in method is in general faster.

Note that the example is only an example, there could be dozens of indexes in the list, and hundreds of rows in the tensor.

Output:
have a look at torch builtin index_select() method. It would be helpful to you.
or
You can do this using slicing.

tensor = [[1,2,3],
            [4,5,6],
            [7,8,9]]

new_tensor = tensor[0::2]
print(new_tensor)

Output:

[[1, 2, 3], [7, 8, 9]]


Instruction: Derived Class of Pytorch nn.Module Cannot be Loaded by Module Import in Python

Input:
Using Python 3.6 with Pytorch 1.3.1. I have noticed that some saved nn.Modules cannot be loaded when the whole module is being imported into another module. To give an example, here is the template of a minimum working example.

#!/usr/bin/env python3
#encoding:utf-8
# file 'dnn_predict.py'

from torch import nn
class NN(nn.Module):##NN network
    # Initialisation and other class methods

networks=[torch.load(f=os.path.join(resource_directory, 'nn-classify-cpu_{fold}.pkl'.format(fold=fold))) for fold in range(5)]
...
if __name__=='__main__':
    # Some testing snippets
    pass

The whole file works just fine when I run it in the shell directly. However, when I want to use the class and load the neural network in another file using this code, it fails.

#!/usr/bin/env python3
#encoding:utf-8
from dnn_predict import *

The error reads AttributeError: Can't get attribute 'NN' on <module '__main__'>

Does loading of saved variables or importing modules happen differently in Pytorch than other common Python libraries? Some help or pointer to the root cause will be really appreciated.

Output:
When you save a model with torch.save(model, PATH) the whole object gets serialised with pickle, which does not save the class itself, but a path to the file containing the class, hence when loading the model the exact same directory and file structure is required to find the correct class. When running a Python script, the module of that file is __main__, therefore if you want to load that module, your NN class must be defined in the script you're running.

That is very inflexible, so the recommended approach is to not save the entire model, but instead just save the state dictionary, which only saves the parameters of the model.

# Save the state dictionary of the model
torch.save(model.state_dict(), PATH)

Afterwards, the state dictionary can be loaded and applied to your model.

from dnn_predict import NN

# Create the model (will have randomly initialised parameters)
model = NN()

# Load the previously saved state dictionary
state_dict = torch.load(PATH)

# Apply the state dictionary to the model
model.load_state_dict(state_dict)

More details on the state dictionary and saving/loading the models: PyTorch - Saving and Loading Models


Instruction: How to vectorize matrix inversion while handling runtime error in pytorch

Input:
I need to invert some matrices in pytorch. However, some of the matrices are not invertible, which leads to the code throwing runtime error as follows,

matrices = torch.randn([5,3,3])
matrices[[2,3]] = torch.zeros([3,3])
inverses = torch.inverse(matrices)

RuntimeError: inverse_cpu: For batch 2: U(1,1) is zero, singular U.

I have a fallback technique for such situations. However, I can't figure out which of the matrices throw the error. Currently, I have replaced the code with non-vectorized version, but it has become a bottleneck.

Is there a way to handle this without giving up vectorization?

Output:
The best way I can think of is to first calculate the determinate of each matrix, then calculate inverses of those that have a abs(det)>0.

matrices = torch.randn([5,3,3])
matrices[[2,3]] = torch.zeros([3,3])
determinants = torch.det(matrices)
inverses = torch.inverse(matrices[determinants.abs()>0.])

You'll have to handle the removal of singular matrices, but that shouldn't be too hard since you have the index values of those matrices from determinants.abs()==0.. This allows you to keep the inversion vectorized.


Instruction: How is KL-divergence in pytorch code related to the formula?

Input:
In VAE tutorial, kl-divergence of two Normal Distributions is defined by:

And in many code, such as here, hereand here, the code is implemented as:

 KL_loss = -0.5 * torch.sum(1 + logv - mean.pow(2) - logv.exp())

or

def latent_loss(z_mean, z_stddev):
    mean_sq = z_mean * z_mean
    stddev_sq = z_stddev * z_stddev
    return 0.5 * torch.mean(mean_sq + stddev_sq - torch.log(stddev_sq) - 1)

How are they related? why there is not any "tr" or ".transpose()" in code?

Output:
The expressions in the code you posted assume X is an uncorrelated multi-variate Gaussian random variable. This is apparent by the lack of cross terms in the determinant of the covariance matrix. Therefore the mean vector and covariance matrix take the forms

Using this we can quickly derive the following equivalent representations for the components of the original expression

Substituting these back into the original expression gives


Instruction: Why does by torch.optim.SGD method learning rate change?

Input:
With SGD learning rate should not be changed during epochs but it is. Help me understand why it happens please and how to prevent this LR changing?

import torch
params = [torch.nn.Parameter(torch.randn(1, 1))]
optimizer = torch.optim.SGD(params, lr=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
for epoch in range(5):
    print(scheduler.get_lr())
    scheduler.step()

Output is:

[0.9]
[0.7290000000000001]
[0.6561000000000001]
[0.5904900000000002]
[0.5314410000000002]

My torch version is 1.4.0

Output:
Since you are using the command  torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9) (meaning actually torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.9)) thus you are multiplying the learning rate by gamma=0.9 every step_size=1 step:

0.9     = 0.9
0.729   = 0.9*0.9*0.9
0.6561  = 0.9*0.9*0.9*0.9
0.59049 = 0.9*0.9*0.9*0.9*0.9

The only "strange" point is that it missing 0.81=0.9*0.9 at the second step (UPDATE: see Szymon Maszke answer for an explanation)

To prevent early decreasing, if you have N samples in your dataset, and the batch size is D, then set torch.optim.lr_scheduler.StepLR(optimizer, step_size=N/D, gamma=0.9) to decrease at each epoch. To decrease each E epoch set torch.optim.lr_scheduler.StepLR(optimizer, step_size=E*N/D, gamma=0.9)


Instruction: How Can I change the value of "maxdets" in Faster R-CNN by Pytorch?

Input:
I am implementing a faster RCNN network on pytorch. I have followed the next tutorial.

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

There are images in which I have more than 100 objects to classify. However, with this tutorial I can only detect a maximum of 100 objects, since the parameter "maxdets" = 100.

Is there a way to change this value to adapt it to my project?

IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.235
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.655
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.105
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.238
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.006
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.066
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.331
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.331
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

If only change next param, it would be the problem solved?

cocoeval.Params.setDetParams.maxDets = [1, 10, 100]

Thank you!

Output:
"There are images in which I have more than 100 objects to classify."

maxDets = 100 doesn't mean it will classify only 100 images but it refers to % AverageRecall given 100 detections per image

inshort maxDets is realted to metrics not actual no. of images classified.

for more info visit :
http://cocodataset.org/#detection-eval

Tensorboard graph recall

https://github.com/matterport/Mask_RCNN/issues/663

 # Limit to max_per_image detections **over all classes**
    if number_of_detections > self.detections_per_img > 0:
        cls_scores = result.get_field("scores")
        image_thresh, _ = torch.kthvalue(
            cls_scores.cpu(), number_of_detections - self.detections_per_img + 1
        )
        keep = cls_scores >= image_thresh.item()
        keep = torch.nonzero(keep).squeeze(1)
        result = result[keep]
    return result

according to this code snippet I found that it checks the no. of detection
so model.roi_heads.detections_per_img=300 is correct for your purpose.
And I haven't found much proper documentation on maxdets but i guess the above code should work.

 # non-maximum suppression, independently done per class
   keep = box_ops.batched_nms(boxes, scores, labels, self.nms_thresh)
 # keep only topk scoring predictions
   keep = keep[:self.detections_per_img]

this code snippet says that we can filter out only some top detections we want to have in our model.


Instruction: Pytorch C++ (libtorch) outputs different results if I change shape

Input:
So I'm learning Neural networks right now and I noticed something really really strange in my network.
I have an input layer created like this

convN1 = register_module("convN1", torch::nn::Conv2d(torch::nn::Conv2dOptions(4, 256, 3).padding(1)));

and an output layer that is a tanh function.

So it's expecting a torch::Tensor of shape
{/batchSize/, 4, /sideLength/, /sideLength/} which is and will output a tensor of just 1 float value.

As such for testing I have created a custom Tensor of shape {4, 15, 15}.

The really weird part is what's happening below

auto inputTensor = torch::zeros({ 1, 4, 15, 15});
inputTensor[0] = customTensor;
std::cout << network->forward(inputTensor); // Outputs something like 0.94142

inputTensor = torch::zeros({ 32, 4, 15, 15});
inputTensor[0] = customTensor;
std::cout << network->forward(inputTensor); // Outputs something like 0.1234 then 0.8543 31 times

So why is the customTensor getting 2 different values from my network just from the fact that the batchsize has changed? Am I not understanding some parts of how tensors work?

P.S. I did check and the above block of code was operating under eval mode.

Edit: Since it's been asked here's a more indepth look at my network

convN1 = register_module("convN1", torch::nn::Conv2d(torch::nn::Conv2dOptions(4, 256, 3).padding(1)));
batchNorm1 = register_module("batchNorm1", torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));

m_residualBatch1 = register_module(batch1Name, torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));
m_residualBatch2 = register_module(batch2Name, torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));
m_residualConv1 = register_module(conv1Name, torch::nn::Conv2d(torch::nn::Conv2dOptions(256, 256, 3).padding(1)));
m_residualConv2 = register_module(conv2Name, torch::nn::Conv2d(torch::nn::Conv2dOptions(256, 256, 3).padding(1)));

valueN1 = register_module("valueN1", torch::nn::Conv2d(256, 2, 1));
batchNorm3 = register_module("batchNorm3", torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(2)));
valueN2 = register_module("valueN2", torch::nn::Linear(2 * BOARD_LENGTH, 64));
valueN3 = register_module("valueN3", torch::nn::Linear(64, 1));

And how it forwards is like so

torch::Tensor Net::forwadValue(torch::Tensor x)
{
    x = convN1->forward(x);
    x = batchNorm1->forward(x);
    x = torch::relu(x);

    torch::Tensor residualCopy = x.clone();
    x = m_residualConv1->forward(x);
    x = m_residualBatch1->forward(x);
    x = torch::relu(x);
    x = m_residualConv2->forward(x);
    x = m_residualBatch2->forward(x);
    x += residualCopy;
    x = torch::relu(x);

    x = valueN1->forward(x);
    x = batchNorm3->forward(x)
    x = torch::relu(x);
    x = valueN2->forward(x.reshape({ x.sizes()[0], 30 }))
    x = torch::relu(x);
    x = valueN3->forward(x)
    return torch::tanh(x);
}

Output:
THANK YOU @MichaelJungo turns out you were right in that one of my BatchNorm2d wasn't being set to eval mode. I was unclear how registering modules worked in the beginning (still am to an extent) so I overloaded the ::train() function to manually set all my modules to the necessary mode.

In it I forgot to set one of my BatchNorm modules to the correct mode, doing so made it all consistent thanks a bunch!


Instruction: How to do cubic spline interpolation and integration in Pytorch

Input: In Pytorch, is there cubic spline interpolation similar to Scipy's? Given 1D input tensors x and y, I want to interpolate through those points and evaluate them at xs to obtain ys. Also, I want an integrator function that finds Ys, the integral of the spline interpolation from x[0] to xs.

Output:
Here is a gist I made doing this with Cubic Hermite Splines in Pytorch efficiently and with autograd support.
For convenience, I'll also put the code here.
import torch as T

def h_poly_helper(tt):
  A = T.tensor([
      [1, 0, -3, 2],
      [0, 1, -2, 1],
      [0, 0, 3, -2],
      [0, 0, -1, 1]
      ], dtype=tt[-1].dtype)
  return [
    sum( A[i, j]*tt[j] for j in range(4) )
    for i in range(4) ]

def h_poly(t):
  tt = [ None for _ in range(4) ]
  tt[0] = 1
  for i in range(1, 4):
    tt[i] = tt[i-1]*t
  return h_poly_helper(tt)

def H_poly(t):
  tt = [ None for _ in range(4) ]
  tt[0] = t
  for i in range(1, 4):
    tt[i] = tt[i-1]*t*i/(i+1)
  return h_poly_helper(tt)

def interp_func(x, y):
  "Returns integral of interpolating function"
  if len(y)>1:
    m = (y[1:] - y[:-1])/(x[1:] - x[:-1])
    m = T.cat([m[[0]], (m[1:] + m[:-1])/2, m[[-1]]])
  def f(xs):
    if len(y)==1: # in the case of 1 point, treat as constant function
      return y[0] + T.zeros_like(xs)
    I = T.searchsorted(x[1:], xs)
    dx = (x[I+1]-x[I])
    hh = h_poly((xs-x[I])/dx)
    return hh[0]*y[I] + hh[1]*m[I]*dx + hh[2]*y[I+1] + hh[3]*m[I+1]*dx
  return f

def interp(x, y, xs):
  return interp_func(x,y)(xs)

def integ_func(x, y):
  "Returns interpolating function"
  if len(y)>1:
    m = (y[1:] - y[:-1])/(x[1:] - x[:-1])
    m = T.cat([m[[0]], (m[1:] + m[:-1])/2, m[[-1]]])
    Y = T.zeros_like(y)
    Y[1:] = (x[1:]-x[:-1])*(
        (y[:-1]+y[1:])/2 + (m[:-1] - m[1:])*(x[1:]-x[:-1])/12
        )
    Y = Y.cumsum(0)
  def f(xs):
    if len(y)==1:
      return y[0]*(xs - x[0])
    I = P.searchsorted(x[1:].detach(), xs)
    dx = (x[I+1]-x[I])
    hh = H_poly((xs-x[I])/dx)
    return Y[I] + dx*(
        hh[0]*y[I] + hh[1]*m[I]*dx + hh[2]*y[I+1] + hh[3]*m[I+1]*dx
        )
  return f

def integ(x, y, xs):
  return integ_func(x,y)(xs)

# Example
if __name__ == "__main__":
  import matplotlib.pylab as P # for plotting
  x = T.linspace(0, 6, 7)
  y = x.sin()
  xs = T.linspace(0, 6, 101)
  ys = interp(x, y, xs)
  Ys = integ(x, y, xs)
  P.scatter(x, y, label='Samples', color='purple')
  P.plot(xs, ys, label='Interpolated curve')
  P.plot(xs, xs.sin(), '--', label='True Curve')
  P.plot(xs, Ys, label='Spline Integral')
  P.plot(xs, 1-xs.cos(), '--', label='True Integral')
  P.legend()
  P.show()


Instruction: Understanding input shape to PyTorch LSTM

Input:
This seems to be one of the most common questions about LSTMs in PyTorch, but I am still unable to figure out what should be the input shape to PyTorch LSTM.

Even after following several posts (1, 2, 3) and trying out the solutions, it doesn't seem to work.

Background: I have encoded text sequences (variable length) in a batch of size 12 and the sequences are padded and packed using pad_packed_sequence functionality. MAX_LEN for each sequence is 384 and each token (or word) in the sequence has a dimension of 768. Hence my batch tensor could have one of the following shapes: [12, 384, 768] or [384, 12, 768].

The batch will be my input to the PyTorch rnn module (lstm here).

According to the PyTorch documentation for LSTMs, its input dimensions are (seq_len, batch, input_size) which I understand as following.
seq_len - the number of time steps in each input stream (feature vector length).
batch - the size of each batch of input sequences.
input_size - the dimension for each input token or time step.

lstm = nn.LSTM(input_size=?, hidden_size=?, batch_first=True)

What should be the exact input_size and hidden_size values here?

Output:
You have explained the structure of your input, but you haven't made the connection between your input dimensions and the LSTM's expected input dimensions.

Let's break down your input (assigning names to the dimensions):

batch_size: 12
seq_len: 384
input_size / num_features: 768

That means the input_size of the LSTM needs to be 768.

The hidden_size is not dependent on your input, but rather how many features the LSTM should create, which is then used for the hidden state as well as the output, since that is the last hidden state. You have to decide how many features you want to use for the LSTM.

Finally, for the input shape, setting batch_first=True requires the input to have the shape [batch_size, seq_len, input_size], in your case that would be [12, 384, 768].

import torch
import torch.nn as nn

# Size: [batch_size, seq_len, input_size]
input = torch.randn(12, 384, 768)

lstm = nn.LSTM(input_size=768, hidden_size=512, batch_first=True)

output, _ = lstm(input)
output.size()  # => torch.Size([12, 384, 512])


Instruction: My train accuracy remains at 10% when I add weight_decay parameter to my optimizer in PyTorch. I am using CIFAR10 dataset and LeNet CNN model

Input:
I am training CIFAR10 dataset on LeNet CNN model. I am using PyTorch on Google Colab. The code runs only when I use Adam optimizer with model.parameters() as the only parameter. But when I change my optimizer or use weight_decay parameter then the accuracy remains at 10% through all the epochs. I cannot understand the reason why it is happening.

# CNN Model - LeNet
class LeNet_ReLU(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn_model = nn.Sequential(nn.Conv2d(3,6,5),
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2),
                                       nn.Conv2d(6,16,5),
                                       nn.ReLU(),
                                       nn.AvgPool2d(2, stride=2))
        self.fc_model = nn.Sequential(nn.Linear(400, 120),
                                      nn.ReLU(),
                                      nn.Linear(120,84),
                                      nn.ReLU(),
                                      nn.Linear(84,10))

    def forward(self, x):
        x = self.cnn_model(x)
        x = x.view(x.size(0), -1)
        x = self.fc_model(x)
        return x

# Importing dataset and creating dataloader
batch_size = 128
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True,
                                    transform=transforms.ToTensor())
trainloader = utils_data.DataLoader(trainset, batch_size=batch_size, shuffle=True)
testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True,
                                    transform=transforms.ToTensor())
testloader = utils_data.DataLoader(testset, batch_size=batch_size, shuffle=False)

# Creating instance of the model
net = LeNet_ReLU()

# Evaluation function
def evaluation(dataloader):
    total, correct = 0, 0
    for data in dataloader:
        inputs, labels = data

        outputs = net(inputs)
        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (pred==labels).sum().item()
    return correct/total * 100

# Loss function and optimizer
loss_fn = nn.CrossEntropyLoss()
opt = optim.Adam(net.parameters(), weight_decay = 0.9)

# Model training
loss_epoch_arr = []
max_epochs = 16

for epoch in range(max_epochs):
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data

        outputs = net(inputs)
        loss = loss_fn(outputs, labels)
        loss.backward()
        opt.step()

        opt.zero_grad()

    loss_epoch_arr.append(loss.item())

    print('Epoch: %d/%d, Test acc: %0.2f, Train acc: %0.2f'
    % (epoch,max_epochs, evaluation(testloader), evaluation(trainloader)))

plt.plot(loss_epoch_arr)

Output:
The weight decay mechanism sets a penalty for high value wieghts, i.e. it stricts the weights to have relatively small values by adding their sum multiplied by the weight_decay argument you gave it. That can be seen as a quadratic regularization term.

When passing large weight_decay value, you may strict your network too much and prevent it from learning, that's probably the reason it had 10% of accuracy which is related to non-learning at all and just guessing the answer (since you have 10 classes you receive 10% of acc, when the output isn't a function of your input at all).

The solution would be to play around with different values, train for weight_decay of 1e-4 or some other values in that area. Note that when you reach values closer to zero you should have results which are closer to your initial train without using the weight decay.

Hope that helps.


Instruction: I am getting Attribute error as 'int' object has no attribute 'to'

Input:
I am writing a python code in Kaggle notebook for Image Classification. In the training part, I am getting an error

AttributeError                            Traceback (most recent call last)
<ipython-input-22-052723d8ce9d> in <module>
      5     test_loss = 0.0
      6     for images,label in enumerate(train_loader):
----> 7         images,label = images.to(cuda),label.to(cuda)
      8         optimizer.zero_grad()
      9

AttributeError: 'int' object has no attribute 'to'

This is the following code, (I am giving only 2 parts, pls tell if you need more)

train_loader = torch.utils.data.DataLoader(train_data,batch_size = 128,num_workers =0,shuffle =True)
test_loader = torch.utils.data.DataLoader(test_data,batch_size = 64,num_workers =0,shuffle =False)

epoch = 10

for e in range(epoch):
    train_loss = 0.0
    test_loss = 0.0
    for images,label in enumerate(train_loader):
        images,label = images.to(cuda),label.to(cuda)
        optimizer.zero_grad()

        output = model(images)
        _,predict = torch.max(output.data, 1)
        loss = criterion(output,labels)
        loss.backward()
        optimizer.step()

        train_loss += loss.item()
        train_size += label.size(0)
        train_success += (predict==label).sum().item()

        print("train_accuracy is {.2f}".format(100*(train_success/train_size)) )

Output:
I don't know much about the environment you're working in, but this is what goes wrong:

for images, label in enumerate (train_loader):
Puts whatever is in train_loader into label, while images is given a number.

Try this to see what I mean, and to see what goes wrong:

for images, label in enumerate(train_loader):
    print(images)
    return

And since images is a number (int), there is no images.to() method associated with images


Instruction: Validation set augmentations PyTorch example

Input:
In this PyTorch vision example for transfer learning, they are performing validation set augmentations, and I can't figure out why.

# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

As far as I know, data augmentations are to be made solely on the training set (and sometimes on the test set, in what is called Test Time Augmentations).

Why is it done here as well?
Also, why not just resizing straight to 224?

Output:
To clarify, random data augmentation is only allowed on the training set. You can apply data augmentation to the validation and test sets provided that none of the augmentations are random. You will see this clearly in the example you provided.

The training set uses many random augmentations (augmentations that use randomness usually have "random" in the name). However, the validation set only uses augmentations that don't introduce any randomness to the data.

One last important detail: when you use normalization on the validation and test set you MUST use the same exact factors you used for the training set. You will see that the example above kept the numbers the same.

The need to resize and then center crop comes from the fact the val set needs to come from the same domain of the train set, thus if the former was randomly resized and cropped to 224, the val set needs to deterministically resize and crop.


Instruction: How to extract patches from an image in pytorch?

Input:
I want to extract image patches from an image with patch size 128 and stride 32, so I have this code, but it gives me an error :

from PIL import Image
img = Image.open("cat.jpg")
x = transforms.ToTensor()(img)

x = x.unsqueeze(0)

size = 128 # patch size
stride = 32 # patch stride
patches = x.unfold(1, size, stride).unfold(2, size, stride).unfold(3, size, stride)
print(patches.shape)

and the error I get is :

RuntimeError: maximum size for tensor at dimension 1 is 3 but size is 128

This is the only method I've found so far. but it gives me this error

Output:
The size of your x is [1, 3, height, width]. Calling x.unfold(1, size, stride) tries to create slices of size 128 from dimension 1, which has size 3, hence it is too small to create any slice.

You don't want to create slices across dimension 1, since those are the channels of the image (RGB in this case) and they need to be kept as they are for all patches. The patches are only created across the height and width of an image.

patches = x.unfold(2, size, stride).unfold(3, size, stride)

The resulting tensor will have size [1, 3, num_vertical_slices, num_horizontal_slices, 128, 128]. You can reshape it to combine the slices to get a list of patches i.e. size of [1, 3, num_patches, 128, 128]:

patches = patches.reshape(1, 3, -1, size, size)