Instruction: Extracting tensor data with index in pytorch

Input:
I would like to have the tensor indexed a certain way.
Suppose my data, tensor X shaped (1, 3, 16, 9) is
tensor([[[[ 0.,  0.,  0.,  0.,  1.,  2.,  0.,  5.,  6.],
      [ 0.,  0.,  0.,  1.,  2.,  3.,  5.,  6.,  7.],
      [ 0.,  0.,  0.,  2.,  3.,  4.,  6.,  7.,  8.],
      [ 0.,  0.,  0.,  3.,  4.,  0.,  7.,  8.,  0.],
      [ 0.,  1.,  2.,  0.,  5.,  6.,  0.,  9., 10.],
      [ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
      [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
      [ 3.,  4.,  0.,  7.,  8.,  0., 11., 12.,  0.],
      [ 0.,  5.,  6.,  0.,  9., 10.,  0., 13., 14.],
      [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
      [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.],
      [ 7.,  8.,  0., 11., 12.,  0., 15., 16.,  0.],
      [ 0.,  9., 10.,  0., 13., 14.,  0.,  0.,  0.],
      [ 9., 10., 11., 13., 14., 15.,  0.,  0.,  0.],
      [10., 11., 12., 14., 15., 16.,  0.,  0.,  0.],
      [11., 12.,  0., 15., 16.,  0.,  0.,  0.,  0.]],

     [[ 0.,  0.,  0.,  0., 17., 18.,  0., 21., 22.],
      [ 0.,  0.,  0., 17., 18., 19., 21., 22., 23.],
      [ 0.,  0.,  0., 18., 19., 20., 22., 23., 24.],
      [ 0.,  0.,  0., 19., 20.,  0., 23., 24.,  0.],
      [ 0., 17., 18.,  0., 21., 22.,  0., 25., 26.],
      [17., 18., 19., 21., 22., 23., 25., 26., 27.],
      [18., 19., 20., 22., 23., 24., 26., 27., 28.],
      [19., 20.,  0., 23., 24.,  0., 27., 28.,  0.],
      [ 0., 21., 22.,  0., 25., 26.,  0., 29., 30.],
      [21., 22., 23., 25., 26., 27., 29., 30., 31.],
      [22., 23., 24., 26., 27., 28., 30., 31., 32.],
      [23., 24.,  0., 27., 28.,  0., 31., 32.,  0.],
      [ 0., 25., 26.,  0., 29., 30.,  0.,  0.,  0.],
      [25., 26., 27., 29., 30., 31.,  0.,  0.,  0.],
      [26., 27., 28., 30., 31., 32.,  0.,  0.,  0.],
      [27., 28.,  0., 31., 32.,  0.,  0.,  0.,  0.]],

     [[ 0.,  0.,  0.,  0., 33., 34.,  0., 37., 38.],
      [ 0.,  0.,  0., 33., 34., 35., 37., 38., 39.],
      [ 0.,  0.,  0., 34., 35., 36., 38., 39., 40.],
      [ 0.,  0.,  0., 35., 36.,  0., 39., 40.,  0.],
      [ 0., 33., 34.,  0., 37., 38.,  0., 41., 42.],
      [33., 34., 35., 37., 38., 39., 41., 42., 43.],
      [34., 35., 36., 38., 39., 40., 42., 43., 44.],
      [35., 36.,  0., 39., 40.,  0., 43., 44.,  0.],
      [ 0., 37., 38.,  0., 41., 42.,  0., 45., 46.],
      [37., 38., 39., 41., 42., 43., 45., 46., 47.],
      [38., 39., 40., 42., 43., 44., 46., 47., 48.],
      [39., 40.,  0., 43., 44.,  0., 47., 48.,  0.],
      [ 0., 41., 42.,  0., 45., 46.,  0.,  0.,  0.],
      [41., 42., 43., 45., 46., 47.,  0.,  0.,  0.],
      [42., 43., 44., 46., 47., 48.,  0.,  0.,  0.],
      [43., 44.,  0., 47., 48.,  0.,  0.,  0.,  0.]]]]

I would like to have those rows where (row_index % n) == i (say n = 4 and i = 0 to 3) is saved in another tensor Y.
For example, for the data X[0][0]:
[[ 0.,  0.,  0.,  0.,  1.,  2.,  0.,  5.,  6.],
 [ 0.,  0.,  0.,  1.,  2.,  3.,  5.,  6.,  7.],
 [ 0.,  0.,  0.,  2.,  3.,  4.,  6.,  7.,  8.],
 [ 0.,  0.,  0.,  3.,  4.,  0.,  7.,  8.,  0.],
 [ 0.,  1.,  2.,  0.,  5.,  6.,  0.,  9., 10.],
 [ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
 [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
 [ 3.,  4.,  0.,  7.,  8.,  0., 11., 12.,  0.],
 [ 0.,  5.,  6.,  0.,  9., 10.,  0., 13., 14.],
 [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
 [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.],
 [ 7.,  8.,  0., 11., 12.,  0., 15., 16.,  0.],
 [ 0.,  9., 10.,  0., 13., 14.,  0.,  0.,  0.],
 [ 9., 10., 11., 13., 14., 15.,  0.,  0.,  0.],
 [10., 11., 12., 14., 15., 16.,  0.,  0.,  0.],
 [11., 12.,  0., 15., 16.,  0.,  0.,  0.,  0.]]

I would like to have a tensor containing the following data, which is basically collection of the rows where row_index % 4 == 0 (here i = 0):
[[ 0.,  0.,  0.,  0.,  1.,  2.,  0.,  5.,  6.],
 [ 0.,  1.,  2.,  0.,  5.,  6.,  0.,  9., 10.],
 [ 0.,  5.,  6.,  0.,  9., 10.,  0., 13., 14.],
 [ 0.,  9., 10.,  0., 13., 14.,  0.,  0.,  0.]]

Similarly, where i = 1, row_index % 4 == i will look like:
[[ 0.,  0.,  0.,  1.,  2.,  3.,  5.,  6.,  7.],
 [ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
 [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
 [ 9., 10., 11., 13., 14., 15.,  0.,  0.,  0.]]

when i = 2,  row_index % 4 == i:
[[ 0.,  0.,  0.,  2.,  3.,  4.,  6.,  7.,  8.],
 [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
 [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.],
 [10., 11., 12., 14., 15., 16.,  0.,  0.,  0.]]

when i = 3,  row_index % 4 == i:
[[ 0.,  0.,  0.,  3.,  4.,  0.,  7.,  8.,  0.],
 [ 3.,  4.,  0.,  7.,  8.,  0., 11., 12.,  0.],
 [ 7.,  8.,  0., 11., 12.,  0., 15., 16.,  0.],
 [11., 12.,  0., 15., 16.,  0.,  0.,  0.,  0.]]

I have tried hard coding it and it doesn't seem practical when the data becomes larger and the size becomes dynamic and I assume that there would be a better way to come about it.
temp0 = data[0][0][0][:]
temp1 = data[0][0][4][:]
temp2 = data[0][0][8][:]
temp3 = data[0][0][12][:]
temp = torch.stack([temp0,temp1,temp2,temp3],dim = 0)

Also, it would be great if the result can come back in one tensor like :
tensor Y = ([[[ 0.,  0.,  0.,  0.,  1.,  2.,  0.,  5.,  6.],
              [ 0.,  1.,  2.,  0.,  5.,  6.,  0.,  9., 10.],
              [ 0.,  5.,  6.,  0.,  9., 10.,  0., 13., 14.],
              [ 0.,  9., 10.,  0., 13., 14.,  0.,  0.,  0.]],

             [[ 0.,  0.,  0.,  1.,  2.,  3.,  5.,  6.,  7.],
              [ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
              [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
              [ 9., 10., 11., 13., 14., 15.,  0.,  0.,  0.]],

             [[ 0.,  0.,  0.,  2.,  3.,  4.,  6.,  7.,  8.],
              [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
              [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.],
              [10., 11., 12., 14., 15., 16.,  0.,  0.,  0.]],

             [[ 0.,  0.,  0.,  3.,  4.,  0.,  7.,  8.,  0.],
              [ 3.,  4.,  0.,  7.,  8.,  0., 11., 12.,  0.],
              [ 7.,  8.,  0., 11., 12.,  0., 15., 16.,  0.],
              [11., 12.,  0., 15., 16.,  0.,  0.,  0.,  0.]]])

Output:
You can achieve this by first constructing a tensor containing the selected rows, then using torch.gather to assemble the final tensor.
Assuming we two lists I and N containing the values of i and n respectively:
I = [0, 1, 2, 3]
N = [4, 4, 4, 4]

First we construct the index tensor:
>>> index = torch.stack([(torch.arange(16) % n == i).nonzero() for i, n in zip(I, N)])
tensor([[[ 0],
         [ 4],
         [ 8],
         [12]],

        [[ 1],
         [ 5],
         [ 9],
         [13]],

        [[ 2],
         [ 6],
         [10],
         [14]],

        [[ 3],
         [ 7],
         [11],
         [15]]])

Then some expanding and reshaping is required:
>>> index_ = index[None].flatten(1,2).expand(X.size(0), -1, X.size(-1))
tensor([[[ 0,  0,  0,  0,  0,  0,  0,  0,  0],
         [ 4,  4,  4,  4,  4,  4,  4,  4,  4],
         [ 8,  8,  8,  8,  8,  8,  8,  8,  8],
         [12, 12, 12, 12, 12, 12, 12, 12, 12],
         [ 1,  1,  1,  1,  1,  1,  1,  1,  1],
         [ 5,  5,  5,  5,  5,  5,  5,  5,  5],
         [ 9,  9,  9,  9,  9,  9,  9,  9,  9],
         [13, 13, 13, 13, 13, 13, 13, 13, 13],
         [ 2,  2,  2,  2,  2,  2,  2,  2,  2],
         [ 6,  6,  6,  6,  6,  6,  6,  6,  6],
         [10, 10, 10, 10, 10, 10, 10, 10, 10],
         [14, 14, 14, 14, 14, 14, 14, 14, 14],
         [ 3,  3,  3,  3,  3,  3,  3,  3,  3],
         [ 7,  7,  7,  7,  7,  7,  7,  7,  7],
         [11, 11, 11, 11, 11, 11, 11, 11, 11],
         [15, 15, 15, 15, 15, 15, 15, 15, 15]]])

As a rule of thumb, we want index_ to have the same number of dimensions as X.
Now we can apply torch.gather and reshape to the final form:
>>> X.gather(1, index_).reshape(len(X), *index.shape[:2], -1)
tensor([[[[ 0.,  0.,  0.,  0.,  1.,  2.,  0.,  5.,  6.],
          [ 0.,  1.,  2.,  0.,  5.,  6.,  0.,  9., 10.],
          [ 0.,  5.,  6.,  0.,  9., 10.,  0., 13., 14.],
          [ 0.,  9., 10.,  0., 13., 14.,  0.,  0.,  0.]],

         [[ 0.,  0.,  0.,  1.,  2.,  3.,  5.,  6.,  7.],
          [ 1.,  2.,  3.,  5.,  6.,  7.,  9., 10., 11.],
          [ 5.,  6.,  7.,  9., 10., 11., 13., 14., 15.],
          [ 9., 10., 11., 13., 14., 15.,  0.,  0.,  0.]],

         [[ 0.,  0.,  0.,  2.,  3.,  4.,  6.,  7.,  8.],
          [ 2.,  3.,  4.,  6.,  7.,  8., 10., 11., 12.],
          [ 6.,  7.,  8., 10., 11., 12., 14., 15., 16.],
          [10., 11., 12., 14., 15., 16.,  0.,  0.,  0.]],

         [[ 0.,  0.,  0.,  3.,  4.,  0.,  7.,  8.,  0.],
          [ 3.,  4.,  0.,  7.,  8.,  0., 11., 12.,  0.],
          [ 7.,  8.,  0., 11., 12.,  0., 15., 16.,  0.],
          [11., 12.,  0., 15., 16.,  0.,  0.,  0.,  0.]]]])

This method can be extended to batch tensors:
>>> index = torch.stack([(torch.arange(16) % n == i).nonzero() for i, n in zip(I, N)])
>>> index_  = index[None,None].flatten(2,3).expand(X.size(0), X.size(1), -1, X.size(-1))

>>> X.gather(2, index_).reshape(*X.shape[:2], *index.shape[:2], -1)


Instruction: Pytorch / Numpy dimensions math: Why [n] + [n, 1] = [n, n]

Input:
I'm trying to understand the logic for inferring the sematics of addition in numpy/torch. Here is an example that caused a bug in my program:
import numpy as np

x = np.arange(2)     # shape (2,)
y = x.reshape(-1, 1) # shape (2, 1)
z = y + x            # expected (2,) or (2, 1)
print(z.shape)       # (2, 2)

So basically reshape happened in unrelated operation, however x and y still had same number of elements, and I've expected to get resulting shape either [2,] or [2, 1] as the addition happens on the axis where all elements live.
My questions:

why do I get [2,2] shape?
What's bigger picture behind it that can help expect this outcome in similar, but different scenarios?

Output:
This is caused by broadcasting, where the following example is given:
x  = np.arange(4)   # shape (4,)
xx = x.reshape(4,1) # shape (4,1)
y  = np.ones(5)     # shape (5,)

x + y  # ValueError: operands could not be broadcast together with shapes (4,) (5,)
xx + y  # shape (4, 5)

"When either of the dimensions compared is 1, the other is used. In other words, dimensions with size 1 are stretched or “copied” to match the other."


Instruction: How do I use autograd for a separate function independent of backpropagate in PyTorch?

Input:
I have two variables, x and theta. I am trying to minimise my loss with respect to theta only, but as part of my loss function I need the derivative of a different function (f) with respect to x. This derivative itself is not relevant to the minimisation, only its output. However, when implementing this in PyTorch I am getting a Runtime error.
A minimal example is as follows:
# minimal example of two different autograds
import torch

from torch.autograd.functional import jacobian
def f(theta, x):
    return torch.sum(theta * x ** 2)

def df(theta, x):
    J = jacobian(lambda x: f(theta, x), x)
    return J

# example evaluations of the autograd gradient
x = torch.tensor([1., 2.])
theta = torch.tensor([1., 1.], requires_grad = True)

# derivative should be 2*theta*x (same as an analytical)
with torch.no_grad():
    print(df(theta, x))
    print(2*theta*x)

tensor([2., 4.])

tensor([2., 4.])

# define some arbitrary loss as a fn of theta
loss = torch.sum(df(theta, x)**2)
loss.backward()

gives the following error

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

If I provide an analytic derivative (2*theta*x), it works fine:
loss = torch.sum((2*theta*x)**2)
loss.backward()

Is there a way to do this in PyTorch? Or am I limited in some way?
Let me know if anyone needs any more details.
PS
I am imagining the solution is something similar to the way that JAX does autograd, as that is what I am more familiar with. What I mean here is that in JAX I believe you would just do:
from jax import grad
df = grad(lambda x: f(theta, x))

and then df would just be a function that can be called at any point. But is PyTorch the same? Or is there some conflict within .backward() that causes this error?

Output:
PyTorch's jacobian does not create a computation graph unless you explicitely ask for it
J = jacobian(lambda x: f(theta, x), x, create_graph=True)

.. with create_graph argument.
The documentation is quite clear about it

create_graph (bool, optional) – If True, the Jacobian will be computed in a differentiable manner


Instruction: How to change the directory of mlflow logs?

Input:
I am using MLflow to log the metrics but I want to change the default saving logs directory. So, instead of writing log files besides my main file, I want to store them to /path/outputs/lg . I don't know how to change it. I use it without in the Model.
import os
from time import time

import mlflow
import numpy as np
import torch
import tqdm

# from segmentation_models_pytorch.utils import metrics
from AICore.emergency_landing.metrics import IoU, F1
from AICore.emergency_landing.utils import AverageMeter
from AICore.emergency_landing.utils import TBLogger

class Model:
    def __init__(self, model, num_classes=5, ignore_index=0, optimizer=None, scheduler=None, criterion=None,
                 device=None, epochs=30, train_loader=None, val_loader=None, tb_logger: TBLogger = None,
                 logger=None,
                 best_model_path=None,
                 model_check_point_path=None,
                 load_from_best_model=None,
                 load_from_model_checkpoint=None,
                 early_stopping=None,
                 debug=False):

        self.debug = debug

        self.early_stopping = {
            'init': early_stopping,
            'changed': 0
        }
        self.optimizer = optimizer
        self.scheduler = scheduler
        self.criterion = criterion
        self.device = device
        self.epochs = epochs
        self.train_loader = train_loader
        self.val_loader = val_loader

        self.model = model.to(device)

        self.tb_logger = tb_logger
        self.logger = logger

        self.best_loss = np.Inf

        if not os.path.exists(best_model_path):
            os.makedirs(best_model_path)
        self.best_model_path = best_model_path

        if not os.path.exists(model_check_point_path):
            os.makedirs(model_check_point_path)
        self.model_check_point_path = model_check_point_path

        self.load_from_best_model = load_from_best_model
        self.load_from_model_checkpoint = load_from_model_checkpoint

        if self.load_from_best_model is not None:
            self.load_model(path=self.load_from_best_model)
        if self.load_from_model_checkpoint is not None:
            self.load_model_checkpoint(path=self.load_from_model_checkpoint)

        self.train_iou = IoU(num_classes=num_classes, ignore_index=ignore_index)
        self.val_iou = IoU(num_classes=num_classes, ignore_index=ignore_index)
        self.test_iou = IoU(num_classes=num_classes, ignore_index=ignore_index)

        self.train_f1 = F1(num_classes=num_classes, ignore_index=ignore_index, mdmc_average='samplewise')
        self.val_f1 = F1(num_classes=num_classes, ignore_index=ignore_index, mdmc_average='samplewise')
        self.test_f1 = F1(num_classes=num_classes, ignore_index=ignore_index, mdmc_average='samplewise')

    def metrics(self, is_train=True):
        if is_train:
            train_losses = AverageMeter('Training Loss', ':.4e')
            train_iou = AverageMeter('Training iou', ':6.2f')
            train_f_score = AverageMeter('Training F_score', ':6.2f')

            return train_losses, train_iou, train_f_score
        else:
            val_losses = AverageMeter('Validation Loss', ':.4e')
            val_iou = AverageMeter('Validation mean iou', ':6.2f')
            val_f_score = AverageMeter('Validation F_score', ':6.2f')

            return val_losses, val_iou, val_f_score

    def fit(self):

        self.logger.info("\nStart training\n\n")
        start_training_time = time()

        with mlflow.start_run():
            for e in range(self.epochs):
                start_training_epoch_time = time()
                self.model.train()
                train_losses_avg, train_iou_avg, train_f_score_avg = self.metrics(is_train=True)
                with tqdm.tqdm(self.train_loader, unit="batch") as tepoch:
                    tepoch.set_description(f"Epoch {e}")
                    for image, target in tepoch:
                        # Transfer Data to GPU if available
                        image = image.to(self.device)
                        target = target.to(self.device)
                        # Clear the gradients
                        self.optimizer.zero_grad()
                        # Forward Pass
                        # out = self.model(image)['out']
                        # if unet == true => remove ['out']
                        out = self.model(image)
                        # Find the Loss
                        loss = self.criterion(out, target)
                        # Calculate Loss
                        train_losses_avg.update(loss.item(), image.size(0))
                        # Calculate gradients
                        loss.backward()
                        # Update Weights
                        self.optimizer.step()

                        iou = self.train_iou(out.cpu(), target.cpu()).item()
                        train_iou_avg.update(iou)

                        f1_score = self.train_f1(out.cpu(), target.cpu()).item()
                        train_f_score_avg.update(f1_score)

                        tepoch.set_postfix(loss=train_losses_avg.avg,
                                           iou=train_iou_avg.avg,
                                           f_score=train_f_score_avg.avg)
                        if self.debug:
                            break

                self.tb_logger.log(log_type='criterion/training', value=train_losses_avg.avg, epoch=e)
                self.tb_logger.log(log_type='iou/training', value=train_iou_avg.avg, epoch=e)
                self.tb_logger.log(log_type='f_score/training', value=train_f_score_avg.avg, epoch=e)

                mlflow.log_metric('criterion/training', train_losses_avg.avg, step=e)
                mlflow.log_metric('iou/training', train_iou_avg.avg, step=e)
                mlflow.log_metric('f_score/training', train_f_score_avg.avg, step=e)

                end_training_epoch_time = time() - start_training_epoch_time
                print('\n')
                self.logger.info(
                    f'Training Results - [{end_training_epoch_time:.3f}s] Epoch: {e}:'
                    f' f_score: {train_f_score_avg.avg:.3f},'
                    f' IoU: {train_iou_avg.avg:.3f},'
                    f' Loss: {train_losses_avg.avg:.3f}')

                # validation step
                val_loss = self.evaluation(e)
                # apply scheduler
                if self.scheduler:
                    self.scheduler.step()
                # early stopping
                if self.early_stopping['init'] >= self.early_stopping['changed']:
                    self._early_stopping_model(val_loss=val_loss)
                else:
                    print(f'The model can not learn more, Early Stopping at epoch[{e}]')
                    break

                # save best model
                if self.best_model_path is not None:
                    self._best_model(val_loss=val_loss, path=self.best_model_path)

                # model check points
                if self.model_check_point_path is not None:
                    self.save_model_check_points(path=self.model_check_point_path, epoch=e, net=self.model,
                                                 optimizer=self.optimizer, loss=self.criterion,
                                                 avg_loss=train_losses_avg.avg)

                # log mlflow
                if self.scheduler:
                    mlflow.log_param("get_last_lr", self.scheduler.get_last_lr())
                    mlflow.log_param("scheduler", self.scheduler.state_dict())

                self.tb_logger.flush()
                if self.debug:
                    break

            end_training_time = time() - start_training_time
            print(f'Finished Training after {end_training_time:.3f}s')
            self.tb_logger.close()

    def evaluation(self, epoch):
        print('Validating...')
        start_validation_epoch_time = time()
        self.model.eval()  # Optional when not using Model Specific layer
        with torch.no_grad():
            val_losses_avg, val_iou_avg, val_f_score_avg = self.metrics(is_train=False)
            with tqdm.tqdm(self.val_loader, unit="batch") as tepoch:
                for image, target in tepoch:
                    # Transfer Data to GPU if available
                    image = image.to(self.device)
                    target = target.to(self.device)
                    # out = self.model(image)['out']
                    # if unet == true => remove ['out']
                    out = self.model(image)
                    # Find the Loss
                    loss = self.criterion(out, target)
                    # Calculate Loss
                    val_losses_avg.update(loss.item(), image.size(0))

                    iou = self.val_iou(out.cpu(), target.cpu()).item()
                    val_iou_avg.update(iou)

                    f1_score = self.val_f1(out.cpu(), target.cpu()).item()
                    val_f_score_avg.update(f1_score)

                    tepoch.set_postfix(loss=val_losses_avg.avg,
                                       iou=val_iou_avg.avg,
                                       f_score=val_f_score_avg.avg)
                    if self.debug:
                        break
            print('\n')
            self.tb_logger.log(log_type='criterion/validation', value=val_losses_avg.avg, epoch=epoch)
            self.tb_logger.log(log_type='iou/validation', value=val_iou_avg.avg, epoch=epoch)
            self.tb_logger.log(log_type='f_score/validation', value=val_f_score_avg.avg, epoch=epoch)

            mlflow.log_metric('criterion/validation', val_losses_avg.avg, step=epoch)
            mlflow.log_metric('iou/validation', val_iou_avg.avg, step=epoch)
            mlflow.log_metric('f_score/validation', val_f_score_avg.avg, step=epoch)

            end_validation_epoch_time = time() - start_validation_epoch_time
            self.logger.info(
                f'validation Results - [{end_validation_epoch_time:.3f}s] Epoch: {epoch}:'
                f' f_score: {val_f_score_avg.avg:.3f},'
                f' IoU: {val_iou_avg.avg:.3f},'
                f' Loss: {val_losses_avg.avg:.3f}')
            print('\n')
            return val_losses_avg.avg

    def _save_model(self, name, path, params):
        torch.save(params, path)

    def _early_stopping_model(self, val_loss):
        if self.best_loss < val_loss:
            self.early_stopping['changed'] += 1
        else:
            self.early_stopping['changed'] = 0

    def _best_model(self, val_loss, path):
        if self.best_loss > val_loss:
            self.best_loss = val_loss
            name = f'/best_model_loss_{self.best_loss:.2f}'.replace('.', '_')
            self._save_model(name, path=f'{path}/{name}.pt', params={
                'model_state_dict': self.model.state_dict(),
            })

            print(f'The best model is saved with criterion: {self.best_loss:.2f}')

    def save_model_check_points(self, path, epoch, net, optimizer, loss, avg_loss):
        name = f'/model_epoch_{epoch}_loss_{avg_loss:.2f}'.replace('.', '_')
        self._save_model(name, path=f'{path}/{name}.pt', params={
            'epoch': epoch,
            'model_state_dict': net.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'criterion': loss,
        })
        print(f'model checkpoint is saved at model_epoch_{epoch}_loss_{avg_loss:.2f}')

    def load_model_checkpoint(self, path):
        checkpoint = torch.load(path)
        self.model.load_state_dict(checkpoint['model_state_dict'])
        self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        epoch = checkpoint['epoch']
        self.criterion = checkpoint['criterion']

        return epoch

    def load_model(self, path):
        best_model = torch.load(path)
        self.model.load_state_dict(best_model['model_state_dict'])

Output:
The solution is:
mlflow.set_tracking_uri(uri=f'file://{hydra.utils.to_absolute_path("../output/mlruns")}')
exp = mlflow.get_experiment_by_name(name='Emegency_landing')
if not exp:
    experiment_id = mlflow.create_experiment(name='Emegency_landing',
                                                 artifact_location=f'file://{hydra.utils.to_absolute_path("../output/mlruns")}')
else:
    experiment_id = exp.experiment_id

And then you should pass the experiment Id to:
with mlflow.start_run(experiment_id=experiment_id):
     pass

If you don't mention the /path/mlruns, when you run the command of mlflow ui, it will create another folder automatically named mlruns. so, pay attention to this point to have the same name as mlruns.


Instruction: Efficient way to get "neuron-edge-neuron" values in a neural network

Input:
I'm working on a visual networks project where I'm trying to plot several node-edge-node values in an interactive graph.
I have several neural networks (this is one example):
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 1)

    def forward(self, x):
        x1 = self.fc1(x)
        x = torch.relu(x1)
        x2 = self.fc2(x)
        x = torch.relu(x2)
        x3 = self.fc3(x)
        return x3, x2, x1

net = Model()

How can I get the node-edge-node (neuron-edge-neuron) values in the network in an efficient way? Some of these networks have a large number of parameters. Note that for the first layer it will be input-edge-neuron rather than neuron-edge-neuron.
I tried saving each node values after the fc layers (ie x1,x2,x3) so I won't have to recompute them, but I'm not sure how to do the edges and match them to their corresponding neurons in an efficient way.
The output I'm looking for is a list of lists of node-edge-node values. Though it can also be a tensor of tensors if it's easier. For example, in the above network from the first layer I will have 2 triples (1x2), from the 2nd layer I will have 6 of them (2x3), and in the last layer I will have 3 triples (3x1). The issue is matching nodes (ie neurons) values (one from layer n-1 and one from layer n) with the corresponding edges in an efficient way.

Output:
Confession: Let's start by saying that I modified your code a bit to make it convenient. You can do everything in the form it originally was. I also changed the specific number of neurons just for playing around (I am sure you can revert them back).
I created a summary object (returned by .forward() function) that contains entire execution trace of the network, i.e. (input, weight, output) tuples for *every layer.
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(3, 5)
        self.fc2 = nn.Linear(5, 7)
        self.fc3 = nn.Linear(7, 2)

    def forward(self, x):
        summary = []
        running_x = x
        for layer in self.children():
            out = layer(running_x)
            # triplet of (input, weight, output) for each layer
            summary.append((running_x, layer.weight, out))
            running_x = out

        return summary

model = Model()
batch_size = 32
X = torch.rand(batch_size, 3)
summary = model(X)

The core logic is only this much
for L in summary: # iterate over the (ip, weight, out) tuple for each layer
    ip, weight, out = L # unpack them

    ip = ip[:, :, None, None].repeat(1, 1, out.shape[-1], 1)
    weight = weight.T[None, :, :, None].repeat(batch_size, 1, 1, 1)
    out = out[:, None, :, None].repeat(1, ip.shape[1], 1, 1)
    triplets = torch.cat([ip, weight, out], -1)

So the triplets variable (one for each layer) is all you are looking for. It has a size
(batch_size, layer_in_dim, layer_out_dim, 3)

Let's see specifically the triplets for first layer.
>> triplets.shape
(32, 3, 5, 3)

E.g., given a sample index b = 12, input neuron index i = 1 and output neuron index j = 3, you have exactly node-edge-node tuples
>> triplets[b][i][j]
tensor([0.7080, 0.3442, 0.7344], ...)

Verify: Let's manually verify the correctness.
The 12th sample's 1st dimension is
# Its the first layer we are looking, so input comes from user
>> X[12][1]
tensor(0.7080)

CHECK.
The connecting weight between 1st input neuron and 3rd output neuron for first layer
>> model.fc1.weight.T[1][3] # weight matrix is transposed, so had to do .T
tensor(0.3442, ...)

CHECK.
The output of 3rd neuron for 12th sample can be retrieved from its activation tensor
>> _, _, out = summary[0] # first layer's output tensor
>> out[12][3]
tensor(0.7344, ...)

ALSO CHECK.

I hope that's what you wanted. If anymore info/changed needed, feel free to comment. I don't think it can get any more efficient that that.


Instruction: OSError: [WinError 127] The specified procedure could not be found

Input:
While importing torch (import torch) I'm facing the following error message:
OSError: [WinError 127] The specified procedure could not be found. Error loading "C:\Users\myUserName\anaconda3\lib\site-packages\torch\lib\jitbackend_test.dll" or one of its dependencies.
I tried the suggestion from this article but without success.
Any ideas how to fix it?
My environment:

NVIDIA GeForce GTX 1650
Windows 11
Cuda 11.5
Conda 4.10.3
Python 3.8.5
Torch 1.10
Microsoft Visual C++ Redistributable installed (https://aka.ms/vs/17/release/vc_redist.x64.exe)

Output:
Fortunately, after extensive research, I found a solution.
Someone suggested me to create a new conda environment. And that worked for me!
Solution:

create new conda env by: conda create --name new-env
install python: conda install python=3.8.5
run: conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
test cuda: import torch; print(torch.version.cuda); print(torch.cuda.is_available())


Instruction: I can predict one image but not a set of images with a pytorch resnet18 model, how can i predict a set of images in a list using pytorch models?

Input:
x is a list of (36, 60, 3) images. I am trying to predict with a pytorch pretrained resnet18 the output on my images. I took x as a list of 2 images. when I take only 1 image, i get the prediction with no errors as it follows:
im = x[0]
preprocess = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )])
# Pass the image for preprocessing and the image preprocessed
img_preprocessed = preprocess(im)
# Reshape, crop, and normalize the input tensor for feeding into network for evaluation
batch_img_tensor = torch.unsqueeze(img_preprocessed, 0)
resnet18.eval()
out = resnet18(batch_img_tensor).flatten()

but it does not work when i set im=x. Something goes wrong in preprocessing line and I get this error:
TypeError: pic should be PIL Image or ndarray. Got <class 'list'>
I tried Variable (torch.tensot(x)) as follows :
x=dataset(source_p)
y=Variable(torch.tensor(x))
print(y.shape)
resnet18(y)

I get the following error :
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[2, 36, 60, 3] to have 3 channels, but got 36 channels instead

My question is : how can I predict all images in x list at once?
Thanks!

Output:
Eventually I created a class that takes x and transforms all elements :
class formDataset(Dataset):

    def __init__(self, imgs, transform=None):

        self.imgs = imgs
        self.transform = transform

    def __len__(self):
        return len(self.imgs)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        image = self.imgs[idx]
        sample = {image}

        if self.transform:
            sample = self.transform(sample)

        return sample

after I call
l_set=formDataset(imgs=x,transform=preprocess)
l_loader = DataLoader(l_set, batch_size=2)

for data in (l_loader):
     features=resnet(outputs)


Instruction: Check equality of any top k entries in rows of tensor A against argmax in rows of tensor B

Input:
New to tensors/pytorch.
I have two 2d tensors, A and B.
A contains floats that represent the probability assigned to a certain index. B contains a one-hot binary vector in the correct index.
A
tensor([[0.1, 0.4, 0.5],
        [0.5, 0.4, 0.1],
        [0.4, 0.5, 0.1]])

B
tensor([[0, 0, 1],
        [0, 1, 0],
        [0, 0, 1]])

I would like to find the number of rows where the index of any top-k values of A match the one-hot index in B. In this case, k=2.
My attempt:
tops = torch.topk(A, 2, dim=1)

top_idx = tops.indices

top_2_matches = torch.where((torch.any(top_idx, 1) == B.argmax(dim=1)))

If done properly, the example should return a tensor([0, 1]), since the first 2 rows have top-2 matches, but I get (tensor([1]),) as a return.
Not sure where I'm going wrong here. Thanks for any help!

Output:
Try this:
top_idx = torch.topk(A, 2, dim=1).indices

row_indicator = (top_idx == B.argmax(dim=1).unsqueeze(dim=1)).any(dim=1)

top_2_matches = torch.arange(len(row_indicator))[row_indicator]

For example:
>>> import torch
>>> A = torch.tensor([[0.1, 0.4, 0.5],
...                   [0.5, 0.4, 0.1],
...                   [0.4, 0.5, 0.1]])
>>> B = torch.tensor([[0, 0, 1],
...                   [0, 1, 0],
...                   [0, 0, 1]])
>>> tops = torch.topk(A, 2, dim=1)
>>>tops
torch.return_types.topk(
values=tensor([[0.5000, 0.4000],
               [0.5000, 0.4000],
               [0.5000, 0.4000]]),
indices=tensor([[2, 1],
                [0, 1],
                [1, 0]]))
>>> top_idx = tops.indices
>>> top_idx
tensor([[2, 1],
        [0, 1],
        [1, 0]])
>>> index_indicator = top_idx == B.argmax(dim=1).unsqueeze(dim=1)
>>> index_indicator
tensor([[ True, False],
        [False,  True],
        [False, False]])
>>> row_indicator = index_indicator.any(dim=1)
>>> row_indicator
tensor([ True,  True, False])
>>> top_2_matches = torch.arange(len(row_indicator))[row_indicator]
>>> top_2_matches
tensor([0, 1])


Instruction: cross_entropy_loss(): argument 'target' (position 2) must be Tensor, not numpy.ndarray

Input:
Data preparation, and building the model：
dataset = datasets.load_iris()
data = dataset.data
target = dataset.target
data_tensor=torch.from_numpy(data).float()
target_tensor=torch.from_numpy(target).long()

model = nn.Sequential(
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=4, out_features=100),
    nn.ReLU(),
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=100, out_features=3),
)
cross_entropy_loss = nn.CrossEntropyLoss()
klloss = bnn.BKLLoss(reduction='mean', last_layer_only=False)
klweight = 0.01
optimizer = optim.Adam(model.parameters(), lr=0.01)

Run the model: Error occurred
for step in range(3000):
    models = model(data_tensor)
    cross_entropy = cross_entropy_loss(models, target)
    kl = klloss(model)
    total_cost = cross_entropy + klweight*kl

    optimizer.zero_grad()
    total_cost.backward()
    optimizer.step()

_, predicted = torch.max(models.data, 1)
final = target_tensor.size(0)
correct = (predicted == target_tensor).sum()
print('- Accuracy: %f %%' % (100 * float(correct) / final))
print('- CE : %2.2f, KL : %2.2f' % (cross_entropy.item(), kl.item()))

TypeError
----> 3     cross_entropy = cross_entropy_loss(models, target)
TypeError: cross_entropy_loss(): argument 'target' (position 2) must be Tensor, not numpy.ndarray

I copied code from a web-learning page, but it shows the attached error, any suggestions? many thanks !!

Output:
You used the wrong variable for target.
cross_entropy_loss(models, target)

would be
cross_entropy_loss(models, target_tensor)


Instruction: Scikit learn train_test_split into Pytorch Dataloader

Input:
I have a dataset for binary classification with PNGs titled as in the attachment below, where the first 0 or 1 in the title determines its class. They're in a folder called "annotation_class", and I have a small script to separate these:
import cv2,glob
import numpy as np
from sklearn.model_selection import train_test_split

filelist = glob.glob('annotation_class'+'/*.png')
size_row, size_col = 256, 256
X,y = [],[]

for name in filelist:
        img = cv2.imread(name)
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        img = cv2.resize(img,(size_row, size_col))
        X.append(img)
        y.append(int(name.split('\\')[-1].split('_')[1]))

x_train, x_test, y_train, y_test= train_test_split(X, y, test_size=0.2, train_size=0.8, random_state=4)

The returns are all lists. I'm using Pytorch for this project and would like to make a custom Dataset to use Dataloader, but I'm not sure how best to include these after I've used train_test_split. Should I scrap that altogether and use something else? I'd like to end up with two DataLoader's for training and testing.

Output:
You don't have to rewrite. You can reuse your core data loading logic inside PyTorch Dataset
import cv2,glob
import numpy as np
from sklearn.model_selection import train_test_split

from torch.utils.data import Dataset

class MyCoolDataset(Dataset):

    def __init__(self, dir, train=True):
        filelist = glob.glob(dir + '/*.png')
        ...
        # all your data loading logic using cv2, glob ..
        x_train, x_test, y_train, y_test = train_test_split(X, y, test_size=0.2, train_size=0.8, random_state=4)

        # two modes - train and test
        if train:
            self.x_data, self.y_data = x_train, y_train
        else:
            self.x_data, self.y_data = x_test, y_test

    def __getitem__(self, i):
        return self.x_data[i], self.y_data[i]

Then use a DataLoader as usual
dl = DataLoader(MyCoolDataset(...), batch_size=...)
for X, Y in dl:
    pass


Instruction: IndexError: Target 11 is out of bounds. cross-entropy

Input:
How can I change the attached model to fit my dataset for the Bayesian model? my data include 5 variables and 32 results
model = nn.Sequential(
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=5, out_features=100),
    nn.ReLU(),
    bnn.BayesLinear(prior_mu=0, prior_sigma=0.1, in_features=100, out_features=3),
)

cross_entropy_loss = nn.CrossEntropyLoss()
klloss = bnn.BKLLoss(reduction='mean', last_layer_only=False)
klweight = 0.01
optimizer = optim.Adam(model.parameters(), lr=0.01)

Next is to run the model
for step in range(3000):
    models = model(data_tensor)
    cross_entropy = cross_entropy_loss(models, target_tensor)
    kl = klloss(model)
    total_cost = cross_entropy + klweight*kl

    optimizer.zero_grad()
    total_cost.backward()
    optimizer.step()

_,predicted = torch.max(models.data, 1)
final = target_tensor.size(0)
correct = (predicted == target_tensor).sum()
print('- Accuracy: %f %%' % (100 * float(correct) / final))
print('- CE : %2.2f, KL : %2.2f' % (cross_entropy.item(), kl.item()))

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_19752/2114183600.py in <module>
      1 for step in range(4000):
      2     models = model(data_tensor)
----> 3     cross_entropy = cross_entropy_loss(models, target_tensor)
      4     kl = klloss(model)
      5     total_cost = cross_entropy + klweight*kl

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used
   1104         full_backward_hooks, non_full_backward_hooks = [], []

~\anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
   1148
   1149     def forward(self, input: Tensor, target: Tensor) -> Tensor:
-> 1150         return F.cross_entropy(input, target, weight=self.weight,
   1151                                ignore_index=self.ignore_index, reduction=self.reduction,
   1152                                label_smoothing=self.label_smoothing)

~\anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)
   2844     if size_average is not None or reduce is not None:
   2845         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2846     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
   2847
   2848

IndexError: Target 11 is out of bounds.

data_tensor.shape:torch.Size([640, 5])
target_tensor.shape:  torch.Size([640])
data properties:
data_tensor=torch.from_numpy(data).float()
target_tensor=torch.from_numpy(target).float()
target_tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
        18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,  0,  1,  2,  3,
         4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
        22, 23, 24, 25, 26, 27, 28, 29, 30, 31,  0,  1,  2,  3,  4,  5,  6,  7,
         8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
        26, 27, 28, 29, 30, 31,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11,
        12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
        30, 31,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
        16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,  0,  1,
         2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,  0,  1,  2,  3,  4,  5,
         6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
        24, 25, 26, 27, 28, 29, 30, 31,  0,  1,  2,  3,  4,  5,  6,  7,  8,  9,
        10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
        28, 29, 30, 31])

I am not sure what's wrong with my target_sensor and cross_entropy_Loss. Target sensors are repeat numbers between 0-31, for 20 times

Output:
I met the same question with you, please check whether the output number in final Linear part of your model is equal to the num_classes.
I am not good at English, please ignore the gramme problem hhh


Instruction: Pytorch CNN: Expected input to have 1 channel but got 60000 channels instead

Input:
While implementing a NN for Fashion MNIST dataset, I'm getting the following error:
RuntimeError: Given groups=1, weight of size [6, 1, 5, 5], expected input[1, 60000, 28, 28] to have 1 channels, but got 60000 channels instead

I'm inferring that 60000 is the length of my entire dataset, but not sure why is the algorithm giving this error. Can someone help me fix this please?
My dataset:
(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()
train_data = []
test_data = []
train_data.append([X_train, y_train])
test_data.append([X_test, y_test])

trainloader = torch.utils.data.DataLoader(train_data, shuffle=True, batch_size=100)
testloader = torch.utils.data.DataLoader(test_data, shuffle=True, batch_size=100)

I'm getting the error in following order (as per stack trace):
      8     #making predictions
----> 9     y_pred = model(images)

     32     #first hidden layer
---> 33     x = self.conv1(x)

Update 1
Added the line:
images = images.transpose(0, 1)

to transpose the image as pointed out by Ivan but now getting the error:
RuntimeError: expected scalar type Byte but found Float

Output:
You input is shaped (1, 60000, 28, 28), while it should be shaped (60000, 1, 28, 28). You can fix this by transposing the first two axes:
>>> x.transpose(0, 1)


Instruction: What is the correct way to update an input variable during training?

Input:
I have an input
inp = torch.tensor([1.0])

and a neural network
class Model_updater(nn.Module):
    def __init__(self):
        super(Model_updater, self).__init__()
        self.fc1 = nn.Linear(1, 2)
        self.fc2 = nn.Linear(2, 3)
        self.fc3 = nn.Linear(3, 2)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net_updater = Model_updater()

opt_updater = optim.Adam(net_updater.parameters())

I'm trying to update my input using the neural network's output:
inp = torch.tensor([1.0])
epochs = 3

for i in range(epochs):
    opt_updater.zero_grad()

    inp_copy = inp.detach().clone()

    mu, sigma = net_updater(inp_copy)
    dist1 = Normal(mu, torch.abs(sigma))
    a = dist1.rsample()

    inp += a

    loss = torch.tensor(5.0) - inp

    loss.backward(retain_graph=True)
    opt_updater.step()

But getting the error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 2]], which is output 0 of TBackward, is at version 2; expected version 1

I also tried changing the loss calculations with
loss = torch.tensor(5.0) - inp_copy

But got the error
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I also tried without the retain_graph=True but I get
RuntimeError: Trying to backward through the graph a second time,
but the saved intermediate results have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the first time.

Which doesn't really makes sense to me because I don't see where I'm calling backward() twice

Output:
Most likely, this is what you want
inp1 = inp + a  # create a separate variable for updated value
inp.data = inp1.data # update the value without touching the graph

loss = torch.tensor(5.0) - inp1 # use updated value which has gradient


Instruction: How to convert horizontal bounding box to oriented bounding box in object detection task

Input:
I am trying to detect oriented bounding boxes with faster rcnn for a long time but I couldn't make it to do so. I aim to detect objects in DOTA dataset. I was using built-in faster rcnn model in pytorch but realized that it does not support OBB. Then I found another library named detectron2 that is built on the pytorch framework. Built-in faster rcnn network in detectron2 is actually compatible with OBB but I could not make that model work with DOTA. Because I could not convert DOTA box annotations to (cx, cy, w, h, a). In DOTA, objects are annotated by coordinates of 4 corners which are (x1,y1,x2,y2,x3,y3,x4,y4).
I cant come up with a solution that converts these 4 coordinates to (cx, cy, w, h, a), where cx and cy are the center point of OBB and w, h and a are width, height and angle respectively.
Is there any suggestion?

Output:
If you have your boxes in an Nx8 tensor/array, you can conver them to (cx, cy, w, h, a) by doing (assuming first point is top left, second point is bottom left, then bottom right, then top right...):
def DOTA_2_OBB(boxes):
    #calculate the angle of the box using arctan (degrees here)
    angle = (torch.atan((boxes[:,7] - boxes[:,5])/(boxes[:,6] - boxes[:,4]))*180/np.pi).float()
    #centrepoint is the mean of adjacent points
    cx = boxes[:,[4,0]].mean(1)
    cy = boxes[:,[7,3]].mean(1)
    #calculate w and h based on the distance between adjacent points
    w = ((boxes[:,7] - boxes[:,1])**2+(boxes[:,6] - boxes[:,0])**2)**0.5
    h = ((boxes[:,1] - boxes[:,3])**2+(boxes[:,0] - boxes[:,2])**2)**0.5
    return torch.stack([cx,cy,w,h,angle]).T

Then giving this a test...
In [40]: boxes = torch.tensor([[0,2,1,0,2,2,1,3],[4,12,8,2,12,12,8,22]]).float()

In [43]: DOTA_2_OBB(boxes)
Out[43]:
tensor([[  1.0000,   1.5000,   1.4142,   2.2361, -45.0000],
        [  8.0000,  12.0000,  10.7703,  10.7703, -68.1986]])


Instruction: Reshape tensor in custom order (PyTorch)

Input:
I have the following tensor
t = torch.tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 ,15, 16, 17])

I want to reshape it in the following way:
t_reshape = torch.tensor([[0, 1, 2, 6, 7, 8, 12, 13, 14],
                          [3, 4, 5, 9, 10, 11, 15, 16, 17]])

Are there ways to efficiently reshape tensors in that fashion?

Output:
You can achieve this by reshaping, transposing and reshaping back:
>>> t.reshape(3,2,-1).transpose(0,1).reshape(2,-1)
tensor([[ 0,  1,  2,  6,  7,  8, 12, 13, 14],
        [ 3,  4,  5,  9, 10, 11, 15, 16, 17]])


Instruction: PyTorch - RuntimeError: Sizes of tensors must match except in dimension 2. Got 55 and 54 (The offending index is 0)

Input:
I used a 3DUnet with resblock to segment a CT image with input torch size of [1, 1, 96, 176, 176], but it throws the following error:
RuntimeError: Sizes of tensors must match except in dimension 2. Got 55 and 54 (The offending index is 0)
Hence I traced back, I found the error comes from
outputs = self.decoder_stage2(torch.cat([short_range6, long_range3], dim=1)) + short_range6

The short_range6 has torch.Size([1, 64, 24, 55, 40]) while the long_range3 has torch.Size([1, 128, 24, 54, 40]). I think this is because something not being a power of 2, but cannot find where to modify.
Below is the complete structure of the network, really thanks for any help!
class ResUNet(nn.Module):
    def __init__(self, in_channel=1, out_channel=2 ,training=True):
        super().__init__()

        self.training = training
        self.dorp_rate = 0.2

        self.encoder_stage1 = nn.Sequential(
            nn.Conv3d(in_channel, 16, 3, 1, padding=1),
            nn.PReLU(16),

            nn.Conv3d(16, 16, 3, 1, padding=1),
            nn.PReLU(16),
        )

        self.encoder_stage2 = nn.Sequential(
            nn.Conv3d(32, 32, 3, 1, padding=1),
            nn.PReLU(32),

            nn.Conv3d(32, 32, 3, 1, padding=1),
            nn.PReLU(32),

            nn.Conv3d(32, 32, 3, 1, padding=1),
            nn.PReLU(32),
        )

        self.encoder_stage3 = nn.Sequential(
            nn.Conv3d(64, 64, 3, 1, padding=1),
            nn.PReLU(64),

            nn.Conv3d(64, 64, 3, 1, padding=2, dilation=2),
            nn.PReLU(64),

            nn.Conv3d(64, 64, 3, 1, padding=4, dilation=4),
            nn.PReLU(64),
        )

        self.encoder_stage4 = nn.Sequential(
            nn.Conv3d(128, 128, 3, 1, padding=3, dilation=3),
            nn.PReLU(128),

            nn.Conv3d(128, 128, 3, 1, padding=4, dilation=4),
            nn.PReLU(128),

            nn.Conv3d(128, 128, 3, 1, padding=5, dilation=5),
            nn.PReLU(128),
        )

        self.decoder_stage1 = nn.Sequential(
            nn.Conv3d(128, 256, 3, 1, padding=1),
            nn.PReLU(256),

            nn.Conv3d(256, 256, 3, 1, padding=1),
            nn.PReLU(256),

            nn.Conv3d(256, 256, 3, 1, padding=1),
            nn.PReLU(256),
        )

        self.decoder_stage2 = nn.Sequential(
            nn.Conv3d(128 + 64, 128, 3, 1, padding=1),
            nn.PReLU(128),

            nn.Conv3d(128, 128, 3, 1, padding=1),
            nn.PReLU(128),

            nn.Conv3d(128, 128, 3, 1, padding=1),
            nn.PReLU(128),
        )

        self.decoder_stage3 = nn.Sequential(
            nn.Conv3d(64 + 32, 64, 3, 1, padding=1),
            nn.PReLU(64),

            nn.Conv3d(64, 64, 3, 1, padding=1),
            nn.PReLU(64),

            nn.Conv3d(64, 64, 3, 1, padding=1),
            nn.PReLU(64),
        )

        self.decoder_stage4 = nn.Sequential(
            nn.Conv3d(32 + 16, 32, 3, 1, padding=1),
            nn.PReLU(32),

            nn.Conv3d(32, 32, 3, 1, padding=1),
            nn.PReLU(32),
        )

        self.down_conv1 = nn.Sequential(
            nn.Conv3d(16, 32, 2, 2),
            nn.PReLU(32)
        )

        self.down_conv2 = nn.Sequential(
            nn.Conv3d(32, 64, 2, 2),
            nn.PReLU(64)
        )

        self.down_conv3 = nn.Sequential(
            nn.Conv3d(64, 128, 2, 2),
            nn.PReLU(128)
        )

        self.down_conv4 = nn.Sequential(
            nn.Conv3d(128, 256, 3, 1, padding=1),
            nn.PReLU(256)
        )

        self.up_conv2 = nn.Sequential(
            nn.ConvTranspose3d(256, 128, 2, 2),
            nn.PReLU(128)
        )

        self.up_conv3 = nn.Sequential(
            nn.ConvTranspose3d(128, 64, 2, 2),
            nn.PReLU(64)
        )

        self.up_conv4 = nn.Sequential(
            nn.ConvTranspose3d(64, 32, 2, 2),
            nn.PReLU(32)
        )

        # 256*256
        self.map4 = nn.Sequential(
            nn.Conv3d(32, out_channel, 1, 1),
            nn.Upsample(scale_factor=(1, 1, 1), mode='trilinear', align_corners=False),
            nn.Softmax(dim=1)
        )

        # 128*128
        self.map3 = nn.Sequential(
            nn.Conv3d(64, out_channel, 1, 1),
            nn.Upsample(scale_factor=(2, 2, 2), mode='trilinear', align_corners=False),
            nn.Softmax(dim=1)
        )

        # 64*64
        self.map2 = nn.Sequential(
            nn.Conv3d(128, out_channel, 1, 1),
            nn.Upsample(scale_factor=(4, 4, 4), mode='trilinear', align_corners=False),

            nn.Softmax(dim=1)
        )

        # 32*32
        self.map1 = nn.Sequential(
            nn.Conv3d(256, out_channel, 1, 1),
            nn.Upsample(scale_factor=(8, 8, 8), mode='trilinear', align_corners=False),
            nn.Softmax(dim=1)
        )

    def forward(self, inputs):

        long_range1 = self.encoder_stage1(inputs) + inputs

        short_range1 = self.down_conv1(long_range1)

        long_range2 = self.encoder_stage2(short_range1) + short_range1
        long_range2 = F.dropout(long_range2, self.dorp_rate, self.training)

        short_range2 = self.down_conv2(long_range2)

        long_range3 = self.encoder_stage3(short_range2) + short_range2
        long_range3 = F.dropout(long_range3, self.dorp_rate, self.training)

        short_range3 = self.down_conv3(long_range3)

        long_range4 = self.encoder_stage4(short_range3) + short_range3
        long_range4 = F.dropout(long_range4, self.dorp_rate, self.training)

        short_range4 = self.down_conv4(long_range4)

        outputs = self.decoder_stage1(long_range4) + short_range4
        outputs = F.dropout(outputs, self.dorp_rate, self.training)

        output1 = self.map1(outputs)

        short_range6 = self.up_conv2(outputs)

        outputs = self.decoder_stage2(torch.cat([short_range6, long_range3], dim=1)) + short_range6
        outputs = F.dropout(outputs, self.dorp_rate, self.training)

        output2 = self.map2(outputs)

        short_range7 = self.up_conv3(outputs)

        outputs = self.decoder_stage3(torch.cat([short_range7, long_range2], dim=1)) + short_range7
        outputs = F.dropout(outputs, self.dorp_rate, self.training)

        output3 = self.map3(outputs)

        short_range8 = self.up_conv4(outputs)

        outputs = self.decoder_stage4(torch.cat([short_range8, long_range1], dim=1)) + short_range8

        output4 = self.map4(outputs)

        if self.training is True:
            return output1, output2, output3, output4
        else:
            return output4```

Output:
You can pad your image's dimensions to be multiple of 32's. By doing this, you won't have to change the 3DUnet's parameters.
I will provide you a simple code to show you the way.
# I assume that you named your input image as img

padding1_mult = math.floor(img.shape[3] / 32) + 1
padding2_mult = math.floor(img.shape[4] / 32) + 1
pad1 = (32 * padding1_mult) - img.shape[3]
pad2 = (32 * padding2_mult) - img.shape[4]

padding = nn.ReplicationPad2d((0, pad2, pad1, 0, 0 ,0))

img = padding(img)

After this operation, your image shape must be torch.Size([1, 1, 96, 192, 192])


Instruction: Reshaping a PyTorch tensor to 3 dimensions when it is originally 2 dimensions?

Input:
I would like to take a PyTorch tensor that I have, originally of shape torch.Size([15000, 23]) and reshape it such that it is compatible to run in spiking neural network (snnTorch is the framework I am using in PyTorch). The shape of the tensor to input into the SNN should [time x batch_size x feature_dimensions] (more information on this can be found here.
Right now, I am using the following code:
    # Create data of dimensions [time x batch_size x feature_dimensions]
    time_steps = 200
    batch_size = 1
    feature_dimensions = torch_input_tensor.size(dim = 1)
    torch_input_tensor_reshaped = torch.reshape(torch_input_tensor, (time_steps, batch_size, feature_dimensions))
    print(torch_input_tensor_reshaped.size())
    print(torch_input_tensor_reshaped)

When I run this code, I get the following error:
RuntimeError: shape '[200, 1, 23]' is invalid for input of size 345000

I may be using the wrong function to do this, but the idea is that I currently have 15000 data points, and 23 input features. I want to essentially feed in the same data point (23 features, 1 data point) 200 times (200 time steps).
In the example provided in the link, the use the following code:
spk_in = spikegen.rate_conv(torch.rand((200, 784))).unsqueeze(1)
The unsqueeze function is for the input along dim=1 to indicate 'one batch' of data.
How can I make my data shape compatible to run in an SNN?

Output:
The thing with SNNs is that they are time-varying, so if your data is time-static, then your options are either to:

pass the same sample at every time step to the network, or
convert it into a spike-train before passing it in.

You appear to be going for (2), although (1) might be easier.
During training, you would pass the same sample to the network over and over again:
for step in range(num_steps):
    cur1 = self.fc1(x)

If your input was time varying, you would have to change x to x[step] to iterate through each time step. An example of this with MNIST is given here.
If the above code doesn't help, then it'd be useful to see how you define your network. Try something like:
# Define Network
class Net(nn.Module):
    def __init__(self):
        super().__init__()

        # Initialize layers
        self.fc1 = nn.Linear(23, 100) # 23 inputs, 100 hidden neurons
        self.lif1 = snn.Leaky(beta=0.9) # randomly chose 0.9
        self.fc2 = nn.Linear(100, num_outputs) # change num_outputs to your number of classes
        self.lif2 = snn.Leaky(beta=0.9)

    def forward(self, x):

        # Initialize hidden states at t=0
        mem1 = self.lif1.init_leaky()
        mem2 = self.lif2.init_leaky()

        # Record the final layer
        spk2_rec = []
        mem2_rec = []

        for step in range(num_steps):
            cur1 = self.fc1(x)
            spk1, mem1 = self.lif1(cur1, mem1)
            cur2 = self.fc2(spk1)
            spk2, mem2 = self.lif2(cur2, mem2)
            spk2_rec.append(spk2)
            mem2_rec.append(mem2)

        return torch.stack(spk2_rec, dim=0), torch.stack(mem2_rec, dim=0)


Instruction: Using multiprocessing with AllenNLP decoding is sluggish compared to non-multiprocessing case

Input:
I'm using the AllenNLP (version 2.6) semantic role labeling model to process a large pile of sentences. My Python version is 3.7.9. I'm on MacOS 11.6.1. My goal is to use multiprocessing.Pool to parallelize the work, but the calls via the pool are taking longer than they do in the parent process, sometimes substantially so.
In the parent process, I have explicitly placed the model in shared memory as follows:
from allennlp.predictors import Predictor
from allennlp.models.archival import load_archive
import allennlp_models.structured_prediction.predictors.srl
PREDICTOR_PATH = "...<srl model path>..."

archive = load_archive(PREDICTOR_PATH)
archive.model.share_memory()
PREDICTOR = Predictor.from_archive(archive)

I know the model is only being loaded once, in the parent process. And I place the model in shared memory whether or not I'm going to make use of the pool. I'm using torch.multiprocessing, as many recommend, and I'm using the spawn start method.
I'm calling the predictor in the pool using Pool.apply_async, and I'm timing the calls within the child processes. I know that the pool is using the available CPUs (I have six cores), and I'm nowhere near running out of physical memory, so there's no reason for the child processes to be swapped to disk.
Here's what happens, for a batch of 395 sentences:

Without multiprocessing: 638 total processing seconds (and elapsed time).
With a 4-process pool: 293 seconds elapsed time, 915 total processing seconds.
With a 12-process pool: 263 seconds elapsed time, 2024 total processing seconds.

The more processes, the worse the total AllenNLP processing time - even though the model is explicitly in shared memory, and the only thing that crosses the process boundary during the invocation is the input text and the output JSON.
I've done some profiling, and the first thing that leaps out at me is that the function torch._C._nn.linear is taking significantly longer in the multiprocessing cases. This function takes two tensors as arguments - but there are no tensors being passed across the process boundary, and I'm decoding, not training, so the model should be entirely read-only. It seems like it has to be a problem with locking or competition for the shared model resource, but I don't understand at all why that would be the case. And I'm not a torch programmer, so my understanding of what's happening is limited.
Any pointers or suggestions would be appreciated.

Output: Turns out that I wasn't comparing exactly the right things. This thread: https://github.com/allenai/allennlp/discussions/5471 goes into all the detail. Briefly, because pytorch can use additional resources under the hood, my baseline test without multiprocessing wasn't taxing my computer enough when running two instances in parallel; I had to run 4 instances to see the penalty, and in that case, the total processing time was essentially the same for 4 parallel nonmultiprocessing invocations, or one multiprocessing case with 4 subprocesses.


Instruction: Map each input element to array in pytorch

Input:
I have a (A,B) tensor, and I'm looking for a performant way to map each value from that tensor to an array to create a new tensor of size (A,B,N). Here's a functioning piece of code showing what I'm trying to do.
A, B, N = 3, 4, 5
my_old_tensor = torch.ones((A,B), dtype=torch.float32)
my_new_tensor = torch.zeros((A, B, N), dtype=torch.float32)
for val in range(N):
    my_new_tensor[:,:,val] = (val - my_old_tensor)/2

My code is currently quite slow, and I think the for-loop is the problem. Is there a more pytorch-performant way of doing this that eliminates the for-loop? I've tried something like this
x = torch.arange(0, N, 1, dtype=torch.float32)
my_new_tensor = (x - my_old_tensor)/2

but that gives "RuntimeError: The size of tensor a (5) must match the size of tensor b (4) at non-singleton dimension 1"
Any help would be appreciated!

Output:
Use unsqueeze to broadcast my_old_tensor:
my_new_tensor = (torch.arange(N, dtype=torch.float32) - my_old_tensor.unsqueeze(-1))/2


Instruction: Using PyTorch tensors with scikit-learn

Input:
Can I use PyTorch tensors instead of NumPy arrays while working with scikit-learn?
I tried some methods from scikit-learn like train_test_split and StandardScalar, and it seems to work just fine, but is there anything I should know when I'm using PyTorch tensors instead of NumPy arrays?
According to this question on https://scikit-learn.org/stable/faq.html#how-can-i-load-my-own-datasets-into-a-format-usable-by-scikit-learn :

numpy arrays or scipy sparse matrices. Other types that are convertible to numeric arrays such as pandas DataFrame are also acceptable.

Does that mean using PyTorch tensors is completely safe?

Output:
I don't think PyTorch tensors are directly supported by scikit-learn. But you can always get the underlying numpy array from PyTorch tensors
my_nparray = my_tensor.numpy()

and then use it with scikit learn functions.


Instruction: Installed PyTorch but VS code wont import torch

Input:
I have installed PyTorch by just using pip install torch. I also have the correct version of python installed (I don't have two different versions).
When I ran the following in VS code it returned the correct version, and when I check if PyTorch is installed with pip it works.

import torch
print(torch.__version__)

But for some reason VS code doesn't recognise torch when I try and import it, or try to inheret from nn.Module in a class.
I just get the error "Import torch could not be resolved" and "nn is not defined"
I'm really confused as to what to do as I cant find any other people having this issue and all PyTorch VS code examples I look at just install the python extension and have no issue.

Output:
Check if vscode is using the same python interpreter and environment in which pytorch was installed.
Hit cmd + shift + P and search for Interpreter. Click on Python Interpreter and choose the correct one.
Check the image shown below


Instruction: Error while using Vgg16 (transfter learning) - RuntimeError: Failed to run torchinfo

Input:
I'm trying to use VGG16 with transfer learning, but getting errors:
model = torchvision.models.vgg16(pretrained=True)
print(model)
for param in model.parameters():
    param.requires_grad = False

input_size             = model.classifier[0].in_features
model.classifier[0]   = nn.Sequential(
    nn.Linear(input_size, 128), nn.ReLU(),
    nn.Linear(128, 2))

torchinfo.summary(model, (64, 3, 224, 224))

VGG16:
VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(7, 7))
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace=True)
    (2): Dropout(p=0.5, inplace=False)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace=True)
    (5): Dropout(p=0.5, inplace=False)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

Error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/torchinfo/torchinfo.py in forward_pass(model, x, batch_dim, cache_forward_pass, device, **kwargs)
    260             if isinstance(x, (list, tuple)):
--> 261                 _ = model.to(device)(*x, **kwargs)
    262             elif isinstance(x, dict):

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1101                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1102             return forward_call(*input, **kwargs)
   1103         # Do not call functions when jit is used

~/.local/lib/python3.8/site-packages/torchvision/models/vgg.py in forward(self, x)
     51         x = torch.flatten(x, 1)
---> 52         x = self.classifier(x)
     53         return x

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1119
-> 1120         result = forward_call(*input, **kwargs)
   1121         if _global_forward_hooks or self._forward_hooks:

~/.local/lib/python3.8/site-packages/torch/nn/modules/container.py in forward(self, input)
    140         for module in self:
--> 141             input = module(input)
    142         return input

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1119
-> 1120         result = forward_call(*input, **kwargs)
   1121         if _global_forward_hooks or self._forward_hooks:

~/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py in forward(self, input)
    102     def forward(self, input: Tensor) -> Tensor:
--> 103         return F.linear(input, self.weight, self.bias)
    104

~/.local/lib/python3.8/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-> 1848     return torch._C._nn.linear(input, weight, bias)
   1849

RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x2 and 4096x4096)

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_8204/406510959.py in <module>
     11     nn.Linear(128, 2))
     12
---> 13 torchinfo.summary(model, (64, 3, 224, 224))

~/.local/lib/python3.8/site-packages/torchinfo/torchinfo.py in summary(model, input_size, input_data, batch_dim, cache_forward_pass, col_names, col_width, depth, device, dtypes, row_settings, verbose, **kwargs)
    192         input_data, input_size, batch_dim, device, dtypes
    193     )
--> 194     summary_list = forward_pass(
    195         model, x, batch_dim, cache_forward_pass, device, **kwargs
    196     )

~/.local/lib/python3.8/site-packages/torchinfo/torchinfo.py in forward_pass(model, x, batch_dim, cache_forward_pass, device, **kwargs)
    268     except Exception as e:
    269         executed_layers = [layer for layer in summary_list if layer.executed]
--> 270         raise RuntimeError(
    271             "Failed to run torchinfo. See above stack traces for more details. "
    272             f"Executed layers up to: {executed_layers}"

RuntimeError: Failed to run torchinfo. See above stack traces for more details. Executed layers up to: [Sequential: 1, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, MaxPool2d: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, MaxPool2d: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, MaxPool2d: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, MaxPool2d: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, Conv2d: 2, ReLU: 2, MaxPool2d: 2, AdaptiveAvgPool2d: 1, Sequential: 2, Linear: 3, ReLU: 3, Linear: 3, ReLU: 2, Dropout: 2]

I'm using the following pytorch packages versions:
torch==1.10.0
torchinfo==1.5.3
torchvision==0.11.1

What is wrong ?
WHat do I need to change in order to use VGG16 (with transfer learning) ?

Output:
In case you're trying to change the final classifier, you should change the whole, not only one layer:
model.classifier   = nn.Sequential(
    nn.Linear(input_size, 128), nn.ReLU(),
    nn.Linear(128, 2))


Instruction: Understanding key_dim and num_heads in tf.keras.layers.MultiHeadAttention

Input:
For example, I have input with shape (1, 1000, 10) (so, src.shape wil be (1, 1000, 10)). Then:

This works

class Model(tf.keras.Model):
        def __init__(self):
            super(Model, self).__init__()
            self.attention1 = tf.keras.layers.MultiHeadAttention(num_heads=20, key_dim=9)
            self.dense = tf.keras.layers.Dense(10, activation="softmax")

        def call(self, src):
            output = self.attention1(src, src)
            output = tf.reshape(output, [1, 10000])
            output = self.dense(output)
            return output

And this:

class Model(tf.keras.Model):
        def __init__(self):
            super(Model, self).__init__()
            self.attention1 = tf.keras.layers.MultiHeadAttention(num_heads=123, key_dim=17)
            self.dense = tf.keras.layers.Dense(10, activation="softmax")

        def call(self, src):
            output = self.attention1(src, src)
            output = tf.reshape(output, [1, 10000])
            output = self.dense(output)
            return output

So, this layer works with whatever num_heads and key_dim but secuence length (i.e. 1000) should be divisible by num_heads. WHY? Is it a bug? For example, the same code for Pytorch doesn't work. Also, what is a key_dim then... Thanks in advance.

Output:
There are two dimensions d_k and d_v in the original paper.

key_dim corresponds to d_k, which is the size of the key and query dimensions for each head. d_k can be more or less than d_v.

d_v = embed_dim/num_head. d_v is the size of the value for each head.

In their paper, Vaswani et al. set d_k = d_v. This, however, is not required. Conceptually, you can have d_k << d_v or even d_k >> d_v. In the former, you will have dimensionality reduction for each key/query in each head and in the latter, you will have dimensionality expansion for each key/query in each attention head. The change in dimension is transparently handled in the dimensionality of the weight matrix that is multiplied into each query/key/value.


Instruction: freezing layers in a neural network in pytorch

Input:
I have a cascaded neural network whereby the the output of first network become the input of second network. The first neural network is pretrained so I just initialise it with those pretrained weights. However, I want to freeze the first neural network so that when training its only updating weights of the second neural network. How can I do that? My network looks like:

###First network

class LambdaBase(nn.Sequential):
    def __init__(self, fn, *args):
        super(LambdaBase, self).__init__(*args)
        self.lambda_func = fn

    def forward_prepare(self, input):
        output = []
        for module in self._modules.values():
            output.append(module(input))
        return output if output else input

class Lambda(LambdaBase):
    def forward(self, input):
        return self.lambda_func(self.forward_prepare(input))

class LambdaMap(LambdaBase):
    def forward(self, input):
        return list(map(self.lambda_func,self.forward_prepare(input)))

class LambdaReduce(LambdaBase):
    def forward(self, input):
        return reduce(self.lambda_func,self.forward_prepare(input))

def get_first_model(load_weights = True):
    pretrained_model_reloaded_th = nn.Sequential( # Sequential,
        nn.Conv2d(4,300,(19, 1)),
        nn.BatchNorm2d(300),
        nn.ReLU(),
        nn.MaxPool2d((3, 1),(3, 1)),
        nn.Conv2d(300,200,(11, 1)),
        nn.BatchNorm2d(200),
        nn.ReLU(),
        nn.MaxPool2d((4, 1),(4, 1)),
        nn.Conv2d(200,200,(7, 1)),
        nn.BatchNorm2d(200),
        nn.ReLU(),
        nn.MaxPool2d((4, 1),(4, 1)),
        Lambda(lambda x: x.view(x.size(0),-1)), # Reshape,
        nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(2000,1000)), # Linear,
        nn.BatchNorm1d(1000,1e-05,0.1,True),#BatchNorm1d,
        nn.ReLU(),
        nn.Dropout(0.3),
        nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(1000,1000)), # Linear,
        nn.BatchNorm1d(1000,1e-05,0.1,True),#BatchNorm1d,
        nn.ReLU(),
        nn.Dropout(0.3),
        nn.Sequential(Lambda(lambda x: x.view(1,-1) if 1==len(x.size()) else x ),nn.Linear(1000,164)), # Linear,
        nn.Sigmoid(),
    )
    if load_weights:
        sd = torch.load('pretrained_model.pth')
        pretrained_model_reloaded_th.load_state_dict(sd)
    return  pretrained_model_reloaded_th

### second network

def next_model_architecture():
    next_model = nn.Sequential(
    nn.Linear(164, 64),
    nn.ReLU(),
    nn.Linear(64, 1),
    nn.Sigmoid())

    return next_model

### joining two networks
def cascading_model(first_model,next_model):
    network = nn.Sequential(first_model, next_model)
    return network

first_model = get_first_model(load_weights = True)
next_model = next_model_architecture()
network = cascading_model(first_model,next_model)

If I do:

first_model = first_model.eval()

Will this freeze my first neural network and only update weights of second network during training?

Output:
Freezing any parameter is done by setting it's .requires_grad to False. Do so by iterating over all parameters of the module (that you want to freeze)
for p in first_model.parameters():
    p.requires_grad = False


Instruction: Deploy pytorch .pth model in a python script

Input:
After successfully training my yolact model using a custom dataset I'm happy with the inference results outputted by eval.py using this command from anaconda terminal:
python eval.py --trained_model=./weights/yolact_plus_resnet50_abrasion_39_10000.pth --config=yolact_resnet_abrasion_config --score_threshold=0.8 --top_k=15 --images=./images:output_images

Now I want to run this inference from my own python script instead of using the anaconda terminal.
I wanna be able to get the bounding boxes of detections made on webcam frames obtained by this code below. Any idea ?
import cv2

src = cv2.VideoCapture(0)

while True:
    ret, frame = src.read()
    cv2.imshow('frame', frame)
    key = cv2.waitKey(5)
    if key == (27):
        break

The eval.py code is here at Yolact repository https://github.com/dbolya/yolact/blob/master/eval.py

Output:
I will just write the pseudocode here for you.
Step 1: Try loading the model using the lines starting from here and ending here
Step 2: Use this function for evaluation. Instead of cv2.imread, you just need to send your frame
Step 3:  Follow this function to get the bounding boxes. Especially this line. Just trackback the 't' variable and you will get your bounding boxes.
Hope it helps. Let me know if you need more clarification.


Instruction: How to transform output of neural network and still train?

Input:
I have a neural network which outputs output. I want to transform output before the loss and backpropogation happen.
Here is my general code:
with torch.set_grad_enabled(training):
                  outputs = net(x_batch[:, 0], x_batch[:, 1]) # the prediction of the NN
                  # My issue is here:
                  outputs = transform_torch(outputs)
                  loss = my_loss(outputs, y_batch)

                  if training:
                      scheduler.step()
                      loss.backward()
                      optimizer.step()

I have a transformation function which I put my output through:
def transform_torch(predictions):
    torch_dimensions = predictions.size()
    torch_grad = predictions.grad_fn
    cuda0 = torch.device('cuda:0')
    new_tensor = torch.ones(torch_dimensions, dtype=torch.float64, device=cuda0, requires_grad=True)
    for i in range(int(len(predictions))):
      a = predictions[i]
      # with torch.no_grad(): # Note: no training happens if this line is kept in
      new_tensor[i] = torch.flip(torch.cumsum(torch.flip(a, dims = [0]), dim = 0), dims = [0])
    return new_tensor

My problem is that I get an error on the next to last line:
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

Any suggestions? I have already tried using "with torch.no_grad():" (commented), but this results in very poor training and I believe that the gradients don't backpropogate properly after the transformation function.
Thanks!

Output:
The error is quite correct about what the issue is - when you create a new tensor with requires_grad = True, you create a leaf node in the graph (just like parameters of a model) and not allowed to do in-place operation on it.
The solution is simple, you do not need to create the new_tensor in advance. It is not supposed to be a leaf node; just create it on the fly
new_tensor = [ ]
for i in range(int(len(predictions))):
    a = predictions[i]
    new_tensor.append(torch.flip(torch.cumsum(torch.flip(a, ...), ...), ...))

new_tensor = torch.stack(new_tensor, 0)

This new_tensor will inherit all properties like dtype, device from predictions and will have require_grad = True already.


Instruction: How to change my model's parameters manually?

Input:
I have a model:
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 3)
        self.fc2 = nn.Linear(3, 2)
        self.fc3 = nn.Linear(2, 1)

    def forward(self, x):
        x1 = self.fc1(x)
        x = torch.relu(x1)
        x2 = self.fc2(x)
        x = torch.relu(x2)
        x3 = self.fc3(x)
        return x3, x2, x1

net = Model()

I'm trying to manually update the parameters with
i, j = torch.meshgrid(torch.arange(3), torch.arange(2))
i = i.reshape(-1)
j = j.reshape(-1)
update = torch.ones(6,1)
print(i)
print(j)
print(update.squeeze())
print(net.fc2.weight[j,i].data)
net.fc2.weight[j,i].data += update.squeeze()
print(net.fc2.weight[j,i].data)

>>> tensor([0, 0, 1, 1, 2, 2])
    tensor([0, 1, 0, 1, 0, 1])
    tensor([1., 1., 1., 1., 1., 1.])
    tensor([-0.0209, -0.3770,  0.4982, -0.2123, -0.2630, -0.5580])
    tensor([-0.0209, -0.3770,  0.4982, -0.2123, -0.2630, -0.5580])

But nothing seems to change.
However, if I do
print(net.fc2.weight[1].data)
net.fc2.weight[1].data += 1
print(net.fc2.weight[1].data)

>>> tensor([-0.3770, -0.2123, -0.5580])
    tensor([0.6230, 0.7877, 0.4420])

They do change.
What am I doing wrong in the first approach and how can I make it work?

Output:
The point you are missing is simple: when you do a "constant indexing", you get a "view" of the tensor, otherwise (i.e. indexing with another tensor) you get a new tensor or a new node in the computation graph.
PyTorch provides a .data_ptr() method to peek into the underlying memory pointer.
>> net.fc2.weight.data.data_ptr()
2911054070464

>> net.fc2.weight[1].data.data_ptr()
2911054070464

Constant indexing did not change the underlying raw data. However, indexing with a tensor creates a new node and hence a new underlying raw memory location
>> net.fc2.weight[j, i].data.data_ptr()
2911054068672

So, in your case, you are creating a new tensor/node with net.fc2.weight[j,i] and assigning new value to it. That's why your original tensor remains unchanged. In the constant indexing case, you are changing the same memory location, hence the change is reflected.

To fix your problem, instead of doing this
net.fc2.weight[j,i].data += update.squeeze()

do this
net.fc2.weight.data[j,i] += update.squeeze()

.. essentially grabbing the underlying .data first and then indexing it, which means the indexing operation is entirely out of autograd's tracking machinery.


Instruction: How to index tensor with high dimensional tensor

Input:
I have two tensors x for values and y for indexing. x.shape and y.shape are the same except the last dimension.
For example:
x=torch.tensor([[1, 6, 7, 5, 6],
        [8, 6, 7, 8, 4],
        [2, 8, 3, 5, 6]])
# x.shape:(3,5)
y=torch.tensor([[1, 2],
        [2, 3],
        [2, 2]])
# y.shape:(3,2)

Is there a simple way to slice it x[y]so that the result is:
torch.tensor([6,7],[7,8],[3,3])

What if x and y are higher dimensional tensors:
x.shape=(a,b,c,d)
y.shape=(a,b,c,e)
# a, b, c, d, e are positive integer

Output:
That's exactly what torch.gather does.
>> torch.gather(x, -1, y) # index along the last dimension
tensor([[6, 7],
        [7, 8],
        [3, 3]])

The exact same command works with your high-dimensional case as well.


Instruction: Calling a custom module in pythorch with two parameters

Input:
I've tried to create three custom modules as follows:
import torch

class VerySimple(torch.nn.Module):
  def __init__(self):
    super(VerySimple, self).__init__()

  def forward(self, x):
    return x * 3.0

class VerySimple2(torch.nn.Module):
  def __init__(self):
    super(VerySimple, self).__init__()

  def forward(self, x, y):
    return x * y * 3.0

After that I created two very simple networks as such:
vs = VerySimple()
vs2 = VerySimple2()
print(vs(2.0))
print(vs2(2.0, 3.0))

The examples work as I expect when I call this which outputs 6.0 and 18.0
Now I try to create something a little more interesting like so:
class Simple2(torch.nn.Module):
  def __init__(self):
    super(Simple2, self).__init__()
    self.model1 = torch.nn.Sequential(
        torch.nn.Linear(1, 3),
        torch.nn.ReLU(),
        torch.nn.Linear(3, 1)
    )
    self.model2 = torch.nn.Sequential(
        torch.nn.Linear(1, 3),
        torch.nn.ReLU(),
        torch.nn.Linear(3, 1)
    )

  def forward(self, x, y):
    x1 = self.model1(x)
    y2 = self.model2(y)
    return torch.cat((x1,y2),1)

But now when I get an "AttributeError" with the code below:
s2 = Simple2()
s2(2,3)

What am I doing wrong with the s2(2,3)?
Alternatively: What is the minimal working example with s2(2,3)?
As requested I add the full log here:
6.0
18.0
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-f3b0dc51220a> in <module>
     43
     44 s2 = Simple2()
---> 45 s2(2,3)

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-3-f3b0dc51220a> in forward(self, x, y)
     38
     39   def forward(self, x, y):
---> 40     x1 = self.model1(x)
     41     y2 = self.model2(y)
     42     return torch.cat((x1,y1),1)

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

/opt/app-root/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1686         if any([type(t) is not Tensor for t in tens_ops]) and has_torch_function(tens_ops):
   1687             return handle_torch_function(linear, tens_ops, input, weight, bias=bias)
-> 1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
   1690         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'int' object has no attribute 'dim'

I tried the example with tensors below from Tamir as such:
x = torch.tensor([2.0])
y = torch.tensor([3.0])
s2(x,y)

But I end up with this error instead:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-8-c61a0803c9b9> in <module>
     43
     44 s2 = Simple2()
---> 45 s2(torch.tensor([2.0]), torch.tensor([3.0]))
     46

/opt/app-root/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-8-c61a0803c9b9> in forward(self, x, y)
     40     x1 = self.model1(x)
     41     y2 = self.model2(y)
---> 42     return torch.cat((x1,y2),1)
     43
     44 s2 = Simple2()

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Last NOTE:
I had to modify the Simple2 example to this instead to get it to work with Tamir's solution:
class Simple2(torch.nn.Module):
  def __init__(self):
    super(Simple2, self).__init__()
    self.model1 = torch.nn.Sequential(
        torch.nn.Linear(1, 3),
        torch.nn.ReLU(),
        torch.nn.Linear(3, 1)
    )
    self.model2 = torch.nn.Sequential(
        torch.nn.Linear(1, 3),
        torch.nn.ReLU(),
        torch.nn.Linear(3, 1)
    )

  def forward(self, x, y):
    x1 = self.model1(x)
    y2 = self.model2(y)
    # replaced this with row below: return torch.cat((x1,y2),1)
    return x1 + y2

Output:
This is probably a type issue, Pytorch Linear and ReLU layer expect Tensors as inputs and your are passing integers.
Do something like
x = torch.tensor([2])
y = torch.tensor([3])
s2(x,y)


Instruction: Error: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Input:
I am trying to train a DNN model using pytorch, and I want to use GPU to train my model. I am able to successfully copy my model to the GPU using model.to(device), where device = cuda:0.
However, the standard methods for copying input to the GPU, (RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same), that is, X.to(device) and X.cuda() does not give me the desired output. Following is the method I am currently implementing:
def train_loop(self, dataloader, device):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        print(device)
        X.to(device)
        print(X.is_cuda)
        y.to(device)

        pred = self.model(X)
        loss = self.loss_fn(pred, y)

On printing the device value print(device) it shows as: cuda:0. But when I run print(X.is_cuda) it returns false. (Screenshot attached below).

Please let me know where I am going wrong. Thank you!

Output:
X.to(device) does nothing.
change it to:
x=x.to(device)

Of course this should be done to any parameter\variable you want on the GPU