Instruction: AttributeError: 'tuple' object has no attribute 'train_dataloader'

Input:
I have a 3 file. In the datamodule file, I have created data and used the basic format of the PyTorch Lightning. In the linear_model I made a linear regression model based on this page. Finally, I have a train file, I am calling the model and trying to fit the data. But I am getting this error
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/test_train.py", line 10, in <module>
    train_dataloader=datamodule.DataModuleClass().setup().train_dataloader(),
AttributeError: 'tuple' object has no attribute 'train_dataloader'

Sample datamodule file
class DataModuleClass(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.sigma = 5
        self.batch_size = 10
        self.prepare_data()

def prepare_data(self):
    x = np.random.uniform(0, 10, 10)
    e = np.random.normal(0, self.sigma, len(x))

    y = x + e

    X = np.transpose(np.array([x, e]))

    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(y).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)
    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [8, 2])
    return self.train_data, self.val_data

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)

Sample training file
from . import datamodule, linear_model

model = linear_model.LinearRegression(input_dim=2, l1_strength=1, l2_strength=1)

trainer = pl.Trainer()
trainer.fit(model,
            train_dataloader=datamodule.DataModuleClass().setup().train_dataloader(),
            val_dataloaders=datamodule.DataModuleClass().setup().val_dataloaders())

Let me know if you need more code or explanation.
Update (Based on the comment)
Now, I am getting the following error after removing self.prepare_data() from the __init__() of the DataModuleClass(), removed return self.train_data, self.val_data from setup(), and changed the test file to
data_module = datamodule.DataModuleClass()

trainer = pl.Trainer()
trainer.fit(model,data_module)

Error:
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/test_train.py", line 10, in <module>
    train_dataloader=datamodule.DataModuleClass().train_dataloader(),
  File "/home/mostafiz/Dropbox/MSc/Thesis/regreesion_EC/src/datamodule.py", line 54, in train_dataloader
    return DataLoader(self.train_data)
AttributeError: 'DataModuleClass' object has no attribute 'train_data'

Output:
Most of the things were correct, except few things like:
def prepare_data(self):

This function was right except that it should not return anything.
Another thing was
def setup(self,stage=None):

Requires stage variable which can be set to a default value of none in case we don't want to switch between different test and train stage.
Putting everything together, here is the code:
from argparse import ArgumentParser
import numpy as np
import pytorch_lightning as pl
from torch.utils.data import random_split, DataLoader, TensorDataset
import torch
from torch.autograd import Variable
from torchvision import transforms
import pytorch_lightning as pl
import torch
from torch import nn
from torch.nn import functional as F
from torch.optim import Adam
from torch.optim.optimizer import Optimizer

class LinearRegression(pl.LightningModule):
    def __init__(
        self,
        input_dim: int = 2,
        output_dim: int = 1,
        bias: bool = True,
        learning_rate: float = 1e-4,
        optimizer: Optimizer = Adam,
        l1_strength: float = 0.0,
        l2_strength: float = 0.0
    ):
        super().__init__()
        self.save_hyperparameters()
        self.optimizer = optimizer

        self.linear = nn.Linear(in_features=self.hparams.input_dim, out_features=self.hparams.output_dim, bias=bias)

    def forward(self, x):
        y_hat = self.linear(x)
        return y_hat

    def training_step(self, batch, batch_idx):
        x, y = batch

        # flatten any input
        x = x.view(x.size(0), -1)

        y_hat = self(x)

        loss = F.mse_loss(y_hat, y, reduction='sum')

        # L1 regularizer
        if self.hparams.l1_strength > 0:
            l1_reg = sum(param.abs().sum() for param in self.parameters())
            loss += self.hparams.l1_strength * l1_reg

        # L2 regularizer
        if self.hparams.l2_strength > 0:
            l2_reg = sum(param.pow(2).sum() for param in self.parameters())
            loss += self.hparams.l2_strength * l2_reg

        loss /= x.size(0)

        tensorboard_logs = {'train_mse_loss': loss}
        progress_bar_metrics = tensorboard_logs
        return {'loss': loss, 'log': tensorboard_logs, 'progress_bar': progress_bar_metrics}

    def validation_step(self, batch, batch_idx):
        x, y = batch
        x = x.view(x.size(0), -1)
        y_hat = self(x)
        return {'val_loss': F.mse_loss(y_hat, y)}

    def validation_epoch_end(self, outputs):
        val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        tensorboard_logs = {'val_mse_loss': val_loss}
        progress_bar_metrics = tensorboard_logs
        return {'val_loss': val_loss, 'log': tensorboard_logs, 'progress_bar': progress_bar_metrics}

    def configure_optimizers(self):
        return self.optimizer(self.parameters(), lr=self.hparams.learning_rate)

np.random.seed(42)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class DataModuleClass(pl.LightningDataModule):
  def __init__(self):
      super().__init__()
      self.sigma = 5
      self.batch_size = 10

  def prepare_data(self):
      x = np.random.uniform(0, 10, 10)
      e = np.random.normal(0, self.sigma, len(x))

      y = x + e

      X = np.transpose(np.array([x, e]))

      self.x_train_tensor = torch.from_numpy(X).float().to(device)
      self.y_train_tensor = torch.from_numpy(y).float().to(device)

      training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)
      self.training_dataset = training_dataset

  def setup(self,stage=None):
      data = self.training_dataset
      self.train_data, self.val_data = random_split(data, [8, 2])

  def train_dataloader(self):
      return DataLoader(self.train_data)

  def val_dataloader(self):
      return DataLoader(self.val_data)

model = LinearRegression(input_dim=2, l1_strength=1, l2_strength=1)
trainer = pl.Trainer()
dummy = DataModuleClass()
trainer.fit(model,dummy)


Instruction: Does pytorch register modules that are assigned to object properties?

Input:
Say I have a module like:
class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = CNN(params)

And then I do:
module = MyModule()
module.cnn = CNN(some_other_params)

Is the replacement registered? Will there be any nasty side effects down the line?

Output: Yes. This is the standard way to reassign attributes. Whether there are nasty side effects depends on whether your replacement is specified compatibly.


Instruction: Nearly Constant training and validation accuracy

Input:
I’m new to pytorch and my problem may be a little naive
I’m training a pretrained VGG16 network on my dataset which it’s size is near 33000 images in 8 classes with labels [1,2,…,8] and my classes are imbalanced. my problem is that during training, validation and training accuracy is low and doesn’t increase, is there any problem in my code?
if not, what do you suggest to improve training?
'''
import torch
import time
import torch.nn as nn
import numpy as np
from sklearn.model_selection import train_test_split
from torch.optim import Adam
import cv2
import torchvision.models as models
from classify_dataset import Classification_dataset
from torchvision import transforms

transform = transforms.Compose([transforms.Resize((224,224)),
                                    transforms.RandomHorizontalFlip(p=0.5),
                                    transforms.RandomVerticalFlip(p=0.5),
                                    transforms.RandomRotation(degrees=45),
                                    transforms.ToTensor(),
                                    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))
                                    ])
dataset = Classification_dataset(root_dir=r'//home/arisa/Desktop/Hamid/IQA/Hamid_Dataset',
                                     csv_file=r'/home/arisa/Desktop/Hamid/IQA/new_label.csv',transform=transform)

target = dataset.labels - 1

train_indices, test_indices = train_test_split(np.arange(target.shape[0]), stratify=target)
test_dataset = torch.utils.data.Subset(dataset, indices=test_indices)
train_dataset = torch.utils.data.Subset(dataset, indices=train_indices)

class_sample_count = np.array([len(np.where(target[train_indices] == t)[0]) for t in np.unique(target)])
weight = 1. / class_sample_count
samples_weight = np.array([weight[t] for t in target[train_indices]])
samples_weight = torch.from_numpy(samples_weight)
samples_weight = samples_weight.double()

sampler = torch.utils.data.WeightedRandomSampler(samples_weight, len(samples_weight), replacement = True)

train_loader = torch.utils.data.DataLoader(train_dataset,
                                           batch_size=64,
                                           sampler=sampler)
test_loader = torch.utils.data.DataLoader(test_dataset,
                                           batch_size=64,
                                           shuffle=False)
for param in model.parameters():
     param.requires_grad = False

num_ftrs = model.classifier[0].in_features
model.classifier = nn.Linear(num_ftrs,8)

optimizer = Adam(model.parameters(), lr = 0.0001 )
criterion = nn.CrossEntropyLoss()
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.01)

path = '/home/arisa/Desktop/Hamid/IQA/'
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)
def train_model(model, train_loader,valid_loader, optimizer, criterion, scheduler=None, num_epochs=10 ):

        min_valid_loss = np.inf
        model.train()
        start = time.time()
        TrainLoss = []
        model = model.to(device)
        for epoch in range(num_epochs):
            total = 0
            correct = 0
            train_loss = 0
            #lr_scheduler.step()
            print('Epoch {}/{}'.format(epoch+1, num_epochs))
            print('-' * 10)

            train_loss = 0.0
            for x,y in train_loader:
                x = x.to(device)
                #print(y.shape)
                y = y.view(y.shape[0],).to(device)
                y = y.to(device)
                y -= 1
                out = model(x)

                loss = criterion(out, y)
                optimizer.zero_grad()
                loss.backward()

                TrainLoss.append(loss.item()* y.shape[0])
                train_loss += loss.item() * y.shape[0]
                _,predicted = torch.max(out.data,1)
                total += y.size(0)
                correct += (predicted == y).sum().item()
                optimizer.step()
                lr_scheduler.step()
            accuracy = 100*correct/total
            valid_loss = 0.0
            val_loss = []
            model.eval()
            val_correct = 0
            val_total = 0

            with torch.no_grad():

                for x_val, y_val in test_loader:
                    x_val = x_val.to(device)
                    y_val = y_val.view(y_val.shape[0],).to(device)
                    y_val -= 1
                    target = model(x_val)
                    loss = criterion(target, y_val)
                    valid_loss += loss.item() * y_val.shape[0]

                    _,predicted = torch.max(target.data,1)
                    val_total += y_val.size(0)
                    val_correct += (predicted == y_val).sum().item()

                    val_loss.append(loss.item()* y_val.shape[0])
                val_acc = 100*val_correct / val_total

                print(f'Epoch {epoch + 1} \t\t Training Loss: {train_loss / len(train_loader)} \t\t Validation Loss: {valid_loss / len(test_loader)} \t\t Train Acc:{accuracy} \t\t Validation Acc:{val_acc}')
                if min_valid_loss > (valid_loss / len(test_loader)):
                    print(f'Validation Loss Decreased({min_valid_loss:.6f}--->{valid_loss / len(test_loader):.6f}) \t Saving The Model')
                    min_valid_loss = valid_loss / len(test_loader)
                    state = {'state_dict': model.state_dict(),'optimizer': optimizer.state_dict(),}
                    torch.save(state,'/home/arisa/Desktop/Hamid/IQA/checkpoint.t7')

        end = time.time()
        print('TRAIN TIME:')
        print('%.2gs'%(end-start))

    train_model(model=model, train_loader=train_loader, optimizer=optimizer, criterion=criterion, valid_loader= test_loader,num_epochs=500  )

Thanks in advance
here is the result of 15 epoch
Epoch 1/500
----------
Epoch 1          Training Loss: 205.63448420514916       Validation Loss: 233.89266112356475         Train Acc:39.36360386127994         Validation Acc:24.142040038131555
Epoch 2/500
----------
Epoch 2          Training Loss: 199.05699240435197       Validation Loss: 235.08799531243065         Train Acc:41.90998291820601         Validation Acc:24.27311725452812
Epoch 3/500
----------
Epoch 3          Training Loss: 199.15626737127448       Validation Loss: 236.00033430619672         Train Acc:41.1035633416756          Validation Acc:23.677311725452814
Epoch 4/500
----------
Epoch 4          Training Loss: 199.02581041173886       Validation Loss: 233.60767459869385         Train Acc:41.86628530568466         Validation Acc:24.606768350810295
Epoch 5/500
----------
Epoch 5          Training Loss: 198.61493769454472       Validation Loss: 233.7503859202067          Train Acc:41.53656695665991         Validation Acc:25.0
Epoch 6/500
----------
Epoch 6          Training Loss: 198.71323942956585       Validation Loss: 234.17176149830675         Train Acc:41.639852222619474        Validation Acc:25.369399428026693
Epoch 7/500
----------
Epoch 7          Training Loss: 199.9395153770592        Validation Loss: 234.1744423635078          Train Acc:40.98041552456998         Validation Acc:24.84509056244042
Epoch 8/500
----------
Epoch 8          Training Loss: 199.3533399020355        Validation Loss: 235.4645173188412          Train Acc:41.26643626107337         Validation Acc:24.165872259294567
Epoch 9/500
----------
Epoch 9          Training Loss: 199.6451746921249        Validation Loss: 233.33387595956975         Train Acc:40.96452548365312         Validation Acc:24.59485224022879
Epoch 10/500
----------
Epoch 10         Training Loss: 197.9305159737011        Validation Loss: 233.76405122063377         Train Acc:41.8782028363723          Validation Acc:24.6186844613918
Epoch 11/500
----------
Epoch 11         Training Loss: 199.33247244055502       Validation Loss: 234.41085289463854         Train Acc:41.59218209986891         Validation Acc:25.119161105815063
Epoch 12/500
----------
Epoch 12         Training Loss: 199.87399289874256       Validation Loss: 234.23621463775635         Train Acc:41.028085647320545        Validation Acc:24.49952335557674
Epoch 13/500
----------
Epoch 13         Training Loss: 198.85540591944292       Validation Loss: 234.33149099349976         Train Acc:41.206848607635166        Validation Acc:24.857006673021925
Epoch 14/500
----------
Epoch 14         Training Loss: 199.92641723337513       Validation Loss: 233.37722391070741         Train Acc:41.15520597465539         Validation Acc:24.988083889418494
Epoch 15/500
----------
Epoch 15         Training Loss: 197.82172771698328       Validation Loss: 234.4943131533536          Train Acc:41.69943987605768         Validation Acc:24.380362249761678

Output: my problem was in model.train(). This phrase should be inside the training loop. but in my case I put it outside the training loop and when it comes to model.eval(), model maintained in this mode


Instruction: Pytorch semantic segmentation loss function

Input:
I’m new to segmentation model.
I would like to use the deeplabv3_resnet50 model.
My image has shape (256, 256, 3) and my label has shape (256, 256). Each pixel in my label has a class value(0-4). And the batch size set in the DataLoader is 32.
Therefore, the shape of my input batch is [32, 3, 256, 256] and the shape of corresponding target is [32, 256, 256]. I believe this is correct.
I was trying to use nn.BCEWithLogitsLoss().

Is this the correct loss function for my case? Or should I use
CrossEntropy instead?
If this is the right one, the output of my model is [32, 5, 256, 256]. Each image prediction has the shape [5,256, 256], does layer 0 means the unnomarlized probabilities of class 0? In order to make a [32, 256, 256] tensor to match the target to feed into the BCEWithLogitsLoss, do I need to transform the unnomarlized probabilities to classes?
If I should use CrossEntropy, what the size of my output and label should be?

Thank you everyone.

Output:
You are using the wrong loss function.
nn.BCEWithLogitsLoss() stands for Binary Cross-Entropy loss: that is a loss for Binary labels. In your case, you have 5 labels (0..4).
You should be using nn.CrossEntropyLoss: a loss designed for discrete labels, beyond the binary case.
Your models should output a tensor of shape [32, 5, 256, 256]: for each pixel in the 32 images of the batch, it should output a 5-dim vector of logits. The logits are the "raw" scores for each class, to be later on normalize to class probabilities using softmax function.
For numerical stability and computational efficiency, nn.CrossEntropyLoss does not require you to explicitly compute the softmax of the logits, but does it internally for you. As the documentation read:

This criterion combines LogSoftmax and NLLLoss in one single class.


Instruction: How can I more efficiently multiply every element in a batch of tensors with every other batch element, except for itself?

Input:
So, I have this code that multiplies every element in a batch of tensors with every other element, except for itself. The code works, but it becomes painfully slow with larger batch sizes (Ideally I want to be able to use it with batch sizes of up to 1000 or more, but even a couple hundred is okay). It basically freezes when using the PyTorch autograd system and large batch sizes (like 50 or greater).
I need help making the code faster and more efficient, while still getting the same output. Any help would be appreciated!
import torch

tensor = torch.randn(50, 512, 512)

batch_size = tensor.size(0)
list1 = []
for i in range(batch_size):
    list2 = []
    for j in range(batch_size):
        if j != i:
            x_out = (tensor[i] * tensor[j]).sum()
            list2.append(x_out )
    list1.append(sum(list2))

out = sum(list1)

I thought that torch.prod might be able to be used, but it doesn't seem to result in the same output as the code above. NumPy answers are acceptable as long as they can be recreated in PyTorch.

Output:
You could do the following:
import torch
tensor = torch.randn(50, 512, 512)

batch_size = tensor.size(0)
tensor = tensor.reshape(batch_size, -1)
prod = torch.matmul(tensor, tensor.transpose(0,1))
out = torch.sum(prod) - torch.trace(prod)

Here, you first flatten each element. Then, you multiply the matrix where each row is an element with its own transpose, which gives a batch_size x batch_size matrix, where the ijth element equals the product of tensor[i] with tensor[j]. So, summing up over the values in this matrix and subtracting its trace (i.e., sum of diagonal elements) gives the desired result.
I tried both methods with a batch_size of 1000, and the time taken dropped from 61.43s to 0.59s.


Instruction: Is there a way to load torchvision model by string?

Input:
Currently, I load pretrained torchvision model using following code:
import torchvision
torchvision.models.resnet101(pretrained=True)

However, I'd love to have model name as string parameter and then load the pretrained model using that string. A pseudo-code that would do so would be something like:
model_name = 'resnet101'
torchvision.models.get(model_name)(pretrained=True)

Is there a way to accomplish this in a rather simple manner?

Output:
You can use getattr
getattr(torchvision.models, 'resnet101')(pretrained=True)


Instruction: pytorch (cpu only) on osx fails with symbol not found

Input:
I am trying to get started with PyTorch - on a mac osx computer. However, basic steps fail:
from torch_sparse import coalesce, SparseTensor

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-1-dad8246d5249> in <module>
----> 1 from torch_sparse import coalesce, SparseTensor

/usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/site-packages/torch_sparse/__init__.py in <module>
     10         '_saint', '_padding'
     11 ]:
---> 12     torch.ops.load_library(importlib.machinery.PathFinder().find_spec(
     13         library, [osp.dirname(__file__)]).origin)
     14

/usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/site-packages/torch/_ops.py in load_library(self, path)
    102             # static (global) initialization code in order to register custom
    103             # operators with the JIT.
--> 104             ctypes.CDLL(path)
    105         self.loaded_libraries.add(path)
    106

/usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371
    372         if handle is None:
--> 373             self._handle = _dlopen(self._name, mode)
    374         else:
    375             self._handle = handle

OSError: dlopen(/usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/site-packages/torch_sparse/_version.so, 6): Symbol not found: __ZN3c105ErrorC1ENS_14SourceLocationERKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEE
  Referenced from: /usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/site-packages/torch_sparse/_version.so
  Expected in: flat namespace
 in /usr/local/Caskroom/miniconda/base/envs/my_conda_env/lib/python3.8/site-packages/torch_sparse/_version.so

I am using a conda environment of:
name: my_conda_env
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - python>=3.8
  - pytorch
  - pytorch_geometric

and instantiated it using:
conda env create --force -f environment.yml

Output:
https://github.com/rusty1s/pytorch_sparse/issues/135
Indeed, https://github.com/conda-forge/pytorch_sparse-feedstock/issues/13 is the problem
The manual installation of pytorch geometric and its dependencies such as pytorch sparse using pip https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html is a currently suitable workaround


Instruction: Pytorch getting RuntimeError: Found dtype Double but expected Float

Input:
I am trying to implement a neural net in PyTorch but it doesn't seem to work. The problem seems to be in the training loop. I've spend several hours into this but can't get it right. Please help, thanks.
I haven't added the data preprocessing parts.
# importing libraries
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import torch.nn.functional as F

# get x function (dataset related stuff)
def Getx(idx):
    sample = samples[idx]
    vector = Calculating_bottom(sample)
    vector = torch.as_tensor(vector, dtype = torch.float64)

    return vector

# get y function (dataset related stuff)
def Gety(idx):
    y = np.array(train.iloc[idx, 4], dtype = np.float64)
    y = torch.as_tensor(y, dtype = torch.float64)

    return y

# dataset
class mydataset(Dataset):

    def __init__(self):
        super().__init__()

    def __getitem__(self, index):
        x = Getx(index)
        y = Gety(index)

        return x, y

    def __len__(self):
        return len(train)

dataset = mydataset()

# sample dataset value
print(dataset.__getitem__(0))

(tensor([ 5.,  5.,  8., 14.], dtype=torch.float64), tensor(-0.3403, dtype=torch.float64))
# data-loader
dataloader = DataLoader(dataset, batch_size = 1, shuffle = True)

# nn architecture
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 4)
        self.fc2 = nn.Linear(4, 2)
        self.fc3 = nn.Linear(2, 1)

    def forward(self, x):
        x = x.float()
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Net()

# device
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)

# hyper-parameters
criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

# training loop

for epoch in range(5):

    for batch in dataloader:

        # unpacking
        x, y = batch
        x.to(device)
        y.to(device)

        # reset gradients
        optimizer.zero_grad()

        # forward propagation through the network
        out = model(x)

        # calculate the loss
        loss = criterion(out, y)

        # backpropagation
        loss.backward()

        # update the parameters
        optimizer.step()

Error:
/opt/conda/lib/python3.7/site-packages/torch/nn/modules/loss.py:446: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-3f68fcee9ff3> in <module>
     20
     21         # backpropagation
---> 22         loss.backward()
     23
     24         # update the parameters

/opt/conda/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222
    223     def register_hook(self, hook):

/opt/conda/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
--> 132         allow_unreachable=True)  # allow_unreachable flag
    133
    134

RuntimeError: Found dtype Double but expected Float

Output:
You need the data type of the data to match the data type of the model.
Either convert the model to double (recommended for simple nets with no serious performance problems such as yours)
# nn architecture
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(4, 4)
        self.fc2 = nn.Linear(4, 2)
        self.fc3 = nn.Linear(2, 1)
        self.double()

or convert the data to float.
class mydataset(Dataset):

    def __init__(self):
        super().__init__()

    def __getitem__(self, index):
        x = Getx(index)
        y = Gety(index)

        return x.float(), y.float()


Instruction: How to replace torch.norm with other pytorch function?

Input:
I would like to replace the torch.norm function using the other Pytorch function.
I was able to replace torch.norm in the case where x is not a matrix, as shown in the following code.
import torch
x = torch.randn(9)
out1 = torch.norm(x)
out2 = sum(abs(x)**2)**(1./2)
out1 == out2
>> tensor(True)

But I don't know how to replace it when x is a matrix.
Especially, I want to replace it in my case of dim=1 and keepdim=True.
x = torch.randn([3, 136, 64, 64])
out1 = torch.norm(x, dim=1, keepdim=True)
out2 = ???
out1 == out2

Background:
I'm converting a Pytorch model to CoreML, but the _VF.frobenius_norm operator defined in the torch.norm function is not implemented with CoreMLTools.
(The implementation inside torch.norm can be found here.)
A few people have trouble with this problem, but CoreMLTools is still unsupported (You can check from this issue).
So I'd like to replace it without the operator used in torch.norm.
I have tried torch.linalg.norm() and numpy.linalg.norm but they were not supported.
I have created a simple colaboratory notebook that reproduces this.
Please test it using the following colab.
https://colab.research.google.com/drive/11o6rTxHzEgZ_Rc7nFZHd3TvPugybB88h?usp=sharing

Output:
You could try the following:
import torch
x = torch.randn([3, 136, 64, 64])
out1 = torch.norm(x, dim=1, keepdim=True)
out2 = torch.square(x).sum(dim=1, keepdim=True).sqrt()

Note that out1 == out2 won't give exactly all True due to small errors in precision. You can check that the errors are in the order of 1e-7 for float32.
Here, the norm is computed directly using its mathematical definition. You can see this reference from Wolfram MathWorld for more details.


Instruction: Understanding torch.nn.Flatten

Input:
I understand that Flatten removes all of the dimensions except for one. For example, I understand flatten():
> t = torch.ones(4, 3)
> t
tensor([[1., 1., 1.],
    [1., 1., 1.],
    [1., 1., 1.],
    [1., 1., 1.]])

> flatten(t)
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

However, I don't get Flatten, especially I don't get meaning of this snippet from the doc:
>>> input = torch.randn(32, 1, 5, 5)
>>> m = nn.Sequential(
>>>     nn.Conv2d(1, 32, 5, 1, 1),
>>>     nn.Flatten()
>>> )
>>> output = m(input)
>>> output.size()
torch.Size([32, 288])

I felt the output should have size [160], because 32*5=160.
Q1. So why it outputted size [32,288]?
Q2. I also don't get meaning of shape information given in the doc:

Q3. And also meaning of parameters:

Output:
It is a difference in the default behaviour. torch.flatten flattens all dimensions by default, while torch.nn.Flatten flattens all dimensions starting from the second dimension (index 1) by default.
You can see this behaviour in the default values of the start_dim and end_dim arguments. The start_dim argument denotes the first dimension to be flattened (zero-indexed), and the end_dim argument denotes the last dimension to be flattened. So, when start_dim=1, which is the default for torch.nn.Flatten, the first dimension (index 0) is not flattened, but it is included when start_dim=0, which is the default for torch.flatten.
The reason behind this difference is probably because torch.nn.Flatten is intended to be used with torch.nn.Sequential, where typically a series of operations are performed on a batch of inputs, where each input is treated independently of the others. For example, if you have a batch of images and you call torch.nn.Flatten, the typical use case would be to flatten each image separately, and not flatten the whole batch.
If you do want to flatten all dimensions using torch.nn.Flatten, you can simply create the object as torch.nn.Flatten(start_dim=0).
Finally, the shape information in the docs just covers how the shape of the tensor will be affected, illustrating that the first (index 0) dimension is left as it is. So, if you have an input tensor of shape (N, *dims), where *dims is an arbitrary sequence of dimensions, the output tensor will have the shape (N, product of *dims), since all dimensions except the batch dimension are flattened. For example, an input of shape (3,10,10) will have an output of shape (3, 10 x 10) = (3, 100).


Instruction: What is the relation between a learning rate scheduler and an optimizer?

Input:
If I have a model:
import torch
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(2, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 4)

        def forward(self, x):
            x=self.fc1(x)
            x=self.fc2(x)
            x=self.out(x)
            return x

nx = net_x()

And then I'm defining my inputs, optimizer (with lr=0.1), scheduler (with base_lr=1e-3), and training:
r = torch.tensor([1.0,2.0])
optimizer = optim.Adam(nx.parameters(), lr = 0.1)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-3, max_lr=0.1, step_size_up=1, mode="triangular2", cycle_momentum=False)

path = 'opt.pt'
for epoch in range(10):
    optimizer.zero_grad()
    net_predictions = nx(r)
    loss = torch.sum(torch.randint(0,10,(4,)) - net_predictions)
    loss.backward()
    optimizer.step()
    scheduler.step()
    print('loss:' , loss)

    #save state dict
    torch.save({    'epoch': epoch,
                    'net_x_state_dict': nx.state_dict(),
                    'optimizer_state_dict': optimizer.state_dict(),
                    'scheduler': scheduler.state_dict(),
                    }, path)
#loading state dict
checkpoint = torch.load(path)
nx.load_state_dict(checkpoint['net_x_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
scheduler.load_state_dict(checkpoint['scheduler'])

The optimizer seems to take the learning rate of the scheduler
for g in optimizer.param_groups:
    print(g)
>>>
{'lr': 0.001, 'betas': (0.9, 0.999), 'eps': 1e-08, 'weight_decay': 0, 'amsgrad': False, 'initial_lr': 0.001, 'params': [Parameter containing:

Does the learning rate scheduler overwrite the optimizer? How does it connect to it? Trying to understand the relation between them (i.e how they interact, etc.)

Output:
TL;DR: The LR scheduler contains the optimizer as a member and alters its parameters learning rates explicitly.

As mentioned in PyTorch Official Documentations, the learning rate scheduler receives the optimizer as a parameter in its constructor, and thus has access to its parameters.
The common use is to update the LR after every epoch:
scheduler = ... # initialize some LR scheduler
for epoch in range(100):
    train(...) # here optimizer.step() is called numerous times.
    validate(...)
    scheduler.step()

All optimizers inherit from a common parent class torch.nn.Optimizer and are updated using the step method implemented for each of them.
Similarly, all LR schedulers (besides ReduceLROnPlateau) inherit from a common parent class named _LRScheduler. Observing its source code uncovers that in the step method the class indeed changes the LR of the parameters of the optimizer:
...
for i, data in enumerate(zip(self.optimizer.param_groups, values)):
            param_group, lr = data
            param_group['lr'] = lr
...


Instruction: Pytorch min and max of tensor

Input:
I have following pytorch tensor:
>>> mean_actions
tensor([[-5.7547e-04,  1.4318e-02,  1.9328e-04, -2.5660e-03,  3.5269e-03,
         -1.3797e-02, -6.1871e-04, -2.7425e-03,  1.1661e-03,  1.6873e-03,
          3.9045e-03,  1.8047e-03,  4.8656e-03,  5.7182e-03, -4.8501e-03,
         -5.5913e-03, -4.4574e-03, -3.3154e-03, -4.9826e-03, -1.0071e-02,
         -2.3483e-03, -1.1413e-02, -4.9613e-03, -1.1648e-02,  2.4752e-03,
         -1.9764e-03,  3.1063e-03, -6.3481e-05,  7.6710e-03,  5.7503e-04]])

I am getting following min and max values out of tensor:
>>> th.min(mean_actions)
tensor(-0.0138)
>>> th.max(mean_actions)
tensor(0.0143)

However, I dont see -0.0138 and 0.0143 present in the tensor. What I am missing?
Here are the screenshots from debug session:

Output:
1.4318e-02 is scientific notation for 0.014318
and
-1.3797e-02 is scientific notation for -0.013797
See Wikipedia - E Notation


Instruction: PyTorch: two binary masks union?

Input:
I have two binary masks of shape (batch_size, width, heigh) that I want to create a binary mask which indicates the union of elements between the two.
To find the intersection, I can use torch.where(A == B, 1, 0), but how can I find the union?

Output:
When working with binary masks, you should use logical operations such as:
logical_or(), logical_and().
The intersection is then the binary mask:
intersection = A.logical_and(B)

and the union is:
union = A.logical_or(B)

BTW,
I'll leave it to you as an exercise to check why the intersection you computed (A == B) is not correct.


Instruction: CNN Pytorch only batches of spatial targets supported error

Input:
So I've architected the following model which I will use to classify the MNIST Fashion data.
class CNN(nn.Module):
    def __init__(self, **kwargs):
      super().__init__()

      self.conv1    = nn.Conv2d(784, 64, 2, 1, padding=5)
      self.maxpool  = nn.MaxPool2d(kernel_size=2, stride=2)
      self.conv2    = nn.Conv2d(64, 128, 2, 2, padding = 0)
      self.conv2_bn = nn.BatchNorm2d(128)
      self.relu     = nn.ReLU()
      self.dense    = nn.Linear(1, 128)
      self.softmax  = nn.Softmax()

    def forward(self, x): # you can add any additional parameters you want

        x = self.conv1(x)
        x = F.max_pool2d(F.relu(x), kernel_size=2)
        x = self.conv2(x)
        x = self.conv2_bn(x)
        x = F.max_pool2d(F.relu(x), kernel_size=2)
        print(x.shape)
        x = self.dense(x)
        x = F.relu(x)

        return F.log_softmax(x)

And this is where I run my code:
for epoch in range(max_epoch):

    print('EPOCH='+str(epoch))

    correct = 0
    total = 0

    running_loss = 0

    for data, label in tzip(TRAX, TRAY):

        #train = data.view(64,1,2,2)

        DAAA = data.view(1,784,1,1)
        #zeroing the parameter
        optimizer.zero_grad()
        label = torch.tensor([label]).type(torch.LongTensor)
        #forwards prop
        outputs = model2(DAAA)
        loss = criterion(outputs, label)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        '========================================'
        _, predicted = torch.max(outputs.data, 1)
        total += label.size(0)
        correct += (predicted == label).sum().item()

        '========================================'

    print('\n')
    print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
    print('\n')
    print(str(epoch)+'loss= '+str(running_loss))

    lossjournal.append(running_loss)
    accjournal.append(100 * correct / total)

print('Finished Training')

<ipython-input-378-27ce013b2c10> in <module>
     55         #forwards prop
     56         outputs = model2(DAAA)
---> 57         loss = criterion(outputs, label)
     58         loss.backward()
     59         optimizer.step()

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
   1045     def forward(self, input: Tensor, target: Tensor) -> Tensor:
   1046         assert self.weight is None or isinstance(self.weight, Tensor)
-> 1047         return F.cross_entropy(input, target, weight=self.weight,
   1048                                ignore_index=self.ignore_index, reduction=self.reduction)
   1049

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2691     if size_average is not None or reduce is not None:
   2692         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2693     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2694
   2695

/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2388         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2389     elif dim == 4:
-> 2390         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2391     else:
   2392         # dim == 3 or dim > 4

RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1

When I run my model I got this error but I don't know what to do from now on? What adjustments should I make for this model to work? I know the problem is with the criterion but is it because of the output of the model, which is of shape [1, 128, 1, 128]?

Output:
MNIST has 10 classes so your output should be of size [batch_size, 10]. Change the last linear layer to self.dense = nn.Linear(128,10).  Then, since your label is of size [batch_size,1], you should use torch.nn.CrossEntropyLoss as the criterion.
Additionally, you need not include the last softmax layer during training, as the aforementioned loss function performs a softmax operation during computation. You can instead use softmax or argmax for inference only.


Instruction: How to implement Flatten layer with batch size > 1 in Pytorch (Pytorch_Geometric)

Input:
I am new to Pytorch and am trying to transfer my previous code from Tensorflow to Pytorch due to memory issues.  However, when trying to reproduce Flatten layer, some issues kept coming out.
In my DataLoader object, batch_size is mixed with the first dimension of input (in my GNN, the input unpacked from DataLoader object is of size [batch_size*node_num, attribute_num], e.g. [4*896, 32] after the GCNConv layers). Basically, if I implement torch.flatten() after GCNConv, samples are mixed together (to [4*896*32]) and there would be only 1 output from this network, while I expect #batch_size outputs. And if I use nn.Flatten() instead, nothing seems to happen (still [4*896, 32]).  Should I set batch_size as the first dim of the input at the very beginning, or should I directly use view() function?  I tried directly using view() and it (seemed to have) worked, although I am not sure if this is the same as Flatten.  Please refer to my code below.  I am currently using global_max_pool because it works (it can separate batch_size directly).
By the way, I am not sure why training is so slow in Pytorch...  When node_num is raised to 13000, I need an hour to go through an epoch, and I have 100 epoch per test fold and 10 test folds.  In tensorflow the whole training process only takes several hours.  Same network architecture and raw input data, as shown here in another post of mine, which also described the memory issues I met when using TF.
Have been quite frustrated for a while.  I checked this and this post, but it seems their problems somewhat differ from mine.  Would greatly appreciate any help!
Code:
# Generate dataset
class STDataset(InMemoryDataset):
    def __init__(self, root, transform=None, pre_transform=None):
        super(STDataset, self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])

    @property
    def raw_file_names(self):
        return []

    @property
    def processed_file_names(self):
        return ['pygdata.pt']

    def download(self):
        pass

    def process(self):
        data_list= []
        for i in range(sample_size):
            data = Data(x=torch.tensor(X_all[i],dtype=torch.float),edge_index=edge_index,y=torch.FloatTensor(y_all[i]))
            data_list.append(data)

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])

dataset = STDataset(root=save_dir)
train_dataset = dataset[:len(X_train)]
val_dataset = dataset[len(X_train):(len(X_train)+len(X_val))]
test_dataset = dataset[(len(X_train)+len(X_val)):]

# Build network

from torch_geometric.nn import GCNConv, GATConv, TopKPooling, global_max_pool, global_mean_pool
from torch.nn import Flatten, Linear, ELU
import torch.nn.functional as F

class GCN(torch.nn.Module):
    def __init__(self):
        super(GCN, self).__init__()
        self.conv1 = GCNConv(in_channels = feature_num, out_channels = 32)
        self.conv2 = GCNConv(in_channels = 32, out_channels = 32)
        self.fc1 = Flatten()
#         self.ln1 = Linear(in_features = batch_size*N*32, out_features = 512)
        self.ln1 = Linear(in_features = 32, out_features = 32)
        self.ln2 = Linear(in_features = 32, out_features = 1)

    def forward(self,x,edge_index,batch):
#         x, edge_index, batch = data.x, data.edge_index, data.batch
#         print(np.shape(x),np.shape(edge_index),np.shape(batch))
        x = F.elu(self.conv1(x,edge_index))
#         x = x.squeeze(1)
        x = F.elu(self.conv2(x,edge_index))
        print(np.shape(x))
        x = self.fc1(x)
#         x = torch.flatten(x,0)
#         x = torch.cat([global_max_pool(x,batch),global_mean_pool(x,batch)],dim=1)
        print(np.shape(x))
        x = self.ln1(x)
        x = F.relu(x)
        ## Dropout?
        print("o")
        x = torch.sigmoid(self.ln2(x))
        return x

# training
def train():
    model.train()
    loss_all=0
    correct = 0
    for i, data in enumerate(train_loader, 0):
        data = data.to(device)
        optimizer.zero_grad()
        output = model(data.x, data.edge_index,data.batch)
        label = data.y.to(device)
        loss = loss_func(output, label)
        loss.backward()
        loss_all += loss.item()

        output = output.detach().cpu().numpy().squeeze()
        label = label.detach().cpu().numpy().squeeze()
        correct += (abs(output-label)<0.5).sum()

        optimizer.step()

    return loss_all / len(train_dataset), correct / len(train_dataset)

device = torch.device('cuda')
model = GCN().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
loss_func = torch.nn.BCELoss()  # binary cross-entropy
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle = True)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle = True)
for epoch in range(num_epochs):
    gc.collect()
    train_loss, train_acc = train()

Error message for using torch.nn.Flatten(start_dim = 1) (code above):
ValueError                                Traceback (most recent call last)
<ipython-input-42-c96e8b058742> in <module>
     65 for epoch in range(num_epochs):
     66     gc.collect()
---> 67     train_loss, train_acc = train()

<ipython-input-42-c96e8b058742> in train()
     10         output = model(data.x, data.edge_index,data.batch)
     11         label = data.y.to(device)
---> 12         loss = loss_func(output, label)
     13         loss.backward()
     14         loss_all += loss.item()

~/miniconda3/envs/ST-Torch/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/miniconda3/envs/ST-Torch/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    496
    497     def forward(self, input, target):
--> 498         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    499
    500

~/miniconda3/envs/ST-Torch/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2068     if input.numel() != target.numel():
   2069         raise ValueError("Target and input must have the same number of elements. target nelement ({}) "
-> 2070                          "!= input nelement ({})".format(target.numel(), input.numel()))
   2071
   2072     if weight is not None:

ValueError: Target and input must have the same number of elements. target nelement (4) != input nelement (3584)

Output:
The way you want the shape to be batch_size*node_num, attribute_num is kinda weird.
Usually it should be batch_size, node_num*attribute_num as you need to match the input to the output. And Flatten in Pytorch does exactly that.
If what you want is really batch_size*node_num, attribute_num then you left with only reshaping the tensor using view or reshape. And actually Flatten itself just calls .reshape.

tensor.view: This will reshape the existing tensor to a new shape, if you edit this new tensor the old one will change too.

tensor.reshape: This will create a new tensor using the data from old tensor but with new shape.

    def forward(self,x,edge_index,batch):
        x = F.elu(self.conv1(x,edge_index))
        x = F.elu(self.conv2(x,edge_index))

        # print(np.shape(x)) # don't use this
        print(x.size())  # use this

        # x = self.fc1(x)  # this is the old one
        ## choose one of these
        x = x.view(4*896, 32)
        x = x.reshape(4*896, 32)

        # print(np.shape(x)) # don't use this
        print(x.size())  # use this

        x = self.ln1(x)
        x = F.relu(x)
        ## Dropout?
        print("o")
        x = torch.sigmoid(self.ln2(x))
        return x

Edit 2 reshape
Let's say we have an array of [[[1, 1, 1], [2, 2, 2]]] which shape (1, 2, 3), which represent (batch, length, channel) in Tensorflow.
If you want to use this data properly in Pytorch you need to make it (batch, channel, length), which is (1, 3, 2).
Here's the difference between permute and reshape
>>> x = torch.tensor([[[1, 1, 1], [2, 2, 2]]])
>>> x.size()
torch.Size([1, 2, 3])
>>> x[0, 0, :]
tensor([1, 1, 1])
>>> y = x.reshape((1, 3, 2))
>>> y
tensor([[[1, 1],
         [1, 2],
         [2, 2]]])
>>> y[0, :, 0]
tensor([1, 1, 2])
>>> z = x.permute(0, 2, 1)
>>> z
tensor([[[1, 2],
         [1, 2],
         [1, 2]]])
>>> z[0, :, 0]
tensor([1, 1, 1])

As you can see, the first channel of both x and z are [1, 1, 1] which is what we want while y is [1, 1, 2].


Instruction: Pytorch: a similar process to reverse pooling and replicate padding?

Input:
I have a tensor A that has shape (batch_size, width, height). Assume that it has these values:
A = torch.tensor([[[0, 1],
                   [1, 0]]])

I am also given a number K that is a positive integer. Let K=2 in this case. I want to do a process that is similar to reverse pooling and replicate padding. This is the expected output:
B = torch.tensor([[[0, 0, 1, 1],
                   [0, 0, 1, 1],
                   [1, 1, 0, 0],
                   [1, 1, 0, 0]]])

Explanation: for each element in A, we expand it to the matrix of shape (K, K), and put it in the result tensor. We continue to do this with other elements, and let the stride between them equals to the kernel size (that is, K).
How can I do this in PyTorch? Currently, A is a binary mask, but it could be better if I can expand it to non-binary case.

Output:
Square expansion
You can get your desired output by expanding twice:
def dilate(t, k):
  x = t.squeeze()
  x = x.unsqueeze(-1).expand([*x.shape,k])
  x = x.unsqueeze(-1).expand([*x.shape,k])
  x = torch.cat([*x], dim=1)
  x = torch.cat([*x], dim=1)
  x = x.unsqueeze(0)
  return x

B = dilate(A, k)

Resizing / interpolating nearest
If you don't mind corners potentially 'bleeding' in larger expansions (since it uses Euclidean as opposed to Manhattan distance when determining 'nearest' points to interpolate), a simpler method is to just resize:
import torchvision.transforms.functional as F

B = F.resize(A, A.shape[-1]*k)

For completeness:

MaxUnpool2d takes in as input the output of MaxPool2d including the indices of the maximal values and computes a partial inverse in which all non-maximal values are set to zero.


Instruction: Using PyTorch's autograd efficiently with tensors by calculating the Jacobian

Input:
In my previous question I found how to use PyTorch's autograd with tensors:
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 4) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()

#input
t = torch.tensor([1.0, 2.0, 3.2], requires_grad = True) #input vector
t = torch.reshape(t, (3,1)) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)
dx = torch.diagonal(torch.diagonal(dx, 0, -1), 0)[0] #first vector
#dx = torch.diagonal(torch.diagonal(dx, 1, -1), 0)[0] #2nd vector
#dx = torch.diagonal(torch.diagonal(dx, 2, -1), 0)[0] #3rd vector
#dx = torch.diagonal(torch.diagonal(dx, 3, -1), 0)[0] #4th vector
dx
>>>
tensor([-0.0142, -0.0517, -0.0634])

The issue is that grad only knows how to propagate gradients from a scalar tensor (which my network's output is not), which is why I had to calculate the Jacobian.
However, this is not very efficient and a bit slow as my matrix is large and calculating the entire Jacobian takes a while (and I'm also not using the entire Jacobian matrix).
Is there a way to calculate only the diagonals of the Jacobian (to get the 4 vectors in this example)?
There appears to be an open feature request but it doesn't appear to have gotten much attention.
Update 1:
I tried what @iacob said about setting torch.autograd.functional.jacobian(vectorize=True).
However, this seems to be slower. To test this I changed my network output from 4 to 400, and my input t to be:
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)) #reshape for batch

Without vectorized = True:
Wall time: 10.4 s

With:
Wall time: 14.6 s

Output:
OK, results first:
The performance (My laptop has an RTX-2070 and PyTorch is using it):
# Method 1: Use the jacobian function
CPU times: user 34.6 s, sys: 165 ms, total: 34.7 s
Wall time: 5.8 s

# Method 2: Sample with appropriate vectors
CPU times: user 1.11 ms, sys: 0 ns, total: 1.11 ms
Wall time: 191 µs

It's about 30000x faster.

Why should you use backward instead of jacobian (in your case)
I'm not a pro with PyTorch. But, according to my experience, it's pretty inefficient to calculate the jacobi-matrix if you do not need all the elements in it.
If you only need the diagonal elements, you can use backward function to calculate vector-jacobian multiplication with some specific vectors. If you set the vectors correctly, you can sample/extract specific elements from the Jacobi matrix.
A little linear algebra:
j = np.array([[1,2],[3,4]]) # 2x2 jacobi you want
sv = np.array([[1],[0]])     # 2x1 sampling vector

first_diagonal_element = sv.T.dot(j).dot(sv)  # it's j[0, 0]

It's not that powerful for this simple case. But if PyTorch needs to calculate all jacobians along the way (j could be the result of a long sequence of matrix-matrix multiplications), it would be way too slow. In contrast, if we calculate a sequence of vector-jacobian multiplications, the computation would be super fast.

Solution
Sample elements from jacobian:
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 400) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()

#input

val = 100
a = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(a, (val,1)) #reshape for batch

#method
%time dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)
dx = torch.diagonal(torch.diagonal(dx, 0, -1), 0)[0] #first vector
#dx = torch.diagonal(torch.diagonal(dx, 1, -1), 0)[0] #2nd vector
#dx = torch.diagonal(torch.diagonal(dx, 2, -1), 0)[0] #3rd vector
#dx = torch.diagonal(torch.diagonal(dx, 3, -1), 0)[0] #4th vector
print(dx)

out = nx(t)
m = torch.zeros((val,400))
m[:, 0] = 1
%time out.backward(m)
print(a.grad)

a.grad should be equal to the first tensor dx. And, m is the sampling vector that corresponds to what is called the "first vector" in your code.

but if I run it again the answer will change.

Yeah, you've already figured it out. The gradients accumulate every time you call backward. So you have to set a.grad to zero first if you have to run that cell multiple times.

can you explain the idea behind the m method? Both using the torch.zeros and setting the column to 1. Also, how come the grad is on a rather than on t?

The idea behind the m method is: what the function backward calculates is actually a vector-jacobian multiplication, where the vector represents the so-called "upstream gradient" and the Jacobi-matrix is the "local gradient" (and this jacobian is also the one you get with the jacobian function, since your lambda could be viewed as a single "local" operation). If you need some elements from the jacobian, you can fake (or, more precisely, construct) some "upstream gradient", with which you can extract specific entries from the jacobian. However, sometimes these upstream gradients might be hard to find (for me at least) if complicated tensor operations are involved.
PyTorch accumulates gradients on leaf nodes of the computational graph. And, your original line of code t = torch.reshape(t, (3,1)) loses handle to the leaf node, and t refers now to an intermediate node instead of a leaf node. In order to have access to the leaf node, I created the tensor a.


Instruction: Training Will Be Stop After a While in GRU Layer Pytorch

Input:
I use my custom dataset class to convert audio files to mel-Spectrogram images. the shape will be padded to (128,1024). I have 10 classes. after a while of training in the first epoch, my network will be crashed inside the hidden layer in GRU shapes due to this error:
Current run is terminating due to exception: Expected hidden size (1, 7, 32), got [1, 16, 32]
Engine run is terminating due to exception: Expected hidden size (1, 7, 32), got [1, 16, 32]
Traceback (most recent call last):
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-b8f3a45f8e35>", line 1, in <module>
    runfile('/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py', wdir='/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools')
  File "/home/omid/OMID/program/pycharm-professional-2020.2.4/pycharm-2020.2.4/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/home/omid/OMID/program/pycharm-professional-2020.2.4/pycharm-2020.2.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 60, in <module>
    main()
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 56, in main
    train(cfg)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/tools/train_net.py", line 35, in train
    do_train(
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/engine/trainer.py", line 79, in do_train
    trainer.run(train_loader, max_epochs=epochs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 745, in _internal_run
    time_taken = self._run_once_on_dataset()
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 850, in _run_once_on_dataset
    self._handle_exception(e)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/engine.py", line 833, in _run_once_on_dataset
    self.state.output = self._process_function(self, self.state.batch)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/ignite/engine/__init__.py", line 103, in _update
    y_pred = model(x)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/modeling/model.py", line 113, in forward
    x, h1 = self.gru1(x, h0)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 819, in forward
    self.check_forward_args(input, hx, batch_sizes)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 229, in check_forward_args
    self.check_hidden_size(hidden, expected_hidden_size)
  File "/home/omid/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/nn/modules/rnn.py", line 223, in check_hidden_size
    raise RuntimeError(msg.format(expected_hidden_size, list(hx.size())))
RuntimeError: Expected hidden size (1, 7, 32), got [1, 16, 32]

My Network Is:
import torch
import torch.nn as nn
import torch.nn.functional as F

print('cuda', torch.cuda.is_available())

class MusicClassification(nn.Module):
    def __init__(self, cfg):
        super(MusicClassification, self).__init__()
        device = cfg.MODEL.DEVICE
        num_class = cfg.MODEL.NUM_CLASSES

        self.np_layers = 4
        self.np_filters = [64, 128, 128, 128]
        self.kernel_size = (3, 3)

        self.pool_size = [(2, 2), (4, 2)]

        self.channel_axis = 1
        self.frequency_axis = 2
        self.time_axis = 3

        # self.h0 = torch.zeros((1, 16, 32)).to(device)

        self.bn0 = nn.BatchNorm2d(num_features=self.channel_axis)
        self.bn1 = nn.BatchNorm2d(num_features=self.np_filters[0])
        self.bn2 = nn.BatchNorm2d(num_features=self.np_filters[1])
        self.bn3 = nn.BatchNorm2d(num_features=self.np_filters[2])
        self.bn4 = nn.BatchNorm2d(num_features=self.np_filters[3])

        self.conv1 = nn.Conv2d(1, self.np_filters[0], kernel_size=self.kernel_size)
        self.conv2 = nn.Conv2d(self.np_filters[0], self.np_filters[1], kernel_size=self.kernel_size)
        self.conv3 = nn.Conv2d(self.np_filters[1], self.np_filters[2], kernel_size=self.kernel_size)
        self.conv4 = nn.Conv2d(self.np_filters[2], self.np_filters[3], kernel_size=self.kernel_size)

        self.max_pool_2_2 = nn.MaxPool2d(self.pool_size[0])
        self.max_pool_4_2 = nn.MaxPool2d(self.pool_size[1])

        self.drop_01 = nn.Dropout(0.1)
        self.drop_03 = nn.Dropout(0.3)

        self.gru1 = nn.GRU(input_size=128, hidden_size=32, batch_first=True)
        self.gru2 = nn.GRU(input_size=32, hidden_size=32, batch_first=True)

        self.activation = nn.ELU()

        self.dense = nn.Linear(32, num_class)
        self.softmax = nn.LogSoftmax(dim=1)

    def forward(self, x):
        # x [16, 1, 128,938]
        x = self.bn0(x)
        # x [16, 1, 128,938]
        x = F.pad(x, (0, 0, 2, 1))
        # x [16, 1, 131,938]
        x = self.conv1(x)
        # x [16, 64, 129,936]
        x = self.activation(x)
        # x [16, 64, 129,936]
        x = self.bn1(x)
        # x [16, 64, 129,936]
        x = self.max_pool_2_2(x)
        # x [16, 64, 64,468]
        x = self.drop_01(x)
        # x [16, 64, 64,468]
        x = F.pad(x, (0, 0, 2, 1))
        # x [16, 64, 67,468]
        x = self.conv2(x)
        # x [16, 128, 65,466]
        x = self.activation(x)
        # x [16, 128, 65,466]
        x = self.bn2(x)
        # x [16, 128, 65,455]
        x = self.max_pool_4_2(x)
        # x [16, 128, 16,233]
        x = self.drop_01(x)
        # x [16, 128, 16,233]
        x = F.pad(x, (0, 0, 2, 1))
        # x [16, 128, 19,233]
        x = self.conv3(x)
        # x [16, 128, 17,231]
        x = self.activation(x)
        # x [16, 128, 17,231]
        x = self.bn3(x)
        # x [16, 128, 17,231]
        x = self.max_pool_4_2(x)
        # x [16, 128, 4,115]
        x = self.drop_01(x)
        # x [16, 128, 4,115]
        x = F.pad(x, (0, 0, 2, 1))
        # x [16, 128, 7,115]
        x = self.conv4(x)
        # x [16, 128, 5,113]
        x = self.activation(x)
        # x [16, 128, 5,113]
        x = self.bn4(x)
        # x [16, 128, 5,113]
        x = self.max_pool_4_2(x)
        # x [16, 128, 1,56]
        x = self.drop_01(x)
        # x [16, 128, 1,56]

        x = x.permute(0, 3, 1, 2)
        # x [16, 56, 128,1]
        resize_shape = list(x.shape)[2] * list(x.shape)[3]
        # x [16, 128, 56,1], reshape size is 128
        x = torch.reshape(x, (list(x.shape)[0], list(x.shape)[1], resize_shape))
        # x [16, 56, 128]
        device = torch.device("cuda" if torch.cuda.is_available()
                              else "cpu")
        h0 = torch.zeros((1, 16, 32)).to(device)
        x, h1 = self.gru1(x, h0)
        # x [16, 56, 32]
        x, _ = self.gru2(x, h1)
        # x [16, 56, 32]
        x = x[:, -1, :]
        x = self.dense(x)
        # x [16,10]
        x = self.softmax(x)
        # x [16, 10]
        # x = torch.argmax(x, 1)
        return x

My Dataset is:

from __future__ import print_function, division

import os

import librosa
import matplotlib.pyplot as plt
import numpy as np
import torch
import torchaudio
from sklearn.preprocessing import OneHotEncoder, LabelEncoder
from torch.utils.data import Dataset
from utils.util import pad_along_axis

print(torch.__version__)
print(torchaudio.__version__)

# Ignore warnings
import warnings

warnings.filterwarnings("ignore")

plt.ion()

import pathlib

print(pathlib.Path().absolute())

class GTZANDataset(Dataset):
    def __init__(self,
                 genre_folder='/home/omid/OMID/projects/python/mldl/NeuralMusicClassification/data/dataset/genres_original',
                 one_hot_encoding=False,
                 sr=16000, n_mels=128,
                 n_fft=2048, hop_length=512,
                 transform=None):

        self.genre_folder = genre_folder
        self.one_hot_encoding = one_hot_encoding
        self.audio_address, self.labels = self.extract_address()
        self.sr = sr
        self.n_mels = n_mels
        self.n_fft = n_fft
        self.transform = transform
        self.le = LabelEncoder()
        self.hop_length = hop_length

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        address = self.audio_address[index]
        y, sr = librosa.load(address, sr=self.sr)
        S = librosa.feature.melspectrogram(y, sr=sr,
                                           n_mels=self.n_mels,
                                           n_fft=self.n_fft,
                                           hop_length=self.hop_length)

        sample = librosa.amplitude_to_db(S, ref=1.0)
        sample = np.expand_dims(sample, axis=0)
        sample = pad_along_axis(sample, 1024, axis=2)
        # print(sample.shape)
        sample = torch.from_numpy(sample)

        label = self.labels[index]
        # label = torch.from_numpy(label)
        print(sample.shape,label)
        if self.transform:
            sample = self.transform(sample)
        return sample, label

    def extract_address(self):
        label_map = {
            'blues': 0,
            'classical': 1,
            'country': 2,
            'disco': 3,
            'hiphop': 4,
            'jazz': 5,
            'metal': 6,
            'pop': 7,
            'reggae': 8,
            'rock': 9
        }
        labels = []
        address = []
        # extract all genres' folders
        genres = [path for path in os.listdir(self.genre_folder)]
        for genre in genres:
            # e.g. ./data/generes_original/country
            genre_path = os.path.join(self.genre_folder, genre)
            # extract all sounds from genre_path
            songs = os.listdir(genre_path)

            for song in songs:
                song_path = os.path.join(genre_path, song)
                genre_id = label_map[genre]
                # one_hot_targets = torch.eye(10)[genre_id]
                labels.append(genre_id)
                address.append(song_path)

        samples = np.array(address)
        labels = np.array(labels)
        # convert labels to one-hot encoding
        # if self.one_hot_encoding:
        #     labels = OneHotEncoder(sparse=False).fit_transform(labels)
        # else:
        #     labels = LabelEncoder().fit_transform(labels)

        return samples, labels

and trainer :

# encoding: utf-8

import logging

from ignite.engine import Events, create_supervised_trainer, create_supervised_evaluator
from ignite.handlers import ModelCheckpoint, Timer
from ignite.metrics import Accuracy, Loss, RunningAverage

def do_train(
        cfg,
        model,
        train_loader,
        val_loader,
        optimizer,
        scheduler,
        loss_fn,
):
    log_period = cfg.SOLVER.LOG_PERIOD
    checkpoint_period = cfg.SOLVER.CHECKPOINT_PERIOD
    output_dir = cfg.OUTPUT_DIR
    device = cfg.MODEL.DEVICE
    epochs = cfg.SOLVER.MAX_EPOCHS

    model = model.to(device)

    logger = logging.getLogger("template_model.train")
    logger.info("Start training")
    trainer = create_supervised_trainer(model, optimizer, loss_fn, device=device)
    evaluator = create_supervised_evaluator(model, metrics={'accuracy': Accuracy(),
                                                            'ce_loss': Loss(loss_fn)}, device=device)
    checkpointer = ModelCheckpoint(output_dir, 'mnist', None, n_saved=10, require_empty=False)
    timer = Timer(average=True)

    trainer.add_event_handler(Events.EPOCH_COMPLETED, checkpointer, {'model': model.state_dict(),
                                                                     'optimizer': optimizer.state_dict()})
    timer.attach(trainer, start=Events.EPOCH_STARTED, resume=Events.ITERATION_STARTED,
                 pause=Events.ITERATION_COMPLETED, step=Events.ITERATION_COMPLETED)

    RunningAverage(output_transform=lambda x: x).attach(trainer, 'avg_loss')

    @trainer.on(Events.ITERATION_COMPLETED)
    def log_training_loss(engine):
        iter = (engine.state.iteration - 1) % len(train_loader) + 1

        if iter % log_period == 0:
            logger.info("Epoch[{}] Iteration[{}/{}] Loss: {:.2f}"
                        .format(engine.state.epoch, iter, len(train_loader), engine.state.metrics['avg_loss']))

    @trainer.on(Events.EPOCH_COMPLETED)
    def log_training_results(engine):
        evaluator.run(train_loader)
        metrics = evaluator.state.metrics
        avg_accuracy = metrics['accuracy']
        avg_loss = metrics['ce_loss']
        logger.info("Training Results - Epoch: {} Avg accuracy: {:.3f} Avg Loss: {:.3f}"
                    .format(engine.state.epoch, avg_accuracy, avg_loss))

    if val_loader is not None:
        @trainer.on(Events.EPOCH_COMPLETED)
        def log_validation_results(engine):
            evaluator.run(val_loader)
            metrics = evaluator.state.metrics
            avg_accuracy = metrics['accuracy']
            avg_loss = metrics['ce_loss']
            logger.info("Validation Results - Epoch: {} Avg accuracy: {:.3f} Avg Loss: {:.3f}"
                        .format(engine.state.epoch, avg_accuracy, avg_loss)
                        )

    # adding handlers using `trainer.on` decorator API
    @trainer.on(Events.EPOCH_COMPLETED)
    def print_times(engine):
        logger.info('Epoch {} done. Time per batch: {:.3f}[s] Speed: {:.1f}[samples/s]'
                    .format(engine.state.epoch, timer.value() * timer.step_count,
                            train_loader.batch_size / timer.value()))
        timer.reset()

    trainer.run(train_loader, max_epochs=epochs)

Output:
Errors like this are usually due to your data changing in some unexpected way, as the model is fixed and (as you said) working until a point. I think your error comes from this line in your model.forward() call:
h0 = torch.zeros((1, 16, 32)).to(device)

As this is hard-coded to be 16, when it appears that your dataloader is passing a batch_size of 7 at some stage. There are two ways to fix this, one of which is to set the h0 batch size dynamically:
h0 = torch.zeros((1,list(x.shape[0]),list(x.shape[2]))

or by passing drop_last=True in your dataloader creation.


Instruction: Assign value to sub-tensor in Pytorch

Input:
Suppose I have a PyTorch tensor t of shape (3, 3, 3, 3), and I want to set t[0, 1, x, y] to z for all x and y. I noticed all the following syntaxes work:

t[0, 1] = z
t[0, 1, :] = z
t[0, 1, :, :] = z
t[0, 1, ...] = z

Is there any difference in terms of how things are executed under the hood? Is any one of these methods preferred over the others?

Output:
There is a subtle difference in how : and ... are translated to C++ under the hood:

The one-to-one translation between Python and C++ index types is as follows:

Python
C++ (assuming using namespace torch::indexing)

None
None

Ellipsis
Ellipsis

...
"..."

: or ::
Slice() or Slice(None, None) or Slice(None, None, None)

Practically this has no effect on performance however. Use whichever style is most readable in your context.


Instruction: Python pytorch function consumes memory excessively quickly

Input:
I am writing a function with pytorch that feeds inputs through a transformer model, then condenses the last embedding layer by computing the average along a specific axis (using a subset of indices defined by a mask). Since the output of the model is very very large, I need to process the inputs in batches.
My question does not concern the logic of this function as I believe I have the correct implementation. My issue is that the function I wrote consumes memory excessively quickly and practically makes it unusable.
Here is my function:
def get_chunk_embeddings(encoded_dataset, batch_size):
  chunk_embeddings = torch.empty([0,768])
  for i in range(len(encoded_dataset['input_ids'])//batch_size):
    input_ids = encoded_dataset['input_ids'][i*batch_size:i*batch_size + batch_size]
    attention_mask = encoded_dataset['attention_mask'][i*batch_size:i*batch_size + batch_size]
    embeddings = model.forward(input_ids=input_ids, attention_mask=attention_mask)['last_hidden_state']
    embeddings = embeddings * attention_mask[:,:,None]
    embeddings = embeddings.sum(dim=1)/attention_mask.sum(dim=1)[:,None]
    chunk_embeddings = torch.cat([chunk_embeddings, embeddings],0)
  return chunk_embeddings

Now let's talk memory (the numbers below assume that I pass a batch_size of 8):

I am using google colab and I have ~25 GB of RAM available
The model is a BERT model and consumes 413 MB
encoded_dataset consumes 0.48 GB
input_ids consumes 0.413 MB
attention_mask consumes 4.096 KB
embeddings at its peak consumption consume 12.6 MB
chunk_embeddings adds 0.024576 MB with each iteration

So from my understanding, I should be able to allow chunk_embeddings to grow up to: 25GB - 413MB - 0.48GB - 0.413MB - 4.096KB - 12.6MB ~= 24 GB. Enough for almost 1 million iterations..
Here I will walk through an example of what I am experiencing:

Before running my function, google colab tells me that I have plenty of memory

Now, for the sake of example, I will run the function (for only 3 iterations)
To be explicit, I put this at the end of my for loop:
if (i == 2):return chunk_embeddings

Now I run the code val = get_chunk_embeddings(train_encoded_dataset, 8)
So even with just 3 iterations, somehow I consume almost 5.5 GB of RAM.

Why is this happening? Also after I have returned from the function all of the local variables should be deleted and there is no way that val is so large.
Could someone tell me what it is I am doing wrong or am not understanding? Please let me know if further information is needed.

Output:
To expand upon @GoodDeeds' answer, by default the computations within a pytorch.nn module (model) create the computation graph and preserve the gradient (unless you are using with torch.no_grad() or something similar. That means that at every iteration of your loop, the computational graph for embeddings is stored within the tensor embeddings. embeddings.grad is likely much larger than embeddings itself because the gradient of each layer value with respect to each previous layer's value is maintained. Next, since you use torch.cat, you append embeddingsd and the associated gradient to chunk_embeddings. That means that after a few iterations, chunk_embeddings stores a huge number of gradient values, which is where your memory is going. There are a few solutions:

If you need to use chunk embeddings for backpropogation (i.e. training), you should move your loss calculation and optimizer step within the loop, such that the gradients are automatically cleared afterwards.

If this function is only used during inference, you can disable the gradient computation entirely (which should also speed computation slightly) using torch.no_grad(), or you can use torch.detach() on embeddings at each iteration as suggested in the comments.

Example:
def get_chunk_embeddings(encoded_dataset, batch_size):
  with torch.no_grad():
    chunk_embeddings = torch.empty([0,768])
    for i in range(len(encoded_dataset['input_ids'])//batch_size):
      input_ids = encoded_dataset['input_ids'][i*batch_size:i*batch_size + batch_size]
      attention_mask = encoded_dataset['attention_mask'][i*batch_size:i*batch_size + batch_size]
      embeddings = model.forward(input_ids=input_ids, attention_mask=attention_mask)['last_hidden_state']
      embeddings = embeddings * attention_mask[:,:,None]
      embeddings = embeddings.sum(dim=1)/attention_mask.sum(dim=1)[:,None]
      chunk_embeddings = torch.cat([chunk_embeddings, embeddings],0)
  return chunk_embeddings


Instruction: PyTorch: Why create multiple instances of the same type of layer?

Input:
This code is from PyTorch transformer:
    self.linear1 = Linear(d_model, dim_feedforward, **factory_kwargs)
    self.dropout = Dropout(dropout)
    self.linear2 = Linear(dim_feedforward, d_model, **factory_kwargs)
    self.norm1 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.norm2 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.norm3 = LayerNorm(d_model, eps=layer_norm_eps, **factory_kwargs)
    self.dropout1 = Dropout(dropout)
    self.dropout2 = Dropout(dropout)
    self.dropout3 = Dropout(dropout)

Why do they add self.dropout1, ...2, ...3 when self.dropout already exists and is the exact same function?
Also, what is the difference between (self.linear1, self.linear2) and self.linear?

Output: That's because to separate one Linear layer or Dropout layer from one another. That's very simple logic. You are creating different instances or layers in the network of the Dropout function using self.dropout = Dropout(dropout).


Instruction: TensorFlow equivalent of PyTorch's transforms.Normalize()

Input:
I'm trying to inference a TFLite model that was originally built in PyTorch. I have been following along the lines of the PyTorch implementation and have to preprocess images along the RGB channels. I found the closest TensorFlow equivalent of transforms.Normalize() to be tf.image.per_image_standardization() (documentation). Although this is a pretty good match, tf.image.per_image_standardization() does this by taking mean and std across the channels and applies it to them. Here's their full implementation from here
def per_image_standardization(image):
  """Linearly scales `image` to have zero mean and unit norm.
  This op computes `(x - mean) / adjusted_stddev`, where `mean` is the average
  of all values in image, and
  `adjusted_stddev = max(stddev, 1.0/sqrt(image.NumElements()))`.
  `stddev` is the standard deviation of all values in `image`. It is capped
  away from zero to protect against division by 0 when handling uniform images.
  Args:
    image: 3-D tensor of shape `[height, width, channels]`.
  Returns:
    The standardized image with same shape as `image`.
  Raises:
    ValueError: if the shape of 'image' is incompatible with this function.
  """
  image = ops.convert_to_tensor(image, name='image')
  _Check3DImage(image, require_static=False)
  num_pixels = math_ops.reduce_prod(array_ops.shape(image))

  image = math_ops.cast(image, dtype=dtypes.float32)
  image_mean = math_ops.reduce_mean(image)

  variance = (math_ops.reduce_mean(math_ops.square(image)) -
              math_ops.square(image_mean))
  variance = gen_nn_ops.relu(variance)
  stddev = math_ops.sqrt(variance)

  # Apply a minimum normalization that protects us against uniform images.
  min_stddev = math_ops.rsqrt(math_ops.cast(num_pixels, dtypes.float32))
  pixel_value_scale = math_ops.maximum(stddev, min_stddev)
  pixel_value_offset = image_mean

  image = math_ops.subtract(image, pixel_value_offset)
  image = math_ops.div(image, pixel_value_scale)
  return image

whereas PyTorch's transforms.Normalize() allows us to mention the mean and std to be applied across each channel like below.
# transformation
    pose_transform = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225]),
    ])

What would be a way to get this functionality in TensorFlow 2.x?
Edit:
I created a quick botch that seems to solve this by defining a function as such:
def normalize_image(image, mean, std):
    for channel in range(3):
        image[:,:,channel] = (image[:,:,channel] - mean[channel])/std[channel]

    return image

I'm not sure how efficient this is but seems to get the job done. I still have to convert the output to a tensor before inputing to the model.

Output:
The workaround that you mentioned seems ok. But using for...loop to compute normalization to each RGB channel for a single image can be a bit problematic when you deal with a large dataset in the data pipeline (generator or tf.data). But it's ok anyway. Here is the demonstration of your approach, and later we will provide two possible alternatives that might work for you easily.
from PIL import Image
from matplotlib.pyplot import imshow, subplot, title, hist

# load image (RGB)
img = Image.open('/content/9.jpg')

def normalize_image(image, mean, std):
    for channel in range(3):
        image[:,:,channel] = (image[:,:,channel] - mean[channel]) / std[channel]
    return image

OP_approach = normalize_image(np.array(img) / 255.0,
                            mean=[0.485, 0.456, 0.406],
                            std=[0.229, 0.224, 0.225])

Now, let's observe the transform properties afterward.
plt.figure(figsize=(25,10))
subplot(121); imshow(OP_approach); title(f'Normalized Image \n min-px: \
    {OP_approach.min()} \n max-pix: {OP_approach.max()}')
subplot(122); hist(OP_approach.ravel(), bins=50, density=True); \
                                    title('Histogram - pixel distribution')

The range of minimum and maximum pixel after normalization are (-2.1179039301310043, 2.6399999999999997) respectively.
Option 2
We can use the tf. keras...Normalization preprocessing layer to do the same. It takes two important arguments which are mean and, variance (square of the std).
from tensorflow.keras.experimental.preprocessing import Normalization

input_data = np.array(img)/255
layer = Normalization(mean=[0.485, 0.456, 0.406],
                      variance=[np.square(0.299),
                                np.square(0.224),
                                np.square(0.225)])

plt.figure(figsize=(25,10))
subplot(121); imshow(layer(input_data).numpy()); title(f'Normalized Image \n min-px: \
   {layer(input_data).numpy().min()} \n max-pix: {layer(input_data).numpy().max()}')
subplot(122); hist(layer(input_data).numpy().ravel(), bins=50, density=True);\
   title('Histogram - pixel distribution')

The range of minimum and maximum pixel after normalization are (-2.0357144, 2.64) respectively.
Option 3
This is more like subtracting the average mean and divide by the average std.
norm_img = ((tf.cast(np.array(img), tf.float32) / 255.0) - 0.449) / 0.226

plt.figure(figsize=(25,10))
subplot(121); imshow(norm_img.numpy()); title(f'Normalized Image \n min-px: \
{norm_img.numpy().min()} \n max-pix: {norm_img.numpy().max()}')
subplot(122); hist(norm_img.numpy().ravel(), bins=50, density=True); \
title('Histogram - pixel distribution')

The range of minimum and maximum pixel after normalization are (-1.9867257, 2.4380531) respectively. Lastly, if we compare to the pytorch way, there is not that much difference among these approaches.
import torchvision.transforms as transforms

transform_norm = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                            std=[0.229, 0.224, 0.225]),
])
norm_pt = transform_norm(img)

plt.figure(figsize=(25,10))
subplot(121); imshow(np.array(norm_pt).transpose(1, 2, 0));\
  title(f'Normalized Image \n min-px: \
  {np.array(norm_pt).min()} \n max-pix: {np.array(norm_pt).max()}')
subplot(122); hist(np.array(norm_pt).ravel(), bins=50, density=True); \
  title('Histogram - pixel distribution')

The range of minimum and maximum pixel after normalization are (-2.117904, 2.64) respectively.


Instruction: Why is using the GPU slower than using the CPU?

Input:
Consider the following network:
%%time
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 400) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()

#input
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)

This outputs
CPU times: user 11.1 s, sys: 3.52 ms, total: 11.1 s
Wall time: 11.1 s

However, when I change to using the GPU with .to(device):
%%time
import torch
from torch.autograd import grad
import torch.nn as nn
import torch.optim as optim
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(1, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 400) #a,b,c,d

        def forward(self, x):
            x=torch.tanh(self.fc1(x))
            x=torch.tanh(self.fc2(x))
            x=self.out(x)
            return x

nx = net_x()
nx.to(device)
#input
val = 100
t = torch.rand(val, requires_grad = True) #input vector
t = torch.reshape(t, (val,1)).to(device) #reshape for batch

#method
dx = torch.autograd.functional.jacobian(lambda t_: nx(t_), t)

This outputs:
CPU times: user 18.6 s, sys: 1.5 s, total: 20.1 s
Wall time: 19.5 s

Update 1:
Checking the timing of the process of moving the inputs and model to device:
%%time
nx.to(device)
t.to(device)

This outputs:
CPU times: user 2.05 ms, sys: 0 ns, total: 2.05 ms
Wall time: 2.13 ms

Update 2:
Looks like @Gulzar was right. I changed my batch size to 1000 (val=1000) and the CPU outputs:
Wall time: 8min 44s
While the GPU outputs:
Wall time: 3min 12s

Output:
A hand-wavey answer
GPUs are "weaker" computers, with much more computing cores than CPUs.
Data has to be passed to them from RAM memory to GRAM in a "costly" manner, every once in a while, so they can process it.
If data is "large", and processing can be parallelized on that data, it is likely computing there will be faster.
If data is not "big enough", the cost of transferring the data, or the cost of using weaker cores and synchronizing them can outweigh the benefit of parallelization.

When will GPU be useful?

For larger networks, or for heavier computations, such as convolutions, or larger fully connected layers (larger matrix multiplications)
For larger batches - batches are a very easy way to parallelize computation, as they are (almost*) independent. *Almost, as they do need to be synchronized programmatically at some point.


Instruction: Why does Dropout not change my input tensor?

Input:
Please see the following code associated with output,
import torch
import torch.nn as nn
inputTensor = torch.tensor([1.0, 2.0, 3, 4, 5])

outplace_dropout = nn.Dropout(p=0.4)
print(inputTensor)
output_afterDropout = outplace_dropout(inputTensor)
print(output_afterDropout)
print(inputTensor)

The output is:
tensor([1., 2., 3., 4., 5.])
tensor([1.6667, 3.3333, 0.0000, 6.6667, 0.0000])
tensor([1., 2., 3., 4., 5.])

Could you please elaborate why the input tensor values are still unchanged?

Output:
From the documentation of torch.nn.Dropout, you can see that the inplace argument defaults to False. If you wish to change the input tensor in place, change the initialization to:
outplace_dropout = nn.Dropout(p=0.4, inplace=True)


Instruction: PyTorch NN : RuntimeError: mat1 dim 1 must match mat2 dim 0

Input:
I'm new to the Neural Network domain and I have stuck on a problem.
I'm trying to create a NN with dropout with 0.1 probability for the hidden fully connected layer.
When I code like below:
class ConvNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = torch.nn.Sequential(

            #layer1
            torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=16),
            torch.nn.ReLU(),

            #layer2
            torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.ReLU(),

            #layer3
            torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.Flatten(),
            torch.nn.Linear(32,16),
            torch.nn.Dropout2d(p=0.1),
            torch.nn.Linear(16, 2)
        )

    def forward(self, x):
        return self.layers(x)

test_convnet = ConvNet().to('cuda')
test_input = torch.randn(16, 3, 100, 100, device='cuda')
test_output = test_convnet(test_input)
print(test_output.shape)

And then I got the error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-27-b5ce1c300266> in <module>()
     48 test_convnet = ConvNet().to('cuda')
     49 test_input = torch.randn(16, 3, 100, 100, device='cuda')
---> 50 test_output = test_convnet(test_input)
     51 print(test_output.shape)

6 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: mat1 dim 1 must match mat2 dim 0
Thanks in advance for all of your help

Output:
In layer 3
 #layer3
            torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.Flatten(),
            torch.nn.Linear(288,16),
            torch.nn.Dropout2d(p=0.1),
            torch.nn.Linear(16, 2)

Corrected Line = torch.nn.Linear(288,16)
This is the wrong shape: torch.nn.Linear(32,16)
Full code:
from argparse import ArgumentParser
import numpy as np
from torch.utils.data import random_split, DataLoader, TensorDataset
import torch
from torch.autograd import Variable
from torchvision import transforms
import torch
from torch import nn
from torch.nn import functional as F
from torch.optim import Adam
from torch.optim.optimizer import Optimizer

class ConvNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = torch.nn.Sequential(

            #layer1
            torch.nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=16),
            torch.nn.ReLU(),

            #layer2
            torch.nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.ReLU(),

            #layer3
            torch.nn.Conv2d(in_channels=32, out_channels=32, kernel_size=3, stride=1),
            torch.nn.MaxPool2d(kernel_size=2, stride=3),
            torch.nn.BatchNorm2d(num_features=32),
            torch.nn.Flatten(),
            torch.nn.Linear(288,16),
            torch.nn.Dropout2d(p=0.1),
            torch.nn.Linear(16, 2)
        )

    def forward(self, x):
        return self.layers(x)

test_convnet = ConvNet()
test_input = torch.randn(16, 3, 100, 100)
test_output = test_convnet(test_input)
print(test_output.shape)


Instruction: Problem with freezing pytorch model - requires_grad is always true

Input:
I have tried to freeze part of my model but it does not work. Gradient computation is still enabled for each layer. Is that some sort of bug or am I doing something wrong? :)
model = models.resnet18(pretrained=True)

# To freeze the residual layers
for param in model.parameters():
    param.require_grad = False
for param in model.fc.parameters():
    param.require_grad = True

# Replace last layer
num_features = model.fc.in_features
model.fc = nn.Linear(num_features, 2)
model.fc = nn.Dropout(0.5)

# Find total parameters and trainable parameters
total_params = sum(p.numel() for p in model.parameters())
print(f'{total_params:,} total parameters.')
# >>> 21,284,672 total parameters.

total_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'{total_trainable_params:,} training parameters.')
# >>> 21,284,672 training parameters.

Output:
This is because of a typo. Change require_grad to requires_grad:
for param in model.parameters():
    param.requires_grad = False
for param in model.fc.parameters():
    param.requires_grad = True

Currently, you are declaring a new attribute for the model and assigning it to True and False as appropriate, so it has no effect.


Instruction: How to efficiently multiply by torch tensor with repeated rows without storing all the rows in memory or iterating?

Input:
Given a torch tensor:
# example tensor size 2 x 4
a = torch.Tensor([[1, 2, 3, 4], [5, 6, 7, 8]])

and another where every n rows are repeated:
# example tensor size 4 x 3 where every 2 rows repeated
b = torch.Tensor([[1, 2, 3], [4, 5, 6], [1, 2, 3], [4, 5, 6]])

how can one perform matrix multiplication:
>>> torch.mm(a, b)
tensor([[ 28.,  38.,  48.],
        [ 68.,  94., 120.]])

without copying the whole repeated row tensor into memory or iterating?
i.e. only store the first 2 rows:
# example tensor size 2 x 3 where only the first two rows from b are actually stored in memory
b_abbreviated = torch.Tensor([[1, 2, 3], [4, 5, 6]])

since these rows will be repeated.
There is a function
torch.expand()

but this does work when repeating more than a single row, and also, as this question:
Repeating a pytorch tensor without copying memory
indicates and my own tests confirm often ends up copying the whole tensor into memory anyway when calling
.to(device)

It is also possible to do this iteratively, but this is relatively slow.
Is there some way to perform this operation efficiently without storing the whole repeated row tensor in memory?
Edit explanation:
Sorry, for not initially clarifying: One was used as the first dimension of the first tensor to keep the example simple, but I am actually looking for a solution to the general case for any two tensors a and b such that their dimensions are compatible for matrix multiplication and the rows of b repeat every n rows. I have updated the example to reflect this.

Output:
Assuming that the first dimension of a is 1 as in your example, you could do the following:
a = torch.Tensor([[1, 2, 3, 4]])
b_abbreviated = torch.Tensor([[1, 2, 3], [4, 5, 6]])
torch.mm(a.reshape(-1, 2), b_abbreviated).sum(axis=0, keepdim=True)

Here, instead of repeating the rows, you multiply a in chunks, then add them up column-wise to get the same result.

If the first dimension of a is not necessarily 1, you could try the following:
torch.cat(torch.split(torch.mm(a.reshape(-1,2),b_abbreviated), a.shape[0]), dim=1).sum(
dim=0, keepdim=True).reshape(a.shape[0], -1)

Here, you do the following:

With torch.mm(a.reshape(-1,2),b_abbreviated, you again split each row of a into chunks of size 2 and stack them one over the other, and then stack each row over the other.
With torch.split(torch.mm(a.reshape(-1,2),b_abbreviated), a.shape[0]), these stacks are then separated row-wise, so that each resultant component of the split corresponds to chunks of a single row.
With torch.cat(torch.split(torch.mm(a.reshape(-1,2),b_abbreviated), a.shape[0]), dim=1) these stacks are then concatenated column-wise.
With .sum(dim=0, keepdim=True), results corresponding to separate chunks of individual rows in a are added up.
With .reshape(a.shape[0], -1), rows of a that were concatenated column-wise are again stacked row-wise.

It seems quite slow compared to direct matrix multiplication, which is not surprising, but I have not yet checked in comparison to explicit iteration. There are likely better ways of doing this, will edit if I think of any.