Instruction: How to solve dist.init_process_group from hanging (or deadlocks)?

Input:
I was to set up DDP (distributed data parallel) on a DGX A100 but it doesn't work. Whenever I try to run it simply hangs. My code is super simple just spawning 4 processes for 4 gpus (for the sake of debugging I simply destroy the group immediately but it doesn't even reach there):
def find_free_port():
    """ https://stackoverflow.com/questions/1365265/on-localhost-how-do-i-pick-a-free-port-number """
    import socket
    from contextlib import closing

    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return str(s.getsockname()[1])

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.

    export NCCL_SOCKET_IFNAME=eth0
    export NCCL_IB_DISABLE=1

    https://stackoverflow.com/questions/61075390/about-pytorch-nccl-error-unhandled-system-error-nccl-version-2-4-8

    https://pytorch.org/docs/stable/distributed.html#common-environment-variables
    """
    if rank != -1:  # -1 rank indicates serial code
        print(f'setting up rank={rank} (with world_size={world_size})')
        # MASTER_ADDR = 'localhost'
        MASTER_ADDR = '127.0.0.1'
        MASTER_PORT = find_free_port()
        # set up the master's ip address so this child process can coordinate
        os.environ['MASTER_ADDR'] = MASTER_ADDR
        print(f"{MASTER_ADDR=}")
        os.environ['MASTER_PORT'] = MASTER_PORT
        print(f"{MASTER_PORT}")

        # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
        if torch.cuda.is_available():
            # unsure if this is really needed
            # os.environ['NCCL_SOCKET_IFNAME'] = 'eth0'
            # os.environ['NCCL_IB_DISABLE'] = '1'
            backend = 'nccl'
        print(f'{backend=}')
        # Initializes the default distributed process group, and this will also initialize the distributed package.
        dist.init_process_group(backend, rank=rank, world_size=world_size)
        # dist.init_process_group(backend, rank=rank, world_size=world_size)
        # dist.init_process_group(backend='nccl', init_method='env://', world_size=world_size, rank=rank)
        print(f'--> done setting up rank={rank}')
        dist.destroy_process_group()

mp.spawn(setup_process, args=(4,), world_size=4)

why is this hanging?
nvidia-smi output:
$ nvidia-smi
Fri Mar  5 12:47:17 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04   Driver Version: 450.102.04   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      On   | 00000000:07:00.0 Off |                    0 |
| N/A   26C    P0    51W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   1  A100-SXM4-40GB      On   | 00000000:0F:00.0 Off |                    0 |
| N/A   25C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   2  A100-SXM4-40GB      On   | 00000000:47:00.0 Off |                    0 |
| N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   3  A100-SXM4-40GB      On   | 00000000:4E:00.0 Off |                    0 |
| N/A   25C    P0    51W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   4  A100-SXM4-40GB      On   | 00000000:87:00.0 Off |                    0 |
| N/A   30C    P0    52W / 400W |      3MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   5  A100-SXM4-40GB      On   | 00000000:90:00.0 Off |                    0 |
| N/A   29C    P0    53W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   6  A100-SXM4-40GB      On   | 00000000:B7:00.0 Off |                    0 |
| N/A   29C    P0    52W / 400W |      0MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+
|   7  A100-SXM4-40GB      On   | 00000000:BD:00.0 Off |                    0 |
| N/A   48C    P0   231W / 400W |   7500MiB / 40537MiB |     99%      Default |
|                               |                      |             Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    7   N/A  N/A    147243      C   python                           7497MiB |
+-----------------------------------------------------------------------------+

How do I set up ddp in this new machine?

Update
btw I've successfully installed APEX because some other links say to do that but it still fails. For I did:
went to: https://github.com/NVIDIA/apex follwed their instructions
git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

but before the above I had to update gcc:
conda install -c psi4 gcc-5

it did install it as I successfully imported it but it didn't help.

Now it actually prints an error msg:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
KeyboardInterrupt
Process SpawnProcess-3:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 252, in train
    setup_process(rank, world_size=opts.world_size)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/distributed.py", line 85, in setup_process
    dist.init_process_group(backend, rank=rank, world_size=world_size)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 436, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/home/miranda9/miniconda3/envs/metalearning/lib/python3.8/site-packages/torch/distributed/rendezvous.py", line 179, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: connect() timed out.

During handling of the above exception, another exception occurred:

related:

https://github.com/pytorch/pytorch/issues/9696
https://discuss.pytorch.org/t/dist-init-process-group-hangs-silently/55347/2
https://forums.developer.nvidia.com/t/imagenet-hang-on-dgx-1-when-using-multiple-gpus/61919
apex suggestion: https://discourse.mozilla.org/t/hangs-on-dist-init-process-group-in-distribute-py/44686
https://github.com/pytorch/pytorch/issues/15638
https://github.com/pytorch/pytorch/issues/53395

Output:
The following fixes are based on Writing Distributed Applications with PyTorch, Initialization Methods.
Issue 1:
It will hang unless you pass in nprocs=world_size to mp.spawn().  In other words, it's waiting for the "whole world" to show up, process-wise.

Issue 2:
The MASTER_ADDR and MASTER_PORT need to be the same in each process' environment and need to be a free address:port combination on the machine where the process with rank 0 will be run.

Both of these are implied or directly read from the following quote from the link above (emphasis added):

Environment Variable
We have been using the environment variable initialization method
throughout this tutorial. By setting the following four environment
variables on all machines, all processes will be able to properly
connect to the master, obtain information about the other processes,
and finally handshake with them.
MASTER_PORT: A free port on the machine that will host the process with rank 0.
MASTER_ADDR: IP address of the machine that will host the process with rank 0.
WORLD_SIZE: The total number of processes, so that the master knows how many workers to wait for.
RANK: Rank of each process, so they will know whether it is the master of a worker.

Here's some code to demonstrate both of those in action:
import torch
import torch.multiprocessing as mp
import torch.distributed as dist
import os

def find_free_port():
    """ https://stackoverflow.com/questions/1365265/on-localhost-how-do-i-pick-a-free-port-number """
    import socket
    from contextlib import closing

    with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
        s.bind(('', 0))
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        return str(s.getsockname()[1])

def setup_process(rank, master_addr, master_port, world_size, backend='gloo'):
    print(f'setting up {rank=} {world_size=} {backend=}')

    # set up the master's ip address so this child process can coordinate
    os.environ['MASTER_ADDR'] = master_addr
    os.environ['MASTER_PORT'] = master_port
    print(f"{master_addr=} {master_port=}")

    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)
    print(f"{rank=} init complete")
    dist.destroy_process_group()
    print(f"{rank=} destroy complete")

if __name__ == '__main__':
    world_size = 4
    master_addr = '127.0.0.1'
    master_port = find_free_port()
    mp.spawn(setup_process, args=(master_addr,master_port,world_size,), nprocs=world_size)


Instruction: unable to import pytorch-lightning

Input:
I installed pytorch-lightning using pip, and I'm running on Mac.
I tried:
! pip install pytorch-lightning --upgrade
! pip install pytorch-lightning-bolts

(finished successfully)
and then:
import pytorch_lightning as pl

and what I get is:
--
-------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-f3b4217dcea1> in <module>
      7 from torchvision.datasets import MNIST
      8 from torchvision import transforms
----> 9 import pytorch_lightning as pl
     10 from pytorch_lightning.metrics.functional import accuracy
     11 tmpdir = os.getcwd()

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/__init__.py in <module>
     60     # We are not importing the rest of the lightning during the build process, as it may not be compiled yet
     61 else:
---> 62     from pytorch_lightning import metrics
     63     from pytorch_lightning.callbacks import Callback
     64     from pytorch_lightning.core import LightningDataModule, LightningModule

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.classification import (  # noqa: F401
     15     Accuracy,
     16     AUC,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.classification.accuracy import Accuracy  # noqa: F401
     15 from pytorch_lightning.metrics.classification.auc import AUC  # noqa: F401
     16 from pytorch_lightning.metrics.classification.auroc import AUROC  # noqa: F401

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/accuracy.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.metrics.functional.accuracy import _accuracy_compute, _accuracy_update
     19 from pytorch_lightning.metrics.metric import Metric
     20

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.functional.accuracy import accuracy  # noqa: F401
     15 from pytorch_lightning.metrics.functional.auc import auc  # noqa: F401
     16 from pytorch_lightning.metrics.functional.auroc import auroc  # noqa: F401

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/accuracy.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.metrics.classification.helpers import _input_format_classification, DataType
     19
     20

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/helpers.py in <module>
     17 import torch
     18
---> 19 from pytorch_lightning.metrics.utils import select_topk, to_onehot
     20 from pytorch_lightning.utilities import LightningEnum
     21

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/utils.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.utilities import rank_zero_warn
     19
     20 METRIC_EPS = 1e-6

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/__init__.py in <module>
     16 import numpy
     17
---> 18 from pytorch_lightning.utilities.apply_func import move_data_to_device  # noqa: F401
     19 from pytorch_lightning.utilities.distributed import (  # noqa: F401
     20     AllGatherGrad,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py in <module>
     23
     24 from pytorch_lightning.utilities.exceptions import MisconfigurationException
---> 25 from pytorch_lightning.utilities.imports import _TORCHTEXT_AVAILABLE
     26
     27 if _TORCHTEXT_AVAILABLE:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py in <module>
     54 _TORCH_GREATER_EQUAL_1_7 = _compare_version("torch", operator.ge, "1.7.0")
     55 _TORCH_QUANTIZE_AVAILABLE = bool([eg for eg in torch.backends.quantized.supported_engines if eg != 'none'])
---> 56 _APEX_AVAILABLE = _module_available("apex.amp")
     57 _BOLTS_AVAILABLE = _module_available('pl_bolts')
     58 _DEEPSPEED_AVAILABLE = not _IS_WINDOWS and _module_available('deepspeed')

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py in _module_available(module_path)
     32     """
     33     try:
---> 34         return find_spec(module_path) is not None
     35     except AttributeError:
     36         # Python 3.6

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/util.py in find_spec(name, package)
     92         parent_name = fullname.rpartition('.')[0]
     93         if parent_name:
---> 94             parent = __import__(parent_name, fromlist=['__path__'])
     95             try:
     96                 parent_path = parent.__path__

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/apex/__init__.py in <module>
     11                                 ISessionFactory)
     12 from pyramid.security import NO_PERMISSION_REQUIRED
---> 13 from pyramid.session import UnencryptedCookieSessionFactoryConfig
     14 from pyramid.settings import asbool
     15

ImportError: cannot import name 'UnencryptedCookieSessionFactoryConfig' from 'pyramid.session' (unknown location

Output: I guess this is an outdated issue as we have cut out TorchMetrics to a standalone package. Please, check out the latest PytorchLightning.


Instruction: pytorch deep learning loading data sequentially and efficiently

Input:
I have been doing neural network analysis on 20 thousand "images", each image represented in the form of the intensity of 100 * 100 * 100 neurons.
x = np.loadtxt('imgfile')
x = x.reshape(-1, img_channels, 100, 100, 100)
//similarly for target variable 'y'

Above, the first dimension of x will be the number of images. Am using DataLoader to get appropriate number of images for training during each iteration as shown below.
batch_size = 16
traindataset = TensorDataset(Tensor(x[:-testdatasize]), Tensor(y[:-testdatasize]) )
train_loader = DataLoader(dataset=traindataset, batch_size=batch_size, shuffle=True)

for epoch in range(num_epochs):
    for i, (data,targets) in enumerate(train_loader):
        ...

I hope to increase the number of images to 50k but am restricted by the computer memory  (imgfile is ~50 GB).
I was wondering if there is an efficient way to handle all the data? Like, rather than loading the whole imgfile, can we first divide them into sets, each with batch_size number of images, and load the sets periodically during training. I am not completely sure how to implement this.
I found some similar ideas using Keras here: https://machinelearningmastery.com/how-to-load-large-datasets-from-directories-for-deep-learning-with-keras/
Please point me towards any similar ideas implemented with pytorch or you have any ideas.

Output:
Digging a while after posting the question, found out there is, of course, a way using torch.utils.data.Dataset. Each image-data can be saved in a separate file and all the filenames are listed in 'filelistdata'. Only the batch_size number of images will be loaded into memory when called using DataLoader (in the background, getitem method will fetch the images). The following worked for me:
traindataset = CustDataset(filename='filelistdata', root_dir=root_dir)

train_loader = DataLoader(dataset=traindataset, batch_size=batch_size, num_workers = 16)

num_workers is really important for performance and should be higher than the number of Cpus you are using (I am using 4 cpus above). Found the following resources useful for answering this question.
How to split and load huge dataset that doesn't fit into memory into pytorch Dataloader?
https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel
https://www.youtube.com/watch?v=ZoZHd0Zm3RY


Instruction: How to do elementwise comparison sum without for loop

Input:
The for loop makes my program very slow. I would've used np.sum(target==output) but I need the argmax value for each row in the output. How can I speed this up?
The output is a tensor data type
for i, x in enumerate(target):
  if target[i] == torch.argmax(output[i]):
    correct_class += 1

Output:
You could vectorize the above using np.argmax's axis argument, to obtain the indices of the maximum value across the rows:
(target==np.argmax(output, axis=1)).sum()

For instance:
output = np.random.choice([0,1],(4,2))
print(output)
array([[1, 1],
       [0, 1],
       [0, 1],
       [0, 1]])
target = np.array([[0,1,0,1]])
(target==np.argmax(output, axis=1)).sum()
# 3


Instruction: MMDetection loading from own training checkpoint for inference produces garbage detections

Input:
I've trained up a very simple model using the MMDetection colab tutorial and then verifying the result using:
img = mmcv.imread('/content/mmdetection/20210301_145246_123456.jpg')
img = cv2.resize(img, (0,0), fx=0.25, fy=0.25)

model.cfg = cfg
result = inference_detector(model, img)
show_result_pyplot(model, img, result)

confirms that it's working great.
I then follow the same steps as for training but instead I load my own training checkpoint, and I don't train. Then running the verification snippet above produces garbage results.
Here's that in code
from mmcv import Config
cfg = Config.fromfile('configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.py')

from mmdet.apis import set_random_seed

# Modify dataset type and path
cfg.dataset_type = 'SamplesDataset'
cfg.data_root = 'samples_dataset/'

cfg.data.test.type = 'SamplesDataset'
cfg.data.test.data_root = 'samples_dataset/'
cfg.data.test.ann_file = 'train.txt'
cfg.data.test.img_prefix = 'o2h'

cfg.data.train.type = 'SamplesDataset'
cfg.data.train.data_root = 'samples_dataset/'
cfg.data.train.ann_file = 'train.txt'
cfg.data.train.img_prefix = 'o2h'

cfg.data.val.type = 'SamplesDataset'
cfg.data.val.data_root = 'samples_dataset/'
cfg.data.val.ann_file = 'val.txt'
cfg.data.val.img_prefix = 'o2h'

# modify num classes of the model in box head
cfg.model.roi_head.bbox_head.num_classes = 1
# We can still use the pre-trained Mask RCNN model though we do not need to
# use the mask branch
# cfg.load_from = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'
cfg.load_from = './experiments/epoch_1.pth'

# Set up working dir to save files and logs.
cfg.work_dir = './experiments'

# The original learning rate (LR) is set for 8-GPU training.
# We divide it by 8 since we only use one GPU.
cfg.optimizer.lr = 0.02 / 8
cfg.lr_config.warmup = None
cfg.log_config.interval = 10
cfg.runner = dict(type='EpochBasedRunner', max_epochs=1)
cfg.total_epochs = 1

# Change the evaluation metric since we use customized dataset.
cfg.evaluation.metric = 'mAP'
# We can set the evaluation interval to reduce the evaluation times
# cfg.evaluation.interval = 12
# We can set the checkpoint saving interval to reduce the storage cost
cfg.checkpoint_config.interval = 1

# Set seed thus the results are more reproducible
cfg.seed = 0
set_random_seed(0, deterministic=False)
cfg.gpu_ids = range(1)

# We can initialize the logger for training and have a look
# at the final config used for training
# print(f'Config:\n{cfg.pretty_text}')

from mmdet.datasets import build_dataset
from mmdet.models import build_detector
from mmdet.apis import train_detector

# Build dataset
# datasets = [build_dataset(cfg.data.train)]

# Build the detector
model = build_detector(cfg.model)
# Add an attribute for visualization convenience
model.CLASSES = datasets[0].CLASSES

# Create work_dir
# mmcv.mkdir_or_exist(osp.abspath(cfg.work_dir))
# train_detector(model, datasets, cfg, distributed=False, validate=True)

Obviously, I wouldn't normally do all that just for validating my model, but this is one of many debugging steps for me, as my goal is to download and run the model locally. This is what I'm trying to do locally:
import sys
import glob
import time

sys.path.insert(0, '../mmdetection')
from mmdet.apis import init_detector, inference_detector, show_result_pyplot
from mmdet.models import build_detector
import mmcv
import numpy as np

file_paths = glob.glob('samples/o2h/*.jpg')

cfg = mmcv.Config.fromfile('../mmdetection/configs/faster_rcnn/faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.py')
cfg.model.roi_head.bbox_head.num_classes = 1
cfg.load_from = 'models/mmdet_faster_rcnn_r50_caffe_fpn_mstrain_1x_coco.pth' # my own checkpoint
model = build_detector(cfg.model)
model.CLASSES = ('hash',)
model.cfg = cfg

file_path = np.random.choice(file_paths)
print(file_path)

start = time.time()
result = inference_detector(model, file_path)
print(f"Time taken for inference: {time.time() - start:.2f}s")
show_result_pyplot(model, file_path, result)

Output:
One of the mistakes in your code is that you have not updated num_classes for mask_head.
Our aim here should be to replicate the same config file that was used for training should also be used for testing/validation. If you have trained the model using 1 num_classes for bbox_head and mask_head in the config file but for validation/testing you are using 80 num_classes as default, then that will cause a mismatch in the testing process, leading to garbage detections and segmentations.
There are 2 solutions for achieving the best result:

Change the num_classes in config file before doing inference
Save the model and config file as pickle as soon as training is completed.

  Note: The first solution is best one.

Change the num_classes in config file before doing inference.
First, find the total number of classes in your dataset. Here num_classes is total number of classes in the training dataset.
Locate to this path:
mmdetection/configs/model_name (model_name is name used for training)
Here, inside model_name folder, find the ..._config.py that you have used for training.
Inside this config file, if you have found model = dict(...) then change the num_classes for each of these keys: bbox_head, mask_head.
bbox_head might be list. so, change num_classes for each keys in the list.
If model = dict(...) is not found, then at the first line there is
_base_ = '...' So, open that config file and check whether model=dict(...) is found or not. If not found keep on opening the file location of _base_.
After changing the num_classes, use this code for inference:

Code after changing the num_classes:
from mmdet.apis import init_detector, inference_detector
import mmcv
import numpy as np
import cv2
import os
import matplotlib.pyplot as plt
%matplotlib inline

config_file = './configs/scnet/scnet_x101_64x4d_fpn_20e_coco.py' #(I have used SCNet for training)

checkpoint_file = 'tutorial_exps/epoch_40.pth' #(checkpoint saved after training)

model = init_detector(config_file, checkpoint_file, device='cuda:0') #loading the model

img = 'test.png'

result = inference_detector(model, img)

#visualize the results in a new window
im1 = cv2.imread(img)[:,:,::-1]
#im_ones = np.ones(im1.shape, dtype='uint')*255
# model.show_result(im_ones, result, out_file='fine_result6.jpg')
plt.imshow(model.show_result(im1, result))

Save the model and config as pickle as soon as training is completed.
Another solution is to save both model and config as pickle as soon as the training is completed, irrespective of depending on mmdetection to do it.
When you load default config file (without any change), it won't produce the required result. Your config should be exact to the config used for training. So, it is good idea to save the model and config as pickle instead of loading them.
 Note: The pickle files should be saved right after training is completed.

 Code for saving as pickle:
import pickle

with open('mdl.pkl','wb') as f:
    pickle.dump(model, f)

with open('cfg.pkl','wb') as f:
    pickle.dump(cfg, f)

You can use this model wherever and whenever you want. For inference with the saved model, use this:
import pickle, mmcv
from mmdet.apis import inference_detector, show_result_pyplot

model = pickle.load(open('mdl.pkl','rb'))
cfg = pickle.load(open('cfg.pkl','rb'))

img = mmcv.imread('images/test.png')

model.cfg = cfg
result = inference_detector(model, img)
show_result_pyplot(model, img, result)


Instruction: Unable to import pytorch_lightning on google colab

Input:
I have done the following:
!pip install pytorch_lightning -qqq
import pytorch_lightning

But get the following error:
ImportError                               Traceback (most recent call last)
<ipython-input-7-d883b15aac58> in <module>()
----> 1 import pytorch_lightning

----------------------------------9 frames------------------------------------------------
/usr/local/lib/python3.7/dist-packages/pytorch_lightning/utilities/apply_func.py in <module>()
     26
     27 if _TORCHTEXT_AVAILABLE:
---> 28     from torchtext.data import Batch
     29 else:
     30     Batch = type(None)

ImportError: cannot import name 'Batch' from 'torchtext.data' (/usr/local/lib/python3.7/dist-packages/torchtext/data/__init__.py)

What could the issue be?

Output:
As said in Issue #6415 on Github, try installing from the GitHub.
It worked for me.
!pip install git+https://github.com/PyTorchLightning/pytorch-lightning
import pytorch_lightning as pl
print(pl.__version__)

Output:
1.3.0dev

It seems that the error is coming from Issue #6210 and they say it was fixed. I guess it wasn't uploaded to PyPi.


Instruction: One hot encoding in pytorch

Input:
I am really new to coding, right now I am trying to turn my label to one hot encoding. I have already done transferring the np.array to tensor as shown below

tensor([4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 4., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 3., 3., 3., 3., 3., 3.,
        3., 3., 3., 3., 3., 3., 3., 3., 3., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.,
        2., 2., 2.], dtype=torch.float64)

and I am using code to do one hot encoding

aaa = F.one_hot(torch_qyh, num_classes=5)

However, there is an error showing "RuntimeError: one_hot is only applicable to index tensor." Any help would appreciated.

Output:
You will have to convert it in long type. Can't do it with float. F.one_hot only takes LongTensor.
F.one_hot(t.long())


Instruction: How to run a pytorch model server inside docker?

Input:
I am trying to create a docker image to make it run as a server for serving a model in pytorch.
I converted the .pt model file to .MAR file in my local machine
and i copied the .MAR file inside the docker image. I created a dockerfile:
FROM ubuntu:18.04
ENV TZ=Asia/Shanghai
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
    && apt-get install --yes --no-install-recommends \
    tzdata

RUN ln -snf /usr/share/zoneinfo/${TZ} /etc/localtime\
    && echo ${TZ} > /etc/timezone \
    && dpkg-reconfigure -f noninteractive tzdata

RUN apt-get install python3 python3-dev python3-pip openjdk-11-jre-headless git wget curl -y
RUN python3 -m pip install torch torchvision torch-model-archiver torchserve==0.2.0
COPY densenet161.mar /model_store/
CMD torchserve --start --model-store model_store --models densenet161=densenet161.mar
EXPOSE 8080

I was able to create the image but I was not able to access that container
when I tried to open the image and run the code it works
docker exec -it 4b126bd87f21 sh

# torchserve --start --ncs --model-store model_store --models densenet161.mar

The server is running. When I run the docker image it is not working. Docker container is running but I was not able to access the server.
I don't know what is the problem.

Output:
I hope I understand the problem.
When you do docker run torchserve:local ...., by default it runs the CMD which is torchserve --start --model-store model_store --models densenet161=densenet161.mar but since the command runs in the background, your newly created docker container will immediately exit. Due to this same problem, i.e. to prevent docker exit it is possible to add tail -f /dev/null.
Look at the official docker entry point of torchserve https://github.com/pytorch/serve/blob/master/docker/dockerd-entrypoint.sh#L12
They tail it at the end to prevent docker exit


Instruction: how to import a torch 1.7.1 when torch 1.4.0 is also installed

Input:
how to import a torch 1.7.1 when torch 1.4.0 is also installed
When I run the command: ! pip list
It lists all libraries with : torch    1.7.1
Now when I run:
>>>import torch
>>>torch.__version__
'1.4.0'

How Do I import torch==1.7.1 in the python program?
I am using python 3.8.3 and windows 10

Output:
Slightly different way to answer your question, but if you want to have two versions of torch installed simultaneously for different purposes (e.g. running different programs), my recommendation would be to use torch 1.7.1 and torch 1.4.1 in separate virtual environments.  Please see the links below for guides on getting started with these:

virtualenv:

# Create virtual environment
python3 -m venv torch-env

# Activate virtual environment
torch-env\Scripts\activate.bat

# Any packages you install now are ONLY for this virtual environment
pip install torch==1.7.1  # Or 1.4.1 in a separate environment

conda:

# Create virtual environment with Python 3.8.3
conda create --name torch-env python=3.8.3

# Activate virtual environment
conda activate torch-env

# Install torch
pip install torch==1.7.1  # Or 1.4.1

Creating separate virtual environments will allow you to have multiple versions of torch simultaneously without any issues.  You should then be able to import as you do above and use them interchangeably simply by switching environments.


Instruction: Understanding Jacobian tensor gradients in pytorch

Input:
I was going through official pytorch tut, where it explains tensor gradients and Jacobian products as follows:

Instead of computing the Jacobian matrix itself, PyTorch allows you to compute Jacobian Product  for a given input vector v=(v1…vm). This is achieved by calling backward with v as an argument:

inp = torch.eye(5, requires_grad=True)
out = (inp+1).pow(2)
out.backward(torch.ones_like(inp), retain_graph=True)
print("First call\n", inp.grad)
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nSecond call\n", inp.grad)
inp.grad.zero_()
out.backward(torch.ones_like(inp), retain_graph=True)
print("\nCall after zeroing gradients\n", inp.grad)

Ouptut:
First call
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])

Second call
 tensor([[8., 4., 4., 4., 4.],
        [4., 8., 4., 4., 4.],
        [4., 4., 8., 4., 4.],
        [4., 4., 4., 8., 4.],
        [4., 4., 4., 4., 8.]])

Call after zeroing gradients
 tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.],
        [2., 2., 2., 2., 4.]])

Though I get what is Jacobian matrix is, I didnt get how is this Jacobian product is calculated.
Here, are different tensors I tried to print out to get understanding:
>>> out
tensor([[4., 1., 1., 1., 1.],
        [1., 4., 1., 1., 1.],
        [1., 1., 4., 1., 1.],
        [1., 1., 1., 4., 1.],
        [1., 1., 1., 1., 4.]], grad_fn=<PowBackward0>)
>>> torch.eye(5)
tensor([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]])
>>> torch.ones_like(inp)
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])
>>> inp
tensor([[1., 0., 0., 0., 0.],
        [0., 1., 0., 0., 0.],
        [0., 0., 1., 0., 0.],
        [0., 0., 0., 1., 0.],
        [0., 0., 0., 0., 1.]], requires_grad=True)

But I didnt get how the tuts output is calculated. Can someone explain a bit of Jacobian matrix with calculations done in this example?

Output:
We will go through the entire process: from computing the Jacobian to applying it to get the resulting gradient for this input. We're looking at the operation f(x) = (x + 1)², in the simple scalar setting, we get df/dx = 2(x + 1) as complete derivative.
In the multi-dimensional setting, we have an input x_ij, and an output y_mn, indexed by (i, j), and (m, n) respectively. The function mapping is defined as y_mn = (x_mn + 1)².
First, we should look at the Jacobian itself, this corresponds to the tensor J containing all partial derivatives J_ijmn = dy_mn/dx_ij. From the expression of y_mn we can say that for all i, j, m, and n: dy_mn/dx_ij = d(x_mn + 1)²/dx_ij which is 0 if m≠i or n≠j. Else, i.e. m=i or n=j, we have that d(x_mn + 1)²/dx_ij = d(x_ij + 1)²/dx_ij = 2(x_ij + 1).
As a result, J_ijmn can be simply defined as
         ↱ 2(x_ij + 1) if i=m, j=n
J_ijmn =
         ↳ 0 else

From the rule chain the gradient of the output with respect to the input x is denoted as dL/dx = dL/dy*dy/dx. From a PyTorch perspective we have the following relationships:

x.grad = dL/dx, shaped like x,
dL/dy is the incoming gradient: the gradient argument in the backward function
dL/dx is the Jacobian tensor described above.

As explained in the documentation, applying backward doesn't actually provide the Jacobian. It computes the chain rule product directly and stores the gradient (i.e. dL/dx inside x.grad).
In terms of shapes, the Jacobian multiplication dL/dy*dy/dx = gradient*J reduces itself to a tensor of the same shape as x.
The operation performed is defined by: [dL/dx]_ij = ∑_mn([dL/dy]_ij * J_ijmn).

If we apply this to your example. We have x = 1(i=j) (where 1(k): (k == True) -> 1 is the indicator function), essentially just the identity matrix.
We compute the Jacobian:
         ↱ 2(1(i=j) + 1) =  if i=m, j=n
J_ijmn =
         ↳ 0 else

which becomes
         ↱ 2(1 + 1) = 4  if i=j=m=n
J_ijmn = → 2(0 + 1) = 2  if i=m, j=n, i≠j
         ↳ 0 else

For visualization purposes, we will stick with x = torch.eye(2):
>>> f = lambda x: (x+1)**2
>>> J = A.jacobian(f, inp)
tensor([[[[4., 0.],
          [0., 0.]],

         [[0., 2.],
          [0., 0.]]],

        [[[0., 0.],
          [2., 0.]],

         [[0., 0.],
          [0., 4.]]]])

Then computing the matrix multiplication using torch.einsum (I won't go into details, look through this, then this for an in-depth overview of the EinSum summation operator):
>>> torch.einsum('ij,ijmn->mn', torch.ones_like(inp), J)
tensor([[4., 2.],
        [2., 4.]])

This matches what you get when back propagating from out with torch.ones_like(inp) as incoming gradient:
>>> out = f(inp)
>>> out.backward(torch.ones_like(inp))
>>> inp.grad
tensor([[4., 2.],
        [2., 4.]])

If you backpropagate twice (while retaining the graph of course) you end up computing the same operation which accumulating on the parameter's grad attribute. So, naturally, after two backward passes you have twice the gradient:
>>> out = f(inp)
>>> out.backward(torch.ones_like(inp), retain_graph=True)
>>> out.backward(torch.ones_like(inp))
>>> inp.grad
tensor([[8., 4.],
        [4., 8.]])

Those gradients will accumulate, you can reset them by calling the inplace function zero_: inp.grad.zero_(). From there if you backpropagate again you will stand with one accumulate gradient only.
In practice, you would register your parameters on an optimizer, from which you can call zero_grad enabling you to handle and reset all parameters in that collection in one go.

I have imported torch.autograd.functional as A


Instruction: HTTP Error when trying to download MNIST data

Input:
I am using Google Colab for training a LeNet-300-100 fully-connected neural network on MNIST using Python3 and PyTorch 1.8.
To apply the transformations and download the MNIST dataset, the following code is being used:
# MNIST dataset statistics:
# mean = tensor([0.1307]) & std dev = tensor([0.3081])
mean = np.array([0.1307])
std_dev = np.array([0.3081])

transforms_apply = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean = mean, std = std_dev)
    ])

which gives the error:

Downloading
http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to
./data/MNIST/raw/train-images-idx3-ubyte.gz
--------------------------------------------------------------------------- HTTPError                                 Traceback (most recent call
last)  in ()
2 train_dataset = torchvision.datasets.MNIST(
3         root = './data', train = True,
----> 4         transform = transforms_apply, download = True
5         )
6
11 frames /usr/lib/python3.7/urllib/request.py in
http_error_default(self, req, fp, code, msg, hdrs)
647 class HTTPDefaultErrorHandler(BaseHandler):
648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
650
651 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 503: Service Unavailable

What's wrong?

Output:
I was having the same 503 error and this worked for me
!wget www.di.ens.fr/~lelarge/MNIST.tar.gz
!tar -zxvf MNIST.tar.gz

from torchvision.datasets import MNIST
from torchvision import transforms

train_set = MNIST('./', download=True,
transform=transforms.Compose([
transforms.ToTensor(),
]), train=True)

test_set = MNIST('./', download=True,
transform=transforms.Compose([
transforms.ToTensor(),
]), train=False)


Instruction: Bound optimization using pytorch

Input:
How to include bounds when using optimization method in pytorch. I have a tensor of variables, each variable has different bound.
upper_bound = torch.tensor([1,5,10], requires_grad=False)
lower_bound = torch.tensor([-1,-5,-10], requires_grad=False)
X           = torch.tensor([10, -60, 105], require_grad=True)
for _ in range(100):
    optimizer.zero_grad()
    loss = ..
    loss.backward()
    optimizer.step()
    X[:] = X.clamp(lower_bound, upper_bound)

But, clamp only uses a single number. Since each variable is bounded differently, I need to include the upper and lower bounds tensors.

Output:
Gradient descent is not the best method to achieve constrained optimization, but here you can enforce your constraints with :
x = ((X-lower_bound).clamp(min=0)+lower_bound-upper_bound).clamp(max=0)+upper_bound

Requires two clamp instead of one but I could not find any native way to achieve this.


Instruction: RuntimeError: mean is not implemented for type torch.ByteTensor

Input:
I am getting this error after running my code:
"RuntimeError: mean is not implemented for type torch.ByteTensor"?
Do anyone know what I am doing wrong here?
accuracy = torch.mean(output)

Output:
Got it, basically torch.mean() isn't implemented on torch.ByteTensor so we can convert it to FloatTensor which is supported by torch.mean().
So the code will change to:
accuracy = torch.mean(output.type(torch.FloatTensor))


Instruction: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED using pytorch

Input:
I am trying to run a simple pytorch sample code. It's works fine using CPU. But when using GPU, i get this error message:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 263, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 260, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

The code i am trying to run is the following:
import torch
from torch import nn
m = nn.Conv1d(16, 33, 3, stride=2)
m=m.to('cuda')
input = torch.randn(20, 16, 50)
input=input.to('cuda')
output = m(input)

I am running this code in a NVIDIA docker with CUDA version 10.2 and my GPU is a RTX 2070

Output:
There is some discussion regarding this here. I had the same issue but using cuda 11.1 resolved it for me.
This is the exact pip command
pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: How to shift columns (or rows) in a tensor with different offsets in PyTorch?

Input:
In PyTorch, the build-in torch.roll function is only able to shift columns (or rows) with same offsets. But I want to shift columns with different offsets. Suppose the input tensor is
[[1,2,3],
 [4,5,6],
 [7,8,9]]

Let's say, I want to shift with offset i for the i-th column. Thus, the expected output is
[[1,8,6],
 [4,2,9],
 [7,5,3]]

An option to do so is to separately shift every column using torch.roll and concat each of them. But for the consideration of effectiveness and code compactness, I don't want to introduce the loop structure. Is there a better way？

Output:
I was sceptical about the performance of torch.gather so I searched for similar questions with numpy and found this post.
Similar solution from NumPy to Pytorch
I took the solution from @Andy L and translated it into pytorch. However, take it with a grain of salt, because I don't know how the strides work:
from numpy.lib.stride_tricks import as_strided
# NumPy solution:
def custom_roll(arr, r_tup):
    m = np.asarray(r_tup)
    arr_roll = arr[:, [*range(arr.shape[1]),*range(arr.shape[1]-1)]].copy() #need `copy`
    #print(arr_roll)
    strd_0, strd_1 = arr_roll.strides
    #print(strd_0, strd_1)
    n = arr.shape[1]
    result = as_strided(arr_roll, (*arr.shape, n), (strd_0 ,strd_1, strd_1))

    return result[np.arange(arr.shape[0]), (n-m)%n]

# Translated to PyTorch
def pcustom_roll(arr, r_tup):
    m = torch.tensor(r_tup)
    arr_roll = arr[:, [*range(arr.shape[1]),*range(arr.shape[1]-1)]].clone() #need `copy`
    #print(arr_roll)
    strd_0, strd_1 = arr_roll.stride()
    #print(strd_0, strd_1)
    n = arr.shape[1]
    result = torch.as_strided(arr_roll, (*arr.shape, n), (strd_0 ,strd_1, strd_1))

    return result[torch.arange(arr.shape[0]), (n-m)%n]

Here is also the solution from @Daniel M as plug and play.
def roll_by_gather(mat,dim, shifts: torch.LongTensor):
    # assumes 2D array
    n_rows, n_cols = mat.shape

    if dim==0:
        #print(mat)
        arange1 = torch.arange(n_rows).view((n_rows, 1)).repeat((1, n_cols))
        #print(arange1)
        arange2 = (arange1 - shifts) % n_rows
        #print(arange2)
        return torch.gather(mat, 0, arange2)
    elif dim==1:
        arange1 = torch.arange(n_cols).view(( 1,n_cols)).repeat((n_rows,1))
        #print(arange1)
        arange2 = (arange1 - shifts) % n_cols
        #print(arange2)
        return torch.gather(mat, 1, arange2)

Benchmarking
First, I ran the methods on CPU.
Surprisingly, the gather solution from above is the fastest:
n_cols = 10000
n_rows = 100
shifts = torch.randint(-100,100,size=[n_rows,1])
data = torch.arange(n_rows*n_cols).reshape(n_rows,n_cols)
npdata = np.arange(n_rows*n_cols).reshape(n_rows,n_cols)
npshifts = shifts.numpy()
%timeit roll_by_gather(data,1,shifts)
%timeit pcustom_roll(data,shifts)
%timeit custom_roll(npdata,npshifts)
>> 2.41 ms ± 68.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
>> 90.4 ms ± 882 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
>> 247 ms ± 6.08 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Running the code on GPU shows similar results:
%timeit roll_by_gather(data,shifts)
%timeit pcustom_roll(data,shifts)
131 µs ± 6.79 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
3.29 ms ± 46.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

(Note: You need torch.arange(...,device='cuda:0') within the roll_by_gather method)


Instruction: GPU in docker container for deep learning task

Input:
Recently, I have started working on using docker images. I want to deploy PyTorch based text classification model which requires GPU to run on. When the docker image is called upon, then it's not able to detect GPU in the VM. Hence, my code is failing by throwing no Cuda device found error.
This is my base image FROM gcr.io/deeplearning-platform-release/pytorch-gpu.1-0. I don't know what all steps to follow to install Nvidia drivers in the docker. Please help me out.

Output: Installing Nvidia drivers into the container doesn't help since the GPU device isn't exposed to the container. Running with nvidia-docker instead does.


Instruction: Pytorch showing the error: 'NoneType' object has no attribute 'zero_'

Input:
I am using Python 3.8 and VSCode.
I tried to create a basic Neural Network without activations and biases but because of the error, I'm not able to update the gradients of the weights.
Matrix Details:
Layer Shape: (1, No. of Neurons)
Weight Layer Shape: (No. of Neurons in the previous layer, No. of Neurons in the next layer)
Here's my code:
# sample neural network

# importing libraries
import torch
import numpy as np
torch.manual_seed(0)

# hyperparameters
epochs = 100
lr = 0.01 # learning rate

# data
X_train = torch.tensor([[1]], dtype = torch.float)
y_train = torch.tensor([[2]], dtype = torch.float)

'''
Network Architecture:
    1 neuron in the first layer
    4 neurons in the second layer
    4 neurons in the third layer
    1 neuron in the last layer

    * I haven't added bias and activation.
'''

# initializing the weights
weights = []
weights.append(torch.rand((1, 4), requires_grad = True))
weights.append(torch.rand((4, 4), requires_grad = True))
weights.append(torch.rand((4, 1), requires_grad = True))

# calculating y_pred
y_pred = torch.matmul(torch.matmul(torch.matmul(X_train, weights[0]), weights[1]), weights[2])

# calculating loss
loss = (y_pred - y_train)**2

# calculating the partial derivatives
loss.backward()

# updating the weights and zeroing the gradients
with torch.no_grad():
    for i in range(len(weights)):
        weights[i] = weights[i] - weights[i].grad
        weights[i].grad.zero_()

It it showing the error:
File "test3.py", line 43, in <module>
weights[i].grad.zero_()
AttributeError: 'NoneType' object has no attribute 'zero_'

I don't understand why it is showing this error. Can someone please explain?

Output:
Your model doesn't have any trainable parameters for the grad to be calculated. Use torch's Parameter. See this link for creating a module with learnable parameters.
  torch.nn.parameter.Parameter
        A kind of Tensor that is to be considered a module parameter.


Instruction: Why does my pytorch NN return a tensor of nan?

Input:
I have a quite simple neural network which takes a flattened 6x6 grid as input and should output the values of four actions to take on that grid, so a 1x4 tensor of values.
Sometimes after a few runs though for some reason I am getting a 1x4 tensor of nan
tensor([[nan, nan, nan, nan]], grad_fn=<ReluBackward0>)

My model looks like this with input dim being 36 and output dim being 4:
class Model(nn.Module):
    def __init__(self, input_dim, output_dim):
        # super relates to nn.Module so this initializes nn.Module
        super(Model, self).__init__()
        # Gridsize as input,
        # last layer needs 4 outputs because of 4 possible actions: left, right, up, down
        # output values are Q Values need activation function for those like argmax
        self.lin1 = nn.Linear(input_dim, 24)
        self.lin2 = nn.Linear(24, 24)
        self.lin3 = nn.Linear(24, output_dim)

    # function to feed the input through the net
    def forward(self, x):
        # rectified linear as activation function for the first two layers
        if isinstance(x, np.ndarray):
            x = torch.tensor(x, dtype=torch.float)

        activation1 = F.relu(self.lin1(x))
        activation2 = F.relu(self.lin2(activation1))
        output = F.relu(self.lin3(activation2))

        return output

The input was:
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3333,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6667]])

What are possible causes for a nan output and how can i fix those?

Output: nan values as outputs just mean that the training is instable which can have about every possible cause including all kinds of bugs in the code. If you think your code is correct you can try addressing the instability by lowering the learning rate or use gradient clipping.


Instruction: tuple unpacking in X_test

Input:
i am following a simple tutorial in PyTorch. The tutorial is using the diabetes dataset and built a simple two-layer network. In some part of the tutorial there exist the following part of the code where there exists an iteration on the X_test using enumerate.
predictions=[]
with torch.no_grad():

    for i, data in enumerate(X_test):
        y_pred=model(data)

        predictions.append(y_pred.argmax().item())

        print(y_pred.argmax().item())

while I thought that I was understanding enumerate I am a little confused. What the i and data referring to in the above code?
in general, I understandnd that is a typle unpacking procedure

Output:
enumerate simply enumerates iterable items.
e.g.
my_list = ['cat', 'dog', 'elephant']
print(list(enumerate(my_list)))
>> [(0, 'cat'), (1, 'dog'), (2, 'elephant')]

In your case i is an index of data variable in some loop state (it is not used tho) and data is a feature tensor of one row in your dataset.


Instruction: GPU underutilized in Actor Critic (A2C) Stable Baselines3 implementation

Input:
I am trying to use A2C of StablesBaselines3 for training an agent on my custom environment. My problem is that my GPU Utilization is very less (around 10 % only) while my CPU utilization has hit the ceiling. Because of this the training is very very slow. I have tried the following things as per this discussion thread addressed by @araffin
[https://github.com/hill-a/stable-baselines/issues/308]

Expanding the Actor and Critic architecture to a three layer neural
network having 256, 256 and 128 neurons respectively. The GPU
utilization did increase after that but it was only marginal
(increased from 10 % to 15 %) as in this suggestion
changed device argument of A2C method to 'cuda' from the default which is 'auto' -
No improvement

It is not entirely clear what worked for the user in the above mentioned discussion.
My A2C is intialized as follows:
import gym
from stable_baselines3 import A2C
policy_kwargs = dict(activation_fn=th.nn.ReLU,
                     net_arch=[dict(pi=[256, 256, 128], vf=[256, 256, 128])])
model = A2C("MlpPolicy", env, verbose=1,  learning_rate=linear_schedule(0.001),\
 n_steps=50, gamma=0.8, gae_lambda=1.0, ent_coef=0.2, vf_coef=0.2, \
tensorboard_log=logdir,  policy_kwargs=policy_kwargs, seed=50, device='cuda' )
model.learn(total_timesteps=50000)

GPU configurations are:

Cuda version - 9.0
GPU - NVIDIA RTX 2080i
GPU mem - 11GB (out of that only about 1600MB is being used)

CPU configurations

Threads per Core - 2
Core per socket - 8
Intel i9 core 9900 K @ 3.6 Ghz
CPU Mem - 32GB
SW versions
Torch - 1.6.0
Tensorflow GPU - 2.4.1 (although I know SB3 implementation uses Torch)
Python 3.7 on anaconda environment

Any help will be appreciated!

Output:
Stable baselines is using your gpu ... if you look task manager
on second tab click on your gpu and instead of 3D
select cuda and you will see the usage of cuda
I had some troubles with my env because my env uses pandas .. and pandas use cpu .. on windows is not possible easly to use cudf so my cpu was using 100%
I used pandas and numpy on my custom env ..
to improve it I should use cunumery and cudf instead, but to do it I need linux.
so for now, my cpu is using 100% gpu is using cuda 70% and gpu memory almost 1.7 gb


Instruction: how to solve this (Pytorch RuntimeError: 1D target tensor expected, multi-target not supported)

Input:
i am newbie in pytorch and deep learning
my data set 53502 x 58,
i have problem this my code
model = nn.Sequential(
    nn.Linear(58,64),
    nn.ReLU(),
    nn.Linear(64,32),
    nn.ReLU(),
    nn.Linear(32,16),
    nn.ReLU(),
    nn.Linear(16,2),
    nn.LogSoftmax(1)
)

criterion = nn.NLLLoss()
optimizer = optim.AdamW(model.parameters(), lr = 0.0001)

epoch = 500
train_cost, test_cost = [], []
for i in range(epoch):
    model.train()
    cost = 0
    for feature, target in trainloader:
        output = model(feature)          #feedforward
        loss = criterion(output, target) #loss
        loss.backward()                  #backprop

        optimizer.step()                 #update weight
        optimizer.zero_grad()            #zero grad

        cost += loss.item() * feature.shape[0]
    train_cost.append(cost / len(train_set))

    with torch.no_grad():
        model.eval()
        cost = 0
        for feature, target in testloader:
            output = model(feature)          #feedforward
            loss = criterion(output, target) #loss

            cost += loss.item() * feature.shape
        test_cost.append(cost / len(test_set))

    print(f'\repoch {i+1}/{epoch} | train_cost: {train_cost[-1]} | test_cost : {test_cost[-1]}', end = "")

and then i get problem like this
   2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
-> 2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2265     elif dim == 4:
   2266         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

whats wrong?
how to solve this problem?
why this happend?
Thank you very much in advance!

Output:
When using NLLLoss the target tensor must contain the index representation of the labels and not one-hot. So for example:
I guess this is what your target looks like:
target = [0, 0, 1, 0]

Just convert it to just the number which is the index of the 1:
[0, 0, 1, 0] -> [2]
[1, 0, 0, 0] -> [0]
[0, 0, 0, 1] -> [3]

And then convert it to long tensor, ie:
target = [2]
target = torch.Tensor(target).type(torch.LongTensor)

It might be confusing, that your output is a tensor with the length of classes and your target is an number but that how it is.
You can check it out yourself here.


Instruction: Difference between Xavier weights pytorch implementation

Input: What is the difference between nn.init.xavier_uniform and nn.init.xavier_uniform_ when initialising weights?

Output: The _ convention in nn.init.xavier_uniform_ is PyTorch's way of doing an operation in place. This convention applies to many of its functions.


Instruction: Pass an arbitrary image size to cnn in pytorch

Input:
I'm trying to train a lenet model in pytorch, The ideia is to put images of any size in it, so I started doing with nn.AdaptiveAvgPool2d but the error comes as

mat1 dim 1 must match mat2 dim 0

Here is my code
class LeNet5(nn.Module):
  def __init__(self, num_classes=10):
    super(LeNet5, self).__init__()

    self.conv_1 = nn.Conv2d(
        in_channels=1, out_channels=32, kernel_size=5, bias=False
    )
    self.relu_1 = nn.ReLU(inplace=True)
    self.maxpool_1 = nn.MaxPool2d(kernel_size=2, stride=2)
    self.conv_2 = nn.Conv2d(
        in_channels=32, out_channels=256, kernel_size=5, bias=False
    )
    self.relu_2 = nn.ReLU(inplace=True)
    self.maxpool_2 = nn.MaxPool2d(kernel_size=2, stride=2)
    self.avgpool = nn.AdaptiveAvgPool2d(output_size=1)
    self.flatten = nn.Flatten()
    self.fc_1 = nn.Linear(in_features=4096, out_features=120, bias=False)
    self.fc_2 = nn.Linear(in_features=120, out_features=84)
    self.fc_3 = nn.Linear(in_features=84, out_features=num_classes)

  def forward(self, input):
    conv_1_output = self.conv_1(input)
    relu_1_output = self.relu_1(conv_1_output)
    maxpool_1_output = self.maxpool_1(relu_1_output)
    conv_2_output = self.conv_2(maxpool_1_output)
    relu_2_output = self.relu_2(conv_2_output)
    maxpool_2_output = self.maxpool_2(relu_2_output)
    flatten_output = self.flatten((self.avgpool(maxpool_2_output).view(maxpool_2_output.shape[0], -1)))
    fc_1_output = self.fc_1(flatten_output)
    fc_2_output = self.fc_2(fc_1_output)
    fc_3_output = self.fc_3(fc_2_output)

    return fc_3_output

Output:
if you read the theory on AdaptiveAvgPool2d, this is what it says " we specify the output size And the stride and kernel-size are automatically selected to adapt to the needs"
More info available here
Hence Your spatial dimension is reduced by AdaptiveAvgPool2d and not the depth of feature maps.
So, the spatial dimension will be 1x1 and depth will still be 256 , making your
self.fc_1 = nn.Linear(in_features=256, out_features=120, bias=False) and not self.fc_1 = nn.Linear(in_features=4096, out_features=120, bias=False)


Instruction: Performance in reading dicom files with SimpleITK and PyTorch

Input:
I want to directly load image from memory to python in pytorch tensor format.
I modified GetArrayViewFromImage() function by replacing those lines:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = numpy.asarray(image_memory_view).view(dtype = dtype)

by:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = torch.as_tensor(image_memory_view, dtype = dtype)

in practise it is so slow I replaced it with:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = numpy.asarray(image_memory_view).view(dtype = dtype)
array_view  = torch.as_tensor(array_view)

Now I have two questions:

it is much slower, and I don't really know why reading it with numpy and converting it is faster.
even though I add the dtype argument and it returns a tensor with a correct dtype it reads it wrong (ex. -1000 in numpy is read as 252 no matter what torch.dtype I choose) which is not a problem when reading with numpy and converting, why is that happening?

Output: While this does not directly answer your question, I strongly recommend using the torchio package, instead of dealing with these IO issues yourself (torchio uses SimpleITK under the hood).


Instruction: How to vectorize loss for a LSTM doing sequential Language modelling

Input:
So I have an assignment involving Language Modelling and I passed all the unit tests but my code is too slow to run. I think it's because of the way I compute my loss. The formula we're given is the following:

My naive implementation is the following:
losses_batch_list = []
batch_size = log_probas.size(0)
for b in range(batch_size):

    seq_length = max([i for i, e in enumerate(mask[b,:]) if e != 0]) + 1

    loss_batch = 0
    for t in range(seq_length):
        for n in range(self.vocabulary_size):
            if targets[b, t] == n:
                loss_batch += log_probas[b, t, n].detach()

    loss_batch = - loss_batch / seq_length
    losses_batch_list.append(loss_batch)

loss = torch.tensor(np.mean(losses_batch_list))

return loss

But that loop runs for ever since the vocabulary size is the same approximately as GPT1 (~40 000) and the sequence length is up to 255 (something it is shorter because of padding, hence the mask).
Does anyone have any tips on how to vectorize/speed this up?
I know it's correct but I can't report any results with it...
Thanks!

Output:
B = batch_size
T = sequence_length (padded)
N = vocab_size

if type(mask_b) == torch.bool:
    mask = mask.view(-1) # (B, T) -> (B*T,)
else:
    mask = mask.bool().view(-1) # (B, T) -> (B*T,)
log_probas = log_probas.view(-1, N) # (B, T, N) -> (B*T, N)
targets = target.view(-1, 1) # (B, T) -> (B*T, 1)
loss = torch.gather(log_probas[mask], -1, target[mask]) # loss without padded tokens
loss = loss.mean()


Instruction: Pytorch MNIST autoencoder to learn 10-digit classification

Input:
I'm trying to build a simple autoencoder for MNIST, where the middle layer is just 10 neurons. My hope is that it will learn to classify the 10 digits, and I assume that would lead to the lowest error in the end (wrt reproducing the original image).
I have the following code, which I've already played around with a fair amount. If I run it for up-to 100 epochs, the loss doesn't really go below 1.0, and if I evaluate it, it's obviously not working. What am I missing?
Training:
import torch
import torchvision as tv
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torchvision.utils import save_image

num_epochs = 100
batch_size = 64

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
trainset = tv.datasets.MNIST(root='./data',  train=True, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=4)

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder,self).__init__()
        self.encoder = nn.Sequential(
            # 28 x 28
            nn.Conv2d(1, 4, kernel_size=5),
            nn.Dropout2d(p=0.2),
            # 4 x 24 x 24
            nn.ReLU(True),
            nn.Conv2d(4, 8, kernel_size=5),
            nn.Dropout2d(p=0.2),
            # 8 x 20 x 20 = 3200
            nn.ReLU(True),
            nn.Flatten(),
            nn.Linear(3200, 10),
            nn.ReLU(True),
            # 10
            nn.Softmax(),
            # 10
            )
        self.decoder = nn.Sequential(
            # 10
            nn.Linear(10, 400),
            nn.ReLU(True),
            # 400
            nn.Unflatten(1, (1, 20, 20)),
            # 20 x 20
            nn.Dropout2d(p=0.2),
            nn.ConvTranspose2d(1, 10, kernel_size=5),
            # 24 x 24
            nn.ReLU(True),
            nn.Dropout2d(p=0.2),
            nn.ConvTranspose2d(10, 1, kernel_size=5),
            # 28 x 28
            nn.ReLU(True),
            nn.Sigmoid(),
            )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = Autoencoder().cpu()
distance = nn.MSELoss()
#optimizer = torch.optim.Adam(model.parameters(), weight_decay=1e-5)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = Variable(img).cpu()
        output = model(img)
        loss = distance(output, img)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print('epoch [{}/{}], loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

Already the training loss indicates that the thing is not working, but printing out the confusion matrix (which in this case should not necessarily be the identity matrix, since the neurons can be ordered arbitrarily, but should be row-col-reordarable and approximate the identity, if this would work):
import numpy as np

confusion_matrix = np.zeros((10, 10))

batch_size = 20*1000

testset = tv.datasets.MNIST(root='./data',  train=False, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=True, num_workers=4)

for data in dataloader:
    imgs, labels = data
    imgs = Variable(imgs).cpu()
    encs = model.encoder(imgs).detach().numpy()
    for i in range(len(encs)):
        predicted = np.argmax(encs[i])
        actual = labels[i]
        confusion_matrix[actual][predicted] += 1
print(confusion_matrix)

Output:
Autoencoder is technically not used as a classifier in general. They learn how to encode a given image into a short vector and reconstruct the same image from the encoded vector. It is a way of compressing image into a short vector:

Since you want to train autoencoder with classification capabilities, we need to make some changes to model. First of all, there will be two different losses:

MSE loss: Current autoencoder reconstruction loss. This will force network to output an image as close as possible to given image by taking the compressed representation.
Classification loss: Classic cross entropy should do the trick. This loss will take compressed representation (C dimensional) and target labels to calculate negative log likelihood loss. This loss will force encoder to output compressed representation such that it aligns well with the target class.

I've done a couple of changes to your code to get the combined model working. Firstly, let's see the code:
 import torch
 import torchvision as tv
 import torchvision.transforms as transforms
 import torch.nn as nn
 import torch.nn.functional as F
 from torch.autograd import Variable
 from torchvision.utils import save_image

 num_epochs = 10
 batch_size = 64
 transform = transforms.Compose([
     transforms.ToTensor(),
     transforms.Normalize((0.1307,), (0.3081,))
 ])

 trainset = tv.datasets.MNIST(root='./data',  train=True, download=True, transform=transform)
 testset  = tv.datasets.MNIST(root='./data',  train=False, download=True, transform=transform)
 dataloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=4)
 testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=True, num_workers=4)
 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

 class Autoencoderv3(nn.Module):
     def __init__(self):
         super(Autoencoderv3,self).__init__()
         self.encoder = nn.Sequential(
             nn.Conv2d(1, 4, kernel_size=5),
             nn.Dropout2d(p=0.1),
             nn.ReLU(True),
             nn.Conv2d(4, 8, kernel_size=5),
             nn.Dropout2d(p=0.1),
             nn.ReLU(True),
             nn.Flatten(),
             nn.Linear(3200, 10)
             )
         self.softmax = nn.Softmax(dim=1)
         self.decoder = nn.Sequential(
             nn.Linear(10, 400),
             nn.ReLU(True),
             nn.Unflatten(1, (1, 20, 20)),
             nn.Dropout2d(p=0.1),
             nn.ConvTranspose2d(1, 10, kernel_size=5),
             nn.ReLU(True),
             nn.Dropout2d(p=0.1),
             nn.ConvTranspose2d(10, 1, kernel_size=5)
             )

     def forward(self, x):
         out_en = self.encoder(x)
         out = self.softmax(out_en)
         out = self.decoder(out)
         return out, out_en

 model = Autoencoderv3().to(device)
 distance   = nn.MSELoss()
 class_loss = nn.CrossEntropyLoss()

 optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

 mse_multp = 0.5
 cls_multp = 0.5

 model.train()

 for epoch in range(num_epochs):
     total_mseloss = 0.0
     total_clsloss = 0.0
     for ind, data in enumerate(dataloader):
         img, labels = data[0].to(device), data[1].to(device)
         output, output_en = model(img)
         loss_mse = distance(output, img)
         loss_cls = class_loss(output_en, labels)
         loss = (mse_multp * loss_mse) + (cls_multp * loss_cls)  # Combine two losses together
         optimizer.zero_grad()
         loss.backward()
         optimizer.step()
         # Track this epoch's loss
         total_mseloss += loss_mse.item()
         total_clsloss += loss_cls.item()

     # Check accuracy on test set after each epoch:
     model.eval()   # Turn off dropout in evaluation mode
     acc = 0.0
     total_samples = 0
     for data in testloader:
         # We only care about the 10 dimensional encoder output for classification
         img, labels = data[0].to(device), data[1].to(device)
         _, output_en = model(img)
         # output_en contains 10 values for each input, apply softmax to calculate class probabilities
         prob = nn.functional.softmax(output_en, dim = 1)
         pred = torch.max(prob, dim=1)[1].detach().cpu().numpy() # Max prob assigned to class
         acc += (pred == labels.cpu().numpy()).sum()
         total_samples += labels.shape[0]
     model.train()   # Enables dropout back again
     print('epoch [{}/{}], loss_mse: {:.4f}  loss_cls: {:.4f}  Acc on test: {:.4f}'.format(epoch+1, num_epochs, total_mseloss / len(dataloader), total_clsloss / len(dataloader), acc / total_samples))

This code should now train the model both as a classifier and a generative autoencoder. In general though, this type of approach can be a bit tricky to get the model training. In this case, MNIST data is simple enough to get those two complementary losses train together. In more complex cases like Generative Adversarial Networks (GAN), they apply model training switching, freezing one model etc. to get whole model trained. This autoencoder model trains easily on MNIST without doing those types of tricks:
 epoch [1/10], loss_mse: 0.8928  loss_cls: 0.4627  Acc on test: 0.9463
 epoch [2/10], loss_mse: 0.8287  loss_cls: 0.2105  Acc on test: 0.9639
 epoch [3/10], loss_mse: 0.7803  loss_cls: 0.1574  Acc on test: 0.9737
 epoch [4/10], loss_mse: 0.7513  loss_cls: 0.1290  Acc on test: 0.9764
 epoch [5/10], loss_mse: 0.7298  loss_cls: 0.1117  Acc on test: 0.9762
 epoch [6/10], loss_mse: 0.7110  loss_cls: 0.1017  Acc on test: 0.9801
 epoch [7/10], loss_mse: 0.6962  loss_cls: 0.0920  Acc on test: 0.9794
 epoch [8/10], loss_mse: 0.6824  loss_cls: 0.0859  Acc on test: 0.9806
 epoch [9/10], loss_mse: 0.6733  loss_cls: 0.0797  Acc on test: 0.9814
 epoch [10/10], loss_mse: 0.6671  loss_cls: 0.0764  Acc on test: 0.9813

As you can see, both mse loss and classification loss is decreasing, and accuracy on test set is increasing. In the code, MSE loss and classification loss are added together. This means respective gradients calculated from each loss are fighting against each other to force the network into their direction. I've added loss multiplier to control the contribution from each loss. If MSE has a higher multiplier, network will have more gradients from MSE loss, meaning it will better learn reconstruction, if CLS loss has a higher multiplier, network will get better classification accuracies. You can play with those multiplier to see how end result is changing, but MNIST is a very easy dataset so differences might be hard to see maybe. Currently, it doesn't do too bad at reconstructing inputs:
 import numpy as np
 import matplotlib.pyplot as plt

 model.eval()
 img, labels = list(dataloader)[0]
 img = img.to(device)
 output, output_en = model(img)
 inp = img[0:10, 0, :, :].squeeze().detach().cpu()
 out = output[0:10, 0, :, :].squeeze().detach().cpu()

 # Just some trick to concatenate first ten images next to each other
 inp = inp.permute(1,0,2).reshape(28, -1).numpy()
 out = out.permute(1,0,2).reshape(28, -1).numpy()
 combined = np.vstack([inp, out])

 plt.imshow(combined)
 plt.show()

I am sure with more training and fine tuning loss multipliers, you can get better results.
Lastly, decoder receives softmax of encoder output. This mean decoder tries to create output image from 0 - 1 probabilities of the input. So if the softmax probability vector is 0.98 at input location 0 and close to zero elsewhere, decoder should output an image that looks like 0.0. Here I give network input to create 0 to 9 reconstructions:
 test_arr = np.zeros([10, 10], dtype = np.float32)
 ind = np.arange(0, 10)
 test_arr[ind, ind] = 1.0

 model.eval()
 img = torch.from_numpy(test_arr).to(device)
 out = model.decoder(img)
 out = out[0:10, 0, :, :].squeeze().detach().cpu()
 out = out.permute(1,0,2).reshape(28, -1).numpy()
 plt.imshow(out)
 plt.show()

I've also done a few small changes in the code, printing epoch average loss etc. which doesn't really change the training logic, so you can see those changes in the code and let me know if anything looks weird.


Instruction: pytorch collections.OrderedDict' object has no attribute 'to'

Input:
this is my main code,but I don't know how to fix the problem?
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = torch.load('./checkpoints/fcn_model_5.pth')  # 加载模型
model = model.to(device)

Output:
You are loading the checkpoint as a state dict, it is not a nn.module object.
checkpoint = './checkpoints/fcn_model_5.pth'

model = your_model() # a torch.nn.Module object
model.load_state_dict(torch.load(checkpoint ))
model = model.to(device)


Instruction: How to calculate the mean and the std of cifar10 data

Input:
Pytorch is using the following values as the mean and std for the cifar10 data:
transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
I need to understand the concept behind calculating it because this data is 3 channel image and I do not understand what is summed and divided over what and so on.
Also if someone can share a code for calculating the mean and the std, would be so thankful.

Output:
The 0.5 values are just approximates for cifar10 mean and std values over the three channels (r,g,b). The precise values for cifar10 train set are

mean: 0.49139968, 0.48215827 ,0.44653124
std: 0.24703233 0.24348505 0.26158768

You may calculate these using the following script:
import torch
import numpy
import torchvision.datasets as datasets
from torchvision import transforms

cifar_trainset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transforms.ToTensor())

imgs = [item[0] for item in cifar_trainset] # item[0] and item[1] are image and its label
imgs = torch.stack(imgs, dim=0).numpy()

# calculate mean over each channel (r,g,b)
mean_r = imgs[:,0,:,:].mean()
mean_g = imgs[:,1,:,:].mean()
mean_b = imgs[:,2,:,:].mean()
print(mean_r,mean_g,mean_b)

# calculate std over each channel (r,g,b)
std_r = imgs[:,0,:,:].std()
std_g = imgs[:,1,:,:].std()
std_b = imgs[:,2,:,:].std()
print(std_r,std_g,std_b)

Also, you may find the same mean and std values here and here


Instruction: How to incoporate mask into negative likelihood loss (torch.nn.functional.nll_loss)

Input:
Hello I am implementing a lstm for language modelling
for homework and I am at the loss implementation phase. Our instructor told us to use F.nll_loss but the sequences are padded and we have to take into account a mask that is given which tells us when the sequences stop.
input:

log_probas (batch_size, sequence_length(padded), vocabulary size)
targets (batch_size, sequence_length(padded))
mask (batch_size, sequence_length(padded)

naive implementation which works without taking into account the mask:
import torch.nn.functional as F
loss = F.nll_loss(log_probas.transpose(1, 2), targets)

I've been crawling the internet and banging my head but can't seem to find an answer on how to incorporate a mask into the averaging scheme of the loss.

Output:
you could reshape the tensors and use mask to select non-padded tokens, and compute the loss
vocab_size = log_probas.size(-1)
log_probas = log_probas.view(-1, vocab_size)
target = target.view(-1)
mask = mask.view(-1).bool()
loss = F.nll_loss(log_probas[mask], targets[mask])


Instruction: PyTorch can't find the name?? (NameError: name 'device' is not defined)

Input:
sorry - I'm a complete beginner!
I am trying to build a 'mini-system' using the Torchreid libraries from https://kaiyangzhou.github.io/deep-person-reid/index.html#
In their version they use CUDA but my Mac is not compatible with CUDA and it doesn't have a
CUDA enabled GPU so I installed the CPU-only version of PyTorch instead - therefore I changed model = model.cuda() to model = model.to(device) and added in device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') as you can see below. I thought this would work but I keep getting the NameError: name 'device' is not defined and I don't know what to do.
Please help!
(I also tried putting device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') at the top instead of the bottom to see if it made any difference but I just got another error - NameError: name 'torch' is not defined)
model = torchreid.models.build_model(
    name='resnet50',
    num_classes=datamanager.num_train_pids,
    loss='softmax',
    pretrained=True
)

model = model.to(device)

optimizer = torchreid.optim.build_optimizer(
    model,
    optim='adam',
    lr=0.0003
)

scheduler = torchreid.optim.build_lr_scheduler(
    optimizer,
    lr_scheduler='single_step',
    stepsize=20
)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Output:
Define device variable before the usage:
import torch
...
model = torchreid.models.build_model(
    name='resnet50',
    num_classes=datamanager.num_train_pids,
    loss='softmax',
    pretrained=True
)

# Just right before the actual usage
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
...


Instruction: Why is pytorch softmax function not working?

Input:
so this is my code
import torch.nn.functional as F
import torch

inputs = [1,2,3]
input = torch.tensor(inputs)
output = F.softmax(input, dim=1)
print(output)

is the reason why the code not working because of the dim?
the error here:
  File "c:\Users\user\Desktop\AI\pytorch_jovian\linear_reg.py", line 19, in <module>
    output = F.softmax(input, dim=1)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\functional.py", line 1583, in softmax
    ret = input.softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Output:
Apart from dim=0, there is another issue in your code. Softmax doesn't work on a long tensor, so it should be converted to a float or double tensor first
>>> input = torch.tensor([1, 2, 3])
>>> input
tensor([1, 2, 3])

>>> F.softmax(input.float(), dim=0)
tensor([0.0900, 0.2447, 0.6652])


Instruction: Mapping tensor in pytorch

Input:
I have the following two tensors:

img is a RGB image of shape (224,224,3)
uvs is a tensor with same spacial size e.g. (224, 224, 2) that maps to coordinates (x,y). In other words it provides (x,y) coordinates for every pixel of the input image.

I want to create now a new output image tensor that contains on index (x,y) the value of the input image. So the output should be an image as well with the pixels rearranged according to the mapping tensor.
Small toy example:
img = [[c1,c2], [c3, c4]] where c is a RGB color [r, g, b]
uvs = [[[0,0], [1,1]],[[0,1], [1,0]]]
out = [[c1, c3], [c4, c2]]

How would one achieve such a thing in pytorch in a fast vectorized manner?

Output:
Try with:
out = img[idx[...,0], idx[...,1]]


Instruction: Define manually sorted MNIST dataset with batch size = 1 in PyTorch

Input:
[] : this indicates a batch. For example, if the batch size is 5, then the batch will look something like this [1,4,7,4,2]. The length of [] indicates the batch size.
What I want to make a training set something looks like this:
[1] -> [1] -> [1] -> [1] -> [1] -> [7] -> [7] -> [7] -> [7] -> [7] -> [3] -> [3] -> [3] -> [3] -> [3] -> ... and so on
Which means that firstly five 1s (batch size = 1), secondly five 7s (batch size = 1), thirdly five 3s (batch size = 1) and so on...
Can someone please provide me an idea?
It will be very helpful if someone can explain how to implement this with codes.
Thank you! :)

Output:
If you want a DataLoader where you just want to define the class label for each sample then you can make use of the torch.data.utils.Subset class. Despite its name it doesn't necessarily need to define a subset of dataset. For example
import torch
import torchvision
import torchvision.transforms as T
from itertools import cycle

mnist = torchvision.datasets.MNIST(root='./', train=True, transform=T.ToTensor())

# not sure what "...and so on" implies, but define this list however you like
target_classes = [1, 1, 1, 1, 1, 7, 7, 7, 7, 7, 3, 3, 3, 3, 3]

# create cyclic iterators of indices for each class in MNIST
indices = dict()
for label in torch.unique(mnist.targets).tolist():
    indices[label] = cycle(torch.nonzero(mnist.targets == label).flatten().tolist())

# define the order of indices in the new mnist subset based on target_classes
new_indices = []
for t in target_classes:
    new_indices.append(next(indices[t]))

# create a Subset of MNIST based on new_indices
mnist_modified = torch.utils.data.Subset(mnist, new_indices)
dataloader = torch.utils.data.DataLoader(mnist_modified, batch_size=1, shuffle=False)

for idx, (x, y) in enumerate(dataloader):
    # training loop
    print(f'Batch {idx+1} labels: {y.tolist()}')


Instruction: How to find partial derivative in pytorch

Input:
I have a model u(x,t) with layers 2X50, then 50X50, and 50X1.
I train the model with input x,t of size [100,2]. In the final layer I get u and now I want to differentiate it w.r.t to x and t and double differentiate w.r.t to x.  How do I do this in PyTorch?

Output:
You can use PyTorch's autograd engine like so:
import torch

x = torch.randn(100, requires_grad=True)
t = torch.randn(2, requires_grad=True)
u = u(x,t)

# 1st derivatives
dt = torch.autograd.grad(u, t)[0]
dx = torch.autograd.grad(u, x, create_graph=True)[0]

# 2nd derivatives (higher orders require `create_graph=True`)
ddx = torch.autograd.grad(dx, x)[0]


Instruction: How can i use pytorch model in C#?

Input:
I have a pytorch model in NLP and a script for use it in python. Now i want to use this script in C#. I tried run python script from C# and it worked. I get user sentence in C#, pass it to python and its outputs use in C#. The problem is that i want to do this work in a loop until user select exit but every time it goes to python code, it must load pytorch model and its time consuming.
Is there a way to load model one time and then in a loop get input from user and inference with loaded model?

Output: you can export the model in ONNX format and then use the OpenCV DNN module or tensorrt for inference purposes. It will give you a significant boost in speed and your whole code will be in C#.


Instruction: PyTorch How to code Multi Head Self Attention in parallel?

Input:
I want to encode the word (embedding) sequence with 16-Head Self-Attention.
Currently I use a nn.ModuleList together with a for loop to generate the output of each head then concatenate all of them. This approach is extremely slow and I wonder if there's way to code MHA in parallel?
To generalize, I would like to know if I can 'stack' multiple nn.Linear, as I feed the input vector, the multiple outputs will be computed in parallel.

Output:
I figured it out. Since nn.Linear is acctually an affine transformation with a weights matrix and a bias matrix, one can easily wrap such matrices in nn.Parameter and take advantage of broadcast semantics to achieve the goal.

Edit: I also find a nn.Linear(d_model, n_heads*d_key) functions identically.


Instruction: PyTorch using LR-Scheduler with param groups of different LR's

Input:
I have defined the following optimizer with different learning rates for each parameter group:
  optimizer = optim.SGD([
          {'params': param_groups[0], 'lr': CFG.lr, 'weight_decay': CFG.weight_decay},
          {'params': param_groups[1], 'lr': 2*CFG.lr, 'weight_decay': 0},
          {'params': param_groups[2], 'lr': 10*CFG.lr, 'weight_decay': CFG.weight_decay},
          {'params': param_groups[3], 'lr': 20*CFG.lr, 'weight_decay': 0},
      ], lr=CFG.lr, momentum=0.9, weight_decay=CFG.weight_decay, nesterov=CFG.nesterov)

Now I want to use a LR-Scheduler to update all the learning rates and not only the first one, because by deafult, a scheduler would only update the param_groups[0]?
scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=5, T_mult=2, eta_min=CFG.min_lr, last_epoch=-1, verbose=True)

Giving me:
Parameter Group 0
    dampening: 0
    initial_lr: 0.001
    lr: 0.0009999603905218616
    momentum: 0.9
    nesterov: True
    weight_decay: 0.0001

Parameter Group 1
    dampening: 0
    initial_lr: 0.002
    lr: 0.002
    momentum: 0.9
    nesterov: True
    weight_decay: 0

Parameter Group 2
    dampening: 0
    initial_lr: 0.01
    lr: 0.01
    momentum: 0.9
    nesterov: True
    weight_decay: 0.0001

Parameter Group 3
    dampening: 0
    initial_lr: 0.02
    lr: 0.02
    momentum: 0.9
    nesterov: True
    weight_decay: 0
)

after one update.
Any idea how to update all the learning rates with a scheduler?

Output:
You are right, learning rate scheduler should update each group's learning rate one by one. After a bit of testing, it looks like, this problem only occurs with CosineAnnealingWarmRestarts scheduler. I've tested CosineAnnealingLR and couple of other schedulers, they updated each group's learning rate:
 scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, 100, verbose=True)

Then, to find the cause of the problem I had a look at the source code of learning rate scheduler: https://github.com/pytorch/pytorch/blob/master/torch/optim/lr_scheduler.py
From a quick look through, it looks like there are some differences between the implementation of CosineAnnealingLR and CosineAnnealingWarmRestarts get_lr() functions:
 # CosineAnnealingLR:
 def get_lr(self):
     if not self._get_lr_called_within_step:
         warnings.warn("To get the last learning rate computed by the scheduler, "
                       "please use `get_last_lr()`.", UserWarning)

     if self.last_epoch == 0:
         return self.base_lrs
     elif (self.last_epoch - 1 - self.T_max) % (2 * self.T_max) == 0:
         return [group['lr'] + (base_lr - self.eta_min) *
                 (1 - math.cos(math.pi / self.T_max)) / 2
                 for base_lr, group in
                 zip(self.base_lrs, self.optimizer.param_groups)]
     return [(1 + math.cos(math.pi * self.last_epoch / self.T_max)) /
             (1 + math.cos(math.pi * (self.last_epoch - 1) / self.T_max)) *
             (group['lr'] - self.eta_min) + self.eta_min
             for group in self.optimizer.param_groups]

 # CosineAnnealingWarmRestarts:
 def get_lr(self):
     if not self._get_lr_called_within_step:
         warnings.warn("To get the last learning rate computed by the scheduler, "
                       "please use `get_last_lr()`.", UserWarning)

     return [self.eta_min + (base_lr - self.eta_min) * (1 + math.cos(math.pi * self.T_cur / self.T_i)) / 2
             for base_lr in self.base_lrs]

So after looking through the code, I feel like this issue is a bug. Even the documentation of CosineAnnealingWarmRestart suggests that "Set the learning rate of each parameter group using a cosine annealing schedule".


Instruction: How to join two tensors

Input:
I have two tensors of dimension [3,1]. I need to join them as a [3,2] tensor.
t = torch.tensor([[3.],[1],[2]], requires_grad=True)
x = torch.tensor([[1.],[4],[5]], requires_grad=True)

I tried torch.cat and torch.stack but neither work for me.

Output:
With cat you need to specify the dimension the tensors are concatenated along. By default this is 0, but you wish to use 1:
import torch
res = torch.cat([t,x], axis=1)


Instruction: RuntimeError: 1D target tensor expected, multi-target not supported Python: NumPy

Input:
I am dealing with a CNN and I get the following error on the line loss = criterion(outputs, data_y):
Here is the relevant code snippet:
def run(model, X_train, Y_train, X_test, Y_test, learning_rate=0.01,
          num_epochs=100, minibatch_size=8, print_cost=True):

    seed = 0                                         # to keep results consistent (numpy seed)
    (m, n_H0, n_W0, n_C0) = X_train.shape
    costs = []                                       # To keep track of the cost

    criterion = nn.NLLLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

    # Training loop
    model.train() # Turn on the training mode
    for epoch in range(num_epochs):

        minibatch_cost = 0.
        num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set
        seed = seed + 1
        minibatches = utils.generate_minibatch(X_train, Y_train, minibatch_size, seed)

        for minibatch in minibatches:
            (batch_x, batch_y) = minibatch
            data_x = torch.Tensor(batch_x)
            data_y = torch.LongTensor(batch_x)

            ### START YOUR CODE ###
            # Zero the gradients
            optimizer.zero_grad() # Hint: call zero_grad()

            # Forward pass and compute loss
            outputs = model(data_x) # Hint: use model as a callable
            loss = criterion(outputs, data_y)  # Hint: use criterion as a callable

            # Backward and optimize
            loss.backward() # Hint: call backward()
            optimizer.step()  # Hint: call step()
            ### END YOUR CODE ###

            minibatch_cost += loss.item()

        # Print the cost every epoch
        minibatch_cost /= num_minibatches
        if print_cost and epoch % 5 == 0:
            print ("Cost after epoch %i: %f" % (epoch, minibatch_cost))
        costs.append(minibatch_cost)

    # Calculate accuracy on the train and test datasets
    data_x = torch.Tensor(X_test)
    data_y = torch.LongTensor(Y_test)
    model.eval() # Turn on the evaluation mode
    with torch.no_grad():
        test_pred = model(data_x)
        num_correct = (torch.argmax(test_pred, dim=1).view(data_y.size()).data == data_y.data).float().sum()
        test_acc = num_correct / test_pred.size()[0]
    print("Test Accuracy:", test_acc.item())

model = CNN_Model()

torch.manual_seed(0)
run(model, X_train, Y_train, X_test, Y_test)

And here is the error I'm getting:
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-9839fc42e5c2> in <module>
      3
      4 torch.manual_seed(0)
----> 5 run(model, X_train, Y_train, X_test, Y_test)
      6
      7 # NOTE: It could be slow to run 100 epochs. Make sure that your costs for after each epoch

<ipython-input-5-05ddcdc9ddf5> in run(model, X_train, Y_train, X_test, Y_test, learning_rate, num_epochs, minibatch_size, print_cost)
     40             # Forward pass and compute loss
     41             outputs = model(data_x) # Hint: use model as a callable
---> 42             loss = criterion(outputs, data_y)  # Hint: use criterion as a callable
     43
     44             # Backward and optimize

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    214     def forward(self, input: Tensor, target: Tensor) -> Tensor:
    215         assert self.weight is None or isinstance(self.weight, Tensor)
--> 216         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
    217
    218

C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2383         )
   2384     if dim == 2:
-> 2385         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2386     elif dim == 4:
   2387         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

Output:
This error usually appears when you pass a one-hot-encoded target to CrossEntropy or NLLLoss (instead of a single class index), but your problem is simpler - you just have a typo here:
data_y = torch.LongTensor(batch_x) # <- should be `batch_y`


Instruction: Average Pooling layer in Deep Learning and gradient artifacts

Input:
I know that in Convolution layers the kernel size needs to be a multiplication of stride or else it will produce artefacts in gradient calculations like the checkerboard problem.
Now does it also work like that in Pooling layers? I read somewhere that max pooling can also cause problems like that. Take this line in the discriminator for example:
  self.downsample = nn.AvgPool2d(3, stride=2, padding=1, count_include_pad=False)

I have a model (MUNIT) with it, and this is the image it produced:

It looks like the checkerboard problem, or at least a gradient problem but I checked my Convolution layers and didn't found the error described above. They all are of size 4 with stride 2 or an uneven size with stride of 1.

Output:
This doesn't look like a checkerboard artifact honestly. Also I don't think discriminator would be the problem, it's usually about image restoration (generator or decoder).
Took a quick look at the MUNIT and what they use in Decoder is torch.nn.Upsample with nearest neighbor upsampling (exact code line here).
You may try to use torch.nn.Conv2d followed by torch.nn.PixelShuffle, something like this:
import torch

in_channels = 32
upscale_factor = 2
out_channels = 16

upsampling = torch.nn.Sequential(
    torch.nn.Conv2d(
        in_channels,
        out_channels * upscale_factor * upscale_factor,
        kernel_size=3,
        padding=1,
    ),
    torch.nn.PixelShuffle(upscale_factor),
)

image = torch.randn(1, 32, 16, 16)

upsampling(image).shape  # [1, 16, 32, 32]

This allows neural network to learn how to upsample the image instead of merely using torch.nn.Upsample which the network has no control over (and using below trick it should also be free of checkerboard artifacts).
Additionally, ICNR initialization for Conv2d should also help (possible implementation here or here). This init scheme initializes weights to act similar to nearest neighbor upsampling at the beginning (research paper here).


Instruction: Pytorch RNN with no nonlinearity

Input: Is it possible to implement an RNN layer with no nonlinearity in Pytorch like in Keras where one can set the activation to linear? By removing the nonlinearlity, I want to implement a first-order infinite-impulse-response (IIR) filter with a differentiable parameter and integrate it into my model for end-to-end learning. I can obviously implement the filter in Pytorch but I thought using an inbuilt function may be more efficient.

Output:
Removing non-linearity from RNN turns it into a linear dense layer without any activation.
If that is what you want, then simply use nn.linear and set activation to None
Explanation
Here is why this happens. Fundamentally, an RNN for timesteps works as below -

h(t) = tanh(U.x(t) + W.h(t−1) + b)

h(0) = tanh(U0.x(0) + b0)
h(1) = tanh(U1.x(1) + W1.h(0) + b1)
h(2) = tanh(U2.x(2) + W2.h(1) + b2)

#... and so on.

If you remove linearity by removing the tanh, here is what happens -
h(0) = U0.x(0) + b0
h(1) = U1.x(1) + W1.h(0) + b1
     = U1.x(1) + W1.(U0.x(0) + b0) + b1 #expanding x(0)
     = U1.x(1) + W1.U0.x(0) + W1.b0 + b1
     = U1.x(1) + W1.U0.x(0) + W1.b0 + b1
     = V1.x(1) + V0.x(0) + C    #Can be rewritten with new weights
     = V . x + C    #General form

So the final form of the state of an RNN after 2 timesteps is simply Wx+b like the linear layer without activation.
In other words, removing the non-linearity from an RNN turns it into a linear dense layer without any activation, completely removing the notion of time-steps.