Instruction: RuntimeError Pytoch Unable to find a valid cuDNN algorithm to run convolution

Input:
I want to test a github for my work:
https://github.com/tufts-ml/GAN-Ensemble-for-Anomaly-Detection
so I did a
git clone https://github.com/tufts-ml/GAN-Ensemble-for-Anomaly-Detection

Unfortunately, I have an error when I do the command
sh experiments/run_mnist_en_fanogan.sh

(from the github README)
sh experiments/run_mnist_en_fanogan.sh                                                                                 1 ✘

/home/svetlana/.local/lib/python3.9/site-packages/torch/cuda/__init__.py:106: UserWarning:
NVIDIA GeForce RTX 3080 Laptop GPU with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3080 Laptop GPU GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
/home/svetlana/.local/lib/python3.9/site-packages/torchvision/datasets/mnist.py:498:      UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:180.)
  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
Traceback (most recent call last):
  File "/home/svetlana/Documents/git/GAN-Ensemble-for-Anomaly-Detection/train.py", line 30, in <module>
    main()
  File "/home/svetlana/Documents/git/GAN-Ensemble-for-Anomaly-Detection/train.py", line 24, in main
    model.train()
  File "/home/svetlana/Documents/git/GAN-Ensemble-for-Anomaly-Detection/models/f_anogan.py", line 155, in train
    self.gan_training(epoch)
  File "/home/svetlana/Documents/git/GAN-Ensemble-for-Anomaly-Detection/models/f_anogan.py", line 93, in gan_training
    fake_imgs = self.net_Gds[i_G](z)
  File "/home/svetlana/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/svetlana/Documents/git/GAN-Ensemble-for-Anomaly-Detection/models/networks.py", line 175, in forward
    output = self.main(input)
  File "/home/svetlana/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/svetlana/.local/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/svetlana/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/svetlana/.local/lib/python3.9/site-packages/torch/nn/modules/conv.py", line 916, in forward
    return F.conv_transpose2d(
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

I thought my installation is ok but now I have doubts. This is my installation:
Python 3.9.6 (default, Jun 30 2021, 10:22:16)

 nvcc  --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jul_14_19:41:19_PDT_2021
Cuda compilation tools, release 11.4, V11.4.100
Build cuda_11.4.r11.4/compiler.30188945_0

import torch
print(torch.__version__)
1.9.0+cu102

I installed cudnn-11.4 from nvidia website (https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html),I don't know the command to check the version, I tried this one:
cat /opt/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

but it returns nothing
I tried solutions found here: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize,
without succes (to show VRAM, I used nvtop)

Output:
@Berriel
You right, I was focus on the error.
To solve my problem, I did
pip uninstall torch torchvision torchaudio

Then
pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

According to
https://pytorch.org/get-started/locally/

(this link is from the warning message)


Instruction: what is the difference of torch.nn.Softmax, torch.nn.funtional.softmax, torch.softmax and torch.nn.functional.log_softmax

Input:
I tried to find documents but cannot find anything about torch.softmax.
What is the difference among torch.nn.Softmax, torch.nn.funtional.softmax, torch.softmax and torch.nn.functional.log_softmax?
Examples are appreciated.

Output:
import torch

x = torch.rand(5)

x1 = torch.nn.Softmax()(x)
x2 = torch.nn.functional.softmax(x)
x3 = torch.nn.functional.log_softmax(x)

print(x1)
print(x2)
print(torch.log(x1))
print(x3)

tensor([0.2740, 0.1955, 0.1519, 0.1758, 0.2029])
tensor([0.2740, 0.1955, 0.1519, 0.1758, 0.2029])
tensor([-1.2946, -1.6323, -1.8847, -1.7386, -1.5952])
tensor([-1.2946, -1.6323, -1.8847, -1.7386, -1.5952])

torch.nn.Softmax and torch.nn.functional.softmax gives identical outputs, one is a class (pytorch module), another one is a function.
log_softmax applies log after applying softmax.
NLLLoss takes log-probabilities (log(softmax(x))) as input. So, you would need log_softmax for NLLLoss, log_softmax is numerically more stable, usually yields better results.


Instruction: Use of torch.stack()

Input:
t1 = torch.tensor([1,2,3])
t2 = torch.tensor([4,5,6])
t3 = torch.tensor([7,8,9])

torch.stack((t1,t2,t3),dim=1)

When implementing the torch.stack(), I can't understand how stacking is done for different dim.
Here stacking is done for columns but I can't understand the details as to how it is done. It becomes more complicated dealing with 2-d or 3-D tensors.
tensor([[1, 4, 7],
        [2, 5, 8],
        [3, 6, 9]])

Output:
Imagine have n tensors. If we stay in 3D, those correspond to volumes, namely rectangular cuboids. Stacking corresponds to combining those n volumes on an additional dimension: here a 4th dimension is added to host the n 3D volumes. This operation is in clear contrast with concatenation, where the volumes would be combined on one of the existing dimensions. So concatenation of three-dimensional tensors would result in a 3D tensor.
Here is a possible representation of the stacking operations for limited dimensions sizes (up to three-dimensional inputs):

Where you chose to perform the stacking defines along which new dimension the stack will take place. In the above examples, the newly created dimension is last, hence the idea of "added dimension" makes more sense.
In the following visualization, we observe how tensors can be stacked on different axes. This in turn affects the resulting tensor shape

For the 1D case, for instance, it can also happen on the first axis, see below:

With code:
>>> x_1d = list(torch.empty(3, 2))     # 3 lines

>>> torch.stack(x_1d, 0).shape         # axis=0 stacking
torch.Size([3, 2])

>>> torch.stack(x_1d, 1).shape         # axis=1 stacking
torch.Size([2, 3])

Similarly for two-dimensional inputs:

With code:
>>> x_2d = list(torch.empty(3, 2, 2))   # 3 2x2-squares

>>> torch.stack(x_2d, 0).shape          # axis=0 stacking
torch.Size([3, 2, 2])

>>> torch.stack(x_2d, 1).shape          # axis=1 stacking
torch.Size([2, 3, 2])

>>> torch.stack(x_2d, 2).shape          # axis=2 stacking
torch.Size([2, 2, 3])

With this state of mind, you can intuitively extend the operation to n-dimensional tensors.


Instruction: Pytorch Tensor - How to get the index of a tensor given a multidimensional tensor

Input:
I have a the following tensor lets call it lookup_table:
tensor([266, 103,  84,  12,  32,  34,   1, 523,  22, 136, 268, 432,  53,  63,
        201,  51, 164,  69,  31,  42, 122, 131, 119,  36, 245,  60,  28,  81,
          9, 114, 105,   3,  41,  86, 150,  79, 104, 120,  74, 420,  39, 427,
         40,  59,  24, 126, 202, 222, 145, 429,  43,  30,  38,  55,  10, 141,
         85, 121, 203, 240,  96,   7,  64,  89, 127, 236, 117,  99,  54,  90,
         57,  11,  21,  62,  82,  25, 267,  75, 111, 518,  76,  56,  20,   2,
         61, 516,  80,  78, 555, 246, 133, 497,  33, 421,  58, 107,  92,  68,
         13, 113, 235, 875,  35,  98, 102,  27,  14,  15,  72,  37,  16,  50,
        517, 134, 223, 163,  91,  44,  17, 412,  18,  48,  23,   4,  29,  77,
          6, 110,  67,  45, 161, 254, 112,   8, 106,  19, 498, 101,   5, 157,
         83, 350, 154, 238, 115,  26, 142, 143])

And I have another tensor lets call it data, which looks like this:
tensor([[517, 235, 236,  76,  81,  25, 110,  59, 245,  39],
        [523, 114, 350, 246,  30, 222,  39, 517, 106,   2],
        [ 35, 235, 120,  99, 266,  63, 236, 133, 412,  38],
        [134,   2, 497,  21,  78,  60, 142, 498,  24,  89],
        [ 60, 111, 120, 145,  91, 141, 164,  81, 350,  55]])

Now I want something which looks similar to this:
tensor([112, 100, ..., 40],
       [7, 29, ..., 2],
       ...,          ])

I want to use my data tensor to get the index of the lookup table.
Basically I want to vectorize this:
(lookup_table == data).nonzero()

So that this works for multidimensional arrays.
I have read this, but they are not working for my case:
How Pytorch Tensor get the index of specific value
How Pytorch Tensor get the index of elements?
Pytorch tensor - How to get the indexes by a specific tensor
EDIT:
I am basically searching for an optimized/vectorized version of this:
x_data = torch.stack([(lookuptable == data[0][i]).nonzero(as_tuple=False) for i in range(len(data[0]))]).flatten().unsqueeze(0)
print(x_data.size())
for o in range(1, len(data)):
    x_data = torch.cat((x_data, torch.stack([(lookuptable == data[o][i]).nonzero(as_tuple=False) for i in range(len(data[o]))]).flatten().unsqueeze(0)), dim=0)

EDIT 2 Minimal example:
We have the data tensor:
data = torch.Tensor([
        [523, 114, 350, 246,  30, 222,  39, 517, 106,   2],
        [ 35, 235, 120,  99, 266,  63, 236, 133, 412,  38],
        [555, 104,  14,  81,  55, 497, 222,  64,  57, 131]
])

And we have the lookup_table tensor, see above.
If we apply this code to the 2 tensors:
 # convert champion keys into index notation
x_data = torch.stack([(lookuptable == x[0][i]).nonzero(as_tuple=False) for i in range(len(x[0]))]).flatten().unsqueeze(0)
for o in range(1, len(data) - 1):
    x_data = torch.cat((x_data, torch.stack([(lookuptable == x[o][i]).nonzero(as_tuple=False) for i in range(len(x[o]))]).flatten().unsqueeze(0)), dim=0)

We get an output of this:
tensor([[  7,  29, 141,  89,  51,  47,  40, 112, 134,  83],
        [102, 100,  37,  67,   0,  13,  65,  90, 119,  52],
        [ 88,  36, 106,  27,  53,  91,  47,  62,  70,  21]
       ])

This output is what I want, and like I said above its the index of where each value of the tensor data lies on the tensor lookuptable.
The problem is that this is not vectorized.
And I have no Idea how to vectorize it.

Output:
Using searchsorted:
Scanning the whole lookup_table array for each input element is quite inefficient. How about sorting the lookup table first (this only needs to be done once)
sorted_lookup_table, indexes = torch.sort(lookup_table)

and then using searchsorted
index_into_sorted = torch.searchsorted(sorted_lookup_table, data)

If you need an index into the original lookup_table, you can get it with
index_into_lookup_table = indexes[index_into_sorted]


Instruction: Why can't torchtext find a symbol _ZN2at6detail10noopDeleteEPv?

Input:
Why can't torchtext find this symbol?
(synthesis) miranda9~/ultimate-utils $ python ~/type-parametric-synthesis/src/main.py --reproduce_10K --serial --debug --num_workers 0

Traceback (most recent call last):
  File "/home/miranda9/type-parametric-synthesis/src/main.py", line 32, in <module>
    from data_pkg.data_preparation import get_dataloaders, get_simply_type_lambda_calc_dataloader_from_folder
  File "/home/miranda9/type-parametric-synthesis/src/data_pkg/data_preparation.py", line 10, in <module>
    from torchtext.vocab import Vocab, vocab
  File "/home/miranda9/miniconda3/envs/synthesis/lib/python3.9/site-packages/torchtext/__init__.py", line 5, in <module>
    from . import vocab
  File "/home/miranda9/miniconda3/envs/synthesis/lib/python3.9/site-packages/torchtext/vocab.py", line 13, in <module>
    from torchtext._torchtext import (
ImportError: /home/miranda9/miniconda3/envs/synthesis/lib/python3.9/site-packages/torchtext/_torchtext.so: undefined symbol: _ZN2at6detail10noopDeleteEPv

Output:
Reinstall torchtext with the current version of pytorch:
e.g.
conda install -y torchtext -c pytorch

or for older versions of pytorch torchtext ImportError in colab
conda install -y torchtext==0.8.0 -c pytorch

Though in general it seems in my experience that torchtext install the right version of pytorch on its own...
Ref: https://github.com/PyTorchLightning/pytorch-lightning/issues/4533


Instruction: Installing pyTorch / Torch

Input:
I am trying to install PyTorch / torch on pyCharm Community edition. It gave the following error:

ERROR: Command errored out with exit status 1:
command: 'c:\users\joshu\uiuc\research\ai sound-20210814t005717z-001\ai sound\venv\scripts\python.exe' -u -c
'import io, os, sys, setuptools, tokenize; sys.argv[0] =
'"'"'C:\Users\Public\Documents\Wondershare\CreatorTemp\pip-install-ygcl9r4a\pytorch_62f432ab0d344f46a572a5a74f2b015b\setup.py'"'"';
file='"'"'C:\Users\Public\Documents\Wondershare\CreatorTemp\pip-install-ygcl9r4a\pytorch_62f432ab0d344f46a572a5a74f2b015b\setup.py'"'"';f
= getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import
setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))'
install --record
'C:\Users\Public\Documents\Wondershare\CreatorTemp\pip-record-5cueumzp\install-record.txt'
--single-version-externally-managed --compile --install-headers 'c:\users\joshu\uiuc\research\ai sound-20210814t005717z-001\ai
sound\venv\include\site\python3.8\pytorch'
cwd: C:\Users\Public\Documents\Wondershare\CreatorTemp\pip-install-ygcl9r4a\pytorch_62f432ab0d344f46a572a5a74f2b015b
Complete output (5 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Public\Documents\Wondershare\CreatorTemp\pip-install-ygcl9r4a\pytorch_62f432ab0d344f46a572a5a74f2b015b\setup.py",
line 11, in
raise Exception(message)
Exception: You tried to install "pytorch". The package named for PyTorch is "torch"

sorry that this is a noob question, but help is appreciated :)

Output:
torch is the name of the package, not pytorch.
Type the following in your terminal.
pip install torch


Instruction: How to sample from a distribution with constraints in PyTorch?

Input:
I got a simple situation like:

Generate samples and log_prob from an uniform distribution for 2-dim variables (m1, m2) which is satisfying m1~U(5, 80), m2~U(5, 80) with constraint m1+m2 < 100.

from torch import distributions
prior = distributions.Uniform(torch.tensor([5.0, 5.0]),
                              torch.tensor([80.0, 80.0]))

I try coding in PyTorch like the above, but I don't know how to construct a torch.distribution with constraint condition. By the way, I see some implementations about torch.distributions.constraints but I can't figure out how to use it.

Output:
This can be achieved using rejection sampling:
d = torch.distributions.Uniform(5, 80)

m1 = 80
m2 = 80

while m1 + m2 >= 100:
    m1 = d.sample()
    m2 = d.sample()
    print(m1, m2)

Example output:
tensor(52.3322) tensor(67.8245)
tensor(68.3232) tensor(40.0983)
tensor(44.7374) tensor(9.9690)


Instruction: How do I match samples with their predictions when doing inference with PyTorch's DistributedSampler?

Input:
I have trained a torch model for NLP tasks and would like to perform some inference using a multi GPU machine (in this case with two GPUs).
Inside the processing code, I use this
dataset = TensorDataset(encoded_dict['input_ids'], encoded_dict['attention_mask'])
sampler = DistributedSampler(
    dataset, num_replicas=args.nodes * args.gpus, rank=args.node_rank * args.gpus +    gpu_number, shuffle=False
)
dataloader = DataLoader(dataset, batch_size=batch_size, sampler=sampler)

For those familiar with NLP, encoded_dict is the output from the tokenizer.batch_encode_plus function where the tokenizer is an instance of transformers.BertTokenizer.
The issue I’m having is that when I call the code through the torch.multiprocessing.spawn function, each GPU is doing predictions (i.e. inference) on a subset of the full dataset, and saving the predictions separately; for example, if I have a dataset with 1000 samples to predict, each GPU is predicting 500 of them. As a result, I have no way of knowing which samples out of the 1000 were predicted by which GPU, as their order is not preserved, therefore the model predictions are meaningless as I cannot trace each of them back to their input sample.
I have tried to save the dataloader instance (as a pickle) together with the predictions and then extracting the input_ids by using dataloader.dataset.tensors, however this requires a tokeniser decoding step which I rather avoid, as the tokenizer will have slightly changed the text (for example double whitespaces would be removed, words with dashes will have been split and so on).
What is the cleanest way to save the input text samples together with their predictions when doing inference in distributed mode, or alternatively to keep track of which prediction refers to which sample?

Output:
As I understand it, basically your dataset returns for an index idx [data,label] during training and [data] during inference. The issue with this is that the idx is not preserved by the dataloader object, so there is no way to obtain the idx values for the minibatch after the fact.
One way to handle this issue is to define a very simple custom dataset object that also returns [data,id] instead of only data during inference. Probably the easiest way to do this is to make the dataset return a dictionary object with keys id and data. The dictionary return type is convenient because Pytorch collates (converts data structures to batches) this type automatically, otherwise you'd have to define a custom collate_fn  and pass it to the dataloader object, which is itself not very hard but is an extra step.
In any case, here's I would define a new dataset object as follows which should be almost a one-to-one substitute for your current dataset (I believe):
def TensorDictDataset(torch.data.Dataset):
    def __init__(self,ids,attention_mask):
        self.ids = ids
        self.mask = attention_mask

    def __len__(self):
        return len(self.ids)

    def __getitem(self,idx):
        datum = {
            "mask": self.mask[idx],
            "id":ids[idx]
                }
        return datum

The only change you'll then have to make is that rather than returning mask your dataset will now return dict{"mask":mask,"id":id} so you'll have to parse that appropriately.


Instruction: Importing torchsparse (PyTorch) on Windows 10 not working

Input:
On Windows 10. I am testing someone's code which has the following imports:
import torch.nn as nn
import torchsparse.nn as spnn
from torchsparse.point_tensor import PointTensor

So on my machine I successfully installed via
pip install torch-scatter torch-sparse -f https://data.pyg.org/whl/torch-1.9.0+cu111.html

As I have CUDA 11.1. However, there seems to be a syntax difference, as the above imports give:
    import torchsparse.nn as spnn
ModuleNotFoundError: No module named 'torchsparse'

I found that when I am in Python I can do the following:
>>> import torchsparse
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torchsparse'
>>> import torchsparse.nn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torchsparse'
>>> import torch_sparse
>>> import torch_sparse.nn
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch_sparse.nn'
>>>

So I can only import torch_sparse. Does anyone know how I can get the equivalent imports to test my buddy's code? Much appreciated.
---- EDIT ----
Trying Ivan's answer, I got the following:
pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
WARNING: Ignoring invalid distribution -ip (c:\python39\lib\site-packages)
WARNING: Ignoring invalid distribution -ip (c:\python39\lib\site-packages)
Collecting git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
  Cloning https://github.com/mit-han-lab/torchsparse.git (to revision v1.4.0) to c:\users\iijds\appdata\local\temp\pip-req-build-gvmcjx1m
  Running command git clone -q https://github.com/mit-han-lab/torchsparse.git 'C:\Users\iiJDS\AppData\Local\Temp\pip-req-build-gvmcjx1m'
  Running command git checkout -q 74099d10a51c71c14318bce63d6421f698b24f24
  Resolved https://github.com/mit-han-lab/torchsparse.git to commit 74099d10a51c71c14318bce63d6421f698b24f24
Building wheels for collected packages: torchsparse
  Building wheel for torchsparse (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"'; __file__='"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\iiJDS\AppData\Local\Temp\pip-wheel-rz9wo9lc'
       cwd: C:\Users\iiJDS\AppData\Local\Temp\pip-req-build-gvmcjx1m\
  Complete output (53 lines):
  running bdist_wheel
  c:\python39\lib\site-packages\torch\utils\cpp_extension.py:370: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
    warnings.warn(msg.format('we could not find ninja.'))
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.9
  creating build\lib.win-amd64-3.9\torchsparse
  copying torchsparse\operators.py -> build\lib.win-amd64-3.9\torchsparse
  copying torchsparse\tensor.py -> build\lib.win-amd64-3.9\torchsparse
  copying torchsparse\version.py -> build\lib.win-amd64-3.9\torchsparse
  copying torchsparse\__init__.py -> build\lib.win-amd64-3.9\torchsparse
  creating build\lib.win-amd64-3.9\torchsparse\nn
  copying torchsparse\nn\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn
  creating build\lib.win-amd64-3.9\torchsparse\utils
  copying torchsparse\utils\collate.py -> build\lib.win-amd64-3.9\torchsparse\utils
  copying torchsparse\utils\quantize.py -> build\lib.win-amd64-3.9\torchsparse\utils
  copying torchsparse\utils\utils.py -> build\lib.win-amd64-3.9\torchsparse\utils
  copying torchsparse\utils\__init__.py -> build\lib.win-amd64-3.9\torchsparse\utils
  creating build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\activation.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\conv.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\count.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\crop.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\devoxelize.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\downsample.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\hash.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\pooling.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\query.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\voxelize.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  copying torchsparse\nn\functional\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
  creating build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\activation.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\bev.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\conv.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\crop.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\norm.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\pooling.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  copying torchsparse\nn\modules\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
  creating build\lib.win-amd64-3.9\torchsparse\nn\utils
  copying torchsparse\nn\utils\apply.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
  copying torchsparse\nn\utils\kernel.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
  copying torchsparse\nn\utils\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
  running build_ext
  c:\python39\lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
    warnings.warn(f'Error checking compiler version for {compiler}: {error}')
  building 'torchsparse.backend' extension
  error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
  Error in atexit._run_exitfuncs:
  Traceback (most recent call last):
    File "c:\python39\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
      return stream.closed
  ValueError: underlying buffer has been detached
  ----------------------------------------
  ERROR: Failed building wheel for torchsparse
  Running setup.py clean for torchsparse
Failed to build torchsparse
WARNING: Ignoring invalid distribution -ip (c:\python39\lib\site-packages)
Installing collected packages: torchsparse
    Running setup.py install for torchsparse ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"'; __file__='"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\iiJDS\AppData\Local\Temp\pip-record-p7zn1m6h\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python39\Include\torchsparse'
         cwd: C:\Users\iiJDS\AppData\Local\Temp\pip-req-build-gvmcjx1m\
    Complete output (53 lines):
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.9
    creating build\lib.win-amd64-3.9\torchsparse
    copying torchsparse\operators.py -> build\lib.win-amd64-3.9\torchsparse
    copying torchsparse\tensor.py -> build\lib.win-amd64-3.9\torchsparse
    copying torchsparse\version.py -> build\lib.win-amd64-3.9\torchsparse
    copying torchsparse\__init__.py -> build\lib.win-amd64-3.9\torchsparse
    creating build\lib.win-amd64-3.9\torchsparse\nn
    copying torchsparse\nn\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn
    creating build\lib.win-amd64-3.9\torchsparse\utils
    copying torchsparse\utils\collate.py -> build\lib.win-amd64-3.9\torchsparse\utils
    copying torchsparse\utils\quantize.py -> build\lib.win-amd64-3.9\torchsparse\utils
    copying torchsparse\utils\utils.py -> build\lib.win-amd64-3.9\torchsparse\utils
    copying torchsparse\utils\__init__.py -> build\lib.win-amd64-3.9\torchsparse\utils
    creating build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\activation.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\conv.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\count.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\crop.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\devoxelize.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\downsample.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\hash.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\pooling.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\query.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\voxelize.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    copying torchsparse\nn\functional\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\functional
    creating build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\activation.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\bev.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\conv.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\crop.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\norm.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\pooling.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    copying torchsparse\nn\modules\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\modules
    creating build\lib.win-amd64-3.9\torchsparse\nn\utils
    copying torchsparse\nn\utils\apply.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
    copying torchsparse\nn\utils\kernel.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
    copying torchsparse\nn\utils\__init__.py -> build\lib.win-amd64-3.9\torchsparse\nn\utils
    running build_ext
    c:\python39\lib\site-packages\torch\utils\cpp_extension.py:370: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
      warnings.warn(msg.format('we could not find ninja.'))
    c:\python39\lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
      warnings.warn(f'Error checking compiler version for {compiler}: {error}')
    building 'torchsparse.backend' extension
    error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
    Error in atexit._run_exitfuncs:
    Traceback (most recent call last):
      File "c:\python39\lib\site-packages\colorama\ansitowin32.py", line 59, in closed
        return stream.closed
    ValueError: underlying buffer has been detached
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"'; __file__='"'"'C:\\Users\\iiJDS\\AppData\\Local\\Temp\\pip-req-build-gvmcjx1m\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\iiJDS\AppData\Local\Temp\pip-record-p7zn1m6h\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\python39\Include\torchsparse' Check the logs for full command output.

Output:
It appears the package you are trying to import comes from this Github repo, which is different to the package you installed: torch-sparse you tried using. You should try with this instead:
pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0

Also, you can uninstall the other one with:
pip uninstall torch-sparse


Instruction: What is different between The MaxPool1D API in tensorflow 2.X and MaxPool1d in pytorch

Input:
I'm trying to re-implement code generated in tensorflow into pytroch, but I came across maxpooling, looked into the documentation of the two frameworks, and found that their behavior is not the same. Can someone please explain to me why they are different, and which one is more efficient (I ask this because they give a different result)?
import tensorflow
from tensorflow.keras.layers import  GlobalMaxPool1D
tf_tensor = tensorflow.random.normal([8, 6, 5])
tf_maxpool = GlobalMaxPool1D()
print("output shape : ", tf_maxpool(tf_tensor).shape)

output shape :  (8, 5)

import torch
import torch.nn as nn

torch_tensor = torch.tensor(tf_tensor.numpy())
maxpool = nn.MaxPool1d(kernel_size=2)
print("output shape : ", maxpool(torch_tensor).shape)

output shape :  torch.Size([8, 6, 2])

Output:
MaxPool vs GlobalMaxPool
torch.nn.MaxPool1d pools every N adjacent values by performing max operation.
For these values:
[1, 2, 3, 4, 5, 6, 7, 8]

with kernel_size=2 as you've specified, you would get the following values:
[2, 4, 6, 8]

which means a sliding window of size 2 gets the maximum value and moves on to the next pair.
Global Pooling is a similar operation, but gets the maximum value from the whole list, as pointed out in Ivan's answer. In our case, we would simply get one 8 value.
This operation, in PyTorch, is called torch.nn.AdaptiveAvgPool1d (optionally followed by torch.nn.Flatten):
import torch

tensor = torch.randn(8, 6, 5)

global_max_pooling = torch.nn.Sequential(
    torch.nn.AdaptiveMaxPool1d(1), # (8, 6, 1) shape
    torch.nn.Flatten(), # (8, 6) after removing unnecessary 1 dimension
)

global_max_pooling(tensor) # (8, 6)

The above explanation is simplified as this operation is carried across specific dimension.
Tensorflow vs PyTorch shape difference
As one could notice, in the case of Tensorflow the output is of shape (8, 5), while in the case of PyTorch it is (8, 6).
This difference stems from different channels dimensions (see here for channels last in PyTorch), namely:

PyTorch assumes data layout of (batch, channels, sequence)
Tensorflow assumes data layout of (batch, sequence, channels) (a.k.a. channels last)

One has to permute the data in case of PyTorch to get exactly the same results:
tensor = tensor.permute(0, 2, 1) # (8, 5, 6)
global_max_pooling(tensor) # (8, 5)

Efficiency
Use torch.nn.AdaptiveAvgPool1d when you want to perform pooling with specified output size (different than 1) as it skips some unnecessary operations torch.nn.MaxPool1d performs (going over the same elements more than once, which is out of scope of this question).
In general case, when we perform global pooling both are roughly equivalent and perform the same number of operations.


Instruction: Unfreeze model Layer by Layer in PyTorch

Input:
I'm working with a PyTorch model from here (T2T_ViT_7).
I'm trying to freeze all layers except the last (head) layer and train it on my dataset. I want to evaluate its performance, and then unfreeze layers one by one and train it each time and see how it performs.
To initially freeze all the layers and and just unfreeze the head layer, I used:
for param in model.parameters():
    param.requires_grad_(False)

model.head.requires_grad_(True)

Now I want to start from the bottom, and start unfreezing layers one by one. How can I do this? Do I use model.modules() or maybe model.children()?
Thank you!

Output:
If by layers you mean each block inside of model.blocks, then you can use nn.Module.children (// nn.Module.named_children). This will return all direct submodules, while the nn.Module.modules returns all submodules recursively.
Since model.blocks is a nn.ModuleList, you can slice the blocks to only select the last n layers. Something like that:
model.blocks[-n:].requires_grad_(False)


Instruction: Pytorch: Building 3D Dense Network run into error adaptive_avg_pool3d: output_size must be 3

Input:
I ran into such an error when trying to train my 3D Dense Network. There is an average pooling layer at the end of convolution blocks. As can be seen in the message below, it says that my code out = F.adaptive_avg_pool3d(input=out, output_size=[1,1,1]) does not give the right-sized output.
I have tried with output_size=1, output_size=(1,1,1) and tried to use layer F.avg_pool instead, but all ran into such error. It's quiet strange since my output does have the right size.
P.S. I have imported torch.nn.functional as F

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

/tmp/ipykernel_37419/694291867.py in <module>
     11 trainer = Trainer(model, device, optimizer, loss_fn)
     12
---> 13 trainer.fit(N_EPOCHS, train_dataloader, val_dataloader, "4C", N_EPOCHS / 3, )
     14
     15 trainer.display_plots("4C")

/tmp/ipykernel_37419/579275755.py in fit(self, epochs, train_dataloader, valid_dataloader, mrimodule, patience)
     26             self.info_message("EPOCH: {}", n_epoch)
     27
---> 28             train_loss, train_auc, train_time = self.train_epoch(train_dataloader)
     29             valid_loss, valid_auc, valid_time = self.valid_epoch(valid_dataloader)
     30

/tmp/ipykernel_37419/579275755.py in train_epoch(self, train_dataloader)
     71             Y = Y.to(self.device)
     72             self.optimizer.zero_grad()
---> 73             pred = self.model(X).squeeze(1)
     74             loss = self.loss_fn(pred, Y)
     75

~/miniconda3/envs/MGMT_Classify/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/MGMT_Classify_Scripts/DenseNet3D.py in forward(self, x)
    216         features = self.features(x)
    217         out = F.relu(features, inplace=True)
--> 218         out = F.adaptive_avg_pool3d(input=out, output_size=[1,1,1])
    219         out = torch.flatten(out, 1)
    220         out = self.classifier(out)

~/miniconda3/envs/MGMT_Classify/lib/python3.7/site-packages/torch/nn/functional.py in adaptive_avg_pool3d(input, output_size)
    954                 adaptive_avg_pool3d, (input,), input, output_size)
    955     _output_size = _list_with_default(output_size, input.size())
--> 956     return torch._C._nn.adaptive_avg_pool3d(input, _output_size)
    957
    958

RuntimeError: adaptive_avg_pool3d: output_size must be 3

Output:
I have at least found one temporary solution to this by comparing my script with other 3D densenet models on GitHub.
I used the code:
out = F.adaptive_avg_pool3d(input=out, output_size=(1, 1, 1)).view(features.size(0), -1)

In place of the following, from the original script:
out = F.adaptive_avg_pool3d(input=out, output_size=[1,1,1])
out = torch.flatten(out, 1)

After that, it works. I have not tried it on my task now, but I have confirmed that it can overfit a very small test dataset.
I am still very curious what on earth lead to the error and why it can be fixed in this way. If you have insight, please offer it in the comments. Otherwise, this offers a workaround for anyone else that needs it.


Instruction: How to Change the torchtext LabelField value

Input:
I'm using PyTorch to create several models which each one is run in a separate notebook.
When using torch text Field to create vocab it is assigning a number for each class that is correct and my original class labels also are numbers. But the assigned label for each class is not the same as the original class label. I was wondering is there a way to assign an exact number class for my Label vocab.
my code that creates torch text Field:
LABEL = data.LabelField()
LABEL.build_vocab(train_data)

my result's like this:
print(LABEL.vocab.stoi)
defaultdict(None, {'1': 0, '2': 1, '0': 2})

the result's I want:
defaultdict(None, {'0': 0, '1': 1, '2': 2})

I write this code for the solution. Is it correct to create vocab like this?
 LABEL.build_vocab({'0': 0, '1': 1, '2': 2})

p.s: I know this assigning is just used in models and everything works fine but I was worried about the time I comparison models results on test data and was more worried about my confusion each time I look at the confession matrix.

Output:
I don't think this is going to give you what you want. build_vocab iterates over a dataset and maps an item to an index if it appears in the dataset above some min_freq (default of min_freq=1). I think what you are giving it in your last example will tell build_vocab that the item '0' appears 0 times, so it won't be included in your dataset.
If you are concerned about mixing things up in your review process, you can always write a script to get the index of a certain label, then get whatever is at that index, and map it to a new dict with the index you want. This will probably be much easier than messing with the way torchtext is building your vocabulary.'
EDIT: A better solution for you mght be setting use_vocab=False when defining your Label field:
LABEL = data.LabelField(use_vocab=False)

This will work in your case, when your data is already numerical. From the torchtext 0.8 docs:

use_vocab: Whether to use a Vocab object. If False, the data in this
field should already be numerical. Default: True.


Instruction: how is stacked rnn (num layers > 1) implemented on pytorch?

Input:
The GRU layer in pytorch takes in a parameter called num_layers, where you can stack RNNs. However, it is unclear how exactly the subsequent RNNs use the outputs of the previous layer.
According to the documentation:
Number of recurrent layers. E.g., setting num_layers=2 would mean stacking two GRUs together to form a stacked GRU, with the second GRU taking in outputs of the first GRU and computing the final results.
Does this mean that the output of the final cell of the first layer of the GRU is fed as input to the next layer? Or does it mean the outputs of each cell (at each timestep) is fed as an input to the cell at the same timestep of the next layer?

Output:
Does this mean that the output of the final cell of the first layer of the GRU is fed as input to the next layer? Or does it mean the outputs of each cell (at each timestep) is fed as an input to the cell at the same timestep of the next layer?

The latter. Each time step's output from the first layer is used as input for the same time step of the second layer.
This figure from a Keras tutorial shows how multilayer RNNs are structured:


Instruction: Could not find the pytorch 1.9.1 in conda's current channels

Input:
I create a new conda virtual environment and try to install the pytorch 1.9.1 by using conda install pytorch=1.9.1. But, conda reports the PackageNotFoundError as follows.
PackagesNotFoundError: The following packages are not available from current channels:

pytorch==1.9.1

Current channels:

https://conda.anaconda.org/python/win-64
https://conda.anaconda.org/python/noarch
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-64
https://repo.anaconda.com/pkgs/msys2/noarch
Help！！！ Thanks

Output:
The right commands are listed on the pytorch web site. They should use the pytorch channel, e.g. with cuda:
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia


Instruction: How does one pip install torch 1.9.x with cuda 11.1 when errors related with memory issue arise?

Input:
I was trying to install torch 1.9.x with pip3 but I get this error:
(metalearning_gpu) miranda9~/automl-meta-learning $ pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu111
ERROR: Exception:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 203, in wrapper
    return func(self, options, args)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 315, in run
    requirement_set = resolver.resolve(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
    result = self._result = resolver.resolve(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 341, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 140, in __bool__
    return any(self)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 128, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 32, in _iter_built
    candidate = func()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 204, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 295, in __init__
    super().__init__(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 550, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 239, in unpack_url
    file = get_http_url(
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 102, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/download.py", line 132, in __call__
    resp = _http_get_download(self._session, link)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/download.py", line 115, in _http_get_download
    resp = session.get(target_url, headers=HEADERS, stream=True)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/session.py", line 454, in request
    return super().request(method, url, *args, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/cachecontrol/adapter.py", line 44, in send
    cached_response = self.controller.cached_request(request)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_vendor/cachecontrol/controller.py", line 139, in cached_request
    cache_data = self.cache.get(cache_url)
  File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/pip/_internal/network/cache.py", line 54, in get
    return f.read()
MemoryError

It says there is a memory error but I don't understand where and how? How do I start debugging this?
Nothing in my home seems like a problem:
(metalearning_gpu) miranda9~/automl-meta-learning $ du -hs ~
150G    /home/miranda9

Output:
Not sure why this is the case but it seems once I got a node to run the installs (with a gpu) the install worked...is that normal?!
(synthesis) miranda9~/automl-meta-learning $ condor_submit -i interactive.sub
Submitting job(s).
1 job(s) submitted to cluster 17192.

Could not find conda environment: metalearning_cpu
You can list all discoverable environments with `conda info --envs`.

Welcome to slot1_1@vision-02.cs.illinois.edu!
Could not find conda environment: metalearning_cpu
You can list all discoverable environments with `conda info --envs`.

(synthesis) miranda9~/automl-meta-learning $
(synthesis) miranda9~/automl-meta-learning $
(synthesis) miranda9~/automl-meta-learning $
(synthesis) miranda9~/automl-meta-learning $
(synthesis) miranda9~/automl-meta-learning $
(synthesis) miranda9~/automl-meta-learning $ conda activate metalearning_gpu
(metalearning_gpu) miranda9~/automl-meta-learning $ pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.9.0+cu111
  Using cached https://download.pytorch.org/whl/cu111/torch-1.9.0%2Bcu111-cp39-cp39-linux_x86_64.whl (2041.4 MB)
Collecting torchvision==0.10.0+cu111
  Using cached https://download.pytorch.org/whl/cu111/torchvision-0.10.0%2Bcu111-cp39-cp39-linux_x86_64.whl (23.1 MB)
Collecting torchaudio==0.9.0
  Using cached torchaudio-0.9.0-cp39-cp39-manylinux1_x86_64.whl (1.9 MB)
Collecting typing-extensions
  Using cached typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
Collecting numpy
  Using cached numpy-1.21.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.8 MB)
Collecting pillow>=5.3.0
  Using cached Pillow-8.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
Installing collected packages: typing-extensions, torch, pillow, numpy, torchvision, torchaudio
Successfully installed numpy-1.21.2 pillow-8.3.2 torch-1.9.0+cu111 torchaudio-0.9.0 torchvision-0.10.0+cu111 typing-extensions-3.10.0.2
(metalearning_gpu) miranda9~/automl-meta-learning $

you might want to try this command instead:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html


Instruction: How to visualize a graph from DGL's datasets?

Input:
The following snippet comes from the tutorial https://cnvrg.io/graph-neural-networks/. How can I visualize a graph from the dataset? Using something like matplotlib if possible.
import dgl
import torch
import torch.nn as nn
import torch.nn.functional as F
import dgl.data
dataset = dgl.data.CoraGraphDataset()
g = dataset[0]

Output:
import dgl.data
import matplotlib.pyplot as plt
import networkx as nx

dataset = dgl.data.CoraGraphDataset()
g = dataset[0]
options = {
    'node_color': 'black',
    'node_size': 20,
    'width': 1,
}
G = dgl.to_networkx(g)
plt.figure(figsize=[15,7])
nx.draw(G, **options)

It is a huge graph so you might have to play with the sizes (node, width, figure, etc.).
Here are some useful links:

https://networkx.org/documentation/stable/tutorial.html

https://docs.dgl.ai/en/0.5.x/generated/dgl.to_networkx.html#dgl.to_networkx

https://docs.dgl.ai/en/0.5.x/api/python/dgl.data.html


Instruction: Unble to install requirements for summarization

Input:
I am following this huggingface github(https://github.com/huggingface/transformers/tree/master/examples/pytorch/summarization) for summarization but not able to install packages from requirements.
Command used:

pip install -r requirements.txt

For your information I am trying this Intel oneapi devcloud.Please find the below error.

env: ‘pkg-config’: No such file or directory
    Failed to find sentencepiece pkg-config
    ----------------------------------------
ERROR: Command "/home/uxxxxx/.conda/envs/env/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/home/uxxxx/tmp/pip-install-ld2o4xe1/sentencepiece/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /home/uxxxxx/tmp/pip-record-tl0qz8f6/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /home/uxxxxx/tmp/pip-install-ld2o4xe1/sentencepiece/

Please use "pip install --user <package>" to install user packages.
Please visit the forums at: https://software.intel.com/en-us/forums/intel-devcloud

Thanks in Advance!

Output:
Try installing pkg-config and sentencepiece separately.
sudo apt-get install pkg-config
pip3 install sentencepiece== 0.19.2

Related source: Link
Also, follow the steps which @Abhijeet has mentioned below. That's kinda obvious so not reiterating it.


Instruction: How to find input layers names for intermediate layer in PyTorch model?

Input:
I have some complicated model on PyTorch. How can I print names of layers (or IDs) which connected to layer's input. For start I want to find it for Concat layer. See example code below:
class Concat(nn.Module):
    def __init__(self, dimension=1):
        super().__init__()
        self.d = dimension

    def forward(self, x):
        return torch.cat(x, self.d)

class SomeModel(nn.Module):
    def __init__(self):
        super(SomeModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.conv2 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        self.conc = Concat(1)
        self.linear = nn.Linear(8192, 1)

    def forward(self, x):
        out1 = F.relu(self.bn1(self.conv1(x)))
        out2 = F.relu(self.conv2(x))
        out = self.conc([out1, out2])
        out = F.avg_pool2d(out, 4)
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out

if __name__ == '__main__':
    model = SomeModel()
    print(model)
    y = model(torch.randn(1, 3, 32, 32))
    print(y.size())
    for name, m in model.named_modules():
        if 'Concat' in m.__class__.__name__:
            print(name, m, m.__class__.__name__)
            # Here print names of all input layers for Concat

Output:
You can use type(module).__name__ to get the nn.Module class name:
>>> model = SomeModel()
>>> y = model(torch.randn(1, 3, 32, 32))
>>> for name, m in model.named_modules():
...     if 'Concat' == type(m).__name__:
...         print(name, m)
conc Concat()

Edit: You can actually manage to get the list of operators used to compute the inputs of Concat. However, I don't think you can actually get the attribute names of the nn.Module associated with these operators. This kind of information is not available - and needed - at model inference.
This solution requires you to register a forward hook on the layer with nn.Module.register_forward_hook. Then perform one inference to trigger it, then you can remove the hook. In the forward hook, you have access to the list of inputs and extract the name of the operator from the grad_fn attribute callback. Using nn.Module.register_forward_pre_hook here would be more appropriate since we are only looking at the inputs, and do not need the output.
>>> def op_name(x)
...     return type(x.grad_fn).__name__.replace('Backward0', '')

>>> def forward_hook(module, ins):
...     print([op_name(x) for x in ins[0]])

Attach the hook on model.conc, trigger it and then clean up:
>>> handle = model.conc.register_forward_pre_hook(forward_hook)
>>> model(torch.empty(2, 3, 10, 10, requires_grad=True))
['Relu', 'Relu']

>>> handle.remove()


Instruction: Large, exploding loss in Pytorch transformer model

Input:
I am trying to solve a sequence to sequence problem with a transformer model. The data is derived from a set of crossword puzzles.
The positional encoding and transformer classes are as follows:
class PositionalEncoding(nn.Module):

    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 3000):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        position = torch.arange(max_len).unsqueeze(1)

        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))

        pe = torch.zeros(1, max_len, d_model)

        pe[0, :, 0::2] = torch.sin(position * div_term)
        pe[0, :, 1::2] = torch.cos(position * div_term)

        self.register_buffer('pe', pe)

    def debug(self, x):
        return x.shape, x.size()

    def forward(self, x: Tensor) -> Tensor:

        x = x + self.pe[:, :x.size(1), :]

        return self.dropout(x)

class Transformer(nn.Module):

    def __init__(
        self,
        num_tokens,
        dim_model,
        num_heads,
        num_encoder_layers,
        num_decoder_layers,
        batch_first,
        dropout_p,
    ):
        super().__init__()

        self.model_type = "Transformer"
        self.dim_model = dim_model

        self.positional_encoder = PositionalEncoding(
            d_model=dim_model, dropout=dropout_p, max_len=3000
        )
        self.embedding = nn.Embedding.from_pretrained(vec_weights, freeze=False)#nn.Embedding(num_tokens, dim_model)
        self.transformer = nn.Transformer(
            d_model=dim_model,
            nhead=num_heads,
            num_encoder_layers=num_encoder_layers,
            num_decoder_layers=num_decoder_layers,
            dropout=dropout_p,
            batch_first = batch_first
        )

        self.out = nn.Linear(dim_model, num_tokens)

    def forward(self, src, tgt, tgt_mask=None, src_pad_mask=None, tgt_pad_mask=None):

        src = self.embedding(src)*math.sqrt(self.dim_model)
        tgt = self.embedding(tgt)*math.sqrt(self.dim_model)
        src = self.positional_encoder(src)
        tgt = self.positional_encoder(tgt)

        transformer_out = self.transformer(src, tgt, tgt_mask=tgt_mask, src_key_padding_mask=src_pad_mask, tgt_key_padding_mask=tgt_pad_mask)
        out = self.out(transformer_out)

        return out

    def get_tgt_mask(self, size) -> torch.tensor:
        mask = torch.tril(torch.ones(size, size) == 1)
        mask = mask.float()
        mask = mask.masked_fill(mask == 0, float('-inf'))
        mask = mask.masked_fill(mask == 1, float(0.0))

        return mask

    def create_pad_mask(self, matrix: torch.tensor, pad_token: int) -> torch.tensor:
        return (matrix == pad_token)

The input tensors are a source tensor of size N by S, where N is the batch size and S is the source sequence length, and a target tensor of size N by T, where T is the target sequence length. S is about 10 and T is about 5, while the total number of items is about 160,000-200,000, divided into batch sizes of 512.  They are torch.IntTensors, with elements in the range from 0 to V, where V is the vocabulary length.
The first layer is an embedding layer that takes the input from N by S to N by S by E, where E is the embedding dimension (300), or to N by T by E in the case of the target. The second layer adds position encoding without changing the shape. Then both tensors are passed through the transformer layer, which outputs an N by T by E tensor. Finally, we pass this output through a linear layer, which produces an N by T by V output, where V is the size of the vocabulary used in the problem. Here V is about 56,697. The most frequent tokens (words) appear about 50-60 times in the target tensor.
The transformer class also contains the functions for implementing the masking matrices.
Then we create the model and run it (this process is wrapped in a function).
device = "cuda"

src_train, src_test = torch.utils.data.random_split(src_t, [int(0.9*len(src_t)), len(src_t)-int(0.9*len(src_t))])
src_train, src_test = src_train[:512], src_test[:512]
tgt_train, tgt_test = torch.utils.data.random_split(tgt_t, [int(0.9*len(tgt_t)), len(tgt_t)-int(0.9*len(tgt_t))])
tgt_train, tgt_test = tgt_train[:512], tgt_test[:512]
train_data, test_data = list(zip(src_train, tgt_train)), list(zip(src_test, tgt_test))
train, test = torch.utils.data.DataLoader(dataset=train_data), torch.utils.data.DataLoader(dataset=test_data)

model = Transformer(num_tokens=ntokens, dim_model=300, num_heads=2, num_encoder_layers=3, num_decoder_layers=3, batch_first = True, dropout_p=0.1).to(device)
loss_function = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.0000001)

n_epochs = 50

def train_model(model, optimizer, loss_function, n_epochs):
    loss_value=0
    for epoch in range(n_epochs):
        print(f"Starting epoch {epoch}")
        for batch, data in enumerate(train):

            x, y = data
            if batch%100 == 0:
                print(f"Batch is {batch}")

            batch += 1
            optimizer.zero_grad()

            x, y = torch.tensor(x).to(device), torch.tensor(y).to(device)
            y_input, y_base = y[:, :-1], y[:, 1:]
            y_input, y_base = y_input.to(device), y_base.to(device)

            tgt_mask = model.get_tgt_mask(y_input.shape[1]).to(device)
            pad_token = vocabulary_table[embeddings.key_to_index["/"]]
            src_pad_mask = model.create_pad_mask(x, pad_token).to(device)
            tgt_pad_mask = model.create_pad_mask(y_input, pad_token).to(device)

            z = model(x, y_input, tgt_mask, src_pad_mask, tgt_pad_mask)
            z = z.permute(0, 2, 1).to(device)

            y_base = y_base.long().to(device)

            loss = loss_function(z, y_base).to(device)

            loss.backward()
            nn.utils.clip_grad_norm_(model.parameters(), max_norm=2.0, norm_type=2)
            optimizer.step()

            loss_value += float(loss)

            if batch%100 == 0:
                print(f"For epoch {epoch}, batch {batch} the cross-entropy loss is {loss_value}")

            #Free GPU memory.
            del z
            del x
            del y
            del y_input
            del y_base
            del loss
            torch.cuda.empty_cache()

    return model.parameters(), loss_value

Basically, we split the data into test and training sets and use an SGD optimizer and cross-entropy loss. We create a masking matrix for the padding for both the target and source tensors, and a masking matrix for unseen elements for the target tensor. We then do the usual gradient update steps. Right now, there is no validation loop, because I cannot even get the training loss to decrease.
The loss is very high, reaching more than 1000 after 100 batches. More concerningly, the loss also increases rapidly during training, rather than decreasing. In the code that I included, I tried clipping the gradients, lowering the learning rate, and using a much smaller sample to debug the code.
What could be causing this behavior?

Output:
You are only adding things to your loss, so naturally it can only increase.
loss_value += float(loss)

You're supposed to set it to zero after every epoch. Now you set it to zero once, in the beginning of the training process. There is a training loop template here if you're interested (https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html). This explains the increasing loss. To further troubleshoot (if needed) I'd throw in a validation loop.


Instruction: Are PyTorch-trained models transferable between GPUs and TPUs?

Input: After using a GPU for some to train a PyTorch model, can I use the saved weights to continue training my model on a TPU?

Output:
After using GPU for some time can I use the saved weights to train my model using TPU?

Yes, if you saved your GPU-trained model with, say
    torch.save(model.save_dict(), 'model.pt')

you can load it again for use on a TPU (using https://github.com/pytorch/xla)  in a separate program run with
    import torch_xla.utils.serialization as xser

    model.load_state_dict(xser.load('model.pt'))


Instruction: Getting pytorch backward's RuntimeError: Trying to backward through the graph a second time... when slicing a tensor

Input:
Upon running the code snippet (PyTorch 1.7.1; Python 3.8),
import numpy as np
import torch

def batch_matrix(vector_pairs, factor=2):
    baselen = len(vector_pairs[0]) // factor
    split_batch = []

    for j in range(factor):
        for i in range(factor):
            start_j = j * baselen
            end_j = (j+1) * baselen if j != factor - 1 else None
            start_i = i * baselen
            end_i = (i+1) * baselen if i != factor - 1 else None

            mini_pairs = vector_pairs[start_j:end_j, start_i:end_i, :]
            split_batch.append(mini_pairs)
    return split_batch

def concat_matrix(vectors_):
    vectors = vectors_.clone()
    seq_len, dim_vec = vectors.shape
    project_x = vectors.repeat((1, 1, seq_len)).reshape(seq_len, seq_len, dim_vec)
    project_y = project_x.permute(1, 0, 2)
    matrix = torch.cat((project_x, project_y), dim=-1)
    matrix_ = matrix.clone()

    return matrix_

if __name__ == "__main__":
    vector_list = []
    for i in range(10):
        vector_list.append(torch.randn((5,), requires_grad=True))
    vectors = torch.stack(vector_list, dim=0)
    pmatrix = concat_matrix(vectors)

    factor = np.ceil(vectors.shape[0]/6).astype(int)
    batched_feats = batch_matrix(pmatrix, factor=factor)

    for i in batched_feats:
        i = i + 5
        print(i.shape)
        summed = torch.sum(i)
        summed.backward()

I get the output and error as below:
torch.Size([5, 5, 10])
torch.Size([5, 5, 10])
Traceback (most recent call last):
  File "/home/user/PycharmProjects/project/run.py", line 43, in <module>
    summed.backward()
  File "/home/user/anaconda3/envs/diff/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/user/anaconda3/envs/diff/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

I have read all the existing posts on the issue and could not resolve it myself. Passing retain_graph=True in backward() fixes the issue in the provided snippet, however, the snippet is only an oversimplified version of a large network where retain_graph=True changes the error to the following:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3000, 512]], which is output 0 of TBackward, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I tried setting torch.autograd.set_detect_anomaly(True) and determining the point of failure, but all that I tried failed and the error persisted.
I suspect that if I can understand the cause of error in the current situation then it will help me resolve this error in actual codebase.
Therefore, I want to understand why is it that backward() works fine for first two tensors in batched_feats, while fails for the third one? I would really appreciate if someone can help me see the reuse of an intermediate result that has been freed.
Thanks a lot!

Output:
After backpropagation, the leaf nodes' gradients are stored in their Tensor.grad attributes. The gradients of non-leaf nodes (i.e. the intermediate results to which the error refers) are freed by default, as PyTorch assumes you won't need them. In your example, your leaf nodes are those in vector_list created from torch.randn().
Calling backward() multiple times consecutively accumulates gradients via summation by default (this is useful for recurrent neural networks). This is problematic when existing intermediate results have been freed; the leaf nodes' gradients have not; and the call to backward() involves some of the same leaf nodes and intermediate results as a previous call to backward(). This is the problem you're facing; some of your tensor slices reference the same underlying tensors, and you're not zeroing all the relevant gradients between calls to backward(), but you are implicitly zeroing intermediate gradients.
If you wish to accumulate gradients in the leaf nodes via summation, simply call backward like so: summed.backward(retain_graph = True).
However, if you wish to compute gradients with respect to your batches independently (rather than w.r.t. the leaf nodes in vector_list), then you can just detach your batches at the beginning of each iteration. This will prevent gradients from propagating through them all the way to their common leaf nodes in vector_list (i.e. they become leaf nodes themselves in their own graphs). Detaching a tensor disables gradients for it, so you'll have to re-enable them manually:
for i in batched_feats:
    i = i.detach()
    i.requires_grad = True
    j = i + 5
    print(j.shape)
    summed = torch.sum(j)
    summed.backward()
    print(i.grad) # Prints the gradients stored in i

This is how some data loaders work; they load the data from disk, convert them to tensors, perform augmentation / other preprocessing, and then detach them so that they can serve as leaf nodes in a fresh computational graph. If the application developer wants to compute gradients w.r.t. the data tensors, they do not have to save intermediate results since the data tensors have been detached and thus serve as leaf nodes.


Instruction: How can I fit a curve to a 3d point cloud?

Input:
My goal is to fit a line through a point cloud. The point cloud is approximately cylindrical but can be curved, which is why the fitted line should not be straight.
I’ve tried several things, but this is my current approach :
I use PyTorch to optimise on a surface equation and compute the loss for each point.
However, this does not seem to lead to “good” results, since the plane/surface does not cut the point cloud vertically, which I would expect to lead to the least error.  Since I’m new to PyTorch I don’t know whether it’s a mistake in my code or whether there’s a mathematical problem with the idea. This approach is in the code below.
Additionally, once I’ve fitted this surface, I would like to obtain a line on it, but I’m unsure on the best way to do this.
My questions are :
Why is the fit of the plane/surface not more centred ?
How could I then obtain a curved line of best fit on the resulting surface ?
Other approaches I've tried :

SVD, but like I said, im looking for a line that is not straight
Computing the mean of each z-slice (i.e. each horizontal slice) in the y and x direction and using the resulting points to fit a spline (using python’s splrep). The problem with this approach is that a couple of top or bottom slices can consist of very few points that are not necessarily in the “center” of the point cloud (e.g. are on the left extreme of the top slice), which forces the spline in that direction. It’s really important for the project that the line is “central”.

The data in this code example is just toy data, the real data would have a more complex shape. To check the idea I'm fitting a plane, but in the end I'd like a more complex surface. The code results in this figure.
import numpy as np
import torch
import matplotlib.pyplot as plt
from torch import nn
from torch.functional import F

ix = np.random.uniform(0,2, 1000)
iy = np.random.uniform(0,2, 1000)
iz = np.random.uniform(0,100, 1000)

x = torch.tensor(np.array([  ix, iy ]).T).float()
y = torch.tensor(iz).float()
degree =1

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        if degree == 1 :
            weights = torch.distributions.Uniform(-1, 1).sample((4,))
        if degree == 3 :
            weights = torch.distributions.Uniform(-1, 1).sample((8,))

        self.weights = nn.Parameter(weights)

    def forward(self, X):
        if degree ==1 :
            a_1, a_2, a_3, a_4 = self.weights
            return (a_1 *X[:,0] + a_2*X[:,1] +a_3)
        if degree == 3 :
            a_1, a_2, a_3, a_4, a_5, a_6, a_7, a_8 = self.weights
            return (a_1 * X[:,0]**3 + a_2*X[:,1]**3  + a_3*X[:,0]**2 + a_4 *X[:,1] **2  + a_5 *X[:,0] + a_6 *X[:,1] + a_7)

def training_loop(model, optimizer):
    losses = []
    loss = 10000
    it = 0
    if degree == 3 :
        lim = 0.1
    if degree == 1 :
        lim = 0.1
    while loss > lim:
        it +=1
        if it > 5000:
            break
        preds1= model(x).float()
        l1 = torch.nn.L1Loss()
        loss = l1(preds1, y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        losses.append(loss.detach().numpy())
        print(loss)
    return losses

m = Model()
if degree == 1 :
    opt = torch.optim.Adam(m.parameters(), lr=0.01)
    losses = np.array(training_loop(m, opt))
if degree == 3 :
    opt= torch.optim.Adam(m.parameters(), lr=0.001)
    losses = np.array(training_loop(m, opt))

params=list(m.parameters())[0].detach().numpy()

X = np.arange(0, 2, 0.1)
Y = np.arange(0, 2, 0.1)
X, Y = np.meshgrid(X, Y)

if degree == 1 :
    Z = (params[0] * X + params[1]*Y + params[2])
if degree == 3:
    Z = (params[0] * X**3 + params[1]*Y**3 + params[2]*X**2 + params[3]*Y**2 + params[4]*X + params[5]*Y + params[6])

fig = plt.figure()
ax = plt.axes(projection='3d')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
surf = ax.plot_surface(X, Y, Z, color='tab:orange', alpha = 0.5,linewidth=0, antialiased=False)
ax.scatter3D(ix,iy,iz, alpha = 0.3, s=2)
plt.show()

Edit : I tried the approach in this post : Fit Curve-Spline to 3D Point Cloud
However this forces me to specify a source and target for the shortest path. Simply choosing the centers of the lowest and highest slice results in this :

Output: You can use Delaunay/Voronoi methods to get an approximation of the medial axis of the point cloud and pass a spline curve through it. See my previous answer here, which does exactly that for points sampled on a cylindrical surface. The figure below was taken from that answer. If your point cloud also has points from inside the bounding surface (and not just samples from the outer surface), you can compute the 3D alpha-shape using the code from this answer and then just take the points on the outer surface and approximate the medial axis as I describe in the answer (or use a different method to extract the medial curve from the triangulated boundary surface).


Instruction: Exponential moving covariance matrix

Input:
I have time series data of a specific dimensionality (e.g. [T, 32]). I filter the data in an online fashion using an exponential moving average and variance (according to Wikipedia and this paper):
mean_n = alpha * mean_{n-1} + (1-alpha) * sample
var_n = alpha * (var_{n-1} + (1-alpha) * (sample - mean_{n-1}) * (sample - mean_{n-1}))

I wanted to replace the moving variance with a moving covariance matrix in order to capture the correlation between the data dimensions (e.g. 32). So I have simply replaced the element-wise variance calculation with an outer product:
covmat_n = alpha * (covmat_{n-1} + (1-alpha) * np.outer((sample - mean_{n-1}), (sample - mean_{n-1})))

However this does not seem to give correct results. For example, when I try to initialize a pytorch multivariate gaussian distribution with such a covariance matrix, it sometimes throws an error saying that the covariance matrix contains invalid values. Since it is always symmetric I assume it breaks the positive-definiteness constraint. Other downstream tasks suggested  the same (e.g. computing the KL-Divergence between two gaussians with such covariance matrices sometimes gave negative results).
Does anyone know what I am doing wrong here? Where is the error in my calculations?
And as a bonus question: Are my calculations for the simple moving variance correct? It seeems strange to multiply the new sample variance with alpha again, but the sources suggest that it is the correct way.

Output:
I have found the answer myself. It seemed to be a numerical problem. Since the eigenvalues of a positive definite matrix must be positive, I could solve it by applying an eigenvalue decomposition to every sample's covariance matrix and ensure that its eigenvalues are larger than zero:
diff = sample - last_mean
sample_covmat = np.outer(diff, diff)
w, v = np.linalg.eigh(sample_covmat)
w += 1e-3 # Note: Could also achieve this by e.g. w = np.maximum(w, 0)
sample_covmat = v @ np.diag(w) @ np.linalg.inv(v)


Instruction: Pytorch nn.parallel.DistributedDataParallel model load

Input:
Model saved with:
torch.distributed.init_process_group(backend="nccl")
local_rank = torch.distributed.get_rank()
torch.cuda.set_device(local_rank)
device = torch.device("cuda", local_rank)
save_model = f'./model'
Path(save_model).mkdir(parents=True, exist_ok=True)
net = Net(args)  # .to(device)
model_name = f"{save_model}/net.pt"
torch.save(net.state_dict(), model_name)   #
model = Model(net, args).to(device)
model_name = f"{save_model}/model.pt"

if torch.cuda.device_count() > 1:
    model = nn.parallel.DistributedDataParallel(model, device_ids=[local_rank],
                                                output_device=local_rank)
model.module.fit(tr_data, val_data, args)
torch.save(maml, model_name)

I tried to load the model with:
save_model = f'./model'
net = Net(args)  # .to(device)
model_name = f"{save_model}/net.pt"
net.load_state_dict(
    torch.load(model_name, map_location=torch.torch.device("cpu")))
maml = Model(net, args).to(device)
model_name = f"{save_model}/model.pt"
maml = torch.load(model_name, map_location=torch.torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"))  # .load_state_dict

The "net" can be loaded successfully, but I got the error when loading "model":
File "D:\Research\Traffic Prediction\maml\venv\lib\site-packages\torch\serialization.py", line 607, in load return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
File "D:\Research\Traffic Prediction\maml\venv\lib\site-packages\torch\serialization.py", line 882, in _load result = unpickler.load()
TypeError: () missing 1 required positional argument: 'ddp_join_throw_on_early_termination'
Any input would be really appreciated.

Output:
In general, with PyTorch’s DistributedDataParallel, same model is kept across all nodes (as it’s ‘synchronised’ during backpropagation).
Best way to save it is to just save the model instead of the whole DistributedDataParallel (usually on main node or multiple if possible node failure is a concern):
# or not only local_rank 0
if local_rank == 0:
    torch.save(model.module.cpu(), path)

Please notice, if your model is wrapped within DistributedDataParallel the model you are after is kept within module attribute.
Another thing - cast your model to CPU, no mapping will be necessary in such case (as you might use multiple GPUs and you would have to map it appropriately on other devices, which might not have GPU).


Instruction: How to remove the model of transformers in GPU memory

Input:
from transformers import CTRLTokenizer, TFCTRLLMHeadModel
tokenizer_ctrl = CTRLTokenizer.from_pretrained('ctrl', cache_dir='./cache', local_files_only=True)
model_ctrl = TFCTRLLMHeadModel.from_pretrained('ctrl', cache_dir='./cache', local_files_only=True)
print(tokenizer_ctrl)
gen_nlp  = pipeline("text-generation", model=model_ctrl, tokenizer=tokenizer_ctrl, device=1, return_full_text=False)

Hello, my codes can load the transformer model, for example, CTRL here, into the gpu memory.
How to remove it from GPU after usage, to free more gpu memory?
show I use torch.cuda.empty_cache() ?
Thanks.

Output:
You can simply del tokenizer_ctrl and then use torch.cuda.empty_cache().
See this thread from pytorch forum discussing it.


Instruction: The output of my neural network is negative even if i am using ReLU on every layer

Input:
I am a beginner at deep learning.I am using PyTorch to implement a neural network to train some chemical data. the input range between (0 to 1 )  with no negative values and I am using the ReLu activation function on every layer so I didn't expect to see a negative value in the output
my input size: 9 features
the output size: 7 Features
number of layers: 5
I can predict 6 of the 7 features correctly without problem only I found one of them has always negative values which I don't know why !. As far as i know, ReLU can't generate negative values
This is my neural network model:
class NNModel(nn.Module):

    def __init__(self, in_size, hidden_size, out_size):
        super().__init__()
        # hidden layers
        self.linear1 = nn.Linear(in_size, hidden_size)
        self.relu1 = nn.ReLU()

        self.linear2 = nn.Linear(hidden_size, hidden_size)
        self.relu2 = nn.ReLU()

        self.linear3 = nn.Linear(hidden_size, hidden_size)
        self.relu3 = nn.ReLU()

        self.linear4 = nn.Linear(hidden_size, hidden_size)
        self.relu4 = nn.ReLU()
        # output layer
        self.linear5 = nn.Linear(hidden_size, out_size)

    def forward(self, xb):

        # Get intermediate outputs using hidden layer
        out = self.linear1(xb)
        # Apply activation function
        out = self.relu1(out)

        out=self.linear2(out)
        out = self.relu2(out)

        out=self.linear3(out)
        out = self.relu3(out)

        out=self.linear4(out)
        out = self.relu4(out)
        # Get predictions using output layer
        out = self.linear5(out)
        return out

    def training_step(self, batch):
        inputs, targets = batch
        out = self(inputs)                  # Generate predictions
        L=nn.MSELoss()
        loss = L(out, targets) # Calculate loss

        return loss

    def validation_step(self, batch):
        inputs, targets = batch
        out = self(inputs)                    # Generate predictions
        L=nn.MSELoss()
        loss = L(out, targets)   # Calculate loss
        return {'val_loss': loss.detach() }

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses
        return {'val_loss': epoch_loss.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], val_loss: {}".format(epoch+1, result['val_loss']))

Any help, tips, or advice will be appreciated

Output:
You are using a linear layer after relu, you can rewrite your last layer as:
out = self.relu(self.linear5(out))

and your model definition from:
self.relu1 = nn.ReLU()
self.relu2 = nn.ReLU()
self.relu3 = nn.ReLU()

to a single definition:
self.relu = nn.ReLU()

and reuse this self.relu, as it is just a function without any learnable parameters.
Based on you features, i would recommend some things:
use pyramidal like neurons by layer, eg:
layer 1: 512
layer 2: 256
layer 3: 125
last layer activation function:
use a linear then a sigmoid (don't use relu in the mid,otherwise you will clip you outputs in the range 0.5 to 1).
Normalize your output data, and store mean and variance for reversing the transformation after that. Also, take a closer look, it is very likely that the values you are failing to predict are in a different range/distribution compared to the ones you are doing ok.
Try adam optimizer with 0.001 learning rate.
Try different batch sizes, ranging from 1,2,8,16,32,64,128.
Try add dropout.
After all, you may need more data.


Instruction: How to downgrade from CUDA 11.4 to 10.2 & add sm_35 - CUDA error: no kernel image is available for execution on the device

Input:
I'm trying to run a piece of code on Pytorch, but I get the error:
RuntimeError: CUDA error: no kernel image is available for execution on the device

I've narrowed down the issue to be a missmatch of CUDA versions. My machine has 2 GPUs:
a GeForce GTX 650 (compute capability 3.0) and a Tesla K40c (compute capability 3.5). I've checked the compute capabilities here: https://developer.nvidia.com/cuda-gpus.
My nvidia-smi command gives the following:
nvidia-smi output (Driver Version: 470.57.02 & CUDA Version: 11.4)
While my nvcc -V command gives the following:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

The 10.1 version exists because I tried to install that CUDA version, specifically, following the instructions elsewhere (for example: https://medium.com/@anarmammadli/how-to-install-cuda-10-2-cudnn-7-6-5-and-samples-on-ubuntu-18-04-2493124478ca)
Also, I've installed cudatoolkit with conda, and so on my conda list I have the following entry:
...
cudatoolkit               10.1.243             h6bb024c_0
...

In accordance with https://github.com/moi90/pytorch_compute_capabilities/blob/main/table.md I've also installed the 1.8.0 PyTorch version.
However, in Python 3.7.11:
Python 3.7.11 (default, Jul 27 2021, 14:32:16)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1'
>>> torch.version.cuda
'10.1'
>>> torch.cuda.get_arch_list()
['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'compute_37']
>>> torch.cuda.is_available()
True

I don't have sm_35 available, which I need to use the Tesla K40. I believe this is the reason why I keep getting the CUDA error: no kernel image is available for execution on the device error. I've also tried all the above for version 10.2 of CUDA, same result.

Output:
I've solved my problem. As stated in the comments, I required a version of PyTorch that supports sm_35 compute capability. It had little to do with the current CUDA version. In the end, I found these binaries:
https://blog.nelsonliu.me/2020/10/13/newer-pytorch-binaries-for-older-gpus/
I finally fixed the issue by creating a new environment and running:
pip install torch==1.3.1+cu92 -f https://nelsonliu.me/files/pytorch/whl/torch_stable.html


Instruction: Correct use of Cross-entropy as a loss function for sequence of elements

Input:
I have a sequece labeling task.
So as input, I have a sequence of elements with shape [batch_size, sequence_length] and where each element of this sequence should be assigned with some class.
And as a loss function during training a neural net, I use a Cross-entropy.
How should I correctly use it?
My variable target_predictions has shape [batch_size, sequence_length, number_of_classes] and target has shape [batch_size, sequence_length].
Documentation says:

I know if I use CrossEntropyLoss(target_predictions.permute(0, 2, 1), target), everything will work fine. But I have concerns that torch is intepreting my sequence_length as d_1 variable as on screenshot and will think that it is a multidimential loss, which is not the case.
How should I correctly do it?

Output:
Using CE Loss will give you loss instead of labels. By default mean will be taken which is what you are probably after and the snippet with permute will be fine (using this loss you can train your nn via backward).
To get predicted class just take argmax across appropriate dimension, in the case without permutation it would be:
labels = torch.argmax(target_predictions, dim=-1)

This will give you (batch, sequence_length) output containing classes.


Instruction: Understanding backpropagation in PyTorch

Input:
I am exploring PyTorch, and I do not understand the output of the following  example:
# Initialize x, y and z to values 4, -3 and 5
x = torch.tensor(4., requires_grad = True)
y = torch.tensor(-3., requires_grad = True)
z = torch.tensor(5., requires_grad = True)

# Set q to sum of x and y, set f to product of q with z
q = x + y
f = q * z

# Compute the derivatives
f.backward()

# Print the gradients
print("Gradient of x is: " + str(x.grad))
print("Gradient of y is: " + str(y.grad))
print("Gradient of z is: " + str(z.grad))

Output
Gradient of x is: tensor(5.)
Gradient of y is: tensor(5.)
Gradient of z is: tensor(1.)

I have little doubt that my confusion originates with a minor misunderstanding. Can someone explain in a stepwise manner?

Output:
I hope you understand that When you do f.backward(), what you get in x.grad is .
In your case
.
So, simply (with preliminary calculus)

If you put your values for x, y and z, that explains the outputs.
But, this isn't really "Backpropagation" algorithm. This is just partial derivatives (which is all you asked in the question).
Edit:
If you want to know about the Backpropagation machinery behind it, please see @Ivan's answer.


Instruction: How to get outputs in the same order as inputs with multiple spawned processes running on multiple GPUs and batches of data processed by each?

Input:
I am using Pytorch Distributed Data Parallel approach and spawning multiple processes, each running on separate GPU.I am using Pytorch Distributed Data Sampler along with Data Loader for loading batches of input data to each process.
My questions:

Under the hood, how does Pytorch Distributed Data Sampler, Data Loader make slices of input data? Just for simplicity say we have 4 GPUs, and 400 input samples and batch size of say 50, then will Pytorch Distributed Data Sampler (together with Data Loader) make first 50 samples go to GPU-0, next 50 to GPU-1., next 50 to GPU-2, then GPU-3 and then again next 50 to GPU-0 i.e. in the order of GPU device number? or the order of GPU to select for next batch of input is random based on which GPU has finished its previous batch first? or is it like 400 samples get divided into 4 parts first and then GPU-0 would get first 100 samples of input data (50 at a time ),  GPU-1 will get next 100 samples ( 50 at a time) and so on..and in this case no matter if say GPU-3 gets its second batch started earlier than GPU-0, but still with respect to input data, GPU-0 would still have first 100 samples and GPU-3 would have last 100?

2). My Second question is how to retrieve output data in same order as input data so that final consolidated output ( having outputs from all processes combined in one data structure) is in same order as original inputs and each output corresponds to the right input

Output:
The PyTorch documentation on DistributedSampler doesn't provide any guarantees regarding how data is distributed across processes and devices, other than the fact that it is, in fact, distributed across processes and devices. You shouldn't design your application to be dependent on an implementation detail of an external package; otherwise, your application could suddenly fail one day after updating PyTorch, and you'd have no idea why (or potentially that it's even failing to begin with). If, for some reason, you absolutely need the data to be distributed in a very specific way, you should roll your own solution. The documentation for DistributedDataParallel suggests that, if you're using a single host with N GPUs, you should spin up N processes, each designated a single GPU. A simple solution would be to set the process's rank equal to the designated GPU device ID; this could in-turn be used in a custom sampler class to select the appropriate sample indices.
You could try to control the order in which outputs are returned by the various distributed processes, but this introduces unnecessary synchronization which would defeat much of the purpose of parallelization. A better solution is to simply return outputs in an arbitrary order, and then sort them after-the-fact. If you'd like the outputs to be sorted in the same order as the inputs, you can just associate each input with, say, an integer index (input 0 gets index 0, input 1 gets index 1, and so on). When returning the output, also return the index of the associated input (e.g. as a tuple). Afterwards, you can just sort the outputs by their corresponding indices.


Instruction: Is there any option at Pytorch autograd function for these problem?

Input:
Sorry for the vague title because I don't know exactly how to ask the question.
I'm using pytorch's autograd function right now and I'm struggling with the results I don't understand.
In common sense, the grad calculated by the loss is how far each parameter should go in the direction where the loss is minimized. Because it doesn't make sense for the value to change just because the scale has changed.
It means  $$ grad(loss) = 5grad(loss \frac 1 5) $$
But my actual result doesn;t. So
formulation explanation
And this is my actual code:)
from torch.autograd import grad
train_loss = loss(models(adaptation_data), adaptation_labels)
grads = grad(train_loss , models.parameters(),create_graph=True)
grads_02 = grad(train_loss*0.2 , models.parameters(),create_graph=True)
grads[-1] == grads_02[-1] * 5
#result : False

the whole code
Maybe I'm doing something wrong or there is an option for this for the grad function, but can anyone tell me? please

Output:
Your code screenshot shows that the two tensors are different due to floating point truncation error. Do not compare them with == sign, use isclose() function instead
torch.isclose(grads[-1], grads_02[-1] * 5)


Instruction: what is model.training in pytorch?

Input:
hi i'm going through pytorch tutorial about transfer learning.
(https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html)
what is model.training for??
enter def visualize_model(model,num_images=6):
was_training=model.training
model.eval()
images_so_far=0
fig=plt.figure()

with torch.no_grad():
    for i, (inputs,labels) in enumerate(dataloaders['val']):
        inputs=inputs.to(device)
        labels=labels.to(device)

        outputs=model(inputs)
        _,pred=torch.max(outputs,1)

        for j in range(inputs.size()[0]):
            images_so_far+=1
            ax=plt.subplot(num_images//2,2,images_so_far)
            ax.axis('off')
            ax.set_title('predicted: {}'.format(class_names[preds[j]]))
            imshow(inputs.cpu().data[j])

            if images_so_far==num_images:
                model.train(mode=was_training)
                return
    model.train(mode=was_training)code here

i cannot understand "model.train(model=was_training)". any help?? thank you so much

Output:
I think this will help (link)
All nn.Modules have an internal training attribute, which is changed by calling model.train() and model.eval() to switch the behavior of the model.
The was_training variable stores the current training state of the model, calls model.eval(), and resets the state at the end using model.train(training=was_training).
You can find great answers in pytorch discuss forum ;)


Instruction: I cannot draw bounding box correctly

Input:
I can't draw the bounding box correctly.
img = Image.open("/content/drive/MyDrive/58125_000893_Sideline_frame298.jpg").convert('RGB')
convert_tensor = torchvision.transforms.ToTensor()

img=convert_tensor(img)
width=15
top=456
height=16
left=1099
img=img*255
boxs=[left,top,top+width,left+height]
print(boxs)
boxs=torch.tensor(boxs,dtype=torch.int)
a=torch.tensor(img,dtype=torch.uint8)
a=torchvision.utils.draw_bounding_boxes(image=a,boxes=boxs.unsqueeze(0),width=2,colors=(0,0,255))
a=a.permute(1,2,0)
plt.imshow(a)
plt.show()

I need to draw a bounding box to the player's helmet. But bounding was drawn in a different location. I get bounding attributes from CSV files. Can someone help me fix this issue?

Output:
You are using a top-left coordinate system while PyTorch uses x: horizontal left->right and y: vertical bottom->top. The bonding box provided to torchvision.utils.draw_bounding_boxes is defined as (xmin, ymin, xmax, ymax).
Your mapping should therefore be:
xmin = left
ymin = top + height
xmax = left + width
ymax = top