Instruction: FastGAN - RuntimeError: Error(s) in loading state_dict for Generator

Input:
I'm running FastGAN (https://github.com/odegeasslbc/FastGAN-pytorch) on Google Colab and now trying to resume training from a saved .pth generated by the network. However, it keeps throwing this error:

Traceback (most recent call last):
  File "train.py", line 202, in
    train(args)
  File "train.py", line 117, in train
    netG.load_state_dict(ckpt['g'])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1407, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Generator:
    Missing key(s) in state_dict: "init.init.0.weight_orig", "init.init.0.weight", "init.init.0.weight_u", "init.init.0.weight_orig", "init.init.0.weight_u", "init.init.0.weight_v", "init.init.1.weight", "init.init.1.bias", "init.init.1.running_mean", "init.init.1.running_var", "feat_8.1.weight_orig", "feat_8.1.weight", "feat_8.1.weight_u", "feat_8.1.weight_orig", "feat_8.1.weight_u", "feat_8.1.weight_v", "feat_8.2.weight", "feat_8.3.weight", "feat_8.3.bias", "feat_8.3.running_mean", "feat_8.3.running_var", "feat_8.5.weight_orig", "feat_8.5.weight", "feat_8.5.weight_u", "feat_8.5.weight_orig", "feat_8.5.weight_u", "feat_8.5.weight_v", "feat_8.6.weight", "feat_8.7.weight", "feat_8.7.bias", "feat_8.7.running_mean", "feat_8.7.running_var", "feat_16.1.weight_orig", "feat_16.1.weight", "feat_16.1.weight_u", "feat_16.1.weight_orig", "feat_16.1.weight_u", "feat_16.1.weight_v", "feat_16.2.weight", "feat_16.2.bias", "feat_16.2.running_mean", "feat_16.2.running_var", "feat_32.1.weight_orig", "feat_32.1.weight", "feat_32.1.weight_u", "feat_32.1.weight_orig", "feat_32.1.weight_u", "feat_32.1.weight_v", "feat_32.2.weight", "feat_32.3.weight", "feat_32.3.bias", "feat_32.3.running_mean", "feat_32.3.running_var", "feat_32.5.weight_orig", "feat_32.5.weight", "feat_32.5.weight_u", "feat_32.5.weight_orig", "feat_32.5.weight_u", "feat_32.5.weight_v", "feat_32.6.weight", "feat_32.7.weight", "feat_32.7.bias", "feat_32.7.running_mean", "feat_32.7.running_var", "feat_64.1.weight_orig", "feat_64.1.weight", "feat_64.1.weight_u", "feat_64.1.weight_orig", "feat_64.1.weight_u", "feat_64.1.weight_v", "feat_64.2.weight", "feat_64.2.bias", "feat_64.2.running_mean", "feat_64.2.running_var", "feat_128.1.weight_orig", "feat_128.1.weight", "feat_128.1.weight_u", "feat_128.1.weight_orig", "feat_128.1.weight_u", "feat_128.1.weight_v", "feat_128.2.weight", "feat_128.3.weight", "feat_128.3.bias", "feat_128.3.running_mean", "feat_128.3.running_var", "feat_128.5.weight_orig", "feat_128.5.weight", "feat_128.5.weight_u", "feat_128.5.weight_orig", "feat_128.5.weight_u", "feat_128.5.weight_v", "feat_128.6.weight", "feat_128.7.weight", "feat_128.7.bias", "feat_128.7.running_mean", "feat_128.7.running_var", "feat_256.1.weight_orig", "feat_256.1.weight", "feat_256.1.weight_u", "feat_256.1.weight_orig", "feat_256.1.weight_u", "feat_256.1.weight_v", "feat_256.2.weight", "feat_256.2.bias", "feat_256.2.running_mean", "feat_256.2.running_var", "se_64.main.1.weight_orig", "se_64.main.1.weight", "se_64.main.1.weight_u", "se_64.main.1.weight_orig", "se_64.main.1.weight_u", "se_64.main.1.weight_v", "se_64.main.3.weight_orig", "se_64.main.3.weight", "se_64.main.3.weight_u", "se_64.main.3.weight_orig", "se_64.main.3.weight_u", "se_64.main.3.weight_v", "se_128.main.1.weight_orig", "se_128.main.1.weight", "se_128.main.1.weight_u", "se_128.main.1.weight_orig", "se_128.main.1.weight_u", "se_128.main.1.weight_v", "se_128.main.3.weight_orig", "se_128.main.3.weight", "se_128.main.3.weight_u", "se_128.main.3.weight_orig", "se_128.main.3.weight_u", "se_128.main.3.weight_v", "se_256.main.1.weight_orig", "se_256.main.1.weight", "se_256.main.1.weight_u", "se_256.main.1.weight_orig", "se_256.main.1.weight_u", "se_256.main.1.weight_v", "se_256.main.3.weight_orig", "se_256.main.3.weight", "se_256.main.3.weight_u", "se_256.main.3.weight_orig", "se_256.main.3.weight_u", "se_256.main.3.weight_v", "to_128.weight_orig", "to_128.weight", "to_128.weight_u", "to_128.weight_orig", "to_128.weight_u", "to_128.weight_v", "to_big.weight_orig", "to_big.weight", "to_big.weight_u", "to_big.weight_orig", "to_big.weight_u", "to_big.weight_v", "feat_512.1.weight_orig", "feat_512.1.weight", "feat_512.1.weight_u", "feat_512.1.weight_orig", "feat_512.1.weight_u", "feat_512.1.weight_v", "feat_512.2.weight", "feat_512.3.weight", "feat_512.3.bias", "feat_512.3.running_mean", "feat_512.3.running_var", "feat_512.5.weight_orig", "feat_512.5.weight", "feat_512.5.weight_u", "feat_512.5.weight_orig", "feat_512.5.weight_u", "feat_512.5.weight_v", "feat_512.6.weight", "feat_512.7.weight", "feat_512.7.bias", "feat_512.7.running_mean", "feat_512.7.running_var", "se_512.main.1.weight_orig", "se_512.main.1.weight", "se_512.main.1.weight_u", "se_512.main.1.weight_orig", "se_512.main.1.weight_u", "se_512.main.1.weight_v", "se_512.main.3.weight_orig", "se_512.main.3.weight", "se_512.main.3.weight_u", "se_512.main.3.weight_orig", "se_512.main.3.weight_u", "se_512.main.3.weight_v", "feat_1024.1.weight_orig", "feat_1024.1.weight", "feat_1024.1.weight_u", "feat_1024.1.weight_orig", "feat_1024.1.weight_u", "feat_1024.1.weight_v", "feat_1024.2.weight", "feat_1024.2.bias", "feat_1024.2.running_mean", "feat_1024.2.running_var".
    Unexpected key(s) in state_dict: "module.init.init.0.weight_orig", "module.init.init.0.weight_u", "module.init.init.0.weight_v", "module.init.init.1.weight", "module.init.init.1.bias", "module.init.init.1.running_mean", "module.init.init.1.running_var", "module.init.init.1.num_batches_tracked", "module.feat_8.1.weight_orig", "module.feat_8.1.weight_u", "module.feat_8.1.weight_v", "module.feat_8.2.weight", "module.feat_8.3.weight", "module.feat_8.3.bias", "module.feat_8.3.running_mean", "module.feat_8.3.running_var", "module.feat_8.3.num_batches_tracked", "module.feat_8.5.weight_orig", "module.feat_8.5.weight_u", "module.feat_8.5.weight_v", "module.feat_8.6.weight", "module.feat_8.7.weight", "module.feat_8.7.bias", "module.feat_8.7.running_mean", "module.feat_8.7.running_var", "module.feat_8.7.num_batches_tracked", "module.feat_16.1.weight_orig", "module.feat_16.1.weight_u", "module.feat_16.1.weight_v", "module.feat_16.2.weight", "module.feat_16.2.bias", "module.feat_16.2.running_mean", "module.feat_16.2.running_var", "module.feat_16.2.num_batches_tracked", "module.feat_32.1.weight_orig", "module.feat_32.1.weight_u", "module.feat_32.1.weight_v", "module.feat_32.2.weight", "module.feat_32.3.weight", "module.feat_32.3.bias", "module.feat_32.3.running_mean", "module.feat_32.3.running_var", "module.feat_32.3.num_batches_tracked", "module.feat_32.5.weight_orig", "module.feat_32.5.weight_u", "module.feat_32.5.weight_v", "module.feat_32.6.weight", "module.feat_32.7.weight", "module.feat_32.7.bias", "module.feat_32.7.running_mean", "module.feat_32.7.running_var", "module.feat_32.7.num_batches_tracked", "module.feat_64.1.weight_orig", "module.feat_64.1.weight_u", "module.feat_64.1.weight_v", "module.feat_64.2.weight", "module.feat_64.2.bias", "module.feat_64.2.running_mean", "module.feat_64.2.running_var", "module.feat_64.2.num_batches_tracked", "module.feat_128.1.weight_orig", "module.feat_128.1.weight_u", "module.feat_128.1.weight_v", "module.feat_128.2.weight", "module.feat_128.3.weight", "module.feat_128.3.bias", "module.feat_128.3.running_mean", "module.feat_128.3.running_var", "module.feat_128.3.num_batches_tracked", "module.feat_128.5.weight_orig", "module.feat_128.5.weight_u", "module.feat_128.5.weight_v", "module.feat_128.6.weight", "module.feat_128.7.weight", "module.feat_128.7.bias", "module.feat_128.7.running_mean", "module.feat_128.7.running_var", "module.feat_128.7.num_batches_tracked", "module.feat_256.1.weight_orig", "module.feat_256.1.weight_u", "module.feat_256.1.weight_v", "module.feat_256.2.weight", "module.feat_256.2.bias", "module.feat_256.2.running_mean", "module.feat_256.2.running_var", "module.feat_256.2.num_batches_tracked", "module.se_64.main.1.weight_orig", "module.se_64.main.1.weight_u", "module.se_64.main.1.weight_v", "module.se_64.main.3.weight_orig", "module.se_64.main.3.weight_u", "module.se_64.main.3.weight_v", "module.se_128.main.1.weight_orig", "module.se_128.main.1.weight_u", "module.se_128.main.1.weight_v", "module.se_128.main.3.weight_orig", "module.se_128.main.3.weight_u", "module.se_128.main.3.weight_v", "module.se_256.main.1.weight_orig", "module.se_256.main.1.weight_u", "module.se_256.main.1.weight_v", "module.se_256.main.3.weight_orig", "module.se_256.main.3.weight_u", "module.se_256.main.3.weight_v", "module.to_128.weight_orig", "module.to_128.weight_u", "module.to_128.weight_v", "module.to_big.weight_orig", "module.to_big.weight_u", "module.to_big.weight_v", "module.feat_512.1.weight_orig", "module.feat_512.1.weight_u", "module.feat_512.1.weight_v", "module.feat_512.2.weight", "module.feat_512.3.weight", "module.feat_512.3.bias", "module.feat_512.3.running_mean", "module.feat_512.3.running_var", "module.feat_512.3.num_batches_tracked", "module.feat_512.5.weight_orig", "module.feat_512.5.weight_u", "module.feat_512.5.weight_v", "module.feat_512.6.weight", "module.feat_512.7.weight", "module.feat_512.7.bias", "module.feat_512.7.running_mean", "module.feat_512.7.running_var", "module.feat_512.7.num_batches_tracked", "module.se_512.main.1.weight_orig", "module.se_512.main.1.weight_u", "module.se_512.main.1.weight_v", "module.se_512.main.3.weight_orig", "module.se_512.main.3.weight_u", "module.se_512.main.3.weight_v", "module.feat_1024.1.weight_orig", "module.feat_1024.1.weight_u", "module.feat_1024.1.weight_v", "module.feat_1024.2.weight", "module.feat_1024.2.bias", "module.feat_1024.2.running_mean", "module.feat_1024.2.running_var", "module.feat_1024.2.num_batches_tracked".

Any idea what might be happening here?
Thanks so much for any help!

Output:
This is common when changing the attribute name of the submodules in your nn.Module.
Notice how most of your layer keys here differ from the ones contained in the loaded state dict because of their prefix: all keys in the dictionary have a 'module.' prefix.
A quick fix you be to slice away this prefix. You could for instance usea dict comprehension:
loaded_state = {k.replace('module.', ''): v for k, v in ckpt['g'].items()}
netG.load_state_dict(loaded_state)


Instruction: How can i implement the __getitem__ method in PyTorch for the sinus

Input:
I am starting with PyTorch and i am trying to create a Network that is predicting the sinus of x. I tried to create a DataSet like this:
 class SinusDataset(Dataset):

    def __init__(self, size: int = 1000):
        self.size = size

    def __len__(self):
        return self.size

    def __getitem__(self, idx: int)->Tensor:
        if idx > self.size:
            raise ValueError

        return idx, math.sin(idx)

I do not think that is the proper way to implement this. How should I Implemented the `__get__´ method?

Output:
You could initialize your input and labels on init and save those in lists. Then, in your __getitem__ function, pick instances from those two using the provided idx integer. Something like:
class SinusDataset(Dataset):
    def __init__(self, size: int = 1000):
        self.x = torch.linspace(0, 1, size)
        self.y = torch.sin(self.x)

    def __len__(self) -> int:
        return len(self.x)

    def __getitem__(self, idx: int):
        return return self.x[idx][None], self.y[idx][None]

Then you can use the dataset by wrapping a torch.utils.data.DataLoader:
>>> dl = DataLoader(SinusDataset(100), batch_size=4, shuffle=True)

>>> for x, y in dl:
...     print(x, y)
...     break
tensor([0.2452, 0.6116, 0.0791, 0.6667]) tensor([0.2428, 0.5742, 0.0790, 0.6184])

In this case it would be more appropriate to inherit from torch.utils.data.TensorDataset directly. This comes with both __len__ and __getitem__ implemented for you (see source):
class SinusDataset(TensorDataset):
    def __init__(self, size: int = 1000):
        x = torch.linspace(0, 1, 1000)[:,None]
        y = torch.sin(x)[:,None]
        super().__init__(x, y)

This is slightly more advanced but it is considered best practice to inherit from the closest built-in torch.utils.data.Dataset class instead of writing the same methods yourself.

Inference example:
>>> model = nn.Sequential(nn.Linear(1, 4),
                          nn.ReLU(),
                          nn.Linear(4, 1))

>>> x, y = next(iter(dl))
>>> model(x)
tensor([[-0.0640],
        [ 0.1461],
        [-0.0882],
        [ 0.2259]], grad_fn=<AddmmBackward>)


Instruction: unable to load pytorch model for evaluation

Input:
I have a .pth model saved and I am trying to load to do inference using the following code
model = GatherModel()
model.load_state_dict(torch.load('/content/CIGIN/weights/cigin.tar'))

and I am getting this error shown below. why I am getting this.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-3bff0e426886> in <module>()
----> 1 model.load_state_dict(torch.load('/content/CIGIN/weights/cigin.tar'))

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1405         if len(error_msgs) > 0:
   1406             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1407                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1408         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1409

RuntimeError: Error(s) in loading state_dict for GatherModel:
    Missing key(s) in state_dict: "lin0.weight", "lin0.bias", "set2set.lstm.weight_ih_l0", "set2set.lstm.weight_hh_l0", "set2set.lstm.bias_ih_l0", "set2set.lstm.bias_hh_l0", "message_layer.weight", "message_layer.bias", "conv.bias", "conv.edge_func.0.weight", "conv.edge_func.0.bias", "conv.edge_func.2.weight", "conv.edge_func.2.bias".
    Unexpected key(s) in state_dict: "solute_pass.U_0.weight", "solute_pass.U_0.bias", "solute_pass.U_1.weight", "solute_pass.U_1.bias", "solute_pass.U_2.weight", "solute_pass.U_2.bias", "solute_pass.M_0.weight", "solute_pass.M_0.bias", "solute_pass.M_1.weight", "solute_pass.M_1.bias", "solute_pass.M_2.weight", "solute_pass.M_2.bias", "solvent_pass.U_0.weight", "solvent_pass.U_0.bias", "solvent_pass.U_1.weight", "solvent_pass.U_1.bias", "solvent_pass.U_2.weight", "solvent_pass.U_2.bias", "solvent_pass.M_0.weight", "solvent_pass.M_0.bias", "solvent_pass.M_1.weight", "solvent_pass.M_1.bias", "solvent_pass.M_2.weight", "solvent_pass.M_2.bias", "lstm_solute.weight_ih_l0", "lstm_solute.weight_hh_l0", "lstm_solute.bias_ih_l0", "lstm_solute.bias_hh_l0", "lstm_solvent.weight_ih_l0", "lstm_solvent.weight_hh_l0", "lstm_solvent.bias_ih_l0", "lstm_solvent.bias_hh_l0", "lstm_gather_solute.weight_ih_l0", "lstm_gather_solute.weight_hh_l0", "lstm_gather_solute.bias_ih_l0", "lstm_gather_solute.bias_hh_l0", "lstm_gather_solvent.weight_ih_l0", "lstm_gather_solvent.weight_hh_l0", "lstm_gather_solvent.bias_ih_l0", "lstm_gather_solvent.bias_hh_l0", "first_layer.weight", "first_layer.bias", "second_layer.weight", "second_layer.bias", "third_layer.weight", "third_layer.bias", "fourth_layer.weight", "fourth_layer.bias".

I have tried using strict=False in state_dict but I am getting this error
_IncompatibleKeys(missing_keys=['lin0.weight', 'lin0.bias', 'set2set.lstm.weight_ih_l0', 'set2set.lstm.weight_hh_l0', 'set2set.lstm.bias_ih_l0', 'set2set.lstm.bias_hh_l0', 'message_layer.weight', 'message_layer.bias', 'conv.bias', 'conv.edge_func.0.weight', 'conv.edge_func.0.bias', 'conv.edge_func.2.weight', 'conv.edge_func.2.bias'], unexpected_keys=['solute_pass.U_0.weight', 'solute_pass.U_0.bias', 'solute_pass.U_1.weight', 'solute_pass.U_1.bias', 'solute_pass.U_2.weight', 'solute_pass.U_2.bias', 'solute_pass.M_0.weight', 'solute_pass.M_0.bias', 'solute_pass.M_1.weight', 'solute_pass.M_1.bias', 'solute_pass.M_2.weight', 'solute_pass.M_2.bias', 'solvent_pass.U_0.weight', 'solvent_pass.U_0.bias', 'solvent_pass.U_1.weight', 'solvent_pass.U_1.bias', 'solvent_pass.U_2.weight', 'solvent_pass.U_2.bias', 'solvent_pass.M_0.weight', 'solvent_pass.M_0.bias', 'solvent_pass.M_1.weight', 'solvent_pass.M_1.bias', 'solvent_pass.M_2.weight', 'solvent_pass.M_2.bias', 'lstm_solute.weight_ih_l0', 'lstm_solute.weight_hh_l0', 'lstm_solute.bias_ih_l0', 'lstm_solute.bias_hh_l0', 'lstm_solvent.weight_ih_l0', 'lstm_solvent.weight_hh_l0', 'lstm_solvent.bias_ih_l0', 'lstm_solvent.bias_hh_l0', 'lstm_gather_solute.weight_ih_l0', 'lstm_gather_solute.weight_hh_l0', 'lstm_gather_solute.bias_ih_l0', 'lstm_gather_solute.bias_hh_l0', 'lstm_gather_solvent.weight_ih_l0', 'lstm_gather_solvent.weight_hh_l0', 'lstm_gather_solvent.bias_ih_l0', 'lstm_gather_solvent.bias_hh_l0', 'first_layer.weight', 'first_layer.bias', 'second_layer.weight', 'second_layer.bias', 'third_layer.weight', 'third_layer.bias', 'fourth_layer.weight', 'fourth_layer.bias'])

Output: The error basically says that there are weights defined by architecture you are using that are not in the state_dict, and also there are weights that are not defined by the architecture, but are present in the state_dict. Are you sure that whatever is defined by GatherModel() is the same architecture that created the state_dict in the first place? Because this error indicates that the answer is no.


Instruction: maxpooling what should the indices be when there're multiple max values?

Input:
What to do when there are multiple values in the kernel that equals to the max? For example, for these values:
array([[0., 0.],
       [0., 0.]])

The max is simply 0. What should max indices look like? should it be True's for all occurrence of max:
array([[ True,  True],
       [ True,  True]])

Or the first occurrence of the max:
array([[ True, False],
       [False, False]])

Pytorch uses the first occurrence of the max, while some source uses the other one (he uses pos = np.where(result == view, 1, 0), which essentially records 1 for all occurrence of max).
Edit: for forward prop it doesn't matter, but during backpropagation, for example, if the upstream gradient is [[1,1],[1,1]], then the gradient of current node will be different: [[1,1],[1,1]] (all occurrence) vs [[1,0],[0,0]] (first occurrence)

Output: After even a single back-propagation it is very unlikely that the kernel values will remain equal. So it shouldn't be a real issue, should it?


Instruction: Why are torch.version.cuda and deviceQuery reporting different versions?

Input:
I have a doubt about the CUDA version installed on my system and being effectively used by my software.
I have done some research online but could not find a solution to my doubt.
The issue which helped me a bit in my understanding and is the most related to what I will ask below is this one.
Description of the problem:
I created a virtualenvironment with virtualenvironmentwrapper and then I installed pytorch in it.
After some time I realized I did not have CUDA installed on my system.
You can find it out by  doing:
nvcc –V
If nothing is returned it means that you did not install CUDA (as far as I understood).
Therefore, I followed the instructions here
And I installed CUDA with this official link.
Then, I installed the nvidia-development-kit simply with
sudo apt install nvidia-cuda-toolkit
Now, if in my virtualenvironment I do:
nvcc -V
I get:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

However, if (always in the virtualenvironment) I do:
python -c "import torch; print(torch.version.cuda)"
I get:
10.2
This is the first thing I don't understand. Which version of CUDA am I using in my virtualenvironment?
Then, if I run the sample deviceQuery (from the cuda-samples folder - the samples can be installed by following this link) I get:
./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA GeForce RTX 2080 Super with Max-Q Design"
  CUDA Driver Version / Runtime Version          11.4 / 11.4
  CUDA Capability Major/Minor version number:    7.5
  Total amount of global memory:                 7974 MBytes (8361279488 bytes)
  (048) Multiprocessors, (064) CUDA Cores/MP:    3072 CUDA Cores
  GPU Max Clock rate:                            1080 MHz (1.08 GHz)
  Memory Clock rate:                             5501 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        65536 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1024
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 3 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Managed Memory:                Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.4, NumDevs = 1
Result = PASS

Why is it now mentioned CUDA version 11.4? Is it because I am using the NVIDIA_CUDA-11.4_Samples I guess?
Another information is the following. If I check in my /usr/local folder I see three folders related to CUDA.
If I do:
cd /usr/local && ll | grep -i CUDA
I get:
lrwxrwxrwx  1 root root   22 Oct  7 11:33 cuda -> /etc/alternatives/cuda/
lrwxrwxrwx  1 root root   25 Oct  7 11:33 cuda-11 -> /etc/alternatives/cuda-11/
drwxr-xr-x 16 root root 4096 Oct  7 11:33 cuda-11.4/

Is that normal?
Thanks for your help.

Output:
torch.version.cuda is just defined as a string. It doesn't query anything. It doesn't tell you which version of CUDA you have installed. It only tells you that the PyTorch you have installed is meant for that (10.2) version of CUDA. But the version of CUDA you are actually running on your system is 11.4.
If you installed PyTorch with, say,
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

then you should also have the necessary libraries (cudatoolkit) in your Anaconda directory, which may be different from your system-level libraries.
However, note that these depend on the NVIDIA display drivers:

Installing cudatoolkit does not install the drivers (nvidia.ko), which you need to install separately on your system.


Instruction: How can you iterate through dict items at the same time with variable dict length?

Input:
I want to iterate through a dict that has n keys and values.
eg:
d = {0:"batched data", 1:"batched data", 2:"batched data", ..., n: "batched data"}

Since the batched data is of the pytorch DataLoader form, it is not indexable only iterable.
Thus, how can I iterate through all of the dict batched data at once:
I was thinking zip may be useful, but I was unsure how to compute this when the length is variable.
For instance if n=3:
for index, data in enumerate(zip(d[0],d[1],d[2])):

would work, but how can this be generalized for any number of dict elements?

Output:
Looking at the output of your command:
for index, data in enumerate(zip(d[0],d[1],d[2])):
    print("{} -> {}".format(index, data))

it seems to me that this will handle the generic case:
d = {0:"batched data", 1:"batched data", 2:"batched data", 3: "batched data"}

for index, data in enumerate(zip(*d.values())):
    print("{} -> {}".format(index, data))

Output:
0 -> ('b', 'b', 'b', 'b')
1 -> ('a', 'a', 'a', 'a')
2 -> ('t', 't', 't', 't')
3 -> ('c', 'c', 'c', 'c')
4 -> ('h', 'h', 'h', 'h')
5 -> ('e', 'e', 'e', 'e')
6 -> ('d', 'd', 'd', 'd')
7 -> (' ', ' ', ' ', ' ')
8 -> ('d', 'd', 'd', 'd')
9 -> ('a', 'a', 'a', 'a')
10 -> ('t', 't', 't', 't')
11 -> ('a', 'a', 'a', 'a')


Instruction: Why do torch.add and torch.einsum return different results?

Input:
t=m+n
x=torch.einsum('xyzw,xyzw->xyzw',m,n)

When I try this code, I get x that's different from t, which is surprising. Why does this happen?

Output:
einsum is actually a product of its inputs. It's only a sum over the indexes that do not occur in the output.
So,
x=torch.einsum('xyzw,xyzw->xyzw',m,n)

is actually equivalent to
x = m * n


Instruction: '1D target tensor expected' error when using pytorch tensorDataset class

Input:
I am wondering why this error is occuring. My hunch tells me that the tensorDataset reads the last column as being the labels, but I don't know why it would behave that way if I input a separate dataset for labels as the second argument. Also, can someone explain exactly how one-hot encoding works and how I can fix this problem because I only want one label per item?
Error: return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported

Code:
if __name__ == '__main__':

inputs_file = pd.read_csv('dataset.csv')
targets_file = pd.read_csv('labels.csv')

inputs = inputs_file.iloc[1:1001].values
targets = targets_file.iloc[1:1001].values

inputs = torch.tensor(inputs, dtype=torch.float32)
targets = torch.tensor(targets)

dataset = TensorDataset(inputs, targets)

val_size = 200
test_size = 100
train_size = len(dataset) - (val_size + test_size)

# Divide dataset into 3 unique random subsets
training_data, validation_data, test_data = random_split(dataset, [train_size, val_size, test_size])

batch_size = 50

train_loader = DataLoader(training_data, batch_size, shuffle=True, num_workers=4, pin_memory=True)
valid_loader = DataLoader(validation_data, batch_size*2, num_workers=4, pin_memory=True)

Output:
From what I gather from the comments discussion, the error is reproduced by the following.
import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset, random_split

inputs = torch.randn(999, 11, dtype=torch.float32)
targets = torch.randint(5, (999, 1), dtype=torch.long)

# you need this to adapt from pandas, but not for this example code
# inputs = torch.tensor(inputs, dtype=torch.float32)
# targets = torch.tensor(targets)

dataset = TensorDataset(inputs, targets)

val_size = 200
test_size = 100
train_size = len(dataset) - (val_size + test_size)

# Divide dataset into 3 unique random subsets
training_data, validation_data, test_data = random_split(dataset, [train_size, val_size, test_size])

batch_size = 50

train_loader = DataLoader(training_data, batch_size, shuffle=True, num_workers=4, pin_memory=True)
valid_loader = DataLoader(validation_data, batch_size*2, num_workers=4, pin_memory=True)

# guess model. More on this in a moment
model = nn.Sequential(
    nn.Linear(11, 8),
    nn.Linear(8, 5),
)

loss_func = nn.CrossEntropyLoss()

for features, labels in train_loader:
    out = model(features)
    loss = loss_func(out, labels)
    print(f"{loss = }")
    break

Solution 1
Add labels.squeeze(-1) to the loop body a la
for features, labels in train_loader:
    out = model(features)
    labels = labels.squeeze()
    loss = loss_func(out, labels)
    print(f"{loss = }")
    break

Solution 2
Flatten your targets initially with
targets = torch.tensor(targets[:, 0])

In response to

Now I am getting this error: RuntimeError: mat1 and mat2 shapes cannot be multiplied (11x1 and 11x8) I should also add that I am using a hidden layer of size 8 and i have 5 classes

My architecture is a guess at what you're using, but as the code above is resolved by the target reshape, I'll need more to be more helpful.
Perhaps some documentation to assist? CrossEntropyLoss The example code shows the expected shape of the targets being N, rather than N, 1 or N, classes.


Instruction: PyTorch Batch masked select implementation

Input:
How can I perform a batched masked_select?
Given:
x = torch.tensor([[1., 2., 2., 2., 3.],
                  [1., 2., 4., 3., 2.]])

The desired output would be:
tensor([[1., 3., 1., 1., 1.],
        [1., 4., 3., 1., 1.]])

Here is a possible approach:
x = torch.tensor([[1., 2., 2., 2., 3.],
                  [1., 2., 4., 3., 2.]])

ones = torch.tensor([[1., 1., 1., 1., 1.],
                     [1., 1., 1., 1., 1.]])

masks = torch.tensor([[ True, False, False, False,  True],
                      [ True, False,  True,  True, False]])

for i in range(x.size(0)):
    mask = masks[i]
    s = torch.masked_select(x[i], mask)
    ones[i][:s.size(0)] = s

Are there any alternative solutions?

Output:
The main issue with those kinds of problems is that the intermediate result is non-homogenous: in your batch elements will have different numbers of masked values. This is an issue if we want to apply PyTorch builtins. Here I am providing two solutions to perform this operation.

1- Using a list comprehension
Go through the batch elements, mask and pad by the appropriate amount:
>>> pad = lambda v: F.pad(v, [0, len(m)-len(v)], value=1)
>>> torch.stack([pad(r[m]) for r, m in zip(x, masks)])
tensor([[1., 3., 1., 1., 1.],
        [1., 4., 3., 1., 1.]])

This is rather straightforward and similar to your approach.

2- Using torch.scatter
A vectorized alternative is to construct the correct value and index tensors such that we can apply torch.scatter and obtain the desired result. The trick here is to work with flattened tensors. From x and masks we first want to get access to nz and idx defined as:

nz: the non-masked values from x (using masks of course), i.e. we need to find:
tensor([1., 3., 1., 4., 3.])

idx: their corresponding indices in the output tensor when flattened.
tensor([ 0,  1,  5,  6,  7])

Then we can apply scatter with something like out = ones.scatter(dim=0, idx, nz) which will effectively perform: out[idx[i]] = nz[i].
To construct nz, we can directly index x with masksindices of non-zero values using  on masks:
>>> nz = x[masks]
tensor([1., 3., 1., 4., 3.])

For idx it will be a little bit more tricky. We can sort the mask itself, flatten it and get the non-zero values using torch.Tensor.nonzero. When sorted, the True values end up at the beginning of each rows:
>>> idx = masks.sort(1, True).values.view(-1).nonzero()[:,0]
tensor([ 0,  1,  5,  6,  7])

Finally we can apply torch.scatter and reshape to get the desired result:
>>> torch.ones(x.numel()).scatter(0, idx, nz).view_as(x)
tensor([[1., 3., 1., 1., 1.],
        [1., 4., 3., 1., 1.]])

The usage of torch.scatter here is limited since the inputs are one-dimensional. An equivalent approach is to simply to:
>>> o = torch.ones(x.numel())
>>> o[idx] = nz
>>> o.view_as(x)

Complete approach:
>>> idx = masks.sort(1, True)[0].view(-1).nonzero()[:,0]
>>> torch.ones(x.numel()).scatter(0, idx, x[masks]).view_as(x)
tensor([[1., 3., 1., 1., 1.],
        [1., 4., 3., 1., 1.]])


Instruction: How can I solve the wrong shape in DataLoader?

Input:
I have a text dataset that I want to use for a GAN and it should turn to onehotencode and this is how I Creating a Custom Dataset for my files
class Dataset2(torch.utils.data.Dataset):
    def __init__(self, list_, labels):
        'Initialization'
        self.labels = labels
        self.list_IDs = list_

    def __len__(self):
        'Denotes the total number of samples'
        return len(self.list_IDs)

    def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample
        mylist = self.list_IDs[index]

        # Load data and get label
        X = F.one_hot(mylist, num_classes=len(alphabet))
        y = self.labels[index]

        return X, y

It is working well and every time I call it, it works just fine but the problem is when I use DataLoader and try to use it, its shape is not the same as it just came out of the dataset, this is the shape that came out of the dataset
x , _ = dataset[1]
x.shape

torch.Size([1274, 22])

and this is the shape that came out dataloader
dataloader = DataLoader(dataset, batch_size=64, shuffle=True)

one = []
for epoch in range(epochs):
    for i, (real_data, _) in enumerate(dataloader):
        one.append(real_data)
one[3].shape

torch.Size([4, 1274, 22])

this 4 is number of samples in my data but it should not be there, how can I fix this problem?

Output:
You confirmed you only had four elements in your dataset. You have wrapped your dataset with a data loader with batch_size=64 which is greater than 4. This means the dataloader will only output a single batch containing 4 elements.
In turn, this means you only append a single element per epoch, and one[3].shape is a batch (the only batch of the data loader), shaped (4, 1274, 22).


Instruction: Increasing num of layers in LSTM increases the input dimensions in Pytorch?

Input:
This is my LSTM model and I am finding a peculiar problem while training it.
class LSTM1(nn.Module):
def __init__(self, num_classes, input_size, hidden_size, num_layers, seq_length,drop_prob=0.0):
    super(LSTM1, self).__init__()
    self.num_classes = num_classes #number of classes
    self.num_layers = num_layers #number of layers
    self.input_size = input_size #input size
    self.hidden_size = hidden_size #hidden state
    self.seq_length = seq_length #sequence length

    self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                      num_layers=num_layers, dropout=drop_prob,batch_first=True) #lstm
    # self.dropout = nn.Dropout(drop_prob)
    # self.fc_1 =  nn.Linear(hidden_size, num_classes)
    self.fc_1 =  nn.Linear(hidden_size, 64) #fully connected 1
    self.fc = nn.Linear(64, num_classes) #fully connected last layer

    self.relu = nn.ReLU()

def forward(self,x):
    # h_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size)).to(device) #hidden state
    # c_0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size)).to(device) #internal state
    # Propagate input through LSTM
    output, (hn, cn) = self.lstm(x) #lstm with input, hidden, and internal state

    hn = hn.view(-1, self.hidden_size) #reshaping the data for Dense layer next
    #out = self.dropout(hn)
    out = self.relu(hn)
    out = self.fc_1(out) #first Dense
    out = self.relu(out) #relu
    out = self.fc(out) #Final Output
    out = self.relu(out) #relu

    return out

My training data has these dimensions where the first one is the input and second the labels
Training Shape torch.Size([8051, 1, 201]) torch.Size([8051, 1])
When I  train with num_layers variable = 1 for the LSTM layer it works fine.
However, when I increase the num_layers to 2 I get the error which says
    Training Shape torch.Size([8051, 1, 201]) torch.Size([8051, 1])
Testing Shape torch.Size([4930, 1, 201]) torch.Size([4930, 1])
C:\Users\adity\miniconda3\envs\pytorch3\lib\site-packages\torch\nn\modules\loss.py:528: UserWarning: Using a target size (torch.Size([8051, 1])) that is different to the input size (torch.Size([16102, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
Traceback (most recent call last):
  File "C:\Users\adity\OneDrive - Louisiana State University\Documents\CSC 7343 HW\hw1.py", line 135, in <module>
    loss = criterion(outputs.to(device), y_train_tensors.to(device))
  File "C:\Users\adity\miniconda3\envs\pytorch3\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\adity\miniconda3\envs\pytorch3\lib\site-packages\torch\nn\modules\loss.py", line 528, in forward
    return F.mse_loss(input, target, reduction=self.reduction)
  File "C:\Users\adity\miniconda3\envs\pytorch3\lib\site-packages\torch\nn\functional.py", line 3089, in mse_loss
    expanded_input, expanded_target = torch.broadcast_tensors(input, target)
  File "C:\Users\adity\miniconda3\envs\pytorch3\lib\site-packages\torch\functional.py", line 73, in broadcast_tensors
    return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
RuntimeError: The size of tensor a (16102) must match the size of tensor b (8051) at non-singleton dimension 0

When I changed layers to 3 the error says the size of tensor should be 24153.
Why is the input dimensions changing when I increase num_layers?

Output: The issue is that you are flattening hn, according to the documentation page, its shape is (D*num_layers, N, Hout), i.e. it depends on the number of hidden layers. So you will either have to change the fully connected that is following or only take the last hidden state of your LSTM.


Instruction: How to delete tensors from pytorch graph?

Input:
I'm doing predictions on images for object detection in a for loop.
I've actually ran into the same issue with tensorflow and hoped I could solve it with pytorch.
At least now it seems I have found out what the issue is (naively assuming it's the same for tensorflow)
I predict like this
 model = detection.fasterrcnn_resnet50_fpn(pretrained=True,
    progress=True,pretrained_backbone=True).to(DEVICE)
    for i in tqdm(range(train.shape[0])):
        image = cv2.imread(train_img_paths[i])
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        image = image.transpose((2, 0, 1))
        image = image / 255.0
        image = np.expand_dims(image, axis=0)
        image = torch.FloatTensor(image)
        image = image.to(DEVICE)
        predictions = model(image)[0]

Now through the garbage collector I found that each and every image stays in the graph.
Is there away to avoid it?
I have not been able to use dataloader or dataset with the detection models (same with tensorflow hub)

Output:
Don't forget when you're doing testing to turn gradient accumulation off. You can do this by either wrapping your code like:
with torch.no_grad():
     model.eval()
     out = model(x)

or if your code is a function, using a decorator to do the same thing:
@torch.no_grad()
def model_proc(model,x):
    model.eval()
    return model(x)


Instruction: Pytorch argmax across multiple dimensions

Input:
I have a 4D tensor, and I would like to get the argmax across the last two dimensions. torch.argmax only accepts integers as the "dim" argument, not tuples.
How can I accomplish this?
Here's what I had in mind, but I can't figure out how to match up the dimensions of my two "indices" tensors. original_array is shape [1, 512, 37, 59].
max_vals, indices_r = torch.max(original_array, dim=2)
max_vals, indices_c = torch.max(max_vals, dim=2)
indices = torch.hstack((indices_r, indices_c))

Output:
As others mentioned, its best to flatten the last two dimensions and apply argmax
original_array = torch.rand(1, 512, 37, 59)
original_flatten = original_array.view(1, 512, -1)
_, max_ind = original_flatten.max(-1)

.. you will get the linear index of the maximum values. In case you want the 2D indecies of the maximum values, you can do "unflatten" the indecies using the number of columns
# 59 is the number of columns for the (37, 59) part
torch.stack([max_ind // 59, max_ind % 59], -1)

this will give you a (1, 512, 2) where each last 2 dim contains 2D coordinates.


Instruction: How to Define Model in Native Pytorch Format and Import Into LightningModule Without Copy and Pasting?

Input:
Let's say I have a native pytorch model like this
class NormalAutoEncoder(nn.Module)):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(28 * 28, 64), nn.ReLU(), nn.Linear(64, 3))
        self.decoder = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 28 * 28))

    def forward(self, x):
        # in lightning, forward defines the prediction/inference actions
        embedding = self.encoder(x)
        return embedding

How can I have the __init__ and forward functions (basically, the whole network) into a pytorch lighting module without copying and pasting?

Output:
Simple. Make use of Python's inheritance mechanism.
If the following is a native PyTorch module
class NormalAutoEncoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = ...
        self.decoder = ...

    def forward(self, x):
        embedding = ...
        return embedding

Then have your new LightningAutoEncoder that also inherits from NormalAutoEncoder
class LightningAutoEncoder(LightningModule, NormalAutoEncoder):
    def __init__(self, ...):
        LightningModule.__init__(self) # only LightningModule's init
        NormalAutoEncoder.__init__(self, ...) # this basically executes __init__() of the NormalAutoEncoder

    def forward(self, x):
        # offloads its execution to NormalAutoEncoder's forward() method
        return NormalAutoEncoder.forward(self, x)

That's it. No copy-pasting.


Instruction: How to specify the axis over which to get the elements in PyTorch

Input:
I was looking for an answer around SO, but tbh, don't even know how to phrase a question.
Given a tensor, how do I get indices over a specified axis
Here is a simple example
indices = torch.tensor([2, 4, 5])

# Depending on the context I need either
y = x[indices]

# Or
y = x[:, indices]

# Or any other axis
y = x[:, :, :, indices]

Here is a usecase where I need this behavior

def remove_weaklings(x: Tensor, percentage: float, axis: int) -> Tensor:
    all_axes = set(range(x.ndim)) - set([axis])
    y = x
    # Sumup over all other axes
    for a in all_axes:
        y = y.sum(axis=a, keepdim=True)
    y = y.squeeze()
    # Get the sorted list
    _, idx = torch.sort(y)
    # Get only a fraction of the list
    idx = idx[:int(percentage * len(idx))]
    # Get the indices over some axis
    # !!! This part not sure how to solve !!!
    return x.get_over_axis(axis=axis, indices=idx)

Output:
I think you are looking for topk:
def remove_weaklings(x, percentage, axis):
  k = int(percentage * x.shape[axis])
  return torch.topk(x, k, dim=axis)

If you want a more generic solution, you can use numpy's slicing:
def get_over_axis(x, indices, axis):
  i = []
  for d_ in range(x.dim()):
    if d_ == axis:
      i.append(indices)
    else:
      i.append(np.s_[:])
  return x[tuple(i)]


Instruction: pytorch - Model_heplers.py in is_overridden > raise ValueError(“Expected a parent”)

Input:
I am running on a new remote server a code that used to work on another remote server. I think I setup things in the same way, but when I run my training script, I get this error:
Traceback (most recent call last):
  File "/home/andrea/code/vertikal-machine-learning/source/model/hss_bearing_mk2/hss_bearing_mk2/models/train_model.py", line 144, in <module>
    seq_len=seq_len, mname=mname)
  File "/home/andrea/code/vertikal-machine-learning/source/model/hss_bearing_mk2/hss_bearing_mk2/models/pytorch_models.py", line 321, in train_test
    trainer.fit(model, datamodule=dm)
  File "/home/andrea/anaconda3/envs/hss_bearing_mk2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/home/andrea/anaconda3/envs/hss_bearing_mk2/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 849, in _run
    self.config_validator.verify_loop_configurations(model)
  File "/home/andrea/anaconda3/envs/hss_bearing_mk2/lib/python3.7/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 34, in verify_loop_configurations
    self.__verify_train_loop_configuration(model)
  File "/home/andrea/anaconda3/envs/hss_bearing_mk2/lib/python3.7/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 49, in __verify_train_loop_configuration
    has_training_step = is_overridden("training_step", model)
  File "/home/andrea/anaconda3/envs/hss_bearing_mk2/lib/python3.7/site-packages/pytorch_lightning/utilities/model_helpers.py", line 45, in is_overridden
    raise ValueError("Expected a parent")
ValueError: Expected a parent

Here is the part of code that looks buggy for some reason:
    model = get_model(mname=mname)

    dm = DataModule(
        X_train=X_train,
        y_train=y_train,
        X_val=X_val,
        y_val=y_val,
        X_test=X_test,
        y_test=y_test,
        keys_train=keys_train,
        keys_val=keys_val,
        keys_test=keys_test,
        seq_len=seq_len,
        batch_size=batch_size,
        num_workers=4
    )
    # trainer.logger_connector.callback_metrics
    trainer.fit(model, datamodule=dm)

Is it something related to environment setup? Something overridden by something something??
Can someone point me in the right direction?
EDIT: I tried to run my project locally in a newly created environment and I have the same error.
EDIT 2: My DataModule inherits from LightningDataModule
class DataModule(pl.LightningDataModule):

Output: The problem was that model was inheriting from nn.Module instead of from pl.LightningModule


Instruction: How to use soft labels in computer vision with PyTorch?

Input:
I have an image dataset with soft labels (i.e. the images don't belong to a single class, but rather I have a probability distribution saying that there's a 66% chance this image belong in one class and 33% chance it belongs in some other class).
I am struggling to figure out how to setup my PyTorch code to allow this to be represented by the model and outputted correctly. The probabilities are saved in a csv file. I have looked at the PyTorch docs and other resources which mention the cross entropy loss function but I am still unclear how to import the data successfully and make use of soft labels.

Output:
What you are trying to solve is a multi-label classification task, i.e. instances can be classified with more than one label at a time. You cannot use torch.CrossEntropyLoss since it only allows for single-label targets. So you have two options:

Either use a soft version of the nn.CrossEntropyLoss function, this can be done by implementing the loss by hand allowing for soft targets. You can find such implementation on Soft Cross Entropy in PyTorch.

Or consider the task a multiple "independent" binary classification tasks, in this case, you would use nn.BCEWithLogitsLoss (this layer contains a sigmoid function).


Instruction: How can I download to my pc an image dataset after loading it into a dataloader in pytorch?

Input:
I've been following this article:
https://github.com/moein-shariatnia/Deep-Learning/tree/main/Image%20Colorization%20Tutorial
And I have a problem. I'm trying to download the images I have loaded into the pytorch dataloader, but since they are chosen randomly from a larger dataset I can't figure it out.
How can I download the exact images that I've loaded into "train_dl" to my PC (as a folder) after they are being chosen? I have already trained my model on these exact images, so I'll need them for future work.

Output:
Dataloaders should have a dataset attribute which you can use to find the paths in your code
In the code:
class ColorizationDataset(Dataset):
    def __init__(self, paths, split='train'):
        ...

        self.split = split
        self.size = SIZE
        self.paths = paths #<<<<- IMPORTANT LINE HERE

    def __getitem__(self, idx):
        ...

    def __len__(self):
        return len(self.paths)

def make_dataloaders(batch_size=16, n_workers=4, pin_memory=True, **kwargs): # A handy function to make our dataloaders
    dataset = ColorizationDataset(**kwargs)
    dataloader = DataLoader(dataset, batch_size=batch_size, num_workers=n_workers,
                            pin_memory=pin_memory)
    return dataloader

So, to get the items from your dataloader you can run:
train_paths = train_dl.dataset.paths

You can loop through this list to save the images individually (not recommended) or just save it as text with the train/val/test splits as separate files.


Instruction: Indexing one element per row in PyTorch matrix

Input:
Let price be a PyTorch tensor with shape (num_days, num_products).
Let purchased_product_by_day be an integer tensor with shape (num_days,), with values in range(num_products).
Intuitively, price lists the price, for each day, of each product, and purchased_product_by_day are the indices of purchased products, one per day.
To obtain a tensor containing the expense per day, I can write
price[list(range(num_days)), purchased_product_by_day]
but this iterates over days at the Python level. I would like to obtain the same tensor on the C level.
I tried
price[:, purchased_product_by_day]
but that does not work, being equivalent to indexing each row by purchased_product_by_day for each day and stacking the results.
Is there a way to do that without iteration at the Python level?

Output:
Based on Row-wise Element Indexing in PyTorch for C++, the solution in Python is
price.gather(1, purchased_product_by_day.unsqueeze(1)).squeeze()


Instruction: Working of numpy array to torch tensor conversion

Input:
I want to know how the conversion from a NumPy array to a tensor happens in PyTorch as well as Tensorflow. Does it create a copy of the entire array or there is some in-place mechanism?
Secondly, how to clear the memory that was used earlier by the NumPy array that was later converted into a tensor.

Output:
When device is CPU in PyTorch, PyTorch and Numpy uses the same internal representation of n-dimensional arrays in memory, so when converted from a Numpy array to a PyTorch tensor no copy operation is performed, only the way they are represented internally is changed. Refer here.

Python garbage collector uses reference counts for clearing unused memory similar to shared pointers in C++.


Instruction: PyTorch CUDA: "to(device)" vs "device=device" for tensors?

Input:
I found a somewhat similar question here What is the difference between model.to(device) and model=model.to(device)?, but I would like to check again if the same applies to my example:

Using .to(self.device)

mask = torch.tril(torch.ones(len_q, len_k)).type(torch.BoolTensor).to(self.device)`

and

Using device=self.device

mask = torch.tril(torch.ones((trg_len, trg_len), device = self.device)).bool()

Are they both accomplishing the same thing - ensuring that mask goes to the GPU?

Output:
The torch.Tensor.to function will make a copy of your tensor on the destination device. While setting the device option on initialization will place it there on init, so there is no copy involved.
So in your case you would rather do:
>>> mask = torch.tril(torch.ones(len_q, len_k), device=self.device)

But to give an answer to your question, both have the effect of placing mask on self.device. The only difference is that in the former you will have a copy of your data on both devices.

The same can be said for torch.Tensor.bool vs. initializing with dtype:
>>> torch.randint(0, 1, (10,)).bool()

Will make a copy, while the following won't:
>>> torch.randint(0, 1, (10,), dtype=torch.bool)

However, torch.tril doesn't provide a dtype option, so it is not relevant here.


Instruction: How to assign numbers to characters in dictionary?

Input:
I have a sequence generation and in the final result, I have a tensor that contains my sequence, I have that tensor which is one_hot_encoded and an alphabet and I try to inverse that one_hot_encoded to characters, I try dictionary and argmax function but I could not find any solution that can do it all tensor at once and most of the answers do just one character
int_to_char = dict((i, c) for i, c in enumerate(alphabet))
inverted = int_to_char[argmax(onehot_encoded[0])]
print(inverted)

this is how I use argmax and its answer
a.argmax(-1)

tensor([10,  4, 17,  ..., 19, 16, 20])

these are alphabet and expected result
alphabet = ['A', 'C', 'D', 'E', 'F', 'G','H', 'I', 'K', 'L', 'M', 'N', 'P', 'Q', 'R', 'S', 'T', 'V', 'W', 'Y', '*', 'X']

data = "MFVFLVLLPLVSSQCVNLTTRTQLPPAYTNSFTRGVYYPDKVFRSSVLHSTQDLFLPFFSNVTWFHAIHVSGTNGTKRFDNPVLPFNDGVYFASTEKSNIIRGWIFGTTLDSKTQSLLIVNNATNVVIKVCEFQFCNDPFLGVYYHKNNKSLMESEFRVYSSANNCTFEYVSQPFLMDLEGKQGNFKNXREFVFKNIDGYFKIYSKHTPINLVRDLPQGFSALEPLVDLPIGINITRFQTLLALHRSYLTPGDSSSGWTAGAAAYYVGYLQPRTFLLKYNENGTITDAVDCALDPLSETKCTLKSFTVEKGIYQTSNFRVQPTESIVRFPNITNLCPFGEVFNATRFASVYAWNRKRISNCVADYSVLYNSASFSTFKCYGVSPTKLNDLCFTNVYADSFVIRGDEVRQIAPGQTGKIADYNYKLPDDFTGCVIAWNSNNLDSKVGGNYNYLYRLFRKSNLKPFERDISTEIYQAGSTPCNGVKGFNCYFPLQSYGFQPTNGVGYQPYRVVVLSFELLHAPATVCGPKKSTNLVKNKCVNFNFNGLTGTGVLTESNKKFLPFQQFGRDIADTTDAVRDPQTLEILDITPCSFGGVSVITPGTNTSNQVAVLYQGVNCTEVPVAIHADQLTPTWRVYSTGSNVFQTRAGCLIGAEHVNNSYECDIPIGAGICASYQTQTNSPRRARSVASQSIIAYTMSLGAENSVAYSNNSIAIPTNFTISVTTEILPVSMTKTSVDCTMYICGDSTECSNLLLQYGSFCTQLNRALTVIAVEQDKNTQEVFAQVKQIYKTPPIKDFGGFNFSQILPDPSKPSKRSFIEDLLFNKVTLADAGFIKQYGDCLGDIAARDLICAQKFNGLTVLPPLLTDEMIAQYTSALLAGTITSGWTFGAGAALQIPFAMQMAYRFNGIGVTQNVLYENQKLIANQFNSAIGKIQDSLSSTASALGKLQDVVNQNAQALNTLVKQLSSNFGAISSVLNDILSRLDKVEAEVQIDRLITGRLQSLQTYVTQQLIRAAEIRASANLAATKMSECVLGQSKRVDFCGKGYHLMSFPQSAPHGVVFLHVTYVPAQEKNFTTAPAICHDGKAHFPREGVFVSNGTHWFVTQRNFYEPQIITTDNTFVSGNCDVVIGIVNNTVYDPLQPELDSFKEELDKYFKNHTSPDVDLGDISGINASVVNIQKEIDRLNEVAKNLNESLIDLQELGKYEQYIKWPWYIWLGFIAGLIAIVMVTIMLCCMTSCCSCLKGCCSCGSCCKFDEDDSEPVLKGVKLHYT*"

I want to turn this tensor into characters based on an alphabet, How can I do that?

Output:
Given a sequence of ints, here is a sample:
encoded = torch.tensor([10,  4, 17, 19, 16, 20])

You could apply a list comprehension on your sequence, accessing alphabet to extract the letter corresponding to each index:
>>> ''.join(alphabet[e] for e in encoded)
MFVYT*

Given a str sequence of letters:
data = 'MFVFLVLLPLVSSQCVNLTTRTQLPPAYTNSFTRGVYYPDKVFRSSVLHSTQDLFLPFFSNVTWFHAIHVSGTNGTKRFDNPVLPFNDGVYFASTEKSNIIRGWIFGTTLDSKTQSLLIVNNATNVVIKVCEFQFCNDPFLGVYYHKNNKSLMESEFRVYSSANNCTFEYVSQPFLMDLEGKQGNFKNXREFVFKNIDGYFKIYSKHTPINLVRDLPQGFSALEPLVDLPIGINITRFQTLLALHRSYLTPGDSSSGWTAGAAAYYVGYLQPRTFLLKYNENGTITDAVDCALDPLSETKCTLKSFTVEKGIYQTSNFRVQPTESIVRFPNITNLCPFGEVFNATRFASVYAWNRKRISNCVADYSVLYNSASFSTFKCYGVSPTKLNDLCFTNVYADSFVIRGDEVRQIAPGQTGKIADYNYKLPDDFTGCVIAWNSNNLDSKVGGNYNYLYRLFRKSNLKPFERDISTEIYQAGSTPCNGVKGFNCYFPLQSYGFQPTNGVGYQPYRVVVLSFELLHAPATVCGPKKSTNLVKNKCVNFNFNGLTGTGVLTESNKKFLPFQQFGRDIADTTDAVRDPQTLEILDITPCSFGGVSVITPGTNTSNQVAVLYQGVNCTEVPVAIHADQLTPTWRVYSTGSNVFQTRAGCLIGAEHVNNSYECDIPIGAGICASYQTQTNSPRRARSVASQSIIAYTMSLGAENSVAYSNNSIAIPTNFTISVTTEILPVSMTKTSVDCTMYICGDSTECSNLLLQYGSFCTQLNRALTVIAVEQDKNTQEVFAQVKQIYKTPPIKDFGGFNFSQILPDPSKPSKRSFIEDLLFNKVTLADAGFIKQYGDCLGDIAARDLICAQKFNGLTVLPPLLTDEMIAQYTSALLAGTITSGWTFGAGAALQIPFAMQMAYRFNGIGVTQNVLYENQKLIANQFNSAIGKIQDSLSSTASALGKLQDVVNQNAQALNTLVKQLSSNFGAISSVLNDILSRLDKVEAEVQIDRLITGRLQSLQTYVTQQLIRAAEIRASANLAATKMSECVLGQSKRVDFCGKGYHLMSFPQSAPHGVVFLHVTYVPAQEKNFTTAPAICHDGKAHFPREGVFVSNGTHWFVTQRNFYEPQIITTDNTFVSGNCDVVIGIVNNTVYDPLQPELDSFKEELDKYFKNHTSPDVDLGDISGINASVVNIQKEIDRLNEVAKNLNESLIDLQELGKYEQYIKWPWYIWLGFIAGLIAIVMVTIMLCCMTSCCSCLKGCCSCGSCCKFDEDDSEPVLKGVKLHYT'

You could apply a list comprehension on your sequence, this time retrieving the index in alphabet for each letter:
>>> encoding = [alphabet.index(x) for x in data]

Alternatively you can use map:
>>> encoding = map(lambda x: alphabet.index(x), data)


Instruction: Correct way to compute VGG features for Perceptual loss

Input:
While computing VGG Perceptual loss, although I have not seen, I feel it is alright to wrap the computation of VGG features for the GT image inside torch.no_grad().
So basically I feel the following will be alright,
with torch.no_grad():
    gt_vgg_features = self.vgg_features(gt)

nw_op_vgg_features = self.vgg_features(nw_op)

# Now compute L1 loss

or one should use,
gt_vgg_features = self.vgg_features(gt)
nw_op_vgg_features = self.vgg_features(nw_op)

In both approaches requires_grad for VGG parameters is set False and VGG put in eval() mode.
The first approach will save a lot of GPU resources and feel should be numerically equal to the second one as no backpropagation is required through GT images. But in most implementations, I find the second approach used for computing VGG perceptual loss.
So which option we should go for implementing VGG perceptual loss in PyTorch?

Output:
The first way:
with torch.no_grad():
    gt_vgg_features = self.vgg_features(gt)

nw_op_vgg_features = self.vgg_features(nw_op)

Although VGG is in eval mode and its parameters are kept fix, you still need to propagate gradients through it from the loss on the features to the output nw_op.
However, there's no reason to compute these gradients w.r.t gt.


Instruction: How to concatenate along a dimension of a single pytorch tensor?

Input: I wrote a custom pytorch Dataset and the __getitem__() function return a tensor with shape (250, 150), then I used DataLoader to generate a batch of data with batch size 10. My intension was to have a batch with shape (2500, 150) as concatenation of these 10 tensors along dimension 0, but the output of DataLoader has shape (10, 250, 150). How do I transform the output of DataLoader into shape (2500, 150) as concatenation along dimension 0?

Output:
PyTorch DataLoader will always add an extra batch dimension at 0th index. So, if you get a tensor of shape (10, 250, 150), you can simple reshape it with
# x is of shape (10, 250, 150)
x_ = x.view(-1, 150)
# x_ is of shape (2500, 150)

Or, to be more correct, you can supply a custom collator to your dataloader
def custom_collate(batch):
    # each item in batch is (250, 150) as returned by __getitem__
    return torch.cat(batch, 0)

dl = DataLoader(dataset, batch_size=10, collate_fn=custom_collate, ...)

This will create properly sized tensor right in the dataloder itself, so no need for any post-processing with .view().


Instruction: PyTorch `torch.no_grad` vs `torch.inference_mode`

Input:
PyTorch has new functionality torch.inference_mode as of v1.9 which is "analogous to torch.no_grad... Code run under this mode gets better performance by disabling view tracking and version counter bumps."
If I am just evaluating my model at test time (i.e. not training), is there any situation where torch.no_grad is preferable to torch.inference_mode? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn't surface as a runtime error then I assume it is indeed preferable to use torch.inference_mode).
More details on why inference mode was developed are mentioned in the PyTorch Developer Podcast.

Output: Yes, torch.inference_mode is indeed preferable to torch.no_grad in all situations where inference mode does not throw a runtime error. Check here and here.


Instruction: How padding works in PyTorch

Input:
Normally if I understood well PyTorch implementation of the Conv2D layer, the padding parameter will expand the shape of the convolved image with zeros to all four sides of the input. So, if we have an image of shape (6,6) and set padding = 2 and strides = 2 and kernel = (5,5), the output will be an image of shape (1,1).  Then, padding = 2 will pad with zeroes (2 up, 2 down, 2 left and 2 right) resulting in a convolved image of shape (5,5)
However when running the following script :
import torch
from torch import nn
x = torch.ones(1,1,6,6)
y = nn.Conv2d(in_channels= 1, out_channels=1,
              kernel_size= 5, stride = 2,
              padding = 2,)(x)

I got the following outputs:
y.shape
==> torch.Size([1, 1, 3, 3]) ("So shape of convolved image = (3,3) instead of (5,5)")

y[0][0]
==> tensor([[0.1892, 0.1718, 0.2627, 0.2627, 0.4423, 0.2906],
    [0.4578, 0.6136, 0.7614, 0.7614, 0.9293, 0.6835],
    [0.2679, 0.5373, 0.6183, 0.6183, 0.7267, 0.5638],
    [0.2679, 0.5373, 0.6183, 0.6183, 0.7267, 0.5638],
    [0.2589, 0.5793, 0.5466, 0.5466, 0.4823, 0.4467],
    [0.0760, 0.2057, 0.1017, 0.1017, 0.0660, 0.0411]],
   grad_fn=<SelectBackward>)

Normally it should be filled with zeroes. I'm confused. Can anyone help please?

Output:
The input is padded, not the output. In your case, the conv2d layer will apply  a two-pixel padding on all sides just before computing the convolution operation.
For illustration purposes,
>>> weight = torch.rand(1, 1, 5, 5)

Here we apply a convolution with padding=2:
>>> x = torch.ones(1,1,6,6)
>>> F.conv2d(x, weight, stride=2, padding=2)
tensor([[[[ 5.9152,  8.8923,  6.0984],
          [ 8.9397, 14.7627, 10.8613],
          [ 7.2708, 12.0152,  9.0840]]]])

And we don't use any padding but instead apply it ourselves on the input:
>>> x_padded = F.pad(x, (2,)*4)
tensor([[[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 1., 1., 1., 1., 1., 1., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
          [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]]])

>>> F.conv2d(x_padded, weight, stride=2)
tensor([[[[ 5.9152,  8.8923,  6.0984],
          [ 8.9397, 14.7627, 10.8613],
          [ 7.2708, 12.0152,  9.0840]]]])


Instruction: forward() not overridden in implementation of nn.Module in an example

Input:
In this example, we see the following implementation of nn.Module:
class Net(torch.nn.Module):
    def __init__(self, in_channels, hidden_channels, out_channels):
        super().__init__()
        self.conv1 = GCNConv(in_channels, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, out_channels)

    def encode(self, x, edge_index):
        x = self.conv1(x, edge_index).relu()
        return self.conv2(x, edge_index)

    def decode(self, z, edge_label_index):
        return (z[edge_label_index[0]] * z[edge_label_index[1]]).sum(dim=-1)

    def decode_all(self, z):
        prob_adj = z @ z.t()
        return (prob_adj > 0).nonzero(as_tuple=False).t()

However, in the docs we have that 'forward(*input)' "Should be overridden by all subclasses."
Why is that not then the case in this example?

Output:
This Net module is meant to be used via two separate interfaces encoder and decode, at least it seems so... Since it doesn't have a forward implementation, then yes it is improperly inheriting from nn.Module. However, the code is still "valid", and will run properly but may have some side effects if you are using forward hooks.
The standard way of performing inference on a nn.Module is to call the object, i.e. calling the __call__ function. This __call__ function is implemented by the parent class nn.Module and will in turn do two things:

handle forward hooks before or after the inference call
call the forward function of the class.

The __call__ function acts as a wrapper of forward.
So for this reason the forward function is expected to be overridden by the user-defined nn.Module. The only caveat of violating this design pattern is that it will effectively ignore any hooks applied on the nn.Module.


Instruction: Doing inference using pytorch saved weights in .tar file using django

Input:
I am doing a machine learning project where I need to display the predictions on a webpage. The webpage is build using Django. I have predictions function and the weights of the model but how to integrate the predictions function, model, and weights in the Django code and do predictions.
My prediction code
def predicting(model, device, loader):
    model.eval()
    total_preds = torch.Tensor()
    total_labels = torch.Tensor()
    with torch.no_grad():
        for solute_graphs, solvent_graphs, solute_lens, solvent_lens in loader:
            outputs, i_map = model(
            [solute_graphs.to(device), solvent_graphs.to(device), torch.tensor(solute_lens).to(device),
             torch.tensor(solvent_lens).to(device)])
            print(outputs)
            total_preds = torch.cat((total_preds, outputs.cpu()), 0)
    return total_preds.numpy().flatten()

I have saved the weights in .tar file so I need to run the model while loading the weights for prediction. I have no idea where to keep my PyTorch model and the weights to do inference using Django. please help.

Output:
Deploy a PyTorch deep learning classifier at Heroku using Django in 30 minutes
https://www.youtube.com/watch?v=MLk2We1rJPs
Using PyTorch Inside a Django App:
https://stefanbschneider.github.io/blog/pytorch-django
I hope this will help you get started!!!


Instruction: Deep learning with PyTorch:Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 2-dimensional input of size [32, 1728] instead

Input:
I am using deep learning with PyTorch to do some image classification. Whenever I try to train my model, the function forward fails. Could somebody explain me what are the reasons why the input size is wrong and how to fix this, please?
This is the code for my model, as well as my training loss and my optimizer:
model.fc = nn.Sequential(
        nn.Conv2d(1, 6, 9, padding=0),  # 64040
        nn.ReLU(),  #
        nn.AvgPool2d(2, stride=2),  # max 62020
        nn.Conv2d(6, 16, 11, padding=0),  # 161010
        nn.ReLU(),  # 161010
        nn.AvgPool2d(2, stride=2),  # 1655 = 400
        nn.Flatten(),
        nn.Linear(400, 200),
        nn.ReLU(),
        nn.Linear(200, 100),
        nn.ReLU(),
        nn.Linear(100, 3),
        nn.LogSoftmax(dim=1))

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.fc.parameters(), lr=0.003)
model.to(device)

This is the validation function:
def validation(model, val_dataloader, criterion):

val_loss = 0
accuracy = 0

for images, labels in iter(val_dataloader):

    images, labels = images.to('cuda'), labels.to('cuda')

    output = model.forward(images)
    val_loss += criterion(output, labels).item()

    probabilities = torch.exp(output)

    equality = (labels.data == probabilities.max(dim=1)[1])
    accuracy += equality.type(torch.FloatTensor).mean()

return val_loss, accuracy

Finally, this is my training function:
def train_classifier():
  epochs = 10
  steps = 0
  print_every = 40

  model.to('cuda')

  for e in range(epochs):

      model.train()

      running_loss = 0

      for images, labels in iter(train_dataloader):
        images = images.view(images.shape[0], -1)  #this flattens it?

        steps += 1

        images, labels = images.to('cuda'), labels.to('cuda')

        optimizer.zero_grad()

        # training
        output = model.forward(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

        if steps % print_every == 0:

            model.eval()

            # Turn off gradients for validation, saves memory and computations
            with torch.no_grad():
                validation_loss, accuracy = validation(model, validate_loader, criterion)

            print("Epoch: {}/{}.. ".format(e+1, epochs),
                  "Training Loss: {:.3f}.. ".format(running_loss/print_every),
                  "Validation Loss: {:.3f}.. ".format(validation_loss/len(validate_loader)),
                  "Validation Accuracy: {:.3f}".format(accuracy/len(validate_loader)))

            running_loss = 0
            model.train()

train_classifier()

Error code:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-32-60a435d940e1> in <module>()
     49             model.train()
     50
---> 51 train_classifier()

5 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    438                             _pair(0), self.dilation, self.groups)
    439         return F.conv2d(input, weight, bias, self.stride,
--> 440                         self.padding, self.dilation, self.groups)
    441
    442     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 2-dimensional input of size [32, 1728] instead

Any help is appreciated! Thank you!

Output: Your network has both Conv2d layers as well as fully connected Linear layers - this is the source of the problem: Conv2d expects its inputs to be 4D: batch-channel-height-width. On the other hand, nn.Linear works of "flattened" features: batch-channel. Therefore, you need to "flatten" your data, but not before applying the network, but rather during its processing, where you have a Flattening layer.


Instruction: How to convert the prediction of pytorch into normal text

Input:
I have a PyTorch model and i am doing prediction on it. After doing prediction i am getting the output as
tensor([[-3.4333]], grad_fn=<AddmmBackward>)

But i need it as normal integer -3.4333. How can i do it.

Output: Call .item on your tensor to convert it to a standard python number.


Instruction: Initializing a differentiable param in pytorch

Input:
I'm trying to define a set of new parameters B in a pytorch model. I would like to initialize the new params with current weights of the model W.
Question: I want these params B to be differentiable, but autograd should not track their history to W (so B should have a new memory with no reference to W). What is the correct function to use?
I understand B = W.clone() will result in autograd tracking history of B to W while differentiating. Also I understand that B = W.detach().clone() will not be differentiable.

EDIT:
I believe B = nn.Parameter(W.detach().clone()) should be the correct function. Is this correct and if yes, is this the simplest function to use?

Output:
There are several ways to do it and one of them is
B = W.clone().detach()

Another elegant one comes to my mind is
B = torch.new_tensor(x, requires_grad=True)

which is much more readable and the doc says they are equivalent.
If you want it as a parameter to nn.Module, of course wrap it with
self.B = nn.Parameter(...)


Instruction: What is the correct way to maximize one loss and minimize another during NN training?

Input:
I have a simple NN:
import torch
import torch.nn as nn
import torch.optim as optim

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(1, 5)
        self.fc2 = nn.Linear(5, 10)
        self.fc3 = nn.Linear(10, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = torch.relu(x)
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Model()

opt = optim.Adam(net.parameters())

I also have some input features:
features = torch.rand((3,1))

I can train it normally with a simple loss function that will be minimized:
for i in range(10):
    opt.zero_grad()
    out = net(features)
    loss = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    print('loss:', loss)
    loss.backward()
    opt.step()

However, if I'll add another loss component to this that I'd want to maximize--loss2:
loss2s = []
for i in range(10000):
    opt.zero_grad()
    out = net(features)
    loss1 = torch.mean(torch.square(torch.tensor(5) - torch.sum(out)))
    loss2 = torch.sum(torch.tensor([torch.sum(w_arr) for w_arr in net.parameters()]))
    loss2s.append(loss2)
    loss = loss1 + loss2
    loss.backward()
    opt.step()

It becomes seemingly unstable as the 2 losses have different scales. Also, I'm not sure that this is the correct way because how would the loss know to maximize one part and minimize the other. Note that this is just an example, obviously there's no point in increasing the weights.
import matplotlib.pyplot as plt
plt.plot(loss2s, c='r')
plt.plot(loss1s, c='b')

And also I believe that minimizing functions is the common way to train in ML, so I wasn't sure if changing the maximization problem into minimization problem in some way will be better.

Output:
The standard way to denote "minimization" and "maximization" is changing the sign. PyTorch always minimizes a loss if the following is done
loss.backward()

So, if another loss2 needs to be maximized, we add negative of it
overall_loss = loss + (- loss2)
overall_loss.backward()

since minimizing a negative quantity is equivalent to maximizing the original positive quantity.
With regard to "scale", yes scales do matter. Often the following is done in order to match scales
overall_loss = loss + alpha * (- loss2)

where alpha is a fraction denoting relative importance of one loss w.r.t to the other. Its a hyperparameter and needs to experimented with.

Keeping technicalities aside, whether the resulting loss will be stable depends a lot on the specific problem and loss functions involved. If the losses are contradicting, you may experience instability. The ways to deal them is itself a research problem and much beyond the scope of this question.


Instruction: Torch: Nccl available but not used (?)

Input:
I use PyTorch 1.9.0 but get the following error when trying to run a distributed version of a model:
File "/home/ferdiko/fastmoe/examples/transformer-xl/train.py", line 315, in <module>
    para_model = DistributedGroupedDataParallel(model).to(device)
  File "/home/ferdiko/anaconda3/envs/fastmoe/lib/python3.9/site-packages/fastmoe-0.2.1-py3.9-linux-x86_64.egg/fmoe/distributed.py", line 45, in __init__
    self.comms["dp"] = get_torch_default_comm()
  File "/home/ferdiko/anaconda3/envs/fastmoe/lib/python3.9/site-packages/fastmoe-0.2.1-py3.9-linux-x86_64.egg/fmoe/utils.py", line 30, in get_torch_default_comm
    raise RuntimeError("Unsupported PyTorch version")

if I run torch.cuda.nccl.version() I get 2708. The developers suggested to run:
x = torch.rand(10).cuda()
print(torch.cuda.nccl.is_available(x))

which gives me False. Does this actually mean that there's a problem with PyTorch and NCCL?

Output:
torch.cuda.nccl.is_available takes a sequence of tensors, and if they are on different devices, there is hope that you'll get a True:
    In [1]: import torch

    In [2]: x = torch.rand(1024, 1024, device='cuda:0')

    In [3]: y = torch.rand(1024, 1024, device='cuda:1')

    In [4]: torch.cuda.nccl.is_available([x, y])
    Out[4]: True

If you give it just one tensor, torch.cuda.nccl.is_available will iterate through it instead, but different parts of the same tensor are always on the same device, so you'll always get a False:
    In [5]: torch.cuda.nccl.is_available(x)
    Out[5]: False

    In [6]: torch.cuda.nccl.is_available([x])
    Out[6]: True


Instruction: What is the predicted output label from a PyTorch model?

Input:
I am currently training a ResNet18 model with a custom optimizer in PyTorch.
I am using CrossEntropyLoss() and the ResNet18 model from PyTorch.
In tensorflow the outputs are of the desired shape, but in pytorch it is necessary to find the argmax of the predicted labels in order to find the accuracy.
If my batch size = 64 with the resnet model, why is the predicted label of shape [64, 1000]?
What do the 1000 values correspond to?

Output:
The predicted quantity is not "label", it is the probability (soft score) of the input being one of 1000 classes.
The output of (64, 1000) contains a 1000 length vector for each input in a batch. If you want discrete labels (i.e. 0 to 999), perform an argmax over it
labels = torch.argmax(output, 1)

By argmax over each probability vector, we compute which class (among 1000) has the highest probability for the input.


Instruction: Tensor Reshape Query

Input:
I have a image tensor of shape :-
N,C,H,W = 5,512,13,13

I need to take a mean across H and W dimensions so that the output is of shape :-
N,C,1,1

I am trying doing for loop but is there some better way to do so using reshape. .

Output:
 import torch

 tz = torch.rand(5, 512, 13, 13)
 tzm = tz.mean(dim=(2,3), keepdim=True)
 tzm.shape

Output
torch.Size([5, 512, 1, 1])


Instruction: How to sample data from the proximity of existing data?

Input:
I have data for xor as below -

x
y
z
x ^ y ^ z

0
0
1
1

0
1
0
1

1
0
0
1

1
1
1
1

Kept only the ones that make the xor of all three equal to 1.
I want to generate synthetic data around the already available data within some range uniformly at random. The above table can be thought of as seed data. An example of expected table will be as follows:

x
y
z
x ^ y ^ z

0.1
0.3
0.8
0.9

0.25
0.87
0.03
0.99

0.79
0.09
0.28
0.82

0.97
0.76
0.91
0.89

Above table is sampled with a range of 0 to 0.3 for 0 value and with range 0.7 to 1 for value 1.
I want to achieve this using pytorch.

Output:
For a problem such as this, you are able to completely synthesise data without using a reference because it has a simple solution. For zero (0-0.3) you can use the torch.rand function to generate uniformly random data for 0-1 and scale it. For one (0.7-1) you can do the same and just offset it:
N = 5
p = 0.5 #change this to bias your outputs
x_is_1 = torch.rand(N)>p #decide if x is going to be 1 or 0
y_is_1 = torch.rand(N)>p #decide if y is going to be 1 or 0
not_all_0 = ~(x_is_1 & y_is_1) #get rid of the x ^ y ^ z = 0 elements
x_is_1,y_is_1 = x_is_1[not_all_0],y_is_1[not_all_0]
N = x_is_1.shape[0]
x = torch.rand(N) * 0.3
x = torch.where(x_is_1,x+0.7,x)
y = torch.rand(N) * 0.3
y = torch.where(y_is_1,y+0.7,y)
z = torch.logical_xor(x_is_1,y_is_1).float()
triple_xor = 1 - torch.rand(z.shape)*0.3
print(torch.stack([x,y,z,triple_xor]).T)
       #x        y       z       x^y^z
tensor([[0.2615, 0.7676, 1.0000, 0.8832],
    [0.9895, 0.0370, 1.0000, 0.9796],
    [0.1406, 0.9203, 1.0000, 0.9646],
    [0.1799, 0.9722, 1.0000, 0.9327]])

Or, to treat your data as the basis (for more complex data), there is a preprocessing tool known as gaussian noise injection which seems to be what you're after. Or you can just define a function and call it a bunch of times.
def add_noise(x,y,z,triple_xor,range=0.3):
     def proc(dat,range):
        return torch.where(dat>0.5,torch.rand(dat.shape)*range+1-range,torch.rand(dat.shape)*range)
     return proc(x,range),proc(y,range),proc(z,range),proc(triple_xor,range)

gen_new_data = torch.cat([torch.stack(add_noise(x,y,z,triple_xor)).T for _ in range(5)])


Instruction: __init__() got an unexpected keyword argument 'chi1'

Input:
I am new in python. I am trying to import arguments into my class "inverse_model" . I call a function "get_models" to do that. But it gives me the error "init() got an unexpected keyword argument 'zz':
I appreciate help. Please see code below:
    def get_models(args):

    zz=torch.tensor(args.chi_Initialize)
    inverse_net = inverse_model(in_channels=len(args.chi),zz=zz,resolution_ratio=args.resolution_ratio,nonlinearity=args.nonlinearity)

    return inverse_net

class inverse_model(nn.Module):
    def __init__(self, in_channels,zz,resolution_ratio=6,nonlinearity="tanh"):
        super(inverse_model, self).__init__()
        self.in_channels = in_channels
        self.zz=zz
        self.resolution_ratio = resolution_ratio #vertical scale mismtach between seismic and EI
        self.activation =  nn.ReLU() if nonlinearity=="relu" else nn.Tanh()

Output:
After default argument, python does not allow to have non default argument.
Modify your constructor to
def __init__(self, in_channels,
                 chi1,chi2,chi3,chi4,chi5,chi6,chi7,chi8,chi9,
                 chi10,chi11,chi12, resolution_ratio=6,nonlinearity="tanh"):

Updated answer : Revision 1 (updated question)
import torch
from torch import nn

class inverse_model(nn.Module):
    def __init__(self, in_channels, zz, resolution_ratio=6, nonlinearity="tanh"):
        super(inverse_model, self).__init__()
        self.in_channels = in_channels
        self.zz = zz
        self.resolution_ratio = resolution_ratio  # vertical scale mismtach between seismic and EI
        self.activation = nn.ReLU() if nonlinearity == "relu" else nn.Tanh()

def get_models(args):
    zz = torch.tensor(args.chi_Initialize)
    inverse_net = inverse_model(in_channels=len(args.chi), zz=zz, resolution_ratio=args.resolution_ratio,
                                nonlinearity=args.nonlinearity)

    return inverse_net

It returns exit 0 as status.


Instruction: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same at summary() method call

Input:
I am trying to implement this program which is available on github to display the summary of models in Pytorch: :
import torch as th
from torch.autograd import Variable
from torch import nn

from collections import OrderedDict

def summary(input_size, model):
    def register_hook(module):
        def hook(module, input, output):
            class_name = str(module.__class__).split('.')[-1].split("'")[0]
            module_idx = len(summary)

            m_key = '%s-%i' % (class_name, module_idx+1)
            summary[m_key] = OrderedDict()
            summary[m_key]['input_shape'] = list(input[0].size())
            summary[m_key]['input_shape'][0] = -1
            summary[m_key]['output_shape'] = list(output.size())
            summary[m_key]['output_shape'][0] = -1

            params = 0
            if hasattr(module, 'weight'):
                params += th.prod(th.LongTensor(list(module.weight.size())))
                if module.weight.requires_grad:
                    summary[m_key]['trainable'] = True
                else:
                    summary[m_key]['trainable'] = False
            if hasattr(module, 'bias'):
                params +=  th.prod(th.LongTensor(list(module.bias.size())))
            summary[m_key]['nb_params'] = params

        if not isinstance(module, nn.Sequential) and \
             not isinstance(module, nn.ModuleList) and \
             not (module == model):
            hooks.append(module.register_forward_hook(hook))

    dtype = th.cuda.FloatTensor

    # check if there are multiple inputs to the network
    if isinstance(input_size[0], (list, tuple)):
        x = [Variable(th.rand(1,*in_size)).type(dtype) for in_size in input_size]
    else:
        x = Variable(th.rand(1,*input_size)).type(dtype)

    print(x.shape)
    print(type(x[0]))

    # create properties
    summary = OrderedDict()
    hooks = []
    # register hook
    model.apply(register_hook)
    # make a forward pass
    model(x)
    # remove these hooks
    for h in hooks:
        h.remove()

    print('----------------------------------------------------------------')
    line_new = '{:>20}  {:>25} {:>15}'.format('Layer (type)', 'Output Shape', 'Param #')
    print(line_new)
    print('================================================================')
    total_params = 0
    trainable_params = 0
    for layer in summary:
        ## input_shape, output_shape, trainable, nb_params
        line_new = '{:>20}  {:>25} {:>15}'.format(layer, str(summary[layer]['output_shape']), summary[layer]['nb_params'])
        total_params += summary[layer]['nb_params']
        if 'trainable' in summary[layer]:
            if summary[layer]['trainable'] == True:
                trainable_params += summary[layer]['nb_params']
        print(line_new)
    print('================================================================')
    print('Total params: ' + str(total_params))
    print('Trainable params: ' + str(trainable_params))
    print('Non-trainable params: ' + str(total_params - trainable_params))
    print('----------------------------------------------------------------')
    return summary

  To test it you can use a ready model from the library torchvision, like this (such as resnext50_32x4d)
import torchvision.models as models

resnext50_32x4d = models.resnext50_32x4d(pretrained=True)

   But when calling the function
summary((3, 300, 300, resnext50_32x4d)

   I get the following.
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Output:
The error message you provided is quite clear about the issue.

Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) ..

Your input is on GPU but your weight (i.e. model) is not. So, just put your model on GPU
resnext50_32x4d = resnext50_32x4d.cuda()


Instruction: Implementing custom learning rate scheduler in Pytorch?

Input:
I would like to implement this learning rate method as in the paper Attention is all you need. I have this code in Tensorflow, but I would like to implement it in Pytorch too. I know that Pytorch has modules for this (https://pytorch.org/docs/stable/_modules/torch/optim/lr_scheduler.html), but how could I go about making a custom scheduler? Or perhaps one of the above lr_scheduler already fulfils the same function?
Tensorflow code:
class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
  def __init__(self, d_model, warmup_steps=4000):
    super(CustomSchedule, self).__init__()

    self.d_model = d_model
    self.d_model = tf.cast(self.d_model, tf.float32)

    self.warmup_steps = warmup_steps

  def __call__(self, step):
    arg1 = tf.math.rsqrt(step)
    arg2 = step * (self.warmup_steps ** -1.5)

    return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)

learning_rate = CustomSchedule(d_model)

optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98,
                                     epsilon=1e-9)

Pytorch?
import torch

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)

scheduler =

Output:
Since this is a scheduler used in a popular paper (Attention is all you need), reasonably good implementations already exist online.
You can grab a PyTorch implementation from this repository by @jadore801120.
Once you have it, then simply
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)
sched = ScheduledOptim(optimizer, d_model=..., n_warmup_steps=...)

also make sure to invoke the scheduler at the right time
for i, batch in enumerate(dataloader):
    sched.zero_grad()
    ...
    loss.backward()
    sched.step_and_update_lr()


Instruction: Model parallelism, CUDA out of memory in Pytorch

Input:
I am trying to  build autoencoder model, where input/output is RGB images with size of 256 x 256. I tried to train model on 1 GPU with 12 GB of memory but I always caught CUDA OOM (I tried differen batchsizes and even batch size of 1 is failing). So I read about model parallelism in Pytorch and tried this:
class Autoencoder(nn.Module):
    def __init__(self, input_output_size):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_output_size, 1024),
            nn.ReLU(True),
            nn.Linear(1024, 200),
            nn.ReLU(True)
            ).cuda(0)

        self.decoder = nn.Sequential(
           nn.Linear(200, 1024),
           nn.ReLU(True),
           nn.Linear(1024, input_output_size),
           nn.Sigmoid()).cuda(1)

        print(self.encoder.get_device())
        print(self.decoder.get_device())

    def forward(self, x):
        x = x.cuda(0)
        x = self.encoder(x)
        x = x.cuda(1)
        x = self.decoder(x)
        return x

So I have moved my encoder and decoder on different GPUs. But now I get this exception:
Expected tensor for 'out' to have the same device as tensor for argument #2 'mat1'; but device 0 does not equal 1 (while checking arguments for addmm)

It appear when I do x = x.cuda(1) in forward method.
Moreover, here is my "train" code, maye you can give me some advices about optimizations? Is images of 3 x 256 x 256 too large for training? (I cannot reduce them). Thank you in advance.
Training:
input_output_size = 3 * 256 * 256
model = Autoencoder(input_output_size).to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
criterion = nn.MSELoss()

for epoch in range(100):
    epoch_loss = 0
    for batch_idx, (images, _) in enumerate(dataloader):
        images = torch.flatten(images, start_dim=1).to(device)
        output_images = model(images).to(device)
        train_loss = criterion(output_images, images)

        train_loss.backward()
        optimizer.step()

        if batch_idx % 5 == 0:
            with torch.no_grad():
                model.eval()
                pred = model(test_set).to(device)
                model.train()

                test_loss = criterion(pred, test_set)

                wandb.log({"MSE train": train_loss})
                wandb.log({"MSE test": test_loss})
                del pred, test_loss

        if batch_idx % 200 == 0:
            # here I send testing images from output to W&B
            with torch.no_grad():
                model.eval()
                pred = model(test_set).to(device)
                model.train()
                wandb.log({"PRED": [wandb.Image((pred[i].cpu().reshape((3, 256, 256)).permute(1, 2, 0) * 255).numpy().astype(np.uint8), caption=str(i)) for i in range(20)]})
                del pred
        gc.collect()
        torch.cuda.empty_cache()
        epoch_loss += train_loss.item()
        del output_images, train_loss
    epoch_loss = epoch_loss / len(dataloader)
    wandb.log({"Epoch MSE train": epoch_loss})
    del epoch_loss

Output:
Three issues that I'm seeing:

model(test_set)

This is when you are sending the entirety of your test set (presumably huge) as a single batch through your model.

I don't know what wandb is, but another likely source of memory growth is these lines:
wandb.log({"MSE train": train_loss})
wandb.log({"MSE test": test_loss})

You seem to be saving train_loss and test_loss, but these contain not only the numbers themselves, but the computational graphs (living on the GPU) needed for backprop. Before saving them, you want to convert them into float or numpy.

Your model contains two 3*256*256 x 1024 weight blocks. When used in Adam, these will require 3*256*256 x 1024 * 3 * 4 bytes = 2.25GB of VRAM each (Possibly more, if it's inefficiently implemented) This looks like a poor architecture for other reasons also.


Instruction: Using Pytorch model trained on RTX2080 on RTX3060

Input:
I try to run my PyTorch model (trained on a Nvidia RTX2080) on the newer Nvidia RTX3060 with CUDA support. It is possible to load the model and to execute it. If I run it on the CPU with the --no_cuda flag it runs smootly and gives back the correct predictions, but if I want to run it with CUDA, it only returns wrong predictions which make no sense.
Does the different GPU-architecture of the cards affect the prediction?

Output: Ok it seemed that the problem was the different floating point of the two architectures. The flag torch.backends.cuda.matmul.allow_tf32 = false needs to be set, to provide a stable execution of the model of a different architecture.


Instruction: How does grad() works in PyTorch?

Input:
I need some conceptual clarity with the inputs of the Pytorch grad() function.
Please see the following code:
import torch
a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
Q = 1/3*a**3 - 1/2*b**2

Here, I have defined 3 tensors, I was trying to compute the derivative of Q w.r.t a.
The following line would simply compute the first and second derivatives.
Q_a = torch.autograd.grad(Q.sum(), a, create_graph=True)[0]
Q_aa = torch.autograd.grad(Q_a.sum(), a, create_graph=True)[0]
print('Q_a =',Q_a.detach().numpy())
print('Q_aa =',Q_aa.detach().numpy())

The output is:
Q_a = [4. 9.]
Q_aa = [4. 6.]

I am wondering that, why do I need to pass Q.sum() or Q_a.sum() which is just 1 value and the second argument a has two values.
>>> print(Q.sum())
>>> tensor(-14.3333, grad_fn=<SumBackward0>)
>>> print(a)
>>> tensor([2., 3.], requires_grad=True)

Can someone explain to me how does Q.sum() helps in computing the correct gradient. Is it possible to compute the derivatives with just Q not Q.sum()?

Output:
Well, your question is based on wrong assumption. You said ..

I was trying to compute the derivative of Q w.r.t a

NO. you are not. In the code sample you provided, you are trying to compute the derivative of Q.sum() w.r.t a - they are different things.
"Derivative of Q w.r.t a" is matrix called Jacobian, whereas ..
"Derivative of Q.sum() w.r.t a" is a vector known as gradient.
Both can be computed and are used in different places for achieving different things. Its your decision which one you want.


Instruction: Pytorch: using CUDA prevents optimization from working

Input:
I have a very simple optimization: a straight line. Here is the code:
use_gpu = torch.cuda.is_available()
learning_rate = 0.05
loss_function = nn.MSELoss()

train_inputs = torch.FloatTensor([1,2,3,4,5,6]).T.unsqueeze(0)
y_truth = torch.FloatTensor([10, 15, 20, 25, 30, 35]).unsqueeze(0)

W = torch.nn.Parameter(torch.rand(1), requires_grad=True)
b = torch.nn.Parameter(torch.rand(1), requires_grad=True)
optimizer = optim.Adam([b, W], lr=learning_rate)

# if use_gpu:
#     y_truth = y_truth.cuda()
#     W = W.cuda()
#     b = b.cuda()
#     train_inputs = train_inputs.cuda()

for epoch in range(1000):
    optimizer.zero_grad()
    y_preds = b + W * train_inputs
    loss = loss_function(y_truth, y_preds)
    loss.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print(loss.data, W.data, b.data)

That code works fine if I do not put the data on the GPU. If I uncomment the if use_gpu bloc, the code runs, but does not minimize anything and the variables do not update.
I would expect the code to work similarly on the GPU or not. Any idea what is happening?
Thanks!

Output:
Any idea what is happening?

Yes, the parameters you are training, W and b stayed on the host (CPU).
When you did
W = W.cuda()
b = b.cuda()

you just chose to ignore the actual parameters being optimized.

If you wish to use the GPU for this, you could try:
W = torch.nn.Parameter(torch.rand(1).cuda())
b = torch.nn.Parameter(torch.rand(1).cuda())

instead.


Instruction: How to add a dropout layer in LSTM to avoid overfitting

Input:
While implementing a hybrid quantum LSTM model, the model is overfitting and thus giving low accuracy. I tried setting dropout = 1 in nn.LSTM but no improvement. I have used a single hidden layer. How do I add the dropout layer to reduce overfitting?
Model parameters:
input_dim = 16
hidden_dim = 100
layer_dim = 1
output_dim = 1

Model class:
class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim

        self.layer_dim = layer_dim

        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, dropout=1, batch_first=True, )

        self.fc = nn.Linear(hidden_dim, output_dim)
        self.hybrid = Hybrid(qiskit.Aer.get_backend('qasm_simulator'), 100, np.pi / 2)

    def forward(self, x):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        x, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        x = self.fc(x[:, -1, :])
        x = self.hybrid(x)
        return T.cat((x, 1 - x), -1)

Output:
Pytorch's LSTM layer takes the dropout parameter as the probability of the layer having its nodes zeroed out. When you pass 1, it will zero out the whole layer. I assume you meant to make it a conventional value such as 0.3 or 0.5.
As @ayandas says above, too, it applies dropout to each layer except the last (see the link above), so it won't work for a single-layer LSTM. You can always apply your own dropout using  nn.dropout at the output of your LSTM layers if you wish.


Instruction: add a nn.Parameter tensor to the optim

Input:
let's say I have a network with a parameter tensor:
self.inital_value = nn.Parameter(torch.randn(1,96,4,4))

and two layers:
self.rgb_layers = nn.ModuleList([nn.Conv2d(96, 3, 1, 1, 0)]*2)

so the forward function is as follows:
v = torch.clone(self.inital_value)
for i in range(2):
    v = self.rgb_layers[i](v)
return v

now I define an optimiser. if I want to train only the first layer I can do:
opt = optim.RMSprop(model.rgb_layers[0].parameters(),lr = lr)

but if I want to train only the tensor self.inital_value and I do the same I get this error:
'Parameter' object has no attribute 'parameters'

how can I bypass this limitation?

Output:
Well, it takes a sequence of Parameters. And you have one Parameter, called self.initial_value.
So how about [self.initial_value] ?


Instruction: How to give single number parameter in PyTorch?

Input:
I want to use a single number as a parameter.
I tried to insert parameter as follows
self.p1 = nn.Parameter() in   def __init__(self, ...)
and self.p1 = 1.0 in def _initialize(self):
But I am getting the error
    raise TypeError("cannot assign '{}' as parameter '{}' "
TypeError: cannot assign 'float' as parameter 'p1' (torch.nn.Parameter or None expected)

How can I use a single number as a parameter in my model initialized with 1?

Output:
The argument passed to nn.Parameter should be a torch.Tensor:
>>> self.p1 = nn.Parameter(torch.tensor([1.0]))


Instruction: Pytorch equivalent of tensorflow keras StringLookup?

Input:
I'm working with pytorch now, and I'm missing a layer: tf.keras.layers.StringLookup that helped with the processing of ids. Is there any workaround to do something similar with pytorch?
An example of the functionality I'm looking for:
vocab = ["a", "b", "c", "d"]
data = tf.constant([["a", "c", "d"], ["d", "a", "b"]])
layer = tf.keras.layers.StringLookup(vocabulary=vocab)
layer(data)

Outputs:
<tf.Tensor: shape=(2, 3), dtype=int64, numpy=
array([[1, 3, 4],
       [4, 1, 2]])>

Output:
Package torchnlp,
pip install pytorch-nlp

from torchnlp.encoders import LabelEncoder

data = ["a", "c", "d", "e", "d"]
encoder = LabelEncoder(data, reserved_labels=['unknown'], unknown_index=0)

enl = encoder.batch_encode(data)

print(enl)

tensor([1, 2, 3, 4, 3])