Instruction: How to change PyTorch tensor into a half size and/or double size with different dimension?

Input:
I'm new to PyTorch and tensor data thing. I have a problem about switching shape of tensors.

I have two questions.

First, what should I do if I have a tensor with torch.Size([8, 512, 16, 16]) and I want to change it into torch.Size([8, 256, 32, 32]) which is double size of the original tensor.

Second, what should I do if I have a tensor with torch.Size([8, 256, 32, 32]) and I want to change it into torch.Size([8, 512, 16, 16]) which is half size of the original tensor.

In the first question, I've tried on ZeroPadding2D(8) function to reshape it into torch.Size([8, 512, 32, 32]) but I don't know how to change the 2nd dimension which is 512 into 256.

The actual usage in the first question is something like this.

x = input # torch.Size([8, 512, 16, 16])
x = layer(x) # torch.Size([8, 256, 32, 32]
x = x + input # what I want to do is adding tensor values before and after passing the layer together (like skip connection)

I expect the output of adding two tensors to be success but the actual output is an error about unequal size in dimensions

Output:
For the first case, use resize_() to change second dimension from 512 to 256 and then allocate a tensor with your padding value and the target dimensions and assign the portion for which you have data.

import torch

target_output = torch.zeros(8, 256, 32, 32)

in_tensor = torch.randn(8, 512, 16, 16)
out_temp = in_tensor.resize_((8, 256, 16, 16))

target_output[:, :, :16, :16] = out_temp
print(target_output.shape)

# output:
# torch.Size([8, 256, 32, 32])

You can also use torch.nn.ConstantPad2d() and then resize_() as below:

in_tensor = torch.randn(8, 512, 16, 16)
m = nn.ConstantPad2d((8, 8, 8, 8), 0)
out_tensor = m(in_tensor).resize_(8, 256, 16, 16)
print(out_tensor.shape)

# output:
# torch.Size([8, 256, 32, 32])

Alternatively, you can also use torch.nn.ConstantPad2d() and copy_() like below:

import torch.nn as nn
in_tensor = torch.randn(8, 512, 16, 16)  # shape [8, 512, 16, 16]
m = nn.ConstantPad2d((8, 8, 8, 8), 0)
temp = m(in_tensor)                      # shape [8, 512, 32, 32]
out_tensor = torch.zeros(8, 256, 32, 32) # shape [8, 256, 32, 32]
out_tensor = out_tensor[:,:,:,:].copy_(temp[:,:256,:,:]) # shape [8, 256, 32, 32]

You can read more about reshaping a tensor with padding in pytorch from here.
For the second case, you can simply use resize_() for resizing your tensor to half the size.

in_tensor = torch.randn(8, 256, 32, 32)
out_tensor = in_tensor.resize_(8, 512, 16, 16)
print(out_tensor.shape)

# output:
# torch.Size([8, 512, 16, 16])

Alternatively, you can use copy_ as below:

in_tensor = torch.randn(8, 256, 32, 32)
temp_tensor = torch.zeros(8, 512, 16, 16) # shape [8, 512, 16, 16]
temp_tensor[:,:256,:,:].copy_(in_tensor[:,:,:16,:16]) # shape [8, 512, 16, 16]
out_tensor = temp_tensor # shape [8, 512, 16, 16]

Without using copy_():

in_tensor = torch.randn(8, 256, 32, 32)
out_tensor = torch.zeros(8, 512, 16, 16) # shape [8, 512, 16, 16]
out_tensor[:,:256,:,:] = in_tensor[:,:,:16,:16]


Instruction: How to fix RuntimeError "Expected object of scalar type Float but got scalar type Double for argument"?

Input:
I'm trying to train a classifier via PyTorch. However, I am experiencing problems with training when I feed the model with training data.
I get this error on y_pred = model(X_trainTensor):

  RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

Here are key parts of my code:

# Hyper-parameters
D_in = 47  # there are 47 parameters I investigate
H = 33
D_out = 2  # output should be either 1 or 0

# Format and load the data
y = np.array( df['target'] )
X = np.array( df.drop(columns = ['target'], axis = 1) )
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size = 0.8)  # split training/test data

X_trainTensor = torch.from_numpy(X_train) # convert to tensors
y_trainTensor = torch.from_numpy(y_train)
X_testTensor = torch.from_numpy(X_test)
y_testTensor = torch.from_numpy(y_test)

# Define the model
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
    nn.LogSoftmax(dim = 1)
)

# Define the loss function
loss_fn = torch.nn.NLLLoss()

for i in range(50):
    y_pred = model(X_trainTensor)
    loss = loss_fn(y_pred, y_trainTensor)
    model.zero_grad()
    loss.backward()
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad

Output:
Reference is from this github issue.

When the error is RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1', you would need to use the .float() function since it says Expected object of scalar type Float.

Therefore, the solution is changing y_pred = model(X_trainTensor) to y_pred = model(X_trainTensor.float()).

Likewise, when you get another error for loss = loss_fn(y_pred, y_trainTensor), you need y_trainTensor.long() since the error message says Expected object of scalar type Long.

You could also do model.double(), as suggested by @Paddy
.


Instruction: PyTorch - Getting the 'TypeError: pic should be PIL Image or ndarray. Got ' error

Input:
I am getting the error TypeError: pic should be PIL Image or ndarray. Got <class 'numpy.ndarray'> when I try to load a non-image dataset through the DataLoader. The versions of torch and torchvision are 1.0.1, and 0.2.2.post3, respectively. Python's version is 3.7.1 on a Windows 10 machine.

Here is the code:

class AndroDataset(Dataset):
    def __init__(self, csv_path):
        self.transform = transforms.Compose([transforms.ToTensor()])

        csv_data = pd.read_csv(csv_path)

        self.csv_path = csv_path
        self.features = []
        self.classes = []

        self.features.append(csv_data.iloc[:, :-1].values)
        self.classes.append(csv_data.iloc[:, -1].values)

    def __getitem__(self, index):
        # the error occurs here
        return self.transform(self.features[index]), self.transform(self.classes[index])

    def __len__(self):
        return len(self.features)

And I set the loader:

training_data = AndroDataset('android.csv')
train_loader = DataLoader(dataset=training_data, batch_size=batch_size, shuffle=True)

Here is the full error stack trace:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1758, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2018.1.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 231, in <module>
    main()
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 149, in main
    for i, (images, labels) in enumerate(train_loader):
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torch\utils\data\dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torch\utils\data\dataloader.py", line 615, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:/Users/talha/Documents/PyCharmProjects/DeepAndroid/deep_test_conv1d.py", line 102, in __getitem__
    return self.transform(self.features[index]), self.transform(self.classes[index])
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\transforms.py", line 60, in __call__
    img = t(img)
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\transforms.py", line 91, in __call__
    return F.to_tensor(pic)
  File "C:\Users\talha\Documents\PyCharmProjects\DeepAndroid\venv\lib\site-packages\torchvision\transforms\functional.py", line 50, in to_tensor
    raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'numpy.ndarray'>

Output:
Expanding on @MiriamFarber's answer, you cannot use transforms.ToTensor() on numpy.ndarray objects. You can convert numpy arrays to torch tensors using torch.from_numpy() and then cast your tensor to the required datatype.

Eg:

>>> import numpy as np
>>> import torch
>>> np_arr = np.ones((5289, 38))
>>> torch_tensor = torch.from_numpy(np_arr).long()
>>> type(np_arr)
<class 'numpy.ndarray'>
>>> type(torch_tensor)
<class 'torch.Tensor'>


Instruction: PyTorch DataLoader - "IndexError: too many indices for tensor of dimension 0"

Input:
I am trying to implement a CNN to identify digits in the MNIST dataset and my code comes up with the error during the data loading process. I don't understand why this is happening.

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5), (0.5))
])

trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=20, shuffle=True, num_workers=2)

testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=20, shuffle=False, num_workers=2)

for i, data in enumerate(trainloader, 0):
    inputs, labels = data[0], data[1]

Error:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-6-b37c638b6114> in <module>
      2
----> 3     for i, data in enumerate(trainloader, 0):
      4         inputs, labels = data[0], data[1]

# ...

IndexError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torchvision/datasets/mnist.py", line 95, in __getitem__
    img = self.transform(img)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 164, in __call__
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "/opt/conda/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 208, in normalize
    tensor.sub_(mean[:, None, None]).div_(std[:, None, None])
IndexError: too many indices for tensor of dimension 0

Output:
The problem is that the mean and std have to be sequences (e.g., tuples), therefore you should add a comma after the values:

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5,), (0.5,))
])

Note the difference between (0.5) and (0.5,). You can check here how these values are used. If you apply the same process you'll see that:

import torch

x1 = torch.as_tensor((0.5))
x2 = torch.as_tensor((0.5,))

print(x1.shape, x1.ndim)  # output: torch.Size([]) 0
print(x2.shape, x2.ndim)  # output: torch.Size([1]) 1

Maybe you don't know, but they are also different in Python:

type((0.5))   # <type 'float'>
type((0.5,))  # <type 'tuple'>


Instruction: Perform max pooling on Integer tensor in Pytorch

Input:
I am defining a max pool layer and passing in an integer tensor as follows.

max_pool = nn.MaxPool2d(3, stride=2)
max_pool(torch.IntTensor(3,5,5).random_(0, 10))

It throws the following error:

RuntimeError: _thnn_max_pool2d_with_indices_forward not supported on CPUType for Int

Output:
As the error message suggests, nn.MaxPoll2d only supports floating point input tensors.
You'll need to cast your input int tensor to torch.float before applying the pooling.


Instruction: How to vectorize the sum? tensor[i,:,:,:] + tensor[i]

Input:
I want to vectorize the following code:

def style_noise(self, y, style):
    n = torch.randn(y.shape)
    for i in range(n.shape[0]):
        n[i] = (n[i] - n.mean(dim=(1, 2, 3))[i]) * style.std(dim=(1, 2, 3))[i] / n.std(dim=(1, 2, 3))[i] + style.mean(dim=(1, 2, 3))[i]
    noise = Variable(n, requires_grad=False).to(y.device)
    return noise

I didn't find a way nice way of doing so.

y and style are 4d tensors, say  style.shape = y.shape = [64, 3, 128, 128].

I want to return the noise tensor, noise.shape = [64, 3, 128, 128].

Please let me know in the comments if the question is not clear.

Output:
Your use case is exactly why the .mean and .std methods come with a keepdim parameter. You can make use of this to enable broadcasting semantics to vectorize things for you:

def style_noise(self, y, style):
    n = torch.randn(y.shape)
    n_mean = n.mean(dim=(1, 2, 3), keepdim=True)
    n_std = n.std(dim=(1, 2, 3), keepdim=True)
    style_mean = style.mean(dim=(1, 2, 3), keepdim=True)
    style_std = style.std(dim=(1, 2, 3), keepdim=True)
    n = (n - n_mean) * style_std / n_std + style_mean
    noise = Variable(n, requires_grad=False).to(y.device)
    return noise


Instruction: Where do I get Visual Studio 15.4 that I need for pyTorch 0.4.1

Input:
I have a problem with the following dependency chain:

MedicalDetectionToolkit (MDT) needs pyTorch 0.4.1 since in pyTorch 1.0 torch.utils.ffi is depricated
pyTorch 0.4.1 needs CUDA 9.0 (does not work with 10.0)
CUDA 9.0 does not work with Visual Studio > 15.4
Visual Studio 15.4 is not available anywhere, the MSVS installer will install 15.9

The Visual Studio Installer allows to install the "Compiler Tools 15.4" side-by.side, but MDT uses torch.utils.ffi at some point, which just uses the 15.9.

So

How do I install Visual Studio 15.4?

Or

How do I tell torch.utils.ffi to use the 15.4 compiler?

(MedicalDetectionToolkit will not be updated by the author. I may try it, but would like to have a working version first.)

Output:
There are the "Build Tools for Visual Studio 2017 (version 15.0)" available on
https://my.visualstudio.com/Downloads?q=Visual%20Studio%202017

You may first have to join the "Essentials" program. After joining the downloads became visible.


Instruction: How to permanently install package in GoogleColab using conda?

Input:
I am trying yo use a PyTorch library SparseConvNet (https://github.com/facebookresearch/SparseConvNet) in Google Colaboratory. In order to install it properly, you need to first install Conda, and then using Conda install the SparseConvNet package. Here is the code I am using (following the instructions from scn readme file):

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local

import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')

!conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
!conda install google-sparsehash -c bioconda
!conda install -c anaconda pillow

!git clone https://github.com/facebookresearch/SparseConvNet.git
!cd SparseConvNet/
!bash develop.sh

When I run this it is working and I can successfully import sparseconvnet package, but I need to do it every time I enter the notebook or restart  runtime, and it's taking a lot of time. Is it possible to install these packages permanently?

There is one similar question, and the answer suggest that I should install it
on my drive, but I don't know how to do it using conda.
Thanks!

Output:
You can specify the directory for conda to install to using

conda install -p path_to_your_dir

So, you can mount your google drive and conda install there to make it permanent.


Instruction: Backprop through Pytorch Element-Wise Operation if input contains NaNs

Input:
Suppose I'd like to compute the element-wise quotient between two tensors. If one of the tensors contains NaNs, the resulting quotient will also contain NaNs, and this I understand. But why does the gradient becomes non-existent for the entire operation? And how can I preserve the gradient for the non-NaN entries?

For example:

>>> x = torch.tensor([1.0, np.NaN])
>>> y = torch.tensor([2.0, 3.0])
>>> z = torch.div(y, x)
>>> z
tensor([2., nan])
>>> z.backward()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Output:
There are a lot of mistakes in your code, which I hereby address, in hope to enlighten you :)

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn is NOT because of the NaN. It's because none of your input variables require a gradient, therefore z doesn't have the possibility to call backward(). You need to start the backpropagation tree somewhere.
You can't just .backward() on a tensor with multiple values. You need to sum() first, or some similar operation, but that would produce NaN in your case. You can backpropagate both parts of the z separately, by calling .backward(torch.Tensor([1.0,1.0])).

Therefore, if you fix all the bugs, it should work:

import torch
import numpy as np

x = torch.tensor([1.0, np.NaN], requires_grad=True)
y = torch.tensor([2.0, 3.0])
z = torch.div(y, x)

z.backward(torch.Tensor([1.0,1.0]))
print(x.grad)

tensor([-2., nan])


Instruction: How to train the original U-Net model with PyTorch?

Input:
I’m trying to implement and train the original U-Net model, but I’m stuck in when I’m trying to train the model using the ISBI Challenge Dataset.

According with the original U-Net model, the network outputs an image with 2 channels and size of 388 x 388. So, my data loader for training generates a tensor with size of [batch, channels=1, width=572, height=572] for the input images and [batch, channels=2, width=388, width=388] for target/output images.

My problem actually is that when I’m trying to use the nn.CrossEntropyLoss() the following error is raised:

  RuntimeError: invalid argument 3: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4 at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THNN/generic/SpatialClassNLLCriterion.c:59

I’m just starting with PyTorch (newbie here)… so, I’ll really appreciate if someone could help me to overcome this problem.

The sourcecode is available on GitHub:

https://github.com/dalifreire/cnn_unet_pytorch
https://github.com/dalifreire/cnn_unet_pytorch/blob/master/unet_pytorch.ipynb

Best regards!

UPDATE

I just remove the channel dimension from my masks and everything works well… now I’m generating masks with the shape 1 [width=388, height=388].

After that, I’m working with input images (X), target masks (y) and predicted output masks (y_hat) as follow:

X     --> torch.Size([10, 1, 572, 572])
y     --> torch.Size([10, 388, 388])
y_hat --> torch.Size([10, 2, 388, 388])

But, I don’t understand why target masks (y) and predicted masks (y_hat) must have different shapes? It’s so weird for me…

Output:
From the CrossEntropyLoss docstring of PyTorch:

Shape:

    - Input: :math:`(N, C)` where `C = number of classes`, or
      :math:`(N, C, d_1, d_2, ..., d_K)` with :math:`K \geq 1`
      in the case of `K`-dimensional loss.

    - Target: :math:`(N)` where each value is :math:`0 \leq \text{targets}[i] \leq C-1`, or
      :math:`(N, d_1, d_2, ..., d_K)` with :math:`K \geq 1` in the case of
      K-dimensional loss.

    - Output: scalar.
      If :attr:`reduction` is ``'none'``, then the same size as the target:
      :math:`(N)`, or
      :math:`(N, d_1, d_2, ..., d_K)` with :math:`K \geq 1` in the case
      of K-dimensional loss.

If your targets contain the class indices already, you should remove the channel dimension.

Source


Instruction: AWS tensorboard Segmentation fault (core dumped)

Input:
I am trying to use tensorboardX to debug a pytorch NN that is running in a p2.xlarge instance of AWS.

I followed this tutorial to open the port 6006.

The model is running and tensorboardX is making its writer file. I get the following warning there. I am not sure how relevant it is.

  WARNING:root:tuple appears in op that does not forward tuples
  (VisitNode at /pytorch/torch/csrc/jit/passes/lower_tuples.cpp:117)
  frame #0: std::function::operator()() const + 0x11
  (0x7fbe3dd04441 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) +
  0x2a (0x7fbe3dd03d7a in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #2:  + 0xaf61f5 (0x7fbe3cdc41f5 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #3:  + 0xaf6464 (0x7fbe3cdc4464 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #4:
  torch::jit::LowerAllTuples(std::shared_ptr&) + 0x13
  (0x7fbe3cdc44a3 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #5:  + 0x3f84b4 (0x7fbe7d2cb4b4 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  frame #6:  + 0x130cfc (0x7fbe7d003cfc in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
   frame #40: __libc_start_main + 0xf0
  (0x7fbe8d69c830 in /lib/x86_64-linux-gnu/libc.so.6)

  WARNING:root:tuple appears in op that does not forward tuples
  (VisitNode at /pytorch/torch/csrc/jit/passes/lower_tuples.cpp:117)
  frame #0: std::function::operator()() const + 0x11
  (0x7fbe3dd04441 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) +
  0x2a (0x7fbe3dd03d7a in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libc10.so)
  frame #2:  + 0xaf61f5 (0x7fbe3cdc41f5 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #3:  + 0xaf6464 (0x7fbe3cdc4464 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #4:
  torch::jit::LowerAllTuples(std::shared_ptr&) + 0x13
  (0x7fbe3cdc44a3 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch.so.1)
  frame #5:  + 0x3f84b4 (0x7fbe7d2cb4b4 in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
  frame #6:  + 0x130cfc (0x7fbe7d003cfc in
  /home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
   frame #40: __libc_start_main + 0xf0
  (0x7fbe8d69c830 in /lib/x86_64-linux-gnu/libc.so.6)

The problem is that I don't have access to the tensorboard browser user interface. I take the following steps:

$ cd PATH_TO_FOLDER_CONTAINING_runs
$ source activate pytorch_p36
$ tensorboard --logdir=runs

Where I get the error message:

  Segmentation fault (core dumped)

When I check the syslog var/log/syslog I see that following:

  Jun 26 09:06:40 ip-172-xx-xx-xxx kernel: [515315.598917] tensorboard[1446]: segfault at 0 ip           (null) sp 00007ffd64c5f178 error 14 in python2.7[55d8673d1000+1000]

My googling skills were far from enough. How can I access tensorboard through the browser with it running in the ASW instance?

Please let me know if something is unclear or if some info is missing.

Output:
Even though the code has to run in the environment pytorch_p36, tensorboard actually has to run on a different environment.

The sequence of commands in the terminal should be:

$ cd PATH_TO_FOLDER_CONTAINING_runs
$ source activate tensorflow_p27
$ tensorboard --logdir=runs

Then the designated port opens.


Instruction: Modifying the forward function in vgg model

Input:
I need to modify the existing forward method in VGG16 so that it can  pass through the two classifiers and return the value

I tried creating the custom forward method manually and overriding the existing method but I get the following error

vgg.forward = forward

forward() missing 1 required positional argument: 'x'

My custom forward function

def forward(self,x):
    x = self.features(x)
    x = self.avgpool(x)
    x = x.view(x.size(0), -1)
    x = self.classifier(x)
    y = self.classifier_2(x)
    return x,y

I have modified the default vgg16_bn with one additional classifier as

vgg = models.vgg16_bn()
final_in_features = vgg.classifier[6].in_features
mod_classifier = list(vgg.classifier.children())[:-1]
mod_classifier.extend([nn.Linear(final_in_features, 10)])
vgg.add_module('classifier_2',vgg.classifier)

My model looks like this after the addition of above classifier

(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=10, bias=True)
  )
  (classifier_2): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=10, bias=True)
  )

My convolutional layers results are supposed to be passed through two separate FFN layers. So how do i modify my forward pass

Output:
I think the best way to achieve what you want is to create a new model extending the nn.Module. I'd do something like:
from torchvision import models
from torch import nn

class MyVgg (nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        vgg = models.vgg16_bn(pretrained=True)

        # Here you get the bottleneck/feature extractor
        self.vgg_feature_extractor = nn.Sequential(*list(vgg.children())[:-1])

        # Now you can include your classifiers
        self.classifier1 = nn.Sequential(layers1)
        self.classifier2 = nn.Sequential(layers2)

    # Set your own forward pass
    def forward(self, img, extra_info=None):

        x = self.vgg_convs(img)
        x = x.view(x.size(0), -1)
        x1 = self.classifier1(x)
        x2 = self.classifier2(x)

        return x1, x2


Instruction: How to train a CNN model?

Input:
When trying to train the CNN model, I came across a code shown below:

def train(n_epochs, loaders, model, optimizer, criterion):

    for epoch in range(1,n_epochs):
        train_loss = 0
        valid_loss = 0

        model.train()
        for i, (data,target) in enumerate(loaders['train']):

            # zero the parameter (weight) gradients
            optimizer.zero_grad()
            # forward pass to get outputs
            output = model(data)
            # calculate the loss
            loss = criterion(output, target)
            # backward pass to calculate the parameter gradients
            loss.backward()
            # update the parameters
            optimizer.step()

Can someone please tell me why is the second for loop used?
i.e; for i, (data,target) in enumerate(loaders['train']):

And why optimizer.zero_grad() and optimizer.step() is used?

Output:
torch.utils.data.DataLoader comes in handy when you need to prepare data batches (and perhaps shuffle them before every run).

data_train_loader = DataLoader(data_train, batch_size=64, shuffle=True)

In the above code, first for-loop iterates through the number of epochs while second loop iterates through the training dataset converted into batches via above code. For example:

for batch_idx, samples in enumerate(data_train_loader):
    # samples will be a 64 x D dimensional tensor
    # batch_idx is each batch index

Learn more about torch.utils.data.DataLoader from here.
Optimizer.zero_gradient(): Before the backward pass, use the optimizer object to zero all of the gradients for the tensors it will update (which are the learnable weights of the model)
optimizer.step(): We generally use optimizer.step() to make the gradient descent step. Calling the step function on an Optimizer makes an update to its parameters.

Learn more about these from here.


Instruction: RuntimeError: The size of tensor a (133) must match the size of tensor b (10) at non-singleton dimension 1

Input:
I am training a CNN model. I am facing issue while doing the training iteration for my model. The code is as below:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        #convo layers
        self.conv1 = nn.Conv2d(3,32,3)
        self.conv2 = nn.Conv2d(32,64,3)
        self.conv3 = nn.Conv2d(64,128,3)
        self.conv4 = nn.Conv2d(128,256,3)
        self.conv5 = nn.Conv2d(256,512,3)

        #pooling layer
        self.pool = nn.MaxPool2d(2,2)

        #linear layers
        self.fc1 = nn.Linear(512*5*5,2048)
        self.fc2 = nn.Linear(2048,1024)
        self.fc3 = nn.Linear(1024,133)

        #dropout layer
        self.dropout = nn.Dropout(0.3)
        def forward(self, x):
        #first layer
        x = self.conv1(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #second layer
        x = self.conv2(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #third layer
        x = self.conv3(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)
        #fourth layer
        x = self.conv4(x)
        x = F.relu(x)
        x = self.pool(x)
        #fifth layer
        x = self.conv5(x)
        x = F.relu(x)
        x = self.pool(x)
        #x = self.dropout(x)

        #reshape tensor
        x = x.view(-1,512*5*5)
        #last layer
        x = self.dropout(x)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.dropout(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)

        return x

        #loss func
        criterion = nn.MSELoss()
        optimizer = optim.Adam(net.parameters(), lr = 0.0001)
        #criterion = nn.CrossEntropyLoss()
        #optimizer = optim.SGD(net.parameters(), lr = 0.05)

        def train(n_epochs,model,loader,optimizer,criterion,save_path):
           for epoch in range(n_epochs):
              train_loss = 0
              valid_loss = 0
              #training
              net.train()
              for batch, (data,target) in enumerate(loaders['train']):
                   optimizer.zero_grad()
                   outputs = net(data)
                   #print(outputs.shape)
                   loss = criterion(outputs,target)
                   loss.backward()
                   optimizer.step()

When I use the CrossEntropy Loss function and SGD optimizer, I able able to train the model with no error.
When I use MSE loss function and Adam optimizer, I am facing the following error:

RuntimeError Traceback (most recent call last) <ipython-input-20-2223dd9058dd> in <module>
      1 #train the model
      2 n_epochs = 2
----> 3 train(n_epochs,net,loaders,optimizer,criterion,'saved_model/dog_model.pt')

<ipython-input-19-a93d145ef9f7> in train(n_epochs, model, loader, optimizer, criterion, save_path)
     22
     23             #calculate loss
---> 24             loss = criterion(outputs,target)
     25
     26             #backward prop

RuntimeError: The size of tensor a (133) must match the size of tensor b (10) at non-singleton dimension 1.

Does the selected loss function and optimizer effect the training of the model? Can anyone please help on this?

Output:
The error message clearly suggests that the error occurred at the line

loss = criterion(outputs,target)

where you are trying to compute the mean-squared error between the input and the target.
See this line: criterion = nn.MSELoss().

I think you should modify your code where you are estimating loss between (output, target) pair of inputs,i.e., loss = criterion(outputs,target) to something like below:

loss = criterion(outputs,target.view(1, -1))

Here, you are making target shape same as outputs from model on line

outputs = net(data)

One more think to notice here is the output of the net model, i.e., outputs will be of shape batch_size X output_channels, where batch size if the first dimension of input images as during the training you will get batches of images, so your shape in the forward method will get an additional batch dimension at dim0: [batch_size, channels, height, width], and ouput_channels is number of output features/channels from the last linear layer in the net model.

And, the the target labels will be of shape batch_size, which is 10 in your case, check batch_size you passed in torch.utils.data.DataLoader(). Therefore, on reshaping it using view(1, -1), it will be of converted into a shape 1 X batch_size, i.e., 1 X 10.

That's why, you are getting the error:

  RuntimeError: input and target shapes do not match: input [10 x 133],
  target [1 x 10]

So, a way around is to replace loss = criterion(outputs,target.view(1, -1)) with loss = criterion(outputs,target.view(-1, 1)) and change the output_channels of last linear layer to 1 instead of 133. In this way, both of outputs and target shape will be equal and we can compute MSE value then.

Learn more about pytorch MSE loss function from here.


Instruction: RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[4, 5000, 5000, 3] to have 3 channels, but got 5000 channels instead

Input:
So, I have a U-Net model and I'm feeding images of 5000x5000x3 into the model and I and getting the error above.

So here is my model.

import torch
import torch.nn as nn

def double_conv(in_channels, out_channels):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, 3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(out_channels, out_channels, 3, padding=1),
        nn.ReLU(inplace=True)
    )

class UNeT(nn.Module):
    def __init__(self, n_class):
        super().__init__()
        self.dconv_down1 = double_conv(3, 64)
        self.dconv_down2 = double_conv(64, 128)
        self.dconv_down3 = double_conv(128, 256)
        self.dconv_down4 = double_conv(256, 512)
        self.maxpool = nn.MaxPool2d(2)
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear',
                                    align_corners=True)
        self.dconv_up3 = double_conv(256 + 512, 256)
        self.dconv_up2 = double_conv(128 + 256, 128)
        self.dconv_up1 = double_conv(128 + 64, 64)
        self.conv_last = nn.Conv2d(64, n_class, 1)

    def forward(self, x):
        conv1 = self.dconv_down1(x)
        x = self.maxpool(conv1)
        conv2 = self.dconv_down2(x)
        x = self.maxpool(conv2)
        conv3 = self.dconv_down3(x)
        x = self.maxpool(conv3)
        x = self.dconv_down4(x)
        x = self.upsample(x)
        x = torch.cat([x, conv3], dim=1)
        x = self.dconv_up3(x)
        x = self.upsample(x)
        x = torch.cat([x, conv2], dim=1)
        x = self.dconv_up2(x)
        x = self.upsample(x)
        x = torch.cat([x, conv1], dim=1)
        x = self.dconv_up1(x)
        out = self.conv_last(x)
        return out

I tried to do model(inputs.unsqueeze_(0)) but I got a different error.

Output:
The order of dimensions in pytorch is different than what you expect. Your input tensor has shape of 4x5000x5000x3 which you interpret as a batch of size 4, with images of 5000x5000 pixels, each pixel has 3 channels. That is, your dimensions are batch-height-width-channel.

However, pytorch expects tensor dimensions to be in a different order: batch-channel-height-width. That is, the channel dimension should precede the width and height spatial dimensions.

You need to permute the dimensions of your input tensor to solve your problem:

model(inputs.permute(0, 3, 1, 2))

For more information, see the documentation of nn.Conv2d.


Instruction: I am trying to install intel optimized pytorch in different ways

Input:
I am very first time using pyTorch. I am trying to install it. In how many ways I can do this?
Please provide the steps for that.

Output:
You can install PyTorch in 3 ways.

Using pip
Using conda
From source

1.Intel Optimized Pytorch Installation

Install the stable version (v 1.0) on Linux via Pip for Python 3.6.

pip install https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl

pip install torchvision

2.Conda Pytorch Installation

conda install pytorch-cpu torchvision-cpu -c pytorch

3.PyTorch Installation from source

Create a new environment:

conda create -n <env_name> python=3.6

export CMAKE_PREFIX_PATH=/home/user/.conda/envs/<env_name>

source activate <env_name>

Install dependencies:

conda install numpy pyyaml mkl mkl-include setuptools cmake cffi typing

conda install -c conda-forge opencv

conda install Pillow

Get the PyTorch source:

git clone --recursive https://github.com/intel/pytorch

cd pytorch

mv caffe2/contrib/cuda-convnet2/ /tmp
# Fix the bug removing the old package out

Install PyTorch:

python setup.py install 2>&1 | tee build.out

Hope this will help you.


Instruction: How torch.Tensor.backward() works?

Input:
I recently study Pytorch and backward function of the package.
I understood how to use it, but when I try

x = Variable(2*torch.ones(2, 2), requires_grad=True)
x.backward(x)
print(x.grad)

I expect

tensor([[1., 1.],
        [1., 1.]])

because it is an identity function. However, it returns

tensor([[2., 2.],
        [2., 2.]]).

Why this happens?

Output:
Actually, this is what you are looking for:

Case 1: when z = 2*x**3 + x

import torch
from torch.autograd import Variable
x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x*x*x*2+x
z.backward(torch.ones_like(z))
print(x.grad)

output:

tensor([[25., 25.],
        [25., 25.]])

Case 2: when z = x*x

x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x*x
z.backward(torch.ones_like(z))
print(x.grad)

output:

tensor([[4., 4.],
        [4., 4.]])

Case 3: when z = x (your case)

x = Variable(2*torch.ones(2, 2), requires_grad=True)
z = x
z.backward(torch.ones_like(z))
print(x.grad)

output:

tensor([[1., 1.],
        [1., 1.]])

To learn more how to calculate gradient in pytorch, check this.


Instruction: Regarding odd image dimensions in Pytorch

Input:
So I am currently building a 2-channel (also called double channel) convolution neural network for measuring the similarity between 2 (binary) images.

The problem I am having is the following:

My input images are 40 x 50, and after 1 conv and 1 pooling layer (for example), the output size is 18 x 23. So how does one do more pooling without ending up with non-integer output sizes? For example, pooling a 18 x 23 image with size 2 x 2, the output size is given by 9 x 11.5.

I cannot seem to find any suitable kernel sizes to avoid such a problem, which in my opinion is a result of the fact that the original input image dimensions are not powers of 2. For example, input images of size 64 x 64 doesn't have this issue with the correct padding size and so on.

Any help is much appreciated.

Output:
Regarding your question:

  So how does one do more pooling without ending up with non-integer output sizes?

Let's say you have:

import torch
from torch import nn
from torch.nn import functional as F

# equivalent to your (18 x 23) activation volume
x = torch.rand(1, 1, 4, 3)

print(x)
# tensor([[[[0.5005, 0.3433, 0.5252],
#           [0.4878, 0.5266, 0.0237],
#           [0.8600, 0.8092, 0.8912],
#           [0.1623, 0.4863, 0.3644]]]])

If you apply pooling (I will use MaxPooling in this example and I assume you meant a 2x2 pooling with stride=2 based on your expected output shape):

p = nn.MaxPool2d(2, stride=2)
y = p(x)

print(y.shape)
# torch.Size([1, 1, 2, 1])

print(y)
# tensor([[[[0.5266],
#           [0.8600]]]])

If you would like to have a [1, 1, 2, 2], you can set the ceil_mode=True of MaxPooling:

p = nn.MaxPool2d(2, stride=2, ceil_mode=True)
y = p(x)

print(y.shape)
# torch.Size([1, 1, 2, 2])

print(y)
# tensor([[[[0.5266, 0.5252],
#           [0.8600, 0.8912]]]])

You can also pad the volume to achieve the same (here I assume the volume has min=0 as if it was after a ReLU):

p = nn.MaxPool2d(2, stride=2)
y = p(F.pad(x, (0, 1), "constant", 0))

print(y.shape)
# torch.Size([1, 1, 2, 2])

print(y)
# tensor([[[[0.5266, 0.5252],
#           [0.8600, 0.8912]]]])

Regarding:

  I cannot seem to find any suitable kernel sizes to avoid such a problem, which in my opinion is a result of the fact that the original input image dimensions are not powers of 2.

Well, if you want to use Pooling operations that change the input size in half (e.g., MaxPooling with kernel=2 and stride=2), then using an input with a power of 2 shape is quite convenient (after all, you'll be able to do many of these /2 operations). However, this is not required. You can change the stride of the pooling, you can always pool with ceil_mode=True, you can also pad asymmetrically, and many other things. All of them are decisions you'll have to make when building your model :)


Instruction: Regarding the consistency of convolutional neural networks

Input:
I am currently building a 2-channel (also called double-channel) convolutional neural network in order to classify 2 binary images (containing binary objects) as 'similar' or 'different'.

The problem I am having is that it seems as though the network doesn't always converge to the same solution. For example, I can use exactly the same ordering of training pairs and all the same parameters and so forth, and when I run the network multiple times, each time produces a different solution; sometimes converging to below 2% error rates, and other times I get 50% error rates.

I have a feeling that it has something to do with the random initialization of the weights of the network, which results in different optimization paths each time the network is executed. This issue even occurs when I use SGD with momentum, so I don't really know how to 'force' the network to converge to the same solution (global optima) every time?

Can this have something to do with the fact that I am using binary images instead of grey-scale or color images, or is there something intrinsic to neural networks that is causing this issue?

Output:
There are several sources of randomness in training.
Initialization is one. SGD itself is of course stochastic since the content of the minibatches is often random. Sometimes, layers like dropout are inherently random too. The only way to ensure getting identical results is to fix the random seed for all of them.

Given all these sources of randomness and a model with many millions of parameters, your quote

  "I don't really know how to 'force' the network to converge to the same solution (global optima) every time?"

is something pretty much something anyone should say - no one knows how to find the same solution every time, or even a local optima, let alone the global optima.

Nevertheless, ideally, it is desirable to have the network perform similarly across training attempts (with fixed hyper-parameters and dataset). Anything else is going to cause problems in reproducibility, of course.

Unfortunately, I suspect the problem is inherent to CNNs.
You may be aware of the bias-variance tradeoff. For a powerful model like a CNN, the bias is likely to be low, but the variance very high. In other words, CNNs are sensitive to data noise, initialization, and hyper-parameters. Hence, it's not so surprising that training the same model multiple times yields very different results. (I also get this phenomenon, with performances changing between training runs by as much as 30% in one project I did.) My main suggestion to reduce this is stronger regularization.

  Can this have something to do with the fact that I am using binary images instead of grey-scale or color images, or is there something intrinsic to neural networks that is causing this issue?

As I mentioned, this problem is present inherently for deep models to an extent. However, your use of binary images may also be a factor, since the space of the data itself is rather discontinuous. Perhaps consider "softening" the input (e.g. filtering the inputs) and using data augmentation. A similar approach is known to help in label smoothing, for example.


Instruction: torch.argmax() fails to find a maximum value in a tensor containing data

Input:
I have a tensor of shape [batch_size, channel, depth, height, width]:

torch.Size([1, 1, 32, 64, 64])

with data:

tensor([[[[[-1.8540, -2.8068, -2.7348,  ..., -1.9074, -1.8227, -1.4540],
           [-2.7012, -4.2785, -3.7421,  ..., -3.1961, -2.7786, -1.8042],
           [-2.1924, -4.2202, -4.4361,  ..., -3.1203, -2.9282, -2.3800],
           ...,
           [-2.7429, -4.3133, -4.4029,  ..., -4.4971, -5.3288, -2.8659],
           [-3.0169, -4.0198, -3.6886,  ..., -3.7542, -4.5010, -2.4040],
           [-1.6174, -2.5340, -2.3974,  ..., -1.9249, -2.4107, -1.2664]],

          [[-2.7840, -3.2442, -3.6118,  ..., -3.1365, -2.8342, -1.9516],
           [-3.5764, -4.9253, -5.9196,  ..., -4.8373, -4.2233, -3.3809],
           [-3.1701, -5.0826, -5.6424,  ..., -5.2955, -4.6438, -3.4820],
           ...,
           [-4.0111, -6.1946, -5.6582,  ..., -6.7947, -6.5305, -4.2866],
           [-4.2103, -6.6177, -6.0420,  ..., -5.8076, -6.2128, -3.2093],
           [-2.3174, -4.1081, -3.7369,  ..., -3.5552, -3.1871, -1.9736]],

          [[-2.8441, -4.1575, -3.8233,  ..., -3.5065, -3.4313, -2.3030],
           [-4.0076, -5.4939, -6.2451,  ..., -4.6663, -4.9835, -3.1530],
           [-3.4737, -5.6347, -6.0232,  ..., -5.6191, -5.2626, -3.6109],
           ...,
           [-3.8026, -5.3676, -6.1460,  ..., -7.6695, -6.7640, -4.1681],
           [-4.4012, -6.1293, -6.1859,  ..., -6.0011, -6.1012, -3.5307],
           [-2.7917, -4.2264, -4.1388,  ..., -4.2080, -3.5555, -1.6384]],

          ...,

          [[-2.2204, -3.5705, -4.3114,  ..., -4.2249, -3.9628, -2.9190],
           [-3.6343, -5.3445, -6.1638,  ..., -6.3998, -6.7561, -4.8491],
           [-3.4870, -5.5835, -5.6436,  ..., -6.8527, -7.2536, -4.8143],
           ...,
           [-2.4492, -3.7896, -5.4344,  ..., -6.2853, -6.0766, -3.7538],
           [-2.4723, -3.8393, -4.8480,  ..., -5.6503, -5.0375, -3.5580],
           [-1.6161, -2.9843, -3.2865,  ..., -3.2627, -3.2887, -2.5750]],

          [[-2.1509, -3.8303, -4.2807,  ..., -3.7945, -3.7561, -3.0863],
           [-3.1012, -5.1321, -6.1387,  ..., -6.5191, -6.3268, -4.4283],
           [-2.8346, -5.0640, -5.4868,  ..., -6.6515, -6.5529, -4.3672],
           ...,
           [-2.7278, -4.2538, -4.9776,  ..., -6.4153, -6.0100, -3.9929],
           [-2.8002, -4.0473, -4.7455,  ..., -5.4203, -4.7286, -3.4111],
           [-1.7964, -3.2307, -3.6329,  ..., -3.2750, -2.3952, -1.9714]],

          [[-1.4447, -2.1572, -2.4487,  ..., -2.3859, -2.9540, -1.8451],
           [-1.8075, -2.8380, -3.5621,  ..., -3.8641, -3.5828, -2.7304],
           [-1.7862, -2.9849, -3.8364,  ..., -4.3380, -4.4745, -2.8476],
           ...,
           [-1.8043, -2.5662, -2.7296,  ..., -4.2772, -3.9882, -2.8654],
           [-1.2364, -2.5228, -2.7190,  ..., -4.1142, -3.6160, -2.2325],
           [-1.0395, -1.7621, -2.5738,  ..., -2.0349, -1.5140, -1.1625]]]]]

Now to get the prediction from this I use

torch.argmax(data, 1)

which should give me the location of maximum values in the channel dimension, but instead I get a tensor containing only zeros. Even max(torch.argmax()) produces 0.

How can this be, the tensor is only a single dimension and a single batch, how can it return a 0?

To get rid of the negative values I applied torch.nn.Sigmoid() on it, but still argmax failed to find a maximum value. Which I dont understand, how can there not be a maximum value?

numpy.argmax(output.detach().numpy(), 1) gives the same output, all 0.

Am I not using argmax correctly?

Output:
On this page everything is confusing for argmax.
The example they selected is 4x4 so you cannot spot the difference

a = torch.randn(5, 3)
print(a)
print(torch.argmax(a, dim=0))
print(torch.argmax(a, dim=1))

Out

tensor([[-1.0329,  0.2042,  2.5499],
        [ 0.9893,  0.3913,  0.5096],
        [ 0.4951,  0.2260, -0.3810],
        [-1.8953, -0.6823,  0.8349],
        [-0.6217,  0.4068, -1.0846]])
tensor([1, 4, 0])
tensor([2, 0, 0, 2, 1])

See how in dim=0 we have 3 values. This is the dimension of columns.
So it is telling you the element with index 1 in the first column is the max in that column.
The other dim=1 is the dimension of rows, so we have 5 values.

For your example you can calculate the shape of result for argmax:

for i in range(data.dim()):
    print("dim", i)
    r =torch.argmax(data,i)
    print(r.shape)

dim 0
torch.Size([1, 32, 64, 64])
dim 1
torch.Size([1, 32, 64, 64])
dim 2
torch.Size([1, 1, 64, 64])
dim 3
torch.Size([1, 1, 32, 64])
dim 4
torch.Size([1, 1, 32, 64])

And for dim=0 and dim=0 you should have all 0 sine the dim is 1 (index = 0).

  I tried that, but then how do I extract the maximum values from argmax and which dimension should it look against?

data = torch.randn(32, 64, 64)

values, indices = data.max(0)
print(values, indices)

values, indices = values.max(0)
print(values, indices)

values, indices = values.max(0)
print(values, indices

)

tensor([[1.9918, 1.6041, 2.6535,  ..., 1.5768, 1.7320, 1.8234],
        [1.6700, 2.4574, 1.8548,  ..., 1.8770, 1.7674, 1.6194],
        [1.8361, 1.6800, 1.8982,  ..., 1.7983, 2.7109, 2.2166],
        ...,
        [2.7439, 1.6215, 2.9740,  ..., 1.7031, 1.4445, 1.6681],
        [1.9437, 1.4507, 1.8551,  ..., 2.5853, 1.9753, 2.4046],
        [1.4198, 2.5250, 1.8949,  ..., 3.2618, 2.8547, 2.0487]]) tensor([[ 4,  7, 21,  ..., 27, 28, 17],
        [16, 27, 18,  ..., 29, 30, 19],
        [ 6, 16, 14,  ..., 22, 24, 29],
        ...,
        [16, 16,  8,  ..., 21, 27, 22],
        [15,  0,  0,  ...,  9, 12,  3],
        [30, 14,  9,  ..., 23, 20, 14]])
tensor([3.2089, 4.1386, 3.2650, 3.3497, 4.4210, 3.0439, 3.5144, 3.2356, 3.3058,
        3.2702, 2.9981, 3.6997, 3.1719, 3.4962, 3.0889, 3.6220, 3.9256, 4.1314,
        3.0804, 3.3636, 3.5517, 3.2052, 3.6548, 3.7064, 3.6531, 4.5144, 3.1287,
        4.1465, 3.1906, 3.1493, 3.1996, 3.6754, 3.7610, 3.5968, 3.2109, 3.6037,
        3.2799, 3.0069, 3.0386, 3.0240, 3.5372, 3.6539, 3.5571, 3.2047, 3.1218,
        4.2479, 3.1230, 3.0372, 3.0258, 3.8679, 3.6409, 3.0938, 3.1246, 2.9426,
        4.0824, 3.8124, 3.4226, 3.3459, 4.1600, 3.6566, 3.0351, 3.3969, 3.5842,
        3.0997]) tensor([17, 21, 30, 62, 62, 63, 43, 31, 45, 63, 20,  4, 58, 23, 22, 43, 54, 30,
        15, 28, 13,  4,  4, 28,  6, 52, 53, 19, 33, 20,  3,  1, 14, 40,  0,  0,
        46, 62, 58, 45, 28, 50,  4, 55, 25,  5, 21, 16, 27, 32, 10, 19, 38, 30,
        48, 27, 20,  9,  2, 39, 55, 58, 32,  6])
tensor(4.5144) tensor(25)

This was by dimension, or simple

m = values.max()

will give you the max value.

a = torch.argmax(values)
idx = np.unravel_index(a, values.shape)

will give you the index.


Instruction: Dimensions Not Matching In PyTorch Linear Layer

Input:
Following the training lesson in PyTorch on this page:  https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py

It's basically their 'Hello World!' version of an image classifier.

What I'm trying to do is manually code the training steps in the network to make sure I understand each one, but I am currently getting a dimension mismatch in one of my linear layers, which has me stumped.  Especially since (AFAIK) I'm recreating the steps in the tutorial exactly.

Anyways........

MY NETWORK:

class net(nn.Module):
def __init__(self):
    super(net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool  = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1   = nn.Linear(16*5*5, 120)
    self.fc2   = nn.Linear(120, 84)
    self.fc2   = nn.Linear(84, 10)

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)

    return x

net = net()

I believe this is exactly as they have it on their own page.

I'm trying to calculate the following step without a loop:

for epoch in range(2):  # loop over the dataset multiple times

  running_loss = 0.0
  for i, data in enumerate(trainloader, 0):
    # get the inputs; data is a list of [inputs, labels]
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

  print('Finished Training')

What I'm doing is this:

data = enumerate(trainloader)
inputs, labels = next(data)[1]
outputs = net(inputs)

And the last line gives me the following traceback:

RuntimeError                              Traceback (most recent call last)
<ipython-input-285-d4be5abf5bb1> in <module>
----> 1 outputs = net(inputs)

~\Anaconda\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-282-a6eca2e3e9db> in forward(self, x)
    14         x = x.view(-1, 16 * 5 * 5)
    15         x = F.relu(self.fc1(x))
---> 16         x = F.relu(self.fc2(x))
    17         x = self.fc3(x)

Which closes out with:

RuntimeError: size mismatch, m1: [4 x 120], m2: [84 x 10] at
c:\a\w\1\s\tmp_conda_3.7_110206\conda\conda-
bld\pytorch_1550401474361\work\aten\src\th\generic/THTensorMath.cpp:940

I know this means my dimension values don't match, and I suspect it has to do with the line x = x.view(-1, 16 * 5 * 5) where I go from the convolutional to linear layer, but I have two confusions:

As far as I can tell my network matches exactly what's on the PyTorch page
My error occurs on the second linear layer, not the first, and columns of the preceeding layer match the rows of the current one, so I find it confusing why this error is happening.

Output:
Actually, there is no self.fc3(x) in __init__() as you have mentioned in forward() function. Try running you code by changing

self.fc2   = nn.Linear(84, 10) in __init__() function to
self.fc3   = nn.Linear(84, 10).

Above mistake is the reason why you are getting the error. As you are initializing self.fc2 twice in the above code, see below lines:

self.fc2   = nn.Linear(120, 84)
self.fc2   = nn.Linear(84, 10)

Here, first value of self.fc2 is overriden by later value. So, finally it is initialized with a Linear layer with input channels 84 and output channels 10.
Later on, in the forward function you are passing the output channels of x = F.relu(self.fc1(x)), i.e., 120 as an input channels to x = F.relu(self.fc2(x)), which has been changed to 84 because of the above explained reasons, you are getting the error.

Apart from this, I don't think if something wrong with your code.


Instruction: ValueError: Only one class present in y_true. ROC AUC score is not defined in that case

Input:
I want to calculate AUROC using dataset with all instance zero. However, the following error occurred:

ValueError: Only one class present in y_true. ROC AUC score is not defined in that case

def computeAUROC (dataGT, dataPRED, classCount):

    outAUROC = []

    datanpGT = dataGT.cpu().numpy()
    datanpPRED = dataPRED.cpu().numpy()

    for i in range(classCount):
        outAUROC.append(roc_auc_score(datanpGT[:, i], datanpPRED[:, i]))

    return outAUROC

Output: You cannot have an ROC curve without both positive and negative examples in your dataset. With only one class in the dataset, you cannot measure your false-positive rate, and therefore cannot plot an ROC curve. This is why you get this error message.


Instruction: PyTorch - BCELoss: ValueError: Target and input must have the same number of elements

Input:
When I use BCELoss as the loss function of my neural network, getting the ValueError: Target and input must have the same number of elements.

Here is my code for test phase (which is a quite typical test phase code):

network.eval()
test_loss = 0
correct = 0
with torch.no_grad():
    for data, target in test_loader:
        data, target = data.to(device), target.to(device)

    output = network(data)
    output = output.to(device)
    test_loss += loss_function(output, target).item() # error happens here
    _, predicted = torch.max(output.data, 1)
    correct += (predicted == target).sum().item()

The shape of the variable output is [1000, 10] as there are 10 target classes (in MNIST dataset), and the shape of the variable target is [1000] as it contains the target classes of the tested batch (the batch size for test is set to 10). So, the question is how can I apply BCELoss as the loss function of a CNN network?

p.s. The dataset I use is the MNIST dataset which is provided by the torchvision library.

p.s. The answer provided to a similar question here does not propose a solution for my case.

Output:
The answer you claim does not propose a solution, does in fact solves your problem:

  your targets are incomplete! If there are multiple classes, you should work with torch.nn.CrossEntropyLoss instead of torch.nn.BCELoss()

To recap, torch.nn.BCELoss() is intended to be used for a task of classifying c independant binary attributes per input example. You, on the other hand, have the task of classifying each output into one of c mutually exclusive classes. For this task you need a different loss, torch.nn.CrossEntropyLoss().
The different tasks, represented by the different loss functions call for different supervision (labels). If you want to classify each example to one of c mutually exclusive classes, you only need one integer label for each example (as you have in your mnist example). However, if you want to classify each example into c independent binary attributes, you need, for each example c binary labels - and this is why pytorch gives you an error.


Instruction: What is Tensorflow equivalent of pytorch's conv1d?

Input:
Just wondering how I can perform 1D convolution in tensorflow. Specifically, looking to replace this code to tensorflow:

inputs = F.pad(inputs, (kernel_size-1,0), 'constant', 0)

output = F.conv1d(inputs, weight, padding=0, groups=num_heads)

Output:
Tensorflow equivalent of PyTorch's

torch.nn.functional.conv1d() is
tf.nn.conv1d() and torch.nn.functional.pad() is tf.pad().

For Example:

(PyTorch code)

import torch.nn as nn
import torch

inputs = torch.tensor([1, 0, 2, 3, 0, 1, 1], dtype=torch.float32)
filters = torch.tensor([2, 1, 3], dtype=torch.float32)

inputs = inputs.unsqueeze(0).unsqueeze(0)                   # torch.Size([1, 1, 7])
filters = filters.unsqueeze(0).unsqueeze(0)                 # torch.Size([1, 1, 3])
conv_res = F.conv1d(inputs, filters, padding=0, groups=1)   # torch.Size([1, 1, 5])
pad_res = F.pad(conv_res, (1, 1), mode='constant', value=0) # torch.Size([1, 1, 7])

output:

tensor([[[ 0.,  8., 11.,  7.,  9.,  4.,  0.]]])

(Tensorflow code)

import tensorflow as tf
tf.enable_eager_execution()

i = tf.constant([1, 0, 2, 3, 0, 1, 1], dtype=tf.float32)
k = tf.constant([2, 1, 3], dtype=tf.float32, name='k')

data = tf.reshape(i, [1, int(i.shape[0]), 1], name='data')
kernel = tf.reshape(k, [int(k.shape[0]), 1, 1], name='kernel')

res = tf.nn.conv1d(data, kernel, 1, 'VALID')
res = tf.pad(res[0], [[1, 1], [0, 0]], "CONSTANT")

output:

<tf.Tensor: id=555, shape=(7, 1), dtype=float32, numpy=
array([[ 0.],
       [ 8.],
       [11.],
       [ 7.],
       [ 9.],
       [ 4.],
       [ 0.]], dtype=float32)>


Instruction: Pytorch: Is it possible to load model when parameters or size have changed?

Input:
A Pytorch model (graph, weights, and biases)  is saved with :

torch.save(self.state_dict(), file)

and loaded with :

self.load_state_dict(torch.load(file))

But if parameters are changed, the model won't load with error for example :

RuntimeError: Error(s) in loading state_dict for LeNet5:
    size mismatch for conv1.weight:

Is it possible to load to the model with changed size?
Sort of filling the rest of weights like in initialization (if there are more weights) and clip if there are fewer weights?

Output:
There is no automatic way of doing so - because you need to explicitly decide what to do when things do not match.

Personally, when I need to "force" a pre-trained weights on a slightly changed model. I find that working with the state_dict itself is the most convenient way.

new_model = model( ... )  # construct the new model
new_sd = new_model.state_dict()  # take the "default" state_dict
pre_trained_sd = torch.load(file)  # load the old version pre-trained weights
# merge information from pre_trained_sd into new_sd
# ...
# after merging the state dict you can load it:
new_model.load_state_dict(new_sd)


Instruction: Why do I converge only using a parameter wrapper in PyTorch?

Input:
I'm writing an implementation of Bayesian Dropout, and the following code worked after several hours of debugging:

class DropoutLayer(nn.Module):

    def __init__(self, n_in, n_out, dropout_rate):
        super(DropoutLayer,self).__init__()
        self.M = Parameter(torch.normal(torch.zeros(n_in,n_out),0.01))
        self.m = Parameter(torch.rand(n_out))

        # 1-p is used to be consistent with original dropout definition
        self.z = Bernoulli(torch.tensor([1-dropout_rate]))

        self.W = Parameter(torch.mm(
        torch.diagflat(self.z.sample(sample_shape=(n_in,))),
        self.M))

    def forward(self,x,act):
        activation = torch.mm(x,self.W) + self.m

        out = act(activation)
        return out

However, I do not seem to understand, why the matrix multiplication in self.W needs a parameter wrapping. I would assume once I nominate self.M to be a parameter in autograd, I would not need to do it for any values which use it further down the line. Why do I need it here?

After adding this wrapper, the neural net converges without a problem.
Could you tell me why is it the case and if there is a simpler substitute for this solution which I don't like?

Output:
First of all, you don't need to pass self.M into Parameter, as the gradient does not go through it. You forward function only uses self.W and self.m (plus activation, although you really should pass it to the constructor, not forward...).

All self.M is some kind of random normal tensor created at the same time as module and it's simply initializing your self.W matrix to concrete values. So self.W is a separate tensor, it's not computationally dependent on self.M in any way.


Instruction: Discrepancy between tensorflow's conv1d and pytorch's conv1d

Input:
I am trying to import some pytorch code to tensorflow, I came to know that torch.nn.functional.conv1d() is  tf.nn.conv1d() but I am afraid there are still some discrepancies in tf's versions. Specifically, I cannot find the group parameter in tf.conv1d. For example: the following codes output two different results:

Pytorch:

inputs = torch.Tensor([[[1, 1, 1, 1],[2, 2, 2, 2],[3, 3, 3, 3]]]) #batch_sizex seq_length x embed_dim,
inputs = inputs.transpose(2,1) #batch_size x embed_dim x seq_length
batch_size, embed_dim, seq_length = inputs.size()

kernel_size = 3
in_channels = 2
out_channels = in_channels
weight = torch.ones(out_channels, 1, kernel_size)

inputs = inputs.contiguous().view(-1, in_channels, seq_length) #batch_size*embed_dim/in_channels x in_channels x seq_length
inputs = F.pad(inputs, (kernel_size-1,0), 'constant', 0)
output = F.conv1d(inputs, weight, padding=0, groups=in_channels)
output = output.contiguous().view(batch_size, embed_dim, seq_length).transpose(2,1)

Output:

tensor([[[1., 1., 1., 1.],
         [3., 3., 3., 3.],
         [6., 6., 6., 6.]]])

Tensorflow:

inputs = tf.constant([[[1, 1, 1, 1],[2, 2, 2, 2],[3, 3, 3, 3]]], dtype=tf.float32) #batch_sizex seq_length x embed_dim
inputs = tf.transpose(inputs, perm=[0,2,1])
batch_size, embed_dim, seq_length = inputs.get_shape()

print(batch_size, seq_length, embed_dim)
kernel_size = 3
in_channels = 2
out_channels = in_channels
weight = tf.ones([kernel_size, in_channels, out_channels])

inputs = tf.reshape(inputs, [(batch_size*embed_dim)//in_channels, in_channels, seq_length], name='inputs')
inputs = tf.transpose(inputs, perm=[0, 2, 1])
padding = [[0, 0], [(kernel_size - 1), 0], [0, 0]]
padded = tf.pad(inputs, padding)

res = tf.nn.conv1d(padded, weight, 1, 'VALID')
res = tf.transpose(res, perm=[0, 2, 1])
res = tf.reshape(res, [batch_size, embed_dim, seq_length])
res = tf.transpose(res, perm=[0, 2, 1])
print(res)

Output:

[[[ 2.  2.  2.  2.]
  [ 6.  6.  6.  6.]
  [12. 12. 12. 12.]]], shape=(1, 3, 4), dtype=float32)

Output:
Different results

There is no discrepancy between those versions, you are just setting up different things. To get exactly same results as in Tensorflow change the lines specifying weights to:

weight = torch.ones(out_channels, 2, kernel_size)

, because your input has two input channels, as you have correctly declared in TF:

weight = tf.ones([kernel_size, in_channels, out_channels])

Groups parameter

You have misunderstood what is groups parameter responsible for in pytorch. It restricts the number of channels each filter uses (in this case only one as 2 input_channels divided by 2 give us one).

See here for more intuitive explanation for 2D convolution.


Instruction: LSTM returning only zeros (PyTorch)

Input:
After succesfully implementing a LSTM "from scratch" based on linear layers, I decided to start using the existing LSTM class to make things easier and gain in performance.
But somehow when I try it, it only returns tensors full of zeros.
Here is the model :

class pytorchLSTM(nn.Module):
    def __init__(self,input_size,hidden_size):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.softmax = nn.LogSoftmax(dim = 1)

    def forward(self, input):
        out, hidden = self.lstm(input)
        out = self.softmax(out)
        return out, hidden

the input is a (1,1,60) tensor representing a one-hot encoded letter :

tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
          0., 0., 0., 0., 0., 0., 0., 1., 0.]]])

and the models returns, invariably (I tried modifying the values inside of the input, but the result is always the same) :

tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
           0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]],
        grad_fn=<LogSoftmaxBackward>)

Any idea where my mistake is and what I understood wrong about the LSTM class ?

Output:
You are passing input of shape (1,1,60), which can be described as (sequence_length, batch_size, feature_size), where feature_size is binary.

I'm not sure why you are using LogSoftmax at all as it's discouraged and numerically unstable and it's advised to use torch.nn.CrossEntropyLoss taking logits straight out of your network, but that's not the problem here (might be later).

IIUC you are trying to predict another tensor of shape (sequence_length, batch_size, feature_size), e.g. next letter in the word (or the same later as inputted, dunno).
You need to do LogSoftmax(dim=2) (not sure if dim=-1 will work), currently you are softmaxing batch dimension.


Instruction: Adding layers and bidirectionality to custom LSTM cell in pytorch

Input:
I use a very custom LSTM-cell inspired by http://mlexplained.com/2019/02/15/building-an-lstm-from-scratch-in-pytorch-lstms-in-depth-part-1/.

I use it to look at intermediate gating values. My question is, how would I expand this class to have an option for adding more layers and for adding bidirectionality? Should it be wrapped in a new class or added in the present one?

class Dim(IntEnum):
    batch = 0
    seq = 1

class simpleLSTM(nn.Module):
    def __init__(self, input_sz: int, hidden_sz: int):
        super().__init__()
        self.input_size = input_sz
        self.hidden_size = hidden_sz
        # input gate
        self.W_ii = Parameter(torch.Tensor(input_sz, hidden_sz))
        self.W_hi = Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_i = Parameter(torch.Tensor(hidden_sz))
        # forget gate
        self.W_if = Parameter(torch.Tensor(input_sz, hidden_sz))
        self.W_hf = Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_f = Parameter(torch.Tensor(hidden_sz))
        # ???
        self.W_ig = Parameter(torch.Tensor(input_sz, hidden_sz))
        self.W_hg = Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_g = Parameter(torch.Tensor(hidden_sz))
        # output gate
        self.W_io = Parameter(torch.Tensor(input_sz, hidden_sz))
        self.W_ho = Parameter(torch.Tensor(hidden_sz, hidden_sz))
        self.b_o = Parameter(torch.Tensor(hidden_sz))

        self.init_weights()

        self.out = nn.Linear(hidden_sz, len(TRG.vocab))

    def init_weights(self):
        for p in self.parameters():
            if p.data.ndimension() >= 2:
                nn.init.xavier_uniform_(p.data)
            else:
                nn.init.zeros_(p.data)

    def forward(self, x, init_states=None ):
        """Assumes x is of shape (batch, sequence, feature)"""
        seq_sz, bs, = x.size()
        hidden_seq = []
        prediction = []

        if init_states is None:
            h_t, c_t = torch.zeros(self.hidden_size).to(x.device), torch.zeros(self.hidden_size).to(x.device)
        else:
            h_t, c_t = init_states
        for t in range(seq_sz): # iterate over the time steps
            x_t = x[t, :].float()
            #LOOK HERE!!!
            i_t = torch.sigmoid(x_t @ self.W_ii + h_t @ self.W_hi + self.b_i)
            f_t = torch.sigmoid(x_t @ self.W_if + h_t @ self.W_hf + self.b_f)
            g_t = torch.tanh(x_t @ self.W_ig + h_t @ self.W_hg + self.b_g)
            o_t = torch.sigmoid(x_t @ self.W_io + h_t @ self.W_ho + self.b_o)
            c_t = f_t * c_t + i_t * g_t
            h_t = o_t * torch.tanh(c_t)

            hidden_seq.append(h_t.unsqueeze(Dim.batch))

            pred_t = self.out(h_t.unsqueeze(Dim.batch))
            #pred_t = F.softmax(pred_t)
            prediction.append(pred_t)

        hidden_seq = torch.cat(hidden_seq, dim=Dim.batch)
        prediction = torch.cat(prediction, dim=Dim.batch)
        # reshape from shape (sequence, batch, feature) to (batch, sequence, feature)
        hidden_seq = hidden_seq.transpose(Dim.batch, Dim.seq).contiguous()
        prediction = prediction.transpose(Dim.batch, Dim.seq).contiguous()

        return prediction, hidden_seq, (h_t, c_t)

I call it and train using the following as an example.

lstm = simpleLSTM(1, 100)
hidden_size = lstm.hidden_size

optimizer = optim.Adam(lstm.parameters())

h_0, c_0 = (torch.zeros(hidden_size, requires_grad=True),
            torch.zeros(hidden_size, requires_grad=True))

grads = []
h_t, c_t = h_0, c_0

N_EPOCHS = 10

for epoch in range(N_EPOCHS):
    epoch_loss = 0
    for i, batch in enumerate(train):

        optimizer.zero_grad()

        src, src_len = batch.src
        trg = batch.trg
        trg = trg.view(-1)

        predict, output, hidden_states = lstm(src)
        predict = predict.t().unsqueeze(1)
        predict= predict.view(-1, predict.shape[-1])

        loss = criterion(predict,trg)
        loss.backward()

        optimizer.step()
        epoch_loss += loss.item()

    print(epoch_loss)

Output:
The easiest would be to create another module (say Bidirectional) and pass any cell you want to it.

Implementation itself is quite easy to do. Notice that I'm using concat operation for joining bi-directional output, you may want to specify other modes like summation etc.

Please read the comments in the code below, you may have to change it appropriately.

import torch

class Bidirectional(torch.nn.Module):
    def __init__(self, cell):
        super().__init__()
        self.cell = cell

    def __call__(self, x, init_states=None):
        prediction, hidden_seq, (h_t, c_t) = self.cell(x, init_states)
        backward_prediction, backward_hidden_seq, (
            backward_h_t,
            backward_c_t,
            # Assuming sequence is first dimension, otherwise change 0 appropriately
            # Reverses sequences so the LSTM cell acts on the reversed sequence
        ) = self.cell(torch.flip(x, (0,)), init_states)
        return (
            # Assuming you transpose so it has (batch, seq, features) dimensionality
            torch.cat((prediction, backward_prediction), 2),
            torch.cat((hidden_seq, backward_hidden_seq), 2),
            # Assuming it has (batch, features) dimensionality
            torch.cat((h_t, backward_ht), 1),
            torch.cat((c_t, backward_ct), 1),
        )

When it comes to multiple layers you could do something similiar in principle:

import torch

class Multilayer(torch.nn.Module):
    def __init__(self, *cells):
        super().__init__()
        self.cells = torch.nn.ModuleList(cells)

    def __call__(self, x, init_states=None):
        inputs = x
        for cell in self.cells:
            prediction, hidden_seq, (h_t, c_t) = cell(inputs, init_states)
            inputs = hidden_seq
        return prediction, hidden_seq, (h_t, c_t)

Please note you have to pass created cell objects into Multilayer e.g.:

# For three layers of LSTM, each needs features to be set up correctly
multilayer_LSTM = Multilayer(LSTM(), LSTM(), LSTM())

You may also pass classes instead of instances into constructor and create those inside Multilayer (so hidden_size matches automatically), but those ideas should get you started.


Instruction: 'DataLoader' object does not support indexing

Input:
I have downloaded the ImageNet dataset via this pytorch api by setting download=True. But I cannot iterate through the dataloader.

The error says "'DataLoader' object does not support indexing"

trainset = torch.utils.data.DataLoader(
    datasets.ImageNet('/media/farshid/DataStore/temp/Imagenet/', split='train',
                      download=False))
trainloader = torch.utils.data.DataLoader(trainset, batch_size=1, shuffle=False, num_workers=1)

I tried a simple approach I just tried to run the following,

trainloader[0]

In the root directory, the pattern is

root/
    train/
          n01440764/
          n01443537/
                   n01443537_2.jpg

The docs in the official website doesnt say anything else. https://pytorch.org/docs/stable/torchvision/datasets.html#imagenet

What am I doing wrong ?

Output:
Solution

input_transform = standard_transforms.Compose([
    transforms.Resize((255,255)), # to Make sure all the
    transforms.CenterCrop(224),   # imgs are at the same size
    transforms.ToTensor()
])

# torch.utils.data.Dataset object
trainset = datasets.ImageNet('/media/farshid/DataStore/temp/Imagenet/',
                             split='train', download=False, transform = input_transform)
# torch.utils.data.DataLoader object
trainloader =torch.utils.data.DataLoader(trainset, batch_size=2, shuffle=False)

for batch_idx, data in enumerate(trainloader, 0):
    x, y = data
    break


Instruction: Nonexistant pytorch gradients when dotting tensors in loss function

Input:
For the purposes of this MWE I'm trying to fit a linear regression using a custom loss function with multiple terms. However, I'm running into strange behavior when trying to weight the different terms in my loss function by dotting a weight vector with my losses. Just summing the losses works as expected; however, when dotting the weights and losses the backpropagation gets broken somehow and the loss function doesn't decrease.

I've tried enabling and disabling requires_grad on both tensors, but have been unable to replicate the expected behavior.

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Hyper-parameters
input_size = 1
output_size = 1
num_epochs = 60
learning_rate = 0.001

# Toy dataset
x_train = np.array([[3.3], [4.4], [5.5], [6.71], [6.93], [4.168],
                    [9.779], [6.182], [7.59], [2.167], [7.042],
                    [10.791], [5.313], [7.997], [3.1]], dtype=np.float32)

y_train = np.array([[1.7], [2.76], [2.09], [3.19], [1.694], [1.573],
                    [3.366], [2.596], [2.53], [1.221], [2.827],
                    [3.465], [1.65], [2.904], [1.3]], dtype=np.float32)

# Linear regression model
model = nn.Linear(input_size, output_size)

# Loss and optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

def loss_fn(outputs, targets):
    l1loss = torch.norm(outputs - targets, 1)
    l2loss = torch.norm(outputs - targets, 2)

    # This works as expected
    # loss = 1 * l1loss + 1 * l2loss
    # Loss never changes, no matter what combination of
    # requires_grad I set
    loss = torch.dot(torch.tensor([1.0, 1.0], requires_grad=False),
            torch.tensor([l1loss, l2loss], requires_grad=True))
    return loss

# Train the model
for epoch in range(num_epochs):
    # Convert numpy arrays to torch tensors
    inputs = torch.from_numpy(x_train)
    targets = torch.from_numpy(y_train)

    # Forward pass
    outputs = model(inputs)
    loss = loss_fn(outputs, targets)

    # Backward and optimize
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if (epoch+1) % 5 == 0:
        print ('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

# Plot the graph
predicted = model(torch.from_numpy(x_train)).detach().numpy()
plt.plot(x_train, y_train, 'ro', label='Original data')
plt.plot(x_train, predicted, label='Fitted line')
plt.legend()
plt.show()

Expected result: loss function decreases and the linear regression is fitted (see output below)

Epoch [5/60], Loss: 7.9943
Epoch [10/60], Loss: 7.7597
Epoch [15/60], Loss: 7.6619
Epoch [20/60], Loss: 7.6102
Epoch [25/60], Loss: 7.4971
Epoch [30/60], Loss: 7.4106
Epoch [35/60], Loss: 7.3942
Epoch [40/60], Loss: 7.2438
Epoch [45/60], Loss: 7.2322
Epoch [50/60], Loss: 7.1012
Epoch [55/60], Loss: 7.0701
Epoch [60/60], Loss: 6.9612

Actual result: no change in loss function

Epoch [5/60], Loss: 73.7473
Epoch [10/60], Loss: 73.7473
Epoch [15/60], Loss: 73.7473
Epoch [20/60], Loss: 73.7473
Epoch [25/60], Loss: 73.7473
Epoch [30/60], Loss: 73.7473
Epoch [35/60], Loss: 73.7473
Epoch [40/60], Loss: 73.7473
Epoch [45/60], Loss: 73.7473
Epoch [50/60], Loss: 73.7473
Epoch [55/60], Loss: 73.7473
Epoch [60/60], Loss: 73.7473

I'm pretty confused as to why such a simple operation is breaking the backpropagation gradients and would really appreciate it if anyone had some insights on why this isn't working.

Output:
Use torch.cat((loss1, loss2)), you are creating new Tensor from existing tensors destroying graph.

Anyway you shouldn't do that unless you are trying to generalize your loss function, it's pretty unreadable. Simple addition is way better.


Instruction: My custom loss function in Pytorch does not train

Input:
My custom loss function in Pytorch does not update during training. The loss stays exactly the same. I am trying to write this custom loss function based on the false positive and negative rates. I am giving you a simplified version of the code. Any idea what could be happening? Does the backpropagation turns to 0? Is this not the correct way of defining a custom loss function?

I have already checked that during backpropagation the Gradient always stays TRUE (assert requires_grad). I have also tried to make a class (torch.nn.module) of the function false_pos_neg_rate, but that did not work. The Assert Requires_grad turned out to be negative and I left it out afterwards.
There is no error, the training does continue.

def false_pos_neg_rate(outputs, truths):
    y = truths
    y_predicted = outputs
    cut_off= torch.tensor(0.5, requires_grad=True)
    y_predicted =torch.where(y_predicted <= cut_off, zeros, ones)
    tp, fp, tn, fn = confusion_matrix(y_predicted, y)
    fp_rate = fp / (fp+tn).float()
    fn_rate = fn / (fn+tp).float()
    loss = fn_rate + fp_rate
    return loss

for i, (samples, truths) in enumerate(train_loader):
    samples = Variable(samples)
    truths = Variable(truths)
    outputs = model(samples)
    loss = false_pos_neg_rate_torch(outputs, truths)
    loss.backward()
    optimizer.step()

I expect the loss function to update the model and be smaller every training step. Instead the loss stays exactly the same and nothing happens.

Please help me, what happens? Why does the model not train during training steps?

Output:
As pointed out by Umang Gupta your loss function is not differentiable. If you write, mathematically, what you are trying to do you'll see that your loss has zero gradient almost everywhere and it behaves like a "step function".
In order to train models using gradient-descent methods you must have meaningful gradients for the loss function.


Instruction: Weights of nn.ModuleList() are adjusted even if no forward propagation was applied

Input:
I'm trying to use nn.ModuleList() to conduct some multitask learning, but the weights of all list elements (i.e. tasks) are adjusted when they were trained before. The following code (based on this notebook) creates a neural network object called MTL.

import torch
import torch.nn as nn
import numpy as np
import os
from torch.autograd import Variable
import math
import sklearn.preprocessing as sk
from sklearn.model_selection import KFold
from sklearn import metrics
from sklearn.feature_selection import VarianceThreshold
from sklearn.linear_model import Ridge
from sklearn.linear_model import RidgeCV
from sklearn.model_selection import train_test_split
import random

#creating training data
seed = 42
random.seed(seed)
torch.cuda.manual_seed_all(seed)

N = 10000
M = 100
c = 0.5
p = 0.9
k = np.random.randn(M)
u1 = np.random.randn(M)
u1 -= u1.dot(k) * k / np.linalg.norm(k)**2
u1 /= np.linalg.norm(u1)
k /= np.linalg.norm(k)
u2 = k
w1 = c*u1
w2 = c*(p*u1+np.sqrt((1-p**2))*u2)
X = np.random.normal(0, 1, (N, M))
eps1 = np.random.normal(0, 0.01)
eps2 = np.random.normal(0, 0.01)
Y1 = np.matmul(X, w1) + np.sin(np.matmul(X, w1))+eps1
Y2 = np.matmul(X, w2) + np.sin(np.matmul(X, w2))+eps2
split = list(np.random.permutation(N))

X_train = X[split[0:8000],:]
Y1_train = Y1[split[0:8000]]
Y2_train = Y2[split[0:8000]]
X_valid = X[8000:9000,:]
Y1_valid = Y1[8000:9000]
Y2_valid = Y2[8000:9000]
X_test = X[9000:10000,:]
Y1_test = Y1[9000:10000]
Y2_test = Y2[9000:10000]

X_train = torch.from_numpy(X_train)
X_train = X_train.float()
Y1_train = torch.tensor(Y1_train)
Y1_train = Y1_train.float()
Y2_train = torch.tensor(Y2_train)
Y2_train = Y2_train.float()

X_valid = torch.from_numpy(X_valid)
X_valid = X_valid.float()
Y1_valid = torch.tensor(Y1_valid)
Y1_valid = Y1_valid.float()
Y2_valid = torch.tensor(Y2_valid)
Y2_valid = Y2_valid.float()

X_test = torch.from_numpy(X_test)
X_test = X_test.float()
Y1_test = torch.tensor(Y1_test)
Y1_test = Y1_test.float()
Y2_test = torch.tensor(Y2_test)
Y2_test = Y2_test.float()

input_size, feature_size = X.shape

LR = 0.001
epoch = 50
mb_size = 100

#the network
class MTLnet(nn.Module):
    def __init__(self):
        super(MTLnet, self).__init__()

        self.sharedlayer = nn.Sequential(
            nn.Linear(feature_size, 64),
            nn.ReLU(),
            nn.Dropout()
        )

        output = ['tower1', 'tower2']
        self.scoring_list = nn.ModuleList()

        for task, lab in enumerate(output):
            tower = nn.Sequential(
                nn.Linear(64, 32),
                nn.ReLU(),
                nn.Dropout(),
                nn.Linear(32, 16),
                nn.ReLU(),
                nn.Dropout(),
                nn.Linear(16, 1)
            )
            self.scoring_list.append(tower)

    def forward(self, x, task_id):
        h_shared = self.sharedlayer(x)
        logits = self.scoring_list[task_id](h_shared)

        return logits

def random_mini_batches(XE, R1E, R2E, mini_batch_size = 3, seed = 42):
    # Creating the mini-batches
    np.random.seed(seed)
    m = XE.shape[0]
    mini_batches = []
    permutation = list(np.random.permutation(m))
    shuffled_XE = XE[permutation,:]
    shuffled_X1R = R1E[permutation]
    shuffled_X2R = R2E[permutation]
    num_complete_minibatches = math.floor(m/mini_batch_size)
    for k in range(0, int(num_complete_minibatches)):
        mini_batch_XE = shuffled_XE[k * mini_batch_size : (k+1) * mini_batch_size, :]
        mini_batch_X1R = shuffled_X1R[k * mini_batch_size : (k+1) * mini_batch_size]
        mini_batch_X2R = shuffled_X2R[k * mini_batch_size : (k+1) * mini_batch_size]
        mini_batch = (mini_batch_XE, mini_batch_X1R, mini_batch_X2R)
        mini_batches.append(mini_batch)
    Lower = int(num_complete_minibatches * mini_batch_size)
    Upper = int(m - (mini_batch_size * math.floor(m/mini_batch_size)))
    if m % mini_batch_size != 0:
        mini_batch_XE = shuffled_XE[Lower : Lower + Upper, :]
        mini_batch_X1R = shuffled_X1R[Lower : Lower + Upper]
        mini_batch_X2R = shuffled_X2R[Lower : Lower + Upper]
        mini_batch = (mini_batch_XE, mini_batch_X1R, mini_batch_X2R)
        mini_batches.append(mini_batch)

    return mini_batches

MTL = MTLnet()
optimizer = torch.optim.Adam(MTL.parameters(), lr=LR)
loss_func = nn.MSELoss()

The neural network has the following structure:

<bound method Module.parameters of MTLnet(
  (sharedlayer): Sequential(
    (0): Linear(in_features=100, out_features=64, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.5)
  )
  (scoring_list): ModuleList(
    (0): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.5)
      (3): Linear(in_features=32, out_features=16, bias=True)
      (4): ReLU()
      (5): Dropout(p=0.5)
      (6): Linear(in_features=16, out_features=1, bias=True)
    )
    (1): Sequential(
      (0): Linear(in_features=64, out_features=32, bias=True)
      (1): ReLU()
      (2): Dropout(p=0.5)
      (3): Linear(in_features=32, out_features=16, bias=True)
      (4): ReLU()
      (5): Dropout(p=0.5)
      (6): Linear(in_features=16, out_features=1, bias=True)
    )
  )
)>

The initial values of the weights are (yours will differ):

print(MTL.state_dict()['scoring_list.0.6.weight'])
print(MTL.state_dict()['scoring_list.1.6.weight'])

Output:

tensor([[-0.0240, -0.1798, -0.2393, -0.2149, -0.1393,  0.1718, -0.1476,  0.0346,
          0.2485, -0.0305, -0.1574,  0.1500, -0.2356, -0.0597,  0.0291,  0.0521]])
tensor([[ 0.2046, -0.1277, -0.2103, -0.1006, -0.1311,  0.1902, -0.0969, -0.0953,
          0.1340,  0.1506, -0.1222, -0.0638, -0.0661,  0.1118, -0.1009, -0.1438]])

The following code trains the neural network and prints the weights after every epoch. The first epoch will train the shared layer and the first element of nn.ModuleList() (i.e. task1). The second epoch will train the shared layer and the second element of the nn.ModuleList() (i.e. task2).

trainTask1 = True
epoch = 2

for it in range(epoch):
    minibatches = random_mini_batches(X_train, Y1_train, Y2_train, mb_size)
    for minibatch in minibatches:
        XE, YE1, YE2  = minibatch

        if trainTask1:
            Yhat = MTL(XE, 0)
            loss = loss_func(Yhat, YE1.view(-1,1))
        else:
            Yhat = MTL(XE, 1)
            loss = loss_func(Yhat, YE2.view(-1,1))

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        #Shows you the weights of task 1 and they get adjusted during the 2 epoch even if no training has happened
        #print("Task 1 weights {}".format(MTL.state_dict()['scoring_list.0.6.weight']))

    #@prosti suggested to freeze the layers of the task which aren't trained
    if trainTask1:
        trainTask1 = False
        for param in MTL.scoring_list[0].parameters():
            param.requires_grad = False
    else:
        trainTask1 = True
        for param in MTL.scoring_list[1].parameters():
            param.requires_grad = False

    print(it)
    print(MTL.state_dict()['scoring_list.0.6.weight'])
    print(MTL.state_dict()['scoring_list.1.6.weight'])

Output:

0
tensor([[-0.0283, -0.2025, -0.2181, -0.2183, -0.1438,  0.1605, -0.1715,  0.0863,
          0.2765, -0.0153, -0.1519,  0.1704, -0.2453, -0.0539,  0.0220,  0.0739]])
tensor([[ 0.2046, -0.1277, -0.2103, -0.1006, -0.1311,  0.1902, -0.0969, -0.0953,
          0.1340,  0.1506, -0.1222, -0.0638, -0.0661,  0.1118, -0.1009, -0.1438]])
1
tensor([[-0.0311, -0.2114, -0.2162, -0.2214, -0.1463,  0.1614, -0.1800,  0.1003,
          0.2850, -0.0148, -0.1576,  0.1809, -0.2511, -0.0575,  0.0221,  0.0844]])
tensor([[ 0.2693, -0.0921, -0.2313, -0.1483, -0.0995,  0.2497, -0.1028, -0.1108,
          0.1405,  0.1997, -0.1266, -0.0725, -0.0871,  0.1472, -0.0924, -0.0994]])

After the first epoch the weights of task1 were adjusted but not the weights of task2 (as expected), but after the second epoch the weights of both tasks were adjusted. That shouldn't happen.

You can also see that when you print the weights during the over minibatches (just uncomment the print), they are always get adjusted for the first task even if all layers were freezed and no calculation has happened.

Is there another cache I have to clear besides optimizer.zero_grad()?

Output:
If you monitor MTL.scoring_list[0][6].weight.grad and MTL.scoring_list[1][6].weight.grad during training, you'll notice that, in the first epoch, MTL.scoring_list[1][6].weight.grad is None, while in the second epoch, MTL.scoring_list[0][6].weight.grad is a zero tensor.

Looking at the source of .step() for various optimizers, it appears that they don't check for .requires_grad. They only check if .grad is None. So even if .grad is a zero tensor, optimizer.step will still do its thing. Whether this ends up affecting the frozen weights or not depends on the exact computations that the optimizer performs.

As a quick fix, you can add param.grad = None, after param.requires_grad = False, so that the optimizer ignores those parameters entirely. This seems to fix the problem. But you might still want to think about any implications it might have for the optimizer's calculations for future epochs.


Instruction: Nsight Compute can't profile Waveglow (PyTorch application)

Input:
I tried to profile  https://github.com/NVIDIA/waveglow  by this command:

nv-nsight-cu-cli --export ./nsight_output ~/.virtualenvs/waveglow/bin/python3 inference.py -f <(ls mel_spectrograms/*.pt) -w waveglow_256channels.pt -o . --is_fp16 -s 0.6

Python command is from instruction of https://github.com/NVIDIA/waveglow#generate-audio-with-our-pre-existing-model ,
and it works with Nsight System, not Nsight Compute.

Profiling doesn't end printing this log; so I pressed Ctrl+C.
Also, It profiles only one kernel but I have more kernels. (checked by Nsight Systems)

...
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 286: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 287: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 288: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 289: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 290: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 291: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 292: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 293: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 294: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 295: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 296: 0%....50%...^C
==PROF== Received signal, trying to shutdown target application
 - 43 passes
==ERROR== Failed to profile kernel "weight_norm_fwd_first_dim_ker..." in process
==ERROR== An error occurred while trying to profile.
==ERROR== An error occurred while trying to profile
==PROF== Report: nsight_compute_result.nsight-cuprof-report

OS:  CentOS Linux 7, Nsight Compute (2019.3.1, Build 26317742),
GPU: Tesla V100-PCIE-32GB

How can I fix this?

Output:
I don't think there is any error here, the tool behaves as expected. It does not profile only one kernel, it profiled 296 kernel launches already in your log output (which appear to all be from one kernel function).

You can control the number or types of kernels that are profiled using e.g. the --launch-count or the --kernel-regex options. You can also control the metrics collected for each kernel using --metrics and --section, as collecting fewer metrics reduces the overhead of the tool.

See https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html#command-line-options for more available command line options.


Instruction: LSTM getting caught up in loop

Input:
I recently implemented a name generating RNN "from scratch" which was doing ok but far from perfect. So I thought about trying my luck with pytorch's LSTM class to see if it makes a difference. Indeed it does and the outpus looks way better for the first 7 ~ 8 characters. But then the networks gets caught in a loop and outputs things like "laulaulaulau" or "rourourourou" (it is supposed the generate french names).

Is it a often occuring problem ? If so do you know a way to fix it ? I'm concern about the fact the network doesn't produce EOS tokens...
This is an issue which has already been asked here Why does my keras LSTM model get stuck in an infinite loop?
but not really answered hence my post.

here is the model :

class pytorchLSTM(nn.Module):
    def __init__(self,input_size,hidden_size):
        super(pytorchLSTM,self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.output_layer = nn.Linear(hidden_size,input_size)
        self.tanh = nn.Tanh()
        self.softmax = nn.LogSoftmax(dim = 2)

    def forward(self, input, hidden)
            out, hidden = self.lstm(input,hidden)
            out = self.tanh(out)
            out = self.output_layer(out)
            out = self.softmax(out)
        return out, hidden

The input and target are two sequences of one-hot encoded vectors respectively with a start of sequence and end of sequence vector at the start and the end. They represent the characters inside of a name taken from the name list (database).
I use a  and  token on each name from the database. here are the function I use

def inputTensor(line):
#tensor starts with <start of sequence> token.
    tensor = torch.zeros(len(line)+1, 1, n_letters)
    tensor[0][0][n_letters - 2] = 1
    for li in range(len(line)):
        letter = line[li]
        tensor[li+1][0][all_letters.find(letter)] = 1
    return tensor

# LongTensor of second letter to end (EOS) for target
def targetTensor(line):
    letter_indexes = [all_letters.find(line[li]) for li in range(len(line))]
    letter_indexes.append(n_letters - 1) # EOS
    return torch.LongTensor(letter_indexes)

training loop :

def train_lstm(model):
    start = time.time()
    criterion = nn.NLLLoss()
    optimizer = torch.optim.Adam(model.parameters())
    n_iters = 20000
    print_every = 1000
    plot_every = 500
    all_losses = []
    total_loss = 0
    for iter in range(1,n_iters+1):
        line = randomChoice(category_line)
        input_line_tensor = inputTensor(line)
        target_line_tensor = targetTensor(line).unsqueeze(-1)
        optimizer.zero_grad()
        loss = 0
        output, hidden = model(input_line_tensor)
        for i in range(input_line_tensor.size(0)):
            l = criterion(output[i], target_line_tensor[i])
            loss += l
        loss.backward()
        optimizer.step()

the sampling function :

def sample():
    max_length = 20
    input = torch.zeros(1,1,n_letters)
    input[0][0][n_letters - 2] = 1
    output_name = ""
    hidden = (torch.zeros(2,1,lstm.hidden_size),torch.zeros(2,1,lstm.hidden_size))

    for i in range(max_length):
        output, hidden = lstm(input)
        output = output[-1][:][:]
        l = torch.multinomial(torch.exp(output[0]),num_samples = 1).item()
        if l == n_letters - 1:
            break
        else:
            letter = all_letters[l]
            output_name += letter
        input = inputTensor(letter)
    return output_name

The typical sampled output looks something like that :

Laurayeerauerararauo
Leayealouododauodouo
Courouauurourourodau

Do you know how I can improve that ?

Output:
I found the explanation :

When using instances of the LSTM class as part of a RNN, the default input dimensions are (seq_length,batch_dim,input_size). To be able to interpret the output of the lstm as a probability (over the set of inputs) I needed to pass it to a Linear layer before the Softmax call, which is where the problem happens : Linear instances expects the input to be in the format (batch_dim,seq_length,input_size).

To fix this, one needs to pass batch_first = True as an argument to the LSTM upon creation, and then feed the RNN with an input of the form (batch_dim, seq_length, input_size).