Instruction: How to use tensor.item() ? IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

Input:
Im pretty new to Siamese Neural Networks and recently found this example and Colab notebook.

When running the code i get the following error:

  IndexError: invalid index of a 0-dim tensor. Use tensor.item() to
  convert a 0-dim tensor to a Python number

on the line:

result=torch.max(res,1)[1][0][0][0].data[0].tolist()

I found something about the tensor.item() but i really just don't know how to use it here.

EDIT:

test_dataloader = DataLoader(test_dataset,num_workers=6,batch_size=1,shuffle=True)
accuracy=0
counter=0
correct=0
for i, data in enumerate(test_dataloader,0):
x0, x1 , label = data
# onehsot applies in the output of 128 dense vectors which is then  converted to 2 dense vectors
output1,output2 = model(x0.to(device),x1.to(device))
res=torch.abs(output1.cuda() - output2.cuda())
label=label[0].tolist()
label=int(label[0])
result=torch.max(res,1)[1][0][0][0].data.item().tolist()
if label == result:
correct=correct+1
counter=counter+1
#   if counter ==20:
#      break

accuracy=(correct/len(test_dataloader))*100
print("Accuracy:{}%".format(accuracy))

Thats the code i get the error on.

Output:
What this error message says is that you're trying to index into an array which has just one item in it. For example,

In [10]: aten = torch.tensor(2)

In [11]: aten
Out[11]: tensor(2)

In [12]: aten[0]
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-12-5c40f6ab046a> in <module>
----> 1 aten[0]

IndexError: invalid index of a 0-dim tensor.  Use tensor.item() to convert a 0-dim
tensor to a Python number

In the above case, aten is a tensor with just a single number in it. So, using an index (or more) to retrieve that number throws the IndexError.

The correct way to extract the number(item) out of the tensor is to use tensor.item(), here aten.item() as in:

In [14]: aten.item()
Out[14]: 2


Instruction: PyTorch: How to write a neural network that only returns the weights?

Input:
I'm training a neural network that learns some weights and based on those weights, I compute transformations that produce the predicted model in combination with the weights. My network doesn't learn properly and therefore I'm writing a different network that does nothing but returning the weights independent from the input x (after normalization with softmax and transpose). This way, I want to find out whether the problem lies in the network or in the transformation estimation outside the network. But this doesn't work. This is what I've got.

class DoNothingNet(torch.nn.Module):
    def __init__(self, n_vertices=6890, n_joints=14):
        super(DoNothingNet, self).__init__()
        self.weights = nn.parameter.Parameter(torch.randn(n_vertices, n_joints))

    def forward(self, x, indices):
        self.weights = F.softmax(self.weights, dim=1)
        return self.weights.transpose(0,1)

But the line self.weights = F.softmax(self.weights, dim=1) doesn't work and produces the error TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weights' (torch.nn.Parameter or None expected). How do I fix this? And does the code even make sense?

Output:
nn.Module tracks all fields of type nn.Parameter for training. In your code every forward call you try to change parameters weights by assigning it to Tensor type, so the error occurs.

The following code outputs normalised weights without changing the stored ones. Hope this will help.

import torch
from torch import nn
from torch.nn import functional as F

class DoNothingNet(torch.nn.Module):
    def __init__(self, n_vertices=6890, n_joints=14):
        super(DoNothingNet, self).__init__()
        self.weights = nn.parameter.Parameter(torch.randn(n_vertices, n_joints))

    def forward(self, x, indices):
        output = F.softmax(self.weights, dim=1)
        return output.transpose(0,1)


Instruction: How to exponentially decay a value along an axis?

Input:
I have a tensor of shape (sequence_length,batch_size,1) (which is (5,5,1) in the following example). Every sequence has a single entry that I want to exponentially decay along the sequence (for example with a decay factor of 0.99) for n steps (here: n = 2).

Example:

Is there an efficient way to realize something like that (either in PyTorch or numpy).

Edit (visualizing the additive nature):

Output:
This looks like convolution to me:

import numpy as np
from scipy.ndimage import convolve1d

# just for printing:
import numpy.lib.recfunctions as nlr
def show(x): print(nlr.unstructured_to_structured(x))

a = "00000","00100","10000","03502","00010"
A = np.array(a).view('U1').reshape(5,5,1).astype(float)

show(A)
# [[(0.,) (0.,) (0.,) (0.,) (0.,)]
#  [(0.,) (0.,) (1.,) (0.,) (0.,)]
#  [(1.,) (0.,) (0.,) (0.,) (0.,)]
#  [(0.,) (3.,) (5.,) (0.,) (2.,)]
#  [(0.,) (0.,) (0.,) (1.,) (0.,)]]

show(convolve1d(A,np.r_[np.logspace(2,0,3,base=0.99),0,0],axis=0,mode="constant"))
# [[(0.9801,) (0.    ,) (0.99  ,) (0.    ,) (0.    ,)]
#  [(0.99  ,) (2.9403,) (5.9005,) (0.    ,) (1.9602,)]
#  [(1.    ,) (2.97  ,) (4.95  ,) (0.9801,) (1.98  ,)]
#  [(0.    ,) (3.    ,) (5.    ,) (0.99  ,) (2.    ,)]
#  [(0.    ,) (0.    ,) (0.    ,) (1.    ,) (0.    ,)]]


Instruction: Does PyTorch have a RandomState-like object for random number generation?

Input:
in numpy i can

import numpy as np

rs = np.random.RandomState(seed=0)

and then pass that object around, eg for dependency injection.

Does PyTorch have a similar interface? I can't find anything in the docs, but maybe i'm missing something.

Output: The closest thing would be torch.manual_seed, which sets the seed for generating random numbers and returns a torch.Generator. This thread here has more information, apparently there may be some inconsistencies depending on whether you are using GPU or a CPU.


Instruction: Images change color after unfolding using pytorch

Input:
I am new to pytorch, for one of my projects I am dividing a large image into smaller tiles/patches. I am using unfold to make this happen. My code is as follows

data = training_set[1][0].data.unfold(1, 64, 64).unfold(2, 64, 64).unfold(3, 64, 64)

After doing this I transpose the resultant matrix since the images are flipped, like this sample code

torch.t(data [0][0][0][0])

but the resultant images lose color, or get discolored for some reason, and I am worried that this might affect any calculations I do based on these patches.

The following is a screenshot of the problem

The top is the patch and the bottom one is the complete picture

Any help is appreciated, thanks

Output:
I think your dataset is probably fine.

I had a similar issue in the past related to this. In this situation I have the feeling that the culprit is the matplotlib.imshow() function.

It would be helpful if you could share the complete code you have used to plot the matplotlib figure.

You are most likely taking a RGBA instead of a regular RGB as input into the plt.imshow() function. Thus the color are just because you also display an alpha value (A) on top of the regular red green blue (RGB).

If that is the case I would suggest that you try to plot this

image = torch.t(data [0][0][0])


Instruction: I've installed PyTorch, but CUDA is not recognized

Input:
I installed PyTorch using this command:

pip3 install torch===1.2.0 torchvision===0.4.0 -f https://download.pytorch.org/whl/torch_stable.html

And it seemed to work. When importing torch, there isn't any error.
My laptop has a GeForce 1060 TI, which I assumed would work with CUDA.

Here is the error in the IDE (Eclipse):

Output:
You should include code, not a graph!

Remove PyTorch and install it again with this command:

conda install pytorch -c pytorch
pip3 install torchvision

after this try checking it with this command:

import torch
import torchvision
train_on_gpu = torch.cuda.is_available()

if train_on_gpu:
    print('CUDA is available, Training on GPU ...')
else:
    print('CUDA is not available!  Training on CPU ...')


Instruction: PyTorch [1 if x > 0.5 else 0 for x in outputs ] with tensors

Input:
I have a list outputs from a sigmoid function as a tensor in PyTorch

E.g

output (type) = torch.Size([4]) tensor([0.4481, 0.4014, 0.5820, 0.2877], device='cuda:0',

As I'm doing binary classification I want to turn all values bellow 0.5 to 0 and above 0.5 to 1.

Traditionally with a NumPy array you can use list iterators:

output_prediction = [1 if x > 0.5 else 0 for x in outputs ]

This would work, however I have to later convert output_prediction back to a tensor to use

torch.sum(ouput_prediction == labels.data)

Where labels.data is a binary tensor of labels.

Is there a way to use list iterators with tensors?

Output:
prob = torch.tensor([0.3,0.4,0.6,0.7])

out = (prob>0.5).float()
# tensor([0.,0.,1.,1.])

Explanation: In pytorch, you can directly use prob>0.5 to get a torch.bool type tensor. Then you can convert to float type via .float().


Instruction: Pytorch differentiable conditional (index-based) sum

Input:
I have an idx array like [0, 1, 0, 2, 3, 1] and another 2d array data like the following:

[[0,  1,  2],
 [3,  4,  5],
 [6,  7,  8],
 [9,  10, 11],
 [12, 13, 14],
 [15, 16, 17]]

I want my output to be 4x3 in which 4 is the max of idx and 3 is the feature size (data.shape[1]) and in the output each element is the sum of features with corresponding index in the idx. Then output in this example would be:

[[6,  8,  10],
 [18, 20, 22],
 [9,  10, 11],
 [12, 13, 14]]

I can do it with iterating over range(3) and creating a mask on data and summing up them but it's not differentiable (I suppose). Is there any function(s) in Pytorch for this purpose? something like scatter().

Update: It seems I am looking for something named scatter sum which is implemented in this repository.

Output:
You are looking for index_add_:

import torch

x = torch.tensor([[ 0.,  1.,  2.],
        [ 3.,  4.,  5.],
        [ 6.,  7.,  8.],
        [ 9., 10., 11.],
        [12., 13., 14.],
        [15., 16., 17.]], dtype=torch.float)
idx = torch.tensor([0, 1, 0, 2, 3, 1], dtype=torch.long)  # note the dtype here, must be "long"
# init the sums to zero
y = torch.zeros((idx.max()+1, x.shape[1]), dtype=x.dtype)

# do the magic
y.index_add_(0, idx, x)

Gives the desired output

tensor([[ 6.,  8., 10.],
        [18., 20., 22.],
        [ 9., 10., 11.],
        [12., 13., 14.]])


Instruction: Attention Text Generation in Character-by-Character fashion

Input:
I am searching the web for a couple of days for any text generation model that would use only attention mechanisms.

The Transformer architecture that made waves in the context of Seq-to-Seq models is actually based solely on Attention mechanisms but is mainly designed and used for translation or chat bot tasks so it doesn't fit to the purpose, but the principle does.

My question is:

Does anyone knows or heard of a text generation model based solely on Attention without any recurrence?

Thanks a lot!

P.S. I'm familiar with PyTorch.

Output:
Building a character-level self-attentive model is a challenging task. Character-level models are usually based on RNNs. Whereas in a word/subword model, it is clear from the beginning what are the units carrying meaning (and therefore the units the attention mechanism can attend to), a character-level model needs to learn word meaning in the following layers. This makes it quite difficult for the model to learn.

Text generation models are nothing more than conditional languages model. Google AI recently published a paper on Transformer character language model, but it is the only work I know of.

Anyway, you should consider either using subwords units (as BPE, SentencePiece) or if you really need to go for character level, use RNNs instead.


Instruction: padding zeroes to a tensor on both dimensions

Input:
I have a tensor t

1 2
3 4
5 6
7 8

And I would like to make it

0 0 0 0
0 1 2 0
0 3 4 0
0 5 6 0
0 7 8 0
0 0 0 0

I tried stacking with new=torch.tensor([0. 0. 0. 0.]) tensor four times but that did not work.

t = torch.arange(8).reshape(1,4,2).float()
print(t)
new=torch.tensor([[0., 0., 0.,0.]])
print(new)
r = torch.stack([t,new])  # invalid argument 0: Tensors must have same number of dimensions: got 4 and 3
new=torch.tensor([[[0., 0., 0.,0.]]])
print(new)
r = torch.stack([t,new])  # invalid argument 0: Sizes of tensors must match except in dimension 0.

I also tried cat, that did not work either.

Output:
It is probably better to initialize an array of the desired shape first, and then add the data at the appropriate indices.

import torch

t = torch.arange(8).reshape(1,4,2).float()
x = torch.zeros((1, t.shape[1]+2, t.shape[2]+2))
x[:, 1:-1, 1:-1] = t

print(x)

On the other hand, if you just want to pad your tensor with zeroes (and not just add extra zeroes somewhere), you can use torch.nn.functional.pad:

import torch

t = torch.arange(8).reshape(1, 4, 2).float()
x = torch.nn.functional.pad(t, (1, 1, 1, 1))

print(x)


Instruction: How to add to pytorch tensor at indices?

Input:
I have to admit, I'm a bit confused by the scatter* and index* operations - I'm not sure any of them do exactly what I'm looking for, which is very simple:

Given some 2-D tensor

z = tensor([[1., 1., 1., 1.],
            [1., 1., 1., 1.],
            [1., 1., 1., 1.]])

And a list (or tensor?) of 2-d indexes:

inds = tensor([[0, 0],
               [1, 1],
               [1, 2]])

I want to add a scalar to z at those indexes (and do it efficiently):

znew = z.something_add(inds, 3)
->
znew = tensor([[4., 1., 1., 1.],
               [1., 4., 4., 1.],
               [1., 1., 1., 1.]])

If I have to I can make that scalar a tensor of whatever shape (where all elements = 3), but I'd rather not...

Output:
You must provide two lists to your indexing. The first having the row positions and the second the column positions. In your example, it would be:

z[[0, 1, 1], [0, 1, 2]] += 3

torch.Tensor indexing follows Numpy. See https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#integer-array-indexing for more details.


Instruction: Truncate SVD decomposition of Pytorch tensor without transfering to cpu

Input:
I'm training a model in Pytorch and I want to use truncated SVD decomposition of input. For calculating SVD I transfer input witch is a Pytorch Cuda Tensor to CPU and using TruncatedSVD from scikit-learn perform truncate, after that, I transfer the result back to GPU. The following is code for my model:

 class ImgEmb(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(ImgEmb, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.drop = nn.Dropout(0.2)
        self.mlp = nn.Linear(input_size/2, hidden_size)
        self.relu = nn.Tanh()
        self.svd = TruncatedSVD(n_components=input_size/2)

    def forward(self, input):
        svd=self.svd.fit_transform(input.cpu())
        svd_tensor=torch.from_numpy(svd)
        svd_tensor=svd_tensor.cuda()
        mlp=self.mlp(svd_tensor)
        res = self.relu(mlp)
        return res

I wonder is a way to implement truncated SVD without transferring back and forth to GPU? (Because it's very time consuming and is not efficient at all)

Output:
You could directly use PyTorch's SVD and truncate it manually, or you can use the truncated SVD from TensorLy, with the PyTorch backend:

import tensorly as tl
tl.set_backend('pytorch')

U, S, V = tl.truncated_svd(matrix, n_eigenvecs=10)

However, the GPU SVD does not scale very well on large matrices. You can also use TensorLy's partial svd which will still copy your input to CPU but will be much faster if you keep only a few eigenvalues as it will use a sparse eigendecomposition. In Scikit-learn's truncated SVD, you can also use 'algorithm = arpack' to use Scipy's sparse SVD which again might be faster if you only need a few components.


Instruction: sklearn pipeline with multiple inputs/outputs

Input:
How can I build a sklearn pipeline to do the following?

What I have:

A, B = getAB(X_train)
X_train = transform(X_train)
model(A, B, X_train)

What I want:

pipe = Pipeline([
(‘ab’, getAB),
(‘tranf’, transform),
(‘net’, net)
]
pipe.fit(X_train, y_train)

Please help!

Output:
Yes it is doable by writing a custom transformer that has a fit/transform function. This can be your class:

from sklearn.base import BaseEstimator, TransformerMixin

def getABTransformer(BaseEstimator, TransformerMixin):
    def __init__(self): # no *args or **kargs
        pass

    def fit(self, X, y=None):
        return self # nothing else to do

    def transform(self, X, y=None):
        return getAB(X)

Then you can create your ColumnTransformer as following:

from sklearn.compose import ColumnTransformer

clm_pipe = ColumnTransformer([
(‘ab’, getABTransformer, np.arange(0, len(X_train)),  # list of columns indices
(‘tranf’, transform, np.arange(0, len(X_train))),  # list of columns indices
]

and a final pipeline with the model:

pipe = Pipeline([
(‘clm_pipe’, clm_pipe),
(‘net’, net)
]

You can read more about ColumnTransformer


Instruction: Pytorch's stack() adds dimension?

Input:
I am looking to stack one shape=(1,2) tensor array on top of another shape=(1,2) array, across dim=1, using pytorch's stack() method.

>>> import numpy as np
>>> import torch
>>> np_a = np.array([[1,2]])
>>> np_b = np.array([[3,4]])
>>> print(np_a)
[[1 2]]
>>> print(np_b)
[[3 4]]
>>> t_a = torch.from_numpy(np_a)
>>> t_b = torch.from_numpy(np_b)
>>> print(t_a)
tensor([[1, 2]])
>>> print(t_b)
tensor([[3, 4]])
>>> t_stacked = torch.stack((t_a, t_b), dim=1)
>>> print(t_stacked)
tensor([[[1, 2],
         [3, 4]]])

The resulting tensor has an added dimension and now has a shape=(1,2,2).  Why doesn't pytorch's stack() behave like numpy's vstack()?  See below:

>>> import numpy as np
>>> np_a = np.array([[1,2]])
>>> np_b = np.array([[3,4]])
>>> stacked = np.vstack((np_a, np_b))
>>> print(stacked)
[[1 2]
 [3 4]]

How do I make pytorch not add a dimension?

Output:
I can't tell you why it was decided to have pytorch's stack behave differently from numpy (maybe compatibility with luatorch?). Anyway, to get the desired outcome you can use torch.cat:

>>> torch.cat((t_a, t_b), dim=0)
tensor([[1, 2],
        [3, 4]])


Instruction: Training a model with multiple learning rate in PyTorch

Input:
I am new to PyTorch and getting used to some concepts.

I need to train a Neural Network. For optimization, I need to use Adam optimizer with 4 different learning rates = [2e-5, 3e-5, 4e-5, 5e-5]

The optimizer function is defined as below

def optimizer(no_decay = ['bias', 'gamma', 'beta'], lr=2e-5):
    param_optimizer = list(model.named_parameters())
    optimizer_grouped_parameters = [
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.01},
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.0}
     ]
     # This variable contains all of the hyperparemeter information our training loop needs
     optimizer = BertAdam(optimizer_grouped_parameters, lr, warmup=.1)
     return optimizer

How do I make sure the optimizer uses my specified set of learning rate and returns the best model?

During training, we use the optimizer like below where I don't see a way to tell it to try different learning rate

def model_train():
    #other code
    # clear out the gradient
    optimizer.zero_grad()
    # Forward pass
    loss = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
    train_loss_set.append(loss.item())
    # Backward pass
    loss.backward()
    # Update parameters and take a step using the computed gradient
    optimizer.step()

I know that the optimizer.step() internally steps through to optimize the gradient. But how do I make sure the optimizer tries my specified set of learning rate and returns the best model to me?

Please suggest.

Output:
If you want to train four times with four different learning rates and then compare you need not only four optimizers but also four models: Using different learning rate (or any other meta-parameter for this matter) yields a different trajectory of the weights in the high-dimensional "parameter space". That is, after a few steps its not only the learning rate that differentiate between the models, but the trained weights themselves - this is what yield the actual difference between the models.

therefore, you need to train 4 times using 4 separate instances of model using 4 instances of optimizer with different learning rates.


Instruction: Pytorch simple model not improving

Input:
I am making a simple PyTorch neural net to approximate the sine function on x = [0, 2pi]. This is a simple architecture I use with different deep learning libraries to test whether I understand how to use it or not. The neural net, when untrained, always produces a straight horizontal line, and when trained, produces a straight line at y = 0. In general, it always produces a straight line at y = (The mean of the function). This leads me to believe something is wrong with the forward prop portion of it, as the boundary should not just be a straight line when untrained. Here is the code for the net:

class Net(nn.Module):
    def __init__(self):
      super(Net, self).__init__()
      self.model = nn.Sequential(
      nn.Linear(1, 20),
      nn.Sigmoid(),
      nn.Linear(20, 50),
      nn.Sigmoid(),
      nn.Linear(50, 50),
      nn.Sigmoid(),
      nn.Linear(50, 1)
      )

    def forward(self, x):
        x = self.model(x)
        return x

Here is the training loop

def train(net, trainloader, valloader, learningrate, n_epochs):
    net = net.train()
    loss = nn.MSELoss()
    optimizer = torch.optim.SGD(net.parameters(), lr = learningrate)

    for epoch in range(n_epochs):

        for X, y in trainloader:
            X = X.reshape(-1, 1)
            y = y.view(-1, 1)
            optimizer.zero_grad()

            outputs = net(X)

            error   = loss(outputs, y)
            error.backward()
            #net.parameters()  net.parameters() * learningrate
            optimizer.step()

        total_loss = 0
        for X, y in valloader:
            X = X.reshape(-1, 1).float()
            y = y.view(-1, 1)
            outputs = net(X)
            error   = loss(outputs, y)
            total_loss += error.data

        print('Val loss for epoch', epoch, 'is', total_loss / len(valloader) )

it is called as:

net = Net()
losslist = train(net, trainloader, valloader, .0001, n_epochs = 4)

Where trainloader and valloader are the training and validation loaders. Can anyone help me see what's wrong with this? I know its not the learning rate since its the one I use in other frameworks, and I know its not the fact im using SGD or sigmoid activation functions, although I have a suspicion the error is in the activation functions somewhere.

Does anyone know how to fix this? Thanks.

Output:
After a while playing with some hyperparameters, modifying the net and changing the optimizer (following this excellent recipe) I ended up with changing the line optimizer = torch.optim.SGD(net.parameters(), lr = learningrate) to optimizer = torch.optim.Adam(net.parameters()) (the default optimizer parameters was used), running for 100 epochs and batch size equal to 1.

The following code was used (tested on CPU only):

import torch
import torch.nn as nn
from torch.utils import data
import numpy as np
import matplotlib.pyplot as plt

# for reproducibility
torch.manual_seed(0)
np.random.seed(0)

class Dataset(data.Dataset):

    def __init__(self, init, end, n):

        self.n = n
        self.x = np.random.rand(self.n, 1) * (end - init) + init
        self.y = np.sin(self.x)

    def __len__(self):

        return self.n

    def __getitem__(self, idx):

        x = self.x[idx, np.newaxis]
        y = self.y[idx, np.newaxis]

        return torch.Tensor(x), torch.Tensor(y)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.model = nn.Sequential(
        nn.Linear(1, 20),
        nn.Sigmoid(),
        nn.Linear(20, 50),
        nn.Sigmoid(),
        nn.Linear(50, 50),
        nn.Sigmoid(),
        nn.Linear(50, 1)
        )

    def forward(self, x):
        x = self.model(x)
        return x

def train(net, trainloader, valloader, n_epochs):

    loss = nn.MSELoss()
    # Switch the two following lines and run the code
    # optimizer = torch.optim.SGD(net.parameters(), lr = 0.0001)
    optimizer = torch.optim.Adam(net.parameters())

    for epoch in range(n_epochs):

        net.train()
        for x, y in trainloader:
            optimizer.zero_grad()
            outputs = net(x).view(-1)
            error   = loss(outputs, y)
            error.backward()
            optimizer.step()

        net.eval()
        total_loss = 0
        for x, y in valloader:
            outputs = net(x)
            error   = loss(outputs, y)
            total_loss += error.data

        print('Val loss for epoch', epoch, 'is', total_loss / len(valloader) )

    net.eval()

    f, (ax1, ax2) = plt.subplots(1, 2, sharey=True)

    def plot_result(ax, dataloader):
        out, xx, yy = [], [], []
        for x, y in dataloader:
            out.append(net(x))
            xx.append(x)
            yy.append(y)
        out = torch.cat(out, dim=0).detach().numpy().reshape(-1)
        xx = torch.cat(xx, dim=0).numpy().reshape(-1)
        yy = torch.cat(yy, dim=0).numpy().reshape(-1)
        ax.scatter(xx, yy, facecolor='green')
        ax.scatter(xx, out, facecolor='red')
        xx = np.linspace(0.0, 3.14159*2, 1000)
        ax.plot(xx, np.sin(xx), color='green')

    plot_result(ax1, trainloader)
    plot_result(ax2, valloader)
    plt.show()

train_dataset = Dataset(0.0, 3.14159*2, 100)
val_dataset = Dataset(0.0, 3.14159*2, 30)

params = {'batch_size': 1,
          'shuffle': True,
          'num_workers': 4}

trainloader = data.DataLoader(train_dataset, **params)
valloader = data.DataLoader(val_dataset, **params)

net = Net()
losslist = train(net, trainloader, valloader, n_epochs = 100)

Result with Adam optimizer:

Result with SGD optimizer:


Instruction: How to have incrementing batch size in pytorch

Input:
In pytorch, DataLoader will split a dataset into batches of set size with additional options of shuffling etc, which one can then loop over.

But if I need the batch size to increment, such as first 10 batch of size 50, next 5 batch of size 100 and so on, what's the best way of doing so?

I tried splitting the tensor then concat them:

#10x50 + 5*100
originalTensor = torch.randn(1000, 80)
split1=torch.split(originalTensor, 500, dim=0)
split2=torch.split(list(split1)[0], 100, dim=0)

Thereafter is there a way to pass the concatenated tensor into dataLoader or any other way to directly turn the concat tensor into a generator (which might lose shuffling and other functionalities)?

Output:
I think you can do that by simply providing a non-default batch_sampler to your DataLoader.
For instance:

class VaryingSizeBatchSampler(Sampler):
    r"""Wraps another sampler to yield a varying-size mini-batch of indices.

    Args:
        sampler (Sampler): Base sampler.
        batch_size_fn (function): Size of current mini-batch.
        drop_last (bool): If ``True``, the sampler will drop the last batch if
            its size would be less than ``batch_size``
    """

    def __init__(self, sampler, batch_size_fn, drop_last):
        if not isinstance(sampler, Sampler):
            raise ValueError("sampler should be an instance of "
                             "torch.utils.data.Sampler, but got sampler={}"
                             .format(sampler))
        self.sampler = sampler
        self.batch_size_fn = batch_size_fn
        self.drop_last = drop_last
        self.batch_counter = 0

    def __iter__(self):
        batch = []
        cur_batch_size = self.batch_size_fn(self.batch_counter)  # get current batch size
        for idx in self.sampler:
            batch.append(idx)
            if len(batch) == cur_batch_size:
                yield batch
                self.batch_counter += 1
                cur_batch_size = self.batch_size_fn(self.batch_counter)  # get current batch size
                batch = []
        if len(batch) > 0 and not self.drop_last:
            yield batch

    def __len__(self):
        raise NotImplementedError('You need to implement it yourself!')


Instruction: why does the Accuracy decrease when using a ReLu activation after Linear layers

Input:
so I started using Pytorch and I'm building a very basic CNN on the FashionMNIST Dataset. I noticed some weird Behaviour while using the NN and I don't know why this is happening, in the Forward Function the Accuracy of the NN decrease when I use a Relu function after every Linear Layer.

here is the Code for my custom NN:

# custom class neural network
class FashionMnistClassifier(nn.Module):
  def __init__(self, n_inputs, n_out):
    super().__init__()
    self.cnn1 = nn.Conv2d(n_inputs, out_channels=32, kernel_size=5).cuda(device)
    self.cnn2 = nn.Conv2d(32, out_channels=64, kernel_size=5).cuda(device)
    #self.cnn3 = nn.Conv2d(n_inputs, out_channels=32, kernel_size=5)
    self.fc1 = nn.Linear(64*4*4, out_features=100).cuda(device)
    self.fc2 = nn.Linear(100, out_features=n_out).cuda(device)
    self.relu = nn.ReLU().cuda(device)
    self.pool = nn.MaxPool2d(kernel_size=2).cuda(device)
    self.soft_max = nn.Softmax().cuda(device)

  def forward(self, x):
    x.cuda(device)
    out = self.relu(self.cnn1(x))
    out = self.pool(out)
    out = self.relu(self.cnn2(out))
    out = self.pool(out)
    #print("out shape in classifier forward func: ", out.shape)
    out = self.fc1(out.view(out.size(0), -1))
    #out = self.relu(out) # if I uncomment these then the Accuracy decrease from 90 to 50!!!
    out = self.fc2(out)
    #out = self.relu(out) # this too
    return out

n_batch = 100
n_outputs = 10
LR = 0.001

model = FashionMnistClassifier(1, 10).cuda(device)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.CrossEntropyLoss()

so if I use the ReLu only after the CNN layers I get an Accuracy of 90% but when I uncomment that section and use the Relu activation after the Linear Layers the Accuracy decrease to 50% I have no idea why is this happening since I thought it is always better to use activation after every linear layer to get a better accuracy for Classification. I thought always that we should always use the activation function if we have a Classification Problem and for Linear Regression we don't have to do this, but here in my case although it is a Classification Problem, I get a better performance if I don't use the Activation Function after the Linear Layers. Can someone maybe clarify this to me?

Output:
CrossEntropyLoss requires you to pass in unnormalized logits (output from last Linear layer).

If you use ReLU as the output from last layer you are only outputting values in the range [0, inf), while neural network tends to go with small values for incorrect labels and high for the correct ones (we may say it's overconfident in it's predictions). Oh, and the one with highest logit value is chosen by argmax as correct label.

So it won't definitely work with this line:

# out = self.relu(out) # this too

Though it should with the ReLU before it. Just remember, more nonlinearity isn't always good for the network.


Instruction: How to use tensorboard debugger with pytorch?

Input: I know pytorch starts to support tensorboard since the version 1.11. But I am wondering is it possible for us to use the debugger plugin for tensorboard with pytorch? I didn't find any information about this. If pytorch can also support tensorboard debugger, it would be extremely convenient and could save us a lot of time.

Output: No, it is not and to doubt it ever will be possible. The plugin heavily relies on how TensorFlow internally represents graph nodes which is quite different in PyTorch.


Instruction: Extracting reduced dimension data from autoencoder in pytorch

Input:
I have defined my autoencoder in pytorch as following:

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 64, kernel_size=1, stride=1),
    nn.ReLU()
)

self.decoder = nn.Sequential(
    nn.Conv2d(64, 64, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Sigmoid()
)

I need to get a reduced dimension encoding which requires creating a new linear layer of the dimension N much lower than the image dimension so that I can extract the activations.

If anybody can help me with fitting a linear layer in the decoder part I would appreciate (i know how to Flatten() the data, but I guess I need to "unflatten" it again to interface with the Conv2d layer again)

Update:

I have come up with the following based on the first answer (it gives me a 8-dimensional bottleneck at the output of the encoder which works fine torch.Size([1, 8, 1, 1]) ).

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(64, 8, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.MaxPool2d(7, stride=1)
)

self.decoder = nn.Sequential(
    nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Sigmoid()
)

What I cannot do is train the autoencoder with

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

The decoder gives me an error:

Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size

I would like to thank the person who provided the first answer.

Output:
In the decoder part, you need to upsample to larger size, which can be done via nn.ConvTranspose2d. I notice that in your encoder part, seems like you didn't downsample your feature maps, because your stride is alwasy 1. Here is a toy example.

self.encoder = nn.Sequential(
            nn.Conv2d(32, 16, 3, stride=1, padding=1),  # b, 16, 32, 32
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 16, 16
            nn.Conv2d(16, 32, 3, stride=1, padding=1),  # b, 32, 16, 16
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2)  # b, 32, 8, 8
        )
self.decoder = nn.Sequential(
            nn.ConvTranspose2d(32, 16, 3, stride=2,padding=1,output_padding=1),  # b, 16, 16, 16
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1),  # b, 1, 32, 32
            nn.Sigmoid()
        )


Instruction: How to generate new tensor by given indexes and tensor in pytorch?

Input:
I have a tensor like this:

x = torch.tensor([[3, 4, 2], [0, 1, 5]])

and I have a indexes like this:

ind = torch.tensor([[1, 1, 0], [0, 0, 1]])

then I want to generate a new tensor by x and ind:

z = torch.tensor([0, 1, 2], [3, 4, 5])

I impement it with python like this:

# -*- coding: utf-8 -*-

import torch

x = torch.tensor([[3, 4, 2], [0, 1, 5]])
ind = torch.tensor([[1, 1, 0], [0, 0, 1]])

z = torch.zeros_like(x)
for i in range(x.shape[0]):
    for j in range(x.shape[1]):
        z[i, j] = x[ind[i][j]][j]

print(z)

I want to know how to solve this by pytorch?

Output:
You are looking for torch.gather

In [1]: import torch

In [2]: x = torch.tensor([[3, 4, 2], [0, 1, 5]])

In [3]: ind = torch.tensor([[1, 1, 0], [0, 0, 1]])

In [4]: torch.gather(x, 0, ind)
Out[4]:
tensor([[0, 1, 2],
        [3, 4, 5]])


Instruction: Pytorch: ValueError: Expected input batch_size (32) to match target batch_size (64)

Input:
Tried to run the CNN examples on MNIST dataset, batch size=64, channel =1, n_h=28, n_w=28, n_iters = 1000. The program runs for first 500 interation and then gives the above mentioned error.
There are same topics already being discussed on the forum such as : topic 1
and topic 2, but none of them could help me identify the mistake in the following code:

class CNN_MNIST(nn.Module):
def __init__(self):
    super(CNN_MNIST,self).__init__()

    # convolution layer 1
    self.cnn1 = nn.Conv2d(in_channels=1, out_channels= 32, kernel_size=5,
                          stride=1,padding=2)

    # ReLU activation
    self.relu1 = nn.ReLU()

    # maxpool 1
    self.maxpool1 = nn.MaxPool2d(kernel_size=2,stride=2)

    # convolution 2
    self.cnn2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=5,
                          stride=1,padding=2)

    # ReLU activation
    self.relu2 = nn.ReLU()

    # maxpool 2
    self.maxpool2 = nn.MaxPool2d(kernel_size=2,stride=2)

    # fully connected 1
    self.fc1 = nn.Linear(7*7*64,1000)
    # fully connected 2
    self.fc2 = nn.Linear(1000,10)

def forward(self,x):

    # convolution 1
    out = self.cnn1(x)
    # activation function
    out = self.relu1(out)
    # maxpool 1
    out = self.maxpool1(out)

    # convolution 2
    out = self.cnn2(out)
    # activation function
    out = self.relu2(out)
    # maxpool 2
    out = self.maxpool2(out)

    # flatten the output
    out = out.view(out.size(0),-1)

    # fully connected layers
    out = self.fc1(out)
    out = self.fc2(out)

    return out
# model trainning
count = 0
loss_list = []
iteration_list = []
accuracy_list = []

for epoch in range(int(n_epochs)):
    for i, (image,labels) in enumerate(train_loader):

    train = Variable(image)
    labels = Variable(labels)

    # clear gradient
    optimizer.zero_grad()

    # forward propagation
    output = cnn_model(train)

    # calculate softmax and cross entropy loss
    loss = error(output,label)

    # calculate gradients
    loss.backward()

    # update the optimizer
    optimizer.step()

    count += 1

    if count % 50 ==0:
        # calculate the accuracy
        correct = 0
        total = 0

        # iterate through the test data
        for image, labels in test_loader:

            test = Variable(image)

            # forward propagation
            output = cnn_model(test)

            # get prediction
            predict = torch.max(output.data,1)[1]

            # total number of labels
            total += len(labels)

            # correct prediction
            correct += (predict==labels).sum()

        # accuracy
        accuracy = 100*correct/float(total)

        # store loss, number of iteration, and accuracy
        loss_list.append(loss.data)
        iteration_list.append(count)
        accuracy_list.append(accuracy)

        # print loss and accurcay as the algorithm progresses
        if count % 500 ==0:
            print('Iteration :{}    Loss :{}    Accuracy :

{}'.format(count,loss.item(),accuracy))

The error is as follows:

    ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-19-9e93a242961b> in <module>
     18
     19         # calculate softmax and cross entropy loss
---> 20         loss = error(output,label)
     21
     22         # calculate gradients

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~\Anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   1993     if size_average is not None or reduce is not None:
   1994         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 1995     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   1996
   1997

~\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1820     if input.size(0) != target.size(0):
   1821         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 1822                          .format(input.size(0), target.size(0)))
   1823     if dim == 2:
   1824         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (32) to match target batch_size (64).

Output:
You are providing the wrong target to your loss:
loss = error(output, label)

While your loader gives you
    for i, (image,labels) in enumerate(train_loader):

      train = Variable(image)
      labels = Variable(labels)

So you have a variable name labels (with s)  from the loader, yet you feed label (no s) to your loss.
Batch size is the least of your worries.


Instruction: Pytorch autograd.grad how to write the parameters for multiple outputs?

Input:
In the documentation of torch.autograd.grad, it is stated that, for parameters,

  parameters:

  outputs (sequence of Tensor) – outputs of the differentiated function.

  inputs (sequence of Tensor) – Inputs w.r.t.
  which the gradient will be returned (and not accumulated into .grad).

I try the following:

a = torch.rand(2, requires_grad=True)
b = torch.rand(2, requires_grad=True)
c = a+b
d = a-b

torch.autograd.grad([c, d], [a, b]) #ValueError: only one element tensors can be converted to Python scalars
torch.autograd.grad(torch.tensor([c, d]), torch.tensor([a, b])) #RuntimeError: grad can be implicitly created only for scalar outputs

I would like to get gradients of a list of tensors w.r.t another list of tensors. What is the correct way to feed the parameters?

Output:
As the torch.autograd.grad mentioned, torch.autograd.grad computes and returns the sum of gradients of outputs w.r.t. the inputs. Since your c and d are not scalar values, grad_outputs are required.

import torch

a = torch.rand(2,requires_grad=True)
b = torch.rand(2, requires_grad=True)

a
# tensor([0.2308, 0.2388], requires_grad=True)

b
# tensor([0.6314, 0.7867], requires_grad=True)

c = a*a + b*b
d = 2*a+4*b

torch.autograd.grad([c,d], inputs=[a,b], grad_outputs=[torch.Tensor([1.,1.]), torch.Tensor([1.,1.])])
# (tensor([2.4616, 2.4776]), tensor([5.2628, 5.5734]))

Explanation:
dc/da = 2*a = [0.2308*2, 0.2388*2]
dd/da = [2.,2.]
So the first output is dc/da*grad_outputs[0]+dd/da*grad_outputs[1] = [2.4616, 2.4776]. Same calculation for the second output.

If you just want to get the gradient of c and d w.r.t. the inputs, probably you can do this:

a = torch.rand(2,requires_grad=True)
b = torch.rand(2, requires_grad=True)

a
# tensor([0.9566, 0.6066], requires_grad=True)
b
# tensor([0.5248, 0.4833], requires_grad=True)

c = a*a + b*b
d = 2*a+4*b

[torch.autograd.grad(t, inputs=[a,b], grad_outputs=[torch.Tensor([1.,1.])]) for t in [c,d]]
# [(tensor([1.9133, 1.2132]), tensor([1.0496, 0.9666])),
# (tensor([2., 2.]), tensor([4., 4.]))]


Instruction: Pytorch GPU memory increase after load operation

Input:
I have a pytorch model which is of the size 386MB, but when i load the model

state = torch.load(f, flair.device)

My GPU memory takes up to 900MB, why does this happen and is there a way to resolve this?

This is how i save the model

model_state = self._get_state_dict()

# additional fields for model checkpointing
model_state["optimizer_state_dict"] = optimizer_state
model_state["scheduler_state_dict"] = scheduler_state
model_state["epoch"] = epoch
model_state["loss"] = loss

torch.save(model_state, str(model_file), pickle_protocol=4)

Output:
It's probably the optimizer_state that takes the extra space. Some optimizers (e.g., Adam) track statistics of each trainable parameter, such as first and second order moments. This information takes up space, as you can tell.

You can load to CPU first:

state = torch.load(f, map_location=torch.device('cpu'))


Instruction: Compute cross entropy loss for classification in pytorch

Input:
I am trying to build two neural network for classification. One for Binary and the second is for multi-class classification. I am trying to use the torch.nn.CrossEntropyLoss() as a loss function, but I try to train my first neural network I get the following error:

multi-target not supported at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THNN/generic/ClassNLLCriterion.c:22

From my analysis, I found that the my dataset has two problems that caused the error.

My data set is one hot encoded. I used one hot encoding to pre processes my dataset. The first target Y_binary variable has the shape of torch.Size([125973, 1]) full of 0s and 1 indicating classes 'No' and 'Yes'.
My data has the wrong dimensions? I found that I can't use a simple vector with the cross entropy loss function. Some people used the following code to reshape their target vector before feeding to the loss function.

out = out.permute(0, 2, 3, 1).contiguous().view(-1, class_number)

But I didn't really understand the reasoning behind this code. But it seems for my that I need to keep track of the following variables: Class_Number, Batch_size, Dimension_Output. For my code here are the dimensions

X_train.shape: (125973, 122)
Y_train2.shape: (125973, 1)
batch_size = 64
K = len(set(Y_train2)) # Binary classification For multi class classification use K = len(set(Y_train5))

Should the target value be one hot encoded? If not, how I can feed a nominal feature to the loss function?
If I use reshape the output, can you help me do this for my code ?

I am trying to use this loss function for both my neural networks.

Thank you in advance,

Output:
The error is due to the usage of torch.nn.CrossEntropyLoss() which can be used if you want to predict 1 class out of N classes. For multiclass classification, you should use torch.nn.BCEWithLogitsLoss() which combines a Sigmoid layer and the BCELoss in one single class.

In case of multi-class, and if you use Sigmoid + BCELoss, then you need the target to be one-hot encoding, i.e. something like this per sample: [0 1 0 0 0 1 0 0 1 0], where 1 will be at the locations of classes present.


Instruction: Upsampling an autoencoder in pytorch

Input:
I have defined my autoencoder in pytorch as following (it gives me a 8-dimensional bottleneck at the output of the encoder which works fine torch.Size([1, 8, 1, 1])):

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(64, 8, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.MaxPool2d(7, stride=1)
)

self.decoder = nn.Sequential(
    nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Sigmoid()
)

What I cannot do is train the autoencoder with

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

The decoder gives me an error that the decoder cannot upsample the tensor:

Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size

Output:
You are not upsampling enough via ConvTranspose2d, shape of your encoder is only 1 pixel (width x height), see this example:

import torch

layer = torch.nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1)
print(layer(torch.randn(64, 8, 1, 1)).shape)

This prints your exact (3,3) shape after upsampling.

You can:

Make the kernel smaller - instead of 4 in first Conv2d in decoder use 3 or 2 or even 1
Upsample more, for example: torch.nn.ConvTranspose2d(8, 64, kernel_size=7, stride=2) would give you 7x7
What I would do personally: downsample less in encoder, so output shape after it is at least 4x4 or maybe 5x5. If you squash your image so much there is no way to encode enough information into one pixel, and even if the code passes the network won't learn any useful representation.


Instruction: Concept of mini batch in deep generative model using pyro

Input:
I am new to probabilistic programming and ML. I am following a code on deep Markov model given on pyro's website. The link to the github page to that code is:

https://github.com/pyro-ppl/pyro/blob/dev/examples/dmm/dmm.py

I understand most part of the code. The part I don't understand is mini batch idea they are using from line 175.

Question 1:
Could someone explain what are they doing there when they are using mini-batch?

In pyro documentation they say

  mini_batch is a three dimensional tensor, with the first dimension being the batch dimension, the second dimension being the temporal dimension, and the final dimension being the features (88-dimensional in our case)'

Question 2:
What does temporal dimension means here?

Because I want to use this code on my dataset which is a sequential data. I have done one hot encoding of my data such that it's dimension is (10000,500,20) where 10000 is the number of examples/Sequences, 500 is the length of each of these sequences and 20 is the number of features.

Question 3:
How can I use my one hot encoded data as mini batch here?

I'm sorry if it is a really basic question but, insights will be appreciated.

Link to that documentation is:

https://pyro.ai/examples/dmm.html

Output:
  Question 1: Could someone explain what are they doing there when they are using mini-batch?

To optimize most of the deep learning models, we use mini-batch gradient descent. Here, A mini_batch refers to a small number of examples. Let's say, we have 10,000 training examples and we want to create mini-batches of 50 examples. So, in total there will be 200 mini-batches and we will perform 200 parameter updates during one iteration over the entire dataset.

  Question 2: What does the temporal dimension mean here?

In your data: (10000, 500, 20), the second dimension refers to the temporal dimension. You can consider you have examples with 500 timesteps (t1, t2, ..., t500).

  Question 3: How can I use my one-hot encoded data as mini-batch here?

In your scenario, you can split your data (10000, 500, 20) into 200 small batches of size (50, 500, 20) where 50 is the number of examples/Sequences in the mini-batch, 500 is the length of each of these sequences and 20 is the number of features.

How do we decide the mini-batch size? Basically, we can tune the batch size just like any other hyperparameters of our model.


Instruction: Not Able to Import PyTorch in Conda Env

Input:
Last week I had a working conda env I was using for a project. I have not touched the project in a week. I just went to run a python file (python file.py) that had been running with no errors.

Now I get the following error:

Traceback (most recent call last):
  File "file.py", line 2, in <module>
    from torch.utils.data import Dataset, DataLoader
ModuleNotFoundError: No module named 'torch'

In an attempt to troubleshoot, I opened a python console and ran the following code:

>>> import torch

The result was the following error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'

If I check all the installed packages using conda list -n <env_name>, I can see that PyTorch is in fact installed, just as it was last week.

...
pytorch                   1.2.0           py3.7_cuda9.2.148_cudnn7.6.2_0    pytorch
...
torchvision               0.4.0                 py37_cu92    pytorch
...

Here is what I see when I start python console using python:

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux

The output of python -c 'import sys; print(sys.path) in the base env is:

['', '/home/<name>/anaconda3/lib/python37.zip', '/home/<name>/anaconda3/lib/python3.7', '/home/<name>/anaconda3/lib/python3.7/lib-dynload', '/home/<name>/anaconda3/lib/python3.7/site-packages'] I have not personally made any chages to PYTHONPATH.

If I run python -c 'import sys; print(sys.path)' with my conda env (non-base) active, I get:

['', '/home/<name>/anaconda3/envs/<env_name>/lib/python37.zip', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7/lib-dynload', '/home/<name>/anaconda3/envs/<env_name>/lib/python3.7/site-packages']

This is totally bizarre, can't figure out what is going on, and what could have happened, over the course of the last week, without me touching the code or making any changes to Anaconda.

Output:
open anaconda-prompt then run this

conda install PyTorch -c PyTorch
If you didn't upgrade your pip.use this command to update

python -m pip install –upgrade pip
After first step run this

pip3 install torchvision

hope it will work.


Instruction: How are token vectors calculated in spacy-pytorch-transformers

Input:
I am currently working with the spacy-pytorch-transformer package to experiment with the respective embeddings.
When reading the introductionary article (essentially the GitHub README), my understanding was that the token-level embeddings are the mean over the embeddings of all corresponding word pieces, i.e. embed(complex) would be the same as 1/2 * embed(comp#) * embed(#lex).

According to the BERT paper, this should simply utilize the last_hidden_state property of the network, but my MCVE below shows that this is not the same for Spacy 2.1.8 and spacy-pytorch-transformers 0.4.0, for at least BERT and RoBERTa (have not verified it for more models):

import spacy
import numpy as np
nlp = spacy.load("en_pytt_robertabase_lg")  # either this or the BERT model
test = "This is a test"  # Note that all tokens are directly aligned, so no mean has to be calculated.
doc = nlp(test)
# doc[0].vector and doc.tensor[0] are equal, so the results are equivalent.
print(np.allclose(doc[0].vector, doc._.pytt_last_hidden_state[1, :]))
# returns False

The offset of 1 for the hidden states is due to the <CLS> token as the first input, which corresponds to the sentence classification task; I even checked with any available other token for my sentence (which has no token alignment problems according to doc._.pytt_alignment), so there is no way I missed something here.

According to the source code, the corresponding hook overwrites simply to return the corresponding row in the tensor, so I do not see any transformation here. Is there something obvious that I am missing here, or is this deviating from the expected behavior?

Output:
It seems that there is a more elaborate weighting scheme behind this, which also accounts for the [CLS] and [SEP] token outputs in each sequence.

This has also been confirmed by an issue post from the spaCy developers.

Unfortunately, it seems that this part of the code has since moved with the renaming to spacy-transformers.


Instruction: How to fix 'Input and hidden tensors are not at the same device' in pytorch

Input:
When I want to put the model on the GPU, I get the following error:

"RuntimeError: Input and hidden tensors are not at the same device, found input tensor at cuda:0 and hidden tensor at cpu"

However, all of the above had been put on the GPU:
for m in model.parameters():
    print(m.device) #return cuda:0

if torch.cuda.is_available():
    model = model.cuda()
    test = test.cuda() # test is the Input

Windows 10 server
Pytorch 1.2.0 + cuda 9.2
cuda 9.2
cudnn 7.6.3 for cuda 9.2

Output:
You need to move the model, the inputs, and the targets to Cuda:
if torch.cuda.is_available():
   model.cuda()
   inputs = inputs.cuda()
   target = target.cuda()


Instruction: How to create variable names in loop for layers in pytorch neural network

Input:
I am implementing a straightforward feedforward neural newtork in PyTorch. However I am wondern if theres a nicer way to add a flexible amount of layer to the network? Maybe by naming them during a loop, but i heard thats impossible?

Currently I am doing it like this

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, input_dim, output_dim, hidden_dim):
        super(Net, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.layer_dim = len(hidden_dim)
        self.fc1 = nn.Linear(self.input_dim, self.hidden_dim[0])
        i = 1
        if self.layer_dim > i:
            self.fc2 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc3 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc4 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc5 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc6 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc7 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        if self.layer_dim > i:
            self.fc8 = nn.Linear(self.hidden_dim[i-1], self.hidden_dim[i])
            i += 1
        self.fcn = nn.Linear(self.hidden_dim[-1], self.output_dim)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.relu(self.fc1(x))
        i = 1
        if self.layer_dim > i:
            x = F.relu(self.fc2(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc3(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc4(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc5(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc6(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc7(x))
            i += 1
        if self.layer_dim > i:
            x = F.relu(self.fc8(x))
            i += 1
        x = F.softmax(self.fcn(x))
        return x

Output:
You can put your layers in a ModuleList container:

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self, input_dim, output_dim, hidden_dim):
        super(Net, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        current_dim = input_dim
        self.layers = nn.ModuleList()
        for hdim in hidden_dim:
            self.layers.append(nn.Linear(current_dim, hdim))
            current_dim = hdim
        self.layers.append(nn.Linear(current_dim, output_dim))

    def forward(self, x):
        for layer in self.layers[:-1]:
            x = F.relu(layer(x))
        out = F.softmax(self.layers[-1](x))
        return out

It is very important to use pytorch Containers for the layers, and not just a simple python lists. Please see this answer to know why.


Instruction: Choosing loss function for lstm trained on word2vec vectors when target is a vector of same dimensions

Input:
I have an lstm I'm using as a sequence-generator trained on word2vec vectors. The previous implementation produced a probability distribution for all the different labels. There was one label for every word in the vocabulary. This implementation used Pytorch's CrossEntropyLoss. I now want to change this so the lstm outputs a vector that has the same dimensions as the vectors used for training. This way I could use the euclydian distance measure to match wit to nearby vectors in the vocabulary. The problem is that in order to do this, I have to use a different loss function, because CrossEntropyLoss is appropriate for classifiers and not for regression problems.

I tried changing the format of the target vector, but torch's CrossEntropyLoss function requires integer input, and I have a word vector. Having looked at a few options it seems Cosine Embedding Loss might be a good idea but I don't understand how it works and what kind of input it takes.

I have already changed my fully connected layer to output vectors of the same dimensions as the Word Embeddings used for training:

nn.Linear(in_features=self.cfg.lstm.lstm_num_hidden,out_features=self.cfg.lstm.embedding_dim,bias=True)

Any advice and examples would be much appreciated.

Output:
As the documentation of CosineEmbeddingLoss says:

  Creates a criterion that measures the loss given two input tensors and a Tensor label with values 1 or -1.

In your scenario, you should always provide 1 as the Tensor label.

batch_size, seq_len, w2v_dim = 32, 100, 200
x1 = torch.randn(batch_size, seq_len, w2v_dim)
x2 = torch.randn(batch_size, seq_len, w2v_dim)
y = torch.ones(batch_size, seq_len)
loss_fn = torch.nn.CosineEmbeddingLoss(reduction='none')
loss = loss_fn(x1.view(-1, w2v_dim),
               x2.view(-1, w2v_dim),
               y.view(-1))
loss = loss.view(batch_size, seq_len)

Here, I assume x1 is the word embeddings, x2 is the output of the LSTM followed by some transformation.

  Why should I always provide 1 as the Tensor label?

First, you should see the loss function.

In your scenario, the higher the cosine similarity is, the lower the loss should be. In other words, you want to maximize the cosine similarity. So, you need to provide 1 as the label.

On the other hand, if you want to minimize the cosine similarity, you need to provide -1 as the label.


Instruction: How to use DataLoader for PyTorch on iPython Console of Spyder

Input:
I check this Tutorial and can't figure out a way to actually use my DataLoader to train a ANN.
When iterating over my DataLoader a cmd prompt pops up and immediately closes itself, afterwards nothing happens. My original data are both np.arrays.

import torch
from torch.utils import data
import numpy as np

class Dataset(data.Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, datax, labels):
        'Initialization'
        self.labels = torch.tensor(labels)
        self.datax = torch.tensor(datax)
        self.len = len(datax)

  def __len__(self):
        'Denotes the total number of samples'
        return self.len

  def __getitem__(self, index):
        'Generates one sample of data'
        # Load data and get label
        X = self.datax[index]
        y = self.labels[index]
        return X, y

params = {'batch_size': 64,
          'shuffle': True,
          'num_workers': 1}
training_set = Dataset(datax=X, labels=labels)
training_generator = data.DataLoader(training_set, **params)

for x in training_generator:
    print(1)

I tried many times and had a glimpse at the commandprompt which says something like

OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0
OMP: Info #156: KMP_AFFINITY: 4 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 2 cores/pkg x 2 threads/core (2 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 0 thread 1
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 1 thread 0
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 1 thread 1
OMP: Info #250: KMP_AFFINITY: pid 10264 tid 2388 thread 0 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 10264 tid 3288 thread 1 bound to OS proc set 2

Output:
Here is how I do that:

class myDataset(Dataset):
    '''
    a dataset for PyTorch
    '''
    def __init__(self, X, y):
        self.X = X
        self.y = y
    def __getitem__(self, index):
        return self.X[index], self.y[index]
    def __len__(self):
        return len(self.X)

then you can simple add to the loader:

full_dataset = myDataset(X,y)
train_loader = DataLoader(full_dataset, batch_size=batch_size)

Also, X, y are just numpy arrays.

And for the training you can access your data with a for loop:

for data, target in train_loader:
        if train_on_gpu:
            data, target = data.double().cuda(), target.double().cuda()


Instruction: How to use the lbfgs optimizer with pytorch-lightning?

Input:
I have a problem in using the LBFGS optimizer from pytorch with lightning.
I use the template from here to start a new project and here is the code that I tried (only the training portion):

def training_step(self, batch, batch_nb):
    x, y = batch
    x = x.float()
    y = y.float()
    y_hat = self.forward(x)
    return {'loss': F.mse_loss(y_hat, y)}

def configure_optimizers(self):
    optimizer = torch.optim.LBFGS(self.parameters())
    return optimizer

def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i):
    def closure():
        optimizer.zero_grad()
        l = self.training_step(batch, batch_nb)
        loss = l['loss']
        loss.backward()
        return loss

    optimizer.step(closure)

The LBFGS optimizer from pytorch requires a closure function (see here and here), but I don't know how to define it inside the template, specially I don't know how the batch data is passed to the optimizer. I tried to define a custom optimizer_step function but I have some problems to passing the batch inside the closure function.

I will be very thankful of any advise that helps me to solve this problem or points me to the right direction.

Environment:

PyTorch version:  1.2.0+cpu
Lightning version: 0.4.9
Test-tube version: 0.7.1

Output: The support for lbfgs optimizer was added on #310, now it is not required to define a closure function.


Instruction: Convert image of dimension height,width,number of channels to n_masks, image_height, image_width

Input:
I have an RGB image, it contains masks of different colors, each color represents a particular class.
I want to convert it into the format - n_masks, image_height, image_width where n_masks is number of masks present in the image. and each slice of the matrix along the 0th axis represents one binary mask.
So far, I have been able to convert it into the format of image_height, image_width where each array value represents which class it belongs to but I am kind of struck after it.
Below is my code to convert it into image_height,image_width format-

def mask_to_class(mask):
    target = torch.from_numpy(mask)
    h,w = target.shape[0],target.shape[1]
    masks = torch.empty(h, w, dtype=torch.long)
    colors = torch.unique(target.view(-1,target.size(2)),dim=0).numpy()
    target = target.permute(2, 0, 1).contiguous()
    mapping = {tuple(c): t for c, t in zip(colors.tolist(), range(len(colors)))}
    for k in mapping:
        idx = (target==torch.tensor(k, dtype=torch.uint8).unsqueeze(1).unsqueeze(2))
        validx = (idx.sum(0) == 3)
        masks[validx] = torch.tensor(mapping[k], dtype=torch.long)
    return masks

It converts an image of let's say format (512,512,3) to (512,512) where each pixel values represents the class it belongs to, but I don't have any idea how to proceed further.
P.S- I am coding it in pytorch but any approach involving numpy is also welcomed.

Output:
Suppose you already have (512,512) mask. You can first use mask.unique() to get all classes pixel values. Then for each class value, torch.where(mask==cls_val, torch.tensor(1), torch.tensor(0)) will return the mask of one certain class. In the end, you stack all outputs.

mask = torch.tensor([[1,2,3],
                     [2,4,5],
                     [1,2,3]])
cls = mask.unique()
res = torch.stack([torch.where(mask==cls_val, torch.tensor(1), torch.tensor(0)) for cls_val in cls])

#tensor([[[1, 0, 0],
#         [0, 0, 0],
#         [1, 0, 0]],
#
#        [[0, 1, 0],
#         [1, 0, 0],
#         [0, 1, 0]],
#
#        [[0, 0, 1],
#         [0, 0, 0],
#         [0, 0, 1]],
#
#        [[0, 0, 0],
#         [0, 1, 0],
#         [0, 0, 0]],
#
#        [[0, 0, 0],
#         [0, 0, 1],
#         [0, 0, 0]]])


Instruction: Suppress use of Softmax in CrossEntropyLoss for PyTorch Neural Net

Input:
I know theres no need to use a nn.Softmax() Function in the output layer for a neural net when using nn.CrossEntropyLoss as a loss function.

However I need to do so, is there a way to suppress the implemented use of softmax in nn.CrossEntropyLoss and instead use nn.Softmax() on my output layer of the neural network itself?

Motivation: I am using shap package to analyze the features influences afterwards, where I can only feed my trained model as an input. The outputs however don't make any sense then because I am looking at unbound values instead of probabilites.

Example: Instead of -69.36 as an output value for one class of my model, I want something between 0 and 1, summing up to 1 for all classes. As I can't alter it afterwards, the outputs need to be like this already during training.

Output:
The documentation of nn.CrossEntropyLoss says,

  This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.

I suggest you stick to the use of CrossEntropyLoss as the loss criterion. However, you can convert the output of your model into probability values by using the softmax function.

Please note, you can always play with the output values of your model, you do not need to change the loss criterion for that.

But if you still want to use Softmax() in your network, then you can use the NLLLoss() as the loss criterion, only apply log() before feeding model's output to the criterion function. Similarly, if you use LogSoftmax instead in your network, you can apply exp() to get the probability values.

Update:

To use log() on the Softmax output, please do:

torch.log(prob_scores + 1e-20)

By adding a very small number (1e-20) to the prob_scores, we can avoid the log(0) issue.


Instruction: Remove certain elements of all the tensors in a list of dictionary of tensors in pytorch

Input:
I am new to pytorch, and trying to learn by playing with simple codes. What I have is a list of length zero, populated by a dictionary, where values in this dictionary are tensors. This is what this list looks like:

A = [{'boxes': tensor([[ 142.1232,  142.9373, 1106.0452,  971.3792],
    [ 259.1277,  618.4834, 1100.1293, 1028.8989],
    [ 232.1346,  692.5888,  763.3408, 1028.6766],
    [ 206.8070,  312.2080, 1137.1434, 1013.4373],
    [ 495.9471,  675.7287,  978.5932, 1012.7568]], grad_fn=<StackBackward>), 'labels': tensor([16,  1,  1,  1,  1]), 'scores': tensor([0.9988, 0.9489, 0.5228, 0.3500, 0.0639], grad_fn=<IndexBackward>), 'masks': tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]]], grad_fn=<UnsqueezeBackward0>)}]

This list is the output of Mask-RCNN and I want to remove certain element of all the tensors in the nested dictionary. In this case, I only want to keep information related with class label "1". Information related to each class ('boxes', 'labels', 'scores', and 'masks') are all in the same location(s) (index) in each of the tensors. So, I find the index of all the "1"s in the tensor with key "labels":

idxOfClass = [i for i, x in enumerate(list(pred[0]['labels'])) if x == 1]

which gives me: [1, 2, 3, 4].
Then, I want to keep all the values located at indices in idxOfClass in all of the tensors in the nested dictionary. If I do something like this:

Anew = [{pred[0]['boxes'][idxOfClass],pred[0]['labels'][idxOfClass],pred[0]['masks'][idxOfClass],pred[0]['scores'][idxOfClass]}]

I get:

[{tensor([[ 259.1277,  618.4834, 1100.1293, 1028.8989],
    [ 232.1346,  692.5888,  763.3408, 1028.6766],
    [ 206.8070,  312.2080, 1137.1434, 1013.4373],
    [ 495.9471,  675.7287,  978.5932, 1012.7568]], grad_fn=
    <IndexBackward>), tensor([0.9489, 0.5228, 0.3500, 0.0639],
    grad_fn=
    <IndexBackward>), tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]]], grad_fn=<IndexBackward>),
    tensor([1, 1, 1, 1])}]

But, this is not a list of dictionary with tensors as the values of the dictionary. This is a list of tensors, with no key-value structure of the nested dictionary. My question is, "is there any way to keep the original structure of the list when I remove certain elements of all the tensors using indices of specific elements?".

Output:
You can just add key names when you construct the new pred res.

Anew = [{'boxes': pred[0]['boxes'][idxOfClass],'labels': pred[0]['labels'][idxOfClass],'masks': pred[0]['masks'][idxOfClass],'scores': pred[0]['scores'][idxOfClass]}]


Instruction: libcudart.so.9.0: cannot open shared object file: No such file or directory

Input:
I'm using Pytorch  under Ubuntu 18.04 and trying to import torchvision, but I get an error libcudart.so.9.0: cannot open shared object file: No such file or directory.

Someone could help to fix it? Thanks.

The infos below are detailed error logs:

Traceback (most recent call last):
  File "/home/x/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-6dd351122000>", line 1, in <module>
    import torchvision
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
import torch
import torch.nn as nn
import torchvision.transforms as transforms
Traceback (most recent call last):
  File "/home/x/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2882, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-677acbcfae34>", line 1, in <module>
    import torchvision.transforms as transforms
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "/home/x/.local/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
  File "/home/x/pycharm-2019.2/helpers/pydev/_pydev_bundle/pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

Output:
If you are using anaconda, the following may fix your problem.

conda install -c anaconda cudatoolkit==9.0

You can also try the followings.

Make sure the CUDA version is 9.0. And add the following 2 lines to ~/.bashrc.

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Then, run:

source  ~/.bashrc

Add the following lines to /etc/ld.so.conf.d/cuda.conf

/usr/local/cuda/lib64

And run:

sudo ldconfig


Instruction: RuntimeError: Given groups=1, weight of size 16 1 5 5, expected input[100, 3, 256, 256] to have 1 channels, but got 3 channels instead

Input:
I try to run the following programe for images classification problem in Pytorch:

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Hyper parameters
num_epochs = 5
num_classes = 10
batch_size = 100
learning_rate = 0.001

TRAIN_DATA_PATH = "train/"
TEST_DATA_PATH = "test/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_dataset = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_loader = data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True,  num_workers=4)
test_dataset = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_loader  = data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True, num_workers=4)

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7 * 7 * 32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

model = ConvNet(num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))

# Test the model
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model/model.ckpt')

But I get a RuntimeError:

Traceback (most recent call last):

RuntimeError: Given groups=1, weight of size 16 1 5 5, expected input[100, 3, 256, 256] to have 1 channels, but got 3 channels instead

Someone could help to fix the bug? Thanks a lot.

Reference related:

https://discuss.pytorch.org/t/given-groups-1-weight-16-1-5-5-so-expected-input-100-3-64-64-to-have-1-channels-but-got-3-channels-instead/28831/17

RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[3, 1, 224, 224] to have 3 channels, but got 1 channels instead

Output:
Your input layer self.layer1 starts with a 2d convolution nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2). This conv layer expects an input with two spatial dimensions and one channel, and outputs a tesnor with the same spatial dimensions and 16 channels.
However, your input has three channels and not one (RGB image instead of gray level image).

Make sure your net and data are in synch.


Instruction: How the function nn.LSTM behaves within the batches/ seq_len?

Input:
I’m currently learning to use nn.LSTM with pytorch and had to ask how the function is working.

Basically I’m trying to feed my dataset matrix (M x N).
Since the dataset is a matrix, I wanted to feed the dataset recursively(as timesteps) into the LSTM network with Dataloader(utils.data.Dataset).

The point where i got confused was the size of input(seq_len, batch, input_size)

Let’s say I’m getting my data_loader with batch_size=10.
In order to generate the train_loader with the right form, I had to make the previous size of (M x N) into the size including the sequence_length which could simply be transformed to (M/seq_len, seq_len, N).

Then the input size of my nn.LSTM would be like:
(M/seq_len/batch_size, seq_len, N)

So, my main question comes:

If i feed this data size into the LSTM model nn.LSTM(N, hidden_size),
is the LSTM model already doing the recursive feed-forward within the whole batch?
I'm also confused with the seq_len, while seq_len>1, the output will get the dimension of seq_len.
Would that mean the output contains the recursive operations of sequences?

I’m not sure i made the questions clear, but my understanding is getting quite messed up..lol
Hope somebody could help me organizing the right understanding.

Output:
Yes, provided each sample's sequence length is the same (which seems to be the case here). If not, you have to pad with torch.nn.utils.rnn.pad_sequence for example.
Yes, LSTM is expanded to each timestep and there is output for each timestep already. Hence you don't have to apply it for each element separately.


Instruction: For a given condition, get indices of values in 2D tensor A, use those to index a 3D tensor B

Input:
For a given 2D tensor I want to retrieve all indices where the value is 1. I expected to be able to simply use torch.nonzero(a == 1).squeeze(), which would return tensor([1, 3, 2]). However, instead, torch.nonzero(a == 1) returns a 2D tensor (that's okay), with two values per row (that's not what I expected). The returned indices should then be used to index the second dimension (index 1) of a 3D tensor, again returning a 2D tensor.

import torch

a = torch.Tensor([[12, 1, 0, 0],
                  [4, 9, 21, 1],
                  [10, 2, 1, 0]])

b = torch.rand(3, 4, 8)

print('a_size', a.size())
# a_size torch.Size([3, 4])
print('b_size', b.size())
# b_size torch.Size([3, 4, 8])

idxs = torch.nonzero(a == 1)
print('idxs_size', idxs.size())
# idxs_size torch.Size([3, 2])

print(b.gather(1, idxs))

Evidently, this does not work, leading to aRunTimeError:

  RuntimeError: invalid argument 4: Index tensor must have same
  dimensions as input tensor at
  C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorEvenMoreMath.cpp:453

It seems that idxs is not what I expect it to be, nor can I use it the way I thought. idxs is

tensor([[0, 1],
        [1, 3],
        [2, 2]])

but reading through the documentation I don't understand why I also get back the row indices in the resulting tensor. Now, I know I can get the correct idxs by slicing idxs[:, 1] but then still, I cannot use those values as indices for the 3D tensor because the same error as before is raised. Is it possible to use the 1D tensor of indices to select items across a given dimension?

Output:
You could simply slice them and pass it as the indices as in:
In [193]: idxs = torch.nonzero(a == 1)
In [194]: c = b[idxs[:, 0], idxs[:, 1]]

In [195]: c
Out[195]:
tensor([[0.3411, 0.3944, 0.8108, 0.3986, 0.3917, 0.1176, 0.6252, 0.4885],
        [0.5698, 0.3140, 0.6525, 0.7724, 0.3751, 0.3376, 0.5425, 0.1062],
        [0.7780, 0.4572, 0.5645, 0.5759, 0.5957, 0.2750, 0.6429, 0.1029]])

Alternatively, an even simpler & my preferred approach would be to just use torch.where() and then directly index into the tensor b as in:
In [196]: b[torch.where(a == 1)]
Out[196]:
tensor([[0.3411, 0.3944, 0.8108, 0.3986, 0.3917, 0.1176, 0.6252, 0.4885],
        [0.5698, 0.3140, 0.6525, 0.7724, 0.3751, 0.3376, 0.5425, 0.1062],
        [0.7780, 0.4572, 0.5645, 0.5759, 0.5957, 0.2750, 0.6429, 0.1029]])

A bit more explanation about the above approach of using torch.where(): It works based on the concept of advanced indexing. That is, when we index into the tensor using a tuple of sequence objects such as tuple of tensors, tuple of lists, tuple of tuples etc.
# some input tensor
In [207]: a
Out[207]:
tensor([[12.,  1.,  0.,  0.],
        [ 4.,  9., 21.,  1.],
        [10.,  2.,  1.,  0.]])

For basic slicing, we would need a tuple of integer indices:
   In [212]: a[(1, 2)]
   Out[212]: tensor(21.)

To achieve the same using advanced indexing, we would need a tuple of sequence objects:
# adv. indexing using a tuple of lists
In [213]: a[([1,], [2,])]
Out[213]: tensor([21.])

# adv. indexing using a tuple of tuples
In [215]: a[((1,), (2,))]
Out[215]: tensor([21.])

# adv. indexing using a tuple of tensors
In [214]: a[(torch.tensor([1,]), torch.tensor([2,]))]
Out[214]: tensor([21.])

And the dimension of the returned tensor would always be one dimension less than the dimension of the input tensor.


Instruction: Freezing Individual Weights in Pytorch

Input:
The following question is not a duplicate of How to apply layer-wise learning rate in Pytorch? because this question aims at freezing a subset of a tensor from training rather than the entire layer.

I am trying out a PyTorch implementation of Lottery Ticket Hypothesis.

For that, I want to freeze the weights in a model that are zero. Is the following a correct way to implement it?

for name, p in model.named_parameters():
            if 'weight' in name:
                tensor = p.data.cpu().numpy()
                grad_tensor = p.grad.data.cpu().numpy()
                grad_tensor = np.where(tensor == 0, 0, grad_tensor)
                p.grad.data = torch.from_numpy(grad_tensor).to(device)

Output:
What you have seems like it would work provided you did it after loss.backward() and before optimizer.step() (referring to the common usage for these variable names). That said, it seems a bit convoluted. Also, if your weights are floating point values then comparing them to exactly zero is probably a bad idea, we could introduce an epsilon to account for this. IMO the following is a little cleaner than the solution you proposed:

# locate zero-value weights before training loop
EPS = 1e-6
locked_masks = {n: torch.abs(w) < EPS for n, w in model.named_parameters() if n.endswith('weight')}

...

for ... #training loop

    ...

    optimizer.zero_grad()
    loss.backward()
    # zero the gradients of interest
    for n, w in model.named_parameters():
        if w.grad is not None and n in locked_masks:
            w.grad[locked_masks[n]] = 0
    optimizer.step()


Instruction: Why pytorch needs much more memory than it should?

Input:
I'm just playing around with pytorch and I'm wondering why it consumes so much memory of my GPU?

I'm using Cuda 10.0 with pythorch 1.2.0 and torchvision 0.4.0.

import torch
gpu = torch.device("cuda")
x = torch.ones(int(4e8), device=gpu)
y = torch.ones(int(1e5), device=gpu)

Running the above code I get the error:
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 2.00 GiB total capacity; 1.49 GiB already allocated; 0 bytes free; 0 bytes cached)

So, does pytorch needs ~500MB of the gpu memory as overhead? Or what is the problem here?

Output:
More information and testing done by xymeng in github could be seen in the given link

Referencing xymeng's words :

  PyTorch has its own cuda kernels. From my measurement the cuda runtime allocates ~1GB memory for them. If you compile pytorch with cudnn enabled the total memory usage is 1GB + 750M + others = 2GB+
  Note that this is just my speculation as there is no official documentation about this. What puzzles me is that the cuda runtime allocates much more memory than the actual code size (they are approx. linearly correlated. If I remove half of pytorch's kernels the memory usage is also reduced by half). I suspect either the kernel binaries have been compressed or they have to be post-processed by the runtime.

Seems it suits your situation.


Instruction: Why Pytorch officially use mean=[0.485, 0.456, 0.406] and std=[0.229, 0.224, 0.225] to normalize images?

Input:
In this page (https://pytorch.org/vision/stable/models.html), it says that "All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape (3 x H x W), where H and W are expected to be at least 224. The images have to be loaded in to a range of [0, 1] and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]".
Shouldn't the usual mean and std of normalization be [0.5, 0.5, 0.5] and [0.5, 0.5, 0.5]? Why is it setting such strange values?

Output: Using the mean and std of Imagenet is a common practice. They are calculated based on millions of images. If you want to train from scratch on your own dataset, you can calculate the new mean and std. Otherwise, using the Imagenet pretrianed model with its own mean and std is recommended.


Instruction: Should I transpose a Tensor when feeding it into a CNN

Input: I am using a custom dataset with images of different sizes in the Lab format (Lightness, a, b) which are feed into a CNN. The input layer has 3 in-channels and so my idea was to split all 3 channels (L, a, b) and feed those into the network. Next I was wondering if each tensor needs to be transposed? My doubt is that it would lose its dimensions which are variable from image to image and I would not be able to reconstruct the image in the end. Any thoughts or ideas how I should normalize the image?

Output:
You can normalise without the need of transposing the image or splitting it based on its channels

torchvision.transforms.Normalize(mean=[l_channel_mean, a_channel_mean , b_channel_mean], std= [l_channel_mean, a_channel_mean , b_channel_mean])

The only required transform is the one that converts the images to tensors :

torchvision.transforms.ToTensor()


Instruction: RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'mat2' how to fix it?

Input:
import torch.nn as nn
import torch
import torch.optim as optim
import itertools

class net1(nn.Module):
    def __init__(self):
        super(net1,self).__init__()

        self.pipe = nn.Sequential(
            nn.Linear(10,10),
            nn.ReLU()
        )

    def forward(self,x):
        return self.pipe(x.long())

class net2(nn.Module):
    def __init__(self):
        super(net2,self).__init__()

        self.pipe = nn.Sequential(
            nn.Linear(10,20),
            nn.ReLU(),
            nn.Linear(20,10)
        )

    def forward(self,x):
        return self.pipe(x.long())

netFIRST = net1()
netSECOND = net2()

learning_rate = 0.001

opt = optim.Adam(itertools.chain(netFIRST.parameters(),netSECOND.parameters()), lr=learning_rate)

epochs = 1000

x = torch.tensor([1,2,3,4,5,6,7,8,9,10],dtype=torch.long)
y = torch.tensor([10,9,8,7,6,5,4,3,2,1],dtype=torch.long)

for epoch in range(epochs):
    opt.zero_grad()

    prediction = netSECOND(netFIRST(x))
    loss = (y.long() - prediction)**2
    loss.backward()

    print(loss)
    print(prediction)
    opt.step()

error:

  line 49, in  prediction = netSECOND(netFIRST(x))

  line 1371, in linear; output = input.matmul(weight.t())

  RuntimeError: Expected object of scalar type Long but got scalar type
  Float for argument #2 'mat2'

I don't really see what I'm doing wrong. I have tried to turn everything in a Long in every possible way. I don't really get the way typing works for pytorch. Last time I tried something with just one layer and it forced me to use type int.
Could someone explain how the typing is established in pytorch and how to prevent and fix errors like this??
A lot I mean an awful lot of thanks in advance, this problem really bothers me and I can't seem to fix it no matter what I try.

Output:
The weights are Floats, the inputs are Longs. This is not allowed. In fact, I don't think torch supports anything else than Floats in neural networks.

If you remove all calls to long, and define your input as floats, it will work (it does, I tried).

(You will then get another unrelated error: you need to sum your loss)


Instruction: Deep Learning methods for Text Generation (PyTorch)

Input:
Greetings to everyone,

I want to design a system that is able to generate stories or poetry based on a large dataset of text, without being needed to feed a text description/start/summary as input at inference time.

So far I did this using RNN's, but as you know they have a lot of flaws. My question is, what are the best methods to achieve this task at the time?
I searched for possibilities using Attention mechanisms, but it turns out that they are fitted for translation tasks.

I know about GPT-2, Bert, Transformer, etc., but all of them need a text description as input, before the generation and this is not what I'm seeking. I want a system able to generate stories from scratch after training.

Thanks a lot!

Output:
edit

so the comment was: I want to generate text from scratch, not starting from a given sentence at inference time. I hope it makes sense.

yes, you can do that, that's just simple code manipulation on top of the ready models, be it BERT, GPT-2 or LSTM based RNN.

How? You have to provide random input to the model. Such random input can be randomly chosen word or phrase or just a vector of zeroes.

Hope it helps.

You have mixed up several things here.

You can achieve what you want either using LSTM based or transformer based architecture.

When you said you did it with RNN, you probably mean that you have tried LSTM based sequence to sequence model.

Now, there is attention in your question. So you can use attention to improve your RNN but it is not a required condition. However, if you use transformer architecture, then it is built in the transormer blocks.

GPT-2 is nothing but a transformer based model. Its building block is a transformer architecture.

BERT is also another transformer based architecture.

So to answer your question, you should and can try using LSTM based or transformer based architecture to achieve what you want. Sometimes such architecture is called GPT-2, sometimes BERT depending on how it is realized.

I encourage you to read this classic from Karpathy, if you understand it then you have cleared most of your questions:

http://karpathy.github.io/2015/05/21/rnn-effectiveness/