Instruction: Efficiently find centroid of labelled image regions

Input:
I have a segmented image as a 2 dimensional matrix of unique labels 1 ... k. For example:

img =
    [1 1 2 2 2 2 2 3 3]
    [1 1 1 2 2 2 2 3 3]
    [1 1 2 2 2 2 3 3 3]
    [1 4 4 4 2 2 2 2 3]
    [4 4 4 5 5 5 2 3 3]
    [4 4 4 5 5 6 6 6 6]
    [4 4 5 5 5 5 6 6 6]

I am trying to determine the region centroids. That is, per label, what is the X,Y coordinate of the center of mass? For example, the centroid of label 1 is (1.25, 0.625). Just average up the row numbers ((0 + 0 + 1 + 1 + 1 + 2 + 2 + 3) / 8 = 1.25) and the column numbers ((0 + 0 + 0 + 0 + 1 + 1 + 1 + 2) / 8 = 0.625)

The only way I know how to do this is to use a for loop from 1 to k (or in the case of my example, 1 through 6), find the indices of the points for each label, and average their coordinates by indexing a meshgrid of the image.

However, I am looking to do this in a way optimized for GPU computations. Hence, the use of a for loop is less than ideal (takes about 1 sec per image on a nice GPU for a few hundred labels). I am using PyTorch, but really any numpy solution should suffice.

Is there a GPU-efficient solution for this task?

Output:
One idea would be to use bincount to accumulate the row and column indices for each region using the numbers in the input array as the bins and thus have a vectorized solution, like so -

m,n = a.shape
r,c = np.mgrid[:m,:n]
count = np.bincount(a.ravel())
centroid_row = np.bincount(a.ravel(),r.ravel())/count
centroid_col = np.bincount(a.ravel(),c.ravel())/count

Sample run -

In [77]: a
Out[77]:
array([[1, 1, 2, 2, 2, 2, 2, 3, 3],
       [1, 1, 1, 2, 2, 2, 2, 3, 3],
       [1, 1, 2, 2, 2, 2, 3, 3, 3],
       [1, 4, 4, 4, 2, 2, 2, 2, 3],
       [4, 4, 4, 5, 5, 5, 2, 3, 3],
       [4, 4, 4, 5, 5, 6, 6, 6, 6],
       [4, 4, 5, 5, 5, 5, 6, 6, 6]])

In [78]: np.c_[centroid_row, centroid_col]
Out[78]:
array([[  nan,   nan],
       [ 1.25,  0.62], # centroid for region-1
       [ 1.56,  4.44], # centroid for region-2
       [ 1.9 ,  7.4 ], # centroid for region-3 and so on.
       [ 4.36,  1.18],
       [ 5.11,  3.67],
       [ 5.43,  6.71]])


Instruction: Why grad_output requires_grad is False in pytorch?

Input:
Here is the customLayer.py.

I am quite confused about the following things:

The input of the inner layer is not a Variable. Then in backward it becomes a Variable and requires gradient. Why?
grad_output is a Variable yet requires_grad is False. Why is not true?
In my custom layer, I need customize forward and backward operations. It is quite complicated. See the same link. I have posted questions in it.

Output:
The gradients are updated through your loss computation and are required for the backpropagation. If you don't have gradients, you cant train your network.
Probably, because you don't want the gradients last on the variable. It's temporally only for one backward phase.
Why do you need a custom backward function? Do you need extra operations on your backpropagation?


Instruction: How to do reinforcement learning with an LSTM in PyTorch?

Input:
Due to observations not revealing the entire state, I need to do reinforcement with a recurrent neural network so that the network has some sort of memory of what has happened in the past. For simplicity let's assume that we use an LSTM.

Now the in-built PyTorch LSTM requires you to feed it a an input of shape Time x MiniBatch x Input D and it outputs a tensor of shape Time x MiniBatch x Output D.

In reinforcement learning however, to know the input at time t+1, I need to know the output at time t, because I am doing actions in an environment.

So is it possible to use the in-built PyTorch LSTM to do BPTT in a reinforcement learning setting? And if it is, how could I do it?

Output:
Maybe you can feed your input sequence in a loop to your LSTM. Something, like this:

h, c = Variable(torch.zeros()), Variable(torch.zeros())
for i in range(T):
    input = Variable(...)
    _, (h, c) = lstm(input, (h,c))

Every timestep you can use (h,c) and input to evaluate action for instance. As long as you do not break computational graph you can backpropagate as Variables keep all the history.


Instruction: Pytorch RNN memory allocation error in DataLoader

Input:
I am writing an RNN in Pytorch.
I have the following line of code:

data_loader = torch.utils.data.DataLoader(
    data,
    batch_size=args.batch_size,
    shuffle=True,
    num_workers=args.num_workers,
    drop_last=True)

If I set num_workers to 0, I get a segmentation fault.
If I set num_workers to > 0, then I have the traceback:

Traceback (most recent call last):
File "rnn_model.py", line 352, in <module>
train_model(train_data, dev_data, test_data, model, args)
File "rnn_model.py", line 212, in train_model
loss = run_epoch(train_data, True, model, optimizer, args)
File "rnn_model.py", line 301, in run_epoch
for batch in tqdm.tqdm(data_loader):
File "/home/username/miniconda3/lib/python2.7/site-packages/tqdm/_tqdm.py",
line 872, in __iter__
for obj in iterable:
File "/home/username/miniconda3/lib/python2.7/site-
packages/torch/utils/data/dataloader.py", line 303, in __iter__
return DataLoaderIter(self)
File "/home/username/miniconda3/lib/python2.7/site-
packages/torch/utils/data/dataloader.py", line 162, in __init__
w.start()
File "/home/username/miniconda3/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/home/username/miniconda3/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Output:
You are trying to load more data than your system can hold in its RAM.
You can either try to load only parts of your data or use/write a data loader which only loads the data that is needed for the current batch.


Instruction: Teacher forcing with pytorch RNN

Input:
The pytorch tutorials do a great job of illustrating a bare-bones RNN by defining the input and hidden layers, and manually feeding the hidden layers back into the network to remember the state.  This flexibility then allows you to very easily perform teacher forcing.

Question 1: How do you perform teacher forcing when using the native nn.RNN() module (since the entire sequence is fed at once)?  Example simple RNN network would be:

class SimpleRNN(nn.Module):

    def __init__(self, vocab_size,
                 embedding_dim,
                 batch_sz,
                 hidden_size=128,
                 nlayers=1,
                 num_directions=1,
                 dropout=0.1):

        super(SimpleRNN, self).__init__()

        self.batch_sz = batch_sz
        self.hidden_size = hidden_size

        self.encoder = nn.Embedding(vocab_size, embedding_dim)
        self.rnn = nn.RNN(embedding_dim, hidden_size, nlayers, dropout=0.5)
        self.decoder = nn.Linear(hidden_size, vocab_size)

    def init_hidden(self):
        return autograd.Variable(torch.zeros(nlayers, batch_sz, hidden_size)).cuda()

    def forward(self, inputs, hidden):

        # -- encoder returns:
        # -- [batch_sz, seq_len, embed_dim]
        encoded = self.encoder(inputs)
        _, seq_len, _ = encoded.size()

        # -- rnn returns:
        # -- output.size() = [seq_len, batch_sz, hidden_sz]
        # -- hidden.size() = [nlayers, batch_sz, hidden_sz]
        output, hidden = self.rnn(encoded.view(seq_len, batch_sz, embedding_dim), hidden)

        # -- decoder returns:
        # -- output.size() = [batch_sz, seq_len, vocab_size]
        output = F.log_softmax(decoder(output.view(batch_sz, seq_len, self.hidden_size)))

        return output, hidden

Where I can call the network with:

model = SimpleRNN(vocab_size, embedding_dim, batch_sz).cuda()
x_data, y_data = get_sequence_data(train_batches[0])
output, hidden = model(x_data, model.init_hidden())

Just for completeness, here are my shapes of x_data,  output, and hidden:

print(x_data.size(), output.size(), hidden.size())
torch.Size([32, 80]) torch.Size([32, 80, 4773]) torch.Size([1, 32, 128])

Question 2: would it be possible to use this SimpleRNN network to then generate a sequence word-by-word, by first feeding it a <GO_TOKEN> and iterating until an <END_TOKEN> is reached?  I ask because when I run this:

x_data = autograd.Variable(torch.LongTensor([[word2idx['<GO>']]]), volatile=True).cuda()
output, hidden = model(x_data, model.init_hidden(1))

print(output, output.sum())

I get an output of all 0s, and the output.sum() = 0.  I get this even after training the network and backpropagating the loss.  Any ideas why?

Question 3: If not terribly inefficient, is it possible to train the SimpleRNN network above word-by-word, analogous to the pytorch tutorial shown (here)[http://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html] (albeit there they're training character-by-character).

Output:
Question 1.

Teacher forcing is performed implicitly in this case, since your x_data is [seq_len, batch_size] it will feed in each item in seq_len as input and not use the actual output for the next input.

Question 2.

Your model.init_hidden does not take any input however it looks like you're trying to add the batch size in, maybe you could check that, everything else seems fine. Though you will need to do an max() or multinomial() on the output  before you can feed it back through.

Question 3.

Yes you can do this, yes it is terribly inefficient. This is a limitation of the CUDNN LSTM kernel


Instruction: Running out of memory during evaluation in Pytorch

Input:
I'm training a model in pytorch. Every 10 epochs, I'm evaluating the train and test error on the entire train and test dataset. For some reason the evaluation function is causing out-of-memory on my GPU. This is strange because I have the same batchsize for training and evaluation. I believe it's due to the net.forward() method being called repeated and having all the hidden values stored in memory but I'm not sure how to get around this?

def evaluate(self, data):
    correct = 0
    total = 0
    loader = self.train_loader if data == "train" else self.test_loader
    for step, (story, question, answer) in enumerate(loader):
        story = Variable(story)
        question = Variable(question)
        answer = Variable(answer)
        _, answer = torch.max(answer, 1)

        if self.config.cuda:
            story = story.cuda()
            question = question.cuda()
            answer = answer.cuda()

        pred_prob = self.mem_n2n(story, question)[0]
        _, output_max_index = torch.max(pred_prob, 1)
        toadd = (answer == output_max_index).float().sum().data[0]
        correct = correct + toadd
        total = total + captions.size(0)

    acc = correct / total
    return acc

Output:
I think it fails during Validation because you don't use optimizer.zero_grad(). The zero_grad executes detach, making the tensor a leaf. It is commonly used every epoch in the training part.

The use of volatile flag in Variable from PyTorch 0.4.0 has been removed.
Ref - migration_guide_to_0.4.0

Starting from 0.4.0, to avoid the gradient being computed during validation, use torch.no_grad()

Code example from the migration guide.

# evaluate
with torch.no_grad():                   # operations inside don't track history
  for input, target in test_loader:
      ...

For 0.3.X, using volatile should work.


Instruction: How to train a simple linear regression model with SGD in pytorch successfully?

Input:
I was trying to train a simple polynomial linear regression model in pytorch with SGD. I wrote some self contained (what I thought would be extremely simple code), however, for some reason my model does not train as I thought it should.

I have 5 points sampled from a sine curve and try to fit it with a polynomial of degree 4. This is a convex problem so GD or SGD should find a solution with zero train error eventually as long as we have enough iterations and small enough step size. For some reason however my model does not train well (even though it seems that it is changing the parameters of the model. Anyone have an idea why? Here is the code (I tried making it self contained and minimal):

import numpy as np
from sklearn.preprocessing import PolynomialFeatures

import torch
from torch.autograd import Variable

from maps import NamedDict

from plotting_utils import *

def index_batch(X,batch_indices,dtype):
    '''
    returns the batch indexed/sliced batch
    '''
    if len(X.shape) == 1: # i.e. dimension (M,) just a vector
        batch_xs = torch.FloatTensor(X[batch_indices]).type(dtype)
    else:
        batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    return batch_xs

def get_batch2(X,Y,M,dtype):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = index_batch(X,batch_indices,dtype)
    batch_ys = index_batch(Y,batch_indices,dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

def get_sequential_lifted_mdl(nb_monomials,D_out, bias=False):
    return torch.nn.Sequential(torch.nn.Linear(nb_monomials,D_out,bias=bias))

def train_SGD(mdl, M,eta,nb_iter,logging_freq ,dtype, X_train,Y_train):
    ##
    N_train,_ = tuple( X_train.size() )
    #print(N_train)
    for i in range(nb_iter):
        # Forward pass: compute predicted Y using operations on Variables
        batch_xs, batch_ys = get_batch2(X_train,Y_train,M,dtype) # [M, D], [M, 1]
        ## FORWARD PASS
        y_pred = mdl.forward(batch_xs)
        ## LOSS + Regularization
        batch_loss = (1/M)*(y_pred - batch_ys).pow(2).sum()
        ## BACKARD PASS
        batch_loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
        ## SGD update
        for W in mdl.parameters():
            delta = eta*W.grad.data
            W.data.copy_(W.data - delta)
        ## train stats
        if i % (nb_iter/10) == 0 or i == 0:
            current_train_loss = (1/N_train)*(mdl.forward(X_train) - Y_train).pow(2).sum().data.numpy()
            print('i = {}, current_loss = {}'.format(i, current_train_loss ) )
        ## Manually zero the gradients after updating weights
        mdl.zero_grad()
##
logging_freq = 100
dtype = torch.FloatTensor
## SGD params
M = 3
eta = 0.0002
nb_iter = 20*1000
##
lb,ub = 0,1
f_target = lambda x: np.sin(2*np.pi*x)
N_train = 5
X_train = np.linspace(lb,ub,N_train)
Y_train = f_target(X_train)
## degree of mdl
Degree_mdl = 4
## pseudo-inverse solution
c_pinv = np.polyfit( X_train, Y_train , Degree_mdl )[::-1]
## linear mdl to train with SGD
nb_terms = c_pinv.shape[0]
mdl_sgd = get_sequential_lifted_mdl(nb_monomials=nb_terms,D_out=1, bias=False)
## Make polynomial Kernel
poly_feat = PolynomialFeatures(degree=Degree_mdl)
Kern_train = poly_feat.fit_transform(X_train.reshape(N_train,1))
Kern_train_pt, Y_train_pt = Variable(torch.FloatTensor(Kern_train).type(dtype), requires_grad=False), Variable(torch.FloatTensor(Y_train).type(dtype), requires_grad=False)
train_SGD(mdl_sgd, M,eta,nb_iter,logging_freq ,dtype, Kern_train_pt,Y_train_pt)

the error seems to hover on 2ish:

i = 0, current_loss = [ 2.08996224]
i = 2000, current_loss = [ 2.03536892]
i = 4000, current_loss = [ 2.02014995]
i = 6000, current_loss = [ 2.01307297]
i = 8000, current_loss = [ 2.01300406]
i = 10000, current_loss = [ 2.01125693]
i = 12000, current_loss = [ 2.01162267]
i = 14000, current_loss = [ 2.01296973]
i = 16000, current_loss = [ 2.00951076]
i = 18000, current_loss = [ 2.00967121]

which is weird cuz it should be able to reach zero.

I also plotted the learned function:

the code for the plotting:

##
x_horizontal = np.linspace(lb,ub,1000).reshape(1000,1)
X_plot = poly_feat.fit_transform(x_horizontal)
X_plot_pytorch = Variable( torch.FloatTensor(X_plot), requires_grad=False)
##
fig1 = plt.figure()
#plots objs
p_sgd, = plt.plot(x_horizontal, [ float(f_val) for f_val in mdl_sgd.forward(X_plot_pytorch).data.numpy() ])
p_pinv, = plt.plot(x_horizontal, np.dot(X_plot,c_pinv))
p_data, = plt.plot(X_train,Y_train,'ro')
## legend
nb_terms = c_pinv.shape[0]
legend_mdl = f'SGD solution standard parametrization, number of monomials={nb_terms}, batch-size={M}, iterations={nb_iter}, step size={eta}'
plt.legend(
        [p_sgd,p_pinv,p_data],
        [legend_mdl,f'linear algebra soln, number of monomials={nb_terms}',f'data points = {N_train}']
    )
##
plt.xlabel('x'), plt.ylabel('f(x)')
plt.show()

I actually went ahead and implemented a TensorFlow version. That one does seem to train the model. I tried having both of them match by giving them the same initialization:

mdl_sgd[0].weight.data.fill_(0)

but that still didn't work. Tensorflow code:

graph = tf.Graph()
with graph.as_default():
    X = tf.placeholder(tf.float32, [None, nb_terms])
    Y = tf.placeholder(tf.float32, [None,1])
    w = tf.Variable( tf.zeros([nb_terms,1]) )
    #w = tf.Variable( tf.truncated_normal([Degree_mdl,1],mean=0.0,stddev=1.0) )
    #w = tf.Variable( 1000*tf.ones([Degree_mdl,1]) )
    ##
    f = tf.matmul(X,w) # [N,1] = [N,D] x [D,1]
    #loss = tf.reduce_sum(tf.square(Y - f))
    loss = tf.reduce_sum( tf.reduce_mean(tf.square(Y-f), 0))
    l2loss_tf = (1/N_train)*2*tf.nn.l2_loss(Y-f)
    ##
    learning_rate = eta
    #global_step = tf.Variable(0, trainable=False)
    #learning_rate = tf.train.exponential_decay(learning_rate=eta, global_step=global_step,decay_steps=nb_iter/2, decay_rate=1, staircase=True)
    train_step = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss)
    with tf.Session(graph=graph) as sess:
        Y_train = Y_train.reshape(N_train,1)
        tf.global_variables_initializer().run()
        # Train
        for i in range(nb_iter):
            #if i % (nb_iter/10) == 0:
            if i % (nb_iter/10) == 0 or i == 0:
                current_loss = sess.run(fetches=loss, feed_dict={X: Kern_train, Y: Y_train})
                print(f'i = {i}, current_loss = {current_loss}')
            ## train
            batch_xs, batch_ys = get_batch(Kern_train,Y_train,M)
            sess.run(train_step, feed_dict={X: batch_xs, Y: batch_ys})

I also tried changing the initialization but it didn't change anything, which makes sense cuz it shouldn't make a big difference:

mdl_sgd[0].weight.data.normal_(mean=0,std=0.001)

Original post:

https://discuss.pytorch.org/t/how-to-train-a-simple-linear-regression-model-with-sgd-in-pytorch-successfully/9620

This is how it should look like:

SOLUTION:

it seems that there is an issue with the result being returned as a vector instead of a number causing the issue. i.e. the following code fixed things:

    y_pred = model(batch_xs).view(-1) # change this to "y_pred = model(batch_xs)" to get the incorrect results
    loss = (y_pred - batch_ys).pow(2).mean()

which seems completely mysterious to me. Does someone know why this fixed the issue? it just seems like magic.

Output:
The bug is really subtle but essentially it's because pytorch is using numpy broadcasting rules. So when a column vector (3,1) and an array (i.e. dim is (3,) ) then what happens is that broadcasting produces a (3,3) matrix (note this wouldn't happen when you subtract a row vector (1,3) vector with a (3,) array, I guess arrays are treated as row vectors). This is really bad because it means that we compute the matrix of all pairwise differences between every label and every prediction. Of course this is nonsensical and produces a bug because we don't want the prediction of the first label point to match the prediction of every other label in the data set. Of course that won't produce anything sensible.

So it seems the answer is just to avoid wrong numpy broadcasting by either reshaping things during training or before the data is fed. Either one should work.

To avoid the error one can attach use this code:

def check_vectors_have_same_dimensions(Y,Y_):
    '''
    Checks that vector Y and Y_ have the same dimensions. If they don't
    then there might be an error that could be caused due to wrong broadcasting.
    '''
    DY = tuple( Y.size() )
    DY_ = tuple( Y_.size() )
    if len(DY) != len(DY_):
        return True
    for i in range(len(DY)):
        if DY[i] != DY_[i]:
            return True
    return False


Instruction: How to change the picture size in PyTorch

Input:
I'm trying to convert CNN Keras model for Emotion Recognition using FER2013 dataset to PyTorch model and I have following error:

Traceback (most recent call last):
  File "VGG.py", line 112, in <module>
    transfer.keras_to_pytorch(keras_network, pytorch_network)
  File "/home/eorg/NeuralNetworks/user/Project/model/nntransfer.py", line 121, in keras_to_pytorch
    pytorch_model.load_state_dict(state_dict)
  File "/home/eorg/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 334, in load_state_dict
    own_state[name].copy_(param)
RuntimeError: inconsistent tensor size at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorCopy.c:51

I understood that the error is related to the shape of images. In Keras the input size is defined to be 48 by 48.

And my question is how to define in PyTorch models that of my pictures are the shape of 48x48? I couldn't find such function in the documentation and examples.

Any help would be useful!

Output:
In order to automatically resize your input images you need to define a preprocessing pipeline all your images go through. This can be done with torchvision.transforms.Compose() (Compose docs). To resize Images you can use torchvision.transforms.Scale() (Scale docs) from the torchvision package.

See the documentation:
Note, in the documentation it says that .Scale() is deprecated and .Resize() should be used instead. Resize docs

This would be a minimal working example:

import torch
from torchvision import transforms

p = transforms.Compose([transforms.Scale((48,48))])

from PIL import Image

img = Image.open('img.jpg')

img.size
# (224, 224) <-- This will be the original dimensions of your image

p(img).size
# (48, 48) <-- This will be the rescaled/resized dimensions of your image


Instruction: Pytorch: how to convert data into tensor

Input:
I am a beginner for Pytorch.
I was trying to write CNN code referring Pytorch tutorial.
Below is a part of the code, but it shows error "RuntimeError: Variable data has to be a tensor, but got list". I tried to cast input data to tensor but didn't work well. If anybody know the solution, please help me out...

    def read_labels(file):
      dic = {}
      with open(file) as f:
        reader = f
        for row in reader:
            dic[row.split(",")[0]]  = row.split(",")[1].rstrip() #rstrip(): eliminate "\n"
      return dic

    image_names= os.listdir("./train_mini")
    label_dic = read_labels("labels.csv")

    names =[]
    labels = []
    images =[]

    for name in image_names[1:]:
        images.append(cv2.imread("./train_mini/"+name))
        labels.append(label_dic[os.path.splitext(name)[0]])

    """
    Data distribution
    """
    N = len(images)
    N_train = int(N * 0.7)
    N_test = int(N*0.2)

    X_train, X_tmp, Y_train, Y_tmp = train_test_split(images, labels, train_size=N_train)
    X_validation, X_test, Y_validation, Y_test = train_test_split(X_tmp, Y_tmp, test_size=N_test)

    """
    Model Definition
    """

    class CNN(nn.Module):
        def __init__(self):
            super(CNN, self).__init__()
            self.head = nn.Sequential(
                nn.Conv2d(in_channels=1, out_channels=10,
                          kernel_size=5, stride=1),
                nn.MaxPool2d(kernel_size=2),
                nn.ReLU(),
                nn.Conv2d(10, 20, kernel_size=5),
                nn.MaxPool2d(kernel_size=2),
                nn.ReLU())
            self.tail = nn.Sequential(
                nn.Linear(320, 50),
                nn.ReLU(),
                nn.Linear(50, 10))

        def forward(self, x):
            x = self.head(x)
            x = x.view(-1, 320)
            x = self.tail(x)
            return F.log_softmax(x)

    CNN = CNN()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(CNN.parameters(), lr=0.001, momentum=0.9)

    """
    Training
    """
    batch_size = 50
    for epoch in range(2):  # loop over the dataset multiple times
        running_loss = 0.0
        for i in range(N / batch_size):
        #for i, data in enumerate(trainloader, 0):
            batch = batch_size * i

            # get the inputs
            images_batch = X_train[batch:batch + batch_size]
            labels_batch = Y_train[batch:batch + batch_size]

            # wrap them in Variable
            images_batch, labels_batch = Variable(images_batch), Variable(labels_batch)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = CNN(images_batch)
            loss = criterion(outputs, labels_batch)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.data[0]
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

    print('Finished Training')

And error is happening here

# wrap them in Variable
images_batch, labels_batch = Variable(images_batch), Variable(labels_batch)

Output:
If my guess is correct, you are probably getting error in the following line.

# wrap them in Variable
images_batch, labels_batch = Variable(images_batch), Variable(labels_batch)

It means, images_batch and/or labels_batch are lists. You can simple convert them to numpy array and then convert to tensor as follows.

# wrap them in Variable
images_batch = torch.from_numpy(numpy.array(images_batch))
labels_batch = torch.from_numpy(numpy.array(labels_batch))

It should solve your problem.

Edit: If you get the following error while running the above snippet of code:

  "RuntimeError: can't convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8."

You can create the numpy array by giving a data type. For example,

images_batch = torch.from_numpy(numpy.array(images_batch, dtype='int32'))

I am assuming images_batch contains pixel information of images, so I used int32. For more information, see official documentation.


Instruction: pytorch, AttributeError: module 'torch' has no attribute 'Tensor'

Input:
I'm working with Python 3.5.1 on a computer having CentOS Linux 7.3.1611 (Core) operating system.

I'm trying to use PyTorch and I'm getting started with this tutorial.

Unfortunately, the #4 line of the example creates troubles:

>>> torch.Tensor(5, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch' has no attribute 'Tensor'

I cannot understand this error... of course in Torch the 'torch' does have an attribute 'Tensor'. The same command works in Torch.

How can I solve this problem?

Output:
The Python binary that you are running does not have torch installed. It does have a directory named torch on the module search path, and it is treated as a namespace package:

$ pwd
/some/path
$ python3 -c 'import torch; print(torch); print(torch.__path__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
$ mkdir torch
$ python3 -c 'import torch; print(torch); print(torch.__path__)'
<module 'torch' (namespace)>
_NamespacePath(['/some/path/torch'])

Any directory without a __init__.py file present in it, located on your module search path, will be treated as a namespace, provided no other Python modules or packages by that name are found anywhere else along the search path.

This means that if torch was installed for your Python binary, it doesn't matter if there is a local torch directory:

$ ls -ld torch/
drwxr-xr-x  2 mjpieters users  68 Nov 23 13:57 torch/
$ mkdir -p additional_path/torch/
$ touch additional_path/torch/__init__.py
$ PYTHONPATH="./additional_path" python3 -c 'import os.path as p, sys; print(*(t for t in (p.join(e, "torch") for e in sys.path) if p.exists(t)), sep="\n")'
torch
/some/path/additional_path/torch
$ PYTHONPATH="./additional_path" python3 -c 'import torch; print(torch); print(torch.__path__)'
<module 'torch' from '/some/path/additional_path/torch/__init__.py'>
['/some/path/additional_path/torch']

The above shows that sys.path lists the torch directory first, followed by additional_path/torch, but the latter is loaded as the torch module when you try to import it. That's because Python gives priority to top-level modules and packages before loading a namespace package.

You need to install torch correctly for your current Python binary, see the project homepage; when using pip you may want to use the Python binary with the -m switch instead:

python3.5 -m pip install http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
python3.5 -m pip install torchvision

So replace the pip3 the homepage instructions use with python3.5 -m pip; python3.5 can also be the full path to your Python binary.

Do use the correct download.pytorch.org URL for the latest version.

You don't have to move the directory aside, but if you do want to and don't know where it is located, use print(torch.__path__) as I've shown above.

Again, note that if you do have an __init__.py file in a local torch directory, it becomes a regular package and it'll mask packages installed by pip into the normal site-packages location. If you have such a package, or a local torch.py single-file module, you need to rename those. The diagnostic information looks different in that case:

$ pwd
/some/path
$ python3 -c 'import torch; print(torch); print(torch.__path__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
$ mkdir torch
$ touch torch/__init__.py  # make it a package
$ python3 -c 'import torch; print(torch); print(torch.__path__)'
<module 'torch' from '/some/path/torch/__init__.py'>
['/some/path/torch']
$ rm -rf torch/
$ touch torch.py           # make it a module
$ python3 -c 'import torch; print(torch); print(torch.__file__)'
<module 'torch' from '/some/path/torch.py'>
/some/path/torch.py

Note the differences; a namespace package, above, uses <module 'name' (namespace)>, while a regular package uses ), while a plain module uses`.

Such packages and modules (not namespace packages) are found first and stop the search. If the found package or module is not the one you wanted, you need to move them aside or rename them.


Instruction: From 2D to 3D using convolutional autoencoder

Input:
I’d like to reconstruct 3D object from 2D images.
For that, I try to use convolutional auto encoder. However, in which layer should I lift the dimensionality?

I wrote a code below, however, it shows an error:

  “RuntimeError: invalid argument 2: size ‘[1 x 1156 x 1156]’ is invalid for input of with 2312 elements at pytorch-src/torch/lib/TH/THStorage.c:41”

class dim_lifting(nn.Module):
    def __init__(self):
        super(dim_lifting, self).__init__()
        self.encode = nn.Sequential(
            nn.Conv2d(1, 34, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.Conv2d(34, 16, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 8, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.LeakyReLU()
        )

        self.fc1 = nn.Linear(2312, 2312)
        self.decode = nn.Sequential(
            nn.ConvTranspose3d(1, 16, kernel_size=5, padding=2),
            nn.LeakyReLU(),
            nn.ConvTranspose3d(16, 32, kernel_size=5, padding=2),
            nn.LeakyReLU(),
            nn.MaxPool2d(2))

    def forward(self, x):
        out = self.encode(x)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = out.view(1, 1156, 1156)
        out = self.decode(out)
        return out

Error happens here

out = out.view(1, 1156, 1156)

Output:
I cannot test my suggestion because your example is not complete.
I think your line should like

out = out.view(x.size(0), -1)

this way you're flattening out your input.


Instruction: How Pytorch do row normalization for each matrix in a 3D Tensor(Variable)?

Input:
If I have a 3D Tensor (Variable) with size [a,b,c].
consider it as a b*c matrix, and I hope that all these a matrix got row normalized.

Output:
You can use the normalize function.

import torch.nn.functional as f
f.normalize(input, p=2, dim=2)

The dim=2 argument tells along which dimension to normalize (divide each row vector by its p-norm.


Instruction: Does tensorflow have the function similar to pytorch's "masked_fill_"

Input:
I want to set INF value to matrix by mask matrix, just like pytorch code:

scores.data.masked_fill_(y_mask.data, -float('inf'))

I try to use tf.map_fn to implement that, but the performance is too slow. So does tensorflow have any efficient function to implement that?

Output:
I have used a math calculate method to instead. It's valid and much faster.

def mask_fill_inf(matrix, mask):
    negmask = 1 - mask
    num = 3.4 * math.pow(10, 38)
    return (matrix * mask) + (-((negmask * num + num) - num))

Do anyone have the better method?


Instruction: No module named 'torch.utils.data.distributed'

Input:
I have installed anaconda and pytorch on my windows 10 and there was no errors when I installed it.

But I don't know why I don't have the modules or packages in pytorch.
Have you experienced the same problem?

This is the program I'm testing: https://github.com/pytorch/examples/blob/master/imagenet/main.py

Output:
Because you didn't provide any additional information, there are couple of things you can try:

1) first make sure that you've already installed torchvision

2) Then try the following import:

# this import is necessary
import torch.utils.data


Instruction: `THIndexTensor_(size)(target, 0) == batch_size' failed. at d:\projects\pytorch\torch\lib\thnn\generic/ClassNLLCriterion.c:54

Input:
I am trying to train my neural networks on dog breeds data set. After feed-forward, during the loss computation it throws this error :

RuntimeError: Assertion `THIndexTensor_(size)(target, 0) == batch_size' failed.  at d:\projects\pytorch\torch\lib\thnn\generic/ClassNLLCriterion.c:54

Code:

criterion =nn.CrossEntropyLoss()
optimizer=optim.Adam(net.parameters(),lr=0.001)

for epoch in range(10):  # loop over the dataset multiple times
        running_loss = 0.0
        print(len(trainloader))
        for i, data in enumerate(trainloader, 0):
            # get the inputs
            inputs, labels  = data

            # wrap them in Variable
            inputs, labels = Variable(inputs).float(), Variable(labels).float().type(torch.LongTensor)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)

            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            # print statistics
            running_loss += loss.data[0]
            if i % 2000 == 1999:    # print every 2000 mini-batches
                print('[%d, %5d] loss: %.3f' %
                      (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

print('Finished Training')

Error is generated in this line :

loss = criterion(outputs, labels)

What's the issue ??

Output:
I think problem is that you are missing the batch dimension on the tensor labels. The error says that the size of the 0th dimension is not equal to the batch size.

Try changing this:

loss = criterion(outputs, labels.unsqueeze(0))

Please note, the outputs tensor should have one more dimension than the labels tensor corresponding to a score for each label and the labels should just contain the index of the correct label.


Instruction: why is my simple feedforward neural network diverging (pytorch)?

Input:
I am experimenting with a simple 2 layer neural network with pytorch, feeding in only three inputs of size 10 each, with a single value as output. I have normalised inputs and lowered learning rate. It is my understanding that a two layer fully connected neural network should be able to trivially fit to this data

Features:

0.8138  1.2342  0.4419  0.8273  0.0728  2.4576  0.3800  0.0512  0.6872  0.5201
1.5666  1.3955  1.0436  0.1602  0.1688  0.2074  0.8810  0.9155  0.9641  1.3668
1.7091  0.9091  0.5058  0.6149  0.3669  0.1365  0.3442  0.9482  1.2550  1.6950
[torch.FloatTensor of size 3x10]

Targets
[124, 125, 122]
[torch.FloatTensor of size 3]

The code is adapted from a simple example and I am using MSELoss as the loss function. The loss diverges to infinity after just a few iterations:

features = torch.from_numpy(np.array(features))

x_data = Variable(torch.Tensor(features))
y_data = Variable(torch.Tensor(targets))

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(10,5)
        self.linear2 = torch.nn.Linear(5,1)

    def forward(self, x):
        l_out1 = self.linear(x)
        y_pred = self.linear2(l_out1)
        return y_pred

model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optim = torch.optim.SGD(model.parameters(), lr = 0.001)

def main():
    for iteration in range(1000):
        y_pred = model(x_data)
        loss = criterion(y_pred, y_data)

        print(iteration, loss.data[0])
        optim.zero_grad()

        loss.backward()
        optim.step()

Any help would be appreciated. Thanks

EDIT:

Indeed it seems that this was simply due to the learning rate being too high. Setting to 0.00001 fixes convergence issues, albeit giving very slow convergence.

Output:
This is because you're not using a non-linearity between layers, and your network is still Linear.

You can use Relu in order to make it non linear. You can change the forward method like this :

...
y_pred = torch.nn.functional.F.relu(self.linear2(l_out1))
...


Instruction: Why tensorflow GPU memory usage decreasing when I increasing the batch size?

Input:
Recently I implemented a VGG-16 network using both Tensorflow and PyTorch, data set is CIFAR-10. Each picture is 32 * 32 RGB.

I use a 64 batch size in beginning, while I found PyTorch using much less GPU memory than tensorflow. Then I did some experiments and got a figure, which is posted below.

After some researching, I known the tensorflow using BFC algorithm to manage memory. So it's can explain why tensorflow's memory using decreasing or increasing by 2048, 1024, ... MB and sometimes the memory use not increasing when batch size is bigger.

But I am still confused, why the memory use is lower when batch size is 512 than batch size is 384, 448 etc. which has a smaller batch size. The same as when batch size is from 1024 to 1408, and batch size is 2048 to 2688.

Here is my source code:

PyTorch:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16-pytorch.py

Tensorflow:https://github.com/liupeng3425/tesorflow-vgg/blob/master/vgg-16.py

edit:
I have two Titan XP on my computer, OS: Linux Mint 18.2 64-bit.

I determine GPU memory usage with command nvidia-smi.

My code runs on GPU1, which is defined in my code:

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"

And I am sure there only one application using GPU1.

GPU memory usage can be determined by the application list below.
For example, like the posted screen shot below, process name is /usr/bin/python3 and its GPU memory usage is 1563 MiB.

Output:
As noted in the comments, by default TensorFlow always takes up all memory on a GPU. I assume you have disabled that function for this test, but it does show that the algorithms do not generally attempt to minimize the memory that is reserved, even if it's not all utilized in the calculations.
To find the optimal configuration for your device and code, TensorFlow often runs (parts of) the first calculation multiple times. I suspect that this included settings for pre-loading data onto the GPU. This would mean that the numbers you see happen to be the optimal values for your device and configuration.
Since TensorFlow doesn't mind using more memory, 'optimal' here is measured by speed, not memory usage.


Instruction: Q-value keeps stepping down when training a DQN

Input:
I am training a DQN and the Q-value keeps going down. The curve looks very weird (see below).

Every step corresponds to an update to target network.
Any possible reason why this happens?

Output:
Does the step correspond to the Target Q network update? If so try to:

1) update the TargetQ network less frequently

2) increase the discount factor (e.g. to .99 if you were using .5)

3) use a smooth update for the TargetQ network in the form (1 - tau)old + tauv1


Instruction: Loss dimensionality issue in PyTorch (sequence to label learning)

Input:
I am doing a sequence to label learning model in PyTorch. I have two sentences and I am classifying whether they are entailed or not (SNLI dataset). I concatenate two 50 word sentences together (sometimes padded) into a vector of length 100. I then send in minibatches into word embeddings -> LSTM -> Linear layer. I am doing cross entropy loss but I need a vector of [mini_batch, C] to go into the CrossEntropyLoss function. Instead I still have the 100 words in my vector as [mini_batch, 100, C]

Here is my model:

class myLSTM(nn.Module):
    def __init__(self, h_size=128, v_size=10, embed_d=300, mlp_d=256):
        super(myLSTM, self).__init__()
        self.embedding = nn.Embedding(v_size, embed_d)
        self.lstm = nn.LSTM(embed_d, h_size, num_layers=1, bidirectional=True, batch_first=True)
        self.mlp = nn.Linear(mlp_d, 1024)

        # Set static embedding vectors
        self.embedding.weight.requires_grad = False

        #self.sm = nn.CrossEntropyLoss()

    def display(self):
        for param in self.parameters():
            print(param.data.size())

    def filter_params(self):
        # Might not be compatible with python 3
        #self.parameters = filter(lambda p: p.requires_grad, self.parameters())
        pass

    def init_hidden(self):
        # Need to init hidden weights in LSTM
        pass

    def forward(self, sentence):
        print(sentence.size())
        embeds = self.embedding(sentence)
        print(embeds.size())
        out, _ = self.lstm(embeds)
        print(out.size())
        out = self.mlp(out)
        return out

My training sequences with output:

batch_size = 3
SGD_optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.01, weight_decay=1e-4)
ADM_optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.01)

criterion = nn.CrossEntropyLoss()
num_epochs = 50

from torch.autograd import Variable
from torch import optim

for epoch in range(num_epochs):
    print("Epoch {0}/{1}: {2}%".format(epoch, num_epochs, float(epoch)/num_epochs))
    for start, end in tqdm(batch_index_gen(batch_size, len(n_data))):

        # Convert minibatch to numpy
        s1, s2, y = convert_to_numpy(n_data[start:end])

        # Convert numpy to Tensor
        res = np.concatenate((s1,s2), axis=1) # Attach two sentences into 1 input vector
        input_tensor = torch.from_numpy(res).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.FloatTensor)
        data, target = Variable(input_tensor), Variable(target_tensor)

        # Zero gradients
        SGD_optimizer.zero_grad()

        # Forward Pass
        output = model.forward(data)
        print("Output size: ")
        print(output.size())
        print("Target size: ")
        print(target.size())
        # Calculate loss with respect to training labels
        loss = criterion(output, target)

        # Backprogogate and update optimizer
        loss.backward()
        SGD_optimizer.step()
        #ADAM_optimizer.step()

output:

Epoch 0/50: 0.0%
torch.Size([3, 100])
torch.Size([3, 100, 300])
torch.Size([3, 100, 256])
Output size:
torch.Size([3, 100, 1024])
Target size:
torch.Size([3])

error:

ValueError: Expected 2 or 4 dimensions (got 3)

EDITED -------------------------------------------------------------------

I have now got my model training but I am getting low accuracy. Is there an issue with my LSTM outputs being concatenated and then condensed to a smaller tensor to go through my linear layer?

New Model:

class myLSTM(nn.Module):
    def __init__(self, h_size=128, v_size=10, embed_d=300, mlp_d=256, num_classes=3, lstm_layers=1):
        super(myLSTM, self).__init__()
        self.num_layers = lstm_layers
        self.hidden_size = h_size
        self.embedding = nn.Embedding(v_size, embed_d)
        self.lstm = nn.LSTM(embed_d, h_size, num_layers=lstm_layers, bidirectional=True, batch_first=True)
        self.mlp = nn.Linear(2 * h_size * 2, num_classes)

        # Set static embedding vectors
        self.embedding.weight.requires_grad = False

    def forward(self, s1, s2):
        # Set initial states
        #h0 = Variable(torch.zeros(self.num_layers*2, s1.size(0), self.hidden_size)).cuda() # 2 for bidirection
        #c0 = Variable(torch.zeros(self.num_layers*2, s1.size(0), self.hidden_size)).cuda()

        batch_size = s1.size()[0]
        embeds_1 = self.embedding(s1)
        embeds_2 = self.embedding(s2)
        _, (h_1_last, _) = self.lstm(embeds_1)#, (h0, c0)) #note the change here. Last hidden state is taken
        _, (h_2_last, _) = self.lstm(embeds_2)#, (h0, c0))
        concat = torch.cat( (h_1_last, h_2_last), dim=2) #double check the dimension
        concat = concat.view(batch_size, -1)
        scores = self.mlp(concat)
        return scores

New Training

batch_size = 64
SGD_optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.001, weight_decay=1e-4)

criterion = nn.CrossEntropyLoss()
num_epochs = 10
model.train()

if cuda:
    model = model.cuda()
    criterion = criterion.cuda()

from torch.autograd import Variable
from torch import optim

epoch_losses = []

for epoch in range(num_epochs):
    print("Epoch {0}/{1}: {2}%".format(epoch, num_epochs, 100*float(epoch)/num_epochs))

    # Batch loss aggregator
    losses = []

    for start, end in tqdm(batch_index_gen(batch_size, len(n_data))):
        # Convert minibatch to numpy
        s1, s2, y = convert_to_numpy(n_data[start:end])

        # Convert numpy to Tensor
        s1_tensor = torch.from_numpy(s1).type(torch.LongTensor)
        s2_tensor = torch.from_numpy(s2).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.LongTensor)

        s1 = Variable(s1_tensor)
        s2 = Variable(s2_tensor)
        target = Variable(target_tensor)

        if cuda:
            s1 = s1.cuda()
            s2 = s2.cuda()
            target = target.cuda()

        # Zero gradients
        SGD_optimizer.zero_grad()

        # Forward Pass
        output = model.forward(s1,s2)

        # Calculate loss with respect to training labels
        loss = criterion(output, target)
        losses.append(loss.data[0])

        # Backprogogate and update optimizer
        loss.backward()
        SGD_optimizer.step()

    # concat losses to epoch losses
    epoch_losses += losses

training with tensor sizes printed:

Epoch 0/10: 0.0%
Batch size: 64
Sentences
torch.Size([64, 50])
torch.Size([64, 50])
torch.Size([64, 50, 300])
torch.Size([64, 50, 300])
Hidden states
torch.Size([2, 64, 128])
torch.Size([2, 64, 128])
Concatenated hidden states
torch.Size([2, 64, 256])
Reshaped tensors for linear layer
torch.Size([64, 512])
Linear propogation
torch.Size([64, 3])

Evaluation

def eval_model(model, mode='dev'):
    file_name = 'snli_1.0/snli_1.0_dev.jsonl' if mode == 'dev' else 'snli_1.0/snli_1.0_test.jsonl'

    dev_data, _ = obtain_data(file_name)
    dev_n_data = vocab.process_data(dev_data)

    print("Length of data: {}".format(len(dev_n_data)))

    eval_batch_size = 1024
    model.eval()

    total = len(dev_n_data)
    hit = 0
    correct = 0

    # Batch dev eval
    for start, end in batch_index_gen(eval_batch_size, len(dev_n_data)):

        s1, s2, y = convert_to_numpy(dev_n_data[start:end])

        s1_tensor = torch.from_numpy(s1).type(torch.LongTensor)
        s2_tensor = torch.from_numpy(s2).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.LongTensor)

        s1 = Variable(s1_tensor, volatile=True)
        s2 = Variable(s2_tensor, volatile=True)
        target = Variable(target_tensor, volatile=True)

        if cuda:
            s1 = s1.cuda()
            s2 = s2.cuda()
            target = target.cuda()

        output = model.forward(s1,s2)
        loss = criterion(output, target)

        #print("output size: {}".format(output.size()))
        #print("target size: {}".format(target.size()))
        pred = output.data.max(1)[1] # get the index of the max log-probability
        #print(pred[:5])
        #print(output[:])
        correct += pred.eq(target.data).cpu().sum()

    return correct / float(total)

eval_model(model)

Output:
I think there is an issue in a way you are trying to solve an entailment problem.

Maybe you can do it this way:

design your module to accept two sentences as input
embed both of them with your embeddings
encode them using the LSTM module.
now you have two fixed length vector representations of two sentences. Simpliest thing to do is to just concatenate them
together.
Add liner layer on top to evaluate scores for each entailment class (3 I suppose)
apply softmax to get a proper probability distribution

So your model can look like this (double check the dimensions):

class myLSTM(nn.Module):
    def __init__(self, h_size=128, v_size=10, embed_d=300, num_classes = 3):
        super(myLSTM, self).__init__()
        self.embedding = nn.Embedding(v_size, embed_d)
        self.lstm = nn.LSTM(embed_d, h_size, num_layers=1, bidirectional=True, batch_first=True)
        self.mlp = nn.Linear(2*h_size*2, num_classes) #<- change here

    def forward(self, sentence1, sentence2):
        embeds_1 = self.embedding(sentence1)
        embeds_2 = self.embedding(sentence2)
        _, (h_1_last, _) = self.lstm(embeds_1) #note the change here. Last hidden state is taken
        _, (h_2_last, _) = self.lstm(embeds_2)
        concat = torch.concat([h_1_last, h_2_last], dim=1) #double check the dimension
        scores = self.mlp(concat)
        probas = F.softmax(scores) #from torch.functional ...

Then you can play around with adding more hidden layers or thinking how combining two sentences can be done in more intelligent way (attention, etc).
Double check what CrossEntropyLoss accepts as input and target and adjust (is it unnormalized class scores or probability distribution). Check http://pytorch.org/docs/master/nn.html#lstm for LSTM module documentation to clarify what LSTM returns (do you need hidden states for every word or just the representation after the last one).


Instruction: What does the following error mean, when I tried to import torch library on iPython?

Input:
RuntimeError                              Traceback (most recent call last)

RuntimeError: module compiled against API version 0xb but this version of numpy is 0xa

ImportError                               Traceback (most recent call last)
 in ()
----> 1 import torch

~/anaconda3/lib/python3.6/site-packages/torch/init.py in ()
     51 sys.setdlopenflags(_dl_flags.RTLD_GLOBAL | _dl_flags.RTLD_NOW)
     52
---> 53 from torch._C import *
     54
     55 all += [name for name in dir(_C)

ImportError: numpy.core.multiarray failed to import

Output:
You may need to update your numpy installation.

Run this in terminal:

pip install -U numpy

If this doesn't work, try uninstalling and then reinstalling it:

pip uninstall numpy
pip install numpy


Instruction: PyTorch ValueError: Target and input must have the same number of elements

Input:
I am kinda new to PyTorch, but I am trying to understand how the sizes of target and input work in torch.nn.BCELoss() when computing loss function.

import torch
import torch.nn as nn
from torch.autograd import Variable

time_steps = 15
batch_size = 3
embeddings_size = 100
num_classes = 2

model = nn.LSTM(embeddings_size, num_classes)
input_seq = Variable(torch.randn(time_steps, batch_size, embeddings_size))
lstm_out, _ = model(input_seq)
last_out = lstm_out[-1]
print(last_out)

loss = nn.BCELoss()
target = Variable(torch.LongTensor(batch_size).random_(0, num_classes))
print(target)
err = loss(last_out.long(), target)
err.backward()

I received the following error:

Warning (from warnings module):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 767
"Please ensure they have the same size.".format(target.size(), input.size()))

File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/functional.py", line 770, in binary_cross_entropy
"!= input nelement ({})".format(target.nelement(), input.nelement()))
ValueError: Target and input must have the same number of elements. target nelement (3) != input nelement (6)

This error definitely comes from the different sizes of last_out (size 3x2) and target (size 3). So my question is how can I convert last_out into something like target (of size 3 and containing just 0s and 1s) to compute loss function?

Output:
The idea of nn.BCELoss() is to implement the following formula:

Both o and tare tensors of arbitrary (but same!) size and i simply indexes each element of the two tensor to compute the sum above.

Typically, nn.BCELoss() is used in a classification setting: o and i will be matrices of dimensions N x D. N will be the number of observations in your dataset or minibatch. D will be 1 if you are only trying to classify a single property, and larger than 1 if you are trying to classify multiple properties. t, the target matrix, will only hold 0 and 1, as for every property, there are only two classes (that's where the binary in binary cross entropy loss comes from). o will hold the probability with which you assign every property of every observation to class 1.

Now in your setting above, it is not clear how many classes you are considering and how many properties there are for you. If there is only one property as suggested by the shape of your target, you should only output a single quantity, namely the probability of being in class 1 from your model. If there are two properties, your targets are incomplete! If there are multiple classes, you should work with torch.nn.CrossEntropyLossinstead of torch.nn.BCELoss().

As an aside, often it is desirable for the sake of numerical stability to use torch.nn.BCEWithLogitsLossinstead of torch.nn.BCELoss() following nn.Sigmoid() on some outputs.


Instruction: How does one use Hermite polynomials with Stochastic Gradient Descent (SGD)?

Input:
I was trying to train a simple polynomial linear model with pytorch using Hermite polynomials since they seem to have a better conditioned Hessian.

To do that I decided to use the hermvander since it gives the Vandermonde matrix with each entry being a Hermite term. To do that I just made my feature vectors be the outpute of hermvander:

Kern_train = hermvander(X_train,Degree_mdl)

however, when I proceeded to train I get NaN all the time. I suspected it could have been a step size issue but I decided to use the step size suggested by this question that already has my example working in R, so there is no need to search for a step size I thought. However, when I tried it it does not work.

Anyone has any idea whats going on?

Same error occurs in tensorflow:

import pdb

import numpy as np
from numpy.polynomial.hermite import hermvander

import random

import tensorflow as tf

def get_batch(X,Y,M):
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = X[batch_indices,:]
    batch_ys = Y[batch_indices]
    return batch_xs, batch_ys

##
D0=1
logging_freq = 100
## SGD params
M = 5
eta = 0.1
#eta = lambda i: eta/(i**0.6)
nb_iter = 500*10
##
lb,ub = 0,1
freq_sin = 4 # 2.3
f_target = lambda x: np.sin(2*np.pi*freq_sin*x)
N_train = 10
X_train = np.linspace(lb,ub,N_train)
Y_train = f_target(X_train).reshape(N_train,1)
x_horizontal = np.linspace(lb,ub,1000).reshape(1000,1)
## degree of mdl
Degree_mdl = N_train-1
## Hermite
Kern_train = hermvander(X_train,Degree_mdl)
print(f'Kern_train.shape={Kern_train.shape}')
Kern_train = Kern_train.reshape(N_train,Kern_train.shape[1])
##
Kern_train_pinv = np.linalg.pinv( Kern_train )
c_pinv = np.dot(Kern_train_pinv, Y_train)
nb_terms = c_pinv.shape[0]
##
condition_number_hessian = np.linalg.cond(Kern_train)
##
graph = tf.Graph()
with graph.as_default():
    X = tf.placeholder(tf.float32, [None, nb_terms])
    Y = tf.placeholder(tf.float32, [None,1])
    w = tf.Variable( tf.zeros([nb_terms,1]) )
    #w = tf.Variable( tf.truncated_normal([Degree_mdl,1],mean=0.0,stddev=1.0) )
    #w = tf.Variable( 1000*tf.ones([Degree_mdl,1]) )
    ##
    f = tf.matmul(X,w) # [N,1] = [N,D] x [D,1]
    #loss = tf.reduce_sum(tf.square(Y - f))
    loss = tf.reduce_sum( tf.reduce_mean(tf.square(Y-f), 0))
    l2loss_tf = (1/N_train)*2*tf.nn.l2_loss(Y-f)
    ##
    learning_rate = eta
    #global_step = tf.Variable(0, trainable=False)
    #learning_rate = tf.train.exponential_decay(learning_rate=eta, global_step=global_step,decay_steps=nb_iter/2, decay_rate=1, staircase=True)
    train_step = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss)
    with tf.Session(graph=graph) as sess:
        Y_train = Y_train.reshape(N_train,1)
        tf.global_variables_initializer().run()
        # Train
        for i in range(nb_iter):
            #if i % (nb_iter/10) == 0:
            if i % (nb_iter/10) == 0 or i == 0:
                current_loss = sess.run(fetches=loss, feed_dict={X: Kern_train, Y: Y_train})
                print(f'tf: i = {i}, current_loss = {current_loss}')
            ## train
            batch_xs, batch_ys = get_batch(Kern_train,Y_train,M)
            sess.run(train_step, feed_dict={X: batch_xs, Y: batch_ys})
print(f'condition_number_hessian = {condition_number_hessian}')
print('\a')

Totally self contained code in pytorch:

import numpy as np
from numpy.polynomial.hermite import hermvander

import random

import torch
from torch.autograd import Variable

def vectors_dims_dont_match(Y,Y_):
    '''
    Checks that vector Y and Y_ have the same dimensions. If they don't
    then there might be an error that could be caused due to wrong broadcasting.
    '''
    DY = tuple( Y.size() )
    DY_ = tuple( Y_.size() )
    if len(DY) != len(DY_):
        return True
    for i in range(len(DY)):
        if DY[i] != DY_[i]:
            return True
    return False

def index_batch(X,batch_indices,dtype):
    '''
    returns the batch indexed/sliced batch
    '''
    if len(X.shape) == 1: # i.e. dimension (M,) just a vector
        batch_xs = torch.FloatTensor(X[batch_indices]).type(dtype)
    else:
        batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    return batch_xs

def get_batch2(X,Y,M,dtype):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = index_batch(X,batch_indices,dtype)
    batch_ys = index_batch(Y,batch_indices,dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

def get_sequential_lifted_mdl(nb_monomials,D_out, bias=False):
    return torch.nn.Sequential(torch.nn.Linear(nb_monomials,D_out,bias=bias))

def train_SGD(mdl, M,eta,nb_iter,logging_freq ,dtype, X_train,Y_train):
    ##
    #pdb.set_trace()
    N_train,_ = tuple( X_train.size() )
    #print(N_train)
    for i in range(1,nb_iter+1):
        # Forward pass: compute predicted Y using operations on Variables
        batch_xs, batch_ys = get_batch2(X_train,Y_train,M,dtype) # [M, D], [M, 1]
        ## FORWARD PASS
        y_pred = mdl.forward(batch_xs)
        ## Check vectors have same dimension
        if vectors_dims_dont_match(batch_ys,y_pred):
            raise ValueError('You vectors don\'t have matching dimensions. It will lead to errors.')
        ## LOSS + Regularization
        batch_loss = (1/M)*(y_pred - batch_ys).pow(2).sum()
        ## BACKARD PASS
        batch_loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
        ## SGD update
        for W in mdl.parameters():
            delta = eta(i)*W.grad.data
            W.data.copy_(W.data - delta)
        ## train stats
        if i % (nb_iter/10) == 0 or i == 0:
            #X_train_, Y_train_ = Variable(X_train), Variable(Y_train)
            X_train_, Y_train_ = X_train, Y_train
            current_train_loss = (1/N_train)*(mdl.forward(X_train_) - Y_train_).pow(2).sum().data.numpy()
            print('\n-------------')
            print(f'i = {i}, current_train_loss = {current_train_loss}\n')
            print(f'eta*W.grad.data = {eta*W.grad.data}')
            print(f'W.grad.data = {W.grad.data}')
        ## Manually zero the gradients after updating weights
        mdl.zero_grad()
    final_sgd_error = current_train_loss
    return final_sgd_error
##
D0=1
logging_freq = 100
#dtype = torch.cuda.FloatTensor
dtype = torch.FloatTensor
## SGD params
M = 5
eta = 0.1
eta = lambda i: eta/(i**0.6)
nb_iter = 500*10
##
lb,ub = 0,1
freq_sin = 4 # 2.3
f_target = lambda x: np.sin(2*np.pi*freq_sin*x)
N_train = 10
X_train = np.linspace(lb,ub,N_train)
Y_train = f_target(X_train).reshape(N_train,1)
x_horizontal = np.linspace(lb,ub,1000).reshape(1000,1)
## degree of mdl
Degree_mdl = N_train-1
## Hermite
Kern_train = hermvander(X_train,Degree_mdl)
Kern_train = Kern_train.reshape(N_train,Kern_train.shape[2])
##
Kern_train_pinv = np.linalg.pinv( Kern_train )
c_pinv = np.dot(Kern_train_pinv, Y_train)
##
condition_number_hessian = np.linalg.cond(Kern_train)
## linear mdl to train with SGD
nb_terms = c_pinv.shape[0]
mdl_sgd = get_sequential_lifted_mdl(nb_monomials=nb_terms,D_out=1, bias=False)
mdl_sgd[0].weight.data.normal_(mean=0,std=0.001)
mdl_sgd[0].weight.data.fill_(0)
## Make polynomial Kernel
Kern_train_pt, Y_train_pt = Variable(torch.FloatTensor(Kern_train).type(dtype), requires_grad=False), Variable(torch.FloatTensor(Y_train).type(dtype), requires_grad=False)
final_sgd_error = train_SGD(mdl_sgd, M,eta,nb_iter,logging_freq ,dtype, Kern_train_pt,Y_train_pt)
## PRINT ERRORS
from plotting_utils import *

train_error_pinv = (1/N_train)*(np.linalg.norm(Y_train-np.dot(Kern_train,c_pinv))**2)
print('\n-----------------')
print(f'N_train={N_train}')
print(f'train_error_pinv = {train_error_pinv}')
print(f'final_sgd_error = {final_sgd_error}')

print(f'condition_number_hessian = {condition_number_hessian}')
print('\a')

Output: Maybe it's a bit late, but you might have a look at this https://github.com/Orcuslc/OrthNet


Instruction: what is the fastest way of loading images?

Input:
I have about 200,000 high resolution images, and loading such high quality images every time is time consuming.
Preloading all images might occupy too much memory.
How about saving each images into .npz file format and loading .npz instead of .jpg?  Would it be boosting speed?

Output:
You do not need to load all the image to memory at once. Considering also that we need to do data augmentation on the dataset during model training, it is impossible to load all images.

In PyTorch, you can use Dataset to store your training and validation set. The Dataset class has a parameter transforms(e.g., Scale, RandomCrop, etc.), which is used to transform the training image on the fly during training. Several ready-made dataset are also provided by torchvision package, see here.

Basic methold

PyTorch's builtin Dataloader has a num_worker, which is used to control how many subprocess you use for loading the data. Since your dataset is not so huge, that would be enough for your need. About how to set the appropriate number of worker, see here.

More references

There are discussion on PyTorch forum on fast image loading, use post1 and post2 as a start.


Instruction: Spiral Loss Function for Music Encoding

Input:
I am trying to develop an autoencoder for music generation; in pursuit of that end, I am attempting to develop a loss function which captures musical relationships.

My current idea is a 'Spiral' loss function, which is to say that if the system predicts the same note in a different octave, the loss should be smaller than if the note is just wrong. Additionally, notes that are close to the correct note, such as B and D to C should also have small losses. One can conceptually think of this as finding the distance between two points on a coil or spiral, such that the same notes in different octaves lie along a line tangent to the coil, but separated by some loop distance.

I am working in PyTorch, and my input representation is a 36 by 36 Tensor, where the rows represent the notes (MIDI range 48:84, the middle three octaves of a piano) and the columns represent time steps (1 column = 1/100th of a second). The values in the matrix are either 0 or 1, signifying that a note is on at a particular time.

Here is my current implementation of the loss:

def SpiralLoss():
    def spiral_loss(input, output):
        loss = Variable(torch.FloatTensor([0]))
        d = 5
        r = 10
        for i in xrange(input.size()[0]):
            for j in xrange(input.size()[3]):
                # take along the 1 axis because it's a column vector
                inval, inind = torch.max(input[i, :, :, j], 1)
                outval, outind = torch.max(output[i, :, :, j], 1)
                note_loss = (r*30*(inind%12 - outind%12)).float()
                octave_loss = (d*(inind/12 - outind/12)).float()
                loss += torch.sqrt(torch.pow(note_loss, 2) + torch.pow(octave_loss, 2))
        return loss
    return spiral_loss

The problem with this loss is that the max function is not differentiable. I cannot think of a way to make this loss differentiable, and was wondering if anyone might have any ideas or suggestions?

I'm not sure if this is the right place for a post like this, and so if it isn't, I would really appreciate any pointers towards a better location.

Output:
Taking the maximum here is not only problematic because of differentiability: If you only take the maximum of the output, and it is at the right place, slightly lower values in wrong positions don't get punished.

One rough idea would be to use a normal L1 or L2 loss on the difference of the input and a modified output vector: The output could be multiplied by some weight mask that punishes octave and note difference differently, like for example:

def create_mask(input_column):
    r = 10
    d = 5
    mask = torch.FloatTensor(input_column.size())
    _, max_ind = torch.max(input_column, 0)
    max_ind = int(max_ind[0])
    for i in range(mask.size(0)):
        mask[i] = r*abs(i-max_ind)%12 + d*abs(i-max_ind)/12
    return mask

This is just roughly written, not something ready but in theory it should do the job. The mask vector should be set to requires_grad=False since it is an exact constant we compute for each input. Thus, you can use the maximum on the input but don't use the max on the output.

I hope it helps!


Instruction: pytorch data loader multiple iterations

Input:
i use iris-dataset to train a simple network with pytorch.

trainset = iris.Iris(train=True)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=150,
                                          shuffle=True, num_workers=2)

dataiter = iter(trainloader)

the dataset itself has only 150 data points, and pytorch dataloader iterates jus t once over the whole dataset, because of the batch size of 150.

My question is now, is there generally any way to tell dataloader of pytorch to repeat over the dataset if it's once done with iteration?

thnaks

update

got it runnning :)
just created a sub class of dataloader and implemented my own __next__()

Output:
To complement the previous answers. To be comparable between datasets, it is often better to use the total number of steps instead of the total number of epochs as a hyper-parameter. That is because the number of iterations should not relly on the dataset size, but on its complexity.

I am using the following code for training. It ensures that the data loader re-shuffles the data every time it is re-initiated.

# main training loop
    generator = iter(trainloader)
    for i in range(max_steps):

        try:
            # Samples the batch
            x, y = next(generator)
        except StopIteration:
            # restart the generator if the previous generator is exhausted.
            generator = iter(trainloader)
            x, y = next(generator)

I will agree that is not the most elegant solution, but it keeps me from having to rely on epochs for training.


Instruction: Example CrossEntropyLoss for 3D semantic segmentation in pytorch

Input:
I have a network performing 3D convolutions on a 5D input tensor. The output of my network if of size (1, 12, 60, 36, 60) corresponding to ( BatchSize, NumClasses, x-dim, y-dim, z-dim). I need to compute a voxel-wise cross entropy loss. However I keep on getting errors.

When trying to compute cross entropy loss using torch.nn.CrossEntropyLoss(), I keep on getting the following error message:

RuntimeError: multi-target not supported at .../src/THCUNN/generic/ClassNLLCriterion.cu:16

here is the extract of my code:

import torch
import torch.nn as nn
from torch.autograd import Variable
criterion = torch.nn.CrossEntropyLoss()
images = Variable(torch.randn(1, 12, 60, 36, 60)).cuda()
labels = Variable(torch.zeros(1, 12, 60, 36, 60).random_(2)).long().cuda()
loss = criterion(images.view(1,-1), labels.view(1,-1))

Same happens when I create a one-hot tensor for the labels:

nclasses = 12
labels = (np.random.randint(0,12,(1,60,36,60))) # Random labels with values between [0..11]
labels = (np.arange(nclasses) == labels[..., None] - 1).astype(int) # Converts labels to one_hot_tensor
a = np.transpose(labels,(0,4,3,2,1)) #  Reorder dimensions to match shape of "images" ([1, 12, 60, 36, 60])
b = Variable(torch.from_numpy(a)).cuda()
loss = criterion(images.view(1,-1), b.view(1,-1))

Any idea what I'm doing wrong?
Can someone provide an example of computing cross entropy on a 5D output tensor?

Output:
Just checked some implementation (fcn) for 2D semantic segmentation, and tried to adapt it to 3D semantic segmentation. No guarantee that this is correct, I'll have to double check...

 import torch
 import torch.nn.functional as F
 def cross_entropy3d(input, target, weight=None, size_average=True):
    # input: (n, c, h, w, z), target: (n, h, w, z)
    n, c, h, w , z = input.size()
    # log_p: (n, c, h, w, z)
    log_p = F.log_softmax(input, dim=1)
    # log_p: (n*h*w*z, c)
    log_p = log_p.permute(0, 4, 3, 2, 1).contiguous().view(-1, c) # make class dimension last dimension
    log_p = log_p[target.view(n, h, w, z, 1).repeat(1, 1, 1, 1, c) >= 0] # this looks wrong -> Should rather be a one-hot vector
    log_p = log_p.view(-1, c)
    # target: (n*h*w*z,)
    mask = target >= 0
    target = target[mask]
    loss = F.nll_loss(log_p, target.view(-1), weight=weight, size_average=False)
    if size_average:
        loss /= mask.data.sum()
    return loss
images = Variable(torch.randn(5, 3, 16, 16, 16))
labels = Variable(torch.LongTensor(5, 16, 16, 16).random_(3))
cross_entropy3d(images, labels, weight=None, size_average=True)


Instruction: How to install pytorch in windows?

Input:
I am trying to install pytorch on windows and there is one which is available for it but shows an error.

conda install -c peterjc123 pytorch=0.1.12

Output:
Warning: Unless you have a very specific reason not to, just follow the official installation instructions from https://pytorch.org. They are far more likely to be accurate and up-to-date.

Here is how to install the PyTorch package from the official channel, on Windows using Anaconda, as of the time of writing this comment (31/03/2020):

PyTorch without CUDA:

conda install pytorch torchvision cpuonly -c pytorch

PyTorch with CUDA 10.1:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch


Instruction: Python (Pytorch) Multiprocessing throwing errors: Connection reset by peer and File Not Found

Input:
I'm trying to get something working similarly to keras' "fit_generator" method. Basically, I have a (very) large data file of mini-batches and I want to have my CPU grab mini-batches and populate a queue parallel to my GPU taking mini-batches from the queue and training on them. By having the CPU work in parallel to the GPU (as opposed to having the CPU grab a batch and making the GPU wait for the CPU before it trains on that batch) I should be able to reduce my training time by about half. I have benchmarked how long it takes the CPU to grab a mini-batch, and it's taking a comparable amount of time to how long it takes my GPU to train on one mini-batch, so parallelizing the CPU and GPU should work alright. I haven't found a built-in method in pytorch to do this, if there is one, please let me know.

So I have tried to use the torch.multiprocessing module to do what I want, but I'm not able to complete the training as I always get some sort of error right before training is completed. The torch.multiprocessing module should be a wrapper with essentially all the same functionalities as the regular multiprocessing module except it allows pytorch tensors to be shared between processes. Basically, I have set up my code to have 2 functions, a loader function, and a trainer function like so:

    def data_gen(que,PATH,epochs,steps_per_epoch,batch_size=32):
        for epoch in range(epochs):
            for j in range(steps_per_epoch):
                with h5py.File(PATH,'r') as f:
                    X = f['X'][j*batch_size:(j+1)*batch_size]
                    Y = f['Y'][j*batch_size:(j+1)*batch_size]

                    X = autograd.Variable(torch.Tensor(X).resize_(batch_size,256,25).cpu())
                    Y = autograd.Variable(torch.Tensor(Y).cpu())

                    que.put((X,Y))
            que.put('stop')
        que.close()
        return

    def train_network(que,net,optimizer,epochs):
        print('Training for %s epochs...' %epochs)
        for epoch in range(epochs):
            while(True):
                data = que.get()
                if(data == 'stop'):
                    break
                net.zero_grad()
                net.hid = net.init_hid()
                inp,labels = data
                inp   = inp.cuda()
                labels = labels.cuda()
                out,hid = net(inp)
                loss = F.binary_cross_entropy(out,labels)
                loss.backward()
                optimizer.step()
            print('Epoch end reached')
        return

And then I run the two processes in parallel like so:

    if __name__ == '__main__':
        tmp.set_start_method('spawn')
        que = tmp.Queue(maxsize=10)
        loader = tmp.Process(target=data_gen, args=(que,PATH,epochs,steps), kwargs={'batch_size':batch_size})
        loader.start()
        trainer = tmp.Process(target=train_network, args=(que,net,optimizer,epochs,steps))
        trainer.start()
        loader.join()
        trainer.join()

I have the que put in a 'stop' value at the end of each epoch so I can break out of the loop in the trainer and go to the next epoch. This "poison pill" method appears to work because the code runs for multiple epochs, and the trainer does in fact print the end of epoch verification message. The code runs, and it does appear to speed up the training process (I've been trying to prototype this code on a small subset of the data so it's sometimes hard to tell how much speed up I'm getting), but at the end of training (and always at the end, no matter how many epochs I specify), I always get an error:

    Process Process-2:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
        self._target(*self._args, **self._kwargs)
      File "/media/digitalstorm/Storage/RNN_Prototype/Lazuli_rnnprototype.py", line 307, in train_network
        data = que.get()
      File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
        return _ForkingPickler.loads(res)
      File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
        fd = df.detach()
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
        with _resource_sharer.get_connection(self._id) as conn:
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
        c = Client(address, authkey=process.current_process().authkey)
      File "/usr/lib/python3.6/multiprocessing/connection.py", line 487, in Client
        c = SocketClient(address)
      File "/usr/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
        s.connect(address)
    FileNotFoundError: [Errno 2] No such file or directory

or, if I mess around a bit with various options, I sometimes get an error like this:

    Process Process-2:
    Traceback (most recent call last):
      File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
        self._target(*self._args, **self._kwargs)
      File "/media/digitalstorm/Storage/RNN_Prototype/Lazuli_rnnprototype.py", line 306, in train_network
        data = que.get()
      File "/usr/lib/python3.6/multiprocessing/queues.py", line 113, in get
        return _ForkingPickler.loads(res)
      File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/reductions.py", line 70, in rebuild_storage_fd
        fd = df.detach()
      File "/usr/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
        return reduction.recv_handle(conn)
      File "/usr/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
        return recvfds(s, 1)[0]
      File "/usr/lib/python3.6/multiprocessing/reduction.py", line 153, in recvfds
        msg, ancdata, flags, addr = sock.recvmsg(1, socket.CMSG_LEN(bytes_size))
    ConnectionResetError: [Errno 104] Connection reset by peer

I don't know where I'm going wrong. Admittedly, I am a novice at multiprocessing, so it's hard for me to debug what went wrong exactly. Any help would be appreciated, thanks!

Output: Since there hasn't been any movement on this question, I'll just post my own workaround to this problem. Basically the loader process was shutting down the que after it finished processing and queuing up the examples. It wasn't waiting for the trainer process to finish, so when the trainer process was going to get the next minibatch, it couldn't find it. I don't quite know why the loader process was prematurely shutting down the que, the documentation for que.close() says that this should only tell the que no more objects are being sent to the que, but it shouldn't actually shut down the que. Also, deleting que.close() did not solve the issue, so I don't think the error had to do with that command. What solved this issue for me was putting a time.sleep(2) command after the que.close() command. This forces the que to sleep for a couple of seconds after it finishes putting everything in the que, and it allows the program to complete and exit without error.


Instruction: pix2pixHD error with own dataset

Input:
I am trying to generate my own images using the pix2pixHD pre-trained model. Github repo found here

The images inside the dataset has to be in grayscale with no alpha channel. The images in the repo has a size of 16 bitPerSample and I have both images in size 8 and 16 bitsPerSample.

When I check both my images and the images in the repo using sips -g all. This is the outcome I get:

pixelWidth: 2048
pixelHeight: 1024
typeIdentifier: public.png
format: png
formatOptions: default
dpiWidth: 72.000
dpiHeight: 72.000
samplesPerPixel: 1
bitsPerSample: 16
hasAlpha: no
space: Gray

The strange thing is that it works with the images that has 8 bitPerSample.
This is the outcome I get:

Grayscale input

Converted label map

Final output

When I run test.py with 16 bitsPerSample images, it doesn't work.
This is the error it gives me:

model [Pix2PixHDModel] was created
Traceback (most recent call last):
  File "test.py", line 26, in <module>
    for i, data in enumerate(dataset):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 210, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 230, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 42, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 42, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/paperspace/Documents/pix2pixHD/data/aligned_dataset.py", line 41, in __getitem__
    label_tensor = transform_label(label) * 255.0
  File "/usr/local/lib/python3.5/dist-packages/torch/tensor.py", line 309, in __mul__
    return self.mul(other)
TypeError: mul received an invalid combination of arguments - got (float), but expected one of:
 * (int value)
      didn't match because some of the arguments have invalid types: (float)
 * (torch.IntTensor other)
      didn't match because some of the arguments have invalid types: (float)

I am new fairly to Tensorflow and I have never used pytorch before.

Any idea what this error mean and how can I resolve it?

Output:
Yes, I think I can help you.
I haven't checked the repository, but from the error trace the problem appears to be following:

You are performing a multiplication operation betweenn the output of transform_label(label) (presumably a tensor) and a scalar 255.0. This is fine as long as both your scalar and your tensor are of the same datatype. From the error trace however, it looks as if the output of transform_label() is of data type Int / Long, while 255.0 is a float.

I suggest you try 255 or int(255.0) instead of 255.0.

If this does not resolve your problem, let me know what data type the output of transform_label() is.


Instruction: PyTorch 2d Convolution with sparse filters

Input:
I am trying to perform a spatial convolution (e.g. on an image) in pytorch on dense input using a sparse filter matrix.

Sparse Tensors are implemented in PyTorch. I tried to use a sparse Tensor, but it ends up with a segmentation fault.

import torch
from torch.autograd import Variable
from torch.nn import functional as F

# build sparse filter matrix
i = torch.LongTensor([[0, 1, 1],[2, 0, 2]])
v = torch.FloatTensor([3, 4, 5])
filter =  Variable(torch.sparse.FloatTensor(i, v, torch.Size([3,3])))

inputs = Variable(torch.randn(1,1,6,6))

F.conv2d(inputs, filter)

Can anyone just give me a hint how to do that?

Thanks in advance!
dymat

Output:
I know this question is outdated but I also know that there are still people looking for an answer (like myself) so here goes...

On sparse filters

If you'd like sparse convolution without the freedom to specify the sparsity pattern yourself, take a look at dilated conv (also called atrous conv). This is implemented in PyTorch and you can control the degree of sparsity by adjusting the dilation param in Conv2d.

If you'd like to specify the sparsity pattern yourself, to the best of my knowledge, this feature is not currently available in PyTorch. But you may want to check this out if you are ok with using Tensorflow. There is also a blog post providing more details on this repo.

On sparse input

A list of existing and TODO sparse tensor operations is available here.

This talks about the current state of sparse tensors in PyTorch.

This lets you propose your own sparse tensor use case to the PyTorch contributors.

But at the time of this writing, I did not see conv on sparse tensors being an implemented feature or on the TODO list. nn.Linear on sparse input, however, is supported.

And if you build a sparse tensor and apply a conv layer to it, PyTorch (1.1.0) throws an exception:

>>> a = torch.zeros((1, 3, 2, 2), layout=torch.sparse_coo)
>>> net = torch.nn.Conv2d(1, 1, 1)
>>> b = net(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: sparse tensors do not have is_contiguous
>>> torch.__version__
'1.1.0'

Changing to a linear layer and it would work:

>>> c = torch.zeros((1, 2), layout=torch.sparse_coo)
>>> another_net = torch.nn.Linear(2, 1)
>>> d = another_net(c)
>>> d
tensor([[0.1944]], grad_fn=<AddmmBackward>)
>>> d.backward()
>>> another_net.weight.grad
tensor([[0., 0.]])
>>> another_net.bias.grad
tensor([1.])


Instruction: PyTorch DataLoader

Input:
I'm trying to use multiple torch.utils.data.DataLoaders to create datasets that have different transforms applied to them. Currently, my code is roughly

d_transforms = [
    transforms.RandomHorizontalFlip(),
    # Some other transforms...
]
loaders = []
for i in range(len(d_transforms)):
    dataset = datasets.MNIST('./data',
            train=train,
            download=True,
            transform=d_transforms[i]
    loaders.append(
        DataLoader(dataset,
            shuffle=True,
            pin_memory=True,
            num_workers=1)
        )

This works, but it's extremely slow. kernprof shows that nearly all of the time in my code is spent on lines like

x, y = next(iter(train_loaders[i]))

I suspect that this is due to the fact that i'm using multiple instances of DataLoader, each with their own worker, which tries to read the same data files.

My question is, what is a better way to do this? Ideally, I would subclass torch.utils.data.DataSet and specify the transform i'd like to apply when sampling, but this doesn't seem possible due to __getitem__ not being able to take arguments.

Output:
__getitem__ does take an argument which is the index of the content you want to load . For eg.

transform = transforms.Compose(
    [transforms.ToTensor(),
     normalize])

class CountDataset(Dataset):

def __init__(self, file,transform=None):

    self.transform = transform
    #self.vocab = vocab
    with open(file,'rb') as f:
        self.data = pickle.load(f)
    self.y = self.data['answers']
    self.I = self.data['images']

def __len__(self):
    return len(self.y)

def __getitem__(self, idx):
    img_name = self.I[idx]
    label = self.y[Idx]
    fname = '/'.join(img_name.split("/")[-2:]) #/train2014/xx.jpg
    DIR = '/hdd/manoj/VQA/Images/mscoco/'
    img_full_path = os.path.join(DIR,fname)
    img = Image.open(img_full_path).convert("RGB")
    img_tensor = self.transform(img.resize((224,224)))
    return img_tensor,label

testset = CountDataset(file = 'testdat.pkl',
                        transform = transform)

testloader = DataLoader(testset, batch_size=32,
                         shuffle=False, num_workers=4)

You dont call the data loader in loop.


Instruction: How can I use LSTM in pytorch for classification?

Input:
My code is as below:

class Mymodel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers, batch_size):
        super(Discriminator, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.num_layers = num_layers
        self.batch_size = batch_size

        self.lstm = nn.LSTM(input_size, hidden_size)
        self.proj = nn.Linear(hidden_size, output_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (Variable(torch.zeros(self.num_layers, self.batch_size, self.hidden_size)),
                Variable(torch.zeros(self.num_layers, self.batch_size, self.hidden_size)))

    def forward(self, x):
        lstm_out, self.hidden = self.lstm(x, self.hidden)
        output = self.proj(lstm_out)
        result = F.sigmoid(output)
        return result

I want to use LSTM to classify a sentence to good (1) or bad (0). Using this code, I get the result which is time_step * batch_size * 1 but not 0 or 1. How to edit the code in order to get the classification result?

Output:
Theory:

Recall that an LSTM outputs a vector for every input in the series. You are using sentences, which are a series of words (probably converted to indices and then embedded as vectors). This code from the LSTM PyTorch tutorial makes clear exactly what I mean (***emphasis mine):

lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [autograd.Variable(torch.randn((1, 3)))
          for _ in range(5)]  # make a sequence of length 5

# initialize the hidden state.
hidden = (autograd.Variable(torch.randn(1, 1, 3)),
          autograd.Variable(torch.randn((1, 1, 3))))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)

# alternatively, we can do the entire sequence all at once.
# the first value returned by LSTM is all of the hidden states throughout
# the sequence. the second is just the most recent hidden state
# *** (compare the last slice of "out" with "hidden" below, they are the same)
# The reason for this is that:
# "out" will give you access to all hidden states in the sequence
# "hidden" will allow you to continue the sequence and backpropagate,
# by passing it as an argument  to the lstm at a later time
# Add the extra 2nd dimension
inputs = torch.cat(inputs).view(len(inputs), 1, -1)
hidden = (autograd.Variable(torch.randn(1, 1, 3)), autograd.Variable(
torch.randn((1, 1, 3))))  # clean out hidden state
out, hidden = lstm(inputs, hidden)
print(out)
print(hidden)

One more time: compare the last slice of "out" with "hidden" below, they are the same. Why? Well...

If you're familiar with LSTM's, I'd recommend the PyTorch LSTM docs at this point. Under the output section, notice h_t is output at every t.

Now if you aren't used to LSTM-style equations, take a look at Chris Olah's LSTM blog post. Scroll down to the diagram of the unrolled network:

As you feed your sentence in word-by-word (x_i-by-x_i+1), you get an output from each timestep. You want to interpret the entire sentence to classify it. So you must wait until the LSTM has seen all the words. That is, you need to take h_t where t is the number of words in your sentence.

Code:

Here's a coding reference. I'm not going to copy-paste the entire thing, just the relevant parts. The magic happens at self.hidden2label(lstm_out[-1])

class LSTMClassifier(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, label_size, batch_size):
        ...
        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim)
        self.hidden2label = nn.Linear(hidden_dim, label_size)
        self.hidden = self.init_hidden()

    def init_hidden(self):
        return (autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim)),
                autograd.Variable(torch.zeros(1, self.batch_size, self.hidden_dim)))

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        x = embeds.view(len(sentence), self.batch_size , -1)
        lstm_out, self.hidden = self.lstm(x, self.hidden)
        y  = self.hidden2label(lstm_out[-1])
        log_probs = F.log_softmax(y)
        return log_probs


Instruction: Using PyTorch for scientific computation

Input:
I would like to use PyTorch as a scientific computation package. It has much to recommend it in that respect - its Tensors are basically GPU-accelerated numpy arrays, and its autograd mechanism is potentially useful for a lot of things besides neural networks.

However, the available tutorials and documentation seem strongly geared towards quickly getting people up and running using it for machine learning. Although there is lots of good information available on the Tensor and Variable classes (and I understand that material reasonably well), the nn and optim packages always seem to be introduced by example rather than by explaining the API, which makes it hard to figure out exactly what's going on.

My main question at this point is whether I can use the optim package without also using the nn package, and if so how to do so. Of course I can always implement my simulations as subclasses of nn.Module even though they are not neural networks, but I would like to understand what happens under the hood when I do this, and what benefits/drawbacks it would give for my particular application.

More broadly, I would appreciate pointers to any resource that gives more of a logical overview of the API (for nn and optim specifically), rather than just presenting examples.

Output:
This is a partial self-answer to the specific question about using optim without using nn. The answer is, yes, you can do that. In fact, from looking at the source code, the optim package doesn't know anything about nn and only cares about Variables and tensors.

The documentation gives the following incomplete example:

optimizer = optim.Adam([var1, var2], lr = 0.0001)

and then later:

for input, target in dataset:
    optimizer.zero_grad()
    output = model(input)
    loss = loss_fn(output, target)
    loss.backward()
    optimizer.step()

The function model isn't defined anywhere and looks like it might be something to do with nn, but in fact it can just be a Python function that computes output from input using var1 and var2 as parameters, as long as all the intermediate steps are done using Variables so that it can be differentiated. The call to optimizer.step() will update the values of var1 and var2 automatically.

In terms of the structure of PyTorch overall, it seems that optim and nn are independent of one another, with nn being basically just a convenient way to chain differentiable functions together, along with a library of such functions that are useful in machine learning. I would still appreciate pointers to a good technical overview of the whole package, though.


Instruction: Pytorch installation problems with Anaconda

Input:
After having upgraded my environment's python to Python 3.61, I attempted to install pytorch using this command:

conda install -c peterjc123 pytorch

However I got this error:

Fetching package metadata .............
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

-pytorch
-pyqt

I also used the commands
conda install -c peterjc123 pytorch cuda90
conda install -c peterjc123 pytorch cuda80

But the result is still the same. Anyone got a clue how to solve this?

Output:
The problem was solved after downgrading from Python 3.6.2 to Python 3.5.1 after running:

conda install -c anaconda python=3.5.1

After running this command, run:

conda install -c peterjc123 pytorch

Pytorch should install as per normal. A similar issue occurs for openCV as well


Instruction: Everytime I use cuda() to remove Variable from CPU to GPU in pytorch,it takes about 5 to 10 minitues

Input:
I just do this:

t = Variable(torch.randn(5))
t =t.cuda()
print(t)

but it takes 5 to 10 minitues,everytime.
I used cuda samples to test bandwidth, it's fine.
Then I used pdb to find which takes the most time.
I find in /anaconda3/lib/python3.6/site-packages/torch/cuda/__init__:

def _lazy_new(cls, *args, **kwargs):
    _lazy_init()
    # We need this method only for lazy init, so we can remove it
    del _CudaBase.__new__
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)

it takes about 5 minitues in the return
I don't know how to solve my problem by these imformation.
My environment is: Ubuntu 16.04 + CUDA 9.1

Output:
There’s a cuda version mismatch between the cuda my pytorch was compiled with the cuda I'm running.I divided the official installation commond

conda install pytorch torchvision cuda90 -c pytorch

into two section：

conda install -c soumith magma-cuda90
conda install pytorch torchvision -c soumith

The second commond installed pytorch-0.2.0 by default,which mathchs CUDA8.0. After I update my pytorch to 0.3.0,this commond only takes one second.


Instruction: Dynamic addition of hidden units in pytorch

Input:
I am trying to add hidden units to a 3-layered neural network(input, hidden,output) dynamically as I train it. I want to keep the weights of trained part of the network as I add new hidden units.This is my code,

class my_network(torch.nn.Module):
    def __init__(self,input_dim,hidden_dim,output_dim):
        super(my_network,self).__init__()
        self.I = input_dim
        self.H = hidden_dim
        self.O = output_dim
        self.layer1 = torch.nn.Linear(input_dim,hidden_dim)
        self.layer2 = torch.nn.Linear(hidden_dim,output_dim)

    def add_neurons(self,no_of_neurons,flag):
        if flag == 1:
            weights = [self.layer1.weight.data,self.layer2.weight.data]
            self.layer1 = torch.nn.Linear(self.I,self.H+no_of_neurons)
            self.layer2 = torch.nn.Linear(self.H+no_of_neurons,self.O)
            self.layer1.weight.data[0:-no_of_neurons,:] = weights[0]
            self.layer2.weight.data[:,0:-no_of_neurons] = weights[1]
            self.H = self.H + no_of_neurons
        return self.layer1.weight.shape[0]

    def forward(self,x):
        temp = self.layer1(x)
        out = self.layer2(temp)
        return out

I have noticed that once I call “add_neurons” method, the weights stop updating(while gradients are generated). Any help would be much appreciated.

Output: The optimizer might not be informed about new parameters that you added to your model. The easiest probably would be to recreate optimizer object with the updated list of your model parameters.