Instruction: How can I create a PyCUDA GPUArray from a gpu memory address?

Input: I'm working with PyTorch and want to do some arithmetic on Tensor data with the help of PyCUDA. I can get a memory address of a cuda tensor t via t.data_ptr(). Can I somehow use this address and my knowledge of the size and data type to initialize a GPUArray? I am hoping to avoid copying the data, but that would also be an alternative.

Output:
It turns out this is possible.
We need a pointer do the data, which needs some additional capabilities:

class Holder(PointerHolderBase):

    def __init__(self, tensor):
        super().__init__()
        self.tensor = tensor
        self.gpudata = tensor.data_ptr()

    def get_pointer(self):
        return self.tensor.data_ptr()

    def __int__(self):
        return self.__index__()

    # without an __index__ method, arithmetic calls to the GPUArray backed by this pointer fail
    # not sure why, this needs to return some integer, apparently
    def __index__(self):
        return self.gpudata

We can then use this class to instantiate GPUArrays. The code uses Reikna arrays which are a subclass but should work with pycuda arrays as well.

def tensor_to_gpuarray(tensor, context=pycuda.autoinit.context):
    '''Convert a :class:`torch.Tensor` to a :class:`pycuda.gpuarray.GPUArray`. The underlying
    storage will be shared, so that modifications to the array will reflect in the tensor object.
    Parameters
    ----------
    tensor  :   torch.Tensor
    Returns
    -------
    pycuda.gpuarray.GPUArray
    Raises
    ------
    ValueError
        If the ``tensor`` does not live on the gpu
    '''
    if not tensor.is_cuda:
        raise ValueError('Cannot convert CPU tensor to GPUArray (call `cuda()` on it)')
    else:
        thread = cuda.cuda_api().Thread(context)
    return reikna.cluda.cuda.Array(thread, tensor.shape, dtype=torch_dtype_to_numpy(tensor.dtype), base_data=Holder(tensor))

We can go back with this code. I have not found a way to do this without copying the data.

def gpuarray_to_tensor(gpuarray, context=pycuda.autoinit.context):
    '''Convert a :class:`pycuda.gpuarray.GPUArray` to a :class:`torch.Tensor`. The underlying
    storage will NOT be shared, since a new copy must be allocated.
    Parameters
    ----------
    gpuarray  :   pycuda.gpuarray.GPUArray
    Returns
    -------
    torch.Tensor
    '''
    shape = gpuarray.shape
    dtype = gpuarray.dtype
    out_dtype = numpy_dtype_to_torch(dtype)
    out = torch.zeros(shape, dtype=out_dtype).cuda()
    gpuarray_copy = tensor_to_gpuarray(out, context=context)
    byte_size = gpuarray.itemsize * gpuarray.size
    pycuda.driver.memcpy_dtod(gpuarray_copy.gpudata, gpuarray.gpudata, byte_size)
    return out

Old answer

from pycuda.gpuarray import GPUArray

def torch_dtype_to_numpy(dtype):
    dtype_name = str(dtype)[6:]     # remove 'torch.'
    return getattr(np, dtype_name)

def tensor_to_gpuarray(tensor):
    if not tensor.is_cuda:
        raise ValueError('Cannot convert CPU tensor to GPUArray (call `cuda()` on it)')
    else:
        array = GPUArray(tensor.shape, dtype=torch_dtype_to_numpy(tensor.dtype),
                         gpudata=tensor.data_ptr())
        return array.copy()

Unfortunately, passing an int as the gpudata keyword (or a subtype of pycuda.driver.PointerHolderBase as was suggested in the pytorch forum) seems to work on the surface, but many operations fail with seemingly unrelated errors. Copying the array seems to transform it into a useable format though.
I think it is related to the fact that the gpudata member should be a pycuda.driver.DeviceAllocation object which cannot be instantiated from Python it seems.

Now how to go back from the raw data to a Tensor is another matter.


Instruction: How to iterate over two dataloaders simultaneously using pytorch?

Input:
I am trying to implement a Siamese network that takes in two images. I load these images and create two separate dataloaders.

In my loop I want to go through both dataloaders simultaneously so that I can train the network on both images.

for i, data in enumerate(zip(dataloaders1, dataloaders2)):

    # get the inputs
    inputs1 = data[0][0].cuda(async=True);
    labels1 = data[0][1].cuda(async=True);

    inputs2 = data[1][0].cuda(async=True);
    labels2 = data[1][1].cuda(async=True);

    labels1 = labels1.view(batchSize,1)
    labels2 = labels2.view(batchSize,1)

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs1 = alexnet(inputs1)
    outputs2 = alexnet(inputs2)

The return value of the dataloader is a tuple.
However, when I try to use zip to iterate over them, I get the following error:

OSError: [Errno 24] Too many open files
Exception NameError: "global name 'FileNotFoundError' is not defined" in <bound method _DataLoaderIter.__del__ of <torch.utils.data.dataloader._DataLoaderIter object at 0x7f2d3c00c190>> ignored

Shouldn't zip work on all iterable items? But it seems like here I can't use it on dataloaders.

Is there any other way to pursue this? Or am I approaching the implementation of a Siamese network incorrectly?

Output:
I see you are struggling to make a right dataloder function. I would do:
class Siamese(Dataset):

    def __init__(self, transform=None):

       #init data here

    def __len__(self):
        return   #length of the data

    def __getitem__(self, idx):
        #get images and labels here
        #returned images must be tensor
        #labels should be int
        return img1, img2 , label1, label2


Instruction: In Pytorch F.nll_loss() Expected object of type torch.LongTensor but found type torch.FloatTensor for argument #2 'target'

Input:
Why does this error occur.

I am trying to write a custom loss function, that finally has a negative log likelihood.

As per my understanding the NLL is calculated between two probability values?

>>> loss = F.nll_loss(sigm, trg_, ignore_index=250, weight=None, size_average=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home//lib/python3.5/site-packages/torch/nn/functional.py", line 1332, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: Expected object of type torch.LongTensor but found type torch.FloatTensor for argument #2 'target'

The inputs here being the following:

>>> sigm.size()
torch.Size([151414, 80])
>>> sigm
tensor([[ 0.3283,  0.6472,  0.8278,  ...,  0.6756,  0.2168,  0.5659],
        [ 0.6603,  0.5957,  0.8375,  ...,  0.2274,  0.4523,  0.4665],
        [ 0.5262,  0.4223,  0.5009,  ...,  0.5734,  0.3151,  0.2076],
        ...,
        [ 0.4083,  0.2479,  0.5996,  ...,  0.8355,  0.6681,  0.7900],
        [ 0.6373,  0.3771,  0.6568,  ...,  0.4356,  0.8143,  0.4704],
        [ 0.5888,  0.4365,  0.8587,  ...,  0.2233,  0.8264,  0.5411]])

And my target tensor is:

>>> trg_.size()
torch.Size([151414])
>>> trg_
tensor([-7.4693e-01,  3.5152e+00,  2.9679e-02,  ...,  1.6316e-01,
         3.6594e+00,  1.3366e-01])

If I convert this to long I loose all data:

>>> sigm.long()
tensor([[ 0,  0,  0,  ...,  0,  0,  0],
        [ 0,  0,  0,  ...,  0,  0,  0],
        [ 0,  0,  0,  ...,  0,  0,  0],
        ...,
        [ 0,  0,  0,  ...,  0,  0,  0],
        [ 0,  0,  0,  ...,  0,  0,  0],
        [ 0,  0,  0,  ...,  0,  0,  0]])
>>> trg_.long()
tensor([ 0,  3,  0,  ...,  0,  3,  0])

If I convert the raw values of target tensor to sigmoid too:

>>> F.sigmoid(trg_)
tensor([ 0.3215,  0.9711,  0.5074,  ...,  0.5407,  0.9749,  0.5334])
>>> loss = F.nll_loss(sigm, F.sigmoid(trg_), ignore_index=250, weight=None, size_average=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/lib/python3.5/site-packages/torch/nn/functional.py", line 1332, in nll_loss
    return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
RuntimeError: Expected object of type torch.LongTensor but found type torch.FloatTensor for argument #2 'target'

This does calculate the loss happily, but it is again just make belief as I have lost data in the long conversion:

>>> loss = F.nll_loss(sigm, F.sigmoid(trg_).long(), ignore_index=250, weight=None, size_average=True)
>>> loss
tensor(-0.5010)

>>> F.sigmoid(trg_).long()
tensor([ 0,  0,  0,  ...,  0,  0,  0])

Output:
"As per my understanding, the NLL is calculated between two probability values?"

No, NLL is not calculated between two probability values. As per the pytorch docs
(See shape section), It is usually used to implement cross entropy loss. It takes input which is expected to be log-probability and is of size (N, C) when N is data size and C is the number of classes. Target is a long tensor of size (N,) which tells the true class of the sample.

Since in your case, target for sure is not the true class, you might have to implement your own version of loss and you may not be able to use NLLLoss. If you add more details about what loss you want to code up, I can help/explain more on how to do that (if possible by using existing function in torch).


Instruction: How do I get words from an embedded vector?

Input:
How can I convert them into their original words when I generate word vectors in the generator?
I used the nn.Embedding module built into pytorch to embed words.

Output:
Since you didn't provide any code, I am using below code with comments to answers your query. Feel free to add more information for your particular use case.

import torch
# declare embeddings
embed = torch.nn.Embedding(5,10)

# generate embedding for word [4] in vocab
word = torch.tensor([4])

# search function for searching through embedding
def search(vector, distance_fun):
    weights = embed.weight
    min = torch.tensor(float('inf'))
    idx = -1
    v, e = weights.shape

    # each vector in embeding is corresponding to one of the word.
    # use a distance function to compare with vector
    for i in range(v):
        dist = distance_fun(vector, weights[i])
        if (min<dist):
            min = dist
            idx = i
    return i
# searching with squared distance
search(word, lambda x,y: ((x-y)**2).sum()


Instruction: Pytorch nn.embedding error

Input:
I was reading pytorch documentation on Word Embedding.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(5)

word_to_ix = {"hello": 0, "world": 1, "how":2, "are":3, "you":4}
embeds = nn.Embedding(2, 5)  # 2 words in vocab, 5 dimensional embeddings
lookup_tensor = torch.tensor(word_to_ix["hello"], dtype=torch.long)
hello_embed = embeds(lookup_tensor)
print(hello_embed)

Output:

tensor([-0.4868, -0.6038, -0.5581,  0.6675, -0.1974])

This looks good but if I replace  line lookup_tensor by

lookup_tensor = torch.tensor(word_to_ix["how"], dtype=torch.long)

I am getting the error as:

RuntimeError: index out of range at /Users/soumith/minicondabuild3/conda-bld/pytorch_1524590658547/work/aten/src/TH/generic/THTensorMath.c:343

I don't understand why it gives RunTime error on line hello_embed = embeds(lookup_tensor).

Output:
When you declare embeds = nn.Embedding(2, 5) the vocab size is 2 and embedding size is 5. i.e each word will be represented by a vector of size 5 and there are only 2 words in vocab.

lookup_tensor = torch.tensor(word_to_ix["how"], dtype=torch.long) embeds will try to look up vector corresponding to the third word in vocab, but embedding has vocab size of 2. and that is why you get the error.

If you declare embeds = nn.Embedding(5, 5) it should work fine.


Instruction: PyTorch Tensors of Inputs and Labels in LSTM

Input:
I am new to PyTorch, and I'm working on a simple project to generate text, in order to get my hands on pytorch. I am using the concept of this code and converting it to PyTorch: https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/
I have 10 timesteps and 990 samples. For each of these 990 samples, there are 10 values, corresponding to the indexes (scaled) of the sequence. My output samples are the rest of the letters (not including the first letter). For example if my sample is 'Hello Worl" my corresponding output is "ello World". My input size (features) is 1 since I want to feed in one letter at a time. Therefore, my final input shape is (990, 10, 1). I then convert the output tensor into one hot vector, so the final shape is (9900, 42), where 42 are the number of elements in the one hot vector. When I run the output, I can get a shape of (9900, 42), so this is the output of all my timesteps, and each one of them contains the corresponding one-hot vector. But when I calculate the loss, then there is an error:

  multi-target not supported

Can I understand what I did wrong? Thanks . Below is my code

#The file contains 163780 characters
#The file contains 1000 characters
#There are 42 unique characters

char2int = {char:value for (value,char) in enumerate (unique)}
int2char = {value:char for (value,char) in enumerate (unique)}

learning_rate = 0.01
num_epochs = 5
input_size = 1              #The number of input neurons (features) to our RNN
units = 100
num_layers = 2
num_classes = len(unique)   #The number of output neurons

timesteps = 10
datax = []
datay = []
for index in range(0, len(file) - timesteps, 1):
    prev_letters = file[index:index + timesteps]
    output = file[index + 1: index + timesteps + 1]
    #Convert the 10 previous characters to their integers and put in a list. Append that list to the dataset
    datax.append([char2int[c] for c in prev_letters])
    datay.append([char2int[c] for c in output])
print('There are {} Sequences in the dataset'.format(len(datax)))
#There are 990 Sequences in the dataset

x = np.array(datax)
x = x / float(len(unique))
x = torch.FloatTensor(x)
x = x.view(x.size(0), timesteps, input_size)
print(x.shape)   #torch.Size([990, 10, 1])

y = torch.LongTensor(datay)
print(y.shape)   #torch.Size([990, 10])
y_one_hot = torch.zeros(y.shape[0] * y.shape[1], num_classes)
index = y.long()
index = index.view(-1,1)          #The expected shape for the scatter function
y_one_hot.scatter_(1,index,1)    #(dim (1 for along rows and 0 for along cols), index, number to insert)
y_one_hot = y_one_hot.view(-1, num_classes)    # Make the tensor of shape(rows, cols)
y_one_hot = y_one_hot.long()
print(y_one_hot.shape)
#torch.Size([9900, 42])

inputs = Variable(x)
labels = Variable(y_one_hot)

class TextGenerator(nn.Module):
    def __init__(self,input_size,units,num_layers,num_classes,timesteps):
        super(TextGenerator,self).__init__()
        self.units = units
        self.num_layers = num_layers
        self.timesteps = timesteps
        self.input_size = input_size
        # When batch_first=true, inputs are of shape (batch_size/samples, sequence_length, input_dimension)
        self.lstm = nn.LSTM(input_size = input_size, hidden_size = units, num_layers = num_layers, batch_first = True)
        #The output layer
        self.fc = nn.Linear(units, num_classes)
    def forward(self,x):
        #Initialize the hidden state
        h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.units))
        #Initialize the cell state
        c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.units))
        out,_ = self.lstm(x, (h0,c0))
        #Reshape the outout from (samples,timesteps,output_features) to a shape appropriate for the FC layer
        out = out.contiguous().view(-1, self.units)
        out = self.fc(out)
        return out

net = TextGenerator(input_size,units,num_layers,num_classes,timesteps)
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
out = net(inputs)
out.shape   #(9900, 42)
loss_fn(out,labels)

Output: In PyTorch, when using the CrossEntropyLoss, you need to give the output labels as integers in [0..n_classes-1] instead of as one-hot vectors. Right now pytorch thinks you are trying to predict multiple outputs.


Instruction: Highlighting important words in a sentence using Deep Learning

Input:
I am trying to highlight important words in imdb dataset which contributed finally to the sentiment analysis prediction .

The dataset is like :

X_train - A review as string .

Y_train - 0 or 1

Now after using Glove embeddings for embedding the X_train value I can feed it to a neural net .

Now my question is , how can I highlight the most important words probability wise ? just like deepmoji.mit.edu ?

What have I tried :

I tried splitting the input sentences into bi-grams and using a 1D CNN to train it . Later when we want to find the important words of X_test , we just split the X_test in bigrams and find their probabilities . It works but not accurate .
I tried using prebuilt Hierarchical Attention Networks and succeeded . I got what I wanted but I couldn't figure out every line and concepts from the code .It's like a black box to me .

I know how a neural net works and I can code it using numpy with manual back propagation from scratch . I have detailed knowledge of how a lstm works and what forget , update , and output gates actually outputs . But I couldn't still figure out how to extract attention weights and how to make the data as a 3D array ( what is the timestep in our 2D data ? )

So , any type of guidance is welcome

Output:
Here is a version with Attention (not Hierarchical) but you should be able to figure out how to make it work with hierarchy too - if not I can help out too. The trick is to define 2 models and use 1 for the training (model) and the other one to extract attention values (model_with_attention_output):

# Tensorflow 1.9; Keras 2.2.0 (latest versions)
# should be backwards compatible upto Keras 2.0.9 and tf 1.5
from keras.models import Model
from keras.layers import *
import numpy as np

dictionary_size=1000

def create_models():
  #Get a sequence of indexes of words as input:
  # Keras supports dynamic input lengths if you provide (None,) as the
  #  input shape
  inp = Input((None,))
  #Embed words into vectors of size 10 each:
  # Output shape is (None,10)
  embs = Embedding(dictionary_size, 10)(inp)
  # Run LSTM on these vectors and return output on each timestep
  # Output shape is (None,5)
  lstm = LSTM(5, return_sequences=True)(embs)
  ##Attention Block
  #Transform each timestep into 1 value (attention_value)
  # Output shape is (None,1)
  attention = TimeDistributed(Dense(1))(lstm)
  #By running softmax on axis 1 we force attention_values
  # to sum up to 1. We are effectively assigning a "weight" to each timestep
  # Output shape is still (None,1) but each value changes
  attention_vals = Softmax(axis=1)(attention)
  # Multiply the encoded timestep by the respective weight
  # I.e. we are scaling each timestep based on its weight
  # Output shape is (None,5): (None,5)*(None,1)=(None,5)
  scaled_vecs = Multiply()([lstm,attention_vals])
  # Sum up all scaled timesteps into 1 vector
  # i.e. obtain a weighted sum of timesteps
  # Output shape is (5,) : Observe the time dimension got collapsed
  context_vector = Lambda(lambda x: K.sum(x,axis=1))(scaled_vecs)
  ##Attention Block over
  # Get the output out
  out = Dense(1,activation='sigmoid')(context_vector)

  model = Model(inp, out)
  model_with_attention_output = Model(inp, [out, attention_vals])
  model.compile(optimizer='adam',loss='binary_crossentropy')
  return model, model_with_attention_output

model,model_with_attention_output = create_models()

model.fit(np.array([[1,2,3]]),[1],batch_size=1)
print ('Attention Over each word: ',model_with_attention_output.predict(np.array([[1,2,3]]),batch_size=1)[1])

The output will be the numpy array with attention value of each word - the higher the value the more important the word was

EDIT: You might want to replace lstm in multiplication with embs to get better interpretations but it will lead to worse performance...


Instruction: Initialising weights and bias with PyTorch - how to correct dimensions?

Input:
Using this model I'm attempting to initialise my network with my predefined weights and bias :

dimensions_input = 10
hidden_layer_nodes = 5
output_dimension = 10

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(dimensions_input,hidden_layer_nodes)
        self.linear2 = torch.nn.Linear(hidden_layer_nodes,output_dimension)

        self.linear.weight = torch.nn.Parameter(torch.zeros(dimensions_input,hidden_layer_nodes))
        self.linear.bias = torch.nn.Parameter(torch.ones(hidden_layer_nodes))

        self.linear2.weight = torch.nn.Parameter(torch.zeros(dimensions_input,hidden_layer_nodes))
        self.linear2.bias = torch.nn.Parameter(torch.ones(hidden_layer_nodes))

    def forward(self, x):
        l_out1 = self.linear(x)
        y_pred = self.linear2(l_out1)
        return y_pred

model = Model()

criterion = torch.nn.MSELoss(size_average = False)
optim = torch.optim.SGD(model.parameters(), lr = 0.00001)

def train_model():
    y_data = x_data.clone()
    for i in range(10000):
        y_pred = model(x_data)
        loss = criterion(y_pred, y_data)

        if i % 5000 == 0:
            print(loss)
        optim.zero_grad()

        loss.backward()
        optim.step()

RuntimeError:

  The expanded size of the tensor (10) must match the existing size (5)
  at non-singleton dimension 1

My dimensions appear correct as they match the corresponding linear layers ?

Output:
The code provided doesn't run due to the fact that x_data isn't defined, so I can't be sure that this is the issue, but one thing that strikes me is that you should replace

self.linear2.weight = torch.nn.Parameter(torch.zeros(dimensions_input,hidden_layer_nodes))
self.linear2.bias = torch.nn.Parameter(torch.ones(hidden_layer_nodes))

with

self.linear2.weight = torch.nn.Parameter(torch.zeros(hidden_layer_nodes, output_dimension))
self.linear2.bias = torch.nn.Parameter(torch.ones(output_dimension))


Instruction: Pytorch Torch.save FileNotFoundError

Input:
When I try to call "torch.save" to save my model in a "tmp_file", it rises a FileNotFoundError. the trace back is as follow:

  Traceback (most recent call last):
  File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 234, in
  test_hdr_continuous()
  File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 195, in test_hdr_continuous
  model = fit_mdn(X[:split], y[:split], nepochs=20)
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\simulations\continuous.py”, line 192, in fit_mdn
  torch.save(model, tmp_file)
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 161, in save
  return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 116, in _with_file_like
  f = open(f, mode)
  FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp_file_4358f298-a1d9-4c81-9e44-db4d8f1b4319’

It is weird that everything works perfectly on my mac, but I got this error on my Windows desktop.

Output:
As shmee observed, you are trying to write to /tmp/[...] on a Windows machine. Therefore you get FileNotFoundError.
To make your code OS agnostic, you may find python's tempfile package useful, especially NamedTemporaryFile: this function creates a temporary file and returns its name, so you can access/use it in your program.


Instruction: How to extract fc7 features from AlexNet in pytorch as numpy array?

Input:
I want to extract the 4096-dimensional feature vector from the fc7 layer of my finetuned AlexNet. My goal is to use this layer for clustering later on.
This is how I extract it:

alexnet = models.alexnet(pretrained=True);
fc7 = alexnet.classifier[6];

However, when I print it, fc7 is a Linear object:

Linear(in_features=4096, out_features=1000, bias=True)

What I am looking for is how to turn this Linear object into a numpy array so that I can do further manipulations on it.
What I am thinking of is to call its method'def forward(self, input)', but am not sure which input to provide? Should I provide the input image or should I provide the output the fc6 layer?

And I want the 4096-dim input array and get rid of the 1000 output array (presumably, since I don't think it will help me for clustering).

Output:
This could be done by creating a new model with all the same layers (and associated parameters) as alexnet except for the last layer.

new_model = models.alexnet(pretrained=True)
new_classifier = nn.Sequential(*list(new_model.classifier.children())[:-1])
new_model.classifier = new_classifier

You should now be able to provide the input image to new_model and extract a 4096-dimensional feature vector.

If you do need a particular layer as a numpy array for some reason, you could do the following: fc7.weight.data.numpy().

(on PyTorch 0.4.0)


Instruction: Calculate the accuracy every epoch in PyTorch

Input:
I am working on a Neural Network problem, to classify data as 1 or 0. I am using Binary cross entropy loss to do this. The loss is fine, however, the accuracy is very low and isn't improving. I am assuming I did a mistake in the accuracy calculation. After every epoch, I am calculating the correct predictions after thresholding the output, and dividing that number by the total number of the dataset. Is there any thing wrong I did in the accuracy calculation? And why isn't it improving, but getting more worse?
This is my code:

net = Model()
criterion = torch.nn.BCELoss(size_average=True)
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

num_epochs = 100
for epoch in range(num_epochs):
    for i, (inputs,labels) in enumerate (train_loader):
        inputs = Variable(inputs.float())
        labels = Variable(labels.float())
        output = net(inputs)
        optimizer.zero_grad()
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

    #Accuracy
    output = (output>0.5).float()
    correct = (output == labels).float().sum()
    print("Epoch {}/{}, Loss: {:.3f}, Accuracy: {:.3f}".format(epoch+1,num_epochs, loss.data[0], correct/x.shape[0]))

And this is the strange output I get:

Epoch 1/100, Loss: 0.389, Accuracy: 0.035
Epoch 2/100, Loss: 0.370, Accuracy: 0.036
Epoch 3/100, Loss: 0.514, Accuracy: 0.030
Epoch 4/100, Loss: 0.539, Accuracy: 0.030
Epoch 5/100, Loss: 0.583, Accuracy: 0.029
Epoch 6/100, Loss: 0.439, Accuracy: 0.031
Epoch 7/100, Loss: 0.429, Accuracy: 0.034
Epoch 8/100, Loss: 0.408, Accuracy: 0.035
Epoch 9/100, Loss: 0.316, Accuracy: 0.035
Epoch 10/100, Loss: 0.436, Accuracy: 0.035
Epoch 11/100, Loss: 0.365, Accuracy: 0.034
Epoch 12/100, Loss: 0.485, Accuracy: 0.031
Epoch 13/100, Loss: 0.392, Accuracy: 0.033
Epoch 14/100, Loss: 0.494, Accuracy: 0.030
Epoch 15/100, Loss: 0.369, Accuracy: 0.035
Epoch 16/100, Loss: 0.495, Accuracy: 0.029
Epoch 17/100, Loss: 0.415, Accuracy: 0.034
Epoch 18/100, Loss: 0.410, Accuracy: 0.035
Epoch 19/100, Loss: 0.282, Accuracy: 0.038
Epoch 20/100, Loss: 0.499, Accuracy: 0.031
Epoch 21/100, Loss: 0.446, Accuracy: 0.030
Epoch 22/100, Loss: 0.585, Accuracy: 0.026
Epoch 23/100, Loss: 0.419, Accuracy: 0.035
Epoch 24/100, Loss: 0.492, Accuracy: 0.031
Epoch 25/100, Loss: 0.537, Accuracy: 0.031
Epoch 26/100, Loss: 0.439, Accuracy: 0.033
Epoch 27/100, Loss: 0.421, Accuracy: 0.035
Epoch 28/100, Loss: 0.532, Accuracy: 0.034
Epoch 29/100, Loss: 0.234, Accuracy: 0.038
Epoch 30/100, Loss: 0.492, Accuracy: 0.027
Epoch 31/100, Loss: 0.407, Accuracy: 0.035
Epoch 32/100, Loss: 0.305, Accuracy: 0.038
Epoch 33/100, Loss: 0.663, Accuracy: 0.025
Epoch 34/100, Loss: 0.588, Accuracy: 0.031
Epoch 35/100, Loss: 0.329, Accuracy: 0.035
Epoch 36/100, Loss: 0.474, Accuracy: 0.033
Epoch 37/100, Loss: 0.535, Accuracy: 0.031
Epoch 38/100, Loss: 0.406, Accuracy: 0.033
Epoch 39/100, Loss: 0.513, Accuracy: 0.030
Epoch 40/100, Loss: 0.593, Accuracy: 0.030
Epoch 41/100, Loss: 0.265, Accuracy: 0.036
Epoch 42/100, Loss: 0.576, Accuracy: 0.031
Epoch 43/100, Loss: 0.565, Accuracy: 0.027
Epoch 44/100, Loss: 0.576, Accuracy: 0.030
Epoch 45/100, Loss: 0.396, Accuracy: 0.035
Epoch 46/100, Loss: 0.423, Accuracy: 0.034
Epoch 47/100, Loss: 0.489, Accuracy: 0.033
Epoch 48/100, Loss: 0.591, Accuracy: 0.029
Epoch 49/100, Loss: 0.415, Accuracy: 0.034
Epoch 50/100, Loss: 0.291, Accuracy: 0.039
Epoch 51/100, Loss: 0.395, Accuracy: 0.033
Epoch 52/100, Loss: 0.540, Accuracy: 0.026
Epoch 53/100, Loss: 0.436, Accuracy: 0.033
Epoch 54/100, Loss: 0.346, Accuracy: 0.036
Epoch 55/100, Loss: 0.519, Accuracy: 0.029
Epoch 56/100, Loss: 0.456, Accuracy: 0.031
Epoch 57/100, Loss: 0.425, Accuracy: 0.035
Epoch 58/100, Loss: 0.311, Accuracy: 0.039
Epoch 59/100, Loss: 0.406, Accuracy: 0.034
Epoch 60/100, Loss: 0.360, Accuracy: 0.035
Epoch 61/100, Loss: 0.476, Accuracy: 0.030
Epoch 62/100, Loss: 0.404, Accuracy: 0.034
Epoch 63/100, Loss: 0.382, Accuracy: 0.036
Epoch 64/100, Loss: 0.538, Accuracy: 0.031
Epoch 65/100, Loss: 0.392, Accuracy: 0.034
Epoch 66/100, Loss: 0.434, Accuracy: 0.033
Epoch 67/100, Loss: 0.479, Accuracy: 0.031
Epoch 68/100, Loss: 0.494, Accuracy: 0.031
Epoch 69/100, Loss: 0.415, Accuracy: 0.034
Epoch 70/100, Loss: 0.390, Accuracy: 0.036
Epoch 71/100, Loss: 0.330, Accuracy: 0.038
Epoch 72/100, Loss: 0.449, Accuracy: 0.030
Epoch 73/100, Loss: 0.315, Accuracy: 0.039
Epoch 74/100, Loss: 0.450, Accuracy: 0.031
Epoch 75/100, Loss: 0.562, Accuracy: 0.030
Epoch 76/100, Loss: 0.447, Accuracy: 0.031
Epoch 77/100, Loss: 0.408, Accuracy: 0.038
Epoch 78/100, Loss: 0.359, Accuracy: 0.034
Epoch 79/100, Loss: 0.372, Accuracy: 0.035
Epoch 80/100, Loss: 0.452, Accuracy: 0.034
Epoch 81/100, Loss: 0.360, Accuracy: 0.035
Epoch 82/100, Loss: 0.453, Accuracy: 0.031
Epoch 83/100, Loss: 0.578, Accuracy: 0.030
Epoch 84/100, Loss: 0.537, Accuracy: 0.030
Epoch 85/100, Loss: 0.483, Accuracy: 0.035
Epoch 86/100, Loss: 0.343, Accuracy: 0.036
Epoch 87/100, Loss: 0.439, Accuracy: 0.034
Epoch 88/100, Loss: 0.686, Accuracy: 0.023
Epoch 89/100, Loss: 0.265, Accuracy: 0.039
Epoch 90/100, Loss: 0.369, Accuracy: 0.035
Epoch 91/100, Loss: 0.521, Accuracy: 0.027
Epoch 92/100, Loss: 0.662, Accuracy: 0.027
Epoch 93/100, Loss: 0.581, Accuracy: 0.029
Epoch 94/100, Loss: 0.322, Accuracy: 0.034
Epoch 95/100, Loss: 0.375, Accuracy: 0.035
Epoch 96/100, Loss: 0.575, Accuracy: 0.031
Epoch 97/100, Loss: 0.489, Accuracy: 0.030
Epoch 98/100, Loss: 0.435, Accuracy: 0.033
Epoch 99/100, Loss: 0.440, Accuracy: 0.031
Epoch 100/100, Loss: 0.444, Accuracy: 0.033

Output: Is x the entire input dataset? If so, you might be dividing by the size of the entire input dataset in correct/x.shape[0] (as opposed to the size of the mini-batch). Try changing this to correct/output.shape[0]


Instruction: Accessing reduced dimensionality of trained autoencoder

Input:
Here is a autoencoder trained on mnist using PyTorch :

import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available() # True if cuda is available, False otherwise
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
print('Training on %s' % ('GPU' if cuda else 'CPU'))

# Loading the MNIST data set
transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,))])
mnist = torchvision.datasets.MNIST(root='../data/', train=True, transform=transform, download=True)

# Loader to feed the data batch by batch during training.
batch = 100
data_loader = torch.utils.data.DataLoader(mnist, batch_size=batch, shuffle=True)

autoencoder = nn.Sequential(
                # Encoder
                nn.Linear(28 * 28, 512),
                nn.PReLU(512),
                nn.BatchNorm1d(512),

                # Low-dimensional representation
                nn.Linear(512, 128),
                nn.PReLU(128),
                nn.BatchNorm1d(128),

                # Decoder
                nn.Linear(128, 512),
                nn.PReLU(512),
                nn.BatchNorm1d(512),
                nn.Linear(512, 28 * 28))

autoencoder = autoencoder.type(FloatTensor)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.005)

epochs = 10
data_size = int(mnist.train_labels.size()[0])

for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
        images = images.view(images.size(0), -1) # from (batch 1, 28, 28) to (batch, 28, 28)
        images = Variable(images).type(FloatTensor)

        autoencoder.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()
    print('Epoch %i/%i loss %.2f' % (i + 1, epochs, loss.data[0]))

print('Optimization finished.')

I'm attempting to compare the lower dimensionality representation of each image.

Printing the dimensionality of each layer :

for l in autoencoder.parameters() :
    print(l.shape)

displays :

torch.Size([512, 784])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([128, 512])
torch.Size([128])
torch.Size([128])
torch.Size([128])
torch.Size([128])
torch.Size([512, 128])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([784, 512])
torch.Size([784])

So appears the dimensionality is not stored in learned vectors ?

In other words if I have 10000 images each containing 100 pixels, executing above autoencoder that reduces dimensionality to 10 pixels should allow to access the 10 pixel dimensionality of all 10000 images ?

Output:
I'm not very familiar with pyTorch, but splitting the autoencoder into an encoder and decoder model seems to work (I changed the size of the hidden layer from 512 to 64, and the dimension of the encoded image from 128 to 4, to make the example run faster):

import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available() # True if cuda is available, False otherwise
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
print('Training on %s' % ('GPU' if cuda else 'CPU'))

# Loading the MNIST data set
transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,))])
mnist = torchvision.datasets.MNIST(root='../data/', train=True, transform=transform, download=True)

# Loader to feed the data batch by batch during training.
batch = 100
data_loader = torch.utils.data.DataLoader(mnist, batch_size=batch, shuffle=True)

encoder = nn.Sequential(
                # Encoder
                nn.Linear(28 * 28, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),

                # Low-dimensional representation
                nn.Linear(64, 4),
                nn.PReLU(4),
                nn.BatchNorm1d(4))

decoder = nn.Sequential(
                # Decoder
                nn.Linear(4, 64),
                nn.PReLU(64),
                nn.BatchNorm1d(64),
                nn.Linear(64, 28 * 28))

autoencoder = nn.Sequential(encoder, decoder)

encoder = encoder.type(FloatTensor)
decoder = decoder.type(FloatTensor)
autoencoder = autoencoder.type(FloatTensor)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.005)

epochs = 10
data_size = int(mnist.train_labels.size()[0])

for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
        images = images.view(images.size(0), -1) # from (batch 1, 28, 28) to (batch, 28, 28)
        images = Variable(images).type(FloatTensor)

        autoencoder.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()
    print('Epoch %i/%i loss %.2f' % (i + 1, epochs, loss.data[0]))

print('Optimization finished.')

# Get the encoded images here
encoded_images = []
for j, (images, _) in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))


Instruction: Custom Loss in Pytorch where object does not have attribute backward()

Input:
I am new to pytorch and I tried creating my own custom loss. This has been really challenging. Below is what I have for my loss.

class CustomLoss(nn.Module):

    def __init__(self,  size_average=True, reduce=True):
        """

        Args:
            size_average (bool, optional): By default, the losses are averaged
               over observations for each minibatch. However, if the field
               size_average is set to ``False``, the losses are instead summed for
               each minibatch. Only applies when reduce is ``True``. Default: ``True``
            reduce (bool, optional): By default, the losses are averaged
               over observations for each minibatch, or summed, depending on
               size_average. When reduce is ``False``, returns a loss per input/target
               element instead and ignores size_average. Default: ``True``
        """
        super(CustomLoss, self).__init__()

    def forward(self, S, N, M, type='softmax',):

        return self.loss_cal(S, N, M, type)

    ### new loss cal
    def loss_cal(self, S, N, M, type="softmax",):
        """ calculate loss with similarity matrix(S) eq.(6) (7)
        :type: "softmax" or "contrast"
        :return: loss
        """

        self.A = torch.cat([S[i * M:(i + 1) * M, i:(i + 1)]
                               for i in range(N)], dim=0)
        self.A = torch.autograd.Variable(self.A)

        if type == "softmax":
            self.B = torch.log(torch.sum(torch.exp(S.float()), dim=1, keepdim=True) + 1e-8)
            self.B = torch.autograd.Variable(self.B)
            total = torch.abs(torch.sum(self.A - self.B))
        else:
            raise AssertionError("loss type should be softmax or contrast !")
        return total

When I run the following:

loss = CustomLoss()
(loss.loss_cal(S=S,N=N,M=M))
loss.backward()

I get the following error:

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2113             magic_arg_s = self.var_expand(line, stack_depth)
   2114             with self.builtin_trap:
-> 2115                 result = fn(magic_arg_s, cell)
   2116             return result
   2117

<decorator-gen-60> in time(self, line, cell, local_ns)

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magic.py in <lambda>(f, *a, **k)
    186     # but it's overkill for just that one bit of state.
    187     def magic_deco(arg):
--> 188         call = lambda f, *a, **k: f(*a, **k)
    189
    190         if callable(arg):

C:\Program Files\Anaconda3\lib\site-packages\IPython\core\magics\execution.py in time(self, line, cell, local_ns)
   1178         else:
   1179             st = clock2()
-> 1180             exec(code, glob, local_ns)
   1181             end = clock2()
   1182             out = None

<timed exec> in <module>()

C:\Program Files\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __getattr__(self, name)
    530                 return modules[name]
    531         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 532             type(self).__name__, name))
    533
    534     def __setattr__(self, name, value):

AttributeError: 'CustomLoss' object has no attribute 'backward'

Why am I getting this error? I did not face this error with TF. My understanding is, it has to do with the autograd? If someone can explain why I am facing this error, I can figure the rest out.

Output:
Hi!

the problem is that you try to call the backward function on the module, but not on the variable (as you probably want to). As you have not implemented a backward function on the module, the interpreter cannot find one. So what you want to do instead is:

loss_func = CustomLoss()
loss = loss_func.loss_cal(S=S,N=N,M=M)
loss.backward()

As a general remark:
You are using a nn.Module without it actually having parameters. While that works, this is not what nn.Modules are there for - and should therefore be avoided. Instead, simply make a pure function - after all, the function you have there is static anyways. If you really want to go for the class, think of the type of class you want to create - a loss. Losses, however, can have special pytorch properties. So you should read up on a discussion here.


Instruction: Why is PyTorch called PyTorch?

Input:
I have been looking into deep learning frameworks lately and have been wondering about the origin of the name of PyTorch.
With Keras, their home page nicely explains the name's origin, and with something like TensorFlow, the reasoning behind the name seems rather clear. For PyTorch, however, I cannot seem to come across why it is so named.
Of course, I understand the "Py-" prefix and also know that PyTorch is a successor in some sense of Torch. But I am still wondering: what is the original idea behind the "-Torch" part? Is it known what the origin of the name is?

Output:
Here a short answer, formed as another question:
Torch, SMORCH ???
PyTorch developed from Torch7. A precursor to the original Torch was a library called SVM-Torch, which was developed around 2001. The SVM stands for Support Vector Machines.
SVM-Torch is a decomposition algorithm similar to SVM-Light, but adapted to regression problems, according to this paper.
Also around this time, G.W.Flake described the sequential minimal optimization algorithm (SMO), which could be used to train SVMs on sparse data sets, and this was incorporated into NODElib.
Interestingly, this was called the SMORCH algorithm.
You can find out more about SMORCH in the NODElib docs

Optimization of the SVMs is:

performed by a variation of John Platt's sequential minimal
optimization (SMO) algorithm.  This version of SMO is generalized
for regression, uses kernel caching, and incorporates several
heuristics; for these reasons, we refer to the optimization
algorithm as SMORCH.

So SMORCH =
Sequential
Minimal
Optimization
Regression
Caching
Heuristics
I can't answer definitively, but my thinking is "Torch" is a riff or evolution of "Light" from SVM-Light combined with a large helping of SMORCHiness.  You'd need to check in with the authors of SVMTorch and SVM-Light to confirm that this is indeed what "sparked" the name. It is reasonable to assume that the "TO" of Torch stands for some other optimization, rather than SMO, such as  Tensor Optimization, but I haven't found any direct reference... yet.


Instruction: Is "input" a keyword that causes errors when used as a parameter name (in PyTorch)?

Input:
So I have a line of code:

packed_embeddings = pack_padded_sequence(input=embeddings,
                                                 lengths=lengths,
                                                 batch_first=True)

That throws me this error:

  File "/Users/kwj/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 130, in might_trace
    first_arg = args[0]

IndexError: tuple index out of range

But magically fixes itself if I take out the "input":

    packed_embeddings = pack_padded_sequence(embeddings,
                                             lengths=lengths,
                                             batch_first=True)

Here is the function specification in the PyTorch docs:

https://pytorch.org/docs/stable/_modules/torch/nn/utils/rnn.html#pack_padded_sequence

I'm using Python3 and PyTorch 0.4. Am I missing something really basic?
Not sure if this is my issue, or a PyTorch specific issue...pretty confused here.

Thanks

Output:
What's happening here is that pack_padded_sequence is decorated to return a partially applied function, and within the decorating code there is a function that accepts arguments as *args, **kwargs. This function passes args to another function, which inspects the first arg.  When you pass all the arguments to packed_padded_sequence as keyword arguments, args is empty, and so args[0] raises an IndexError.  If you pass input as a positional argument,args is not empty, and the IndexError is not raised.

This example code demonstrates the behaviour (the Pytorch code is not easy to read).

def decorator(func):
    def wrapper(*args, **kwargs):
        print('Args:', repr(args))
        print('Kwargs:', repr(kwargs))
        return func(*args, **kwargs)
    return wrapper

@decorator
def f(a, b=0, c=0):
    return a, b, c

if __name__ == '__main__':
    print('Positional argument...')
    print(f(1, b=2, c=3))
    print('All keyword arguments...')
    print(f(a=1, b=2, c=3))

The code produces this output:

Positional argument...
Args: (1,)                        <- Args is populated
Kwargs: {'b': 2, 'c': 3}
(1, 2, 3)
All keyword arguments...
Args: ()                          <- Args is empty
Kwargs: {'a': 1, 'b': 2, 'c': 3}
(1, 2, 3)


Instruction: Which part of Pytorch tensor represents channels?

Input:
Surprisingly I have not found an answer to this question after looking around the internet. I am specifically interested in a 3d tensor. From doing my own experiments, I have found that when I create a tensor:

h=torch.randn(5,12,5)

And then put a convolutional layer on it defined as follows:

conv=torch.nn.Conv1d(12,48,3,padding=1)

The output is a (5,48,5) tensor. So, am I correct in assuming that for a 3d tensor in pytorch the middle number represents the number of channels?

Edit: It seems that when running a conv2d, the input dimension is the first entry in the tensor, and I need to make it a 4d tensor (1,48,5,5) for example. Now I am very confused...

Any help is much appreciated!

Output:
For a conv2D, input should be in (N, C, H, W) format. N is the number of samples/batch_size. C is the channels. H and W are height and width resp.

See shape documentation at https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d

For conv1D, input should be (N,C,L) see documentation at https://pytorch.org/docs/stable/nn.html#conv1d


Instruction: Implementing a custom dataset with PyTorch

Input:
I'm attempting to modify this feedforward network taken from https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/01-basics/feedforward_neural_network/main.py
to utilize my own dataset.

I define a custom dataset of two 1 dim arrays as input and two scalars the corresponding output :

x = torch.tensor([[5.5, 3,3,4] , [1 , 2,3,4], [9 , 2,3,4]])
print(x)

y = torch.tensor([1,2,3])
print(y)

import torch.utils.data as data_utils

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=50, shuffle=True)

I've updated the hyperparameters to match new input_size (2) & num_classes (3).

I've also changed images = images.reshape(-1, 28*28).to(device) to images = images.reshape(-1, 4).to(device)

As the training set is minimal I've changed the batch_size to 1.

Upon making these modifications I receive error when attempting to train :

  RuntimeError                              Traceback (most recent call
  last)  in ()
       51
       52         # Forward pass
  ---> 53         outputs = model(images)
       54         loss = criterion(outputs, labels)
       55

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
      489             result = self._slow_forward(*input, **kwargs)
      490         else:
  --> 491             result = self.forward(*input, **kwargs)
      492         for hook in self._forward_hooks.values():
      493             hook_result = hook(self, input, result)

   in forward(self, x)
       31
       32     def forward(self, x):
  ---> 33         out = self.fc1(x)
       34         out = self.relu(out)
       35         out = self.fc2(out)

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
      489             result = self._slow_forward(*input, **kwargs)
      490         else:
  --> 491             result = self.forward(*input, **kwargs)
      492         for hook in self._forward_hooks.values():
      493             hook_result = hook(self, input, result)

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
       53
       54     def forward(self, input):
  ---> 55         return F.linear(input, self.weight, self.bias)
       56
       57     def extra_repr(self):

  /home/.local/lib/python3.6/site-packages/torch/nn/functional.py
  in linear(input, weight, bias)
      990     if input.dim() == 2 and bias is not None:
      991         # fused op is marginally faster
  --> 992         return torch.addmm(bias, input, weight.t())
      993
      994     output = input.matmul(weight.t())

  RuntimeError: size mismatch, m1: [3 x 4], m2: [2 x 3] at
  /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249

How to amend code to match expected dimensionality ? I'm unsure what code to change as I've changed all parameters that require updating ?

Source prior to changes :

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Hyper-parameters
input_size = 784
hidden_size = 500
num_classes = 10
num_epochs = 5
batch_size = 100
learning_rate = 0.001

# MNIST dataset
train_dataset = torchvision.datasets.MNIST(root='../../data',
                                           train=True,
                                           transform=transforms.ToTensor(),
                                           download=True)

test_dataset = torchvision.datasets.MNIST(root='../../data',
                                          train=False,
                                          transform=transforms.ToTensor())

# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 28*28).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.reshape(-1, 28*28).to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model.ckpt')

Source post changes :

x = torch.tensor([[5.5, 3,3,4] , [1 , 2,3,4], [9 , 2,3,4]])
print(x)

y = torch.tensor([1,2,3])
print(y)

import torch.utils.data as data_utils

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=50, shuffle=True)

print(my_train)

print(my_train_loader)

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Hyper-parameters
input_size = 2
hidden_size = 3
num_classes = 3
num_epochs = 5
batch_size = 1
learning_rate = 0.001

# MNIST dataset
train_dataset = my_train

# Data loader
train_loader = my_train_loader

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 4).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.reshape(-1, 4).to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model.ckpt')

Output:
You need to change input_size to 4 (2*2), and not 2 as your modified code currently shows.
If you compare it to the original MNIST example, you'll see that input_size is set to 784 (28*28) and not just to 28.


Instruction: How to make a class in pytorch use GPU

Input:
So I am running some code and getting the following error in Pytorch: "RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same"

From what I understand, this means that my model may not be pushed to the GPU, while the input data already is using the GPU. I can share my code if that would help (I am refraining from doing it right now since it is longer than a small code snippet).

I am aware that I can do something like

    myModel=Model()
    myModel.cuda()

However, I am making a class as part of a custom module that will be added to a Sequential wrapper. So, I can't really make an object out of it (I'm not good with OOP terminology, so I apologize for any technical writing mistakes). I was wondering if there is a way to get around this issue, and make the class always use the GPU, even though I never explicitly define an object?

If this was not clear enough, I can post my code, but as previously warned it may take some time to go through (not too long, but not very convenient either).

Any help is much appreciated.

Edit: Here is the code, I presume the issue is in the RLSTM class, since there was not an error before I added this.

class VGG(nn.Module):
'''
VGG model
'''
def __init__(self, features): # features represents the layers array
    super(VGG, self).__init__()
    self.features = features
    self.classifier = nn.Sequential(
        nn.Dropout(),
        nn.Linear(512,512),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(512, 512),
        nn.ReLU(True),
        nn.Linear(512, 10),
    )
     # Initialize weights
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
            m.bias.data.zero_()

def forward(self, x): # x is the image, we run x through the layers
    print(x.size())
    x = self.features(x) # runs through all features, where each feature is a function
    x = x.view(x.size(0), -1)
    # after running through features, does sequential steps to finally classify
    x = self.classifier(x)
    # print(x)
    return x

def make_layers(cfg, batch_norm=False):
   # print("Making layers!")
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
            layers+=[RLSTM()]

    return nn.Sequential(*layers)

class RLSTM(nn.Module):
def __init__(self):
    super(RLSTM,self).__init__()

def forward(self, image):
    print("going in rowlstm")
    global current
    global _layer
    global isgates
    size = image.size()
    b = size[0]
    indvs = list(image.split(1,0)) # split up the batch into individual images
    #print(indvs[0].size())
    tensor_array = []
    for i in range(b):
        current = 0
        _layer = []
        isgates = []
        tensor_array.append(self.RowLSTM(indvs[i]))

    seq=tuple(tensor_array)
    trans = torch.cat(seq,0)
    return trans.cuda() # trying to make floattensor error go away
def RowLSTM(self, image):
    global current
    global _layer
    global isgates

    # input-to-state (K_is * x_i) : 3x1 convolution. generate 4h x n x n tensor. 4hxnxn tensor contains all i -> s info

# the input to state convolution should only be computed one time
    if current==0:
        n = image.size()[2]
        ch=image.size()[1]
        input_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1))
        isgates = self.splitIS(input_to_state(image)) # convolve, then split into gates (4 per row)
        cell=RowLSTMCell(0,torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1))
        # now have dummy, learnable variables for first row
        _layer.append(cell)

    else:
        Cell_prev = _layer[current-1] # access previous row
        hidPrev = Cell_prev.getHiddenState()
        ch = image.size()[1]
    #   print("about to apply conv1d")
        state_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1)) # error is here: hidPrev is an array - not a valid number of input channel
    #   print("applied conv1d")
        prevHid=Cell_prev.getHiddenState()
        ssgates = self.splitSS(state_to_state(prevHid.unsqueeze(0))) #need to unsqueeze (Ex: currently 16x5, need to make 1x16x5)
        gates = self.addGates(isgates,ssgates,current)
        # split gates
        ig, og, fg, gg = gates[0], gates[1], gates[2], gates[3] # into four, ADD SIGMOID!
        cell = RowLSTMCell(Cell_prev,ig,og,fg,gg,0,0)
        cell.compute()
        _layer.append(cell)
    # attempting to eliminate requirement of getting size

    #print(current)
    try:

        current+=1
        y=(isgates[0][0][1][current])
        return self.RowLSTM(image)
    except Exception as error:
        concats=[]
        for cell in _layer:
            tensor=torch.unsqueeze(cell.h,0)

            concats.append(tensor)
        seq=tuple(concats)
        tensor=torch.cat(seq,3)
        return tensor

def splitIS(tensor): #always going to be splitting into 4 pieces, so no need to add extra parameters
    inputStateGates={}
    size=tensor.size() # 1 x 4h x n x n
    out_ft=size[1] # get 4h for the nxnx4h tensor
    num=size[2] # get n for the nxn image
    hh=out_ft/4 # we want to split the tensor into 4, for the gates
    tensor = torch.squeeze(tensor) # 4h x n x n

    # First, split by row: Creates n tensors of 4h x n x 1
    rows = list(tensor.split(1,2))

    for i in range(num):
        # Each row is a tensor of 4h x n x 1, split it into 4 of h x n x 1
        row=rows[i]
        inputStateGates[i]=list(row.split(hh,0))

    return inputStateGates

def splitSS(tensor): # 1 x 4h x n x 1, create 4 of 1 x h x n x 1
    size=tensor.size()
    out_ft=size[1] # get 4h for the 1x4hxn tensor
    num=size[2] # get n for the 1xhxn row
    hh=out_ft/4 # we want to split the tensor into 4, for the gates
    tensor = tensor.squeeze(0) # 4h x n x 1
    splitted=list(tensor.split(hh,0))
    return splitted

def addGates(i2s,s2s,key):
    """ these dictionaries are of form {key : [[i], [o], [f], [g]]}
        we want to add pairwise elemeents """

    # i2s is of form key: [[i], [o], [f], [g]] where each gate is hxn
    # s2s is of form [[h,n],[h,n],[h,n], [h,n]]
    gateSum = []
    for i in range(4): # always of length 4, representing the gates
        gateSum.append(torch.sigmoid(i2s[key][i] + s2s[i]))

    return gateSum

Output:
You have to define the child modules inside the __init__ function so that they can be registered as parameters of the module. If they are not parameters, .cuda() would not be call on them when you call .cuda() for the parent.

If you really needed dynamic parameters/modules declaration, take a look here. The key is apaszke's answer.


Instruction: How to progressively grow a neural network in pytorch?

Input:
I am trying to make a progressive autoencoder and I have thought a couple ways of growing my network during training. However, I am always stuck on this one part where I don't know if changing the input(encoder) and output(decoder) channel would affect my network. See the example below.

X = torch.randn( 8, 1, 4, 4,) # A batch of 8 grayscale images of 4x4 pixels in size

Encoder = nn.Sequential( Conv2D( 1, 16, 3, 1, 1 ), nn.ReLU() ) # starting setup 16 3x3 kernels

if I print the above weights from the network I would get a size of [ 1, 16, 3, 3 ], 16 kernels each of size 3x3
if I want to grow the network I would need to do save those weight because hopefully its already well trained on those 4x4 image inputs.

X = torch.randn( 8, 1, 8, 8) # increase the image size from 4x4 to 8x8

...
new_model = nn.Sequential()

then do...
# copy the previous layer and its weights from the original encoder

# BTW My issue starts here.

# Add/grow the new_model with new layers concat with the old layer, also modify the input channela so they can link correctly

# Final result would be like something below.

new_model = nn.Sequential( Conv2D( **1**, 8, 3, 1, 1 ), nn.ReLU(), Conv2D( **8**, 16,  3, 1, 1 ), nn.ReLU()   )

Encoder = new_model

# Repeat process

Everything looks good however because I change the input channel the size of the weights changed as well, and this is the issue that I have been stuck on for a while now. You can simply check this by running,

foo_1 = nn.Conv2d(1, 1, 3, 1, 1) # You can think this as the starting Conv2D from the starting encoder

foo_2 = nn.Conv2d(3, 1, 3, 1, 1) # You can think this as the modfiied starting Conv2D with an outer layer outputting 3 channels connecting to it

print(foo_1.weight.size()) # torch.Size([1, 1, 3, 3])

print(foo_2.weight.size()) # torch.Size([1, 3, 3, 3])

Initially, I thought foo_1 and foo_2 would both have the same weight size, as in both would only use one 3x3 kernel but it doesn't see to be the case. I hope you can see my dilemma now, after x amount of epochs I need to grow another convolution and I have to mess with the input size to make new layers chain properly but if I change the input size the shape of the weight is different and I don't know how pasting the old state would work.

I have been looking at pro gan implementations in pytorch and IMO they are not easy to read. How can I build more institutions on how to properly progressively grow your network?

Output:
By progressive autoencoder I assume you are referring to something like Pioneer Networks: Progressively Growing Generative Autoencoder which referred Progressive Growing of GANs for Improved Quality, Stability, and Variation.

First of all, don't use nn.Sequential. It is great for modeling simple and direct network structure, which is definitely not the case here. You should use simple nn.Conv2d, F.ReLU modules instead of building a nn.Sequential object.

Second, this isn't really about implementation but theory. You cannot magically convert a convolution layer from accepting 1 channels to 8 channels. There are numbers of ways to expand your convolution filter like appending random weights but I think it is not what you wanted.

From the second paper (It is a GAN, but the idea is the same), it does not expand any filters. Instead, the filters maintain their shape in the entire training process. Meaning that, you would have a

Conv2D(8, 16, 3, 1, 1)

from the very beginning (assuming you only have those two layers). An obvious problem pops out -- your grayscale image is a 1-channel input, but your convolution requires a 8-channel input in the first stage of training. In the second paper, it uses an extra 1x1 convolution layer to map RGB <-> feature maps. In your case, that would be

Conv2D(1, 8, 1)

which maps 1-channel input to 8-channel output. This can be thrown away after you are done with first stage.

There are other techniques like gradually fading in using a weight term stated in the paper. I suggest you read them, especially the second one.


Instruction: Issues with using cuda and float tensor

Input:
I have some code, and when I run it, I get the following error: Expected object of type torch.cuda.FloatTensor but found type torch.FloatTensor for argument #2 'other'

From this error message, I assume there a problem with pushing my models to the GPU. However, I am not sure precisely where the problem lies.

I will place the code wherein I think the problem may lie at the end of this question. Could someone please describe what the error exactly means and how to fix it? Any help is much appreciated.

class VGG(nn.Module):
    '''
    VGG model
    '''
    def __init__(self, features): # features represents the layers array
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512,512),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(512, 512),
            nn.ReLU(True),
            nn.Linear(512, 10),
        )
         # Initialize weights
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                m.bias.data.zero_()

    def forward(self, x): # x is the image, we run x through the layers
        print("Running through features")
        x = self.features(x) # runs through all features, where each feature is a function
        print("Finsihed features, going to classifier")
        x = x.view(x.size(0), -1)
        # after running through features, does sequential steps to finally classify
        x = self.classifier(x)
        return x

def make_layers(cfg, batch_norm=False):
   # print("Making layers!")
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
            rlstm =RLSTM(v)
            rlstm.input_to_state = torch.nn.DataParallel(rlstm.input_to_state)
            rlstm.state_to_state = torch.nn.DataParallel(rlstm.state_to_state)
            rlstm=rlstm.cuda()
            layers+=[rlstm]

    return nn.Sequential(*layers)

class RLSTM(nn.Module):
    def __init__(self,ch):
       # torch.set_default_tensor_type('torch.cuda.FloatTensor')
        super(RLSTM,self).__init__()
        self.ch=ch
        self.input_to_state = torch.nn.Conv2d(self.ch,4*self.ch,kernel_size=(1,3),padding=(0,1))
        self.state_to_state = torch.nn.Conv2d(self.ch,4*self.ch,kernel_size=(1,3),padding=(0,1)) # error is here: hidPrev is an array - not a valid number of input channel
       # self.input_to_state = self.input_to_state.cuda()
        #self.state_to_state = self.state_to_state.cuda()

    def forward(self, image):
      #  print("going in row forward")
        global current
        global _layer
        global isgates
        size = image.size()
        print("size: "+str(size))
        b = size[0]
        indvs = list(image.split(1,0)) # split up the batch into individual images
        #print(indvs[0].size())
        tensor_array = []
        for i in range(b):
            current = 0
            _layer = []
            isgates = []
            print(len(tensor_array))
            tensor_array.append(self.RowLSTM(indvs[i]))

        seq=tuple(tensor_array)
        trans = torch.cat(seq,0)
        print(trans.size())
        return trans.cuda() # trying to make floattensor error go away
    def RowLSTM(self, image):
    #    print("going in rowlstm")
        global current
        global _layer
        global isgates

        # input-to-state (K_is * x_i) : 3x1 convolution. generate 4h x n x n tensor. 4hxnxn tensor contains all i -> s info

    # the input to state convolution should only be computed one time
        if current==0:
            n = image.size()[2]
            ch=image.size()[1]
           # input_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1))
          #  print("about to do convolution")
            isgates = self.splitIS(self.input_to_state(image)) # convolve, then split into gates (4 per row)

            cell=RowLSTMCell(0,torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1))
            # now have dummy, learnable variables for first row
            _layer.append(cell)
            print("layeres: "+str(len(_layer)))
        else:
            Cell_prev = _layer[current-1] # access previous row
            hidPrev = Cell_prev.getHiddenState()
            ch = image.size()[1]
        #   print("about to apply conv1d")
           # state_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1)) # error is here: hidPrev is an array - not a valid number of input channel
        #   print("applied conv1d")
            prevHid=Cell_prev.getHiddenState()
            ssgates = self.splitSS(self.state_to_state(prevHid.unsqueeze(0))) #need to unsqueeze (Ex: currently 16x5, need to make 1x16x5)
            gates = self.addGates(isgates,ssgates,current)
            # split gates
            ig, og, fg, gg = gates[0], gates[1], gates[2], gates[3] # into four, ADD SIGMOID!
            cell = RowLSTMCell(Cell_prev,ig,og,fg,gg,0,0)
            cell.compute()
            _layer.append(cell)
        # attempting to eliminate requirement of getting size

        #print(current)
        try:
            print("adding one to current")
            current+=1
            y=(isgates[0][0][1][current])

            return self.RowLSTM(image) #expecting floattensor, but gets cuda floattensor

        except Exception as error:
            print(error)
            concats=[]
            print(len(_layer))
            for cell in _layer:
                tensor=torch.unsqueeze(cell.h,0)
                concats.append(tensor)
            seq=tuple(concats)

            print("non catted tensor: "+str(tensor.size()))
            tense=torch.cat(seq,3)
            print("catted lstm tensor "+str(tense.size()))
            return tensor

The code runs, but when trying to go through the try/except block,  the error is thrown. I am guessing the mistake lies somewhere here?

Edit: Using print statements to see where the program exactly terminates, it seems that there is a mistake in code that I have note posted yet! I will post that now, it looks like the error is in the compute() function, since the statement "finished computing" never gets printed.

class RowLSTMCell(): #inherit torch.nn.LSTM?
    def __init__(self,prev_row, i, o, f, g, c, h):
        #super(RowLSTMCell,self).__init__()
        self.c=c

        #self.c = self.c.cuda()
        self.h=h
       # self.h = self.h.cuda()
        self.i=i
        self.i = self.i.cuda()
        self.o=o
        self.o = self.o.cuda()
        self.g=g
        self.g = self.g.cuda()
        self.f=f
        self.f = self.f.cuda()
        self.prev_row=prev_row
    def getStateSize(self):
        return self._state_size

    def getOutputSize(self):
        return self._output_size

    def compute(self):
        print("computing")
        c_prev = self.prev_row.getCellState()
        h_prev = self.prev_row.getHiddenState()

        self.c = self.f * c_prev + self.i * self.g
        self.h = torch.tanh(self.c) * self.o
        print("finished computing")
    def getHiddenState(self):
        return self.h

    def getCellState(self):
        return self.c

Output: self.c and self.h were not cuda! I guess you really have to make sure that each tensor is using cuda. I just put .cuda() at the end of self.c and self.h's computation in the compute() method.


Instruction: How do I load custom image based datasets into Pytorch for use with a CNN?

Input:
I have searched for hours on the internet to find a good solution to my issue. Here is some relevant background information to help you answer my question.

This is my first ever deep learning project and I have no idea what I am doing. I know the theory but not the practical elements.

The data that I am using can be found on kaggle at this link:
(https://www.kaggle.com/alxmamaev/flowers-recognition)

I am aiming to classify flowers based on the images provided in the dataset using a CNN.

Here is some sample code I have tried to use to load data in so far, this is my best attempt but as I mentioned I am clueless and Pytorch docs didn't offer much help that I could understand at my level.
(https://pastebin.com/fNLVW1UW)

    # Loads the images for use with the CNN.
def load_images(image_size=32, batch_size=64, root="../images"):
    transform = transforms.Compose([
        transforms.Resize(32),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    train_set = datasets.ImageFolder(root=root, train=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)

    return train_loader

# Defining variables for use with the CNN.
classes = ('daisy', 'dandelion', 'rose', 'sunflower', 'tulip')
train_loader_data = load_images()

# Training samples.
n_training_samples = 3394
train_sampler = SubsetRandomSampler(np.arange(n_training_samples, dtype=np.int64))

# Validation samples.
n_val_samples = 424
val_sampler = SubsetRandomSampler(np.arange(n_training_samples, n_training_samples + n_val_samples, dtype=np.int64))

# Test samples.
n_test_samples = 424
test_sampler = SubsetRandomSampler(np.arange(n_test_samples, dtype=np.int64))

Here are my direct questions that I require answers too:

How do I fix my code to load in the dataset in an 80/10/10 split for training/test/validation?
How do i create the required labels/classes for these images which are already divided by folders in /images ?

Output:
Looking at the data from Kaggle and your code, there are problems in your data loading.

The data should be in a different folder per class label for PyTorch ImageFolder to load it correctly. In your case, since all the training data is in the same folder, PyTorch is loading it as one train set. You can correct this by using a folder structure like - train/daisy, train/dandelion, test/daisy, test/dandelion and then passing the train and the test folder to the train and test ImageFolder respectively. Just change the folder structure and you should be good. Take a look at the official documentation of torchvision.datasets.Imagefolder which has a similar example.

As you said, these images which are already divided by folders in /images. PyTorch ImageFolder assumes that images are organized in the following way. But this folder structure is only correct if you are using all the images for train set:

```
/images/daisy/100080576_f52e8ee070_n.jpg
/images/daisy/10140303196_b88d3d6cec.jpg
.
.
.
/images/dandelion/10043234166_e6dd915111_n.jpg
/images/dandelion/10200780773_c6051a7d71_n.jpg
```

where 'daisy', 'dandelion' etc. are class labels.

The correct folder structure if you want to split the dataset into train and test set in your case (note that I know you want to split the dataset into train, validation, and test set, but it doesn't matters as this is just an example to get the idea out):

```
/images/train/daisy/100080576_f52e8ee070_n.jpg
/images/train/daisy/10140303196_b88d3d6cec.jpg
.
.
/images/train/dandelion/10043234166_e6dd915111_n.jpg
/images/train/dandelion/10200780773_c6051a7d71_n.jpg
.
.
/images/test/daisy/300080576_f52e8ee070_n.jpg
/images/test/daisy/95140303196_b88d3d6cec.jpg
.
.
/images/test/dandelion/32143234166_e6dd915111_n.jpg
/images/test/dandelion/65200780773_c6051a7d71_n.jpg
```

Then, you can refer to the following full code example on how to write a dataloader:

import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms

EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "./images/train/"
TEST_DATA_PATH = "./images/test/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)

class CNN(nn.Module):
    # omitted...

if __name__ == '__main__':

    print("Number of train samples: ", len(train_data))
    print("Number of test samples: ", len(test_data))
    print("Detected Classes are: ", train_data.class_to_idx) # classes are detected by folder structure

    model = CNN()
    optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE)
    loss_func = nn.CrossEntropyLoss()

    # Training and Testing
    for epoch in range(EPOCHS):
        for step, (x, y) in enumerate(train_data_loader):
            b_x = Variable(x)   # batch x (image)
            b_y = Variable(y)   # batch y (target)
            output = model(b_x)[0]
            loss = loss_func(output, b_y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if step % 50 == 0:
                test_x = Variable(test_data_loader)
                test_output, last_layer = model(test_x)
                pred_y = torch.max(test_output, 1)[1].data.squeeze()
                accuracy = sum(pred_y == test_y) / float(test_y.size(0))
                print('Epoch: ', epoch, '| train loss: %.4f' % loss.data[0], '| test accuracy: %.2f' % accuracy)


Instruction: Data loading with variable batch size?

Input:
I am currently working on patch based super-resolution. Most of the papers divide an image into smaller patches and then use the patches as input to the models.I was able to create patches using custom dataloader. The code is given below:

import torch.utils.data as data
from torchvision.transforms import CenterCrop, ToTensor, Compose, ToPILImage, Resize, RandomHorizontalFlip, RandomVerticalFlip
from os import listdir
from os.path import join
from PIL import Image
import random
import os
import numpy as np
import torch

def is_image_file(filename):
    return any(filename.endswith(extension) for extension in [".png", ".jpg", ".jpeg", ".bmp"])

class TrainDatasetFromFolder(data.Dataset):
    def __init__(self, dataset_dir, patch_size, is_gray, stride):
        super(TrainDatasetFromFolder, self).__init__()
        self.imageHrfilenames = []
        self.imageHrfilenames.extend(join(dataset_dir, x)
                                     for x in sorted(listdir(dataset_dir)) if is_image_file(x))
        self.is_gray = is_gray
        self.patchSize = patch_size
        self.stride = stride

    def _load_file(self, index):
        filename = self.imageHrfilenames[index]
        hr = Image.open(self.imageHrfilenames[index])
        downsizes = (1, 0.7, 0.45)
        downsize = 2
        w_ = int(hr.width * downsizes[downsize])
        h_ = int(hr.height * downsizes[downsize])
        aug = Compose([Resize([h_, w_], interpolation=Image.BICUBIC),
                       RandomHorizontalFlip(),
                       RandomVerticalFlip()])

        hr = aug(hr)
        rv = random.randint(0, 4)
        hr = hr.rotate(90*rv, expand=1)
        filename = os.path.splitext(os.path.split(filename)[-1])[0]
        return hr, filename

    def _patching(self, img):

        img = ToTensor()(img)
        LR_ = Compose([ToPILImage(), Resize(self.patchSize//2, interpolation=Image.BICUBIC), ToTensor()])

        HR_p, LR_p = [], []
        for i in range(0, img.shape[1] - self.patchSize, self.stride):
            for j in range(0, img.shape[2] - self.patchSize, self.stride):
                temp = img[:, i:i + self.patchSize, j:j + self.patchSize]
                HR_p += [temp]
                LR_p += [LR_(temp)]

        return torch.stack(LR_p),torch.stack(HR_p)

    def __getitem__(self, index):
        HR_, filename = self._load_file(index)
        LR_p, HR_p = self._patching(HR_)
        return LR_p, HR_p

    def __len__(self):
        return len(self.imageHrfilenames)

Suppose the batch size is 1, it takes an image and gives an output of size [x,3,patchsize,patchsize]. When batch size is 2, I will have two different outputs of size [x,3,patchsize,patchsize] (for example image 1 may give[50,3,patchsize,patchsize], image 2 may give[75,3,patchsize,patchsize] ). To handle this a custom collate function was required that stacks these two outputs along dimension 0. The collate function is given below:

def my_collate(batch):
    data = torch.cat([item[0] for item in batch],dim = 0)
    target = torch.cat([item[1] for item in batch],dim = 0)

    return [data, target]

This collate function concatenates along x (From the above example, I finally get [125,3,patchsize,pathsize]. For training purposes, I need to train the model using a minibatch size of say 25. Is there any method or any functions which I can use to directly get an output of size [25 , 3, patchsize, pathsize] directly from the dataloader using the necessary number of images as input to the Dataloader?

Output:
The following code snippet works for your purpose.

First, we define a ToyDataset which takes in a list of tensors (tensors) of variable length in dimension 0. This is similar to the samples returned by your dataset.

import torch
from torch.utils.data import Dataset
from torch.utils.data.sampler import RandomSampler

class ToyDataset(Dataset):
    def __init__(self, tensors):
        self.tensors = tensors

    def __getitem__(self, index):
        return self.tensors[index]

    def __len__(self):
        return len(tensors)

Secondly, we define a custom data loader. The usual Pytorch dichotomy to create datasets and data loaders is roughly the following: There is an indexed dataset, to which you can pass an index and it returns the associated sample from the dataset. There is a sampler which yields an index, there are different strategies to draw indices which give rise to different samplers. The sampler is used by a batch_sampler to draw multiple indices at once (as many as specified by batch_size). There is a dataloader which combines sampler and dataset to let you iterate over a dataset, importantly the data loader also owns a function (collate_fn) which specifies how the multiple samples retrieved from the dataset using the indices from the batch_sampler should be combined. For your use case, the usual PyTorch dichotomy does not work well, because instead of drawing a fixed number of indices, we need to draw indices until the objects associated with the indices exceed the cumulative size we desire. This means we need immediate inspection of the objects and use this knowledge to decide whether to return a batch or keep drawing indices. This is what the custom data loader below does:

class CustomLoader(object):

    def __init__(self, dataset, my_bsz, drop_last=True):
        self.ds = dataset
        self.my_bsz = my_bsz
        self.drop_last = drop_last
        self.sampler = RandomSampler(dataset)

    def __iter__(self):
        batch = torch.Tensor()
        for idx in self.sampler:
            batch = torch.cat([batch, self.ds[idx]])
            while batch.size(0) >= self.my_bsz:
                if batch.size(0) == self.my_bsz:
                    yield batch
                    batch = torch.Tensor()
                else:
                    return_batch, batch = batch.split([self.my_bsz,batch.size(0)-self.my_bsz])
                    yield return_batch
        if batch.size(0) > 0 and not self.drop_last:
            yield batch

Here we iterate over the dataset, after drawing an index and loading the associated object, we concatenate it to the tensors we drew before (batch). We keep doing this until we reach the desired size, such that we can cut out and yield a batch. We retain the rows in batch, which we did not yield. Because it may be the case that a single instance exceeds the desired batch_size, we use a while loop.

You could modify this minimal CustomDataloader to add more features in the style of PyTorch's dataloader. There is also no need to use a RandomSampler to draw in indices, others would work equally well. It would also be possible to avoid repeated concats, in case your data is large by using for example a list and keeping track of the cumulative length of its tensors.

Here is an example, that demonstrates it works:

patch_size = 5
channels = 3
dim0sizes = torch.LongTensor(100).random_(1, 100)
data = torch.randn(size=(dim0sizes.sum(), channels, patch_size, patch_size))
tensors = torch.split(data, list(dim0sizes))

ds = ToyDataset(tensors)
dl = CustomLoader(ds, my_bsz=250, drop_last=False)
for i in dl:
    print(i.size(0))


Instruction: Linear regression with pytorch

Input:
I tried to run linear regression on ForestFires dataset.
Dataset is available on Kaggle and gist of my attempt is here:
https://gist.github.com/Chandrak1907/747b1a6045bb64898d5f9140f4cf9a37

I am facing two problems:

Output from prediction is of shape 32x1 and target data shape is 32.

  input and target shapes do not match: input [32 x 1], target [32]¶

Using view I reshaped predictions tensor.

  y_pred = y_pred.view(inputs.shape[0])

Why there is a mismatch in shapes of predicted tensor and actual tensor?

SGD in pytorch never converges. I tried to compute MSE manually using

  print(torch.mean((y_pred - labels)**2))

This value does not match

  loss = criterion(y_pred,labels)

Can someone highlight where is the mistake in my code?

Thank you.

Output:
Problem 1

This is reference about MSELoss from Pytorch docs: https://pytorch.org/docs/stable/nn.html#torch.nn.MSELoss

Shape:
 - Input: (N,∗) where * means, any number of additional dimensions
 - Target: (N,∗), same shape as the input

So, you need to expand dims of labels: (32) -> (32,1), by using: torch.unsqueeze(labels, 1) or labels.view(-1,1)

https://pytorch.org/docs/stable/torch.html#torch.unsqueeze

  torch.unsqueeze(input, dim, out=None) → Tensor

  Returns a new tensor with a dimension of size one inserted at the specified position.

  The returned tensor shares the same underlying data with this tensor.

Problem 2

After reviewing your code, I realized that you have added size_average param to MSELoss:

criterion = torch.nn.MSELoss(size_average=False)

  size_average (bool, optional) – Deprecated (see reduction). By default, the losses are averaged over each loss element in the batch. Note that for some losses, there multiple elements per sample. If the field size_average is set to False, the losses are instead summed for each minibatch. Ignored when reduce is False. Default: True

That's why 2 computed values not matched. This is sample code:

import torch
import torch.nn as nn

loss1 = nn.MSELoss()
loss2 = nn.MSELoss(size_average=False)
inputs = torch.randn(32, 1, requires_grad=True)
targets = torch.randn(32, 1)

output1 = loss1(inputs, targets)
output2 = loss2(inputs, targets)
output3 = torch.mean((inputs - targets) ** 2)

print(output1)  # tensor(1.0907)
print(output2)  # tensor(34.9021)
print(output3)  # tensor(1.0907)


Instruction: Why doesn't my simple pytorch network work on GPU device?

Input:
I built a simple network from a tutorial and I got this error:

  RuntimeError: Expected object of type torch.cuda.FloatTensor but found
  type torch.FloatTensor for argument #4 'mat1'

Any help? Thank you!

import torch
import torchvision

device = torch.device("cuda:0")
root = '.data/'

dataset = torchvision.datasets.MNIST(root, transform=torchvision.transforms.ToTensor(), download=True)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.out = torch.nn.Linear(28*28, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.out(x)
        return x

net = Net()
net.to(device)

for i, (inputs, labels) in enumerate(dataloader):
    inputs.to(device)
    out = net(inputs)

Output:
TL;DR
This is the fix

inputs = inputs.to(device)

Why?!
There is a slight difference between torch.nn.Module.to() and torch.Tensor.to(): while Module.to() is an in-place operator, Tensor.to() is not. Therefore

net.to(device)

Changes net itself and moves it to device. On the other hand

inputs.to(device)

does not change inputs, but rather returns a copy of inputs that resides on device. To use that "on device" copy, you need to assign it into a variable, hence

inputs = inputs.to(device)


Instruction: Understanding PyTorch prediction

Input:
For my trained model this code :

model(x[0].reshape(1,784).cuda())

returns :

tensor([[-1.9903, -4.0458, -4.1143, -4.0074, -3.5510,  7.1074]], device='cuda:0')

My network model is defined as :

# Hyper-parameters
input_size = 784
hidden_size = 50
num_classes = 6
num_epochs = 5000
batch_size = 1
learning_rate = 0.0001

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

        class NeuralNet(nn.Module):
            def __init__(self, input_size, hidden_size, num_classes):
                super(NeuralNet, self).__init__()
                self.fc1 = nn.Linear(input_size, hidden_size)
                self.relu = nn.ReLU()
                self.fc2 = nn.Linear(hidden_size, num_classes)

            def forward(self, x):
                out = self.fc1(x)
                out = self.relu(out)
                out = self.fc2(out)
                return out

I'm attempting to understand the returned value :

tensor([[-1.9903, -4.0458, -4.1143, -4.0074, -3.5510,  7.1074]], device='cuda:0')

The value 7.1074 is the most probable as it maximum value in tensor array ? As 7.1074 is at position 5, is the significance here that the associated output value being predicted for input x[0] is 5 ? If so what is the intuition behind this ?

Output:
Disclaimer: I don't really know pytorch, but i'm guessing based on other libraries and general standard practice as i know it.

I believe it's the outputs of the last layer, which would be that fc2 linear transformation. So, the predicted category would be category 5, having the highest value.

You could think of it as being a sort of 'raw' probability distribution-esque range, that is commonly then squashed into the range (0,1] via softmax.


Instruction: PyTorch: passing numpy array for weight initialization

Input:
I'd like to initialize the parameters of RNN with np arrays.

In the following example, I want to pass w to the parameters of rnn. I know pytorch provides many initialization methods like Xavier, uniform, etc., but is there way to initialize the parameters by passing numpy arrays?

import numpy as np
import torch as nn
rng = np.random.RandomState(313)
w = rng.randn(input_size, hidden_size).astype(np.float32)

rnn = nn.RNN(input_size, hidden_size, num_layers)

Output:
First, let's note that nn.RNN has more than one weight variable, c.f. the documentation:

  Variables:

  weight_ih_l[k] – the learnable input-hidden weights of the k-th layer, of shape (hidden_size * input_size) for k = 0. Otherwise,
  the shape is (hidden_size * hidden_size)
  weight_hh_l[k] – the learnable hidden-hidden weights of the k-th layer, of shape (hidden_size * hidden_size)
  bias_ih_l[k] – the learnable input-hidden bias of the k-th layer, of shape (hidden_size)
  bias_hh_l[k] – the learnable hidden-hidden bias of the k-th layer, of shape (hidden_size)

Now, each of these variables (Parameter instances) are attributes of your nn.RNN instance. You can access them, and edit them, two ways, as show below:

Solution 1: Accessing all the RNN Parameter attributes by name (rnn.weight_hh_lK, rnn.weight_ih_lK, etc.):

import torch
from torch import nn
import numpy as np

input_size, hidden_size, num_layers = 3, 4, 2
use_bias = True
rng = np.random.RandomState(313)

rnn = nn.RNN(input_size, hidden_size, num_layers, bias=use_bias)

def set_nn_parameter_data(layer, parameter_name, new_data):
    param = getattr(layer, parameter_name)
    param.data = new_data

for i in range(num_layers):
    weights_hh_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    weights_ih_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    set_nn_parameter_data(rnn, "weight_hh_l{}".format(i),
                          torch.from_numpy(weights_hh_layer_i))
    set_nn_parameter_data(rnn, "weight_ih_l{}".format(i),
                          torch.from_numpy(weights_ih_layer_i))

    if use_bias:
        bias_hh_layer_i = rng.randn(hidden_size).astype(np.float32)
        bias_ih_layer_i = rng.randn(hidden_size).astype(np.float32)
        set_nn_parameter_data(rnn, "bias_hh_l{}".format(i),
                              torch.from_numpy(bias_hh_layer_i))
        set_nn_parameter_data(rnn, "bias_ih_l{}".format(i),
                              torch.from_numpy(bias_ih_layer_i))

Solution 2: Accessing all the RNN Parameter attributes through rnn.all_weights list attribute:

import torch
from torch import nn
import numpy as np

input_size, hidden_size, num_layers = 3, 4, 2
use_bias = True
rng = np.random.RandomState(313)

rnn = nn.RNN(input_size, hidden_size, num_layers, bias=use_bias)

for i in range(num_layers):
    weights_hh_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    weights_ih_layer_i = rng.randn(hidden_size, hidden_size).astype(np.float32)
    rnn.all_weights[i][0].data = torch.from_numpy(weights_ih_layer_i)
    rnn.all_weights[i][1].data = torch.from_numpy(weights_hh_layer_i)

    if use_bias:
        bias_hh_layer_i = rng.randn(hidden_size).astype(np.float32)
        bias_ih_layer_i = rng.randn(hidden_size).astype(np.float32)
        rnn.all_weights[i][2].data = torch.from_numpy(bias_ih_layer_i)
        rnn.all_weights[i][3].data = torch.from_numpy(bias_hh_layer_i)