Instruction: What's the reason of the error ValueError: Expected more than 1 value per channel?

Input:
reference fast.ai

github repository of fast.ai
(as the code elevates the library which is built on top of PyTorch)

  Please scroll the discussion a bit

I am running the following code, and get an error while trying to pass the data to the predict_array function

The code is failing when i am trying to use it to predict directly on a single image but it run's perfectly when that same image is in a test folder

from fastai.conv_learner import *
from planet import f2

PATH = 'data/shopstyle/'

metrics=[f2]
f_model = resnet34

def get_data(sz):
    tfms = tfms_from_model(f_model, sz, aug_tfms=transforms_side_on, max_zoom=1.05)
    return ImageClassifierData.from_csv(PATH, 'train', label_csv, tfms=tfms, suffix='.jpg', val_idxs=val_idxs, test_name='test')

def print_list(list_or_iterator):
        return "[" + ", ".join( str(x) for x in list_or_iterator) + "]"

label_csv = f'{PATH}prod_train.csv'
n = len(list(open(label_csv)))-1
val_idxs = get_cv_idxs(n)

sz = 64
data = get_data(sz)

print("Loading model...")
learn = ConvLearner.pretrained(f_model, data, metrics=metrics)
learn.load(f'{sz}')
#learn.load("tmp")

print("Predicting...")
learn.precompute=False
trn_tfms, val_tfrms = tfms_from_model(f_model, sz)
#im = val_tfrms(open_image(f'{PATH}valid/4500132.jpg'))
im = val_tfrms(np.array(PIL.Image.open(f'{PATH}valid/4500132.jpg')))
preds = learn.predict_array(im[None])
p=list(zip(data.classes, preds))
print("predictions = " + print_list(p))

Here's the Traceback I am Getting

  Traceback (most recent call last):
  File "predict.py", line 34, in <module>
    preds = learn.predict_array(im[None])
  File "/home/ubuntu/fastai/courses/dl1/fastai/learner.py", line 266, in predict_array
    def predict_array(self, arr): return to_np(self.model(V(T(arr).cuda())))
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py", line 37, in forward
    self.training, self.momentum, self.eps)
  File "/home/ubuntu/src/anaconda3/envs/fastai/lib/python3.6/site-packages/torch/nn/functional.py", line 1011, in batch_norm
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size [1, 1024]

Things I have Tried

np.expand_dims(IMG,axis=0) or image = image[..., np.newaxis]
Tried a different way of reading the image

img = cv2.imread(img_path)
img = cv2.resize(img, dsize = (200,200))
img = np.einsum('ijk->kij', img)
img = np.expand_dims(img, axis =0)
img = torch.from_numpy(img)
learn.model(Variable(img.float()).cuda())

BTW the error still remains

ValueError: Expected more than 1 value per channel when training, got input size [1, 1024]

Can't find any reference in The Google search also..

Output:
It will fail on batches of size 1 if we use feature-wise batch normalization.

As Batch normalization computes:

y = (x - mean(x)) / (std(x) + eps)

If we have one sample per batch then mean(x) = x, and the output will be entirely zero (ignoring the bias). We can't use that for learning...


Instruction: BatchNorm momentum convention PyTorch

Input: Is the batchnorm momentum convention (default=0.1) correct as in other libraries e.g. Tensorflow it seems to usually be 0.9 or 0.99 by default? Or maybe we are just using a different convention?

Output:
It seems that the parametrization convention is different in pytorch than in tensorflow, so that 0.1 in pytorch is equivalent to 0.9 in tensorflow.

To be more precise:

In Tensorflow:

running_mean = decay*running_mean + (1-decay)*new_value

In PyTorch:

running_mean = (1-decay)*running_mean + decay*new_value

This means that a value of decay in PyTorch is equivalent to a value of (1-decay) in Tensorflow.


Instruction: How to model Convolutional recurrent network ( CRNN ) in Keras

Input:
I was trying to port CRNN model to Keras.

But, I got stuck while connecting output of Conv2D layer to LSTM layer.

Output from CNN layer will have a shape of ( batch_size, 512, 1, width_dash) where first one depends on batch_size, and last one depends on input width of input ( this model can accept variable width input )

For eg: an input with shape [2, 1, 32, 829] was resulting output with shape of (2, 512, 1, 208)

Now, as per Pytorch model, we have to do squeeze(2)  followed by permute(2, 0, 1)
it will result a tensor with shape [208, 2, 512 ]

I was trying to implement this is Keras, but I was not able to do that because, in Keras we can not alter batch_size dimension in a keras.models.Sequential model

Can someone please guide me how to port above part of this model to Keras?

Current state of ported CNN layer

Output:
You don't need to permute the batch axis in Keras. In a pytorch model you need to do it because a pytorch LSTM expects an input shape (seq_len, batch, input_size). However in Keras, the LSTM layer expects (batch, seq_len, input_size).

So after defining the CNN and squeezing out axis 2, you just need to permute the last two axes. As a simple example (in 'channels_first' Keras image format),

model = Sequential()
model.add(Conv2D(512, 3, strides=(32, 4), padding='same', input_shape=(1, 32, None)))
model.add(Reshape((512, -1)))
model.add(Permute((2, 1)))
model.add(LSTM(32))

You can verify the shapes with model.summary():

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_4 (Conv2D)            (None, 512, 1, None)      5120
_________________________________________________________________
reshape_3 (Reshape)          (None, 512, None)         0
_________________________________________________________________
permute_4 (Permute)          (None, None, 512)         0
_________________________________________________________________
lstm_3 (LSTM)                (None, 32)                69760
=================================================================
Total params: 74,880
Trainable params: 74,880
Non-trainable params: 0
_________________________________________________________________


Instruction: Converting state-parameters of Pytorch LSTM to Keras LSTM

Input:
I was trying to port an existing trained PyTorch model into Keras.

During the porting, I got stuck at LSTM layer.

Keras implementation of LSTM network seems to have three state kind of state matrices while Pytorch implementation have four.

For eg, for an Bidirectional LSTM with hidden_layers=64, input_size=512 & output size=128 state parameters where as follows

State params of Keras LSTM

[<tf.Variable 'bidirectional_1/forward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/forward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/kernel:0' shape=(512, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/recurrent_kernel:0' shape=(64, 256) dtype=float32_ref>,
 <tf.Variable 'bidirectional_1/backward_lstm_1/bias:0' shape=(256,) dtype=float32_ref>]

State params of PyTorch LSTM

 ['rnn.0.rnn.weight_ih_l0', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0', torch.Size([256])],
 ['rnn.0.rnn.weight_ih_l0_reverse', torch.Size([256, 512])],
 ['rnn.0.rnn.weight_hh_l0_reverse', torch.Size([256, 64])],
 ['rnn.0.rnn.bias_ih_l0_reverse', torch.Size([256])],
 ['rnn.0.rnn.bias_hh_l0_reverse', torch.Size([256])],

I tried to look in to the code of both implementation but not able to understand much.

Can someone please help me to transform 4-set of state params from PyTorch  into 3-set of state params in Keras

Output:
They are really not that different. If you sum up the two bias vectors in PyTorch, the equations will be the same as what's implemented in Keras.

This is the LSTM formula on PyTorch documentation:

PyTorch uses two separate bias vectors for the input transformation (with a subscript starts with i) and recurrent transformation (with a subscript starts with h).

In Keras LSTMCell:

        x_i = K.dot(inputs_i, self.kernel_i)
        x_f = K.dot(inputs_f, self.kernel_f)
        x_c = K.dot(inputs_c, self.kernel_c)
        x_o = K.dot(inputs_o, self.kernel_o)
        if self.use_bias:
            x_i = K.bias_add(x_i, self.bias_i)
            x_f = K.bias_add(x_f, self.bias_f)
            x_c = K.bias_add(x_c, self.bias_c)
            x_o = K.bias_add(x_o, self.bias_o)

        if 0 < self.recurrent_dropout < 1.:
            h_tm1_i = h_tm1 * rec_dp_mask[0]
            h_tm1_f = h_tm1 * rec_dp_mask[1]
            h_tm1_c = h_tm1 * rec_dp_mask[2]
            h_tm1_o = h_tm1 * rec_dp_mask[3]
        else:
            h_tm1_i = h_tm1
            h_tm1_f = h_tm1
            h_tm1_c = h_tm1
            h_tm1_o = h_tm1
        i = self.recurrent_activation(x_i + K.dot(h_tm1_i,
                                                  self.recurrent_kernel_i))
        f = self.recurrent_activation(x_f + K.dot(h_tm1_f,
                                                  self.recurrent_kernel_f))
        c = f * c_tm1 + i * self.activation(x_c + K.dot(h_tm1_c,
                                                        self.recurrent_kernel_c))
        o = self.recurrent_activation(x_o + K.dot(h_tm1_o,
                                                  self.recurrent_kernel_o))

There's only one bias added in the input transformation. However, the equations would be equivalent if we sum up the two biases in PyTorch.

The two-bias LSTM is what's implemented in cuDNN (see the developer guide). I'm really not that familiar with PyTorch, but I guess that's why they use two bias parameters. In Keras, the CuDNNLSTM layer also has two bias weight vectors.


Instruction: Pytorch: Updating numpy array not updating the corresponding tensor

Input:
When I run the following code,

import numpy as np
a = np.ones(3)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)

Both a and b are 2s.

However, When I run:

import numpy as np
a = np.ones(3)
b = torch.from_numpy(a)
a = a+1
print(a)
print(b)

b remains as 1s while a has been updated to 2s.

Is this an expected behaviour?

Output:
Yes, as @hpaulj pointed out in his comment, the operation

a = a + 1

creates copy of the original array a and adds 1 using broadcasting. And after addition, since we assign it to a, so a gets updated to the result of addition operation. But, b still shares the memory of the original array a (i.e. the array a that was created before updation.)

So, we see result like this:

In [75]: a = np.ones(3)
    ...: b = torch.from_numpy(a)
    ...: a = a+1     # <========= creates copy of `a` and modifies it
    ...: print(a)
    ...: print(b)
    ...:
[ 2.  2.  2.]

 1
 1
 1
[torch.DoubleTensor of size 3]

But, see what happens when you rather do:

In [72]: a = np.ones(3)
    ...: b = torch.from_numpy(a)
    ...: a += 1      # <========== in-place modification of `a`
    ...: print(a)
    ...: print(b)
    ...:

[ 2.  2.  2.]

 2
 2
 2
[torch.DoubleTensor of size 3]

Observe how the += operation is making modification to the original array in-place whereas somearr = somearr + 1 creates copy of the array somearray and then makes modification to it.


Instruction: RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)

Input:
Im using a Pytorch Unet model to which i am feeding in a image as input and along with that i am feeding the label as the input image mask and traning the dataset on it.
The Unet model i have picked up from somewhere else, and i am using the cross-entropy loss as a loss function but i get this dimension out of range error,
RuntimeError
Traceback (most recent call last)
<ipython-input-358-fa0ef49a43ae> in <module>()
     16 for epoch in range(0, num_epochs):
     17     # train for one epoch
---> 18     curr_loss = train(train_loader, model, criterion, epoch, num_epochs)
     19
     20     # store best loss and save a model checkpoint

<ipython-input-356-1bd6c6c281fb> in train(train_loader, model, criterion, epoch, num_epochs)
     16         # measure loss
     17         print (outputs.size(),labels.size())
---> 18         loss = criterion(outputs, labels)
     19         losses.update(loss.data[0], images.size(0))
     20

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     _ _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

<ipython-input-355-db66abcdb074> in forward(self, logits, targets)
      9         probs_flat = probs.view(-1)
     10         targets_flat = targets.view(-1)
---> 11         return self.crossEntropy_loss(probs_flat, targets_flat)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in     __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
  --> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/loss.py in f orward(self, input, target)
    599         _assert_no_grad(target)
    600         return F.cross_entropy(input, target, self.weight, self.size_average,
--> 601                                self.ignore_index, self.reduce)
    602
    603

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     cross_entropy(input, target, weight, size_average, ignore_index, reduce)
   1138         >>> loss.backward()
   1139     """
-> 1140     return nll_loss(log_softmax(input, 1), target, weight, size_average, ignore_index, reduce)
   1141
   1142

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in     log_softmax(input, dim, _stacklevel)
    784     if dim is None:
    785         dim = _get_softmax_dim('log_softmax', input.dim(),      _stacklevel)
--> 786     return torch._C._nn.log_softmax(input, dim)
    787
    788

RuntimeError: dimension out of range (expected to be in range of [-1, 0], but got 1)

Part of my code looks like this
class crossEntropy(nn.Module):
    def __init__(self, weight = None, size_average = True):
        super(crossEntropy, self).__init__()
        self.crossEntropy_loss = nn.CrossEntropyLoss(weight, size_average)

    def forward(self, logits, targets):
        probs = F.sigmoid(logits)
        probs_flat = probs.view(-1)
        targets_flat = targets.view(-1)
        return self.crossEntropy_loss(probs_flat, targets_flat)

class UNet(nn.Module):
    def __init__(self, imsize):
        super(UNet, self).__init__()
        self.imsize = imsize

        self.activation = F.relu

        self.pool1 = nn.MaxPool2d(2)
        self.pool2 = nn.MaxPool2d(2)
        self.pool3 = nn.MaxPool2d(2)
        self.pool4 = nn.MaxPool2d(2)
        self.conv_block1_64 = UNetConvBlock(4, 64)
        self.conv_block64_128 = UNetConvBlock(64, 128)
        self.conv_block128_256 = UNetConvBlock(128, 256)
        self.conv_block256_512 = UNetConvBlock(256, 512)
        self.conv_block512_1024 = UNetConvBlock(512, 1024)

        self.up_block1024_512 = UNetUpBlock(1024, 512)
        self.up_block512_256 = UNetUpBlock(512, 256)
        self.up_block256_128 = UNetUpBlock(256, 128)
        self.up_block128_64 = UNetUpBlock(128, 64)

        self.last = nn.Conv2d(64, 2, 1)

    def forward(self, x):
        block1 = self.conv_block1_64(x)
        pool1 = self.pool1(block1)

        block2 = self.conv_block64_128(pool1)
        pool2 = self.pool2(block2)

        block3 = self.conv_block128_256(pool2)
        pool3 = self.pool3(block3)

        block4 = self.conv_block256_512(pool3)
        pool4 = self.pool4(block4)

        block5 = self.conv_block512_1024(pool4)

        up1 = self.up_block1024_512(block5, block4)

        up2 = self.up_block512_256(up1, block3)

        up3 = self.up_block256_128(up2, block2)

        up4 = self.up_block128_64(up3, block1)

        return F.log_softmax(self.last(up4))

Output:
According to your code:

probs_flat = probs.view(-1)
targets_flat = targets.view(-1)
return self.crossEntropy_loss(probs_flat, targets_flat)

You are giving two 1d tensor to nn.CrossEntropyLoss but according to documentation, it expects:

Input: (N,C) where C = number of classes
Target: (N) where each value is 0 <= targets[i] <= C-1
Output: scalar. If reduce is False, then (N) instead.

I believe that is the cause of the problem you are encountering.


Instruction: pytorch model.cuda() runtime error

Input:
I'm building a text classifier using pytorch, and got into some trouble with .cuda() method. I know that .cuda() moves all parameters into gpu so that the training procedure can be faster. However, error occurred in .cuda() method like this:

start_time  = time.time()

for model_type in ('lstm',):

    hyperparam_combinations = score_util.all_combination(hyperparam_dict[model_type].values())
     # for selecting best scoring model

    for test_idx, setting in enumerate(hyperparam_combinations):
        args = custom_dataset.list_to_args(setting,model_type=model_type)
        print(args)
        tsv = "test %d\ttrain_loss\ttrain_acc\ttrain_auc\tval_loss\tval_acc\tval_auc\n"%(test_idx) # tsv record
        avg_score = [] # cv_mean score

        ### 4 fold cross validation
        for cv_num,(train_iter,val_iter) in enumerate(cv_splits):

            ### model initiation
            model = model_dict[model_type](args)

            if args.emb_type is not None: # word embedding init
                emb = emb_dict[args.emb_type]
                emb = score_util.embedding_init(emb,tr_text_field,args.emb_type)
                model.embed.weight.data.copy_(emb)

            model.cuda()

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-ff6cfce73c10> in <module>()
     23                 model.embed.weight.data.copy_(emb)
     24
---> 25             model.cuda()
     26
     27             optimizer= torch.optim.Adam(model.parameters(),lr=args.lr)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in cuda(self, device_id)
    145                 copied to that device
    146         """
--> 147         return self._apply(lambda t: t.cuda(device_id))
    148
    149     def cpu(self, device_id=None):

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    116     def _apply(self, fn):
    117         for module in self.children():
--> 118             module._apply(fn)
    119
    120         for param in self._parameters.values():

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    122                 # Variables stored in modules are graph leaves, and we don't
    123                 # want to create copy nodes, so we have to unpack the data.
--> 124                 param.data = fn(param.data)
    125                 if param._grad is not None:
    126                     param._grad.data = fn(param._grad.data)

RuntimeError: Variable data has to be a tensor, but got torch.cuda.FloatTensor

These are error traceback and I can't see why this happens.
This code worked very well before I set epoch parameter to 1 to run some tests. I set epoch to 1000 again, but the problem lingers on.
Aren't torch.cuda.FloatTensor object also Tensors? Any help would be much appreciated.

my model looks like this :

class TR_LSTM(nn.Module):
    def __init__(self,args,
                 use_hidden_average=False,
                 pretrained_emb = None):

        super(TR_LSTM,self).__init__()
        # arguments
        self.emb_dim = args.embed_dim
        self.emb_num = args.embed_num
        self.num_hidden_unit = args.hidden_state_dim
        self.num_lstm_layer = args.num_lstm_layer
        self.use_hidden_average = use_hidden_average
        self.batch_size = args.batch_size

        # layers
        self.embed = nn.Embedding(self.emb_num, self.emb_dim)
        if pretrained_emb is not None:
            self.embed.weight.data.copy_(pretrained_emb)

        self.lstm_layer = nn.LSTM(self.emb_dim, self.num_hidden_unit, self.num_lstm_layer, batch_first = True)
        self.fc_layer = nn.Sequential(nn.Linear(self.num_hidden_unit,self.num_hidden_unit),
                                      nn.Linear(self.num_hidden_unit,2))

    def forward(self,x):
        x = self.embed(x) # batch * max_seq_len * emb_dim
        h_0,c_0 = self.init_hidden(x.size(0))
        x, (_, _) = self.lstm_layer(x, (h_0,c_0)) # batch * seq_len * hidden_unit_num

        if not self.use_hidden_average:
            x = x[:,x.size(1)-1,:]
            x = x.squeeze(1)
        else:
            x = x.mean(1).squeeze(1)
        x = self.fc_layer(x)

        return x

    def init_hidden(self,batch_size):
        h_0, c_0 = torch.zeros(self.num_lstm_layer,batch_size , self.num_hidden_unit),\
                   torch.zeros(self.num_lstm_layer,batch_size , self.num_hidden_unit)
        h_0, c_0 = h_0.cuda(), c_0.cuda()
        h_0_param, c_0_param = torch.nn.Parameter(h_0), torch.nn.Parameter(c_0)
        return h_0_param, c_0_param

Output:
model.cuda() is called inside your training/test loop, which is the problem. As the error message suggests, you repeatedly convert parameters(tensors) in your model to cuda, which is not the right way to convert model into cuda tensor.

model object should be created and cuda-ize outside the loop. Only training/test instances shall be convert to cuda tensor every time you feed your model. I also suggest you read examples code from pytorch document site.


Instruction: Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small

Input:
I am trying to train a U-Net which looks like this

`class UNet(nn.Module):
def __init__(self, imsize):
    super(UNet, self).__init__()
    self.imsize = imsize

    self.activation = F.relu
    self.pool1 = nn.MaxPool2d(2)
    self.pool2 = nn.MaxPool2d(2)
    self.pool3 = nn.MaxPool2d(2)
    self.pool4 = nn.MaxPool2d(2)
    self.conv_block1_64 = UNetConvBlock(4, 64)
    self.conv_block64_128 = UNetConvBlock(64, 128)
    self.conv_block128_256 = UNetConvBlock(128, 256)
    self.conv_block256_512 = UNetConvBlock(256, 512)
    self.conv_block512_1024 = UNetConvBlock(512, 1024)

    self.up_block1024_512 = UNetUpBlock(1024, 512)
    self.up_block512_256 = UNetUpBlock(512, 256)
    self.up_block256_128 = UNetUpBlock(256, 128)
    self.up_block128_64 = UNetUpBlock(128, 64)

    self.last = nn.Conv2d(64, 1, 1)`

The loss function i am using is

`class BCELoss2d(nn.Module):

def __init__(self, weight=None, size_average=True):
    super(BCELoss2d, self).__init__()
    self.bce_loss = nn.BCELoss(weight, size_average)

def forward(self, logits, targets):
    probs = F.sigmoid(logits)
    probs_flat = probs.view(-1)
    targets_flat = targets.view(-1)
    return self.bce_loss(probs_flat, targets_flat)`

The input image tensor is [1,1,68,68] and labels are also of the same shape

I get this error:

<ipython-input-72-270210759010> in forward(self, x)
 75
 76         block4 = self.conv_block256_512(pool3)
---> 77         pool4 = self.pool4(block4)
     78
  79         block5 = self.conv_block512_1024(pool4)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py in _    _call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/usr/local/lib/python3.5/dist-packages/torch/nn/modules/pooling.py in forward(self, input)
    141         return F.max_pool2d(input, self.kernel_size, self.stride,
    142                             self.padding, self.dilation, self.ceil_mode,
--> 143                             self.return_indices)
    144
    145     def __repr__(self):

/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py in max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)
    332     See :class:`~torch.nn.MaxPool2d` for details.
    333     """
--> 334     ret = torch._C._nn.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
    335     return ret if return_indices else ret[0]
    336

RuntimeError: Given input size: (128x1x1). Calculated output size: (128x0x0). Output size is too small at /pytorch/torch/lib/THCUNN/generic/SpatialDilatedMaxPooling.cu:69

I'm guessing I'm making a mistake in my channel size or pooling size but i'm not sure where exactly is the mistake.

Output: Your problem is that before the Pool4 your image has already reduced to a 1x1pixel size image. So you need to either feed an much larger image of size at least around double that (~134x134) or remove a pooling layer in your network.


Instruction: pytorch lstm tutorial initializing Variable

Input:
I am going through the pytorch tutorial for lstm and here's the code they use:

lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [autograd.Variable(torch.randn((1, 3)))
          for _ in range(5)]  # make a sequence of length 5

# initialize the hidden state.
hidden = (autograd.Variable(torch.randn(1, 1, 3)),
          autograd.Variable(torch.randn((1, 1, 3))))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)

For variable hidden, it initializes into a tuple and the result is:

(Variable containing:
(0 ,.,.) =
  0.4251 -1.2328 -0.6195
[torch.FloatTensor of size 1x1x3]
, Variable containing:
(0 ,.,.) =
  1.5133  1.9954 -0.6585
[torch.FloatTensor of size 1x1x3]
)

What I don't understand is

Is (0, ., .) an index? And shouldn't it initialize all three numbers since we said (torch.randn(1,1,3))?
What is the difference between torch.randn(1, 1, 3) and torch.randn((1,1,3))?

Output:
First to quickly answer number 2: They are identical. I don't know why they would do them differently.

Next, to answer question 1:

hidden is a tuple that contains two Variables that are essentially a 1 x 1 x 3 tensor.

Let's focus on what (0 ,.,.). If instead of a 1 x 1 x 3 tensor you had a 2 x 2 tensor, you could simply print out something like:

0.1 0.2
0.3 0.4

But it's kind of hard to represent 3 dimensional things on the screen. Even though it's a bit silly, having the aditional 1 at the beginning changes what would otherwise be a 2 dimensional tensor into a 3 dimensional one. So, instead Pytorch prints out "slices" of the tensor. In this case, you only have one "slice" which happens to be the zeroith slice. Thus you get the additional (0, ,.,.) instead of it just printing out

  0.4251 -1.2328 -0.6195

If instead the dimensions were 2 x 1 x 3 you could expect an output like:

(0 ,.,.) =
 -0.3027 -1.1077  0.4724

(1 ,.,.) =
  1.0063 -0.5936 -1.1589
[torch.FloatTensor of size 2x1x3]

And as you can see every element in the tensor is actually initialized.


Instruction: Pytorch transfer learning predictions

Input:
I have been following the pytorch transfer learning tutorial,and following the tutorial to my own dataset i have arrived at this

model_conv = train_model(model_conv, criterion, optimizer_conv,
                     exp_lr_scheduler, num_epochs=25)

Epoch 1/1
......
......
......
Epoch 24/24
train Loss: 0.8674 Acc: 0.5784
val Loss: 1.2930 Acc: 0.4583
Training complete in 43m 24s
Best val Acc: 0.486111

The tutorial ends here,but im not exaclty sure how do i get predictions for my new images?

Any suggestions would be really helpful.
Thanks in advance.

Output:
Simply feed the new image (in the same format as the images in your training dataset were) to the model:

labels = model_conv(new_images)


Instruction: How to setup pytorch in google-cloud-ml

Input:
I try to throw job with Pytorch code in google-cloud-ml.
so I code the "setup.py" file. And add option "install_requires"

"setup.py"

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl','torchvision']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My keras trainer application package.'
)

and throw the job to the google-cloud-ml, but it doesn't work

with error message

{
 insertId:  "3m78xtf9czd0u"
 jsonPayload: {
  created:  1516845879.49039
  levelname:  "ERROR"
  lineno:  829
  message:  "Command '['pip', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', u'trainer-0.1.tar.gz']' returned non-zero exit status 1"
  pathname:  "/runcloudml.py"
 }
 labels: {
  compute.googleapis.com/resource_id:  "6637909247101536087"
  compute.googleapis.com/resource_name:  "cmle-training-master-5502b52646-0-ql9ds"
  compute.googleapis.com/zone:  "us-central1-c"
  ml.googleapis.com/job_id:  "run_ml_engine_pytorch_test_20180125_015752"
  ml.googleapis.com/job_id/log_area:  "root"
  ml.googleapis.com/task_name:  "master-replica-0"
  ml.googleapis.com/trial_id:  ""
 }
 logName:  "projects/exem-191100/logs/master-replica-0"
 receiveTimestamp:  "2018-01-25T02:04:55.421517460Z"
 resource: {
  labels: {…}
  type:  "ml_job"
 }
 severity:  "ERROR"
 timestamp:  "2018-01-25T02:04:39.490387916Z"
}

====================================================================

See detailed message here

so how can i use pytorch in google cloud ml engine?

Output:
i find solution about setting up PYTORCH in google-cloud-ml

first
you have to get a .whl file about pytorch and store it to google storage bucket.
and you will get the link for bucket link.

gs://bucketname/directory/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl

the .whl file is depend on your python version or cuda version....

second
you write the command line and setup.py because you have to set up the google-cloud-ml setting.
related link is this  submit_job_to_ml-engine
you write the setup.py file to describe your setup.
the related link is this write_setup.py_file

this is my command code and setup.py file

=====================================================================
 "command"

#commandline code
JOB_NAME="run_ml_engine_pytorch_test_$(date +%Y%m%d_%H%M%S)"
REGION=us-central1
OUTPUT_PATH=gs://yourbucket
gcloud ml-engine jobs submit training $JOB_NAME \
    --job-dir $OUTPUT_PATH \
    --runtime-version 1.4 \
    --module-name models.pytorch_test \
    --package-path models/ \
    --packages gs://yourbucket/directory/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl \
    --region $REGION \
    -- \
    --verbosity DEBUG

=====================================================================
"setup.py"

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['torchvision']
setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My pytorch trainer application package.'
)

=====================================================================

third
if you have experience submitting job to the ml-engine.
you might know the file structure about submitting ml-engine
packaging_training_model.
you have to follow above link and know how to pack files.


Instruction: PyTorch: training with GPU gives worse error than training the same thing with CPU

Input:
I have a next step prediction model on times series which is simply a GRU with a fully-connected layer on top of it. When I train it using CPU after 50 epochs I get a loss of 0.10 but when I train it with GPU the loss is 0.15 after 50 epochs. Doing more epochs doesnt really lower the losses in either cases.

Why is performance after training on CPU better than GPU?

I have tried changing the random seeds for both data and model, and these results are independent of the random seeds.

I have:

Python 3.6.2

PyTorch 0.3.0

CUDNN_MAJOR 7

CUDNN_MINOR 0

CUDNN_PATCHLEVEL 5

Edit:

I also use PyTorch's weight normalizaton torch.nn.utils.weight_norm on the GRU and on the fully-connected layer.

Output:
After trying many things I think I found the problem. Apparently the CUDNN libraries are sub-optimal in PyTorch. I don't know if it is a bug in PyTorch or a bug in CUDNN but doing

torch.backends.cudnn.enabled = False

solves the problem. With the above line, training with GPU or CPU gives the same loss at the same epoch.

Edit:

It seems that it is the interaction of weight normalization and CUDNN which results in things going wrong. If I remove weight normalization it works. If I remove CUDNN it works. It seems that only in combination they do not work in PyTorch.


Instruction: unknown resampling filter error when trying to create my own dataset with pytorch

Input:
I am trying to create a CNN implemented with data augmentation in pytorch to classify dogs and cats. The issue that I am having is that when I try to input my dataset and enumerate through it I keep getting this error:

  Traceback (most recent call last):

  File "<ipython-input-55-6337e0536bae>", line 75, in <module>
    for i, (inputs, labels) in enumerate(trainloader):

  File "/usr/local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 188, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])

  File "/usr/local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 188, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])

  File "/usr/local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 124, in __getitem__
    img = self.transform(img)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 42, in __call__
    img = t(img)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 147, in __call__
    return F.resize(img, self.size, self.interpolation)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 197, in resize
    return img.resize((ow, oh), interpolation)

  File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 1724, in resize
    raise ValueError("unknown resampling filter")

ValueError: unknown resampling filter

and I really dont know whats wrong with my code. I have provided the code below:

# Creating the CNN

# Importing the libraries
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torchvision
from torchvision import transforms

#Creating the CNN Model
class CNN(nn.Module):
    def __init__(self, nb_outputs):
        super(CNN, self).__init__() #activates the inheritance and allows the use of all the tools in the nn.Module
        #making the 3 convolutional layers that will be used in the convolutional neural network
        self.convolution1 = nn.Conv2d(in_channels = 1, out_channels = 32, kernel_size = 5) #kernal_size -> the deminson of the feature detector e.g kernel_size = 5 => feature detector of size 5x5
        self.convolution2 = nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 2)
        #making 2 full connections one to connect the inputs of the ANN to the hidden layer and another to connect the hidden layer to the outputs of the ANN
        self.fc1 = nn.Linear(in_features = self.count_neurons((1, 64,64)), out_features = 40)
        self.fc2 = nn.Linear(in_features = 40, out_features = nb_outputs)

    def count_neurons(self, image_dim):
        x = Variable(torch.rand(1, *image_dim)) #this variable repersents a fake image to allow us to compute the number of neruons
        #in order to pass the elements of the tuple image_dim into our function as a list of arguments we need to add a * before image_dim
        #since x will be going into our neural network we need to convert it into a torch variable using the Variable() function
        x = F.relu(F.max_pool2d(self.convolution1(x), 3, 2)) #first we apply the convolution to x then apply max_pooling to the convolutional fake images and then activate all the neurons in the pooling layer
        x = F.relu(F.max_pool2d(self.convolution2(x), 3, 2)) #the signals are now propragated up to the thrid convoulational layer
        #Now to flatten x to obtain the number of neurons in the flattening layer
        return x.data.view(1, -1).size(1) #this will flatten x into a huge vector and returns the size of the vector, that size repersents the number of neurons that will be inputted into the ANN
        #even though x is not a real image from the game since the size of the flattened vector only depends on the dimention of the inputted image we can just set x to have the same dimentions as the image

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.convolution1(x), 3, 2)) #first we apply the convolution to x then apply max_pooling to the convolutional fake images and then activate all the neurons in the pooling layer
        x = F.relu(F.max_pool2d(self.convolution2(x), 3, 2))
        #flattening layer of the CNN
        x = x.view(x.size(0), -1)
        #x is now the inputs to the ANN
        x = F.relu(self.fc1(x)) #we propagte the signals from the flatten layer to the full connected layer and activate the neruons by breaking the linearilty with the relu function
        x = F.sigmoid(self.fc2(x))
        #x is now the output neurons of the ANN
        return x

train_tf = transforms.Compose([transforms.RandomHorizontalFlip(),
                               transforms.Resize(64,64),
                               transforms.RandomRotation(20),
                               transforms.RandomGrayscale(.2),
                               transforms.ToTensor()])

test_tf = transforms.Compose([transforms.Resize(64,64),
                              transforms.ToTensor()])

training_set = torchvision.datasets.ImageFolder(root = './dataset/training_set',
                                                transform = train_tf)
test_set = torchvision.datasets.ImageFolder(root = './dataset/test_set',
                                                transform = transforms.Compose([transforms.Resize(64,64),
                              transforms.ToTensor()]) )

trainloader = torch.utils.data.DataLoader(training_set, batch_size=32,
                                              shuffle=True, num_workers=0)
testloader = torch.utils.data.DataLoader(test_set, batch_size= 32,
                                             shuffle=False, num_workers=0)

#training the model
cnn = CNN(1)
cnn.train()
loss = nn.BCELoss()
optimizer = optim.Adam(cnn.parameters(), lr = 0.001) #the optimizer => Adam optimizer
nb_epochs = 25

for epoch in range(nb_epochs):
    train_loss = 0.0
    train_acc = 0.0
    total = 0.0
    for i, (inputs, labels) in enumerate(trainloader):
        inputs, labels = Variable(inputs), Variable(labels)
        cnn.zero_grad()
        outputs = cnn(inputs)
        loss_error = loss(outputs, labels)
        optimizer.step()

        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        train_loss += loss_error.data[0]
        train_acc += (pred == labels).sum()

    train_loss = train_loss/len(training_loader)
    train_acc = train_acc/total
    print('Epoch: %d, loss: %.4f, accuracy: %.4f' %(epoch+1, train_loss, train_acc))

The folder arrangement for the code is /dataset/training_set and inside the training_set folder are two more folders one for all the cat images and the other for all the dog images. Each image is name either dog.xxxx.jpg or cat.xxxx.jpg, where the xxxx represents the number so for the first cat image it would be cat.1.jpg up to cat.4000.jpg. This is the same format for the test_set folder. The number of training images is 8000 and the number of test images is 2000. If anyone can point out my error I would greatly appreciate it.

Thank you

Output:
Try to set the desired size in transforms.Resize as a tuple:

transforms.Resize((64, 64))

PIL is using the second argument (in your case 64) as the interpolation method.


Instruction: Problems with PyTorch MLP when training the MNIST dataset retrieved from Keras

Input:
I have finished a PyTorch MLP model for the MNIST dataset, but got two different results: 0.90+ accuracy when using MNIST dataset from PyTorch, but ~0.10 accuracy when using MNIST dataset from Keras.
Below is my code with dependency: PyTorch 0.3.0.post4, keras 2.1.3, tensorflow backend 1.4.1 gpu version.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import torch as pt
import torchvision as ptv
from keras.datasets import mnist
from torch.nn import functional as F
from torch.utils.data import Dataset, DataLoader

# training data from PyTorch
train_set = ptv.datasets.MNIST("./data/mnist/train", train=True, transform=ptv.transforms.ToTensor(), download=True)
test_set = ptv.datasets.MNIST("./data/mnist/test", train=False, transform=ptv.transforms.ToTensor(), download=True)
train_dataset = DataLoader(train_set, batch_size=100, shuffle=True)
test_dataset = DataLoader(test_set, batch_size=10000, shuffle=True)

class MLP(pt.nn.Module):
    """The Multi-layer perceptron"""
    def __init__(self):
        super(MLP, self).__init__()
        self.fc1 = pt.nn.Linear(784, 512)
        self.fc2 = pt.nn.Linear(512, 128)
        self.fc3 = pt.nn.Linear(128, 10)
        self.use_gpu = True

    def forward(self, din):
        din = din.view(-1, 28 * 28)
        dout = F.relu(self.fc1(din))
        dout = F.relu(self.fc2(dout))
        # return F.softmax(self.fc3(dout))
        return self.fc3(dout)

model = MLP().cuda()
print(model)

# loss func and optim
optimizer = pt.optim.SGD(model.parameters(), lr=1)
criterion = pt.nn.CrossEntropyLoss().cuda()

def evaluate_acc(pred, label):
    pred = pred.cpu().data.numpy()
    label = label.cpu().data.numpy()
    test_np = (np.argmax(pred, 1) == label)
    test_np = np.float32(test_np)
    return np.mean(test_np)

def evaluate_loader(loader):
    print("evaluating ...")
    accurarcy_list = []
    for i, (inputs, labels) in enumerate(loader):
        inputs = pt.autograd.Variable(inputs).cuda()
        labels = pt.autograd.Variable(labels).cuda()
        outputs = model(inputs)
        accurarcy_list.append(evaluate_acc(outputs, labels))
    print(sum(accurarcy_list) / len(accurarcy_list))

def training(d, epochs):
    for x in range(epochs):
        for i, data in enumerate(d):

            optimizer.zero_grad()

            (inputs, labels) = data
            inputs = pt.autograd.Variable(inputs).cuda()
            labels = pt.autograd.Variable(labels).cuda()

            outputs = model(inputs)

            loss = criterion(outputs, labels)
            loss.backward()

            optimizer.step()

            if i % 200 == 0:
                print(i, ":", evaluate_acc(outputs, labels))

# Training MLP for 4 epochs with MNIST dataset from PyTorch
training(train_dataset, 4)

# The accuracy is ~0.96.
evaluate_loader(test_dataset)

print("###########################################################")

def load_mnist():
    (x, y), (x_test, y_test) = mnist.load_data()
    x = x.reshape((-1, 1, 28, 28)).astype(np.float32)
    x_test = x_test.reshape((-1, 1, 28, 28)).astype(np.float32)
    y = y.astype(np.int64)
    y_test = y_test.astype(np.int64)
    print("x.shape", x.shape, "y.shape", y.shape,
          "\nx_test.shape", x_test.shape, "y_test.shape", y_test.shape,
          )

    return x, y, x_test, y_test

class TMPDataset(Dataset):
    """Dateset for loading Keras MNIST dataset."""
    def __init__(self, a, b):
        self.x = a
        self.y = b

    def __getitem__(self, item):
        return self.x[item], self.y[item]

    def __len__(self):
        return len(self.y)

x_train, y_train, x_test, y_test = load_mnist()
# Create dataloader for MNIST dataset from Keras.
test_loader = DataLoader(TMPDataset(x_test, y_test), num_workers=1, batch_size=10000)
train_loader = DataLoader(TMPDataset(x_train, y_train), shuffle=True, batch_size=100)

# Evaluate the performance of MLP trained on PyTorch dataset and the accurach is ~0.96.
evaluate_loader(test_loader)
evaluate_loader(train_loader)

model = MLP().cuda()
print(model)

optimizer = pt.optim.SGD(model.parameters(), lr=1)
criterion = pt.nn.CrossEntropyLoss().cuda()

# Train now on MNIST dataset from Keras.
training(train_loader, 4)

# Evaluate the trianed model on MNIST dataset from Keras and result in performance ~0.10...
evaluate_loader(test_loader)
evaluate_loader(train_loader)

I had checked some samples from Keras MNIST dataset and found no error.
I am wondering what is wrong with the datasets?
The code can run without error, run it to see the results.

Output:
The MNIST data coming from Keras are not normalized; following the Keras MNIST MLP example, you should do it manually, i.e. you should include the following in your load_data() function:

x /= 255
x_test /= 255

Not sure about PyTorch, but it would seem that the MNIST data from their own utility functions come already normalized (as is the case with Tensorflow - see the third point in my answer here).

A 10% accuracy (i.e. equivalent to random guessing) in case of not-normalized input data is perfectly consistent.


Instruction: How to find the index of a tensor in a list?

Input:
I want to find the index of the smallest tensor (by some key function) in a list li.
So I did min and afterwards li.index(min_el). My MWE suggests that somehow tensors don't work with index.

import torch
li=[torch.ones(1,1), torch.zeros(2,2)]
li.index(li[0])
0
li.index(li[1])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../local/lib/python2.7/site-packages/torch/tensor.py", line 330, in __eq__
    return self.eq(other)
RuntimeError: inconsistent tensor size at /b/wheel/pytorch-src/torch/lib/TH/generic/THTensorMath.c:2679

I can of course make my own index function that first checks for size and then element wise. E.g.

def index(list, element):
    for i,el in enumerate(list):
        if el.size() == element.size():
            diff = el - element
            if (1- diff.byte()).all():
               return i
    return -1

I was just wondering, why doesn't index work?
Is there maybe a smart way of doing this not by hand that I'm missing?

Output:
You could directly find the index by using enumerate and a key function that operates on the second element of each tuple. For instance, if your key function compares the first element of the each tensor, you could use

ix, _ = min(enumerate(li), key=lambda x: x[1][0, 0])

I think the reason why index succeeds for the first element is that Python probably does something that is equivalent to x is value or x == value, where x is the current element in the list. Since value is value, the failing equality comparison never happens. This is why li.index(li[0]) works but this fails:

y = torch.ones(1,1)
li.index(y)


Instruction: Loss is increasing from first epoch itself

Input:
I am training my siamese network for nlp. I have used lstm in it. and BCELoss. My loss is increasing from the first epoch. The first 36 epoch loss is
error after  0 is
 272.4357
[torch.FloatTensor of size 1]

error after  1 is
 271.8972
[torch.FloatTensor of size 1]

error after  2 is
 271.5598
[torch.FloatTensor of size 1]

error after  3 is
 271.6979
[torch.FloatTensor of size 1]

error after  4 is
 271.7315
[torch.FloatTensor of size 1]

error after  5 is
 272.3965
[torch.FloatTensor of size 1]

error after  6 is
 273.3982
[torch.FloatTensor of size 1]

error after  7 is
 275.1197
[torch.FloatTensor of size 1]

error after  8 is
 275.8228
[torch.FloatTensor of size 1]

error after  9 is
 278.3311
[torch.FloatTensor of size 1]

error after  10 is
 277.1054
[torch.FloatTensor of size 1]

error after  11 is
 277.8418
[torch.FloatTensor of size 1]

error after  12 is
 279.0189
[torch.FloatTensor of size 1]

error after  13 is
 278.4090
[torch.FloatTensor of size 1]

error after  14 is
 281.8813
[torch.FloatTensor of size 1]

error after  15 is
 283.4077
[torch.FloatTensor of size 1]

error after  16 is
 286.3093
[torch.FloatTensor of size 1]

error after  17 is
 287.6292
[torch.FloatTensor of size 1]

error after  18 is
 297.2318
[torch.FloatTensor of size 1]

error after  19 is
 307.4176
[torch.FloatTensor of size 1]

error after  20 is
 304.6649
[torch.FloatTensor of size 1]

error after  21 is
 328.9772
[torch.FloatTensor of size 1]

error after  22 is
 300.0669
[torch.FloatTensor of size 1]

error after  23 is
 292.3902
[torch.FloatTensor of size 1]

error after  24 is
 300.8633
[torch.FloatTensor of size 1]

error after  25 is
 305.1822
[torch.FloatTensor of size 1]

error after  26 is
 333.9984
[torch.FloatTensor of size 1]

error after  27 is
 346.2062
[torch.FloatTensor of size 1]

error after  28 is
 354.6148
[torch.FloatTensor of size 1]

error after  29 is
 341.3568
[torch.FloatTensor of size 1]

error after  30 is
 369.7580
[torch.FloatTensor of size 1]

error after  31 is
 366.1615
[torch.FloatTensor of size 1]

error after  32 is
 368.2455
[torch.FloatTensor of size 1]

error after  33 is
 391.4102
[torch.FloatTensor of size 1]

error after  34 is
 394.3190
[torch.FloatTensor of size 1]

error after  35 is
 401.0990
[torch.FloatTensor of size 1]

error after  36 is
 422.3723
[torch.FloatTensor of size 1]

Output:
Probably your learning rate is too high. Try decreasing your learning rate. A too large learning rate is the most common reason for loss increasing from the first epoch.

Also your loss is very high. It is unusual to have such a high lost. You probably have a sum in your loss function, it might be wiser to replace that sum with a mean. While this makes no difference if you use the Adam optimizer, if you use simple SGD with or without momentum using a sum instead of a mean, means that you will need to tune your learning rate differently if the dimensions (or the length of your sequence processed by your lstm) of your system changes.


Instruction: Pytorch Test Loss increases while accuracy increases

Input:
I am trying to implement
End to End Memory Network using Pytorch and BabI dataset.
The network architecture is :

MemN2N (
(embedding_A): Embedding(85, 120, padding_idx=0)
(embedding_B): Embedding(85, 120, padding_idx=0)
(embedding_C): Embedding(85, 120, padding_idx=0)
(match): Softmax ()
)

85 is the vocabulary size and 120 is embedding size.
Loss function is cross entropy and optimizer is RmsProp.
 The results is

Epoch    Train Loss  Test Loss  TrainAcc TestAcc
10       0.608         11.213       1.0     0.99
20       0.027         11.193       1.0     0.99
30       0.0017        11.740       1.0     0.99
40       0.0006        12.190       1.0     0.99
50       5.597e-05     12.319       1.0     0.99
60       3.366-05      12.379       1.0     0.99
70       2.72e-05      12.361       1.0     0.99
80       2.64e-05      12.333       1.0     0.99
90       2.63e-05      12.329       1.0     0.99
100      2.63e-05      12.329       1.0     0.99
110      2.63e-05      12.329       1.0     0.99
120      2.63e-05      12.329       1.0     0.99
Final TrainAcc TestAcc
    1.0     0.999

I know the accuracy is good, but I wonder the behaviour of the test loss.
Since training loss decreases, the test loss increases.
The calculation is the same for each loss value. Shouldn't it decrease too? I used Task 1 to display, but the behaviour is the same with other tasks.

Do you have any idea about this behaviour?

Output:
When training loss continues to decrease but test loss starts to increase, that is the moment you are starting to overfit, that means that your network weights are fitting the data you are training on better and better, but this extra fitting will not generalize to new unseen data. This means that that is the moment you should stop training.

You are embedding 80 words in 120 dimensions, so you have no information bottle neck at all, you have much too many dimensions for only 80 words. You have so many free parameters you can fit anything, even noise. Try changing 120 for 10 and probably you will not overfit anymore. If you try using 2 dimensions instead of 120, then you will probably underfit.

Overfitting: When your model has enough capacity to fit particularities of your training data which doesn't generalize to new data from the same distribution.

Underfitting: When your model does not have enough capacity to fit even your training data (you cannot bring your training loss "close" to zero).

In your case, I am guessing that your model becomes over-confident on your training data (output probabilities too close to 1 or 0) which is justified in the case of the training data but which is too confident for your test data (or any other data you didn't train on).


Instruction: How to calculate cost for softmax regression with pytorch

Input:
I would to calculate the cost for the softmax regression.
The cost function to calculate is given at the bottom of the page.

For numpy I can get the cost as follows:

"""
X.shape = 2,300 # floats
y.shape = 300,  # integers
W.shape = 2,3
b.shape = 3,1
"""
import numpy as np
np.random.seed(100)

# Data and labels
X = np.random.randn(300,2)
y = np.ones(300)
y[0:100] = 0
y[200:300] = 2
y = y.astype(np.int)

# weights and bias
W = np.random.randn(2,3)
b = np.random.randn(3)

N = X.shape[0]
scores = np.dot(X, W) + b
hyp = np.exp(scores-np.max(scores, axis=0, keepdims=True))
probs = hyp / np.sum(hyp, axis = 0)
logprobs = np.log(probs[range(N),y])
cost_data = -1/N * np.sum(logprobs)

print("hyp.shape = {}".format(hyp.shape)) # hyp.shape = (300, 3)
print(cost_data)

But, when I tried torch I could not get this.
So far I have got this:

"""
X.shape = 2,300 # floats
y.shape = 300,  # integers
W.shape = 2,3
b.shape = 3,1
"""
import numpy as np
import torch
from torch.autograd import Variable
np.random.seed(100)

# Data and labels
X = np.random.randn(300,2)
y = np.ones(300)
y[0:100] = 0
y[200:300] = 2
y = y.astype(np.int)

X = Variable(torch.from_numpy(X),requires_grad=True).type(torch.FloatTensor)
y = Variable(torch.from_numpy(y),requires_grad=True).type(torch.LongTensor)

# weights and bias
W = Variable(torch.randn(2,3),requires_grad=True)
b = Variable(torch.randn(3),requires_grad=True)

N = X.shape[0]
scores = torch.mm(X, W) + b
hyp = torch.exp(scores - torch.max(scores))
probs = hyp / torch.sum(hyp)
correct_probs = probs[range(N),y] # got problem HERE
# logprobs = np.log(correct_probs)
# cost_data = -1/N * torch.sum(logprobs)

# print(cost_data)

I got problem calculating the correct probabilities for the classes.

How can we solve this problem and get the correct cost value.

The cost function to calculate is given below:

Output:
Your problem is that you cannot use range(N) with pytorch, use the slice 0:N instead:

hyp = torch.exp(scores - torch.max(scores))
probs = hyp / torch.sum(hyp)
correct_probs = probs[0:N,y] # problem solved
logprobs = torch.log(correct_probs)
cost_data = -1/N * torch.sum(logprobs)

Another point is that your labels y do not require gradients, you would better have:

y = Variable(torch.from_numpy(y),requires_grad=False).type(torch.LongTensor)


Instruction: How to take the average of the weights of two networks?

Input: Suppose in PyTorch I have model1 and model2 which have the same architecture. They were further trained on same data or one model is an earlier version of the othter, but it is not technically relevant for the question. Now I want to set the weights of model to be the average of the weights of model1 and model2. How would I do that in PyTorch?

Output:
beta = 0.5 #The interpolation parameter
params1 = model1.named_parameters()
params2 = model2.named_parameters()

dict_params2 = dict(params2)

for name1, param1 in params1:
    if name1 in dict_params2:
        dict_params2[name1].data.copy_(beta*param1.data + (1-beta)*dict_params2[name1].data)

model.load_state_dict(dict_params2)

Taken from pytorch forums. You could grab the parameters, transform and load them back but make sure the dimensions match.

Also I would be really interested in knowing about your findings with these..


Instruction: ImportError when importing pytorch

Input:
When I try importing pytorch inside a jupyter notebook i get the following error:

  ImportError: dlopen: cannot load any more object with static TLS
  pytorch

import torch

No error when i import torch from command line (not jupyter notebook)

Output:
For reasons i cannot yet explain, i found out that this is happening only when i install Keras too in the same virtual environment, and it goes away when i uninstall Keras.

So the solution was

pip uninstall keras


Instruction: pytorch variable index lost one dimension

Input:
I am going to get every horizontal tensor in a variable but I got one dimension lost.
This is my code:
import torch
from torch.autograd import Variable
t = torch.rand((2,2,4))
x = Variable(t)
print(x)
shape = x.size()
for i in range(shape[0]):
    for j in range(shape[1]):
        print(x[i,j])

and the output is :
Variable containing:
(0 ,.,.) =
  0.6717  0.8216  0.5100  0.9106
  0.3280  0.8182  0.5781  0.3919

(1 ,.,.) =
  0.8823  0.4237  0.6620  0.0817
  0.5781  0.4187  0.3769  0.0498
[torch.FloatTensor of size 2x2x4]

Variable containing:
 0.6717
 0.8216
 0.5100
 0.9106
[torch.FloatTensor of size 4]

Variable containing:
 0.3280
 0.8182
 0.5781
 0.3919
[torch.FloatTensor of size 4]

Variable containing:
 0.8823
 0.4237
 0.6620
 0.0817
[torch.FloatTensor of size 4]

Variable containing:
 0.5781
 0.4187
 0.3769
 0.0498
[torch.FloatTensor of size 4]

and how can I get [torch.FloatTensor of size 1x4]?

Output:
In your case, x is a 2x2x4 tensor. So when you do x[0] you obtain the 2x4 tensor which is in the first row. And if you do x[i,j] you obtain the 4 dimensional vector in position (i,j). If you want to retain one of the dimensions you can either use a slice: x[i,j:j+1] OR reshape the tensor: x[i,j].view(1,4). Thus your code would look like:

import torch
from torch.autograd import Variable
t = torch.rand((2,2,4))
x = Variable(t)
print(x)
shape = x.size()
for i in range(shape[0]):
    for j in range(shape[1]):
        print(x[i,j:j+1])

or

import torch
from torch.autograd import Variable
t = torch.rand((2,2,4))
x = Variable(t)
print(x)
shape = x.size()
for i in range(shape[0]):
    for j in range(shape[1]):
        print(x[i,j].view(1,4)

Will give you the desired result.

Edit:

Yes, or as mentioned in the answer by nnnmmm,  torch.unsqueeze(x[i, j], 0) also works as it adds a dimension of size 1 in the 0th position.


Instruction: How can I find the equivalent 'batch_size' used in Keras from this Pytorch code?

Input:
I am using Pytorch code from github

I am trying to port this over to Keras.
In particular, Keras uses model.fit for training the Neural net and has a batch_size parameter. I am trying to set this but cannot determine it in the Pytorch script linked above.

In the script, there is a function called sliding_window in box 4 where there is a parameter called step. I am not sure if this is the equivalent setting of batch_size.

Also, I am looking into how to set the learning rate scheduler from the box 11 in the Pytorch code:

base_lr = 0.01
params_dict = dict(net.named_parameters())
params = []
for key, value in params_dict.items():
    if '_D' in key:
        # Decoder weights are trained at the nominal learning rate
        params += [{'params':[value],'lr': base_lr}]
    else:
        # Encoder weights are trained at lr / 2 (we have VGG-16 weights as initialization)
        params += [{'params':[value],'lr': base_lr / 2}]

optimizer = optim.SGD(net.parameters(), lr=base_lr, momentum=0.9, weight_decay=0.0005)
# We define the scheduler
scheduler = optim.lr_scheduler.MultiStepLR(optimizer, [25, 35, 45], gamma=0.1)

I have always used the default learning rate from Keras. Any insight on how to go about converting this scheduler into Keras code  would also be appreciated.

Output:
Below find an example on how to write a Learning Rate scheduler in Keras:

from keras.callbacks import Callback
from keras import backed as K

class LRSchedule(Callback):
    def __init__(self, schedule):
        super(LRSchedule, self).__init__()
        self.schedule = schedule

    def on_train_begin(self, logs = {}):
        self.epoch_counter = 0
        self.schedule_index = 0

    def on_epoch_end(self, epoch, logs = {}):
        self.epoch_counter += 1

        if len(self.schedule)  > self.schedule_index + 1:
            next_epoch = self.schedule[self.schedule_index + 1]
            if self.epoch_counter == next_epoch:
                K.set_value(self.model.optimizer.lr, self.model.optimizer.lr / 2.0)
                self.schedule_index += 1

In your code, you will call the callback in this way:

lr_scheduler = LRSchedule([25, 35, 45])
model.fit_generator(..., callbacks = [lr_scheduler])

Notice that this scheduler, when an epoch is reached, sets the learning to a lower value dividing by 2.  Modifying it to more fancy schedule policies is trivial.


Instruction: Kaggle Pytorch run length encoding

Input:
Im working on the DSB problem in pytorch,I have my predictions but im not sure how to get those into the run length encoding format that is required for the submission,In short this is what it is

===================================================================

In order to reduce the submission file size, our metric uses run-length encoding on the pixel values. Instead of submitting an exhaustive list of indices for your segmentation, you will submit pairs of values that contain a start position and a run length. E.g. '1 3' implies starting at pixel 1 and running a total of 3 pixels (1,2,3).

The competition format requires a space delimited list of pairs. For example, '1 3 10 5' implies pixels 1,2,3,10,11,12,13,14 are to be included in the mask. The pixels are one-indexed and numbered from top to bottom, then left to right: 1 is pixel (1,1), 2 is pixel (2,1), etc.

===================================================================

I get the predictions like this

model = model.eval()
for data in testdataloader:
    data = t.autograd.Variable(data.cuda(device = 1))
    pred = model(data)

but now that i have my prediction im not sure how to move forward,
I found this script online,but im not sure how to modify this for my use case

def rle_encoding(x):
    dots = np.where(x.T.flatten() == 1)[0]
    run_lengths = []
    prev = -2
    for b in dots:
        if (b>prev+1): run_lengths.extend((b + 1, 0))
        run_lengths[-1] += 1
        prev = b
    return run_lengths

def prob_to_rles(x, cutoff=0.5):
    lab_img = label(x > cutoff)
    for i in range(1, lab_img.max() + 1):
        yield rle_encoding(lab_img == i)

Any suggestions on how may i get started or how do i modify this would be really helpfull!

Output:
Try if something like that works

def rle_encode(image):
    """
    receives a masked image and encodes it to RLE
    :param mask_image:
    :return: string corresponding to the rle of the input image
    """
    pixels = image.flatten()
    # We avoid issues with '1' at the start or end (at the corners of
    # the original image) by setting those pixels to '0' explicitly.
    # We do not expect these to be non-zero for an accurate mask,
    # so this should not harm the score.
    pixels[0] = 0
    pixels[-1] = 0
    runs = np.where(pixels[1:] != pixels[:-1])[0] + 2
    runs[1::2] = runs[1::2] - runs[:-1:2]
    return ' '.join(str(x) for x in runs)

# create the file path
f = open(args.submit_dir + args.arch + '.csv', 'w')

# add the header of the csv file
f.write('img,rle_mask\n')

# NOTE: put this part in the test loop to generate the output file
f.write(image_name ',' + data_utils.mask_to_RLEstring(net_output.numpy()) + '\n')


Instruction: Where is the len function used in PyTorch Dataset?

Input:
I am looking to use the code from here .
However, I am looking at box 5, where there is the following function;

def __len__(self):
    # Default epoch size is 10 000 samples
    return 10000

I do not see  anywhere throughout this script where this function is being used.
Clarification on this would be appreciated.

Also, I want to determine the number of image patches used for training  this convolutional neural network. Is this len function linked to the number of patches?

Output:
This is a function of the Dataset class. The __len__() function specifies the size of the dataset. In your referenced code, in box 10, a dataset is initialized and passed to a DataLoader object:

train_set = ISPRS_dataset(train_ids, cache=CACHE)
train_loader = torch.utils.data.DataLoader(train_set,batch_size=BATCH_SIZE)

You see that in the DataLoader the dataset object is passed as well as the batch size. The DataLoader object then uses the __len__ function of the Dataset to create the batches. This happens in box 13, where it is iterated over the DataLoader.


Instruction: Result of slicing in an empty tensor

Input:
Can anyone help me to understand why this works:

lens = list(range(170,1,-1))
xs = Variable(torch.randn(169, 200, 1))
packed = torch.nn.utils.rnn.pack_padded_sequence(xs, lens, batch_first=True)

and this does not:

lens = [294, 289, 288, 282, 273, 270, 261, 260, 240, 235, 231, 228, 228, 227, 226, 226, 199, 195, 194, 192, 190, 189, 177, 176, 165, 165, 161, 156, 153, 149, 149, 142, 142, 137, 136, 136, 135, 134, 134, 132, 131, 129, 122, 121, 121, 114, 113, 113, 112, 110, 109, 108, 107, 107, 106, 105, 105, 103, 102, 100, 99, 99, 98, 96, 95, 93, 92, 91, 91, 90, 88, 88, 87, 79, 78, 78, 77, 76, 75, 74, 73, 72, 72, 71, 71, 71, 71, 69, 69, 69, 68, 68, 68, 68, 68, 68, 67, 67, 66, 66, 65, 65, 64, 64, 64, 63, 63, 61, 61, 61, 61, 60, 60, 59, 59, 59, 59, 57, 57, 57, 57, 57, 56, 56, 55, 55, 54, 54, 54, 54, 54, 53, 53, 52, 52, 52, 51, 51, 51, 51, 51, 50, 50, 50, 49, 49, 49, 48, 47, 47, 47, 47, 46, 46, 46, 45, 44, 44, 44, 44, 43, 42, 39, 38, 36, 30, 30, 25, 23]
xs = Variable(torch.randn(169, 200, 1))
packed = torch.nn.utils.rnn.pack_padded_sequence(xs, lens, batch_first=True)

I am getting a "ValueError: result of slicing is an empty tensor"

Output:
Answering my own question.

In fact it was answered by SimomW at the Pytorch forum: https://discuss.pytorch.org/t/result-of-slicing-is-an-empty-tensor/13306

It happens because the first len has values all <= 200, but the second has many > 200, which is the maximum seq_len a tensor of shape 169,200,1 can have.


Instruction: Simple Pytorch Example - Loss on training doesnt decrease

Input:
I am just starting to try and learn pytorch and am finding it frustrating regardless of how it is advertised :)

Here I am running a simple regression as an experiment but since the loss doesn't seem to be decreasing with each epoch (on the training) I must be doing something wrong -- either in training or how I am collecting the MSE?

from __future__ import division
import numpy as np
import matplotlib.pyplot as plt

import torch
import torch.utils.data as utils_data
from torch.autograd import Variable
from torch import optim, nn
from torch.utils.data import Dataset
import torch.nn.functional as F

from sklearn.datasets import load_boston

cuda=True

#regular old numpy
boston = load_boston()

x=boston.data
y=boston.target

x.shape

training_samples = utils_data.TensorDataset(x, y)
data_loader = utils_data.DataLoader(training_samples, batch_size=10)

len(data_loader) #number of batches in an epoch

#override this
class Net(nn.Module):
    def __init__(self):
         super(Net, self).__init__()

         #all the layers
         self.fc1   = nn.Linear(x.shape[1], 50)
         self.drop = nn.Dropout(p=0.2)
         self.fc2   = nn.Linear(50, 1)

    #
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.drop(x)
        x = self.fc2(x)

        return x

net=Net()
if cuda:
    net.cuda()
print(net)

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss()

# create a stochastic gradient descent optimizer
optimizer = optim.Adam(net.parameters())
# create a loss function (mse)
loss = nn.MSELoss()

# run the main training loop
epochs =20
hold_loss=[]

for epoch in range(epochs):
    cum_loss=0.
    for batch_idx, (data, target) in enumerate(data_loader):
        tr_x, tr_y = Variable(data.float()), Variable(target.float())
        if cuda:
            tr_x, tr_y = tr_x.cuda(), tr_y.cuda()

        # Reset gradient
        optimizer.zero_grad()

        # Forward pass
        fx = net(tr_x)
        output = loss(fx, tr_y) #loss for this batch
        cum_loss += output.data[0]

        # Backward
        output.backward()

        # Update parameters based on backprop
        optimizer.step()
    hold_loss.append(cum_loss)
    #print(epoch+1, cum_loss) #

plt.plot(np.array(hold_loss))

Output:
Well I just changed the line:

training_samples = utils_data.TensorDataset(torch.from_numpy(x), torch.from_numpy(y))

Adding the torch.from_numpy (otherwise, it was throwing an error, thus nor running) and I get a learning curve that looks something like this:


Instruction: Installing Pytorch on Windows 10

Input:
I am trying to install Pytorch on Windows 10 anaconda environment with Python 3.6 using the following command:

conda install -c peterjc123 pytorch

But it gives the following error:

  UnsatisfiableError: The following specifications were found to be in conflict:
    - curl -> krb5=1.14 -> *[track_features=vc14]
    - curl -> libssh2=1.8 -> vc==14
    - pytorch
  Use "conda info " to see the dependencies for each package.

The same goes for the new updated December code

conda install -c peterjc123 pytorch cuda80

Is there a way which will help me install Pytorch on Windows ?

Output: I solved the issue by side loading the pytorch's tar.bz2 file


Instruction: reshaping a tensor with padding in pytorch

Input: How do I reshape a tensor with dimensions (30, 35, 49) to (30, 35, 512) by padding it?

Output:
While @nemo's solution works fine, there is a pytorch internal routine, torch.nn.functional.pad, that does the same - and which has a couple of properties that a torch.ones(*sizes)*pad_value solution does not (namely other forms of padding, like reflection padding or replicate padding ... it also checks some gradient-related properties):

import torch.nn.functional as F
source = torch.rand((5,10))
# now we expand to size (7, 11) by appending a row of 0s at pos 0 and pos 6,
# and a column of 0s at pos 10
result = F.pad(input=source, pad=(0, 1, 1, 1), mode='constant', value=0)

The semantics of the arguments are:

input: the source tensor,
pad: a list of length 2 * len(source.shape) of the form (begin last axis, end last axis, begin 2nd to last axis, end 2nd to last axis, begin 3rd to last axis, etc.) that states how many dimensions should be added to the beginning and end of each axis,
mode: 'constant', 'reflect' or 'replicate'. Default: 'constant' for the different kinds of padding
value for constant padding.


Instruction: How can I download and skip VGG weights that have no counterpart with my CNN in Keras?

Input:
I would like to follow the Convolutional Neural Net (CNN) approach here. However, this code in github uses Pytorch, whereas I am using Keras.

I want to reproduce boxes 6,7 and 8 where pre-trained weights from VGG-16 on ImageNet is downloaded and is used to make the CNN converge faster.

In particular, there is a portion (box 8) where weights are downloaded and skipped from VGG-16 that have no counterpart in SegNet (the CNN model). In my work, I am using a CNN model called U-Net instead of Segnet. The U-Net Keras code that I am using can be found here.

I am new to Keras and would appreciate any insight in Keras code on how I can go about downloading and skipping the VGG weights that have no counterpart with my U-Netmodel.

Output:
The technique you are addressing is called "Transfer Learning" - when a pre-trained model on a different dataset is used as part of the model as a starting point for better convergence. The intuition behind it is simple: we assume that after training on such a large and rich dataset as ImageNet, the convolution kernels of the model will learn useful representations.

In your specific case, you want to stack VGG16 weights in the bottom and deconvolution blocks on the top. I will go step-by-step, as you pointed out that you are new to Keras. This answer is organized as a step-by-step tutorial and will provide small snippets for you to use in your own code.

Loading weights

In the PyTorch code you linked to above, the model was first defined, and only then the weights are copied. I found this approach abundant, as it contains lots of not necessary code. Here, we will load VGG16 first, and then stack the other layers on the top.

from keras import applications
from keras.layers import Input

# Loading without top layers, since you only need convolution. Note that by not
# specifying the shape of top layers, the input tensor shape is (None, None, 3),
# so you can use them for any size of images.
vgg_model = applications.VGG16(weights='imagenet', include_top=False)

# If you want to specify input tensor shape, e.g. 256x256 with 3 channels:
input_tensor = Input(shape=(256, 256, 3))
vgg_model = applications.VGG16(weights='imagenet',
                               include_top=False,
                               input_tensor=input_tensor)

# To see the models' architecture and layer names, run the following
vgg_model.summary()

Defining the U-Net computation graph with VGG16 on the bottom

As pointed in previous paragraph, you do not need to define a model and copy the weights over. Just stack other layers on top of vgg_model:

# Import the layers to be used in U-Net
from keras.layers import ...

# From the U-Net code you provided
def make_conv_block(nb_filters, input_tensor, block):
    ...

# Creating dictionary that maps layer names to the layers
layers = dict([(layer.name, layer) for layer in vgg_model.layers])

# Getting output tensor of the last VGG layer that we want to include.
# I don't know much about U-Net, but according to the code you provided,
# you don't need the last pooling layer, right?
vgg_top = layers['block5_conv3'].output

# Now getting bottom layers for multi-scale skip-layers
block1_conv2 = layers['block1_conv2'].output
block2_conv2 = layers['block2_conv2'].output
block3_conv3 = layers['block3_conv3'].output
block4_conv3 = layers['block4_conv3'].output

# Stacking the remaining layers of U-Net on top of it (modified from
# the U-Net code you provided)
up6 = Concatenate()([UpSampling2D(size=(2, 2))(vgg_top), block4_conv3])
conv6 = make_conv_block(256, up6, 6)
up7 = Concatenate()([UpSampling2D(size=(2, 2))(conv6), block3_conv3])
conv7 = make_conv_block(128, up7, 7)
up8 = Concatenate()([UpSampling2D(size=(2, 2))(conv7), block2_conv2])
conv8 = make_conv_block(64, up8, 8)
up9 = Concatenate()([UpSampling2D(size=(2, 2))(conv8), block1_conv2])
conv9 = make_conv_block(32, up9, 9)
conv10 = Conv2D(nb_labels, (1, 1), name='conv_10_1')(conv9)
x = Reshape((nb_rows * nb_cols, nb_labels))(conv10)
x = Activation('softmax')(x)
outputs = Reshape((nb_rows, nb_cols, nb_labels))(x)

I want to emphasize that what we've done in this paragraph is just defining the computation graph for U-Net. This code is written specifically for VGG16, but you can modify it for other architectures as you wish.

Creating a model

After the previous step, we've got a computational graph (I assume that you use Tensorflow backend for Keras. If you're using Theano, I recommend you to switch to Tensorflow since this framework has achieved a state of maturity now). Now, we need to do the following things:

Create a model on top of this computation graph
Freeze the bottom layers, since you don't want to wreck your pre-trained weights

# Creating new model. Please note that this is NOT a Sequential() model
# as in commonly found tutorials on the internet.
from keras.models import Model
custom_model = Model(inputs=vgg_model.input, outputs=outputs)

# Make sure that the pre-trained bottom layers are not trainable.
# Here, I freeze all the layers of VGG16 (layers 0-18, including the
# pooling ones.
for layer in custom_model.layers[:19]:
    layer.trainable = False

# Do not forget to compile it before training
custom_model.compile(loss='your_loss',
                     optimizer='your_optimizer',
                     metrics=['your_metrics'])

"I got confused"

Assuming that you're new to Keras and to Deep Learning in general (as you admitted in your question), I recommend the following articles to read to further understand the process of Fine Tuning and Transfer Learning on Keras:

How CNNs see the world - a great short article that will give you intuitive understanding of the dirty magic behind Transfer Learning.
Building powerful image classification models using very little data - This one will give you more insight on how to adjust learning rates and "release" the frozen layers.

When you're learning a framework, documentation is your best friend. Fortunately, Keras has an incredible documentation.

Q&A

  The deconvolution blocks we put on top of VGG are from the UNET achitecture (i.e. up6 to conv10)? Please confirm.

Yes, it's the same as here, just with different names of the skip-connection layers (e.g. block1_conv2 instead of conv1)

  We leave out the conv layers (i.e., conv1 to conv5). Can you please share with me as to why this is so?

We don't leave or throw any layers from the VGG network. The VGG16 network architecture and the bottom architecture of U-Net (up to conv5) is very similar. In fact, they are made of 5 blocks of the following format:

+-----------------+-------------------+
| VGG conv blocks | U-Net conv blocks |
+-----------------+-------------------+
| blockX_conv1    | convN             |
| ...             | poolN             |
| blockX_convN    |                   |
| blockX_pool     |                   |
+-----------------+-------------------+

Here is a better visualization. So, the only difference between VGG16 and bottom part of U-Net is that each block of VGG16 contains multiple convolution layers instead of one. That's why, the alternative of connecting conv3 to conv6 is connecting block3_conv3 to conv6. The U-Net architecture remains the same, just with more convolution layers on the bottom.

  Is there anyway to incorporate the Max pooling in the conv layers (in your opinion what are we doing here by leaving them out, and would you say it is insignificant?)

We don't leave them out. The only pooling layer that I threw away is block5_pool (which is the last layer in bottom part of VGG16) - because in the original U-Net (refer to the code) it seems like the last convolution block in the bottom part is not followed by a pooling layer (we have conv5 but don't have pool5). I kept all the layers of VGG16.

  We see Maxpooling being used on the convolution blocks. Would we also just simply drop these pooling layers (as we are doing here with Unet) if we wanted to combine Segnet with VGG?

As I explained in the question above, we are not dropping any pooling layers.
However, you would need to stack a different type of pooling layers instead of the simple MaxPooling2D that is used in the default VGG16, because SegNet preserves max-indexes. This can be achieved with tf.nn.max_pool_with_argmax and using the trick of replacing middle layers of Keras model (I won't cover the detailed information in this answer to keep it clean). The replacement is harmless and doesn't require re-training because pooling layers don't contain any trained weights.

  The U-NET from here is different from what I am using, can you tell what is the impact of such a difference between the two?

It is a more shallow U-Net. The one in your original question has 5 convolution blocks on the bottom (conv1 - conv5), while the later only has 3. Choose how many blocks you need depending on the data (e.g. for simple data as cells you might want to use only 2-3 blocks, while gray matter or tissue segmentation might require 5 blocks for better quality. See this link to have an insight of what convolution kernels "see".

  Also, what do you think about the VGGSegnet from here. Does it use the trick of the middle layers you mentioned in Q&A? And is it the equivalent of the Pytorch code I initially posted?

Interesting. It is an incorrect implementation, and is not equivalent to the Pytorch code you posted. I have opened an issue in that repository.

  Final question....is it always a rule in Transfer Learning to put the pretrained model (i.e., the model w/ pretrained weights) at the bottom?

Generally it is. Think of the convolution kernels as "features": the first layer detects small edges, colors. The following layers combines those edges and colors into more complicated detections, like "yellow lines" or "blue circle". Then the upper convolution layers detects more abstract shapes as "eyes", "nose", etc. based on detections of lower layers. So replacing the bottom layers (while the upper layers depends on the bottom representation) is illogic.


Instruction: Error when implementing RBF kernel bandwidth differentiation in Pytorch

Input:
I'm implementing an RBF network by using some beginer examples from Pytorch Website. I have a problem when implementing the kernel bandwidth differentiation for the network. Also, Iwould like to know whether my attempt ti implement the idea is fine. This is a code sample to reproduce the issue. Thanks

# -*- coding: utf-8 -*-
import torch
from torch.autograd import Variable

def kernel_product(x,y, mode = "gaussian", s = 1.):
    x_i = x.unsqueeze(1)
    y_j = y.unsqueeze(0)
    xmy = ((x_i-y_j)**2).sum(2)

    if   mode == "gaussian" : K = torch.exp( - xmy/s**2) )
    elif mode == "laplace"  : K = torch.exp( - torch.sqrt(xmy + (s**2)))
    elif mode == "energy"   : K = torch.pow(   xmy + (s**2), -.25 )

    return torch.t(K)

class MyReLU(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """

    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

dtype = torch.cuda.FloatTensor
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold input and outputs, and wrap them in Variables.
x = Variable(torch.randn(N, D_in).type(dtype), requires_grad=False)
y = Variable(torch.randn(N, D_out).type(dtype), requires_grad=False)

# Create random Tensors for weights, and wrap them in Variables.
w1 = Variable(torch.randn(H, D_in).type(dtype), requires_grad=True)
w2 = Variable(torch.randn(H, D_out).type(dtype), requires_grad=True)

# I've created this scalar variable (the kernel bandwidth)
s = Variable(torch.randn(1).type(dtype), requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # To apply our Function, we use Function.apply method. We alias this as 'relu'.
    relu = MyReLU.apply

    # Forward pass: compute predicted y using operations on Variables; we compute
    # ReLU using our custom autograd operation.
#    y_pred = relu(x.mm(w1)).mm(w2)
    y_pred = relu(kernel_product(w1, x, s)).mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.data[0])

    # Use autograd to compute the backward pass.
    loss.backward()

    # Update weights using gradient descent
    w1.data -= learning_rate * w1.grad.data
    w2.data -= learning_rate * w2.grad.data

    # Manually zero the gradients after updating weights
    w1.grad.data.zero_()
    w2.grad.data.zero_()

However I get this error, which dissapears when I simply use a fixed scalar in the default input parameter of kernel_product():

RuntimeError: eq() received an invalid combination of arguments - got (str), but expected one of:
 * (float other)
      didn't match because some of the arguments have invalid types: (str)
 * (Variable other)
      didn't match because some of the arguments have invalid types: (str)

Output: Well, you are calling kernel_product(w1, x, s) where w1, x and s are torch Variable while the definition of the function is: kernel_product(x,y, mode = "gaussian", s = 1.). Seems like s should be a string specifying the mode.


Instruction: setuptools: installing pytorch from download link: 403 Forbidden

Input:
I am trying to include pytorch in the requirements list for setuptools:

install_requires=[
      'torch'
      ],
dependency_links=[
      'http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl' '@develop#egg=torch'
      ],

But after running python setup.py develop I receive:

error: Can't download http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl@develop#egg=torch: 403 Forbidden

I also tried using the source install link rather than the one for pip:

install_requires=[
      'torch'
      ],
  dependency_links=[
      'https://github.com/pytorch/pytorch#from-source' '@develop#egg=torch'
      ],

But then:

RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

What is the correct way to install pytorch through setuptools? (Note: using anaconda is not an option in my case)

Thank you.

EDIT:

As suggested in the answer, I also tried:

dependency_links=[
      'https://github.com/pytorch/pytorch'
      ],

dependency_links=[
      'https://github.com/pytorch/pytorch#egg=torch'
      ],

dependency_links=[
      'http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl'
      ],

And received:

RuntimeError: PyTorch does not currently provide packages for PyPI (see status at https://github.com/pytorch/pytorch/issues/566).

Output:
First error: if you use direct URL to a wheel file:

http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl

You must not use @develop#egg=torch. That part is for installing from VCS like git.

The second URL

https://github.com/pytorch/pytorch#from-source

is also wrong. It should be

https://github.com/pytorch/pytorch

and here @develop#egg=torch is exactly right if you want to install develop branch.


Instruction: Mini batch training for inputs of variable sizes

Input:
I have a list of LongTensors, and another list of labels. I'm new to PyTorch and RNN's so I'm quite confused as to how to implement minibatch training for the data I have. There is much more to this data, but I want to keep it simple, so I can understand only how to implement the minibatch training part. I'm doing multiclass classification based on the final hidden state of an LSTM/GRU trained on variable length inputs. I managed to get it working with batch size 1(basically SGD) but I'm struggling with implementing minibatches.

Do I have to pad the sequences to the maximum size and create a new tensor matrix of larger size which holds all the elements? I mean like this:

inputs = pad(sequences)
train = DataLoader(inputs, batch_size=batch_size, shuffle=True)
for i, data in train:
   #do stuff using LSTM and/or GRU models

Is this the accepted way of doing minibatch training on custom data? I couldn't find any tutorials on loading custom data using DataLoader(but I assume that's the way to create batches using pyTorch?)

Another doubt I have is with regards to padding. The reason I'm using LSTM/GRU is because of the variable length of the input. Doesn't padding defeat the purpose? Is padding necessary for minibatch training?

Output:
Yes. The issue with minibatch training on sequences which have different lengths is that you can't stack sequences of different lengths together.

Normally one would do.

for e in range(epochs):
    sequences = shuffle(sequences)
    for mb in range(len(sequences)/mb_size):
        batch = torch.stack(sequences[mb*mb_size:(mb+1)*mb_size])

and then you apply your neural network on your batch. But because your sequences are of different lengths, the torch.stack will fail. So indeed what you have to do is to pad your sequences with zeros so that they all have the same length (at least in a minibatch). So you have 2 options:

1) At the very very beginning, pad all your sequences with initial zeros so that they all have the same length as your longest sequence of all your data.

OR

2) On the fly, for each minibatch, before stacking the sequences together, pad all the sequences that will go into the minibatch with initial zeros so that they all have the same length as the longest sequence of the minibatch.


Instruction: Reading multiple images as custom dataset for PyTorch?

Input:
I want to read in multiple images for the main_image set and blur_image set. For example, 5 main images and 5 blurred images. The goal is determine what values for the kernel in the convolutional layer convert the main images to the blurred images. The assumption is that the same kernel is used to blur each of the 5 original images to produce the 5 blurred images

My code is available at: https://pastebin.com/PWf7rjd4 and https://pastebin.com/VxryDb7g

However, it seems to only be processing the first image, that is "1.png" for the main and blurred images. It is not processing images 2.png, 3.png, 4.png, and 5.png

How can I fix this?

Output:
In your class BlurDataset you only return one image in the __getitem__ method. In your main method you call

for batch_idx, (main, blur) in enumerate(train_loader)

The torch.utils.data.Dataset class that you inherit from then calls __getitem__ with the index given by enumerate. It will give you one pair of pictures in each iteration of the loop. If you want to get all 5 pairs of pictures in the first iteration you have to change your __getitem__ method to loop over images and return them all.


Instruction: PyTorch: How do the means and stds get calculated in the Transfer Learning tutorial?

Input:
I'm going through the PyTorch Transfer Learning tutorial at: link

In the data augmentation stage, there is the following step to normalize images:

transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

I can understand why it's doing this but I can't find how the mean and std values get calculated? I tried to calculate the mean on the train data set and the mean values are:

array([ 0.11727478,  0.04542569, -0.28624609], dtype=float32)

Output:
Your numbers don't seem right to me; since the ToTensor transform has output in the range [0.0, 1.0] it shouldn't be possible to get a negative mean.

If I calculate the mean with

traindata = datasets.ImageFolder(data_dir + '/train', transforms.ToTensor())
image_means = torch.stack([t.mean(1).mean(1) for t, c in traindata])
image_means.mean(0)

I get (0.5143, 0.4760, 0.3487) and for the validation set (0.5224, 0.4799, 0.3564). These are closer to the numbers in the tutorial. Searching for the specific numbers, you'll see that they appear in the Imagenet example, so my guess is that they are the means of the Imagenet dataset, of which the tutorial dataset is a subset.


Instruction: How to use PyTorch multiprocessing?

Input:
I'm trying to use python's multiprocessing Pool method in pytorch to process a image. Here's the code:

from multiprocessing import Process, Pool
from torch.autograd import Variable
import numpy as np
from scipy.ndimage import zoom

def get_pred(args):

  img = args[0]
  scale = args[1]
  scales = args[2]
  img_scale = zoom(img.numpy(),
                     (1., 1., scale, scale),
                     order=1,
                     prefilter=False,
                     mode='nearest')

  # feed input data
  input_img = Variable(torch.from_numpy(img_scale),
                     volatile=True).cuda()
  return input_img

scales = [1,2,3,4,5]
scale_list = []
for scale in scales:
    scale_list.append([img,scale,scales])
multi_pool = Pool(processes=5)
predictions = multi_pool.map(get_pred,scale_list)
multi_pool.close()
multi_pool.join()

I'm getting this error:

`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

`
In this line:

predictions = multi_pool.map(get_pred,scale_list)

Can anyone tell me what I'm doing wrong ?

Output:
As stated in pytorch documentation the best practice to handle multiprocessing is to use torch.multiprocessing instead of multiprocessing.

Be aware that sharing CUDA tensors between processes is supported only in Python 3, either with spawn or forkserver as start method.

Without touching your code, a workaround for the error you got is replacing

from multiprocessing import Process, Pool

with:

from torch.multiprocessing import Pool, Process, set_start_method
try:
     set_start_method('spawn')
except RuntimeError:
    pass


Instruction: Google Ngram Viewer - English One Million

Input:
I'm training a language model in PyTorch and I'd need the most common one million words in English to serve as dictionary.

From what I've understood, the Google Ngram English One Million (1-grams) might suit to this task, but after downloading every part (0-9) of this dataset and using tail on them to check if they were what I supposed, I found out that no part of this dataset contains words beyond the F letter.

As far as I understood, any Version 1 file has its ngrams alphabetically and cronologically sorted and I'm concerned if it might be possible that the most common one million words do not go beyond the F?

Or am I missing the point of this dataset and it isn't the most commond one million words?

Output: Try shuf <file> to get a random sorting and you will see the data covers all letters. What you see at the end of the files is not an f but the ligature ﬂ.


Instruction: LSTM in Pytorch

Input:
I'm new to PyTorch. I came across some this GitHub repository (link to full code example) containing various different examples.

There is also an example about LSTMs, this is the Network class:

# RNN Model (Many-to-One)
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # Set initial states
        h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
        c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))

        # Forward propagate RNN
        out, _ = self.lstm(x, (h0, c0))

        # Decode hidden state of last time step
        out = self.fc(out[:, -1, :])
        return out

So my question is about the following lines:

h0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))
c0 = Variable(torch.zeros(self.num_layers, x.size(0), self.hidden_size))

As far as I understand it, forward() is called for every training example. But this would mean, that the hidden state and cell state would be resettet i.e. replaced with a matrix of zeros on every training example.

The names h0 and c0 indicate that this is only the hidden/cell state at t=0, but why then are theses zeros matrices handed over to the lstm with every training example?

Even if they are just ignored after the first call, it would not be a very nice solution.

When testing the code it states an accuracy of 97% on the MNIST set, so it seems to work this way, but it doesn't make sense to me.

Hope someone can help me out with this.

Thanks in advance!

Output:
Obviously I was on the wrong track with this. I was confusing hidden units and hidden/cell state.
Only the hidden units in the LSTM are trained during the training step. Cell state and hidden state are resetet at the beginning of every sequence. So it just makes sense that it is programmed this way.

Sorry for this..


Instruction: ouputs in semantic segmentation task

Input:
Im working on the Kaggle semantic segmentation task,

In the testing part of my code,

model = model.eval()
predictions =[]
for data in testdataloader:
    data = t.autograd.Variable(data, volatile=True).cuda()
    output = model.forward(data)
    _,preds = t.max(output, 1, keepdim = True)

when i do the preds part,the array is only filled with ,i was hoping it to be an array of maximum locations im not sure what is going wrong.
The output part works well,I have attached a screenshot for visualization of output

Any sugestions on what is going wrong would be really helpful.

thanks

Output:
Assuming your data is of the form MiniBatch x Dim what you are doing now is looking at which minibatch has the highest value. If you are testing it with a single sample (MB = 1) then you will always get 0 as your answer. Thus, you might want to try:

_,preds = t.max(output, 0, keepdim = False)