Instruction: Pytorch register_hook to Keras implementation

Input:
Im trying to implement the following project into Tensorflow/Keras.
https://github.com/jacobgil/pytorch-pruning

Im having a hard time understanding what register_hook does? It can be found in finetune.py, row 66.
x.register_hook(self.compute_rank)

I've searched for clear explanations regarding this function and tried to find Keras-equivalents, without any luck. Do you have answers to these questions?

Output:
First things first, here's the documentation:

http://pytorch.org/docs/master/autograd.html#torch.autograd.Variable.register_hook

This allows you to register a method to a Variable that is called whenever the Variable's .grad is updated, i.e. in a backward pass, and takes the grad as input. The method can return a Variable that would replace the original .grad or None if you just want to read the gradients to do something else.
If you update the gradients this way, the nodes further down in the compute graph see the new updated gradient in the backward pass and will have their respective gradients calculated with the updated value.

I'm not a Tensorflow expert, but the RegisterGradient decorators (documentation) seem to be able to do the same, for an example see this answer.


Instruction: Tensor sizes CrossEntropyLoss

Input:
In the snippet:

criterion = nn.CrossEntropyLoss()
raw_loss = criterion(output.view(-1, ntokens), targets)

output size is torch.Size([5, 5, 8967]), targets size is torch.Size([25]), and ntokens is 8967

After modifying the code, my

output size is torch.Size([5, 8967]) and targets size is torch.Size([25])

which rises dimensionality issues when computing the loss.

Is it sensible to increase the size of my Linear activation that produces the output by 5, so that I can resize the output later to be of the size torch.Size([5, 5, 8967])?

The problem with increasing the size of the tensor is that ntokens can become quite large and I can easily run out of memory because of that. Is there an alternative approach?

Output:
You should do something like this:

ntokens = 8000
output = Variable(torch.randn(5, 5, ntokens))
targets = Variable(torch.from_numpy(np.random.randint(0, ntokens, size=25)))
criterion = nn.CrossEntropyLoss()
loss = criterion(output.view(-1, ntokens), targets)
print(loss)

This prints:

Variable containing:
 9.4613
[torch.FloatTensor of size 1]

Here, I am assuming output contains predictions of next word for 5 sentences (minibatch size is 5) and each sentence is of length 5 (sequence length is 5). 8000 is the vocabulary size, so your model is predicting a probability distribution over the entire vocabulary.
Now, you can compute the loss of predicting each word as your target shape is 25 as required.

Please note, CrossEntropyLoss expects input to contain scores for each class. So, input has to be a 2D Tensor of size (minibatch, C) and the target has to be a class index (0 to C-1) for each value of a 1D tensor of size minibatch.


Instruction: Implementing WNGrad in Pytorch?

Input:
I'm trying to implement the WNGrad (technically WN-Adam, algorithm 4 in the paper) optimizier (WNGrad) in pytorch.  I've never implemented an optimizer in pytorch before so I don't know if I've done it correctly (I started from the adam implementation).  The optimizer does not make much progress and falls down like I would expect (bj values can only monotonically increase, which happens quickly so no progress is made) but I'm guessing I have a bug.  Standard optimizers (Adam, SGD) work fine on the same model I'm trying to optimize.

Does this implementation look correct?

from torch.optim import Optimizer

class WNAdam(Optimizer):
    """Implements WNAdam algorithm.

    It has been proposed in `WNGrad: Learn the Learning Rate in Gradient Descent`_.

    Arguments:
        params (iterable): iterable of parameters to optimize or dicts defining
            parameter groups
        lr (float, optional): learning rate (default: 0.1)
        beta1 (float, optional): exponential smoothing coefficient for gradient.
                                When beta=0 this implements WNGrad.

    .. _WNGrad\: Learn the Learning Rate in Gradient Descent:
        https://arxiv.org/abs/1803.02865
    """

    def __init__(self, params, lr=0.1, beta1=0.9):
        if not 0.0 <= beta1 < 1.0:
            raise ValueError("Invalid beta1 parameter: {}".format(beta1))
        defaults = dict(lr=lr, beta1=beta1)
        super().__init__(params, defaults)

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad.data
                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p.data)
                    # Learning rate adjustment
                    state['bj'] = 1.0

                exp_avg = state['exp_avg']
                beta1 = group['beta1']

                state['step'] += 1
                state['bj'] += (group['lr']**2)/(state['bj'])*grad.pow(2).sum()

                # update exponential moving average
                exp_avg.mul_(beta1).add_(1 - beta1, grad)

                bias_correction = 1 - beta1 ** state['step']
                p.data.sub_(group['lr'] / state['bj'] / bias_correction, exp_avg)
        return loss

Output: The paper's author has an open sourced implementation on GitHub.


Instruction: Binary One Hot Encoding For Dimension Reduction

Input:
Introduction:

I've been studying machine learning this last couple of months, more specifically neural networks. I found out that categorical data, such as cities, could be transformed to integer data, but shouldn't because there's no linearity. What I mean by this is: there is no reason why New York should be 1 or 100. A better solution to this is one hot encoding. However, it greatly increases the dimensionality of the problem (Picture a network input with the number of cities as input nodes).

Example:

New York = [1, 0, 0]

Buenos Aires = [0, 1, 0]

Tokio = [0, 0, 1]

You need N inputs, N being the number of cities.

Possible Solution:

I don't really know if this already exists or if it could work, I haven't tested it yet. That being said, in the example above, I'm using decimal inputs. Could you use binary inputs, in order to reduce the dimensionality of the problem?

Example using binary inputs

New York = [0, 1]

Buenos Aires = [1, 0]

Tokio = [1, 1]

In this way, you only need ≈ log(N) (base 2) number of inputs which greatly increases, specially if there are a great number of features. For example: if you have 1000 categorical data inputs, it would only result in 10 inputs for a neural network.

Thank you in advance. Remember I'm only learning.

Output:
I do not think that you can translate One Hot Encoding (OHE) to binary inputs.
The meaning of One Hot Encoding is that you have as many features as you have cities. No two cities share value for any feature, as they are distinct. After you translation to binary inputs, various cities randomly share values for the same feature.

E.g. both Buenos Aires and Tokio would have 1 as the first feature. The neural network would think that they really have this feature in common. Yet it is not the case, depending on your ordering, New York could easily share value of the first feature with Tokio:

Buenos Aires = [0, 1]
Tokio = [1, 0]
New York = [1, 1]

Now, Tokio and New York seem more similar to each other.


Instruction: I need some explanations about GAN codes

Input:
Here is github codes

for epoch in range(num_epoch):
for i, (img, _) in enumerate(dataloader):
    num_img = img.size(0)
    # =================train discriminator
    img = img.view(num_img, -1)
    real_img = Variable(img).cuda()
    real_label = Variable(torch.ones(num_img)).cuda()
    fake_label = Variable(torch.zeros(num_img)).cuda()

I don`t understand whats the torch.ones and torch.zeros in training codes.

Can anyone explain about this?

Output:
As you probably know: In GAN, generator tries to fool the discriminator by convincing that a fake example is a true example. Discriminator trained to distinguish true examples and fake examples. On the other hand, the generator is trained to generate (fake) examples that look very close to the real examples.

Analysis of the code/example (in the link) you shared.

Generator: is a simple feed-forward neural network. The generator generates 28 * 28 images from random (noisy) distribution. The goal of the generator is to generate images that look like real images.

Discriminator: is a simple feed-forward neural network. The discriminator provides a sigmoid ([0, 1]) score given an image. The goal of the discriminator is to give low score (~0) to fake images and high score (~1) to real images. In essence, the discriminator wants to distinguish real images from fake ones.

How does the code work?

First, the discriminator is provided examples of real images and a loss is computed based on discriminator's predicted score.

# compute loss of real_img
real_out = D(real_img)
d_loss_real = criterion(real_out, real_label)
real_scores = real_out  # closer to 1 means better

Then the discriminator is provided the fake images generated by the generator. A loss is computed based on discriminator's score on fake examples.

# compute loss of fake_img
z = Variable(torch.randn(num_img, z_dimension)).cuda()
fake_img = G(z)
fake_out = D(fake_img)
d_loss_fake = criterion(fake_out, fake_label)
fake_scores = fake_out  # closer to 0 means better

Essentially, the generator and discriminator are competing against each other to become expert in achieving their goal. We can think in this way: if we have a perfect generator, then it will create fake example exactly look like real ones and the discriminator will fail to distinguish them and vice versa.

The code you provided above is just creating labels using torch.zeros() and torch.ones(). You can simply consider it as binary labels for real and fake images.


Instruction: Loading huge text files for neural machine translation with Pytorch

Input:
In PyTorch, I have written a dataset loading class for loading 2 text files as source and targets, for neural machine translation purpose. Each file has 93577946 lines, and each of them allocates 8GB memory on Hard Disc.

The class is as the following:

class LoadUniModal(Dataset):
    sources = []
    targets = []
    maxlen = 0
    lengths = []

    def __init__(self, src, trg, src_vocab, trg_vocab):
        self.src_vocab = src_vocab
        self.trg_vocab = trg_vocab

        with codecs.open(src, encoding="utf-8") as f:
            for line in f:
                tokens = line.replace("\n", "").split()
                self.maxlen = max(self.maxlen, len(tokens))
                self.sources.append(tokens)
        with codecs.open(trg, encoding="utf-8") as f:
            for line in f:
                tokens = line.replace("\n", "").split()
                self.maxlen = max(self.maxlen, len(tokens))
                self.targets.append(tokens)
                self.lengths.append(len(tokens)+2)

    # Overrride to give PyTorch access to any image on the dataset
    def __getitem__(self, index):

        # Source sentence processing
        tokens = self.sources[index]
        ntokens = [self.src_vocab['<START>']]
        for a in range(self.maxlen):
            if a <= (len(tokens) - 1):
                if tokens[a] in self.src_vocab.keys():
                    ntokens.append(self.src_vocab[tokens[a]])
                else:
                    ntokens.append(self.src_vocab['<UNK>'])
            elif a == len(tokens):
                ntokens.append(self.src_vocab['<END>'])
            elif a > len(tokens):
                ntokens.append(self.src_vocab['<PAD>'])

        source = torch.from_numpy(np.asarray(ntokens)).long()

        # Target sentence processing
        tokens = self.targets[index]
                ntokens = [self.trg_vocab['<START>']]
                for a in range(self.maxlen):
                        if a <= (len(tokens) - 1):
                                if tokens[a] in self.trg_vocab.keys():
                                        ntokens.append(self.trg_vocab[tokens[a]])
                                else:
                                        ntokens.append(self.trg_vocab['<UNK>'])
                        elif a == len(tokens):
                                ntokens.append(self.trg_vocab['<END>'])
                        elif a > len(tokens):
                                ntokens.append(self.trg_vocab['<PAD>'])

                target = torch.from_numpy(np.asarray(ntokens)).long()

        length = self.lengths[index]

        return [0], source, target, length

    def __len__(self):
        return len(self.sources)

I use the class in order to load dataset as follows:

def load_text_train_data(train_dir, src_vocab, trg_vocab, lang_pair, batch_size):

        tpl = ast.literal_eval(lang_pair)
        slang = tpl[1]
        tlang = tpl[2]

        strain_file = os.path.join(train_dir, "train"+slang)
        ttrain_file = os.path.join(train_dir, "train"+tlang)

        data_iter = LoadUniModal(strain_file, ttrain_file, src_vocab, trg_vocab)
        data_iter = DataLoader(data_iter, batch_size=batch_size)

        return data_iter

When I am trying to load the data, I get memory error.

How would it be possible to load the data without memory problem?

Thanks,

Output:
This shouldn't give you an error unless you load the entire data in memory at once. One suggestion I want to give you is: don't pad all sentences to a maximum length. In machine translation data, in general, sentence lengths vary a lot.

Also, you can try smaller mini-batches of size x (ex., 32, 64) which your memory can afford. Only pad the elements of the current mini-batch and move to cuda tensor and then pass it to your model. Hopefully, it will solve your problem.


Instruction: How to deploy pytorch in docker image?

Input: I want to deploy pytorch on a docker image. Is there a way that I can do that? I faced so much problems installing pytorch, their official installation links does not seem to be working; neither pip/conda works. Does anyone have any alternative solution in installing pytorch?

Output:
You can find pytorch images on Dockerhub.

If those images are not sufficient, you can check their Dockerfiles to see how you can build you own custom image.


Instruction: In pytorch, the meaning of y.backward([0.1, 1.0, 0.0001])

Input:
In pytorch
what is the meaning of y.backward([0.1, 1.0, 0.0001])?

I understand that y.backward() means doing backpropagation.
But what is the meaning of [0.1, 1.0, 0.0001] in y.backward([0.1, 1.0, 0.0001])?

Output:
The expression y.backward([0.1, 1.0, 0.0001]) is actually wrong. It should be y.backward(torch.Tensor([0.1, 1.0, 0.0001])), where torch.Tensor([0.1, 1.0, 0.0001]) are the Variables of which the derivative will be computed.

Example:

x = Variable(torch.ones(2, 2), requires_grad=True)
y = (x + 2).mean()
y.backward(torch.Tensor([1.0]))
print(x.grad)

Here, y = (x + 2)/4 and so, dy/dx_i = 0.25 since x_i = 1.0. Also note, y.backward(torch.Tensor([1.0])) and y.backward() are equivalent.

If you do:

y.backward(torch.Tensor([0.1]))
print(x.grad)

it prints:

Variable containing:
1.00000e-02 *
  2.5000  2.5000
  2.5000  2.5000
[torch.FloatTensor of size 2x2]

It is simply 0.1 * 0.25 = 0.025. So, now if you compute:

y.backward(torch.Tensor([0.1, 0.01]))
print(x.grad)

Then it prints:

Variable containing:
1.00000e-02 *
  2.5000  0.2500
  2.5000  0.2500
[torch.FloatTensor of size 2x2]

Where, dy/dx_11 = dy/d_x21 = 0.025 and dy/dx_12 = dy/d_x22 = 0.0025.

See the function prototype of backward(). You may consider looking into this example.


Instruction: CTC loss goes down and stops

Input:
I’m trying to train a captcha recognition model. Model details are resnet pretrained CNN layers + Bidirectional LSTM + Fully Connected. It reached 90% sequence accuracy on captcha generated by python library captcha. The problem is that these generated captcha seems to have similary location of each character. When I randomly add spaces between characters, the model does not work any more. So I wonder is LSTM learning segmentation during learning? Then I try to use CTC loss. At first, loss goes down pretty quick. But it stays at about 16 without significant drop later. I tried different layers of LSTM, different number of units. 2 Layers of LSTM reach lower loss, but still not converging. 3 layers are just like 2 layers. The loss curve:

#encoding:utf8
import os
import sys
import torch
import warpctc_pytorch
import traceback

import torchvision
from torch import nn, autograd, FloatTensor, optim
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import MultiStepLR
from tensorboard import SummaryWriter
from pprint import pprint

from net.utils import decoder

from logging import getLogger, StreamHandler
logger = getLogger(__name__)
handler = StreamHandler(sys.stdout)
logger.addHandler(handler)

from dataset_util.utils import id_to_character
from dataset_util.transform import rescale, normalizer
from config.config import MAX_CAPTCHA_LENGTH, TENSORBOARD_LOG_PATH, MODEL_PATH

class CNN_RNN(nn.Module):
    def __init__(self, lstm_bidirectional=True, use_ctc=True, *args, **kwargs):
        super(CNN_RNN, self).__init__(*args, **kwargs)
        model_conv = torchvision.models.resnet18(pretrained=True)
        for param in model_conv.parameters():
            param.requires_grad = False

        modules = list(model_conv.children())[:-1]  # delete the last fc layer.
        for param in modules[8].parameters():
            param.requires_grad = True

        self.resnet = nn.Sequential(*modules)            # CNN with fixed parameters from resnet as feature extractor
        self.lstm_input_size = 512 * 2 * 2
        self.lstm_hidden_state_size = 512
        self.lstm_num_layers = 2
        self.chracter_space_length = 64
        self._lstm_bidirectional = lstm_bidirectional
        self._use_ctc = use_ctc
        if use_ctc:
            self._max_captcha_length = int(MAX_CAPTCHA_LENGTH * 2)
        else:
            self._max_captcha_length = MAX_CAPTCHA_LENGTH

        if lstm_bidirectional:
            self.lstm_hidden_state_size = self.lstm_hidden_state_size * 2           # so that hidden size for one direction in bidirection lstm is the same as vanilla lstm
            self.lstm = self.lstm = nn.LSTM(self.lstm_input_size, self.lstm_hidden_state_size // 2, dropout=0.5, bidirectional=True, num_layers=self.lstm_num_layers)
        else:
            self.lstm = nn.LSTM(self.lstm_input_size, self.lstm_hidden_state_size, dropout=0.5, bidirectional=False, num_layers=self.lstm_num_layers)  # dropout doen't work for one layer lstm

        self.ouput_to_tag = nn.Linear(self.lstm_hidden_state_size, self.chracter_space_length)
        self.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)
        # self.dropout_lstm = nn.Dropout()

    def init_hidden_status(self, batch_size):
        if self._lstm_bidirectional:
            self.hidden = (autograd.Variable(torch.zeros((self.lstm_num_layers * 2, batch_size, self.lstm_hidden_state_size // 2))),
                           autograd.Variable(torch.zeros((self.lstm_num_layers * 2, batch_size, self.lstm_hidden_state_size // 2)))) # number of layers, batch size, hidden dimention
        else:
            self.hidden = (autograd.Variable(torch.zeros((self.lstm_num_layers, batch_size, self.lstm_hidden_state_size))),
                           autograd.Variable(torch.zeros((self.lstm_num_layers, batch_size, self.lstm_hidden_state_size)))) # number of layers, batch size, hidden dimention

    def forward(self, image):
        '''
        :param image:  # batch_size, CHANNEL, HEIGHT, WIDTH
        :return:
        '''
        features = self.resnet(image)                 # [batch_size, 512, 2, 2]
        batch_size = image.shape[0]
        features = [features.view(batch_size, -1) for i in range(self._max_captcha_length)]
        features = torch.stack(features)
        self.init_hidden_status(batch_size)
        output, hidden = self.lstm(features, self.hidden)
        # output = self.dropout_lstm(output)
        tag_space = self.ouput_to_tag(output.view(-1, output.size(2)))      # [MAX_CAPTCHA_LENGTH * BATCH_SIZE, CHARACTER_SPACE_LENGTH]
        tag_space = tag_space.view(self._max_captcha_length, batch_size, -1)

        if not self._use_ctc:
            tag_score = F.log_softmax(tag_space, dim=2)             # [MAX_CAPTCHA_LENGTH, BATCH_SIZE, CHARACTER_SPACE_LENGTH]
        else:
            tag_score = tag_space

        return tag_score

    def train_net(self, data_loader, eval_data_loader=None, learning_rate=0.008, epoch_num=400):
        try:
            if self._use_ctc:
                loss_function = warpctc_pytorch.warp_ctc.CTCLoss()
            else:
                loss_function = nn.NLLLoss()

            # optimizer = optim.SGD(filter(lambda p: p.requires_grad, self.parameters()), momentum=0.9, lr=learning_rate)
            # optimizer = MultiStepLR(optimizer, milestones=[10,15], gamma=0.5)

            # optimizer = optim.Adadelta(filter(lambda p: p.requires_grad, self.parameters()))
            optimizer = optim.Adam(filter(lambda p: p.requires_grad, self.parameters()))
            self.tensorboard_writer.add_scalar("learning_rate", learning_rate)

            tensorbard_global_step=0
            if os.path.exists(os.path.join(TENSORBOARD_LOG_PATH, "resume_step")):
                with open(os.path.join(TENSORBOARD_LOG_PATH, "resume_step"), "r") as file_handler:
                    tensorbard_global_step = int(file_handler.read()) + 1

            for epoch_index, epoch in enumerate(range(epoch_num)):
                for index, sample in enumerate(data_loader):
                    optimizer.zero_grad()
                    input_image = autograd.Variable(sample["image"])        # batch_size, 3, 255, 255
                    tag_score = self.forward(input_image)

                    if self._use_ctc:
                        tag_score, target, tag_score_sizes, target_sizes = self._loss_preprocess_ctc(tag_score, sample)
                        loss = loss_function(tag_score, target, tag_score_sizes, target_sizes)
                        loss = loss / tag_score.size(1)

                    else:
                        target = sample["padded_label_idx"]
                        tag_score, target = self._loss_preprocess(tag_score, target)
                        loss = loss_function(tag_score, target)

                    print("Training loss: {}".format(float(loss)))
                    self.tensorboard_writer.add_scalar("training_loss", float(loss), tensorbard_global_step)
                    loss.backward()
                    optimizer.step()

                    if index % 250 == 0:
                        print(u"Processing batch: {} of {}, epoch: {}".format(index, len(data_loader), epoch_index))
                        self.evaluate(eval_data_loader, loss_function, tensorbard_global_step)

                    tensorbard_global_step += 1

                self.save_model(MODEL_PATH + "_epoch_{}".format(epoch_index))

        except KeyboardInterrupt:
            print("Exit for KeyboardInterrupt, save model")
            self.save_model(MODEL_PATH)

            with open(os.path.join(TENSORBOARD_LOG_PATH, "resume_step"), "w") as file_handler:
                file_handler.write(str(tensorbard_global_step))

        except Exception as excp:
            logger.error(str(excp))
            logger.error(traceback.format_exc())

    def predict(self, image):
        # TODO ctc version
        '''
        :param image: [batch_size, channel, height, width]
        :return:
        '''
        tag_score = self.forward(image)
        # TODO ctc
        # if self._use_ctc:
        #     tag_score = F.softmax(tag_score, dim=-1)
        #     decoder.decode(tag_score)

        confidence_log_probability, indexes = tag_score.max(2)

        predicted_labels = []
        for batch_index in range(indexes.size(1)):
            label = ""
            for character_index in range(self._max_captcha_length):
                if int(indexes[character_index, batch_index]) != 1:
                    label += id_to_character[int(indexes[character_index, batch_index])]
            predicted_labels.append(label)

        return predicted_labels, tag_score

    def predict_pil_image(self, pil_image):
        try:
            self.eval()
            processed_image = normalizer(rescale({"image": pil_image}))["image"].view(1, 3, 255, 255)
            result, tag_score = self.predict(processed_image)
            self.train()

        except Exception as excp:
            logger.error(str(excp))
            logger.error(traceback.format_exc())
            return [""], None

        return result, tag_score

    def evaluate(self, eval_dataloader, loss_function, step=0):
        total = 0
        sequence_correct = 0
        character_correct = 0
        character_total = 0
        loss_total = 0
        batch_size = eval_data_loader.batch_size
        true_predicted = {}
        self.eval()
        for sample in eval_dataloader:
            total += batch_size
            input_images = sample["image"]
            predicted_labels, tag_score = self.predict(input_images)

            for predicted, true_label in zip(predicted_labels, sample["label"]):
                if predicted == true_label:                  # dataloader is making label a list, use batch_size=1
                    sequence_correct += 1

                for index, true_character in enumerate(true_label):
                    character_total += 1
                    if index < len(predicted) and predicted[index] == true_character:
                        character_correct += 1

                true_predicted[true_label] = predicted

            if self._use_ctc:
                tag_score, target, tag_score_sizes, target_sizes = self._loss_preprocess_ctc(tag_score, sample)
                loss_total += float(loss_function(tag_score, target, tag_score_sizes, target_sizes) / batch_size)

            else:
                tag_score, target = self._loss_preprocess(tag_score, sample["padded_label_idx"])
                loss_total += float(loss_function(tag_score, target))  # averaged over batch index

        print("True captcha to predicted captcha: ")
        pprint(true_predicted)
        self.tensorboard_writer.add_text("eval_ture_to_predicted", str(true_predicted), global_step=step)

        accuracy = float(sequence_correct) / total
        avg_loss = float(loss_total) / (total / batch_size)
        character_accuracy = float(character_correct) / character_total
        self.tensorboard_writer.add_scalar("eval_sequence_accuracy", accuracy, global_step=step)
        self.tensorboard_writer.add_scalar("eval_character_accuracy", character_accuracy, global_step=step)
        self.tensorboard_writer.add_scalar("eval_loss", avg_loss, global_step=step)
        self.zero_grad()
        self.train()

    def _loss_preprocess(self, tag_score, target):
        '''
        :param tag_score:  value return by self.forward
        :param target:     sample["padded_label_idx"]
        :return:           (processed_tag_score, processed_target)  ready for NLLoss function
        '''
        target = target.transpose(0, 1)
        target = target.contiguous()
        target = target.view(target.size(0) * target.size(1))
        tag_score = tag_score.view(-1, self.chracter_space_length)

        return tag_score, target

    def _loss_preprocess_ctc(self, tag_score, sample):
        target_2d = [
            [int(ele) for ele in sample["padded_label_idx"][row, :] if int(ele) != 0 and int(ele) != 1]
            for row in range(sample["padded_label_idx"].size(0))]
        target = []
        for ele in target_2d:
            target.extend(ele)
        target = autograd.Variable(torch.IntTensor(target))

        # tag_score = F.softmax(F.sigmoid(tag_score), dim=-1)
        tag_score_sizes = autograd.Variable(torch.IntTensor([self._max_captcha_length] * tag_score.size(1)))
        target_sizes = autograd.Variable(sample["captcha_length"].int())

        return tag_score, target, tag_score_sizes, target_sizes

    # def visualize_graph(self, dataset):
    #     '''Since pytorch use dynamic graph, an input is required to visualize graph in tensorboard'''
    #     # warning: Do not run this, the graph is too large to visualize...
    #     sample = dataset[0]
    #     input_image = autograd.Variable(sample["image"].view(1, 3, 255, 255))
    #     tag_score = self.forward(input_image)
    #     self.tensorboard_writer.add_graph(self, tag_score)

    def save_model(self, model_path):
        self.tensorboard_writer.close()
        self.tensorboard_writer = None          # can't be pickled
        torch.save(self, model_path)
        self.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)

    @classmethod
    def load_model(cls, model_path=MODEL_PATH, *args, **kwargs):
        net = cls(*args, **kwargs)
        if os.path.exists(model_path):
            model = torch.load(model_path)
            if model:
                model.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)
                net = model

        return net

    def __del__(self):
        if self.tensorboard_writer:
            self.tensorboard_writer.close()

if __name__ == "__main__":
    from dataset_util.dataset import dataset, eval_dataset
    data_loader = DataLoader(dataset, batch_size=2, shuffle=True)
    eval_data_loader = DataLoader(eval_dataset, batch_size=2, shuffle=True)

    net = CNN_RNN.load_model()

    net.train_net(data_loader, eval_data_loader=eval_data_loader)
    # net.predict(dataset[0]["image"].view(1, 3, 255, 255))

    # predict_pil_image test code
    # from config.config import IMAGE_PATHS
    # import glob
    # from PIL import Image
    #
    # image_paths = glob.glob(os.path.join(IMAGE_PATHS.get("EVAL"), "*.png"))
    # for image_path in image_paths:
    #     pil_image = Image.open(image_path)
    #     predicted, score = net.predict_pil_image(pil_image)
    #     print("True value: {}, predicted: {}".format(os.path.split(image_path)[1], predicted))

    print("Done")

The above codes are main part. If you need other components that makes it running, leave a comment. Got stuck here for quite long. Any advice for training crnn + ctc is appreciated.

Output:
I've been training with ctc loss and encountered the same problem. I know this is a rather late answer but hopefully it'll help someone else who's researching on this. After trial and error and a lot of research there are a few things that's worth knowing when it comes to training with ctc (if your model is set up correctly):

The quickest way for the model to lower cost is to predict only blanks. This is  noted in a few papers and blogs: see http://www.tbluche.com/ctc_and_blank.html
The model learns to predict only blanks first, then it starts picking up on the error signal in regards to the correct underlying labels. This is also explained in the above link. In practice, I noticed that my model starts to learn the real underlying labels/targets after a couple hundred epochs and the loss starts decreasing dramatically again. Similar to what is shown for the toy example here: https://thomasmesnard.github.io/files/CTC_Poster_Mesnard_Auvolat.pdf
These parameters have a great impact on whether your model converges or not - learning rate, batch size and epoch number.


Instruction: Difference between 'ctx' and 'self' in python?

Input:
While using the deep learning library PyTorch I came across definitions like this. Does ctx have the same behavior as self?

class LinearFunction(Function):

    @staticmethod
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, weight, bias = ctx.saved_variables
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias

Output:
A static method (@staticmethod) is called using the class type directly, not an instance of this class:

LinearFunction.backward(x, y)

Since you have no instance, it does not make sense to use self in a static method.

Here, ctx is just a regular argument that you'll have to pass when calling your methods.


Instruction: cffi error when compiling faster rcnn pytorch

Input:
I'm trying to compile faster_rcnn_pytorch using the instructions given here: https://github.com/longcw/faster_rcnn_pytorch

I get this error:

(p27) [$USER@compute-1-5 faster_rcnn]$ ./make.sh
Traceback (most recent call last):
  File "setup.py", line 59, in <module>
    CUDA = locate_cuda()
  File "setup.py", line 45, in locate_cuda
    raise EnvironmentError('The nvcc binary could not be '
EnvironmentError: The nvcc binary could not be located in your $PATH. Either add it to your path, or set $CUDAHOME
Compiling roi pooling kernels by nvcc...
./make.sh: line 10: nvcc: command not found
/home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling
generating /tmp/tmpP8mucv/_roi_pooling.c
setting the current directory to '/tmp/tmpP8mucv'
running build_ext
building '_roi_pooling' extension
creating home
creating home/$USER
creating home/$USER/play
creating home/$USER/play/hw2
creating home/$USER/play/hw2/gitdir
creating home/$USER/play/hw2/gitdir/hw2-release
creating home/$USER/play/hw2/gitdir/hw2-release/code
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling
creating home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/data2/$USER/anaconda2/envs/p27/include/python2.7 -c _roi_pooling.c -o ./_roi_pooling.o
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/data2/$USER/anaconda2/envs/p27/include/python2.7 -c /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.c -o ./home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.o
gcc -pthread -shared -L/data2/$USER/anaconda2/envs/p27/lib -Wl,-rpath=/data2/$USER/anaconda2/envs/p27/lib,--no-as-needed ./_roi_pooling.o ./home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/roi_pooling.o /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/cuda/roi_pooling.cu.o -L/data2/$USER/anaconda2/envs/p27/lib -lpython2.7 -o ./_roi_pooling.so
gcc: error: /home/$USER/play/hw2/gitdir/hw2-release/code/faster_rcnn/roi_pooling/src/cuda/roi_pooling.cu.o: No such file or directory
Traceback (most recent call last):
  File "build.py", line 34, in <module>
    ffi.build()
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 167, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 103, in _build_extension
    ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/api.py", line 697, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/recompiler.py", line 1520, in recompile
    compiler_verbose, debug)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/data2/$USER/anaconda2/envs/p27/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

As far as I can tell, there are 2 problems:

nvcc is not found
cffi error

Regarding nvcc, I'm not sure how to add it to my path. It looks like I have both CUDA 8 and CUDA 9 so maybe I need to add it to only CUDA 9?

(p27) [$USER@compute-1-5 faster_rcnn]$ which nvcc
/usr/bin/which: no nvcc in (/data2/$USER/anaconda2/envs/p27/bin:/data2/$USER/anaconda2/bin:/opt/afni:/data2/$USER/anaconda2/bin:/opt/afni:/data2/$USER/anaconda2/bin:/opt/fsl/bin:/opt/openmpi/bin:/usr/lib64/qt-3.3/bin:/opt/afni:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/ganglia/bin:/opt/ganglia/sbin:/usr/java/latest/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbin:/opt/pdsh/bin:/opt/rocks/bin:/opt/rocks/sbina:/home/$USER/bin:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbina:/opt/maui/bin:/opt/torque/bin:/opt/torque/sbina)

(p27) [$USER@compute-1-5 faster_rcnn]$ find /opt -name nvcc
/opt/cuda-8.0/bin/nvcc
/opt/freesurfer/lib/cuda/bin/nvcc

(p27) [$USER@compute-1-5 faster_rcnn]$ echo $LD_LIBRARY_PATH
/home/$USER/lib/cuDNN/v7/cuda/lib64:/home/$USER/cuda-9.0/lib64:/home/$USER/cuda-9.0/extras/CUPTI/lib64:/home/$USER/lib/cuDNN/v7/cuda/lib64:/home/$USER/cuda-9.0/lib64:/home/$USER/cuda-9.0/extras/CUPTI/lib64:/opt/fsl/lib:/opt/openmpi/lib:/home/$USER/cuda/lib64:/home/$USER/cuda/lib64

Regarding cffi error, I have installed both gcc and cffi using conda but I'm not sure why it still complains.

(p27) [$USER@compute-1-5 faster_rcnn]$ conda list | grep cffi
cffi                      1.11.5           py27h9745a5d_0    anaconda
(p27) [$USER@compute-1-5 faster_rcnn]$ conda list | grep gcc
gcc                       4.8.5                         7    anaconda
libgcc-ng                 7.2.0                hdf63c60_3    anaconda

Thank you for helping!!

Output:
So looks like the problem was because I had 2 CUDA installations. For some reason, when I added CUDA 8.0 to the path, it did not work but when I added CUDA 9.0 to the path, it worked.

I added this to my bashrc:

export PATH=/home/$USER/cuda-9.0/bin:"$PATH"

To check whether nvcc is correctly added to the path, I ran this command as shown here :

cat $(which nvcc).profile


Instruction: PyTorch - modifications of autograd variables

Input:
In my PyTorch program, I have a matrix which is continuously updated over runtime.

I wonder how to perform this update. I tried using something like this:

matrix[0, index] = hidden[0]

Both matrix and hidden are autograd Variables. When using the example above I'm getting the following error message:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I wonder how to get around this and perform the update without using in-place operations.

Probably slicing the matrix and creating a new matrix with torch.cat would work, but this doesn't seem to be a very nice solution.

Is there a better way doing this?

Thanks in advance!

Output:
Maybe posting a piece of code could help, but have you tried using a dataset ? You could sequentially run through data efficiently with it.

http://pytorch.org/docs/master/data.html#torch.utils.data.TensorDataset


Instruction: Loss doesn't decrease in training the pytorch RNN

Input:
Here is the RNN network I designed for a sentiment.

class rnn(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.i2h = nn.Linear(input_size, hidden_size)
        self.h2o = nn.Linear(hidden_size, output_size)
        self.h2h = nn.Linear(hidden_size , hidden_size)
        self.relu = nn.Tanh()
        self.sigmoid = nn.LogSigmoid()

    def forward(self, input, hidden):
        hidden_new = self.relu(self.i2h(input)+self.h2h(hidden))
        output = self.h2o(hidden)
        output = self.sigmoid(output)
        return output, hidden_new

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

Then, I create and train the network as :

RNN = rnn(50, 50, 1)
learning_rate = 0.0005
criteria = nn.MSELoss()
optimizer = optim.Adam(RNN.parameters(), lr=learning_rate)
hidden = RNN.init_hidden()
epochs = 2
for epoch in range(epochs):
    for i in range(len(train['Phrase'])):
        input = convert_to_vectors(train['Phrase'][i])
        for j in range(len(input)):
            temp_input = Variable(torch.FloatTensor(input[j]))
            output, hidden = RNN(temp_input, hidden)
        temp_output = torch.FloatTensor([np.float64(train['Sentiment'][i])/4])
        loss = criteria( output, Variable(temp_output))
        loss.backward(retain_graph = True)
        if (i%20 == 0):
            print('Current loss is ', loss)

The problem is that the loss of the network isn't decreasing. It increases, then decreases and so on. It isn't stable at all. I tried using a smaller learning rate but it doesn't seem to help.

Why is this happening and how can I rectify this?

Output:
You just need to call optimizer.step() after you do loss.backward().

Which, by the way, illustrates a common misconception: Backpropagation is not a learning algorithm, it's just a cool way of computing the gradient of the loss w.r.t. your parameters. You then use some variant of Gradient Descent (eg. plain SGD, AdaGrad, etc., in your case Adam) to update the weights given the gradients.


Instruction: PyTorch: purpose of addmm function

Input:
What is the purpose of the following PyTorch function (doc):

torch.addmm(beta=1, mat, alpha=1, mat1, mat2, out=None)

More specifically, is there any reason to prefer this function instead of just using

beta * mat + alpha * (mat1 @ mat2)

Output:
The addmm function is an optimized version of the equation beta*mat + alpha*(mat1 @ mat2). I ran some tests and timed their execution.

If beta=1, alpha=1, then the execution of both the statements (addmm and manual) is approximately the same (addmm is just a little faster), regardless of the matrices size.
If beta and alpha are not 1, then addmm is two times faster than the manual execution for smaller matrices (with total elements in order of 105). But, if matrices are large (in order of 106), the speedup seems negligible (39ms v/s 41ms)


Instruction: Import error when importing torch

Input:
I didn't make any change but it suddenly doesn't work.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cheng/anaconda/envs/py36/lib/python3.6/site-packages/torch/__init__.py", line 56, in <module>
    from torch._C import *
ImportError: numpy.core.multiarray failed to import

I tried pip install numpy -I but it didn't work.

My OS is Mac OS X, and I tried to install by both pip and conda. Reinstall the env didn't work as well.

pytorch version 0.3.1

numpy version 1.14.2

Solution:

Do not name your file tokenize.py!

Output:
Solution according to OP:

A file named tokenize.py in the current directory caused this issue.


Instruction: how to serve pytorch or sklearn models using tensorflow serving

Input:
I have found tutorials and posts which only says to serve tensorflow models using tensor serving.
In model.conf file, there is a parameter model_platform in which tensorflow or any other platform can be mentioned. But how, do we export other platform models in tensorflow way so that it can be loaded by tensorflow serving.

Output:
I'm not sure if you can. The tensorflow platform is designed to be flexible, but if you really want to use it, you'd probably need to implement a C++ library to load your saved model (in protobuf) and give a serveable to tensorflow serving platform. Here's a similar question.

I haven't seen such an implementation, and the efforts I've seen usually go towards two other directions:

Pure python code serving a model over HTTP or GRPC for instance. Such as what's being developed in Pipeline.AI
Dump the model in PMML format, and serve it with a java code.


Instruction: Pytorch super(class, self).__init__() vs super().__init__()

Input:
So far out of all the paper and its corresponding code I see people use super(class, self).__init__() instead of super().__init__() within their def __init__ function regardless of which python version people are using. Why? I thought they are the same thing but different syntax for different version of python. Am I paranoid or is there an actual reason to use the old format in pytorch?

Edited: It's for backward compatibility and nothing else.

Output: I asked on the official form and it seems to be just for backward compatibility and nothing else.


Instruction: How is the output h_n of an RNN (nn.LSTM, nn.GRU, etc.) in PyTorch structured?

Input:
The docs say

  h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len

Now, the batch and hidden_size dimensions are pretty much self-explanatory. The first dimension remains a mystery, though.

I assume, that the hidden states of all "last cells" of all layers are included in this output. But then what is the index of, for example, the hidden state of the "last cell" in the "uppermost layer"? h_n[-1]? h_n[0]?

Is the output affected by the batch_first option?

Output:
The implementation of LSTM and GRU in pytorch automatically includes the possibility of stacked layers of LSTMs and GRUs.

You give this with the keyword argument nn.LSTM(num_layers=num_layers). num_layers is the number of stacked LSTMs (or GRUs) that you have. The default value is 1, which gives you the basic LSTM.

num_directions is either 1 or 2. It is 1 for normal LSTMs and GRUs, and it is 2 for bidirectional RNNs.

So in your case, you probably have a simple LSTM or GRU so the value of num_layers * num_directions would then be one.

h_n[0] is the hidden state of the bottom-most layer (the one which takes in the input), and h_n[-1] of the top-most layer (the one which outputs the output of the network).

batch_first puts the batch dimension before the time dimension (the default being the time dimension before the batch dimension), because the hidden state doesn't have a time dimension, batch_first has no effect on the hidden state shape.


Instruction: How to combine multiple h5 files?

Input: Limited by the device, I could only produce several h5 files (the format of each file are same with shape of [idx, 1, 224, 224]) for huge dataset (>100GB) and now I'm confused about the solution to combine these files into a single one for further training on PyTorch. enter image description here

Output:
In h5py, groups and files support copy(), which can be used to move groups (including the root group) and their contents between files.

See the docs here (scroll down a bit to find copy()):

http://docs.h5py.org/en/latest/high/group.html

The HDF5 distribution also includes a command-line tool called h5copy that can be used to move things around, and the C API has an H5Ocopy() function.


Instruction: cuda runtime error (48): no kernel image is available for execution on the device

Input:
I'm new to pytorch. I took the code from this repository https://github.com/ruotianluo/ImageCaptioning.pytorch and wanted to make captions to the images. Installed "CUDA" and when I run the script to create the annotations produces this:

  $ CUDA_LAUNCH_BLOCKING = 1 python eval.py --model model.pth -
  -infos_path infos.pkl --image_folder blah --num_images 1 /home/azat/anaconda2/lib/python2.7/site-packages/h5py/init.py:36:
  FutureWarning: Conversion of the second argument of issubdtype from
  float tonp.floating is deprecated. In the future, it will be
  treated as np.float64 == np.dtype (float) .type.   from ._conv
  import register_converters as _register_converters
  /home/azat/anaconda2/lib/python2.7/site-packages/torch/cuda/init.py:97:
  UserWarning:
      Found GPU0 GeForce 820M which is of cuda capability 2.1.
      PyTorch no longer supports this GPU because it is too old.
         warnings.warn (old_gpu_warn% (d, name, major, capability 1)) DataLoaderRaw loading images from folder: blah 0 listing all images in
  directory blah DataLoaderRaw found 8 images THCudaCheck FAIL file = /
  pytorch / torch / lib / THC / generic / THCTensorMathPairwise.cu line
  = 40 error = 48: no kernel image is available for execution on the device Traceback (most recent last call last):   File "eval.py", line
  122, in
      vars (opt))   File "/ home / azat / Programing / Python / techno_atom_neuro / Others Implementation /
  ImageCaptioning.pytorch-master / eval_utils.py", line 82, in
  eval_split
      data = loader.get_batch (split)   File "/ home / azat / Programing / Python / techno_atom_neuro / Others Implementation /
  ImageCaptioning.pytorch-master / dataloaderraw.py", line 112, in
  get_batch
      img = Variable (preprocess (img), volatile = True)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py",
  line 42, in call
      img = t (img)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/transforms.py",
  line 118, in call
      return F.normalize (tensor, self.mean, self.std)   File "/home/azat/anaconda2/lib/python2.7/site-packages/torchvision/transforms/functional.py",
  line 161, in normalize
      t.sub_ (m) .div_ (s) RuntimeError: cuda runtime error (48): no kernel image is available for execution on the device at
  /pytorch/torch/lib/THC/generic/THCTensorMathPairwise.cu:40

So, I want to find out what this error is, hardware or software. And how can I solve this problem. Thanks.

PyTorch
OS: Ubuntu 16.04
PyTorch version: 0.3.1 pip
Python version: Python 2.7.14 :: Anaconda custom (64-bit)
CUDA/cuDNN version: 9.1
GPU models and configuration:

$ nvidia-smi
Wed Apr 11 21:34:08 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.48                 Driver Version: 390.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 820M        Off  | 00000000:08:00.0 N/A |                  N/A |
| N/A   54C    P0    N/A /  N/A |    114MiB /  1985MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

Output:
The answer is buried in that huge error message you got:

  UserWarning: Found GPU0 GeForce 820M which is of cuda capability 2.1. PyTorch no longer supports this GPU because it is too old.


Instruction: Google Colaboratory local runtime using local GPU

Input:
I'm using Colaboratory and Pytorch to run a GAN that uses an unusual dataset, which is currently stored locally on my machine. To access these files I connected to a local runtime (as per https://research.google.com/colaboratory/local-runtimes.html). However, Colaboratory now uses my own GPU when running now, which it did not do on previous runs. I know this because current runs run much much slower, as they are using my GTX 1060 6GB instead of Colab's Tesla K80.

I checked this using

torch.cuda.get_device_name(0)

Which returns "GeForce GTX 1060 6G" when I am connected locally. This is the case even with Edit -> Notebook Settings -> Hardware Accelerator -> "GPU" selected.

However, when I am not connected locally, and instead use the (default) "Connect to hosted runtime" option,

torch.cuda.get_device_name(0) does return "Tesla K80".

I've had trouble uploading my dataset to Drive, as it is a large image dataset, and would like to carry on using the local runtime.

How do I use both the local runtime and Colab's amazing Tesla K80?
Any help would be much appreciated.

Output: Colab is using your GPU because you connected it to a local runtime. That's what connecting it to your own runtime means. It means that you're using your machine instead of handling the process on Google's servers. If you want to still use Google's servers and processing capabilities, I'd suggest looking into connecting your Google Drive to the Colaboratory runtime.


Instruction: Could not import pytorch in windows

Input:
I am trying to install torch in windows to use with python. I have install torch and lua somehow.
Then I tried using:

conda install -c peterjc123 pytorch

from this answer here.
It seems to have succeeded and asks for a new package to be installed:

  The following NEW packages will be INSTALLED:

pytorch: 0.3.1-py36_cuda80_cudnn6he774522_2 peterjc123

I don't have a GPU but thought it might use CPU (I am not sure about that but I proceeded anyway).

The problem is that after the installation I cannot import it:

import PyTorch

  ModuleNotFoundError: No module named 'pytorch'

The code I tried was from here. Does this seem normal? I would expect an error regarding functionality or something not about importing the module.

P.S. My initial intention is to run this code which uses torch. I am not sure how torch and pytorch are connected but I was hoping to make it work.

Output: Use import torch to import the complete pyTorch library.


Instruction: How to install pytorch in Anaconda with conda or pip?

Input:
I am trying to install pytorch in Anaconda to work with Python 3.5 in Windows. Following the instructions in pytorch.org I introduced the following code in Anaconda:

pip3 install torch torchvision

But the following error came in:

Command "python setup.py egg_info" failed with error code 1 in C:\Users\sluis\AppData\Local\Temp\pip-install-qmrvz7b9\torch\

By searching on the web I found out that it may be because of setuptools being out of date but I checked and have it updated. I also tried:

conda install -c peterjc123 pytorch cuda80

But the following error arise:

The following specifications were found to be in conflict:
  - pytorch
Use "conda info <package>" to see the dependencies for each package.

I also tried to load the pytorch's tar.bz2 file which I download in the following website:

anaconda.org/peterjc123/pytorch/files

And then just do:

$ conda install filename.tar.bz2

But I got the following error:

Error: HTTPError: 404 Client Error: None for url: file:///C|/Users/sluis/pytorch-0.3.1-py36_cuda80_cudnn6he774522_2.tar.bz2: file:///C|/Users/sluis/pytorch-0.3.1-py36_cuda80_cudnn6he774522_2.tar.bz2

I am quite new to this programming world so I don't really know how to dig more on the errors. Anyone knows how to get pytorch installed?

Edit: As suggested in the comments I tried:

conda install pytorch torchivsion -c pytorch

And I got the following error:

Error: Packages missing in current win-64 channels:
 - pytorch
 - torchvision

I did:

anaconda search -t conda torchvision

And tried to install dericlk/torchvision using the following command:

conda install -c derickl torchvision

But I am getting the same error:

Error: Package missing in current win-64 channels:
  - torchvision

I couldn't find any torchvisionpackages for win-64.

conda list is giving me the following:

# packages in environment at C:\Users\aaaa\AppData\Local\Continuum\Anaconda3\envs\torchenv2:
#
mkl-include               2018.0.2                      1    anaconda
certifi                   2016.2.28                py35_0
cffi                      1.10.0                   py35_0
cmake                     3.6.3                    vc14_0  [vc14]
openmp                    2018.0.0                intel_8    intel
mkl                       2017.0.3                      0
numpy                     1.13.1                   py35_0
pip                       10.0.0                    <pip>
pip                       9.0.1                    py35_1
pycparser                 2.18                     py35_0
python                    3.5.4                         0
pyyaml                    3.12                     py35_0
setuptools                36.4.0                   py35_1
typing                    3.6.2                    py35_0
vc                        14                            0
vs2015_runtime            14.0.25420                    0
wheel                     0.29.0                   py35_0
wincertstore              0.2                      py35_0
zlib                      1.2.11                   vc14_0  [vc14]

=======

Output:
The following worked for me. First install MKL:

conda install -c anaconda mkl

After this, install pytorch and torchvision:

conda install -c pytorch pytorch torchvision


Instruction: How does one implement adversarial examples in pytorch?

Input:
I wanted to reproduce:

from the paper https://arxiv.org/pdf/1312.6199.pdf. I was wondering, how does one actually implement this in pytorch? My main confusion is that for loss_f I am using a torch.nn.CrossEntropy() criterion for example. Do i just need to change the code that I already have from:

loss = criterion(outputs+r, labels)
loss.backward()

to:

loss = criterion(outputs+r, labels)
loss = loss + c * r.norm(2)
loss.backward()

or something along those lines (of course include r in the optimizer!). I know its not quite right cuz I did not explicitly show how I implemented x+r or the hypercube constraint but those are parts that I still need to figure out.

I think for the moment I want to focus first without the hypercube constraint. If we assume I am ok with going out of that is the above correct? I just want to know if:

loss = loss + c * r.norm(2)

works as it should.

Now if we do include the hypercube constraint how does my solution change? Is that were the "penalty function method" come into place?

https://discuss.pytorch.org/t/how-does-one-implement-adversarial-examples-in-pytorch/15668

Output:
This is how i did it. Hope that helps you.

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Sat Nov 18 12:39:16 2016

@author: manojacharya
"""

import torch
import torch.nn as nn
from torch.autograd import Variable
from torchvision import models,transforms
import numpy as np
from scipy.misc import imread, imresize
import os
import matplotlib.pyplot as plt
import torch.nn.functional as F
import json

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])

transform = transforms.Compose(
    [transforms.ToTensor(),
     normalize])

def imshow(inp, title=None):
    """Imshow for Tensor."""
    plt.figure()
    inp = inp.data[0]
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    plt.imshow(inp)
    plt.axis('off')
    if title is not None:
        plt.title(title)

with open('imagenet.json') as f:
    imagenet_labels = json.load(f)

# In[model]:

model = models.vgg16(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

def predict(img):
    pred_raw = model(img)
    pred = F.softmax(pred_raw)
    _,indices = torch.topk(pred,k=1)
    for ind in indices.data.numpy().ravel():
        print ("%.2f%% , class: %s , %s" %(100*pred.data[0][ind],str(ind),imagenet_labels[ind]))

# In[image ]:

peppers = imread("dog.jpg")
img =  imresize(peppers,(224,224))
imgtensor = transform(img)
imgvar = Variable(imgtensor.unsqueeze(0),requires_grad=False)
imgvard = Variable(imgtensor.unsqueeze(0),requires_grad=True)
optimizer = torch.optim.Adam([imgvard], lr = 0.1)
loss_fn =  nn.CrossEntropyLoss()

label  =  torch.LongTensor(1)
#classify the object as this label
label[0] = 80
label = Variable(label)
eps = 2/255.0

#%%
Nepochs = 50
print ("Starting ...........",predict(imgvar))

for epoch in range(Nepochs):
    optimizer.zero_grad()
    pred_raw = model(imgvard)
    loss  =  loss_fn(pred_raw,label)

    diff = imgvard.data - imgvar.data
    imgvard.data = torch.clamp(torch.abs(diff),max=eps) + imgvar.data

    loss.backward()
    optimizer.step()

    print('epoch: {}/{}, loss: {}'.format(
                epoch + 1,Nepochs, loss.data[0]))
    predict(imgvard)
print('Finished Training')

#%%
imshow(imgvard)

#%%
plt.figure()
diffimg = diff[0].numpy()
diffimg = diffimg.transpose((1,2,0))
plt.imshow(diffimg)


Instruction: RNN Implementation

Input:
I am going to implement RNN using Pytorch . But , before that , I am having some difficulties in understanding the character level one-hot encoding which is asked in the question .

Please find below the question

Choose the text you want your neural network to learn, but keep in mind that your
data set must be quite large in order to learn the structure! RNNs have been trained
on highly diverse texts (novels, song lyrics, Linux Kernel, etc.) with success, so you
can get creative. As one easy option, Gutenberg Books is a source of free books where
you may download full novels in a .txt format.
We will use a character-level representation for this model. To do this, you may use
extended ASCII with 256 characters. As you read your chosen training set, you will
read in the characters one at a time into a one-hot-encoding, that is, each character
will map to a vector of ones and zeros, where the one indicates which of the characters
is present:

char → [0, 0, · · · , 1, · · · , 0, 0]
Your RNN will read in these length-256 binary vectors as input.

So , For example , I have read a novel in python. Total unique characters is 97. and total characters is somewhere around 300,000 .

So , will my input be 97 x 256 one hot encoded matrix ?

or will it be 300,000 x 256 one hot encoded matrix ?

Output:
One hot assumes each of your vector should be different in one place. So if you have 97 unique character then i think you should use a 1-hot vector of size ( 97 + 1 = 98). The extra vector maps all the unknown character to that vector. But you can also use a 256 length vector. So you input will be:

B x N x V ( B = batch size, N = no of characters , V = one hot vector size).

But if you are using libraries they usually ask the index of characters in vocabulary and they handle index to one hot conversion. Hope that helps.


Instruction: PyTorch - RuntimeError: Assertion 'cur_target >= 0 && cur_target < n_classes' failed

Input:
I'm trying to make a neural network with PyTorch to predict student's final exam grades. I've done it like this -

# Hyper Parameters
input_size = 2
hidden_size = 50
num_classes =21
num_epochs = 500
batch_size = 5
learning_rate = 0.1

# define a customise torch dataset
class DataFrameDataset(torch.utils.data.Dataset):
def __init__(self, df):
    self.data_tensor = torch.Tensor(df.as_matrix())

# a function to get items by index
def __getitem__(self, index):
    obj = self.data_tensor[index]
    input = self.data_tensor[index][0:-1]
    target = self.data_tensor[index][-1] - 1
    return input, target

# a function to count samples
def __len__(self):
    n, _ = self.data_tensor.shape
    return n

# load all data
data_i = pd.read_csv('dataset/student-mat.csv', header=None,delimiter=";")
data = data_i.iloc[:,30:33]

# normalise input data
for column in data:
# the last column is target
if column != data.shape[1] - 1:
    data[column] = data.loc[:, [column]].apply(lambda x: (x - x.mean()) / x.std())

# randomly split data into training set (80%) and testing set (20%)
msk = np.random.rand(len(data)) < 0.8
train_data = data[msk]
test_data = data[~msk]

# define train dataset and a data loader
train_dataset = DataFrameDataset(df=train_data)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

# Neural Network
class Net(nn.Module):
def __init__(self, input_size, hidden_size, num_classes):
    super(Net, self).__init__()
    self.fc1 = nn.Linear(input_size, hidden_size)
    self.sigmoid = nn.Sigmoid()
    self.fc2 = nn.Linear(hidden_size, num_classes)

def forward(self, x):
    out = self.fc1(x)
    out = self.sigmoid(out)
    out = self.fc2(out)
    return out

net = Net(input_size, hidden_size, num_classes)

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Rprop(net.parameters(), lr=learning_rate)

# store all losses for visualisation
all_losses = []

# train the model by batch
for epoch in range(num_epochs):
    for step, (batch_x, batch_y) in enumerate(train_loader):
        # convert torch tensor to Variable
        X = Variable(batch_x)
        Y = Variable(batch_y.long())

        # Forward + Backward + Optimize
        optimizer.zero_grad()  # zero the gradient buffer
        outputs = net(X)
        loss = criterion(outputs, Y)
        all_losses.append(loss.data[0])
        loss.backward()
        optimizer.step()

        if epoch % 50 == 0:
            _, predicted = torch.max(outputs, 1)
            # calculate and print accuracy
            total = predicted.size(0)
            correct = predicted.data.numpy() == Y.data.numpy()

            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Accuracy: %.2f %%'
                  % (epoch + 1, num_epochs, step + 1,
                      len(train_data) // batch_size + 1,
                      loss.data[0], 100 * sum(correct)/total))

I'm getting an error at line loss = criterion(outputs, Y) which says -
RuntimeError: Assertion 'cur_target >= 0 && cur_target < n_classes' failed.  at /pytorch/torch/lib/THNN/generic/ClassNLLCriterion.c:62

I'm not able to figure out what I'm doing wrong as I'm pretty new to this and I've already checked the other posts here but, they don't seem to help.
The data dataframe looks like -

     30  31  32

0     5   6   6
1     5   5   6
2     7   8  10
3    15  14  15
4     6  10  10
5    15  15  15

Can anyone tell me what I am doing wrong and how can I correct it. Any help is appreciated! Thanks! :)

Output:
I had the same error in my program
and i just realized that the problem was in the number of output nodes in my network
In my program the number of output nodes of my model was not equal to the number of labels of the data
the number of output was 1 and the number of target labels was 10. then i changed the number of output to 10, there was no error


Instruction: How does one create a data set in pytorch and save it into a file to later be used?

Input:
I want to extract data from cifar10 in a specific order according to some criterion f(Image,label) (for the sake of an example lets say f(Image,label) simply computes the sum of all the pixels in Image). Then it I want to generate 1 file for the train set and 1 file for the test set that I can later load in a dataloader to use for training a neural net.

How do I do this? My current idea was simply to loop through the data with data loader with shuffle off and remember the indices of the images and the score and then sort the indices according to the score and then loop through everything again and create some giant numpy array and save it. After I save it I’d use torch.utils.data.TensorDataset(X_train, X_test) to wrap with TensorDataset and feed to DataLoader.

I think it might work for a small data set like cifar10 at the very least, right?

Another very important thing for me is that I also want to only train on the first K images (especially since I already sorted them the first K have a special meaning which I want to keep) so respecting but training only with a fraction will be important.

https://discuss.pytorch.org/t/how-does-one-create-a-data-set-in-pytorch-and-save-it-into-a-file-to-later-be-used/16742

Output:
the simplest way to save this is to just read to an array and then do numpy.save('file',data,allow_pickle =False)
to load it you then to data = numpy.load('file')

remember to set the batch size to 1 and torch.to_numpy() everything

once you do this its fairly simple to just rebuild your data loader and reload data loader with your dataset

use numpy.load('file', mmap_mode='r') if you need to get at the data without loading it all to ram (helps with those pesky 600gb datasets)

for those asking why numpy.save() whole datasets? :
sometimes your data needs post processing, batching, reshaping, and this takes a lot of CPU time. You don't want to be using that cpu time re-crunching your data before you send it to your model.

the next step up is to start using databases and servers, it does this, but better with more SQL!

in terms of using slices of data its just a mater of reloading your dataset with dataset(data[k:],label[k:]) instead of dataset(data,label)


Instruction: How can I build an RNN without using nn.RNN

Input:
I need to build an RNN (without using nn.RNN) with following specifications :

It should have set of weights [

It is a chanracter RNN.
It should have 1 hidden layer
Wxh (from input layer to hidden layer )
Whh (from the recurrent connection in the hidden layer)
W ho  (from hidden layer to output layer)
I need to use Tanh for hidden layer
I need to use softmax for output layer.

I have implemented the code . I am using CrossEntropyLoss() as loss function .
Which is giving me error as

RuntimeError                              Traceback (most recent call last)
<ipython-input-33-94b42540bc4f> in <module>()
     25         print("target ",target_tensor[timestep])
     26
---> 27         loss += criterion(output,target_tensor[timestep].view(1,n_vocab))
     28
     29     loss.backward()

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    323         for hook in self._forward_pre_hooks.values():
    324             hook(self, input)
--> 325         result = self.forward(*input, **kwargs)
    326         for hook in self._forward_hooks.values():
    327             hook_result = hook(self, input, result)

/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    145         _assert_no_grad(target)
    146         return F.nll_loss(input, target, self.weight, self.size_average,
--> 147                           self.ignore_index, self.reduce)
    148
    149

/opt/anaconda/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1047         weight = Variable(weight)
   1048     if dim == 2:
-> 1049         return torch._C._nn.nll_loss(input, target, weight, size_average, ignore_index, reduce)
   1050     elif dim == 4:
   1051         return torch._C._nn.nll_loss2d(input, target, weight, size_average, ignore_index, reduce)

RuntimeError: multi-target not supported at /opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THNN/generic/ClassNLLCriterion.c:22

Here is my code for model :

class CharRNN(torch.nn.Module):

    def __init__(self,input_size,hidden_size,output_size, n_layers = 1):

        super(CharRNN, self).__init__()
        self.input_size  = input_size
        self.hidden_size = hidden_size
        self.n_layers    = 1

        self.x2h_i = torch.nn.Linear(input_size + hidden_size, hidden_size)
        self.x2h_f = torch.nn.Linear(input_size + hidden_size, hidden_size)
        self.x2h_o = torch.nn.Linear(input_size + hidden_size, hidden_size)
        self.x2h_q = torch.nn.Linear(input_size + hidden_size, hidden_size)
        self.h2o   = torch.nn.Linear(hidden_size, output_size)
        self.sigmoid = torch.nn.Sigmoid()
        self.softmax = torch.nn.Softmax()
        self.tanh    = torch.nn.Tanh()

    def forward(self, input, h_t, c_t):

        combined_input = torch.cat((input,h_t),1)

        i_t = self.sigmoid(self.x2h_i(combined_input))
        f_t = self.sigmoid(self.x2h_f(combined_input))
        o_t = self.sigmoid(self.x2h_o(combined_input))
        q_t = self.tanh(self.x2h_q(combined_input))

        c_t_next = f_t*c_t + i_t*q_t
        h_t_next = o_t*self.tanh(c_t_next)

        output = self.softmax(h_t_next)
        return output, h_t, c_t

    def initHidden(self):
        return torch.autograd.Variable(torch.zeros(1, self.hidden_size))

    def weights_init(self,model):

        classname = model.__class__.__name__
        if classname.find('Linear') != -1:
            model.weight.data.normal_(0.0, 0.02)
            model.bias.data.fill_(0)

`

and this is the code for training the model :

`
input_tensor  = torch.autograd.Variable(torch.zeros(seq_length,n_vocab))
target_tensor = torch.autograd.Variable(torch.zeros(seq_length,n_vocab))

model   = CharRNN(input_size = n_vocab, hidden_size = hidden_size, output_size = output_size)
model.apply(model.weights_init)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr = learning_rate)

for i in range(n_epochs):
    print("Iteration", i)

    start_idx    = np.random.randint(0, n_chars-seq_length-1)
    train_data   = raw_text[start_idx:start_idx + seq_length + 1]

    input_tensor = torch.autograd.Variable(seq2tensor(train_data[:-1],n_vocab), requires_grad = True)
    target_tensor= torch.autograd.Variable(seq2tensor(train_data[1:],n_vocab), requires_grad = False).long()

    loss = 0

    h_t = torch.autograd.Variable(torch.zeros(1,hidden_size))
    c_t = torch.autograd.Variable(torch.zeros(1,hidden_size))

    for timestep in range(seq_length):

        output, h_t, c_t = model(input_tensor[timestep].view(1,n_vocab), h_t, c_t)

        loss += criterion(output,target_tensor[timestep].view(1,n_vocab))

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

    x_t = input_tensor[0].view(1,n_vocab)
    h_t = torch.autograd.Variable(torch.zeros(1,hidden_size))
    c_t = torch.autograd.Variable(torch.zeros(1,hidden_size))

    gen_seq = []

    for timestep in range(100):
        output, h_t, c_t = model(x_t, h_t, c_t)
        ix = np.random.choice(range(n_vocab), p=output.data.numpy().ravel())
        x_t = torch.autograd.Variable(torch.zeros(1,n_vocab))
        x_t[0,ix] = 1
        gen_seq.append(idx2char[ix])

    txt = ''.join(gen_seq)
    print ('----------------------')
    print (txt)
    print ('----------------------')

Can you please help me ?

Thanks in advance.

Output:
The problem is with your target tensor. It is of shape 1, n_classes, a 2D tensor, but CrossEntropyLoss expects a 1D tensor.

Or stated in other terms, you are providing a one-hot encoded target tensor, but the loss function is expecting class number from 0 to n_classes-1. Change your loss calculation to -

one_hot_target = target_tensor[timestep].view(1,n_vocab)
_, class_target = torch.max(one_hot_target, dim=1)
loss += criterion(output, class_target)


Instruction: Converting Pytorch model .pth into onnx model

Input:
I have one pre-trained model into format of .pth extension. I want to convert that into Tensorflow protobuf. But I am not finding any way to do that. I have seen onnx can convert models from pytorch into onnx and then from onnx to Tensorflow. But with that approach I got following error in the first stage of conversion.

from torch.autograd import Variable
import torch.onnx
import torchvision
import torch

dummy_input = Variable(torch.randn(1, 3, 256, 256))
model = torch.load('./my_model.pth')
torch.onnx.export(model, dummy_input, "moment-in-time.onnx")`

It gives error like this.

File "t.py", line 9, in <module>
    torch.onnx.export(model, dummy_input, "moment-in-time.onnx")
  File "/usr/local/lib/python3.5/dist-packages/torch/onnx/__init__.py", line 75, in export
    _export(model, args, f, export_params, verbose, training)
  File "/usr/local/lib/python3.5/dist-packages/torch/onnx/__init__.py", line 108, in _export
    orig_state_dict_keys = model.state_dict().keys()
AttributeError: 'dict' object has no attribute 'state_dict'

What is possible solution ?

Output:
try changing your code to this

from torch.autograd import Variable

import torch.onnx
import torchvision
import torch

dummy_input = Variable(torch.randn(1, 3, 256, 256))
state_dict = torch.load('./my_model.pth')
model.load_state_dict(state_dict)
torch.onnx.export(model, dummy_input, "moment-in-time.onnx")


Instruction: Does batch_first affect hidden tensors in Pytorch LSTMs?

Input:
Does batch_first affect hidden tensors in Pytorch LSTMs?

That is if batch_first parameter is true,
Will the hidden state be (numlayer*direction,num_batch,encoding_dim)
or (num_batch,numlayer*direction,encoding_dim)

I've tested both, both give no error.

Output:
I was thinking about the same question some time ago. Like laydog outlined, in the documentation it says

  batch_first – If True, then the input and output tensors are provided
  as (batch, seq, feature)

As I understand the question we are talking about the hidden / cell state tuple, not the actual inputs and outputs.

For me it seems pretty obvious that this does not affect the hidden state as they mention:

  (batch, seq, feature)

This clearly refers to inputs and outputs, not the state tuple which consists of two tuples with shape:

  (num_layers * num_directions, batch, hidden_size)

So I'm pretty certain the hidden and cell states are not affected by this, it also would not make sense to me changing the order hidden state tuple.

Hope this helps.


Instruction: PyTorch - How to set Activation Rules of neurons to increase efficiency of Neural Network?

Input:
I'm trying to make a Back Propagation Neural Network with PyTorch. I can successfully execute and test its accuracy, but it doesn't work very efficiently. Now, I'm supposed to increase its efficiency by setting different activation rules for neurons, so that those neurons that don't contribute to the final output get excluded (pruned) from the computations, thereby increasing the time and accuracy.

My code looks like this (extracted snippets) -

# Hyper Parameters
input_size = 20
hidden_size = 50
num_classes =130
num_epochs = 500
batch_size = 5
learning_rate = 0.1

# normalise input data
for column in data:
# the last column is target
if column != data.shape[1] - 1:
    data[column] = data.loc[:, [column]].apply(lambda x: (x - x.mean()) / x.std())

# randomly split data into training set (80%) and testing set (20%)
msk = np.random.rand(len(data)) < 0.8
train_data = data[msk]
test_data = data[~msk]

# define train dataset and a data loader
train_dataset = DataFrameDataset(df=train_data)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

# Neural Network
class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        return out
net = Net(input_size, hidden_size, num_classes)

# train the model by batch
for epoch in range(num_epochs):
    for step, (batch_x, batch_y) in enumerate(train_loader):
        # convert torch tensor to Variable
        X = Variable(batch_x)
        Y = Variable(batch_y.long())

        # Forward + Backward + Optimize
        optimizer.zero_grad()  # zero the gradient buffer
        outputs = net(X)
        loss = criterion(outputs, Y)
        all_losses.append(loss.data[0])
        loss.backward()
        optimizer.step()

        if epoch % 50 == 0:
            _, predicted = torch.max(outputs, 1)
            # calculate and print accuracy
            total = predicted.size(0)
            correct = predicted.data.numpy() == Y.data.numpy()

            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Accuracy: %.2f %%' % (epoch + 1, num_epochs, step + 1, len(train_data) // batch_size + 1, loss.data[0], 100 * sum(correct)/total))

Can someone tell me how to do that in PyTorch as I'm very new to PyTorch.

Output:
I'm not sure if that question is supposed to be on stackoverflow, but I will give you a hint anyway. You are working with a sigmoid activation function at the moment, the gradient of which vanishes if the input value is too large to small. A commonly used approach is to use the ReLU activation function (stands for rectified linear unit).

ReLU(x) is the identity for the positive domain and 0 for the negative domain, in Python that would be written as follows:

def ReLU(x):
    if(x > 0):
        return x
    else:
        return 0

It should be readily available in PyTorch


Instruction: PyTorch print output to 2dp

Input:
Is there a way to print the output of a CNN model evaluation to 2dp given that the output is a multi element FloatTensor?

eg.

prediction = torch.exp(model(image2))
print(prediction)

Out:

Variable containing:
 2.84e-01  1.68e-07  7.16e-01
[torch.FloatTensor of size 1x3]

It would be better if I could output the value as:

Variable containing:
 0.28  0.00  0.72
[torch.FloatTensor of size 1x3]

I've tried:

print("%.2f" % prediction)

and using:

torch.set_printoptions(precision=2)

But neither give the desired effect.

I had a look on the documentation page:

http://pytorch.org/docs/master/torch.html#creation-ops

...under 'torch.set_printoptions' but I can't see how any of the arguments might help in this situation.

Many thanks in advance!

Output:
This has now been implemented. Use

torch.set_printoptions(sci_mode=False)

https://pytorch.org/docs/stable/torch.html#torch.set_printoptions


Instruction: pytorch cnn stride error

Input:
I am now using pytorch 0.4.0 in windows to build a CNN and here is my code:

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=(1,3),stride=1 )

        self.conv2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(1,3), stride=1)

        self.dense1 = nn.Linear(32 * 28 * 24, 60)
        self.out = nn.Linear(60,3)

    def forward(self, input):
        x = F.relu(self.conv1(input))
        x = F.relu(self.conv2(x))
        x = x.view(x.size(0), -1) # flatten（batch，32*7*7）
        x = self.dense1(x)
        output = self.out(x)
        return output

but I get the error that

File "D:\Anaconda\lib\site-packages\torch\nn\modules\conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)

RuntimeError: expected stride to be a single integer value or a list of 1 values to match the convolution dimensions, but got stride=[1, 1]

I think it shows that I made some mistakes in the code above, but I don't know how to fix it, can any one help me out? Thanks in advance!

Output:
Fine, maybe I know what's happening cause I've faced same runtime error just 4 or 5 hours ago.

Here is my solution in my case(I defined the dataset by myself):

The image I feed into the net is 1 channel, same as your code(self.conv1 = nn.Conv2d(in_channels=1,...)). And the attribute of the image which would bring runtime error is as followed:

error_img

The image I fixed is as followed:

fixed_img

Can you feel the difference? The input image's channel should be 1, so the img.shape()should be tuple! Use img.reshape(1,100,100)to fix it and the net's forward function would proceed.

I hope it can help you.


Instruction: Is it possible to view the code of a torch pretrained network

Input:
If you're thinking such a noob while reading the title - yes I am.

I've googled but didn't find a single guide that allowed me to view how a pre-trained torch neural network is designed/coded. I have already downloaded the pre-trained network (file format .t7) and I have torch installed. Can anyone help me view how it is coded (what size filters used, parameters used etc.)?

May be it's not on google because it's not possible? Will be happy to answer any additional questions you have or if anything isn't clear.

Thank You.

Output:
I think it is not possible to get the underlying code. But you can get a summary of the model which includes the layers and the main parameters just by using print.

model = SumModel(vocab_size=vocab_size, hiddem_dim=hidden_dim, batch_size=batch_size)
# saving model
torch.save(model, 'test_model.save')
# print summary of original
print(' - original model summary:')
print(model)
print()

# load saved model
loaded_model = torch.load('test_model.save')
# print summary of loaded model
print(' - loaded model summary:')
print(loaded_model)

This will output a summary which looks like this.

  - original model summary:
SumModel(
  (word_embedding): Embedding(530734, 128)
  (encoder): LSTM(128, 128, batch_first=True)
  (decoder): LSTM(128, 128, batch_first=True)
  (output_layer): Linear(in_features=128, out_features=530734, bias=True)
)

 - loaded model summary:
SumModel(
  (word_embedding): Embedding(530734, 128)
  (encoder): LSTM(128, 128, batch_first=True)
  (decoder): LSTM(128, 128, batch_first=True)
  (output_layer): Linear(in_features=128, out_features=530734, bias=True)
)

Tested with Pytorch 0.4.0

As you can see both outputs for the original and the loaded model are consistent.

I hope this helps.


Instruction: Pytorch vs. Keras: Pytorch model overfits heavily

Input:
For several days now, I'm trying to replicate my keras training results with pytorch. Whatever I do, the pytorch model will overfit far earlier and stronger to the validation set then in keras. For pytorch I use the same XCeption Code from https://github.com/Cadene/pretrained-models.pytorch.

The dataloading, the augmentation, the validation, the training schedule etc. are equivalent. Am I missing something obvious? There must be a general problem somewhere. I tried thousands of different module constellations, but nothing seems to come even close to the keras training. Can somebody help?

Keras model: val accuracy > 90%

# base model
base_model = applications.Xception(weights='imagenet', include_top=False, input_shape=(img_width, img_height, 3))

# top model
x = base_model.output
x = GlobalMaxPooling2D()(x)
x = Dense(512, activation='relu')(x)
x = Dropout(0.5)(x)
predictions = Dense(4, activation='softmax')(x)

# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)

# Compile model
from keras import optimizers
adam = optimizers.Adam(lr=0.0001)
model.compile(loss='categorical_crossentropy',
optimizer=adam, metrics=['accuracy'])

# LROnPlateau etc. with equivalent settings as pytorch

Pytorch model: val accuracy ~81%

from xception import xception
import torch.nn.functional as F

# modified from https://github.com/Cadene/pretrained-models.pytorch
class XCeption(nn.Module):
    def __init__(self, num_classes):
        super(XCeption, self).__init__()

        original_model = xception(pretrained="imagenet")

        self.features=nn.Sequential(*list(original_model.children())[:-1])
        self.last_linear = nn.Sequential(
             nn.Linear(original_model.last_linear.in_features, 512),
             nn.ReLU(),
             nn.Dropout(p=0.5),
             nn.Linear(512, num_classes)
        )

    def logits(self, features):
        x = F.relu(features)
        x = F.adaptive_max_pool2d(x, (1, 1))
        x = x.view(x.size(0), -1)
        x = self.last_linear(x)
        return x

    def forward(self, input):
        x = self.features(input)
        x = self.logits(x)
        return x

device = torch.device("cuda")
model=XCeption(len(class_names))
if torch.cuda.device_count() > 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs
    model = nn.DataParallel(model)
model.to(device)

criterion = nn.CrossEntropyLoss(size_average=False)
optimizer = optim.Adam(model.parameters(), lr=0.0001)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.2, patience=5, cooldown=5)

Thank you very much!

Update:
Settings:

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=lr)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.2, patience=5, cooldown=5)

model = train_model(model, train_loader, val_loader,
                        criterion, optimizer, scheduler,
                        batch_size, trainmult=8, valmult=10,
                        num_epochs=200, epochs_top=0)

Cleaned training function:

def train_model(model, train_loader, val_loader, criterion, optimizer, scheduler, batch_size, trainmult=1, valmult=1, num_epochs=None, epochs_top=0):
  for epoch in range(num_epochs):
    for phase in ['train', 'val']:
        running_loss = 0.0
        running_acc = 0
        total = 0
        # Iterate over data.
        if phase=="train":
            model.train(True)  # Set model to training mode
            for i in range(trainmult):
                for data in train_loader:
                    # get the inputs
                    inputs, labels = data
                    inputs, labels = inputs.to(torch.device("cuda")), labels.to(torch.device("cuda"))
                    # zero the parameter gradients
                    optimizer.zero_grad()
                    # forward
                    outputs = model(inputs) # notinception
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
                    # backward + optimize only if in training phase
                    loss.backward()
                    optimizer.step()
                    # statistics
                    total += labels.size(0)
                    running_loss += loss.item()*labels.size(0)
                    running_acc += torch.sum(preds == labels)
                    train_loss=(running_loss/total)
                    train_acc=(running_acc.double()/total)
        else:
            model.train(False)  # Set model to evaluate mode
            with torch.no_grad():
                for i in range(valmult):
                    for data in val_loader:
                        # get the inputs
                        inputs, labels = data
                        inputs, labels = inputs.to(torch.device("cuda")), labels.to(torch.device("cuda"))
                        # zero the parameter gradients
                        optimizer.zero_grad()
                        # forward
                        outputs = model(inputs)
                        _, preds = torch.max(outputs, 1)
                        loss = criterion(outputs, labels.data)
                        # statistics
                        total += labels.size(0)
                        running_loss += loss.item()*labels.size(0)
                        running_acc += torch.sum(preds == labels)
                        val_loss=(running_loss/total)
                        val_acc=(running_acc.double()/total)
            scheduler.step(val_loss)
    return model

Output:
it may be because type of weight initialization you are using
otherwise this should not happen
try with same initializer in both the models


Instruction: Pytorch LongTensor of large integer gives float float value

Input:
x = torch.LongTensor([108500])
print (x)

gives

1.0850e+05
[torch.LongTensor of size 1]

Why is it so? Can't we have LongTensor of large integer?
How can I convert the value of x to be integer so that I can get the embedding of x.

Output:
As you can see in the documentation, torch.LongTensor uses the data type '64-bit integer (signed)'.

So what you are seeing in your output is actually an integer, it's only the exponential notation.


Instruction: Pytorch AttributeError: module 'torch' has no attribute 'set_grad_enabled'

Input:
AttributeError: module 'torch' has no attribute 'set_grad_enabled'

Traceback (most recent call last):
  File
   "/home/thaqafi/PycharmProjects/transfer_learning/tutorial_transfer.py", line 280, in

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=25)

File
 "/home/thaqafi/PycharmProjects/transfer_learning/tutorial_transfer.py", line 179, in train_model
    with torch.set_grad_enabled(True):

AttributeError: module 'torch' has no attribute 'set_grad_enabled'

using source code form Pytorch webiste.

http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

Output:
As far as I know this feature was not available before version 0.4.0, so probably you are using a version lower than 0.4.0 (e.g. 0.3.1.).

Try updating PyTorch to version 0.4.0. if possible.


Instruction: DataLoader not randomly sampling in PyTorch

Input:
My DataLoader is returning me the same image with each epoch. My model is only looking at the same single image (indexed '0') each time (batch size is 1...although nothing changes with different batch sizes, anyways).

Here's my dataset, stripped down to the important bits:

class MyDataset(Dataset):

    def __init__(self, path, loader=pil_loader):
        self.path = path
        self.images = os.listdir(path)

    def __getitem__(self, index):
        image = self.images[index]

    . . .

And here's the DataSet:

train_ds = MyDataset('/data')

And here's my sampler:

train_sampler = RandomSampler(train_ds)

And here's my DataLoader:

train_dl = DataLoader(train_ds, batch_size=1, sampler=train_sampler)

I'm not sure why it is returning me the same image each time, during
 training.

Do I have RandomSampler incompletely set up? Or maybe I wrote the __getitem__ incorrectly? I can't figure it out.

Output:
Aha. Well, if anyone ends up here with the same issue, I figured out what it is and maybe this will help.

My definition of __len__ was wrong.

I guess the random sampler depends on how you've set up the length method.

Mine was temporarily mocked up as

def __len__(self):
    return len(0)

instead of something real, like:

def __len__(self):
    return len(self.images)