Instruction: Unable to load data Google Colab

Input:
I am trying to load data after downloading it through kaggle cli. This exercise is from this course on Udacity.

!kaggle competitions download -c dogs-vs-cats
!unzip {content}/competitions/dogs-vs-cats/train.zip -d {content}/competitions/dogs-vs-cats/
!unzip {content}/competitions/dogs-vs-cats/test1.zip -d {content}/competitions/dogs-vs-cats/

Counting the number of images

!ls '{content}/competitions/dogs-vs-cats/train/' | wc -l
# 25000

Then I try to load the data

data_dir = '{content}/competitions/dogs-vs-cats/train/'

transform = transforms.Compose([transforms.Resize(255),
                               transforms.CenterCrop(224),
                               transforms.ToTensor()]) # TODO: compose transforms here
dataset = datasets.ImageFolder(data_dir, transform=transform) # TODO: create the ImageFolder
dataloader = torch.utils.data.DataLoader(dataset,batch_size=32,shuffle=True) # TODO: use the ImageFolder dataset to create the DataLoader

Error

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-70-9c49d0bdcdb0> in <module>()
      4                                transforms.CenterCrop(224),
      5                                transforms.ToTensor()]) # TODO: compose transforms here
----> 6 dataset = datasets.ImageFolder(data_dir, transform=transform) # TODO: create the ImageFolder
      7 dataloader = torch.utils.data.DataLoader(dataset,batch_size=32,shuffle=True) # TODO: use the ImageFolder dataset to create the DataLoader

1 frames
/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in __init__(self, root, transform, target_transform, loader, is_valid_file)
    207                                           transform=transform,
    208                                           target_transform=target_transform,
--> 209                                           is_valid_file=is_valid_file)
    210         self.imgs = self.samples

/usr/local/lib/python3.6/dist-packages/torchvision/datasets/folder.py in __init__(self, root, loader, extensions, transform, target_transform, is_valid_file)
     95         if len(samples) == 0:
     96             raise (RuntimeError("Found 0 files in subfolders of: " + self.root + "\n"
---> 97                                 "Supported extensions are: " + ",".join(extensions)))
     98
     99         self.loader = loader

RuntimeError: Found 0 files in subfolders of: {content}/competitions/dogs-vs-cats/train/
Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp

Output:
Looks like you are using f-strings in a wrong way. Simply add f here:

data_dir = f'{content}/competitions/dogs-vs-cats/train/'

to include content value to the path; without f you are just using {content}/competitions... string as a path, as you can see in your error message.


Instruction: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Input:
In a pytorch model training process I get this error:

  RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.LongTensor [128, 1]] is at version 8; expected version 7 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

with stack trace

   sys:1: RuntimeWarning: Traceback of forward call that caused the error:
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel_launcher.py", line 16, in <module>
    app.launch_new_instance()
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 499, in start
    self.io_loop.start()
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/tornado/ioloop.py", line 1073, in start
    handler_func(fd_obj, events)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/arash/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 456, in _handle_events
    self._handle_recv()
  File "/home/arash/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 486, in _handle_recv
    self._run_callback(callback, msg)
  File "/home/arash/.local/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 438, in _run_callback
    callback(*args, **kwargs)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 233, in dispatch_shell
    handler(stream, idents, msg)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
    user_expressions, allow_stdin)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 208, in do_execute
    res = shell.run_cell(code, store_history=store_history, silent=silent)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 537, in run_cell
    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2714, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2818, in run_ast_nodes
    if self.run_code(code, result):
  File "/home/arash/anaconda2/envs/mzh27/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-71-a5b255596e11>", line 33, in <module>
    sampled_captions, sampled_log_probs=predict_captions(out_enc,hid_enc,enc_pp,'sample')
  File "<ipython-input-70-a6ea511f0678>", line 18, in predict_captions
    out_dec, hid_dec, word_logits = dec.forward(r, last_enc, img_features)
  File "<ipython-input-21-0601dad4805f>", line 21, in forward
    emb = self.embedding(input)
  File "/home/arash/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/arash/.local/lib/python2.7/site-packages/torch/nn/modules/sparse.py", line 117, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/home/arash/.local/lib/python2.7/site-packages/torch/nn/functional.py", line 1506, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-71-a5b255596e11> in <module>()
     54                  torch.stack(sampled_log_probs).reshape(-1)).mean()
     55         torch.autograd.set_detect_anomaly(True)
---> 56         loss.backward()
     57         clip_grad_value_(enc_optim.param_groups[0]['params'], 5.0)
     58         clip_grad_value_(dec_optim.param_groups[0]['params'], 5.0)

/home/arash/.local/lib/python2.7/site-packages/torch/tensor.pyc in backward(self, gradient, retain_graph, create_graph)
    105                 products. Defaults to ``False``.
    106         """
--> 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    108
    109     def register_hook(self, hook):

/home/arash/.local/lib/python2.7/site-packages/torch/autograd/__init__.pyc in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---> 93         allow_unreachable=True)  # allow_unreachable flag
     94
     95

But I'm unable to locate in-place operation that causes the error.
Here is my code:

for epoch in xrange(0, 13):
    print ("Starting New Epoch: %d" % epoch)
    rewards = {
        'sample_cider': [],
        'sample_context': [],
        'sample_reward': [],  # actual reward, controlled by beta
        'greedy_cider': [],
        'greedy_context': [],
        'greedy_reward': []
    }

    order = np.arange(enc_padded_text.shape[0])
    np.random.shuffle(order)
    enc_padded_text = enc_padded_text[order]
    input_text=[input_text[i] for i in order]
    dec_text_tensor.data = dec_text_tensor.data[order]

    for i in xrange(num_batches):
        s = i * BATCH_SIZE
        e = (i+1) * BATCH_SIZE

        _, enc_pp, dec_pp, enc_lengths = make_packpadded_s2s(s, e, enc_padded_text, dec_text_tensor)

        enc.zero_grad()
        dec.zero_grad()

        hid = enc.initHidden(BATCH_SIZE)

        out_enc, hid_enc = enc.forward(enc_pp, hid, enc_lengths)

        hid_enc = torch.cat([hid_enc[0,:, :], hid_enc[1,:,:]], dim=1).unsqueeze(0)
        gt_dict = dict(zip(_,input_text[s:e]))
        sampled_captions, sampled_log_probs=predict_captions(out_enc,hid_enc,enc_pp,'sample')
        sampled_dict = dict(zip(_, sampled_captions))
        with torch.no_grad():
            greedy_captions = predict_captions(out_enc,hid_enc,enc_pp, 'greedy')
            greedy_dict = dict(zip(_, greedy_captions))

        sample_cider_score, sample_context_score, sample_reward = get_scores(
            dec_pp[:,1:], sampled_captions, gt_dict, sampled_dict)
        greedy_cider_score, greedy_context_score, greedy_reward = get_scores(
            dec_pp[:,1:], greedy_captions, gt_dict, greedy_dict)

        # self-critical: score from sampling - score from test time
        advantages = torch.Tensor((sample_cider_score - greedy_cider_score).reshape(-1))

        # normalize advantages
        advantages = ((advantages - advantages.mean()) /
                      advantages.std() + 1e-9)
        if cuda:
            advantages = advantages.cuda()
        loss = -(advantages *
                 torch.stack(sampled_log_probs).reshape(-1)).mean()
        torch.autograd.set_detect_anomaly(True)
        loss.backward()
        clip_grad_value_(enc_optim.param_groups[0]['params'], 5.0)
        clip_grad_value_(dec_optim.param_groups[0]['params'], 5.0)
        enc_optim.step()
        dec_optim.step()

        rewards['sample_cider'].extend(sample_cider_score)
        rewards['sample_context'].extend(sample_context_score)
        rewards['sample_reward'].extend(sample_reward)
        rewards['greedy_cider'].extend(greedy_cider_score)
        rewards['greedy_context'].extend(greedy_context_score)
        rewards['greedy_reward'].extend(greedy_reward)

        if (b + 1) % 100 == 0:
            print('\t[Batch {} running metrics] - R train {:.2f} - R train (greedy): {:.2f}'.format(
                b + 1, np.mean(rewards['sample_reward']), np.mean(rewards['greedy_reward'])))

predict_captions function:

def predict_captions(img_features,hid_enc,enc_pp, mode='sample', constrain=False,L=22):
    dec_tensor = torch.ones((enc_pp.shape[0]), L+1, dtype=torch.long) * Toks.SOS
    global cuda
    if cuda:
        dec_tensor = dec_tensor.cuda(device=device)
    last_enc = hid_enc
    if mode == 'beam_search':
        return self.beam_search(img_features, state, lstm_states)

    predictions = []
    log_probs = []

    # this should store the index of the first occurrence of <EOS>
    # for each sample in the batch
    EOS_tracker = np.full(img_features.shape[0], None)
    for i in range(L):
        r=dec_tensor[:,i].unsqueeze(1)
        out_dec, hid_dec, word_logits = dec.forward(r, last_enc, img_features)
        out_dec[:, 0, Toks.UNK] = -np.inf # ignore unknowns
        l=out_dec[:,0]
        chosen = torch.argmax(l,dim=1)
        dec_tensor[:, i+1] = chosen
        last_enc = hid_dec
        # decoding stuff
        probs = F.softmax(word_logits, dim=2)
        probs=probs.reshape(128,20004)

        if constrain:
            # enforce constraint that the same word can't be predicted
            # twice in a row. zero-out the probability of previous words
            for p, prev_idx in zip(probs, state['prev_word_indeces']):
                p[prev_idx] = 0

        if mode == 'sample':
            idxs = torch.multinomial(probs, 1)
        else:
            idxs = torch.argmax(probs, dim=1)
        if cuda:
            idxs = idxs.cpu()
        words = [dec_idx_to_word[index] for index in idxs]
        predictions.append(np.array(words).reshape(-1))

        # get the respective log probability of chosen word
        # for each sample in the batch
        log_probs.append([lp[i] for (lp, i)
                         in zip(torch.log(probs), idxs)])

        # inefficient but this should be fast enough anyway... ? :(
        eos_idxs = (np.array(words)==dec_idx_to_word[2]).nonzero()[0]
        for idx in eos_idxs:
            if EOS_tracker[idx] is None:
                EOS_tracker[idx] = i + 1

        # finish loop if they're all done
        if len(EOS_tracker[EOS_tracker == None])==0:
            break

    # build the actual sentences, up until the first occurrence of <EOS>
    captions = [
        [' '.join(w[:eos_idx])] for (w, eos_idx) in
        zip(np.array(predictions).T, EOS_tracker)
    ]
    print captions
    # do this only when training. not needed otherwise.
    if mode == 'sample':
        log_probs = [lp[:eos_idx].sum() for (lp, eos_idx) in zip(np.array(log_probs).T, EOS_tracker)]
        return captions, log_probs

    return captions

models:

class Encoder_s2s(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(Encoder_s2s, self).__init__()
        assert hidden_size % 2 == 0

        self.hidden_size = hidden_size
        self.input_size = input_size

        self.hidden_init_tensor = torch.zeros(2, 1, self.hidden_size/2, requires_grad=True)
        nn.init.normal_(self.hidden_init_tensor, mean=0, std=0.05)
        self.hidden_init = torch.nn.Parameter(self.hidden_init_tensor, requires_grad=True)

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.emb_drop = nn.Dropout(0.2)
        self.gru = nn.GRU(hidden_size, hidden_size/2, batch_first=True, bidirectional=True)
        self.gru_out_drop = nn.Dropout(0.2)
        self.gru_hid_drop = nn.Dropout(0.3)

    def forward(self, input, hidden, lengths):
        emb = self.emb_drop(self.embedding(input))
        #emb = embedded_dropout(self.embedding, input, dropout=0.2 if self.training else 0)
        pp = torch.nn.utils.rnn.pack_padded_sequence(emb, lengths, batch_first=True)
        out, hidden = self.gru(pp, hidden)
        out = torch.nn.utils.rnn.pad_packed_sequence(out, batch_first=True)[0]
        out = self.gru_out_drop(out)
        hidden = self.gru_hid_drop(hidden)
        return out, hidden

    def initHidden(self, bs):
        return self.hidden_init.expand(2, bs, self.hidden_size/2).contiguous()

class DecoderAttn(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, out_bias):
        super(DecoderAttn, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.emb_drop = nn.Dropout(0.2)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.gru_drop = nn.Dropout(0.2)
        self.mlp = nn.Linear(hidden_size*2, output_size)
        if out_bias is not None:
            out_bias_tensor = torch.tensor(out_bias, requires_grad=False)
            self.mlp.bias.data[:] = out_bias_tensor
        self.logsoftmax = nn.LogSoftmax(dim=2)

        self.att_mlp = nn.Linear(hidden_size, hidden_size, bias=False)
        self.attn_softmax = nn.Softmax(dim=2)

    def forward(self, input, hidden, encoder_outs):
        emb = self.embedding(input)
        emb=self.emb_drop(emb)
        out, hidden = self.gru(emb, hidden)

        out_proj = self.att_mlp(out)
        enc_out_perm = encoder_outs.permute(0, 2, 1)
        e_exp = torch.bmm(out_proj, enc_out_perm)
        attn = self.attn_softmax(e_exp)

        ctx = torch.bmm(attn, encoder_outs)

        full_ctx = torch.cat([self.gru_drop(out), ctx], dim=2)

        out = self.mlp(full_ctx)
        out1 = self.logsoftmax(out)
        return out1, hidden, out

Output:
A tensor matching this description torch.cuda.LongTensor [128, 1], should narrow down your search.

A quick google search revealed that, LongTensors are most commonly  returned by min , max, sort. so the lines

l=out_dec[:,0]
chosen = torch.argmax(l,dim=1)
dec_tensor[:, i+1] = chosen

Most probably the line dec_tensor[:, i+1] = chosen seems problematic.


Instruction: What is the correct way to use OHE lookup table for a pytorch RNN?

Input:
I am currently trying to build a LSTM RNN using pytorch. One input vector is represented as an array of 50 integers corresponding to a sequence of  at most 50 tokens with padding where each integer corresponds to an element from my vocabulary and the index of the 1 in the OHE vector. I want to have an embedding layer that just uses a lookup table to One-hot encode the integer-- kind of like tensorflow's OHE layer.

Something like this "kind of" works

import torch
import numpy as np
import torch.nn as nn

# vocab_size is the number of words in your train, val and test set
# vector_size is the dimension of the word vectors you are using
vocab_size, vector_size = 5, 5
embed = nn.Embedding(vocab_size, vector_size)

# intialize the word vectors, pretrained_weights is a
# numpy array of size (vocab_size, vector_size) and
# pretrained_weights[i] retrieves the word vector of
# i-th word in the vocabulary
pretrained_weights = np.zeros((vocab_size, vector_size))
np.fill_diagonal(pretrained_weights, 1)
tmp =torch.from_numpy(pretrained_weights)
embed.weight = nn.Parameter(tmp,requires_grad=False )

# Then turn the word index into actual word vector
vocab = {"some": 0, "words": 1}
word_indexes = torch.from_numpy(np.array([vocab[w] for w in ["some", "words"]]))
word_vectors = embed(word_indexes)
word_vectors.data.numpy()

>>>output
array([[1., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0.]])

but it is very hacky, and doesn't play nicely with batches of input vectors.

What is the correct way to declare a OHE embedding layer at the begining of an RNN?

Output:
The method you are looking for is torch.nn.functional.one_hot. It was added to PyTorch later, so it's hard to find when you Google for it.

 >>> import torch
>>> x = torch.arange(0,5)
>>> x
tensor([0, 1, 2, 3, 4])
>>> torch.nn.functional.one_hot(x)
tensor([[1, 0, 0, 0, 0],
      [0, 1, 0, 0, 0],
      [0, 0, 1, 0, 0],
      [0, 0, 0, 1, 0],
      [0, 0, 0, 0, 1]])


Instruction: Why does my tensorboard say: 'No scalar data was found' when using SummaryWriter.add_scalar?

Input:
I am attempting to visualize a loss vs. epoch graph using tensorboard (by calling .addscalar()) so that I can appropriately tune my neural network model. However, after training the model, I check my tensorboard in the web browser and it reads: 'No scalar data was found.'

I've tried checking my events statistics and it seems like there are scalars:

Event statistics for runs/tf_druggability/classification_ffnn_1:
audio -
graph -
histograms -
images -
scalars
   first_step           0
   last_step            90
   max_step             90
   min_step             0
   num_steps            10
   outoforder_steps     [(90L, 0L), (90L, 0L)]
sessionlog:checkpoint -
sessionlog:start -
sessionlog:stop -
tensor -

Here's my code:

from torch.utils.tensorboard import SummaryWriter

sess = tensorflow.Session()
writer = SummaryWriter('runs/tf_druggability/classification_ffnn_1', sess.graph)

for epoch in range(100):

    # Wrap input data and labels in Variable to can gradient descent (in place of DataLoader)
    inputs = Variable(traindf_to_tensor)
    labels = Variable(trainlabels_to_tensor)

    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    if(epoch%10 == 0):

        # HERE I WRITE to the LOG FILE for TENSORBOARD:
        writer.add_scalar('training loss', loss.item(), epoch)
        print('epoch {}, loss {}'.format(epoch, loss.item()))

writer.close()
print('Finished training! :)')

When I try to open the events file in Jupyter Notebooks it has this in it (... means I've omitted the middle part):

1800 0000 0000 0000 a37f 4b22 09f2 3d6b
a51e 58d7 411a 0d62 7261 696e 2e45 7665
6e74 3a32 ac51 0a16 2100 0000 0000 0000
...
f723 0000 0000 0000 00b0 9f77 4309 d00a
0ea7 1e58 d741 105a 2a16 0a14 0a0d 7472
6169 6e69 6e67 5f6c 6f73 7315 c193 403f

Output: I discovered that the graphs only displayed if I was in the exact same directory as the program I'm running. Being in a higher directory than that of the program seems to have caused this inability to display graphs.


Instruction: Implementation of Focal loss for multi label classification

Input:
trying to write focal loss for multi-label classification

class FocalLoss(nn.Module):
    def __init__(self, gamma=2, alpha=0.25):
        self._gamma = gamma
        self._alpha = alpha

    def forward(self, y_true, y_pred):
        cross_entropy_loss = torch.nn.BCELoss(y_true, y_pred)
        p_t = ((y_true * y_pred) +
               ((1 - y_true) * (1 - y_pred)))
        modulating_factor = 1.0
        if self._gamma:
            modulating_factor = torch.pow(1.0 - p_t, self._gamma)
        alpha_weight_factor = 1.0
        if self._alpha is not None:
            alpha_weight_factor = (y_true * self._alpha +
                                   (1 - y_true) * (1 - self._alpha))
        focal_cross_entropy_loss = (modulating_factor * alpha_weight_factor *
                                    cross_entropy_loss)
        return focal_cross_entropy_loss.mean()

But when i run this i get

  File "train.py", line 82, in <module>
    loss = loss_fn(output, target)
  File "/home/bubbles/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 538, in __call__
    for hook in self._forward_pre_hooks.values():
  File "/home/bubbles/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'FocalLoss' object has no attribute '_forward_pre_hooks'

Any suggestions would be really helpful, Thanks in advance.

Output:
You shouldn't inherit from torch.nn.Module as it's designed for modules with learnable parameters (e.g. neural networks).
Just create normal functor or function and you should be fine.
BTW. If you inherit from it, you should call super().__init__() somewhere in your __init__().
EDIT
Actually inheriting from nn.Module might be a good idea, it allows you to use the loss as part of neural network and is common in PyTorch implementations/PyTorch Lightning.


Instruction: Use PyTorch to adjust Tensor matrix values based on numbers I calculate from the Tensors?

Input:
I have two tensors (matrices) that I've initialized:

sm=Var(torch.randn(20,1),requires_grad=True)
sm = torch.mm(sm,sm.t())
freq_m=Var(torch.randn(12,20),requires_grad=True)

I am creating two lists from the data inside these 2 matrices, and I am using spearmanr to get a correlation value between these 2 lists. How I am creating the lists is not important, but the goal is to adjust the values inside the matrices so that the calculated correlation value is as close to 1 as possible.

If I were to solve this problem manually, I would tweak values in the matrices by .01 (or some small number) each time and recalculate the lists and correlation score. If the new correlation value is higher than the previous one, I would save the 2 matrices and tweak a different value until I get the 2 matrices that give me the highest correlation score possible.

Is PyTorch capable of doing this automatically? I know PyTorch can adjust based on an equation but the way I want to adjust the matrix values is not against an equation, it's against a correlation value that I calculate. Any guidance with this is greatly appreciated!

Output:
Pytorch has an autograd package, that means if you have variable and you pass them through differentiable functions and get a scalar result, you can perform a gradient descent to update the variable to lower or augment the scalar result.

So what you need to do is to define a function f that works on tensor level such that f(sm, freq_m) will give you the desired correlation.

Then, you should do something like:

lr = 1e-3
for i in range(100):
  # 100 updates
  loss = 1 - f(sm, freq_m)
  print(loss)
  loss.backward()
  with torch.no_grad():
    sm -= lr * sm.grad
    freq_m -= lr * freq_m.grad
    # Manually zero the gradients after updating weights
    sm.grad.zero_()
    freq_m.grad.zero_()

The learning rate is basically the size of the step you do, a learning rate too high will cause the loss to explode, and a learning rate too little will cause a slow convergence, I suggest you experiment.

Edit : To answer the comment on loss.backward : for any differentiable function f, f is a function of multiple tensors t1, ..., tn with requires_grad=True as a result, you can calculate the gradient of the loss with respect to each of those tensors. When you do loss.backward, it calculates those gradients and store those in t1.grad, ..., tn.grad. Then you update t1, ..., tn using gradient descent in order to lower the value of f. This update doesn't need a computational graph, so this is why you use with torch.no_grad().
At the end of the loop, you zero the gradients because .backward doesn't overwrite the gradients but rather add the new gradients to them. More on that here : https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903


Instruction: PyTorch: wrapping multiple records in one file?

Input:
Is there a standard way of encoding multiple records (in this case, data from multiple .png or .jpeg images) in one file that PyTorch can read? Something similar to TensorFlow's "TFRecord" or MXNet's "RecordIO", but for PyTorch.

I need to download image data from S3 for inference, and it's much slower if my image data is in many small .jpg files rather than fewer files.

Thanks.

Output: One thing is to store batches of images together in a single npz file. Numpy's np.savez lets you save multiple arrays compressed into a single file. Then load the file as np arrays and use torch.from_numpy to convert to tensors.


Instruction: Loading custom dataset of images using PyTorch

Input:
I'm using the coil-100 dataset which has images of 100 objects, 72 images per object taken from a fixed camera by turning the object 5 degrees per image. Following is the folder structure I'm using:

  data/train/obj1/obj01_0.png, obj01_5.png ... obj01_355.png
  .
  .
  data/train/obj85/obj85_0.png, obj85_5.png ... obj85_355.png
  .
  .
  data/test/obj86/obj86_0.ong, obj86_5.png ... obj86_355.png
  .
  .
  data/test/obj100/obj100_0.ong, obj100_5.png ... obj100_355.png

I have used the imageloader and dataloader classes. The train and test datasets loaded properly and I can print the class names.

train_path = 'data/train/'
test_path = 'data/test/'
data_transforms = {
    transforms.Compose([
    transforms.Resize(224, 224),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
}

train_data = torchvision.datasets.ImageFolder(
    root=train_path,
    transform= data_transforms
)
test_data = torchvision.datasets.ImageFolder(
    root = test_path,
    transform = data_transforms
)
train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=None,
    num_workers=1,
    shuffle=False
)
test_loader = torch.utils.data.DataLoader(
    test_data,
    batch_size=None,
    num_workers=1,
    shuffle=False
)

print(len(train_data))
print(len(test_data))
classes = train_data.class_to_idx
print("detected classes: ", classes)

In my model I wish to pass every image through pretrained resnet and make a dataset from the output of resnet to feed into a biderectional LSTM.
For which I need to access the images by classname and index.
for ex. pre_resnet_train_data['obj01'][0] should be obj01_0.png and post_resnet_train_data['obj01'][0] should be the resnet output of obj01_0.png and so on.
I'm a beginner in Pytorch and for the past 2 days, I have read many tutorials and stackoverflow questions about creating a custom dataset class but couldn't figure out how to achieve what I want.
please help!

Output:
Assuming you only plan on running resent on the images once and save the output for later use, I suggest you write your own data set, derived from ImageFolder.
Save each resnet output at the same location as the image file with .pth extension.

class MyDataset(torchvision.datasets.ImageFolder):
  def __init__(self, root, transform):
    super(MyDataset, self).__init__(root, transform)

  def __getitem__(self, index):
    # override ImageFolder's method
    """
    Args:
      index (int): Index
    Returns:
      tuple: (sample, resnet, target) where target is class_index of the target class.
    """
    path, target = self.samples[index]
    sample = self.loader(path)
    if self.transform is not None:
      sample = self.transform(sample)
    if self.target_transform is not None:
      target = self.target_transform(target)
    # this is where you load your resnet data
    resnet_path = os.path.join(os.path.splitext(path)[0], '.pth')  # replace image extension with .pth
    resnet = torch.load(resnet_path)  # load the stored features
    return sample, resnet, target


Instruction: Finetune PyTorch model after training fc layers

Input:
I am trying to do Transfer Learning using PyTorch. I wan to train fc layers first and then to finetune the whole network. Unfortunately after training fc layers and then passing my network to finetune, I am losing the accuracy that was acquired in the first training. Is this an expected behaviour or am I doing something wrong here?

Here is the code:

model = torchvision.models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False

num_ftrs = model.fc.in_features
model.fc = nn.Linear(num_ftrs, 4)
model = model.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

model = trainer.fit_model(dataloader, model, criterion, optimizer, num_epochs=10)
# fit model is basic PyTorch training function found here: https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html#convnet-as-fixed-feature-extractor The only difference is that scheduler is an optional param.

for param in model.parameters():
    param.requires_grad = True

torch.cuda.empty_cache()
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

# Here I am finetuning the model
model_ft = trainer.fit_model(
    dataloader, model, criterion, optimizer, scheduler=exp_lr_scheduler, num_epochs=10
)

Am I missing something here or should I just train the model once?

Output:
That is something that can happen when performing transfer learning called
 catastrophic forgetting. Basically, you update your pretrained weights too much and you 'forget' what was previously learned. This can happen notably if your learning rate is too high. I would suggest trying at first a lower learning rate, or using diffentiable learning rate (different learning rate for the head of the network and the pretrained part, so that you can have a higher learning rate on the fc layers than for the rest of the network).


Instruction: Implementing Batch for Image Segmentation

Input:
I wrote a Python 3.5 script for doing street segmentation. Since I'm new in Image Segementation, I did not use predefined dataloaders from pytorch, instead I wrote them by my self (for better understanding). Until now I only use a batch size of 1. Now I want to generalize this for arbitrary batch sizes.

This is a snippet of my Dataloader:

def augment_data(batch_size):

    # [...] defining some paths and data transformation (including ToTensor() function)

    # The images are named by numbers (Frame numbers), this allows me to find the correct label image for a given input image.
    all_input_image_paths = {int(elem.split('\\')[-1].split('.')[0]) : elem for idx, elem in enumerate(glob.glob(input_dir + "*"))}
    all_label_image_paths = {int(elem.split('\\')[-1].split('.')[0]) : elem for idx, elem in enumerate(glob.glob(label_dir + "*"))}

    dataloader = {"train":[], "val":[]}
    all_samples = []
    img_counter = 0

    for key, value in all_input_image_paths.items():
        input_img = Image.open(all_input_image_paths[key])
        label_img = Image.open(all_label_image_paths[key])

        # Here I use my own augmentation function which crops the input and label on the same position and do other things.
        # We get a list of new augmented data
        augmented_images = generate_augmented_images(input_img, label_img)
        for elem in augmented_images:
            input_as_tensor = data_transforms['norm'](elem[0])
            label_as_tensor = data_transforms['val'](elem[1])

            input_as_tensor.unsqueeze_(0)
            label_as_tensor.unsqueeze_(0)

            is_training_data = random.uniform(0.0, 1.0)
            if is_training_data <= 0.7:
                dataloader["train"].append([input_as_tensor, label_as_tensor])
            else:
                dataloader["val"].append([input_as_tensor, label_as_tensor])
            img_counter += 1

    shuffle(dataloader["train"])
    shuffle(dataloader["val"])
    dataloader_batched =  {"train":[], "val":[]}

    # Here I group my data to a given batch size
    for elem in dataloader["train"]:
        batch = []
        for i in range(batch_size):
            batch.append(elem)
        dataloader_batched["train"].append(batch)

    for elem in dataloader["val"]:
        batch = []
        for i in range(batch_size):
            batch.append(elem)
        dataloader_batched["val"].append(batch)

    return dataloader_batched

This is a snippet of my training method with batch size 1:

    while epoch <= num_epochs:
        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step(3)
                model.train()  # Set model to training mode
            else:
                model.eval()  # Set model to evaluate mode

            running_loss = 0.0

            counter = 0
            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                counter += 1
                max_num = len(dataloaders[phase])

                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)

            epoch_loss = running_loss / dataset_sizes[phase]

If I execute this, I get of course the error:

for inputs, labels in dataloaders[phase]:
ValueError: not enough values to unpack (expected 2, got 1)

I understand why, because now I have a list of images and not only an input and label image as before. So guessed I need a second for loop which iterates over these batches. So I tried this:

            # Iterate over data.
            for elem in dataloaders[phase]:
                for inputs, labels in elem:
                    counter += 1
                    max_num = len(dataloaders[phase])

                    inputs = inputs.to(device)
                    labels = labels.to(device)

                    # zero the parameter gradients
                    optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    # _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

But for me it looks like the optimization step (back-prop) is only applied on the last image of the batch. Is that true? And if so, how can I fix this? I guess if I indent the with-Block, then I get again a batch size 1 optimization.

Thanks in advance

Output:
  But for me it looks like the optimization step (back-prop) is only applied on the last image of the batch.

It should not apply only based on the last image. It should apply based on the batch size.
If you set bs=2 and it should apply to the batch of two images.

Optimization step actually will update the params of your network. Backprop is a fancy name for PyTorch autograd system that computes the first order gradients.


Instruction: How can I combine and overlap two tensors?

Input:
I have two tensors that should together overlap each other to form a larger tensor. To illustrate:

a = torch.Tensor([[1, 2, 3], [1, 2, 3]])
b = torch.Tensor([[5, 6, 7], [5, 6, 7]])

a = [[1 2 3]    b = [[5 6 7]
     [1 2 3]]        [5 6 7]]

I want to combine the two tensors and have them partially overlap by a single column, with the average being taken for those elements that overlap.

e.g.

result = [[1 2 4 6 7]
          [1 2 4 6 7]]

The first two columns are the first two columns of  'a'. The last two columns are the last two columns of 'b'. The middle column is the average of 'a's last column and 'b's first column.

I know how to merge two tensors side by side or in a new dimension. But doing this eludes me.

Can anyone help?

Output:
This is not a trivial operation, and this solution is not very trivial or intuitive either.

Looking at result with shape=(2, 5), you can think of a and b as two 2x3 patches of result taken with stride=2. Like this illustration:

We can use pytorch's unfold to "recover" the green (a) and blue (b) patches from result ("recover" up to the averaged values):

from torch.nn import functional as nnf

recovered = nnf.unfold(result, kernel_size=(2,3), stride=2)

The result is:

tensor([[[1., 4.],
         [2., 6.],
         [4., 7.],
         [1., 4.],
         [2., 6.],
         [4., 7.]]])

The patches were recovered (as column vectors).

Now that we understand how to get a and b from result, we can use fold to perform the "inverse" operation and go from b and b to result.
First, we need to flatten concatenate a and b to the shape fold expects (mimicking the output of unfold, two "flatten" patches of 3x2 elements):

uf = torch.cat((a.view(1, 6, 1), b.view(1, 6, 1)), dim=2)

We can now "fold" the patches

raw = nnf.fold(uf, (2,5), kernel_size=(2,3), stride=2)

We are not there yet, when there are overlapping elements fold sums up the overlapping elements, resulting with

tensor([[[[1., 2., 8., 6., 7.],
          [1., 2., 8., 6., 7.]]]])

To count how many elements were summed for each entry in result, we can simply "fold" an all ones tensor

counter = nnf.fold(torch.ones_like(uf), (2, 5), kernel_size=(2, 3), stride=2)

And finally, we can recover result:

result = raw / counter

tensor([[[[1., 2., 4., 6., 7.],
          [1., 2., 4., 6., 7.]]]])

Piece of cake.


Instruction: How to create torch.tensor object and to update only some of its elements?

Input:
Let's say I want to create torch.tensor object of size [2,3] filled with random elements, and I intend to use this matrix in the network and optimize it's values. However, I want to update only some of the the values in the matrix.

I know that it can be done for a tensor by setting up parameter requires_grad To True or False. However, the following code

z = torch.rand([2,3], requires_grad=True)
z[-1][-1].requires_grad=False

does not work as expected

RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().

How to fix this RuntimeError? How to initialize torch tensor and then define which elements there would have requires_grad =True?

If I write code in a similar manner:

z = torch.rand([2,3], requires_grad=False)
z[-1][-1].requires_grad=True

There will be no error, but no change of the requires_grad as well.

Output:
It does not really make much sense to have a single tensor which requires_grad for only part of its entries.
Why not have two separate tensors one that us updated (requires_grad=True) and another one fixed (requires_grad=False)? You can then merge them for computational ease:

fixed = torch.rand([2, 3], require_grad=False)
upd = torch.rand([2, 3], require_grad=True)
mask = torch.tensor([[0, 1, 0], [1, 0, 1]], require_grad=False)  # how to combine the two
# combine them using fixed "mask":
z = mask * fixed + (1-mask) * upd

You can obviously have other methods of combining fixed and upd other than using a binary mask.
For example, if upd occupies the first two columns of z and fixed the rest, then:

fixed = torch.rand([2, 1], require_grad=False)
upd = torch.rand([2, 2], require_grad=True)
# combine them using concatination
z = torch.cat((upd, fixed),dim=1)

Or, if you know the indices

fidx = torch.tensor([0, 2], dtype=torch.long)
uidx = torch.tensor([1, 3, 4, 5], dtype=torch.long)
fixed = torch.rand([2,], require_grad=False)
upd = torch.rand([4,], require_grad=True)
z = torch.empty([2, 3])
z[fidx] = fixed
z[uidx] = upd


Instruction: Trouble installing PyTorch for CUDA 9.0

Input:
I am trying to install PyTorch for CUDA 9.0 (Ubuntu 18.04 LTS). I followed instructions as per official documentation, which says

Download the whl file with the desired version from the following html page:
https://download.pytorch.org/whl/cu90/torch_stable.html

$ pip install [downloaded file]

But above link is broken, I am unable to open it.

Output: It turns out https://download.pytorch.org/whl/cu90/torch_stable.html is a list of <a href> tags, which we can see by going into the source from the browser. We can then download the appropriate version and install via pip install [downloaded file].


Instruction: How to display more than 10 images in Tensorboard?

Input:
I noticed that it doesn't matter how many image I save to the tensorboard log file, tensorboard will only ever show 10 of them (per tag).

  How can we increase the number of images or at least select which ones are displayed?

To reproduce what I mean run following MCVE:

import torch
from torch.utils.tensorboard import SummaryWriter
tb = SummaryWriter(comment="test")

for k in range(100):
    # create an image with some funny pattern
    b = [n for (n, c) in enumerate(bin(k)) if c == '1']
    img = torch.zeros((1,10,10))
    img[0, b, :] = 0.5
    img =img + img.permute([0, 2, 1])

    # add the image to the tensorboard file
    tb.add_image(tag="test", img_tensor=img, global_step=k)

This creates a folder runs in which the data is saved. From the same folder execute tensorboard --logdir runs, open the browser and go to localhost:6006 (or replace 6006 with whatever port tensorboard happens to display after starting it). Then go to the tab called "images" and move the slider above the grayscale image.

In my case it only displayed the images from steps

k = 3, 20, 24, 32, 37, 49, 52, 53, 67, 78

which isn't even an nice even spacing, but looks pretty random. I'd prefer to have

see more than just 10 of the images I saved, and
have a more even spacing of number of steps between each image displayed.

How can I achieve this?

EDIT: I just found the option --samples_per_plugin and tried tensorboard --logdir runs --samples_per_plugin "images=100". This indeed increased the number of images, but it only showed the images from steps k = 0,1,2,3....,78, but none from above 78.

Output:
You probably have to wait a little bit longer to wait for all the data to be loaded, but this is indeed the correct solution, see --help:

  --samples_per_plugin: An optional comma separated list of plugin_name=num_samples pairs to explicitly specify how many samples
  to keep per tag for that plugin. For unspecified plugins, TensorBoard
  randomly downsamples logged summaries to reasonable values to prevent
  out-of-memory errors for long running jobs. This flag allows fine
  control over that downsampling. Note that 0 means keep all samples of
  that type. For instance, "scalars=500,images=0" keeps 500 scalars and
  all images. Most users should not need to set this flag. (default: '')

Regarding the random samples: This is also true, there is some sort of randomness to it, from  the the FAQ:

  Is my data being downsampled? Am I really seeing all the data?

  TensorBoard uses reservoir sampling to downsample your data so that it
  can be loaded into RAM. You can modify the number of elements it will
  keep per tag in tensorboard/backend/application.py.


Instruction: Neural network from scratch - predict single example

Input:
Here is a neural network I've modified from Coursera Deep Learning Specialization to train on a dataset containing a flattened array of training data :

%reset -s -f

import numpy as np
import math

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def initialize_with_zeros(dim):

    w = np.zeros(shape=(dim, 1))
    b = 0

    return w, b

X = np.array([[1,1,1,1],[1,0,1,0] , [1,1,1,0], [0,0,0,0], [0,1,0,0], [0,1,0,1]])
Y = np.array([[1,0,1,1,1,1]])

X = X.reshape(X.shape[0], -1).T
Y = Y.reshape(Y.shape[0], -1).T

print('X shape' , X.shape)
print('Y shape' , Y.shape)

b = 1
w, b = initialize_with_zeros(4)

def propagate(w, b, X, Y) :

    m = X.shape[1]

    A = sigmoid(np.dot(w.T, X) + b)  # compute activation
    cost = (- 1 / m) * np.sum(Y * np.log(A) + (1 - Y) * (np.log(1 - A)))  # compute cost
    dw = (1./m)*np.dot(X,((A-Y).T))
    db = (1./m)*np.sum(A-Y, axis=1)
    cost = np.squeeze(cost)

    grads = {"dw": dw,
             "db": db}

    return grads, cost

propagate(w , b , X , Y)

learning_rate = .001
costs = []

def optimize(w , b, X , Y) :
    for i in range(2):

        grads, cost = propagate(w=w, b=b, X=X, Y=Y)

        dw = grads["dw"]
        db = grads["db"]

        w = w - learning_rate*dw
        b = b -  learning_rate*db

        if i % 100 == 0:
            costs.append(cost)

    return w , b

w , b = optimize(w , b , X , Y)

def predict(w, b, X):

    m = 6
    Y_prediction = np.zeros((1,m))
#     w = w.reshape(X.shape[0], 1)

    A = sigmoid(np.dot(w.T, X) + b)
    for i in range(A.shape[1]):

        if A[0, i] >= 0.5:
            Y_prediction[0, i] = 1

        else:
            Y_prediction[0, i] = 0

    return Y_prediction

predict(w , b, X)

This works as expected but I'm struggling to predict a single example.

If I use :

predict(w , b, X[0])

returns error :

ValueError: shapes (6,4) and (6,) not aligned: 4 (dim 1) != 6 (dim 0)

How to re-arrange matrix operation in order to predict a single instance ?

Output:
Try

predict(w, b, X[:1])

It seems like you predict function expects X to be 2-d, when passing only one X it should have a singleton second dimension (i.e., shape=(6,1)) rather than being a single dimension (i.e., shape=(6,)).


Instruction: Ground Truth Image to One Hot Encoded Array (Semantic Segmentation)

Input:
I'm creating my own Dataset for People and Street Segmentation. Below, you see a labled Ground Truth (GT) Image.

In the past I did a simple Regression between the model output and the GT Image (in the past I only used Streets). Now I read, Cross Entropy Loss is more common in that case. Since, my GT and also the model output Image has the same width w and height h as the input image, I have to create an array of size h x w x c, where c is the number of classes (in my case 3, background, street, people). I think, this is called One-Hot-Encoded Array.

I solved this as follows:

        for height in range(len(img_as_np_array)):
            for width in range(len(img_as_np_array[0])):
                temp = np.zeros(classes)
                if get_class(img_as_np_array[height,width]) == 1:
                    temp[1] = 1
                    one_hot_label[height,width] = temp
                if get_class(img_as_np_array[height,width]) == 2:
                    temp[2] = 1
                    one_hot_label[height,width] = temp

where the method get_class(channels) decides the pixel class by the color of the pixel.

def get_class(channels):
    threshold = 40
    # Class 1 corresponds to streets, roads
    if channels[0] in np.arange(243-threshold,243+threshold,1) and \
        channels[1] in np.arange(169-threshold,169+threshold,1) and \
        channels[2] in np.arange(0,threshold,1):
        return 1

    # Class 2 corresponds to people
    if channels[0] in np.arange(0,threshold,1) and \
        channels[1] in np.arange(163-threshold,163+threshold,1) and \
        channels[2] in np.arange(232-threshold,232+threshold,1):
        return 2

    # Class 0 corresponds to background respectively other things
    return 0

I have two questions:

My approach is very slow (about 3 minutes for a Full HD Image), is there a way to speed this up?
I noticed that the colors differ in the sense of channel values. For example, orange should be [243,169,0] (RGB), but I found entries like this [206,172,8] or even this [207,176,24] could that happen, because I store my labels as jpg? Is there a better way to find the orange and blue pixels than my idea above with the threshold?

EDIT:

I solved the first question by my self. This takes 2 or 3 seconds for an Full HD Image:

threshold = 40
class_1_shape_cond_1 = (img_as_array[:, :, 0] >= 243 - threshold) * (img_as_array[:, :, 0] <= 243 + threshold)
class_1_shape_cond_2 = (img_as_array[:, :, 1] >= 171 - threshold) * (img_as_array[:, :, 1] <= 171 + threshold)
class_1_shape_cond_3 = (img_as_array[:, :, 2] >= 0) * (img_as_array[:, :, 2] <= threshold)
class_1_shape = (class_1_shape_cond_1 * class_1_shape_cond_2 * class_1_shape_cond_3)

Then I do the same for class 2 and for class 3 (everything else) I can do:

class_3_shape = 1 - (class_1_shape + class_2_shape)

After that I have to adjust the type with:

class_1_shape = class_1_shape.astype(np.uint8)
class_2_shape = class_2_shape.astype(np.uint8)
class_3_shape = class_3_shape.astype(np.uint8)

Question 2 is still an open.

Output:
Do NOT save labels as JPEG images!
jpeg is a lossy compression method - that is, it is designed to save images using fewer bits even if it changes a bit the pixel values as long as it "looks good" to a human observer.
This is NOT the case for training labels stored as images! You cannot afford inaccuracies in the labels. You must use a lossless compression method, e.g., png.
Better still, store your labels as indexed RGB images to begin with and save you all the trouble of inferring the discrete labels from the RGB values.


Instruction: Being confused about tensor dimensions in pytorch

Input:
I'm new to pytorch and machine learning in general and I'm trying to create a simple convolutional neural net that classifies the MNIST handwritten digits.
Unfortunately when I'm trying to train it, I get the following error:

ValueError: Expected input batch_size (288) to match target batch_size (64).

Here is the neural network code.

from torch import nn
from torch.nn.functional import relu, log_softmax

class MNIST_SimpleConv(nn.Module):
    def __init__(self):
        super(MNIST_SimpleConv, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3,  out_channels=32, kernel_size=3, stride=1)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1)

        self.pool1 = nn.MaxPool2d(2, 2)

        self.dense1 = nn.Linear(4*4*64, 100)
        self.dense2 = nn.Linear(100, 10)

    def forward(self, x):
        x = relu(self.conv1(x))
        x = relu(self.conv2(x))
        x = self.pool1(x)

        x = x.view(-1, 4*4*64)

        x = relu(self.dense1(x))
        return log_softmax(self.dense2(x), dim=1)

And the training code is as follows:

from nets.conv import MNIST_SimpleConv
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from torch.nn.functional import nll_loss
import torch.optim as optim
import torch
from torch import nn

MNIST_ROOT = "data/MNIST"

#prepare dataset
mnist_train_ds = datasets.ImageFolder(root=MNIST_ROOT+"/train", transform=transforms.Compose([
                           transforms.ToTensor()]))
mnist_test_ds  = datasets.ImageFolder(root=MNIST_ROOT+"/test", transform=transforms.Compose([
                           transforms.ToTensor()]))
mnist_train = DataLoader(mnist_train_ds, batch_size=64, shuffle=True, num_workers=6)
mnist_test  = DataLoader(mnist_test_ds, batch_size=64, shuffle=True, num_workers=6)

criterion = nn.CrossEntropyLoss()

def train(model, device, train_loader, optimizer, epoch):
    model.train()

    for batch_idx, (data, target) in enumerate(train_loader, 0):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = MNIST_SimpleConv().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

for epoch in range(1, 10):
    train(model, device, mnist_train , optimizer, epoch)

So far I have investigated how the dimensions of 'x' change while x is forwarded through the network.

Input:
torch.Size([64, 3, 28, 28])

After x = relu(self.conv1(x)):
torch.Size([64, 32, 26, 26])

After x = relu(self.conv2(x)):
torch.Size([64, 64, 24, 24])

After x = self.pool1(x):
torch.Size([64, 64, 12, 12])

After x = x.view(-1, 4*4*64)
torch.Size([576, 1024])

After x = relu(self.dense1(x))
torch.Size([576, 100])

After x = log_softmax(self.dense2(x), dim=1)
torch.Size([576, 10])

The error is probably caused by x = x.view(-1, 4*4*64) for some reason producing a tensor with the shape [576, 1024] instead of [64, 1024].
(If I understand this correctly the first dimension should be equal to the batch size which in my case is 64.)

What am I doing wrong?

Output:
Passing a value of -1 to any dimension in view means that value of that particular dimension will be determined by other dimensions. for example:

x = torch.rand(1,10) # x.shape = [1,10]
x = x.view(-1, 5)  # x.shape = [2, 5]

In your case, if you want to merge all dimensions of output of pool1, then it should be something like this:

x = x.view(-1, 64*12,*12) # x.shape = [64, 9216]

Also, we have to update the input channels for  self.dense1 in this case:

self.dense1 = nn.Linear(64*12*12, 100)

However, one thing we need to make sure is that output dimensions of self.pool1 are always going to be batch_size x 64 x 12 x 12, specifically the last two dimensions should stay 12 for the whole process. This can be made sure by fixing the input image dimensions across dataset.


Instruction: How do I pass a keyword argument to the forward used by a pre-forward hook?

Input:
Given a torch's nn.Module with a pre-forward hook, e.g.

import torch
import torch.nn as nn

class NeoEmbeddings(nn.Embedding):
    def __init__(self, num_embeddings:int, embedding_dim:int, padding_idx=-1):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.register_forward_pre_hook(self.neo_genesis)

    @staticmethod
    def neo_genesis(self, input, higgs_bosson=0):
        if higgs_bosson:
            input = input + higgs_bosson
        return input

It's possible to let an input tensor go through some manipulation before going to the actual forward() function, e.g.

>>> x = NeoEmbeddings(10, 5, 1)
>>> x.forward(torch.tensor([0,2,5,8]))
tensor([[-1.6449,  0.5832, -0.0165, -1.3329,  0.6878],
        [-0.3262,  0.5844,  0.6917,  0.1268,  2.1363],
        [ 1.0772,  0.1748, -0.7131,  0.7405,  1.5733],
        [ 0.7651,  0.4619,  0.4388, -0.2752, -0.3018]],
       grad_fn=<EmbeddingBackward>)

>>> print(x._forward_pre_hooks)
OrderedDict([(25, <function NeoEmbeddings.neo_genesis at 0x1208d10d0>)])

How could we pass the arguments (*args or **kwargs) that the pre-forward hook needs but not accepted by the default forward() function?

Without modification/overriding the forward() function, this is not possible:

>>> x = NeoEmbeddings(10, 5, 1)
>>> x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

----------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-102-8705a40a3cc2> in <module>
      1 x = NeoEmbeddings(10, 5, 1)
----> 2 x.forward(torch.tensor([0,2,5,8]), higgs_bosson=2)

TypeError: forward() got an unexpected keyword argument 'higgs_bosson'

Output:
Torchscript incompatible (as of 1.2.0)

First of all, your example torch.nn.Module has some minor mistakes (probably by an accident).

Secondly, you can pass anything to forward and register_forward_pre_hook will just get the argument that will be passed your your torch.nn.Module (be it layer or model or anything) else. You indeed cannot do it without modifying forward call, but why would you want to avoid that? You could simply forward the arguments to base function as can be seen below:

import torch

class NeoEmbeddings(torch.nn.Embedding):
    def __init__(self, num_embeddings: int, embedding_dim: int, padding_idx=-1):
        super().__init__(num_embeddings, embedding_dim, padding_idx)
        self.register_forward_pre_hook(NeoEmbeddings.neo_genesis)

    # First argument should be named something like module, as that's what
    # you are registering this hook to
    @staticmethod
    def neo_genesis(module, inputs):  # No need for self as first argument
        net_input, higgs_bosson = inputs  # Simply unpack tuple here
        return net_input

    def forward(self, inputs, higgs_bosson):
        # Do whatever you want here with both arguments, you can ignore
        # higgs_bosson if it's only needed in the hook as done here
        return super().forward(inputs)

if __name__ == "__main__":
    x = NeoEmbeddings(10, 5, 1)
    # You should call () instead of forward so the hooks register appropriately
    print(x(torch.tensor([0, 2, 5, 8]), 1))

You can't do it in more succinct way, but the limitation is base's class forward method, not the hook itself (and tbh I wouldn't want it to be more succinct as it would become unreadable IMO).

Torchscript compatible

If you want to use torchscript (tested on 1.2.0) you could use composition instead of inheritance. All you have to change are merely two lines and your code may look something like this:

import torch

# Inherit from Module and register embedding as submodule
class NeoEmbeddings(torch.nn.Module):
    def __init__(self, num_embeddings: int, embedding_dim: int, padding_idx=-1):
        super().__init__()
        # Just use it as a container inside your own class
        self._embedding = torch.nn.Embedding(num_embeddings, embedding_dim, padding_idx)
        self.register_forward_pre_hook(NeoEmbeddings.neo_genesis)

    @staticmethod
    def neo_genesis(module, inputs):
        net_input, higgs_bosson = inputs
        return net_input

    def forward(self, inputs: torch.Tensor, higgs_bosson: torch.Tensor):
        return self._embedding(inputs)

if __name__ == "__main__":
    x = torch.jit.script(NeoEmbeddings(10, 5, 1))
    # All arguments must be tensors in torchscript
    print(x(torch.tensor([0, 2, 5, 8]), torch.tensor([1])))


Instruction: plot function does not plot curves just plotted legend in PyCharm

Input:
I am doing some homework about image classification using PyTorch. For this purpose I have visited the following link: https://github.com/Bjarten/early-stopping-pytorch/blob/master/MNIST_Early_Stopping_example.ipynb and obtained some results like this related to training losses and validation losses

[ 1/10] train_loss: 1.56952 valid_loss: 1.54557
Validation loss decreased (inf --> 1.545571).  Saving model ...
[ 2/10] train_loss: 1.52198 valid_loss: 1.50932
Validation loss decreased (1.545571 --> 1.509324).  Saving model ...
[ 3/10] train_loss: 1.48958 valid_loss: 1.48514
Validation loss decreased (1.509324 --> 1.485136).  Saving model ...
[ 4/10] train_loss: 1.46960 valid_loss: 1.46938
Validation loss decreased (1.485136 --> 1.469381).  Saving model ...

But the problem is that whenever I try to plot loss curves it does not show anything.

my plotted figure

I would expect my curve to be like the one in this picture.
Here is the code that I used to draw this curve:

  # visualize the loss as the network trained
    fig = plt.figure(figsize=(10, 8))
    plt.plot(range(1, len(train_loss) + 1), train_loss, label='Training Loss')
    plt.plot(range(1, len(valid_loss) + 1), valid_loss, label='Validation Loss')

        # find position of lowest validation loss
    minposs = valid_loss.index(min(valid_loss)) + 1
    plt.axvline(minposs, linestyle='--', color='r', label='Early Stopping Checkpoint')

    plt.xlabel('epochs')
    plt.ylabel('loss')
    plt.ylim(0, 0.5)  # consistent scale
    plt.xlim(0, len(train_loss) + 1)  # consistent scale
    plt.grid(True)
    plt.legend()
    plt.tight_layout()
    plt.show()
    fig.savefig('loss_plot.png', bbox_inches='tight')

Here is the value of length of the training loss and the testing loss

len of train loss: 1
train loss :[1.6059992909431458]

Output:
Since len(train_loss) == 1 and train_loss = [1.6059992909431458], your plot is exactly the same as if you called plt.plot(range(1, 2), [1.6059992909431458]), which, as you can check, is empty (this is because plt.plot draws lines between data points, and as there is a single data point, there can be no line drawn). You can change plt.plot to plt.scatter in order to see that single data point.

So this problem has nothing to do with plotting really, and you should rather figure out why your algorithm only runs for a single epoch and troubleshoot that, rather than plotting.


Instruction: using transforms.LinearTransformation to apply whitening in PyTorch

Input:
I need to apply ZCA whitening in PyTorch. I think I have found a way this can be done by using transforms.LinearTransformation and I have found a test in the PyTorch repo which gives some insight into how this is done (see final code block or link below)

https://github.com/pytorch/vision/blob/master/test/test_transforms.py

I am struggling to work out how I apply something like this myself.

Currently I have transforms along the lines of:

    transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(np.array([125.3, 123.0, 113.9]) / 255.0,
                         np.array([63.0, 62.1, 66.7]) / 255.0),
])

The documents say they way to use LinearTransformation is as follows:

torchvision.transforms.LinearTransformation(transformation_matrix, mean_vector)

  whitening transformation: Suppose X is a column vector zero-centered
  data. Then compute the data covariance matrix [D x D] with
  torch.mm(X.t(), X), perform SVD on this matrix and pass it as
  transformation_matrix.

I can see from the tests I linked above and copied below that they are using torch.mm to calculate what they call a principal_components:

def test_linear_transformation(self):
    num_samples = 1000
    x = torch.randn(num_samples, 3, 10, 10)
    flat_x = x.view(x.size(0), x.size(1) * x.size(2) * x.size(3))
    # compute principal components
    sigma = torch.mm(flat_x.t(), flat_x) / flat_x.size(0)
    u, s, _ = np.linalg.svd(sigma.numpy())
    zca_epsilon = 1e-10  # avoid division by 0
    d = torch.Tensor(np.diag(1. / np.sqrt(s + zca_epsilon)))
    u = torch.Tensor(u)
    principal_components = torch.mm(torch.mm(u, d), u.t())
    mean_vector = (torch.sum(flat_x, dim=0) / flat_x.size(0))
    # initialize whitening matrix
    whitening = transforms.LinearTransformation(principal_components, mean_vector)
    # estimate covariance and mean using weak law of large number
    num_features = flat_x.size(1)
    cov = 0.0
    mean = 0.0
    for i in x:
        xwhite = whitening(i)
        xwhite = xwhite.view(1, -1).numpy()
        cov += np.dot(xwhite, xwhite.T) / num_features
        mean += np.sum(xwhite) / num_features
    # if rtol for std = 1e-3 then rtol for cov = 2e-3 as std**2 = cov
    assert np.allclose(cov / num_samples, np.identity(1), rtol=2e-3), "cov not close to 1"
    assert np.allclose(mean / num_samples, 0, rtol=1e-3), "mean not close to 0"

    # Checking if LinearTransformation can be printed as string
    whitening.__repr__()

How do I apply something like this? do I use it where I define my transforms or apply it in my training loop where I am iterating over my training loop?

Thanks in advance

Output:
ZCA whitening is typically a preprocessing step, like center-reduction, which basically aims at making your data more NN-friendly (additional info below). As such, it is supposed to be applied once, right before training.

So right before you starts training your model with a given dataset X, compute the whitened dataset Z, which is simply the multiplication of X with the ZCA matrix W_zca that you can learn to compute here. Then train your model on the whitened dataset.
Finally, you should have something that looks like this

class MyModule(torch.nn.Module):
    def __init__(self):
        super(MyModule,self).__init__()
        # Feel free to use something more useful than a simple linear layer
        self._network = torch.nn.Linear(...)
        # Do your stuff
        ...

    def fit(self, inputs, labels):
    """ Trains the model to predict the right label for a given input """
        # Compute the whitening matrix and inputs
        self._zca_mat = compute_zca(inputs)
        whitened_inputs = torch.mm(self._zca_mat, inputs)

        # Apply training on the whitened data
        outputs = self._network(whitened_inputs)
        loss = torch.nn.MSEloss()(outputs, labels)
        loss.backward()
        optimizer.step()

     def forward(self, input):
         # You always need to apply the zca transform before forwarding,
         # because your network has been trained with whitened data
         whitened_input = torch.mm(self._zca_mat, input)
         predicted_label = self._network.forward(whitened_input)
         return predicted_label

Additional info

Whitening your data means decorrelating its dimensions so that the correlation matrix of the whitened data is the identity matrix. It is a rotation-scaling operation (thus linear), and there are actually an infinity of possible ZCA transforms. To understand the maths behind ZCA, read this


Instruction: Understanding deep Markov model code on pyro

Input:
I am going through an a deep Markov model code given on pyro's website:

https://pyro.ai/examples/dmm.html

I am really confused about combiner module that they've implemented. You an find combiner module on line 104 on this GitHub page:

https://github.com/pyro-ppl/pyro/blob/dev/examples/dmm/dmm.py

The article they are following is:

https://arxiv.org/abs/1609.09869

They explain combiner module in section 4 (Structured inference networks). I am really confused about why they are doing three linear transformations in the code starting from line 104 on GitHub. Aren't they just suppose to use RNN to produce the distribution or am I missing something? Insights would be appreciated.

Output:
The Combiner module implementation corresponds to the formulas described at the bottom of page 4, "Combiner Function for Structured Approximations (for DKS)". mu_t is loc and sigma_t**2 is scale.

The RNN state is used to parameterize the distribution, but that distribution is parameterized by two variables. These variables are extracted from the RNN state via the transformations in question.


Instruction: Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Input:
I'm getting the following error when calling .backward():

Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Here's the code:

for i, j, k in zip(X, Y, Z):
    A[:, i, j] = A[:, i, j] + k

I've tried .clone(), torch.add(), and so on.

Please help!

Output:
After the comments I'm a bit confused about what you want to accomplish. The code you gave gives me an error using the dimensions you provided in the comments

Traceback (most recent call last):
    A[:, i, j] = A[:, i, j] + k
RuntimeError: The size of tensor a (32) must match the size of tensor b (200) at non-singleton dimension 0

But here's what I think you want to do, please correct me in the comments if this is wrong...

Given tensors X, Y, and Z, each entry of X, Y, and Z correspond to a coordinate (x,y) and a value z. What you want is to add z to A at coordinate (x,y). For most cases the batch dimension is kept independent, although its not clear that's the case in the code you posted. For now that's what I'll assume you want to do.

For example lets say A contains all zeros and has shape 3x4x5 and X,Y are shape 3x3 and Z is shape 3x3x1. For this example let's assume A contains all zeros to start, and X, Y, and Z have the following values

X = tensor([[1, 2, 3],
            [1, 2, 3],
            [2, 2, 2]])
Y = tensor([[1, 2, 3],
            [1, 2, 3],
            [1, 1, 1]])
Z = tensor([[[0.1], [0.2], [0.3]],
            [[0.4], [0.5], [0.6]],
            [[0.7], [0.8], [0.9]]])

Then we would expect A to have the following values after the operation

A = tensor([[[0,   0,   0,   0,   0],
             [0,   0.1, 0,   0,   0],
             [0,   0,   0.2, 0,   0],
             [0,   0,   0,   0.3, 0]],

            [[0,   0,   0,   0,   0],
             [0,   0.4, 0,   0,   0],
             [0,   0,   0.5, 0,   0],
             [0,   0,   0,   0.6, 0]],

            [[0,   0,   0,   0,   0],
             [0,   0,   0,   0,   0],
             [0,   2.4, 0,   0,   0],
             [0,   0,   0,   0,   0]]])

In order to accomplish this we can make use to the index_add function which allows us to add to a list of indices. Since this only supports 1-dimensional operations we first need to convert X,Y to a linear index for flattened tensor A. Afterwards we can un-flatten to the original shape.

layer_size = A.shape[1] * A.shape[2]
index_offset = torch.arange(0, A.shape[0] * layer_size, layer_size).unsqueeze(1)
indices = (X * A.shape[2] + Y) + index_offset
A = A.view(-1).index_add(0, indices.view(-1), Z.view(-1)).view(A.shape)


Instruction: What is the PyTorch equivalent to Numpy's linalg.solve?

Input: I have looked it up and there are some posts, for example this one, that suggest to use torch.gesv but I can't seem to find it in the PyTorch documentation.

Output:
Pytorch provides the function torch.solve, which behaves like numpy.linalg.solve. It will output the solution of the the linear system and the LU factorization that has been used to compute it.
More information and example codes here.


Instruction: How to change input type(image) to list or array when using PyTorch tutorial code

Input:
I have searched the code that uses list or array input data for training DQN code. But I have could not find any code.

Currently, I reference the reinforcement learning tutorial(DQN) of Pytorch.
However, this code uses image input data.

I want to know how to change the image input data to list or array input data.

(I need help to resolve my research that uses list input data. List input data shape is 1 by 9. )

Output:
In PyTorch, we deal with tensors. Images, text, even sounds can be transformed to tensors and then PyTorch models can learn on the data.

In PyTorch image classifier examples, you often see something like this, to transform images to tensors:

train_transform = transforms.Compose([
                                      transforms.Resize(x),
                                      ...
                                      transforms.ToTensor()
                                     ])

If your input is a numpy array x, you can convert it to a tensor like this:

torch.from_numpy(x)

You also have to pay attention to tensor dimensions, your input data needs to match what the model expects in the first layer.


Instruction: How do I get the value of a tensor in PyTorch?

Input:
Printing a tensor x gives:
>>> x = torch.tensor([3])
>>> print(x)
tensor([3])

Indexing x.data gives:
>>> x.data[0]
tensor(3)

How do I get just a regular non-tensor value 3?

Output: You can use x.item() to get a Python number from a Tensor that has one element.


Instruction: Display result of convolution in PyTorch

Input:
PyTorch newbie here. I wrote a script (code below) that performs the following operations: load an image, perform a 2D convolution operation and then display the output and the input.

At present I have the image below, which seems off. How can I plot the feature map correctly?

import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import matplotlib.pyplot as plt
import imageio
import sys

A = imageio.imread('LiT.png')
# Define how the convolution operation works
conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)

image_d = torch.FloatTensor(np.asarray(A.reshape(1, 3, A.shape[0] , A.shape[1])))
fc = conv2(image_d)
fc1 = fc.permute(0, 2, 3, 1).reshape([516, 780, 3])

plt.figure(figsize=(16,8))
plt.subplot(1,2,1)
plt.imshow(A)
plt.subplot(1,2,2)
plt.imshow(fc1.data.numpy())

plt.show()

Output:
The issue with your code is this line

image_d = torch.FloatTensor(np.asarray(A.reshape(1, 3, A.shape[0] , A.shape[1])))

You can't just reshape the image you need to transpose the channels. As a remark for the future, if you get a stripy result like you did it's most likely some permutation/transposition or reshaping operation that's not correct.

Other than that I also scaled the input image to [0, 1] to show it properly. Below is the working code:

import numpy as np
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import matplotlib.pyplot as plt
import imageio
import sys

A = imageio.imread('LiT.png')
# Define how the convolution operation works
conv2 = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=3, stride=1, padding=1)

# from [H, W, C] to [C, H, W]
transposed_image = A.transpose((2, 0, 1))
# add batch dim
transposed_image = np.expand_dims(transposed_image, 0)

image_d = torch.FloatTensor(transposed_image)
fc = conv2(image_d)
fc1 = fc.permute(0, 2, 3, 1)[0]
result = fc1.data.numpy()
max_ = np.max(result)
min_ = np.min(result)
result -= min_
result /= max_

plt.figure(figsize=(16,8))
plt.subplot(1,2,1)
plt.imshow(A)
plt.subplot(1,2,2)
plt.imshow(result)

plt.show()


Instruction: Check the number of parameters from state_dict in pytorch

Input:
SO has an answer about how to check the total # of params from the model:
pytorch_total_params = sum(p.numel() for p in model.parameters())

However, how does one check the total # of params from
the state_dict?

state_dict = torch.load(model_path, map_location='cpu')?

Output:
You can count the number of saved entries in the state_dict:

sum(p.numel() for p in state_dict.values())

However, there's a snag here: a state_dict stores both parameters and persistent buffers (e.g., BatchNorm's running mean and var). There's no way (AFAIK) to tell them apart from the state_dict itself, you'll need to load them into the model and use sum(p.numel() for p in model.parameters() to count only the parameters.

For instance, if you checkout resnet50

from torchvision.models import resnet50
model = resnet50(pretrained=True)
state_dict = torch.load('~/.torch/models/resnet50-19c8e357.pth')
num_parameters = sum(p.numel() for p in model.parameters())
num_state_dict = sum(p.numel() for p in state_dict.values())
print('num parameters = {}, stored in state_dict = {}, diff = {}'.format(num_parameters, num_state_dict, num_state_dict - num_parameters))

Resulting with

num parameters = 25557032, stored in state_dict = 25610152, diff = 53120

As you can see there can be quite a gap between the two values.


Instruction: Using imbalanced-learn with Pandas DataFrame

Input:
My dataset is quite imbalanced. The two minority classes each contain half of the sample in the majority class. My RNN model is not able to learn anything about the least populated class.

I'm trying to use the imbalanced-learn library. For instance:

sm = SMOTE(random_state=42, n_jobs=-1, k_neighbors=10)
X_train, y_train = sm.fit_resample(train.drop(['label], axis=1), train['label'])

works if train.drop(['label] contains just the values of the used features. The problem is that my DataFrame contains one additional columns containing strings as values: I cannot drop it since those strings are the input for my RNN. And if I drop it, I would not be able to tell to which row of the oversampled dataset those strings belong to.

Is there a way to keep all the columns and tell the function which columns to use for oversampling?

Output: For those who need to do something similar, a co-author of the library suggested me to use SMOTENC, which can handle also categorical variables (like strings).


Instruction: How to vectorize this pytorch code over (at least) the batch dimension?

Input:
I want to implement a code to build an adjacency matrix such that (for example):

If X[0] : [0, 1, 2, 0, 1, 0], then,

A[0, 1] = 1
A[1, 2] = 1
A[2, 0] = 1
A[0, 1] = 1
A[1, 0] = 1

The following code works fine, however, it's too slow! So, please help me to vectorize this code on the batch (first) dimension at least:

A = torch.zeros((3, 3, 3), dtype = torch.float)
X = torch.tensor([[0, 1, 2, 0, 1, 0], [1, 0, 0, 2, 1, 1], [0, 0, 2, 2, 1, 1]])
for a, x in zip(A, X):
    for i, j in zip(x, x[1:]):
        a[i, j] = 1

Thanks! :)

Output:
I am pretty sure that there is a much simpler way of doing this, but I tried to keep within the realm of torch function calls, to make sure that any gradient operation could be properly tracked.

In case this is not required for backpropagation, I strongly suggest you look into solution that maybe utilize some numpy functions, because I think there is a stronger guarantee to find something suitable here. But, without further ado, here is the solution I came up with.

It essentially transforms your X vector into a series of tuple entries that correspond to the position in A. For this, we need to align some of the indices (specifically, the first dimension is only implicitly given in X, since the first list in X corresponds to A[0,:,:], the second list to A[1,:,:], and so on.
This is also probably where you can start optimizing the code, because I did not find a reasonable description of such a matrix, and therefore had to come up with my own way of creating it.

# Start by "aligning" your shifted view of X
# Essentially, take the all but the last element,
# and put it on top of all but the first element.
X_shift = torch.stack([X[:,:-1], X[:,1:]], dim=2)
# X_shift.shape: (3,5,2) in your example

# To assign this properly, we need to turn it into a "concatenated" list,
# where each entry corresponds to a 2D tuple in the respective dimension of A.
temp_tuples = X_shift.view(-1,2).transpose(0,1)
# temp_tuples.shape: (2,15) in your example. Below are the values:
tensor([[0, 1, 2, 0, 1, 1, 0, 0, 2, 1, 0, 0, 2, 2, 1],
        [1, 2, 0, 1, 0, 0, 0, 2, 1, 1, 0, 2, 2, 1, 1]])

# Now we have to create a matrix do indicate the proper "first dimension index"
fix_dims = torch.repeat_interleave(torch.arange(0,3,1), len(X[0])-1, 0).unsqueeze(dim=0)
# fix_dims.shape: (1,15)
# Long story short, this creates the following vector.
tensor([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2]])
# Note that the unsqueeze is necessary to properly concatenate the two matrices:
access_tuples = tuple(torch.cat([fix_dims, temp_tuples], dim=0))
A[access_tuples] = 1

This further assumes that every dimension in X has the same number of tuples changed. If that is not the case, then you have to manually create a fix_dims vector, where each increment is repeated the length of X[i] times. If it is equal as in your example, you can safely use the proposed solution.


Instruction: keras.preprocessing.text.Tokenizer equivalent in Pytorch?

Input: Basically the title; is there any equivalent tokeras.preprocessing.text.Tokenizer in Pytorch? I have yet to find any that gives all the utilities without handcrafting things.

Output:
I find Torchtext more difficult to use for simple things. PyTorch-NLP can do this in a more straightforward way:

from torchnlp.encoders.text import StaticTokenizerEncoder， stack_and_pad_tensors, pad_tensor

loaded_data = ["now this ain't funny", "so don't you dare laugh"]
encoder = StaticTokenizerEncoder(loaded_data, tokenize=lambda s: s.split())
encoded_data = [encoder.encode(example) for example in loaded_data]

print(encoded_data)

  [tensor([5, 6, 7, 8]), tensor([ 9, 10, 11, 12, 13])]

encoded_data = [pad_tensor(x, length=10) for x in encoded_data]
print(stack_and_pad_tensors(encoded_data))
# alternatively, use encoder.batch_encode()

  BatchedSequences(tensor=tensor([[ 5,  6,  7,  8,  0,  0,  0,  0,  0,  0], [ 9, 10, 11, 12, 13,  0,  0,  0,  0,  0]]), lengths=tensor([10, 10]))

It comes with other types of encoders, such as spaCy's tokenizer, subword encoder, etc.


Instruction: How do I concatenate tensors along a given axis?

Input:
import torch

a = torch.Tensor(2,2,2)

b = myfunction(a)

print(a)
>>> [[[1,2],
      [5,6]],
     [[7,8],
      [9,10]]]

print(b)
>>> [[1,2,7,8],
     [5,6,9,10]]

How do I code myfunction to get b from a?

Is there some pytorch functions that transforms a in such way?

Output:
You can achieve this by using transpose to swap the first two axes (cf. e.g. np.swapaxes), and reshape to get your desired shape:

In [12]: a
Out[12]:
tensor([[[  1.,   2.],
         [  5.,   6.]],

        [[  7.,   8.],
         [  9.,  10.]]])

In [13]: a.transpose(0, 1).reshape(2, 4)
Out[13]:
tensor([[  1.,   2.,   7.,   8.],
        [  5.,   6.,   9.,  10.]])


Instruction: why Gradient Descent doesn't work as expected with pytorch

Input:
so I'm starting with Pytorch and tried to start with an easy Linear Regression Example. Actually I made an easy Implementation of Linear Regression with Pytorch to calculate the equation 2*x+1 but the loss stay stuck at 120 and there is a Problem with Gradient Descent because it doesn't converge to a small loss value. I don't know why this is happening and it made me crazy because I don't see what's wrong. actually this example should be very easy to solve. this is the Code I'm using

import torch
import torch.nn.functional as F
from torch.utils.data import TensorDataset, DataLoader
import numpy as np

X = np.array([i for i in np.arange(1, 20)]).reshape(-1, 1)
X = torch.tensor(X, dtype=torch.float32, requires_grad=True)
y = np.array([2*i+1 for i in np.arange(1, 20)]).reshape(-1, 1)
y = torch.tensor(y, dtype=torch.float32, requires_grad=True)
print(X.shape, y.shape)

class LR(torch.nn.Module):
    def __init__(self, n_features, n_hidden1, n_out):
        super(LR, self).__init__()
        self.linear = torch.nn.Linear(n_features, n_hidden1)

        self.predict = torch.nn.Linear(n_hidden1, n_out)

    def forward(self, x):
        x = F.relu(self.linear(x))
        x = self.predict(x)
        return x

model = LR(1, 10, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()

def train(epochs=100):
    for e in range(epochs):

        pred = model(X)
        loss = loss_fn(pred, y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(f"epoch: {e} and loss= {loss}")

desired output is a small loss value and that the model train to give a good prediction later.

Output: Your learning rate is too large. The model takes a few steps in the right direction, but it can't land on an actually good minimizer and henceforth zigzags around it. If you try lr=0.001 instead,  your performance will be much better. This is why it's often useful to decay your learning rate over time when using first order optimizers.


Instruction: ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))

Input:
ValueError                                Traceback (most recent call last)
<ipython-input-30-33821ccddf5f> in <module>
     23         output = model(data)
     24         # calculate the batch loss
---> 25         loss = criterion(output, target)
     26         # backward pass: compute gradient of the loss with respect to model parameters
     27         loss.backward()

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    593                                                   self.weight,
    594                                                   pos_weight=self.pos_weight,
--> 595                                                   reduction=self.reduction)
    596
    597

C:\Users\mnauf\Anaconda3\envs\federated_learning\lib\site-packages\torch\nn\functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2073
   2074     if not (target.size() == input.size()):
-> 2075         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2076
   2077     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)

ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))

I am training a CNN. Working on the Horses vs humans dataset. This is my code. I am using criterion = nn.BCEWithLogitsLoss() and optimizer = optim.RMSprop(model.parameters(), lr=0.01). My final layer is self.fc2 = nn.Linear(512, 1). Out last neuron, will output 1 for horse and 0 for human, right? or should I choose 2 neurons for output?

16 is the batch size. Since the error says ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1])). I don't understand, where do I need to make change, to rectify the error.

Output: target = target.unsqueeze(1), before passing target to criterion, changed the target tensor size from [16] to [16,1]. Doing it solved the issue. Furthermore, I also needed to do target = target.float() before passing it to criterion, because our outputs are in float. Besides, there was another error in the code. I was using sigmoid activation function in the last layer, but I shouldn’t because the criterion I am using already comes with sigmoid builtin.


Instruction: PyTorch dot product does not work for some cases

Input:
I am struggling with an issue I cannot solve in PyTorch code.  My task is simple: just get the inner product of two tensors.  However, the output becomes zero in certain cases instead of returning the correct value.

For example, the code below demonstrates the issue.

a = torch.tensor([3,1])
b = torch.tensor([3.,1.])
a.dot(a)  # this returns 10, which is correct
b.dot(b)  # this returns 0., which is not correct

I am using PyTorch version 1.2 and this has never happened to me before...  Is there something simple that I am missing around the way I define the tensors?

Thank you for your time!

Output: I just ran the same code using torch version '1.0.1.post2' and saw the results tensor(10)and tensor(10.) respectively for a.dot(a) and b.dot(b). Please confirm if you are missing something else in the code.


Instruction: How to create custom neural network with custom weight initialization in tensorflow or pytorch

Input: I'm trying to create a small neural network with custom connections between neurons. The connections should exist over several layers and not be fully connected (sparse) as shown in the picture. I would also like to do the weight initialization manually and not completely randomly. My goal is to determine whether a connection is positive or negative. Is it possible to create such a neural net in tensorflow (python/js) or pytorch?

Output:
To summarize:
Can you do it? -- Yes, absolutely.
Is it going to be pretty? -- No, absolutely not.

In my explanation, I will further focus on PyTorch, as this is the library that I am more comfortable with, and that is especially more useful if you have custom operations that you can easily express in a pythonic manner. Tensorflow also has eager execution mode (more serious integration from version 2, if I remember that correctly), but it is traditionally done with computational graphs, which make this whole thing a little uglier than it needs to be.

As you hopefully know, backpropagation (the "learning" step in any ANN) is basically an inverse pass through the network, to calculate gradients, or at least close enough to the truth for our problem at hand. Importantly, torch functions store this "reverse" direction, which makes it trivial for the user to call backpropagation functions.

To model a simple network as described in your image, we have only one major disadvantage:
The available operations are usually excelling at what they are doing because they are simply and can be optimized quite heavily. In your case, you have to express different layers as custom operations, which generally scales incredibly poorly, unless you can express the functionals as some form of matrix operation, which I do not see straigt away in your example. I am further assuming that you are applying some form of non-linearity, as it would otherwise be a network that would fail for any non-linearly separable problem.

import torch
import torch.nn as nn
class CustomNetwork(nn.module):
    def __init__(self):
        self.h_1_1 = nn.Sequential(nn.Linear(1,2), nn.ReLU) # top node in first layer
        self.h_1_2 = nn.Sequential(nn.Linear(1,2), nn.ReLU) # bottom node in first layer
        # Note that these nodes have no shared weights, which is why we
        # have to initialize separately.
        self.h_2_1 = nn.Sequential(nn.Linear(1,1), nn.ReLU) # top node in second layer
        self.h_2_2 = nn.Sequential(nn.Linear(1,1), nn.ReLU) # bottom node in second layer

        self.h_2_1 = nn.Sequential(nn.Linear(2,1), nn.ReLU) # top node in third layer
        self.h_2_2 = nn.Sequential(nn.Linear(2,1), nn.ReLU) # bottom node in third layer
       # out doesn't require activation function due to pairing with loss function
        self.out = nn.Linear(2,1)

    def forward(self, x):
        # x.shape: (batch_size, 2)

        # first layer. shape of (batch_size, 2), respectively
        out_top = self.h_1_1(x[:,0])
        out_bottom = self.h_1_2(x[:,1])

        # second layer. shape of (batch_size, 1), respectively
        out_top_2 = self.h_2_1(out_top[:,0])
        out_bottom_2 = self.h_2_2(out_bottom[:,0])

        # third layer. shape of (batch_size, 1), respectively
        # additional concatenation of previous outputs required.
        out_top_3 = self.h_3_1(torch.cat([out_top_2, -1 * out_top[:,1]], dim=1))
        out_bottom_3 = self.h_3_2(torch.cat([out_bottom_2, -1 * out_bottom[:,1]], dim=1))
        return self.out(torch.cat([out_top_3, out_bottom_3], dim=1))

As you can see, any computational step is (in this case rather explicitly) given, and very much possible. Again, once you want to scale your number of neurons for each layer, you are going to have to be a little more creative in how you process, but for-loops do very much work in PyTorch as well. Note that this will in any case be much slower than a vanilla linear layer, though.
If you can live with seperately trained weights, you can always also just define separate linear layers of smaller size and put them in a more convenient fashion.


Instruction: Why is my texture synthesis algorithm only producing blocky / noisy, non-sensible output?

Input:
My end-goal is to create a script for neural style transfer, however, during writing code for said task, I stumbled upon a certain problem: the texture synthesis part of the algorithm seemed to have some problems with reproducing the artistic style. In order to solve this, I decided to create another script where I'd try to solve the task of texture synthesis using a neural network on its own.

TL;DR ... even after tackling the problem on its own, my script still produced blocky / noisy, non-sensible output.

I've tried having a look at how other people have solved this task, but most of what I found were more sophisticated solutions ("fast neural-style-transfer", etc.). Also, I couldn't find too many PyTorch implementations.

Since I've already spent the past couple of days on trying to fix this issue and considering that I'm new to the PyTorch-Framework, I have decided to ask the StackOverflow community for help and advice.

I use the VGG16 network for my model ...

class VGG16(nn.Module):
    def __init__(self):
        super(VGG16, self).__init__()
        vgg_fs = models.vgg16(pretrained=True).features

        self.sl1 = nn.Sequential()
        self.sl2 = nn.Sequential()
        self.sl3 = nn.Sequential()
        self.sl4 = nn.Sequential()
        self.sl5 = nn.Sequential()

        for i in range(4):
            self.sl1.add_module(str(i), vgg_fs[i])
        for i in range(4, 9):
            self.sl2.add_module(str(i), vgg_fs[i])
        for i in range(9, 16):
            self.sl3.add_module(str(i), vgg_fs[i])
        for i in range(16, 23):
            self.sl4.add_module(str(i), vgg_fs[i])
        for i in range(23, 30):
            self.sl5.add_module(str(i), vgg_fs[i])

        for p in self.parameters():
            p.requires_grad_(False)

    def forward(self, x):
        h = self.sl1(x)
        h1 = h
        h = self.sl2(h)
        h2 = h
        h = self.sl3(h)
        h3 = h
        h = self.sl4(h)
        h4 = h
        h = self.sl5(h)
        h5 = h

        return_tuple = namedtuple('hidden_states', ['h1', 'h2', 'h3', 'h4', 'h5'])
        ret = return_tuple(h1, h2, h3, h4, h5)

        return ret

Then, I have some functions for gram-matrix computation and normalization ...

def comp_gram(f):
    (b, c, h, w) = f.shape
    f = f.view(b, c, h * w)
    g = f.bmm(f.transpose(1, 2))
    g = g / (c * h * w)
    return g

def norm(b):
    mean    = torch.Tensor([0.485, 0.456, 0.406]).cuda().view(3, 1, 1)
    std     = torch.Tensor([0.229, 0.224, 0.225]).cuda().view(3, 1, 1)
    return (b - mean) / std

And finally, my training-function ...

def train(model, style_img, w=224, h=224, iters=32, lr=1):
    style = torch.from_numpy(np.array(Image.open(style_img))).cuda().float()
    style = style / 255.
    style = style.view(1, style.size()[2], *style.size()[:2])

    style_fts = model(norm(style))
    style_gms = [comp_gram(f) for f in style_fts]

    img = torch.rand(*style.size()[:2], h, w, requires_grad=True, device='cuda')
    optimizer = optim.Adam([img], lr=lr)
    mse_loss = nn.MSELoss()

    plt.ion()
    for i in range(iters):
        optimizer.zero_grad()
        actvs = model(norm(img))

        lss = 0.
        for f, gm in zip(actvs, style_gms):
            g = comp_gram(f)
            lss += mse_loss(g, gm)

        lss.backward()
        optimizer.step()

        if (i % 5 == 0) or (i == iters - 1):
            plt.title('Iter#{:04d}'.format(i))
            plt.imshow(img.detach().cpu().view(*img.shape[2:], img.shape[1]))
            plt.pause(1e-3)

        print('[iter#{:04d}]: Loss\t-> {:}'.format(i, lss.item()))

    plt.ioff()

    plt.subplot(121)
    plt.title('Original')
    plt.imshow(style.cpu().view(*style.size()[2:], style.size()[1]))

    plt.subplot(122)
    plt.title('Style')
    plt.imshow(img.detach().cpu().view(*img.size()[2:], img.size()[1]))

    plt.show()

I expected the algorithm to produce the style of the painting in some way, shape or form, but that didn't happen. ♂

(Since I sadly don't have enough reputation to post an image, you can find a picture of my non-sensible output here: https://mattmoony.github.io/cdn/conv-net_pytorch-style-transfer/problem.png)

I sincerely hope that you will be able to help me out and that I'm able to use this issue as a learning experience. Thank you!

Output:
Hurrah!

After yet another day of researching and testing, I've finally discovered the bug in my code.

The problem doesn't lie with the training process or the model itself, but rather with the lines responsible for loading the style image. (this article helped me discover the issue)

So... I added the following two functions to my script ...

def load_img(p):
    img = Image.open(p).convert('RGB')
    im_transform = transforms.Compose([
        transforms.ToTensor()
    ])
    img = im_transform(img).unsqueeze(0).cuda()

    return img

def to_img(t):
    img = t.cpu().clone().detach()
    img = img.numpy().squeeze()
    img = img.transpose(1, 2, 0)
    img = img.clip(0, 1)

    return img

And used them in my training function ...

def train(...):
...
    style = load_img(style_img)
...
            if i % 5 == 0:
                plt.title('Iter#{:04d}'.format(i))
                plt.imshow(to_img(img))
                plt.pause(1e-3)
...
<etc.>
...

I'm glad that I was able to figure out what the problematic part of my script was and I'll do my best to avoid similar bugs in the future.

I hope that this helps anyone experiencing similar problems or that it at least inspires them to keep on chasing for a solution.