Instruction: how to calculate relative pose given two 4x4 affine matrix

Input:
I have two 4x4 affine matrix, A and B. They represent the pose of two objects in the world coordinate system.
How could I calculate their relative pose via matrix multiplication ? (Actually, I want to know the position(x_A,y_A) in the coordinate system of  object B)
I've tried with relative pose = A * B^-1
relative_pose = torch.multiply(A, torch.inverse(B)).
However, the relative translation is way too big. (A and B are pretty close to each other, while they are far away from origin point in world coordinate.)

test data for pytorch:
import torch
A = torch.tensor([[-9.3793e-01, -3.4481e-01, -3.7340e-02, -4.6983e+03],
    [ 3.4241e-01, -9.3773e-01,  5.8526e-02,  1.0980e+04],
    [-5.5195e-02,  4.2108e-02,  9.9759e-01, -2.3445e+01],
    [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  1.0000e+00]])
B =  torch.tensor([[-9.7592e-01, -2.1022e-01, -5.8136e-02, -4.6956e+03],
         [ 2.0836e-01, -9.7737e-01,  3.6429e-02,  1.0979e+04],
         [-6.4478e-02,  2.3438e-02,  9.9764e-01, -2.3251e+01],
         [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  1.0000e+00]])

Output:
So I assume you are using solid transformation matrices M in homogeneous coordinates, in other words 4x4 matrices containing a 3x3 rotation matrix R, a 3x1 translation vector T and a [0,0,0,1] homogeneous "padding" row vector. And you want to find the transformation to go from one pose to the other (I don't know how to write matrices by block, but that would be something like (R | T \\ 0 | 1)
Then I think your formula is wrong : if Y_1 = M_1 X and Y_2 = M_2 X, then you have Y_2 = M_2 M_1^-1 X, and your relative pose matrix is M_rel = M_2 M_1^-1
So you need to invert your solid transformation matrix M_1 = (R_1 | T_1 \\ 0 | 1)
If you write the equations, and if we note P = R_1^-1, then you'll find that M_1^-1 = (P | -PT \\ 0 | 1)


Instruction: From two Tensors (labels, inputs) to DataLoader

Input:
I'm working with two tensors, inputs and labels, and I want to have them together to train a model. I'm using torch 1.7, but I can't use the function TensorDataset() and then apply DataLoader(), due to some incompatibilities with other packages when I use TensorDataset(). There is another solution to my problem?
Summary:
2 Tensors --> DataLoader without using TensorDataset()

Output:
You can construct your own custom DataSet:
class MyDataSet(torch.utils.data.Dataset):
  def __init__(self, x, y):
    super(MyDataSet, self).__init__()
    # store the raw tensors
    self._x = x
    self._y = y

  def __len__(self):
    # a DataSet must know it size
    return self._x.shape[0]

  def __getitem__(self, index):
    x = self._x[index, :]
    y = self._y[index, :]
    return x, y


Instruction: TensorBoard: Combine two graphs into one graph

Input:
I accidentally ran a model where the training loss goes to "Cos_BN Loss", and the validation loss go to "Gen Cos_BN Loss", that means that I have two separate graphs for train and validation (for the same type of loss).
I want to see them together on the same plot. How?
NOTE: I know I can just run again the model by calling the validation loss - the same name as the training loss, BUT it took around 3 days to ran, on 3 GPUs and I really don't want to train it again all over.
Current state
Thank you

Output:
On the bottom there is a download data link that let's you save the data as a CSV file. You may need to enable it first (top left in following picture):

You can then import this e.g. using pandas and plot it to your liking using `matplotlib.


Instruction: Turn the two highest values of a pytorch tensor / numpy array to 1 and the others to zero

Input:
I have some pytorch tensors (or numpy arrays) and want to turn the two highest numbers to a 1 and every other number to a zero. So this tensor
tensor([0.9998, 0.9997, 0.9991, 0.9998, 0.9996, 0.9996, 0.9997, 0.9995],
   dtype=torch.float64)

Should become this:
tensor([1, 0, 0, 1, 0, 0, 0, 0],
   dtype=torch.float64)

There are some ways to turn the highest number to 1 and the others to 0, but I need the two highest numbers to become 1. Why isn't there a built in function for that? I have a classification problem where I know that two objects belong to class 1. So shouldn't this be a quite common problem?

Output:
You can do this with topk:
x = tensor([0.9998, 0.9997, 0.9991, 0.9998, 0.9996, 0.9996, 0.9997, 0.9995], dtype=torch.float64)

_, idx = x.topk(2)

x.fill_(0)
x[idx] = 1


Instruction: Tensorflow equivalent to torch.Tensor.index_copy

Input:
I am implementing the tensorflow equivalent of the model here originally implemented using pytorch. Everything was going smoothly until I encountered this particular line of code.
batch_current = Variable(torch.zeros(size, self.embedding_dim))

# self.embedding and self.W_c are pytorch network layers I have created
batch_current = self.W_c(batch_current.index_copy(0, Variable(torch.LongTensor(index)),
                                                         self.embedding(Variable(self.th.LongTensor(current_node)))))

If search for the documentation of index_copy and it seems all it does is to copy a group of elements at a certain index and on a common axis and assign it to another tensor. But I don't really want to write some buggy code, so before attempting any self-implementation, I wish to know if you folks have an idea of how I can go about implementing it.
The model is from this paper and yes, I have searched other tensorflow implementations, but they don't seem to make so much sense to me.

Output:
What you need is the tf.tensor_scatter_nd_update in tensorflow to get equivalent operation like Tensor.index_copy_ of pytorch. Here is one demonstration shown below.
In pytorch, you have
import torch

tensor = torch.zeros(5, 3)
indices = torch.tensor([0, 4, 2])
updates= torch.tensor([[1, 2, 3],
                       [4, 5, 6],
                       [7, 8, 9]], dtype=torch.float)
tensor.index_copy_(0, indices, updates)

tensor([[1., 2., 3.],
        [0., 0., 0.],
        [7., 8., 9.],
        [0., 0., 0.],
        [4., 5., 6.]])

And in tensorflow, you can do
import tensorflow as tf

tensor = tf.zeros([5,3])
indices = tf.constant([[0], [4], [2]])
updates  = tf.constant([[1, 2, 3],
                        [4, 5, 6],
                        [7, 8, 9]], dtype=tf.float32)
tensor = tf.tensor_scatter_nd_update(tensor, indices, updates)
tensor.numpy()
array([[1., 2., 3.],
       [0., 0., 0.],
       [7., 8., 9.],
       [0., 0., 0.],
       [4., 5., 6.]], dtype=float32)


Instruction: DQN Pytorch Loss keeps increasing

Input:
I am implementing simple DQN algorithm using pytorch, to solve the CartPole environment from gym. I have been debugging for a while now, and I cant figure out why the model is not learning.
Observations:

using SmoothL1Loss performs worse than MSEloss, but loss increases for both
smaller LR in Adam does not work, I have tested using 0.0001, 0.00025, 0.0005 and default

Notes:

I have debugged various parts of the algorithm individually, and can say with good confidence that the issue is in the learn function. I am wondering if this bug is due to me misunderstanding detach in pytorch or some other framework mistake im making.
I am trying to stick as close to the original paper as possible (linked above)

References:

example: GitHub gist
example: pytroch official

import torch as T
import torch.nn as nn
import torch.nn.functional as F

import gym
import numpy as np

class ReplayBuffer:
    def __init__(self, mem_size, input_shape, output_shape):
        self.mem_counter = 0
        self.mem_size = mem_size
        self.input_shape = input_shape

        self.actions = np.zeros(mem_size)
        self.states = np.zeros((mem_size, *input_shape))
        self.states_ = np.zeros((mem_size, *input_shape))
        self.rewards = np.zeros(mem_size)
        self.terminals = np.zeros(mem_size)

    def sample(self, batch_size):
        indices = np.random.choice(self.mem_size, batch_size)
        return self.actions[indices], self.states[indices], \
            self.states_[indices], self.rewards[indices], \
            self.terminals[indices]

    def store(self, action, state, state_, reward, terminal):
        index = self.mem_counter % self.mem_size

        self.actions[index] = action
        self.states[index] = state
        self.states_[index] = state_
        self.rewards[index] = reward
        self.terminals[index] = terminal
        self.mem_counter += 1

class DeepQN(nn.Module):
    def __init__(self, input_shape, output_shape, hidden_layer_dims):
        super(DeepQN, self).__init__()

        self.input_shape = input_shape
        self.output_shape = output_shape

        layers = []
        layers.append(nn.Linear(*input_shape, hidden_layer_dims[0]))
        for index, dim in enumerate(hidden_layer_dims[1:]):
            layers.append(nn.Linear(hidden_layer_dims[index], dim))
        layers.append(nn.Linear(hidden_layer_dims[-1], *output_shape))

        self.layers = nn.ModuleList(layers)

        self.loss = nn.MSELoss()
        self.optimizer = T.optim.Adam(self.parameters())

    def forward(self, states):
        for layer in self.layers[:-1]:
            states = F.relu(layer(states))
        return self.layers[-1](states)

    def learn(self, predictions, targets):
        self.optimizer.zero_grad()
        loss = self.loss(input=predictions, target=targets)
        loss.backward()
        self.optimizer.step()

        return loss

class Agent:
    def __init__(self, epsilon, gamma, input_shape, output_shape):
        self.input_shape = input_shape
        self.output_shape = output_shape
        self.epsilon = epsilon
        self.gamma = gamma

        self.q_eval = DeepQN(input_shape, output_shape, [64])
        self.memory = ReplayBuffer(10000, input_shape, output_shape)

        self.batch_size = 32
        self.learn_step = 0

    def move(self, state):
        if np.random.random() < self.epsilon:
            return np.random.choice(*self.output_shape)
        else:
            self.q_eval.eval()
            state = T.tensor([state]).float()
            action = self.q_eval(state).max(axis=1)[1]
            return action.item()

    def sample(self):
        actions, states, states_, rewards, terminals = \
            self.memory.sample(self.batch_size)

        actions = T.tensor(actions).long()
        states = T.tensor(states).float()
        states_ = T.tensor(states_).float()
        rewards = T.tensor(rewards).view(self.batch_size).float()
        terminals = T.tensor(terminals).view(self.batch_size).long()

        return actions, states, states_, rewards, terminals

    def learn(self, state, action, state_, reward, done):
        self.memory.store(action, state, state_, reward, done)

        if self.memory.mem_counter < self.batch_size:
            return

        self.q_eval.train()
        self.learn_step += 1
        actions, states, states_, rewards, terminals = self.sample()
        indices = np.arange(self.batch_size)
        q_eval = self.q_eval(states)[indices, actions]
        q_next = self.q_eval(states_).detach()
        q_target = rewards + self.gamma * q_next.max(axis=1)[0] * (1 - terminals)

        loss = self.q_eval.learn(q_eval, q_target)
        self.epsilon *= 0.9 if self.epsilon > 0.1 else 1.0

        return loss.item()

def learn(env, agent, episodes=500):
    print('Episode: Mean Reward: Last Loss: Mean Step')

    rewards = []
    losses = [0]
    steps = []
    num_episodes = episodes
    for episode in range(num_episodes):
        done = False
        state = env.reset()
        total_reward = 0
        n_steps = 0

        while not done:
            action = agent.move(state)
            state_, reward, done, _ = env.step(action)
            loss = agent.learn(state, action, state_, reward, done)

            state = state_
            total_reward += reward
            n_steps += 1

            if loss:
                losses.append(loss)

        rewards.append(total_reward)
        steps.append(n_steps)

        if episode % (episodes // 10) == 0 and episode != 0:
            print(f'{episode:5d} : {np.mean(rewards):5.2f} '
                  f': {np.mean(losses):5.2f}: {np.mean(steps):5.2f}')
            rewards = []
            losses = [0]
            steps = []

    print(f'{episode:5d} : {np.mean(rewards):5.2f} '
          f': {np.mean(losses):5.2f}: {np.mean(steps):5.2f}')
    return losses, rewards

if __name__ == '__main__':
    env = gym.make('CartPole-v1')
    agent = Agent(1.0, 1.0,
                  env.observation_space.shape,
                  [env.action_space.n])

    learn(env, agent, 500)

Output:
The main problem I think is the discount factor, gamma. You are setting it to 1.0, which mean that you are giving the same weight to the future rewards as the current one. Usually in reinforcement learning we care more about the immediate reward than the future, so gamma should always be less than 1.
Just to give it a try I set gamma = 0.99 and run your code:
Episode: Mean Reward: Last Loss: Mean Step
  100 : 34.80 :  0.34: 34.80
  200 : 40.42 :  0.63: 40.42
  300 : 65.58 :  1.78: 65.58
  400 : 212.06 :  9.84: 212.06
  500 : 407.79 : 19.49: 407.79

As you can see the loss still increases (even if not as much as before), but so does the reward. You should consider that loss here is not a good metric for the performance, because you have a moving target. You can reduce the instability of the target by using a target network. With additional parameter tuning and a target network one could probably make the loss even more stable.
Also generally note that in reinforcement learning the loss value is not as important as it is in supervised; a decrease in loss does not always imply an improvement in performance, and vice versa.
The problem is that the Q target is moving while the training steps happen; as the agent plays, predicting the correct sum of rewards gets extremely hard (e.g. more states and rewards explored means higher reward variance), so the loss increases. This is even clearer in more complex the environments (more states, variated rewards, etc).
At the same time the Q network is getting better at approximating the Q values for each action, so the rewards (could) increase.


Instruction: How to discard a branch after training a pytorch model

Input:
I am trying to implement a FCN in pytorch with the overall structure as below:

The code so far looks like below:
class SNet(nn.Module):
    def __init__(self):
        super(SNet, self).__init__()

        self.enc_a = encoder(...)
        self.dec_a = decoder(...)

        self.enc_b = encoder(...)
        self.dec_b = decoder(...)

    def forward(self, x1, x2):
        x1 = self.enc_a(x1)
        x2 = self.enc_b(x2)
        x2 = self.dec_b(x2)
        x1 = self.dec_a(torch.cat((x1, x2), dim=-1)
        return x1, x2

In keras it is relatively easy to do this using the functional API. However, I could not find any concrete example / tutorial to do this in pytorch.

How can I discard the dec_a (decoder part of autoencoder branch) after training?
During joint training the loss will be sum (optionally weighted) of the loss from both the branch?

Output:
You can also define separate modes for your model for training and inference:
class SNet(nn.Module):
  def __init__(self):
    super(SNet, self).__init__()

    self.enc_a = encoder(...)
    self.dec_a = decoder(...)

    self.enc_b = encoder(...)
    self.dec_b = decoder(...)

    self.training = True

  def forward(self, x1, x2):
    if self.training:
        x1 = self.enc_a(x1)
        x2 = self.enc_b(x2)
        x2 = self.dec_b(x2)
        x1 = self.dec_a(torch.cat((x1, x2), dim=-1)
        return x1, x2
    else:
        x1 = self.enc_a(x1)
        x2 = self.enc_b(x2)
        x2 = self.dec_b(x2)
        return x2

These blocks are examples and may not do exactly what you want because I think there is a bit of ambiguity between how you define the training and inference operations in your block chart vs. your code, but in any case you get the idea of how you can use some modules only during training mode. Then you can just set this variable accordingly.


Instruction: how can I connect models in pytorch?

Input:
I made 2 classes:
class Preprocess():
    def __init
     .....
    def forward
     ....

class Model ():
    def __init
     .....
    def forward

I would like to do Preprocess before putting them into Model. Could you kindly tell me how can I achieve that ?

Output:
One could also use torch.nn.Sequential which has the following upsides:

easier on the eyes
one has to just pass the inputs instead of remembering to call Preprocess before it

Code being:
model = torch.nn.Sequential(Preprocess(), Model())


Instruction: Pytorch TypeError: forward() takes 2 positional arguments but 4 were given

Input:
from torch.nn.parameter import Parameter
from torch.nn.modules.module import Module
class Graphconvlayer(nn.Module):
  def __init__(self,adj,input_feature_neurons,output_neurons):
    super(Graphconvlayer, self).__init__()
    self.adj=adj
    self.input_feature_neurons=input_feature_neurons
    self.output_neurons=output_neurons
    self.weights=Parameter(torch.normal(mean=0.0,std=torch.ones(input_feature_neurons,output_neurons)))
    self.bias=Parameter(torch.normal(mean=0.0,std=torch.ones(input_feature_neurons)))

  def forward(self,inputfeaturedata):
    output1= torch.mm(self.adj,inputfeaturedata)
    print(output1.shape)
    print(self.weights.shape)
    print(self.bias.shape)
    output2= torch.matmul(output1,self.weights.t())+ self.bias
    return output2

class GCN(nn.Module):
   def __init__(self,lr,dropoutvalue,adjmatrix,inputneurons,hidden,outputneurons):
     super(GCN, self).__init__()
     self.lr=lr
     self.dropoutvalue=dropoutvalue
     self.adjmatrix=adjmatrix
     self.inputneurons=inputneurons
     self.hidden=hidden
     self.outputneurons=outputneurons
     self.gcn1 = Graphconvlayer(adjmatrix,inputneurons,hidden)
     self.gcn2 = Graphconvlayer(adjmatrix,hidden,outputneurons)

   def forward(self,x,adj):
     x= F.relu(self.gcn1(adj,x,64))
     x= F.dropout(x,self.dropoutvalue)
     x= self.gcn2(adj,x,7)
     return F.log_softmax(x,dim=1)

a=GCN(lr=0.001,dropoutvalue=0.5,adjmatrix=adj,inputneurons=features.shape[1],hidden=64,outputneurons=7)
a.forward(adj,features)

TypeError                                 Traceback (most recent call last)
<ipython-input-85-7d1a2a73ecad> in <module>()
     37
     38 a=GCN(lr=0.001,dropoutvalue=0.5,adjmatrix=adj,inputneurons=features.shape[1],hidden=64,outputneurons=7)
---> 39 a.forward(adj,features)

1 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self.forward(*input, **kwargs)
    888         for hook in itertools.chain(
--> 889                 _global_forward_hooks.values(),
    890                 self._forward_hooks.values()):
    891             hook_result = hook(self, input, result)

TypeError: forward() takes 2 positional arguments but 4 were given

print(a)
>>>
GCN(
  (gcn1): Graphconvlayer()
  (gcn2): Graphconvlayer()
)

This is a graph neural network. What I am trying to get is the output from the forward layer. I am not sure why I am getting the above error and what I should change for the code to work.
Can anyone guide me through this?
Also I am if I pass class graphconvlayer to class GCN, do I have to now separately pass each of it's parameters also to the object ä of class GCN?

Output:
Your GCN is composed of two Graphconvlayer.
As defined in the code you posted, Graphconvlayer's forward method expects only one input argument: inputfeaturedata. However, when GCN calls self.gcn1 or self.gcn2 (in its forward method) it passes 3 arguments: self.gcn1(adj,x,64) and self.gcn2(adj,x,7).
Hence, instead of a single input argument, self.gcn1 and self.gcn2 are receiving 3 -- this is the error you are getting.


Instruction: Cuda out of memory issue with pytorch when training two networks jointly

Input:
I try to train two DNN jointly, The model is trained and goes to the validation phase after every 5 epochs, the problem is after the 5 epochs it is okay and no problem with memory, but after 10 epochs the model complains about Cuda memory. Any help to solve the memory issue.
class Trainer(BaseTrainer):
    def __init__(self, config, resume: bool, model, loss_function, optimizer, train_dataloader, validation_dataloader):
        super(Trainer, self).__init__(config, resume, model, loss_function, optimizer)
        self.train_data_loader = train_dataloader
        self.validation_data_loader = validation_dataloader
        self.model = self.model.double()

    def _train_epoch(self, epoch):
        #loss_total = 0.0
        for i, (mixture, clean, name, label) in enumerate(self.train_data_loader):
            mixture = mixture.to(self.device, dtype=torch.double)
            clean = clean.to(self.device, dtype=torch.double)
            enhanced = self.model(mixture).to(self.device)
            front_loss = self.loss_function(clean, enhanced)

            front_loss.backward(retain_graph=True)

            torch.cuda.empty_cache()
            model_back.train()
            y = model_back(enhanced.double().to(device2))
            back_loss = backend_loss(y[0], label[0].to(device2))
            print("Iteration %d in epoch%d--> loss = %f" % (i, epoch, back_loss.item()), end='\r')

            #optimizer_back.zero_grad()
            back_loss.backward(retain_graph=True)
            self.optimizer.zero_grad()
            self.optimizer.step()
            #optimizer_back.step()
            torch.cuda.empty_cache()
            #loss_total += float(front_loss.item() + back_loss.item())
        dl_len = len(self.train_data_loader)
        #self.writer.add_scalar(f"Train/Loss", loss_total / dl_len, epoch)

    @torch.no_grad()
    def _validation_epoch(self, epoch):
            #visualize_audio_limit = self.validation_custom_config["visualize_audio_limit"]
            #visualize_waveform_limit = self.validation_custom_config["visualize_waveform_limit"]
            #visualize_spectrogram_limit = self.validation_custom_config["visualize_spectrogram_limit"]

            sample_length = self.validation_custom_config["sample_length"]

            stoi_c_n = []  # clean and noisy
            stoi_c_e = []  # clean and enhanced
            stoi_e_n = []
            pesq_c_n = []
            pesq_c_e = []
            pesq_e_n = []
            correct = []

            for i, (mixture, clean, name, label) in enumerate(self.validation_data_loader):
                #assert len(name) == 1, "Only support batch size is 1 in enhancement stage."
                name = name[0]
                padded_length = 0

                mixture = mixture.to(self.device)

                if mixture.size(-1) % sample_length != 0:
                    padded_length = sample_length - (mixture.size(-1) % sample_length)
                    mixture = torch.cat([mixture, torch.zeros(1, 1, padded_length, device=self.device)], dim=-1)

                assert mixture.size(-1) % sample_length == 0 and mixture.dim() == 3
                mixture_chunks = list(torch.split(mixture, sample_length, dim=-1))

                enhanced_chunks = []
                for chunk in mixture_chunks:
                    enhanced_chunks.append(self.model(chunk.double()).detach().cpu())

                enhanced = torch.cat(enhanced_chunks, dim=-1)  # [1, 1, T]
                enhanced = enhanced.to(self.device)
                #print(enhanced)
                if padded_length != 0:
                    enhanced = enhanced[:, :, :-padded_length]
                    mixture = mixture[:, :, :-padded_length]

                torch.cuda.empty_cache()
                model_back.eval()

                y_pred = model_back(enhanced.double().to(self.device))
                pred = torch.argmax(y_pred[0].detach().cpu(), dim=1)
                intent_pred = pred
                correct.append((intent_pred == label[0]).float())
                torch.cuda.empty_cache()
            acc = np.mean(np.hstack(correct))
            intent_acc = acc
            iter_acc = '\n iteration %d epoch %d -->' %(i, epoch)
            print(iter_acc, acc, best_accuracy)
            if intent_acc > best_accuracy:
                improved_accuracy = 'Current accuracy {}, {}'.format(intent_acc, best_accuracy)
                print(improved_accuracy)
                torch.save(model_back.state_dict(), '/home/mnabih/jt/best_model.pkl')

Some solutions I have tried are to use a garbage collector tool (gc library) and to add torch.Cuda.empty_cache() randomly in the training and validation.

Output:
Don't use retain_graph = True on the second backwards pass. This flag is making your code store the computation graphs for each batch, conceivably in perpetuity.
You should only use retain_graph for all but the last call to backward() that back-propagate through the same variables/parameters.


Instruction: fastaiv2 to pytorch for torchserver

Input:
I usually use fastai (v2 or v1) for fast prototyping. Now I'd like to deploy one of my models, trained with fastai, to torchserver.
Let's say that we have a simple model like this one:
    learn = cnn_learner(data,
                    models.resnet34,
                    metrics=[accuracy, error_rate, score])
    # after the training
    torch.save(learn.model.state_dict(), "./test1.pth")
    state = torch.load("./test1.pth")
    model_torch_rep = models.resnet34()
    model_torch_rep.load_state_dict(state)

I've tried many different things with the same result
    RuntimeError Traceback (most recent call last)
    <ipython-input-284-e4dbdce23d43> in <module>
    ----> 1 model_torch_rep.load_state_dict(state);

    /opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
        837         if len(error_msgs) > 0:
        838             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
    --> 839                                self.__class__.__name__, "\n\t".join(error_msgs)))
        840         return _IncompatibleKeys(missing_keys, unexpected_keys)
        841

    RuntimeError: Error(s) in loading state_dict for ResNet:
        Missing key(s) in state_dict: "conv1.weight", "bn1.weight", "bn1.bias", "bn1.running_mean", "bn1.running_var", "layer1.0.conv1.weight", "layer1.0.bn1.weight"

This is happening with fastai 1.0.6 or fastai 2.3.1 + pytorch 1.8.1 ...

Output:
Just figured this out.
For some reason the way you save the state_dict adds a string "module." to each key in the loaded state_dict. (This is because you aren't using Learner class from FastAI to save the model, I assume).
Simply remove the "module." substring from the state_dict and you're all good.
    learn = cnn_learner(data,
                        models.resnet34,
                        metrics=[accuracy, error_rate, score])

    # after the training
    torch.save(learn.model.state_dict(), "./test1.pth")
    state = torch.load("./test1.pth")

    # fix dict keys
    new_state = OrderedDict([(k.partition('module.')[2], v) for k, v in state.items()])

    model_torch_rep = models.resnet34()
    model_torch_rep.load_state_dict(new_state)


Instruction: Why does my convolutional model does not learn?

Input:
I am currently working on building a CNN for sound classification. The problem is relatively simple: I need my model to detect whether there is human speech on an audio record. I made a train / test set containing records of 3 seconds on which there is human speech (speech) or not (no_speech). From these 3 seconds fragments I get a mel-spectrogram of dimension 128 x 128 that is used to feed the model.
Since it is a simple binary problem I thought the a CNN would easily detect human speech but I may have been too cocky. However, it seems that after 1 or 2 epoch the model doesn’t learn anymore, i.e. the loss doesn’t decrease as if the weights do not update and the number of correct prediction stays roughly the same. I tried to play with the hyperparameters but the problem is still the same. I tried a learning rate of 0.1, 0.01 … until 1e-7. I also tried to use a more complex model but the same occur.
Then I thought it could be due to the script itself but I cannot find anything wrong: the loss is computed, the gradients are then computed with backward() and the weights should be updated. I would be glad you could have a quick look at the script and let me know what could go wrong! If you have other ideas of why this problem may occur I would also be glad to receive some advice on how to best train my CNN.
I based the script on the LunaTrainingApp from “Deep learning in PyTorch” by Stevens as I found the script to be elegant. Of course I modified it to match my problem, I added a way to compute the precision and recall and some other custom metrics such as the % of correct predictions.
Here is the script:

import torch
import torch.nn as nn
import argparse
import numpy as np

import logging
logging.basicConfig(level = logging.INFO)
log = logging.getLogger(__name__)

from torch.optim import SGD
from torch.utils.tensorboard import SummaryWriter
from torch.utils.data import DataLoader
from sklearn.metrics import confusion_matrix

from dataset_loader.audiodataset import AudioDataset
from models.vadnet import VADNet
from utils.earlystopping import EarlyStopping

class VADTrainingApp:

    def __init__(self, sys_argv=None):

        parser = argparse.ArgumentParser()

        parser.add_argument("--train_path",
                           help='Path to the training set',
                           required=True,
                           type=str,
        )

        parser.add_argument("--test_path",
                           help='Path to the testing set',
                           required=True,
                           type=str,
        )

        parser.add_argument("--save_path",
                           help='Path to saving the model',
                           required=True,
                           type=str,
        )

        parser.add_argument("--save_es",
            help='Save the checkpoints of early stopping call',
            default="checkpoint.pt",
            type=str,
        )

        parser.add_argument('--num-workers',
            help='Number of worker processes for background data loading',
            default=8,
            type=int,
        )

        parser.add_argument("--batch_size",
                            help='Batch size to use for training',
                            default=32,
                            type=int,)

        parser.add_argument('--epochs',
            help='Number of epochs to train for',
            default=50,
            type=int,
        )

        parser.add_argument('--lr',
            help='Learning rate for th stochastic gradient descent',
            default=0.001,
            type=float,
        )

        self.cli_args = parser.parse_args(sys_argv)

        # related to the hardware
        self.use_cuda = torch.cuda.is_available()
        self.device = torch.device("cuda" if self.use_cuda else "cpu")

        # directly related to the neural network
        self.model = self.initModel()
        self.optimizer = self.initOptimizer()

        # For early stopping
        self.patience = 7

        # For metrics
        self.METRICS_LABELS_NDX = 0
        self.METRICS_PREDS_NDX = 1
        self.METRICS_LOSS_NDX = 2
        self.METRICS_SIZE = 3

    def initModel(self):
        """Initialize the model, if GPU available computation done there"""
        model = VADNet()
        model = model.double()

        if self.use_cuda:
            log.info("Using CUDA; {} devices".format(torch.cuda.device_count()))
            if torch.cuda.device_count() > 1:
                model = nn.DataParallel(model)
            model = model.to(self.device)
        return model

    def initOptimizer(self):

        return SGD(self.model.parameters(), lr=self.cli_args.lr)#, momentum=0.8, weight_decay=0.01)

    def adjust_learning_rate(self):
        """Sets the learning rate to the initial LR decayed by a factor of 10 every 20 epochs"""
        self.cli_args.lr = self.cli_args.lr * (0.1 ** (self.cli_args.epochs // 20))
        for param_group in self.optimizer.param_groups:
            param_group['lr'] = self.cli_args.lr

    def initTrainDL(self):

        trainingset = AudioDataset(self.cli_args.train_path,
                                   n_fft=1024,
                                   hop_length=376,
                                   n_mels=128)

        batch_size = self.cli_args.batch_size
        if self.use_cuda:
            batch_size *= torch.cuda.device_count()

        trainLoader = DataLoader(trainingset,
                                batch_size = batch_size,
                                shuffle=True,
                                num_workers=self.cli_args.num_workers,
                                pin_memory=self.use_cuda)
        return trainLoader

    def initTestDL(self):

        testset = AudioDataset(self.cli_args.test_path,
                                   n_fft=1024,
                                   hop_length=376,
                                   n_mels=128)

        batch_size = self.cli_args.batch_size
        if self.use_cuda:
            batch_size *= torch.cuda.device_count()

        testLoader = DataLoader(testset,
                                batch_size = batch_size,
                                shuffle=True,
                                num_workers=self.cli_args.num_workers,
                                pin_memory=self.use_cuda)
        return testLoader

    def main(self):

        log.info("Start training, {}".format(self.cli_args))

        train_dl = self.initTrainDL()
        test_dl = self.initTestDL()

        trn_writer = SummaryWriter(log_dir='runs' + '-trn')
        val_writer = SummaryWriter(log_dir='runs' + '-val')

        early_stopping = EarlyStopping(patience=self.patience, path=self.cli_args.save_es, verbose=True)

        for epoch_ndx in range(1, self.cli_args.epochs + 1):
            log.info("Epoch {} / {}".format(epoch_ndx, self.cli_args.epochs))

            # Adjust the new learning rate
            self.adjust_learning_rate()

            # Train the model's parameters
            metrics_t = self.do_training(train_dl)
            self.logMetrics(metrics_t, trn_writer, epoch_ndx)

            # Test the model
            metrics_v = self.do_val(test_dl, val_writer)
            self.logMetrics(metrics_v, val_writer, epoch_ndx, train=False)

            # Add the mean loss of the val for the epoch
            early_stopping(metrics_v[self.METRICS_LOSS_NDX].mean(), self.model)

            if early_stopping.early_stop:
                print("Early stopping")
                break

        # Save the model once all epochs have been completed
        torch.save(self.model.state_dict(), self.cli_args.save_path)

    def do_training(self, train_dl):
        """Training loop"""
        self.model.train()

        # Initiate a 3 dimension tensor to store loss, labels and prediction
        trn_metrics = torch.zeros(self.METRICS_SIZE, len(train_dl.dataset), device=self.device)

        for batch_ndx, batch_tup in enumerate(train_dl):

            if batch_ndx%100==0:
                log.info("TRAINING --> Batch {} / {}".format(batch_ndx, len(train_dl)))

            self.optimizer.zero_grad()

            loss = self.ComputeBatchLoss(batch_ndx,
                                        batch_tup,
                                        self.cli_args.batch_size,
                                        trn_metrics)

            loss.backward()
            self.optimizer.step()

        return trn_metrics.to('cpu')

    def do_val(self, test_dl, early_stop):
        """Validation loop"""
        with torch.no_grad():
            self.model.eval()

            val_metrics = torch.zeros(self.METRICS_SIZE, len(test_dl.dataset), device=self.device)

            for batch_ndx, batch_tup in enumerate(test_dl):

                if batch_ndx%100==0:
                    log.info("VAL --> Batch {} / {}".format(batch_ndx, len(test_dl)))

                loss = self.ComputeBatchLoss(batch_ndx,
                                            batch_tup,
                                            self.cli_args.batch_size,
                                            val_metrics)

        return val_metrics.to('cpu')

    def ComputeBatchLoss(self, batch_ndx, batch_tup, batch_size, metrics_mat):
        """
        Return a tensor the loss of the batch
        """

        imgs, labels = batch_tup
        imgs = imgs.to(device=self.device, non_blocking=True)
        labels = labels.to(device=self.device, non_blocking=True)

        outputs = self.model(imgs)
        _, predicted = torch.max(outputs, dim=1)

        loss_func = nn.CrossEntropyLoss(reduction="none")
        loss = loss_func(outputs, labels)

        start_ndx = batch_ndx * self.cli_args.batch_size
        end_ndx = start_ndx + labels.size(0)

        metrics_mat[self.METRICS_LABELS_NDX, start_ndx:end_ndx] = labels.detach()
        metrics_mat[self.METRICS_PREDS_NDX, start_ndx:end_ndx] = predicted.detach()
        metrics_mat[self.METRICS_LOSS_NDX, start_ndx:end_ndx] = loss.detach()

        return loss.mean()

    def logMetrics(self, metrics_mat, writer, epoch_ndx, train=True):
        """
        Function to compute custom metrics: accurracy and recall for both classes
        and % of correct predictions. Log the metrics in a tensorboard writer
        """

        # Confusion matrix to compute precision / recall for each class
        tn, fp, fn, tp = torch.tensor(confusion_matrix(metrics_mat[self.METRICS_LABELS_NDX],
                                                       metrics_mat[self.METRICS_PREDS_NDX],
                                                       labels=[0,1]).ravel())

        precision_no_speech = tp / (tp + fp)
        recall_no_speech = tp / (tp + fn)

        # class speech is labelled 0, so true positive = true negative for speech
        precision_speech = tn / (tn + fn)
        recall_speech = tn / (fp + tn)

        # % of correct predictions - optional metrics that are nice
        no_speech_count = (metrics_mat[self.METRICS_LABELS_NDX] == 0).sum()
        speech_count = (metrics_mat[self.METRICS_LABELS_NDX] == 1).sum()

        no_speech_correct = ((metrics_mat[self.METRICS_PREDS_NDX] == 0) & (metrics_mat[self.METRICS_LABELS_NDX] == 0)).sum()
        speech_correct = ((metrics_mat[self.METRICS_PREDS_NDX] == 1) & (metrics_mat[self.METRICS_LABELS_NDX] == 1)).sum()

        correct_all = (speech_correct + no_speech_correct) / float(speech_count + no_speech_count) * 100
        correct_speech = speech_correct / float(speech_count) * 100
        correct_no_speech = no_speech_correct / float(no_speech_count) * 100

        loss = metrics_mat[self.METRICS_LOSS_NDX].mean()

        writer.add_scalar("loss", loss, epoch_ndx)

        writer.add_scalar("precision/no_speech", precision_no_speech, epoch_ndx)
        writer.add_scalar("recall/no_speech", recall_no_speech, epoch_ndx)
        writer.add_scalar("precision/speech", precision_speech, epoch_ndx)
        writer.add_scalar("recall/speech", recall_speech, epoch_ndx)

        writer.add_scalar("correct/all", correct_all, epoch_ndx)
        writer.add_scalar("correct/speech", correct_speech, epoch_ndx)
        writer.add_scalar("correct/no_speech", correct_no_speech, epoch_ndx)

        if train:
            log.info("[TRAINING] loss: {}, correct/all: {}% , correct/speech: {}%, correct/no_speech: {}%".format(loss,
                                                   correct_all,
                                                   correct_speech,
                                                   correct_no_speech))
        else:
            log.info("[VAL] loss: {}, correct/all: {}% , correct/speech: {}%, correct/no_speech: {}%".format(loss,
                                                   correct_all,
                                                   correct_speech,
                                                   correct_no_speech))

if __name__ == "__main__":
    VADTrainingApp().main()

Regarding the model I am using a simple CNN:

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class VADNet(nn.Module):
    def __init__(self, in_channels=1, conv_channels=8):
        super().__init__()

        self.tail_batchnorm = nn.BatchNorm2d(1)

        self.block1 = ConvBlock(in_channels, conv_channels)
        self.block2 = ConvBlock(conv_channels, conv_channels * 2)
        self.block3 = ConvBlock(conv_channels * 2, conv_channels * 4)
        self.block4 = ConvBlock(conv_channels * 4, conv_channels * 8)

        self.head_linear = nn.Linear(8 * 8 * conv_channels * 8, 2)

        self._init_weights()

    def _init_weights(self):
        for m in self.modules():
            if type(m) in {
                nn.Linear,
                nn.Conv3d,
                nn.Conv2d,
                nn.ConvTranspose2d,
                nn.ConvTranspose3d,
            }:
                nn.init.kaiming_normal_(
                    m.weight.data, a=0, mode='fan_out', nonlinearity='relu',
                )
                if m.bias is not None:
                    fan_in, fan_out = \
                        nn.init._calculate_fan_in_and_fan_out(m.weight.data)
                    bound = 1 / math.sqrt(fan_out)
                    nn.init.normal_(m.bias, -bound, bound)

    def forward(self, input_batch):
        bn_output = self.tail_batchnorm(input_batch)

        block_out = self.block1(bn_output)
        block_out = self.block2(block_out)
        block_out = self.block3(block_out)
        block_out = self.block4(block_out)

        conv_flat = block_out.view(block_out.size(0),-1)
        linear_output = self.head_linear(conv_flat)

        return linear_output

class ConvBlock(nn.Module):
    def __init__(self, in_channels, conv_channels):
        super().__init__()

        self.conv1 = nn.Conv2d(
            in_channels, conv_channels, kernel_size=3, padding=1, bias=True,
        )
        self.relu1 = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(
            conv_channels, conv_channels, kernel_size=3, padding=1, bias=True,
        )
        self.relu2 = nn.ReLU(inplace=True)

        self.maxpool = nn.MaxPool2d(2, 2)

    def forward(self, input_batch):
        block_out = self.conv1(input_batch)
        block_out = self.relu1(block_out)
        block_out = self.conv2(block_out)
        block_out = self.relu2(block_out)

        return self.maxpool(block_out)

Thank you in advance for your help!

Output:
You are applying 2D 3x3 convolutions to spectrograms.
Read it once more and let it sink.
Do you understand now what is the problem?

 A convolution layer learns a static/fixed local patterns and tries to match it everywhere in the input. This is very cool and handy for images where you want to be equivariant to translation and where all pixels have the same "meaning".
 However, in spectrograms, different locations have different meanings - pixels at the top part of the spectrograms mean high frequencies while the lower indicates low frequencies.
 Therefore, if you have matched some local pattern to a local region in the spectrogram, it may mean a completely different thing if it is matched to the upper or lower part of the spectrogram.
 You need a different kind of model to process spectrograms. Maybe convert the spectrogram to a 1D signal with 128 channels (frequencies) and apply 1D convolutions to it?


Instruction: ValueError: not enough values to unpack (expected 3, got 2) in Pytorch

Input:
this is my Define validate function
when I load the model and start prediction using this code I have received the error using PyTorch.and after this, I am iterating through the epoch loop and batch loop and I landed with this error.
def validate_epoch(net, val_loader,loss_type='CE'):
    net.train(False)
    running_loss = 0.0
    sm = nn.Softmax(dim=1)

    truth = []
    preds = []
    bar = tqdm(total=len(val_loader), desc='Processing', ncols=90)
    names_all = []
    n_batches = len(val_loader)
    for i, (batch, targets, names) in enumerate(val_loader):
        if loss_type == 'CE':
                labels = Variable(targets.float())
                inputs = Variable(batch)
        elif loss_type == 'MSE':
                labels = Variable(targets.float())
                inputs = Variable(batch)

        outputs = net(inputs)
        labels = labels.long()
        loss = criterion(outputs, labels)
        if loss_type =='CE':
            probs = sm(outputs).data.cpu().numpy()
        elif loss_type =='MSE':
            probs = outputs
            probs[probs < 0] = 0
            probs[probs > 4] = 4
            probs = probs.view(1,-1).squeeze(0).round().data.cpu().numpy()
        preds.append(probs)
        truth.append(targets.cpu().numpy())
        names_all.extend(names)
        running_loss += loss.item()
        bar.update(1)
        gc.collect()
    gc.collect()
    bar.close()
    if loss_type =='CE':
        preds = np.vstack(preds)
    else:
        preds = np.hstack(preds)
    truth = np.hstack(truth)
    return running_loss / n_batches, preds, truth, names_all

And this is the main function where I call validate function get the error when model is loaded and start prediction on the test loader
criterion = nn.CrossEntropyLoss()

model.eval()

test_losses = []
test_mse = []
test_kappa = []
test_acc = []

test_started = time.time()

test_loss, probs, truth, file_names = validate_epoch(model, test_iterator)

as you can see in traceback error it gives some Terminal shows error:
ValueError                                Traceback (most recent call last)
<ipython-input-27-d2b4a1ca3852> in <module>
     12 test_started = time.time()
     13
---> 14 test_loss, probs, truth, file_names = validate_epoch(model, test_iterator)
     15 preds = probs.argmax(1)
     16

<ipython-input-25-34e29e0ff6ed> in validate_epoch(net, val_loader, loss_type)
      9     names_all = []
     10     n_batches = len(val_loader)
---> 11     for i, (batch, targets, names) in enumerate(val_loader):
     12         if loss_type == 'CE':
     13                 labels = Variable(targets.float())

ValueError: not enough values to unpack (expected 3, got 2)

Output:
From torchvision.datasets.ImageFolder documentation:
"Returns:   (sample, target) where target is class_index of the target class."
So, quite simply, the dataset object you're currently using returns a tuple with 2 items. You'll get an error if you try to store this tuple in 3 variables. The correct line would be:
for i, (batch, targets) in enumerate(val_loader):

If you really need the names (which I assume is the file path for each image) you can define a new dataset object that inherits from the ImageFolder dataset and overload the __getitem__ function to also return this information.


Instruction: I want to train the CSRNet model on UCF_CC_50 dataset which has total number of images 50. and face a problem like

Input:
I want to train the CSRNet model on UCF_CC_50 dataset but occurring this problem

KeyError  Traceback (most recent call last)    <ipython-input-11-78e054690de5> in <module>
     4     img= plt.imread(img_path)
     5     k = np.zeros((img.shape[0],img.shape[1]))
***----> 6     gt = mat["image_info"][0,0][0,0][0]***
     7     for i in range(0,len(gt)):
     8         if int(gt[i][1])<img.shape[0] and int(gt[i][0])<img.shape[1]:

**KeyError: 'image_info'**

----------

enter for img_path in img_paths:
    print (img_path)
    mat = io.loadmat(img_path.replace('.jpg','.mat').replace('images','ground_truth').replace('IMG_','GT_IMG_'))
    img= plt.imread(img_path)
    k = np.zeros((img.shape[0],img.shape[1]))
    gt = mat["image_info"][0,0][0,0][0]
    for i in range(0,len(gt)):
        if int(gt[i][1])<img.shape[0] and int(gt[i][0])<img.shape[1]:
            k[int(gt[i][1]),int(gt[i][0])]=1
    k = gaussian_filter_density(k)
    with h5py.File(img_path.replace('.jpg','.h5').replace('images','groundtruth'), 'w') as hf:
            hf['density'] = kcode here

---------

The file path is

C:\Users\Gigabyte pc\Desktop\COUNTING\CSRNet-pytorch-master\UCF_CC_50\part_A_final/train_data\images\IMG_1.jpg

Output:
Your code does not comply with the structure of the annotation file you are trying to read. Annotations in UCF-50
CC dataset can simply be read by getting the values of the key "annPoints".
You could apply the following changes to your code to read the x and y coordinates of pointwise human-head annotations:
4 img= plt.imread(img_path)
5 k = np.zeros((img.shape[0],img.shape[1]))
6 gt = mat["annPoints"]
7 for i in range(0,len(gt)):


Instruction: Enforce pad_sequence to a certain length

Input:
I have a set of tensors that I'm padding with pad_sequence but I need to guarantee a fixed length for them. I can't do it right now as pad_sequence will extend the shorter tensors up to the longest, if that longest tensor doesn't reach the length I want them I'm screwed. I thought a solution could be adding zeros to one of the tensors to padd up to the length I want so the result of that padding will have my desired length. I don't know how to do it
So lets say I have a tensor with shape torch.Size([44]) and a desired length 50, how can I add zeros to it to reach a shape of torch.Size([50])? This needs to hold regardless of the initial tensor shape.

Output:
You can achieve your logic like so:
from torch.nn.utils.rnn import pad_sequence

# Desired max length
max_len = 50

# 100 seqs of variable length (< max_len)
seq_lens = torch.randint(low=10,high=44,size=(100,))
seqs = [torch.rand(n) for n in seq_lens]

# pad first seq to desired length
seqs[0] = nn.ConstantPad1d((0, max_len - seqs[0].shape[0]), 0)(seqs[0])

# pad all seqs to desired length
seqs = pad_sequence(seqs)


Instruction: convert State integers to onehotvector

Input:
I have a function that I use to convert two states into a one-hot vector and combine it all, and it works fine, but if I have a large L, for example, 3000, it will take time. The function works like this with L=3.
def OH3(x,end=2,len=3):
  x = T.LongTensor([[x]])
  end = T.LongTensor([[end]])
  one_hot_x = T.FloatTensor(len,l)
  one_hot_end = T.FloatTensor(len,l)
  first=one_hot_x.zero_().scatter_(1,x,1)
  second=one_hot_end.zero_().scatter_(1,end,1)
  vector=T.cat((one_hot_x,one_hot_end),dim=1)
  return vector

the output:
OH3(1)
tensor([[0., 1., 0., 0., 0., 1.]])

as I mentioned, the function works fine but slower for a large number of states, and I sent multiple states at once, like 500 states, it will take 4 second
Is there a faster way to do it?

Output:
Provided the assumptions made in the comments above are correct, this should be about as fast as you can get it done, one allocation of a new array and 2 assignment operations. Let me know if I'm missing something.
def OH3(x,end=2,len=3):
    vector = torch.zeros(2*len,dtype = int)
    vector[[x,len+end] = 1
    return vector


Instruction: Is it possible to run multiple CUDA version on windows?

Input:
I am doing an experiment on a chest x-ray Project. and I want multiple versions of the CUDA toolkit but the problem is that my system put the latest version which I installed lastly is appearing.
Is it possible to run any of CUDA like 9.0, 10.2, 11.0 as required to GitHub code?
I have done all the initial steps like path added to an environment variable and added CUDNN copied file and added to the environment.
Now the problem is that I want to use Cuda 9.0 as per my code but my default setting put cuda.11.0 what is the solution or script to switch easily between these version

Output:
You may set CUDA_PATH_V9_0, CUDA_PATH_V10_0, etc properly, then set CUDA_PATH to any one of them (e.g. CUDA_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0).
Then in your VS project, set your cuda library path using the CUDA_PATH (e.g. $CUDA_PATH\lib).
To switch, just set the CUDA_PATH to another version, and clean & rebuild your VS project(s).


Instruction: Using 3-channel (RGB) PyTorch model for classification 4-channel (RGBY) images

Input:
I have labeled dataset with 4-channel images (RGBY). I want to use pretrained classification model (using pytorch and ResNet50 as a model). All of pytorch models for 3 channels though.
So, the question is: How can I use 3-channel pretrained models for 4-channels data? I'm loading the model next way:
import torchvision.models as models
resnet50 = models.resnet50(pretrained=True)

Output:
You can modify the first layer of the CNN such that it expects 4 input channels instead of 3. In your case, the first layer is resnet50.conv1. So:
import torchvision.models as models
resnet50 = models.resnet50(pretrained=True)

# modify first layer so it expects 4 input channels; all other parameters unchanged
resnet50.conv1 = torch.nn.Conv2d(4,64,kernel_size = (7,7),stride = (2,2), padding = (3,3), bias = False)

# test
inp = torch.rand([1,4,512,512])
resnet50.eval()
resnet50.training = False
out = resnet50(inp) # should evaluate without error

The simplicity of this change is made possible by the following implementation detail: For a 2D convolution (also true for other dimensional convolutions), pytorch convolves one kernel for each desired output plane (feature map) with each input plane. This results in n_input_planes x n_output_planes total feature maps (in this case 4 and 64, respectively). Pytorch then sums across all input planes for each output plane, yielding a total of n_output_planes planes regardless of the number of input planes.
The good news is that this means you can add additional input planes (maps) with no modification of the network past the first layer. The (perhaps in some cases) unfavorable part of this is that all of your input feature maps are treated identically and the information from each is fully incorporated by the end of  the first convolution. In some cases it might be desirable to treat the input feature maps differently at the beginning, in which case you'd need to define two separate CNN branches so that the features were not added together at each layer.


Instruction: PyTorch preserving gradient using external libraries

Input:
I have a GAN that returns a predicted torch.tensor. To guide this network, I have a loss function which is a summation of binary cross entropy loss (BCELoss) and Wasserstein distance. However, in order to calculate Wasserstein distance, I am using scipy.stats.wasserstein_distance function from SciPy library. As you might know, this function requires two NumPy arrays as input. So, to use this function, I am converting my predicted tensor and ground-truth tensor to NumPy arrays as follows
pred_np = pred_tensor.detach().cpu().clone().numpy().ravel()
target_np = target_tensor.detach().cpu().clone().numpy().ravel()

W_loss = wasserstein_distance(pred_np, target_np)

Then, total loss is obtained by adding W_loss to BCELoss. I am now showing this part because it is a bit unnecessary and not related to my question.
My concern is I am detaching gradient so I suppose that while optimizing and updating model parameters it will not consider W_loss. I am a bit newbie so I hope my question is clear and appreciate for answers in advance.

Output:
Adding an object that is not a tensor that requires_grad to your loss is essentially adding a constant. The derivative of a constant is zero, so this added term is not doing anything to your network's weights.
tl;dr:
You need to rewrite the loss computation in pytorch (or just find an existing implementation, there's numerous on the internets).


Instruction: How to get argmax for indices in a matrix/tensor?

Input:
Is there any way to perform a top-k operation on a matrix or tensor so that the relevant indices are returned?
For example:
>>> import torch
>>> matrix = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
>>> print(matrix)
tensor([[1, 2, 3],
        [4, 5, 6],
        [7, 8, 9]])
>>> indices = max_index_function(matrix)
>>> print(indices)
tensor([2, 2])

Something like this. If I use torch.argmax it's a little tricky to get what I want. I want to be able to effectively return the indices because I want to extend an argmax to a top-k function as well.
Right now I have something like this:
>>> column_max_idx = torch.unique(torch.argmax(matrix, dim=0))
>>> row_max_idx = torch.unique(torch.argmax(matrix, dim=1))
>>> idx_pairs = torch.cartesian_prod(column_max_idx, row_max_idx)

It does what I want but I'm sure there's a better way. Thanks!
Edit
Another thing I've tried is using two for loops and sorting, but I also am wondering if there's a function I could use without loops.
>>> idxs_and_values = []
>>> for col_idx, col_element in enumerate(matrix):
...     for row_idx, row_element in enumerate(col_element):
...         idxs_and_values.append(((col_idx, row_idx), row_element.item())
>>> sorted(idxs_and_values, key=lambda x: x[-1], reverse=True)
[((2, 2), 9), ((2, 1), 8), ((2, 0), 7), ((1, 2), 6), ((1, 1), 5), \
  ((1, 0), 4), ((0, 2), 3), ((0, 1), 2), ((0, 0), 1)]

Output:
how about topk?
import torch
matrix = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
values, indices = matrix.flatten().topk(k=3)
print(values)
# tensor([9, 8, 7])
print(indices)
# tensor([8, 7, 6])

Note that the indices are now pointing to the flattened vector. You can recover the original d-dimensional indexing with numpy's unravel_index:
indices = [np.unravel_index(i, matrix.shape) for i in indices]
print(indices)
# [(2, 2), (2, 1), (2, 0)]

Nicely wrapped in a function for any tensor order:
from torch import Tensor
from numpy import unravel_index

def mytopk(x: Tensor, k: int) -> list[tuple[int,...]]:
    values, indices = x.flatten().topk(k)
    return [unravel_index(i, x.shape) for i in indices]


Instruction: Child process hangs when performing inference with PyTorch model

Input:
I have a PyTorch model (class Net), together with its saved weights / state dict (net.pth), and I want to perform inference in a multiprocessing environment.
I noticed that I cannot simply create a model instance, load the weights, then share the model with a child process (though I'd have assumed this is possible due to copy-on-write). What happens is that the child hangs on y = model(x), and finally the whole program hangs (due to parent's waitpid).
The following is a minimal reproducible example:
def handler():
    with torch.no_grad():
        x = torch.rand(1, 3, 32, 32)
        y = model(x)

    return y

model = Net()
model.load_state_dict(torch.load("./net.pth"))

pid = os.fork()

if pid == 0:
    # this doesn't get printed as handler() hangs for the child process
    print('child:', handler())
else:
    # everything is fine here
    print('parent:', handler())
    os.waitpid(pid, 0)

If the model loading is done independently for parent & child, i.e. no  sharing, then everything works as expected. I have also tried calling share_memory_ on model's tensors, but to no avail.
Am I doing something obviously wrong here?

Output:
Seems that sharing the state dict and performing the loading operation in each process solves the problem:
LOADED = False

def handler():
    global LOADED
    if not LOADED:
        # each process loads state independently
        model.load_state_dict(state)
        LOADED = True

    with torch.no_grad():
        x = torch.rand(1, 3, 32, 32)
        y = model(x)

    return y

model = Net()

# share the state rather than loading the state dict in parent
# model.load_state_dict(torch.load("./net.pth"))
state = torch.load("./net.pth")

pid = os.fork()

if pid == 0:
    print('child:', handler())
else:
    print('parent:', handler())
    os.waitpid(pid, 0)


Instruction: Why this PyTorch regression program reaches zero loss with periodic oscillations?

Input:
There is one x and one t with with dimensions 3x1. I am trying to find the w (3x3) and b (3,1) so they can please this equation:
t = w*x + b

The program below does oscillate. I tried to debug it with no success. Can someone else take a look? What did I miss?

class fit():
    def __init__(self, w, b):
        self.w = w
        self.b = b

    def forward(self, x):
        return torch.mm(self.w, x) + self.b

w = torch.tensor([[1., 1.1, 1.2],
                  [1., 1.1, 1.2],
                  [1., 1.1, 1.2]], requires_grad=True)

b = torch.tensor([[10.], [11.], [12.]], requires_grad=True)

x = torch.tensor([[1.], [2.], [3.]], requires_grad=False)
t = torch.tensor([[0.], [0.9], [0.1]], requires_grad=False)

model = fit(w, b)

alpha = 0.001

loss = []

arange = np.arange(200)

for i in arange:

    z = model.forward(x)

    l = (z - t)**2

    l = l.sum()
    loss.append(l)

    l.backward()

    model.w.data = model.w.data - alpha * model.w.grad
    model.b.data = model.b.data - alpha * model.b.grad

plt.plot(arange, loss)

If I use the other tools (torch.nn.Linear, torch.optim.sgd, torch.nn.smeloss) from PyTorch everything goes as expected.

Output:
You need to reset the gradient to 0 after each backprop. By default, pytorch accumulates gradients when you call loss.backward().
Replacing the last 2 instructions of your loop with the following lines should fix the issue :
with torch.no_grad():
    model.w.data = model.w.data - alpha * model.w.grad
    model.b.data = model.b.data - alpha * model.b.grad
    model.w.grad.zero_()
    model.b.grad.zero_()


Instruction: PyTorch NN does not learn or learns poorly

Input:
I'm working with PyTorch tutorial, slightly modified to use Titanic dataset. I'm using very simple network of Linear(Dense) with ReLU... I'd like to predict survival status based on age, fare and sex for example.
I experienced a strange behavior with a simple neural network (I'm experimenting on Google Colab). Sometimes when I execute training, the accuracy doesn't change at all. It's strange because I'm recreating the model...
Accuracy: 59.4%, Avg loss: 0.693147
[...50 or more lines like this...]
Accuracy: 59.4%, Avg loss: 0.693147

Sometimes the accuracy is slowly increasing from 60% to 80%.
The other thing is, the accuracy is very low (varies from 60-80%), despite the fact I'm validating with... the very same training set!
I've tried several different combinations of learning rate, batch size and epochs count and also number of neurons, but it still behaves very... unpredictably and weak.
Could you point me why sometimes the network doesn't learn at all? And if I rerun it a few times it starts learning somehow. And what should be done to this network to improve it?
This is my Python notebook:
https://colab.research.google.com/drive/1-50BTqnMgiz_dozv1DjXS9advD1Rxd-B?usp=sharing
Thanks in advance!

Output:
As this is a classification problem, your neural network's last layer should not have a relu activation function.
Code Snippet:
class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.fun = nn.Sequential(
            nn.Linear(3, 32),
            nn.ReLU(),
            nn.Linear(32, 1)
        )

    def forward(self, x):
        return self.fun(x)


Instruction: Division in batches of a 3D tensor (Pytorch)

Input:
I have a 3D tensor of size say 100x5x2 and mean of the tensor across axis=1 which gives shape 100x2.
100 here is the batch size. Normally without batch, the division of tensor of shape 5x2 and 2 works perfectly but in the case of the 3D tensor with batch, I’m receiving error.
a = torch.rand(5,2)
b = torch.rand(2)
z=a/b

gives me expected answer.
a = torch.rand(100,5,2)
b = torch.rand(100,2)
z=a/b

Gives me the following error.
The size of tensor a (5) must match the size of tensor b (100) at non-singleton dimension 1.
How to divide these tensors such that my output is of shape 100x5x2 ? Something like bmm for division?

Output:
Simply do:
z = a / b.unsqueeze(1)

This adds an extra dimension in b and makes it of shape (100, 1, 2) which is compatible for broadcasting with a.


Instruction: IndexError: Target 1 is out of bounds

Input:
When I run the program below, it gives me an error. The problem seems to be in the loss function but I can't find it. I have read the Pytorch Documentation for nn.CrossEntropyLoss but still can't find the problem.
Image size is (1 x 256 x 256),
Batch size is 1
I am new to PyTorch, thanks.
import torch
import torch.nn as nn
from PIL import Image
import numpy as np
torch.manual_seed(0)

x = np.array(Image.open("cat.jpg"))
x = np.expand_dims(x, axis = 0)
x = np.expand_dims(x, axis = 0)
x = torch.from_numpy(x)
x = x.type(torch.FloatTensor) # shape = (1, 1, 256, 256)

def Conv(in_channels, out_channels, kernel=3, stride=1, padding=0):
    return nn.Conv2d(in_channels, out_channels, kernel, stride, padding)

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()

        self.sequential = nn.Sequential(
            Conv(1, 3),
            Conv(3, 5),
            nn.Flatten(),
            nn.Linear(317520, 1),
            nn.Sigmoid()
        )

    def forward(self, x):
        y = self.sequential(x)
        return y

def compute_loss(y_hat, y):
    return nn.CrossEntropyLoss()(y_hat, y)

model = model()
y_hat = model(x)

loss = compute_loss(y_hat, torch.tensor([1]))

Error:
Traceback (most recent call last):
  File "D:/Me/AI/Models/test.py", line 38, in <module>
    **loss = compute_loss(y, torch.tensor([1]))**
  File "D:/Me/AI/Models/test.py", line 33, in compute_loss
    return nn.CrossEntropyLoss()(y_hat, y)
  File "D:\Softwares\Anaconda\envs\deeplearning\lib\site-packages\torch\nn\modules\module.py", line 1054, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\Softwares\Anaconda\envs\deeplearning\lib\site-packages\torch\nn\modules\loss.py", line 1120, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "D:\Softwares\Anaconda\envs\deeplearning\lib\site-packages\torch\nn\functional.py", line 2824, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
**IndexError: Target 1 is out of bounds.**

Process finished with exit code 1

Output:
This looks like a binary classifier model: cat or not cat. But you are using CrossEntropyLoss which is used when you have more than 2 target classes. So what you should use is Binary Cross Entropy Loss.
def compute_loss(y_hat, y):
    return nn.BCELoss()(y_hat, y)


Instruction: How to calculate the percentage of which kind the given image is after machine learning in pytorch?

Input:
I trained my machine with custom vgg model with CIFAR10 data set, and tested with some images in same data set.
airplane :  -16.972412
automobile :  -18.719894
bird :  -6.989656
cat :  -3.8386667
deer :  -7.622768
dog :  0.37765026
frog :  -8.165334
horse :  -7.4519434
sheep :  -21.241518
truck :  -18.978928

This is one of how I got the value of what kind the test image is. Below is what I implemented to print above:
kind = ["airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "sheep", "truck"]
for k in kind:
      print(k,": ", output.cpu().detach().numpy()[0][kind.index(k)])

Here, it is correct that given test image is dog, which is the highest value, yet I want to print every values as percentage, which the sum of all is 100. How could I do this? I used pytorch for code.

Output:
These values are logits, you should apply softmax on them to get the probability values between 0 - 1 and then you can multiply them by 100.
Code Snippet:
pred = output.cpu().detach().numpy()

def softmax(x):
  "Function computes the softmax values for the each element in the given numpy array"
  return np.exp(x) / np.sum(np.exp(x), axis=0)

def probability(x):
  "Function applies softmax for the given logit and returns the classification probability"
  return softmax(x) * 100

output_probs = probability(preds)

or you can use softmax function from scipy
from scipy.special import softmax

output_probs = softmax(preds)


Instruction: How can I change the default image shape in PyTorch?

Input:
I've noticed that PyTorch uses images with shape (channels, width, height). How can I change it to (width, height, channels)?
I am looking for a switch kind of think which switches between the two modes.

Output:
You can use torch.permute:
tt = torch.tensor([[[0, 0], [1,1]], [[0,1], [1,0]], [[2, 2], [2, 2]]])
# > tensor([[[0, 0], [1, 1]],
#           [[0, 1], [1, 0]],
#           [[2, 2], [2, 2]]])
print(tt.shape)
# > torch.Size([3, 2, 2])

tt2 = tt.permute(1, 2, 0)
# > tensor([[[0, 0, 2], [0, 1, 2]],
#           [[1, 1, 2], [1, 0, 2]]])
print(tt2.shape)
# > torch.Size([2, 2, 3])


Instruction: Get max value and its index in numpy array

Input:
I have this torch array.
       tensor([[8.22266e-01, 1.34659e-03, 9.85146e-04, 8.11100e-04],
    [9.35547e-01, 1.22261e-03, 8.70228e-04, 1.25122e-03],
    [9.48730e-01, 1.21975e-03, 9.28402e-04, 8.44955e-04],
    [7.97363e-01, 9.16004e-04, 9.16004e-04, 8.53539e-04],
    [9.26270e-01, 7.69138e-04, 8.47816e-04, 1.12724e-03],
    [9.43848e-01, 7.44820e-04, 8.53539e-04, 8.60691e-04],
    [7.89062e-01, 6.50406e-04, 9.23634e-04, 8.44479e-04],
    [9.29688e-01, 7.02858e-04, 7.60078e-04, 1.19591e-03],
    [9.47266e-01, 5.88894e-04, 8.36849e-04, 9.37462e-04],
    [8.27637e-01, 1.92642e-03, 1.73283e-03, 2.53105e-03],
    [9.22363e-01, 2.23160e-03, 1.37615e-03, 2.46811e-03],
    [9.31641e-01, 1.92928e-03, 1.49632e-03, 2.53296e-03],
    [8.25684e-01, 1.89209e-03, 1.70994e-03, 2.39944e-03],
    [9.21875e-01, 1.90926e-03, 1.28174e-03, 2.44904e-03],
    [9.25781e-01, 1.65272e-03, 1.45912e-03, 2.44141e-03],
    [8.39844e-01, 3.17955e-03, 4.02832e-03, 5.22614e-03],
    [9.17480e-01, 3.13759e-03, 3.37982e-03, 4.72260e-03],
    [9.37012e-01, 2.63405e-03, 3.57056e-03, 4.70734e-03]], device='cuda:0',                                                                              dtype=torch.float16)

I like to get its max value and index for each row in Torch is
conf, j = x[:, 5:].max(1, keepdim=True)

How can I implement in numpy if I have 2D numpy array?

Output: You are looking for numpy.argmax


Instruction: Rowwise numpy.isin for 2D arrays

Input:
I have two arrays:
A = np.array([[3, 1], [4, 1], [1, 4]])
B = np.array([[0, 1, 5], [2, 4, 5], [2, 3, 5]])

Is it possible to use numpy.isin rowwise for 2D arrays? I want to check if A[i,j] is in B[i] and return this result into C[i,j]. At the end I would get the following C:
np.array([[False, True], [True, False], [False, False]])

It would be great, if this is also doable with the == operator, then I could use it also with PyTorch.
Edit:
I also considered check for identical rows in different numpy arrays. The question is somehow similar, but I am not able to apply its solutions to this slightly different problem.

Output:
Not sure that my code solves your problem perfectly. please run it on more test cases to confirm. but i would do smth like i've done in the following code taking advantage of numpys vector outer operations ability (similar to vector outer product). If it works as intended it should work with pytorch as well.
import numpy as np
A = np.array([[3, 1], [4, 1], [1, 4]])
B = np.array([[0, 1, 5], [2, 4, 5], [2, 3, 5]])

AA = A.reshape(3, 2, 1)
BB = B.reshape(3, 1, 3)
(AA == BB).sum(-1).astype(bool)

output:
array([[False,  True],
       [ True, False],
       [False, False]])


Instruction: Trouble understanding behaviour of modified VGG16 forward method (Pytorch)

Input:
I have modified VGG16 in pytorch to insert things like BN and dropout within the feature extractor. By chance I now noticed something strange when I changed the definition of the forward method from:
def forward(self, x):
    x = self.model(x)
    return x

to:
def forward(self, x):
    x = self.model.features(x)
    x = self.model.avgpool(x)
    x = self.model.classifier(x)
    return x

In the second method I am now getting an error that the sizes of the matrices don't match
(mat1 dim 1 must match mat2 dim 0)
Below is the entire code For the editted version of VGG that I've been using.
class Vgg(nn.Module):

    def __init__(self, n_classes, bias= None, dropout = 0.3):

        super().__init__()

        self.model = models.vgg16()
        #self.bn64 = nn.BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        #self.bn128 = nn.BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        #self.bn256 = nn.BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        #self.bn512 = nn.BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

        # change to allow 4 channels input
        self.model.features[0] = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))

        # remove/edit some of the first layers to make it more similar to Resnet
        del self.model.features[2]
        del self.model.features[2]
        del self.model.features[-1]
        self.model.features[2] = nn.MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)

        # add dropout
        for m in self.model.modules():
            if isinstance(m, nn.Dropout):
                m.p = dropout
            else:
                pass

        self.dropout = nn.Dropout(p=dropout)
        self.r = nn.ReLU(inplace=True)

        modules = nn.Sequential(*[self.model.features[0],
                                  #self.bn64,
                                  self.model.features[1:3],
                                  #self.bn64,
                                  self.model.features[3:5],
                                  #self.dropout,

                                  self.model.features[5],
                                  #self.bn128,
                                  self.model.features[6:8],
                                  #self.bn128,
                                  self.model.features[8:10],
                                  #self.dropout,

                                  self.model.features[10],
                                  #self.bn256,
                                  self.model.features[11:13],
                                  #self.bn256,
                                  self.model.features[13:15],
                                  #self.bn256,
                                  self.model.features[15:17],
                                  #self.dropout,

                                  self.model.features[17],
                                  #self.bn512,
                                  self.model.features[18:20],
                                  #self.bn512,
                                  self.model.features[20:22],
                                  #self.bn512,
                                  self.model.features[22:24],
                                  #self.dropout,

                                  self.model.features[24],
                                  #self.bn512,
                                  self.model.features[25:27],
                                  #self.bn512,
                                  self.model.features[27:29],
                                  #self.bn512,
                                  #self.dropout
                                  nn.MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
                                 ])

        self.model.features = modules

        # change the pooling layer
        self.model.avgpool = nn.AdaptiveAvgPool2d(output_size=(1, 1))

        # set output to correct num classes
        self.model.classifier = nn.Linear(in_features=512, out_features=n_classes, bias=True)

        # use predefined bias
        if bias is not None:
            assert isinstance(bias, torch.Tensor), 'bias must be tensor'
            self.model.classifier.bias = nn.Parameter(bias)

    def forward(self, x):
        x = self.model.features(x)
        x = self.model.avgpool(x)
        x = self.model.classifier(x)
        return x

I know it is very ugly and hacky looking. I have tried to re-write it but the re-written version does not work either for some reason, and I am assuming that this current issue is related to that as well. I think that the input is not being fed through the forward method the way I think it is. My assumption is that calling x = self.model(x) does not run 'x' through all of the editted layers I have made, otherwise I would get the same behaviour with the two version of the forward method above. But my question is then, what is happening when I call self.model(x) in forward? Is it running the input through the original vgg16 from pytorch? Because when I print self.model in my console it shows the changes I made to the architecture of self.model.features as well as self.model.avgpool and self.model.classifier.
Edit:
Below is the entire trace of the error. Some extra information. t is a class I made to deal with the training steps (so looping through training and validation modes, ect)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-0b3983ae9702> in <module>
     77                   print_cl_met = True
     78                   )
---> 79 model = t.run()
     80 t.save_to_json()
     81 print(np.max(np.array(t.f1_tracker)))

/home/stevea/treesat/TreeSat/TreeSat/trainers/basetrainer.py in run(self)
    342             for phase in ['training', 'testing']:
    343                 self.phase = phase
--> 344                 self.model_mode()
    345
    346                 if self.phase == 'testing':

/home/stevea/treesat/TreeSat/TreeSat/trainers/basetrainer.py in model_mode(self)
    154         if self.phase == 'training':
    155             print('*********TRAINING PHASE*********')
--> 156             self.trainModel()
    157         else:
    158             print('*********VALIDATION PHASE*********')

/home/stevea/treesat/TreeSat/TreeSat/trainers/basetrainer.py in trainModel(self)
    234         # loop through all batches to perform an epoch
    235         for loaded in self.loaders[self.phase]:
--> 236             epoch_loss = self.train_step(loaded, epoch_loss)
    237
    238         mean_loss = np.mean(np.array(epoch_loss))

/home/stevea/treesat/TreeSat/TreeSat/trainers/basetrainer.py in train_step(self, loaded, epoch_loss)
    262
    263         # process batch through network
--> 264         self.out = self.model(self.img_batch.float())
    265
    266         # get loss value

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/home/stevea/treesat/TreeSat/TreeSat/models/vgg.py in forward(self, x)
    143         x = self.model.features(x)
    144         x = self.model.avgpool(x)
--> 145         x = self.model.classifier(x)
    146         return x
    147

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: mat1 dim 1 must match mat2 dim 0

I also tried to print off the shape of x at each step of the forward method:
def forward(self, x):
        x = self.model.features(x)
        print(x.shape)
        x = self.model.avgpool(x)
        print(x.shape)
        x = self.model.classifier(x)
        print(x.shape)
        return x

And it shows me that the shapes seem to be fine as the classifier should be taking in 512 features:
torch.Size([64, 512, 4, 4])
torch.Size([64, 512, 1, 1])

Output:
I can't run your code, but I believe the issue is because linear layers expect 2d data input (as it is really a matrix multiplication), while you provide 4d input (with dims 2 and 3 of size 1).
Please try squeeze
def forward(self, x):
        x = self.model.features(x)
        x = self.model.avgpool(x)
        x = torch.squeeze(x)
        x = self.model.classifier(x)
        return x

For less hacky-looking code in the squeeze part, see torch einops.


Instruction: Colab pro does not provide more than 16 gb of ram

Input:
Today i upgraded my account to Colab pro. Although it prints the ram as:
Your runtime has 27.3 gigabytes of available RAM

    You are using a high-RAM runtime!

when I start training my model, it gives the error below.
RuntimeError: CUDA out of memory. Tried to allocate 88.00 MiB (GPU 0; 15.90 GiB total capacity; 14.75 GiB already allocated; 75.75 MiB free; 14.95 GiB reserved in total by PyTorch)

Hyperparameters of my model:
args_dict = dict(
    #data_dir="", # path for data files
    output_dir="", # path to save the checkpoints
    model_name_or_path='t5-large',
    tokenizer_name_or_path='t5-large',
    max_seq_length=600,
    learning_rate=3e-4,
    weight_decay=0.0,
    adam_epsilon=1e-8,
    warmup_steps=0,
    train_batch_size=4,
    eval_batch_size=4,
    num_train_epochs=2,
    gradient_accumulation_steps=16,
    n_gpu=1,
    early_stop_callback=False,
    fp_16=True, # if you want to enable 16-bit training then install apex and set this to true
    opt_level='O1', # you can find out more on optimisation levels here https://nvidia.github.io/apex/amp.html#opt-levels-and-properties
    max_grad_norm=1.0, # if you enable 16-bit training then set this to a sensible value, 0.5 is a good default
    seed=42,
)

Colab pro not providing all ram. My code only works if train_batch_size = 1. What causes this? Any ideas?
Note: I get the same error when I run the code in Kaggle (16Gb). So, what I get with colab pro?

Output:
Looking at your error, the 16 GB are referring to the graphics card, not the ram.
As far as I know, using colab-pro enables you to use a graphics card with up to 16GB of VRAM.
You can check the VRAM amount by running the following code.
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
  print('Select the Runtime > "Change runtime type" menu to enable a GPU accelerator, ')
  print('and then re-execute this cell.')
else:
  print(gpu_info)

Maybe you use a smaller batch size than 4?


Instruction: Two models ( with similar hyperparameters) loaded from same checkpoint giving different training results in PYTORCH

Input:
I trained a model (Lenet-5) for 10 epochs and saved the model.
loaded into 2 models ‘new_model’, ‘new_model2’
below is the colab link
https://colab.research.google.com/drive/1qQhyTWNzCgMYn8t0ZtIZilLgk4JptbJG?usp=sharing
trained the new models for 5 epochs, but ended up with different train and test accuracies for each epoch, in spite of loading from same model and setting reproducibility settings.
When I continue training the original model for 5 more epochs, the results are also different from the training results of 2 new models.
Is it possible that the test and train accuracies of original model (15 epochs), 2 new models (5 epochs after loading from the checkpoint) will be same?
(After loaded checkpoint I'm getting same test accuracy for all 3 models, but results are deviating on further training of each of models.)

Output:
You should reset all the seeds to a fixed value right before launching your experiments every time you launch an experiment. In short, this should be the order:

Set Seed
Train new model #1
Set Seed (again) to the same value.
Train new model #2

Reusing some of your code, we could define a function to set the seed, that should be called with the same value in steps 1 and 3:
def set_seed(s):
   th.manual_seed(s)
   th.cuda.manual_seed_all(s)
   th.backends.cudnn.deterministic = True
   th.backends.cudnn.benchmark = False
   np.random.seed(s)
   random.seed(s)
   os.environ['PYTHONHASHSEED'] = str(s)


Instruction: Multiply a 3d tensor with a 2d matrix using torch.matmul

Input:
I have two tensors in PyTorch, z is a 3d tensor of shape (n_samples, n_features, n_views) in which n_samples is the number of samples in the dataset, n_features is the number of features for each sample, and n_views is the number of different views that describe the same (n_samples, n_features) feature matrix, but with other values.
I have another 2d tensor b, of shape (n_samples, n_views), which purpose is to rescale all the features of the samples across the different views. In other words, it encapsulates the importance of the features of each view for the same sample.
For example:
import torch
z = torch.Tensor(([[2,3], [1,1], [4,5]],
                  [[2,2], [1,2], [7,7]],
                  [[2,3], [1,1], [4,5]],
                  [[2,3], [1,1], [4,5]]))

b = torch.Tensor(([1, 0],
                  [0, 1],
                  [0.2, 0.8],
                  [0.5, 0.5]))
print(z.shape, b.shape)

>>>torch.Size([4, 3, 2]) torch.Size([4, 2])

I want to obtain a third tensor r of shape (n_samples, n_features) as a result of operations between z and b.
One possible solution is:
b = b.unsqueeze(1)
r = z * b
r = torch.sum(r, dim=-1)
print(r, r.shape)
>>>tensor([[2.0000, 1.0000, 4.0000],
        [2.0000, 2.0000, 7.0000],
        [2.8000, 1.0000, 4.8000],
        [2.5000, 1.0000, 4.5000]]) torch.Size([4, 3])

Is it possible to achieve that same result using torch.matmul()?. I've tried many times to permute the dimensions of the two vector, but to no avail.

Output:
Yes that's possible. If you have mutiple batch dimensions in both operatns, you can use the broadcasting. In this case the last two dimensions of each operand are interpreted as a matrix size. (I recommend looking it up in the documentation.)
So you need an additional dimension for your vectors b, to make them a n x 1 "matrix" (column vector):
# original implementation
b1 = b.unsqueeze(1)
r1 = z * b1
r1 = torch.sum(r1, dim=-1)
print(r1.shape)

# using torch.matmul
r2 = torch.matmul(z, b.unsqueeze(2))[...,0]
print(r2.shape)
print((r1-r2).abs().sum())  # should be zero if we do the same operation

Alternatively, torch.einsum also makes this very straightforward.
# using torch.einsum
r3 = torch.einsum('ijk,ik->ij', z, b)
print((r1-r3).abs().sum())  # should be zero if we do the same operation

einsum is a very powerful operation that can do a lot of things: you can permute tensor dimensions, sum along them, or perform scalar products, all with or without broadcasting. It is derived from the Einstein summation convention mostly used in physics. The rough idea is that you give every dimension of your operans a name, and then, using these names define what the output should look like. I think it is best to read the documentation. In our case we have a 4 x 3 x 2 tensor as well as a 4 x 2 tensor. So the let's call the dimensions of the first tensor ijk. Here i and k should be considered the same as the dimensions of the second tensor, so this one can be described as ik. Finally the output should have clearly be ij (it mus be a 4 x 3 tensor). From this "signature" ijk, ik -> ij it is clear that the dimension i is preserved, and the dimensions k must be "summe/multiplied" away (scalar product).


Instruction: Is it possible to combine 2 neural networks?

Input:
I have a NET like (exemple from here)
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
net = Net()

and another net like (exemple from here)
class binaryClassification(nn.Module):
    def __init__(self):
        super(binaryClassification, self).__init__()
        # Number of input features is 12.
        self.layer_1 = nn.Linear(12, 64)
        self.layer_2 = nn.Linear(64, 64)
        self.layer_out = nn.Linear(64, 1)
        self.relu = nn.ReLU()
        self.dropout = nn.Dropout(p=0.1)
        self.batchnorm1 = nn.BatchNorm1d(64)
        self.batchnorm2 = nn.BatchNorm1d(64)

    def forward(self, inputs):
        x = self.relu(self.layer_1(inputs))
        x = self.batchnorm1(x)
        x = self.relu(self.layer_2(x))
        x = self.batchnorm2(x)
        x = self.dropout(x)
        x = self.layer_out(x)
        return x

I'd like to change, for exemple "self.fc2 = nn.Linear(120, 84)" in order to have 121 inputs, where the 121th is the x (output) of the binaryClassification network.
The idea is: I'd like to use in the same time, CNN network, and not-CNN network, to train both, with influence one on the other.
Is it possible? How can I perform that? (Keras or Pytorch examples are both ok).
Or maybe the idea is crazy and there is easier way to mix data and image as input of an unique network?

Output:
It is a perfectly valid approach, you are taking two different input data sources, processing them and combining the result to solve a common goal (in this case it seems like a 10-class image classification). You can define the input to your Net network to be a tuple of the image you need for the original Net and the features 12-value vector for your BinaryClassificator. An example code would be:
import torch
import torch.nn as nn

class binaryClassification(nn.Module):
   #> ...same as above

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.binClas = binaryClassification()
        self.fc2 = nn.Linear(121, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, inputs):
        x, features = inputs    # split tuple
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        # Concatenate with BinaryClassification
        x = torch.cat([F.relu(self.fc1(x)), self.binClas(features)])
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
net = Net()

However! Be careful about training them together, it is hard to balance both branches in the network to make them learn. I would recommend you to train them separately for a while before plugging them together (generally speaking, the hyperparameters of one part of the network will probably not be optimal for the other). To do this, you could freeze one part of the network while training the other, and viceversa. (check this link to see how to freeze parts of a torch nn)


Instruction: Convert Pytorch Float Model into Double

Input:
I'm trying to solve cartpole from Gym. It turns out that the states are in double floating point precision whereas the pytorch by default creates model in single floating point precision.
class QNetworkMLP(Module):
    def __init__(self,state_dim,num_actions):
        super(QNetworkMLP,self).__init__()
        self.l1 = Linear(state_dim,64)
        self.l2 = Linear(64,64)
        self.l3 = Linear(64,128)
        self.l4 = Linear(128,num_actions)
        self.relu = ReLU()
        self.lrelu = LeakyReLU()

    def forward(self,x) :
        x = self.lrelu(self.l1(x))
        x = self.lrelu(self.l2(x))
        x = self.lrelu(self.l3(x))
        x = self.l4(x)
        return x

I tried to convert it via
model = QNetworkMLP(4,2).double()

But it still doesn't work I get the same error.
File ".\agent.py", line 117, in update_online_network
    predicted_Qval = self.online_network(states_batch).gather(1,actions_batch)
  File "C:\Users\27abh\anaconda3\envs\gym\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\27abh\Desktop\OpenAI Gym\Cartpole\agent_model.py", line 16, in forward
    x = self.lrelu(self.l1(x))
  File "C:\Users\27abh\anaconda3\envs\gym\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\27abh\anaconda3\envs\gym\lib\site-packages\torch\nn\modules\linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Users\27abh\anaconda3\envs\gym\lib\site-packages\torch\nn\functional.py", line 1674, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat1' in call to _th_addmm

Output:
Can you try this after initializing your model:
 model.to(torch.double)

Also be sure to check if all your inputs to the model are of torch.double data type


Instruction: unable to mmap 1024 bytes - Cannot allocate memory - even though there is more than enough ram

Input:
I'm currently working on a seminar paper on nlp, summarization of sourcecode function documentation. I've therefore created my own dataset with ca. 64000 samples (37453 is the size of the training dataset) and I want to fine tune the BART model. I use for this the package simpletransformers which is based on the huggingface package. My dataset is a pandas dataframe.
An example of my dataset:

My code:
train_df = pd.read_csv(train_path, index_col=0)
train_df.rename(columns={'text':'input_text', 'summary':'target_text'}, inplace=True)

# Logging
logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)

# Hyperparameters
model_args = Seq2SeqArgs()

model_args.num_train_epochs = 10
# bart-base = 32, bart-large-cnn = 16
model_args.train_batch_size = 16
# model_args.no_save = True
# model_args.evaluate_generated_text = True
model_args.evaluate_during_training = True
model_args.evaluate_during_training_verbose = True

model_args.overwrite_output_dir = True
model_args.save_model_every_epoch = False
model_args.save_eval_checkpoints = False
model_args.save_optimizer_and_scheduler = False
model_args.save_steps = -1
best_model_dir = 'drive/MyDrive/outputs/bart-large-cnn/best_model/'
model_args.best_model_dir = best_model_dir

# Initialize model
model = Seq2SeqModel(
    encoder_decoder_type="bart",
    encoder_decoder_name="facebook/bart-base",
    args=model_args,
    use_cuda=True,
)

# Train the model
model.train_model(
    train_df,
    # eval_data=eval_df,
    # matches=count_matches,
)

everything is fine so far BUT I get this error when I start the training.

Here the error from a run I did on a colab notebook:
Exception in thread Thread-14:
Traceback (most recent call last):
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 470, in _handle_results
    task = get()
  File "/usr/lib/python3.7/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/local/lib/python3.7/dist-packages/torch/multiprocessing/reductions.py", line 287, in rebuild_storage_fd
    storage = cls._new_shared_fd(fd, size)
RuntimeError: unable to mmap 1024 bytes from file <filename not specified>: Cannot allocate memory (12)

One would think that I simply have not enough memory but this were my System Monitor ca. 3 sec. after the error:

and this was the lowest my available or free memory get in the time between starting the training and getting the error:

After a lot of tuning I found out that for some reason every thing works fine when I train the model only with a dataset of the size of max. 21000. I doesn't madder if I train the "base" version or the "large-cnn" version of the BART model. I just depends on size of my dataset. The error always occurs in the "Creating features from dataset file at cache_dir/" time.
So what have I already tried:

I added a lot of swap memory (as you can see in the screenshot of my System Monitor)

reduced the numbers of workers to 1

I increased the hard- as well as the softmax of my systems open files limit (-n) to 86000

I also tried to train the model in a google colab notebook but I had the same issue; if the dataset size gets over ca. 21000 the training fails. Even after I doubled the memory of my colab session but still keeping the datset size just a little bit over the 21000 limit.
Desktop:
transformers 4.6.0
simpletransformers 0.61.4
ubuntu 20.04.2 LTS
After trying to solve this by myself for literally weeks I would me more than happy if anyone of you guys have an idea how I can solve this :)
(I am aware of this post mmap returns can not allocate memory, even though there is enough even though there is enough unfortunately it couldn't solve my problem. My vm.max_map_count is at 860000)

Output:
So I just found a simple workaround.
You can just set use_multiprocessing of the model to False:
model_args.use_multiprocessing = False

Now I can run with my whole dataset.