Instruction: How to extract and use BERT encodings of sentences for Text similarity among sentences. (PyTorch/Tensorflow)

Input:
I want to make a text similarity model which I tend to use for FAQ finding and other methods to get the most related text. I want to use the highly optimised BERT model for this NLP task .I tend to use the the encodings of all the sentences to get a similarity matrix using the cosine_similarity and return results.
In the hypothetical conditions, if I have two sentences as hello world and hello hello world then I am assuming the BRT would give me something like [0.2,0.3,0], (0 for padding) and [0.2,0.2,0.3] and I can pass these two inside the sklearn's cosine_similarity.
How am I supposed to extract the embeddings the sentences to use them in the model? I found somewhere that it can be extracted like:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
Using Tensorflow:

import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Is this the right way? Because I read somewhere that there are different type of embeddings that BERT offers.
ALSO please do suggest any other method to find the text similarity

Output:
When you want to compare the embeddings of sentences the recommended way to do this with BERT is to use the value of the CLS token. This corresponds to the first token of the output (after the batch dimension).
last_hidden_states = outputs[0]
cls_embedding = last_hidden_states[0][0]

This will give you one embedding for the entire sentence. As you will have the same size embedding for each sentence you can then easily compute the cosine similarity.
If you do not get satisfactory results using the CLS token you can also try averaging the output embedding for each word in the sentence.


Instruction: Consecutive calls to stack throws RuntimeError: stack expects each tensor to be equal size

Input:
I think both torch.cat and torch.stack cannot fully satisfy my requirement.
Initially, I define an empty tensor. Then I want to append a 1d-tensor to it for multiple times.
x = torch.tensor([]).type(torch.DoubleTensor)
y = torch.tensor([ 0.3981,  0.6952, -1.2320]).type(torch.DoubleTensor)
x = torch.stack([x,y])

This will throw an error:

RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

So I have to initialise x as torch.tensor([0,0,0]) (but can this be avoided?)
x = torch.tensor([0,0,0]).type(torch.DoubleTensor)
y = torch.tensor([ 0.3981,  0.6952, -1.2320]).type(torch.DoubleTensor)
x = torch.stack([x,y])  # this is okay
x = torch.stack([x,y])  # <--- this got error again

But when I run x = torch.stack([x,y]) the second time, I got this error:

RuntimeError: stack expects each tensor to be equal size, but got [2, 3] at entry 0 and [3] at entry 1

What I want to achieve is being able to append a 1d-tensor multiple times (the added 1d-tensor is different at each time, here I use the same one for simplicity)**:
tensor([[ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        ...
        [ 0.3981,  0.6952, -1.2320]], dtype=torch.float64)

How to achieve this?

Output:
From the documentation of torch.cat "All tensors must either have the same shape (except in the concatenating dimension) or be empty".  So, the easiest solution is to add one more dimension (size 1) to the tensor you want to add.  Then, you will have tensors of size (n, whatever) and (1, whatever) which will be concatenated along the 0th dimension, meeting the requirements for torch.cat.
Code:
x = torch.empty(size=(0,3))
y = torch.tensor([ 0.3981,  0.6952, -1.2320])
for n in range(5):
    y1 = y.unsqueeze(dim=0) # same as y but with shape (1,3)
    x = torch.cat([x,y1], dim=0)
print(x)

Output:
tensor([[ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320],
        [ 0.3981,  0.6952, -1.2320]])


Instruction: MSE with predictions not expected

Input:
Here is a regression model where I attempt to predict the y values (outputs) from x values (inputs) . Each class is given a different mean and normalized with l2 normalization:
x_values = sklearn.preprocessing.normalize(x_values, norm="l2")

This may appear as a classification problem attempting to be solved using regression. I'm attempting to understand multiclass regression in PyTorch, as the PyTorch doc gives the following example which suggests multiclass regression is possible:
>>> loss = nn.MSELoss()
>>> input = torch.randn(3, 5, requires_grad=True)
>>> target = torch.randn(3, 5)
>>> output = loss(input, target)
>>> output.backward()

src: https://pytorch.org/docs/master/generated/torch.nn.MSELoss.html
Entire code:
% reset - f

from datetime import datetime
from sklearn import metrics
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import numpy as np
import matplotlib.pyplot as plt
import torch.utils.data as data_utils
import torch.nn as nn
import torch.nn.functional as F
import random
from torch.autograd import Variable
import pandas as pd
import unittest
import time
from collections import Counter
import sklearn

x_values = []
y_values = []
input_size = 17
lr = .1

# Class1
mu, sigma = 0, 0.1  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class2
mu, sigma = 5, 0.5  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class3
mu, sigma = 10, 1.0  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class4
mu, sigma = 15, 1.5  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

# Class5
mu, sigma = 20, 2.0  # mean and standard deviation
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))
x_values.append(np.random.normal(mu, sigma, input_size))

x_values = sklearn.preprocessing.normalize(x_values, norm="l2")

y_values.append(0)
y_values.append(0)
y_values.append(0)

y_values.append(1)
y_values.append(1)
y_values.append(1)

y_values.append(2)
y_values.append(2)
y_values.append(2)

y_values.append(3)
y_values.append(3)
y_values.append(3)

y_values.append(4)
y_values.append(4)
y_values.append(4)

num_classes = len(y_values)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.criterion = torch.nn.MSELoss()
        self.model = torch.nn.Sequential(
            torch.nn.Linear(input_size, 100),
            torch.nn.ReLU(),
            torch.nn.Linear(100, 50),
            torch.nn.ReLU(),
            torch.nn.Linear(50, num_classes)
            #                         torch.nn.ReLU()
        )
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, state, action):
        y_pred = self.model(torch.Tensor(state))
        loss = self.criterion(y_pred, Variable(torch.Tensor(action)))
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss

    def predict(self, s):
        with torch.no_grad():
            return self.model(torch.Tensor(s))

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

model = NeuralNet()
model.apply(weights_init)
print('len(states)', len(x_values))

i = 0

for s in range(7000):

    if i == 15:
        i = 0
    x = x_values[i]
    loss_value = model.update(x, y_values)

    if s % 1000 == 0:
        print('loss_value', loss_value)

    i = i + 1

Predicting on the x_values:
[torch.argmax(model.predict(s)) for s in x_values]

returns:
[tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14),
 tensor(14)]

As I have defined classes with difference means and the final loss value is low (4.7370e-15) I expect the predicted values to be closer to:
[tensor(0)
 tensor(0),
 tensor(0),
 tensor(1),
 tensor(1),
 tensor(1),
 tensor(2),
 tensor(2),
 tensor(2),
 tensor(3),
 tensor(3),
 tensor(3),
 tensor(4),
 tensor(4),
 tensor(4)]

What are the predicted outputs not closed to my expectation?
Have I set up the model incorrectly?

Output:
Are you sure you have a regression problem? When we talk about the output being a specific class, a classification problem is usually used regardless of the input.
Another concept is that you are trying to represent some sort of ordinal categorical variable.
You can pose the problem in two ways:
1 - Consider that you have a classification problem.
class NeuralNet(nn.Module):
    class ExpActivation(nn.Module):
        def __init__(self):
            super().__init__()

        def forward(self, x):
            return torch.exp(x)

    class BoundedPositiveNumber(nn.Module):
        def __init__(self):
            super().__init__()
            self.max_value = 4

        def forward(self, x):
            return self.max_value * torch.sigmoid(x)

    def __init__(self):
        super(NeuralNet, self).__init__()
        self.criterion = torch.nn.CrossEntropyLoss()
        self.model = torch.nn.Sequential(
            torch.nn.Linear(input_size, 100),
            torch.nn.ReLU(),
            torch.nn.Linear(100, 50),
            torch.nn.ReLU(),
            torch.nn.Linear(50, num_classes),
            torch.nn.Softmax()
        )
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, state, action):
        y_pred = self.model(state)
        loss = self.criterion(y_pred, action)
        # print(torch.argmax(y_pred, axis=-1), action, loss)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss

    def predict(self, s):
        with torch.no_grad():
            return self.model(torch.Tensor(s))

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

model = NeuralNet()
#model.apply(weights_init)
print('len(states)', len(x_values))

i = 0

x_values = torch.from_numpy(x_values).float()
y_values = torch.from_numpy(np.array(y_values)).long()

for s in range(700000):

    if i == 15:
        i = 0
    x = x_values[i:i+1]
    y = y_values[i:i+1]
    loss_value = model.update(x_values, y_values)

    if s % 1000 == 0:
        print('loss_value', loss_value)

    i = i + 1

# Example
f = model.model(x)
proba = torch.softmax(f) # obtain the probability distribution
np.argmax(proba.cpu().numpy()) # or np.argmax(f.cpu().numpy()), in this case are equivalent

2 - Consider that you want to get that "number" as a regression and
not as a class. You are not looking for a probability distribution but the value directly. In this case, it is not very common but if you want only positive values it is interesting to use the exponential as activation. So you condense -inf, inf to 0, inf.
class NeuralNet(nn.Module):
    class ExpActivation(nn.Module):
        def __init__(self):
            super().__init__()

        def forward(self, x):
            return torch.exp(x)

    class AcotatedPositiveNumber(nn.Module):
        def __init__(self):
            super().__init__()
            self.max_value = 4

        def forward(self, x):
            return self.max_value * torch.sigmoid(x)

    def __init__(self):
        super(NeuralNet, self).__init__()
        self.criterion = torch.nn.MSELoss()
        self.model = torch.nn.Sequential(
            torch.nn.Linear(input_size, 100),
            torch.nn.ReLU(),
            torch.nn.Linear(100, 50),
            torch.nn.ReLU(),
            torch.nn.Linear(50, 1),
            NeuralNet.ExpActivation()
        )
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, state, action):
        y_pred = self.model(state)
        loss = self.criterion(y_pred, action.unsqueeze(-1))
        # print(torch.round(y_pred.squeeze()).long(), action, loss)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        return loss

    def predict(self, s):
        with torch.no_grad():
            return self.model(torch.Tensor(s))

def weights_init(m):
    if type(m) == nn.Linear:
        m.weight.data.normal_(0.0, 1)

model = NeuralNet()
#model.apply(weights_init)
print('len(states)', len(x_values))

i = 0

x_values = torch.from_numpy(x_values).float()
y_values = torch.from_numpy(np.array(y_values)).float()

for s in range(700000):

    if i == 15:
        i = 0
    x = x_values[i:i+1]
    y = y_values[i:i+1]
    loss_value = model.update(x_values, y_values)

    if s % 1000 == 0:
        print('loss_value', loss_value)

    i = i + 1

# Example
regresion_value = model.model(x)
regresion_value.cpu().numpy()


Instruction: How to avoid iterating over Dataloader while resuming training in Huggingface Trainer class?

Input:
I'm currently using Huggingface's Trainer class to train Distillbert for a regression problem using a custom loss function. I'm using their checkpoints to resume training due to the ephemeral nature of compute / unexpected errors.
The issue I'm facing is that each time I resume training from a checkpoint as per their Trainer class via the model_path in the Trainer.train() method, I noticed that the class iterates over the dataloader until it reaches the iteration count as saved in the checkpoint (see the lines from the Trainer class that match the issue).
This might usually not be a issue, but due to the nature of my dataloader's collate function and the size of the dataset, iterating for such a duration without any training is pretty expensive and slows down the overall training.
I planned on utilizing a custom sampler class something along the lines of this with a parameter to resume the indices from a given location but that too seems quite the hack for the given problem.
What could be an alternative that I could try to save on this wasted compute cycles?

Output:
Well it looks like huggingface has provided a solution to this via the use of ignore_data_skip argument in the TrainingArguments.
Although you would have to be careful using this flag. It will essentially be as if you're starting a new epoch from step 0. But you'd be moving the optimizer / model state to whatever it was from the resume point.


Instruction: Correct way to register a parameter for model in Pytorch

Input:
I tried to define a simple model in Pytorch. The model computes negative log prob for a gaussian distribution:
import torch
import torch.nn as nn

class GaussianModel(nn.Module):

    def __init__(self):
        super(GaussianModel, self).__init__()

        self.register_parameter('mean', nn.Parameter(torch.zeros(1),
                                                     requires_grad=True))

        self.pdf = torch.distributions.Normal(self.state_dict()['mean'],
                                              torch.tensor([1.0]))
    def forward(self, x):
        return -self.pdf.log_prob(x)

model = GaussianModel()

Then I tried to optimize the mean parameter:
optimizer = torch.optim.SGD(model.parameters(), lr=0.002)
for _ in range(5):
  optimizer.zero_grad()
  nll = model(torch.tensor([3.0], requires_grad=True))
  nll.backward()
  optimizer.step()
  print('mean : ', model.state_dict()['mean'],
                 ' - Negative Loglikelihood : ', nll.item())

But it seems the gradient is zero and mean does not change:
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785
mean :  tensor([0.])  - Negative Loglikelihood :  5.418938636779785

Did I register and use the mean parameter correctly? can autograd compute the gradient for torch.distributions.Normal.log_prob or I should implement the backward() for the model?

Output:
You're over complicating registering your parameter. You can just assign a new self.mean attribute to be an nn.Parameter then use it like a tensor for the most part.
nn.Module overrides the __setattr__ method which is called every time you assign a new class attribute. One of the things it does is check to see if you assigned an nn.Parameter type, and if so, it adds it to the modules dictionary of registered parameters.
Because of this, the easiest way to register your parameter is as follows:
import torch
import torch.nn as nn

class GaussianModel(nn.Module):
    def __init__(self):
        super(GaussianModel, self).__init__()
        self.mean = nn.Parameter(torch.zeros(1))
        self.pdf = torch.distributions.Normal(self.mean, torch.tensor([1.0]))

    def forward(self, x):
        return -self.pdf.log_prob(x)


Instruction: Pytorch: all-but-one summation?

Input:
I am in the process of moving some message passing code from Numpy to Pytorch. I am unsure of how to do this single step of a much larger algorithm. Below is the simplest explanation of the step.
Given the following:
index = [[2,0,1], [2,2,0]]
value = [[0.1, 1.2, 2.3], [3.4, 4.5, 5.6]]

I would like to compute the "all-but-one" sum of messages to each index.  Here is a graphical representation:

The answer I am looking for is:
ans = [[7.9, 5.6, 0], [4.6, 3.5, 1.2]]

The explanation is that, for example, index[0][0] points at node 2. The sum of all messages at node 2 is 0.1+3.4+4.5=8. However we want to exclude the message we are considering (value[0][0]=0.1), so we obtain ans[0][0]=7.9. If only a single index points at a node then the answer is 0 (eg. node 1 with ans[0][2]).
I would be happy with computing the sums for each node, and then subtracting out the individual messages.  I am aware that this can lead to loss of significance, but I believe that my use case is very well-behaved (eg. no floating point infinities).
I can also provide the minimal numpy code, but even the minimal example is a bit long. I have looked at pytorch's scatter and gather commands, but I don't think that they're appropriate here.

Output:
After working with pytorch for a while longer, and writing some code for other situations, I realized that there is a much more efficient solution that I hadn't considered. So I am pasting it here for anyone else who comes after me:
import torch

index = [[2, 0, 1], [2, 2, 0]]
value = [[0.1, 1.2, 2.3], [3.4, 4.5, 5.6]]

# convert to tensor
index_tensor = torch.tensor(index)
value_tensor = torch.tensor(value)

num_nodes = 3
totals = torch.zeros(num_nodes)
totals = totals.index_add_(0, index_tensor.flatten(), value_tensor.flatten())
result = totals[index_tensor] - value_tensor
print(result)

It uses much less memory than the scatter_add solution given by Jodag. It avoids all the for loops given in the other solutions. Hooray for much faster code!


Instruction: How to ignore and initialize Missing key(s) in state_dict

Input:
My saved state_dict does not contain all the layers that are in my model.
How can I ignore the Missing key(s) in state_dict error and initialize the remaining weights?

Output:
This can be achieved by passing strict=False to load_state_dict.
load_state_dict(state_dict, strict=False)

Documentation


Instruction: Pytorch C++ RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

Input:
I am calculating word similarity using torch::Embedding module by pretrained wordvector (glove.300d) on Ubuntu 18.04LTS PyTorch C++ (1.5.1, CUDA 10.1). I believe I have moved everything I can to the GPU, but when I execute it, it still says (full error log on the end of the question):
Expected object of device type cuda but got device type cpu for
  argument #1 'self' in call to _th_index_select
  (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)

I have checked my model initialization method in main.cpp, and it is okay if I only do initialization.
SimilarityModel simiModel(args, 400000, 300);
simiModel.to(device);

//model forward
torch::Tensor data = ids.index({Slice(i*batch_size, (i+1)*batch_size), Slice()}).to(torch::kInt64).to(device);        //take a batch
tie(score, indice) = simiModel.forward(data);   //forward and transfer score, indice to cpu for further calculation

and this is how I define SimilarityModel in Similarity.h:
class SimilarityModel : public torch::nn::Module {
    public:
        int64_t topk;       // num of top words;
        Dictionary dict;
        int64_t vocab_size;
        int64_t embedding_dim;
        torch::nn::Embedding embedding{nullptr};
        vector<vector<float> > vec_embed;

        SimilarityModel(unordered_map<string, string> args, int64_t vocab_size, int64_t embed_dim);
        tuple<torch::Tensor, torch::Tensor> forward(torch::Tensor x);
};

At the same time I have done embedding initialization in SimilarityModel function in Similarity.cpp:
SimilarityModel::SimilarityModel(unordered_map<string, string> args, int64_t vocab_size, int64_t embed_dim)
        :embedding(vocab_size, embed_dim) {      //Embedding initialize

    this->topk = stoi(args["topk"]);
    vector<vector<float> > pre_embed;
    tie(pre_embed, dict) = loadwordvec(args);       //load pretrained wordvec from txt file

    this->vocab_size = int64_t(dict.size());
    this->embedding_dim = int64_t(pre_embed[0].size());
    this->vec_embed = pre_embed;
    this->dict = dict;

    vector<float> temp_embed;
    for(const auto& i : pre_embed)      //faltten to 1-d
        for(const auto& j : i)
            temp_embed.push_back(j);
    torch::Tensor data = torch::from_blob(temp_embed.data(), {this->vocab_size, this->embedding_dim}, torch::TensorOptions().dtype(torch::kFloat32)).clone();   //vector to tensor
    register_module("embedding", embedding);
    this->embedding = embedding.from_pretrained(data, torch::nn::EmbeddingFromPretrainedOptions().freeze(true));
}

and forward function in Similarity.cpp:
tuple<torch::Tensor, torch::Tensor> SimilarityModel::forward(torch::Tensor x) {

    auto cuda_available = torch::cuda::is_available();      //copy to gpu
    torch::Device device(cuda_available ? torch::kCUDA : torch::kCPU);

    torch::Tensor wordvec;
    wordvec = this->embedding->forward(x).to(device);      //python:embedding(x)
    torch::Tensor similarity_score = wordvec.matmul(this->embedding->weight.transpose(0, 1)).to(device);
    torch::Tensor score, indice;
    tie(score, indice) = similarity_score.topk(this->topk, -1, true, true);        //Tensor.topk(int64_t k, int64_t dim, bool largest = true, bool sorted = true)

    score = score.to(device);
    indice = indice.to(device);
    score.slice(1, 1, score.size(1));       //Tensor.slice(int64_t dim, int64_t start, int64_t end, int64_t step)
    indice.slice(1, 1, indice.size(1));
    return {score.cpu(), indice.cpu()};   //transfer to cpu for further calculation
}

As for intermediate variables in forward() have been put to GPU too. However, I totally have no idea which one is left in CPU, and the error log does not help so much. I have tried the method in Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select to do SimilarityModel().to(device), but that does not work. I am still having a hard time reading this error log and would like some instructions on how to debug such questions.
Error log:
terminate called after throwing an instance of 'c10::Error'
  what():  Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select (checked_dense_tensor_unwrap at /pytorch/aten/src/ATen/Utils.h:72)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7fb566a27536 in /home/switchsyj/Downloads/libtorch/lib/libc10.so)
frame #1: <unknown function> + 0x101a80b (0x7fb520fa380b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #2: <unknown function> + 0x105009c (0x7fb520fd909c in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #3: <unknown function> + 0xf9d76b (0x7fb520f2676b in /home/switchsyj/Downloads/libtorch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x10c44e3 (0x7fb558d224e3 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #5: at::native::embedding(at::Tensor const&, at::Tensor const&, long, bool, bool) + 0x2e2 (0x7fb558870712 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0x114ef9d (0x7fb558dacf9d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x2bfe42f (0x7fb55a85c42f in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #9: <unknown function> + 0x1187b4d (0x7fb558de5b4d in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #10: <unknown function> + 0x32b63a9 (0x7fb55af143a9 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #11: torch::nn::EmbeddingImpl::forward(at::Tensor const&) + 0x71 (0x7fb55af127b1 in /home/switchsyj/Downloads/libtorch/lib/libtorch_cpu.so)
frame #12: SimilarityModel::forward(at::Tensor) + 0xa9 (0x55c96b8e5793 in ./demo)
frame #13: main + 0xaba (0x55c96b8bfe5c in ./demo)
frame #14: __libc_start_main + 0xe7 (0x7fb51edf5b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #15: _start + 0x2a (0x55c96b8bd74a in ./demo)

Aborted (core dumped)

Output:
Based on the error message, one of the two following Tensors are not in the GPU when you're running SimilarityModel::forward():

this->embedding->weight
x

Given that the error points to the argument #1, I'd say that weight is the one on the CPU.
Here's the call for index.select:
Tensor embedding(const Tensor & weight, const Tensor & indices,
                 int64_t padding_idx, bool scale_grad_by_freq, bool sparse) {
  auto indices_arg = TensorArg(indices, "indices", 1);
  checkScalarType("embedding", indices_arg, kLong);

  // TODO: use tensor.index() after improving perf
  if (indices.dim() == 1) {
    return weight.index_select(0, indices);
  }

  auto size = indices.sizes().vec();
  for (auto d : weight.sizes().slice(1)) {
    size.push_back(d);
  }
  return weight.index_select(0, indices.reshape(-1)).view(size);
}

First, try to directly move the weight to the GPU. If it works, it means that when you called TORCH_MODULE(SimilarityModel), and moved the model to the device, it should have worked too. Remember that you have to change the name to SimilarityModelImpl (Name+Impl) in this case. Otherwise, it won't work as well.


Instruction: lstm:Input batch size 100 doesn't match hidden[0] batch size 1

Input:
I am trying to add an lstm layer to my previous working AI model.
On adding the model I am getting this error when training the batch to my AI.
earlier without the LSTM the error was not there and it worked fine.
Input batch size 100 doesn't match hidden[0] batch size 1.
I am using nn.LSTMCell
can anyone please help a check if I am missing some parameter to init my lstmcell so it can take batch inputs as well.
below is my code...
import os
import time
import random
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
import torch.nn.functional as F
from random import random as rndm
from torch.autograd import Variable
from collections import deque

os.chdir("C:\\Users\\granthjain\\Desktop\\startup_code")

torch.set_default_tensor_type('torch.DoubleTensor')

class ReplayBuffer(object):

  def __init__(self, max_size=1e6):
    self.storage = []
    self.max_size = max_size
    self.ptr = 0

  def add(self, transition):
    if len(self.storage) == self.max_size:
      self.storage[int(self.ptr)] = transition
    else:
      self.storage.append(transition)
    self.ptr = (self.ptr + 1) % self.max_size

  def sample(self, batch_size):

    ind = np.random.randint(0, self.ptr, size=batch_size)
    batch_states, batch_next_states, batch_actions, batch_rewards, batch_dones = [], [], [], [], []
    for i in ind:
      state, next_state, action, reward, done = self.storage[i]

      if state is None:
          continue
      elif next_state is None:
          continue
      elif action is None:
          continue
      elif reward is None:
          continue
      elif done is None:
          continue

      batch_states.append(np.array(state, copy=False))
      batch_next_states.append(np.array(next_state, copy=False))
      batch_actions.append(np.array(action, copy=False))
      batch_rewards.append(np.array(reward, copy=False))
      batch_dones.append(np.array(done, copy=False))

    return np.array(batch_states,dtype=object).astype(float), np.array(batch_next_states,dtype=object).astype(float), np.array(batch_actions,dtype=object).astype(float), np.array(batch_rewards,dtype=object).astype(float), np.array(batch_dones,dtype=object).astype(float)

class Actor(nn.Module):

  def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()
    self.lstm = nn.LSTMCell(state_dim, 256)
    self.layer_1 = nn.Linear(256, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, action_dim)
    self.hx = torch.zeros(1,256)
    self.cx = torch.zeros(1,256)
    self.max_action = max_action

  def forward(self, x):
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))
    x = F.relu(self.layer_1(self.hx))
    x = F.relu(self.layer_2(x))
    x = self.max_action * torch.tanh(self.layer_3(x))
    return x

class Critic(nn.Module):

  def __init__(self, state_dim, action_dim):
    super(Critic, self).__init__()
    # Defining the first Critic neural network
    self.lstm1 = nn.LSTMCell(state_dim + action_dim, 256)
    self.layer_1 = nn.Linear(256, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, 1)
    # Defining the second Critic neural network
    self.lstm2 = nn.LSTMCell(state_dim + action_dim, 256)
    self.layer_4 = nn.Linear(256, 400)
    self.layer_5 = nn.Linear(400, 300)
    self.layer_6 = nn.Linear(300, 1)
    self.hx1 = torch.zeros(1,256)
    self.cx1 = torch.zeros(1,256)
    self.hx2 = torch.zeros(1,256)
    self.cx2 = torch.zeros(1,256)

  def forward(self, x, u):
    xu = torch.cat([x, u], 1)
    # Forward-Propagation on the first Critic Neural Network
    self.hx1,self.cx1 = self.lstm(xu, (self.hx1, self.cx1))
    x1 = F.relu(self.layer_1(self.hx1))
    x1 = F.relu(self.layer_2(x1))
    x1 = self.layer_3(x1)
    # Forward-Propagation on the second Critic Neural Network
    self.hx2,self.cx2 = self.lstm(xu, (self.hx2, self.cx2))
    x2 = F.relu(self.layer_4(self.hx2))
    x2 = F.relu(self.layer_5(x2))
    x2 = self.layer_6(x2)
    return x1, x2

  def Q1(self, x, u):
    xu = torch.cat([x, u], 1)
    self.hx1,self.cx1 = self.lstm(xu, (self.hx1, self.cx1))
    x1 = F.relu(self.layer_1(self.hx1))
    x1 = F.relu(self.layer_2(x1))
    x1 = self.layer_3(x1)
    return x1

# Selecting the device (CPU or GPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Building the whole Training Process into a class

class TD3(object):

  def __init__(self, state_dim, action_dim, max_action):
    self.actor = Actor(state_dim, action_dim, max_action).to(device)
    self.actor_target = Actor(state_dim, action_dim, max_action).to(device)
    self.actor_target.load_state_dict(self.actor.state_dict())
    self.actor_optimizer = torch.optim.Adam(self.actor.parameters())
    self.critic = Critic(state_dim, action_dim).to(device)
    self.critic_target = Critic(state_dim, action_dim).to(device)
    self.critic_target.load_state_dict(self.critic.state_dict())
    self.critic_optimizer = torch.optim.Adam(self.critic.parameters())
    self.max_action = max_action

  def reset_hxcx(self):
    self.actor.cx = torch.zeros(1,256)
    self.actor.hx = torch.zeros(1,256)
    self.actor_target.cx = torch.zeros(1,256)
    self.actor_target.hx = torch.zeros(1,256)
    self.critic.cx1 = torch.zeros(1,256)
    self.critic.cx2 = torch.zeros(1,256)
    self.critic.hx1 = torch.zeros(1,256)
    self.critic.hx2 = torch.zeros(1,256)
    self.critic_target.cx1 = torch.zeros(1,256)
    self.critic_target.cx2 = torch.zeros(1,256)
    self.critic_target.hx1 = torch.zeros(1,256)
    self.critic_target.hx2 = torch.zeros(1,256)

  def select_action(self, state):
    print("state =", type(state))
    return self.actor(state).cpu().data.numpy().flatten()

  def train(self, replay_buffer, iterations, batch_size=50, discount=0.99, tau=0.005, policy_noise=0.2, noise_clip=0.5, policy_freq=2):

    for it in range(iterations):

      # Step 4: We sample a batch of transitions (s, s’, a, r) from the memory
      batch_states, batch_next_states, batch_actions, batch_rewards, batch_dones = replay_buffer.sample(batch_size)

      batch_states=batch_states.astype(float)
      batch_next_states=batch_next_states.astype(float)
      batch_actions=batch_actions.astype(float)
      batch_rewards=batch_rewards.astype(float)
      batch_dones=batch_dones.astype(float)

      state = torch.from_numpy(batch_states)
      next_state = torch.from_numpy(batch_next_states)
      action = torch.from_numpy(batch_actions)
      reward = torch.from_numpy(batch_rewards)
      done = torch.from_numpy(batch_dones)

#      print("actor cx:",self.actor.cx)
#      print("actor hx:",self.actor.hx)
#      print("actor_target cx:",self.actor_target.cx)
#      print("actor_target cx:",self.actor_target.cx)
#      print("self.critic.cx1:",self.critic.cx1)
#      print("self.critic.cx2",self.critic.cx2)
#      print("self.critic.hx1:",self.critic.hx1)
#      print("self.critic.hx2:",self.critic.hx2)
#      print("self.critic_target.cx1:",self.critic_target.cx1)
#      print("self.critic_target.hx1",self.critic_target.hx1)
#      print("self.critic_target.cx2:",self.critic_target.cx2)
#      print("self.critic_target.hx2:",self.critic_target.hx2)

      # Step 5: From the next state s’, the Actor target plays the next action a’
      next_action = self.actor_target(next_state)

      # Step 6: We add Gaussian noise to this next action a’ and we clamp it in a range of values supported by the environment
      noise = torch.Tensor(batch_actions).data.normal_(0, policy_noise).to(device)
      noise = noise.clamp(-noise_clip, noise_clip)
      next_action = (next_action + noise).clamp(-self.max_action, self.max_action)

      # Step 7: The two Critic targets take each the couple (s’, a’) as input and return two Q-values Qt1(s’,a’) and Qt2(s’,a’) as outputs
      target_Q1, target_Q2 = self.critic_target(next_state, next_action)

      # Step 8: We keep the minimum of these two Q-values: min(Qt1, Qt2)
      target_Q = torch.min(target_Q1, target_Q2).double()

      # Step 9: We get the final target of the two Critic models, which is: Qt = r + γ * min(Qt1, Qt2), where γ is the discount factor

      done = done.resize_((done.shape[0],1))

      reward = reward.resize_((reward.shape[0],1))

      target_Q = reward + ((1 - done) * discount * target_Q).detach()

      # Step 10: The two Critic models take each the couple (s, a) as input and return two Q-values Q1(s,a) and Q2(s,a) as outputs
      current_Q1, current_Q2 = self.critic(state, action)

      # Step 11: We compute the loss coming from the two Critic models: Critic Loss = MSE_Loss(Q1(s,a), Qt) + MSE_Loss(Q2(s,a), Qt)
      critic_loss = F.mse_loss(current_Q1, target_Q) + F.mse_loss(current_Q2, target_Q)

      # Step 12: We backpropagate this Critic loss and update the parameters of the two Critic models with a SGD optimizer
      self.critic_optimizer.zero_grad()
      critic_loss.backward()
      self.critic_optimizer.step()

      # Step 13: Once every two iterations, we update our Actor model by performing gradient ascent on the output of the first Critic model
      if it % policy_freq == 0:
        actor_loss = -self.critic.Q1(state, self.actor(state)).mean()
        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

        # Step 14: Still once every two iterations, we update the weights of the Actor target by polyak averaging
        for param, target_param in zip(self.actor.parameters(), self.actor_target.parameters()):
          target_param.data.copy_(tau * param.data + (1 - tau) * target_param.data)

        # Step 15: Still once every two iterations, we update the weights of the Critic target by polyak averaging
        for param, target_param in zip(self.critic.parameters(), self.critic_target.parameters()):
          target_param.data.copy_(tau * param.data + (1 - tau) * target_param.data)

  # Making a save method to save a trained model
  def save(self, filename, directory):
    torch.save(self.actor.state_dict(), '%s/%s_actor.pth' % (directory, filename))
    torch.save(self.critic.state_dict(), '%s/%s_critic.pth' % (directory, filename))

  # Making a load method to load a pre-trained model
  def load(self, filename, directory):
    self.actor.load_state_dict(torch.load('%s/%s_actor.pth' % (directory, filename)))
    self.critic.load_state_dict(torch.load('%s/%s_critic.pth' % (directory, filename)))

#set the parameters

start_timesteps = 1e3 # Number of iterations/timesteps before which the model randomly chooses an action, and after which it starts to use the policy network
eval_freq = 5e1 # How often the evaluation step is performed (after how many timesteps)
max_timesteps = 5e3 # Total number of iterations/timesteps
save_models = True # Boolean checker whether or not to save the pre-trained model
expl_noise = 0.1 # Exploration noise - STD value of exploration Gaussian noise
batch_size = 100 # Size of the batch
discount = 0.99 # Discount factor gamma, used in the calculation of the total discounted reward
tau = 0.005 # Target network update rate
policy_noise = 0.2 # STD of Gaussian noise added to the actions for the exploration purposes
noise_clip = 0.5 # Maximum value of the Gaussian noise added to the actions (policy)
policy_freq = 2 # Number of iterations to wait before the policy network (Actor model) is updated

state_dim = 3
action_dim = 3
max_action = 1
idx = 0

class env1:

    def __init__(self,state_dim,action_dim,data):
        self.state_dim = state_dim
        self.state = torch.zeros(self.state_dim)
        self.state[state_dim-1]=1000.0
        self.next_state = torch.zeros(self.state_dim)
        self.next_state[state_dim-1] = 1000.0
        self.action_dim = action_dim
        self.data = data
        self.idx = 0
        self.count = 0
        self._max_episode_steps = 200
        self.state[1] = self.data[self.idx]
        self.next_state[1] = self.data[self.idx]

    def reset(self):
        self.next_state = torch.zeros(self.state_dim)
        self.next_state[state_dim-1]=1000.0
        self.state = torch.zeros(self.state_dim)
        self.state[state_dim-1]=1000.0
        self.state[1] = self.data[self.idx]
        self.next_state[1] = self.data[self.idx]
        self.count = 0
        ch = self.state[0]
        cp = self.state[1]
        cc = self.state[2]
        st = torch.tensor([ch,cp,cc])
        return st

    def step(self,action):
        done = False
        act_t = torch.argmax(action)
        self.idx += 1
        if(act_t==0):
            num_s = int(self.state[2]/self.state[1])

            self.next_state[0] += num_s
            self.next_state[2] = self.state[2]%self.state[1]

            self.next_state[1] = self.data[self.idx]
        elif(act_t==1):
            self.next_state[1] = self.data[self.idx]
        elif(act_t==2):
            self.next_state[2] = self.state[2]+ self.state[1]*self.state[0]
            self.next_state[0] = 0
            self.next_state[1] = self.data[self.idx]

        reward = self.next_state[2] - self.state[2] + self.next_state[1]*self.next_state[0] - self.state[1]*self.state[0] -1

        self.state[0] = self.next_state[0]
        self.state[1] = self.next_state[1]
        self.state[2] = self.next_state[2]

        ch = self.state[0]
        cp = self.state[1]
        cc = self.state[2]

        st = torch.tensor([ch,cp,cc])

        self.count = (self.count + 1)%100
        if(self.count==0):
            done = True

        return st, reward, done
policy = TD3(state_dim, action_dim, max_action)

#Create the environment
data = pd.read_csv('PAGEIND.csv')
data = data['Close']
data = np.array(data).reshape(-1,1)

max_timesteps = data.shape[0]
sc = StandardScaler()
data = sc.fit_transform(data)
data = torch.DoubleTensor(data)
env = env1(state_dim,action_dim,data)

replay_buffer = ReplayBuffer()

#init training variables

total_timesteps = 0
timesteps_since_eval = 0
episode_num = 0
done = True
t0 = time.time()

# We start the main loop over 500,000 timesteps
while total_timesteps < max_timesteps:

  # If the episode is done
  if done:

    # If we are not at the very beginning, we start the training process of the model
    if total_timesteps != 0:
      print("Total Timesteps: {} Episode Num: {} Reward: {}".format(total_timesteps, episode_num, episode_reward))
      policy.train(replay_buffer, episode_timesteps, batch_size, discount, tau, policy_noise, noise_clip, policy_freq)

    # When the training step is done, we reset the state of the environment
    obs = env.reset()
    policy.reset_hxcx()

    # Set the Done to False
    done = False

    # Set rewards and episode timesteps to zero
    episode_reward = 0
    episode_timesteps = 0
    episode_num += 1

  # Before 1000 timesteps, we play random actions
  if total_timesteps < 0.8*max_timesteps:
#random action
      actn = torch.randn(action_dim)
      action = torch.zeros(action_dim)
      action[torch.argmax(actn)] = 1

  else: # After 1000 timesteps, we switch to the model
    action = policy.select_action(torch.tensor(obs))
    # If the explore_noise parameter is not 0, we add noise to the action and we clip it
    if expl_noise != 0:
      print("policy action:",action)
      actn = (action + torch.randn(action_dim))
      action = torch.zeros(action_dim)
      action[torch.argmax(actn)] = 1

  # The agent performs the action in the environment, then reaches the next state and receives the reward
  new_obs, reward, done = env.step(action)

  # We check if the episode is done
  done_bool = 0 if episode_timesteps + 1 == env._max_episode_steps else float(done)

  # We increase the total reward
  episode_reward += reward

  # We store the new transition into the Experience Replay memory (ReplayBuffer)
  replay_buffer.add((obs, new_obs, action, reward, done_bool))

  # We update the state, the episode timestep, the total timesteps, and the timesteps since the evaluation of the policy
  obs = new_obs
  episode_timesteps += 1
  total_timesteps += 1
  timesteps_since_eval += 1

and below is the error msg:
    policy.train(replay_buffer, episode_timesteps, batch_size, discount, tau, policy_noise, noise_clip, policy_freq)

  File "C:/Users/granthjain/Desktop/startup_code/td3_lstm_try.py", line 196, in train
    next_action = self.actor_target(next_state)

  File "C:\Users\granthjain\Anaconda3_1\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "C:/Users/granthjain/Desktop/startup_code/td3_lstm_try.py", line 79, in forward
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))

  File "C:\Users\granthjain\Anaconda3_1\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\Users\granthjain\Anaconda3_1\lib\site-packages\torch\nn\modules\rnn.py", line 708, in forward
    self.check_forward_hidden(input, hx[0], '[0]')

  File "C:\Users\granthjain\Anaconda3_1\lib\site-packages\torch\nn\modules\rnn.py", line 532, in check_forward_hidden
    input.size(0), hidden_label, hx.size(0)))

RuntimeError: Input batch size 100 doesn't match hidden[0] batch size 1

Output:
If you initialize your cell state and hidden state with zeros, there is no need to provide initialization at all, it will be provided for you (as a default, please see docs). However, should you decide to do it on your own, you should always take into account batch size (which might be different every iteration).
In the end, both cell and hidden state for nn.LSTMCell have shape (batch_size, hidden_size) whereas you initialize those once in the constructor with shape (1, hidden_size). You have to move initialization to forward() and with each call get batch size from x which should be just x.shape[0]

As a side note, you are using nn.LSTMCell which is just a single cell computation. Used once does not really make sense, make sure that's what works for you. Maybe just nn.LSTM instead?


Instruction: Single shot multi-dimension indexing in torch - perhaps with index_select or gather?

Input:
I am performing a multi-index re-arrangement of a matrix based upon its correspondence data. Right now, I and doing this with a pair of index_select calls, but this is very memory inefficient (n^2 in terms of memory usage), and is not exactly ideal in terms of computation efficiency either. Is there some way that I can boil my operation down into a single .gather or .index_select call?
What I essentially want to do is when given a source array of shape (I,J,K), and an array of indices of shape (I,J,2), produce a result which meets the condition:
result[i][j][:] = source[idx[i][j][0]] [idx[i][j][1]] [:]

Here's a runnable toy example of how I'm doing things right now:
source = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])
indices = torch.tensor([[[2,2],[3,1],[0,2]],[[0,2],[0,1],[0,2]],[[0,2],[0,1],[0,2]],[[0,2],[0,1],[0,2]]])

ax1 = torch.index_select(source,0,indices[:,:,0].flatten())
ax2 = torch.index_select(ax1, 1, indices[:,:,1].flatten())

result = ax2.diagonal().reshape(indices.shape(0), indices.shape(1))

This approach works for me only because my images are rather small, so they fit into memory even with the diagonalization issue. Regardless, I am producing a pretty massive amount of data that doesn't need to be. Furthermore, if K becomes large, then this issue gets worse exponentially. Perhaps I'm just missing something obvious in the documentation, but I feel like this is a problem somebody else has to have run into before that can help me out!

Output:
You already have your indices in nice form for integer array indexing so we can simply do
result = source[indices[..., 0], indices[..., 1], ...]


Instruction: pytoch RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1

Input:
I am trying to train a Actor Critic Model with LSTM in both actor and critic.
I am new to all this and can not understand why "RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)" is comming.
I am forwardPropagating from actor and getting error
below is my code and error message.I am using pytorch version 0.4.1
Can someone please help to check what is wrong with this code.
import os
import time
import random
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
import torch.nn.functional as F
from random import random as rndm
from torch.autograd import Variable
from collections import deque

torch.set_default_tensor_type('torch.DoubleTensor')

class Actor(nn.Module):

  def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()
    self.lstm = nn.LSTMCell(state_dim, 256)
    self.layer_1 = nn.Linear(256, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, action_dim)
    self.hx = torch.zeros(1,256)
    self.cx = torch.zeros(1,256)
    self.max_action = max_action

  def forward(self, x):
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))
    x = F.relu(self.layer_1(self.hx))
    x = F.relu(self.layer_2(x))
    x = self.max_action * torch.tanh(self.layer_3(x))
    return x

state_dim = 3
action_dim = 3
max_action = 1

policy = Actor(state_dim, action_dim, max_action)

s = torch.tensor([20,20,100])
next_action = policy(s)

and the error message is :
next_action = policy(s)
Traceback (most recent call last):

  File "<ipython-input-20-de717f0ad3d2>", line 1, in <module>
    next_action = policy(s)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "<ipython-input-4-aed4daf511cb>", line 14, in forward
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 704, in forward
    self.check_forward_input(input)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 523, in check_forward_input
    if input.size(1) != self.input_size:

RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

I am using pytorch version 0.4.1
Can someone please help to check what is wrong with this code.

Output:
Got it.
The input of the lstm layer has different shape.
https://pytorch.org/docs/master/generated/torch.nn.LSTMCell.html


Instruction: PyTorch: How to parallelize over multiple GPU using multiprocessing.pool

Input:
I have the following code which I am trying to parallelize over multiple GPUs in PyTorch:
import numpy as np
import torch
from torch.multiprocessing import Pool

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()

def X_power_func(j):
    X_power = X**j
    return X_power

if __name__ == '__main__':
  with Pool(processes = 2) as p:   # Parallelizing over 2 GPUs
    results = p.map(X_power_func, range(4))

results

But when I ran the code, I am getting this error:
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "<ipython-input-35-6529ab6dac60>", line 11, in X_power_func
    X_power = X**j
RuntimeError: CUDA error: initialization error
"""

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-35-6529ab6dac60> in <module>()
     14 if __name__ == '__main__':
     15   with Pool(processes = 1) as p:
---> 16     results = p.map(X_power_func, range(8))
     17
     18 results

1 frames
/usr/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    642             return self._value
    643         else:
--> 644             raise self._value
    645
    646     def _set(self, i, obj):

RuntimeError: CUDA error: initialization error

Where have I gone wrong? Any help would really be appreciated.

Output:
I think the usual approach is to call model.share_memory() once before multiprocessing, assuming you have a model which subclasses nn.Module.  For tensors, it should be X.share_memory_().  Unfortunately, I had trouble getting that to work with your code, it hangs (without errors) if X.share_memory_() is called before calling pool.map; I'm not sure if the reason is because X is a global variable which is not passed as one of the arguments in map.
What does work is this:
X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

Btw: https://github.com/pytorch/pytorch/issues/15734 mentions that "CUDA API must not be initialized before you fork" (this is likely the issue you were seeing).
Also https://github.com/pytorch/pytorch/issues/17680 if using spawn in Jupyter notebooks "the spawn method will run everything in your notebook top-level" (likely the issue I was seeing when my code was hanging, in a notebook).  In short, I couldn't get either fork or spawn to work, except using the sequence above (which doesn't use CUDA until it's in the forked process).


Instruction: numpy array type not supported?

Input:
I'm trying to copy a model I was able to follow and run through a tutorial, but this time with my own data.
I was able to convert my own MRI images to numpy arrays in the same dimensions as the arrays the tutorial data is.
I tried replacing the numpy arrays in my tutorial with my own arrays and writing my own fictional csv file for normal or abnormal (case, not case).
However when I run it, I get:
(Pytorch) C:\Users\GlaDOS\PythonProjects\dicomnpy>python train.py  -t acl -p sagittal --epochs=10 --prefix_name hue
Traceback (most recent call last):
  File "train.py", line 277, in <module>
    run(args)
  File "train.py", line 214, in run
    mrnet, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every)
  File "train.py", line 34, in train_model
    for i, (image, label, weight) in enumerate(train_loader):
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\GlaDOS\PythonProjects\dicomnpy\dataloader.py", line 56, in __getitem__
    array = self.transform(array)
  File "c:\users\glados\src\torchsample\torchsample\transforms\tensor_transforms.py", line 32, in __call__
    inputs = transform(*inputs)
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 313, in __call__
    return self.lambd(img)
  File "train.py", line 167, in <lambda>
    transforms.Lambda(lambda x: torch.Tensor(x)),
TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.

Now I'm wondering if this error means I somehow didn't convert my MRI's to the "correct" numpy array type? And if so, how do I go about changing them to the correct type?

Output:
You can redefining the variable with astype
your_array = your_array.astype(np.uint16)


Instruction: How to create a 1D sparse tensors from given list of indices and values?

Input:
I have a list of indices and values. I want to create a sparse tensor of size 30000 from this indices and values as follows.
indices = torch.LongTensor([1,3,4,6])
values = torch.FloatTensor([1,1,1,1])

So, I want to build a 30k dimensional sparse tensor in which the indices [1,3,4,6] are ones and the rest are zeros. How can I do that?
I want to store the sequences of such sparce tensors efficiently.

Output:
In general the indices tensor needs to have shape (sparse_dim, nnz) where nnz is the number of non-zero entries and sparse_dim is the number of dimensions for your sparse tensor.
In your case nnz = 4 and sparse_dim = 1 since your desired tensor is 1D. All we need to do to make your indices work is to insert a unitary dimension at the front of indices to make it shape (1, 4).
t = torch.sparse_coo_tensor(indices.unsqueeze(0), values, (30000,))

or equivalently
t = torch.sparse.FloatTensor(indices.unsqueeze(0), values, (30000,))

Keep in mind only a limited number of operations are supported on sparse tensors. To convert a tensor back to it's dense (inefficient) representation you can use the to_dense method
t_dense = t.to_dense()


Instruction: PyTorch Average Accuracy after each epoch

Input:
I am trying to calculate the accuracy of the model after the end of each epoch. After each epoch I would like to calculate the accuracy over the previous epoch. The model only seems to print the  same value as the mean test error.
model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

    global_step += 1

average_accuracy = 0
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)

Output:
IF your model is a classifier, calculating accuracy follows:
acc = (pred.max(dim=1) == target).float().mean()

Where:
pred.shape =  (batch_size, n_classes)
target.shape = (batch_size)

A similar question was asked recently: calculate accuracy for each class using CNN and pytorch
However, you are using MSE loss to train your network. Which is a regression loss. For classification problems CrossEntropyLoss is the GO TO option: https://pytorch.org/docs/master/generated/torch.nn.CrossEntropyLoss.html
IF you are solving an regression task. Than usually accuracy is defined by a threshold. Predictions that are bellow a certain error from the ground truth are considered, the others arent. In this case the calculation is the following:
errors = (pred - target) ** 2  # Squared error
acc = (errors < threshold).float().mean()
error = errors.mean()

Where:
pred.shape = (batch_size, chanels, ...)
target.shap = (batch_size, chanels, ...)

Was it something like that you were looking for?


Instruction: Saving model in pytorch and keras

Input: I have trained model with keras and saved in with the help of pytorch. Will it cause any problems in the future. As far as I know the only difference between them is Keras saves its model's weights as doubles while PyTorch saves its weights as floats.

Output:
You can convert your model to double by doing

model.double()

Note that after this, you will need your input to be DoubleTensor.


Instruction: accuracy is not increasing in classification images

Input:
I try to implement the classification of images with bayesian CNN using dropout.
I have defined two classes:

with dropout for the training phase
without dropout for the test(Don’t drop out on testing? to be confirmed please)

When I started the program I remade that the train/test accuracy remain stable they don’t increase. I don’t see what the problem is.
I don’t know if it’s because of convolution and pooling layer parameters or what? Any idea please.
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5, padding=2)
        self.fc1 = nn.Linear(16 * 8 * 8, 1024)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 192 * 8 * 8)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

            # Lenet with MCDO
    class Net_MCDO(nn.Module):
        def __init__(self):
            super(Net_MCDO, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(16, 192, 5, padding=2)
            self.fc1 = nn.Linear(16 * 8 * 8, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)
            self.dropout = nn.Dropout(p=0.3)

    def forward(self, x):
            x = self.pool(self.dropout(self.conv1(x)))
            x = self.pool(self.dropout(self.conv2(x)))
            x = x.view(-1, 192 * 8 * 8)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(self.dropout(x)))
            x = F.softmax(self.fc3(self.dropout(x)),dim=1)
            return x

    net=Net()
    mcdo=Net_MCDO()

    CE = nn.CrossEntropyLoss()
    learning_rate=0.001
    optimizer=optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
    epoch_num = 30
    train_accuracies=np.zeros(epoch_num)
    test_accuracies=np.zeros(epoch_num)
    for epoch in range(epoch_num):
        average_loss = 0.0
        total=0
        success=0

        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = Variable(inputs), Variable(labels)

            optimizer.zero_grad()
            outputs = mcdo(inputs)
            loss=CE(outputs, labels)
            loss.backward()
            optimizer.step()

            average_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            success += (predicted==labels.data).sum()

        train_accuracy = 100.0*success/total
        succes=0
        total=0

        for (inputs, labels) in testloader:
            inputs, labels = Variable(inputs), Variable(labels)
            outputs = net(inputs)
            _,predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            success += (predicted==labels.data).sum()

        test_accuracy = 100.0*success/total
        print(u"epoch{}, average_loss{}, train_accuracy{},
          test_accuracy{}".format(
          epoch,
          average_loss/n_batches,
          train_accuracy,
          100*success/total
          ))
            #save
            train_accuracies[epoch] = train_accuracy
            test_accuracies[epoch] = 100.0*success/total

    plt.plot(np.arange(1, epoch_num+1), train_accuracies)
    plt.plot(np.arange(1, epoch_num+1), test_accuracies)
    plt.show()

Output:
Pytorch merges Softmax inside the CrossEntroplyLoss for numerical stability (and better training). So you should remove the softmax layer of your models. (check the documentation here: https://pytorch.org/docs/stable/nn.html#crossentropyloss). Keeping the Sofmax layer in your model will lead to slower training and possibly worse metrics, that is because you are squashing the gradient twice, thus the weight update is a lot less significant.
Change your code to:
 class Net_MCDO(nn.Module):
        def __init__(self):
            super(Net_MCDO, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5, padding=2)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(16, 192, 5, padding=2)
            self.fc1 = nn.Linear(16 * 8 * 8, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 10)
            self.dropout = nn.Dropout(p=0.3)

    def forward(self, x):
            x = self.pool(F.relu(self.dropout(self.conv1(x))))  # recommended to add the relu
            x = self.pool(F.relu(self.dropout(self.conv2(x))))  # recommended to add the relu
            x = x.view(-1, 192 * 8 * 8)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(self.dropout(x)))
            x = self.fc3(self.dropout(x)) # no activation function needed for the last layer
            return x

Furthermore, I would recommend you using an activation function, such as ReLU() after every conv or linear layer. Otherwise you are just performing a bunch of linear operations that could be learned in one single layer.
I hope that helps =)


Instruction: Question on restoring training after loading model

Input:
Having trained for 24 hours, the training process saved the model files via torch.save. There was a power-off or other issues caused the process exited. Normally, we can load the model and continue training from the last step.
Why should not we load the states of optimizers (Adam, etc), is it necessary?

Output:
Yes, you can load the model from the last step and retrain it from that very step.
if you want to use it only for inference, you will save the state_dict of the model as
torch.save(model, PATH)

And load it as
model = torch.load(PATH)
model.eval()

However, for your concern you need to save the optimizer state dict as well. For that purpose, you need to save it as
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),
            'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,
            ...
            }, PATH)

and load the model for further training as:
model = TheModelClass(*args, **kwargs)
optimizer = TheOptimizerClass(*args, **kwargs)

checkpoint = torch.load(PATH)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']

model.eval()
# - or -
model.train()

It is necessary to save the optimizer state dictionary, since this contains buffers and parameters that are updated as the model trains.


Instruction: is binary cross entropy an additive function?

Input:
I am trying to train a machine learning model where the loss function is binary cross entropy, because of gpu limitations i can only do batch size of 4 and i'm having lot of spikes in the loss graph. So I'm thinking to back-propagate after some  predefined batch size(>4). So it's like i'll do 10 iterations of batch size 4 store the losses, after 10th iteration add the losses and back-propagate. will it be similar to batch size of 40.
TL;DR
f(a+b) = f(a)+f(b) is it true for binary cross entropy?

Output:
f(a+b) = f(a) + f(b) doesn't seem to be what you're after. This would imply that BCELoss is additive which it clearly isn't. I think what you really care about is if for some index i
# false
f(x, y) == f(x[:i], y[:i]) + f([i:], y[i:])

is true?
The short answer is no, because you're missing some scale factors. What you probably want is the following identity
# true
f(x, y) == (i / b) * f(x[:i], y[:i]) + (1.0 - i / b) * f(x[i:], y[i:])

where b is the total batch size.
This identity is used as motivation for the gradient accumulation method (see below). Also, this identity applies to any objective function which returns an average loss across each batch element, not just BCE.

Caveat/Pitfall: Keep in mind that batch norm will not behave exactly the same when using this approach since it updates its internal statistics based on batch size during the forward pass.

We can actually do a little better memory-wise than just computing the loss as a sum followed by backpropagation. Instead we can compute the gradient of each component in the equivalent sum individually and allow the gradients to accumulate. To better explain I'll give some examples of equivalent operations
Consider the following model
import torch
import torch.nn as nn
import torch.nn.functional as F

class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        num_outputs = 5
        # assume input shape is 10x10
        self.conv_layer = nn.Conv2d(3, 10, 3, 1, 1)
        self.fc_layer = nn.Linear(10*5*5, num_outputs)

    def forward(self, x):
        x = self.conv_layer(x)
        x = F.max_pool2d(x, 2, 2, 0, 1, False, False)
        x = F.relu(x)
        x = self.fc_layer(x.flatten(start_dim=1))
        x = torch.sigmoid(x)   # or omit this and use BCEWithLogitsLoss instead of BCELoss
        return x

# to ensure same results for this example
torch.manual_seed(0)
model = MyModel()
# the examples will work as long as the objective averages across batch elements
objective = nn.BCELoss()
# doesn't matter what type of optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

and lets say our data and targets for a single batch are
torch.manual_seed(1)    # to ensure same results for this example
batch_size = 32
input_data = torch.randn((batch_size, 3, 10, 10))
targets = torch.randint(0, 1, (batch_size, 20)).float()

Full batch
The body of our training loop for an entire batch may look something like this
# entire batch
output = model(input_data)
loss = objective(output, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
loss_value = loss.item()

print("Loss value: ", loss_value)
print("Model checksum: ", sum([p.sum().item() for p in model.parameters()]))

Weighted sum of loss on sub-batches
We could have computed this using the sum of multiple loss functions using
# This is simpler if the sub-batch size is a factor of batch_size
sub_batch_size = 4
assert (batch_size % sub_batch_size == 0)

# for this to work properly the batch_size must be divisible by sub_batch_size
num_sub_batches = batch_size // sub_batch_size

loss = 0
for sub_batch_idx in range(num_sub_batches):
    start_idx = sub_batch_size * sub_batch_idx
    end_idx = start_idx + sub_batch_size
    sub_input = input_data[start_idx:end_idx]
    sub_targets = targets[start_idx:end_idx]
    sub_output = model(sub_input)
    # add loss component for sub_batch
    loss = loss + objective(sub_output, sub_targets) / num_sub_batches
optimizer.zero_grad()
loss.backward()
optimizer.step()

loss_value = loss.item()

print("Loss value: ", loss_value)
print("Model checksum: ", sum([p.sum().item() for p in model.parameters()]))

Gradient accumulation
The problem with the previous approach is that in order to apply back-propagation, pytorch needs to store intermediate results of layers in memory for every sub-batch. This ends up requiring a relatively large amount of memory and you may still run into memory consumption issues.
To alleviate this problem, instead of computing a single loss and performing back-propagation once, we could perform gradient accumulation. This gives equivalent results of the previous version. The difference here is that we instead perform a backward pass on each component of
the loss, only stepping the optimizer once all of them have been backpropagated. This way the computation graph is cleared after each sub-batch which will help with memory usage. Note that this works because .backward() actually accumulates (adds) the newly computed gradients to the existing .grad member of each model parameter. This is why optimizer.zero_grad() must be called only once, before the loop, and not during or after.
# This is simpler if the sub-batch size is a factor of batch_size
sub_batch_size = 4
assert (batch_size % sub_batch_size == 0)

# for this to work properly the batch_size must be divisible by sub_batch_size
num_sub_batches = batch_size // sub_batch_size

# Important! zero the gradients before the loop
optimizer.zero_grad()
loss_value = 0.0
for sub_batch_idx in range(num_sub_batches):
    start_idx = sub_batch_size * sub_batch_idx
    end_idx = start_idx + sub_batch_size
    sub_input = input_data[start_idx:end_idx]
    sub_targets = targets[start_idx:end_idx]
    sub_output = model(sub_input)
    # compute loss component for sub_batch
    sub_loss = objective(sub_output, sub_targets) / num_sub_batches
    # accumulate gradients
    sub_loss.backward()
    loss_value += sub_loss.item()
optimizer.step()

print("Loss value: ", loss_value)
print("Model checksum: ", sum([p.sum().item() for p in model.parameters()]))


Instruction: Splitting custom PyTorch dataset into train loader and validation loader: Length of both same, even though dataset was split?

Input:
I'm trying to split one of the Pytorch custom datasets (MNIST) into a training set and a validation set as follows:
def get_train_valid_splits(data_dir,
                           batch_size,
                           random_seed=1,
                           valid_size=0.2,
                           shuffle=True,
                           num_workers=4,
                           pin_memory=False):

    normalize = transforms.Normalize((0.1307,), (0.3081,))  # MNIST

    # define transforms
    valid_transform = transforms.Compose([
            transforms.ToTensor(),
            normalize
        ])

        train_transform = transforms.Compose([
            transforms.ToTensor(),
            normalize
        ])

    # load the dataset
    train_dataset = datasets.MNIST(root=data_dir, train=True,
                download=True, transform=train_transform)

    valid_dataset = datasets.MNIST(root=data_dir, train=True,
                download=True, transform=valid_transform)

    dataset_size = len(train_dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(valid_size * dataset_size))

    if shuffle == True:
        np.random.seed(random_seed)
        np.random.shuffle(indices)

    train_idx, valid_idx = indices[split:], indices[:split]

    train_sampler = sampler.SubsetRandomSampler(train_idx)
    valid_sampler = sampler.SubsetRandomSampler(valid_idx)

    print(len(train_sampler))
    print(len(valid_sampler))

    train_loader = torch.utils.data.DataLoader(train_dataset,
                    batch_size=batch_size, sampler=train_sampler,
                    num_workers=num_workers, pin_memory=pin_memory)

    valid_loader = torch.utils.data.DataLoader(valid_dataset,
                    batch_size=batch_size, sampler=valid_sampler,
                    num_workers=num_workers, pin_memory=pin_memory)

    print(len(train_loader.dataset))
    print(len(valid_loader.dataset))

    return (train_loader, valid_loader)

After calling the function I notice that the results of the indices to sample look right, 48000 and 12000:
print(len(train_sampler))
print(len(valid_sampler))

But when I look at the length of the data set associated with train_loader and valid_loader:
print(len(train_loader.dataset))
print(len(valid_loader.dataset))

I get the same length for both: 60000! Any idea what is going on here? Why is it giving the same length for both, even though I clearly split it by indices?

Output:
It's because the dataloader doesn't modify the dataset you pass it, but "applies" things like batch size, samplers, etc ... to the data when you try to access by iterating it. You're issue is you're using len(loader.dataset) which gives you the length of the provided dataset without modification, when you really wanted len(loader) which is the length of the dataset after "applying" things like batch size and samplers.
import torch
import numpy as np

dataset = np.random.rand(100,200)
sampler = torch.utils.data.SubsetRandomSampler(list(range(70)))

loader = torch.utils.data.DataLoader(dataset, sampler=sampler)
print(len(loader))
>>> 70
print(len(loader.dataset))
>>> 100

Note: The result of len will be affected by batch size:
# with batch size
loader = torch.utils.data.DataLoader(dataset, sampler=sampler, batch_size=2)
print(len(loader))
>>> 35
print(len(loader.dataset))
>>> 100


Instruction: How to conditionally construct a tensor from two other tensors in PyTorch on the GPU?

Input:
An example:
import torch

pred = torch.tensor([1,2,1,0,0], device='cuda:0')
correct = torch.tensor([1,0,1,1,0], device='cuda:0')
assigned = torch.tensor([1,2,2,1,0], device='cuda:0')

I want result = tensor([1,2,1,1,0], device='cuda:0').

Basically, when pred is same as correct then correct else assigned.

Moreover, I want to exclude this computation from the gradient computation.

Is there a way to do this without iterating over the tensors?

Output:
torch.where does exactly what you're looking for:
import torch

pred = torch.tensor([1,2,1,0,0], device='cuda:0')
correct = torch.tensor([1,0,1,1,0], device='cuda:0')
assigned = torch.tensor([1,2,2,1,0], device='cuda:0')

result = torch.where(pred == correct, correct, assigned)

print(result)
# >>> tensor([1, 2, 1, 1, 0], device='cuda:0')

Since none of these tensors have requires_grad=True, nothing needs to be done to avoid gradient computation. Otherwise, you can do something like this:
import torch

pred = torch.tensor([1.,2.,1.,0.,0.], device='cuda:0')
correct = torch.tensor([1.,0.,1.,1.,0.], device='cuda:0', requires_grad=True)
assigned = torch.tensor([1.,2.,2.,1.,0.], device='cuda:0', requires_grad=True)

with torch.no_grad():
    result = torch.where(pred == correct, correct, assigned)

print(result)
# >>> tensor([1, 2, 1, 1, 0], device='cuda:0')

If you do not use torch.no_grad(), you'll have:
result = torch.where(pred == correct, correct, assigned)
print(result)
# >>> tensor([1., 2., 1., 1., 0.], device='cuda:0', grad_fn=<SWhereBackward>)

which, then, can be detached from the computational graph by using:
result = result.detach()
print(result)
# >>> tensor([1., 2., 1., 1., 0.], device='cuda:0')