Instruction: Pytorch RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

Input:
I am training a model that takes tokenized strings which are then passed through an embedding layer and an LSTM thereafter. However, there seems to be an error in the input, as it does not pass through the embedding layer.

class DrugModel(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim, drug_embed_dim,
            lstm_layer, lstm_dropout, bi_lstm, linear_dropout, char_vocab_size,
            char_embed_dim, char_dropout, dist_fn, learning_rate,
            binary, is_mlp, weight_decay, is_graph, g_layer,
            g_hidden_dim, g_out_dim, g_dropout):

        super(DrugModel, self).__init__()

        # Save model configs
        self.drug_embed_dim = drug_embed_dim
        self.lstm_layer = lstm_layer
        self.char_dropout = char_dropout
        self.dist_fn = dist_fn
        self.binary = binary
        self.is_mlp = is_mlp
        self.is_graph = is_graph
        self.g_layer = g_layer
        self.g_dropout = g_dropout
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # For one-hot encoded SMILES
        if not is_mlp:
            self.char_embed = nn.Embedding(char_vocab_size, char_embed_dim,
                                           padding_idx=0)
            self.lstm = nn.LSTM(char_embed_dim, drug_embed_dim, lstm_layer,
                                bidirectional=False,
                                batch_first=True, dropout=lstm_dropout)
        # Distance function
        self.dist_fc = nn.Linear(drug_embed_dim, 1)

        if binary:
            # Binary Cross Entropy
            self.criterion = lambda x, y: y*torch.log(x) + (1-y)*torch.log(1-x)

    def init_lstm_h(self, batch_size):
        return (Variable(torch.zeros(
                self.lstm_layer*1, batch_size, self.drug_embed_dim)).cuda(),
                Variable(torch.zeros(
                self.lstm_layer*1, batch_size, self.drug_embed_dim)).cuda())

    # Set Siamese network as basic LSTM
    def siamese_sequence(self, inputs, length):
        # Character embedding
        inputs = inputs.long()
        inputs = inputs.cuda()

        self.char_embed = self.char_embed(inputs.to(self.device))
        c_embed = self.char_embed(inputs)
        # c_embed = F.dropout(c_embed, self.char_dropout)
        maxlen = inputs.size(1)

        if not self.training:
            # Sort c_embed
            _, sort_idx = torch.sort(length, dim=0, descending=True)
            _, unsort_idx = torch.sort(sort_idx, dim=0)
            maxlen = torch.max(length)

            # Pack padded sequence
            c_embed = c_embed.index_select(0, Variable(sort_idx).cuda())
            sorted_len = length.index_select(0, sort_idx).tolist()
            c_packed = pack_padded_sequence(c_embed, sorted_len, batch_first=True)

        else:
            c_packed = c_embed

        # Run LSTM
        init_lstm_h = self.init_lstm_h(inputs.size(0))
        lstm_out, states = self.lstm(c_packed, init_lstm_h)

        hidden = torch.transpose(states[0], 0, 1).contiguous().view(
                                 -1, 1 * self.drug_embed_dim)
        if not self.training:
            # Unsort hidden states
            outputs = hidden.index_select(0, Variable(unsort_idx).cuda())
        else:
            outputs = hidden

        return outputs

    def forward(self, key1, key2, targets, key1_len, key2_len, status, predict = False):
        if not self.is_mlp:
            output1 = self.siamese_sequence(key1, key1_len)
            output2 = self.siamese_sequence(key2, key2_len)

After instantiating the class I get the following error when passing the input through the embedding layer:

<ipython-input-128-432fcc7a1e39> in forward(self, key1, key2, targets, key1_len, key2_len, status, predict)
    129     def forward(self, key1, key2, targets, key1_len, key2_len, status, predict = False):
    130         if not self.is_mlp:
--> 131             output1 = self.siamese_sequence(key1, key1_len)
    132             output2 = self.siamese_sequence(key2, key2_len)
    133             set_trace()

<ipython-input-128-432fcc7a1e39> in siamese_sequence(self, inputs, length)
     74         inputs = inputs.cuda()
     75
---> 76         self.char_embed = self.char_embed(inputs.to(self.device))
     77         set_trace()
     78         c_embed = self.char_embed(inputs)

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/miniconda3/lib/python3.7/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

despite the fact that the input (e.g. key1) has already been passed to cuda and has been transformed into long format:

tensor([[25, 33, 30,  ...,  0,  0,  0],
        [25,  7,  7,  ...,  0,  0,  0],
        [25,  7, 30,  ...,  0,  0,  0],
        ...,
        [25,  7, 33,  ...,  0,  0,  0],
        [25, 33, 41,  ...,  0,  0,  0],
        [25, 33, 41,  ...,  0,  0,  0]], device='cuda:0')

Output:
setting model.device to cuda does not change your inner module devices, so self.lstm, self.char_embed, and self.dist_fc are all still on cpu. correct way of doing it is by using DrugModel().to(device)

in general, it's better not to feed a device to your model and write it in a device-agnostic way. to make your init_lstm_h function device-agnostic you can use something like this


Instruction: Cannot convert Pandas Dataframe columns to float

Input:
I am using Pandas to read a CSV file containing several columns that must be converted to floats:

    df = pd.read_csv(r'dataset.csv',  low_memory=False,  sep = ',')
    df.head(2)

    Coal Flow 01    Air Flow 01 Outlet Temp 01  Inlet Temp 01   Bowl DP 01  Current 01  Vibration 01
    0   51.454407   101.432340  64.917089   234.2488932 2.470623    96.727352   1.874374
    1   51.625368   100.953089  64.726890   233.2340394 2.495698    96.309512   1.996391

Next I specify the columns that need to be converted to floats in a variable called features:

    features = ['Coal Flow 01', 'Air Flow 01', 'Outlet Temp 01', 'Inlet Temp 01',
           'Bowl DP 01', 'Current 01', 'Vibration 01']

Then I needed to convert the the value of the columns to float, but I got an error.

features = np.stack([df[col].values for col in features], 1)
features = torch.tensor(features, dtype=torch.float)
features[:5]

and the error that Pandas is showing me is:

  KeyError: "None of [Index([  51.45440668,   101.4323397,   64.91708906, '234.2488932',\n         2.470623484,   96.72735193,    1.87437372],\n      dtype='object')] are in the [columns]"

Output:
Why not just use astype:

df = pd.read_csv(r'dataset.csv',  low_memory=False,  sep = ',')
df[features] = df[features].apply(lambda x: x.apply(lambda x: x[0]).astype(float))


Instruction: Pytorch is installed but is not working on ubuntu 18.04

Input:
I am trying to install Pytorch via pip on ubuntu 18.04.I have python 3.6 and my laptop is HP-Pavilion notebook 15

The installation seems to be right because i get the message:

  Installing collected packages: torch, torchvision Successfully
  installed torch-1.3.1+cpu torchvision-0.4.2+cpu

i run the verification code and it is ok

  from __future__ import print_function

  import torch

  x = torch.rand(5, 3)

  print(x)

However, when i close the terminal or reboot and try to run he same code i get the error:

  Traceback (most recent call last):

  File "torch.py", line 2, in
      import torch

  AttributeError: module 'torch' has no attribute 'rand'

Output:
How are you executing the python script? Which python are you using? Maybe you installed the package in a different python version?

Try to set alias to the python you want to use:

alias python=/usr/local/bin/python3.6

Then pip install the package with that python alias you will always be using.

python pip install <package name>

Python now will install the package in the python files with the alias python - heading to files: /usr/local/bin/python3.6

Let me know if the error still occurs!


Instruction: Is there a way to write a custom BCE loss in pytorch?

Input:
I am writing a custom BCE in pytorch but in some cases it returns -inf and nan most cases. Which is due to the log function.

bce_loss=y_true*torch.log2(y_pred) +(one_torch-y_true)*torch.log2(one_torch-y_pred)

Is there a way to rewrite this? Note y_pred is a sigmoid output which is between 0 and 1.

Output:
You can clamp the preds to stop from log error.

y_pred = torch.clamp(y_pred, 1e-7, 1 - 1e-7)


Instruction: Use skimage feature extractors in PyTorch Transforms

Input:
I want to apply skimage’s Local Binary Pattern feature extraction on my data, and was wondering if there was any possibility of doing this inside my torch’s Transforms, which right now is the following:

data_transforms = {
    'train': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
    'val': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
}

If not, how would I implement it? Would I have to do it when importing the data?

Output:
You can implement the Transform using the lamdba funtion. As @dhananjay correctly pointed out.
Building on that comment, the implementation would be as follows:

def lbp(x):
    radius = 2
    n_points = 8 * radius
    METHOD = 'uniform'

    lbp = local_binary_pattern(x, n_points, radius, METHOD)
    return lbp

data_transforms = {
    'train': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.RandomHorizontalFlip(),
        transforms.Lambda(lbp),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
    'val': transforms.Compose([
        transforms.CenterCrop(178),
        transforms.Lambda(lbp),
        transforms.ToTensor(),
        transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
    ]),
}

BUT. This is a bad idea because it defeats the very purpose of the pytorch tranform.
A transform is ideal for an operation that either
1. Can be computed trivially (at low compute cost) from the original data. Hence there is no advantage to
applying it on your data and storing a copy. Normalize is one such transform.
2. Introduces an element of stochasticity or randomn perturbation in the original data. E.g RandomHorizontalFlip etc.

The key thing to remember is that your transform will be applied at every batch to the dataset while
learning.

Considering the above, you absolutely do not want to implement your lbp as  a transform. It is better
to compute it offline and store it. Else you will be significantly slowing down your batch loading.


Instruction: pytorch: RuntimeError: bool value of Tensor with more than one value is ambiguous

Input:
it works with

x[x >= 0.2] = 1
x[x < 0.2] = 0

x is a tensor here.

but when i am trying to use

x[x > 0 and x < 1] = 1

it reports: RuntimeError: bool value of Tensor with more than one value is ambiguous ?

dose anyone know why?

Output:
Just a syntax thing.

x = torch.randn((1,3,20,20))
x[(x > 0) & (x < 1)] = 1


Instruction: Construct a rotation matrix in Pytorch

Input:
I want to construct a rotation matrix, which have unknown Eular angles. I want to build some regression solution to find the value of Eular angles. My code is here.

roll = yaw = pitch = torch.randn(1,requires_grad=True)
RX = torch.tensor([
                [1, 0, 0],
                [0, cos(roll), -sin(roll)],
                [0, sin(roll), cos(roll)]
            ],requires_grad=True)
RY = torch.tensor([
                [cos(pitch), 0, sin(pitch)],
                [0, 1, 0],
                [-sin(pitch), 0, cos(pitch)]
            ],requires_grad=True)
RZ = torch.tensor([
                [cos(yaw), -sin(yaw), 0],
                [sin(yaw), cos(yaw), 0],
                [0, 0, 1]
            ],requires_grad=True)
R = torch.mm(RZ, RY).requires_grad_()
R = torch.mm(R, RX).requires_grad_()
R = R.mean().requires_grad_()
R.backward()

Matrix cannot differentiate the Euler angles.
There isn't any gradient value of matrix. Can anyone fix my problems? Thanks!
debug result

Output:
torch.tensor is viewed as an operation and that is not able to do backpropgation.

A dirty way to fix your code:

roll = torch.randn(1,requires_grad=True)
yaw = torch.randn(1,requires_grad=True)
pitch = torch.randn(1,requires_grad=True)

tensor_0 = torch.zeros(1)
tensor_1 = torch.ones(1)

RX = torch.stack([
                torch.stack([tensor_1, tensor_0, tensor_0]),
                torch.stack([tensor_0, cos(roll), -sin(roll)]),
                torch.stack([tensor_0, sin(roll), cos(roll)])]).reshape(3,3)

RY = torch.stack([
                torch.stack([cos(pitch), tensor_0, sin(pitch)]),
                torch.stack([tensor_0, tensor_1, tensor_0]),
                torch.stack([-sin(pitch), tensor_0, cos(pitch)])]).reshape(3,3)

RZ = torch.stack([
                torch.stack([cos(yaw), -sin(yaw), tensor_0]),
                torch.stack([sin(yaw), cos(yaw), tensor_0]),
                torch.stack([tensor_0, tensor_0, tensor_1])]).reshape(3,3)

R = torch.mm(RZ, RY)
R = torch.mm(R, RX)
R_mean = R.mean()

R_mean.backward()


Instruction: RuntimeError: Error(s) in loading state_dict for Generator: size mismatch for weights and biases using Pytorch

Input:
I'm training a 3D-GAN to generate MRI volumes. I defined my model as follows:

###### Definition of the generator ######

class Generator(nn.Module):
  def __init__(self, ngpu):
    #super() makes Generator a subclass of nn.Module, so that it inherites all the methods of nn.Module
    super(Generator, self).__init__()
    self.ngpu = ngpu
    #we can use Sequential() since the output of one layer is the input of the next one
    self.main = nn.Sequential(
        # input is latent vector z, going into a convolution
        nn.ConvTranspose3d(nz, ngf * 8, 4, stride=2, padding=0, bias=True), # try to put kernel = (batch_size,4,4,4,512)
        nn.BatchNorm3d(ngf * 8),
        nn.ReLU(True), #True means that it does the operation inplace, default is False

        nn.ConvTranspose3d(ngf * 8, ngf * 4, 4, stride=2, padding=1, bias=True), # try to put kernel = (batch_size,8,8,8,256)
        nn.BatchNorm3d(ngf * 4),
        nn.ReLU(True),

        nn.ConvTranspose3d(ngf * 4, ngf * 2, 4, stride=2, padding=1, bias=True), # try to put kernel = (batch_size,16,16,16,128)
        nn.BatchNorm3d(ngf * 2),
        nn.ReLU(True),

        nn.ConvTranspose3d( ngf * 2, ngf, 4, stride=2, padding=1, bias=True), # try to put kernel = (batch_size,32,32,32,64)
        nn.BatchNorm3d(ngf),
        nn.ReLU(True),

        nn.ConvTranspose3d(ngf, nc, 4, stride=2, padding=1, bias=True), # try to put kernel = (batch_size,64,64,64,1)
        nn.Sigmoid()

        )

  def forward(self, x):
    return self.main(x)

###### Definition of the Discriminator ######

class Discriminator(nn.Module):
    def __init__(self, ngpu):
        super(Discriminator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            nn.Conv3d(nc, ndf, 4, stride=2, padding=1, bias=True),
            nn.BatchNorm3d(ndf),
            nn.LeakyReLU(leak_value, inplace=True),

            nn.Conv3d(ndf, ndf * 2, 4, stride=2, padding=1, bias=True),
            nn.BatchNorm3d(ndf * 2),
            nn.LeakyReLU(leak_value, inplace=True),

            nn.Conv3d(ndf * 2, ndf * 4, 4, stride=2, padding=1, bias=True),
            nn.BatchNorm3d(ndf * 4),
            nn.LeakyReLU(leak_value, inplace=True),

            nn.Conv3d(ndf * 4, ndf * 8, 4, stride=2, padding=1, bias=True),
            nn.BatchNorm3d(ndf * 8),
            nn.LeakyReLU(leak_value, inplace=True),

            nn.Conv3d(ndf * 8, nc, 4, stride=1, padding=0, bias=True),
            nn.Sigmoid()
        )

    def forward(self, x):
        return self.main(x)

I then train the model and save it. When loading the model for evaluation and testing I get the following error:

  RuntimeError: Error(s) in loading state_dict for Generator:
      size mismatch for main.0.weight: copying a param with shape torch.Size([64, 1, 4, 4, 4]) from checkpoint, the shape in current model is torch.Size([200, 512, 4, 4, 4]).
      size mismatch for main.0.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([512]).
      size mismatch for main.1.weight: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([512]).
      size mismatch for main.1.running_mean: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([512]).
      size mismatch for main.1.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([512]).
      size mismatch for main.1.running_var: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([512]).
      size mismatch for main.3.weight: copying a param with shape torch.Size([128, 64, 4, 4, 4]) from checkpoint, the shape in current model is torch.Size([512, 256, 4, 4, 4]).
      size mismatch for main.3.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
      size mismatch for main.4.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
      size mismatch for main.4.running_mean: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
      size mismatch for main.4.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
      size mismatch for main.4.running_var: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([256]).
      size mismatch for main.6.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([128]).
      size mismatch for main.7.weight: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([128]).
      size mismatch for main.7.running_mean: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([128]).
      size mismatch for main.7.bias: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([128]).
      size mismatch for main.7.running_var: copying a param with shape torch.Size([256]) from checkpoint, the shape in current model is torch.Size([128]).
      size mismatch for main.9.weight: copying a param with shape torch.Size([512, 256, 4, 4, 4]) from checkpoint, the shape in current model is torch.Size([128, 64, 4, 4, 4]).
      size mismatch for main.9.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]).
      size mismatch for main.10.weight: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]).
      size mismatch for main.10.running_mean: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]).
      size mismatch for main.10.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]).
      size mismatch for main.10.running_var: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([64]).
      size mismatch for main.12.weight: copying a param with shape torch.Size([1, 512, 4, 4, 4]) from checkpoint, the shape in current model is torch.Size([64, 1, 4, 4, 4]).

What I'm I doing wrong?

Thanks in advance!

Output: The model you loaded and the target model is not identical, so the error raise to inform about mismatches of size, layers, check again your code, or your saved model may not be saved properly


Instruction: What is the fastest Mask R-CNN implementation available

Input:
I'm running a Mask R-CNN model on an edge device (with an NVIDIA GTX 1080). I am currently using the Detectron2 Mask R-CNN implementation and I archieve an inference speed of around 5 FPS.

To speed this up I looked at other inference engines and model implementations. For example ONNX, but I'm not able to gain a faster inference speed.

TensorRT looks very promising to me but I did not found a ready "out-of-the-box" implementation for it.

Are there any other mature and fast inference engines or other techniques to speed up the inference?

Output:
It's almost impossible to get higher inference speed for Mask R-CNN on GTX 1080. You may check detectron2 by Facebook AI Research.
Otherwise, I'd suggest to use YOLACT - (You Only Look At CoefficienTs), it can achieve real-time instance segmentation.

On the other hand, if you don't need instance segmentation, you can use YOLO, SSD, etc for object detection.


Instruction: RuntimeError: Given input size: (10x7x7). Calculated output size: (10x0x0). Output size is too small

Input:
Trying to train the mnist 28x28x1 images

my model is

   def __init__(self):
        super(CNN_mnist, self).__init__()
        self.conv = nn.Sequential(
            # 3 x 128 x 128
            nn.Conv2d(1, 32, 3, 1, 1),
            nn.BatchNorm2d(32),
            nn.LeakyReLU(0.2),

            # 32 x 128 x 128
            nn.Conv2d(32, 64, 3, 1, 1),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(0.2),

            # 64 x 128 x 128
            nn.MaxPool2d(2, 2),

            # 64 x 64 x 64
            nn.Conv2d(64, 128, 3, 1, 1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2),

            # 128 x 64 x 64
            nn.Conv2d(128, 256, 3, 1, 1),
            nn.BatchNorm2d(256),
            nn.LeakyReLU(0.2),

            # 256 x 64 x 64
            nn.MaxPool2d(2, 2),

            # 256 x 32 x 32
            nn.Conv2d(256, 10, 3, 1, 1),
            nn.BatchNorm2d(10),
            nn.LeakyReLU(0.2)
        )
        # 256 x 32 x 32
        self.avg_pool = nn.AvgPool2d(32)
        # 256 x 1 x 1
        self.classifier = nn.Linear(10, 10)

    def forward(self, x):
        features = self.conv(x)
        flatten = self.avg_pool(features).view(features.size(0), -1)
        output = self.classifier(flatten)
        return output, features

and I got following error

  RuntimeError: Given input size: (10x7x7). Calculated output size:
  (10x0x0). Output size is too small

not sure what this error mean and where should I fix it?

Output:
Your [avg_pool] layer expects its input size to be (at least) 32x32, as the kernel size defined for this layer is 32.
However, given the size of the input, the feature map this pooling layer gets is only 7x7 in size. This is too small for kernel size of 32.

You should either increase the input size, or define a smaller (e.g., 7) kernel size for the avg_pooling layer.


Instruction: how to adjust dataloader and make a new dataloader?

Input:
let say I have a data loader of cifar10
if I want to remove some value from the dataloader and make a new dataloader
how should I do it?

def load_data_cifar10(batch_size=128,test=False):
    if not test:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=True,
                                                download=True, transform=transform)
    else:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=False,
                                               download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_dset, batch_size=batch_size, shuffle=True)
    print("LOAD DATA, %d" % (len(train_loader)))
    return train_loader

Output:
You can use the Subset dataset. This takes another dataset as input as well as a list of indices to construct a new dataset. Say you want the first 1000 entries, then you could do

subset_train_dset = torch.utils.data.Subset(train_dset, range(1000))

You can also construct datasets composed of multiple datasets using ConcatDataset dataset, or combinations of ConcatDataset and Subset to build whatever you like

frankenstein_dset = torch.utils.data.ConcatDataset((
    torch.utils.data.Subset(dset1, range(1000)),
    torch.utils.data.Subset(dset2, range(100)))

In your case you would need to either look into the implementation details to determine which indices to keep, or you could write some code to iterate through the original dataset first and save all the indicies you want to keep, then define a Subset with the appropriate indices.


Instruction: loop through the batch image loader pytorch

Input:
Let say I have batch with

  imgs = torch.Size([128, 1, 28, 28])

if I want to loop through the each image

for img in imgs:
   print(img.shpae) -> torch.Size([1, 28, 28])

if I want to get a  torch.Size([1,1, 28, 28]) for each image what should I do?

Output:
unsqueeze Just pass dim, In which position you want to add one extra singleton dimension.

imgs = torch.zeros([128, 1, 28, 28])

# dim (int) – the index at which to insert the singleton dimension
imgs.unsqueeze_(dim = 1)

imgs.shape
>>> torch.Size([128, 1, 1, 28, 28])


Instruction: how to import coco_utils and coco_eval?

Input:
To get a PyTorch script to work, I need to be able to do:

import coco_utils
import coco_eval

I'm using Ubuntu 18.04 with Python 3.

Based on this post:

How to install COCO PythonAPI in python3

I've done the following so far:

cd ~

git clone https://github.com/cocodataset/cocoapi.git

cd cocoapi/PythonAPI

# open Makefile in gedit, change the two instances of "python" to "python3"

python3 setup.py build_ext --inplace

sudo python3 setup.py install

now this works:

import pycocotools

but these still don't:

import coco_utils
import coco_eval

How can I import coco_utils and coco_eval ??

Output:
coco_utils.py and coco_eval.py must be the name of the files that should be in the repository you're trying to use.

Possibly you're looking for the object detection reference training scripts provided in the detection (vision/references/detection/) module of torchvision.


Instruction: RuntimeError: Only Tensors of floating point dtype can require gradients

Input:
RuntimeError: Only Tensors of floating point dtype can require gradients

got following error from

  input = Variable(preprocessed_img, requires_grad = True)

img=train_loader.dataset.data[0]
print(type(img))
img_tensor = torch.tensor(img)
preprocess_image(img)

> def preprocess_image(img):    means=[0.485, 0.456, 0.406]     stds=[0.229,
> 0.224, 0.225]
>
>   preprocessed_img = img.copy()[: , :, ::-1]  for i in range(3):
>       preprocessed_img[:, :, i] = preprocessed_img[:, :, i] - means[i]
>       preprocessed_img[:, :, i] = preprocessed_img[:, :, i] / stds[i]
>   preprocessed_img = \
>       np.ascontiguousarray(np.transpose(preprocessed_img, (2, 0, 1)))
>   preprocessed_img = torch.from_numpy(preprocessed_img)
>   preprocessed_img.unsqueeze_(0)  input = Variable(preprocessed_img,
> requires_grad = True)     return input

Output: Use torch.Tensor(img) instead of torch.tensor(img). This should probably solve your issue.


Instruction: How do I make a minimal and reproducible example for neural networks?

Input:
I would like to know how to make a minimal and reproducible deep learning example for Stack Overflow. I want to make sure that people have enough information to pinpoint the exact problem with my code. Is it enough to just provide the traceback?

    c:\users\samuel\appdata\local\programs\python\python35\lib\site-packages\keras\engine\training_utils.py
                         in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
        135                             ': expected ' + names[i] + ' to have shape ' +
        136                             str(shape) + ' but got array with shape ' +
    --> 137                             str(data_shape))
        138     return data
        139

Or should I simply post the error message?

  Value Error: Error when checking input: expected dense_1_input to have shape
  (4,) but got array with shape (1,)

Output:
Here are a few tips to make a reproducible, minimal deep learning Example. It's good advice whether it be for Keras, Pytorch, or Tensorflow.

We can't use your data, but in most cases, it doesn't matter. All we need is the right shape.

Use randomly generated numbers of the right shape.

E.g., np.random.randint(0, 256, (100, 30, 30, 3) for 100 colored pictures of size 30x30
E.g., np.random.choice(np.arange(10), 100) for 100 samples of 10 categories

We don't need to see your entire pipeline.

Only provide the bare minimum to run your code.

Make the most out of Keras and its debugging abilities.

Include the traceback. It will most likely point out the exact problem.

Neural networks are all about fitting the right shapes.

At a minimum, always provide the input shapes.

Make it easy to test and reproduce.

Post your entire neural network architecture.
Include your library imports. Define all variables.

Here is an example of a perfect minimal and reproducible example:

"I have an error. When I run this code, it gives me this error:"

  ValueError: Error when checking target: expected dense_2 to have shape (10,) but got array with shape

"Here is my architecture, with generated data:"

import numpy as np
import keras
from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D

xtrain, xtest = np.random.rand(2, 1000, 30, 30, 3)
ytrain, ytest = np.random.choice(np.arange(10), 2000).reshape(2, 1000)

model = Sequential([
    Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=xtrain.shape[1:]),
    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2, 2)),
    Flatten(),
    Dense(128, activation='relu'),
    Dense(10, activation='softmax')])

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adam(),
              metrics=['accuracy'])

model.fit(xtrain, ytrain,
          batch_size=16,
          epochs=10,
          validation_data=(xtest, ytest))


Instruction: How to use trained deep learning model to predict in excel?

Input:
After we trained my neural network and saved my model with tensorflow, we can load the model and predict result like following:

from keras.models import load_model
model = load_model('my_model.h5')
result = model.predict(test_input)

Is there any way we can use the trained model in excel to do the similar job? I assume we can build a excel user defined function to take n parameter and give a number. Can we do in VBA? How to load the model file and run the predict?

Output: My solution is to use xlwings (https://www.xlwings.org/) to call python code.


Instruction: How can we convert a .pth model into .pb file?

Input: I have already got the complete model by using pytorch, however I wanna convert the .pth file into .pb, which could be used in Tensorflow. Does anyone have some ideas?

Output:
You can use ONNX: Open Neural Network Exchange Format

To convert .pth file to .pb First, you need to export a model defined in PyTorch to ONNX and then import the ONNX model into Tensorflow (PyTorch => ONNX => Tensorflow)

This is an example of MNISTModel to Convert a PyTorch model to Tensorflow using ONNX from onnx/tutorials

Save the trained model to a file

torch.save(model.state_dict(), 'output/mnist.pth')

Load the trained model from file

trained_model = Net()
trained_model.load_state_dict(torch.load('output/mnist.pth'))

# Export the trained model to ONNX
dummy_input = Variable(torch.randn(1, 1, 28, 28)) # one black and white 28 x 28 picture will be the input to the model
torch.onnx.export(trained_model, dummy_input, "output/mnist.onnx")

Load the ONNX file

model = onnx.load('output/mnist.onnx')

# Import the ONNX model to Tensorflow
tf_rep = prepare(model)

Save the Tensorflow model into a file

tf_rep.export_graph('output/mnist.pb')

AS noted by @tsveti_iko in the comment

  NOTE: The prepare() is build-in in the onnx-tf, so you first need to install it through the console like this pip install onnx-tf, then import it in the code like this: import onnx from onnx_tf.backend import prepare and after that you can finally use it as described in the answer.


Instruction: Local fully connected layer - Pytorch

Input:
Assume we have a feature representation with kN neurons before the classification layer. Now, the classification layer produces an output layer of size N with only local connections.

That is, the kth neuron at the output is computed using input neurons at locations from kN to kN+N. Hence, every N locations in the input layer (with stride N) give single neuron value at the output.

This is done using conv1dlocal in Keras, however, the PyTorch does not seem to have this.

Weight matrix in standard linear layer: kNxN = kN^2 variables

Weight matrix with local linear layer: (kx1)@N times = NK variables

Output:
This is currently triaged on the PyTorch issue tracker, in the mean time you can get a similar behavious using fold and unfold. See this answer:

https://github.com/pytorch/pytorch/issues/499#issuecomment-503962218

class LocalLinear(nn.Module):
    def __init__(self,in_features,local_features,kernel_size,padding=0,stride=1,bias=True):
        super(LocalLinear, self).__init__()
        self.kernel_size = kernel_size
        self.stride = stride
        self.padding = padding

        fold_num = (in_features+2*padding-self.kernel_size)//self.stride+1
        self.weight = nn.Parameter(torch.randn(fold_num,kernel_size,local_features))
        self.bias = nn.Parameter(torch.randn(fold_num,local_features)) if bias else None

    def forward(self, x:torch.Tensor):
        x = F.pad(x,[self.padding]*2,value=0)
        x = x.unfold(-1,size=self.kernel_size,step=self.stride)
        x = torch.matmul(x.unsqueeze(2),self.weight).squeeze(2)+self.bias
        return x


Instruction: How to get top k accuracy in semantic segmentation using PyTorch?

Input:
How do you compute the top k accuracy in semantic segmentation? In classification, we might compute the topk accuracy as:
correct = output.eq(gt.view(1, -1).expand_as(output))

Output:
You are looking for torch.topk function that computes the top k values along a dimension.
The second output of torch.topk is the "arg top k": the k indices of the top values.
Here's how this can be used in the context of semantic segmentation:
Suppose you have the ground truth prediction tensor y of shape b-h-w (dtype=torch.int64).
Your model predicts per-pixel class logits of shape b-c-h-w, with c is the number of classes (including "background"). These logits are the "raw" predictions before softmax function transforms them into class probabilities.
Since we are only looking at the top k, it does not matter if the predictions are "raw" or "probabilities".
# compute the top k predicted classes, per pixel:
_, tk = torch.topk(logits, k, dim=1)
# you now have k predictions per pixel, and you want that one of them will match the true labels y:
correct_pixels = torch.eq(y[:, None, ...], tk).any(dim=1)
# take the mean of correct_pixels to get the overall average top-k accuracy:
top_k_acc = correct_pixels.mean()

Note that this method does not take into account "ignore" pixels. This can be done with a slight modification to the above code:
valid = y != ignore_index
top_k_acc = correct_pixels[valid].mean()


Instruction: How to run lua ML model from Google Colab

Input:
I am trying to run this neural style transfer model from Github but on Google Colab because my computer doesn't have enough memory/CPU/anything.

I have mounted my google drive to my notebook, cloned the repo onto my drive by following this tutorial downloaded the models to my drive folder, and just to test that it works, I'm using the most basic example of running the Brad Pitt example in the readme using:

th neural_style.lua -style_image examples/inputs/picasso_selfport1907.jpg -content_image examples/inputs/brad_pitt.jpg

But for some reason it won't work.

Using subprocess.run() just returned No such file or directory
Using !th neural_style.lua... returns th: command not found

I've tried like four other things and they all give me a variant of the above two error messages. Any thoughts?

Here is the full notebook code to reproduce my setup on Colab from start to end/error:

# Mount the drive
from google.colab import drive
drive.mount('/content/drive')

# Clone the repo onto the drive
!git clone https://github.com/jcjohnson/neural-style

# Install Pytorch
!pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl
!pip3 install torchvision

# Download the models per the github repo instructions
!bash models/download_models.sh

!th neural_style.lua -style_image examples/inputs/picasso_selfport1907.jpg -content_image examples/inputs/brad_pitt.jpg

Output:
First:
%cd /content/
!git clone https://github.com/nagadomi/distro.git torch --recursive
import os
os.chdir('./torch/')
!bash install-deps
!./install.sh
!. ./install/bin/torch-activate

Now it should work using absolute path to th:
!/content/torch/install/bin/th neural_style.lua -style_image examples/inputs/picasso_selfport1907.jpg -content_image examples/inputs/brad_pitt.jpg


Instruction: PureFrameworkTensorFoundError, Runtime error -FedeartedLearning

Input:
I am trying a Linear Regression algorithm with Federated learning using Pytorch and I face the following error. I am implementing it on Colab. According to me this error might be due to some code line in the train() function. Kindly help is you have worked with Pysyft and have faced such error before.

RuntimeError: invalid argument 8: lda should be at least max(1, 0), but have 0 at /pytorch/aten/src/TH/generic/THBlas.cpp:363

And the following is the code:

#import the necessasry packages
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import syft as sy

#create target and data variables as tensors
x_data=Variable(torch.Tensor([[1.0],[0.0],[1.0],[0.0]]))
y_data=Variable(torch.Tensor([[0.0],[0.0],[1.0],[1.0]]))

#Create virtual Workers
hook = sy.TorchHook(torch)
bob = sy.VirtualWorker(hook, id="bob")
alice = sy.VirtualWorker(hook, id="alice")

data_bob = x_data[0:2]
target_bob = y_data[0:2]
data_alice = x_data[2:0]
target_alice = y_data[2:0]

#creating a class that does Linear Regression
class LinearRegression (nn.Module):

  def __init__(self):
    super(LinearRegression,self). __init__ ()
    self.linear = torch.nn.Linear(1,1)

  def forward(self, x):
    y_pred = self.linear(x)
    return y_pred

#assign the function to the variable name 'Model'
model=LinearRegression()

#send the data to the virtual worker pointers
data_bob = data_bob.send(bob)
data_alice = data_alice.send(alice)

target_bob = target_bob.send(bob)
target_alice = target_alice.send(alice)

# organize pointers into a list
datasets = [(data_bob,target_bob),(data_alice,target_alice)]

#create optimizer and calculate the loss
opt = torch.optim.SGD(params=model.parameters(),lr=0.1)
criterion = torch.nn.MSELoss(size_average=False)

def train():
  opt = torch.optim.SGD(params=model.parameters(),lr=0.1)
  for epoch in range (20):
    model.train()
    print("Training started..")

    for x_data,y_data in datasets:

      model.send(x_data.location)

      opt.zero_grad()

       #forwardpass
       #the model here is the linear regression model
      y_pred = model(x_data)

      #ComputeLoss
      loss=criterion(y_pred,y_data)

      #BackwardPass
      loss.backward()

      opt.step()

      model.get()

      print(loss.get())

train()

Output:
you have a typo here:

data_alice = x_data[2:0]
target_alice = y_data[2:0]

Should be [2:]

Because data_alice is failing, you had this error.


Instruction: azure ml & Pytorch: sample conda-dependencies.yml and docker?

Input:
Could you please point me to the documentation sample showcasing how to put together pytorch dependencies for training on AzureML?
Few related questions to the scenario of running pytorch training workloads on AzureML:

How can I set cuda version to 10.1?
Could you please point to sample demonstrating how to use “official” pytorch docker https://hub.docker.com/r/pytorch/pytorch  (which should have all cuda stuff https://github.com/pytorch/pytorch/blob/master/docker/pytorch/Dockerfile)?.
I’ve found distributed-pytorch-with-horovod.yml in the docs but it does not mention any pytorch dependencies  -- am I looking in the right place?

Output:
Install Pytorch with CUDA Version 10.1 with the following command on windows
pip3 install torch===1.3.1 torchvision===0.4.2 -f https://download.pytorch.org/whl/torch_stable.htm.

From .yml file:
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/ml-frameworks/pytorch/deployment/train-hyperparameter-tune-deploy-with-pytorch/train-hyperparameter-tune-deploy-with-pytorch.yml

Please follow the below documents for pytorch on Azure.

https://notebooks.azure.com/pytorch

https://azure.microsoft.com/en-us/blog/pytorch-on-azure-full-support-for-pytorch-1-2/


Instruction: PyTorch regression is producing the same numbers as prediction

Input:
I have applied a simple NN for regression to the traditional boston housing price dataset. The problem I’m getting is, when I make predictions using the trained model it always predicts the same numbers. Here is my code:

import numpy as  np
import pandas as pd
from sklearn import datasets
data = datasets.load_boston()

X = pd.DataFrame(data.data, columns=data.feature_names)
Y = pd.DataFrame(data.target, columns=["MEDV"])

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.20, random_state=1234)

import torch

x_train = torch.tensor(X_train.values, dtype=torch.float)
y_train = torch.tensor(y_train.values, dtype=torch.float)
x_test = torch.tensor(X_test.values, dtype=torch.float)
y_test = torch.tensor(y_test.values, dtype=torch.float)

import torch.nn.functional as F

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)
        self.predict = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = F.relu(self.hidden(x))
        x = self.predict(x)
        return x

net = Net(n_feature=13, n_hidden=50, n_output=1)
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

for t in range(200):
    prediction = net(x_train)

    loss = loss_func(prediction, y_train)
    print(t, loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Here is the loss per epoch log:

0 6414.029296875
1 3.883837028532696e+19
2 5.963952279684907e+18
3 2.1470207536047063e+18
4 7.729279495852524e+17
5 2.7825424945486234e+17
6 1.001715136546734e+17
7 3.606170935335322e+16
8 1.2982211888283648e+16
9 4673591211720704.0
10 1682493577101312.0
11 605696890503168.0
12 218050926215168.0
13 78498328739840.0
14 28259370663936.0
15 10173364043776.0
16 3662410940416.0
17 1318467796992.0
18 474648543232.0
19 170873470976.0
20 61514457088.0
21 22145208320.0
22 7972275200.0
23 2870019072.0
24 1033206912.0
25 371954528.0
26 133903616.0
27 48205360.0
28 17353982.0
29 6247483.5
30 2249145.25
31 809743.25
32 291558.65625
33 105012.1171875
34 37855.3984375
35 13678.982421875
36 4975.46826171875
37 1842.20361328125
38 714.2283325195312
39 308.15716552734375
40 161.97158813476562
41 109.34477233886719
42 90.39911651611328
43 83.57867431640625
44 81.12332153320312
45 80.23938751220703
46 79.92117309570312
47 79.8066177368164
48 79.76537322998047
49 79.75053405761719
50 79.74518585205078
51 79.7432632446289
52 79.74256896972656
53 79.74231719970703
54 79.74223327636719
55 79.74219512939453
56 79.7421875
57 79.74217987060547
58 79.74217987060547
59 79.74217987060547
60 79.74217987060547
61 79.74217987060547
62 79.74217987060547
63 79.74217987060547
64 79.74217987060547
65 79.74217987060547
66 79.74217987060547
67 79.74217987060547
68 79.74217987060547
69 79.74217987060547
70 79.74217987060547
71 79.74217987060547
72 79.74217987060547
73 79.74217987060547
74 79.74217987060547
75 79.74217987060547
76 79.74217987060547
77 79.74217987060547
78 79.74217987060547
79 79.74217987060547
80 79.74217987060547
81 79.74217987060547
82 79.74217987060547
83 79.74217987060547
84 79.74217987060547
85 79.74217987060547
86 79.74217987060547
87 79.74217987060547
88 79.74217987060547
89 79.74217987060547
90 79.74217987060547
91 79.74217987060547
92 79.74217987060547
93 79.74217987060547
94 79.74217987060547
95 79.74217987060547
96 79.74217987060547
97 79.74217987060547
98 79.74217987060547
99 79.74217987060547
100 79.74217987060547
101 79.74217987060547
102 79.74217987060547
103 79.74217987060547
104 79.74217987060547
105 79.74217987060547
106 79.74217987060547
107 79.74217987060547
108 79.74217987060547
109 79.74217987060547
110 79.74217987060547
111 79.74217987060547
112 79.74217987060547
113 79.74217987060547
114 79.74217987060547
115 79.74217987060547
116 79.74217987060547
117 79.74217987060547
118 79.74217987060547
119 79.74217987060547
120 79.74217987060547
121 79.74217987060547
122 79.74217987060547
123 79.74217987060547
124 79.74217987060547
125 79.74217987060547
126 79.74217987060547
127 79.74217987060547
128 79.74217987060547
129 79.74217987060547
130 79.74217987060547
131 79.74217987060547
132 79.74217987060547
133 79.74217987060547
134 79.74217987060547
135 79.74217987060547
136 79.74217987060547
137 79.74217987060547
138 79.74217987060547
139 79.74217987060547
140 79.74217987060547
141 79.74217987060547
142 79.74217987060547
143 79.74217987060547
144 79.74217987060547
145 79.74217987060547
146 79.74217987060547
147 79.74217987060547
148 79.74217987060547
149 79.74217987060547
150 79.74217987060547
151 79.74217987060547
152 79.74217987060547
153 79.74217987060547
154 79.74217987060547
155 79.74217987060547
156 79.74217987060547
157 79.74217987060547
158 79.74217987060547
159 79.74217987060547
160 79.74217987060547
161 79.74217987060547
162 79.74217987060547
163 79.74217987060547
164 79.74217987060547
165 79.74217987060547
166 79.74217987060547
167 79.74217987060547
168 79.74217987060547
169 79.74217987060547
170 79.74217987060547
171 79.74217987060547
172 79.74217987060547
173 79.74217987060547
174 79.74217987060547
175 79.74217987060547
176 79.74217987060547
177 79.74217987060547
178 79.74217987060547
179 79.74217987060547
180 79.74217987060547
181 79.74217987060547
182 79.74217987060547
183 79.74217987060547
184 79.74217987060547
185 79.74217987060547
186 79.74217987060547
187 79.74217987060547
188 79.74217987060547
189 79.74217987060547
190 79.74217987060547
191 79.74217987060547
192 79.74217987060547
193 79.74217987060547
194 79.74217987060547
195 79.74217987060547
196 79.74217987060547
197 79.74217987060547
198 79.74217987060547
199 79.74217987060547

After training the model, I’ve used the following code to make predictions:

with torch.no_grad():
    y_val = net(x_test)

When I print the predictions I get the following results:

tensor([[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099],
[22.4099]])

Output:
It seems to me that the model does not fit to your dataset well.
After inspecting your code, I think I could guess where went wrong.

The way you perform gradient descent is somehow incorrect to me. Bear in mind that we are optimizing on a non-convex function. So pack the whole train dataset in a batch would not work, and your model will be stuck in a local-minium, which always not good enough. This could turn into a more complicated discussion with quite a lengthy explanation. I found a good reference for you to understand why we cannot do that on a non-convex function from Here.

My suggestion is to try to sample your training data into small batch-size and run forward-backward update with small batches with a loop inside your epoch loop. for instance, from 8 to 96, and see how that works. In your toy example, you could shuffle your training data each epoch and select your mini-batch one by one. If you wanted to go fancier(or, more standard for performing deep learning in Pytorch), you could write a PyTorch dataset to handle data loading and batching. The loss of your training data should be very small if you are doing this correctly.

Edit:

You might also want to gradually decrease your learning rate, say, 10 times after every 60 epochs to get a better-optimized loss function.


Instruction: Can't use GPU with Pytorch

Input:
I keep getting this error when trying to use Pytorch.

  RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

I installed Pytorch using conda install pytorch torchvision cudatoolkit=10.1 -c pytorch.

With tensorflow my GPU runs just fine.

Output:
You can fix this error by installing CUDA 10.2 (The Latest Version) and, additionally re-install Pytorch with this command:

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch


Instruction: PyTorch Model can only recognize birds when birds are close to camera

Input:
update 1

The easiest way to stop cropping images is: to pass in a tuple to size parameter.

So it looks like this:

size=(299,299)

Unfortunately, it does not solve my problem. So the question is still opening.

I just trained my birds model. It works fine when I was testing it with close pictures.

But when I moved the pictures further away my camera, the model was not able to detect birds.

My guess is in this line of code

data = ImageDataBunch.from_folder(path=path_img, bs=48, valid_pct= 0.2, ds_tfms=get_transforms(), size=299, classes=['birds', 'others'])

The

  get_transforms()

and

  size=299

the function and size parameters crop my training images to centralize the images, so that birds in the images appear to be closer to the camera.

How to fix it?

Output:
Your dataset is biased toward birds at a certain scale, i.e., their size, in pixels, span a very small range (you can verify this).
Center-cropping the images will not change that - the size of the birds (in pixels) will not change.
Therefore, your model cannot handle scale changes.

In order to overcome this limitation of the model you need to make it more scale-robust.
The simplest way to achieve this is to add scale augmentations to your data loader (before the crop). By introducing random scale to each training image you effectively change the size of the bird (in pixels) thus your model "sees" birds with a wider range of sizes (in pixels).


Instruction: Convert pytorch tensor to opencv mat and vice versa in C++

Input:
I want to convert pytorch tensors to opencv mat and vice versa in C++. I have these two functions:

cv::Mat TensorToCVMat(torch::Tensor tensor)
{
    std::cout << "converting tensor to cvmat\n";
    tensor = tensor.squeeze().detach().permute({1, 2, 0});
    tensor = tensor.mul(255).clamp(0, 255).to(torch::kU8);
    tensor = tensor.to(torch::kCPU);
    int64_t height = tensor.size(0);
    int64_t width = tensor.size(1);
    cv::Mat mat(width, height, CV_8UC3);
    std::memcpy((void *)mat.data, tensor.data_ptr(), sizeof(torch::kU8) * tensor.numel());
    return mat.clone();
}

torch::Tensor CVMatToTensor(cv::Mat mat)
{
    std::cout << "converting cvmat to tensor\n";
    cv::cvtColor(mat, mat, cv::COLOR_BGR2RGB);
    cv::Mat matFloat;
    mat.convertTo(matFloat, CV_32F, 1.0 / 255);
    auto size = matFloat.size();
    auto nChannels = matFloat.channels();
    auto tensor = torch::from_blob(matFloat.data, {1, size.height, size.width, nChannels});
    return tensor.permute({0, 3, 1, 2});
}

In my code I load two images (image1 and image2) and I want to convert them to pytorch tensors and then back to opencv mat to check if it works. The problem is that I get an memory access error on the first call of TensorToCVMat and I cant figure out whats wrong as I do not have much experience with C++ programming.

cv::Mat image1;
image1 = cv::imread(argv[1]);
if (!image1.data)
{
    std::cout << "no image data\n";
    return -1;
}
cv::Mat image2;
image2 = cv::imread(argv[2]);
if (!image2.data)
{
    std::cout << "no image data\n";
    return -1;
}

torch::Tensor tensor1 = CVMatToTensor(image1);
cv::Mat new_image1 = TensorToCVMat(tensor1); // <<< this is where the memory access error is thrown
torch::Tensor tensor2 = CVMatToTensor(image2);
cv::Mat new_image2 = TensorToCVMat(tensor2);

It would be great if you could give me hints or an explanation to solve this problem.

Output:
Not sure if the error is happening at the memcpy step. But you can use the void* data variant of the Mat constructor
Mat (int rows, int cols, int type, void *data, size_t step=AUTO_STEP)

and you can skip the memcpy step
tensor = uint8_tensor //shape: (h, w, 3)
cv::Mat mat = cv::Mat(height, width, CV_8UC3, tensor.data_ptr());
return mat;


Instruction: 'Sequential' object has no attribute 'features' while extracting vgg19 pytorch features

Input:
I'm trying to extract the features of images using VGG19 network (the output should be of dim : [1 , 7 , 7 , 512] per frame
here is the code I have used :

    deep_net = models.vgg19(pretrained=True).cuda()
    deep_net = nn.Sequential(*list(deep_net.children())[:-2])
    deep_net.eval()
    save_file_sample_path = '/media/data1/out.npy'
    input_image = torch.zeros(1, 3, 224, 224)
    output_feat = np.zeros(shape=[1, 49, 512])
    with torch.no_grad():
      im = default_loader('/media/data1/images/frame612.jpg')
      im = transform(im)
      input_image[0, :, :] = im
      input_image = input_image.cuda()
      output_feat = deep_net(input_image)
      output_feat = output_feat.features[:-2].view(1, 512, 49).transpose(1, 2)

But I get the following error :

  AttributeError: 'Sequential' object has no attribute 'features'

At the line :

          output_feat = output_feat.features[:-2].view(1, 512, 49).transpose(1, 2)

Any idea why this does not work anymore? and how to fix?
 Thanks!

Output:
It's because you are rebuilding deep_net with nn.Sequential so it loses the attribute features.

deep_net = models.vgg19(pretrained=True)
deep_net.features

Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): ReLU(inplace=True)
  ...
  (36): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
)

deep_net = nn.Sequential(*list(deep_net.children())[:-2])
deep_net.features

AttributeError: 'Sequential' object has no attribute 'features'

The equivalent you want now is this:

list(deep_net.children())[0][:-2]


Instruction: Fast.Ai EarlyStoppingCallback does not work

Input:
callbacks = [EarlyStoppingCallback(learn, monitor='error_rate', min_delta=1e-5, patience=5)]
learn.fit_one_cycle(30, callbacks=callbacks, max_lr=slice(1e-5,1e-3))

As you can see, I use patience = 5 and min_delta=1e-5 and monitor='error_rate'

  My understanding is: patience tells how many epochs it waits if improvement is less than min_delta on the monitored value, in this case it's error_rate.

So if my understanding was correct, then it would not stop at Epoch 6.

So is this my understanding wrong or the debug in fast.ai lib ?

Output:
It keeps track of the best error rate and compares the min_delta to the difference between this epoch and that value:

class EarlyStoppingCallback(TrackerCallback):
...
if self.operator(current - self.min_delta, self.best):
    self.best,self.wait = current,0
else:
    self.wait += 1
    if self.wait > self.patience:
        print(f'Epoch {epoch}: early stopping')
        return {"stop_training":True}
...

So self.wait only increases if the decrease in error was large enough. Once the 5th time occurs it stops.

np.greater(0.000638 - 1e-5, 0.000729)

False

There does seem to be an issue though, because clearly if the error rate jumped very high we would not want to assign this to self.best. And I believe the point of this callback is to stop training if the error rate starts to increase - which right now it is doing the opposite.

So in TrackerCallback there might need to be a change in:

mode_dict['auto'] = np.less if 'loss' in self.monitor else np.greater

to

mode_dict['auto'] = np.less if 'loss' in self.monitor or 'error' in self.monitor else np.greater


Instruction: torch.max slower with GPU than with CPU when specifying dimension

Input:
t1_h = torch.tensor(np.arange(100000), dtype=torch.float32)
cuda0 = torch.device('cuda:0')
t1_d = torch.tensor(np.arange(100000), dtype=torch.float32, device = cuda0)

%timeit -n 10000 max_h = torch.max(t1_h, 0)
%timeit -n 10000 max_d = torch.max(t1_d, 0)

10000 loops, best of 3: 144 µs per loop
10000 loops, best of 3: 985 µs per loop

As you can see above, GPU takes much more time than CPU. But if I don't specify dimension for calculating max, then GPU is faster.
%timeit -n 10000 max_h = torch.max(t1_h)
%timeit -n 10000 max_d = torch.max(t1_d)

10000 loops, best of 3: 111 µs per loop
10000 loops, best of 3: 41.8 µs per loop

I also tried with argmax instead of max but it is working correctly (GPU faster than CPU).
%timeit -n 10000 cs_h = torch.argmax(t1_h, 0)
%timeit -n 10000 cs_d = torch.argmax(t1_d, 0)

10000 loops, best of 3: 108 µs per loop
10000 loops, best of 3: 18.1 µs per loop

Is there any reason why torch.max is slow on GPU after specifying dimension?

Output:
I discovered this myself, and opened an issue in PyTorch. It looks like it'll be fixed soon - maybe version 1.5 or 1.6? - but in the meantime the suggested workaround is to use

ii=a.argmax(0)
maxval = a.gather(0, ii.unsqueeze(0)).squeeze(0)


Instruction: PyTorch CNN: Loss is unchanging

Input:
I have tried researching a situation for my unchanging loss, and all the answers I found were specific to the code. I just started learning about CNNs and majority of the CNN is from an example and modified to fit the needs of my dataset. I am trying to classify types of ECGs (normal, atrial fibrillation, other, noisy). When I try to train the CNN the loss remains the same, I think this is because my CNN does not learn and only outputs zeros. So far I have tried changing the learning rate/loss function and have made no difference.

I am doing this on Google Colab so feel free to edit the code, and don't forget to change hardware acceleration under the runtime tab to GPU.

Code:

import os
import cv2
import numpy as np
from tqdm import tqdm
from scipy.io import loadmat
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import matplotlib.pyplot as plt

if torch.cuda.is_available():
  device = torch.device("cuda:0")
  print("Running on GPU -", device )
else:
  device = torch.device("cpu")
  print("Running on CPU -", device )

REBUILD_DATA = True # processing data

class ECG_DATA():
  ECG_LENGTH = 3000
  LABEL_SIZE = 485
  DATA = "physionet.org/files/challenge-2017/1.0.0/training/"
  NORMAL = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-normal"
  AF = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-af"
  OTHER = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-other"
  NOISY = "physionet.org/files/challenge-2017/1.0.0/training/RECORDS-noisy"
  LABELS = {NORMAL: 0, AF: 1, OTHER:2, NOISY: 3}
  trainingData = []
  dataCount = {NORMAL: 0, AF: 0, OTHER: 0, NOISY: 0}

  def make_training_data(self):
    for records in self.LABELS:
      with open(records) as label:
        for ecgFile in tqdm(label):
          ecg = loadmat(self.DATA+ecgFile[:-1]+".mat")["val"][0].tolist()

          if records == self.NOISY:
            #self.zero_padding(ecg)
            for x in range(self.ECG_LENGTH, len(ecg), self.ECG_LENGTH):
              if self.dataCount[records] <= self.LABEL_SIZE and x <= len(ecg):
                self.trainingData.append([np.array(ecg[x-self.ECG_LENGTH:x]), np.eye(len(self.LABELS))[self.LABELS[records]]])
                self.dataCount[records] += 1

          elif self.dataCount[records] <= self.LABEL_SIZE and self.ECG_LENGTH <= len(ecg):
            self.trainingData.append([np.array(ecg[:self.ECG_LENGTH]), np.eye(len(self.LABELS))[self.LABELS[records]]])
            self.dataCount[records] += 1

    print(self.dataCount)
    np.random.shuffle(self.trainingData)
    np.save("training_Data.npy", self.trainingData)

  def zero_padding(self, ecg):
    ecg += [0] * (self.ECG_LENGTH-(len(ecg)%self.ECG_LENGTH))
class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv1d(1, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv1d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv1d(64, 128, 5)

        x = torch.randn(1,3000).view(-1,1,3000)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 4) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        x = F.max_pool1d(F.relu(self.conv1(x)), 1) # adjust shape of pooling?
        x = F.max_pool1d(F.relu(self.conv2(x)), 1) # x = F.max_pool1d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool1d(F.relu(self.conv3(x)), 1)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net().to(device)
print(net)

if REBUILD_DATA:
  ECG = ECG_DATA()
  ECG.make_training_data()

training_data = np.load("training_Data.npy", allow_pickle=True)
print(len(training_data))

optimizer = optim.Adam(net.parameters(), lr = 0.01)
loss_function = nn.MSELoss().to(device)

X = torch.Tensor([i[0] for i in training_data])
y = torch.Tensor([i[1] for i in training_data])

VAL_PCT = 0.1
val_size = int(len(X)*VAL_PCT)
print(val_size)

train_X = X[:-val_size]
train_y = y[:-val_size]

test_X = X[-val_size:]
test_y = y[-val_size:]
print(len(train_X), len(test_X))

BATCH_SIZE = 100
EPOCHS = 1
plot = []

for epoch in range(EPOCHS):
    for i in tqdm(range(0, len(train_X), BATCH_SIZE)): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
        #print(f"{i}:{i+BATCH_SIZE}")
        batch_X = train_X[i:i+BATCH_SIZE].view(-1,1,3000).to(device)
        batch_y = train_y[i:i+BATCH_SIZE].to(device)

        net.zero_grad()

        outputs = net(batch_X)
        loss = loss_function(outputs, batch_y)
        loss.backward()
        optimizer.step()    # Does the update

    plot.append([epoch, float(loss)])
    print(f"\nEpoch: {epoch}. Loss: {loss}")

plot = list(map(list, zip(*plot)))
plt.plot(plot[0], plot[1])

Output:
At the end of your network there's a softmax layer but in your training you use MSELoss. This tells me that your model is outputting probabilities but then you are calculating loss as if it is continuous. Not sure exactly how that is working for you but I would suspect this is a reason for faulty loss.

As mentioned below in the comments, you can use CrossEntropyLoss with your softmax layer intact - since your task is about classification.


Instruction: FastAI PyTorch Train_loss and valid_loss look very good, but the model recognize nothing

Input:
Update 1

I’m thinking that it might be the mistake in my detector code.
So, here is my code for using the trained learner/model to predict images.

import requests
import cv2

bytes = b''
stream = requests.get(url, stream=True)
bytes = bytes + stream.raw.read(1024) # I have my mobile video streaming to this url. the resolution for the video streaming is: 2048 x 1080
a = bytes.find(b'\xff\xd8')
b = bytes.find(b'\xff\xd9')
if a != -1 and b != -1:
      jpg = bytes[a:b+2]
      bytes = bytes[b+2:]
      img = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
      processedImg = Image(pil2tensor(img, np.float32).div_(255))
      predict = learn.predict(processedImg)
      self.objectClass = predict[0].obj

and I read the document of imdecode() method, it returns image in B G R order.

Could it because of different channel data used when in training and detecting?

Or

Could it because that I trained with image size 299 x 450, but when detecting the input image size from the video streaming is 2048 x 1080 without resizing it?

new to FastAi, ML and Python. I trained my “Birds Or Not-Birds” model. The train_loss, valid_loss and error_rate were improving. If I only trained 3 epochs, then the model worked(meaning it can recognize whether there are birds or no birds in images), then I increased to 30 epochs, all metrics look very good, but the model does not recognize things anymore, whatever images I input, the model always return Not-Birds.

here is the training output:

Here are the plots of learn.recorder

Here is my code:

from fastai.vision import *
from fastai.metrics import error_rate
from fastai.callbacks import EarlyStoppingCallback,SaveModelCallback
from datetime import datetime as dt
from functools import partial

path_img = '/minidata'
train_folder = 'train'
valid_folder = 'validation'

tunedTransform = partial(get_transforms, max_zoom=1.5)

data = ImageDataBunch.from_folder(path=path_img, train=train_folder, valid=valid_folder, ds_tfms=tunedTransform(),
                                  size=(299, 450), bs=40, classes=['birds', 'others'],
                                  resize_method=ResizeMethod.SQUISH)
data = data.normalize(imagenet_stats)

learn = cnn_learner(data, models.resnet50, metrics=error_rate)
learn.fit_one_cycle(30, max_lr=slice(5e-5,5e-4))

learn.recorder.plot_lr()
learn.recorder.plot()
learn.recorder.plot_losses()

Here is my dataset folder structure:

minidata

train

birds (7500 images)
others (around 7300 images)

validation

birds (1008 images)
others (around 872 images)

Output:
Your learning rate schedule is sub-optimal for this dataset. Try to first figure out the best learning rate for this network and dataset with
LRFinder. This can be done by exploring the loss behavior for different learning rates with

learn.lr_find()
learn.recorder.plot()

Edit:

It looks like you are re-training the last layer in your network. Instead try training more layers from scratch. as:

learn.unfreeze(2)


Instruction: Language translation using TorchText (PyTorch)

Input:
I have recently started with ML/DL using PyTorch. The following pytorch example explains how we can train a simple model for translating from German to English.

https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html

However I am confused on how to use the model for running inference on custom input. From my understanding so far :

1) We will need to save the "vocab" for both German (input) and English(output) [using torch.save()] so that they can be used later for running predictions.

2) At the time of running inference on a German paragraph, we will first need to convert the German text to tensor using the german vocab file.

3) The above tensor will be passed to the model's forward method for translation

4) The model will again return a tensor for the destination language i.e., English in current example.

5) We will use the English vocab saved in first step to convert this tensor back to English text.

Questions:

1) If the above understanding is correct, can the above steps be treated as a generic approach for running inference on any language translation model if we know the source and destination language and have the vocab files for the same? Or can we use the vocab provided by third party libraries like spacy?

2) How do we convert the output tensor returned from model back to target language? I couldn't find any example on how to do that. The above blog explains how to convert the input text to tensor using source-language vocab.

I could easily find various examples and detailed explanation for image/vision models but not much for text.

Output:
Yes globally what you are saying is correct, and of course you can any vocab, e.g. provided by spacy. To convert a tensor into natrual text, one of the most used thechniques is to keep both a dict that maps indexes to words and an other dict that maps words to indexes, the code below can do this:
tok2idx = defaultdict(lambda: 0)
idx2tok = {}

for seq in sequences:
    for tok in seq:
        if not tok in tok2idx:
            tok2idx[tok] = index
            idx2tok[index] = tok
            index += 1

Here sequences is a list of all the sequences (i.e. sentences in your dataset). You can change the model easily if you have only a list of words or tokens, by only keeping the inner loop.


Instruction: Numpy / PyTorch - how to assign value with indices in different dimensions?

Input:
Suppose I have a matrix and some indices

a = np.array([[1, 2, 3], [4, 5, 6]])
a_indices = np.array([[0,2], [1,2]])

Is there any efficient way to achieve following operation?

for i in range(2):
    a[i, a_indices[i]] = 100

# a: np.array([[100, 2, 100], [4, 100, 100]])

Output:
Use np.put_along_axis -

In [111]: np.put_along_axis(a,a_indices,100,axis=1)

In [112]: a
Out[112]:
array([[100,   2, 100],
       [  4, 100, 100]])

Alternaytively, if you want to do with the explicit way, i.e. integer-based indexing -

In [115]: a[np.arange(len(a_indices))[:,None], a_indices] = 100


Instruction: Running a PyTorch dataloader/Dataset on multiple distributed CPUs

Input:
I wonder if there is a way to distributed the dataloader/Dataset to many CPUs, even when using a single GPU.
Specifically, I would like to have a Dataset class, and the __getitem__ function will be distributed across many different CPUs (using mpi maybe? but any other way is also good).
Thanks
EDIT
My title was erroneously edited, I am not trying to distribute the model itslef, I only want to distribute the data loading/parsing of the model
EDIT - 2
Some interesting discussion in this direction is available here

Output:
You can do this of course, but mind you - it is not always very effective for general Machine Learning needs, due to the hefty communication costs.
Use DistributedDataParallel

  Implements distributed data parallelism that is based on
  torch.distributed package at the module level.

  This container parallelizes the application of the given module by
  splitting the input across the specified devices by chunking in the
  batch dimension. The module is replicated on each machine and each
  device, and each such replica handles a portion of the input. During
  the backwards pass, gradients from each node are averaged.

In practice, I'd recommend you utilize the pytorch_lightning package, to reduce some of the boilerplate code you need to write for this to work.

Reference:
DistributedDataParallel,pytorch_lightning


Instruction: Padding a tensor until reaching required size

Input:
I'm working with certian tensors with shape of (X,42) while X can be in a range between 50 to 70.
I want to pad each tensor that I get until it reaches a size of 70. so all tensors will be (70,42).
is there anyway to do this when I the begining size is a variable X? thanks for the help!

Output:
Use torch.nn.functional.pad - Pads tensor.

import torch
import torch.nn.functional as F

source = torch.rand((3,42))
source.shape
>>> torch.Size([3, 42])

# here, pad = (padding_left, padding_right, padding_top, padding_bottom)
source_pad = F.pad(source, pad=(0, 0, 0, 70 - source.shape[0]))

source_pad.shape
>>> torch.Size([70, 42])


Instruction: Why does loss decrease but accuracy decreases too (Pytorch, LSTM)?

Input:
I have built a model with LSTM - Linear modules in Pytorch for a classification problem (10 classes). I am training the model and for each epoch I output the loss and accuracy in the training set. The ouput is as follows:

epoch: 0 start!
  Loss: 2.301875352859497
  Acc: 0.11388888888888889
  epoch: 1 start!
  Loss: 2.2759320735931396
  Acc: 0.29
  epoch: 2 start!
  Loss: 2.2510263919830322
  Acc: 0.4872222222222222
  epoch: 3 start!
  Loss: 2.225804567337036
  Acc: 0.6066666666666667
  epoch: 4 start!
  Loss: 2.199286699295044
  Acc: 0.6511111111111111
  epoch: 5 start!
  Loss: 2.1704766750335693
  Acc: 0.6855555555555556
  epoch: 6 start!
  Loss: 2.1381614208221436
  Acc: 0.7038888888888889
  epoch: 7 start!
  Loss: 2.1007182598114014
  Acc: 0.7194444444444444
  epoch: 8 start!
  Loss: 2.0557992458343506
  Acc: 0.7283333333333334
  epoch: 9 start!
  Loss: 1.9998993873596191
  Acc: 0.7427777777777778
  epoch: 10 start!
  Loss: 1.9277743101119995
  Acc: 0.7527777777777778
  epoch: 11 start!
  Loss: 1.8325848579406738
  Acc: 0.7483333333333333
  epoch: 12 start!
  Loss: 1.712520718574524
  Acc: 0.7077777777777777
  epoch: 13 start!
  Loss: 1.6056485176086426
  Acc: 0.6305555555555555
  epoch: 14 start!
  Loss: 1.5910680294036865
  Acc: 0.4938888888888889
  epoch: 15 start!
  Loss: 1.6259561777114868
  Acc: 0.41555555555555557
  epoch: 16 start!
  Loss: 1.892195224761963
  Acc: 0.3655555555555556
  epoch: 17 start!
  Loss: 1.4949012994766235
  Acc: 0.47944444444444445
  epoch: 18 start!
  Loss: 1.4332982301712036
  Acc: 0.48833333333333334

For loss function I have used nn.CrossEntropyLoss and Adam Optimizer.
Although the loss is constantly decreasing, the accuracy increases until epoch 10 and then begins for some reason to decrease.

Why is this happening ?

Even if my model is overfitting, doesn't that mean that the accuracy should be high ?? (always speaking for accuracy and loss measured on the training set, not the validation set)

Output:
Decreasing loss does not mean improving accuracy always.
I will try to address this for the cross-entropy loss.

CE-loss= sum (-log p(y=i))

Note that loss will decrease if the probability of correct class increases and loss increases if the probability of correct class decreases. Now, when you compute average loss, you are averaging over all the samples, some of the probabilities may increase and some of them can decrease, making overall loss smaller but also accuracy drops.


Instruction: Pytorch on Google VM (Linux) does not recognize GPU

Input:
I created a Google VM instance using this available image:

c1-deeplearning-common-cu100-20191226

Description

Google, Deep Learning Image: Base, m39 (with CUDA 10.0), A Debian based image with CUDA 10.0

I then installed Anaconda onto this VM, then installed Pytorch using the following command line as recommended by the Pytorch website:

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

(this corresponds to Linux, Python 3.7, CUDA 10.1)

From Python, I ran this code to check the GPU detection:

import torch
torch.cuda.is_available()
False

From the nvidia-smi tool, this is the result even after the main body of code is running the training:

(base) redexces.bf@tensorflow-1x-2x:~$ nvidia-smi
Thu Jan  2 01:33:10 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla P4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   37C    P0    22W /  75W |      0MiB /  7611MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Clearly, there are no running processes nor any memory allocated.

This problem appears to be related to Pytorch only; the same VM also has Tensorflow-gpu installed in a separate conda environment which recognizes the GPU and utilizes it as I would expect.

Am I missing any pieces?  Again the same CUDA driver and image are working fine for tensorflow.

Output: I was able to resolve the issue.  Not being a computer science guy, I figured that it could be an nvidia driver compatibility issue.  Since Pytorch was built using CUDA 10.1 driver, and the deep learning image had CUDA 10.0 installed, I created another VM instance but this time instead of using the public image noted earlier, I used the gcloud command line to specify deep learning with cu10.1 driver.  This made it all work as expected.


Instruction: Merge two tensor in pytorch

Input:
Tensor a:

tensor([[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3],
        [1, 2, 3]])

Tensor b:

tensor([4,4,4,4])

Question 1:

How to merge two tensors and get result c:

tensor([[1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4]])

Question 2: How to divide tensor c and get original a and b.

Output:
Question 1: Merge two tensors -

torch.cat((a, b.unsqueeze(1)), 1)
>>> tensor([[1, 2, 3, 4],
            [1, 2, 3, 4],
            [1, 2, 3, 4],
            [1, 2, 3, 4]])

First, we use torch.unsqueeze to add single dim in b tensor to match a dim to be concanate. Then use torch.cat Concatenates tensors a and b.

Question 2:

a = c[:][:,:-1]
a
>>> tensor([[1, 2, 3],
            [1, 2, 3],
            [1, 2, 3],
            [1, 2, 3]])

b = c[:][:,-1:].squeeze(1)
b
>>> tensor([4, 4, 4, 4])


Instruction: What is the difference between model.to(device) and model=model.to(device)?

Input:
Suppose the model is originally stored on CPU, and then I want to move it to GPU0, then I can do:

device = torch.device('cuda:0')
model = model.to(device)
# or
model.to(device)

What is the difference between those two lines?

Output:
No semantic difference. nn.Module.to function moves the model to the device.
But be cautious.
For tensors (documentation):
# tensor a is in CPU
device = torch.device('cuda:0')
b = a.to(device)
# a is still in CPU!
# b is in GPU!
# a and b are different

For models (documentation):
# model a is in CPU
device = torch.device('cuda:0')
b = a.to(device)
# a and b are in GPU
# a and b point to the same model


Instruction: How to implying bagging method for LSTM neural network using pyTorch?

Input:
Like the title, my question is how to apply the Bagging method for LSTM using the PyTorch library? I have built one using TensorFlow on python. But now to implied into the system using C and C++, the requirement is I need to using PyTorch?

Is there any recommendation for not need to use the PyTorch and applying directly the model built on tensorflow into real predicting in the system??

Please help!

Output:
If you want to create an ensemble in PyTorch, you can train multiple models separately and then define a class to use them together:

class MyEnsemble(nn.Module):
    def __init__(self, firstModel, secondModel):
        super(MyEnsemble, self).__init__()
        self.firstModel = modelA
        self.secondModel = modelB
        self.classifier = nn.Linear(in_features, n_classes) #define accordingly
        self.relu = nn.ReLU()

    def forward(self, x1, x2):
        x1 = self.firstModel(x1)
        x2 = self.secondModel(x2)
        x = torch.cat((x1, x2), dim=1)
        x = self.classifier(self.relu(x))
        return x

If you want to use your TensorFlow model, there are multiple ways of doing so. One can export it to C++ Tensorflow -> C++


Instruction: How can I optimize the 5-layer loop using functions provided by torch?

Input:
x is the tensor with the shape of (16, 10, 4, 25, 53), y has the same size as x.
mean's shape is (25, 53), the size of jc and ac are both (16, 10, 4).

How can I optimize the following expression with torch functions?

for k in range(x.size()[0]):
    for s in range(x.size()[1]):
        for u in range(x.size()[2]):
            for i in range(x.size()[3]):
                for j in range(x.size()[4]):
                    num1 += (x[k][s][u][i][j] - mean[i][j] - jc[k][s][u]) * (y[k][s][u][i][j] - mean[i][j] - ac[k][s][u])
                    num2 += (y[k][s][u][i][j] - mean[i][j] - jc[k][s][u]) ** 2
                    num3 += (y[k][s][u][i][j] - mean[i][j] - ac[k][s][u]) ** 2

Output:
I think you are looking at broadcasting your tensors along singleton dimensions.
First, you need the number of dimensions to be the same, so if mean is of shape (25,53) then mean[None, None, None, ...] is of shape (1, 1, 1, 25, 53) - you did not change anything in the underlying data, but the number of dimensions is now 5 instead of only 2 and these singleton dimensions can be broadcast to the corresponding dimensions of x and y.

An optimized code using broadcasting will look something like:

num1 = ((x - mean[None, None, None, ...] - jc[..., None, None]) * (y - mean[None, None, None, ...] - ac[..., None, None])).sum()
num2 = ((y - mean[None, None, None, ...] - jc[..., None, None]) ** 2).sum()  # shouldn't it be x here?
num3 = ((y - mean[None, None, None, ...] - ac[..., None, None]) ** 2).sum()


Instruction: What does the interleave_keys() function in torchtext library do exactly?

Input:
You can find this function at torchtext/data/utils.py file

I have given the official code with documentation below

def interleave_keys(a, b):
    """Interleave bits from two sort keys to form a joint sort key.

    Examples that are similar in both of the provided keys will have similar
    values for the key defined by this function. Useful for tasks with two
    text fields like machine translation or natural language inference.
    """
    def interleave(args):
        return ''.join([x for t in zip(*args) for x in t])
    return int(''.join(interleave(format(x, '016b') for x in (a, b))), base=2)

A more detailed explanation would be helpful to understand how it returns an integer based on how similar the given two strings are.

And the format function used inside it is the commonly used builtin function in python

Output:
So upon breaking down the function I was able to figure out what this function is doing.

format(x, '016b') This piece of code converts the integer (a and b which is actually no of words in the sentences in my case) to 16 digit binary number.

And the interleave function takes out the pairs (of the same position) of binary representations join them like this,

For easy understanding lets assume 4 digit binary for 2 and 11

2's binary representation is  : 0 0 1 0

11's binary representation is: 1 0 1 1

So the output here would be 01001101 (01,00,11,01 has been combined) which when converting to integer will give 77


Instruction: Is there any method to generate a piecewise function for tensors in pytorch?

Input:
I want to get a piecewise function like this for tensors in pytorch. But I don't know how to define it. I use a very stupid method to do it, but it seems not to work in my code.

    def trapezoid(self, X):
        Y = torch.zeros(X.shape)
        Y[X % (2 * pi) < (0.5 * pi)] = (X[X % (2 * pi) < (0.5 * pi)] % (2 * pi)) * 2 / pi
        Y[(X % (2 * pi) >= (0.5 * pi)) & (X % (2 * pi) < 1.5 * pi)] = 1.0
        Y[X % (2 * pi) >= (1.5 * pi)] = (X[X % (2 * pi) >= (1.5 * pi)] % (2 * pi)) * (-2 / pi) + 4
        return Y

could do you help me find out how to design the function trapezoid, so that for tensor X, I can get the result directly using trapezoid(X)?

Output:
Since your function has period 2π we can focus on [0,2π]. Since it's piecewise linear, it's possible to express it as a mini ReLU network on [0,2π] given by:
trapezoid(x) = 1 - relu(x-1.5π)/0.5π - relu(0.5π-x)/0.5π
Thus, we can code the whole function in Pytorch like so:
import torch
import torch.nn.functional as F
from torch import tensor
from math import pi

def trapezoid(X):
  # Left corner position, right corner position, height
  a, b, h = tensor(0.5*pi), tensor(1.5*pi), tensor(1.0)

  # Take remainder mod 2*pi for periodicity
  X = torch.remainder(X,2*pi)

  return h - F.relu(X-b)/a - F.relu(a-X)/a

Plotting to double check produces the correct picture:
import matplotlib.pyplot as plt

X = torch.linspace(-10,10,1000)
Y = trapezoid(X)
plt.plot(X,Y)
plt.title('Pytorch Trapezoid Function')


Instruction: PyTorch: is there a definitive training loop similar to Keras' fit()?

Input:
I'm coming over from Keras to PyTorch, and one of the surprising things I've found is that I'm supposed to implement my own training loop.

In Keras, there is a de facto fit() function that: (1) runs gradient descent and (2) collects a history of metrics for loss and accuracy over both the training set and validation set.

In PyTorch, it appears that the programmer needs to implement the training loop. Since I'm new to PyTorch, I don't know if my training loop implementation is correct. I just want to compare apples-to-apples loss and accuracy metrics with what I'm seeing in Keras.

I've already read through:

the official PyTorch 60-minute blitz, where they provide a sample training loop.
official PyTorch example code, where I've found the training loop placed in-line with other code.
the O'Reilly book Programming PyTorch for Deep Learning with its own training loop.
Stanford CS230 sample code.
various blog posts (e.g. here and here).

So I'm wondering: is there a definitive, universal training loop implementation that does the same thing and reports the same numbers as the Keras fit() function?

My points of frustration:

Pulling data out of the dataloader is not consistent between image data and NLP data.
Correctly computing loss and accuracy is not consistent in any sample code I've seen.
Some code examples use Variable, while others do not.
Unnecessarily detailed: moving data to/from the GPU; knowing when to call zero_grad().

For what it's worth, here is my current implementation. Are there any obvious bugs?

import time

def train(model, optimizer, loss_fn, train_dl, val_dl, epochs=20, device='cuda'):
    '''
    Runs training loop for classification problems. Returns Keras-style
    per-epoch history of loss and accuracy over training and validation data.

    Parameters
    ----------
    model : nn.Module
        Neural network model
    optimizer : torch.optim.Optimizer
        Search space optimizer (e.g. Adam)
    loss_fn :
        Loss function (e.g. nn.CrossEntropyLoss())
    train_dl :
        Iterable dataloader for training data.
    val_dl :
        Iterable dataloader for validation data.
    epochs : int
        Number of epochs to run
    device : string
        Specifies 'cuda' or 'cpu'

    Returns
    -------
    Dictionary
        Similar to Keras' fit(), the output dictionary contains per-epoch
        history of training loss, training accuracy, validation loss, and
        validation accuracy.
    '''

    print('train() called: model=%s, opt=%s(lr=%f), epochs=%d, device=%s\n' % \
          (type(model).__name__, type(optimizer).__name__,
           optimizer.param_groups[0]['lr'], epochs, device))

    history = {} # Collects per-epoch loss and acc like Keras' fit().
    history['loss'] = []
    history['val_loss'] = []
    history['acc'] = []
    history['val_acc'] = []

    start_time_sec = time.time()

    for epoch in range(epochs):

        # --- TRAIN AND EVALUATE ON TRAINING SET -----------------------------
        model.train()
        train_loss         = 0.0
        num_train_correct  = 0
        num_train_examples = 0

        for batch in train_dl:

            optimizer.zero_grad()

            x    = batch[0].to(device)
            y    = batch[1].to(device)
            yhat = model(x)
            loss = loss_fn(yhat, y)

            loss.backward()
            optimizer.step()

            train_loss         += loss.data.item() * x.size(0)
            num_train_correct  += (torch.max(yhat, 1)[1] == y).sum().item()
            num_train_examples += x.shape[0]

        train_acc   = num_train_correct / num_train_examples
        train_loss  = train_loss / len(train_dl.dataset)

        # --- EVALUATE ON VALIDATION SET -------------------------------------
        model.eval()
        val_loss       = 0.0
        num_val_correct  = 0
        num_val_examples = 0

        for batch in val_dl:

            x    = batch[0].to(device)
            y    = batch[1].to(device)
            yhat = model(x)
            loss = loss_fn(yhat, y)

            val_loss         += loss.data.item() * x.size(0)
            num_val_correct  += (torch.max(yhat, 1)[1] == y).sum().item()
            num_val_examples += y.shape[0]

        val_acc  = num_val_correct / num_val_examples
        val_loss = val_loss / len(val_dl.dataset)

        print('Epoch %3d/%3d, train loss: %5.2f, train acc: %5.2f, val loss: %5.2f, val acc: %5.2f' % \
              (epoch+1, epochs, train_loss, train_acc, val_loss, val_acc))

        history['loss'].append(train_loss)
        history['val_loss'].append(val_loss)
        history['acc'].append(train_acc)
        history['val_acc'].append(val_acc)

    # END OF TRAINING LOOP

    end_time_sec       = time.time()
    total_time_sec     = end_time_sec - start_time_sec
    time_per_epoch_sec = total_time_sec / epochs
    print()
    print('Time total:     %5.2f sec' % (total_time_sec))
    print('Time per epoch: %5.2f sec' % (time_per_epoch_sec))

    return history

Output:
Short answer: there is no equivalent training loop for PT and TF.keras and there shall never be one.

First of all, the training loop is syntactical sugar that is supposed to makes one's life easier. From my point of view, "making life easier" is a moto of TF.keras framework and this is the main reason it has it. Training loop can not be formalized as well defined practice, it might vary a lot depending on the task/dataset/procedure/metric/you_name_it and it would require a lot of effort to match all the options for 2 frameworks. Furthermore, creating a defining interface for training loop in Pytorch might be too restrictive for many actual users of the framework.

Matching the outputs of network would require matching behaviors of every operation within 2 frameworks, which would be impossible. First of all, the frameworks don't necessarily provide same sets of operations. Operations can be grouped into higher level abstracts differently. Also, some common functions like sigmoid or BatchNorm might look well mathematically defined on paper, but in reality have dozens of implementation specific details. Also, when improvements are introduced to the operations it is up to the community to integrate these updates into main framework distributions or plane ignore them. Needless to say, developers of 2 frameworks make these decisions independently and likely have different motivation behind them.

To sum it all up, matching high level details of 2 frameworks would require enormous effort and would probably be very disruptive for the existing users.


Instruction: Pass user specified parameters to DataLoader

Input:
I am using U - Net and implementing the weighting technique described in the papers from 2015 (U-Net: Convolutional Networks for Biomedical
Image Segmentation) and 2019 (U-Net – Deep Learning for Cell Counting, Detection, and Morphometry). In that technique there is a variance σ and a weight w_0. I would like, especially the σ, to be a learnable parameter instead of guessing which value is best from dataset to dataset.

From what I found, I can do this using nn.Parameter.
To use the learned σ from epoch to epoch, I need somehow to pass this new value to the get_item function of the DataSet through the DataLoader.

My current take on this, is to extend torch.utils.data.DataLoader where the new init has an extra parameter accepting the user specified/learnable parameters. Given the source code of torch.utils.data.DataLoader, I do not understand where and how the DataLoader calls the DataSet instance and hence to pass these parameters.
Code wise, in the DataSet definition there is the function
def __getitem__(self, index):

that I can change as
def __getitem__(self, index, sigma):

and to make use of the updated, newly learned σ.
My problem is that during training, I iterate through training dataset as
for epoch in range( checkpoint[ 'epoch'], num_epochs):
....
    for ii, ( X, y, y_weight, fname) in enumerate( dataLoader[ phase]):

In that enumeration of DataLoader, how can I pass the new σ to the DataLoader such that the DataLoader will pass it to the DataSet getitem function mentioned above?
EDIT
Currently, I define inside the DataSet class a parameter sigma
class MedicalImageDataset( Dataset):
      def __init__(self, fname, img_transform = None, mask_transform = None, weight_transform = None, sigma = 8):
            ...
            self.sigma = sigma

      def __getitem__(self, index):
            sigma = self.sigma
            ...

which I update through the DataLoader as
dataLoader[ 'train'].dataset.sigma = model.sigma

where,
model.sigma

is a custom parameter defined as
model.register_parameter( name = 'sigma', param = torch.nn.Parameter( torch.tensor( 16, dtype = torch.float16), requires_grad = True))

after creating the model.
My problem is, that model.sigma doesn't look being updated from epoch to epoch. Specifically, is the same as the initial value. Why is this?
Having a look at optimizer.state_dict() I couldn't find any parameter named 'sigma', whereas I can find one in model.named_parameters().
Finally, this parameter sigma is not attached to any layer, it's kinda "free".

Output:
What you need to do is to set sigma as an attribute of the Dataset and change it between epochs.

For the dataset definition

class UNetDataset(object):
    def __init__(self, ..., sigma=5):

        self.sigma = sigma

Now, within __getitem__, you can use the sigma value using self.sigma

Now within your training cycle, after every epoch, you can change the sigma value by setting the sigma attribute of the Dataset

for epoch in range(num_epochs):
    dataset.sigma = #whatever value you want

    for i,(x,y) in enumarate(DataLoader):


Instruction: How could I know whether a function in Pytorch allocates new memory or not?

Input: Recently, I got stuck in a situation that, in my model, the input data really consumed a lot of memory. And this lead to a lot of memory usage when I operate the data in my network layers. I really want to know whether the operations will allocate new memory block or not. I saw the pytorch doc only found how to use the function. I wonder is there a doc or some websites or anything else official to help me out. For example, will functions like view(), permute() or contiguous() allocate new memory block or not and how do you know that. It really caught me, thanks for helping.

Output: If you are running on a GPU, the best way to check memory consumption is to use the linux command nvidia-smi. You can call this in jupyter-notebook using !nvidia-smi. This way, after any Pytorch command, you can check if new memory has been allocated or not


Instruction: How can I get hidden_states from BertForSequenceClassification?

Input:
I read the official tutorial(https://huggingface.co/transformers/model_doc/bert.html) and tried to set config, but it doesn't work.

from transformers import PretrainedConfig
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
model.config.output_hidden_states = True
model.load_state_dict(torch.load('../parameter.pkl'))
model.cuda()
output = model(input)

Output: Output should be a list that holds the hidden states. I expect that because you are loading the parameter.pkl which may not have output hidden states by default, it is overwriting your config.output_hidden_states to False? See what happens if you set it to True after loading the state_dict?


Instruction: What is the correct way to measure the total execution time for a pytorch function running on GPU?

Input:
Following is an example code showing what I am trying to measure. Here I am using time.perf_counter() to measure time. Is this the correct way to measure execution time in this scenario? If not, what is the correct way? My concern is, GPU evaluations are asynchronous and GPU execution might not be completed when ExecTime is measured below.

import torch
import torch.nn.functional as F
import time

Device = torch.device("cuda:0")
ProblemSize = 100
NumChannels = 5
NumFilters = 96
ClassType = torch.float32

X = torch.rand(1, NumChannels, ProblemSize, ProblemSize, dtype=ClassType).to(Device)
weights = torch.rand(NumFilters, NumChannels, 10, 10, dtype=ClassType).to(Device)

#warm up
Y = F.conv2d(X, weights)
Y = F.conv2d(X, weights)

#time
t = time.perf_counter()
Y = F.conv2d(X, weights)
ExecTime = time.perf_counter() - t

Output: I think you are looking for pyotrch's bottleneck profiler.