Instruction: Randomness in VGG16 prediction

Input:
I am using the VGG-16 network available in pytorch out of the box to predict some image index. I found out that for same input file, if i predict multiple time, I get different outcome. This seems counter-intuitive to me. Once the weights are predicted ( since I am using the pretrained model) there should not be any randomness at any step, and hence multiple run with same input file shall return same prediction.

Here is my code:

import torch
import torchvision.models as models
VGG16 = models.vgg16(pretrained=True)
def VGG16_predict(img_path):
  transformer = transforms.Compose([transforms.CenterCrop(224),transforms.ToTensor()])
  data = transformer(Image.open(img_path))
  output = softmax(VGG16(data.unsqueeze(0)), dim=1).argmax().item()
  return output # predicted class index
VGG16_predict(image)

Here is the image

Output:
Recall that many modules have two states for training vs evaluation: "Some models use modules which have different training and evaluation behavior, such as batch normalization. To switch between these modes, use model.train() or model.eval() as appropriate. See train() or eval() for details." (https://pytorch.org/docs/stable/torchvision/models.html)

In this case, the classifier layers include dropout, which is stochastic during training. Run VGG16.eval() if you want the evaluations to be non-random.


Instruction: Passing tensorDataset or Dataloader to skorch

Input:
I want to apply cross validation in Pytorch using skorch, so I prepared my model and my tensorDataset which returns (image,caption and captions_length) and so it has X and Y, so I'll not be able to set Y in the method

net.fit(dataset)

but when I tried that I got an error :

  ValueError: Stratified CV requires explicitly passing a suitable y

Here's part of my code:

start = time.time()
net = NeuralNetClassifier(
        decoder, criterion= nn.CrossEntropyLoss,
        max_epochs=args.epochs,
        lr=args.lr,
        optimizer=optim.SGD,
        device='cuda',  # uncomment this to train with CUDA
       )
net.fit(dataset, y=None)
end = time.time()

Output:
You are (implicitly) using the internal CV split of skorch which uses a stratified split in case of the NeuralNetClassifier which in turn needs information about the labels beforehand.

When passing X and y to fit separately this works fine since y is accessible at all times. The problem is that you are using torch.dataset.Dataset which is lazy and does not give you access to y directly, hence the error.

Your options are the following.

Set train_split=None to disable the internal CV split

net = NeuralNetClassifier(
    train_split=None,
)

You will lose internal validation and, as such, features like early stopping.

Split your data beforehand

Split your dataset into two datasets, dataset_train and dataset_valid,
then use skorch.helper.predefined_split:

net = NeuralNetClassifier(
    train_split=predefined_split(dataset_valid),
)

You lose nothing but depending on your data this might be complicated.

Extract your y and pass it to fit

y_train = np.array([y for X, y in iter(my_dataset)])
net.fit(my_dataset, y=y_train)

This only works if your y fits into memory. Since you are using TensorDataset you can also do the following to extract your y:

y_train = my_dataset.y


Instruction: torch-1.1.0-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform

Input:
I need to use pyTorch when developing a RNN and whenever I try to install it I'm getting an error saying torch-1.1.0-cp37-cp37m-win_amd32.whl is not a supported wheel on this platform.

pip3 install https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-win_amd32.whl

This is how I tried to install it according to https://pytorch.org/.

I'm using,

Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32

Any help would be appreciated. Thanks in advance!

Output:
you are using anaconda you can install pytorch using

conda install pytorch-cpu torchvision-cpu -c pytorch

as written on the main pytorch page.


Instruction: Python __getattr__ called and enter infinite recursion even the attribute is not missing

Input:
I define a NE_Conv2d class and override its __getattr__. When I try to get an existing attribute from one instance of this class, __getattr__ is called and enter infinite recursion.

I know one need  to deal with __getattr__ carefully to avoid infinite recursion, but apparently the conv attribute already exists after __init__ is called. So when I try to get conv attribute, __getattr__ should not be called.

from torch import nn
class NE_Conv2d(nn.Module):
    '''Nonexpansive conv2d'''

    def __init__(self, *k, **kw):
        super(NE_Conv2d, self).__init__()
        self.conv = nn.Conv2d(*k, **kw)
        print('foo')

    def __getattr__(self, attr):
        return getattr(self.conv, attr)

a = NE_Conv2d(3, 32, 5)
print(a)
print(a.conv)

Above code should print info about a and a.conv, but enter infinite recursion when trying to get a.conv.

Output: I seem to know what's wrong... setattr and getattr of nn.Module is reloaded...


Instruction: I am getting error when importing torch and torch vision

Input:
I installed torch and torchvision using pip3 on MAC. When I imported the same, getting the below error.

Environment:

OS : macOS High Sierra
Python : 3.7
pip : 3
Pytorch 1.1

code:

import torch
import torchvision

Error:

  ImportError                               Traceback (most recent call
  last)  in
        1 import numpy as np
        2 from glob import glob
  ----> 3 import torch
        4 import torchvision
        5 from torchvision import datasets

  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/init.py
  in
       77 del _dl_flags
       78
  ---> 79 from torch._C import *
       80
       81 all += [name for name in dir(_C)

  ImportError:
  dlopen(/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/_C.cpython-37m-darwin.so,
  9): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
  Referenced from:
  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/lib/libshm.dylib
  Reason: image not found

Any help on solving this error would be appreciated.

Output:
i think

$ brew install libomp

can help u, cause i solve the same problem by it.

according

github-issue-"libomp.dylib can't be loaded"


Instruction: Loading original images besides the transformed ones using ImageFolder

Input:
I'm trying to train a GAN to colorize images. For that, I'm using ImageFolder of torchvision to load grayscale images but I also need the original data alongwith the transformed ones.
I want it in the fastest way as the data is large. I want to make ImageFolder load both at the same time to reduce the time complexity.
def load_data_bw(opt):
    datapath = '/content/gdrive/My Drive/faces/2003'

    dataset = torchvision.datasets.ImageFolder(datapath,
                       transform=transforms.Compose([
                                                    transforms.Grayscale(num_output_channels=3), #load images as grayscale with three channels
                                                    transforms.RandomChoice(
                                                       [transforms.Resize(opt['loadSize'], interpolation=1),
                                                        transforms.Resize(opt['loadSize'], interpolation=2),
                                                        transforms.Resize(opt['loadSize'], interpolation=3),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=1),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=2),
                                                        transforms.Resize((opt['loadSize'], opt['loadSize']),
                                                                          interpolation=3)]
                                                    ),
                                                    transforms.RandomChoice(
                                                       [transforms.RandomResizedCrop(opt['fineSize'], interpolation=1),
                                                        transforms.RandomResizedCrop(opt['fineSize'], interpolation=2),
                                                        transforms.RandomResizedCrop(opt['fineSize'], interpolation=3)]
                                                    ),
                                                    transforms.ColorJitter(brightness=0.1, contrast=0.1),
                                                    transforms.RandomHorizontalFlip(),
                                                    transforms.ToTensor()
                                               ]))
    return dataset

I'm expecting to get:
for iteration, orig_data, gray_data in enumerate(training_data_loader, 1):
    # code..

Output:
I assume you have 2 dataset variables i.e. dataset_bw and dataset_color that you can load as you mention using ImageFolder. Then you could do the following :

class GAN_dataset(Dataset):
    def __init__(self, dataset_bw, dataset_color):
        self.dataset1 = dataset_bw
        self.dataset2 = dataset_color

    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]

        return x1, x2

    def __len__(self):
        return len(self.dataset1)

dataset = GAN_dataset(dataset_bw, dataset_color)
loader = DataLoader(dataset, batch_size = ...)

This way you when you iterate through loader, you will get two images as you require.


Instruction: Implementing self attention

Input:
I am trying to implement self attention in Pytorch.
I need to calculate the following expressions.

Similarity function S (2 dimensional), P(2 dimensional), C'

S[i][j] = W1 * inp[i] + W2 * inp[j] + W3 * x1[i] * inp[j]

P[i][j] = e^(S[i][j]) / Sum for all j( e ^ (S[i]))

basically, P is a softmax function

C'[i] = Sum (for all j)  P[i][j] * x1[j]

I tried the following code using for loops

        for i in range(self.dim):
            for j in range(self.dim):
                S[i][j] = self.W1 * x1[i] + self.W2 * x1[j] + self.W3 * x1[i] * x1[j]

        for i in range(self.dim):
            for j in range(self.dim):
                P[i][j] = torch.exp(S[i][j]) / torch.sum( torch.exp(S[i]))

        # attend

        for i in range(self.dim):
            out[i] = 0
            for j in range(self.dim):
                out[i] += P[i][j] * x1[j]

Is there any faster way to implement this in Pytorch?

Output:
Here is an example of Self Attention I had implemented in Dual Attention for HSI Imagery

class PAM_Module(Module):
""" Position attention module  https://github.com/junfu1115/DANet/blob/master/encoding/nn/attention.py"""
#Ref from SAGAN
def __init__(self, in_dim):
    super(PAM_Module, self).__init__()
    self.chanel_in = in_dim

    self.query_conv = Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)
    self.key_conv = Conv2d(in_channels=in_dim, out_channels=in_dim//8, kernel_size=1)
    self.value_conv = Conv2d(in_channels=in_dim, out_channels=in_dim, kernel_size=1)

    self.gamma = Parameter(torch.zeros(1))

    self.softmax = Softmax(dim=-1)
def forward(self, x):
    """
        inputs :
            x : input feature maps( B X C X H X W)
        returns :
            out : attention value + input feature
            attention: B X (HxW) X (HxW)
    """
    m_batchsize, C, height, width = x.size()
    proj_query = self.query_conv(x).view(m_batchsize, -1, width*height).permute(0, 2, 1)
    proj_key = self.key_conv(x).view(m_batchsize, -1, width*height)
    energy = torch.bmm(proj_query, proj_key)
    attention = self.softmax(energy)
    proj_value = self.value_conv(x).view(m_batchsize, -1, width*height)

    out = torch.bmm(proj_value, attention.permute(0, 2, 1))
    out = out.view(m_batchsize, C, height, width)

    out = self.gamma*out + x
    #out = F.avg_pool2d(out, out.size()[2:4])

    return out


Instruction: Partial slices in pytorch / numpy with arbitrary and variable number of dimensions

Input:
Given an 2-dimensional tensor in numpy (or in pytorch), I can partially slice along all dimensions at once as follows:

>>> import numpy as np
>>> a = np.arange(2*3).reshape(2,3)
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a[1:,1:]
array([[ 5,  6,  7],
       [ 9, 10, 11]])

How can I achieve the same slicing pattern regardless of the number of dimensions in the tensor if I do not know the number of dimensions at implementation time? (i.e. I want a[1:] if a has only one dimension, a[1:,1:] for two dimensions, a[1:,1:,1:] for three dimensions, and so on)

It would be nice if I could do it in a single line of code like the following, but this is invalid:

a[(1:,) * len(a.shape)]  # SyntaxError: invalid syntax

I am specifically interested in a solution that works for pytorch tensors (just substitute torch for numpy above and the example is the same), but I figure it is likely and best if the solution works for both numpy and pytorch.

Output:
Answer:
Making a tuple of slice objects does the trick:

a[(slice(1,None),) * len(a.shape)]

Explanation:
slice is a builtin python class (not tied to numpy or pytorch) which provides an alternative to the subscript notation for describing slices.  The answer to a different question suggests using this as a way to store slice information in python variables. The python glossary points out that

  The bracket (subscript) notation uses slice objects internally.

Since the __getitem__ methods for numpy ndarrays and pytorch tensors support multi-dimensional indexing with slices, they must also support multidimensional indexing with slice objects, and so we can make a tuple of those slices that of the right length.

Btw, you can see how python uses the slice objects by creating a dummy class as follows and then do slicing on it:

class A(object):
    def __getitem__(self, ix):
        return ix

print(A()[5])  # 5
print(A()[1:])  # slice(1, None, None)
print(A()[1:,1:])  # (slice(1, None, None), slice(1, None, None))
print(A()[1:,slice(1,None)])  #  (slice(1, None, None), slice(1, None, None))


Instruction: Output of a CNN doesn't change much with the input

Input:
I have been trying to implement Actor Critic with a convolutional neural network. There are two different images as the state for the reinforcement learning agent. The output(actions for Actor) of the CNN is the (almost)same for different inputs after (random)initialisation. Due to this even after training the agent never learns anything useful.

State definitions(2 inputs):

Input 1: [1,1,90,128] image with maximum value for the pixel being 45.

Input 2: [1,1,45,80] image with maximum value for the pixel being 45.

Expected output by the actor: [x,y]: a 2-dimensional vector according to the state. Here x is expected in the range [0,160] and y is expected in the range [0,112]

Different types of modifications tried with the input:

1: Feed both the images as it is.

2: Feed both images with normalisation as (img/45) so that pixel values are from [0,1]

3:Feed both images with normalisation as 2*((img/45)-0.5) so that pixel values are from [-1,1]

4: Feed both images with normalisation as (img-mean)/std

Result: The output of the CNN remains almost same.

The code for the definition of actor has been given below.

import numpy as np
import pandas as pd
from tqdm import tqdm
import time
import cv2
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.nn.functional as F

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Actor(nn.Module):
    def __init__(self, action_dim, max_action):
        super(Actor,self).__init__()
        # state image [1,1,90,128]
        self.conv11 = nn.Conv2d(1,16,5)
        self.conv11_bn = nn.BatchNorm2d(16)
        self.conv12 = nn.Conv2d(16,16,5)
        self.conv12_bn = nn.BatchNorm2d(16)
        self.fc11 = nn.Linear(19*29*16,500)
        # dim image [1,1,45,80]
        self.conv21 = nn.Conv2d(1,16,5)
        self.conv21_bn = nn.BatchNorm2d(16)
        self.conv22 = nn.Conv2d(16,16,5)
        self.conv2_bn = nn.BatchNorm2d(16)
        self.fc21 = nn.Linear(8*17*16,250)
        # common pool
        self.pool  = nn.MaxPool2d(2,2)
        # after concatenation
        self.fc2 = nn.Linear(750,100)
        self.fc3 = nn.Linear(100,10)
        self.fc4 = nn.Linear(10,action_dim)
        self.max_action = max_action

    def forward(self,x,y):
        # state image
        x = self.conv11_bn(self.pool(F.relu(self.conv11(x))))
        x = self.conv11_bn(self.pool(F.relu(self.conv12(x))))
        x = x.view(-1,19*29*16)
        x = F.relu(self.fc11(x))
        # state dim
        y = self.conv11_bn(self.pool(F.relu(self.conv21(y))))
        y = self.conv11_bn(self.pool(F.relu(self.conv22(y))))
        y = y.view(-1,8*17*16)
        y = F.relu(self.fc21(y))
        # concatenate
        z = torch.cat((x,y),dim=1)
        z = F.relu(self.fc2(z))
        z = F.relu(self.fc3(z))
        z = self.max_action*torch.tanh(self.fc4(z))
        return z

# to read different sample states for testing
obs = []
for i in range(200):
    obs.append(np.load('eval_episodes/obs_'+str(i)+'.npy',allow_pickle=True))

obs = np.array(obs)

def tensor_from_numpy(state):
    # to add dimensions to tensor to make it [batch_size,channels,height,width]
    state_img = state
    state_img = torch.from_numpy(state_img).float()
    state_img = state_img[np.newaxis, :]
    state_img = state_img[np.newaxis, :].to(device)
    return state_img

actor = Actor(2,torch.FloatTensor([160,112]))
for i in range(20):
    a = tensor_from_numpy(obs[i][0])
    b = tensor_from_numpy(obs[i][2])
    print(actor(a,b))

Output of the above code:

tensor([[28.8616,  3.0934]], grad_fn=<MulBackward0>)
tensor([[27.4125,  3.2864]], grad_fn=<MulBackward0>)
tensor([[28.2210,  2.6859]], grad_fn=<MulBackward0>)
tensor([[27.6312,  3.9528]], grad_fn=<MulBackward0>)
tensor([[25.9290,  4.2942]], grad_fn=<MulBackward0>)
tensor([[26.9652,  4.5730]], grad_fn=<MulBackward0>)
tensor([[27.1342,  2.9612]], grad_fn=<MulBackward0>)
tensor([[27.6494,  4.2218]], grad_fn=<MulBackward0>)
tensor([[27.3122,  1.9945]], grad_fn=<MulBackward0>)
tensor([[29.6915,  1.9938]], grad_fn=<MulBackward0>)
tensor([[28.2001,  2.5967]], grad_fn=<MulBackward0>)
tensor([[26.8502,  4.4917]], grad_fn=<MulBackward0>)
tensor([[28.6489,  3.2022]], grad_fn=<MulBackward0>)
tensor([[28.1455,  2.7610]], grad_fn=<MulBackward0>)
tensor([[27.2369,  3.4243]], grad_fn=<MulBackward0>)
tensor([[25.9513,  5.3057]], grad_fn=<MulBackward0>)
tensor([[28.1400,  3.3242]], grad_fn=<MulBackward0>)
tensor([[28.2049,  2.6622]], grad_fn=<MulBackward0>)
tensor([[26.7446,  2.5966]], grad_fn=<MulBackward0>)
tensor([[25.3867,  5.0346]], grad_fn=<MulBackward0>)

The states(.npy) files can be found here
For different states the actions should vary from [0-160,0-112] but here the outputs just vary a little bit.

Note: the input images are initially sparse(lots of zeros in the image)

Is there a problem with the state pixel values or the network definition?

EDIT: I think the problem has got to do something with the normalisation or the sparsity of the inputs, because I also tried the same network with tensorflow and am facing the same problem there.

Output: The problem with this was that the weight initialization was not appropriate. I used Gaussian initialization with twice the standard deviation as of the default. This helped in giving different outputs for different inputs. Although after a few episodes of training the actor starts giving the same value once again, which is due the the critic network getting saturated.


Instruction: Batch-Matrix multiplication in Pytorch - Confused with the handling of the output's dimension

Input:
I got two arrays :

A
B

Array A contains a batch of RGB images, with shape:

[batch, Width, Height, 3]

whereas Array B contains coefficients needed for a "transformation-like" operation on images, with shape:

[batch, 4, 4, 3]

To put it simply, the operation for a single image is a multiplication that outputs an environment map (normalMap * Coefficients).

The output I want should hold shape:

[batch, Width, Height, 3]

I tried using torch.bmm but failed. Is this possible somehow?

Output:
I think you need to calculate that PyTorch works with
BxCxHxW : number of mini-batches, channels, height, width

format, and also use matmul, since bmm works with tensors or ndim/dim/rank =3.
I know you may find this online, but for any case:
batch1 = torch.randn(10, 3, 20, 10)
batch2 = torch.randn(10, 3, 10, 30)
res = torch.matmul(batch1, batch2)
res.size() # torch.Size([10, 3, 20, 30])


Instruction: No module named 'cupy' on Google Colab

Input:
I'm working on the FastPhotoStyle project:
https://github.com/NVIDIA/FastPhotoStyle

and I follow the steps of its tutorial:
https://github.com/NVIDIA/FastPhotoStyle/blob/master/TUTORIAL.md

I'm running Example 1 on Google Colab where default environment is

CUDA 10.0
Python 3.6
Chainer 5.4.0
CuPy 5.4.0

This is how I tried on Colab Notebook:

https://colab.research.google.com/drive/1oBgdJgXCLlUQhpwPoG1Uom3OKTzHR4BF

After running

!python3 demo.py --output_image_path results/example1.png

Here's the error message I got:

  Traceback (most recent call last):     File "demo.py", line 9, in

  import process_stylization

  File "/content/drive/FastPhotoStyle/process_stylization.py", line 14,
  in

  from smooth_filter import smooth_filter

  File "/content/drive/FastPhotoStyle/smooth_filter.py", line 327, in

  from cupy.cuda import function

  ModuleNotFoundError: No module named 'cupy'

Could someone help me with it please?

Output:
have you tried running the demo with python full path?
!/usr/local/lib/python3.6//bin/python3 demo.py --output_image_path

or running without anaconda installation by simple install all necessary libraries
!pip install cupy

or see installation instructions  here


Instruction: why does linear regression converge slower if we use two layers of linear module?

Input:
I have tested linear regression of 1d input and 1d output by pytorch using three different ways.

One is using the formula from linear algebra,
the other one is using nn.Linear(1,1). These two always give identical solution.
However, when I use two layers: nn.Linear(1,2) and nn.Linear(2,1) sequentially for the third approach, the result does not converge at first. After I set learning rate much smaller and epoch number much bigger, it finally shows its convergence.

In theory, because composition of linear transform is again linear transform, they shall give the same answer no matter one layer and two layers.
Intuitively, I thought more neurons and layers make things more efficient. But this shows its opposite and I do not understand.

The code is in github. Please directly jump in the last shell for the third approach.
The expected result is given in both first and second approaches in the notebook.

Output: This is not surprising. With 2 Linear layers which, as you know, effectively express what a single Linear layer could, you're introducing a bunch of redundant degrees of freedom - different assignments of values to the two layers, which result in the same effective transformation. The optimizer can therefore "walk around" different solutions, which look the same in terms of the loss function (because they mathematically are the same), without converging to a single one. In other words, you cannot converge to a solution if there is an infinite number of them, all looking the same to you.


Instruction: Need a vectorized solution in pytorch

Input:
I'm doing an experiment using face images in PyTorch framework.  The input x is the given face image of size 5 * 5 (height * width) and there are 192 channels.

Objective: To obtain patches of x of patch_size(given as argument).

I have obtained the required result with the help of two for loops. But I want a better-vectorized solution so that the computation cost will be very less than using two for loops.

Used: PyTorch 0.4.1, (12 GB) Nvidia TitanX GPU.

The following is my implementation using two for loops

def extractpatches( x, patch_size): # x is bsx192x5x5
   patches = x.unfold( 2, patch_size ,  1).unfold(3,patch_size,1)
   bs,c,pi,pj, _, _  = patches.size()  #bs,192,
   cnt = 0
   p = torch.empty((bs,pi*pj,c,patch_size,patch_size)).to(device)
   s = torch.empty((bs,pi*pj, c*patch_size*patch_size)).to(device)
  //Want a vectorized method instead of two for loops below
   for i in range(pi):
      for j in range(pj):
          p[:,cnt,:,:,:] = patches[:,:,i,j,:,:]
          s[:,cnt,:] = p[:,cnt,:,:,:].view(-1,c*patch_size*patch_size)
          cnt  = cnt+1
   return s

Thanks for your help in advance.

Output:
I think you can try this as following. I used some parts of your code for my experiment and it worked for me. Here l and f are the lists of tensor patches

l = [patches[:,:,int(i/pi),i%pi,:,:] for i in range(pi * pi)]
f = [l[i].contiguous().view(-1,c*patch_size*patch_size) for i in range(pi * pi)]

You can verify the above code using toy input values.
Thanks.


Instruction: Pytorch DataLoader fails when the number of examples are not exactly divided by the batch size

Input:
I have coded a custom data loader class in the pytorch. But it fails when iterating through all the number of batches inside an epoch. For example, think I have 100 data examples and my batch size is 9. It will fail in the 10th iteration saying batch size is different which will give a batch size 1 instead of 10. I have put my custom data loader in below.  Also I have put how I extract the data from the loader inside the for-loop.

class FlatDirectoryAudioDataset(tdata.Dataset): #customized dataloader

    def __init__(self, data_dir, transform=None):
        self.data_dir = data_dir
        self.transform = transform
        self.files = self.__setup_files()

    def __len__(self):
        """
        compute the length of the dataset
        :return: len => length of dataset
        """
        return len(self.files)

    def __setup_files(self):

        file_names = os.listdir(self.data_dir)
        files = []  # initialize to empty list

        for file_name in file_names:

            possible_file = os.path.join(self.data_dir, file_name)
            if os.path.isfile(possible_file) and (file_name.lower().endswith('.wav') or file_name.lower().endswith('.mp3')): #&& (possible_file.lower().endswith('.wav') or possible_file.lower().endswith('.mp3')):
                files.append(possible_file)

        # return the files list
        return files

    def __getitem__ (self,index):
        sample, _ = librosa.load(self.files[index], 16000)

        if self.transform:
            sample=self.transform(sample)

        sample = torch.from_numpy(sample)
        return sample

from torch.utils.data import DataLoader

    my_dataset=FlatDirectoryAudioDataset(source_directory,source_folder,source_label,transform = None,label=True)

dataloader_my = DataLoader(
        my_dataset,
        batch_size=batch_size,
        num_workers=0,
        shuffle=True)

for (i,batch) in enumerate(dataloader_my,0):
       print(i)
       if batch.shape[0]!=16:
          print(batch.shape)
          assert batch.shape[0]==16,"Something wrong with the batch size"

Output:
use drop_last=True utils.DataLoader(dataset,batch_size=batch_size,shuffle = True,drop_last=True)

https://pytorch.org/docs/stable/data.html


Instruction: Correct data loading, splitting and augmentation in Pytorch

Input:
The tutorial doesn't seem to explain how we should load, split and do proper augmentation.

Let's have a dataset consisting of cars and cats. The folder structure would be:

data
  cat
    0101.jpg
    0201.jpg
    ...
  dogs
    0101.jpg
    0201.jpg
    ...

At first, I loaded the dataset by datasets.ImageFolder function. Image Function has command "TRANSFORM" where we can set some augmentation commands, but we don't want to apply augmentation to test dataset! So let's stay with transform=None.

data = datasets.ImageFolder(root='data')

Apparently, we don't have folder structure train and test and therefore I assume a good approach would be to use split_dataset function

    train_size = int(split * len(data))
    test_size = len(data) - train_size
    train_dataset, test_dataset = torch.utils.data.random_split(data, [train_size, test_size])

Now let's load the data the following way.

    train_loader = torch.utils.data.DataLoader(train_dataset,
                                              batch_size=8,
                                              shuffle=True)
    test_loader = torch.utils.data.DataLoader(test_dataset,
                                              batch_size=8,
                                              shuffle=True)

How can I apply transformations (data augmentation) to the "train_loader" images?

Basically I need to: 1. load data from the folder structure explained above
2. split the data into test/train parts
3. apply augmentations on train part.

Output:
I am not sure if there is a recommended way of doing this, but this is how I would workaround this problem:

Given that torch.utils.data.random_split() returns Subset, we cannot (can we? not 100% sure here I double-checked, we cannot) exploit their inner datasets, because they are the same (the only diference is in the indices). In this context, I would implement a simple class to apply transformations, something like this:

from torch.utils.data import Dataset

class ApplyTransform(Dataset):
    """
    Apply transformations to a Dataset

    Arguments:
        dataset (Dataset): A Dataset that returns (sample, target)
        transform (callable, optional): A function/transform to be applied on the sample
        target_transform (callable, optional): A function/transform to be applied on the target

    """
    def __init__(self, dataset, transform=None, target_transform=None):
        self.dataset = dataset
        self.transform = transform
        self.target_transform = target_transform
        # yes, you don't need these 2 lines below :(
        if transform is None and target_transform is None:
            print("Am I a joke to you? :)")

    def __getitem__(self, idx):
        sample, target = self.dataset[idx]
        if self.transform is not None:
            sample = self.transform(sample)
        if self.target_transform is not None:
            target = self.target_transform(target)
        return sample, target

    def __len__(self):
        return len(self.dataset)

And then use it before passing the dataset to the dataloader:

import torchvision.transforms as transforms

train_transform = transforms.Compose([
    transforms.ToTensor(),
    # ...
])
train_dataset = ApplyTransform(train_dataset, transform=train_transform)

# continue with DataLoaders...


Instruction: Multiply all elements of PyTorch tensor

Input: I wanted to do something like this question in PyTorch i.e. multiply all elements with each other keeping a certain axis constant. Does PyTorch has any pre-defined function for this?

Output:
Yes. torch.prod. Use the dim parameter to tell which along which axis you want the product to be computed.

x = torch.randn((2, 2))
print(x)
print(torch.prod(x, 0)) # product along 0th axis

This prints

tensor([[-0.3661, 1.0693],
           [0.5144, 1.3489]])
tensor([-0.1883, 1.4424])


Instruction: How to install older version of pytorch

Input:
Following this https://pytorch.org/get-started/previous-versions/#via-pip

pip install torch==0.2.0_4 -f https://download.pytorch.org/whl/cpu/stable
Collecting torch==0.2.0_4
  Could not find a version that satisfies the requirement torch==0.2.0_4 (from versions: 0.1.2, 0.1.2.post1, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0)
No matching distribution found for torch==0.2.0_4

How to install older version of pytorch?

Output:
pip install torch==

Collecting torch==

  ERROR: Could not find a version that satisfies the requirement torch== (from versions: 0.1.2, 0.1.2.post1, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0)
      ERROR: No matching distribution found for torch==

This means that version 0.2 is not available

You can download the specific version (wheels seem available) and install it with pip install


Instruction: Why does training accuracy fall after reaching perfect training fit?

Input:
I am training a NN in pytorch on MNIST data. The model start well, improves, reaches good accuracy for both training and testing data, stabilizes for a while and then both the testing and training accuracy collapse, as shown on the following base results graph.

As for MNIST, I use 60000 training images, 10000 testing, training batch size 100 and learning rate 0.01. Neural network consist of two fully connected hidden layers, each with 100 nodes, nodes having ReLU activation functions. F.cross_entropy is used for loss and SGD for gradients calculations.

This is not the over-fitting problem, as it is both the training and testing accuracy which collapse. I suspected that it has something to do with too large learning rate. In the base case I have used 0.01 but when I lower it to 0.001 the whole pattern repeats, just later, as shown on the following graph (please notice x-axis scale change, the pattern is happening roughly 10 times later, which is intuitive). Similar results were obtained using even lower learning rates.

I have tried unit testing, checking the individual parts and making the model smaller. Here are results when I am using only 6 data points in the training set, batch size 2. Perfect fit on training data (here clearly different from the test accuracy, as expected) is unsurprisingly reached, but it still collapses from 100% to 1/6 so no better than a random pick. What needs to happen for the network to spin out from a perfect fit on the training set, can anybody tell me?

Here is the structure of the network (the relevant libraries are added before), although I hope that the aforementioned symptoms will be sufficient for you to recognize what the problem is without it:

class Network(nn.Module):
def __init__(self):
    # call to the super class Module from nn
    super(Network, self).__init__()

    # fc strand for 'fully connected'
    self.fc1 = nn.Linear(in_features=28*28, out_features=100)
    self.fc2 = nn.Linear(in_features=100, out_features=100)
    self.out = nn.Linear(in_features=100, out_features=10)

def forward(self, t):

    # (1) input layer (redundant)
    t = t

    # (2) hidden linear layer
    # As my t consists of 28*28 bit pictures, I need to flatten them:
    t = t.reshape(-1, 28*28)
    # Now having this reshaped input, add it to the linear layer
    t = self.fc1(t)
    # Again, apply ReLU as the activation function
    t = F.relu(t)

    # (3) hidden linear layer
    # As above, but reshaping is not needed now
    t = self.fc2(t)
    t = F.relu(t)

    # (4) output layer
    t = self.out(t)
    t = F.softmax(t, dim=1)

    return t

The main execution of the code:

for b in range(epochs):
print('***** EPOCH NO. ', b+1)
# getting a batch iterator
batch_iterator = iter(batch_train_loader)
# For loop for a single epoch, based on the length of the training set and the batch size
for a in range(round(train_size/b_size)):
    print(a+1)
    # get one batch for the iteration
    batch = next(batch_iterator)
    # decomposing a batch
    images, labels = batch[0].to(device), batch[1].to(device)
    # to get a prediction, as with individual layers, we need to equate it to the network with the samples as input:
    preds = network(images)
    # with the predictions, we will use F to get the loss as cross_entropy
    loss = F.cross_entropy(preds, labels)
    # function for counting the number of correct predictions
    get_num_correct(preds, labels))
    # calculate the gradients needed for update of weights
    loss.backward()
    # with the known gradients, we will update the weights according to stochastic gradient descent
    optimizer = optim.SGD(network.parameters(), lr=learning_rate)
    # with the known weights, step in the direction of correct estimation
    optimizer.step()
    # check if the whole data check should be performed (for taking full training/test data checks only in evenly spaced intervals on the log scale, pre-calculated later)
    if counter in X_log:
        # get the result on the whole train data and record them
        full_train_preds = network(full_train_images)
        full_train_loss = F.cross_entropy(full_train_preds, full_train_labels)
        # Record train loss
        a_train_loss.append(full_train_loss.item())
        # Get a proportion of correct estimates, to make them comparable between train and test data
        full_train_num_correct = get_num_correct(full_train_preds, full_train_labels)/train_size
        # Record train accuracy
        a_train_num_correct.append(full_train_num_correct)
        print('Correct predictions of the dataset:', full_train_num_correct)
        # Repeat for test predictions
        # get the results for the whole test data
        full_test_preds = network(full_test_images)
        full_test_loss = F.cross_entropy(full_test_preds, full_test_labels)
        a_test_loss.append(full_test_loss.item())
        full_test_num_correct = get_num_correct(full_test_preds, full_test_labels)/test_size
        a_test_num_correct.append(full_test_num_correct)
    # update counter
    counter = counter + 1

I have googled and checked here for answers to this questions but people either ask about over-fitting or their NNs do not increase accuracy on the training set at all (i.e. they simply don't work), not about finding a good training fit and then completely loosing it, also on the training set. I hope I did not post something obvious, I am relatively new to NN but I did my best to research the topic before posting it here, thank you for your help and understanding!

Output:
The cause is a bug in the code. We need to add optimizator.zero_grad() at the beginning of the training loop and create the optimizator before the outer training loop, i.e.

optimizator = optim.SGD(...)
for b in range(epochs):

Why do we need to call zero_grad() in PyTorch? explains why.


Instruction: Mat2cell matlab equivalent in tensorflow or pytorch

Input:
I have a square matrix and like to break it into some smaller matrices. For example, assume we have a matrix with the shape of [4,4] and would like to convert it into 4 smaller matrices with size [2,2].

input:

[9, 9, 9, 9,
 8, 8, 8, 8,
 7, 7, 7, 7,
 6, 6, 6, 6]

output:

[[9, 9  | [9, 9,
 8, 8]  | 8, 8],
 ---------------
 [7, 7  | [7, 7,
 6, 6]  | 6, 6]]

Output:
Given a tensor with the shape of 4*4 or 1*16 the easiest way to do this is by view function or reshape:

a = torch.tensor([9, 9, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, 6, 6, 6, 6])
# a = a.view(4,4)
a = a.view(2, 2, 2, 2)

# output:
tensor([[[[9, 9],
          [9, 9]],

         [[8, 8],
          [8, 8]]],

        [[[7, 7],
          [7, 7]],

         [[6, 6],
          [6, 6]]]])


Instruction: Partitioned matrix multiplication in tensorflow or pytorch

Input:
Assume I have matrices P with the size [4, 4] which partitioned (block) into 4 smaller matrices [2,2]. How can I efficiently multiply this block-matrix into another matrix (not partitioned matrix but smaller)?

Let's Assume our original matric is:

P = [ 1 1 2 2
      1 1 2 2
      3 3 4 4
      3 3 4 4]

Which split into submatrices:

P_1 = [1 1    , P_2 = [2 2  , P_3 = [3 3   P_4 = [4 4
       1 1]            2 2]          3 3]         4 4]

Now our P is:

P = [P_1 P_2
     P_3 p_4]

In the next step, I want to do element-wise multiplication between P and smaller matrices which its size is equal to number of sub-matrices:

P * [ 1 0   =   [P_1  0  = [1 1 0 0
      0 0 ]      0    0]    1 1 0 0
                            0 0 0 0
                            0 0 0 0]

Output:
Following is a general Tensorflow-based solution that works for input matrices p (large) and m (small) of arbitrary shapes as long as the sizes of p are divisible by the sizes of m on both axes.

def block_mul(p, m):
   p_x, p_y = p.shape
   m_x, m_y = m.shape
   m_4d = tf.reshape(m, (m_x, 1, m_y, 1))
   m_broadcasted = tf.broadcast_to(m_4d, (m_x, p_x // m_x, m_y, p_y // m_y))
   mp = tf.reshape(m_broadcasted, (p_x, p_y))
   return p * mp

Test:

import tensorflow as tf

tf.enable_eager_execution()

p = tf.reshape(tf.constant(range(36)), (6, 6))
m = tf.reshape(tf.constant(range(9)), (3, 3))
print(f"p:\n{p}\n")
print(f"m:\n{m}\n")
print(f"block_mul(p, m):\n{block_mul(p, m)}")

Output (Python 3.7.3, Tensorflow 1.13.1):

p:
[[ 0  1  2  3  4  5]
 [ 6  7  8  9 10 11]
 [12 13 14 15 16 17]
 [18 19 20 21 22 23]
 [24 25 26 27 28 29]
 [30 31 32 33 34 35]]

m:
[[0 1 2]
 [3 4 5]
 [6 7 8]]

block_mul(p, m):
[[  0   0   2   3   8  10]
 [  0   0   8   9  20  22]
 [ 36  39  56  60  80  85]
 [ 54  57  80  84 110 115]
 [144 150 182 189 224 232]
 [180 186 224 231 272 280]]

Another solution that uses implicit broadcasting is the following:

def block_mul2(p, m):
   p_x, p_y = p.shape
   m_x, m_y = m.shape
   p_4d = tf.reshape(p, (m_x, p_x // m_x, m_y, p_y // m_y))
   m_4d = tf.reshape(m, (m_x, 1, m_y, 1))
   return tf.reshape(p_4d * m_4d, (p_x, p_y))


Instruction: Create multiple images with different ratio PyTorch

Input:
I'm trying to perform some digit recognition using PyTorch. I have implemented a convolutional version of the sliding window with size 32x32. Which makes me able to identify digits of this range of size in a picture.

But now let's imagine I have an image of size 300x300 with a digit that occupies the whole image. I will never be able to identify it...

I have seen people saying that the image needs to be rescaled and resized. Meaning that I need to create various scaled versions of my initial image and then to feed my network with those "new" images.

Does anyone have any idea how I can perform that?

Here is a part of my code, if it can help..

# loading dataset
size=200
height=200
width= 300

transformer_svhn_test = transforms.Compose([
    transforms.Grayscale(3),
    transforms.Resize((height, width)),
    transforms.CenterCrop((size, size)),
    transforms.ToTensor(),
    transforms.Normalize([.5,.5,.5], [.5,.5,.5])
])

SVHN_test = SVHN_(train=False, transform=transformer_svhn_test)
SVHN_test_loader = DataLoader(SVHN_test, batch_size=batch_size, shuffle=False, num_workers=3)

#loading network
model = Network()
model.to(device)
model.load_state_dict(torch.load("digit_classifier_gray_scale_weighted.pth"))

# loading one image and feeding the model with it
image = next(iter(SVHN_test_loader))[0]
image_tensor = image.unsqueeze(0) # creating a single-image batch
image_tensor = image_tensor.to(device)

model.eval()
output = model(image_tensor)

Output:
Please correct me if I understand your question wrong:

Your network takes images with size of 300x300 as input, and does 32x32 sliding window operation within your model, and output the locations of any digits in input images? In this setup, you are framing this problem as an object detection task.

I am imaging the digits in your training data have sizes that are similar to 32x32, and you wanted to use multiple scale evaluation to make sure digits on your testing images will also have similar sizes as those in your training data. As for object detection network, the input size of your network is not fixed.

So the thing you need is actually called multi scale evaluation/testing, and you will find it very common in Computer Vision tasks.

A good starting point would be HERE


Instruction: NameError: name 'nn' is not defined

Input:
Several times when I copy PyTorch code I get this error:

NameError: name 'nn' is not defined

What is missing? What is nn?

To reproduce:

class SLL(nn.Module):
    "single linear layer"
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(10,100)

    def forward(self)->None:
        print("SLL:forward")

m1 = SLL()

Output:
If you got this error you can fix it with the following code:

import torch
import torch.nn as nn

You need to include both lines, since if you set just the second one it may not work if the torch package is not imported.

Where torch and torch.nn (or just nn) are two of the main PyTorch packages. You can help(torch.nn) to confirm this.

It is not uncommon when you include nn to include the functional interface as F like this:

import torch
import torch.nn as nn
import torch.nn.functional as F

To bring you the hints what you imported or what is inside the nn package I provided the list:

['AdaptiveAvgPool1d', 'AdaptiveAvgPool2d', 'AdaptiveAvgPool3d', 'AdaptiveLogSoftmaxWithLoss', 'AdaptiveMaxPool1d', 'AdaptiveMaxPool2d', 'AdaptiveMaxPool3d', 'AlphaDropout', 'AvgPool1d', 'AvgPool2d', 'AvgPool3d', 'BCELoss', 'BCEWithLogitsLoss', 'BatchNorm1d', 'BatchNorm2d', 'BatchNorm3d', 'Bilinear', 'CELU', 'CTCLoss', 'ConstantPad1d', 'ConstantPad2d', 'ConstantPad3d', 'Container', 'Conv1d', 'Conv2d', 'Conv3d', 'ConvTranspose1d', 'ConvTranspose2d', 'ConvTranspose3d', 'CosineEmbeddingLoss', 'CosineSimilarity', 'CrossEntropyLoss', 'CrossMapLRN2d', 'DataParallel', 'Dropout', 'Dropout2d', 'Dropout3d', 'ELU', 'Embedding', 'EmbeddingBag', 'FeatureAlphaDropout', 'Fold', 'FractionalMaxPool2d', 'GLU', 'GRU', 'GRUCell', 'GroupNorm', 'Hardshrink', 'Hardtanh', 'HingeEmbeddingLoss', 'InstanceNorm1d', 'InstanceNorm2d', 'InstanceNorm3d', 'KLDivLoss', 'L1Loss', 'LPPool1d', 'LPPool2d', 'LSTM', 'LSTMCell', 'LayerNorm', 'LeakyReLU', 'Linear', 'LocalResponseNorm', 'LogSigmoid', 'LogSoftmax', 'MSELoss', 'MarginRankingLoss', 'MaxPool1d', 'MaxPool2d', 'MaxPool3d', 'MaxUnpool1d', 'MaxUnpool2d', 'MaxUnpool3d', 'Module', 'ModuleDict', 'ModuleList', 'MultiLabelMarginLoss', 'MultiLabelSoftMarginLoss', 'MultiMarginLoss', 'NLLLoss', 'NLLLoss2d', 'PReLU', 'PairwiseDistance', 'Parameter', 'ParameterDict', 'ParameterList', 'PixelShuffle', 'PoissonNLLLoss', 'RNN', 'RNNBase', 'RNNCell', 'RNNCellBase', 'RReLU', 'ReLU', 'ReLU6', 'ReflectionPad1d', 'ReflectionPad2d', 'ReplicationPad1d', 'ReplicationPad2d', 'ReplicationPad3d', 'SELU', 'Sequential', 'Sigmoid', 'SmoothL1Loss', 'SoftMarginLoss', 'Softmax', 'Softmax2d', 'Softmin', 'Softplus', 'Softshrink', 'Softsign', 'Tanh', 'Tanhshrink', 'Threshold', 'TripletMarginLoss', 'Unfold', 'Upsample', 'UpsamplingBilinear2d', 'UpsamplingNearest2d', 'ZeroPad2d', '_VF', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_functions', '_reduction', 'backends', 'functional', 'grad', 'init', 'modules', 'parallel', 'parameter', 'utils']

Containing many classes where probable the most fundamental one is the PyTorch class nn.Module.

Do not confuse PyTorch class nn.Module with the Python modules.

To fix the SLL model from the question you just have to add the first two lines:

import torch
import torch.nn as nn

class SLL(nn.Module):
    "single linear layer"
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(10,100)

    def forward(self)->None:
        print("SLL:forward")

# we create a module instance m1
m1 = SLL()

And you will get the output:

SLL(
  (l1): Linear(in_features=10, out_features=100, bias=True)
)


Instruction: Efficient metrics evaluation in PyTorch

Input:
I am new to PyTorch and want to efficiently evaluate among others F1 during my Training and my Validation Loop.

So far, my approach was to calculate the predictions on GPU, then push them to CPU and append them to a vector for both Training and Validation. After Training and Validation, I would evaluate both for each epoch using sklearn. However, profiling my code it showed, that pushing to cpu is quite a bottleneck.

for epoch in range(n_epochs):
    model.train()
    avg_loss = 0
    avg_val_loss = 0
    train_pred = np.array([])
    val_pred = np.array([])
    # Training loop (transpose X_batch to fit pretrained (features, samples) style)
    for X_batch, y_batch in train_loader:
        scores = model(X_batch)
        y_pred = F.softmax(scores, dim=1)
        train_pred = np.append(train_pred, self.get_vector(y_pred.detach().cpu().numpy()))

        loss = loss_fn(scores, self.get_vector(y_batch))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)

    model.eval()
    # Validation loop
    for X_batch, y_batch in val_loader:
        with torch.no_grad():
            scores = model(X_batch)
            y_pred = F.softmax(scores, dim=1)
            val_pred = np.append(val_pred, self.get_vector(y_pred.detach().cpu().numpy()))
            loss = loss_fn(scores, self.get_vector(y_batch))
            avg_val_loss += loss.item() / len(val_loader)

    # Model Checkpoint for best validation f1
    val_f1 = self.calculate_metrics(train_targets[val_index], val_pred, f1_only=True)
    if val_f1 > best_val_f1:
        prev_best_val_f1 = best_val_f1
        best_val_f1 = val_f1
        torch.save(model.state_dict(), self.PATHS['xlm'])
        evaluated_epoch = epoch

    # Calc the metrics
    self.save_metrics(train_targets[train_index], train_pred, avg_loss, 'train')
    self.save_metrics(train_targets[val_index], val_pred, avg_val_loss, 'val')

I am certain there is a more efficient way to
a) store the predictions without having to push them to cpu each batch. b) calculate the metrics on GPU directly?

As I am new to PyTorch, I am very grateful for any hints and feedback :)

Output:
You can compute the F-score yourself in pytorch. The F1-score is defined for single-class (true/false) classification only. The only thing you need is to aggregating the number of:

Count of the class in the ground truth target data;
Count of the class in the predictions;
Count how many times the class was correctly predicted.

Let's assume you want to compute F1 score for the class with index 0 in your softmax. In every batch, you can do:

predicted_classes = torch.argmax(y_pred, dim=1) == 0
target_classes = self.get_vector(y_batch)
target_true += torch.sum(target_classes == 0).float()
predicted_true += torch.sum(predicted_classes).float()
correct_true += torch.sum(
    predicted_classes == target_classes * predicted_classes == 0).float()

When all batches are processed:

recall = correct_true / target_true
precision = correct_true / predicted_true
f1_score = 2 * precission * recall / (precision + recall)

Don't forget to take care of cases when precision and recall are zero and when then desired class was not predicted at all.


Instruction: do I need to initialize lstm hidden state when in validation and test set? or just reset it to zero

Input: On training, it is good to initialize hidden state instead of setting it  to 0. But I wonder whether initialize hidden state on validation and testing is good or bad. Thanks

Output:
There's absolutely no reason for custom initializing hidden states to zeros; this is actually the case:

def forward(self, input, hx=None):
    ...

    if hx is None:
        num_directions = 2 if self.bidirectional else 1
        hx = torch.zeros(self.num_layers * num_directions,
                         max_batch_size, self.hidden_size,
                         dtype=input.dtype, device=input.device)
    else:
        # Each batch of the hidden state should match the input sequence that
        # the user believes he/she is passing in.
        hx = self.permute_hidden(hx, sorted_indices)

It checks first if you have passed any custom hidden states values, if you did not, it initializes it to zeros.

Also, you usually, theoretically speaking, do not need to initialize the model's hidden states in the testing mode (either randomly or using predefined values).


Instruction: How to convert 35 classes of cityscapes dataset to 19 classes?

Input:
The following is a small snippet of my code. Using this, I can train my model called 'lolnet' on cityscapes dataset. But the dataset contains 35 classes/labels [0-34].

imports ***

trainloader = torch.utils.data.DataLoader(
    datasets.Cityscapes('/media/farshid/DataStore/temp/cityscapes/', split='train', mode='fine',
                    target_type='semantic', target_transform =trans,
                    transform=input_transform ), batch_size = batch_size, num_workers = 2)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net = lolNet()
criterion = CrossEntropyLoss2d()

net.to(device)
num_of_classes = 34

for epoch in range(int(0), 200000):

    lr = 0.0001

    for batch, data in enumerate(trainloader, 0):

        inputs, labels = data
        labels = labels.long()
        inputs, labels = inputs.to(device), labels.to(device)

        labels = labels.view([-1, ])

        optimizer = optim.Adam(net.parameters(), lr=lr)

        optimizer.zero_grad()
        outputs = net(inputs)

        outputs = outputs.view(-1, num_of_class)

        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        outputs = outputs.to('cpu')
        outputs = outputs.data.numpy()
        outputs = outputs.reshape([-1, num_of_class])

        mask = np.zeros([outputs.shape[0]])
        #
        for i in range(len(outputs)):
            mask[i] = np.argmax(outputs[i])

        mask = mask.reshape([-1, 1])

        IoU = jaccard_score(labels.to('cpu').data, mask, average='micro')

But I want to train my model only on the 19 classes. These 19 classes are found here . The labels to train for are stored as "ignoreInEval" = True. This pytorch Dataloader helper for this dataset doesnt provide any clue.

So my question is how can I train my model on the desired 19 classes of this dataset using pytorch's "datasets.Cityscapes" api.

Output:
It's been a time, but leaving an answer as can be useful for others:
Firstly create a mapping to 19 classes + background. Background is related to not so important classes with ignore flag as said here.
# Mapping of ignore categories and valid ones (numbered from 1-19)
    mapping_20 = {
        0: 0,
        1: 0,
        2: 0,
        3: 0,
        4: 0,
        5: 0,
        6: 0,
        7: 1,
        8: 2,
        9: 0,
        10: 0,
        11: 3,
        12: 4,
        13: 5,
        14: 0,
        15: 0,
        16: 0,
        17: 6,
        18: 0,
        19: 7,
        20: 8,
        21: 9,
        22: 10,
        23: 11,
        24: 12,
        25: 13,
        26: 14,
        27: 15,
        28: 16,
        29: 0,
        30: 0,
        31: 17,
        32: 18,
        33: 19,
        -1: 0
    }

Then for each label image (the gray images where each pixel contains a class, which has pattern "{city}__{number}_{number}_gtFine_labelIds.png") that you load for training, run function below.
It will convert each pixel according to mapping above and your label images (masks) will have now only 20 (19 classes + 1 background) different values, instead of 35.
def encode_labels(mask):
    label_mask = np.zeros_like(mask)
    for k in mapping_20:
        label_mask[mask == k] = mapping_20[k]
    return label_mask

Then you can train your model normally with these new number of classes.


Instruction: Most efficient way to prep data for pytorch lstm from pandas dataframe

Input:
I am trying to feed data into my lstm. I have the data across multiple csv's, so I created a generator to load it in. However, I am having some issues on prepping the data for my lstm.

I understand this code (I got it from the pytorch docs)

seq_len = 5
batch_size= 3
cols_num = 10
hidden_size=20
num_layers = 2

rnn = nn.LSTM(input_size=cols_num, hidden_size=hidden_size, num_layers=num_layers)

data = torch.randn(seq_len, batch_size, cols_num)
h0 = torch.randn(batch_size, seq_len, hidden_size)
c0 = torch.randn(batch_size, seq_len, hidden_size)
output, (hn, cn) = rnn(data)

However, I think my disconnect is with using actual data not the torch.randn().

This is my current generator:

def data_loader(batch_size, fp, dropcol, seq_len):
    while True:
        for f in fp:
            gc.collect()
            df=pd.read_csv(f)
            df=df.replace(np.nan, 0)
            df=df.drop(dropcol,1)
            df['minute'] = df['minute'].apply(lambda x: min_idx(x))
            row_count, col_count = df.shape
            encoder_input = []
            prev = 0
            for idx, b in enumerate(range(1, row_count)):
                end = prev + batch_size
                window = df.iloc[prev:end]
                prev = end - 1
                w = np.array(window, dtype='float64')
                if w.shape[0] != batch_size:  break
                encoder_input.append(w)
                if idx == seq_len:
                    w0 = encoder_input
                    encoder_input = []
                    yield w0

but I get errors when I run this:

loader = data_loader(batch_size=batch_size, fp=<list of csvs>, dropcol=idcol, seq_len=2)
lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers, batch_first=batch_first)

for batch in loader:
    b = torch.tensor(batch)
    output, hidden = lstm(b)

The error:
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

What is the error in how I am thinking? Also, how should I format an h0 or c0 from the data?

Output:
The error is not in how you are thinking, the error is in how Pytorch models accept input. The default data type created in a Pytorch tensor is torch.float64
where the default (and possibly only) data type accepted by models is torch.float32.

To fix this use:

b = torch.tensor(batch, dtype=torch.float32)

This will convert your inputs to torch.float32.


Instruction: What does "Gathers values along an axis specified by dim." mean?

Input:
I want to understand what does "Gathers values along an axis specified by dim." mean in the below code. How to structure the operation of function on data in my head. What is this function doing to data and how ?

Please refer this link https://pytorch.org/docs/stable/torch.html#torch.gather

torch.gather(input, dim, index, out=None, sparse_grad=False)

Gathers values along an axis specified by dim.

For a 3-D tensor the output is specified by:

out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2

Output:
Yes, it goes through the given dim (dimension) of the tensor, and collects into a new tensor the values specified by the index provided. So if I had a 1D tensor (is that allowed?) as

MyValues = torch.tensor([0,2,4,6,8])

and did

torch.gather(MyValues, 0, torch.tensor([0,1,3]))

I'd expect to return a 1D tensor containing [0,2,6]. i.e. the values located at positions 0, 1 and 3.

So it's just picking out the contents using the index tensor as a pointer to the locations of the contents to be extracted from the input tensor.

The dim is the dimension along which you want to index. So for 2D that'd give you the option of indexing by rows or columns, and you can extrapolate that out into as many dimensions as you like.


Instruction: Poor fits for simple 2D Gaussian processes in `GPyTorch`

Input:
I'm having a lot of difficulty fitting a simple 2-dimensional GP out-of-the box using GPyTorch. As you can see below, the fit is very poor, and does not improve much with either swapping out the RBF kernel for something like a Matern. The optimization does appear to converge, but not on anything sensible.

class GPRegressionModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(GPRegressionModel, self).__init__(train_x, train_y, likelihood)

        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(
                gpytorch.kernels.RBFKernel(ard_num_dims=2),
            )

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

Does anyone have good tutorial examples beyond the ones included in the docs?

Output:
I was running into similar issues when trying to fit high-dimensional Gaussian Processes.  A couple of suggestions (not sure if these will work):

Try using a ZeroMean, rather than a constant mean.  It could be that having more hyperparameters (constant mean hyperparameter values) could lead the -mll objective to a local minima, rather than to a global minima.  Using a different optimizer, e.g. lbfgs (which is second-order, rather than adam or sgd, which are both first-order) may help with this as well.

Try normalizing your input data using min-max normalization, and your targets using standard normal N(0,1) normalization.  In short, these normalization steps ensure that your data is consistent with the default priors for these GPR models.  For more info on this, check out this GitHub issue.

Try changing the learning rate for optimizing your hyperparameters, or the number of epochs you train this for.

If you see any issues with numerical precision (particularly after normalization, if you choose to use it), try changing your model and datasets to double, 64-bit precision by converting torch tensors to double-precision tensors with tensor.double().

Again, can't guarantee that these will fix the issues you're running into, but hopefully they help!
Here are some tutorials I've put together as well.


Instruction: How to select parameters for nn.Linear layer while training a CNN?

Input:
I am trying to train a CNN to classify images from the Fashion-MNIST data using Conv2d, Maxpool and Linear layers. I came across a code as mentioned below with in_features = 12*4*4 in nn.Linear layer.

Can I please get help on how to select an in_features parameter for nn.Linear layer?

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)

        self.fc1 = nn.Linear(in_features=12*4*4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)

Output:
Each example in the Fashion-MNIST dataset is a 28 x 28 grayscale image.

Input is 28 x 28
We do a 5 x 5 convolution without padding(since default padding=0) and stride=1(by default), so we lose 2 pixels at each side, we drop down to 24 x 24, i.e.,(28-5)/1 + 1
We then perform maxpooling operation with receptive field of 2 x 2, we cut each dimension by half, down to 12 x 12
We again do another 5 x 5 convolution without padding and stride=1, we drop down to 8 x 8, i.e.,(12-5)/1 + 1
Then, we perform another maxpooling operation, we drop down to 4 x 4

That's why, self.fc1 = nn.Linear(in_features=12*4*4, out_features=120). It's basically, n_features_conv * height * width, where height and width are 4 respectively and n_features_conv  is same as out_channels of the conv2D layer lying just above it.

Note if you change the size of input image, then you will have to perform the above calculations and adjust the first Linear layer accordingly.

Hope this helps you!


Instruction: Custom cross-entropy loss in pytorch

Input:
I have done a custom implementation of the pytorch cross-entropy loss function (as I need more flexibility to be introduced later). The model I intend to train with this will need a considerable amount of time to train and the resources available can't be used to merely test if the function is correct implementation. I have implemented vectorized implementation as it will be quicker to run.

Following is my code for the same:

def custom_cross(my_pred,true,batch_size=BATCH_SIZE):
    loss= -torch.mean(torch.sum(true.view(batch_size, -1) * torch.log(my_pred.view(batch_size, -1)), dim=1))
    return loss

I will really appreciate if you can suggest a more optimized implementation of the same or if I am making a mistake in the present one. The model will use a Nvidia Tesla K-80 to train.

Output:
If you need just cross entropy you can take the advantage PyTorch defined that.

import torch.nn.functional as F
loss_func = F.cross_entropy

  suggest a more optimized implementation

PyTorch has F. loss functions, but you can easily write your own using plain python.
PyTorch will create fast GPU or vectorized CPU code for your function automatically.

So, you may check the PyTorch original implementation but I think is this:

def log_softmax(x):
    return x - x.exp().sum(-1).log().unsqueeze(-1)

And here is the original implementation of cross entropy loss, now you may just alter:

nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

To something you need, and you have it.


Instruction: How to replace infs to avoid nan gradients in PyTorch

Input:
I need to compute log(1 + exp(x)) and then use automatic differentiation on it. But for too large x, it outputs inf because of the exponentiation:

>>> x = torch.tensor([0., 1., 100.], requires_grad=True)
>>> x.exp().log1p()
tensor([0.6931, 1.3133,    inf], grad_fn=<Log1PBackward>)

Since log(1 + exp(x)) ≈ x for large x, I thought I could replace the infs with x using torch.where. But when doing this, I still get nan for the gradient of too large values. Do you know why this happens and if there is another way to make it work?

>>> exp = x.exp()
>>> y = x.where(torch.isinf(exp), exp.log1p())  # Replace infs with x
>>> y  # No infs
tensor([  0.6931,   1.3133, 100.0000], grad_fn=<SWhereBackward>)
>>> y.sum().backward()  # Automatic differentiation
>>> x.grad  # Why is there a nan and how can I get rid of it?
tensor([0.5000, 0.7311,    nan])

Output:
A workaround I've found is to manually implement a Log1PlusExp function with its backward counterpart. Yet it does not explain the bad behavior of torch.where in the question.

>>> class Log1PlusExp(torch.autograd.Function):
...     """Implementation of x ↦ log(1 + exp(x))."""
...     @staticmethod
...     def forward(ctx, x):
...         exp = x.exp()
...         ctx.save_for_backward(x)
...         return x.where(torch.isinf(exp), exp.log1p())
...     @staticmethod
...     def backward(ctx, grad_output):
...         x, = ctx.saved_tensors
...         return grad_output / (1 + (-x).exp())
...
>>> log_1_plus_exp = Log1PlusExp.apply
>>> x = torch.tensor([0., 1., 100.], requires_grad=True)
>>> log_1_plus_exp(x)  # No infs
tensor([  0.6931,   1.3133, 100.0000], grad_fn=<Log1PlusExpBackward>)
>>> log_1_plus_exp(x).sum().backward()
>>> x.grad  # And no nans!
tensor([0.5000, 0.7311, 1.0000])


Instruction: CNTK: "inferred dimension cannot be calculated from input and new shape size."

Input:
I've set up a model for CIFAR-10 using Pytorch, and saved it as an ONNX file.

But it looks like I can't load it from CNTK.

I've already loaded another ONNX file from the same source code (by mistake), so the dependencies look OK. The problem occurs when I call Function.Load()

var deviceDescriptor = DeviceDescriptor.CPUDevice; ;
var function = Function.Load(ONNX_PATH, deviceDescriptor, ModelFormat.ONNX);

I get this exception (Unhandled exception):

   System.ApplicationException : 'Reshape: inferred dimension cannot be calculated from input and new shape size.

    [CALL STACK]
    - CNTK::TrainingParameterSchedule::  GetMinibatchSize
    - CNTK::  XavierInitializer (x6)
    - CNTK::Function::Load
    - CSharp_CNTK_Function__Load__SWIG_0
    - 00007FFB0C41C307 (SymFromAddr() error: Le module spÃ©cifiÃ© est introuvable.)

Output:
It looks like this model can't be loaded in CNTK. CNTK has good support for exporting (saving) to ONNX, importing (loading) can be problematic for some operations.

CNTK development is frozen, what's your motivation to use it?

The recommended way now is to use ONNX Runtime https://github.com/microsoft/onnxruntime for inference, it has first-class support for ONNX.


Instruction: Upsample ONNX gives INVALID_GRAPH when trying to convert bilinear layer to onnx

Input:
when i convert network with bilinear layer trained on Pytorch to ONNX, i get following error

  RuntimeError: [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model
  from test.onnx failed:Type Error: Type 'tensor(int64)' of input
  parameter (11) of operator (Floor) in node () is invalid.

I m not sure on why this error occurs, I tried building ONNX from source but still the issue dosen't seems to go way.

Any ideas on what might cause this error? or how to tackle the issue?

Way to Reproduce-

from torch import nn

import torch
import torch.nn.functional as F
import onnxruntime as rt

class Upsample(torch.nn.Module):
    def forward(self, x):
        #l = nn.Conv2d(3, 3, kernel_size=1, stride=1, padding=1, bias=True)
        return F.interpolate(x, scale_factor=2, mode="bilinear", align_corners=False)

m = Upsample()
v = torch.randn(1,3,128,128, dtype=torch.float32, requires_grad=False)

torch.onnx.export(m, v, "test.onnx")
sess = rt.InferenceSession("test.onnx")

Output:
This error has been fixed in https://github.com/pytorch/pytorch/pull/21434 (the fix is in functional.py), so you should be able to get it if you install pytorch's nightly build.

However, in this same PR, converting Upsample in bilinear mode has been disabled; the reason is that Pytorch's bilinear mode does not align with ONNX's, and Nearest mode is the only mode currently supported.

Upsample (Now called Resize) in ONNX is being updated in opset 11 to support a bilinear mode that aligns with Pytorch in https://github.com/onnx/onnx/pull/2057, but this is not yet pushed.


Instruction: Equivalence of slicing tensor in Pytorch/ATen C++

Input:
In Python given a 2-D tensor, we can use tensor[:,:2] to slice the a 2x2 matrix of the first two elements in the top left of the matrix, e.g. :

x = torch.tensor([[-1.4673,  0.9980, -2.1427, -1.1798, -0.0646, -0.2635, -2.8930, -0.2563,
          0.4559, -0.7947, -0.4540,  3.3224,  0.2295,  5.5568, -8.0451, -2.4529,
          4.8724,  2.1640,  3.3255,  0.6693, -1.2362,  4.4713, -3.5547, -0.0528,
          0.1031, -1.2472, -1.6014,  1.8134],
        [ 2.1636, -1.1497, -5.0298,  2.8261, -0.5684,  0.6389,  2.9009, -5.1609,
          1.7358, -3.1819, -0.9877,  5.5598,  6.7142,  4.5704, -1.2683, -5.3046,
          3.0454,  3.2757, -3.2541,  3.6619, -3.6391, -0.2002,  5.7175,  5.7130,
          0.6632, -0.0744, -0.3502,  4.8116]])

y, z = x[:,:2].chunk(2,1)

print(y)

print(z)

[out]:

tensor([[-1.4673],
        [ 2.1636]])
tensor([[ 0.9980],
        [-1.1497]])

What is right way to do it in C++ for PyTorch's ATen particularly?

For e.g. in the LSTM, there is the gate.chunk(4,1) function at https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp#L253

If I want to do a gate[:,:2].chunk(2,1) to extract different parts of the gates, e.g. auto partial_gates = gates[:,:2].chunk(4, 1);, how can it be done?

template <typename cell_params>
struct LSTMCell : Cell<std::tuple<Tensor, Tensor>, cell_params> {
  using hidden_type = std::tuple<Tensor, Tensor>;
  hidden_type operator()(const Tensor& input, const hidden_type& hidden, const cell_params& params) const override {
    auto hx = std::get<0>(hidden);
    auto cx = std::get<1>(hidden);

    if (input.is_cuda()) {
      auto igates = params.matmul_ih(input);
      auto hgates = params.matmul_hh(hx);
      auto result = at::_thnn_fused_lstm_cell(igates, hgates, cx, params.b_ih, params.b_hh);
      // Slice off the workspace argument (it's needed only for AD).
      return std::make_tuple(std::get<0>(result), std::get<1>(result));
    }

    auto gates = params.linear_ih(input) + params.linear_hh(hx);
    auto chunked_gates = gates.chunk(4, 1);

    auto partial_gates = gates[:,:2].chunk(4, 1);

    auto ingate = chunked_gates[0].sigmoid();
    auto forgetgate = chunked_gates[1].sigmoid();
    auto cellgate = chunked_gates[2].tanh();
    auto outgate = chunked_gates[3].sigmoid();

    auto cy = (forgetgate * cx) + (ingate * cellgate);
    auto hy = outgate * cy.tanh();

    return std::make_tuple(hy, cy);
  }
};

Output:
It's .narrow() from https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/TensorShape.cpp#L364

auto partial_gates = gates.narrow(1,0,2).chunk(4, 1);


Instruction: Expected object of backend CPU but got backend CUDA for argument #2 'source'

Input:
I have tried other answers but the error is not getting removed. Difference from the other question I am getting is that the last term used in error is 'source' which I did not find in any question. If possible please also explain the term 'source' in error. And running code without CPU is working fine.

  I am using Google Colab with GPU enabled.

import torch
from torch import nn
import syft as sy

hook = sy.TorchHook(torch)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = nn.Sequential(nn.Linear(784,256),
                     nn.ReLU(),
                     nn.Linear(256,128),
                     nn.ReLU(),
                     nn.Linear(128,64),
                     nn.ReLU(),
                     nn.Linear(64,10),
                     nn.LogSoftmax(dim = 1))

model = model.to(device)

output :

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-42-136ec343040a> in <module>()
      8                      nn.LogSoftmax(dim = 1))
      9
---> 10 model = model.to(device)

3 frames
/usr/local/lib/python3.6/dist-packages/syft/frameworks/torch/hook/hook.py in data(self, new_data)
    368
    369                 with torch.no_grad():
--> 370                     self.set_(new_data)
    371             return self
    372

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'source'

Output:
This issue is related to PySyft. As you can see in this Issue#1893, the current workaround is to set:

import torch
torch.set_default_tensor_type(torch.cuda.FloatTensor)

right after import torch.

Code:

import torch
from torch import nn
torch.set_default_tensor_type(torch.cuda.FloatTensor)  # <-- workaround

import syft as sy
hook = sy.TorchHook(torch)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

model = nn.Sequential(nn.Linear(784,256),
                     nn.ReLU(),
                     nn.Linear(256,128),
                     nn.ReLU(),
                     nn.Linear(128,64),
                     nn.ReLU(),
                     nn.Linear(64,10),
                     nn.LogSoftmax(dim = 1))

model = model.to(device)
print(model)

Output:

cuda
Sequential(
  (0): Linear(in_features=784, out_features=256, bias=True)
  (1): ReLU()
  (2): Linear(in_features=256, out_features=128, bias=True)
  (3): ReLU()
  (4): Linear(in_features=128, out_features=64, bias=True)
  (5): ReLU()
  (6): Linear(in_features=64, out_features=10, bias=True)
  (7): LogSoftmax()
)


Instruction: Bert-multilingual in pytorch

Input:
I am using bert embedding for french text data. and I have problem with loading model and vocabulary.

I used the following code for tokenization that works well, but to get the vocabulary, it gives me Chinese words!!

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
text = "La Banque Nationale du Canada fête cette année le 110e anniversaire de son bureau de Paris."
marked_text = "[CLS] " + text + " [SEP]"
tokenized_text = tokenizer.tokenize(marked_text)
list(tokenizer.vocab.keys())[5000:5020]

I expected french words in vocabulary but i get chinese words, should i specify the language somewhere in the code?

Output:
You are Getting Chinese text because, you are looking for a specific range of the words from the vocabulary [5000:5020], which corresponds to the Chinese text. Also,bert -base-multilingual-cased is trained on 104 languages.

If you further want to verify your code, you can use this:

tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
text = "La Banque Nationale du Canada fête cette année le 110e anniversaire de son bureau de Paris."
marked_text = "[CLS] " + text + " [SEP]"
tokenized_text = tokenizer.tokenize(marked_text)

which is same as your code, followed by:

token_no=[]
for token in tokenized_text:
    #print(tokenizer.vocab[token]) ### you can use this to check the corresponding index of the token
    token_no.append(tokenizer.vocab[token])

### The below code obtains the tokens from the index, which is similar to what you were trying, but on the correct range.
new_token_list=[]
for i in token_no:
    new_token_list.append(list(tokenizer.vocab.keys())[i])

#print(new_token_list); ### you can use it if you want to check back the tokens.


Instruction: Difference between Parameter vs. Tensor in PyTorch

Input:
I would like to know the difference between PyTorch Parameter and Tensor?

The existing answer is for the old PyTorch where variables are being used?

Output:
This is the whole idea of the Parameter class (attached) in a single image.

Since it is sub-classed from Tensor it is a Tensor.

But there is a trick. Parameters that are inside of a module are added to the list of Module parameters. If m is your module m.parameters() will hold your parameter.

Here is the example:

class M(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(2, 2))
        self.bias = nn.Parameter(torch.zeros(2))

    def forward(self, x):
        return x @ self.weights + self.bias

m=M()
m.parameters()
list(m.parameters())

---

[Parameter containing:
 tensor([[ 0.5527,  0.7096],
         [-0.2345, -1.2346]], requires_grad=True), Parameter containing:
 tensor([0., 0.], requires_grad=True)]

You see how the parameters will show what we defined.
And if we just add a tensor inside a class, like self.t = Tensor, it will not show in the parameters list. That is literally it. Nothing fancy.


Instruction: PyTorch Total CUDA time

Input:
Autograd profiler is a handy tool to measure the execution time in PyTorch as it is shown in what follows:

import torch
import torchvision.models as models

model = models.densenet121(pretrained=True)
x = torch.randn((1, 3, 224, 224), requires_grad=True)

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    model(x)
print(prof)

The output looks like this:

-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                                        CPU time        CUDA time            Calls        CPU total       CUDA total
-----------------------------------  ---------------  ---------------  ---------------  ---------------  ---------------
conv2d                                    9976.544us       9972.736us                1       9976.544us       9972.736us
convolution                               9958.778us       9958.400us                1       9958.778us       9958.400us
_convolution                              9946.712us       9947.136us                1       9946.712us       9947.136us
contiguous                                   6.692us          6.976us                1          6.692us          6.976us
empty                                       11.927us         12.032us                1         11.927us         12.032us

Which will include many lines. My questions are:

1) How can I use autograd profiler to get the entire CUDA time? (i.e., sum of CUDA time column)

2) Is there any solution to use it pragmatically? For example, prof[0].CUDA_Time?

Output:
[item.cuda_time for item in prof.function_events]

will give you a list of CUDA times. Modify it depending on your needs. To get the sum of CUDA times for example:

sum([item.cuda_time for item in prof.function_events])

Be careful though, the times in the list are in microseconds, while they are displayed in milliseconds in print(prof).


Instruction: How to impove the speed of tf.data.experimental.CsvDataset in tensorflow 1.13.1?

Input:
I am coding a toy example for csv data in Tensorflow.
I have implemented three types of data loader in tensorflow and pytorch to compare their speeds. Here is the code:

First, with tensorflow api tf.data.experimental.CsvDataset:

def parse_data(x, n_classes):
    x = tf.convert_to_tensor(x)
    return x[:-1], tf.one_hot(indices=tf.cast(x[-1], tf.int32), depth=n_classes)

if __name__=='__main__':
    dataset_train = tf.data.experimental.CsvDataset('/home/david/Dataset/timit/test.csv', [tf.float32] * 430,
                                                    header=False,
                                                    field_delim=' ')
    dataset_train = dataset_train.map(lambda *x_: parse_data(x_, 1928))
    dataset_train = dataset_train.batch(128)
    dataset_train = dataset_train.prefetch(1)
    iterator = dataset_train.make_initializable_iterator()

    x_in, y = iterator.get_next()

    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x_in)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    logits = tf.layers.Dense(units=1928, activation=None)(x)

    loss = tf.losses.softmax_cross_entropy(y, logits)
    optimizer = tf.train.AdamOptimizer()
    optimizer.minimize(loss)

    sess = tf.Session()
    sess.run(tf.global_variables_initializer())
    sess.run(iterator.initializer)
    running_loss = 0.0
    time_last = time.time()
    epoch = 0
    i = 0
    while True:
        try:
            running_loss += sess.run(loss)  # , feed_dict={x: data, y: labels})
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
            i += 1
        except tf.errors.OutOfRangeError:
            pass

Second, using pandas and tf.placeholder:

if __name__ == '__main__':
    x_in = tf.placeholder(shape=[128, 429], dtype=tf.float32)
    y_in = tf.placeholder(shape=[128], dtype=tf.int32)
    y = tf.one_hot(y_in, depth=1928)

    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x_in)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    x = tf.layers.Dense(units=1024, activation=tf.nn.relu)(x)
    logits = tf.layers.Dense(units=1928, activation=None)(x)

    loss = tf.losses.softmax_cross_entropy(y, logits)
    optimizer = tf.train.AdamOptimizer()
    optimizer.minimize(loss)

    sess = tf.Session()
    sess.run(tf.global_variables_initializer())

    w = pd.read_csv('/home/david/Dataset/timit/test.csv', header=None, delim_whitespace=True).values

    for epoch in range(23):
        running_loss = 0.0
        time_last = time.time()
        i = 0
        indexes = np.random.permutation(w.shape[0])
        w_ = w[indexes, :]
        while True:
            if i * 128 + 128 > w.shape[0]:
                break
            running_loss += sess.run(loss,
                                     feed_dict={x_in: w_[i * 128:i * 128 + 128, :-1],
                                                y_in: w_[i * 128:i * 128 + 128, -1]})
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
            i += 1

Third, with pytorch and pandas:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(429, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 1024)
        self.fc4 = nn.Linear(1024, 1024)
        self.fc5 = nn.Linear(1024, 1928)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x)
        return x

class CsvDataset(data.Dataset):
    """Face Landmarks dataset."""

    def __init__(self, csv_file):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file, header=None, delim_whitespace=True)

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        landmarks = self.landmarks_frame.values[idx, :]
        return landmarks[:-1], landmarks[-1]

if __name__ == '__main__':
    net = Net()
    device = torch.device('cuda:0')
    print(device)
    net.to(device)

    optimizer = optim.Adam(net.parameters(), lr=0.001)
    criterion = nn.CrossEntropyLoss()

    dataset = CsvDataset('/home/david/Dataset/timit/train.csv')
    trainloader = torch.utils.data.DataLoader(dataset, batch_size=128, shuffle=True)

    for epoch in range(23):

        running_loss = 0.0
        time_last = time.time()
        for i, data in enumerate(trainloader):
            inputs, labels = data
            inputs = inputs.float().to(device)
            labels = labels.long().to(device)

            optimizer.zero_grad()

            output = net(inputs)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            if (i + 1) % 5 == 0:
                print('\r[epoch: %2d, batch: %5d, time: %5f] loss: %.3f' % (
                    epoch + 1, i + 1, time.time() - time_last, running_loss / i), end=' ')
                time_last = time.time()
        print('')
    print('Finished Training')

I record the time cost for training in five batches:

First, CsvDataset: 1.382647s
Second, tf.placeholder: 0.013263s
Third, pytorch: 0.042086s

I guess it is partly because that tf.data.experimental.CsvDataset do io operations before each batch to extract data from csv file (Is this ture, or there are other reasons?).

However, it is too slow comparing to the other two. Is there any chance for improvement? How can I set the tf.data.experimental.CsvDataset api to load all csv data at the very beginning?

Or can I say that tf.data.experimental.CsvDataset is implemented only for the csv dataset that is too big to store in the memory? Because the time cost seems like intolerable.

Output:
You may play with the batch size in the first example, and if it reads batches from file every time you can prove it if you make it 2x bigger, you may expect 2x speed improvement. I haven't played with (experimental) class CsvDataset  in TF.

I am sure Pandas reads your document faster and this is part of the reason why you have these times.

Probable the next step you should unset the loss function nn.CrossEntropyLoss(). Most probable have the regression problem and not the classification problem judging by float labels you have at the end.

So try torch.nn.MSELoss as the loss function.


Instruction: PyTorch: Expected input batch_size (12) to match target batch_size (64)

Input:
I tried out PyTorch and wanted to write a program for MNIST. But, I got the error message:

  Expected input batch_size (12) to match target batch_size (64)

I searched for a solution but I don't understand what's wrong with my code.

#kwargs is empty because I don't use cuda
kwargs = {}
train_data = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=True, download=True,
                    transform=transforms.Compose([transforms.ToTensor(),
                    transforms.Normalize((0.1307,),(0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(
    datasets.MNIST('data', train=False,
                    transform=transforms.Compose([transforms.ToTensor(),
                    transforms.Normalize((0.1307,),(0.3081,))])),
    batch_size=64, shuffle=True, **kwargs)

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1,10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 2)
        x = F.relu(x)
        print(x.shape)
        x = x.view(-1, 320)
        x = self.fc1(x)
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=0)

model = Netz()

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)
def train(epoch):
    model.train()

    for batch_id, (data, target) in enumerate(train_data):
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        print(out.shape)
        criterion = nn.CrossEntropyLoss()
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'. format(
            epoch, batch_id * len(data), len(train_data.dataset),
            100. * batch_id / len(train_data), loss.data[0]))

The output should show the epoch and some other information. Actually, I print out the shape of my tensor but I don't know what's wrong. Here is the error message:

/home/michael/Programmierung/Python/PyTorch/venv/bin/python /home/michael/Programmierung/Python/PyTorch/mnist.py
torch.Size([64, 20, 4, 4])
torch.Size([12, 10])
Traceback (most recent call last):
  File "/home/michael/Programmierung/Python/PyTorch/mnist.py", line 69, in <module>
    train(epoch)
  File "/home/michael/Programmierung/Python/PyTorch/mnist.py", line 60, in train
    loss = criterion(out, target)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 942, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 2056, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/michael/Programmierung/Python/PyTorch/venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1869, in nll_loss
    .format(input.size(0), target.size(0)))
ValueError: Expected input batch_size (12) to match target batch_size (64).

Process finished with exit code 1

Output:
The error occurs because your model output, out, has shape (12, 10), while your target has a length of 64.

Since you are using a batch size of 64 and predicting the probabilities of 10 classes, you would expect your model output to be of shape (64, 10), so clearly there is something amiss in the forward() method.

Going through it line by line and noting the size of x at every step, we can try to find out what is going wrong:

    ...
    # x.shape = (64, 20, 4, 4) at this point as seen in your print statement
    x = x.view(-1, 320)             # x.shape = (64, 320)
    x = self.fc1(x)                 # x.shape = (64, 60)
    x = x.view(-1, 320)             # x.shape = (12, 320)
    x = F.relu(self.fc1(x))         # x.shape = (12, 60)
    x = self.fc2(x)                 # x.shape = (12, 10)
    return F.log_softmax(x, dim=0)  # x.shape = (12, 10)

What you actually most likely want is:

    ...
    # x.shape = (64, 20, 4, 4) at this point as seen in your print statement
    x = x.view(-1, 320)             # x.shape = (64, 320)
    x = F.relu(self.fc1(x))         # x.shape = (64, 60)
    x = self.fc2(x)                 # x.shape = (64, 10)
    return F.log_softmax(x, dim=1)  # x.shape = (64, 10)

Note: While not related to the error, note also that you want to softmax over dim=1 since that is the dimension that contains the logits for the classes.


Instruction: Why would we use to() method in pytorch?

Input: I've seen this method multiple times. What are the purposes and advantages of doing this?

Output:
  Why would we use to(device) method in pytorch?

torch.Tensor.to is multipurpose method.

Not only you can do type conversion, but it can also do CPU to GPU tensor move and GPU to CPU tensor move:

tensor = torch.randn(2, 2)
print(tensor)
tensor = tensor.to(torch.float64)
print(tensor) #dtype=torch.float64
tensor = tensor.to("cuda")
print(tensor) #device='cuda:0', dtype=torch.float64)
tensor = tensor.to("cpu")
print(tensor) #dtype=torch.float64
tensor = tensor.to(torch.float32)
print(tensor) # won't print dtype=torch.float32 since it is by default

Since CPU and GPU are different kind memories, there must be a way they communicate.
This is why we have to("cuda"), and to("cpu") that we call on tensor.

Usually when you load training datasets (images):

you download them from URL (like MNIST http://deeplearning.net/data/mnist/mnist.pkl.gz)
unpack them
convert them to numpy arrays
convert numpy arrays to tensors (as this is fast)
move them to GPU for training .to("cuda")

You can create tensors and move them to GPU like this.

torch.zeros(1000).to("cuda")

But there is a trick, sometimes you can even load them directly to GPU without messing the CPU.

torch.zeros(1000, device="gpu")


Instruction: Output hidden state in OpenNMT-py

Input: I just have a short question regarding the pytorch version of OpenNMT. There does not seem to be an option to return the hidden state of encoder and decoder in the options. Am I missing a flag or is this not an option in OpenNMT-py?

Output:
What do you mean by the encoder and decoder does not return hidden state?

If you see the RNNEncoder, it returns encoder_final, memory_bank, lengths where the memory_bank represents the hidden state which is of shape seq_len x batch_size x hidden_size. And the encoder_final is in general used by the decoder in a sequence-to-sequence model.

Now, let's see the RNNDecoder. As we see, the forward() method returns a FlaotTensor and a dictionary of FlaotTensors.

(FloatTensor, dict[str, FloatTensor]):
* dec_outs: output from the decoder (after attn)
  ``(tgt_len, batch, hidden)``.
* attns: distribution over src at each tgt
  ``(tgt_len, batch, src_len)``.

Usually, we use the dec_outs in sequence-to-sequence tasks. For example, in natural language generation tasks, we feed the dec_outs to a softmax layer to predict tokens.

There are several other types of encoders/decoders which you can explore from the following two links.

https://github.com/OpenNMT/OpenNMT-py/tree/master/onmt/encoders
https://github.com/OpenNMT/OpenNMT-py/tree/master/onmt/decoders


Instruction: Can I obtain the expression of derivation by using Pytorch or Tensorflow?

Input:
My question is whether I could obtain the expression after I compute the derivation by using PyTorch or TensorFlow. For better understanding my question I provide an example below:

I have a function (x1, x2, and x3 are all the decision variables)

y = x1 + x1*x2 + x1*x3

The derivation of y in terms of x1 should be:

1 + x2 + x3

My question is, if I use PyTorch or TensorFlow do the computation, could I print out the expression of "1 + x2 + x3"? Very much appreciate if the codes are provided no matter by using PyTorch or TensorFlow.

Output:
I don't know if it's possible in Pytorch or Tensorflow, but you can definitely do it using SymPy python library:

import sympy
x1 = sympy.symbols('x1')
x2 = sympy.symbols('x2')
x3 = sympy.symbols('x3')

y = x1 + x1*x2 + x1*x3
res = sympy.diff(y, x1)

print(res)

output:

x2 + x3 + 1


Instruction: Best algorithm for multi agent continuous space path finding using Reinforcement learning

Input: I am working on project in which I need to find best optimised path from 1 point to another in continuous space in multi agent scenario.  I am looking for best algorithm which suits this problem using Reinforcement learning. I have tried "Multi-agent actor-critic for mixed cooperative-competitive environment" but it does not seems to reach goals in 10000 epesidoes.  How can I improve this algorithm or is there any other algorithm that can help me with this.

Output:
Multi-agent reinforcement learning is quite hard to master and has yet to prove effective for general cases.

The problem is that in multi-agent the environment becomes non-stationary from the perspective of each individual agent. This means that an agents action cannot be mapped to the state directly because other agents are performing action seperately, which "confuse" all of the agents.
There is an in-depth collection of multi-agent research here: https://github.com/LantaoYu/MARL-Papers

If you would like you to pursue the actor-critic method you mentioned, I recommend this for you further research: https://arxiv.org/pdf/1706.02275.pdf if you would like to perfect Multi-Agent Actor Critic (MADDPG)