Instruction: Measuring uncertainty using MC Dropout on pytorch

Input:
I am trying to implement Bayesian CNN using Mc Dropout on Pytorch,
the main idea is that by applying dropout at test time and running over many forward passes , you get predictions from a variety of different models.
I’ve found an application of the Mc Dropout and I really did not get how they applied this method and how exactly they did choose the correct prediction from the list of predictions
here is the code

 def mcdropout_test(model):
    model.train()
    test_loss = 0
    correct = 0
    T = 100
    for data, target in test_loader:
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data, volatile=True), Variable(target)
        output_list = []
        for i in xrange(T):
            output_list.append(torch.unsqueeze(model(data), 0))
        output_mean = torch.cat(output_list, 0).mean(0)
        test_loss += F.nll_loss(F.log_softmax(output_mean), target, size_average=False).data[0]  # sum up batch loss
        pred = output_mean.data.max(1, keepdim=True)[1]  # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nMC Dropout Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

    train()
    mcdropout_test()

I have replaced

data, target = Variable(data, volatile=True), Variable(target)

by adding

with torch.no_grad():  at the beginning

And this is how I have defined my CNN
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 192, 5, padding=2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(192, 192, 5, padding=2)
        self.fc1 = nn.Linear(192 * 8 * 8, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 10)
        self.dropout = nn.Dropout(p=0.3)

        nn.init.xavier_uniform_(self.conv1.weight)
        nn.init.constant_(self.conv1.bias, 0.0)
        nn.init.xavier_uniform_(self.conv2.weight)
        nn.init.constant_(self.conv2.bias, 0.0)
        nn.init.xavier_uniform_(self.fc1.weight)
        nn.init.constant_(self.fc1.bias, 0.0)
        nn.init.xavier_uniform_(self.fc2.weight)
        nn.init.constant_(self.fc2.bias, 0.0)
        nn.init.xavier_uniform_(self.fc3.weight)
        nn.init.constant_(self.fc3.bias, 0.0)

    def forward(self, x):
        x = self.pool(F.relu(self.dropout(self.conv1(x))))  # recommended to add the relu
        x = self.pool(F.relu(self.dropout(self.conv2(x))))  # recommended to add the relu
        x = x.view(-1, 192 * 8 * 8)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(self.dropout(x)))
        x = self.fc3(self.dropout(x))  # no activation function needed for the last layer
        return x

Can anyone help me to get the right implementation of the Monte Carlo Dropout method on CNN?

Output:
Implementing MC Dropout in Pytorch is easy. All that is needed to be done is to set the dropout layers of your model to train mode. This allows for different dropout masks to be used during the different various forward passes. Below is an implementation of MC Dropout in Pytorch illustrating how multiple predictions from the various forward passes are stacked together and used for computing different uncertainty metrics.
import sys

import numpy as np

import torch
import torch.nn as nn

def enable_dropout(model):
    """ Function to enable the dropout layers during test-time """
    for m in model.modules():
        if m.__class__.__name__.startswith('Dropout'):
            m.train()

def get_monte_carlo_predictions(data_loader,
                                forward_passes,
                                model,
                                n_classes,
                                n_samples):
    """ Function to get the monte-carlo samples and uncertainty estimates
    through multiple forward passes

    Parameters
    ----------
    data_loader : object
        data loader object from the data loader module
    forward_passes : int
        number of monte-carlo samples/forward passes
    model : object
        keras model
    n_classes : int
        number of classes in the dataset
    n_samples : int
        number of samples in the test set
    """

    dropout_predictions = np.empty((0, n_samples, n_classes))
    softmax = nn.Softmax(dim=1)
    for i in range(forward_passes):
        predictions = np.empty((0, n_classes))
        model.eval()
        enable_dropout(model)
        for i, (image, label) in enumerate(data_loader):

            image = image.to(torch.device('cuda'))
            with torch.no_grad():
                output = model(image)
                output = softmax(output) # shape (n_samples, n_classes)
            predictions = np.vstack((predictions, output.cpu().numpy()))

        dropout_predictions = np.vstack((dropout_predictions,
                                         predictions[np.newaxis, :, :]))
        # dropout predictions - shape (forward_passes, n_samples, n_classes)

    # Calculating mean across multiple MCD forward passes
    mean = np.mean(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    # Calculating variance across multiple MCD forward passes
    variance = np.var(dropout_predictions, axis=0) # shape (n_samples, n_classes)

    epsilon = sys.float_info.min
    # Calculating entropy across multiple MCD forward passes
    entropy = -np.sum(mean*np.log(mean + epsilon), axis=-1) # shape (n_samples,)

    # Calculating mutual information across multiple MCD forward passes
    mutual_info = entropy - np.mean(np.sum(-dropout_predictions*np.log(dropout_predictions + epsilon),
                                            axis=-1), axis=0) # shape (n_samples,)

Moving on to the implementation which is posted in the question above, multiple predictions from T different forward passes are obtained by first setting the model to train mode (model.train()). Note that this is not desirable because unwanted stochasticity will be introduced in the predictions if there are layers other than dropout such as batch-norm in the model. Hence the best way is to just set the dropout layers to train mode as shown in the snippet above.


Instruction: Converting a pytorch model to nn.Module for exporting to onnx for lens studio

Input:
I am trying to convert pix2pix to a pb or onnx that can run in Lens Studio. Lens studio has strict requirements for the models. I am trying to export this pytorch model to onnx using this guide provided by lens studio. The issue is the pytorch model found here uses its own base class, when in the example it uses Module.nn, and therefore doesnt have methods/variables that the torch.onnx.export function needs to run. So far Ive run into its missing a variable called training and a method called train
Would it be worth it to try to modify the base model, or should I try to build it from scratch using nn.Module? Is there a way to make the pix2pix model inherit from both the abstract base class and nn.module? Am I not understanding the situation? The reason why I want to do it using the lens studio tutorial is because I have gotten it to export onnx different ways but Lens Studio wont accept those for various reasons.
Also this is my first time asking a SO question (after 6 years of coding), let me know if I make any mistakes and I can correct them. Thank you.
This is the important code from the tutorial creating a pytorch model for Lens Studio:
import torch
import torch.nn as nn
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Conv2d(in_channels=3, out_channels=1,
                               kernel_size=3, stride=2, padding=1)

    def forward(self, x):
        out = self.layer(x)
        out = nn.functional.interpolate(out, scale_factor=2,
                                        mode='bilinear', align_corners=True)
        out = torch.nn.functional.softmax(out, dim=1)
        return out

I'm not going to include all the code from the pytorch model bc its large, but the beginning of the baseModel.py is
import os
import torch
from collections import OrderedDict
from abc import ABC, abstractmethod
from . import networks

class BaseModel(ABC):
    """This class is an abstract base class (ABC) for models.
    To create a subclass, you need to implement the following five functions:
        -- <__init__>:                      initialize the class; first call BaseModel.__init__(self, opt).
        -- <set_input>:                     unpack data from dataset and apply preprocessing.
        -- <forward>:                       produce intermediate results.
        -- <optimize_parameters>:           calculate losses, gradients, and update network weights.
        -- <modify_commandline_options>:    (optionally) add model-specific options and set default options.
    """

    def __init__(self, opt):
        """Initialize the BaseModel class.

        Parameters:
            opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions

        When creating your custom class, you need to implement your own initialization.
        In this function, you should first call <BaseModel.__init__(self, opt)>
        Then, you need to define four lists:
            -- self.loss_names (str list):          specify the training losses that you want to plot and save.
            -- self.model_names (str list):         define networks used in our training.
            -- self.visual_names (str list):        specify the images that you want to display and save.
            -- self.optimizers (optimizer list):    define and initialize optimizers. You can define one optimizer for each network. If two networks are updated at the same time, you can use itertools.chain to group them. See cycle_gan_model.py for an example.
        """
        self.opt = opt
        self.gpu_ids = opt.gpu_ids
        self.isTrain = opt.isTrain
        self.device = torch.device('cuda:{}'.format(self.gpu_ids[0])) if self.gpu_ids else torch.device('cpu')  # get device name: CPU or GPU
        self.save_dir = os.path.join(opt.checkpoints_dir, opt.name)  # save all the checkpoints to save_dir
        if opt.preprocess != 'scale_width':  # with [scale_width], input images might have different sizes, which hurts the performance of cudnn.benchmark.
            torch.backends.cudnn.benchmark = True
        self.loss_names = []
        self.model_names = []
        self.visual_names = []
        self.optimizers = []
        self.image_paths = []
        self.metric = 0  # used for learning rate policy 'plateau'

and for pix2pix_model.py
import torch
from .base_model import BaseModel
from . import networks

class Pix2PixModel(BaseModel):
    """ This class implements the pix2pix model, for learning a mapping from input images to output images given paired data.

    The model training requires '--dataset_mode aligned' dataset.
    By default, it uses a '--netG unet256' U-Net generator,
    a '--netD basic' discriminator (PatchGAN),
    and a '--gan_mode' vanilla GAN loss (the cross-entropy objective used in the orignal GAN paper).

    pix2pix paper: https://arxiv.org/pdf/1611.07004.pdf
    """
    @staticmethod
    def modify_commandline_options(parser, is_train=True):
        """Add new dataset-specific options, and rewrite default values for existing options.

        Parameters:
            parser          -- original option parser
            is_train (bool) -- whether training phase or test phase. You can use this flag to add training-specific or test-specific options.

        Returns:
            the modified parser.

        For pix2pix, we do not use image buffer
        The training objective is: GAN Loss + lambda_L1 * ||G(A)-B||_1
        By default, we use vanilla GAN loss, UNet with batchnorm, and aligned datasets.
        """
        # changing the default values to match the pix2pix paper (https://phillipi.github.io/pix2pix/)
        parser.set_defaults(norm='batch', netG='unet_256', dataset_mode='aligned')
        if is_train:
            parser.set_defaults(pool_size=0, gan_mode='vanilla')
            parser.add_argument('--lambda_L1', type=float, default=100.0, help='weight for L1 loss')

        return parser

    def __init__(self, opt):
        """Initialize the pix2pix class.

        Parameters:
            opt (Option class)-- stores all the experiment flags; needs to be a subclass of BaseOptions
        """

(Also sidenote if you see this and it looks like no easy way out let me know, I know what its like seeing someone getting started in something who goes in to deep too early on)

Output:
You can definitely have your model inherit from both the base class and torch.nn.Module (python allows multiple inheritance). However you should take care about the conflicts if both inherited class have functions with identical names (I can see at least one : their base provide the eval function and so to nn.module).
However since you do not need the CycleGan, and a lot of the code is compatibility with their training environment, you'd probably better just re-implement the pix2pix. Just steal the code, have it inherit from nn.Module, copy-paste useful/mandatory functions from the base class, and have everything translated into clean pytorch code. You already have the forward function (which is the only requirement for a pytorch module).
All the subnetworks they use (like the resnet blocks) seem to inherit from nn.Module already so there is nothing to change here (double check that though)


Instruction: PyTorch: `torch.chunk` source code Github location

Input:
I can’t seem to find the source code for torch.chunk in PyTorch’s Github page or in the documentation.
Anyone knows where this is in PyTorch’s Github page?

Output:
Open the main GitHub project https://github.com/pytorch/pytorch
Then type in this page the keyboard shortcut t: you will enter the file finder mode (introduced in 2011). (https://github.com/pytorch/pytorch/find/master)
Type "chunk.h": the first result will be your file.

If the file is not the right one (header but not source), then you need a search:
https://github.com/search?q=%22chunk%22+%22const+Tensor%22+repo%3Apytorch%2Fpytorch+extension%3Acpp+language%3AC%2B%2B+language%3AC%2B%2B&type=Code
Search for chunk and "const Tensor" helps to refine rapidly the search results to TensorShape.cpp#chunk(), as noted in the comment.


Instruction: How to write torch.device('cuda' if torch.cuda.is_available() else 'cpu') as a full if else statement?

Input:
I'm a beginner to Pytorch and wanted to type this statement as a whole if else statement:-
torch.device('cuda' if torch.cuda.is_available() else 'cpu')

Can somebody help me?

Output:
Here is the code as a whole if-else statement:
torch.device('cuda' if torch.cuda.is_available() else 'cpu')

if torch.cuda.is_available():
    torch.device('cuda')
else:
    torch.device('cpu')

Since you probably want to store the device for later, you might want something like this instead:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

if torch.cuda.is_available():
    device = torch.device('cuda')
else:
    device = torch.device('cpu')

Here is a post and discussion about the ternary operator in Python: https://stackoverflow.com/a/2802748/13985765
From that post:
value_when_true if condition else value_when_false


Instruction: How to represent those types of values in a tensor?

Input:
I have a dataset like this:
(0, 1), UpDownUpUp
(2, 3), UpUpUpDownDownDown
(0, 2), DownUp
(0, Undefined), DownUp

How to represent this type of data in a PyTorch tensor? So I can then train a neural network with it?

Output:
Here is one way to do it:

You keep the values as they are, replacing the undefined with 9 (we will use 9 to represent [do nothing])
You encode the labels into integers 5, 6
Convert your variables into lists
For every batch, take the length of the longest sequence
Pad the input and the labels with 9s to match the length you found at previous step

Resulting dataset will look like the following input and target pairs:
[0, 1, 9, 9, 9, 9], [5, 6, 5, 5, 9, 9]
[2 ,3, 9, 9, 9, 9], [5, 5, 5, 6, 6, 6]
[0, 2, 9, 9, 9, 9], [6, 5, 9, 9, 9, 9]
[0, 9, 9, 9, 9, 9], [6, 5, 9, 9, 9, 9]


Instruction: TypeError: 'Image' object does not support item assignment

Input:
I am running the model below and after splitting the image and trying to assign the image back I get the error message stating.
    sample[:, :, 0] = 0
TypeError: 'Image' object does not support item assignment

I have tried different approaches to assigning the image back to the sample but that didn't help. Any help would be greatly appreciated. Many Thanks
Here's the code I am using to do the data loading:
import os

import numpy as np
import torch
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms, utils
from torchvision.transforms import functional
#from torchvision.transforms import Grayscalei
import pandas as pd
import pdb
import cv2
from cv2 import imread
from cv2 import resize

class CellsDataset(Dataset):
    # a very simple dataset

    def __init__(self, root_dir, transform=None, return_filenames=False):
        self.root = root_dir
        self.transform = transform
        self.return_filenames = return_filenames
        self.files = [os.path.join(self.root,filename) for filename in os.listdir(self.root)]
        self.files = [path for path in self.files
                      if os.path.isfile(path) and os.path.splitext(path)[1]=='.png']

    def __len__(self):
        return len(self.files)

        path = self.files[idx]
        sample = Image.open(path)
        sample = np.asarray(sample, dtype=np.uint8)
        sample = cv2.resize(src=sample, dsize=(1024, 1024))
        sample = functional.to_grayscale(sample, num_output_channels=3)

        sample[:, :, 0] = 0
        sample[:, :, 1] = 0

        if self.transform:
            sample = self.transform(sample)

        if self.return_filenames:
            return sample, path
        else:
            return sample

    def duplicate_and_transform(self,transforms):
        currfilelen = self.__len__()

        for pind in range(0,currfilelen-1):
            path = self.files[pind]
            sample = Image.open(path)
            samplet = transforms(sample)
            # save the transformed images and save the associated counts:
            newpath = path[:-4]+'trans.png' #path[:-4] is the path without the .png extension.
            samplet.save(newpath,"PNG")
            metadata = pd.read_csv(os.path.join(self.root,'gt_red.csv'))
            metadata.set_index('filename',inplace=True)
            count = metadata['count'][os.path.split(path)[-1]]
            samplet.save(newpath,"PNG")
            f = open(self.root+'gt.csv','a')
            basepath = os.path.basename(path) # strip away the directory list and just get the filename
            f.write(basepath[:-4]+'trans.png,'+str(count)+'\n')
            # return the new file names and add to self.files.
            self.files.append(newpath)
            #pdb.set_trace()

Output:
You are directly operating on a PIL object. you have to convert it to a numpy array first and then manipulate the values:
sample = Image.open(path)
sample = np.asarray(sample, dtype=np.uint8)


Instruction: Pytorch counting tensor

Input:
I am trying to create a Python function that, given inputs dim and m, generates the tensor of size [m ** dim, dim] of the form
[[1,...,1,1],
 [1,...,1,2],
 ...
 [1,...,1,m],
 [1,...,2,1],
 [1,...,2,2],
 ...
 [m,...,m,m]]

What is the best way of doing this in Pytorch?
Thanks

Output:
I have solved this myself using the following code:
import torch
import numpy as np

def mat_gen(dim, m):
    return torch.from_numpy(np.array(np.meshgrid(*[np.arange(1, m + 1, 1) for i in range(dim)])).T.reshape(m ** dim, dim))

Here is another function that uses Pytorch only:
import torch

def mat_gen(dim, m):
    return torch.stack(torch.meshgrid(*[torch.arange(1, m + 1, 1) for i in range(dim)])).T.reshape(m ** dim, dim)


Instruction: Inverse transform

Input:
I am not able to find an efficient way to give batch input to this function and return the batch output. I want to do this during the training of my neural network.
Inverse_Norm = transforms.Normalize(
   mean = [-m/s for m, s in zip(mean, std)],
   std = [1/s for s in std]
)
inverse_norm_input = Inverse_Norm(input)

Output:
Assuming a tensor of shape (B, C, ...) wheremean and std are iterables of length C then you can use broadcasting semantics to operate across a batch tensor. For example
import torch

def batch_inverse_normalize(x, mean, std):
    # represent mean and std to 1, C, 1, ... tensors for broadcasting
    reshape_shape = [1, -1] + ([1] * (len(x.shape) - 2))
    mean = torch.tensor(mean, device=x.device, dtype=x.dtype).reshape(*reshape_shape)
    std = torch.tensor(std, device=x.device, dtype=x.dtype).reshape(*reshape_shape)
    return x * std + mean


Instruction: Can you integrate opencv SIFT with a tensorflow model?

Input:
I am trying to create a CNN, but using the SIFT algorithm instead of any pooling layers.
Problem is I can't seem to find any Python implementation of the algorithm in Tensorflow or PyTorch. The only implementation I have seen of it is with opencv.
Is it possible to use the opencv SIFT implementation as a layer in a Tensorflow CNN Model?
If so, how would you go about creating it?

Output:
While this is an interesting idea, I believe it has numerous issues that make it highly impracticable to impossible.
Layers of a network have to be differentiable with regards to their input to allow any gradients to be calculated, which are then used to update the weights.
While I think it might be possible to write a fully differentiable sift implementation, this alone will be impractical.
Further SIFT does not have a constant number of outputs and takes a long time to compute, which would slow down the training a lot.
The only practical way to use SIFT with neural networks would be to first run SIFT and then use the top N detected keypoints as input for the first layer. However, I'm not sure this would be successful.


Instruction: How would I apply a nn.conv1d manually, given an input matrix and weight matrix?

Input:
I am trying to understand how a nn.conv1d processes an input for a specific example related to audio processing in a WaveNet model.
I have input data of shape (1,1,8820), which passes through an input layer (1,16,1), to output a shape of (1,16,8820).
That part I understand, because you can just multiply the two matrices. The next layer is a conv1d, kernel size=3, input channels=16, output channels=16, so the state dict shows a matrix with shape (16,16,3) for the weights. When the input of (1,16,8820) goes through that layer, the result is another (1,16,8820).
What multiplication steps occur within the layer to apply the weights to the audio data? In other words, if I wanted to apply the layer(forward calculations only) using only the input matrix, the state_dict matrix, and numpy, how would I do that?
This example is using the nn.conv1d layer from Pytorch. Also, if the same layer had a dilation=2, how would that change the operations?

Output:
A convolution is a specific type of "sliding window operation": that is, applying the same function/operation on overlapping sliding windows of the input.
In your example, you treat each 3 overlapping temporal samples (each in 16 dimensions) as an input to 16 filters. Therefore, you have a weight matrix of 3x16x16.
You can think of it as "unfolding" the (1, 16, 8820) signal into (1, 16*3, 8820) sliding windows. Then multiplying by 16*3 x 16 weight matrix to get an output of shape (1, 16, 8820).
Padding, dilation and strides affect the way the "sliding windows" are formed.
See nn.Unfold for more information.


Instruction: Pinned memory in LibTorch

Input:
I might be missing something really fundamental but I couldn't find any explanation in the documentation or online
I'm trying to copy a GPU at::Tensor to a pinned tensor on the CPU, but once I copy it, the CPU tensor is no longer pinned. I assume it just creates a new copy of the GPU tensor and assigns it, but if that's the case, how do you copy to a pre-allocated pinned memory?
My testing code:
    at::Tensor gpu = at::randn({1025,1025}, device(at::kCUDA));
    at::Tensor pinned = at::empty(gpu.sizes(), device(at::kCPU).pinned_memory(true));
    std::cout << "Is Pinned: " << std::boolalpha << pinned.is_pinned() << std::endl;
    pinned = gpu.to(at::kCPU);
    std::cout << "Is Pinned: " << std::boolalpha << pinned.is_pinned() << std::endl;

The output is
Is Pinned: true
Is Pinned: false

This also happens with torch:: instead of at::
Tested on Ubuntu 16.04 using LibTorch 1.5.0 compiled from sources

Output:
I found a way, and that's using the copy_ function
...
//pinned = gpu.to(torch::kCPU, true);
gpu.copy_(pinned);
std::cout << "Is Pinned: " << std::boolalpha << pinned.is_pinned() << std::endl;

This outputs
Is Pinned: true
Is Pinned: true

I guess it makes sense since the to function returns a tensor rather than manipulating. Though I would expect some variant of to to allow it.
Oh well, it works this way.


Instruction: Welch Transformation to keep the dimensions

Input:
I am trying to extract the scipy.signal.welch signal from a temporal 1d time series and since I am not adept in signal processing I have no idea why the dimensions shrink when returned.
I need to concatenate the temporal to the spectral as another channel so if the temporal is of shape:
[batches, channels, sample_length]

then I expect to get after the concatenation:
[batches, 2*channels, sample_length]

but when I try to train my model an error is thrown because the size of the spectral doesn't match the temporal (the temporal size is 16):
size mismatch, m1: [2 x 9], m2: [16 x 16]

I tried to look at the documentation but they don't mention why it shrinks it and how it can be avoided.

Output:
As @SleuthEye said, I set the the return_onesided to False like so:
scipy.signal.welch(temporal, axis=-1, return_onesided=False, nperseg=temporal.shape[0])


Instruction: Finding column index of a PyTorch tensor that has most 1's

Input:
I have a PyTorch tensor a shaped like below:
import torch
a = torch.tensor([[[1., 0., 0., 0.]],
        [[0., 1., 0., 0.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]]])

Each row of the tensor a has 4 elements, 1's and 0's. Say I index the row and the column of this tensor accordingly. So for instance, the entry in the row 0 (uppermost row) is [[1., 0., 0., 0.]], whereas the entry in the column 3 (rightmost column) is [[0., 0., 0., 1., 0., 1., 0.]].
From a given tensor, I want to identify the index of the column where 1. appear most frequently. For example, for the tensor a, the index of such a column would be 0. If there are ties in the number of 1.'s, I still would like to get all of those tied column indices.
How can I do this task easily on Python?
Thank you,

Output:
If your matrix only contains 0 and 1, you can sum the elements of each column and then search for the sum that is the largest:
import numpy as np

% sum over columns
sumsi = torch.sum(a, dim=1)

% find where maximum
col_idx = np.where(sumsi==np.max(sumsi))


Instruction: PyTorch DatasetLoader tensor should be a torch tensor. Got

Input:
data_dir="D:\ML-ComputerVision\Datasets"
train_transforms=transforms.Compose([transforms.RandomRotation(30),
                                     transforms.RandomResizedCrop(100),
                                     transforms.RandomHorizontalFlip(),
                                    transforms.Normalize([0.5,],[0.5,]),
                                    transforms.ToTensor()])

test_transforms=transforms.Compose([transforms.Normalize([0.5,],[0.5,]),
                                    transforms.ToTensor()])

train_data=datasets.ImageFolder(data_dir + "/Train",transform=train_transforms)

test_data=datasets.ImageFolder(data_dir + "/Test",transform=test_transforms)

trainloader=torch.utils.data.DataLoader(train_data,batch_size=32,shuffle=True)

testloader=torch.utils.data.DataLoader(test_data,batch_size=32,shuffle=False)

images, labels = next(iter(trainloader)) # <-- Error line

I am getting  tensor should be a torch tensor. Got <class 'PIL.Image.Image'> error even though I have implemented transforms.ToTensor(). Any ideas how it can be fixed?

Output:
The way you chained your transforms, transforms.Normalizeis applied before transforms.ToTensor. But even though RandomRotation, RandomResizedCrop and RandomHorizontalFlip are image transforms that work on PIL images, transforms.Normalize only works on tensors (documentation here).
Simply putting ToTensor before Normalize should work.


Instruction: Is normalization necessary for regression problem in Neural network

Input:
I am learning how to build a neural network using PyTorch.
This formula is the target of my code:
y =2X^3 + 7X^2 - 8*X + 120
It is a regression problem.
I used this because it is simple and the output can be calculated so that I can ensure my neural network is able to predict output with the given input.
However, I met some problem during training.
The problem occurs in this line of code:
loss = loss_func(prediction, outputs)

The loss computed in this line is NAN (not a number)
I am using MSEloss as the loss function. 100 datasets are used for training the ANN model. The input X_train is ranged from -1000 to 1000.
I believed that the problem is owing to the value of X_train and MSEloss. X_train should be scaled into some values between 0 and 1 so that MSEloss can compute the loss.
However, is it possible to train the ANN model without scaling the input into value between 0 and 1 in a regression problem?
Here is my code, it does not use MinMaxScaler and it print the loss with NAN:
import torch
import torch.nn as nn
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torch.autograd import Variable

#Load datasets
dataset = pd.read_csv('test_100.csv')

x_temp_train = dataset.iloc[:79, :-1].values
y_temp_train = dataset.iloc[:79, -1:].values
x_temp_test = dataset.iloc[80:, :-1].values
y_temp_test = dataset.iloc[80:, -1:].values

#Turn into tensor
X_train = torch.FloatTensor(x_temp_train)
Y_train = torch.FloatTensor(y_temp_train)
X_test = torch.FloatTensor(x_temp_test)
Y_test = torch.FloatTensor(y_temp_test)

#Define a Artifical Neural Network
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear = nn.Linear(1,1)  #input=1, output=1, bias=True

    def forward(self, x):
        x = self.linear(x)
        return x
net = Net()
print(net)

#Define a Loss function and optimizer
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()

#Training
inputs = Variable(X_train)
outputs = Variable(Y_train)
for i in range(100):      #epoch=100
    prediction = net(inputs)
    loss = loss_func(prediction, outputs)
    optimizer.zero_grad() #zero the parameter gradients
    loss.backward()       #compute gradients(dloss/dx)
    optimizer.step()      #updates the parameters

    if i % 10 == 9:        #print every 10 mini-batches
        #plot and show learning process
        plt.cla()
        plt.scatter(X_train.data.numpy(), Y_train.data.numpy())
        plt.plot(X_train.data.numpy(), prediction.data.numpy(), 'r-', lw=2)
        plt.text(0.5, 0, 'Loss=%.4f' % loss.data.numpy(), fontdict={'size': 10, 'color': 'red'})
        plt.pause(0.1)

plt.show()

Thanks for your time.

Output:
Is normalization necessary for regression problem in Neural Network?

No.
But...
I can tell you that MSELoss works with non-normalised values. You can tell because:
>>> import torch
>>> torch.nn.MSELoss()(torch.randn(1)-1000, torch.randn(1)+1000)
tensor(4002393.)

MSE is a very well-behaved loss function, and you can't really get NaN without giving it a NaN. I would bet that your model is giving a NaN output.
The two most common causes of a NaN are: an accidental divide by 0, and absurdly large weights/gradients.
I ran a variant of your code on my machine using:
x = torch.randn(79, 1)*1000
y = 2*x**3 + 7*x**2 - 8*x + 120

And it got to NaN in about 20 training steps due to absurdly large weights.
A model can get absurdly large weights if the learning rate is too large. You may think 0.2 is not too large, but that's a typical learning rate people use for normalised data, which forces their gradients to be fairly small. Since you are not using normalised data, let's calculate how large your gradients are (roughly).
First, your x is on the order of 1e3, your expected output y scales at x^3, then MSE calculates (pred - y)^2. Then your loss is on the scale of 1e3^3^2=1e18. This propagates to your gradients, and recall that weight updates are += gradient*learning_rate, so it's easy to see why your weights fairly quickly explode outside of float precision.
How to fix this? Well you could use a learning rate of 2e-7. Or you could just normalise your data. I recommend normalising your data; it has other nice properties for training and avoids these kinds of problems.


Instruction: PyTorch - get list of sums of 2D tensors in 3D tensor

Input:
I have a 3D tensor consisting of 2D tensors, e. g.:
t = torch.tensor([[[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]],

                  [[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]],

                  [[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]]
                  ])

I need a list or tensor of sums of those 2D tensors, e. g.: sums = [3, 3, 3]. So far I have:
sizes = [torch.sum(t[i]) for i in range(t.shape[0])]

I think this can be done with PyTorch only, but I've tried using torch.sum() with all possible dimensions and I always get sums over the individual fields of those 2D tensors, e. g.:
[[0, 0, 3],
[0, 3, 0],
[3, 0, 0]]

How can this be done in PyTorch?

Output:
You can do it at once by passing dims as tuple.
t.sum(dim=(0,1))
tensor([3, 3, 3])

or for list
t.sum(dim=(0,1)).tolist()
[3, 3, 3]


Instruction: Error invalid syntax in pytorch nn module

Input:
I have an error in my pytorch code, and i really don't understand why.
  File "<ipython-input-11-89006c750b74>", line 3
    def GaussianBlur(torch.nn.Module):
                          ^
SyntaxError: invalid syntax

The rest of the code is here
import torch.nn as nn
def GaussianBlur(torch.nn.Module):
    def __init__(self, kernel_size, std_dev):
        self.kernel_size = kernel_size
        self.std_dev = std_dev

    def forward(self, img):
        image = np.array(img)
        image_blur = cv2.GaussianBlur(image, self.kernel_size, self.std_dev)
        return Image.fromarray(image_blur)

Anyone knows what's going on? Thanks

Output:
def is for functions, class for classes. This should work :
class GaussianBlur(torch.nn.Module):


Instruction: Training a CNN model with DataLoader on a GPU in PyTorch

Input:
I am trying to classify chest x-rays into two categories: 'normal', and 'pneumonia'. My training and test sets are DataLoader objects with num_workers=0, pin_memory=True. Cuda is available on my device (GTX 1060 6GB). After creating the CNN I call
model = CNN().cuda(). When I try to train the model I get RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'. What do I have to change in order to train the model on my GPU?
The code:
root = 'chest_xray/'

train_data = datasets.ImageFolder(os.path.join(root, 'train'), transform=train_transform)
test_data = datasets.ImageFolder(os.path.join(root, 'test'), transform=test_transform)

train_loader = DataLoader(train_data,batch_size=10,shuffle=True,num_workers=0,pin_memory=True)
test_loader = DataLoader(test_data,batch_size=10,shuffle=False,num_workers=0,pin_memory=True)

class_names = train_data.classes

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 12, 5, 1)
        self.conv2 = nn.Conv2d(12, 24, 5, 1)
        self.conv3 = nn.Conv2d(24, 30, 5, 1)
        self.conv4 = nn.Conv2d(30, 36, 5, 1)
        self.fc1 = nn.Linear(58*58*36, 256)
        self.fc2 = nn.Linear(256, 144)
        self.fc3 = nn.Linear(144, 84)
        self.fc4 = nn.Linear(84, 16)
        self.fc5 = nn.Linear(16, 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv4(x))
        x = F.max_pool2d(x, 2, 2)

        x = x.view(-1, 58*58*36)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        x = self.fc5(x)

        return F.log_softmax(x, dim=1)

model = CNN().cuda()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

epochs = 2

train_losses = []
train_correct = []
for i in range(epochs):
    trn_correct = 0
    tst_correct = 0

    for b, (X_train, y_train) in enumerate(train_loader):
        y_pred = model(X_train)
        loss = criterion(y_pred, y_train)

        predicted = torch.max(y_pred.data, 1)[1]
        batch_correct = (predicted == y_train).sum()
        trn_correct += batch_correct

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if b % 200 == 0:
            print(f'epoch: {i+1} batch: {b} progress: {10*b/len(train_data)} loss: {loss.item()} accuracy: {10*trn_correct/b}%')

    train_losses.append(loss)
    train_correct.append(trn_correct)

Output:
After this line:
for b, (X_train, y_train) in enumerate(train_loader):

Add the following:
X_train, y_train = X_train.cuda(), y_train.cuda()


Instruction: when we reshape numpy array, how is the stride size inferred?

Input:
I have an 1x1024 1-d array (flattened image). To see the image, I want to reshape its size as 32x32.
I can easily achieve this by doing x.reshape(-1,32) and it works as I intended. It doesn't screw the image. It reads the 1d array with 32-width stride each time.
Say this time, there are 4 images, sized 32 by 8. What's the safe way to reshape this?
What is the logic behind how strides are defined? Is it always starting from the largest dimension (say, 3d->2d->1d)? It seems like it..
In [2]: a = np.arange(1024)

In [3]: a.reshape(4,32,8)
Out[3]:
array([[[   0,    1,    2, ...,    5,    6,    7],
        [   8,    9,   10, ...,   13,   14,   15],
        [  16,   17,   18, ...,   21,   22,   23],
        ...,
        [ 232,  233,  234, ...,  237,  238,  239],
        [ 240,  241,  242, ...,  245,  246,  247],
        [ 248,  249,  250, ...,  253,  254,  255]],

       [[ 256,  257,  258, ...,  261,  262,  263],
        [ 264,  265,  266, ...,  269,  270,  271],
        [ 272,  273,  274, ...,  277,  278,  279],
        ...,
        [ 488,  489,  490, ...,  493,  494,  495],
        [ 496,  497,  498, ...,  501,  502,  503],
        [ 504,  505,  506, ...,  509,  510,  511]],

       [[ 512,  513,  514, ...,  517,  518,  519],
        [ 520,  521,  522, ...,  525,  526,  527],
        [ 528,  529,  530, ...,  533,  534,  535],
        ...,
        [ 744,  745,  746, ...,  749,  750,  751],
        [ 752,  753,  754, ...,  757,  758,  759],
        [ 760,  761,  762, ...,  765,  766,  767]],

       [[ 768,  769,  770, ...,  773,  774,  775],
        [ 776,  777,  778, ...,  781,  782,  783],
        [ 784,  785,  786, ...,  789,  790,  791],
        ...,
        [1000, 1001, 1002, ..., 1005, 1006, 1007],
        [1008, 1009, 1010, ..., 1013, 1014, 1015],
        [1016, 1017, 1018, ..., 1021, 1022, 1023]]])

In [4]: a.reshape(4,-1,8)
Out[4]:
array([[[   0,    1,    2, ...,    5,    6,    7],
        [   8,    9,   10, ...,   13,   14,   15],
        [  16,   17,   18, ...,   21,   22,   23],
        ...,
        [ 232,  233,  234, ...,  237,  238,  239],
        [ 240,  241,  242, ...,  245,  246,  247],
        [ 248,  249,  250, ...,  253,  254,  255]],

       [[ 256,  257,  258, ...,  261,  262,  263],
        [ 264,  265,  266, ...,  269,  270,  271],
        [ 272,  273,  274, ...,  277,  278,  279],
        ...,
        [ 488,  489,  490, ...,  493,  494,  495],
        [ 496,  497,  498, ...,  501,  502,  503],
        [ 504,  505,  506, ...,  509,  510,  511]],

       [[ 512,  513,  514, ...,  517,  518,  519],
        [ 520,  521,  522, ...,  525,  526,  527],
        [ 528,  529,  530, ...,  533,  534,  535],
        ...,
        [ 744,  745,  746, ...,  749,  750,  751],
        [ 752,  753,  754, ...,  757,  758,  759],
        [ 760,  761,  762, ...,  765,  766,  767]],

       [[ 768,  769,  770, ...,  773,  774,  775],
        [ 776,  777,  778, ...,  781,  782,  783],
        [ 784,  785,  786, ...,  789,  790,  791],
        ...,
        [1000, 1001, 1002, ..., 1005, 1006, 1007],
        [1008, 1009, 1010, ..., 1013, 1014, 1015],
        [1016, 1017, 1018, ..., 1021, 1022, 1023]]])

In [5]: a.reshape(4,8,32)
Out[5]:
array([[[   0,    1,    2, ...,   29,   30,   31],
        [  32,   33,   34, ...,   61,   62,   63],
        [  64,   65,   66, ...,   93,   94,   95],
        ...,
        [ 160,  161,  162, ...,  189,  190,  191],
        [ 192,  193,  194, ...,  221,  222,  223],
        [ 224,  225,  226, ...,  253,  254,  255]],

       [[ 256,  257,  258, ...,  285,  286,  287],
        [ 288,  289,  290, ...,  317,  318,  319],
        [ 320,  321,  322, ...,  349,  350,  351],
        ...,
        [ 416,  417,  418, ...,  445,  446,  447],
        [ 448,  449,  450, ...,  477,  478,  479],
        [ 480,  481,  482, ...,  509,  510,  511]],

       [[ 512,  513,  514, ...,  541,  542,  543],
        [ 544,  545,  546, ...,  573,  574,  575],
        [ 576,  577,  578, ...,  605,  606,  607],
        ...,
        [ 672,  673,  674, ...,  701,  702,  703],
        [ 704,  705,  706, ...,  733,  734,  735],
        [ 736,  737,  738, ...,  765,  766,  767]],

       [[ 768,  769,  770, ...,  797,  798,  799],
        [ 800,  801,  802, ...,  829,  830,  831],
        [ 832,  833,  834, ...,  861,  862,  863],
        ...,
        [ 928,  929,  930, ...,  957,  958,  959],
        [ 960,  961,  962, ...,  989,  990,  991],
        [ 992,  993,  994, ..., 1021, 1022, 1023]]])

Output:
reshape does not reorder the underlying values.  The array is stored as a 1d array of bytes, plus shape, strides and dtype which are used to view it as particular multidimensional array.
You can look at the strides attribute:
In [513]: arr = np.arange(1024)
In [514]: arr.shape, arr.strides
Out[514]: ((1024,), (8,))
In [515]: arr1=arr.reshape(32,32);arr1.shape, arr1.strides
Out[515]: ((32, 32), (256, 8))
In [516]: arr1=arr.reshape(4,32,8);arr1.shape, arr1.strides
Out[516]: ((4, 32, 8), (2048, 64, 8))

With 1d, it just steps 8 bytes at a time (the size of the int64)
With 2d, 256=32*8; to traverse rows it has to step 256 bytes
With 3d, 2048 = 32 * 8 * 8; the step between blocks.
And for fun, look at a transpose:
In [517]: arr1=arr.reshape(4,32,8).T;arr1.shape, arr1.strides
Out[517]: ((8, 32, 4), (8, 64, 2048))

shape has been reversed, and so has strides.
Often when reshaping an image array into blocks, we need to reshape into small blocks, do partial transpose, and reshape to a target.  The first reshape and transpose create a view, just playing with the shapes and strides.  But the last reshape often requires a copy.


Instruction: Word-sense disambiguation based on sets of words using pre-trained embeddings

Input:
I am interested in identifying the WordNet synset IDs for each word in a set of tags.
The words in the set provide the context for the word sense disambiguation, such as:

{mole, skin}
{mole, grass, fur}
{mole, chemistry}
{bank, river, river bank}
{bank, money, building}

I know of the lesk algorithm and libraries, such as pywsd, which is based on 10+ year old tech (which may still be cutting edge -- that is my question).
Are there better performing algorithms by now that make sense of pre-trained embeddings, like GloVe, and maybe the distances of these embeddings to each other?
Are there ready-to-use implementations of such WSD algorithms?
I know this question is close to the danger zone of asking for subjective preferences - as in this 5-year old thread. But I am not asking for an overview of options or the best software for a problem.

Output:
Transfer learning, particularly models like Allen AI’s ELMO, OpenAI’s Open-GPT, and Google’s BERT allowed researchers to smash multiple benchmarks with minimal task-specific fine-tuning and provided the rest of the NLP community with pretrained models that could easily (with less data and less compute time) be fine-tuned and implemented to produce state of the art results.
these representations will help you accuratley retrieve results matching the customer's intent and contextual meaning(), even if there's no keyword or phrase overlap.
To start off, embeddings are simply (moderately) low dimensional representations of a point in a higher dimensional vector space.
By translating a word to an embedding it becomes possible to model the semantic importance of a word in a numeric form and thus perform mathematical operations on it.
When this was first possible by the word2vec model it was an amazing breakthrough. From there, many more advanced models surfaced which not only captured a static semantic meaning but also a contextualized meaning. For instance, consider the two sentences below:
I like apples.

I like Apple macbooks

Note that the word apple has a different semantic meaning in each sentence. Now with a contextualized language model, the embedding of the word apple would have a different vector representation which makes it even more powerful for NLP tasks.
contextual embedding's like BERT offers an advantage over models like Word2Vec, because while each word has a fixed representation under Word2Vec regardless of the context within which the word appears, BERT produces word representations that are dynamically informed by the words around them.


Instruction: Conda environment : several environment files - specify cpu-only version of Pytorch

Input:
I am using conda 4.8.3 with Python 3.7, I am writing environment files to specify the dependencies of my project. I would like to write several files to be able to install several environments:

main.yml : containing the dependencies of my project, and pytorch CPU-only version
dev.yml : containing dev tools (mypy, flake8, pytest ..)
gpu.yml : containing pytorch-GPU (with a specified version of CUDA)

To get a basic (CPU) installation one would write : conda env update --file main.yml
To get an installation with GPU compatibility would then add conda env update --file gpu.yml
Here is my question : at the moment I cannot find the right way to specify the 'CPU-only' criteria for pytorch in an environment file, does anyone know if it is doable?
The command usually used for that purpose is conda install pytorch torchvision cpuonly -c pytorch, but I cannot find a way to specify it in the yml file.
On the pytorch channel site , there is a pytorch-cpu package, but its version is quite outdated (1.1.0, while the current main is 1.6.0)
Here is my main.yml environment file:
name: my_env
channels:
  - intel
  - conda-forge
  - pytorch

dependencies:

  - numpy
  - scipy
  - scikit-image
  - matplotlib
  - wxpython
  - colorama
  - dill
  - protobuf
  - pytorch   # How to specify the 'cpu' criteria here??
  - torchvision
  - pip:
      - -r env/requirements.txt

Output:
In case anyone else is looking for the answer -- I tried what AMC suggested in a comment above. I can confirm that adding the line:
- cpuonly
to my environment.yml file forced the CPU version of pytorch to be downloaded.


Instruction: ImportError: TensorBoard logging requires TensorBoard version 1.15 or above

Input:
I follow the tutorials in pytorch.org
It occurs error：TensorBoard logging requires TensorBoard version 1.15 or above,but I have install TensorBoard already.
Here is the code:
#from torch.utils.tensorboard import SummaryWriter
from tensorboardX import SummaryWriter
writer = SummaryWriter('runs/fashion_mnist_experiment_1')
#get some random training images
dataiter = iter(trainloader)
images , labels = dataiter.next()
#create grid of images
img_grid = torchvision.utils.make_grid(images)
matplotlib_imshow(img_grid,one_channel=True)
writer.add_image('four_fashion_images',img_grid)
writer.add_graph(net, images)
writer.close()

Error:
ImportError                               Traceback (most recent call last)
<ipython-input-12-d38808675cb4> in <module>
----> 1 writer.add_graph(net, images)
      2 writer.close()

~\anaconda3\envs\torch2\lib\site-packages\tensorboardX\writer.py in add_graph(self, model, input_to_model, verbose)
    791
    792         """
--> 793         from torch.utils.tensorboard._pytorch_graph import graph
    794         self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
    795

~\anaconda3\envs\torch2\lib\site-packages\torch\utils\tensorboard\__init__.py in <module>
      2 from distutils.version import LooseVersion
      3 if not hasattr(tensorboard, '__version__') or LooseVersion(tensorboard.__version__) < LooseVersion('1.15'):
----> 4     raise ImportError('TensorBoard logging requires TensorBoard version 1.15 or above')
      5 del LooseVersion
      6 del tensorboard

ImportError: TensorBoard logging requires TensorBoard version 1.15 or above

Environment:
tensorboard               2.3.0                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorboardx              2.1                      pypi_0    pypi
tensorflow                1.2.1                    py36_0    defaults
pytorch                   1.6.0           py3.6_cuda102_cudnn7_0    pytorch
torchvision               0.7.0                py36_cu102    pytorch
future                    0.18.2                   py36_1    defaults
protobuf                  3.12.3           py36h33f27b4_0    defaults

I use from torch.utils.tensorboard import SummaryWriter at the beginning,but it occurs the error as same as above.Then I use from tensorboardX import SummaryWriter

Output:
Uninstall tensorflow, tensorboard, tensorboardx and tensorboard-plugin-wit.
Install only tensorboard with conda after that.
If this doesn't work, recreate your conda environment only with tensorboard. If you need tensorflow as well install it beforehand.
EDIT:
tensorboard-plugin-wit is a dependency of tensorboard and should be installed automatically as per their pypi description when installing tensorboard itself.


Instruction: rolling statistics in numpy or pytroch

Input:
I have a tensors data of sensors, each tensor is of shape (4,1500)
This is 1500 timepoints and for each time point I have 4 features.
I want to "smooth" the sequences with rolling average or other rolling statistics. The end goal is to try to improve an lstm autoencoder with rolling statistics instead of the long raw sequence.
I am familiar with rolling windows of pandas and currently I am doing this:
#tensor shape:
 data.shape
 (4,1500)

 #convert data to numpy array and then to dataframe and perform rolling mean
 rolled_data=pd.DataFrame(data.numpy().swapaxes(1,0)).rolling(10).mean()[::10]
 rolled_data.shape
 (150, 4)

 # convert back the dataframe to tensor
 tensor_rolled_data=torch.Tensor(rolled_data.to_numpy().swapaxes(1,0))
 tensor_rolled_data.shape
 torch.Size([4, 150])

my question is- is there a better way to do it? a function in numpy/torch that can do rolling statistics in a cleaner or more efficient way?

Output:
Since you're striding the output by the size of the window this is actually more akin to downsampling by averaging than to a computing rolling statistics. We can take advantage of the fact that there are no overlaps by simply reshaping the initial tensor.

Using Tensor.reshape
Assuming your data tensor has a shape divisible by 10 then you can just reshape the tensor to shape (4, 150, 10) and compute the statistic along the last dimension. For example
win_size = 10
tensor_rolled_data = data.reshape(data.shape[0], -1, win_size).mean(dim=2)

This solution doesn't give exactly the same results as your tensor_rolled_data since in this solution the first entry will contain the mean of the first 10 samples, the second entry will contain the mean of the second 10 samples, etc... The pandas solution is a "causal filter" so the first entry will contain the mean of the 10 most recent samples up to and including sample 0, the second will contain the 10 most recent samples up to and including sample 10, etc... (Note that the first entry is nan in the pandas solution since less than 10 preceding samples exist).
If this difference is unacceptable you can recreate the pandas result by first padding with 9 nan values and clipping off the last 9 samples.
import torch.nn.functional as F
win_size = 10
# pad with `nan` to match behavior of pandas
data_padded = F.pad(data[None, :, :-(win_size - 1)], (win_size - 1, 0), 'constant', float('nan')).squeeze(0)
# find mean of groups of N samples
tensor_rolled_data = data_padded.reshape(data.shape[0], -1, win_size).mean(dim=2)

Using Tensor.unfold
To address the comment about what to do when there are overlaps. If you're only interested in the mean statistic then there are a number of ways to compute this (e.g. convolution, average pooling, tensor unfolding). That said, Tensor.unfold gives the most general solution since it could be used to compute any statistic over a window. For example
# same as first example above
win_size = 10
tensor_rolled_data = data.unfold(dimension=1, size=win_size, step=win_size).mean(dim=2)

or
# same as second example above
import torch.nn.functional as F
win_size = 10
data_padded = F.pad(data.unsqueeze(0), (win_size - 1, 0), 'constant', float('nan')).squeeze(0)
tensor_rolled_data = data_padded.unfold(dimension=1, size=win_size, step=win_size).mean(dim=2)

In the above cases, unfolding produces the same result as reshape since size and step are equal. However, unlike reshape, unfolding also supports size != step.
win_size = 10
stride = 2
tensor_rolled_data = data.unfold(1, win_size, stride).mean(dim=2).mean(dim=2)
# produces shape [4, 746]

or you can pad the front of the features with win_size - 1 values to achieve the same result as pandas.
import torch.nn.functional as F
win_size = 10
stride = 2
data_padded = F.pad(data.unsqueeze(0), (win_size - 1, 0), 'constant', float('nan')).squeeze(0)
tensor_rolled_data = data_padded.unfold(1, win_size, stride).mean(dim=2)
# produces shape [4, 750]

Note In practice you probably don't want to pad with NaN since this will probably become quite a headache. Instead you could use zero padding, 'replicate' padding, or 'mirror' padding.


Instruction: Numpy from an array, for each element create a matrix N*M with all values set as the element without for loops

Input:
I have a numpy array like np.array([1, 2, 3])
Without using for loops but just use numpy or pytorch methods, i want a matrix with dimension len(array) * N * M composed by matrices N*M that the first matrix is composed by all ones, the second only by two values and the third only by 3 values.
For istance
N = 4 M = 3
[[[1,1,1,1],[1,1,1,1],[1,1,1,1]],
  [[2,2,2,2],[2,2,2,2],[2,2,2,2]],
  [[3,3,3,3],[3,3,3,3],[3,3,3,3]]]

I tried different methods to achive this matrix like unsqueeze and repeat but i wasn't able to find a solution, any suggestion?

Output:
Here you go:
np.ones((len(a),M,N)) * a[:,None,None]

Or without multiplication:
np.full((len(a),M,N), a[:,None,None])

Output:
array([[[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]],

       [[2., 2., 2., 2.],
        [2., 2., 2., 2.],
        [2., 2., 2., 2.]],

       [[3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.]]])

Note the shape here and given in your expected output is len(a) * M * N, not len(a) * N * M. Swap M,N in np.ones if needed.


Instruction: How to compute the validation loss? (Simple linear regression)

Input:
I am currently learning how to use PyTorch to build a neural network. I have learned keras before and I would like to do the same thing in PyTorch like 'model.fit' and plotting a graph containing both training loss and validation loss.
In order to know whether the model is underfitting or not, I have to plot a graph to compare the training loss and validation loss.
However, I cannot compute the right validation loss. I know that gradients should only be updated during training and it should not be updated during testing/validation. With no change in gradients, does it mean the loss will not change? Sorry, my concept is not clear enough. But I think not, loss should be computed by comparing expected output and prediction using loss function.
In my code, 80 datasets are used for training and 20 datasets are used for validation. In my code, the neural network is prediction this formula:  y =2X^3 + 7X^2 - 8*X + 120
It is easy to compute so I use this for learning how to build neural network through PyTorch.
Here is my code:
import torch
import torch.nn as nn    #neural network model
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torch.autograd import Variable
from sklearn.preprocessing import MinMaxScaler

#Load datasets
dataset = pd.read_csv('test_100.csv')

X = dataset.iloc[:, :-1].values
Y = dataset.iloc[:, -1:].values

X_scaler = MinMaxScaler()
Y_scaler = MinMaxScaler()
print(X_scaler.fit(X))
print(Y_scaler.fit(Y))
X = X_scaler.transform(X)
Y = Y_scaler.transform(Y)

x_temp_train = X[:79]
y_temp_train = Y[:79]
x_temp_test = X[80:]
y_temp_test = Y[80:]

X_train = torch.FloatTensor(x_temp_train)
Y_train = torch.FloatTensor(y_temp_train)
X_test = torch.FloatTensor(x_temp_test)
Y_test = torch.FloatTensor(y_temp_test)

D_in = 1 # D_in is input features
H = 24 # H is hidden dimension
D_out = 1 # D_out is output features.

#Define a Artifical Neural Network model
class Net(nn.Module):
#------------------Two Layers------------------------------
    def __init__(self, D_in, H, D_out):
        super(Net, self).__init__()

        self.linear1 = nn.Linear(D_in, H)
        self.linear2 = nn.Linear(H, D_out)

    def forward(self, x):
        h_relu = self.linear1(x).clamp(min=0)
        prediction = self.linear2(h_relu)
        return prediction
model = Net(D_in, H, D_out)
print(model)

#Define a Loss function and optimizer
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.2) #2e-7

#Training
inputs = Variable(X_train)
outputs = Variable(Y_train)
inputs_val = Variable(X_test)
outputs_val = Variable(Y_test)
loss_values = []
val_values = []
epoch = []
for phase in ['train', 'val']:
    if phase == 'train':
        model.train()  # Set model to training mode
    else:
        optimizer.zero_grad() #zero the parameter gradients
        model.eval()   # Set model to evaluate mode
    for i in range(50):      #epoch=50
        if phase == 'train':
            model.train()
            prediction = model(inputs)
            loss = criterion(prediction, outputs)
            print('train loss')
            print(loss)
            loss_values.append(loss.detach())
            optimizer.zero_grad() #zero the parameter gradients
            epoch.append(i)
            loss.backward()       #compute gradients(dloss/dx)
            optimizer.step()      #updates the parameters
        elif phase == 'val':
            model.eval()
            prediction_val = model(inputs_val)
            loss_val = criterion(prediction_val, outputs_val)
            print('validation loss')
            print(loss)
            val_values.append(loss_val.detach())
            optimizer.zero_grad() #zero the parameter gradients

plt.plot(epoch,loss_values)
plt.plot(epoch, val_values)
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train','validation'], loc='upper left')
plt.show()

Here is the result:
train loss
tensor(0.9788, grad_fn=<MseLossBackward>)
tensor(2.0834, grad_fn=<MseLossBackward>)
tensor(3.2902, grad_fn=<MseLossBackward>)
tensor(0.8851, grad_fn=<MseLossBackward>)
tensor(0.0832, grad_fn=<MseLossBackward>)
tensor(0.0402, grad_fn=<MseLossBackward>)
tensor(0.0323, grad_fn=<MseLossBackward>)
tensor(0.0263, grad_fn=<MseLossBackward>)
tensor(0.0217, grad_fn=<MseLossBackward>)
tensor(0.0181, grad_fn=<MseLossBackward>)
tensor(0.0153, grad_fn=<MseLossBackward>)
tensor(0.0132, grad_fn=<MseLossBackward>)
tensor(0.0116, grad_fn=<MseLossBackward>)
tensor(0.0103, grad_fn=<MseLossBackward>)
tensor(0.0094, grad_fn=<MseLossBackward>)
tensor(0.0087, grad_fn=<MseLossBackward>)
tensor(0.0081, grad_fn=<MseLossBackward>)
tensor(0.0077, grad_fn=<MseLossBackward>)
tensor(0.0074, grad_fn=<MseLossBackward>)
tensor(0.0072, grad_fn=<MseLossBackward>)
tensor(0.0070, grad_fn=<MseLossBackward>)
tensor(0.0068, grad_fn=<MseLossBackward>)
tensor(0.0067, grad_fn=<MseLossBackward>)
tensor(0.0067, grad_fn=<MseLossBackward>)
tensor(0.0066, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0065, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0064, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0063, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0062, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)

validation loss
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)
tensor(0.0061, grad_fn=<MseLossBackward>)

Train Loss Vs. Validation Loss
The validation loss is a flat line. It is not what I want.

Output:
You should do validation after each training epoch to "validate" your model capabilities. Also, in validation phase the model parameters don't change so it is obvious you would get a constant loss in your validations. Your code should be as follows:
training epoch 1
validation
training epoch 2
validation
...
don't forget to use loss.item() instead of loss alone in your loss calculating and averaging. Because loss gives you a grad_function, not a float value.


Instruction: How to find the source code of torch.solve?

Input: I am concerned whether torch.solve() examine the condition of the coefficient matrix for a linear system and employ desirable preconditionings; thus I am curious about its implementation details. I have read through several answers trying to track down the source file but in vain. I hope somebody can help me to locate its definition in the ATen library.

Output:
I think it just uses LAPACK for CPU and CUBLAS for GPU, since torch.solve is listed under "BLAS and LAPACK Operations" on the official docs.
Then we're looking for wrapper code, which I believe is this part.


Instruction: Why does my autoencoder for de-noising in PyTorch learns to zero out everything?

Input:
I need to build a de-noising autoencoder using pytorch for the task of cleaning out signals.
For example, I can take the cosine function and sample it in intervals (where i have two parameters - B and K. B is the number of intervals I take in each example and K is how many sampling points (equally distanced) are in each interval) so for instance, I can take B = 5 intervals and measure K = 8 points in each interval. Hence the distance between each point is 2pi / 8 and i have a total of 40 points. The number of functions I try to generalize is L and I treat this like different channels. Then I add a random starting position for each example (to make it slightly different) then random noise and send it to the autoencoder to train.
The thing is, no matter what architecture or learning rate, gradually it learns to output nothing but zeros. The autoencoder is super simple so I dont reckon there's a problem with it but rather a problem of how I generate my data.
I attach both of the codes anyways:
class ConvAutoencoder(nn.Module):
def __init__(self, enc_channels, dec_channels):
    super(ConvAutoencoder, self).__init__()
    ## encoder layers ##

    encoder_layers = []
    decoder_layers = []

    in_channels = enc_channels[0]

    for i in range(1, len(enc_channels)):
        out_channels = enc_channels[i]

        encoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True),
                           nn.ReLU()]
        in_channels = out_channels

    in_channels = dec_channels[0]

    for i in range(1, len(dec_channels)):
        out_channels = dec_channels[i]

        decoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True),
                           nn.ReLU()]
        in_channels = out_channels

    self.encoder = nn.Sequential(*encoder_layers)
    self.decoder = nn.Sequential(*decoder_layers)

def forward(self, x):

    if len(x.shape) == 3:
        x = x.unsqueeze(dim=-1)

    res = self.decoder(self.encoder(x)).squeeze(-1)

    return res

And the data generation is as follows:
def generate_data(batch_size: int, intervals: int, sample_length: int, channels_functions, noise_scale=1)->torch.tensor:
    channels = len(channels_functions)
    mul_term = 2 * np.pi / sample_length # each sample is 2pi and equally distance

    # each example is K * B long
    positions = np.arange(0, sample_length * intervals)
    x = positions * mul_term

    # creating random start points (from negative to positive)
    random_starting_pos = (np.random.rand(batch_size) - 0.5) * 10000
    start_pos_mat = np.tile(random_starting_pos , (sample_length * intervals, 1))
    start_pos_mat = np.tile(start_pos_mat , (channels, 1)).T
    start_pos_mat = np.reshape(start_pos_mat , (batch_size, channels, sample_length * intervals))

    X = np.tile(x, (channels, 1))
    X = np.repeat(X[np.newaxis, :, :], batch_size, axis=0)

    X += start_pos_mat #adding the random starting position

    # apply each function to a different channel
    for i, function in enumerate(channels_functions):
        X[:, i, :] = function(X[:, i, :])

    clean = X
    noise = np.random.normal(scale=noise_scale, size=clean.shape)
    noisy = clean + noise

    # normalizing each sample
    row_sums = np.linalg.norm(clean, axis=2)
    clean = clean / row_sums[:, :, np.newaxis]
    row_sums = np.linalg.norm(noisy, axis=2)
    noisy = noisy / row_sums[:, :, np.newaxis]

    clean = torch.from_numpy(clean)
    noisy = torch.from_numpy(noisy)

    return clean, noisy

Edit - Added the entire training loop:
if __name__ == '__main__':

func_list = [lambda x: np.cos(x),
             lambda x: np.cos((x**4) / 10),
             lambda x: np.sin(x**3 * np.cos(x**2)),
             lambda x: 0.25*np.cos(x**2) - 10*np.sin(0.25*x)]

L = len(func_list)
K = 3
B = 4

enc_channels = [L, 64, 128, 256]

num_epochs = 100

model = models.ConvAutoencoder(enc_channels, enc_channels[::-1])

criterion = torch.nn.BCELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.005, weight_decay=1e-5)

for epoch in range(num_epochs):
    clean, noisy = util.generate_data(128, K, B, func_list)

    # ===================forward=====================
    output = model(noisy.float())
    loss = criterion(output.float(), clean.float())
    # ===================backward====================
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    # ===================log========================
    print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, num_epochs, loss.data))

    if epoch % 10 == 0:
        show_clean, show_noisy = util.generate_data(1, K, B, func_list)
        print("clean\n{}".format(show_clean))
        print("noisy\n{}".format(show_noisy))
        print("denoised\n{}".format(model(show_noisy.float())))

sure enough after like 10 epochs the model outputs:
clean vector
tensor([[[ 0.3611, -0.1905, -0.3611,  0.1905,  0.3611, -0.1905, -0.3611,
           0.1905,  0.3611, -0.1905, -0.3611,  0.1905],
         [ 0.3387, -0.0575, -0.2506, -0.3531, -0.3035,  0.3451,  0.3537,
          -0.2416,  0.2652, -0.3126, -0.3203, -0.1707],
         [-0.0369,  0.4412, -0.1323,  0.1802, -0.2943,  0.3590,  0.4549,
           0.0827, -0.0164,  0.4350, -0.1413, -0.3395],
         [ 0.3997,  0.3516,  0.2451,  0.1136, -0.0458, -0.1944, -0.3225,
          -0.3925, -0.3971, -0.3382, -0.2457, -0.1153]]], dtype=torch.float64)
noisy vector
tensor([[[-0.1071, -0.0671,  0.0993, -0.2029,  0.1587, -0.4407, -0.0867,
          -0.2598,  0.2426, -0.6939, -0.3011, -0.0870],
         [ 0.0889, -0.3415, -0.1434, -0.2393, -0.4708,  0.0144,  0.2620,
          -0.1186,  0.6424,  0.0886, -0.2192, -0.1562],
         [ 0.1989,  0.2794,  0.0848, -0.2729, -0.2168,  0.1475,  0.5294,
           0.4788,  0.1754,  0.2333, -0.0549, -0.3665],
         [ 0.3611,  0.3535,  0.1957,  0.1980, -0.1115, -0.1912, -0.2713,
          -0.4087, -0.3669, -0.3675, -0.2991, -0.1356]]], dtype=torch.float64)
denoised vector
tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]], grad_fn=<SqueezeBackward1>)

Thanks

Output:
The problem is that you are using a ReLU in the final layer, however your target (clean) contains negative values. When you are using a ReLU in your final layer, you cannot obtain negative values.
Simply replace your decoder with this:
    for i in range(1, len(dec_channels)):
        out_channels = dec_channels[i]

        if i == len(dec_channels) - 1:
            # last layer
            decoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True)]
        else:
            decoder_layers += [nn.ConvTranspose2d(in_channels, out_channels, kernel_size=1, bias=True),
                               nn.ReLU()]
        in_channels = out_channels

And then I would suggest using the L2 loss.


Instruction: RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target'

Input:
I'm running into an issue while calculating the loss for my Neural Net. I'm not sure why the program expects a long object because all my Tensors are in float form. I looked at threads with similar errors and the solution was to cast Tensors as floats instead of longs, but that wouldn't work in my case because all my data is already in float form when passed to the network.
Here's my code:
# Dataloader
from torch.utils.data import Dataset, DataLoader

class LoadInfo(Dataset):
    def __init__(self, prediction, indicator):
        self.prediction = prediction
        self.indicator = indicator
    def __len__(self):
        return len(self.prediction)
    def __getitem__(self, idx):
        data = torch.tensor(self.indicator.iloc[idx, :],dtype=torch.float)
        data = torch.unsqueeze(data, 0)
        label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.float)
        sample = {'data': data, 'label': label}
        return sample

# Trainloader
test_train = LoadInfo(train_label, train_indicators)
trainloader = DataLoader(test_train, batch_size=64,shuffle=True, num_workers=1,pin_memory=True)

# The Network
class NetDense2(nn.Module):

    def __init__(self):
        super(NetDense2, self).__init__()
        self.rnn1 = nn.RNN(11, 100, 3)
        self.rnn2 = nn.RNN(100, 500, 3)
        self.fc1 = nn.Linear(500, 100)
        self.fc2 = nn.Linear(100, 20)
        self.fc3 = nn.Linear(20, 3)

    def forward(self, x):
        x1, h1 = self.rnn1(x)
        x2, h2 = self.rnn2(x1)
        x = F.relu(self.fc1(x2))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# Allocate / Transfer to GPU
dense2 = NetDense2()
dense2.cuda()

# Optimizer
import torch.optim as optim
criterion = nn.CrossEntropyLoss()                                 # specify the loss function
optimizer = optim.SGD(dense2.parameters(), lr=0.001, momentum=0.9,weight_decay=0.001)

# Training
dense2.train()
loss_memory = []
for epoch in range(50):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, samp in enumerate(trainloader):
        # get the inputs
        ins = samp['data']
        targets = samp['label']
        tmp = []
        tmp = torch.squeeze(targets.float())
        ins, targets = ins.cuda(),  tmp.cuda()
        # zero the parameter gradients
        optimizer.zero_grad()
        # forward + backward + optimize
        outputs = dense2(ins)
        loss = criterion(outputs, targets)     # The loss
        loss.backward()
        optimizer.step()
        # keep track of loss
        running_loss += loss.data.item()

I get the error from above in the line " loss = criterion(outputs, targets) "

Output:
As per the documentation and official example at pytorch webpage, The targets passed to nn.CrossEntropyLoss() should be in torch.long format
# official example
import torch
import torch.nn as nn
loss = nn.CrossEntropyLoss()
input = torch.randn(3, 5, requires_grad=True)
target = torch.empty(3, dtype=torch.long).random_(5)

# if you will replace the dtype=torch.float, you will get error

output = loss(input, target)
output.backward()

update this line in your code as
label = torch.tensor(self.prediction.iloc[idx, :],dtype=torch.long) #updated torch.float to torch.long


Instruction: Memory Issue while following LM tutorial

Input:
SPECS:
OS: Windows 10
CUDA: 10.1
GPU: RTX 2060 6G VRAM (x2)
RAM: 32GB
tutorial: https://huggingface.co/blog/how-to-train
Hello I am trying to train my own language model and I have had some memory issues. I have tried to run some of this code in Pycharm on my computer and then trying to replicate in my Collab Pro Notebook.
First, my code
from transformers import RobertaConfig, RobertaTokenizerFast, RobertaForMaskedLM, LineByLineTextDataset
from transformers import DataCollatorForLanguageModeling, Trainer, TrainingArguments

config = RobertaConfig(vocab_size=60000, max_position_embeddings=514, num_attention_heads=12, num_hidden_layers=6,
                       type_vocab_size=1)

tokenizer = RobertaTokenizerFast.from_pretrained("./MODEL DIRECTORY", max_len=512)

model = RobertaForMaskedLM(config=config)

print("making dataset")

dataset = LineByLineTextDataset(tokenizer=tokenizer, file_path="./total_text.txt", block_size=128)

print("making c")

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

training_args = TrainingArguments(output_dir="./MODEL DIRECTORY", overwrite_output_dir=True, num_train_epochs=1,
                                  per_gpu_train_batch_size=64, save_steps=10000, save_total_limit=2)
print("Building trainer")
trainer = Trainer(model=model, args=training_args, data_collator=data_collator, train_dataset=dataset,
                  prediction_loss_only=True)
trainer.train()

trainer.save_model("./MODEL DIRECTORY")

"./total_text.txt" being a 1.7GB text file.
PyCharm Attempt
This code on pycharm builds the dataset and then would throw an error saying that my preferred gpu was running out of memory, and that Torch was already using 3.7GiB of memory.
I tried:

import gc doing a gc clear to try to flush what ever was going on my gpu
Decreasing my batch size for my gpu (training only happened on a batch size of 8 resulting in 200,000+ epochs that all took 1.17 seconds)
Setting my os.environ["CUDA_VISIBLE_OBJECTS"] ="" so that torch would have to use my CPU and not my GPU. Still threw same gpu memory error...

So succumbing to the fact that torch, for the time being, was forcing itself to use my gpu, I decided to go to Collab.
Collab Attempt
Collab has different issues with my code. It does not have the memory to build the dataset, and crashes due to RAM shortages. I purchased a Pro account and then increased the usable RAM to 25GB, still memory shortages.
Cheers!

Output: I came to the conclusion that my text file for training was way to big. From the other examples I found, the training text was around 300MB not 1.7GB. In both instances I was asking PyCharm and Collab to pull off a very resource expensive task.


Instruction: PyTorch calculate MSE and MAE

Input:
I would like to calculate the MSE and MAE of the model below. The model is calculating the MSE after each Epoch. What do I need to do to get the overall MSE value, please? Can I use the same code to calculate the MAE? Many Thanks in advance
model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

    global_step += 1

average_accuracy = 0
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)

Output:
First of all, you would want to keep your batch size as 1 during test phase for simplicity.
This maybe task specific, but calculation of MAE and MSE for a heat map regression model are done based on the following equations:

This means that in your code, you should change the lines where you calculate MAE as following
error = torch.abs(preds - targets).sum().data
squared_error = ((preds - targets)*(preds - targets)).sum().data
runnning_mae += error
runnning_mse += squared_error

and later, after the epoch ends,
mse = math.sqrt(running_mse\len(loader_test))
mae = running_mae\len(loader_test)


Instruction: Any ideas on how to efficiently create this matrix/mask?

Input:
I want to make a torch tensor or numpy array efficiently of a matrix that is a shifting window of 1s.
So for example the matrix below would be a window=3.
The diagonal element has 3 1s to it's right, and 3 1s to it's left but it doesn't wrap round like a circulant matrix, so row 1 just has 4 1s.
Has anyone got any ideas, this is to be used as a mask.

Output:
Pytorch provides the tensor.diagonal method, which gives you access to any diagonal of a tensor. To assign a value to the resulting view of your tensor, you can use tensor.copy_. That would give you something like :
def circulant(n, window):
    circulant_t = torch.zeros(n,n)
    # [0, 1, 2, ..., window, -1, -2, ..., window]
    offsets = [0] + [i for i in range(window)] + [-i for i in range(window)]
    for offset in offsets:
        #size of the 1-tensor depends on the length of the diagonal
        circulant_t.diagonal(offset=offset).copy_(torch.ones(n-abs(offset)))
    return circulant_t


Instruction: differing results when using model to infer on a batch vs individual with pytorch

Input:
I have a neural network which takes input tensor of dimension (batch_size, 100, 1, 1) and produces an output tensor of dimension (batch_size, 3, 64, 64).
I have differing results when using model to infer on a batch of two elements and on inferring on  elements individually.
With the below code I initialize a pytorch tensor of dimension (2, 100, 1, 1). I pass this tensor through the model and  I take the first element of the model output and store in variable result1. For result2  I just directly run the first element of my original input tensor through my model.
inputbatch=torch.randn(2, Z_DIM, 1, 1, device=device)
inputElement=inputbatch[0].unsqueeze(0)

result1=model(inputbatch)[0]
result2=model(inputElement)

My expectation was result1 and result2 would be same. But result1 and result2 are entirely different. Could anyone explain why the two outputs differ.

Output:
This is probably because your model has some random processes that are either training specific and you have not disable them (e.g. by using model.eval()) or needed at the model during inference.
To test the above, use:

model = model.eval()

before obtaining result1.


Instruction: PyTorch loop through epoch then output final values for all epoch

Input:
The code below calculates the MSE and MAE values but I have an issue where the values for MAE and MSE don't get store_MAE and store MSE after the end of each epoch. It appears to use the values of the last epoch only. Any idea what I need to do in the code to save the values for each epoch I hope this makes sense. Thanks for your help
    global_step = 0
best_test_error = 10000
MAE_for_all_epochs = []
MSE_for_all_epochs = []
for epoch in range(4):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # backward pass:
        loss = torch.mean((preds - targets)**2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # logging:
        count_error = torch.abs(preds - targets).mean()
        writer.add_scalar('train_loss', loss.item(), global_step=global_step)
        writer.add_scalar('train_count_error', count_error.item(), global_step=global_step)

        print("Step %d, loss=%f, count error=%f" % (global_step,loss.item(),count_error.item()))

        global_step += 1

    mean_test_error = 0
    model.eval()
    for images, paths in tqdm(loader_test):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # logging:
        #error = torch.abs(preds - targets).sum().data
        #squared_error = ((preds - targets)*(preds - targets)).sum().data

        #runnning_mae += error
        #runnning_mse += squared_error

        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

        global_step += 1
        #store_MAE = 0
        #store_MSE = 0

        mean_test_error = mean_test_error / len(loader_test)
        #store_MAE += mean_test_error
        MAE_for_all_epochs = np.append(MAE_for_all_epochs, mean_test_error)

        mse = math.sqrt(loss / len(loader_test))
        #store_MSE +=mse
        MSE_for_all_epochs = np.append(MSE_for_all_epochs, mse)

        print("Test count error: %f" % mean_test_error)
        print("MSE: %f" % mse)

    if mean_test_error < best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)

print("MAE Total: %f" % store_MAE)
print("MSE Total: %f" % store_MSE)
model_mae= MAE_for_all_epochs / epoch
model_mse= MSE_for_all_epochs / epoch
print("Model MAE: %f" % model_mae)
print("Model MSE: %f" % model_mse)

Output:
np.append() will work for your case like this,
#outside epochs loop
MAE_for_all_epochs = []

#inside loop
#replace this store_MAE with relevant variable

MAE_for_all_epochs = np.append(MAE_for_all_epochs, store_MAE)

Edit: Toy example: as per the usage
import numpy as np
all_var = []

for e in range(1, 10):
    var1 = np.random.random(1)
    all_var = np.append(all_var, var1)

print(all_var)
# output : [0.07660848 0.46824825 0.09432051 0.79462902 0.97798061 0.67299183 0.50996432 0.13084029 0.95100381]


Instruction: Given a set t of tuples containing elements from the set S, what is the most efficient way to build another set whose members are not contained in t?

Input:
For example, suppose I had an (n,2) dimensional tensor t whose elements are all from the set S containing random integers. I want to build another tensor d with size (m,2) where individual elements in each tuple are from S, but the whole tuples do not occur in t.
E.g.
S = [0,1,2,3,7]
t = [[0,1],
     [7,3],
     [3,1]]

d = some_algorithm(S,t)

/*
d =[[2,1],
    [3,2],
    [7,4]]
*/

What is the most efficient way to do this in python? Preferably with pytorch or numpy, but I can work around general solutions.
In my naive attempt, I just use
d = np.random.choice(S,(m,2))
non_dupes = [i not in t for i in d]
d = d[non_dupes]

But both t and S are incredibly large, and this takes an enormous amount of time (not to mention, rarely results in a (m,2) array). I feel like there has to be some fancy tensor thing I can do to achieve this, or maybe making a large hash map of the values in t so checking for membership in t is O(1), but this produces the same issue just with memory. Is there a more efficient way?
An approximate solution is also okay.

Output:
my naive attempt would be a base-transformation function to reduce the problem to an integer set problem:
definitions and assumptions:

let S be a set (unique elements)
let L be the number of elements in S
let t be a set of M-tuples with elements from S
the original order of the elements in t is irrelevant
let I(x) be the index function of the element x in S
let x[n] be the n-th tuple-member of an element of t
let f(x) be our base-transform function (and f^-1 its inverse)

since S is a set we can write each element in t as a M digit number to the base L using elements from S as digits.
for M=2 the transformation looks like
f(x) = I(x[1])*L^1 + I(x[0])*L^0
f^-1(x) is also rather trivial ... x mod L to get back the index of the least significant digit. floor(x/L) and repeat until all indices are extracted. lookup the values in S and construct the tuple.
since now you can represet t as an integer set (read hastable) calculating the inverse set d becomes rather trivial
loop from L^(M-1) to (L^(M+1)-1) and ask your hashtable if the element is in t or d
if the size of S is too big you can also just draw random numbers against the hashtable for a subset of the inverse of t
does this help you?


Instruction: Something wrong with my checkpoint file when using torch.load()

Input:
When I use torch.load to load one checkpoint:
torch.load('./latest_net_G.pth', map_location='cpu')

I got the runtime error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/conda/lib/python3.7/tarfile.py in nti(s)
    186             s = nts(s, "ascii", "strict")
--> 187             n = int(s.strip() or "0", 8)
    188         except ValueError:

ValueError: invalid literal for int() with base 8: '_v2\nq\x03(('

During handling of the above exception, another exception occurred:

InvalidHeaderError                        Traceback (most recent call last)
/opt/conda/lib/python3.7/tarfile.py in next(self)
   2288             try:
-> 2289                 tarinfo = self.tarinfo.fromtarfile(self)
   2290             except EOFHeaderError as e:

/opt/conda/lib/python3.7/tarfile.py in fromtarfile(cls, tarfile)
   1094         buf = tarfile.fileobj.read(BLOCKSIZE)
-> 1095         obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
   1096         obj.offset = tarfile.fileobj.tell() - BLOCKSIZE

/opt/conda/lib/python3.7/tarfile.py in frombuf(cls, buf, encoding, errors)
   1036
-> 1037         chksum = nti(buf[148:156])
   1038         if chksum not in calc_chksums(buf):

/opt/conda/lib/python3.7/tarfile.py in nti(s)
    188         except ValueError:
--> 189             raise InvalidHeaderError("invalid header")
    190     return n

InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

ReadError                                 Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    555         try:
--> 556             return legacy_load(f)
    557         except tarfile.TarError:

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in legacy_load(f)
    466
--> 467         with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \
    468                 mkdtemp() as tmpdir:

/opt/conda/lib/python3.7/tarfile.py in open(cls, name, mode, fileobj, bufsize, **kwargs)
   1590                 raise CompressionError("unknown compression type %r" % comptype)
-> 1591             return func(name, filemode, fileobj, **kwargs)
   1592

/opt/conda/lib/python3.7/tarfile.py in taropen(cls, name, mode, fileobj, **kwargs)
   1620             raise ValueError("mode must be 'r', 'a', 'w' or 'x'")
-> 1621         return cls(name, mode, fileobj, **kwargs)
   1622

/opt/conda/lib/python3.7/tarfile.py in __init__(self, name, mode, fileobj, format, tarinfo, dereference, ignore_zeros, encoding, errors, pax_headers, debug, errorlevel, copybufsize)
   1483                 self.firstmember = None
-> 1484                 self.firstmember = self.next()
   1485

/opt/conda/lib/python3.7/tarfile.py in next(self)
   2300                 elif self.offset == 0:
-> 2301                     raise ReadError(str(e))
   2302             except EmptyHeaderError:

ReadError: invalid header

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-15-2abbf3aab3ae> in <module>
----> 1 torch.load('multi_task/checkpoints/latest_pet/latest_net_G.pth.tar', map_location='cpu')

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--> 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    558             if zipfile.is_zipfile(f):
    559                 # .zip is used for torch.jit.save and will throw an un-pickling error here
--> 560                 raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
    561             # if not a tarfile, reset file offset and proceed
    562             f.seek(0)

RuntimeError: multi_task/checkpoints/latest_pet/latest_net_G.pth.tar is a zip archive (did you mean to use torch.jit.load()?)

And here's how I save the model:
    def save_networks(self, epoch):
        """Save all the networks to the disk.

        Parameters:
            epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
        """
        for name in self.model_names:
            if isinstance(name, str):
                save_filename = '%s_net_%s.pth' % (epoch, name)
                save_path = os.path.join(self.save_dir, save_filename)
                net = getattr(self, 'net' + name)

                if len(self.gpu_ids) > 0 and torch.cuda.is_available():
                    if name == 'Rgr':
                        torch.save(net.state_dict(), save_path)
                    else:
                        torch.save(net.module.cpu().state_dict(), save_path)
                        net.cuda(self.gpu_ids[0])
                else:
                    if name == 'Rgr':
                        torch.save(net.state_dict(), save_path)
                    else:
                        torch.save(net.cpu().state_dict(), save_path)

I don't know what's wrong with my checkpoint file. Because I can actually load my other checkpoint files successfully. Plus, my pytorch version is 1.1.0. Could you help me with this problem?
Thank you.

Output: I have found the solution. Because I'm using different clusters to train and debug,  torch version of each is different. When saving the model, the torch version is 1.6.0 and it's 1.1.0 when loading it.


Instruction: How to right shift each row of a matrix?

Input:
I have a matrix whose shape is (TxK, and K << T). I want to extend it into shape TxT, and right shift the i-th row with i steps.
For an example:
inputs: T= 5, and K = 3
1 2 3
1 2 3
1 2 3
1 2 3
1 2 3

expected outputs:
1 2 3 0 0
0 1 2 3 0
0 0 1 2 3
0 0 0 1 2
0 0 0 0 1

My solutions:
right_pad = T - K + 1
output = F.pad(input, (0, right_pad), 'constant', value=0)
output = output.view(-1)[:-T].view(T, T)

My solution will cause the error -- gradient computation has been modified by an in-place operation. Is there an efficient and feasible way to achieve my purpose?

Output:
Your function is fine and is not a cause of your error (using PyTorch 1.6.0, if you are using other version, please update your dependencies).
Code below works fine:
import torch
import torch.nn as nn
import torch.nn.functional as F

T = 5
K = 3

inputs = torch.tensor(
    [[1, 2, 3,], [1, 2, 3,], [1, 2, 3,], [1, 2, 3,], [1, 2, 3,],],
    requires_grad=True,
    dtype=torch.float,
)

right_pad = T - K + 1
output = F.pad(inputs, (0, right_pad), "constant", value=0)
output = output.flatten()[:-T].reshape(T, T)

output.sum().backward()

print(inputs.grad)

Please notice I have explicitly specified dtype as torch.float as you can't backprop integers.
view and slice will never break backpropagation, as the gradient is connected to single value, no matter whether it is viewed as 1D or unsqueezed 2D or whatever. Those are not modified in-place. In-place modification breaking gradient could be:
output[0, 3] = 15.

Also, your solution returns this:
tensor([[1., 2., 3., 0., 0.],
        [0., 1., 2., 3., 0.],
        [0., 0., 1., 2., 3.],
        [0., 0., 0., 1., 2.],
        [3., 0., 0., 0., 1.]], grad_fn=<ViewBackward>)

so you have a 3 in the bottom left corner. If that's not what you expect, you should add this line (multiplication by upper triangular matrix with 1) after output = output.flatten()[:-T].reshape(T,T):
output *= torch.triu(torch.ones_like(output))

which gives:
tensor([[1., 2., 3., 0., 0.],
        [0., 1., 2., 3., 0.],
        [0., 0., 1., 2., 3.],
        [0., 0., 0., 1., 2.],
        [0., 0., 0., 0., 1.]], grad_fn=<AsStridedBackward>)

And inputs.grad:
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 0.],
        [1., 0., 0.]])