Instruction: Using SGD on MNIST dataset with Pytorch, loss not decreasing

Input:
I tried to use SGD on MNIST dataset with batch size of 32, but the loss does not decrease at all.
I checked my model, loss function and read documentation but couldn't figure out what I've done wrong.
I defined my neural network as below
class classification(nn.Module):
def __init__(self):
    super(classification, self).__init__()

    # construct layers for a neural network
    self.classifier1 = nn.Sequential(
        nn.Linear(in_features=28*28, out_features=20*20),
        nn.Sigmoid(),
    )
    self.classifier2 = nn.Sequential(
        nn.Linear(in_features=20*20, out_features=10*10),
        nn.Sigmoid(),
    )
    self.classifier3 = nn.Sequential(
        nn.Linear(in_features=10*10, out_features=10),
        nn.LogSoftmax(dim=1),
    )

def forward(self, inputs):                 # [batchSize, 1, 28, 28]
    x = inputs.view(inputs.size(0), -1)    # [batchSize, 28*28]
    x = self.classifier1(x)                # [batchSize, 20*20]
    x = self.classifier2(x)                # [batchSize, 10*10]
    out = self.classifier3(x)              # [batchSize, 10]

    return out

And I defined my training process as below

classifier = classification().to("cuda")
#optimizer
optimizer = torch.optim.SGD(classifier.parameters(), lr=learning_rate_value)
#loss function
criterion = nn.NLLLoss()
batch_size=32
epoch = 30
#array to save loss history
loss_train_arr=np.zeros(epoch)

#used DataLoader to make split batch
batched_train = torch.utils.data.DataLoader(training_set, batch_size, shuffle=True)

for i in range(epoch):

    loss_train=0

    #train and compute loss, accuracy
    for img, label in batched_train:
        img=img.to(device)
        label=label.to(device)

        optimizer.zero_grad()
        predicted = classifier(img)

        label_predicted = torch.argmax(predicted,dim=1)
        loss = criterion(predicted, label)
        loss.backward
        optimizer.step()
        loss_train += loss.item()

    loss_train_arr[i]=loss_train/(len(batched_train.dataset)/batch_size)

I am using a model with LogSoftmax layer, so my loss function seems right. But the loss does not decrease at all.

Output: If the code you posted is the exact code you use, the problem is that you don't actually call backward on the loss (missing parentheses ()).


Instruction: How to apply distance IoU loss?

Input:
I'm currently training custom dataset using this repository: https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch.
The result of training is not satisfactory for me, so I'm gonna change the regression loss, which is L1-smooth loss, into distance IoU loss.
The code for regresssion loss for this repo is below:
                anchor_widths_pi = anchor_widths[positive_indices]
                anchor_heights_pi = anchor_heights[positive_indices]
                anchor_ctr_x_pi = anchor_ctr_x[positive_indices]
                anchor_ctr_y_pi = anchor_ctr_y[positive_indices]

                gt_widths = assigned_annotations[:, 2] - assigned_annotations[:, 0]
                gt_heights = assigned_annotations[:, 3] - assigned_annotations[:, 1]
                gt_ctr_x = assigned_annotations[:, 0] + 0.5 * gt_widths
                gt_ctr_y = assigned_annotations[:, 1] + 0.5 * gt_heights

                # efficientdet style
                gt_widths = torch.clamp(gt_widths, min=1)
                gt_heights = torch.clamp(gt_heights, min=1)

                targets_dx = (gt_ctr_x - anchor_ctr_x_pi) / anchor_widths_pi
                targets_dy = (gt_ctr_y - anchor_ctr_y_pi) / anchor_heights_pi
                targets_dw = torch.log(gt_widths / anchor_widths_pi)
                targets_dh = torch.log(gt_heights / anchor_heights_pi)

                targets = torch.stack((targets_dy, targets_dx, targets_dh, targets_dw))
                targets = targets.t()

                # L1 loss
                regression_diff = torch.abs(targets - regression[positive_indices, :])

                regression_loss = torch.where(
                    torch.le(regression_diff, 1.0 / 9.0),
                    0.5 * 9.0 * torch.pow(regression_diff, 2),
                    regression_diff - 0.5 / 9.0

The code that i'm using as distance IoU is below:
    rows = bboxes1.shape[0]
    cols = bboxes2.shape[0]
    dious = torch.zeros((rows, cols))
    if rows * cols == 0:
        return dious
    exchange = False

    bboxes1 = bboxes1.index_select(1, torch.LongTensor([1, 0, 3, 2]).to('cuda'))

    if bboxes1.shape[0] > bboxes2.shape[0]:
        bboxes1, bboxes2 = bboxes2, bboxes1
        dious = torch.zeros((cols, rows))
        exchange = True

    w1 = bboxes1[:, 2] - bboxes1[:, 0]
    h1 = bboxes1[:, 3] - bboxes1[:, 1]
    w2 = bboxes2[:, 2] - bboxes2[:, 0]
    h2 = bboxes2[:, 3] - bboxes2[:, 1]

    area1 = w1 * h1
    area2 = w2 * h2

    center_x1 = (bboxes1[:, 2] + bboxes1[:, 0]) / 2
    center_y1 = (bboxes1[:, 3] + bboxes1[:, 1]) / 2
    center_x2 = (bboxes2[:, 2] + bboxes2[:, 0]) / 2
    center_y2 = (bboxes2[:, 3] + bboxes2[:, 1]) / 2

    inter_max_xy = torch.min(bboxes1[:, 2:],bboxes2[:, 2:])
    inter_min_xy = torch.max(bboxes1[:, :2],bboxes2[:, :2])
    out_max_xy = torch.max(bboxes1[:, 2:],bboxes2[:, 2:])
    out_min_xy = torch.min(bboxes1[:, :2],bboxes2[:, :2])

    inter = torch.clamp((inter_max_xy - inter_min_xy), min=0)
    inter_area = inter[:, 0] * inter[:, 1]
    inter_diag = (center_x2 - center_x1)**2 + (center_y2 - center_y1)**2
    outer = torch.clamp((out_max_xy - out_min_xy), min=0)
    outer_diag = (outer[:, 0] ** 2) + (outer[:, 1] ** 2)
    union = area1+area2-inter_area
    dious = inter_area / union - (inter_diag) / outer_diag
    dious = torch.clamp(dious,min=-1.0,max = 1.0)
    if exchange:
        dious = dious.T

    loss = 1 - dious
    return loss

The question is here:

Should I apply the distance IoU loss for one target bbox to all pred bbox?
For example, there is 2 annotated bboxes, and 1000 predicted bboxes.
Should I calculate losses for twice like each annotated bbox vs 1000 predicted bboxes?

Should I change the predicted bbox into real coordinates for calculation?

Output:
I've seen this done a couple ways, but typically the methods work by assigning the boxes. Calculating 1000x2 array of IOU, you can assign each prediction box a ground truth target and threshold the IOU for good/bad predictions as seen in RetinaNet or assign each ground truth target the best prediction box as seen in older YOLO. Either way, the loss is applied only to the assigned box pairs, not each combination so each assigned prediction focuses on a single target.

DIOU is invariant to scale


Instruction: Determine whether a model is pytorch model or a tensorflow model or scikit model

Input:
If I want to determine the type of model i.e. from which framework was it made programmatically, is there a way to do that?
I have a model in some serialized manner(Eg. a pickle file). For simplicity purposes, assume that my model can be either tensorflow's, pytorch's or scikit learn's. How can I determine programmatically which one of these 3 is the one?

Output:
AFAIK, I have never heard of Tensorflow/Keras and Pytorch models to be saved  with pickle or joblib - these frameworks provide their own functionality for saving & loading models: see the SO threads Tensorflow: how to save/restore a model? and Best way to save a trained model in PyTorch?. Additionally, there is a Github thread reporting all kinds of issues when trying to save Tensorflow models with pickle and joblib.
Given that, if you have loaded a model with, say, pickle, it is trivial to see what type it is using type(model) and model. Here is a short demonstration with a scikit-learn linear regression model:
import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([[1, 1], [1, 2], [2, 2], [2, 3]])
y = np.dot(X, np.array([1, 2])) + 3
reg = LinearRegression()
reg.fit(X, y)

# save it

import pickle

filename = 'model1.pkl'
pickle.dump(reg, open(filename, 'wb'))

Now, loading the model:
loaded_model = pickle.load(open(filename, 'rb'))

type(loaded_model)
# sklearn.linear_model._base.LinearRegression

loaded_model
# LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

This will also work with frameworks like XGBoost, LightGBM, CatBoost etc.


Instruction: Windows keeps crashing when trying to install PyTorch via pip

Input:
I am currently trying to install PyTorch (using the installation commands posted on PyTorch.org for pip) and when I run the command, my computer completely freezes.
I tried this multiple times with the same result. I had to restart the computer a few times as well. On my current try, I put "-v" when trying to install and the pip seems to be stuck on
"Looking up  in the cache".
I do not know how to proceed. As I mentioned, I've already tried this method multiple times. It worked the first time but did not install PyTorch as it gave me an error for not using "--user".
Are there any solutions to this?
EDIT: I did want to add that I have Python 3.8.6 (64bit)

Output:
After troubling shooting and a lot of restart, it seems like the issue came from when pip was trying to load a pre-downloaded file. Essentially, the first time I ran the installation command, pip downloaded files for pytorch but did not install pytorch due to some user privilege issue. The fix is to add --no-cache-dir in the pip install command. This will override the cache (pre-downloaded files) and download the files all over again.
For me specifically, I also needed to add --user.
In other words, the command went from
pip install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

to
pip --no-cache-dir install torch===1.7.0+cu110 torchvision===0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html --user


Instruction: What is tape-based autograd in Pytorch?

Input:
I understand autograd is used to imply automatic differentiation. But what exactly is tape-based autograd in Pytorch and why there are so many discussions that affirm or deny it.
For example:
this

In pytorch, there is no traditional sense of tape

and this

We don’t really build gradient tapes per se. But graphs.

but not this

Autograd is now a core torch package for automatic differentiation. It
uses a tape based system for automatic differentiation.

And for further reference, please compare it with GradientTape in Tensorflow.

Output:
There are different types of automatic differentiation e.g. forward-mode, reverse-mode, hybrids; (more explanation). The tape-based autograd in Pytorch simply refers to the uses of reverse-mode automatic differentiation, source. The reverse-mode auto diff is simply a technique used to compute gradients efficiently and it happens to be used by backpropagation, source.

Now, in PyTorch, Autograd is the core torch package for automatic differentiation. It uses a tape-based system for automatic differentiation. In the forward phase, the autograd tape will remember all the operations it executed, and in the backward phase, it will replay the operations.
Same in TensorFlow, to differentiate automatically, It also needs to remember what operations happen in what order during the forward pass. Then, during the backward pass, TensorFlow traverses this list of operations in reverse order to compute gradients. Now, TensorFlow provides the tf.GradientTape API for automatic differentiation; that is, computing the gradient of computation with respect to some inputs, usually tf.Variables. TensorFlow records relevant operations executed inside the context of a tf.GradientTape onto a tape. TensorFlow then uses that tape to compute the gradients of a recorded computation using reverse mode differentiation.
So, as we can see from the high-level viewpoint, both are doing the same operation. However, during the custom training loop, the forward pass and calculation of the loss are more explicit in TensorFlow as it uses tf.GradientTape API scope whereas in PyTorch it's implicit for these operations but it requires to set required_grad flags to False temporarily while updating the training parameters (weights and biases). For that, it uses torch.no_grad API explicitly. In other words, TensorFlow's tf.GradientTape() is similar to PyTorch's loss.backward(). Below is the simplistic form in the code of the above statements.
# TensorFlow
[w, b] = tf_model.trainable_variables
for epoch in range(epochs):
  with tf.GradientTape() as tape:
    # forward passing and loss calculations
    # within explicit tape scope
    predictions = tf_model(x)
    loss = squared_error(predictions, y)

  # compute gradients (grad)
  w_grad, b_grad = tape.gradient(loss, tf_model.trainable_variables)

  # update training variables
  w.assign(w - w_grad * learning_rate)
  b.assign(b - b_grad * learning_rate)

# PyTorch
[w, b] = torch_model.parameters()
for epoch in range(epochs):
  # forward pass and loss calculation
  # implicit tape-based AD
  y_pred = torch_model(inputs)
  loss = squared_error(y_pred, labels)

  # compute gradients (grad)
  loss.backward()

  # update training variables / parameters
  with torch.no_grad():
    w -= w.grad * learning_rate
    b -= b.grad * learning_rate
    w.grad.zero_()
    b.grad.zero_()

FYI, in the above, the trainable variables (w, b) are manually updated in both frameworks but we generally use an optimizer (e.g. adam) to do the job.
# TensorFlow
# ....
# update training variables
optimizer.apply_gradients(zip([w_grad, b_grad], model.trainable_weights))

# PyTorch
# ....
# update training variables / parameters
optimizer.step()
optimizer.zero_grad()


Instruction: Detecting fixed size objects in variable sized images

Input:
Neural networks can be trained to recognize an object, then detect occurrences of that object in an image, regardless of their position and apparent size. An example of doing this in PyTorch is at https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98
As the text observes,

Most of the code deals with resizing the image to a 416px square while maintaining its aspect ratio and padding the overflow.

So the idea is that the model always deals with 416px images, both in training and in the actual object detection. Detected objects, being only part of the image, will typically be smaller than 416px, but that's okay because the model has been trained to detect patterns in a scale-invariant way. The only thing fixed is the size in pixels of the input image.
I'm looking at a context in which it is necessary to do the reverse: train to detect patterns of a fixed size, then detect them in a variable sized image. For example, train to detect patterns 10px square, then look for them in an image that could be 500px or 1000px square, without resizing the image, but with the assurance that it is only necessary to look for 10px occurrences of the pattern.
Is there an idiomatic way to do this in PyTorch?

Output: Even if you trained your detector with a fixed size image, you can use a different sizes at inference time because everything is convolutional in faster rcnn/yolo architectures. On the other hand, if you only care about 10X10 bounding box detections,  you can easily define this as your anchors. I would recomend to you to use the detectron2 framework which is implemented in pytorch and is easily configurable/hackable.


Instruction: `*** RuntimeError: mat1 dim 1 must match mat2 dim 0` whenever I run model(images)

Input:
    def __init__(self):
        super().__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=5, stride=2, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(),

            nn.Conv2d(64, 64, kernel_size=3, stride=2, bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(),

            nn.Conv2d(64, 64, kernel_size=3, stride=2, bias=False),
            nn.BatchNorm2d(64),

        )

How can I deal with this error? I think the error is with self.fc, but I can't say how to fix it.

Output:
The output from self.conv(x) is of shape torch.Size([32, 64, 2, 2]): 32*64*2*2= 8192 (this is equivalent to (self.conv_out_size). The input to fully connected layer expects a single dimension vector i.e. you need to flatten it before passing to a fully connected layer in the forward function.
i.e.
class Network():
    ...
    def foward():
    ...
        conv_out = self.conv(x)
        print(conv_out.shape)
        conv_out = conv_out.view(-1, 32*64*2*2)
        print(conv_out.shape)
        x = self.fc(conv_out)
        return x

output
torch.Size([32, 64, 2, 2])
torch.Size([1, 8192])

EDIT:
I think you're using self._get_conv_out function wrong.
It should be
    def _get_conv_out(self, shape):
        output = self.conv(torch.zeros(1, *shape)) # not (32, *size)
        return int(numpy.prod(output.size()))

then, in the forward pass, you can use
        conv_out = self.conv(x)
        # flatten the output of conv layers
        conv_out = conv_out.view(conv_out.size(0), -1)
        x = self.fc(conv_out)

For an input of (32, 1, 110, 110), the output should be torch.Size([32, 2]).


Instruction: PyTorch - RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 244 in dimension 2

Input:
I am trying to do semantic segmentation with two classes.
I have 224x224x3 images and 224x224 binary segmentation masks. I am reshaping the masks to be 224x224x1 (I read somewhere that this is the format that I should pass to the model).
When I try to loop through the train data loader it either runs without errors or I get the following error (randomly):
Traceback (most recent call last):
  File "/Users/nikolaykolibarov/Desktop/GATE/rooftop-recognition/rooftop-edge-segmentation-ml/main.py", line 43, in <module>
    for i, sample in enumerate(train_loader):
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/Users/nikolaykolibarov/opt/anaconda3/envs/dtcc/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 224 and 244 in dimension 2 at /tmp/pip-req-build-9oilk29k/aten/src/TH/generic/THTensor.cpp:689

Most of the issues with the same exception I found online don’t have matching numbers like I do (Got 224 and 224). I don’t understand, this exception should occur when they aren’t equal (if I am not wrong).
When I run it, occasionally I get this error a few times, and then it doesn’t occur a few times.
I am not sure if I messed something up with the data types and shapes or it is something else.
Here is the code:
roof_edges_dataset.py
import os
import cv2 as cv
from torch.utils.data import Dataset
from torchvision.transforms import transforms

from utils import create_binary_mask, get_labelme_shapes, plot_segmentation_dataset

class RoofEdgesDataset(Dataset):
    def __init__(self, im_path, ann_path, transform=None):
        self.im_path = im_path
        self.ann_path = ann_path
        self.transform = transform

        self.im_fn_list = sorted(os.listdir(im_path), key=lambda x: int(x.split('.')[0]))
        self.ann_fn_list = sorted(os.listdir(ann_path), key=lambda x: int(x.split('.')[0]))

    def __len__(self):
        return len(self.im_fn_list)

    def __getitem__(self, index):
        im_path = os.path.join(self.im_path, self.im_fn_list[index])
        im = cv.imread(im_path)

        ann_path = os.path.join(self.ann_path, self.ann_fn_list[index])
        ann = create_binary_mask(im, get_labelme_shapes(ann_path))
        ann = ann.reshape(ann.shape[0], ann.shape[1], 1)
        ann = transforms.ToTensor()(ann)

        if self.transform:
            im = self.transform(im)

        return im, ann

main.py:
import torch
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

from roof_edges_dataset import RoofEdgesDataset

# Device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Hyperparameters
in_im_shape = (3, 224, 224)
num_classes = 2  # Edge / Non-edge
learning_rate = 0.001
batch_size = 4
n_epochs = 10

# Data - 60% Train - 20% Val - 20% Test
transformations = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

dataset = RoofEdgesDataset(im_path='data/images', ann_path='data/annotations', transform=transformations)

train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, test_size])

train_size = int(0.8 * len(train_dataset))
val_size = len(train_dataset) - train_size
train_dataset, val_dataset = torch.utils.data.random_split(train_dataset, [train_size, val_size])

train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(dataset=val_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=True)

# Model

# Loss and Optimizer

# Train
for epoch in range(n_epochs):
    for batch_idx, (image, annotation) in enumerate(train_loader):
        image = image.to(device=device)
        annotation = annotation.to(device=device)
        print(image.shape)
        print(annotation.shape)

    break

# Evaluate

I would be also very thankful if you let me know if something is done in a wrong or stupid way and how to do it better.
Thanks in advance!

Output:
It seems like your training images have different sizes, e.g. one is 224x224 while another is 244x244.
Add a crop/resize transformation to all images so they can be concatenated together to form a batch.


Instruction: Organize tensorboard graph with pytorch lightning

Input:
I've added the default tensorboard logger (from pytorch_lightning.loggers import TensorBoardLogger) to my pytorch lightning Trainer with log_graph=True.
When I train my model, the first view of my graph shows three blocks:
inputs => MyNetworkClassName => Outputs
So far so good.
But then, when I expand MyNetworkClassName it gives me absolutely everything that's going on in my net. That's a lot of arrows going everywhere. I would like to organize this graph into simpler blocks with expandable subgraphs. So in my case, where my network has a typical encoder - enhancer - decoder structure, I would like something more like this:

First graph:

inputs => MyNetworkClassName => Outputs

zooming in on MyNetworkClassName:

encoder => enhancer -> decoder

zooming in on encoder:

encoder_layer1 => encoder_layer2 => ...

zooming in on encoder_layer1:

conv2d => batchnorm

What are my options here? Should I put everything into separate classes? Are there any commands that allow me to group certain actions together?

Output:
Refractoring code into classes also affects the tensorboard graph (where refractoring into methods does not). Typical example class that will show up as an expandable block:
class EncoderLayer(nn.Module):
"""Encoder layer class"""

    def __init__(self, activation_function, kernel_num, kernel_size, idx):
        super().__init__()
        self.layer = nn.Sequential(
            ComplexConv2d(
            kernel_num[idx],
            kernel_num[idx + 1],
            kernel_size=(kernel_size, 2),
        ),
        nn.BatchNorm2d(kernel_num[idx + 1])
        activation_function,
    )

def forward(self, x):
    return self.layer(x)


Instruction: How to use the PyTorch Transformer with multi-dimensional sequence-to-seqence?

Input:
I'm trying to go seq2seq with a Transformer model. My input and output are the same shape (torch.Size([499, 128]) where 499 is the sequence length and 128 is the number of features.
My input looks like:

My output looks like:

My training loop is:
    for batch in tqdm(dataset):
        optimizer.zero_grad()
        x, y = batch

        x = x.to(DEVICE)
        y = y.to(DEVICE)

        pred = model(x, torch.zeros(x.size()).to(DEVICE))

        loss = loss_fn(pred, y)
        loss.backward()
        optimizer.step()

My model is:
import math
from typing import final
import torch
import torch.nn as nn

class Reconstructor(nn.Module):
    def __init__(self, input_dim, output_dim, dim_embedding, num_layers=4, nhead=8, dim_feedforward=2048, dropout=0.5):
        super(Reconstructor, self).__init__()

        self.model_type = 'Transformer'
        self.src_mask = None
        self.pos_encoder = PositionalEncoding(d_model=dim_embedding, dropout=dropout)
        self.transformer = nn.Transformer(d_model=dim_embedding, nhead=nhead, dim_feedforward=dim_feedforward, num_encoder_layers=num_layers, num_decoder_layers=num_layers)
        self.decoder = nn.Linear(dim_embedding, output_dim)
        self.decoder_act_fn = nn.PReLU()

        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        nn.init.zeros_(self.decoder.weight)
        nn.init.uniform_(self.decoder.weight, -initrange, initrange)

    def forward(self, src, tgt):

        pe_src = self.pos_encoder(src.permute(1, 0, 2))  # (seq, batch, features)
        transformer_output = self.transformer_encoder(pe_src)
        decoder_output = self.decoder(transformer_output.permute(1, 0, 2)).squeeze(2)
        decoder_output = self.decoder_act_fn(decoder_output)
        return decoder_output

My output has a shape of torch.Size([32, 499, 128]) where 32 is batch, 499 is my sequence length and 128 is the number of features. But the output has the same values:
tensor([[[0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         ...,
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017],
         [0.0014, 0.0016, 0.0017,  ..., 0.0018, 0.0021, 0.0017]]],
       grad_fn=<PreluBackward>)

What am I doing wrong? Thank you so much for any help.

Output:
There are several points to be checked. As you have same output to the different inputs, I suspect that some layer zeros out all it's inputs. So check the outputs of the PositionalEncoding and also Encoder block of the Transformer, to make sure they are not constant. But before that, make sure your inputs differ (try to inject noise, for example).
Additionally, from what I see in the pictures, your input and output are speech signals and was sampled at 22.05kHz (I guess), so it should have ~10k features, but you claim that you have only 128. This is another place to check. Now, the number 499 represent some time slice. Make sure your slices are in reasonable range (20-50 msec, usually 30). If it is the case, then 30ms by 500 is 15 seconds, which is much more you have in your example.  And finally you are masking off a third of a second of speech in your input, which is too much I believe.
I think it would be useful to examine Wav2vec and Wav2vec 2.0 papers, which tackle the problem of self supervised training in speech recognition domain using Transformer Encoder with great success.


Instruction: PyTorch dataloader shows odd behavior with string dataset

Input:
I'm working on an NLP problem and am using PyTorch. For some reason, my dataloader is returning malformed batches. I have input data that comprises sentences and integer labels.
The sentences can either a list of sentences or a list of list of tokens. I will later convert the tokens to integers in a downstream component.
list_labels = [ 0, 1, 0]

# List of sentences.
list_sentences = [ 'the movie is terrible',
                   'The Film was great.',
                   'It was just awful.']

# Or list of list of tokens.
list_sentences = [['the', 'movie', 'is', 'terrible'],
                  ['The', 'Film', 'was', 'great.'],
                  ['It', 'was', 'just', 'awful.']]

I created the following custom dataset:
import torch
from torch.utils.data import DataLoader, Dataset

class MyDataset(torch.utils.data.Dataset):

    def __init__(self, sentences, labels):

        self.sentences = sentences
        self.labels = labels

    def __getitem__(self, i):
        result = {}
        result['sentences'] = self.sentences[i]
        result['label'] = self.labels[i]
        return result

    def __len__(self):
        return len(self.labels)

When I provide input in the form of a list of sentences, the dataloader correctly returns batches of complete sentences. Note that batch_size=2:
list_sentences = [ 'the movie is terrible', 'The Film was great.', 'It was just awful.']
list_labels = [ 0, 1, 0]

dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': ['the movie is terrible', 'The Film was great.'], <-- Great! 2 sentences in batch!
#  'label': tensor([0, 1])}

The batch correctly contains two sentences and two labels because batch_size=2.
However, when I instead enter the sentences as pre-tokenized list of list of token, I get weird results:
list_sentences = [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.'], ['It', 'was', 'just', 'awful.']]
list_labels = [ 0, 1, 0]

dataset = MyDataset(list_sentences, list_labels)
dataloader = DataLoader(dataset, batch_size=2)

batch = next(iter(dataloader))
print(batch)
# {'sentences': [('the', 'The'), ('movie', 'Film'), ('is', 'was'), ('terrible', 'great.')], <-- WHAT?
#  'label': tensor([0, 1])}

Note that this batch's sentences is one single list with tuples of word pairs. I was expecting sentences to be a list of two lists, like this:
{'sentences': [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.']

What is going on?

Output:
This behavior is because the default collate_fn does the following when it has to collate lists (which is the case for ['sentences']):
# [...]
elif isinstance(elem, container_abcs.Sequence):
    # check to make sure that the elements in batch have consistent size
    it = iter(batch)
    elem_size = len(next(it))
    if not all(len(elem) == elem_size for elem in it):
        raise RuntimeError('each element in list of batch should be of equal size')
    transposed = zip(*batch)
    return [default_collate(samples) for samples in transposed]

The "problem" happens because, in the last two lines, it'll recursively call zip(*batch) while the batch is a container_abcs.Sequence (and list is), and zip behaves like this.
As you can see:
batch = [['the', 'movie', 'is', 'terrible'], ['The', 'Film', 'was', 'great.']]
list(zip(*batch))

# [('the', 'The'), ('movie', 'Film'), ('is', 'was'), ('terrible', 'great.')]

I don't see a workaround in your case, except implementing a new collator and passing it to the DataLoader(..., collate_fn=mycollator). For instance, a simple ugly one could be:
def mycollator(batch):
    assert all('sentences' in x for x in batch)
    assert all('label' in x for x in batch)
    return {
        'sentences': [x['sentences'] for x in batch],
        'label': torch.tensor([x['label'] for x in batch])
    }


Instruction: how would someone install pytorch using pip on python 3.9?

Input:
when i tried the command the website used pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
and it returned the error code
ERROR: Could not find a version that satisfies the requirement torch==1.7.0+cpu (from versions: 0.1.2,
0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu

even when i change the code to satisfy the requirement it comes up with more errors like
ModuleNotFoundError: No module named 'tools.setup_helpers'

Output:
Searching for "1.7.0" at the https://download.pytorch.org/whl/torch_stable.html page, you can see that only wheels for Python 3.6, 3.7, and 3.8 are available.
So there is no official wheel for Python 3.9.
Update:
PyTorch currently doesn't work with Python 3.9, there are known issues being resolved: https://github.com/pytorch/pytorch/issues/47776


Instruction: Unsupported ONNX opset version: 11

Input:
I'm following this guide to convert darknet to onnx. However, I'm facing the following error:

"C:\Users\Scott\Anaconda3\envs\pytorch_yolov4\lib\site-packages\torch\onnx\symbolic_helper.py",
line 253, in _set_opset_version
raise ValueError("Unsupported ONNX opset version: " + str(opset_version)) ValueError: Unsupported ONNX opset version: 11

What does this error mean and how to deal with it?

Output:
It looks like you have an old PyTorch version, probably PyTorch 1.2.
The docs here https://github.com/Tianxiaomo/pytorch-YOLOv4#4-pytorch2onnx recommend at least PyTorch 1.4.


Instruction: How to properly reshape a 3D tensor to a 2D forward linear layer then reshape new 3D tensor's fibers corresponding to the old 3D

Input:
I have a 3D tensor of names that comes out of an LSTM that's of shape (batch size x name length x embedding size)
I've been reshaping it to a 2D to put it through a linear layer, because linear layer requires (batch size, linear dimension size) by using the following
y0 = output.contiguous().view(-1, output.size(-1))

this converts outputs to (batchsize, name length * number of possible characters)
then I put y0 through a linear layer and then reshape it back to a 3D using
y = y0.contiguous().view(output.size(0), -1, y0.size(-1))

But I'm not really sure if the fibers of y are correlated properly with the cells of output and I worry this is messing up my learning, because batch size of 1 is actually generating proper names and any larger batch size is generating nonsense.
So what I mean exactly is
output = (batch size * name length, embed size)
y =  (batch size * name length, number of possible characters)
I need to make sure y[i,j,:] is the linear transformed version of output[i,j,:]

Output:
It seems like you are using an older code example. Just 'comment out' the lines of code where you reshape the tensor as there is no need for them.
This link gives you a bit more explaination: https://discuss.pytorch.org/t/when-and-why-do-we-use-contiguous/47588
Try something like this instead and take the output from the LSTM directly into the linear layer:
output, hidden = self.lstm(x, hidden)
output = self.LinearLayer1(output)


Instruction: How to index with list of indices in a multidimensional array in numpy/pytorch

Input:
I have an two multi-dimensional arrays of shape (1,169,5,1+4+4). One contains data, and the other contains zeros that is to be filled.
>>> preds = np.random.random_integers(0,500,(1,169,5,1+4+4))
>>> filtered_preds = np.zeros((1,169,5,1+4+4))

For context the dimensions represent the following:

batch size
number of grid cells
number of anchor boxes
predictions (Confidence score, bounding box coords, number of classes)

I have a third flattened array of shape (169) that contains indicies of the best anchor box
Which looks like the following:
>>> best_bboxes = np.random.random_integers(0,4,(169,))

In order to prevent using loops, my first instinct was to try the following to place the confidence values:
>>> filtered_preds[:,:,best_ab_idx,0:1] = preds[:,:,best_ab_idx,0:1]

However, upon inspecting the array, I do not get the disired output as the shape of filtered_preds[:,:,best_ab_idx,0:1] is (1,169,169,1)
I guess my question so far is, how do I get values from my prediction array with the size of the second dimension and use the value of the third dimension to get the desired value?
I'm still a bit new to array indexing, so any help would be appreciated.
Note: I am using PyTorch but gave my examples in NumPy as it follows the same principle.
Edit:
Here is a solution of what I am trying to acheive using a loop:
for idx,i in enumerate(best_bboxes):
    filtered_preds[:,idx,best_bboxes[idx],0:1] = preds[:,idx,best_bboxes[idx],0:1]
    filtered_preds[:,idx,best_bboxes[idx],1:5] = preds[:,idx,best_bboxes[idx],1:5]
    filtered_preds[:,idx,best_bboxes[idx],5:] = preds[:,idx,best_bboxes[idx],5:]

Output:
I'm a little confused by the description of what you want to achieve, but I can simplify your given solution with the following.
seq_idx = np.arange(best_bboxes.size)
filtered_preds[:, seq_idx, best_bboxes, :] = preds[:, seq_idx, best_bboxes, :]


Instruction: Pytorch executeable works while running from Anaconda prompt but not from Cmd or .exe?

Input:
I packaged (using Pyinstaller) a small variant of the Minimalistic Yolo github repo, found Here, the packaging was done using pyinstaller to run the object detection as a server using Flask.
So while attempting to run the server, it only works when running from Anaconda Prompt (Which is where i wrote the pyinstaller command) other than that, the following error occur.
Error i Get while running from (exe,Cmd,PowerShell) is:
Traceback (most recent call last):
File "flask\app.py", line 2446, in wsgi_app
File "flask\app.py", line 1951, in full_dispatch_request
File "flask\app.py", line 1820, in handle_user_exception
File "flask\_compat.py", line 39, in reraise
File "flask\app.py", line 1949, in full_dispatch_request
File "flask\app.py", line 1935, in dispatch_request
File "FlaskServerV2.py", line 53, in Hello
File "torch\nn\modules\module.py", line 532, in __call__
File "models.py", line 259, in forward
File "torch\nn\modules\module.py", line 532, in __call__
File "models.py", line 177, in forward
RuntimeError: error in LoadLibraryA
127.0.0.1 - - [19/Nov/2020 10:28:53] "GET /detect HTTP/1.1" 500 -

But when running in conda, the code works fine and runs.
so I am suspecting it's an issue with a PyTorch dependency.
Current Code:
from __future__ import division

from flask import Flask, Response, jsonify
app = Flask(__name__)

from models import *
from utils.utils import *
from utils.datasets import *

import os
import sys
import time
import datetime
import argparse

from PIL import Image

import torch
from torch.utils.data import DataLoader
from torchvision import datasets
from torch.autograd import Variable

import matplotlib.pyplot as plt
import matplotlib.patches as patches
from matplotlib.ticker import NullLocator

import cv2
import time
import json

@app.route('/CheckIfRunning')
def CheckIfRunning():
    return '1'

@app.route('/detect')
def Hello():
    global device
    global model
    global classes
    global colors
    global Tensor
    global a
    img=cv2.imread("temp.jpg")
    PILimg = np.array(Image.fromarray(cv2.cvtColor(img,cv2.COLOR_BGR2RGB)))
    imgTensor = transforms.ToTensor()(PILimg)
    imgTensor, _ = pad_to_square(imgTensor, 0)
    imgTensor = resize(imgTensor, 416)
    #add the batch size
    imgTensor = imgTensor.unsqueeze(0)
    imgTensor = Variable(imgTensor.type(Tensor))
    with torch.no_grad():
        detections = model(imgTensor)
        detections = non_max_suppression(detections,0.8, 0.4)
    a.clear()

    Return={}
    ReturnCounter=0
    if detections is not None:
            a.extend(detections)
            b=len(a)
            if len(a)  :
                for detections in a:
                    if detections is not None:
                        detections = rescale_boxes(detections, 416, PILimg.shape[:2])
                        unique_labels = detections[:, -1].cpu().unique()
                        n_cls_preds = len(unique_labels)
                        for x1, y1, x2, y2, conf, cls_conf, cls_pred in detections:
                            box_w = x2 - x1
                            box_h = y2 - y1
                            color = [int(c) for c in colors[int(cls_pred)]]
                            img = cv2.rectangle(img, (x1, y1 + box_h), (x2, y1), color, 2)
                            cv2.putText(img, classes[int(cls_pred)], (x1, y1),     cv2.FONT_HERSHEY_SIMPLEX, 0.5, color, 2)
                            cv2.putText(img, str("%.2f" % float(conf)), (x2, y2 - box_h), cv2.FONT_HERSHEY_SIMPLEX, 0.5,color, 2)
                            Return[ReturnCounter]=    [x1.item(),y1.item(),x2.item(),y2.item(),conf.item(),cls_conf.item(),classes[int(cls_pred)]]
                            ReturnCounter+=1
                        cv2.imwrite("Temp2.jpg",img)
                        return Return

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Set up model
model = Darknet("config/yolov3.cfg", img_size=416).to(device)

model.load_darknet_weights("weights/yolov3.weights")

model.eval()  # Set in evaluation mode

classes = load_classes("data/coco.names")  # Extracts class labels from file
colors = np.random.randint(0, 255, size=(len(classes), 3), dtype="uint8")
Tensor = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor

a=[]
app.run(threaded=True)

Output:
Alright, turns out this is an issue with pyinstaller.
if Pytorch is installed using Conda, it requires the CUDANN , and it won't work with it (ie without that environment)
if you want it to work every where, Pytorch has to be installed using pip.
For reference,
https://github.com/pyinstaller/pyinstaller/issues/2666#issuecomment-508013383


Instruction: What could cause a VAE(Variational AutoEncoder) to output random noise even after training?

Input:
I have trained a VAE on CIFAR10 data-set. However, when I try to generate images from the VAE all I get is a bunch of gray noise back. The implementation of this VAE follows the implementation from the book Generative Deep Learning, but instead of TensorFlow the code uses PyTorch.
The notebook containing the training as well as the generation can be found here, while the actual implementation of the VAE can be found here.
I have tried:

Disabling dropouts.
Increasing the dimension of the latent space.

None of the methods show any improvement at all.
I have verified that:

The input size matches the output size
Back-propagation runs successfully as the loss decreases during training.

Output:
Thanks for providing the code and a link to a Colab notebook! +1!
Also, your code is well-written and easy to read. Unless I missed something, I think there are two problems with your code:

The data normalization
The implementation of the VAE loss.

About 1., your CIFAR10DataModule class normalizes the RGB channels of the CIFAR10 images using mean = 0.5 and std = 0.5. Since the pixel values are initially in [0,1] range, the normalized images have pixel values in the [-1,1] range. However, your Decoder class applies a nn.Sigmoid() activation to the reconstructed images. Therefore, your reconstructed images have pixel values in the [0,1] range. I suggest to remove this mean-std normalization so that both the "true" images and the reconstructed images have their pixel values in the [0,1] range.
About 2.: since you're dealing with RGB images the MSE loss makes sense. The idea behind the MSE loss is the "Gaussian decoder". This decoder assumes the pixel values of a "true image" is generated by independent Gaussian distributions whose mean is the pixel values of the reconstructed image (i.e. the output of the decoder) and with a given variance. Your implementation of the reconstruction loss (namely r_loss = F.mse_loss(predictions, targets)) is equivalent to a fixed variance. Using ideas from this paper, we can do better and obtain an analytic expression for the "optimal value" of this variance parameter. Finally, the reconstruction loss should be summed over all pixels (reduction = 'sum'). To understand why, have a look at analytic expression of the reconstruction loss (see, for instance, this blog post which considers the BCE loss).
Here is what the refactored LitVAE class looks like:
class LitVAE(pl.LightningModule):
    def __init__(self,
                 learning_rate: float = 0.0005,
                 **kwargs) -> None:
        """
        Parameters
        ----------
        - `learning_rate: float`:
            learning rate for the optimizer
        - `**kwargs`:
            arguments to pass to the variational autoencoder constructor
        """
        super(LitVAE, self).__init__()

        self.learning_rate = learning_rate

        self.vae = VariationalAutoEncoder(**kwargs)

    def forward(self, x) -> _tensor_size_3_t:
        return self.vae(x)

    def training_step(self, batch, batch_idx):
        r_loss, kl_loss, sigma_opt = self.shared_step(batch)
        loss = r_loss + kl_loss

        self.log("train_loss_step", loss)
        return {"loss": loss, 'log':{"r_loss": r_loss / len(batch[0]), "kl_loss": kl_loss / len(batch[0]), 'sigma_opt': sigma_opt}}

    def training_epoch_end(self, outputs) -> None:
        # add computation graph
        if(self.current_epoch == 0):
            sample_input = torch.randn((1, 3, 32, 32))
            sample_model = LitVAE(**MODEL_PARAMS)

            self.logger.experiment.add_graph(sample_model, sample_input)

        epoch_loss = self.average_metric(outputs, "loss")
        self.logger.experiment.add_scalar("train_loss_epoch", epoch_loss, self.current_epoch)

    def validation_step(self, batch, batch_idx):
        r_loss, kl_loss, _ = self.shared_step(batch)
        loss = r_loss + kl_loss

        self.log("valid_loss_step", loss)

        return {"loss": loss}

    def validation_epoch_end(self, outputs) -> None:
        epoch_loss = self.average_metric(outputs, "loss")
        self.logger.experiment.add_scalar("valid_loss_epoch", epoch_loss, self.current_epoch)

    def test_step(self, batch, batch_idx):
        r_loss, kl_loss, _ = self.shared_step(batch)
        loss = r_loss + kl_loss

        self.log("test_loss_step", loss)
        return {"loss": loss}

    def test_epoch_end(self, outputs) -> None:
        epoch_loss = self.average_metric(outputs, "loss")
        self.logger.experiment.add_scalar("test_loss_epoch", epoch_loss, self.current_epoch)

    def configure_optimizers(self):
        return optim.Adam(self.parameters(), lr=self.learning_rate)

    def shared_step(self, batch) -> torch.TensorType:
        # images are both samples and targets thus original
        # labels from the dataset are not required
        true_images, _ = batch

        # perform a forward pass through the VAE
        # mean and log_variance are used to calculate the KL Divergence loss
        # decoder_output represents the generated images
        mean, log_variance, generated_images = self(true_images)

        r_loss, kl_loss, sigma_opt = self.calculate_loss(mean, log_variance, generated_images, true_images)
        return r_loss, kl_loss, sigma_opt

    def calculate_loss(self, mean, log_variance, predictions, targets):
        mse = F.mse_loss(predictions, targets, reduction='mean')
        log_sigma_opt = 0.5 * mse.log()
        r_loss = 0.5 * torch.pow((targets - predictions) / log_sigma_opt.exp(), 2) + log_sigma_opt
        r_loss = r_loss.sum()
        kl_loss = self._compute_kl_loss(mean, log_variance)
        return r_loss, kl_loss, log_sigma_opt.exp()

    def _compute_kl_loss(self, mean, log_variance):
        return -0.5 * torch.sum(1 + log_variance - mean.pow(2) - log_variance.exp())

    def average_metric(self, metrics, metric_name):
        avg_metric = torch.stack([x[metric_name] for x in metrics]).mean()
        return avg_metric

After 10 epochs, that's what the reconstructed images look like:


Instruction: Can we reset .requires_grad of all defined tensors in a code to zero at once

Input:
I am using PyTorch 1.6.0 to learn a tensor (lets say x) with autograd.
After x is learnt, how can I reset .requires_grad of every tensor that was a node in the autograd comp. graph to zero?
I know about torch.detach() and about setting .requires_grad to False manually. I am searching for an one-shot instruction.
Ps: I want to do that because I still want to use these tensors after the part of my code that learns x is executed. Plus, some are to be converted to numpy.

Output:
There is no "one shot instruction" to switch .requires_grad for all tensors in graph.
Usually parameters are kept in torch.nn.Module instances but in case they are elsewhere, you can always add them to some list and iterate over it, I'd do something like this:
import torch

class Leafs:
    def __init__(self):
        self.leafs = []

    def add(self, tensor):
        self.leafs.append(tensor)
        return tensor

    def clear(self):
        for leaf in self.leafs:
            leaf.requires_grad_(False)

keeper = Leafs()

x = keeper.add(torch.tensor([1.2], requires_grad=True))
y = keeper.add(torch.tensor([1.3], requires_grad=True))

print(x.requires_grad, y.requires_grad)

keeper.clear()

print(x.requires_grad, y.requires_grad)

Usually there is no need for that, also if you don't want gradient for some part of computation you can always use with torch.no_grad() context manager.


Instruction: Loading n-dimensional tensor for pytorch from text file

Input:
I have a 3-dimensional tensor that I create outside of my python code and am able to encode in any human-readable format. I need to load the values of this tensor as a frozen layer to my pytorch NN. I've tried to encode the tensor as a text file in the form [[[a,b],[c,d]], [[e,f], [g,h]], [[k,l],[m,n]]] which seemed to be the most logical way for that. Then I tried to read its value via
tensor = torch.from_numpy(np.loadtxt("./arrays/tensor.txt"))

but got the exception in npyio.py
ValueError: could not convert string to float: '[[[-2.888356,'

Apparently, that's not how it works and the values are to be written as plain numbers separated by spaces and \n, but then I don't see how to easily read the data of dimension >= 2 with numpy.
What could be other simple methods to write down and read the tensor value into a pytorch tensor?

Output:
Did you try using Python's built-in eval? In case you saved your tensor as a list in text file you may try something as follows:
with open("./arrays/tensor.txt","r") as f:
    loaded_list = eval(f.read())
loaded_tensor = torch.tensor(loaded_list)

eval will take care of converting your string to a list and then just cast the result to a Tensor by using torch.tensor(). The loaded tensor will then be in loaded_tensor as required.


Instruction: removing sample from a Pytorch dataset object

Input:
I'm loading images to pytorch using:
train_data = torchvision.datasets.ImageFolder(train_dir, transform=transform)

Which has some samples I want to omit.
Can i remove certain images from the loader? (by index? by name?)
Thx

Output:
You can modify the imgs attribute of an ImageFolder as follows:
train_data.imgs.remove((PATH, INDEX))

where you should replace PATH with the path to the image you wish to remove and INDEX should be replaced with the respective class index (starting from 0 to the number of classes you have).
In case you are not sure about the INDEX you can print train_data.imgs to obtain the list of all image paths with their classes in tuples.


Instruction: IndexError: index 39092 is out of bounds for axis 0 with size 39092; I'm trying to train a multi label classification

Input:
I think the problem is caused by the way I load the data from the csv file, but I don't know how to fix it
Here is a small portion of my train csv file:
train_datasets
(I use 15 labels from column 1 to the end)
The error looks like this:
IndexError
IndexError
Code==>
import csv
import os
import pandas as pd
import numpy as np
from PIL import Image
import torch

class FashionData(torch.utils.data.Dataset):
    def __init__(self, csv_file, mode='train', transform=None):
        self.mode=mode
        #label(img1) = [0, 0, 0, 1], label(img3) = [1, 0, 1, 0], ...),
        self.transform = transform

        self.data_info = pd.read_csv(csv_file, header=None)
        #print(self.data_info)

        # First column contains the image paths
        self.image_arr = np.asarray(self.data_info.iloc[1:, 0])
        if mode !='test':
            self.label_arr = np.asarray(self.data_info.iloc[1:, 2:]) # columns 1 to N
            self.label_arr=self.label_arr.astype('float32')

        # Calculate len
        self.data_len = len(self.data_info.index)
    def __getitem__(self, index):
        # Get image name from the pandas df
        single_image_name = self.image_arr[index]
        # Open image
        img_as_img = Image.open(single_image_name)

        if self.transform is not None:
            img_as_img = self.transform(img_as_img)
        if self.mode=='test':
            return img_as_img

        # Get label(class) of the image based on the cropped pandas column
        single_image_label = self.label_arr[index]
        #single_image_label = torch.from_numpy(self.label_arr[index]).float()
        #img = torch.from_numpy(img).float().to(device)
        #label = torch.tensor(int(self.labels[index]))

        return (img_as_img, single_image_label)

    def __len__(self):
        return self.data_len

transforms_train = transforms.Compose([transforms.Resize((224,224)),transforms.RandomHorizontalFlip(),transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
transforms_test = transforms.Compose([transforms.Resize((224,224)),transforms.ToTensor(),transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
dataset_train = FashionData('./deep_fashion/train.csv', mode='train', transform=transforms_train)
dataset_val = FashionData('./deep_fashion/val.csv', mode='val', transform=transforms_test)
dataset_test = FashionData('./deep_fashion/test.csv', mode='test', transform=transforms_test)

from torch.utils.data import DataLoader

train_loader = DataLoader(dataset_train, batch_size=128, shuffle=True)
val_loader = DataLoader(dataset_val, batch_size=128, shuffle=False)
test_loader = DataLoader(dataset_test, batch_size=128, shuffle=False)

model=models.resnet50(pretrained=True)
for params in model.parameters():
  params.requires_grad=False
model.fc=nn.Sequential(
    nn.Linear(2048,15),
    nn.Sigmoid()
    )

device=torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model=model.to(device)
print(model)

criterion=nn.BCELoss()
#criterion=nn.BCEWithLogitsLoss()
optimizer=optim.Adam(model.parameters(),lr=0.001)
criterion=criterion.to(device)

def train(train_loader,model,criterion,optimizer):
  model.train()
  loss_list=[]
  total_count=0
  acc_count=0

  for x,y in train_loader:

    x=x.to(device)
    y=y.to(device)

    optimizer.zero_grad()
    output=model(x)

    loss=criterion(output,y)
    loss.backward()
    optimizer.step()

    #_,predicted=torch.max(output,1)
    predicted=(output>0.5).float()
    total_count+=y.size(0)
    acc_count+=(predicted==y).sum().item()

    loss_list.append(loss.item())
  acc=acc_count/total_count
  loss=sum(loss_list)/len(loss_list)
  return acc, loss

def val(valid_loader,model,criterion):
  model.eval()
  loss_list=[]
  total_count=0
  acc_count=0
  with torch.no_grad():
    for x,y in valid_loader:
      x=x.to(device)

      y=y.to(device)

      output=model(x)
      loss=criterion(output,y)
      #_,predicted=torch.max(output,1)
      predicted=(output>0.5).float()

      total_count+=y.size(0)
      acc_count+=(predicted==y).sum().item()
      loss_list.append(loss.item())

  acc=acc_count/total_count
  loss=sum(loss_list)/len(loss_list)
  return acc, loss

train_acc_list = []
train_loss_list = []
val_acc_list = []
val_loss_list = []

for epoch in range(10):
    train_acc, train_loss = train(train_loader, model, criterion, optimizer)
    val_acc, val_loss=val(val_loader, model, criterion)
    train_acc_list.append(train_acc)
    train_loss_list.append(train_loss)
    val_acc_list.append(val_acc)
    val_loss_list.append(val_loss)
    print('epoch',epoch)
    print('Train Acc: {:.6f} Train Loss: {:.6f}'.format(train_acc, train_loss))
    print('  Val Acc: {:.6f}   Val Loss: {:.6f}'.format(val_acc, val_loss))

Did I do wrong in the loading data part? Or it's the other problem?

11111111111111111
11111111111111111
11111111111111111

Output:
Your __len__ is just 1 greater than your actual data, since you loaded the df with header=None.
Simply change the last line of __init__ to self.data_len = len(self.image_arr). This should fix your problem with minimal change.
(Alternatively, set header=True while loading your df, in which case, you'll have to change iloc[1:, ...] to iloc[:, ...], as you no longer need to skip the first row.)


Instruction: Why does using a numpy array instead of a torch tensor use more memory in this example?

Input:
Using this method gives me a train_data size of about 700Mb and test of about 200Mb
data_tensor = torch.Tensor([np.repeat(image[...,np.newaxis],3,-1)/255.0 for image in data])

np.save(f"/home/vedank/Desktop/code/facial_sentiment/{dataset_type}_data.npy",np.array(data_tensor))

However, using the one below increases the size of train_data to 1.6 Gb and test to ~400Mb
data_tensor = np.array([np.repeat(image[...,np.newaxis],3,-1)/255.0 for image in data])

np.save(f"/home/vedank/Desktop/code/facial_sentiment/{dataset_type}_data.npy",data_tensor)

Output:
It seems that the default datatype for numpy (float) arrays is 64 bits.
import numpy as np
print(np.array([2.3]).dtype)
# we get float 64

on the other hand, pytorch default float datatype is float32
That means the size of the array in numpy should be double the one in PyTorch.


Instruction: Data format and stride

Input:
First of all, What is channel last format? Is it NHWC?
For the following code, how stride is calculated?
import torch
N, C, H, W = 10, 3, 32, 32
x = torch.empty(N, C, H, W)
print(x.stride()) # Ouputs: (3072, 1024, 32, 1)

Output is (3072, 1024, 32, 1)
When converted to channel last format, how stride is calculated here?
x = x.contiguous(memory_format=torch.channels_last)
print(x.shape) # Outputs: (10, 3, 32, 32) as dimensions order preserved
print(x.stride()) # Outputs: (3072, 1, 96, 3)

Output is torch.Size([10, 3, 32, 32])
(3072, 1, 96, 3)

Output:
TLDR: stride is the number of elements in the storage that need to be skipped over to obtain the next element along each dimension.

I believe you're referring to this tutorial.
Since every tensor references an underlying storage, you can have multiple tensors referencing the same storage with different strides and dimensions.
For e.g., I might have a 3x3 tensor:
[1 2 4
 0 4 5
 7 8 0]

but all that data is actually stored in a 1D contiguous storage, i.e. [1 2 4 0 4 5 7 8 0]. In order for the tensor to know how it should presented (i.e. 3x3), we need a stride property which is "the number of elements in the storage that need to be skipped over to obtain the next element along each dimension". The stride for this would be (3,1), meaning I need to step over 3 elements in storage to get my next element in the first dimension and step over 1 element to get the next element in the second dimension. E.g.

starts with 1. To get the next value in the row dimension, add 3 (as indicated by stride). So the third element after 1 in storage is 0.
another e.g., let's start with the 2. To get the next value in the second dimension (col), add 1 in storage, which gives me 4.

That's the basics of how stride is calculated.
Back to your original question,

N, C, H, W = 10, 3, 32, 32

To get the stride, say of N, and let's say N is the n'th image, then I need to skip over (32 x 32 x 3 = 3072) elements in order to get to the start of the next image. Hence why stride() = (3072, 1024, 32, 1) on the first dimension. The same logic applies to the other strides.
From the tutorial linked above, channels last is simply a different way to store the underlying storage object in memory (see the pic in link). The stride changes due to how the contiguous memory is allocated, but the way we interpret and understand stride remains the same.


Instruction: Distinguishing images from noise

Input:
Neural networks have been successfully used in supervised learning to classify images.
I'm currently looking at a different problem: distinguishing images that show something, contain some manifest order, versus images that are just white noise. I'd like to get some numeric estimate of just how ordered an image is.
Clearly there are several ways of doing this, e.g. PNG compressing the image and looking at how much it shrank. (Pure noise is not compressible.) But I am currently interested in trying to do it with neural networks; it seems to me that it should be possible to do this with unsupervised learning, using some sort of autoencoder.
Is this something that has been done? Ideally with PyTorch but I would be happy to look at a solution using any language and framework.

Output:
This question is a bit too broad for stackoverflow, it seems more like an research question. The topic would also be more at home in https://stats.stackexchange.com/ , where machine learning people lurk.
Interesting question though. In my experience going with an analytic solution is always better than trying to learn a solution if possible, I see machine learning as something to fall back on when things get too complex. Detecting images of pure noise is relatively simple, one would expect the Shannon entropy of the image to explode. You could also e.g. match the K-L divergence of the image histogram to a Gaussian or Poisson distribution or whatever the noise distribution you'd expect. If the images are really just noise then I'd expect any of these solutions to classify them very easily and fast with very little tweaking. If you want to learn the answer I'd recommend crafting few information theoretic or simple edge based etc. features and classifying them e.g. with Naive Bayes or an SVM.
If you want to use neural networks then sure, I'm sure you could teach a model to get the concept of information content. Deep image prior is a popular recent article which uses the idea of an autoencoder having lower 'impedance' for structured images. They use it to denoise, impute and do superresolution but I'm sure you could use make an unsupervised or semi-supervised NN based nonsense-detector by following the same train of thought. I'd expect it to be orders of magnitude slower than an simpler hand-crafted information theoretic approach with similiar accuracy or worse.


Instruction: pytorch running: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu

Input:
When I running the python code, there is a runtimeError in line 44:RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    42  feats = self.node_features[self.train_mask]
    43  labels = self.node_labels[train_mask]

    44  A = torch.mm(feats.t(), feats) + 1e-05 * torch.eye(feats.size(1))

    45  labels_one_hot = torch.zeros((feats.size(0), self.n_classes))

Can anyone who knows the reason and help me fix it!  Thanks!

Output:
It seems that the tensor torch.eye(...) is on CPU.
you need to pass it as -
44  A = torch.mm(feats.t(), feats) + 1e-05 * torch.eye(feats.size(1)).to(device='cuda')

or
44  A = torch.mm(feats.t(), feats) + 1e-05 * torch.eye(feats.size(1)).cuda()


Instruction: CNN in Pytorch: labels for a custom dataset

Input:
I have a very basic question that would be really helpful for getting me started on a deep learning project I'm running. I'm planning to apply a 3D convolutional network using pytorch to a large number of medical images to predict a single numerical value. My question is, for the labelled instances (ranging from 0-40) that I'll be using to train my CNN model - how exactly are the labels of the training set fed to the model? Is it usually contained within a csv file that you load in or otherwise is the label embedded in the name of the image or alternatively a folder that contain images of the same class (e.g. folder for "0", folder for "1")?
Any help you could provide would be really appreciated!

Output:
This Functions are very useful if your classes are arranged in folder structure
Provided your folder structure to be like this

train

class1
class2
...
class_n

test

class1
class2
...
class_n

train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
print(train_data.classes) # will print all your classes

Second option what you have is:
torch.utils.data.Dataset is an abstract class representing a dataset. Your custom dataset should inherit Dataset and override the following methods:
__len__() so that len(dataset) returns the size of the dataset.
__getitem__() to support the indexing such that dataset[i] can be used to get ith sample
Eg of writing custom Dataset
class CustomDatasetFromCsvLocation(Dataset):
    def __init__(self, csv_path):
        """
        Custom dataset example for reading image locations and labels from csv
        but reading images from files
        Args:
            csv_path (string): path to csv file
        """
        # Transforms
        self.to_tensor = transforms.ToTensor()
        # Read the csv file
        self.data_info = pd.read_csv(csv_path, header=None)
        # First column contains the image paths
        self.image_arr = np.asarray(self.data_info.iloc[:, 0])
        # Second column is the labels
        self.label_arr = np.asarray(self.data_info.iloc[:, 1])
        # Third column is for an operation indicator
        self.operation_arr = np.asarray(self.data_info.iloc[:, 2])
        # Calculate len
        self.data_len = len(self.data_info.index)

    def __getitem__(self, index):
        # Get image name from the pandas df
        single_image_name = self.image_arr[index]
        # Open image
        img_as_img = Image.open(single_image_name)

        # Check if there is an operation
        some_operation = self.operation_arr[index]
        # If there is an operation
        if some_operation:
            # Do some operation on image
            # ...
            # ...
            pass
        # Transform image to tensor
        img_as_tensor = self.to_tensor(img_as_img)

        # Get label(class) of the image based on the cropped pandas column
        single_image_label = self.label_arr[index]

        return (img_as_tensor, single_image_label)

    def __len__(self):
        return self.data_len


Instruction: What is the .optimizer package in Pytorch?

Input:
I am trying to write my own optimizer for pytorch and am looking at the source code https://pytorch.org/docs/stable/_modules/torch/optim/sgd.html#SGD
to get started. When I try to run the code for the SGD, I get an error on the line
from .optimizer import Optimizer, required. I've searched everywhere but I'm not sure where to obtain the .optimizer package. Any help here is greatly appreciated.

Output: Here import .optimizer means import optimizer.py from the same directory as the current .py file, so this file


Instruction: How to use Pytorch to create a custom EfficientNet with the last layer written correctly

Input:
I have a classification problem to predict 8 classes for example, I am using EfficientNetB3 in pytorch from here. However, I got confused on whether my custom class is correctly written. I think I want to strip the last layer of the pre-trained model to suit the 8 outputs right? Did I do it correctly? Because when I print y_preds = model(images) in my DataLoader, it seems to give me 1536 predictions. Is this an expected behavior?
!pip install geffnet
import geffnet

class EfficientNet(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.model = geffnet.create_model(config.effnet, pretrained=True)
        n_features = self.model.classifier.in_features
        # does the name fc matter?
        self.fc = nn.Linear(n_features, config.num_classes)
        self.model.classifier = nn.Identity()

    def extract(self, x):
        x = self.model(x)
        return x

    def forward(self, x):
        x = self.extract(x).squeeze(-1).squeeze(-1)
        return x

model = EfficientNet(config=config)
if torch.cuda.is_available():
    model.cuda()

Sample code for printing y_pred:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
for step, (images, labels) in enumerate(sample_loader):
    images = images.to(device)
    labels = labels.to(device)
    batch_size = images.shape[0]
    y_preds = model(images)
    print('The predictions of the 4 images is as follows\n', y_preds)
    break

Output:
You're not even using self.fc in forward pass.
Either just introduce it as:
def forward(self, x):
    ....
    x = extract(x)...
    x = fc(x)
    return x

Or you can simply replace the layer named classifier (this way you don't need Identity layer):
self.model.classifier = nn.Linear(n_features, config.num_classes)

Also, here config.num_classes should be 8.


Instruction: Pytorch math calculation ( only one element tensors can be converted to Python scalars)

Input:
What does it mean by only one element tensors can be converted to Python scalars in this case and how do I suppose to debug it?
x1 = (max-min)*torch.rand(1, 21) + min
x2 = (max-min)*torch.rand(1, 21) + min
zipped_list = zip(x1, x2)
y = [math.sin(2*x1+2) * math.cos(0.5*x2)+0.5 for (x1, x2) in zipped_list]

output
ValueError: only one element tensors can be converted to Python scalars

Output: problem is because of using math.cos and math.sin. math.cos isn't vectorized, but np.cos is. use np.sin and np.cos or torch.sinand torch.cos.


Instruction: pytorch narrow is not applicable

Input:
How to do slicing in pytorch? I tried narrow and numpy slicing? Both are not working for outputing train data and test data. Is there any suggestion for solving it?
x1 = (max-min)*torch.rand(1, 21, dtype=torch.float) + min
x2 = (max-min)*torch.rand(1, 21, dtype=torch.float) + min
zipped_list = zip(x1, x2)
y = torch.empty(1, 21)
y = [torch.sin(2*x1+2) * torch.cos(0.5*x2)+0.5 for (x1, x2) in zipped_list]

print(y)

train_data = y.narrow(0,1)
test_data = y[11:21]
print(train_data)

Output is
AttributeError: 'list' object has no attribute 'narrow'

However when I do normal slicing, the test data will not be slice properly
train_data = y[0:11]
test_data = y[11:21]

Train Data is: [tensor([-0.0515,  0.4574,  0.5141,  0.4865,  0.9266,  1.0984,  0.5364,  0.7042,
         0.1741, -0.4839,  0.4332,  0.2962,  0.2311,  0.6169,  0.4321,  0.4088,
         0.2443,  0.1982,  0.7978,  0.6651, -0.4453])]
Test Data is: []

Output:
In your code, y is a PyTorch tensor:
y = torch.empty(1, 21)

Then it is replaced with a list of PyTorch tensors (actually, just one):
y = [torch.sin(2*x1+2) * torch.cos(0.5*x2)+0.5 for (x1, x2) in zipped_list]

So you need to get the first element of y that is a tensor and then slice it:
print(y[0][:5])


Instruction: How to solve the ValueError importing torch in python

Input:
After installing pytorch package, I have tried to import pytorch with
import torch

but got the error
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "c:\users\user\appdata\local\programs\python\python38\lib\site-packages\torch\__init__.py", line 190, in <module>
    from torch._C import *
ValueError: module functions cannot set METH_CLASS or METH_STATIC

from searching, I have read that this could be about numpy. So I have tried uninstalling and installing with those
pip uninstall numpy
pip install numpy -I
pip install numpy -U

It didn't work.
I would be happy to hear from you if you have any idea why such an error occurs and how to correct it?
in case needed: Python version 3.8.6 ,
torch version 1.7.0 ,
numpy version 1.19.4

Output:
The latest version of numpy for the moment is 1.19.4
I have uninstalled numpy and installed 1.19.3 version. After a complete restart of the system. The error does not show anymore. Thanks.


Instruction: How can I use an LSTM to classify a series of vectors into two categories in Pytorch

Input:
I have a series of vectors representing a signal over time. I'd like to classify parts of the signal into two categories: 1 or 0. The reason for using LSTM is that I believe the network will need knowledge of the entire signal to classify.
My problem is developing the PyTorch model. Below is the class I've come up with.
class LSTMClassifier(nn.Module):

   def __init__(self, input_dim, hidden_dim, label_size, batch_size):
       self.lstm = nn.LSTM(input_dim, hidden_dim)
       self.hidden2label = nn.Linear(hidden_dim, label_size)
       self.hidden = self.init_hidden()

   def init_hidden(self):
       return (torch.zeros(1, self.batch_size, self.hidden_dim),
               torch.zeros(1, self.batch_size, self.hidden_dim))

   def forward(self, x):
       lstm_out, self.hidden = self.lstm(x, self.hidden)
       y  = self.hidden2label(lstm_out[-1])
       log_probs = F.log_softmax(y)
       return log_probs

However this model is giving a bunch of shape errors, and I'm having trouble understanding everything going on.  I looked at this SO question first.

Output:
You should follow PyTorch documentation, especially inputs and outputs part, always.
This is how the classifier should look like:
import torch
import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, input_dim, hidden_dim, label_size):
        super().__init__()
        self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
        self.hidden2label = nn.Linear(hidden_dim, label_size)

    def forward(self, x):
        _, (h_n, _) = self.lstm(x)
        return self.hidden2label(h_n.reshape(x.shape[0], -1))

clf = LSTMClassifier(100, 200, 1)
inputs = torch.randn(64, 10, 100)
clf(inputs)

Points to consider:

always use super().__init__() as it registers modules in your neural networks, allows for hooks etc.
Use batch_first=True so you can pass inputs of shape (batch, timesteps, n_features)
No need to init_hidden with zeros, it is the default value if left uninitialized
No need to pass self.hidden each time to LSTM. Moreover, you should not do that. It means that elements from each batch of data are somehow next steps, while batch elements should be disjoint and you probably do not need that.
_, (h_n, _) returns last hidden cell from last timestep, exactly of shape: (num_layers * num_directions, batch, hidden_size). In our case num_layers and num_directions is 1 so we get (1, batch, hidden_size) tensor as output
Reshape to (batch, hidden_size) so it can be passed through linear layer
Return logits without activation. Only one if it is a binary case. Use torch.nn.BCEWithLogitsLoss as loss for binary case and torch.nn.CrossEntropyLoss for multiclass case. Also sigmoid is proper activation for binary case, while softmax or log_softmax is appropriate for multiclass.
For binary only one output is needed. Any value below 0 (if returning unnormalized probabilities as in this case) is considered negative, anything above positive.


Instruction: Python Error RuntimeError: expected scalar type Long but found Double

Input:
Firstly, I am fairly new to python/ML in general. I am attempting to utilize the model depicted at stackabuse over my own data set.
Everything flows smoothly until I get ready to run the epochs.
In debugging I see that it is failing on CrossEntropyLoss function and I get the error expected long found double. The data set it appears to fail on is the my tdiff column that I calculated but I can't seem to figure out how to convert it to a long.
Is there something that I'm missing in trying to figure this out?
To be clear, this is what I THINK it is based on my extremely limited knowledge on the function:
import pandas as pd
import sqlite3 as sq
import numpy as np
import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import seaborn as sns

con = sq.connect("filepath")
df = pd.read_sql_query("SELECT b.actcod," +
         "a.trnqty, " +
          "b.dlytrn_id, " +
              "a.usr_id, " +
 " b.oprcod, " +
 "b.usr_id as usrid," +
 " b.tostol as frstol," +
 " a.tostol, " +
 " b.trndte as bdte," +
 " a.trndte as adte," +
 " '' as tdiff" +
 " FROM" +
 " (SELECT  row_number() over(PARTITION by usr_id order by trndte) AS rown," +
 " trndte," +
 " dlytrn_id," +
 " trnqty, " +
 " oprcod," +
 " actcod," +
 " frstol," +
 " tostol," +
 " usr_id" +
 " FROM Master_File" +
 " WHERE actcod = 'PALPCK') a " +
 " JOIN "+
 " (SELECT  row_number() over(PARTITION by usr_id order by trndte) AS rown," +
 " trndte,dlytrn_id, trnqty, oprcod,actcod, frstol, tostol, usr_id" +
 " FROM Master_File" +
 " WHERE actcod = 'PALPCK' ) b" +
 " ON b.rown = (a.rown -1)" +
 " and b.usr_id = a.usr_id" #+
 #" LIMIT 10"
,con)

# Create record Count
record_count = df['dlytrn_id'].count()
test_records = int(record_count * .2)
print(record_count)
print(test_records)

#Calculate time delta between scans

df['tdiff'] = pd.TimedeltaIndex.total_seconds(pd.to_datetime(df['adte'].values) - pd.to_datetime(df['bdte'].values))

# Establish column types

cat_column = ['usrid','actcod','tostol','frstol']
num_column = ['trnqty']
dte_column = ['bdte','adte']
op_column = ['tdiff']

#convert column datatypes
for category in cat_column:
    df[category] = df[category].astype('category')
for category in num_column:
    df[category] = df[category].astype('float64')
for category in dte_column:
    df[category] = df[category].astype('datetime64')

#create categorical tensor

usr = df['usrid'].cat.codes.values
act = df['actcod'].cat.codes.values
to = df['tostol'].cat.codes.values
fr = df['frstol'].cat.codes.values

cat_data = np.stack([usr,act,to,fr], 1)
cat_data = torch.tensor(cat_data, dtype=torch.int64)

#Create Numerical Tensor

num_data = np.stack([df[col].values for col in num_column], 1)
num_data = torch.tensor(num_data, dtype=torch.float)

#convert output array

op_data = torch.tensor(df[op_column].values).flatten()

print(df.shape)
print(cat_data.shape)
print(num_data.shape)
print(op_data.shape)

#convert categorical data into an N-Dimensional vector,
#this will allow better ML analysis on relationships

cat_col_sizes = [len(df[column].cat.categories) for column in cat_column]
cat_embedding_sizes = [(col_size, min(50, (col_size+1)//2)) for col_size in cat_col_sizes]
print(cat_embedding_sizes)

#create test batches

cat_train_data = cat_data[:record_count-test_records]
cat_test_data = cat_data[record_count-test_records:record_count]
num_train_data = num_data[:record_count-test_records]
num_test_data = num_data[record_count-test_records:record_count]
train_op = op_data[:record_count-test_records]
test_op = op_data[record_count-test_records:record_count]

print(len(cat_train_data))
print(len(num_train_data))
print(len(train_op))

print(len(cat_test_data))
print(len(num_test_data))
print(len(test_op))

class Model(nn.Module):

    def __init__(self, embedding_size, num_numerical_cols, output_size, layers, p=0.4):
        super().__init__()
        self.all_embeddings = nn.ModuleList([nn.Embedding(ni, nf) for ni, nf in embedding_size])
        self.embedding_dropout = nn.Dropout(p)
        self.batch_norm_num = nn.BatchNorm1d(num_numerical_cols)

        all_layers = []
        num_categorical_cols = sum((nf for ni, nf in embedding_size))
        input_size = num_categorical_cols + num_numerical_cols

        for i in layers:
            all_layers.append(nn.Linear(input_size, i))
            all_layers.append(nn.ReLU(inplace=True))
            all_layers.append(nn.BatchNorm1d(i))
            all_layers.append(nn.Dropout(p))
            input_size = i

        all_layers.append(nn.Linear(layers[-1], output_size))

        self.layers = nn.Sequential(*all_layers)

    def forward(self, x_categorical, x_numerical):
        embeddings = []
        for i,e in enumerate(self.all_embeddings):
            embeddings.append(e(x_categorical[:,i]))
        x = torch.cat(embeddings, 1)
        x = self.embedding_dropout(x)

        x_numerical = self.batch_norm_num(x_numerical)
        x = torch.cat([x, x_numerical], 1)
        x = self.layers(x)
        return x

model = Model(cat_embedding_sizes, num_data.shape[1], 1, [200,100,50], p=0.4)
print(model)

loss_function = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

epochs = 300
aggregated_losses = []

for i in range(epochs):
    i += 1
    y_pred = model(cat_train_data, num_train_data)
    single_loss = loss_function(y_pred, train_op)
    aggregated_losses.append(single_loss)

    if i%25 == 1:
        print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

    optimizer.zero_grad()
    single_loss.backward()
    optimizer.step()

print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')

print(train_op)
print(type(train_op))

print(op_column)
print(type(op_column))

print(op_data)
print(type(op_data))

con.close()

output:
51779
10355
(51779, 11)
torch.Size([51779, 4])
torch.Size([51779, 1])
torch.Size([51779])
[(185, 50), (1, 1), (302, 50), (303, 50)]
41424
41424
41424
10355
10355
10355
Model(
  (all_embeddings): ModuleList(
    (0): Embedding(185, 50)
    (1): Embedding(1, 1)
    (2): Embedding(302, 50)
    (3): Embedding(303, 50)
  )
  (embedding_dropout): Dropout(p=0.4, inplace=False)
  (batch_norm_num): BatchNorm1d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (layers): Sequential(
    (0): Linear(in_features=152, out_features=200, bias=True)
    (1): ReLU(inplace=True)
    (2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (3): Dropout(p=0.4, inplace=False)
    (4): Linear(in_features=200, out_features=100, bias=True)
    (5): ReLU(inplace=True)
    (6): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): Dropout(p=0.4, inplace=False)
    (8): Linear(in_features=100, out_features=50, bias=True)
    (9): ReLU(inplace=True)
    (10): BatchNorm1d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): Dropout(p=0.4, inplace=False)
    (12): Linear(in_features=50, out_features=1, bias=True)
  )
)
Traceback (most recent call last):

  File  line 193, in <module>
    single_loss = loss_function(y_pred, train_op)

  File anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)

  File anaconda3\lib\site-packages\torch\nn\modules\loss.py", line 961, in forward
    return F.cross_entropy(input, target, weight=self.weight,

  File anaconda3\lib\site-packages\torch\nn\functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)

  File anaconda3\lib\site-packages\torch\nn\functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: expected scalar type Long but found Double

Output:
nn.CrossEntropyLoss() expects target tensors of type Long, but what you're passing is of type Double.
Try to change this line
from:
single_loss = loss_function(y_pred, train_op)
to:
single_loss = loss_function(y_pred, train_op.long())


Instruction: Error when trying to train FasterRCNN with custom backbone on GRAYSCALE images

Input:
I am following instructions from https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#putting-everything-together tutorial in order to create object detector for 1 class on GRAYSCALE images.
Here is my code (note that I am using DenseNet as a BACKBONE - pretrained model by me on my own dataset):
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

num_classes = 2 # 1 class + background

model = torch.load(os.path.join(patch_classifier_model_dir, "densenet121.pt"))

backbone = model.features

backbone.out_channels = 1024

anchor_generator = AnchorGenerator(sizes=((32, 64, 128, 256, 512),),

                                   aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],

                                                output_size=7,

                                                sampling_ratio=2)

# put the pieces together inside a FasterRCNN model

model = FasterRCNN(backbone,

                   num_classes=2,

                   rpn_anchor_generator=anchor_generator,

                   box_roi_pool=roi_pooler)

# move model to the right device

model.to(device)

optimizer = torch.optim.SGD(model.parameters(), lr=0.005,

                            momentum=0.9, weight_decay=0.0005)

# and a learning rate scheduler which decreases the learning rate by

# 10x every 3 epochs

lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,

                                               step_size=3,

                                               gamma=0.1)

This is the error that I am running into:
RuntimeError: Given groups=1, weight of size [64, 1, 7, 7], expected input[2, 3, 1344, 800] to have 1 channels, but got 3 channels instead
Based on FasterRCNN architecture, I assume problem is in the transform component because it tries to normalize images that are initially grayscale, and not RGB:
FasterRCNN(
  (transform): GeneralizedRCNNTransform(
      Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      Resize(min_size=(800,), max_size=1333, mode='bilinear')
  )
  (backbone): Sequential(
    (conv0): Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu0): ReLU(inplace=True)
    (pool0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (denseblock1): _DenseBlock(
      (denselayer1): _DenseLayer(
        (norm1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace=True)
        (conv1): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace=True)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )

      ...............

    (norm5): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (rpn): RegionProposalNetwork(
    (anchor_generator): AnchorGenerator()
    (head): RPNHead(
      (conv): Conv2d(1024, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (cls_logits): Conv2d(1024, 15, kernel_size=(1, 1), stride=(1, 1))
      (bbox_pred): Conv2d(1024, 60, kernel_size=(1, 1), stride=(1, 1))
    )
  )
  (roi_heads): RoIHeads(
    (box_roi_pool): MultiScaleRoIAlign()
    (box_head): TwoMLPHead(
      (fc6): Linear(in_features=50176, out_features=1024, bias=True)
      (fc7): Linear(in_features=1024, out_features=1024, bias=True)
    )
    (box_predictor): FastRCNNPredictor(
      (cls_score): Linear(in_features=1024, out_features=2, bias=True)
      (bbox_pred): Linear(in_features=1024, out_features=8, bias=True)
    )
  )
)

Am I correct? If so, how do I resolve this issue? Is there a STANDARD PRACTICE on dealing with grayscale images and FasterRCNN?
Thanks in advance! Really appreciate it!

Output:
Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) means the normalization process is applied to all the 3 channels of the input image. 0.485 is applied to the R channel, 0.456 is applied to the G channel and 0.406 is applied to the B channel. The same goes for the standard deviation values.
The 1st Conv. layer of the backbone expects a 1 channel input and that's the reason you get this error.
You could do the following to solve the issue.
Re-define the GeneralizedRCNNTransform and attach it to your model. You could do something like this:
# put the pieces together inside a FasterRCNN model

model = FasterRCNN(backbone, num_classes=2, rpn_anchor_generator=anchor_generator, box_roi_pool=roi_pooler)

# Changes
grcnn = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=800, max_size=1333, image_mean=[0.485], image_std=[0.229])
model.transform = grcnn
model.to(device)


Instruction: Why doesn't setting random seed give same performance across runs?

Input:
I am training some deep learning models using pytorch, which also includes the usage of numpy. Since the randomisation is not truly random and it is pseudo-random, why aren't the numbers i.e., accuracy etc same across different runs?
I mean, even if I do not set some random seed, there should be some default random seed according to which my code must run and give same results across different runs. Is there something more to it?

Output:
I don't think the truly random vs. pseudo-random discussion is relevant here. Different numbers may be generated depending on factors like date and time, which is why you should set a seed.
If you involve PyTorch and CUDA, things get a little more complicated. Here is an article talking about randomness and reproducibility.
In short, you need to set seeds for numpy + PyTorch and also set the backend to deterministic operations.


Instruction: Is there a way to define a 'heterogeneous' kernel design to incorporate linear operators into the regression for GPflow (or GPytorch/GPy/...)?

Input:
I'm trying to perform a GP regression with linear operators as described in for example this paper by Särkkä: https://users.aalto.fi/~ssarkka/pub/spde.pdf In this example we can see from equation (8) that I need a different kernel function for the four covariance blocks (of training and test data) in the complete covariance matrix.
This is definitely possible and valid, but I would like to include this in a kernel definition of (preferably) GPflow, or GPytorch, GPy or the like.
However, in the documentation for kernel design in Gpflow, the only possibility is to  define a covariance function that acts on all covariance blocks. In principle, the method above should be straight-forward to add myself (the kernel function expressions can be derived analytically), but I don't see any way of incorporating the 'heterogeneous' kernel functions into the regression or kernel classes. I tried to consult other packages such as Gpytorch and Gpy, but again, the kernel design does not seem to allow this.
Maybe I'm missing something here, maybe I'm not familiar enough with the underlying implementation to asses this, but if someone has done this before or sees the (what should be reasonably straight-forward?) implementation possibility, I would be happy to find out.
Thank you very much in advance for your answer!
Kind regards

Output:
This should be reasonably straightforward, though requires building a custom kernel. Basically, you need a kernel that can know for each input what the linear operator for the corresponding output is (whether this is a function observation/identity operator, integral observation, derivative observation, etc). You can achieve this by including an extra column in your input matrix X, similar to how it's done for the gpflow.kernels.Coregion kernel (see this notebook). You would need to then need to define a new kernel with K and K_diag methods that for each linear operator type find the corresponding rows in the input matrix, and pass it to the appropriate covariance function (using tf.dynamic_partition and tf.dynamic_stitch, this is used in a very similar way in GPflow's SwitchedLikelihood class).
The full implementation would probably take half a day or so, which is beyond what I can do here, but I hope this is a useful starting pointer, and you're very welcome to join the GPflow slack (invite link in the GPflow README) and discuss it in more detail there!


Instruction: How to resize a batch of images for use with Pytorch Linear Regression?

Input:
I am trying to create a simple linear regression neural net for use with batches of images. The input dimensions are [BatchSize, 3, Width, Height]  with the second dimension representing the RGB channels of the input image.
Here is my (broken) attempt at that regression model:
class LinearNet(torch.nn.Module):
    def __init__(self, Chn, W,H, nHidden):
        """
        Input: A [BatchSize x Channels x Width x Height] set of images
        Output: A fitted regression model with weights dimension : [Width x Height]
        """
        super(LinearNet, self).__init__()
        self.Chn = Chn
        self.W = W
        self.H = H
        self.hidden = torch.nn.Linear(Chn*W*H,nHidden)   # hidden layer
        self.predict = torch.nn.Linear(nHidden, Chn*W*H)   # output layer

    def forward(self, x):
        torch.reshape(x, (-1,self.Chn*self.W*self.H)) # FAILS here
        # x = x.resize(-1,self.Chn*self.W*self.H)
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        x = x.resize(-1,self.Chn, self.W,self.H)
        return x

When sending in a batch of images with dimensions [128 x 3 x 96 x 128]  this fails on the indicated line:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (36864x128 and 36864x256)

How should the matrix dimensions be properly manipulated to use these pytorch functions?
Update Based on a (since deleted) comment I have updated the code to use torch.reshape.

Output:
Solution 1 As a possible solution, you can get a batch size from input x with x.shape[0] and use it in reshape later
import torch

batch = torch.zeros([128, 3, 96, 128], dtype=torch.float32)

# -1 will compute last dimension automatically
batch_upd = torch.reshape(batch, (batch.shape[0], -1))

print(batch_upd.shape)

Output for this code is
torch.Size([128, 36864])

Solution 2
As another possible solution you can use flatten
batch_upd = batch.flatten(start_dim=1)

will result in the same output
As to your next problem, consider going through the modified forward code:
def forward(self, x):
    x = x.flatten(1)  # shape: [B, C, W, H] -> [B, C*W*H]
    x = F.relu(self.hidden(x))      # activation function for hidden layer
    x = self.predict(x)             # linear output
    x = x.reshape((-1, self.Chn, self.W, self.H)) # shape: [B, C*W*H] -> [B, C, W, H]
    return x

Here is the successful usage example:
ln = LinearNet(3, 96, 128, 256)
batch = torch.zeros((128, 3, 96, 128))
res = ln(batch)
print(res.shape)  # torch.Size([128, 3, 96, 128])


Instruction: openCv and PyTorch inverser Transform not working

Input:
I have a transforms class which only does:
        if transform is None:
            transform = transforms.Compose([
                transforms.Resize((256, 256)),
                transforms.ToTensor()
            ])
        root = os.path.join(PROJECT_ROOT_DIR, "data")
        super(AttributesDataset, self).__init__()
        self.data = torchvision.datasets.CelebA(
            root=root,
            split=split,
            target_type='attr',
            download=True,
            transform=transform
        )

From the documentation, I understand that this implies just a scale-down of values in the range 0,1 ie all pixel values shall lie between [0,1] (I have verified this as well).
I want to visualize some of the outputs coming from the model. As such, I created a simple method which does:-
    for img, label in dataloader:
        img.squeeze_(0)
        # permute the channels. cv2 expects image in format (h, w, c)
        unscaled_img = img.permute(1, 2, 0)
        # move images to cpu and convert to numpy as required by cv2 library
        unscaled_img = torch.round(unscaled_img * 255)
        unscaled_img = unscaled_img.to(torch.uint8)
        # unscaled_img = np.rint(unscaled_img * 255).astype(np.uint8)
        unscaled_img = cv2.cvtColor(unscaled_img, cv2.COLOR_RGB2BGR)
        cv2.imshow(unscaled_img.numpy())

However, all the images that are created have an unusually blue shade. For instance,

Can someone please tell me what exactly am I doing wrong here? Your help would be highly appreciated

Output:
Solved by @LajosArpad comment. The culprit was
unscaled_img = cv2.cvtColor(unscaled_img, cv2.COLOR_RGB2BGR)

Removing it resulted in correct values.


Instruction: neural networks in pytorch

Input:
I am using pytorch and need to implement this as a part of a neural network. Is there a particular way to code the layers shown in purple (s
def forward(self, images: torch.Tensor) -> torch.Tensor:
x= self.fc1(x)
return x

Output:
Combining all my comments into the answer.
To split the output vector from the first layer, which has a shape [batch_size, 4608], you can use torch.split function as follows
batch = torch.zeros((10, 4608))
sub_batch_1, sub_batch_2 = torch.split(batch, 2304, dim=1)
print(sub_batch_1.shape, sub_batch_2.shape)

This code results in two tensors
(torch.Size([10, 2304]), torch.Size([10, 2304]))

I am not sure about MAX logic you need. If it is about getting the maximum element in a given position from two tensors obtained during split, then torch.maximum function can be used to do so.