Instruction: Optimizing with BCE is not working, nothing will change

Input:
I have the following code:
import torch
import torch.nn as nn
import torch.nn.functional as F
from tqdm import tqdm
import matplotlib.pyplot as plt
import os
import keras
from random import choice
import sys
devicet = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device(devicet)
if devicet == 'cpu':
  print ('Using CPU')
else:
  print ('Using GPU')
cuda0 = torch.device('cuda:0')
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.step1 = nn.Linear(5, 25)
        self.step2 = nn.Linear(25, 50)
        self.step3 = nn.Linear(50, 100)
        self.step4 = nn.Linear(100, 100)
        self.step5 = nn.Linear(100, 10)
        self.step6 = nn.Linear(10, 1)
    def forward(self, x):
      x = F.relu(x)
      x = self.step1(x)
      x = F.relu(x)
      x = self.step2(x)
      x = F.relu(x)
      x = self.step3(x)
      x = F.relu(x)
      x = self.step4(x)
      x = F.relu(x)
      x = self.step5(x)
      x = F.relu(x)
      x = self.step6(x)
      x = F.relu(x)
      return (x)
net = Net()
x = torch.rand(10,5)
num = choice(range(10))
zero_tensor = torch.zeros(num, 1)
one_tensor = torch.ones(10-num, 1)
y = torch.cat((zero_tensor,one_tensor),0)
x.to(devicet)
y.to(devicet)
learning_rate = 1e-3
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
loss_fn = torch.nn.BCELoss()
acc_list = []
for i in tqdm(range(1000),desc='Training'):
    y_pred = net(x)
    loss = loss_fn(y_pred, y)
    loss.backward()
    optimizer.step()
    acc_list.append(abs(net(x).detach().numpy()[0]-y.detach().numpy()[0]))
    with torch.no_grad():
        for param in net.parameters():
            param -= learning_rate * param.grad
    optimizer.zero_grad()
print ('\nFinished training in {} epochs.'.format(len(acc_list)))
plt.plot(range(len(acc_list)),acc_list)
plt.show()
for i in range(10):
  print (str(net(x).detach().numpy()[i][0])+', '+str(y.detach().numpy()[i][0]))

When I run this, it consistently just prints out the following:
Image
Why won't it do any training? It works if I use MSE loss (actually, it only works sometimes with MSE loss, sometimes it does the same thing as in the image) , it's only when I use BCE that it stops working entirely.

Output:
Final layer activation
You are outputting only positive values, those should be between 0 and 1 for starters, these line specifically:
x = F.relu(x)
return (x)

Use torch.sigmoid with BCELoss or even better, just output x and use torch.nn.BCEWithLogitsLoss which uses logits directly
Training
You are using Adam optimizer and doing SGD manually here:
with torch.no_grad():
    for param in net.parameters():
        param -= learning_rate * param.grad

Essentially you are applying optimization step twice which is probably too much and might destroy the weights.
optimizer.step() already does this, no need for both!
Accuracy
This part:
abs(net(x).detach().numpy()[0]-y.detach().numpy()[0])

I assume you want to calculate accuracy, it would go like this (also do not push data through the network twice via net(x), you already have y_pred!):
# Assuming sigmoid activation
def accuracy(y_pred, y_true):
    # For logits use
    # predicted_labels = y_pred > 0.0
    predicted_labels = y_pred > 0.5
    return torch.mean((y_true == predicted_labels).float())


Instruction: Pytorch: File-specific action for each image in the batch

Input:
I have a dataset of images each of them having an additional attribute "channel_no". Each image should be processed with the nn layer according to its channel_no:
 images with channel_no=1 have to be processed with layer1
 images with channel_no=2 have to be processed with layer2
 images with channel_no=3 have to be processed with layer3
etc...

The problem is that when the batch contains more than one image, forward() function gets a torch tensor with the batch of images as input, and each of the images has different channel_no. So it is not clear how to process each image separately.
Here is the code for the case when the batch has 1 image only:
class Net(nn.Module):
    def __init__ (self, weight):
        super(Net, self).__init__()

        self.layer1 = nn.Linear(hidden_sizes[0], hidden_sizes[1])
        self.layer2 = nn.Linear(hidden_sizes[0], hidden_sizes[1])
        self.layer3 = nn.Linear(hidden_sizes[0], hidden_sizes[1])

        self.outp = nn.Linear(hidden_sizes[1], output_size)

    def forward(self, x, channel_no):
        channel_no = channel_no[0] #extract channel_no from the batch list

        x = x.view(-1,hidden_sizes[0])

        if channel_no == 1: x = F.relu(self.layer1(x))
        if channel_no == 2: x = F.relu(self.layer2(x))
        if channel_no == 3: x = F.relu(self.layer3(x))

        x = torch.sigmoid(self.outp(x))

        return x

Is it possible to process each image separately using batch size > 1 ?

Output: To process images separately you probably need separate tensors. I'm not sure if there's a fast way to do it, but you could split the tensor in the batch dimension to get individual image tensors and then iterate through them to sort them by channel number. Then join each group of images with the same channel number into a new tensor and process that tensor specially.


Instruction: Flatten 3D tensor

Input: I have a tensor of the shape T x B x N (training data for a RNN, T is max seq length, B is number of batches, and N number of features) and I'd like to flatten all the features across timesteps, such that I get a tensor of the shape B x TN. Haven't been able to figure out how to do this..

Output:
You need to permute your axes before flattening, like so:
t = t.swapdims(0,1) # (T,B,N) -> (B,T,N)
t = t.view(B,-1)    # (B,T,N) -> (B,T*N) (equivalent to `t.view(B,T*N)`)


Instruction: Passing Parameters in Python/Pytorch Classmethod

Input:
I am confused while passing the parameters in classmethod.
The code is shown below:
def gather_feature(fmap, index):
    # fmap.shape(B, k1, 1)  index.shape(B, k2)
    dim = fmap.size(-1)
    index = index.unsqueeze(len(index.shape)).expand(*index.shape, dim)  # this works

    fmap = fmap.gather(dim=1, index=index)  # out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1.
    return fmap

def gather_feature(fmap, index):
    # fmap.shape(B, k1, 1)  index.shape(B, k2)
    dim = fmap.size(-1)
    index = index.unsqueeze(len(index.shape))
    index = index.expand(*index.shape, dim)  # raise error

    fmap = fmap.gather(dim=1, index=index)  # out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1.
    return fmap

Once index.unsqueeze() has done, the shape of index would be changed to (B, k2, 1).
If the index.shape that pass to expand() classmethod is (B, k2, 1), an error has raised.
However, if writing these tow classmethod in one line, namely index.unsqueeze().expand(), the index.shape passing to expand() classmethod seems to be (B, k2).
Has the index.shape been computed and stored before performing .unsqueeze()?
Therefore, the .unsqueeze() won't affect the index.shape which pass to .expand().
That is my guess, but I cannot figure out another one.
Thank you for your time.

Output:
Consider the first case -
index.unsqueeze(len(index.shape)).expand(*index.shape, dim)

This is equivalent to the following code -
A = index.unsqueeze(len(index.shape))
index = A.expand(*index.shape, dim)

Note that Tensor index has not been changed after the execution of the first line. So when you then execute A.expand(*index.shape, dim) the original shape of index is used.
However in the second case you when you first do index = index.unsqueeze(len(index.shape)) , you are changing index. So in the next step the new unsqueezed index's shape is used.


Instruction: How to iterate over PyTorch tensor

Input:
I have a tensor data of size (1000,110) and I want to iterate over the first index of the tensor and calculate the following.
    data = torch.randn(size=(1000,110)).to(device)

    male_poor = torch.tensor(0).float().to(device)
    male_rich = torch.tensor(0).float().to(device)

    female_poor = torch.tensor(0).float().to(device)
    female_rich = torch.tensor(0).float().to(device)

    for i in data:

        if torch.argmax(i[64:66]) == 0 and torch.argmax(i[108:110]) == 0:
          female_poor += 1
        if torch.argmax(i[64:66]) == 0 and torch.argmax(i[108:110]) == 1:
          female_rich += 1
        if torch.argmax(i[64:66]) == 1 and torch.argmax(i[108:110]) == 0:
          male_poor += 1
        if torch.argmax(i[64:66]) == 1 and torch.argmax(i[108:110]) == 1:
          male_rich += 1

    disparity = ((female_rich/(female_rich + female_poor))) / ((male_rich/(male_rich + male_poor)))

Is there a faster way than for loop to do this?

Output:
The key in pytorch (as well as numpy) is vectorizataion, that is if you can remove loops by operating on matrices it will be a lot faster. Loops in python are quite slow compared to the loops in the underlying compiled C code. On my machine the execution time for your code was about 0.091s, the following vectorized code was about 0.002s so about x50 faster:
import torch
torch.manual_seed(0)
device = torch.device('cpu')

data = torch.randn(size=(1000, 110)).to(device)

import time
t = time.time()
#vectorize over first dimension
argmax64_0 = torch.argmax(data[:, 64:66], dim=1) == 0
argmax64_1 = torch.argmax(data[:, 64:66], dim=1) == 1
argmax108_0 = torch.argmax(data[:, 108:110], dim=1) == 0
argmax108_1 = torch.argmax(data[:, 108:110], dim=1) == 1
female_poor = (argmax64_0 & argmax108_0).sum()
female_rich = (argmax64_0 & argmax108_1).sum()
male_poor = (argmax64_1 & argmax108_0).sum()
male_rich = (argmax64_1 & argmax108_1).sum()

disparity = ((female_rich / (female_rich + female_poor))) / ((male_rich / (male_rich + male_poor)))

print(time.time()-t)
print(disparity)


Instruction: In pytorch, which part cost most time? Loss calculation or backward+step?

Input:
In pytorch, which part cost most of time?

Loss calculation(no backward, just calculating)
backward
step
Please explain a bit if you can.

Output: Not just in Pytorch, but in all framework, backward_step takes more time


Instruction: Is there a vectorized way to create a matrix in which each element is the the row-wise dot product of a matrix?

Input:
Apologize for the vagueness in the title, I spend some time rephrasing but cannot get it very well.
For example, I have a 2*3 matrix in Pytorch tensor
test = torch.tensor([[1, 10, 100],
                    [2, 20, 200]])

What I would like to have a final matrix that is
torch.tensor([10101, 20202],
             [20202, 40404])

Where we can see, the (0,0) position is the first row's dot product with itself, and (0,1) (1,0) is the dot product of first row and second row, (1,1) the the dot product of the second row with itself.
Thanks!

Output:
You can either do matrix multiplication:
test @ test.T

Or a torch.einsum:
torch.einsum('ij,kj->ik', test, test)


Instruction: Calculating the Euclidean distance between 2 points on a circle in D-dimensional space

Input:
It's fairly straightforward to calculate a direct Euclidean distance between 2 points:
import torch
p1 = torch.tensor([1.0, 3.5])
p2 = torch.tensor([5.0, 9.2])

dis = torch.sum(torch.square(p1-p2))
dis
>>> tensor(48.4900)

However, how can I calculate the distance between 2 points on circle without going through the circle? That it, the distance on the circle's perimeter, in D-dimensional space.
Clearly, in 2D a circle is just a circle:
import numpy as np
import matplotlib.pyplot as plt

def circle_points(r, n):
    circles = []
    for r, n in zip(r, n):
        t = np.linspace(0, 2*np.pi, n, endpoint=False)
        x = r * np.cos(t)
        y = r * np.sin(t)
        circles.append(np.c_[x, y])
    return circles

r = [2]
n = [20]
circles = circle_points(r, n)

fig, ax = plt.subplots()
for circle in circles:
    ax.scatter(circle[:, 0], circle[:, 1])
ax.set_aspect('equal')
plt.show()

point_1 = circles[0][0]
point_2 = circles[0][11]
print('point_1: ', point_1, 'point_2: ', point_2)
>>> point_1:  [2. 0.] point_2:  [-1.90211303 -0.61803399]

While in 3D it will be a sphere, 4D hypersphere, etc.

Output:
Let center ben the center of your circle.
Then you can compute the angle between the two center-to-point vectors with the dot product formula, which relates the dot product with the cosine of the angle and the norm of the vectors (look for the geometric definition of the dot product)
normalize = lambda vect: vect/vect.norm()
v1 = normalize(point_1 - center)
v2 = normalize(point_2 - center)
angle = torch.arccos(v1.dot(v2))

Then, the length of the arc is the radius of the circle times the angle, i.e
distance = angle*r


Instruction: How to run a proper Bayesian Logistic Regression

Input:
I'm trying to run  a bayesian logistic regression on the wine dataset provided from the sklearn package. As variables, I decided to use alcohol, color_intensity, flavanoids, hue and magnesium where alcohol is my response variable and the rest the predictors. To do so, I'm using pyro and torch packages:
import pyro
import torch
import pyro.distributions as dist
import pyro.optim as optim
from pyro.infer import SVI, Trace_ELBO
import pandas as pd
import numpy as np
from pyro.infer import Predictive
import torch.distributions.constraints as constraints
from sklearn import datasets
pyro.set_rng_seed(0)

#loading data and prepearing dataframe
 wine = datasets.load_wine()
 data = pd.DataFrame(columns = wine['feature_names'], data=wine['data'] )

#choosiing variables: response and predictors
 variables = data[['alcohol', 'color_intensity', 'flavanoids', 'hue', 'magnesium']]

#standardization
 variables = (variables-variables.min())/(variables.max()-variables.min())

#tensorizing
 alcohol = torch.tensor(variables['alcohol'].values, dtype=torch.float)
 predictors = torch.stack([torch.tensor(variables[column].values, dtype=torch.float)
               for column in ['alcohol', 'color_intensity', 'flavanoids', 'hue', 'magnesium']], 1)

#splitting data
 k = int(0.8 * len(variables))
 x_train, y_train = predictors[:k], alcohol[:k]
 x_test, y_test = predictors[k:], alcohol[k:]

#modelling

def model_alcohol(predictors, alcohol):
    n_observations, n_predictors = predictors.shape

 #weights
    w = pyro.sample('w', dist.Normal(torch.zeros(n_predictors), torch.ones(n_predictors)))
    epsilon = pyro.sample('epsilon', dist.Normal(0.,1.))

#non-linearity
    y_hat = torch.sigmoid((w*predictors).sum(dim=1) + epsilon)
    sigma = pyro.sample("sigma", dist.Uniform(0.,3.))
    with pyro.plate('alcohol', len(alcohol)):
       y=pyro.sample('y', dist.Normal(y_hat, sigma), obs=alcohol)

def guide_alcohol(predictors, alcohol=None):
    n_observations, n_predictors = predictors.shape

    w_loc = pyro.param('w_loc', torch.rand(n_predictors))
    w_scale = pyro.param('w_scale', torch.rand(n_predictors), constraint=constraints.positive)
    w = pyro.sample('w', dist.Normal(w_loc, w_scale))

    epsilon_loc = pyro.param('b_loc', torch.rand(1))
    epsilon_scale = pyro.param('b_scale', torch.rand(1), constraint=constraints.positive)
    epsilon = pyro.sample('epsilon', dist.Normal(epsilon_loc, epsilon_scale))

    sigma_loc = pyro.param('sigma_loc', torch.rand(n_predictors))
    sigma_scale = pyro.param('sigma_scale', torch.rand(n_predictors),
                   constraint=constraints.positive)
    sigma = pyro.sample('sigma', dist.Normal(sigma_loc, sigma_scale))

alcohol_svi = SVI(model=model_alcohol, guide=guide_alcohol, optim=optim.ClippedAdam({'lr' : 0.0002}),
                loss=Trace_ELBO())

losses = []
for step in range(10000):
   loss = alcohol_svi.step(x_train, y_train)/len(x_train)
   losses.append(loss)

As I have to use Stochastic Variational Inference, I have defined both the model and the guide. My problem is now at matching tensor sizes, as I now I get the error:
RuntimeError: The size of tensor a (142) must match the size of tensor b (5) at non-singleton
dimension 0
Trace Shapes:
Param Sites:
Sample Sites:
   w dist   5 |
    value   5 |
 epsilon dist |
    value   1 |
 sigma dist   |
    value   5 |
 alcohol dist |
    value 142 |

I'm kinda new to the idea of modelling on my own, so clearly there are mistakes around the code (hopefully not on the theory behind it). Still, I see I should adjust dimension on the guide maybe? I'm not entirely sure on how to honestly.

Output:
Your main problem is that w is not declared as a single event (.to_event(1)), and your variance (sigma) should have the same dim as your observations (()). The model and guide below fix this; I suggest you look at auto-generated guides in Pyro, and a different prior on sigma.
def model_alcohol(predictors, alcohol):
    n_observations, n_predictors = predictors.shape

    # weights
    # w is a single event
    w = pyro.sample('w', dist.Normal(torch.zeros(n_predictors), torch.ones(n_predictors)).to_event(1))
    epsilon = pyro.sample('epsilon', dist.Normal(0., 1.))

    # non-linearity
    y_hat = torch.sigmoid(predictors @ w + epsilon)  # (predictors * weight).sum(1) == predictors @ w
    sigma = pyro.sample("sigma", dist.Uniform(0., 3.))
    with pyro.plate('alcohol', len(alcohol)):
        pyro.sample('y', dist.Normal(y_hat, sigma), obs=alcohol)

def guide_alcohol(predictors, alcohol=None):
    n_observations, n_predictors = predictors.shape

    w_loc = pyro.param('w_loc', torch.rand(n_predictors))
    w_scale = pyro.param('w_scale', torch.rand(n_predictors), constraint=constraints.positive)
    pyro.sample('w', dist.Normal(w_loc, w_scale).to_event(1))

    epsilon_loc = pyro.param('b_loc', torch.rand(1))
    epsilon_scale = pyro.param('b_scale', torch.rand(1), constraint=constraints.positive)
    epsilon = pyro.sample('epsilon', dist.Normal(epsilon_loc, epsilon_scale))

    sigma_loc = pyro.param('sigma_loc', torch.rand(1))
    sigma_scale = pyro.param('sigma_scale', torch.rand(1),
                             constraint=constraints.positive)
    pyro.sample('sigma', dist.HalfNormal(sigma_loc, sigma_scale))  # MUST BE POSITIVE


Instruction: Transforming the indices back through view in pytorch

Input:
I have the following code, where I define a torch tensor of zeros, change one item to be equal to 1, pass through three reshape functions.
Then, after all the transformations, I obtain the index of 1. I am wondering if it is possible to somehow use the max_idx and the information about the permutations/.view to obtain the index of 1 in the initial B1 tensor (which should be equal to 1234).
A1 = np.zeros(10*18*40*28)
A1[1234] = 1
A1 = A1.reshape(10, 18, 40, 28)
B1 = torch.Tensor(A1)
print('B1: ', B1.shape, torch.nonzero(B1))
C1 = B1.permute(0, 2, 3, 1)
print('C1: ', C1.shape,  torch.nonzero(C1))
D1 = C1.contiguous().view(C1.shape[0], C1.shape[1], C1.shape[2], 3, 6)
print('D1: ', D1.shape, torch.nonzero(D1))
E1 = D1.contiguous().view(D1.shape[0], -1, 6)
print('E1: ', E1.shape, torch.nonzero(E1))

max_idx = torch.nonzero(E1)

I would love to hear any hints on how I can try to do this :)

Output:
For each dimension, you can check in which index the value you refer to lies. After you find the right index, you subtract the number of indices that came before the next dimension and solve the same problem for a smaller sub-array.
Or you can just use the function 'numpy.unravel_index' that does the exact same thing.
import numpy as np
import torch

A1 = np.zeros(10*18*40*28)
idx = 1234
A1[idx] = 1
A1 = A1.reshape(10, 18, 40, 28)
B1 = torch.Tensor(A1)
print('B1: ', B1.shape, torch.nonzero(B1))

idx_temp = idx+0
idxB1 = np.zeros((B1.dim(),), dtype = int)
for i in range(B1.dim()):
    idxB1[i]  = idx_temp//np.prod(B1.shape[i+1:])
    idx_temp -= np.prod(B1.shape[i+1:])*idxB1[i]

idxB1np = np.unravel_index(idx, B1.shape)

print(f'idxB1 = {idxB1}')
print(f'idxB1np = {idxB1np}')

output:
B1:  torch.Size([10, 18, 40, 28]) tensor([[0, 1, 4, 2]])
idxB1 = [0 1 4 2]
idxB1np = (0, 1, 4, 2)


Instruction: Documenting jit-compiled PyTorch Class Method (Sphinx)

Input:
I am having a problem trying to document custom PyTorch models with Sphinx: methods that are jit-compiled show up without docstrings in the documentation. How do I fix this? I checkoed out Python Sphinx autodoc and decorated members and How to autodoc decorated methods with sphinx? but the proposed solutions don't seem to work. When I try using ..automethod I get
AttributeError: '_CachedForward' object has no attribute '__getattr__'

Here's a MWE; at the moment I circumvent the problem by writing a my_forward and calling it in the forward.
from torch import jit, Tensor

class DummyModel(jit.ScriptModule):
    """My dummy model"""

    def __init__(self, const: float):
        super(DummyModel, self).__init__()
        self.const = Tensor(const)

    @jit.script_method
    def forward(self, x: Tensor) -> Tensor:
        """This method does show as a :undoc-member: in the documentation"""
        return self.my_forward(x)

    def my_forward(self,  x: Tensor) -> Tensor:
        """This method shows up as a :member: in the documentation"""
        return x + self.const

Output:
Set the PYTORCH_JIT environment variable to 0 when running Sphinx. This will disable script and tracing annotations (decorators).
See https://pytorch.org/docs/stable/jit.html#disable-jit-for-debugging.


Instruction: How to train pytorch model using batches with partial information

Input:
In this PyTorch model I have two streams data with two different modalities that are input into the model at the same time. The streams of data are blocks of sequential data. So I have modality one M1 = [[1,1,1,1],[2,2,2,2],[3,3,3,3],[4,4,4,4],[5,5,5,5]] and modality two M2 = [[11,11,11,11],[22,22,22,22],[33,33,33,33],[44,44,44,44]]. I want to train this model with a system that during training, batches of sequential data will randomly have partial or full information.
So there will be three possibilities during training: M1 and M2 will have its full sequential data, or M1 will be clipped meaning that the sequential data information will be set to zero (e.g. M1 = [[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0],[0,0,0,0]] ), while M2 will have its full sequential data info, or M2 will be clipped meaning that the sequential data information in M2 will be set to zero, while M1 will have its full sequential data info.
Are there any PyTorch functions that will do that automatically for me, or does anyone know what would be a good way to implement this?

Output:
Let M1 and M2 be 2D tensors or 3D batched tensors for training, and let p1,p2 and p3 be the probabilities that M1, M2, or neither is zeroed:
p1 = 0.5 # for example
p2 = 0.3 # for example
# p3 = 1- p1 - p2 = 0.2

randn = torch.rand(1)

if randn < p1:
   M1 *= 0
elif randn > p1 and randn < p1+p2:
   M2 *= 0

# pass M1 and M2 to your model

As a secondary note, if possible you may want to consider zeroing the gradient for whichever modality had its data erased. You don't really want your network to learn that the zero-values have any significance, which you are at risk of learning.
Depending on the structure of your model, this may be possible. For instance, if M1 is processed by 1 network branch and M2 is processed by a different branch, you could constrain so that loss is only back-propagated through the branch that received non-zeroed inputs. For other network structures though, this may not be possible.


Instruction: Create dataset out of x_train and y_train

Input:
How to put the x_train and y_train into a model for training?
The x_train is a tensor of size (3000, 13).
The y_train is of size (3000, 1)
That is for each element of x_train (1, 13), the respective y label is one digit from y_train.
if I do:
train_data = (train_feat, train_labels)
print(train_data[0].shape)
print(train_data[1].shape)

torch.Size([3082092, 13])
torch.Size([3082092, 1])
train_loader = data.DataLoader(dataset=train_data,
                                batch_size= 7,
                                shuffle=True)

The dataloader  does not return the batch size, but returns the whole dataset instead

Output:
You can use the TensorDataset constructor:
import torch.utils.data as data_utils

dataset = data_utils.TensorDataset(train_feat, train_labels)
train_loader = data_utils.DataLoader(dataset, batch_size=7, shuffle=True)


Instruction: How to replace torch.sparse with other pytorch function?

Input:
I would like to replace the torch.sparse function using the other Pytorch function.
i = torch.LongTensor([[0, 1, 1], [2, 0, 2]])
v = torch.FloatTensor([3, 4, 5])
out1 = torch.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense()
out2 = ???
out1 == out2
> tensor(True)

Background:
I'm converting a Pytorch model to CoreML, but the sparse_coo_tensor operator defined in the torch.norm function is not implemented with CoreMLTools.
A few people have trouble with this problem, but CoreMLTools is still unsupported. So I'd like to replace it without the operator used in torch.sparse.FloatTensor.
I have tried torch.sparse_coo_tensor but it was not supported.
I have created a simple colaboratory notebook that reproduces this. Please test it using the following colab.
https://colab.research.google.com/drive/1TzpeJqEcmCy4IuXhhl6LChFocfZVaR1Q?usp=sharing
I've asked about different operators on stackoverflow before, so please refer to that.
How to replace torch.norm with other pytorch function?

Output:
If I understand sparse_coo format correctly, the two rows of i are just the coordinates at which to copy the values of v. Which means that you can instead create you matrix like :
def dense_from_coo(i, v):
    rows = i[0].max()+1
    cols = i[1].max()+1
    out = torch.zeros(rows, cols)
    out[i[0], i[1]] = v
    return out

print(dense_from_coo(i,v))
>>> tensor([[0., 0., 3.],
            [4., 0., 5.]])

print(torch.sparse.FloatTensor(i, v, torch.Size([2,3])).to_dense())
>>> tensor([[0., 0., 3.],
            [4., 0., 5.]])


Instruction: Why is saving state_dict getting slower as training progresses?

Input:
I'm saving my model's and optimizer's state dict as follows:
if epoch % 50000 == 0:
  #checkpoint save every 50000 epochs
  print('\nSaving model... Loss is: ', loss)
  torch.save({
      'epoch': epoch,
      'model': self.state_dict(),
      'optimizer_state_dict': self.optimizer.state_dict(),
      'scheduler': self.scheduler.state_dict(),
      'loss': loss,
      'losses': self.losses,
      }, PATH)

When I first start the training it saves in less than 5 seconds. However, after a couple of hours of training it takes over a two minutes to save. The only reason I could think of is the list of losses. But I can't see how that would increase the time by that much.
Update 1:
I have my losses as:
self.losses = []

I'm appending the loss at each epoch to this list as follows:
    #... loss calculation
    loss.backward()
    self.optimizer.step()
    self.scheduler.step()

    self.losses.append(loss)

Output:
As mentionned in the comments, the instruction
self.losses.append(loss)

is definitely the culprit, and shoud be replaced with
self.losses.append(loss.item())

The reason is that when you store the tensor loss, you also store the whole computational graph alongside (all the information that is required to perform the backprop). In other words, you are not merely storing a tensor, but also the pointers to all the tensors that have been involved in the computation of the loss and their relations (which ones were added, multiplied etc). So it will grow really big really fast.
When you do loss.item() (or loss.detach(), which would work as well), you detach the tensor from the computational graph, and thus you only store what you intended : the loss value itself, as a simple float value


Instruction: Implementing a 3D gaussian blur using separable 2D convolutions in pytorch

Input:
I'm trying to implement a gaussian-like blurring of a 3D volume in pytorch. I can do a 2D blur of a 2D image by convolving with a 2D gaussian kernel easy enough, and the same approach seems to work for 3D with a 3D gaussian kernel. However, it is very slow in 3D (especially with larger sigmas/kernel sizes). I understand this can also be done instead by convolving 3 times with the 2D kernel which should be much faster, but I can't get this to work. My test case is below.
import torch
import torch.nn.functional as F

VOL_SIZE = 21

def make_gaussian_kernel(sigma):
    ks = int(sigma * 5)
    if ks % 2 == 0:
        ks += 1
    ts = torch.linspace(-ks // 2, ks // 2 + 1, ks)
    gauss = torch.exp((-(ts / sigma)**2 / 2))
    kernel = gauss / gauss.sum()

    return kernel

def test_3d_gaussian_blur(blur_sigma=2):
    # Make a test volume
    vol = torch.zeros([VOL_SIZE] * 3)
    vol[VOL_SIZE // 2, VOL_SIZE // 2, VOL_SIZE // 2] = 1

    # 3D convolution
    vol_in = vol.reshape(1, 1, *vol.shape)
    k = make_gaussian_kernel(blur_sigma)
    k3d = torch.einsum('i,j,k->ijk', k, k, k)
    k3d = k3d / k3d.sum()
    vol_3d = F.conv3d(vol_in, k3d.reshape(1, 1, *k3d.shape), stride=1, padding=len(k) // 2)

    # Separable 2D convolution
    vol_in = vol.reshape(1, *vol.shape)
    k2d = torch.einsum('i,j->ij', k, k)
    k2d = k2d / k2d.sum()
    k2d = k2d.expand(VOL_SIZE, 1, *k2d.shape)
    for i in range(3):
        vol_in = vol_in.permute(0, 3, 1, 2)
        vol_in = F.conv2d(vol_in, k2d, stride=1, padding=len(k) // 2, groups=VOL_SIZE)
    vol_3d_sep = vol_in

    torch.allclose(vol_3d, vol_3d_sep)  # --> False

Any help would be very much appreciated!

Output:
You theoreticaly can compute the 3d-gaussian convolution using three 2d-convolutions, but that would mean you have to reduce the size of the 2d-kernel, as you're effectively convolving in each direction twice.
But computationally more efficient (and what you usually want) is a separation into 1d-kernels. I changed the second part of your function to implement this. (And I must say I really liked your permutation-based appraoch!) Since you're using a 3d volume you can't really use the conv2d or conv1d functions well, so the best thing is really just using conv3d even if you're just computing 1d-convolutions.
Note that allclose uses a threshold of 1e-8 which we do not reach with this method, probably due to cancellation errors.
def test_3d_gaussian_blur(blur_sigma=2):
    # Make a test volume
    vol = torch.randn([VOL_SIZE] * 3) # using something other than zeros
    vol[VOL_SIZE // 2, VOL_SIZE // 2, VOL_SIZE // 2] = 1

    # 3D convolution
    vol_in = vol.reshape(1, 1, *vol.shape)
    k = make_gaussian_kernel(blur_sigma)
    k3d = torch.einsum('i,j,k->ijk', k, k, k)
    k3d = k3d / k3d.sum()
    vol_3d = F.conv3d(vol_in, k3d.reshape(1, 1, *k3d.shape), stride=1, padding=len(k) // 2)

    # Separable 1D convolution
    vol_in = vol[None, None, ...]
    # k2d = torch.einsum('i,j->ij', k, k)
    # k2d = k2d / k2d.sum() # not necessary if kernel already sums to zero, check:
    # print(f'{k2d.sum()=}')
    k1d = k[None, None, :, None, None]
    for i in range(3):
        vol_in = vol_in.permute(0, 1, 4, 2, 3)
        vol_in = F.conv3d(vol_in, k1d, stride=1, padding=(len(k) // 2, 0, 0))
    vol_3d_sep = vol_in
    print((vol_3d- vol_3d_sep).abs().max()) # something ~1e-7
    print(torch.allclose(vol_3d, vol_3d_sep)) # allclose checks if it is around 1e-8

Addendum: If you really want to abuse conv2d to process the volumes you can try
# separate 3d kernel into 1d + 2d
vol_in = vol[None, None, ...]
k2d = torch.einsum('i,j->ij', k, k)
k2d = k2d.expand(VOL_SIZE, 1, len(k), len(k))
# k2d = k2d / k2d.sum() # not necessary if kernel already sums to zero, check:
# print(f'{k2d.sum()=}')
k1d = k[None, None, :, None, None]
vol_in = F.conv3d(vol_in, k1d, stride=1, padding=(len(k) // 2, 0, 0))
vol_in = vol_in[0, ...]
# abuse conv2d-groups argument for volume dimension, works only for 1 channel volumes
vol_in = F.conv2d(vol_in, k2d, stride=1, padding=(len(k) // 2, len(k) // 2), groups=VOL_SIZE)
vol_3d_sep = vol_in

Or using exclusively conv2d you could do:
# separate 3d kernel into 1d + 2d
vol_in = vol[None,  ...]
# 1d kernel
k1d = k[None, None, :,  None]
k1d = k1d.expand(VOL_SIZE, 1, len(k), 1)
# 2d kernel
k2d = torch.einsum('i,j->ij', k, k)
k2d = k2d.expand(VOL_SIZE, 1, len(k), len(k))
vol_in = vol_in.permute(0, 2, 1, 3)
vol_in = F.conv2d(vol_in, k1d, stride=1, padding=(len(k) // 2, 0), groups=VOL_SIZE)
vol_in = vol_in.permute(0, 2, 1, 3)
vol_in = F.conv2d(vol_in, k2d, stride=1, padding=(len(k) // 2, len(k) // 2), groups=VOL_SIZE)
vol_3d_sep = vol_in

These should still be faster than three consecutive 2d convolutions.


Instruction: Python: BERT Model Pooling Error - mean() received an invalid combination of arguments - got (str, int)

Input:
I am writing the code to train a bert model on my dataset. By when I run the code it throws an error in the average pool layer. I am unable to understand what causes this error.
Model
class BERTBaseUncased(nn.Module):
    def __init__(self, bert_path):
        super(BERTBaseUncased, self).__init__()
        self.bert_path = bert_path
        self.bert = transformers.BertModel.from_pretrained(self.bert_path)
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768 * 2, 1)

    def forward(
            self,
            ids,
            mask,
            token_type_ids
    ):
        o1, _ = self.bert(
            ids,
            attention_mask=mask,
            token_type_ids=token_type_ids)

        apool = torch.mean(o1, 1)
        mpool, _ = torch.max(o1, 1)
        cat = torch.cat((apool, mpool), 1)

        bo = self.bert_drop(cat)
        p2 = self.out(bo)
        return p2

Error
Exception in device=TPU:0: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 228, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-12-94e926c1f4df>", line 4, in _mp_fn
    a = _run()
  File "<ipython-input-5-ef9fa564682f>", line 146, in _run
    train_loop_fn(para_loader.per_device_loader(device), model, optimizer, device, scheduler=scheduler)
  File "<ipython-input-5-ef9fa564682f>", line 22, in train_loop_fn
    token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 577, in __call__
    result = self.forward(*input, **kwargs)
  File "<ipython-input-11-9196e0d23668>", line 73, in forward
    apool = torch.mean(o1, 1)
TypeError: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

I am trying to run this on a Kaggle TPU. How to fix this?

Output:
Since one of the 3.X updates, the models return now task-specific output objects (which are dictionaries) instead of plain tuples. You can either force the model to return a tuple by specifying return_dict=False:
o1, _ = self.bert(
            ids,
            attention_mask=mask,
            token_type_ids=token_type_ids,
            return_dict=False)

or by utilizing the basemodeloutputwithpoolingandcrossattentions object:
o = self.bert(
            ids,
            attention_mask=mask,
            token_type_ids=token_type_ids)
#you can view the other attributes with o.keys()
o1 = o.last_hidden_state


Instruction: Is there a significant speed improvement when using transformers tokenizer over batch compared to per item?

Input:
is calling tokenizer on a batch significantly faster than on calling it on each item in a batch? e.g.
encodings = tokenizer(sentences)
# vs
encodings = [tokenizer(x) for x in sentences]

Output:
i ended up just timing both in case it's interesting for someone else
%%timeit
for _ in range(10**4): tokenizer("Lorem ipsum dolor sit amet, consectetur adipiscing elit.")
785 ms ± 24.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
tokenizer(["Lorem ipsum dolor sit amet, consectetur adipiscing elit."]*10**4)
266 ms ± 6.52 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)


Instruction: pytorch cross-entropy-loss weights not working

Input:
I was playing around with some code and and it behaved differently than what i expected. So i dumbed it down to a minimally working example:
import torch

test_act = torch.tensor([[2.,0.]])
test_target = torch.tensor([0])

loss_function_test = torch.nn.CrossEntropyLoss()
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
> tensor(0.1269)

weights=torch.tensor([0.1,0.5])
loss_function_test = torch.nn.CrossEntropyLoss(weight=weights)
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
> tensor(0.1269)

As you can see the outputs are the same regardless if there are weights present or not. But i would expect the second output to be 0.0127
Is there some normalization going on that I dont know about? Or is it possibly bugged?

Output:
In this example, I add a second dataum with a different target class, and the effect of weights is visible.
import torch

test_act = torch.tensor([[2.,1.],[1.,4.]])
test_target = torch.tensor([0,1])

loss_function_test = torch.nn.CrossEntropyLoss()
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
>>> tensor(0.1809)

weights=torch.tensor([0.1,0.5])
loss_function_test = torch.nn.CrossEntropyLoss(weight=weights)
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
>>> tensor(0.0927)

This effect is because "The losses are averaged across observations for each minibatch. If the weight argument is specified then this is a weighted average" but only across the minibatch.
Personally I find this a bit strange and would think it would be useful to apply the weights globally (i.e. even if all classes are not present in each minibatch). One of the prominent uses of the weight parameter would ostensibly be to give more weight to classes that are under-represented in the dataset, but by this formulation the minority classes are only given higher weights for the minibatches in which they are present (which, of course, is a low percentage because they are a minority class).
In any case that is how Pytorch defines this operation.


Instruction: Python image types, shapes, and channels for segmentation

Input:
I am using this tutorial for instance segmentation in PyTorch. The test data the tutorial uses includes images and accompanying image masks from a dataset available here. I have an example of one of the image masks from that data set here (example data for this question). That mask looks like this by default in the dataset:

The tutorial uses this code:
mask.putpalette([
    0, 0, 0, # black background
    255, 0, 0, # index 1 is red
    255, 255, 0, # index 2 is yellow
    255, 153, 0, # index 3 is orange
])

as explanatory step for the mask to make it look like this:

but that code is not requisite in the segmentation process itself. It's just used to show what the mask contains.
I am trying to use my own image data. I created masks for the images in G.I.M.P. This is one of the masks I made. It looks like this by default.

As I try to run the tutorial code, I have problems with the masks. This code chunk creates a class that creates PyTorch datasets.
import os
import numpy as np
import torch
import torch.utils.data
from PIL import Image

class PennFudanDataset(torch.utils.data.Dataset):
    def __init__(self, root, transforms=None):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

dataset = PennFudanDataset('PennFudanPed/')
dataset[0]

The last line returns:
(<PIL.Image.Image image mode=RGB size=559x536 at 0x7FCB4267C390>,
 {'area': tensor([35358., 36225.]), 'boxes': tensor([[159., 181., 301., 430.],
          [419., 170., 534., 485.]]), 'image_id': tensor([0]), 'iscrowd': tensor([0, 0]), 'labels': tensor([1, 1]), 'masks': tensor([[[0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           ...,
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0]],

          [[0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           ...,
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0],
           [0, 0, 0,  ..., 0, 0, 0]]], dtype=torch.uint8)})

When I run this code with my data,
...
dataset = four_chs('drive/MyDrive/chambers/')
dataset[0]

I get this error:
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:38: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-17-12074ae9ab35> in <module>()
      1 len(dataset)
----> 2 dataset[0]

<ipython-input-1-99ab92a46ebe> in __getitem__(self, idx)
     42         boxes = []
     43         for i in range(num_objs):
---> 44             pos = np.where(masks[i])
     45             xmin = np.min(pos[1])
     46             xmax = np.max(pos[1])

TypeError: 'bool' object is not subscriptable

I'm not sure exactly what is going on, but there is a difference between my mask and the mask that is in the test data. They are both PNG file types, but it seems that mine has the red, blue, green channels split out with another channel or something, but I don't know what it is, based on the shape of the object in Python. This comes from one of the masks I made:
mask2 = np.array(mask1)
mask2.shape
(5312, 2988, 4)

For one of the test data masks:
mask2 = np.array(mask)
mask2.shape
(536, 559)

There appears to be only one channel? So with their different shapes, I guess that why I get the error from (this is an excerpt from the code I pasted earlier)
...
        mask_path = os.path.join(self.root, "masks", self.masks[idx])
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
for i in range(num_objs):
             pos = np.where(masks[i])
...

How do I get my masks' shapes to match those of the masks in the test data so that I can use the rest of segmentation code create my PyTorch compatible dataset that will work with the segmentation algorithm? I'm not trying to get the same height and width, but change the number of channels/layers, but I don't think I want it to be grayscale.
EDIT after HansHirse's comment:
I went to back to G.I.M.P and used the Image Mode menu to change the image to grayscale. I exported with that setting. I tried to run the code with that file and it did not work.
I also found a way to convert the R.B.G. image to grayscale with upon import with Image.open().convert("L"). This does not work either.
In both cases, the problem has to do with speckles of colors that I thought were separate being mixed together. For example, I used HansHirse's advice to fill in the areas of interest with grey "colors" of 1,2,3,and 4, while the background stayed 0. Upon importing the file that created, the values of those files are 3,5,8, and 10. And while one of the shapes may be mostly of value 3, there are vagrant pixels with that value in the other shapes, so there is no value that is contained entirely in one shape. With that situation, the code draws bounding boxes that surround all 4 shapes, instead of around one shape.
I'm aware of using the hue, saturation, value (H.S.V.) color space and tried converting to that color space. That still doesn't solve the problem for me.
I'm trying to figure out how to use something like
np.where( mask[<buffered shape1 xmin>,<buffered shape1 xmax>, <buffered shape1 ymin>, <buffered shape1 ymax>,0] == <majority color value for shape>)

to sort of quarter up the mask, filter based on the main color's value for that shape in that quarter, and get the actual x and y values for the shape in that quarter. With those values, I figure I can use the min and max from the actual values to create my bounding boxes.
Another note is that when exporting from G.I.M.P., there is a drop-down menu in the export window to set the file to 8 bit R.G.B. or Gray. Select 8bpc Gray for the format needed.

Output:
Following is an example how to create a grayscale image representing classes for a segmentation task or similar.
On some black background, draw some shapes with fill values in the range of 1, ..., #classes. For visualization purposes, this mask is plotted as perceived as a regular grayscale image as well as scaled to the said value range – to emphasize that the mask looks all black in general, but there's actual content in it. This mask is saved as a lossless PNG image, and then opened using Pillow, and converted to mode P. Last step is to set up a proper palette for the desired number of colors, and apply that palette using Image.putpalette.
import cv2
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image

# Generate mask:  0 - Background  |  1 - Class 1  |  2 - Class 2, and so on.
mask = np.zeros((300, 300), np.uint8)
cv2.rectangle(mask, (30, 40), (75, 60), 1, cv2.FILLED)
cv2.circle(mask, (230, 50), 85, 2, cv2.FILLED)
cv2.ellipse(mask, (230, 230), (60, 40), 0, 0, 360, 3, cv2.FILLED)
cv2.line(mask, (20, 240), (80, 260), 4, 5)

# Save mask as lossless PNG image
cv2.imwrite('mask.png', mask)

# Visualization
plt.figure(1, figsize=(18, 6))
plt.subplot(1, 3, 1), plt.imshow(mask, vmin=0, vmax=255, cmap='gray')
plt.colorbar(), plt.title('Mask when shown as regular image')
plt.subplot(1, 3, 2), plt.imshow(mask, cmap='gray')
plt.colorbar(), plt.title('Mask when shown scaled to values 0 - 4')

# Open mask with Pillow, and convert to mode 'P'
mask = Image.open('mask.png').convert('P')

# Set up and apply palette data
mask.putpalette([  0,   0,   0,         # Background - Black
                 255,   0,   0,         # Class 1 - Red
                   0, 255,   0,         # Class 2 - Green
                   0,   0, 255,         # Class 3 - Blue
                 255, 255,   0])        # Class 4 - Yellow

# More visualization
plt.subplot(1, 3, 3), plt.imshow(mask)
plt.title('Mask when shown as indexed image')
plt.tight_layout(), plt.show()

The first steps generating the actual mask can be done in GIMP, of course. Please be sure to use black background, and fill values in the range 1, ..., #classes. If you have difficulties to do that because these colors are all nearly black, draw your shapes in some bright, distinguishable colors, and later just fill these with values  1, 2, and so on.
----------------------------------------
System information
----------------------------------------
Platform:      Windows-10-10.0.19041-SP0
Python:        3.9.1
PyCharm:       2021.1.1
Matplotlib:    3.4.2
NumPy:         1.20.3
OpenCV:        4.5.2
Pillow:        8.2.0
----------------------------------------


Instruction: Pytorch Model always outputs 0.5 for an unkown reason

Input:
I have a pytorch model I'm trying to use to do facial recognition. I am using the same model structure, loss, and optimizer as a working code, but it seems like the backprop won't do anything, any output of the NN is just 0.5. Here is the code, any help/suggestions is/are appreciated.
import cv2
import numpy as np
import PIL
import torchvision.transforms as transforms
import torch
from tqdm import tqdm
import torch.nn as nn
import torch.nn.functional as F

devicet = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device(devicet)
if devicet == 'cpu':
  print ('Using CPU')
else:
  print ('Using GPU')
cuda0 = torch.device('cuda:0')

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.step1 = nn.Linear(10000, 200)
        self.step2 = nn.Linear(200, 200)
        self.step3 = nn.Linear(200, 1)
    def forward(self, x):
      x = F.relu(x)
      x = self.step1(x)
      x = F.relu(x)
      x = self.step2(x)
      x = F.relu(x)
      x = self.step3(x)
      x = F.relu(x)
      x = torch.sigmoid(x)
      return (x)
net = Net()

transformer = transforms.ToTensor()
original_image = cv2.imread('group.jpg', cv2.IMREAD_GRAYSCALE)
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
detected_faces = face_cascade.detectMultiScale(image=original_image, scaleFactor=1.05, minNeighbors=2)
tensor_collection_faces = []
tensor_collection_output = []
for (x, y, width, height) in detected_faces:
    im_pil = PIL.Image.fromarray(original_image)
    im_pil = im_pil.crop((x,y,x+width,y+height))
    im_pil = im_pil.resize((100,100))
    im_pil = PIL.ImageOps.grayscale(im_pil)
    curr_image_tens = (transformer(im_pil)).reshape((10000)).numpy()
    tensor_collection_faces.append(curr_image_tens)
    display(im_pil)
    tensor_collection_output.append([float(input('Expected Result: '))])
input_tensor = torch.tensor(tensor_collection_faces)
output_tensor = torch.tensor(tensor_collection_output)
loss_fn = torch.nn.BCELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
for i in tqdm(range(500), desc='Training'):
    y_pred = net(input_tensor)
    loss = loss_fn(y_pred, output_tensor)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
print (net(input_tensor))

The output of the code:
Training: 100%|██████████| 500/500 [00:11<00:00, 44.40it/s]tensor([[0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000]], grad_fn=<SigmoidBackward>)

Output:
You applied both relu and sigmoid to your final output. In this case, you want to apply only sigmoid.
def forward(self, x):
    x = F.relu(x)
    x = self.step1(x)
    x = F.relu(x)
    x = self.step2(x)
    x = F.relu(x)
    x = self.step3(x)
    x = F.relu(x) # <---- delete this line
    x = torch.sigmoid(x)
    return (x)

What's happening is your network is outputting negative values in the last layer (before relu or sigmoid are applied), which when passed to relu go to 0. sigmoid(0) = 0.5, which is why you are seeing 0.5.
    x = self.step3(x)    # x = some negative value
    x = F.relu(x)        # relu(negative) = 0
    x = torch.sigmoid(x) # sigmoid(0) = 0.5

There might be other issues with your code, but it's hard to say without having access to the data/labels (or even toy data).


Instruction: Cannot export Yolov5 model using export

Input:
I've trained Yolov5s model in colab.research env. After training I've moved best.pt to main yolov5 directory and renamed file to yolov5s.pt.
After calling export.py i got error listed below
!python models/export.py

Namespace(batch_size=1, img_size=[640, 640], weights='./yolov5s.pt')
Traceback (most recent call last):
  File "models/export.py", line 33, in <module>
    model = attempt_load(opt.weights, map_location=torch.device('cpu'))  # load FP32 model
  File "./models/experimental.py", line 137, in attempt_load
    model.append(torch.load(w, map_location=map_location)['model'].float().fuse().eval())  # load FP32 model
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 587, in load
    with _open_zipfile_reader(opened_file) as opened_zipfile:
  File "/usr/local/lib/python3.7/dist-packages/torch/serialization.py", line 242, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading zip archive: failed finding central directory

I'm trying to export this model to use it in Android App. If I use best.pt I've got other error on loading network:
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: org.pytorch.demo.objectdetection, PID: 6935
    java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.demo.objectdetection/org.pytorch.demo.objectdetection.MainActivity}: com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl
    (no backtrace available)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3449)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3601)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:85)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2066)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)
     Caused by: com.facebook.jni.CppException: [enforce fail at inline_container.cc:222] . file not found: archive/constants.pkl
    (no backtrace available)
        at org.pytorch.NativePeer.initHybridAndroidAsset(Native Method)
        at org.pytorch.NativePeer.<init>(NativePeer.java:27)
        at org.pytorch.PyTorchAndroid.loadModuleFromAsset(PyTorchAndroid.java:31)
        at org.pytorch.demo.objectdetection.MainActivity.onCreate(MainActivity.java:165)
        at android.app.Activity.performCreate(Activity.java:8000)
        at android.app.Activity.performCreate(Activity.java:7984)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1309)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3422)
            ... 11 more
I/Process: Sending signal. PID: 6935 SIG: 9

I haven't found any solutions on my own. Do you have any idea how can I fix this error?
Edit:
Fixed - I could export model if I linked to directory:
!python models/export.py --weights ./runs/train/exp7/weights/best.pt --img 640 --batch 1

Output:
Fixed with linking model in runs directory:
!python models/export.py --weights ./runs/train//exp/weights/best.pt --img 640 --batch 1


Instruction: How to reshape a 32x512 into 128x128 keeping it as consistant 4 32x128 parts stacked?

Input:
So I want to turn 32x512 into 128x128 keeping it as consistant 4 32x128 parts stacked like this:
.
How to do such a thing inside the PyTorch model?

So here is what I tried based on this answer:
import torch
x = torch.arange(32*512*3*2).reshape(2, 3, 32, 512)
print(x.shape)

x = list(x.split(4, dim=3))
print(f" xx {len(x)} {iii.shape for iii in x }")

x = torch.cat(x)
print(f" xs {x.shape}")

And this outputs:
torch.Size([2, 3, 32, 512])
 xx 128 <generator object <genexpr> at 0x7f358f1edad0>
 xs torch.Size([256, 3, 32, 4])

While I wanted  xs torch.Size([2, 3, 128, 128])

Output:
You can use, for tensor t:
torch.cat(t.split(128, dim=1))

To transform in the reverse direction, you can use:
torch.cat(t.split(32, dim=0))

For your updated question:
torch.cat(t.split(128, dim=3), dim=2)

For the reverse:
torch.cat(t.split(32, dim=2), dim=3)

In general, the dim for torch.split tells which dimension you want to split over, and the dim for torch.cat tells which dimension you want to concatenate over.


Instruction: PyTorch: Error>> expected scalar type float but found double

Input:
I've just started using pytorch and I am trying a simple multi-layer perceptron . My ReLU Activation Function is the following:
def ReLU_activation_func(outputs):
    print(type(outputs))
    result = torch.where(outputs > 0, outputs, 0.)
    result = float(result)
    return result

So I am trying to maintain the value which is greater than 0 and change the value to 0 if the value is smaller than 0.
And this is a part of the main code where I use the ReLU Function (where I have the error):
def forward_pass(train_loader):
    for batch_idx, (image, label) in enumerate(train_loader):
        print(image.size())
        x = image.view(-1, 28 * 28)
        print(x.size())

        input_node_num = 28 * 28
        hidden_node_num = 100
        output_node_num = 10
        W_ih = torch.rand(input_node_num, hidden_node_num)
        W_ho = torch.rand(hidden_node_num, output_node_num)
        final_output_n = ReLU_activation_func(torch.matmul(x, W_ih))

and when I run the code, I get the following error:
RuntimeError:
1 forward_pass(train_loader)

in forward_pass(train_loader)
-----14         W_ih = torch.rand(input_node_num, hidden_node_num)
-----15         W_ho = torch.rand(hidden_node_num, output_node_num)
---->16         final_output_n = ReLU_activation_func(torch.matmul(x, W_ih))

in ReLU_activation_func(outputs)
-----10     print(type(outputs))
---->11     result = torch.where(outputs > 0, outputs, 0.)
-----12     result = float(result)
-----13     return result

RuntimeError: expected scalar type float but found double

Any help?

Output:
The issue is not on result, it's either on X, W_ih, or torch.where(outputs > 0, outputs, 0.).
If you don't set an argument for the dtype of torch.rand(), it will assign the dtype based on the pytorch's global default value.
The global variable can be changed using torch.set_default_tensor_type().
Or go the easy route:
def ReLU_activation_func(outputs):
    print((outputs).dtype)
    result = torch.where(outputs > 0, outputs, torch.zeros_like(outputs)).float()
    return result

# for the forward pass function, convert the tensor to floats before matmul
def forward_pass(train_loader):
    for batch_idx, (image, label) in enumerate(train_loader):
        ... <your code>
        X, W_ih = X.float(), W_ih.float()
        final_output_n = ReLU_activation_func(torch.matmul(x, W_ih))


Instruction: Methods to Load Saved Python List in Pytorch?

Input:
I have a question on how to load saved python lists in Pytorch. The reference environment is Google Colab.
Suppose I saved the regular list as follows.
drive.mount('/content/gdrive')
training_sets = ["a","b","c"]
model_save_name = 'training_sets.pt'
path = F"/content/gdrive/My Drive/Documents"
torch.save(training_sets,path)

Does pytorch support loading for regular python lists? Or does it only apply for pytorch tensors?
Thank you.

Output:
Of course yes!
You can save any kind of python object using torch.save
Example
import torch
a = [1,2,3]
torch.save(a,'a.pth')
b = torch.load('a.pth')
>>> b
[1, 2, 3]


Instruction: Pytorch error: 'BiSeNet' object has no attribute 'module'

Input:
I'm new to pytorch and I'm trying to study BiSeNet for image segmentation (code took from a github repo: https://github.com/ooooverflow/BiSeNet/blob/master/train.py). During the training phase, after some training epochs, the net performs validation and tryes to save the parameters of the model if the results of the val are better than the previous ones.
During this last operation, I get this error in line 109 (and also 102 during training):
AttributeError: 'BiSeNet' object has no attribute 'module'
I do not paste all the code here, but just the main steps.
First of all, they built the model like that:
os.environ['CUDA_VISIBLE_DEVICES'] = args.cuda
model = BiSeNet(args.num_classes, args.context_path)
if torch.cuda.is_available() and args.use_gpu:
        model = model.cuda()

So there exists a BiSeNet object created thanks to an imported module called "model" where there is a file named build_BiSeNet.py; In this script the class BiSeNet is defined and there is no attribute named module.
Looking at the pytorch documentation, seems like in the Model class there is an attribute called modules which contains the module I'd like to save.
In the docs, they suggest also to do torch.save(model.state_dict(), ...) in order to save the model, without calling the module attribute Ilike it is done in the line 109)
So, finally, my question is: In order to avoid the error that gives me, should I remove .module in line 109 (and so also in line 102) or maybe change this attribute to .modules?

Output:
Look at their demo.py, they are defining the model:
model = BiSeNet(args.num_classes, args.context_path)
if torch.cuda.is_available() and args.use_gpu:
  model = torch.nn.DataParallel(model).cuda()

Once you wrap the model with nn.DataParallel you get an "extra" .module in your way.
The saved checkpoint does not have this extra .module and therefore, when loading a saved checkpoint:
model.module.load_state_dict(torch.load(args.checkpoint_path))

It is not loaded to the model directly, but rather to the model.module.


Instruction: Mean squared logarithmic error using pytorch

Input: hello I'm new with PyTorch and i would like to use Mean squared logarithmic error as a loss function in my neural  network for training my DQN agent but i can't find the MSLE in the nn.functional in PyTorch what the best way to implement it ?

Output:
well is not hard to do it
the MSLE equation as the photo below shows
now, as some user on the PyTorch form suggested
you can be added as a class like this
class RMSLELoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.mse = nn.MSELoss()

    def forward(self, pred, actual):
        return torch.sqrt(self.mse(torch.log(pred + 1), torch.log(actual + 1)))

then you can call it normally
criterion = RMSLELoss()

rmsle = criterion(pred, actual)


Instruction: Is there a verctorized approach in PyTorch to get this result?

Input:
I would like to get this result with torch functions. Do you have suggestions?
import torch
test_tensor=torch.tensor([[1,  2,  3, 4,  5,  6],
                         [7,  8,  9, 10,  11,  12]]
                        )
print(test_tensor)
'''
I would like to get:
t_1 = torch.tensor([[6], #1+2+3
                  [24]])#7+8+9
t_2 = torch.tensor([[9], #1+3+5
                  [27]])#7+9+11
'''

Output:
Using standard Python stride notation
import torch
test_tensor=torch.tensor([[1,  2,  3, 4,  5,  6],
                         [7,  8,  9, 10,  11,  12]]
                        )

t1 = test_tensor[:, :3].sum(dim=1)
print(t1)
t2 = test_tensor[:, ::2].sum(dim=1)
print(t2)


Instruction: Find euclidean / cosine distance between a tensor and all tensors stored in a column of dataframe efficently

Input:
I have a tensor 'input_sentence_embed' with shape torch.Size([1, 768])
There is a dataframe 'matched_df' which looks like
   INCIDENT_NUMBER           enc_rep
0  INC000030884498      [[tensor(-0.2556), tensor(0.0188), tensor(0.02...
1  INC000029956111      [[tensor(-0.3115), tensor(0.2535), tensor(0.20..
2  INC000029555353      [[tensor(-0.3082), tensor(0.2814), tensor(0.24...
3  INC000029555338      [[tensor(-0.2759), tensor(0.2604), tensor(0.21...

Shape of each tensor element in dataframe looks like
 matched_df['enc_rep'].iloc[0].size()
 torch.Size([1, 768])

I want to find euclidean / cosine similarity between 'input_sentence_embed' and each row of 'matched_df' efficently.
If they were scalar values, I could have easily broadcasted 'input_sentence_embed' as a new column in 'matched_df' and then find cosine similarity between two columns.
I am struggling with two problems

How to broadcast 'input_sentence_embed' as a new column to the
'matched_df'
How to find cosine similarity between tensors stored
in two column

May be someone can also suggest me other easier methods to achieve the end goal of finding similarity between a tensor value and all tensors stored in a column of dataframe efficently.

Output:
Input data:
import pandas as pd
import numpy as np
from torch import tensor

match_df = pd.DataFrame({'INCIDENT_NUMBER': ['INC000030884498',
  'INC000029956111',
  'INC000029555353',
  'INC000029555338'],
 'enc_rep': [[[tensor(0.2971), tensor(0.4831), tensor(0.8239), tensor(0.2048)]],
  [[tensor(0.3481), tensor(0.8104) , tensor(0.2879), tensor(0.9747)]],
  [[tensor(0.2210), tensor(0.3478), tensor(0.2619), tensor(0.2429)]],
  [[tensor(0.2951), tensor(0.6698), tensor(0.9654), tensor(0.5733)]]]})

input_sentence_embed = [[tensor(0.0590), tensor(0.3919), tensor(0.7821) , tensor(0.1967)]]

How to broadcast 'input_sentence_embed' as a new column to the 'matched_df'

match_df["input_sentence_embed"] = [input_sentence_embed] * len(match_df)

How to find cosine similarity between tensors stored in two column

a = np.vstack(match_df["enc_rep"])
b = np.hstack(input_sentence_embed)
match_df["cosine_similarity"] = a.dot(b) / (np.linalg.norm(a) * np.linalg.norm(b))

Output result:
   INCIDENT_NUMBER                                            enc_rep                               input_sentence_embed  cosine_similarity
0  INC000030884498  [[tensor(0.2971), tensor(0.4831), tensor(0.823...  [[tensor(0.0590), tensor(0.3919), tensor(0.782...           0.446067
1  INC000029956111  [[tensor(0.3481), tensor(0.8104), tensor(0.287...  [[tensor(0.0590), tensor(0.3919), tensor(0.782...           0.377775
2  INC000029555353  [[tensor(0.2210), tensor(0.3478), tensor(0.261...  [[tensor(0.0590), tensor(0.3919), tensor(0.782...           0.201116
3  INC000029555338  [[tensor(0.2951), tensor(0.6698), tensor(0.965...  [[tensor(0.0590), tensor(0.3919), tensor(0.782...           0.574257


Instruction: Setting constraints for parameters in pytorch

Input:
I am trying to set some constraints for weight parameters in PyTorch, e.g. the sum of every row of the weight matrix be exactly one for a fully connected layer:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(28*28, 10*10)
        self.layer2 = nn.Linear(10*10, 5*5)
        self.layer3 = nn.Linear(5*5, 10)

    def forward(self, x):
        x=torch.sigmoid(self.layer1(x))
        x=torch.sigmoid(self.layer2(x))
        x=torch.sigmoid(self.layer3(x))

model=Net()

The constraint for this example network would be:
torch.sum(model.linear1.weight,0)==1
torch.sum(model.linear2.weight,0)==1
torch.sum(model.linear3.weight,0)==1

A commonly used method to set a constraint, clamp, is used to set constraints for every element, but in this case, I would be setting a constraint for every row, instead of any particular element of the weight matrix. Are there any ways to implement this kind of constraint?

Output:
A way I can think about is for example to normalize the vector of your choosing by its norm, which will give its direction, with size 1.
w0 = model.linear1.weight[0, :]
w0_hat = w0 / torch.linalg.norm(w0) # direction of w0, norm=1

I don't really see a way of doing this for the .sum, but I also don't see why one would want to.

Using L1 norm would do that for the .sum if you can guarantee the weights are all non-negative. Probably this isn't what you want.

If you insist of normalizing such that the .sum will equal to 1, you have to define the normalization yourself, because there is no single algorithm to decide which weight index gets changed, and by how much.


Instruction: PyTorch: TypeError: 'int' object is not subscriptable

Input:
I'm new to pytorch and I am creating a one hot encoding function for multi-layer perceptron but I'm having some issues. Here is the code:
def one_hot_encoding(label):
    for idx, val in enumerate(label):
        one_hot_outputs = [0]*len(label)
        idx_n = idx[val]
        one_hot_outputs[idx_n] = 1
return one_hot_outputs

I'm having a type error saying:
in one_hot_encoding(label)
  2     for idx, val in enumerate(label):
  3         one_hot_outputs = [0]*len(label)
> 4         idx_n = idx[val]
  5         one_hot_outputs[idx_n] = 1
  6     return one_hot_outputs
TypeError: 'int' object is not subscriptable

Any help?

Output:
This is not a direct answer, but an alternative. PyTorch already has functionality for this: torch.nn.functional.one_hot. So, if you have a label tensor label and n classes, just call:
torch.nn.functional.one_hot(label, num_classes=n)


Instruction: Pytorch Data Loader concatenate an image to input images

Input:
In PyTorch data loader, how could I concatenate an image (let's say x.jpg) in-band wise to each and every input images. ie, in effect I will have 4 band input ( 3 band input jpg with 1 band x.jpg. How to implement it.
Please find below example of my current dataloader just to load the images. To this, I want to add x.jpg to "image"(ie input image, not to mask)
from PIL import Image

class lakeDataSet(Dataset):
   def __init__(self, root, transform):
   super().__init__()
   self.root = root
   self.img_dir = os.path.join(root,'image-c3/c3-crop')   #9UAV
   self.mask_dir = os.path.join(root,'label-c3/c3-crop')
   # self.mask_dir = os.path.join(root,'test')
   self.files = [fname for fname in os.listdir(self.img_dir) if fname.endswith('.jpg')]
self.transform = transform

   def __len__(self):
     return len(self.files)

   def __getitem__(self,I):
     fname = self.files[i]
     img_path = os.path.join(self.img_dir, fname)
     mask_path = os.path.join(self.mask_dir, fname)

     img = self.transform(Image.open(img_path))
     mask = self.transform(Image.open(mask_path))
     return img, mask

Output:
I suppose the self.transform already has ToTensor. Otherwise you should specify it as well.
Then you can just concat the first dimension. Like
x_jpg = self.transform(Image.open('x.jpg'))
img = torch.cat((img, x_jpg), 0)

The x.jpg has to have only 1 channel, if it's an RGB then obviously it'll become 6 channels instead of 4.


Instruction: What and where am I going wrong in this code for pytorch based object detection?

Input:
I am using Yolov5 for this project
Here is my code
import numpy as np
import cv2
import torch
import torch.backends.cudnn as cudnn
from models.experimental import attempt_load
from utils.general import non_max_suppression

weights = '/Users/nidhi/Desktop/yolov5/best.pt'
device = torch.device('cpu')

model = attempt_load(weights, map_location=device)  # load FP32 model
stride = int(model.stride.max())  # model stride
cudnn.benchmark = True

# Capture with opencv and detect object
cap = cv2.VideoCapture('Pothole testing.mp4')
width, height = (352, 352) # quality
cap.set(3, width) # width
cap.set(4, height) # height

while(cap.isOpened()):
    time.sleep(0.2) # wait for 0.2 second
    ret, frame = cap.read()
    if ret ==True:
        now = time.time()
        img = torch.from_numpy(frame).float().to(device).permute(2, 0, 1)
        img /= 255.0  # 0 - 255 to 0.0 - 1.0

        if img.ndimension() == 3:
            img = img.unsqueeze(0)

        pred = model(img, augment=False)[0]
        pred = non_max_suppression(pred, 0.39, 0.45, classes=0, agnostic=True) # img, conf, iou, classes, ...
        print('time -> ', time.time()-now)
     else:
         break

cap.release()

The error I am getting:
  File "run.py", line 38, in <module>
    pred = model(img, augment=False)[0]
  File "/Users/nidhi/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/nidhi/Desktop/yolov5/models/yolo.py", line 118, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "/Users/nidhi/Desktop/yolov5/models/yolo.py", line 134, in forward_once
    x = m(x)  # run
  File "/Users/nidhi/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/nidhi/Desktop/yolov5/models/common.py", line 152, in forward
    return torch.cat(x, self.d)
RuntimeError: Sizes of tensors must match except in dimension 1. Got 108 and 107 in dimension 3 (The offending index is 1)

Operating system: macOS Big Sur 11.2.3
Python version: 3.8.2
The model is used best.pt which I had trained on Google Colab, I used yolov5l model to train the dataset.

Output:
Are you getting your error in the following line?
pred = model(img, augment=False)[0]

It might be because YOLO expects inputs of the image size which are multiple of 32. So 320×320, 352×352 etc. But you are 352x288. You will either have to resize it, or pad the 288 dimension with white/black pixels to make it 352.
If you are not sure about where you are getting the error, can you attach the whole error?


Instruction: No such file or directory

Input:
import os
from PIL import Image
path='D:/SomeExperiments/KITTRawData/2011_09_26/2011_09_26_drive_0091_sync/image_03/data/0000000100.jpg'
with open(path,'rb') as f:
    pass

error as below
[Errno 2] No such file or directory: 'D:/SomeExperiments/KITTRawData/2011_09_26/2011_09_26_drive_0091_sync/image_03/data/0000000224.jpg'

but I can find this path on my computer:

Output:
Possible solution
Hold the shift key and right click on the image file. Select copy as path. Then paste the copied path into your python script. This SHOULD work for sure.
Possible reasons for the error

You are using slashes instead of backslashes - My version of windows and python supports using slashes but I'm not sure if it supports all.

File extension mismatch - Maybe your files are .jpeg and yet you are trying .jpg in your script. Path is not case sensitive in Windows but it's always a good practice to use correct case in the path, for example if your file is aBc.TXT you should use the exact same name not abc.txt.


Instruction: Convert an Array to PyTorch IValue on Android

Input:
Simple question.
I have an array of ints in Android (Kotlin). I need to convert it to an org.pytorch.IValue. I believe that such a conversion can be done, but I can't figure out how to do it. This is my array
val array = Array(34)     { IntArray(1) }
How do I get this wrapped in an IValue?

Output: I found the way to do this by way of the Tensor object. The Tensor class has a static method for creating tensor buffers, such as by Tensor.allocateIntBuffer(). The argument for that call accepts the number of elements that the buffer will contain. The .put() method on the buffer places items within the buffer. Once the buffer is populated, it can be converted to an IValue with IValue.from(sourceTensor).


Instruction: Shuffling along a given axis in PyTorch

Input:
I have the a dataset that gets loaded in  with the following dimension [batch_size, seq_len, n_features] (e.g. torch.Size([16, 600, 130])).
I want to be able to shuffle this data along the sequence length axis=1 without altering the batch ordering or the feature vector ordering in PyTorch.
Further explanation: For exemplification let's say my batch size is 3, sequence length is 3 and number of features is 2.
example:
tensor([[[1,1],[2,2],[3,3]],[[4,4],[5,5],[6,6]],[[7,7],[8,8],[9,9]]])  I want to be able to randomly shuffle the following way:
tensor([[[3,3],[1,1],[2,2]],[[6,6],[5,5],[4,4]],[[8,8],[7,7],[9,9]]])
Are there any PyTorch functions that will do that automatically for me, or does anyone know what would be a good way to implement this?

Output:
You can use torch.randperm.
For tensor t, you can use:
t[:,torch.randperm(t.shape[1]),:]

For your example:
>>> t = torch.tensor([[[1,1],[2,2],[3,3]],[[4,4],[5,5],[6,6]],[[7,7],[8,8],[9,9]]])
>>> t
tensor([[[1, 1],
         [2, 2],
         [3, 3]],

        [[4, 4],
         [5, 5],
         [6, 6]],

        [[7, 7],
         [8, 8],
         [9, 9]]])
>>> t[:,torch.randperm(t.shape[1]),:]
tensor([[[2, 2],
         [3, 3],
         [1, 1]],

        [[5, 5],
         [6, 6],
         [4, 4]],

        [[8, 8],
         [9, 9],
         [7, 7]]])


Instruction: Use other source to download from than pip in PythonVirtualenvOperator

Input:
Say I'm using the PythonVirtualenvOperator and have PyTorch as a requirement. When calling "pip freeze" I get
#requirements.txt
.
.
torch==1.8.1+cpu

and defining my task as
#tasks.py
from airflow.operators.python import PythonVirtualenvOperator

t1= PythonVirtualenvOperator(
        task_id = "test",
        python_version = "3.7",
        python_callable = test_func,
        requirements = ["torch==1.8.1+cpu"]
    )

throws the ERROR: Could not find a version that satisfies the requirement torch==1.8.1+cpu.
In the documentation from PyTorch we install it by pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html i.e downloading it from their webpage and not from pip (if I understand it correctly), which might be why pip fails in the venv. Thus I would like to make the venv (created by airflow for the PythonVirtualOperator) to download torch from the link specified above, instead of pip.
Is that doable? And is there a difference between torch==1.8.1+cpu and just torch==1.8.1 when using the cpu i.e does it make a difference if I just remove the +cpu?

Output:
This seems to be working (tested on Py3.7):
requirements=["torch==1.8.1+cpu", "-f", "https://download.pytorch.org/whl/torch_stable.html"]

Logs from the task:
[2021-05-24 18:37:20,762] {process_utils.py:135} INFO - Executing cmd: virtualenv /tmp/venv9kpx2ahm --system-site-packages --python=python3.7
[2021-05-24 18:37:20,781] {process_utils.py:139} INFO - Output:
[2021-05-24 18:37:21,365] {process_utils.py:143} INFO - created virtual environment CPython3.7.10.final.0-64 in 436ms
[2021-05-24 18:37:21,367] {process_utils.py:143} INFO -   creator CPython3Posix(dest=/tmp/venv9kpx2ahm, clear=False, no_vcs_ignore=False, global=True)
[2021-05-24 18:37:21,369] {process_utils.py:143} INFO -   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
[2021-05-24 18:37:21,370] {process_utils.py:143} INFO -     added seed packages: pip==21.1.1, setuptools==56.0.0, wheel==0.36.2
[2021-05-24 18:37:21,371] {process_utils.py:143} INFO -   activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2021-05-24 18:37:21,386] {process_utils.py:135} INFO - Executing cmd: /tmp/venv9kpx2ahm/bin/pip install torch==1.8.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
[2021-05-24 18:37:21,401] {process_utils.py:139} INFO - Output:
[2021-05-24 18:37:22,455] {process_utils.py:143} INFO - Looking in links: https://download.pytorch.org/whl/torch_stable.html
[2021-05-24 18:37:34,259] {process_utils.py:143} INFO - Collecting torch==1.8.1+cpu
[2021-05-24 18:37:34,820] {process_utils.py:143} INFO -   Downloading https://download.pytorch.org/whl/cpu/torch-1.8.1%2Bcpu-cp37-cp37m-linux_x86_64.whl (169.1 MB)
[2021-05-24 18:41:46,125] {process_utils.py:143} INFO - Requirement already satisfied: numpy in /usr/local/lib/python3.7/site-packages (from torch==1.8.1+cpu) (1.20.3)
[2021-05-24 18:41:46,128] {process_utils.py:143} INFO - Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/site-packages (from torch==1.8.1+cpu) (3.7.4.3)
[2021-05-24 18:41:49,211] {process_utils.py:143} INFO - Installing collected packages: torch
[2021-05-24 18:41:57,106] {process_utils.py:143} INFO - Successfully installed torch-1.8.1+cpu

However, I'm not sure if installing pytroch for every task/DAG run is optimal. By installing required dependencies on workers you can reduce the overhead (in my case installing pytorch took 5 minutes).


Instruction: Calculate `dot` product of image and vector

Input:
I would like to calculate something like dot product of vector and image with shapes:

(3)
(3,1080,1080)

and the output should be (1,1080,1080)
img = torch.rand((3,1080,1080))
R_coefficient=float(0.2126)
G_coefficient=float(0.7152)
B_coefficient=float(0.0722)
Y = img[0,...]*R_coefficient + img[1,...]*G_coefficient img[2,...]*B_coefficient

This code above gives me results, I look for, but I would like to use PyTorch utils like torch.dot, torch.matmul etc.
I have tried these
TensorImage(torch.dot(img[:,...],RGB_vector[:]))
TensorImage(torch.matmul(img,RGB_vector[:]))

These two options give me an errors connecting with shape, I have rejected them.
RGB_vector = torch.tensor([[[0.2126, 0.7152, 0.0722]]], dtype=img.dtype).permute(2,1,0)
    #     torch.t(RGB_vector)
        print(RGB_vector.shape)
        print(img.shape)
        return TensorImage(img[:,...]*RGB_vector)#.unsqueeze_(0)

this sample above works, but I am getting image with shape (3,1080,1080), but I need to get 1 instead of 3 at the first shape's dimension
Current working example
import torch
img = torch.rand((3,1080,1080))
RGB_vector = torch.tensor([[[0.2126, 0.7152, 0.0722]]], dtype=img.dtype).permute(2,1,0)
print(RGB_vector.shape)
print(img.shape)
TensorImage(img[:,...]*RGB_vector).unsqueeze_(0)

Greetings,
DA

Output:
(This can delivers the correct shape but Im not sure if that's what you need)
So a tensor multiplication where the second tensor has dim >2 seems to work when the last dimension of the first tensor equals the second dimension of the second tensor.
v = torch.rand(3).unsqueeze(0)
m = torch.rand(3, 1080, 1080).transpose(0, 1)
r = torch.matmul(v, m).transpose(0, 1)

print(r.shape)

>>> torch.Size([1, 1080, 1080])


Instruction: OOP structure of Pytorch

Input:
I have started learning OOP for implementing my DL models in Pytorch. Where can I find the OOP structure of Pytorch. By structure I mean the class structures (i.e. inheritance structure of classes like Sequential container). I know Module class is base for most structures.
Also I was exploring Pytorch on github and I am confused how import statements in Pytorch works. For example, the Module class is defined in torch/nn/modules/module.py but we only import torch.nn and write nn.Module to represent the Module class. Should it not be nn.modules.module.Module by the way it is stored on github?
# Example of using Sequential
model = nn.Sequential(
nn.Conv2d(1,20,5),
nn.ReLU(),
nn.Conv2d(20,64,5),
nn.ReLU()
)

Sequential class in defined in /torch/nn/modules/container.py file but we just use nn.Sequential here?
Also I didn't find __call__ method in Sequential class then how the above code calling the class as a function.

Output:
In python what you import inside a file becomes part of the file. So when you import  nn.modules.module.Module in __init__.py(which is inside nn folder), it becomes part of the nn module.
Here I will give a quick example.
Let's see we have three files file1.py, file2.py and file3.py, and we have a variable var = 10 defined in file1.py.  If we have imported this variable inside file2.py, file3.py can directly import the variable from file2.py.
file1.py
var = 10

file2.py
from file1 import var

file3.py
from file2 import var
print(var)
# Prints 10

Now let's go back to your question. As you said the Module class is defined inside nn.modules.module.Module. But this class has been imported first inside nn/modules/__init__.py and then it was imported inside  nn/__init__.py. That is why you can import Module class from nn package.
But importing will not change the type of objects you will create from the class.
from torch.nn import Module

module = Module()

print(type(module)) # torch.nn.modules.module.Module


Instruction: Compare two segmentation maps predictions

Input:
I am using consistency between two predicted segmentation maps on unlabeled data. For labeled data, I’m using nn.BCEwithLogitsLoss and Dice Loss.
I’m working on videos that’s why 5 dimensions output.
(batch_size, channels, frames, height, width)
I want to know how can we compare two predicted segmentation maps.
gmentation maps.

# gt_seg - Ground truth segmentation map. - (8, 1, 8, 112, 112)
# aug_gt_seg - Augmented ground truth segmentation map - (8, 1, 8, 112, 112)

predicted_seg_1 = model(data, targets)       # (8, 1, 8, 112, 112)
predicted_seg_2 = model(augmented_data, augmented_targets) #(8, 1, 8, 112, 112)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# Consistency loss
if consistency_loss == "l2":
      consistency_criterion = nn.MSELoss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

elif consistency_loss == "l1":
      consistency_criterion = nn.L1Loss()
      cons_loss = consistency_criterion(predicted_gt_seg_1, predicted_gt_seg_2)

total_supervised_loss = supervised_loss_1 + supervised_loss_2
total_consistency_loss = cons_loss

Is this the right way to apply consistency between two predicted segmentation maps?
I’m mainly confused due to the definition on the torch website. It’s a comparison with input x with target y. I thought it looks correct since I want both predicted segmentation maps similar. But, 2nd segmentation map is not a target. That’s why I’m confused. Because if this could be valid, then every loss function can be applied in some or another way. That doesn’t look appealing to me. If it’s the correct way to compare, can it be extended to other segmentation-based losses such as Dice Loss, IoU Loss, etc.?
One more query regarding loss computation on labeled data:
# gt_seg - Ground truth segmentation map
# aug_gt_seg - Augmented ground truth segmentation map

predicted_seg_1 = model(data, targets)
predicted_seg_2 = model(augmented_data, augmented_targets)

# define criterion
seg_criterion_1 = nn.BCEwithLogitsLoss(size_average=True)
seg_criterion_2 = nn.DiceLoss()

# labeled losses
supervised_loss_1 = seg_criterion_1(predicted_seg_1, gt_seg)
supervised_loss_2 = seg_criterion_2(predicted_seg_1, gt_seg)

# augmented labeled losses
aug_supervised_loss_1 = seg_criterion_1(predicted_seg_2, aug_gt_seg)
aug_supervised_loss_2 = seg_criterion_2(predicted_seg_2, aug_gt_seg)

total_supervised_loss = supervised_loss_1 + supervised_loss_2 + aug_supervised_loss_1 + aug_supervised_loss_2

Is the calculation of total_supervised_loss correct? Can I apply loss.backward() on this?

Output: Yes, this is a valid way to implement consistency loss. The nomenclature used by pytorch documentation lists one input as the target and the other as the prediction, but consider that L1, L2, Dice, and IOU loss are all symmetrical (that is, Loss(a,b) = Loss(b,a)). So any of these functions will accomplish a form of consistency loss with no regard for whether one input is actually a ground-truth or "target".


Instruction: difference between Dataset and TensorDataset in pyTorch

Input: what is the difference between "torch.utils.data.TensorDataset" and "torch.utils.data.Dataset" - the docs are not clear about that and I could not find any answers on google.

Output:
The Dataset class is an abstract class that is used to define new types of (customs) datasets. Instead, the TensorDataset is a ready to use class to represent your data as list of tensors.
You can define your custom dataset in the following way:
class CustomDataset(torch.utils.data.Dataset):

  def __init__(self, *args, **kwargs):
    super().__init__(*args, **kwargs)
    # Your code

    self.instances = your_data

  def __getitem__(self, idx):
    return self.instances[idx] # In case you stored your data on a list called instances

  def __len__(self):
    return len(self.instances)

If you just want to create a dataset that contains tensors for input features and labels, then use the TensorDataset directly:
dataset = TensorDataset(input_features, labels)

Note that input_features and labels must match on the length of the first dimension.


Instruction: Define nn.parameters with a for loop

Input:
I am interested in defining L weights in a custom neural network with Pytorch.
If L is known it is not a problem to define them one by one, but if L is not known I want to use a for loop to define them. My idea is to do something like this (which does not work)
class Network(nn.Module):
    def __init__(self, ):
        super(Network, self).__init__()
        self.nl = nn.ReLU()

        for i in range(L):
          namew = 'weight'+str(i)
          self.namew = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)

This should do something like this (which instead works but is limited to a specific number of weights):
class Network(nn.Module):
    def __init__(self, ):
        super(Network, self).__init__()
        self.nl = nn.ReLU()

        self.weight1 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)
        self.weight2 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)
        self.weight3 = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)

With what I tried to do, there is the problem that instead of a "dynamic" string for 'namew', Pytorch recognizes just the string 'namew'. Therefore instead of L weights, just 1 weight is defined.
Is there some way to solve this problem?

Output:
Best way to accomplish this You can accomplish this by using a ParameterDict or a ModuleDict (for nn.module layers):
class Network(nn.Module):
        def __init__(self):
            super(Network, self).__init__()
            self.nl = nn.ReLU()

            # define some nn.module layers
            self.layers = nn.ModuleDict()
            for i in range(L):
                 self.layers("layer{}".format(i) = torch.nn.Linear(i-1,i)

            # define some non-module layers
            self.weights = torch.nn.ParameterDict()
            for i in range(L):
                 self.weights["weights{}".format(i)] = torch.nn.Parameter(data=torch.Tensor(2,2), requires_grad=True)


Instruction: How to integrate a pytorch model into a dynamic optimization, for example in Pyomo or gekko

Input:
Let's say I have a pytorch-model describing the evolution of some multidimensional system based on its own state x and an external actuator u. So x_(t+1) = f(x_t, u_t) with f being the artificial neural network from pytorch.
Now i want to solve a dynamic optimization problem to find an optimal sequence of u-values to minimize an objective that depends on x. Something like this:
min sum over all timesteps phi(x_t)
s.t.: x_(t+1) = f(x_t, u_t)
Additionally I also have some upper and lower bounds on some of the variables in x.
Is there an easy way to do this using a dynamic optimization toolbox like pyomo or gekko?
I already wrote some code that transforms a feedforward neural network to a numpy-function which can then be passed as a constraint to pyomo. The problem with this approach is, that it requires significant reprogramming-effort every time the structure of the neural network changes, so quick testing becomes difficult. Also integration of recurrent neural networks gets difficult because hidden cell states would have to be added as additional variables to the optimization problem.
I think a good solution could be to do the function evaluations and gradient calculations in torch and somehow pass the results to the dynamic optimizer. I'm just not sure how to do this.
Thanks a lot for your help!

Output:
Tensorflow or Pytorch models can't be directly integrated into the GEKKO at this moment. But, I believe you can retrieve the derivatives from Tensorflow and Pytorch, which allows you to pass them to the GEKKO.
There is a GEKKO Brain module and examples in the link below. You can also find an example that uses GEKKO Feedforward neural network for dynamic optimization.
GEKKO Brain Feedforward neural network examples
MIMO MPC example with GEKKO neural network model
Recurrent Neural Network library in the GEKKO Brain module is currently being developed, which allows using all the GEKKO's dynamic optimization functions easily.
In the meantime, you can use a sequential method by wrapping the TensorFlow or PyTorch models in the available optimization solver such as scipy optimization module.
Check out the below link for a dynamic optimization example with Keras LSTM model and scipy optimize.
Keras LSTM MPC


Instruction: Torch to tensorflow

Input:
Is there a way to convert pytorch code to TensorFlow? While I am a little familiar with TensorFlow, totally new to pytorch. For example,
def get_variation_uncertainty(prediction_score_vectors: List[torch.tensor], matrix_size: Tuple) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:

    prediction_score_vectors = torch.stack(tuple(prediction_score_vectors))

    wt_var = np.var(np.sum(prediction_score_vectors[:, :, 1:].cpu().numpy(), axis=2), axis=0).reshape(matrix_size) * 100
    tc_var = np.var(np.sum(prediction_score_vectors[:, :, [1, 3]].cpu().numpy(), axis=2), axis=0).reshape( matrix_size) *100
    et_var = np.var(prediction_score_vectors[:, :, 3].cpu().numpy(), axis=0).reshape(matrix_size) * 100

    return wt_var.astype(np.uint8), tc_var.astype(np.uint8), et_var.astype(np.uint8)

how can I get the TensorFlow equivalent to the above code?

Output:
Per the comment, I would recommend using more tf functions to improve the performance and reduce the amount of GPU-CPU communication necessary. Here is an example
@tf.function
def get_variation_uncertainty_tf(prediction_score_vectors, matrix_size):
    prediction_score_vectors = tf.stack(prediction_score_vectors)
    wt_var_tmp = tf.math.square(tf.math.reduce_std(tf.reduce_sum(prediction_score_vectors[:, :, 1:], axis=2), axis=0))
    # Two steps because that was getting long
    wt_var = tf.reshape(wt_var_tmp, matrix_size) * 100

    tc_var_tmp = tf.math.square(tf.math.reduce_std(prediction_score_vectors[:, :, 1] + prediction_score_vectors[:, :, 3], axis=0))
    tc_var = tf.reshape(tc_var_tmp, matrix_size) * 100

    et_var_tmp = tf.math.square(tf.math.reduce_std(prediction_score_vectors[:, :, 3], axis=0))
    et_var = tf.reshape(et_var_tmp, matrix_size) * 100
    return tf.cast(wt_var, dtype=tf.uint8), tf.cast(tc_var, dtype=tf.uint8), tf.cast(et_var, dtype=tf.uint8)
    # if you need to return np arrays, do that instead of casting, i.e. (wt_var.numpy()).astype(np.uint8)

Tested and works here, although choosing which method will depend heavily on the shape of your data, feel free to try changing the shape to estimate which is best. In my testing the mostly numpy code is actually better unless you have huge dimensions or will be running it in batches.
https://colab.research.google.com/drive/1miOG6FV9MInanwwQxkYeSXVYirVeUh1r?usp=sharing